Tag Archives: Amazon Aurora

Consistency is the new latency: AI at the data layer

Post Syndicated from Suman Chatterjee original https://aws.amazon.com/blogs/architecture/consistency-is-the-new-latency-ai-at-the-data-layer/

As AI applications scale from reactive bots to autonomous agents, their reliability is bound to the speed and accuracy of the data layer beneath them.

The integrity crisis nobody is talking about

There’s a quiet assumption baked into most AI architectures today regarding data layer consistency, and it’s costing companies more than they realize. The assumption is that the data your AI agent reads is the current state of reality.

In a world of distributed systems, cross-region replication, and autonomous agents making millisecond decisions, this assumption breaks down.

I’ve spent extensive time working with enterprise teams building agentic AI, and a recurring failure pattern emerges.

The breakdown isn’t in the model or the prompts. It’s in how we manage replication consistency when an agent performs the reading.

The context window is the new database row

In a modern agentic Retrieval-Augmented Generation (RAG) architecture, the database is the active memory of your AI. When an agent performs a task, it retrieves data to build its context window, forming the foundation of the large language model’s (LLM) reasoning.

If that data is even slightly out of date, the agent’s entire reasoning chain is invalidated. We must shift from simply managing data availability to strictly verifying contextual integrity.

The silent poison of asynchronous lag

In traditional web applications, asynchronous replication scales global reads with minimal write impact. If a user sees a post 500ms late, nobody notices.

For an autonomous AI agent, a 500ms delay is silent poison. If an agent writes a decision to a primary node and immediately reads from a lagging replica, it treats stale data as ground truth. It then executes a logically coherent, multi-step plan based on factually incorrect inputs.

In the age of AI, a fast answer that is wrong is more expensive than a slightly slower answer that is right.

The anatomy of a stale-read failure: When memory betrays logic

Consider an autonomous Inventory Reconciliation Agent managing a flash sale:

  1. The write: The agent updates available_stock to 500 units on the primary database in us-east-1.
  2. The lag: Network congestion causes a 2-second replication lag to the ap-south-1 (Mumbai) replica.
  3. The read: A secondary agent instance in Mumbai queries the replica and retrieves the old value: 0 units.
  4. The failure: The agent triggers a “Sold Out” notification and halts the sale, despite having 500 units in the warehouse.

The agent didn’t make a reasoning error. It performed logical operations on poisoned context.

Diagram of the stale read cascade, showing how replication lag feeds outdated data into an AI agent’s context

Figure 1: The stale read cascade, showing how replication lag poisons an AI agent’s context

The hallucination debt problem

When an agent writes an incorrect conclusion back to the database, that error becomes long-term memory. Future retrievals pull this poisoned history, creating a self-reinforcing cycle of “Hallucination Debt.”

LLMs amplify this because they lack a temporal compass. They cooperatively treat retrieved database results as current facts without hesitation. The burden of verifying contextual integrity falls entirely on the architecture.

The replication trinity: Choosing your truth

Not all AI tasks have the same consistency requirements. You must match your replication model to the specific “truth requirement” of the task.

Here are three architectural patterns I’ve found most effective.

Pattern A: Precision through global consistency

When an agent manages high-stakes data (user permissions, security policies, financial records, core system instructions), the cost of a stale read is unacceptable. You need strong consistency.

For many workloads, Amazon Aurora Global Database provides the necessary foundation. While its cross-region storage replication is asynchronous by default, you can close the consistency gap by turning on Global Write Forwarding with a GLOBAL consistency level.

To verify Read-Your-Own-Writes integrity, you configure the SESSION consistency level, which makes an agent wait for its own forwarded writes to replicate back before reading.

For the strongest consistency, the GLOBAL level makes a read query wait for replication to catch up to the exact point in time when the read started.

For the next generation of globally distributed AI, Amazon Aurora DSQL addresses this need. Aurora DSQL offers native synchronous strong consistency across multiple regions, so multi-agent systems can scale globally without compromising accuracy.

Every agent, regardless of location, operates on the exact same ground truth.

Best for: Identity metadata, financial ledgers, immutable system prompts.

Why it matters: Eliminates “mid-thought” state changes that cause contradictory behavior between agent instances.

Pattern B: Global availability at scale

For global AI agents that need ultra-low latency at massive scale, Amazon DynamoDB Global Tables offer a multi-leader architecture where data replicates across regions. For replication details, refer to the DynamoDB documentation.

The key technique here is Conditional Writes. By using a ConditionExpression that checks a version timestamp or whether an attribute exists, an agent updates a record only if the data hasn’t changed since it was last retrieved.

If the condition fails, DynamoDB returns a ConditionalCheckFailedException. This is a critical signal: it tells the agent to re-read the current state and reconsider its decision, rather than blindly overwriting another agent’s work.

This pattern prevents the “Lost Update” anomaly (where two agents running in parallel overwrite each other’s reasoning) without requiring synchronous global coordination.

Best for: Conversational history, user session state, personalized agent memory.

Why it matters: Handles concurrent updates from distributed agents while maintaining a shared memory that’s resilient to race conditions.

Pattern C: High-velocity intake

Some AI agents perform real-time anomaly detection or trend analysis on massive streams of telemetry data. In these cases, you need unthrottled ingestion above all else.

A leaderless architecture like Amazon Keyspaces (for Apache Cassandra) is designed for this workload.

Keyspaces provides highly available, predictable performance by automatically replicating data across three Availability Zones.

Every write is durably committed using LOCAL_QUORUM.

To make sure your AI agent doesn’t miss a critical spike in telemetry, you enforce strong consistency by setting its read operations to LOCAL_QUORUM rather than the eventually consistent LOCAL_ONE.

This quorum overlap means the agent retrieves the latest data without slowing down the high-speed ingestion pipeline.

It transforms a noisy, high-frequency data stream into a reliable foundation for real-time AI decision-making.

Best for: Internet of Things (IoT) telemetry, real-time log analysis, high-frequency sensor data.

Why it matters: Throughput is the priority, but you still need a safety valve to confirm the agent doesn’t miss critical spike data.

Conclusion: Becoming a context architect

Our role as architects has evolved.

We can no longer treat database replication as a background infrastructure concern, something to configure once and forget. In the era of autonomous agents, the stability of the data layer is the direct prerequisite for the trustworthiness of the AI. The two are inseparable.

By matching your replication model to your agent’s reasoning requirements, you move beyond simply managing data. You become a Context Architect, someone who works to confirm that every decision your AI makes is grounded in a synchronized version of the truth.

Because in the end, an AI is only as good as the context it operates in. And context is only as good as the data it’s built on.

Get the database layer right, and everything else follows.

References:

About the author

Eclipse Dataspace Components on AWS: Cost optimization strategies

Post Syndicated from Jorge Hernández Suárez original https://aws.amazon.com/blogs/architecture/eclipse-dataspace-components-on-aws-cost-optimization-strategies/

When you deploy Eclipse Dataspace Components (EDC) connectors on AWS, one of the first challenges you face is predicting and controlling the cost of the required infrastructure. Without clear benchmarks, it is difficult to make informed decisions about workload sizing, environment configuration, and long-term investment.

Part 1 of this 3-part blog series covered the fundamentals of data space architectures and the EDC per the International Data Space Association’s (IDSA) standards. Part 2 explored production-ready architecture patterns for deploying EDC connectors on Amazon Web Services (AWS), discussing operational excellence, security, and reliability principles. This final post covers the remaining three AWS Well-Architected Framework pillars: Performance Efficiency, Cost Optimization, and Sustainability.

In this post, you will learn which AWS services drive cost in an EDC connector deployment, how to estimate monthly costs for business-critical and non-critical workloads, and how to apply optimization strategies that can reduce your spending by up to 58%.

Understanding cost drivers in data space deployments

Data spaces are secure and sovereign data environments that enable data sharing across independent organizations. With these architectures, you can collaborate with external organizations while maintaining full control over your data and compliance with data sovereignty principles. Your infrastructure costs can vary significantly. The main factors are your performance and reliability requirements, along with data volume and velocity across the network. It’s also important to distinguish between two types of infrastructure. A Dataspace Governance Authority (DSGA) centrally establishes components such as management, identity, and discovery functions. Participants host other components themselves, including the connector. This blog post focuses only on costs associated with EDC connector deployment on the participant, that is, the data provider and consumer sides.

Fictional usage assumptions

Before diving into the numbers, here are technical and operational assumptions you can use as a baseline for your own estimates.

Technical assumptions

Category Assumption Justification
Data Volume 5 GB per participant Includes 6 months of historical data, and backups
Network Traffic 20 GB/month per participant Data transfers between participants
API Calls 100,000/month per participant Catalog queries, contract negotiations, and data transfers
OAuth Token Requests 1,000/month per participant Machine-to-machine authentication for data plane operations

Table 1: Technical assumptions for EDC connector cost estimation

Operational assumptions

  • Single AWS Region: Spain (eu-south-2)
  • Operating hours: 24/7/365.
  • Growth rate: Not considered in baseline estimates.
  • Disaster recovery: Automated backups only (no cross-region replication)

Deployment architecture and scenarios

Figure 1 shows the reference architecture for deploying production-ready EDC connectors on AWS, covered in depth in Part 2 of this series.

Production-ready EDC connector deployment architecture diagram showing AWS services including Amazon ECS, Amazon Aurora, Network Load Balancer, and supporting services

Figure 1: Production-ready EDC connector deployment

This post considers two cost scenarios depending on the criticality of the workload:

  • Business-critical workloads: Designed for high availability, performance, and reliability of use cases supporting critical business functions.
  • Non-critical workloads: Designed for use cases that tolerate interruptions, testing environments, or production workloads where brief disruptions are acceptable.

Both scenarios follow the architecture patterns described in Part 2 of this post series, with the primary differences being in sizing of compute and database resources.

Cost estimation: Business-critical workloads

Note: These estimates use the assumptions above and illustrate the relative cost contribution of each service. Your actual costs may vary based on your specific usage patterns, data volumes and regional pricing. This post highlights which components represent the highest cost drivers and therefore come with the highest potential for optimization.

AWS Service Configuration Monthly Cost (USD)
Amazon Aurora PostgreSQL-Compatible Edition db.r6g.large (2 vCPU, 16 GB), 20 GB storage + 10 GB backup 276.00
Amazon Elastic Container Service (Amazon ECS) with AWS Fargate 2 vCPU, 4 GB RAM, always on 83.00
Network Load Balancer 20 GB processed data 20.00
AWS Secrets Manager 10 secrets 4.00
Amazon Cognito 1K machine-to-machine (M2M) token requests 2.25
Amazon Elastic Container Registry (Amazon ECR) 2 GB storage, 10 GB transfer 1.00
Amazon API Gateway 100K REST API calls 0.40
Amazon Simple Storage Service (Amazon S3) 5 GB Standard tier 0.10
Total 387.00

Table 2: Estimated monthly cost for business-critical EDC connector deployment

These estimates help identify where your budget goes and where optimization has the most impact. In the business-critical scenario, the main cost driver is Amazon Aurora PostgreSQL. The db.r6g.large configuration is selected for constant workloads that require reliability and speed with high memory and performance. Amazon ECS with AWS Fargate is the second largest cost contributor as it runs containers continuously to maintain environment availability. Network Load Balancer represents a third notable cost component, while the remaining services contribute only a small portion of the total cost.

Cost estimation: Non-critical workloads

If you are running development, testing, or experimentation environments, you can reduce costs by up to 58% through rightsizing and use of Amazon EC2 Spot capacity.

AWS Service Configuration Monthly Cost (USD)
Amazon Aurora PostgreSQL-Compatible db.t4g.medium (2 vCPU, 4 GB), 20 GB storage + 10 GB backup 110.00
Amazon ECS with AWS Fargate Spot 2 vCPU, 4 GB RAM, always on 26.00
Network Load Balancer 20 GB processed data 20.00
AWS Secrets Manager 10 secrets 4.00
Amazon Cognito 1K M2M token requests 2.25
Amazon ECR 2 GB storage, 10 GB transfer 1.00
Amazon API Gateway 100K REST API calls 0.40
Amazon S3 5 GB Standard tier 0.10
Total 164.00

Table 3: Estimated monthly cost for non-critical EDC connector deployment

These figures show that a non-critical configuration can cut costs significantly while maintaining the same data throughput and API capacity. Costs are reduced through the use of smaller and more flexible resources. Amazon Aurora PostgreSQL remains the main cost driver, but the smaller instance type (db.t4g.medium) reduces cost significantly. From a compute perspective, using Amazon ECS with AWS Fargate Spot capacity cuts costs by almost 70% compared to the business-critical setup. In total, this configuration reduces the monthly cost by approximately 58%, while maintaining identical assumptions for data throughput, API calls, and storage.

Key takeaways on cost optimization

This comparison shows that the primary cost contributors in both scenarios are database, compute and load balancing resources, which represent baseline infrastructure costs rather than usage-based charges. Services like Amazon S3, API Gateway, and data transfer charges contribute marginally to overall costs at these volumes. This cost structure indicates that the architecture scales efficiently with increased usage. As you onboard more use cases and increase data volume and velocity, you get more value from your existing infrastructure investment without proportional cost increases.

Well-Architected pillars: Performance efficiency, cost optimization, and sustainability

Part 2 of this series covered EDC best practices along the Operational Excellence, Security, and Reliability pillars of the AWS Well-Architected Framework. This section covers the remaining three pillars as they apply to EDC deployments.

Performance efficiency

Right-size compute resources: Match your Amazon ECS task definitions to actual workload requirements. Start with smaller configurations and scale up based on observed metrics rather than over-provisioning from the start. Amazon CloudWatch Container Insights provides the visibility needed to make informed sizing decisions.

Use the flexibility of Amazon Aurora: For workloads with variable demand patterns, consider Amazon Aurora Serverless v2 which automatically scales database capacity based on application needs. This eliminates the need to provision for peak capacity while maintaining performance during high-demand periods.

Optimize data transfer patterns: Design your data plane operations to minimize unnecessary data movement. Use Amazon S3 Transfer Acceleration for large transfers across geographic distances and consider data compression where appropriate to reduce both transfer times and costs.

Cost optimization

Reduce compute costs for fault-tolerant workloads: With AWS Fargate Spot, you can save up to 70% for workloads that can tolerate interruptions. Non-critical environments, batch processing, and development workloads are ideal candidates. Implement graceful shutdown handling to manage Spot interruptions effectively.

Lower storage costs over time: Configure Amazon S3 Lifecycle policies to automatically transition infrequently accessed data to lower-cost storage classes such as S3 Intelligent-Tiering or S3 Glacier Instant Retrieval. For EDC connector deployments, historical transfer logs and archived assets are good candidates for tiered storage.

Monitor for unexpected cost increases: Use AWS Cost Explorer and set up AWS Budgets with alerts to help detect unexpected cost increases. Tag EDC-related AWS resources consistently so you can accurately allocate costs and identify optimization opportunities.

Lock in lower rates for predictable workloads: For business-critical connectors with predictable, steady-state usage, Savings Plans for Amazon Aurora and AWS Fargate can provide significant discounts compared to On-Demand pricing.

Sustainability

Optimize resource utilization: Higher utilization of provisioned resources means less waste. Use automatic scaling policies to match capacity with demand and shut down non-production environments outside of business hours when possible.

Select efficient instance types: AWS Graviton-based instances (such as the r6g and t4g families used in our example) deliver better price-performance and energy efficiency compared to equivalent x86 instances. AWS Graviton processors offer improved performance per watt of energy use.

Minimize data movement: Each data transfer consumes energy. Design your data space integrations to avoid redundant transfers, cache frequently accessed catalog data of peers locally using the Federated Catalog, and batch operations where possible to reduce the total number of network round trips.

Summary

By rightsizing AWS infrastructure to match actual compute and database capacity needs, data space participants can achieve significant cost savings without compromising on data security and sovereignty aspects that make data spaces valuable. The comparison between business-critical and non-critical workload configurations demonstrates how AWS services like Amazon Aurora, AWS Fargate Spot, and Amazon S3 can be combined effectively to balance data sovereignty, performance, and cost efficiency.

As data spaces grow in adoption across industries and geographies, understanding these cost dynamics becomes increasingly important as you plan your network participation. The patterns and estimates in this post series offer a foundation for planning your cross-organizational data strategy and data spaces journey on AWS.

To get started, assess your workload criticality to determine whether a business-critical or non-critical configuration fits your needs. Then use the AWS Pricing Calculator to estimate costs for your specific data volumes, regions, and usage patterns. For an end-to-end reference implementation, explore the Dataspace Connector on AWS project which combines Infrastructure-as-Code with custom EDC extensions and AI tooling integration.

References

About the authors

Eclipse Dataspace Components on AWS: Architecture patterns in production

Post Syndicated from Jonas Bürkel original https://aws.amazon.com/blogs/architecture/eclipse-dataspace-components-on-aws-architecture-patterns-in-production/

Running Eclipse Dataspace Components (EDC) connectors in production on AWS requires deliberate architecture decisions around isolation, managed services, and security layering. In Part 1 of this series, we covered the fundamentals of data space architectures and EDC per the International Data Space Association’s (IDSA) standards. If you are new to EDC, we recommend starting there. We showed how connector functionality can be customized to support native integration with Amazon Web Services (AWS) Cloud services. Examples include Amazon Simple Storage Service (Amazon S3) for data storage and AWS Secrets Manager for credentials management. In this post, we dive deeper into the connector deployment architecture on AWS and present patterns and practices for production-grade deployments.

Fundamental architecture building blocks

The EDC connector consists of a control plane and a data plane that customers typically ship and deploy as containers. Depending on data integration requirements and support for specific protocols and capabilities, a custom EDC build process may need to be implemented as described in Part 1 of this series. For example, you may need OAuth 2.0 client credentials for the data plane to connect to backend systems. You store the resulting EDC container images in a container registry, such as Amazon Elastic Container Registry (Amazon ECR). Figure 1 shows an example architecture for EDC connector deployments on AWS following best practices for production use.

Architecture diagram showing production-ready EDC connector deployment on AWS with Amazon ECS, Aurora, S3, and API Gateway components

Figure 1: Production-ready EDC connector deployment on AWS

You can split the architecture into four sub-components:

  • Amazon Elastic Container Service (Amazon ECS) and AWS Fargate provide serverless container orchestration. This allows for scalable EDC deployment without managing any of the underlying infrastructure.
  • EDC requires persistence to store secrets and relational control plane data, and a means of vending OAuth 2.0 client credentials. AWS Secrets Manager, Amazon Aurora and Amazon Cognito can provide these capabilities as managed services.
  • Amazon S3 provides durable data storage for handling both inbound and outbound data that is shared and received through the data space.
  • Finally, Amazon API Gateway and Network Load Balancer provide secure, private network connectivity to EDC APIs in an isolated Amazon Virtual Private Cloud (Amazon VPC) using VPC links.

With this approach, all cloud resources belonging to a single EDC connector instance form an isolated architecture cell. You access this cell through the S3 bucket to move in data that is to be shared as part of an EDC asset, or to retrieve data received from a third party as part of an EDC data transfer. Secondly, the API Gateway can be configured to expose selected EDC API resources from its management API, data plane API and Dataspace Protocol (DSP) API. You can protect both means of interacting with the EDC architecture cell using AWS Identity and Access Management (AWS IAM) and the AWS Signature Version 4 (SigV4) protocol.

Larger enterprises participating in data spaces may decide to operate multiple EDCs depending on their requirements on failure isolation, data governance, and separation of shared and received data. A common pattern is to deploy separate connector instances per use case. Infrastructure-as-code such as AWS Cloud Development Kit (CDK) allows for automated, templatized deployment and management of EDC connectors over time while keeping operational effort at bay. Using the Dataspace Connector on AWS reference implementation, a full connector cell deploys from a single CDK command, ready to negotiate contracts and transfer data. Amazon API Gateway also comes with Model Context Protocol (MCP) proxy support. This allows EDC APIs to be consumed by authorized AI agents and MCP clients for autonomous data collection and sharing. Besides integration with agentic systems, customers often follow a workflow-based approach for connecting EDCs with their cloud-based data environments. They interact with both APIs and the peripheral S3 bucket to securely expose and retrieve external information.

Real-world validation of these architecture patterns can be seen in production deployments like the Prometheus-X Data Space Connector, for education sector use cases. This implementation uses the same core architecture we recommend: Amazon ECS with AWS Fargate for container orchestration, S3 for data storage, and event-driven processing with AWS Lambda and Amazon EventBridge. This demonstrates how these patterns work effectively in production environments across different industry sectors.

Key principles for production-readiness

We discuss some of the architecture principles that inform the best practices diagram highlighted in Figure 1 along three of the AWS Well-Architected Framework’s pillars.

Operational Excellence

Infrastructure as Code for Consistency: Define all infrastructure declaratively to support repeatable, version-controlled, and testable deployments. Automated validation, for example using CDK Nag, helps catch misconfigurations and security issues before deployment, shifting security left in the development lifecycle. The code itself serves as living documentation of the architecture.

Observability as a First-Class Concern: Treat monitoring and logging as core infrastructure components. Amazon CloudWatch Container Insights, Amazon CloudWatch Logs, and EDC’s structured health check endpoints provide visibility into system behavior, supporting proactive issue detection and faster troubleshooting. EDC APIs for health checks can be similarly exposed with restricted access through API Gateway and IAM.

Managed Services Over Self-Managed Infrastructure: Use AWS managed services (Aurora, Secrets Manager, Fargate, Cognito) instead of deploying and maintaining compatible self-managed solutions. This shifts undifferentiated heavy lifting to AWS and reduces operational burden. You gain high availability, built-in security best practices, compliance certifications, and automatic updates.

Security

Defense in Depth: Implement security through multiple independent layers rather than relying on a single control. Network isolation (VPC private subnets), security group segmentation (restricting traffic between components), IAM least privilege (scoped permissions per service), and encryption (at rest and in transit) each provide independent controls. These layers work together so that if one layer is bypassed, others continue to provide protection.

Principle of Least Privilege: Every component receives only the minimum permissions required for its specific function. Scope IAM roles to individual services (control plane, data plane) and restrict security groups to necessary ports and sources. The internal-only Network Load Balancer fronted by API Gateway prevents unintended public exposure of EDC APIs and data. This may also support security review and approval of EDC as open-source software, since APIs can be allowlisted and validated individually.

Encryption Everywhere: Encrypt data by default at every stage: at rest (Aurora, S3, Secrets Manager), in transit (TLS enforcement, HTTPS-only egress), and during processing (encrypted environment variables). This provides comprehensive data protection regardless of where information resides in the system.

Reliability

Fail Fast, Recover Automatically: Systems detect failures quickly and recover without manual intervention. ECS circuit breakers can automatically roll back failed deployments, automated health checks remove unhealthy targets, and point-in-time recovery supports rapid database restoration. This minimizes mean time to recovery (MTTR) and reduces the scope of failures, even within a single EDC architecture cell.

Design for Regional Resilience: Cross-zone load balancing distributes traffic across multiple Availability Zones (AZs), Aurora automatically replicates data across AZs, and Fargate tasks can be scheduled in any AZ. The highlighted architecture can tolerate Availability Zone failures in an AWS Region without service disruption. For more information about Availability Zones and Regions, see AWS Global Infrastructure.

Decoupled Components with Clear Boundaries: Deploy the control plane and data plane as independent services with distinct responsibilities, security contexts, and scaling characteristics. This separation enables independent updates, targeted scaling, and failure isolation between coordination logic and data transfer operations.

The remaining three Well-Architected Framework pillars of Performance Efficiency, Cost Optimization, and Sustainability are covered in the third post of this series where we discuss cost optimization strategies for running EDC connectors on AWS.

Conclusion

With the growing popularity of data spaces and EDC as a data space connector, it is important to distinguish between a setup suitable for testing and experimentation and one that is ready for production. Production environments require that business-critical processes depend on timely, successful transmission of confidential information between participants. The architecture defined in this post combines EDC deployment best practices from the community with AWS recommendations to achieve fault tolerance, scalability, and security while keeping operational complexity at a minimum.

In part 3, you will learn about cost optimization strategies to run your production-ready connector efficiently and maximize the value it returns by supporting business use cases for data sharing along your supply network. In the meantime, explore the Dataspace Connector on AWS project and see how the patterns and best practices covered in this post come together in an end-to-end reference implementation.

References

About the authors

Eclipse Dataspace Components on AWS: Data sharing fundamentals

Post Syndicated from Alejandro Esquivias Cañadas original https://aws.amazon.com/blogs/architecture/eclipse-dataspace-components-on-aws-data-sharing-fundamentals/

This three-part blog series guides you through implementing Eclipse Dataspace Components (EDC) on AWS, from foundational concept to production deployment. Part 1 establishes the theoretical foundation with IDSA standards, the Dataspace Protocol (DSP), and core EDC architecture. Part 2 provides production-ready AWS deployment patterns using services like Amazon Elastic Container Service (Amazon ECS), Amazon Aurora, and Amazon API Gateway. Part 3 completes the journey with cost optimization strategies and practical guidance for running efficient, scalable data space infrastructure on AWS.

The International Data Spaces Association (IDSA) is a non-profit organization focused on establishing and promoting standards for data spaces. A data space is defined as a “set of technical services that facilitate interoperable dataset sharing between distinct entities”. At a technical level, a data space has participants: data consumers and data providers. Centrally, there is a Dataspace Governance Authority (DSGA). The DSGA manages the data space, enforces rules and policies and issues membership credentials to participants.

IDSA rules and specifications are implemented in the Dataspace Protocol (DSP). The DSP has become an Eclipse Specification project and can be found in the Dataspace Protocol GitHub repository. Standardization is under development and is currently in the approval phase as ISO/IEC DIS 20151. The Eclipse Dataspace Components (EDC) provide the technical components to implement data spaces according to IDSA requirements.

These components include the federated catalog (FC), the connector, and the identity hub. The FC contains an aggregated repository of catalogs gathered from multiple participants in the data space. These are obtained by periodically crawling participants’ data assets and storing them in a local cache, to eliminate the need to query each participant individually on demand. The connector is the software that enables data to be shared between participants, and identity hub manages a participant’s credentials in the data space. Amazon Web Services (AWS) provides a comprehensive cloud infrastructure that supports the deployment and operation of data space components like the EDC connector. AWS offers scalable compute, storage, and networking services that help you build secure, compliant, and interoperable data spaces aligned with IDSA standards and the ISO/IEC DIS 20151 specifications.

To verify identities in a data space in a decentralized manner, the Decentralized Claims Protocol (DCP) is used. DCP represents an overlay on top of DSP to establish trust between network participants. When an issuer needs to prove their identity to a verifier, they first generate a Decentralized Identifier (DID), which contains information about their identity. The issuer stores their Verifiable Credential (VC) in their identity hub. A verifiable credential is similar to a certificate: while a certificate authority validates a public key, the credential issuer validates the DID with specific attributes. The verifier looks up the DID of the issuer and with the information obtained from the DID document verifies the VC.

High-level diagram showing how DCP establishes trust between an issuer and verifier using DIDs and Verifiable Credentials

Figure 1: High-level diagram of the Decentralized Claims Protocol (DCP)

Simplified overview showing how a verifier resolves a DID document to verify credentials

Figure 2: Simplified overview of DID document processing

The final element of the EDC are the different types of policies, including membership, access, contract and usage policies.

The role of the connector

The EDC connector is divided into two parts: the control plane and the data plane. The control plane handles contract negotiation and sends messages to the data plane to initiate a data transfer. The data plane is responsible for transferring data from a provider’s to a consumer’s EDC connector across distinct legal entities.

Diagram showing the EDC connector architecture with separate control plane and data plane

Figure 3: Overview of the connector control and data plane

Structure of the EDC connector open-source project

The Connector repository as part of the EDC project defines how the connector control and data planes are implemented. We provide an overview of how the repository is structured:

  • /spi – The Service Provider Interface (SPI) is the architectural foundation that defines how modules communicate within the EDC connector. It contains foundational interfaces and contracts that every component must implement, establishing standardized integration patterns. This layer also acts as a blueprint for developers to extend EDC connector functionality while maintaining clean separation of concerns and ensuring compatibility between core and custom components.
  • /core – The Core module represents the core SPI implementation. It houses the actual working code for the connector’s essential operations, including default implementations of key services and business logic for data sharing.
  • /extensions – The Extensions layer showcases the connector’s modular plugin architecture through real-world integrations, including connections to major cloud providers like AWS. These extensions serve both as reference implementations and ready-to-use components for common enterprise integration scenarios.

SPI defines the contracts, Core implements the basics, and Extensions add specialized functionality. These three layers work together to create a flexible, extensible data space connector system. The “vanilla” Eclipse EDC connector does not bundle the AWS extensions by default. To add AWS service-specific functionality, such as integration with Amazon Simple Storage Service (Amazon S3), AWS Secrets Manager, or Amazon DynamoDB, you need to include the respective AWS extensions into a custom EDC control and data plane build.

High-level EDC customization guide

Customizing the EDC connector for native AWS service integration creates a purpose-built solution that can use AWS managed services for storage, security, and scalability. The following steps help your connector natively integrate with services like Amazon S3 for data storage (used for example as Asset Administration Shell “Submodel Server”) and AWS Secrets Manager for credentials management, rather than relying on operations of self-managed components.

EDC uses a modular, plugin-based architecture built on Gradle. The vanilla connector ships only core functionality. To integrate with specific cloud services or persistence backends, you assemble a custom build that combines EDC’s core modules with the extensions you need. The customization revolves around three concepts: a version catalog, launcher modules, and a project settings file.

Step 1: Version catalog

The version catalog (gradle/libs.versions.toml) is the centralized registry of all dependencies and their versions. Here you declare which EDC modules, cloud provider extensions, and third-party libraries your connector needs to use. EDC publishes its AWS extensions under the org.eclipse.edc.aws Maven group. You reference these alongside the core EDC artifacts (org.eclipse.edc) at compatible version numbers. This single file ensures all modules in your project share consistent dependency versions.

Step 2: Launcher modules

The launcher modules are the actual deployable units: one for the control plane and one for the data plane. Each launcher is a small Gradle subproject whose build.gradle.kts lists the extensions to bundle at runtime. A typical control plane launcher might include the core control plane module, a metadata persistence extension (such as PostgreSQL or DynamoDB), a vault extension (such as HashiCorp Vault or AWS Secrets Manager), and any provisioning extensions for your target storage system. The data plane launcher follows the same pattern with data plane-specific modules. Both launchers typically use Gradle plugins to produce a single executable JAR. You only include what you actually use, keeping the connector lightweight and tailored to your unique requirements.

Step 3: Project settings

The project settings file (settings.gradle.kts) registers all modules with Gradle: your launcher subprojects and any custom extensions you may develop locally. If you write your own extension, for example, a DynamoDB-backed asset store, you place it in an extensions/ directory and register it here so your launchers can reference it as a project dependency.

The resulting project structure typically looks like this:

my-connector/
├── control-plane/
│   └── build.gradle.kts       # Control plane launcher
├── data-plane/
│   └── build.gradle.kts       # Data plane launcher
├── extensions/                # Optional: Custom extension code
├── gradle/libs.versions.toml  # All dependency versions
├── build.gradle.kts           # Root build config
└── settings.gradle.kts        # Subproject registrations

An open-source reference implementation for EDC customization on AWS is provided as part of the Dataspace Connector on AWS project. In this post, you learned about the fundamentals behind secure, cross-organizational data sharing using emerging data space architectures. We covered the two protocols included in ISO/IEC DIS 20151 data space standardization, DSP and DCP, and discussed the EDC connector, the main software required for data space participants, and how it can be customized.

In part 2, we explore production-ready deployment patterns for EDC connectors on AWS, examining architecture best practices that support your data space infrastructure’s security, reliability, and operational efficiency. To get started, explore the Dataspace Connector on AWS project and see how its Gradle build settings are configured to support Amazon DynamoDB for serverless, inexpensive metadata persistence.

References

About the authors

AWS Weekly Roundup: AWS Builder Center at 1 year, Network Scanning in Security Hub, Loom for AWS, and more (July 13, 2026)

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-aws-builder-center-at-one-year-network-scanning-in-security-hub-loom-for-aws-and-more-july-13-2026/

AWS Builder Center turned one year old last week. Launched on July 9, 2025, the platform has grown from a community hub with Wishlist voting, community profiles, and a toolbox into a full ecosystem with sandbox environments, workshops, Spaces, and a Builders’ Library. To mark the anniversary, Rick Suttles published a full feature timeline covering everything shipped over the past year: AWS Capabilities by Region (1,500+ services across 37 Regions), Spaces for community-created groups, workshops with category and complexity filters, badges and streaks, article series, view counts, saved items, student status, availability notifications, sign-in with GitHub and Amazon, and sandbox environments.

Jeff Barr published a retrospective summarizing Builder Center’s first year. Since launch, 5,548 authors have published 6,448 articles with more than 10.4 million page views combined. Builders have earned 99,226 badges since the badge system launched in March 2026. Community members have submitted 565 wishes, 10 of which have shipped with another 20 on the near-term roadmap.

The top community article Building an AWS Study Buddy with MCP + Strands Agents SDK by Dineshraj Dhanapathy reached 50,000+ views. Chris Miller’s Migrating an EOL Linux Server to AWS in 8 Hours with Kiro followed at 45,000+, and Yash Aggarwal’s AIdeas: NeuroVoice – Multimodal AI for Early Screening of Neurological Diseases article reached 38,000+.

The week’s headline addition is Sandbox Environments by Rick Suttles. Sandboxes give you a free, pre-provisioned AWS account to complete a workshop exercise. Each environment is active for 8 hours, after which the account and all its resources are automatically de-provisioned. You can have one active sandbox at a time and request one per week. No personal AWS account, credit card, or manual cleanup required.

Last week’s launches
Here’s what else happened this week.

  • AWS Security Hub introduces Network Scanning – Security Hub introduced Network Scanning, a capability that identifies resources in your environment that are reachable from the public internet. Network Scanning probes your resources from the internet to detect actual reachability, complementing the existing network reachability findings in Security Hub that identify configurations that could make a resource reachable. It discovers public IP addresses, virtual machines, and load balancers across your AWS and Azure environments, identifies reachable ports, and determines what services are running behind them. Each reachable port generates a Security Hub finding with evidence of the port and service discovered. Security Hub Exposures then automatically correlates these findings with other findings and resource configurations to determine broader risk. Existing customers can enable Network Scanning in individual accounts and Regions, or across an organization through a configuration policy. For new customers, Network Scanning is on by default. It is included with Security Hub Essentials at no additional cost.
  • Security Hub also extends unified security management to Microsoft Azure – Security Hub now monitors Microsoft Azure resources, providing unified posture management, vulnerability management, and security response across both clouds. It automatically discovers Azure VMs, container images, Function Apps, and identities, and evaluates them for misconfigurations, internet exposure, and software vulnerabilities. AWS and Azure findings appear in the same prioritized view with the same formats and automation workflows.
  • Amazon SageMaker Studio integrates with Hugging Face for one-click model deployment and customization – You can now go from discovering a model on Hugging Face to working with it in SageMaker Studio in a single click. Select any supported model on Hugging Face and choose “Customize on SageMaker AI” or “Deploy on SageMaker AI” to land directly on the corresponding workflow page with the model pre-loaded. New customers receive a Studio environment created in seconds with pre-configured permissions for serverless model customization (including fine-tuning with custom reward functions for reinforcement learning), model evaluation, and deployment to SageMaker or Bedrock endpoints. Verified customers receive default GPU access to G5, G6, and G4dn instances without requesting quota increases, and quota utilization is visible directly inside the Studio environment.
  • Amazon EKS Auto Mode and Amazon ECS Managed Instances reduce GPU management fees by up to 60% – Beginning July 1, 2026, EKS Auto Mode and ECS Managed Instances reduce management fees for accelerated instance types: G-series fees are down 35%, and P-series and AWS Trainium fees are down 60%. The reductions apply automatically to existing clusters and require no action from customers. Both services include capabilities built for accelerated workloads. EKS Auto Mode provides automatic parallel image pulling on GPU instances with local NVMe storage and accelerator-aware node repair. ECS Managed Instances provides GPU metrics through Amazon CloudWatch Container Insights and automatic health monitoring for GPU hardware failures.
  • Amazon Aurora DSQL change data capture (CDC) is now generally available – Aurora DSQL CDC streams the results of insert, update, and delete operations as change events to Amazon Kinesis Data Streams. You can use it to synchronize data across microservices, trigger Lambda functions, or deliver changes to S3, Redshift, and OpenSearch Service through Amazon Data Firehose. CDC streaming is designed to have zero impact on database workload performance and requires no infrastructure to manage.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts you may find useful:

  • Building secure AI agents at scale: Introducing Loom for AWS – Loom is an open-source enterprise platform for building agents with AWS Strands Agents and deploying them on Amazon Bedrock AgentCore Runtime. It provides a unified management UI and backend API with identity provider integration, scope-based authorization, multi-persona navigation, and full lifecycle management for agents, memory, MCP servers, and agent-to-agent integrations. Loom enforces automated resource tagging for cost attribution, implements RBAC and ABAC for multi-tenant security, uses paved-path blueprints for agent deployments, manages identity propagation through delegated actor chains, integrates with AWS Agent Registry for discovery and governance, and supports human-in-the-loop review before sensitive actions. The project is available in AWS Labs on GitHub.
  • Introducing Claude apps gateway for AWS – The Claude apps gateway is a self-hosted control plane that gives organizations centralized control over access, cost, and policy for Claude Code and Claude Desktop. It connects to any OIDC-compliant identity provider, enforces managed settings on every request, routes inference to Amazon Bedrock or Claude Platform on AWS, and supports per-user and per-group spend caps. The gateway runs as a stateless container in your private network, backed by a PostgreSQL database for short-lived sign-in state. No long-lived secrets are stored on developer machines. Deploy it through Amazon Bedrock to keep data within the AWS security boundary, or through Claude Platform on AWS for the native Claude platform experience.
  • Introducing OAuth support for AWS MCP Server – You can now connect agents to the AWS MCP Server using browser-based OAuth with the same credentials you use for the AWS Console or CLI. The new sign-in path supports IAM federation, AWS IAM Identity Center, and root or IAM users. AWS Sign-In issues short-lived access tokens and refresh tokens, with automatic token management so developers stay authenticated across restarts. For headless use cases, a non-interactive flow lets applications with existing AWS credentials obtain OAuth access tokens through the create-oauth2-token-with-iam API. New governance controls include OAuth-specific IAM condition keys, token introspection and revocation, dynamic client registration, and CloudTrail audit elements.

For a full list of AWS blog posts, be sure to keep an eye on the AWS Blogs page.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

Visit the AWS Builder Center to meet other builders, contribute solutions, and find resources that help you keep building.

Wishing everyone a restful and enjoyable summer. Whether you’re building, learning, or recharging, I hope you find time for all three. I’ll be heading to Scandinavia for a few weeks to trade the heat for some cooler weather and longer evenings. Come back next week for more news!

— Esra

AWS Weekly Roundup: Claude Opus 4.8 on AWS, Aurora MySQL with Kiro Powers, and more (June 1, 2026)

Post Syndicated from Micah Walter original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-claude-opus-4-8-on-aws-aurora-mysql-with-kiro-powers-and-more-june-1-2026/

In my last Week in Review post, I shared what I’d been hearing from customers in the AI-Driven Development Lifecycle (AI-DLC) workshops I’ve been delivering. Last week I was back at it, this time in Denver for a two-day AI-DLC workshop, where I helped facilitate 17 teams to deliver nearly 20 separate use cases in just two days. The pace of acceleration that AI-DLC unlocks—especially when paired with tools like Claude Code on Amazon Bedrock—is fundamentally changing how businesses operate. Traditional roles within software development teams are collapsing into smaller, AI-augmented squads, and the paradigm shift is beginning to take place right in front of us. To learn more about how to utilize various AI tools, visit the GitHub repository of AI-DLC workflow.

This shift is also reshaping how AWS account teams (solutions architects, customer solutions managers, and technical account managers) collaborate with customers. It’s becoming less about handing off advisory design documents and more about building alongside them in real time. It’s a genuinely exciting moment to be in the middle of the change, and this week’s headline launch — Anthropic’s most capable model yet, now on AWS — is going to push that pace even further.

Now, let’s get into this week’s AWS news…

Headlines
Claude Opus 4.8 on AWS — Anthropic’s most capable generally available model is now accessible through both Amazon Bedrock and the Claude Platform on AWS. Opus 4.8 is built for agentic coding, knowledge work, and extended autonomous task execution — it sustains longer autonomous sessions with deeper reasoning, recovers from errors, and synthesizes information across lengthy documents. For coding workloads, it reads codebases like an engineer, plans before it edits, and holds context across long sessions. On Amazon Bedrock, you get AWS-managed features like Guardrails, Knowledge Bases, and data residency; on the Claude Platform on AWS, you get Anthropic’s native APIs unified with AWS billing. To learn more, visit the deep-dive blog post.

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Introducing the next generation of AWS Resilience Hub — A reimagined Resilience Hub gives SREs and developers a unified framework to define resilience standards, evaluate applications against them, and demonstrate compliance across an entire portfolio. It introduces modular resilience policies (covering service-level objectives (SLOs), multi-AZ/Region DR, and data recovery), business-oriented application modeling, generative AI-powered assessments aligned with the Well-Architected and Resilience Analysis Frameworks, and automatic dependency discovery via DNS query log analysis. Integration with AWS Organizations enables organization-wide resilience management from a single delegated administrator account.
  • Introducing the next generation of Amazon OpenSearch Serverless for building agentic AI applications — Amazon OpenSearch Serverless is now a fully managed search and vector engine purpose-built for agentic AI applications. It scales from zero to thousands of requests per second—roughly 20x faster than the prior generation—delivers up to 60% cost savings versus peak-provisioned clusters, and adds GPU acceleration plus new SEARCH and VECTORSEARCH collection types. Native integrations with Vercel, Kiro, Claude Code, and Cursor through OpenSearch Agent Skills make it straightforward to plug into your agent stack.
  • New assessment capabilities in AWS Transform — AWS Transform expands with new tools to help you build migration business cases and evaluate TCO before moving workloads to AWS. You can ingest data from RVTools exports, CMDB data, the AWS Transform discovery tool, and third-party discovery tools, then run what-if scenarios across region, utilization, and service mapping for EC2, FSx, S3, SQL Server on EC2, and virtual desktops. The release also adds Agentic Readiness Analysis (ARA) and Modernization Analysis (MODA), which scan code repositories in 5 to 30 minutes per repo to surface severity-tagged findings with file-level evidence and AWS-mapped remediation guidance.
  • Amazon Aurora MySQL with Kiro Powers — Aurora MySQL now integrates with Kiro Powers, drawing from a curated repository of pre-packaged MCP servers, steering files, and hooks validated by Kiro partners. Developers can execute both data plane tasks (queries, schema management) and control plane tasks (cluster management) in natural language, with dynamic guidance for Aurora MySQL Serverless scaling, RDS-to-Aurora migration, and replication setup. The companion Database Blog post explains how the agent produces the API calls, SQL, and configuration for you to review and run — available via one-click install from the Kiro IDE or webpage.
  • Amazon WorkSpaces Applications now supports Windows Desktop OS — You can now bring your own Windows Desktop licenses to Amazon WorkSpaces Applications and stream full Windows desktops and applications from AWS-hosted dedicated hardware. BYOL eliminates OS fees (you pay only for compute and streaming infrastructure), supports eligible Microsoft 365 Apps for enterprise, and gives users a matching experience between local and remote environments — same workflows, shortcuts, and navigation in both.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts and resources that you might find interesting:

For a full list of AWS blog posts, be sure to keep an eye on the AWS Blogs page.

Learn more about AWS, browse and join upcoming AWS-led in-person and virtual events, startup events, and developer-focused events as well as AWS Summits and AWS Community Days. Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development.

That’s all for this week. Check back next Monday for another Weekly Roundup!

-Micah

AWS Weekly Roundup: Anthropic & Meta partnership, AWS Lambda S3 Files, Amazon Bedrock AgentCore CLI, and more (April 27, 2026)

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-anthropic-meta-partnership-aws-lambda-s3-files-amazon-bedrock-agentcore-cli-and-more-april-27-2026/

Late March took me to Seattle for the Specialist Tech Conference, one of the most energizing gatherings of AWS specialists from around the world. It was an incredible opportunity to connect with peers, exchange experiences, and go deep on the latest advancements in Generative AI and Amazon Bedrock — and a powerful reminder of something I truly believe in: when specialists come together to challenge each other, explore edge cases, and co-create solutions, the impact goes far beyond the meeting room. In a fast-moving space like AI, having a strong internal community isn’t a nice-to-have — it’s a competitive advantage.

Now, let’s get into this week’s AWS news…

Headlines

Anthropic partnership: Claude on AWS Trainium and Graviton, and Claude Cowork in Amazon Bedrock – This week, AWS and Anthropic deepened their product collaboration in meaningful ways for builders. Anthropic is now training its most advanced foundation models on AWS Trainium and Graviton infrastructure, co-engineering directly at the silicon level with Annapurna Labs to maximize computational efficiency from the hardware up through the full stack.

Claude Cowork is now available in Amazon Bedrock — Claude Cowork brings Anthropic’s collaborative AI capabilities directly to enterprise builders within the AWS ecosystem, enabling teams to work alongside Claude as a true collaborator, not just a tool. You can now deploy Claude Cowork within your existing Amazon Bedrock environment, keeping your data secure within AWS while leveraging the full power of Claude for team-based AI workflows.

Claude Platform on AWS (Coming soon) — A unified developer experience to build, deploy, and scale Claude-powered applications without leaving AWS. If you’re building with Generative AI on AWS, this is a significant step forward in what you’ll be able to do with Claude directly through Amazon Bedrock.

Meta signs agreement with AWS to power agentic AI on Amazon’s Graviton chips — Meta has signed an agreement to deploy AWS Graviton processors at scale, starting with tens of millions of Graviton cores to power CPU-intensive agentic AI workloads — including real-time reasoning, code generation, search, and multi-step task orchestration.

Last week’s launches

Here are some launches and updates from this past week that caught my attention:

  • AWS Lambda functions can now mount Amazon S3 buckets as file systems with S3 Files — You can now mount Amazon S3 buckets as file systems in AWS Lambda using S3 Files, enabling your functions to perform standard file operations without downloading data for processing. Built on Amazon EFS, S3 Files provides the simplicity of a file system with the scalability, durability, and cost-effectiveness of S3 — and multiple Lambda functions can connect to the same file system simultaneously, sharing data through a common workspace. This is particularly valuable for AI and machine learning workloads where agents need to persist memory and share state across pipeline steps.
  • Amazon EKS Hybrid Nodes gateway for hybrid Kubernetes networking — Amazon Elastic Kubernetes Service now offers the Amazon EKS Hybrid Nodes gateway, which automates networking between your EKS cluster VPC and Kubernetes Pods running on EKS Hybrid Nodes. You can now eliminate the need to make on-premises pod networks routable or coordinate network infrastructure changes, greatly simplifying hybrid Kubernetes environments. The gateway automatically enables pod-to-pod traffic across cloud and on-premises environments, control plane-to-webhook communication, and connectivity for AWS services like Application Load Balancers, and is available at no additional charge.
  • Amazon Aurora Serverless: Up to 30% better performance, smarter scaling, and still scales to zero — Amazon Aurora Serverless just got faster and smarter, with up to 30% better performance than the previous version and an enhanced scaling algorithm designed to handle workloads where multiple tasks compete for resources — like busy APIs and agentic AI applications with bursts of activity and long idle windows. You can now run even more demanding workloads serverlessly, paying only for what you use, and automatically scaling to zero when not in use. All improvements are available in platform version 4 at no additional cost.
  • Amazon Bedrock AgentCore adds new features to help developers build agents faster — Amazon Bedrock AgentCore introduces a managed harness (preview), the AgentCore CLI, and AgentCore skills for coding assistants, helping developers go from idea to working agent prototype faster. The managed harness lets you define an agent by specifying a model, system prompt, and tools and run it immediately with no orchestration code required — and when you’re ready for full control, you can export the harness orchestration as Strands-based code. The AgentCore CLI deploys your agents with the governance and auditability of infrastructure-as-code (AWS CDK today, Terraform coming soon), and is available in 14 AWS Regions at no additional charge.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news

Here are some additional posts and resources that you might find interesting:

  • Introducing granular cost attribution for Amazon Bedrock — This post walks through how Amazon Bedrock’s granular cost attribution works and covers practical example cost tracking scenarios. You can now tag and track Bedrock usage costs at a finer level of detail — useful for organizations running multiple teams or projects on Bedrock who need precise cost visibility and chargeback capabilities.
  • Automating Incident Investigation with AWS DevOps Agent and Salesforce MCP Server — This post (co-written with Salesforce) shows how AWS DevOps Agent, integrated with the Salesforce MCP Server, automates the full lifecycle of infrastructure incident investigation — from identifying issues and diagnosing root causes to notifying customers through Salesforce Service Cloud. It’s a compelling real-world example of how AI agents and MCP-based tool connectivity are reshaping DevOps workflows in production, dramatically reducing mean time to resolution.
  • Microcredentials from AWS are now free — Here’s why that matters — You can now access AWS microcredentials at no cost through AWS Skill Builder in all countries where the platform is offered. Unlike traditional multiple-choice certifications, microcredentials are hands-on assessments that place builders in simulated business scenarios where they configure, troubleshoot, and optimize directly in a live AWS environment — the same way they would on the job. A great opportunity to validate real cloud skills without a cost barrier.
  • Amazon SageMaker AI now supports optimized generative AI inference recommendations — You can now use Amazon SageMaker AI to automatically identify optimized deployment configurations for your generative AI models, including instance type, container, and inference parameters. This new capability takes the guesswork out of tuning inference infrastructure, helping you reduce costs and improve latency for your AI applications in production.

Upcoming AWS events

Check your calendar and sign up for upcoming AWS events:

  • What’s Next with AWS — Tune in on April 28 for What’s Next with AWS, a virtual event featuring the latest announcements and product updates directly from AWS teams. A great opportunity to get up to speed on what’s new before diving into the week’s launches.
  • AWS Summits — AWS Summits are free in-person events where you can explore the latest in cloud and AI innovation, learn best practices, and network with builders and experts. Coming up in May: Singapore (May 6), Tel Aviv (May 6), Warsaw (May 6), Stockholm (May 7), Sydney (May 13–14), Hamburg (May 20), Seoul (May 20), Amsterdam (May 27), Bangkok (May 28), Milan (May 28), and Mumbai (May 28). And in June, join us in Los Angeles (June 10). Check the full schedule and register at the link above.
  • AWS Community Days — Community-led conferences where content is planned, sourced, and delivered by community leaders, featuring technical discussions, workshops, and hands-on labs. Upcoming events include Athens, Greece (April 28), Vancouver, Canada (May 1), İstanbul, Türkiye (May 9), and Panama City, Panama (May 23). If you’re in Latin America, mark your calendar for the AWS Community Day Belo Horizonte (August 22) — registration is open at awscommunityday.com.br.

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— Daniel Abib

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Real-time analytics: Oldcastle integrates Infor with Amazon Aurora and Amazon Quick Sight

Post Syndicated from Avdhesh Paliwal original https://aws.amazon.com/blogs/architecture/real-time-analytics-oldcastle-integrates-infor-with-amazon-aurora-and-amazon-quick-sight/

This post is cowritten with Avdhesh Paliwal from Oldcastle.

Oldcastle APG is one of the largest suppliers of construction materials in North America, including asphalt and concrete. The company also provides construction and paving services across more than 150 facilities. As the company migrated from on-premises systems to Infor Cloud ERP hosted on Amazon Web Services (AWS), they faced a critical challenge: maintaining the real-time operational reporting capabilities that hundreds of users across customer service, finance, logistics, and manufacturing depended on daily.

This post explores how Oldcastle used AWS services to transform their analytics and AI capabilities by integrating Infor ERP with Amazon Aurora and Amazon Quick Sight. We discuss how they overcame the limitations of traditional cloud ERP reporting to deploy real-time dashboards and build a scalable analytics system. This practical, enterprise-grade approach offers a blueprint that organizations can adapt when extending ERP capabilities with cloud-native analytics and AI.

Challenges with cloud ERP reporting

The primary challenge that we faced was finding a solution that could accomplish the following:

  • Maintain real-time data access – Our on-premises environment supported hundreds of complex real-time reports, but Infor ERP Cloud’s configuration-based reporting covered minimal reports of our operational needs.
  • Support complex reporting requirements – Users needed multi-dimensional analysis across customer service, finance, logistics, and manufacturing functions.
  • Provide seamless user experience – Business users demanded integrated reporting within the ERP network without switching between multiple systems.
  • Enable advanced analytics –We needed capabilities for demand forecasting, machine learning (ML) capabilities, and intelligent search across real-time data.
  • Scale efficiently – The solution needed to support over 100 concurrent users and process millions of transactions while maintaining performance.
  • Expose Data using API – The solution needed to expose data through APIs, allowing both external and internal applications to access and consume the data securely and efficiently.

Our existing batch reporting process created significant operational challenges across our organization. We had to wait for batch reports, which consumed valuable time and led to delays in critical decision-making across many of our teams. This lag prevented us from capitalizing on real-time business insights and responding quickly to operational issues or economic changes. Without immediate data visibility, our managers couldn’t make timely, data-driven decisions, resulting in missed opportunities for improvement and competitive advantage. Our IT team also struggled with constant report requests but lacked a scalable system to deliver them efficiently, further compounding the productivity loss across our organization.

Solution overview

AWS Solutions Architects worked closely with our application team to build a comprehensive analytics and AI solution to address these challenges. The architecture uses Infor Data Fabric Stream Pipelines to deliver real-time data to AWS. It powers operational dashboards, artificial intelligence and machine learning (AI/ML) models, and intelligent search capabilities. This approach aligns with Infor’s broader strategy of integrating ERP data, data lake information, machine learning (ML) predictions, and documentation to provide comprehensive end-to-end business solutions.

Real-time data streaming architecture

The foundation of our solution is Infor’s Data Fabric Stream Pipelines, an add-on feature that provides real-time streaming data processing. When data events are ingested into Data Fabric, Stream Pipelines processes them immediately and continuously without waiting for storage in the data lake. This approach minimizes the data journey and accelerates operations, helping us extract insights from our data in real time.

The end-to-end workflow consists of the following components:

Data ingestion – Infor Data Fabric tables stream changes in real-time. We enabled Stream Pipelines as an add-on feature within our Infor Cloud ERP environment. We Configure the specific ERP table that we want to stream (such as sales orders, inventory, financial transactions) to publish change events immediately upon data modification. Stream Pipelines captures insert, update, and delete operations with metadata about the operation type and timestamp.

Load distribution – Because Infor can’t reach our private VPC directly, we use Elastic Load Balancing (ELB) to distribute traffic and provide secure database access. We implemented a Network Load Balancer (NLB) with static Elastic IP addresses in public subnets, giving us stable, allowlisted IP addresses for Infor’s outbound connections. We configured an Amazon Relational Database Service (Amazon RDS) router with Amazon Elastic Compute Cloud (Amazon EC2) instances as NLB targets. These routers forward traffic from the NLB to our Amazon Aurora database in the private subnet using iptables NAT rules. This makes sure that even if the IP of Aurora changes during failover, our static Elastic IPs remain constant. We configured security groups to accept HTTPS traffic (port 443) only from Infor’s IP ranges on the NLB and allow traffic only from the NLB to the RDS routers.

Connection management – We use Amazon RDS Proxy to manage database connections and provide automatic failover. We deployed RDS Proxy in the private subnet between our RDS router instances and Aurora cluster to pool and reuse connections. This is critical for handling our high-frequency streaming data. We configured the proxy with IAM authentication for secure credentials and set connection pool parameters based on our expected concurrent stream volume to handle burst traffic without overwhelming the database. With automatic failover enabled, if our primary Aurora instance fails, RDS Proxy automatically redirects traffic to the promoted replica, maintaining continuous data flow.

Data storage – We store our operational data in Amazon Aurora PostgreSQL- Compatible Edition with multi-Availability Zone deployment for high availability. We provisioned an Aurora PostgreSQL cluster with one writer instance and multiple reader instances across different Availability Zones. We designed our database schema to handle the incoming streaming data, storing it in JSONB columns for flexible querying while using the native JSON functions of Aurora PostgreSQL when we need to parse and normalize specific fields. We created indexes on frequently queried fields to maintain query performance as our data volume grows. We also configured automated backups with point-in-time recovery and set up the automatic storage of Aurora scaling to accommodate our data growth.

Analytics and visualization – Amazon Quick Sight delivers the interactive dashboards and pixel-perfect reports our teams need. We created a Quick Sight account and established a connection to our Aurora PostgreSQL database using VPC connectivity with credentials stored in AWS Secrets Manager. We identified which datasets benefit from SPICE (Super-fast, Parallel, In-memory Calculation Engine) caching—typically aggregated or frequently accessed data—and configured incremental refresh schedules to keep them current. We built our dashboards using the visual interface of Quick Sight, using calculated fields for business logic, parameters for user interactivity, and row-level security rules to make sure that users only see data that they’re authorized to access. For pixel-perfect reports, we use the pixel-perfect report feature of Quick Sight to create formatted documents suitable for printing or regulatory compliance.

Embedded integration – We securely embedded Amazon Quick Sight dashboards within Infor OS through Amazon API Gateway, which generates dynamic URLs for seamless user access. We enabled Quick Sight embedding in our AWS account and registered our Infor domain. We created an API Gateway REST API with Lambda functions that authenticate users, validate Infor session tokens, and call QuickSight’s GenerateEmbedUrlForRegisteredUser API to produce time-limited, signed URLs with row-level security. Our Lambda function maps Infor user roles to Quick Sight permissions and applies dashboard filters based on the user’s organizational context. We configured CORS settings in API Gateway to allow requests from our Infor domain and implemented rate limiting. On the Infor side, we embedded the Quick Sight dashboards using iframe elements that call our API Gateway endpoint, providing a seamless experience where our users access analytics without leaving the ERP interface.

The following diagram illustrates the real-time analytics architecture:

Architecture diagram showing Infor ERP data flowing via Postgres streaming through Amazon Route 53, a Network Load Balancer, and RDS Proxy to Amazon Aurora PostgreSQL inside a VPC, with Amazon QuickSight for visualization and Amazon API Gateway plus Lambda generating embedded dashboard URLs for a reporting application. Amazon CloudWatch and IAM provide monitoring and access control.

This embedded experience aligns with Infor’s broader system strategy of integrating insights seamlessly into workflows.

Results and business impact

The implementation using this architecture on AWS brought substantial benefits, directly addressing the critical challenges that we faced and demonstrating measurable value in employee productivity and core business process optimization.

Business process improvement

The solution successfully addressed Oldcastle’s key operational challenges:

Challenge: Limited visibility into real-time operations

– Solution delivered: Deployed over 50 complex dashboards and reports in eight months, providing immediate visibility across customer service, finance, logistics, and manufacturing.- Technical achievement: Used Infor Data Fabric Stream Pipelines to process data events immediately upon ingestion, alleviating delays from traditional batch processing.- Impact: Real-time streaming architecture using the NDJSON format makes sure decision-makers have access to current operational data when they need it most.

Challenge: Fragmented user experience requiring multiple systems

– Solution delivered: Dashboards embedded directly into the Infor environment through Amazon API Gateway.- Technical achievement: Generated dynamic URLs for secure embedding with single sign-on capabilities.- Impact: Users access insights without leaving their familiar interface, with personalized views based on roles and permissions that maintain context across the application.

Screenshot of an Amazon QuickSight dashboard embedded in Infor M3, showing a Cash Desk Reconciliation report with a transaction table on the left, pie charts breaking down payment types (Visa, Amex, Mastercard, Cash) by cash desk location, and a detailed transaction grid below with customer order numbers, invoice dates, and payment amounts.

Challenge: Inflexible reporting limiting business agility

– Solution delivered: Both interactive dashboards and pixel-perfect reports are available to meet diverse business needs.- Technical achievement: Quick Sight SPICE caching enables subsecond response times on complex analytics across large datasets.- Impact: On-demand access for immediate insights, scheduled distribution, custom formatting aligned with corporate standards, and multiple export formats (PDF, CSV, Excel).

Screenshot of a JIT Report embedded in an ERP application showing two sections: Demand By Day and Projected On Hand by Day. Each section displays a grid of inventory items with daily quantity columns spanning multiple weeks, with backorder and on-hand counts highlighted in blue for items requiring attention.

Challenge: Delayed decision-making due to outdated data

– Solution delivered: Real-time visibility into operations enabled faster, data-driven decisions.- Technical achievement: The Multi-AZ deployment of Amazon Aurora PostgreSQL maintains high availability and continuous data access.- Impact: Decision-makers can respond immediately to operational issues and economic changes with current, reliable data.

Scalability and performance

The architecture delivered exceptional scale and performance, addressing concerns about future growth: – High concurrency: Supports over 100 concurrent users without performance degradation- Data volume handling: Processes millions of transactions daily in real-time- Elastic scaling: Aurora read replicas automatically scale based on demand- Future-ready: Architecture designed to expand to additional regions and use cases- Cost efficiency: AWS services avoided complex third-party integrations, with infrastructure costs scaling efficiently with business growth- API capabilities: Ability to expose data using AWS technologies enables integration with third-party and internal applications

Conclusion

Our journey proves that cloud ERP migrations don’t require sacrificing real-time operational reporting capabilities. By combining Infor Data Fabric Stream Pipelines with AWS analytics and AI services, we’ve maintained real-time data access, accelerated innovation, improved user experience, and built a system that scales efficiently as our business needs evolve. The combination of Infor’s enterprise-grade ERP system with the comprehensive analytics capabilities of AWS has given us the best of both worlds. As we expand our AWS analytics and AI capabilities, we’re not just maintaining parity with on-premises systems, we’re unlocking new sources of business value that weren’t possible before.

Further reading

For more information on the services mentioned in the post, see the following resources:

AWS services:

  • Amazon Aurora PostgreSQL Features – Learn more about the high-availability database that powers Oldcastle’s real-time data storage and Multi-AZ deployment strategy
  • Amazon Quick Sight Embedded Analytics – Explore how to embed interactive dashboards and pixel-perfect reports directly into your enterprise applications, as demonstrated in Oldcastle’s Infor OS integration
  • Amazon Bedrock for Generative AI – Discover opportunities to enhance your analytics system with AI-powered insights and intelligent search capabilities
  • Elastic Load Balancing – Understand how to distribute traffic and secure database connections when integrating cloud ERP systems with AWS services
  • Amazon API Gateway – Learn how to create secure, dynamic URLs for embedding analytics and exposing data through APIs to internal and external applications

Infor Resources:

About the authors

AWS Weekly Roundup: AWS AI/ML Scholars program, Agent Plugin for AWS Serverless, and more (March 30, 2026)

Post Syndicated from Prasad Rao original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-aws-ai-ml-scholars-program-agent-plugin-for-aws-serverless-and-more-march-30-2026/

Last week, what excited me most was the launch of the 2026 AWS AI & ML Scholars program by Swami Sivasubramanian, VP of AWS Agentic AI, to provide free AI education to up to 100,000 learners worldwide. The program has two phases: a Challenge phase where you’ll learn foundational generative AI skills, followed by a fully funded three-month Udacity Nanodegree for the top 4,500 performers. Anyone 18 or older can apply, with no prior AI or ML experience required. Applications close on June 24, 2026. Visit the AWS AI & ML Scholars webpage to learn more and apply.

The AWS AI & ML Scholars Program is back

I’m also excited about the start of AWS Summit season, kicking off with AWS Summit Paris on April 1, followed by London on April 22. AWS Summits are free in-person events where builders and innovators can learn about Cloud and AI, think big, and make new connections. Explore the AWS Summits near you and join us in person.

Now, let’s dive into this week’s AWS news…

Last week’s launches
Here are last week’s launches that caught my attention:

  • Announcing Amazon Aurora PostgreSQL serverless database creation in seconds — Amazon Aurora PostgreSQL now offers express configuration, a streamlined setup with preconfigured defaults that supports creating and connecting to a database in seconds. With just two clicks, you can launch an Aurora PostgreSQL serverless database. You can modify certain settings during or after creation.
  • Amazon Aurora PostgreSQL now available with the AWS Free Tier — Amazon Aurora PostgreSQL is now available on the AWS Free Tier. If you’re new to AWS, you receive $100 in AWS credits upon sign-up and can earn an additional $100 in credits by using services like Amazon Relational Database Service (Amazon RDS).
  • Announcing Agent Plugin for AWS Serverless — With the new Agent Plugin for AWS Serverless, you can easily build, deploy, troubleshoot, and manage serverless applications using AI coding assistants like Kiro, Claude Code, and Cursor. This plugin extends AI assistants with structured capabilities by packaging skills, sub-agents, and Model Context Protocol (MCP) servers into one modular unit. It automatically loads the guidance and expertise you need throughout development to build production-ready serverless applications on AWS.
  • Amazon SageMaker Studio now supports Kiro and Cursor IDEs as remote IDEs — You can now remotely connect from Kiro and Cursor IDEs to Amazon SageMaker Studio. This lets you use your existing Kiro and Cursor setup, including spec-driven development, conversational coding, and automated feature generation, while accessing the scalable compute resources of Amazon SageMaker Studio.
  • Introducing visual customization capability in AWS Management Console — You can now customize your AWS Management Console with visual settings like account color and control which regions and services you see. Hiding unused regions and services helps you focus better and work faster by reducing cognitive load and unnecessary scrolling.
  • Announcing Aurora DSQL connector to simplify building Ruby applications — You can now use the Aurora DSQL Connector for Ruby (pg gem) to easily build Ruby applications on Aurora DSQL. The Ruby Connector simplifies authentication and improves security by automatically generating tokens for each connection, eliminating the risks of traditional passwords while maintaining full compatibility with existing pg gem features.
  • AWS Lambda increases the file descriptor limit for functions running on Lambda Managed Instances — AWS Lambda increases the file descriptor limit from 1,024 to 4,096, a 4x increase, for functions running on Lambda Managed Instances (LMI). You can now run I/O intensive workloads such as high-concurrency web services and file-heavy data processing pipelines without running into file descriptor limits.
  • AWS Lambda now supports up to 32 GB of memory and 16 vCPUs for Lambda Managed Instances — AWS Lambda functions on Lambda Managed Instances now support up to 32 GB of memory and 16 vCPUs. You can run compute-intensive workloads like data processing, media transcoding, and scientific simulations without managing infrastructure. Plus, you can adjust the memory-to-vCPU ratio (2:1, 4:1, or 8:1) to fit your workload.
  • Announcing Bidirectional Streaming API for Amazon Polly — Traditional text-to-speech APIs use a request-response pattern. The new Bidirectional Streaming API for Amazon Polly is designed for conversational AI applications that generate text or audio incrementally, like large language model (LLM) responses. This lets you start synthesizing audio before the full text is available.

For a full list of AWS announcements, be sure to keep an eye on our News Blog channel and the What’s New with AWS page.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • AWS Summits — As I mentioned earlier, join AWS Summits in 2026 for free in-person events where you can explore emerging cloud and AI technologies, learn best practices, and network with industry peers and experts. Upcoming Summits include Paris (April 1), London (April 22), Bengaluru (April 23–24), Singapore (May 6), Tel Aviv (May 6), and Stockholm (May 7).
  • AWS Community Days — Community-led conferences where content is planned, sourced, and delivered by community leaders, featuring technical discussions, workshops, and hands-on labs. Upcoming events include San Francisco (April 10) and Romania (April 23–24).

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse the AWS Events and Webinars for upcoming AWS-led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— Prasad

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Announcing Amazon Aurora PostgreSQL serverless database creation in seconds

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/announcing-amazon-aurora-postgresql-serverless-database-creation-in-seconds/

At re:Invent 2025, Colin Lazier, vice president of databases at AWS, emphasized the importance of building at the speed of an idea—enabling rapid progress from concept to running application. Customers can already create production-ready Amazon DynamoDB tables and Amazon Aurora DSQL databases in seconds. He previewed creating an Amazon Aurora serverless database with the same speed, and customers have since requested quick access and speed to this capability.

Today, we’re announcing the general availability of a new express configuration for Amazon Aurora PostgreSQL, a streamlined database creation experience with preconfigured defaults designed to help you get started in seconds.

With only two clicks, you can have an Aurora PostgreSQL serverless database ready to use in seconds. You have the flexibility to modify certain settings during and after database creation in the new configuration. For example, you can change the capacity range for the serverless instance at the time of create or add read replicas, modify parameter groups after the database is created. Aurora clusters with express configuration are created without an Amazon Virtual Private Cloud (Amazon VPC) network and include an internet access gateway for secure connections from your favorite development tools – no VPN, or AWS Direct Connect required. Express configuration also sets up AWS Identity and Access Management (IAM) authentication for your administrator user by default, enabling passwordless database authentication from the beginning without additional configuration.

After it’s created, you have access to features available for Aurora PostgreSQL serverless, such as deploying additional read replicas for high availability and automated failover capabilities. This launch also introduces a new internet access gateway routing layer for Aurora. Your new serverless instance comes enabled by default with this feature, which allows your applications to connect securely from anywhere in the world through the internet using the PostgreSQL wire protocol from a wide range of developer tools. This gateway is distributed across multiple Availability Zones, offering the same level of high availability as your Aurora cluster.

Creating and connecting to Aurora in seconds means fundamentally rethinking how you get started. We launched multiple capabilities that work together to help you onboard and run your application with Aurora. Aurora is now available on AWS Free Tier, which you gain hands-on experience with Aurora at no upfront cost. After it’s created, you can directly query an Aurora database in AWS CloudShell or using programming languages and developer tools through a new internet accessible routing component for Aurora. With integrations such as v0 by Vercel, you can use natural language to start building your application with the features and benefits of Aurora.

Create an Aurora PostgreSQL serverless database in seconds
To get started, go to the Aurora and RDS console and in the navigation pane, choose Dashboard. Then, choose Create with a rocket icon.

Review pre-configured settings in the Create with express configuration dialog box. You can modify the DB cluster identifier or the capacity range as needed. Choose Create database.

You can also use the AWS Command Line Interface (AWS CLI) or AWS SDKs with the parameter --express-configuration to create both a cluster and an instance within the cluster with a single API call which makes it ready for running queries in seconds.To learn more, visit Creating an Aurora PostgreSQL DB cluster with express configuration.

Here is a CLI command to create the cluster:

$ aws rds create-db-cluster --db-cluster-identifier channy-express-db \
    --engine aurora-postgresql \
    –with-express-configuration

Your Aurora PostgreSQL serverless database should be ready in seconds. A success banner confirms the creation, and the database status changes to Available.

After your database is ready, go to the Connectivity & security tab to access three connection options. When connecting through SDKs, APIs, or third-party tools including agents, choose Code snippets. You can choose various programming languages such as .NET, Golang, JDBC, Node.js, PHP, PSQL, Python, and TypeScript. You can paste the code from each step into your tool and run the commands.

For example, the following Python code is dynamically generated to reflect the authentication configuration:

import psycopg2
import boto3

auth_token = boto3.client('rds', region_name='ap-south-1').generate_db_auth_token(DBHostname='channy-express-db-instance-1.abcdef.ap-south-1.rds.amazonaws.com', Port=5432, DBUsername='postgres', Region='ap-south-1')

conn = None
try:
    conn = psycopg2.connect(
        host='channy-express-db-instance-1.abcdef.ap-south-1.rds.amazonaws.com',
        port=5432,
        database='postgres',
        user='postgres',
        password=auth_token,
        sslmode='require'
    )
    cur = conn.cursor()
    cur.execute('SELECT version();')
    print(cur.fetchone()[0])
    cur.close()
except Exception as e:
    print(f"Database error: {e}")
    raise
finally:
    if conn:
        conn.close()

const { Client } = require('pg');
const AWS = require('aws-sdk');
AWS.config.update({ region: 'ap-south-1' });

async function main() {
  let password = '';
  const signer = new AWS.RDS.Signer({ region: 'ap-south-1', hostname: 'channy-express-db-instance-1.abcdef.ap-south-1.rds.amazonaws.com', port: 5432, username: 'postgres' });
  password = signer.getAuthToken({});

  const client = new Client({
    host: 'channy-express-db-instance-1.abcdef.ap-south-1.rds.amazonaws.com',
    port: 5432,
    database: 'postgres',
    user: 'postgres',
    password,
    ssl: { rejectUnauthorized: false }
  });

  try {
    await client.connect();
    const res = await client.query('SELECT version()');
    console.log(res.rows[0].version);
  } catch (error) {
    console.error('Database error:', error);
    throw error;
  } finally {
    await client.end();
  }
}
main().catch(console.error);

Choose CloudShell for quick access to the AWS CLI which launches directly from the console. When you choose Launch CloudShell, you can see the command is pre-populated with relevant information to connect to your specific cluster. After connecting to the shell, you should see the psql login and the postgres => prompt to run SQL commands.

You can also choose Endpoints to use tools that only support username and password credentials, such as pgAdmin. When you choose Get token, you use an AWS Identity and Access Management (IAM) authentication token generated by the utility in the password field. The token is generated for the master username that you set up at the time of creating the database. The token is valid for 15 minutes at a time. If the tool you’re using terminates the connection, you will need to generate the token again.

Building your application faster with Aurora databases
At re:Invent 2025, we announced enhancements to the AWS Free Tier program, offering up to $200 in AWS credits that can be used across AWS services. You’ll receive $100 in AWS credits upon sign-up and can earn an additional $100 in credits by using services such as Amazon Relational Database Service (Amazon RDS), AWS Lambda, and Amazon Bedrock. In addition, Amazon Aurora is now available across a broad set of eligible Free Tier database services.

Developers are embracing platforms such as Vercel, where natural language is all it takes to build production-ready applications. We announced integrations with Vercel Marketplace to create and connect to an AWS database directly from Vercel in seconds and v0 by Vercel, an AI-powered tool that transforms your ideas into production-ready, full-stack web applications in minutes. It includes Aurora PostgreSQL, Aurora DSQL, and DynamoDB databases. You can also connect your existing databases created through express configuration with Vercel. To learn more, visit AWS for Vercel.

Like Vercel, we’re bringing our databases seamlessly into their experiences and are integrating directly with widely adopted frameworks, AI assistant coding tools, environments, and developer tools, all to unlock your ability to build at the speed of an idea.

We introduced Aurora PostgreSQL integration with Kiro powers, which developers can use to build Aurora PostgreSQL backed applications faster with AI agent-assisted development through Kiro. You can use Kiro power for Aurora PostgreSQL within Kiro IDE and from the Kiro powers webpage for one-click installation. To learn more about this Kiro Power, read Introducing Amazon Aurora powers for Kiro and Amazon Aurora Postgres MCP Server.

Now available
You can create an Aurora PostgreSQL serverless database in seconds today in all AWS commercial Regions. For Regional availability and a future roadmap, visit the AWS Capabilities by Region.

You pay only for capacity consumed based on Aurora Capacity Units (ACUs) billed per second from zero capacity, which automatically starts up, shuts down, and scales capacity up or down based on your application’s needs. To learn more, visit the Amazon Aurora Pricing page.

Give it a try in the Aurora and RDS console and send feedback to AWS re:Post for Aurora PostgreSQL or through your usual AWS Support contacts.

— Channy

Extract data from Amazon Aurora MySQL to Amazon S3 Tables in Apache Iceberg format

Post Syndicated from Kunal Ghosh original https://aws.amazon.com/blogs/big-data/extract-data-from-amazon-aurora-mysql-to-amazon-s3-tables-in-apache-iceberg-format/

If you manage data in Amazon Aurora MySQL-Compatible Edition and want to make it available for analytics, machine learning (ML), or cross-service querying in a modern lakehouse format, you’re not alone.

Organizations often need to run analytics, build ML models, or join data across multiple sources. These are examples of workloads that can be resource-intensive and impractical to run directly against a transactional database. By extracting your Aurora MySQL data into Amazon S3 Tables in Apache Iceberg format, you can offload analytical queries from your production database without impacting its performance, while storing data in a fully managed Iceberg table store optimized for analytics. Built on the open Apache Iceberg standard, Amazon Simple Storage Service (Amazon S3) Table data is queryable from engines like Amazon Athena, Amazon Redshift Spectrum, and Apache Spark without additional data copies. You can also combine relational data with other datasets already in your data lake, enabling richer cross-domain insights.

Apache Iceberg and Amazon S3 Tables

Apache Iceberg is a widely adopted open table format that offers Atomicity, Consistency, Isolation, Durability (ACID) transactions, schema evolution, and time travel capabilities. It enables multiple engines to work concurrently on the same dataset, making it a popular choice for building open lakehouse architectures.

Amazon S3 Tables is a purpose-built, fully managed Apache Iceberg table store designed for analytics workloads. It delivers up to 3x faster query performance and up to 10x more transactions per second compared to self-managed Iceberg tables. It also automatically compacts data and removes unreferenced files to optimize storage and performance.

In this post, you learn how to set up an automated, end-to-end solution that extracts tables from Amazon Aurora MySQL Serverless v2 and writes them to Amazon S3 Tables in Apache Iceberg format using AWS Glue. The entire infrastructure is deployed using a single AWS CloudFormation stack.

Requirements

AWS offers zero-ETL integrations from Amazon Aurora to Amazon Redshift and Amazon SageMaker AI, enabling seamless data flow for analytics and machine learning workloads.

However, there isn’t yet a native zero-ETL integration between Amazon Aurora and Amazon S3 Tables. This means that organizations looking to use Amazon S3 Tables for their Lakehouse architecture currently face several requirements:

  • Setting up ETL pipelines to extract data from Amazon Aurora and transform it into Apache Iceberg format
  • Configuring networking and security for AWS Glue jobs to access Amazon Aurora databases in private subnets
  • Coordinating the provisioning of source databases, ETL pipelines, and target table stores
  • Managing the end-to-end workflow without native automation

Solution overview

In this solution, you automate the extraction of relational database tables from Amazon Aurora MySQL Serverless v2 to Amazon S3 Tables in Apache Iceberg format using AWS Glue 5.0. To help you get started and test this solution, a CloudFormation template is provided. This template provisions the required infrastructure, loads sample data, and configures the Extract, Transform, Load (ETL) pipeline. You can adapt this template for your own scenario.

Solution overview

Sample data

This solution uses the TICKIT sample database, a well-known dataset used in Amazon Redshift documentation. The TICKIT data models a fictional ticket sales system with seven interrelated tables: users, venue, category, date, event, listing, and sales. The dataset is publicly available as mentioned in the Amazon Redshift Getting Started Guide.

Solution flow

The solution flow as shown in the previous architecture diagram:

  1. An AWS Lambda function downloads the TICKIT sample dataset (a fictional ticket sales system used in Amazon Redshift documentation) from a public Amazon S3 bucket to a staging S3 bucket.
  2. A second Lambda function, using PyMySQL (a Python MySQL client library), loads the staged data files into the Aurora MySQL Serverless v2 database using LOAD DATA LOCAL INFILE.
  3. The AWS Glue job reads seven TICKIT tables from Aurora MySQL through a native MySQL connection and writes them to Amazon S3 Tables in Apache Iceberg format using the S3 Tables REST catalog endpoint with SigV4 authentication.
  4. You can query the migrated data in S3 Tables using Amazon Athena.

The solution consists of the following key components:

  1. Amazon Aurora MySQL Serverless v2 as the source relational database containing the TICKIT sample dataset (users, venue, category, date, event, listing, and sales tables)
  2. AWS Secrets Manager to store the Aurora MySQL database credentials securely
  3. Amazon S3 staging bucket for the TICKIT sample data files downloaded from the public redshift-downloads S3 bucket
  4. AWS Lambda functions using PyMySQL to load data into Aurora MySQL
  5. AWS Glue 5.0 job (PySpark) to read tables from Aurora MySQL and write them to S3 Tables in Apache Iceberg format
  6. Amazon S3 Tables as the target storage for the migrated Iceberg tables
  7. Amazon VPC with private subnets and VPC endpoints for Amazon S3, S3 Tables, AWS Glue, Secrets Manager, AWS Security Token Service (AWS STS), CloudWatch Logs, and CloudFormation

Here are some advantages of this architecture:

  • Fully automated setup: A single CloudFormation stack provisions the required infrastructure, loads sample data, and configures the ETL pipeline.
  • Serverless and cost-efficient: Aurora MySQL Serverless v2 and AWS Glue both scale based on demand, minimizing idle costs.
  • Apache Iceberg table format: Data is stored in Apache Iceberg format, enabling ACID transactions, schema evolution, and time travel queries.
  • Network isolation and credential management: The resources run within private subnets with Virtual Private Cloud (VPC) endpoints, and database credentials are managed through AWS Secrets Manager.
  • Extensible pattern: The same approach can be adapted for other relational databases (PostgreSQL, SQL Server) and other target formats supported by AWS Glue.

Prerequisites

To follow along, you need an AWS account. If you don’t yet have an AWS account, you must create one. The CloudFormation stack deployment takes approximately 30-45 minutes to complete and requires familiarity with Amazon S3 Tables, AWS CloudFormation, Apache Iceberg, AWS Glue, Amazon Aurora. This solution will incur AWS costs. The main cost drivers are AWS Glue ETL job runs (billed per DPU-hour, proportional to data volume) and Amazon S3 Tables storage and request charges. Remember to clean up resources when you are done to avoid unnecessary charges.

CloudFormation parameters

You can configure the following parameters before deploying the CloudFormation stack:

Parameter Description Default Required
S3TableBucketName Name of the S3 Tables bucket to create (or use existing) Yes
DatabaseName Name of the initial Aurora MySQL database tickit No
MasterUsername Master username for Aurora MySQL admin No
VpcCidr CIDR block for the VPC 10.1.0.0/16 No
S3TableNamespace Namespace for S3 Tables tickit No

Implementation walkthrough

The following steps walk you through the implementation. These steps are to deploy and test an end-to-end solution from scratch. If you are already running some of these components, you may skip to the relevant step. You can also refer to the aws-samples repository, sample-to-write-aurora-mysql-to-s3tables-using-glue for the entire solution.

Step 1: Deploy the CloudFormation stack

Deploy the CloudFormation template scripts/aurora-mysql-to-s3tables-stack.yaml using the AWS Console or the AWS Command Line Interface (AWS CLI). Provide a name for the S3 Tables bucket; the stack will create it automatically (or use an existing one if it already exists).

To deploy using the AWS Console (recommended), navigate to the AWS CloudFormation Console and use the CloudFormation template. Alternatively, to deploy using the AWS CLI first upload the template to an S3 bucket (the template exceeds the 51,200 byte limit for inline –template-body), then create the stack.

# Upload the template to S3
aws s3 cp scripts/aurora-mysql-to-s3tables-stack.yaml \
  s3://<your-s3-bucket>/aurora-mysql-to-s3tables-stack.yaml \
  --region <your-region>
# Create the stack using the S3 template URL
aws cloudformation create-stack \
  --stack-name aurora-mysql-tickit-stack \
  --template-url https://<your-s3-bucket>.s3.<your-region>.amazonaws.com/aurora-mysql-to-s3tables-stack.yaml \
  --parameters \
    ParameterKey=S3TableBucketName,ParameterValue=<your-s3-table-bucket-name> \
  --capabilities CAPABILITY_NAMED_IAM \
  --region <your-region>

The stack will automatically:

  • Create the S3 Tables bucket (or use existing if it already exists)
  • Create a VPC with private subnets and VPC endpoints
  • Provision an Aurora MySQL Serverless v2 cluster
  • Download TICKIT sample data from the public Amazon S3 bucket
  • Load the sample data into Aurora MySQL via a Lambda function using PyMySQL
  • Create a Glue job configured to migrate data to S3 Tables in Iceberg format

Note: The S3 Tables bucket is retained when the stack is deleted to preserve your data.

Step 2: Verify the Aurora MySQL data

Retrieve the AuroraClusterEndpoint, DatabaseName, and SecretArn values from the CloudFormation stack, make a note of the AuroraClusterEndpoint, DatabaseName, and SecretArn. You can navigate to the Amazon Aurora Console, choose the Query Editor, and enter the values from the CloudFormation stack to connect. You can also choose your preferred method of connecting to an Amazon Aurora DB cluster.

Use the AWS CLI to retrieve the stack outputs: –

aws cloudformation describe-stacks --stack-name aurora-mysql-tickit-stack --region <your-region> --query "Stacks[0].Outputs"

Then run the following SQL commands to verify the data load:

-- Verify if the tables are created
SELECT * FROM information_schema.tables WHERE table_schema = 'tickit';

-- Verify if the data is loaded
SELECT 'users' AS table_name, COUNT(*) AS record_count FROM tickit.users
UNION ALL SELECT 'venue', COUNT(*) FROM tickit.venue
UNION ALL SELECT 'category', COUNT(*) FROM tickit.category
UNION ALL SELECT 'date', COUNT(*) FROM tickit.date
UNION ALL SELECT 'event', COUNT(*) FROM tickit.event
UNION ALL SELECT 'listing', COUNT(*) FROM tickit.listing
UNION ALL SELECT 'sales', COUNT(*) FROM tickit.sales;

Step 3: Run the Glue job

Navigate to the AWS Glue Console, choose ETL jobs under Data Integration and ETL from the left panel. Select the AWS Glue job mysql-tickit-to-iceberg-job and choose Run job to start execution. You can also start the ETL job using the AWS CLI:

aws glue start-job-run --job-name mysql-tickit-to-iceberg-job --region <your-region>

The AWS Glue job performs the following operations for each of the seven TICKIT tables:

  • Reads the table from Aurora MySQL through the native MYSQL Glue connection
  • Converts the data to a Spark DataFrame
  • Creates the Iceberg table in the S3 Tables namespace using CREATE TABLE IF NOT EXISTS with the USING ICEBERG clause
  • Inserts the data using INSERT INTO (or INSERT OVERWRITE if the table already exists)
  • Verifies the record count and displays sample data

Step 4: Verify the results

After the AWS Glue job completes, verify that the tables have been created in your S3 Table bucket by navigating to the Amazon S3 Console. Choose Table buckets under Buckets and select your S3 Table bucket. You can also verify using the AWS CLI:

aws s3tables list-tables \
  --table-bucket-arn arn:aws:s3tables:<your-region>:<your-account-id>:bucket/<your-s3-table-bucket-name> \
  --namespace tickit \
  --region <your-region>

Select a table from the tickit namespace and choose Preview to inspect the data.

Preview S3 data

You can also query the migrated tables using Amazon Athena to validate the data.

Clean up resources

Remember to clean up resources when you no longer need them to avoid unnecessary charges.

Navigate to the CloudFormation console, search for your stack and choose Delete. Alternatively, use the AWS CLI:

aws cloudformation delete-stack --stack-name aurora-mysql-tickit-stack --region <your-region>

The S3 Tables bucket is retained by default. To delete it, use the Amazon S3 console or the AWS CLI to remove the table bucket separately. The staging S3 bucket will be automatically emptied and deleted as part of the stack deletion.

aws s3tables delete-table-bucket --table-bucket-arn arn:aws:s3tables:<your-region>:<your-account-id>:bucket/<your-s3-table-bucket-name> --region <your-region>

Summary

In this post, we showed you how to extract data from Amazon Aurora MySQL Serverless v2 and write it to Amazon S3 Tables in Apache Iceberg format using AWS Glue 5.0. By using the native Iceberg support of AWS Glue and the S3 Tables REST catalog endpoint, you can bridge the gap between relational databases and modern lakehouse storage formats. By automating the entire pipeline through CloudFormation, you can quickly set up and replicate this pattern across multiple environments.

As AWS Glue and Amazon S3 Tables continue to evolve, you can take advantage of future enhancements while maintaining this automated migration pattern.

If you have questions or suggestions, leave us a comment.

About the authors

Kunal Ghosh

Kunal Ghosh

Kunal is a Sr. Solutions Architect at AWS. He is passionate about building efficient and effective solutions on AWS, especially involving generative AI, analytics, data science, and machine learning. Besides family time, he likes reading, swimming, biking, and watching movies.

Arghya Banerjee

Arghya Banerjee

Arghya is a Sr. Solutions Architect at AWS in the San Francisco Bay Area, focused on helping customers adopt and use the AWS Cloud. He is focused on big data, data lakes, streaming and batch analytics services, and generative AI technologies.

Indranil Banerjee

Indranil Banerjee

Indranil is a Sr. Solutions Architect at AWS in the San Francisco Bay Area, focused on helping customers in the hi-tech and semi-conductor sectors solve complex business problems using the AWS Cloud. His special interests are in the areas of legacy modernization and migration, building analytics platforms and helping customers adopt cutting edge technologies such as generative AI.

Vipan Kumar

Vipan Kumar

Vipan is a Sr. Solutions Architect at AWS, where he works with strategic customers. He has extensive experience in machine learning and generative AI. With a background in application development, he is passionate about designing and building enterprise applications for the cloud.

AWS Weekly Roundup: Claude Sonnet 4.6 in Amazon Bedrock, Kiro in GovCloud Regions, new Agent Plugins, and more (February 23, 2026)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-claude-sonnet-4-6-in-amazon-bedrock-kiro-in-govcloud-regions-new-agent-plugins-and-more-february-23-2026/

Last week, my team met many developers at Developer Week in San Jose. My colleague, Vinicius Senger delivered a great keynote about renascent software—a new way of building and evolving applications where humans and AI collaborate as co-developers using Kiro. Other colleagues spoke about building and deploying production-ready AI agents. Everyone stayed to ask and hear the questions related to agent memory, multi-agent patterns, meta-tooling and hooks. It was interesting how many developers were actually building agents.

We are continuing to meet developers and hear their feedback at third-party developer conferences. You can meet us at the dev/nexus, the largest and longest-running Java ecosystem conference on March 4-6 in Atlanta. My colleague, James Ward will speak about building AI Agents with Spring and MCP, and Vinicius Senger and Jonathan Vogel will speak about 10 tools and tips to upgrade your Java code with AI. I’ll keep sharing places for you to connect with us.

Last week’s launches
Here are some of the other announcements from last week:

  • Claude Sonnet 4.6 model in Amazon Bedrock – You can now use Claude Sonnet 4.6 which offers frontier performance across coding, agents, and professional work at scale. Claude Sonnet 4.6 approaches Opus 4.6 intelligence at a lower cost. It enables faster, high-quality task completion, making it ideal for high-volume coding and knowledge work use cases.
  • Amazon EC2 Hpc8a instances powered by 5th Gen AMD EPYC processors – You can use new Hpc8a instances delivering up to 40% higher performance, increased memory bandwidth, and 300 Gbps Elastic Fabric Adapter networking. You can accelerate compute-intensive simulations, engineering workloads, and tightly coupled HPC applications.
  • Amazon SageMaker Inference for custom Amazon Nova models – You can now configure the instance types, auto-scaling policies, and concurrency settings for custom Nova model deployments with Amazon SageMaker Inference to best meet your needs.
  • Nested virtualization on virtual Amazon EC2 instances – You can create nested virtual machines by running KVM or Hyper-V on virtual EC2 instances. You can leverage this capability for use cases such as running emulators for mobile applications, simulating in-vehicle hardware for automobiles, and running Windows Subsystem for Linux on Windows workstations.
  • Server-Side Encryption by default in Amazon Aurora – Amazon Aurora further strengthens your security posture by automatically applying server-side encryption by default to all new databases clusters using AWS-owned keys. This encryption is fully managed, transparent to users, and with no cost or performance impact.
  • Kiro in AWS GovCloud (US) Regions – You can use Kiro for the development teams behind government missions. Developers in regulated environments can now leverage Kiro’s agentic AI tool with the rigorous security controls required.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Additional updates
Here are some additional news items that you might find interesting:

  • Introducing Agent Plugins for AWS – You can see how new open-source Agent Plugins for AWS extend coding agents with skills for deploying applications to AWS. Using the deploy-on-aws plugin, you can generate architecture recommendations, cost estimates, and infrastructure-as-code directly from your coding agent.
  • A chat with Byron Cook on automated reasoning and trust in AI systems – You can hear how to verify AI systems doing the right thing using automated reasoning when they generate code or manage critical decisions. Byron Cook’s team has spent a decade proving correctness in AWS and apply those techniques to agentic systems.
  • Best practices for deploying AWS DevOps Agent in production – You can read best practices for setting up DevOps Agent Spaces that balance investigation capability with operational efficiency. According to Swami Sivasubramanian, AWS DevOps Agent, a frontier agent that resolves and proactively prevents incidents, has handled thousands of escalations, with an estimated root cause identification rate of over 86% within Amazon.

From AWS community
Here are my personal favorite posts from AWS community:

Join the AWS Builder Center to connect with community, share knowledge, and access content that supports your development.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • AWS Summits – Join AWS Summits in 2026, free in-person events where you can explore emerging cloud and AI technologies, learn best practices, and network with industry peers and experts. Upcoming Summits include Paris (April 1), London (April 22), and Bengaluru (April 23–24).
  • Amazon Nova AI Hackathon – Join developers worldwide to build innovative generative AI solutions using frontier foundation models and compete for $40,000 in prizes across five categories including agentic AI, multimodal understanding, UI automation, and voice experiences during this six-week challenge from February 2nd to March 16th, 2026.
  • AWS Community Days – Community-led conferences where content is planned, sourced, and delivered by community leaders, featuring technical discussions, workshops, and hands-on labs. Upcoming events include Ahmedabad (February 28), JAWS Days in Tokyo (March 7), Chennai (March 7), Slovakia (March 11), and Pune (March 21).

Browse here for upcoming AWS led in-person and virtual events, startup events, and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— Channy

Streamline large binary object migrations: A Kafka-based solution for Oracle to Amazon Aurora PostgreSQL and Amazon S3

Post Syndicated from Naresh Dhiman original https://aws.amazon.com/blogs/big-data/streamline-large-binary-object-migrations-a-kafka-based-solution-for-oracle-to-amazon-aurora-postgresql-and-amazon-s3/

Customers migrating from on-premises Oracle databases to AWS face a challenge: efficiently relocating large object data types (LOBs) to object storage while maintaining data integrity and performance. This challenge originates from the traditional enterprise database design where LOBs are stored alongside structured data, leading to storage capacity constraints, backup complexity, and performance bottlenecks during data retrieval and processing. LOBs, which can include images, videos, and other large files, often cause traditional data migrations to suffer from slow speeds and LOB truncation issues. These issues are particularly problematic for long-running migrations that can span several years.

In this post, we present a scalable solution that uses Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon Aurora PostgreSQL-Compatible Edition, and Amazon MSK Connect. The data streaming enables data replication where modifications are sent and received in a continuous flow, allowing the target database to access and apply the changes in real time. This solution generates events for database actions such as insert, update, and delete, triggering AWS Lambda functions to download LOBs from the source Oracle database and upload them to Amazon Simple Storage Service (Amazon S3) buckets. Simultaneously, the streaming events migrate the structured data from the Oracle database to the target database while maintaining proper linking with their respective LOBs.

The complete implementation is available on GitHub, including AWS Cloud Development Kit (AWS CDK) deployment code, configuration files, and setup instructions.

Solution overview

Although traditional Oracle database migrations handle structured data effectively, they struggle with LOBs that can include images, videos, and documents. These migrations often fail due to size limitations and truncation issues, creating significant business risks, including data loss, extended downtime, and project delays that can force you to delay your cloud transformation initiatives. The problem becomes more acute during long-running migrations spanning several years, where maintaining operational continuity is critical. This solution addresses the key challenges of LOB migration, enabling continuous, long-term operations without compromising performance or reliability.

By removing the size limitations associated with traditional migration technologies, our solution provides a robust framework that helps you seamlessly relocate LOBs while facilitating data integrity throughout the process.

Our approach uses a modern streaming architecture to alleviate the traditional constraints of Oracle LOB migration. The solution includes the following core components:

  • Amazon MSK – Provides the streaming infrastructure.
  • Amazon MSK Connect – Using two connectors:
    • Debezium Connector for Oracle as a source connector to capture row-level changes that occur in Oracle database. The connector emits change events and publishes to a Kafka source topic.
    • Debezium Connector for JDBC as a sink connector to consume events from Kafka source topic and then write those events to Aurora PostgreSQL-Compatible by using a JDBC driver.
  • Lambda function – Triggered by an event source mapping to Amazon MSK. The function processes events from the Kafka source topic, extracting the Oracle row primary key from each event payload. It uses this key to download the corresponding BLOB data from the source Oracle database and uploads it to Amazon S3, organizing files by primary key folders to maintain simple linking with the relational database records.
  • Amazon RDS for Oracle – Amazon Relational Database Service (Amazon RDS) for Oracle is used as the source database to simulate an on-premises Oracle database.
  • Aurora PostgreSQL-Compatible – Used as the target database for migrated data.
  • Amazon S3 – Used as object storage for storing the BLOB data from source database.

The following diagram shows the Oracle LOB data migration architecture solution.

Message flow

When data changes occur in the source Amazon RDS for Oracle database, the solution executes the following sequence, moving through event detection and publication, BLOB processing with Lambda, and structured data processing:

  1. The Oracle source connector captures the change data capture (CDC) events, including the change to BLOB data column. This connector configures the BLOB data column to exclude from the Kafka event to optimize the Kafka payload.
  2. The connector publishes this event to an MSK topic.
    1. The MSK event triggers the BLOB Downloader Lambda function for the CDC events.
      1. The Lambda function examines two key conditions: the Debezium event code (specifically checking for create (c) or update(u)) and the configured list of Oracle BLOB table names along with their column names. When a Kafka message matches both the configured table list and valid Debezium events, the Lambda function initiates the BLOB data download from the Oracle source using the primary key and table name; otherwise, the function bypasses the BLOB download process. This selective approach makes sure the Lambda function only executes SQL queries when processing Kafka messages for tables containing BLOB data, optimizing database interactions.
      2. The Lambda function uploads the BLOB to Amazon S3, organizing by primary key folders with unique object names, which enables linking between structured database records and their corresponding BLOB data in Amazon S3.
    2. The PostgreSQL sink connector receives the event from the MSK topic.
      1. The connector applies these changes to the Aurora PostgreSQL database for the Oracle database changes except the BLOB data column. The BLOB data column is excluded by the Oracle source connector.

Key benefits

The solution offers the following key advantages:

  • Cost optimization and licensing – Our approach offers significant cost optimization benefits by reducing the overall size of your database and alleviating your need for expensive licenses associated with traditional databases and replication technologies. By decoupling LOB storage from the database and using Amazon S3, you can reduce your overall database footprint and reduce costs associated with traditional licensing and replication technologies. The streaming architecture also minimizes your infrastructure overhead during long-running migrations.
  • Avoids size constraints and migration failures – Traditional migration tools often impose size limitations on LOB transfers, leading to truncation issues and failed migrations. This solution removes those constraints entirely, so you can migrate LOBs of different sizes while maintaining data integrity. The event-driven architecture enables near real-time data replication, allowing your source systems to remain operational during migration.
  • Business continuity and operational excellence – Changes flow continuously to your target environment, allowing for business continuity. The solution preserves relationships between structured database records and their corresponding LOBs through primary key-based organization in Amazon S3, allowing for referential integrity while providing the flexibility of object storage for large files.
  • Architectural advantages – Storing LOBs in Amazon S3 while maintaining structured data in Aurora PostgreSQL-Compatible creates a clear separation. This architecture simplifies your backup and recovery operations, improves query performance on structured data, and provides flexible access patterns for binary objects through Amazon S3.

Implementation best practices

Consider the following best practices when implementing this solution:

  • Start small and scale gradually – To implement this solution, start with a pilot project using non-production data to validate your approach before committing to full-scale migration. This gives you a chance to work out issues in a controlled environment and refine your configuration without impacting production systems.
  • Monitoring – Set up comprehensive monitoring through Amazon CloudWatch to track key metrics like Kafka lag, Lambda function errors, and replication latency. Establish alerting thresholds early so you can catch and resolve issues quickly before they impact your migration timeline. Size your MSK cluster based on expected CDC volume and configure Lambda reserved concurrency to handle peak loads during initial data synchronization.
  • Security – For security, use encryption in transit and at rest for both structured data and LOBs, and follow the principle of least privilege when setting up AWS Identity and Access Management (IAM) roles and policies for your MSK cluster, Lambda functions, S3 buckets, and database instances. Document your schema mappings between Oracle and Aurora PostgreSQL-Compatible, including how database records link to their corresponding LOBs in Amazon S3.
  • Testing and preparation – Before you go live, test your failover and recovery procedures thoroughly. Validate scenarios like Lambda function failures, MSK cluster issues, and network connectivity problems to ensure you’re prepared for potential issues. Finally, remember that this streaming architecture maintains eventual consistency between your source and target systems, so there might be brief lag times during high-volume periods. Plan your cutover strategy with this in mind.

Limitations and considerations

Although this solution provides a robust approach for migrating Oracle databases with LOBs to AWS, there are several inherent constraints to understand before implementation.

This solution requires network connectivity between your source Oracle database and AWS environment. For on-premises Oracle databases, you must establish AWS Direct Connect or VPN connectivity before deployment. Network bandwidth directly impacts replication speed and overall migration performance, so your connection must be able to handle the expected volume of CDC events and LOB transfers.

The solution uses Debezium Connector for Oracle as the source connector and Debezium Connector for JDBC as the sink connector. This architecture is specifically designed for your Oracle-to-PostgreSQL migrations. Other database combinations require different connector configurations or might not be supported by the current implementation. Migration throughput is also constrained by your MSK cluster capacity and Lambda concurrency limits. You can also exceed AWS service quotas for large-scale migrations and you might need to request quota increases through AWS Enterprise Support.

Conclusion

In this post, we presented a solution that addresses the critical challenge of migrating your large binary objects from Oracle to AWS by using a streaming architecture that separates LOB storage from structured data. This approach avoids size constraints, reduces Oracle licensing costs, and preserves data integrity throughout extended migration periods.

Ready to transform your Oracle migration strategy? Visit the GitHub repository, where you will find the complete AWS CDK deployment code, configuration files, and step-by-step instructions to get started.

About the authors

Naresh Dhiman

Naresh Dhiman

Naresh is a Sr. Solutions Architect at AWS supporting US federal customers. He has over 25 years of experience as a technology leader and is a recognized inventor with six patents. He specializes in containers, machine learning, and generative AI on AWS.

Archana Sharma

Archana Sharma

Archana is a Sr. Database Specialist Solutions Architect, working with Worldwide Public Sector customers. She has years of experience in relational databases, and is passionate about helping customers in their journey to the AWS Cloud with a focus on database migration and modernization.

Ron Kolwitz

Ron Kolwitz

Ron is a Sr. Solutions Architect supporting US Federal Government Sciences customers including NASA and the Department of Energy. He is especially passionate about aerospace and advancing the use of GenAI and quantum-based technologies for scientific research. In his free time, he enjoys spending time with his family of avid water-skiers.

Karan Lakhwani

Karan Lakhwani

Karan is a Sr. Customer Solutions Manager at Amazon Web Services. He specializes in generative AI technologies and is an AWS Golden Jacket recipient. Outside of work, Karan enjoys finding new restaurants and skiing.

AWS Transform announces full-stack Windows modernization capabilities

Post Syndicated from Prasad Rao original https://aws.amazon.com/blogs/aws/aws-transform-announces-full-stack-windows-modernization-capabilities/

Earlier this year in May, we announced the general availability of AWS Transform for .NET, the first agentic AI service for modernizing .NET applications at scale. During the early adoption period of the service, we received valuable feedback indicating that, in addition to .NET application modernization, you would like to modernize SQL Server and legacy UI frameworks. Your applications typically follow a three-tier architecture—presentation tier, application tier, and database tier—and you need a comprehensive solution that can transform all of these tiers in a coordinated way.

Today, based on your feedback, we’re excited to announce AWS Transform for full-stack Windows modernization, to offload complex, tedious modernization work across the Windows application stack. You can now identify application and database dependencies and modernize them in an orchestrated way through a centralized experience.

AWS Transform accelerates full-stack Windows modernization by up to five times across application, UI, database, and deployment layers. Along with porting .NET Framework applications to cross-platform .NET, it migrates SQL Server databases to Amazon Aurora PostgreSQL-Compatible Edition with intelligent stored procedure conversion and dependent application code refactoring. For validation and testing, AWS Transform deploys applications to Amazon Elastic Compute Cloud (Amazon EC2) Linux or Amazon Elastic Container Service (Amazon ECS), and provides customizable AWS CloudFormation templates and deployment configurations for production use. AWS Transform has also added capabilities to modernize ASP.NET Web Forms UI to Blazor.

There is much to explore, so in this post I’ll provide the first look at AWS Transform for full-stack Windows modernization capabilities across all layers.

Create a full-stack Windows modernization transformation job
AWS Transform connects to your source code repositories and database servers, analyzes application and database dependencies, creates modernization waves, and orchestrates full-stack transformations for each wave.

To get started with AWS Transform, I first complete the onboarding steps outlined in the getting started with AWS Transform user guide. After onboarding, I sign in to the AWS Transform console using my credentials and create a job for full-stack Windows modernization.

Create a new job for Windows Modernization
Create a new job by choosing SQL Server Database Modernization

After creating the job, I complete the prerequisites. Then, I configure the database connector for AWS Transform to securely access SQL Server databases running on Amazon EC2 and Amazon Relational Database Service (Amazon RDS). The connector can connect to multiple databases within the same SQL Server instance.

Create new database connector by adding connector name and AWS Account ID

Next, I set up a connector to connect to my source code repositories.

Add a source code connector by adding Connection name, AWS Account ID and Code Connector Arn

Furthermore, I have the option to choose if I would like AWS Transform to deploy the transformed applications. I choose Yes and provide the target AWS account ID and AWS Region for deploying the applications. The deployment option can be configured later as well.

Choose if you would like to deploy transformed apps

After the connectors are set up, AWS Transform connects to the resources and runs the validation to verify IAM roles, network settings, and related AWS resources.

After the successful validation, AWS Transform discovers databases and their associated source code repositories. It identifies dependencies between databases and applications to create waves for transforming related components together. Based on this analysis, AWS Transform creates a wave-based transformation plan.

Start assessment for discovered database and source code repositories

Assessing database and dependent applications
For the assessment, I review the databases and source code repositories discovered by AWS Transform and choose the appropriate branches for code repositories. AWS Transform scans these databases and source code repositories, then presents a list of databases along with their dependent .NET applications and transformation complexity.

Start wave planning of asessed databases and dependent repositories

I choose the target databases and repositories for modernization. AWS Transform analyzes these selections and generates a comprehensive SQL Modernization Assessment Report with a detailed wave plan. I download the report to review the proposed modernization plan. The report includes an executive summary, wave plan, dependencies between databases and code repositories, and complexity analysis.

View SQL Modernization Assessment Report

Wave transformation at scale
The wave plan generated by AWS Transform consists of four steps for each wave. First, it converts the SQL Server schema to PostgreSQL. Second, it migrates the data. Third, it transforms the dependent .NET application code to make it PostgreSQL compatible. Finally, it deploys the application for testing.

Before converting the SQL Server schema, I can either create a new PostgreSQL database or choose an existing one as the target database.

Choose or create target database

After I choose the source and target databases, AWS Transform generates conversion reports for my review. AWS Transform converts the SQL Server schema to PostgreSQL-compatible structures, including tables, indexes, constraints, and stored procedures.

Download Schema conversion reports

For any schema that AWS Transform can’t automatically convert, I can manually address them in the AWS Database Migration Service (AWS DMS) console. Alternatively, I can fix them in my preferred SQL editor and update the target database instance.

After completing schema conversion, I have the option to proceed with data migration, which is an optional step. AWS Transform uses AWS DMS to migrate data from my SQL Server instance to the PostgreSQL database instance. I can choose to perform data migration later, after completing all transformations, or work with test data by loading it into my target database.

Choose if you would like to migrate data

The next step is code transformation. I specify a target branch for AWS Transform to upload the transformed code artifacts. AWS Transform updates the codebase to make the application compatible with the converted PostgreSQL database.

Specify target branch destination for transformed codebase

With this release, AWS Transform for full-stack Windows modernization supports only codebases in .NET 6 or later. For codebases in .NET Framework 3.1+, I first use AWS Transform for .NET to port them to cross-platform .NET. I’ll expand on this in a following section.

After the conversion is completed, I can view the source and target branches along with their code transformation status. I can also download and review the transformation report.

Download transformation report

Modernizing .NET Framework applications with UI layer
One major feature we’re releasing today is the modernization of UI frameworks from ASP.NET Web Forms to Blazor. This is added to existing support for modernizing model-view-controller (MVC) Razor views to ASP.NET Core Razor views.

As mentioned previously, if I have a .NET application in legacy .NET Framework, then I continue using AWS Transform for .NET to port it to cross-platform .NET. For legacy applications with UIs built on ASP.NET Web Forms, AWS Transform now modernizes the UI layer to Blazor along with porting the backend code.

AWS Transform for .NET converts ASP.NET Web Forms projects to Blazor on ASP.NET Core, facilitating the migration of ASP.NET websites to Linux. The UI modernization feature is enabled by default in AWS Transform for .NET on both the AWS Transform web console and Visual Studio extension.

During the modernization process, AWS Transform handles the conversion of ASPX pages, ASCX custom controls, and code-behind files, implementing them as server-side Blazor components rather than web assembly. The following project and file changes are made during the transformation:

From To Description
*.aspx, *.ascx *.razor .aspx pages and .ascx custom controls become .razor files
Web.config appsettings.json Web.config settings become appsettings.json settings
Global.asax Program.cs Global .asax code becomes Program.cs code
*.master *layout.razor Master files become layout.razor files

Image showcasing how the specific project files are transformed

Other new features in AWS Transform for .NET
Along with UI porting, AWS Transform for .NET has added support for more transformation capabilities and enhanced developer experience. These new features include the following:

  • Port to .NET 10 and .NET Standard – AWS Transform now supports porting to .NET 10, the latest Long-Term Support (LTS) release, which was released on November 11, 2025. It also supports porting class libraries to .NET Standard, a formal specification for a set of APIs that are common across all .NET implementations. Furthermore, AWS Transform is now available with AWS Toolkit for Visual Studio 2026.
  • Editable transformation report – After the assessment is complete, you can now view and customize the transformation plan based on your specific requirements and preferences. For example, you can update package replacement details.
  • Real-time transformation updates with estimated remaining time – Depending on the size and complexity of the codebase, AWS Transform can take some time to complete the porting. You can now track transformation updates in real-time along with the estimated remaining time.
  • Next steps markdown – After the transformation is complete, AWS Transform now generates a next steps markdown file with the remaining tasks to complete the porting. You can use this as a revised plan to repeat the transformation with AWS Transform or use AI code-companions to complete the porting.

Things to know
Some more things to know are:

  • AWS Regions – AWS Transform for full-stack Windows modernization is generally available today in the US East (N. Virginia) Region. For Regional availability and future roadmap, visit the AWS Capabilities by Region.
  • Pricing – Currently, there is no added charge for Windows modernization features of AWS Transform. Any resources you create or continue to use in your AWS account using the output of AWS Transform are billed according to their standard pricing. For limits and quotas, refer to the AWS Transform User Guide.
  • SQL Server versions supported – AWS Transform supports the transformation of SQL Server versions from 2008 R2 through 2022, including all editions (Express, Standard, and Enterprise). SQL Server must be hosted on Amazon RDS or Amazon EC2 in the same Region as AWS Transform.
  • Entity Framework versions supported – AWS Transform supports the modernization of Entity Framework versions 6.3 through 6.5 and Entity Framework Core 1.0 through 8.0.
  • Getting started – To get started, visit AWS Transform for full-stack Windows modernization User Guide.

– Prasad

AWS Weekly Roundup: Amazon Aurora 10th anniversary, Amazon EC2 R8 instances, Amazon Bedrock and more (August 25, 2025)

Post Syndicated from Betty Zheng (郑予彬) original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-aurora-10th-anniversary-amazon-ec2-r8-instances-amazon-bedrock-and-more-august-25-2025/

As I was preparing for this week’s roundup, I couldn’t help but reflect on how database technology has evolved over the past decade. It’s fascinating to see how architectural decisions made years ago continue to shape the way we build modern applications. This week brings a special milestone that perfectly captures this evolution in cloud database innovation as Amazon Aurora celebrated 10 years of database innovation.

Birthday cake with words Happy Birthday Amazon Aurora!

Amazon Web Services (AWS) Vice President Swami Sivasubramanian reflected on LinkedIn about his journey with Amazon Aurora, calling it “one of the most interesting products” he’s worked on. When Aurora launched in 2015, it shifted the database landscape by separating compute and storage. Now trusted by hundreds of thousands of customers across industries, Aurora has grown from a MySQL-compatible database to a comprehensive platform featuring innovations such as Aurora DSQL, serverless capabilities, I/O-Optimized pricing, zero-ETL integrations, and generative AI support. Last week’s celebration on August 21 highlighted this decade-long transformation that continues to simplify database scaling for customers.

Last week’s launches

In addition to the inspiring celebrations, here are some AWS launches that caught my attention:

  • AWS Billing and Cost Management introduces customizable Dashboards — This new feature consolidates cost data into visual dashboards with multiple widget types and visualization options, combining information from Cost Explorer, Savings Plans, and Reserved Instance reports to help organizations track spending patterns and share standardized cost reporting across accounts.
  • Amazon Bedrock simplifies access to OpenAI open weight models — AWS has streamlined access to OpenAI’s open weight models (gpt-oss-120b and gpt-oss-20b), making them automatically available to all users without manual activation while maintaining administrator control through IAM policies and service control policies.
  • Amazon Bedrock adds batch inference support for Claude Sonnet 4 and GPT-OSS models —This feature provides asynchronous processing of multiple inference requests with 50 percent lower pricing compared to on-demand inference, optimizing high-volume AI tasks such as document analysis, content generation, and data extraction with Amazon CloudWatch metrics for tracking batch workload progress
  • AWS launching Amazon EC2 R8i and R8i-flex memory-optimized instances — Powered by custom Intel Xeon 6 processors, these new instances deliver up to 20 percent better performance and 2.5 times higher memory throughput than R7i instances, making them ideal for memory-intensive workloads like databases and big data analytics, with R8i-flex offering additional cost savings for applications that don’t fully utilize compute resources.
  • Amazon S3 introduces batch data verification feature — A new capability in S3 Batch Operations that offers efficient verification of billions of objects using multiple checksum algorithms without downloading or restoring data, generating detailed integrity reports for compliance and audit purposes regardless of storage class or object size.

Other AWS news

Here are some additional projects and blog posts that you might find interesting:

  • Amazon introduces DeepFleet foundation models for multirobot coordination — Trained on millions of hours of data from Amazon fulfillment and sortation centers, these pioneering models predict future traffic patterns for robot fleets, representing the first foundation models specifically designed for coordinating multiple robots in complex environments.
  • Building Strands Agents with a few lines of code — A new blog demonstrates how to build multi-agent AI systems with a few lines of code, enabling specialized agents to collaborate seamlessly, handle complex workflows, and share information through standardized protocols for creating distributed AI systems beyond individual agent capabilities.
  • AWS Security Incident Response introduces ITSM integrations — New integrations with Jira and ServiceNow provide bidirectional synchronization of security incidents, comments, and attachments, streamlining response while maintaining existing processes, with open source code available on GitHub for customization and extension to additional IT service management (ITSM) platforms.
  • Finding root-causes using a network digital twin graph and agentic AI — A detailed blog post shows how AWS collaborated with NTT DOCOMO to build a network digital twin using graph databases and autonomous AI agents, helping telecom operators to move beyond correlation to identify true root causes of complex network issues, predict future problems, and improve overall service reliability.

Upcoming AWS events
Check your calendars and sign up for these upcoming AWS events:

  • AWS Summits — Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Toronto (September 4), Los Angeles (September 17), and Bogotá (October 9).
  • AWS re:Invent 2025 — This flagship annual conference is coming to Las Vegas from December 1–5. The event catalog is now available. Mark your calendars for this not to be missed gathering of the AWS community.
  • AWS Community Days — Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Adria (September 5), Baltic (September 10), Aotearoa (September 18), South Africa (September 20), Bolivia (September 20), Portugal (September 27).

Join the AWS Builder Center to learn, build, and connect with builders in the AWS community. Browse here for upcoming in-person and virtual developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Betty

AWS Weekly Roundup: Single GPU P5 instances, Advanced Go Driver, Amazon SageMaker HyperPod and more (August 18, 2025)

Post Syndicated from Prasad Rao original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-single-gpu-p5-instances-advanced-go-driver-amazon-sagemaker-hyperpod-and-more-august-18-2025/

Let me start this week’s update with something I’m especially excited about – the upcoming BeSA (Become a Solutions Architect) cohort. BeSA is a free mentoring program that I host along with a few other AWS employees on a volunteer basis to help people excel in their cloud careers. Last week, the instructors’ lineup was finalized for the 6-week cohort starting September 6. The cohort will focus on migration and modernization on AWS. Visit the BeSA website to learn more.

Another highlight for me last week was the announcement of six new AWS Heroes for their technical leadership and exceptional contributions to the AWS community. Read the full announcement to learn more about these community leaders.

Last week’s launches
Here are some launches from last week that got my attention:

  • Amazon EC2 Single GPU P5 instances are now generally available — You can right-size your machine learning (ML) and high performance computing (HPC) resources cost-effectively with the new Amazon Elastic Compute Cloud (Amazon EC2) P5 instance size with one NVIDIA H100 GPU.
  • AWS Advanced Go Driver is generally available — You can now use the AWS Advanced Go Driver with Amazon Relational Database Service (Amazon RDS) and Amazon Aurora PostgreSQL-Compatible and MySQL-Compatible database clusters for faster switchover and failover times, Federated Authentication, and authentication with AWS Secrets Manager or AWS Identity and Access Management (IAM). You can install the PostgreSQL and MySQL packages for Windows, Mac, or Linux, by following the installation guides in GitHub.
  • Expanded support for Cilium with Amazon EKS Hybrid Nodes — Cilium is a Cloud Native Computing Foundation (CNCF) graduated project that provides core networking capabilities for Kubernetes workloads. Now, you can receive support from AWS for a broader set of Cilium features when using Cilium with Amazon EKS Hybrid Nodes including application ingress, in-cluster load balancing, Kubernetes network policies, and kube-proxy replacement mode.
  • Amazon SageMaker AI now supports P6e-GB200 UltraServers — You can accelerate training and deployment of foundational models (FMs) at trillion-parameter scale by using up to 72 NVIDIA Blackwell GPUs under one NVLink domain with the new P6e-GB200 UltraServer support in Amazon SageMaker HyperPod and Model Training.
  • Amazon SageMaker HyperPod now supports fine-grained quota allocation of compute resources, topology-aware-scheduling of LLM tasks and custom Amazon Machine Images (AMIs) — You can allocate fine-grained compute quota for GPU, Trainium accelerator, vCPU, and vCPU memory within an instance to optimize compute resource distribution. With topology-aware scheduling, you can schedule your large language model (LLM) tasks on an optimal network topology to minimize network communication and enhance training efficiency. Using custom AMIs, you can deploy clusters with pre-configured, security-hardened environments that meet your specific organizational requirements.

Additional updates
Here are some additional news items and blog posts that I found interesting:

Upcoming AWS events
Check your calendars and sign up for upcoming AWS and AWS Community events:

  • AWS re:Invent 2025 (December 1-5, 2025, Las Vegas) — The AWS flagship annual conference offering collaborative innovation through peer-to-peer learning, expert-led discussions, and invaluable networking opportunities.
  • AWS Summits — Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Coming up soon are summits in Johannesburg (August 20) and Toronto (September 4).
  • AWS Community Days — Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Adria (September 5), Baltic (September 10), Aotearoa (September 18), and South Africa (September 20).

Join the AWS Builder Center to learn, build, and connect with builders in the AWS community. Browse here for upcoming in-person and virtual developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Prasad

Celebrating 10 years of Amazon Aurora innovation

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/celebrating-10-years-of-amazon-aurora-innovation/

Ten years ago, we announced the general availability of Amazon Aurora, a database that combined the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases.

As Jeff described it in its launch blog post: “With storage replicated both within and across three Availability Zones, along with an update model driven by quorum writes, Amazon Aurora is designed to deliver high performance and 99.99% availability while easily and efficiently scaling to up to 64 TiB of storage.”

When we started developing Aurora over a decade ago, we made a fundamental architectural decision that would change the database landscape forever: we decoupled storage from compute. This novel approach enabled Aurora to deliver the performance and availability of commercial databases at one-tenth the cost.

This is one of the reasons why hundreds of thousands of AWS customers choose Aurora as their relational database.

Today, I’m excited to invite you to join us for a livestream event on August 21, 2025, to celebrate a decade of Aurora database innovation.

A brief look back at the past
Throughout the evolution of Aurora, we’ve focused on four core innovation themes: security as our top priority, scalability to meet growing workloads, predictable pricing for better cost management, and multi-Region capabilities for global applications. Let me walk you through some key milestones in the Aurora journey.

Aurora Innovtion with Matt Garman

We previewed Aurora at re:Invent 2014, and made it generally available in July 2015. At launch, we presented Aurora as “a new cost-effective MySQL-compatible database engine.”

In June 2016, we introduced reader endpoints and cross-Region read replicas, followed by AWS Lambda integration and the ability to load tables directly from Amazon S3 in October. We added database cloning and export to Amazon S3 capabilities in June 2017 and full compatibility with PostgreSQL in October that year.

The journey continued with the serverless preview in November 2017, which became generally available in August 2018. Global Database launched in November 2018 for cross-Region disaster recovery. We introduced blue/green deployments to simplify database updates, and optimized read instances to improve query performance.

In 2023, we added vector capabilities with pgvector for similarity search for Aurora PostgreSQL, and Aurora I/O-Optimized to provide predictable pricing with up to 40 percent cost savings for I/O-intensive applications. We launched Aurora zero-ETL integration with Amazon Redshift which enables near real-time analytics and ML using Amazon Redshift on petabytes of transactional data from Aurora by removing the need for you to build and maintain complex data pipelines that perform extract, transform, and load (ETL) operations. This year we added Aurora MySQL zero-ETL integration with Amazon Sagemaker, enabling near real-time access of your data in the lakehouse architecture of SageMaker to run a broad range of analytics.

In 2024, we made it as effortless as just one click to select Aurora PostgreSQL as a vector store for Amazon Bedrock Knowledge Bases and launched Aurora PostgreSQL Limitless Database, a serverless horizontal scaling (sharding) capability.

To simplify scaling for customers, we also increased the maximum storage to 128 TiB in September 2020, allowing many applications to operate within a single instance. Last month, we’ve further simplified scaling by doubling the maximum storage to 256 TiB, with no upfront provisioning required and pay-as-you-go pricing based on actual storage used. This enables even more customers to run their growing workloads without the complexity of managing multiple instances while maintaining cost efficiency.

Most recently, at re:Invent 2024, we announced Amazon Aurora DSQL, which became generally available in May 2025. Aurora DSQL represents our latest innovation in distributed SQL databases, offering active-active high availability and multi-Region strong consistency. It’s the fastest serverless distributed SQL database for always available applications, effortlessly scaling to meet any workload demand with zero infrastructure management.

Aurora DSQL builds on our original architectural principles of separation of storage and compute, taking them further with independent scaling of reads, writes, compute, and storage. It provides 99.99% single-Region and 99.999% multi-Region availability, with strong consistency across all Regional endpoints.

Matt Garman introduces Amazon Aurora DSQL

And in June, we launched Model Context Protocol (MCP) servers for Aurora, so you can integrate your AI agents with your data sources and services.

Let’s celebrate 10 years of innovation
Birthday cake with words Happy Birthday Amazon Aurora!By attending the August 21 livestream event, you’ll hear from Aurora technical leaders and founders, including Swami Sivasubramanian, Ganapathy (G2) Krishnamoorthy, Yan Leshinsky, Grant McAlister, and Raman Mittal. You’ll learn directly from the architects who pioneered the separation of compute and storage in cloud databases, with technical insights into Aurora architecture and scaling capabilities. You’ll also get a glimpse into the future of database technology as Aurora engineers share their vision and discuss the complex challenges they’re working to solve on behalf of customers.

The event also offers practical demonstrations that show you how to implement key features. You’ll see how to build AI-powered applications using pgvector, understand cost optimization with the new Aurora DSQL pricing model, and learn how to achieve multi-Region strong consistency for global applications.

The interactive format includes Q&A opportunities with Aurora experts, so you’ll be able to get your specific technical questions answered. You can also receive AWS credits to test new Aurora capabilities.

If you’re interested in agentic AI, you’ll particularly benefit from the sessions on MCP servers, Strands Agents, and how to integrate Strands Agents with Aurora DSQL, which demonstrate how to safely integrate AI capabilities with your Aurora databases while maintaining control over database access.

Whether you’re running mission-critical workloads or building new applications, these sessions will help you understand how to use the latest Aurora features.

Register today to secure your spot and be part of this celebration of database innovation.

To the next decade of Aurora innovation!

— seb

Integrating Amazon OpenSearch Ingestion with Amazon RDS and Amazon Aurora

Post Syndicated from Michael Torio original https://aws.amazon.com/blogs/big-data/integrating-amazon-opensearch-ingestion-with-amazon-rds-and-amazon-aurora/

Unlocking powerful search capabilities for millions of items should be fast, accurate, and effortless while maintaining high relevance. Relational databases are a popular storage method for structured data, and organizations use them extensively to store their core business information. Although relational databases excel at storing and retrieving structured data, they often struggle with searching through large blocks of unstructured text and, for performance reasons, typically don’t index all columns.

In contrast, search engines such as OpenSearch index all fields, enabling rich search capabilities, including semantic search, and powerful aggregations for summarizing and analyzing numeric data. Traditionally, organizations have managed complex, inefficient, and expensive data synchronization processes, including extract, transform, and load (ETL) pipelines, to keep their search indices up to date with their databases. Those looking to enhance their applications with advanced search features need a simpler solution that can maintain search index synchronization with their databases without the overhead of managing custom data sync processes.

We are happy to announce the general availability of the integration of Amazon OpenSearch Service with Amazon Relational Database Service (Amazon RDS) and Amazon Aurora. This new integration eliminates complex data pipelines and enables near real-time data synchronization between Amazon Aurora (including Amazon Aurora MySQL-Compatible Edition and Amazon Aurora PostgreSQL-Compatible Edition) and Amazon RDS databases (including Amazon RDS for MySQL and Amazon RDS for PostgreSQL), and Amazon OpenSearch Service, unlocking advanced search capabilities such as hybrid search, ranked results, and faceted search on transactional databases. You can now deliver low-latency, high-throughput search results, live inventory updates, and personalized recommendations while focusing on creating exceptional customer experiences instead of managing data synchronization. This integration reduces the operational burden of maintaining complex ETL pipelines, reducing costs while providing instant data availability for search operations.

Amazon OpenSearch Ingestion provides near real-time data synchronization between Amazon Aurora or Amazon RDS and OpenSearch Service. Select your Aurora or RDS database, and OpenSearch Ingestion handles the rest, supporting both Aurora MySQL or RDS for MySQL (8.0 and above) and Aurora PostgreSQL or RDS for PostgreSQL (16 and above).

Solution overview

Here’s how these services work together:

  • Data ingestion – OpenSearch Ingestion first loads your database snapshot from Amazon Simple Storage Service (Amazon S3), where Aurora or Amazon RDS has exported the initial data. It then uses Aurora or Amazon RDS change data capture (CDC) streams to replicate further changes in near real time and indexes them into OpenSearch Service. This automated process keeps your data is consistently up to date in OpenSearch, making it readily available for search and analysis without manual intervention.
  • Real-time querying – OpenSearch Service offers powerful query capabilities that enable you to perform complex searches and aggregations on your data. Whether you need to analyze trends, detect anomalies, or perform search queries to return relevant results for your application, OpenSearch Service provides the tools you need.

The following diagram illustrates the solution architecture for Amazon Aurora as a source:

A diagram of a processAI-generated content may be incorrect.

Getting Started

Configuring Your Database Source

Before setting up synchronization, you need to configure your source database’s logging settings. For Aurora MySQL, configure your cluster parameter group with enhanced binary log settings. For Amazon RDS, enable basic binary logging or logical replication through your instance parameter group settings. These logging configurations enable OpenSearch Ingestion to capture and replicate data changes from your database.

The sample HR database with Aurora MySQL is a good example to show how this integration works.

Before creating the view, we now explain how OpenSearch will represent this data. OpenSearch mappings define how documents and their fields are stored and indexed, similar to how a database schema defines tables and columns. The OpenSearch Ingestion pipeline uses dynamic mappings by default, automatically converting Aurora or Amazon RDS data types to appropriate OpenSearch field types. For example, database DATE fields become OpenSearch date types, and numeric fields are mapped to corresponding OpenSearch numeric types. Although you can customize these mappings using index templates, the default mappings typically handle common data types correctly, including dates, numbers, and text fields.

GET employees/_mapping

To demonstrate the integration’s ability to handle complex data relationships, we now examine how OpenSearch Ingestion handles joined data. We create a view in the sample HR database that combines information from multiple related tables into a single, searchable document in OpenSearch. This approach shows how you can transform normalized database structures into denormalized documents that are optimized for search operations.

This employee_details view combines data from multiple tables, creating a rich, denormalized representation of employee information. When replicated to OpenSearch, this view becomes a single, comprehensive document for each employee. This structure is ideal for search operations, allowing for fast and complex queries across what were originally separate tables. For example, you could easily search for employees in a specific department and country or analyze salary distributions across regions—queries that would be more complex and potentially slower in the original normalized database structure.

In the pipeline configuration shown in the following screenshot, you can check how OpenSearch Ingestion connects to the HR database. The configuration identifies the source database and the specific tables we want to replicate. While we created a view to understand the data relationships, the pipeline tracks changes from the underlying base tables (employees, departments, locations, and regions). OpenSearch Ingestion automatically maintains these relationships, which means that changes to these tables are properly reflected in your OpenSearch index, keeping your search data consistent with your source database.

In the gif shown below, you can see a demo of setting up this integration using the visual editor of OpenSearch Ingestion.

You can also specify index mapping templates to map your Aurora or Amazon RDS fields to the correct fields in your OpenSearch Service indexes.

For a comprehensive overview of configuration settings for the pipeline, refer to the OpenSearch Data Prepper documentation. You must set up AWS Identity and Access Management (IAM) roles for the pipeline. For instructions, refer to Configure the pipeline role.

After you configure the integration in OpenSearch Ingestion, the pipeline automatically creates indexes that you can view in OpenSearch Dashboards. OpenSearch Ingestion first triggers an automatic export of your Aurora or Amazon RDS database to Amazon S3, then loads this snapshot data from S3 into your OpenSearch cluster to create the initial indices. After this initial load, OpenSearch Ingestion continually captures changes using binary logs (binlog) for MySQL-based databases or write-ahead logs (WAL) for PostgreSQL-based databases. This way, your OpenSearch indices stay synchronized with your source database in near real time. You can view your indices in OpenSearch Dashboards by invoking:

GET _cat/indices

Example response:

Demonstrating near real time data synchronization

Consider the first five entries in the employee table:

When you make changes to your database, OpenSearch Ingestion updates Amazon OpenSearch Service with the change data. For example, the following code updates an employee’s salary:

UPDATE hr.employees SET SALARY = 26000 WHERE EMPLOYEE_ID = 100;

Amazon Aurora sends out a change notice, your OpenSearch Ingestion pipeline picks it up, and OpenSearch Ingestion sends the changed record to OpenSearch in near real time. You can verify this with an OpenSearch query:

GET employees/_search

Important details about this feature:

  • Monitoring – Track pipeline performance and data synchronization through CloudWatch metrics and the OpenSearch Ingestion dashboard
  • Limitations – Requires same-Region and same-account deployment, primary keys for optimal synchronization, and currently has no data definition language (DDL) statement support

Conclusion

Amazon Aurora or Amazon RDS integration with Amazon OpenSearch Service is now generally available in all AWS Regions where OpenSearch Ingestion is available.

To learn more, refer to the AWS documentation for Aurora or Amazon RDS integration with Amazon OpenSearch Service:

About the authors

Michael Torio is an Associate Specialist Solutions Architect at AWS focused on Amazon OpenSearch Service based out of Mountain View, CA. Michael enjoys helping customers leverage cloud technologies to solve their business challenges.

Sohaib Katariwala is a Senior Specialist Solutions Architect at AWS focused on Amazon OpenSearch Service based out of Chicago, IL. His interests are in all things data and analytics. More specifically he loves to help customers use AI in their data strategy to solve modern day challenges.

Arjun Nambiar is a Product Manager with Amazon OpenSearch Service. He focuses on ingestion technologies that enable ingesting data from a wide variety of sources into Amazon OpenSearch Service at scale. Arjun is interested in large-scale distributed systems and cloud-centered technologies, and is based out of Seattle, Washington.

AWS Weekly Roundup: Amazon Aurora DSQL, MCP Servers, Amazon FSx, AI on EKS, and more (June 2, 2025)

Post Syndicated from Prasad Rao original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-aurora-dsql-mcp-servers-amazon-fsx-ai-on-eks-and-more-june-2-2025/

It’s AWS Summit Season! AWS Summits are free in-person events that take place across the globe in major cities, bringing cloud expertise to local communities. Each AWS Summit features keynote presentations highlighting the latest innovations, technical sessions, live demos, and interactive workshops led by Amazon Web Services (AWS) experts. Last week, events took place at AWS Summit Tel Aviv and AWS Summit Singapore.

The following photo shows the packed keynote at AWS Summit Tel Aviv.

AWS Summit Tel Aviv Keynote

Find an AWS Summit near you and join thousands of AWS customers and cloud professionals taking the next step in their cloud journey.

Last week, the announcement that piqued my interest most was the general availability of Amazon Aurora DSQL, which was introduced in preview at re:Invent 2024. Aurora DSQL is the fastest serverless distributed SQL database that enables you to build always available applications with virtually unlimited scalability, the highest availability, and zero infrastructure management.

Aurora DSQL active-active distributed architecture is designed for 99.99% single-Region and 99.999% multi-Region availability with no single point of failure and automated failure recovery. This means your applications can continue to read and write with strong consistency, even in the rare case an application is unable to connect to a Region cluster endpoint.

Single and multi region deployment of Amazon Aurora DSQL

What’s more fascinating is the journey behind building Aurora DSQL, a story that goes beyond the technology in the pursuit of engineering efficiency. Read the full story in Dr. Werner Vogels’ blog post, Just make it scale: An Aurora DSQL story.

Last week’s launches
Here are the other launches that got my attention:

  • Announcing new Model Context Protocol (MCP) servers for AWS Serverless and Containers – MCP servers are now available for AWS Lambda, Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and Finch. With MCP servers, you can get from idea to production faster by giving your AI assistants access to an up-to-date framework on how to correctly interact with your AWS service of choice. To download and try out the open source MCP servers, visit the aws-labs GitHub repository.
  • Announcing the general availability of Amazon FSx for Lustre Intelligent-Tiering – FSx for Lustre Intelligent-Tiering, a new storage class, automatically optimizes costs by tiering cold data to the applicable lower-cost storage tier based on access patterns and includes an optional SSD read cache to improve performance for your most latency-sensitive workloads.
  • Amazon FSx for NetApp ONTAP now supports write-back mode for ONTAP FlexCache volumes – Write-back mode is a new ONTAP capability that helps you achieve faster performance for your write-intensive workloads that are distributed across multiple AWS Regions and on-premises file systems.
  • AWS Network Firewall Adds Support for Multiple VPC Endpoints – AWS Network Firewall now supports configuring up to 50 Amazon Virtual Private Cloud (Amazon VPC) endpoints per Availability Zone for a single firewall. This new capability gives you more options to scale your Network Firewall deployment across multiple VPCs, using a centralized security policy.
  • Cost Optimization Hub now supports Savings Plans and reservations preferences – You can now use Cost Optimization Hub, a feature within the Billing and Cost Management Console, to configure preferred Savings Plans and reservation term and payment options preferences, so you can see your resulting recommendations and savings potential based on your preferred commitments.
  • AWS Neuron introduces NxD Inference GA, new features, and improved tools – With the release of Neuron 2.23, the NxD Inference library (NxDI) moves from beta to general availability and is now recommended for all multi-chip inference use cases. Neuron 2.23 also introduces new training capabilities, including context parallelism and Odds Ratio Preference Optimization (ORPO), and adds support for PyTorch 2.6 and JAX 0.5.3.
  • AWS Pricing Calculator, now generally available, supports discounts and purchase commitment – We announced the general availability of the AWS Pricing Calculator in the AWS console. You can now create more accurate and comprehensive cost estimates by providing two types of cost estimates: cost estimation for a workload, and estimation of a full AWS bill. You can also import your historical usage or create net new usage when creating a cost estimate. Additionally, with the new rate configuration inclusive of both pricing discounts and purchase commitments, you can gain a clearer picture of potential savings and cost optimizations for your cost scenarios.
  • AWS CDK Toolkit Library is now generally available – AWS CDK Toolkit Library provides programmatic access to core AWS CDK functionalities such as synthesis, deployment, and destruction of stacks. You can use this library to integrate CDK operations directly into your applications, custom CLIs, and automation workflows, offering greater flexibility and control over infrastructure management.
  • Announcing Red Hat Enterprise Linux for AWS – Red Hat Enterprise Linux (RHEL) for AWS, starting with RHEL 10, is now generally available, combining Red Hat’s enterprise-grade Linux software with native AWS integration. RHEL for AWS is built to achieve optimum performance of RHEL running on AWS.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS? page.

Additional updates
Here are some additional projects, blog posts, and news items that you might find interesting:

  • Introducing AI on EKS: powering scalable AI workloads with Amazon EKS – AI on EKS is a new open source initiative from AWS designed to help you deploy, scale, and optimize AI/ML workloads on Amazon EKS. AI on EKS repository includes deployment-ready blueprints for distributed training, LLM inference, generative AI pipelines, multi-model serving, agentic AI, GPU and Neuron-specific benchmarks, and MLOps best practices.
  • Revolutionizing earth observation with geospatial foundation models on AWS – Emerging transformer-based vision models for geospatial data—also called geospatial foundation models (GeoFMs)—offer a new and powerful technology for mapping the earth’s surface at a continental scale. This post explores how Clay Foundation’s Clay foundation model can be deployed for large-scale inference and fine-tuning on Amazon SageMaker. You can use the ready-to-deploy code samples to get started quickly with deploying GeoFMs in your own applications on AWS.

High level solution flow for inference and fine tuning using Geospatial Foundation Models

  • Going beyond AI assistants: Examples from Amazon.com reinventing industries with generative AI – Non-conversational applications offer unique advantages, such as higher latency tolerance, batch processing, and caching, but their autonomous nature requires stronger guardrails and exhaustive quality assurance compared to conversational applications, which benefit from real-time user feedback and supervision. This post examines four diverse Amazon.com examples of non-conversational generative AI applications.

Upcoming AWS events
Check your calendars and sign up for these upcoming AWS events:

  • AWS Summits – Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Stockholm (June 4), Sydney (June 4–5), Hamburg (June 5), Washington (June 10–11), Madrid (June 11), Milan (June 18), Shanghai (June 19–20), and Mumbai (June 19).
  • AWS re:Inforce – Mark your calendars for AWS re:Inforce (June 16–18) in Philadelphia, PA. AWS re:Inforce is a learning conference focused on AWS security solutions, cloud security, compliance, and identity.
  • AWS Community Days – Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Milwaukee, USA (June 5), Mexico (June 14), Nairobi, Kenya (June 14), and Colombia (June 28).

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Prasad