Eclipse Dataspace Components on AWS: Architecture patterns in production

Post Syndicated from Jonas Bürkel original https://aws.amazon.com/blogs/architecture/eclipse-dataspace-components-on-aws-architecture-patterns-in-production/

Running Eclipse Dataspace Components (EDC) connectors in production on AWS requires deliberate architecture decisions around isolation, managed services, and security layering. In Part 1 of this series, we covered the fundamentals of data space architectures and EDC per the International Data Space Association’s (IDSA) standards. If you are new to EDC, we recommend starting there. We showed how connector functionality can be customized to support native integration with Amazon Web Services (AWS) Cloud services. Examples include Amazon Simple Storage Service (Amazon S3) for data storage and AWS Secrets Manager for credentials management. In this post, we dive deeper into the connector deployment architecture on AWS and present patterns and practices for production-grade deployments.

Fundamental architecture building blocks

The EDC connector consists of a control plane and a data plane that customers typically ship and deploy as containers. Depending on data integration requirements and support for specific protocols and capabilities, a custom EDC build process may need to be implemented as described in Part 1 of this series. For example, you may need OAuth 2.0 client credentials for the data plane to connect to backend systems. You store the resulting EDC container images in a container registry, such as Amazon Elastic Container Registry (Amazon ECR). Figure 1 shows an example architecture for EDC connector deployments on AWS following best practices for production use.

Architecture diagram showing production-ready EDC connector deployment on AWS with Amazon ECS, Aurora, S3, and API Gateway components

Figure 1: Production-ready EDC connector deployment on AWS

You can split the architecture into four sub-components:

Amazon Elastic Container Service (Amazon ECS) and AWS Fargate provide serverless container orchestration. This allows for scalable EDC deployment without managing any of the underlying infrastructure.
EDC requires persistence to store secrets and relational control plane data, and a means of vending OAuth 2.0 client credentials. AWS Secrets Manager, Amazon Aurora and Amazon Cognito can provide these capabilities as managed services.
Amazon S3 provides durable data storage for handling both inbound and outbound data that is shared and received through the data space.
Finally, Amazon API Gateway and Network Load Balancer provide secure, private network connectivity to EDC APIs in an isolated Amazon Virtual Private Cloud (Amazon VPC) using VPC links.

With this approach, all cloud resources belonging to a single EDC connector instance form an isolated architecture cell. You access this cell through the S3 bucket to move in data that is to be shared as part of an EDC asset, or to retrieve data received from a third party as part of an EDC data transfer. Secondly, the API Gateway can be configured to expose selected EDC API resources from its management API, data plane API and Dataspace Protocol (DSP) API. You can protect both means of interacting with the EDC architecture cell using AWS Identity and Access Management (AWS IAM) and the AWS Signature Version 4 (SigV4) protocol.

Larger enterprises participating in data spaces may decide to operate multiple EDCs depending on their requirements on failure isolation, data governance, and separation of shared and received data. A common pattern is to deploy separate connector instances per use case. Infrastructure-as-code such as AWS Cloud Development Kit (CDK) allows for automated, templatized deployment and management of EDC connectors over time while keeping operational effort at bay. Using the Dataspace Connector on AWS reference implementation, a full connector cell deploys from a single CDK command, ready to negotiate contracts and transfer data. Amazon API Gateway also comes with Model Context Protocol (MCP) proxy support. This allows EDC APIs to be consumed by authorized AI agents and MCP clients for autonomous data collection and sharing. Besides integration with agentic systems, customers often follow a workflow-based approach for connecting EDCs with their cloud-based data environments. They interact with both APIs and the peripheral S3 bucket to securely expose and retrieve external information.

Real-world validation of these architecture patterns can be seen in production deployments like the Prometheus-X Data Space Connector, for education sector use cases. This implementation uses the same core architecture we recommend: Amazon ECS with AWS Fargate for container orchestration, S3 for data storage, and event-driven processing with AWS Lambda and Amazon EventBridge. This demonstrates how these patterns work effectively in production environments across different industry sectors.

Key principles for production-readiness

We discuss some of the architecture principles that inform the best practices diagram highlighted in Figure 1 along three of the AWS Well-Architected Framework’s pillars.

Operational Excellence

Infrastructure as Code for Consistency: Define all infrastructure declaratively to support repeatable, version-controlled, and testable deployments. Automated validation, for example using CDK Nag, helps catch misconfigurations and security issues before deployment, shifting security left in the development lifecycle. The code itself serves as living documentation of the architecture.

Observability as a First-Class Concern: Treat monitoring and logging as core infrastructure components. Amazon CloudWatch Container Insights, Amazon CloudWatch Logs, and EDC’s structured health check endpoints provide visibility into system behavior, supporting proactive issue detection and faster troubleshooting. EDC APIs for health checks can be similarly exposed with restricted access through API Gateway and IAM.

Managed Services Over Self-Managed Infrastructure: Use AWS managed services (Aurora, Secrets Manager, Fargate, Cognito) instead of deploying and maintaining compatible self-managed solutions. This shifts undifferentiated heavy lifting to AWS and reduces operational burden. You gain high availability, built-in security best practices, compliance certifications, and automatic updates.

Security

Defense in Depth: Implement security through multiple independent layers rather than relying on a single control. Network isolation (VPC private subnets), security group segmentation (restricting traffic between components), IAM least privilege (scoped permissions per service), and encryption (at rest and in transit) each provide independent controls. These layers work together so that if one layer is bypassed, others continue to provide protection.

Principle of Least Privilege: Every component receives only the minimum permissions required for its specific function. Scope IAM roles to individual services (control plane, data plane) and restrict security groups to necessary ports and sources. The internal-only Network Load Balancer fronted by API Gateway prevents unintended public exposure of EDC APIs and data. This may also support security review and approval of EDC as open-source software, since APIs can be allowlisted and validated individually.

Encryption Everywhere: Encrypt data by default at every stage: at rest (Aurora, S3, Secrets Manager), in transit (TLS enforcement, HTTPS-only egress), and during processing (encrypted environment variables). This provides comprehensive data protection regardless of where information resides in the system.

Reliability

Fail Fast, Recover Automatically: Systems detect failures quickly and recover without manual intervention. ECS circuit breakers can automatically roll back failed deployments, automated health checks remove unhealthy targets, and point-in-time recovery supports rapid database restoration. This minimizes mean time to recovery (MTTR) and reduces the scope of failures, even within a single EDC architecture cell.

Design for Regional Resilience: Cross-zone load balancing distributes traffic across multiple Availability Zones (AZs), Aurora automatically replicates data across AZs, and Fargate tasks can be scheduled in any AZ. The highlighted architecture can tolerate Availability Zone failures in an AWS Region without service disruption. For more information about Availability Zones and Regions, see AWS Global Infrastructure.

Decoupled Components with Clear Boundaries: Deploy the control plane and data plane as independent services with distinct responsibilities, security contexts, and scaling characteristics. This separation enables independent updates, targeted scaling, and failure isolation between coordination logic and data transfer operations.

The remaining three Well-Architected Framework pillars of Performance Efficiency, Cost Optimization, and Sustainability are covered in the third post of this series where we discuss cost optimization strategies for running EDC connectors on AWS.

Conclusion

With the growing popularity of data spaces and EDC as a data space connector, it is important to distinguish between a setup suitable for testing and experimentation and one that is ready for production. Production environments require that business-critical processes depend on timely, successful transmission of confidential information between participants. The architecture defined in this post combines EDC deployment best practices from the community with AWS recommendations to achieve fault tolerance, scalability, and security while keeping operational complexity at a minimum.

In part 3, you will learn about cost optimization strategies to run your production-ready connector efficiently and maximize the value it returns by supporting business use cases for data sharing along your supply network. In the meantime, explore the Dataspace Connector on AWS project and see how the patterns and best practices covered in this post come together in an end-to-end reference implementation.

Noise