All posts by Jonas Bürkel

Eclipse Dataspace Components on AWS: Architecture patterns in production

Post Syndicated from Jonas Bürkel original https://aws.amazon.com/blogs/architecture/eclipse-dataspace-components-on-aws-architecture-patterns-in-production/

Running Eclipse Dataspace Components (EDC) connectors in production on AWS requires deliberate architecture decisions around isolation, managed services, and security layering. In Part 1 of this series, we covered the fundamentals of data space architectures and EDC per the International Data Space Association’s (IDSA) standards. If you are new to EDC, we recommend starting there. We showed how connector functionality can be customized to support native integration with Amazon Web Services (AWS) Cloud services. Examples include Amazon Simple Storage Service (Amazon S3) for data storage and AWS Secrets Manager for credentials management. In this post, we dive deeper into the connector deployment architecture on AWS and present patterns and practices for production-grade deployments.

Fundamental architecture building blocks

The EDC connector consists of a control plane and a data plane that customers typically ship and deploy as containers. Depending on data integration requirements and support for specific protocols and capabilities, a custom EDC build process may need to be implemented as described in Part 1 of this series. For example, you may need OAuth 2.0 client credentials for the data plane to connect to backend systems. You store the resulting EDC container images in a container registry, such as Amazon Elastic Container Registry (Amazon ECR). Figure 1 shows an example architecture for EDC connector deployments on AWS following best practices for production use.

Architecture diagram showing production-ready EDC connector deployment on AWS with Amazon ECS, Aurora, S3, and API Gateway components

Figure 1: Production-ready EDC connector deployment on AWS

You can split the architecture into four sub-components:

  • Amazon Elastic Container Service (Amazon ECS) and AWS Fargate provide serverless container orchestration. This allows for scalable EDC deployment without managing any of the underlying infrastructure.
  • EDC requires persistence to store secrets and relational control plane data, and a means of vending OAuth 2.0 client credentials. AWS Secrets Manager, Amazon Aurora and Amazon Cognito can provide these capabilities as managed services.
  • Amazon S3 provides durable data storage for handling both inbound and outbound data that is shared and received through the data space.
  • Finally, Amazon API Gateway and Network Load Balancer provide secure, private network connectivity to EDC APIs in an isolated Amazon Virtual Private Cloud (Amazon VPC) using VPC links.

With this approach, all cloud resources belonging to a single EDC connector instance form an isolated architecture cell. You access this cell through the S3 bucket to move in data that is to be shared as part of an EDC asset, or to retrieve data received from a third party as part of an EDC data transfer. Secondly, the API Gateway can be configured to expose selected EDC API resources from its management API, data plane API and Dataspace Protocol (DSP) API. You can protect both means of interacting with the EDC architecture cell using AWS Identity and Access Management (AWS IAM) and the AWS Signature Version 4 (SigV4) protocol.

Larger enterprises participating in data spaces may decide to operate multiple EDCs depending on their requirements on failure isolation, data governance, and separation of shared and received data. A common pattern is to deploy separate connector instances per use case. Infrastructure-as-code such as AWS Cloud Development Kit (CDK) allows for automated, templatized deployment and management of EDC connectors over time while keeping operational effort at bay. Using the Dataspace Connector on AWS reference implementation, a full connector cell deploys from a single CDK command, ready to negotiate contracts and transfer data. Amazon API Gateway also comes with Model Context Protocol (MCP) proxy support. This allows EDC APIs to be consumed by authorized AI agents and MCP clients for autonomous data collection and sharing. Besides integration with agentic systems, customers often follow a workflow-based approach for connecting EDCs with their cloud-based data environments. They interact with both APIs and the peripheral S3 bucket to securely expose and retrieve external information.

Real-world validation of these architecture patterns can be seen in production deployments like the Prometheus-X Data Space Connector, for education sector use cases. This implementation uses the same core architecture we recommend: Amazon ECS with AWS Fargate for container orchestration, S3 for data storage, and event-driven processing with AWS Lambda and Amazon EventBridge. This demonstrates how these patterns work effectively in production environments across different industry sectors.

Key principles for production-readiness

We discuss some of the architecture principles that inform the best practices diagram highlighted in Figure 1 along three of the AWS Well-Architected Framework’s pillars.

Operational Excellence

Infrastructure as Code for Consistency: Define all infrastructure declaratively to support repeatable, version-controlled, and testable deployments. Automated validation, for example using CDK Nag, helps catch misconfigurations and security issues before deployment, shifting security left in the development lifecycle. The code itself serves as living documentation of the architecture.

Observability as a First-Class Concern: Treat monitoring and logging as core infrastructure components. Amazon CloudWatch Container Insights, Amazon CloudWatch Logs, and EDC’s structured health check endpoints provide visibility into system behavior, supporting proactive issue detection and faster troubleshooting. EDC APIs for health checks can be similarly exposed with restricted access through API Gateway and IAM.

Managed Services Over Self-Managed Infrastructure: Use AWS managed services (Aurora, Secrets Manager, Fargate, Cognito) instead of deploying and maintaining compatible self-managed solutions. This shifts undifferentiated heavy lifting to AWS and reduces operational burden. You gain high availability, built-in security best practices, compliance certifications, and automatic updates.

Security

Defense in Depth: Implement security through multiple independent layers rather than relying on a single control. Network isolation (VPC private subnets), security group segmentation (restricting traffic between components), IAM least privilege (scoped permissions per service), and encryption (at rest and in transit) each provide independent controls. These layers work together so that if one layer is bypassed, others continue to provide protection.

Principle of Least Privilege: Every component receives only the minimum permissions required for its specific function. Scope IAM roles to individual services (control plane, data plane) and restrict security groups to necessary ports and sources. The internal-only Network Load Balancer fronted by API Gateway prevents unintended public exposure of EDC APIs and data. This may also support security review and approval of EDC as open-source software, since APIs can be allowlisted and validated individually.

Encryption Everywhere: Encrypt data by default at every stage: at rest (Aurora, S3, Secrets Manager), in transit (TLS enforcement, HTTPS-only egress), and during processing (encrypted environment variables). This provides comprehensive data protection regardless of where information resides in the system.

Reliability

Fail Fast, Recover Automatically: Systems detect failures quickly and recover without manual intervention. ECS circuit breakers can automatically roll back failed deployments, automated health checks remove unhealthy targets, and point-in-time recovery supports rapid database restoration. This minimizes mean time to recovery (MTTR) and reduces the scope of failures, even within a single EDC architecture cell.

Design for Regional Resilience: Cross-zone load balancing distributes traffic across multiple Availability Zones (AZs), Aurora automatically replicates data across AZs, and Fargate tasks can be scheduled in any AZ. The highlighted architecture can tolerate Availability Zone failures in an AWS Region without service disruption. For more information about Availability Zones and Regions, see AWS Global Infrastructure.

Decoupled Components with Clear Boundaries: Deploy the control plane and data plane as independent services with distinct responsibilities, security contexts, and scaling characteristics. This separation enables independent updates, targeted scaling, and failure isolation between coordination logic and data transfer operations.

The remaining three Well-Architected Framework pillars of Performance Efficiency, Cost Optimization, and Sustainability are covered in the third post of this series where we discuss cost optimization strategies for running EDC connectors on AWS.

Conclusion

With the growing popularity of data spaces and EDC as a data space connector, it is important to distinguish between a setup suitable for testing and experimentation and one that is ready for production. Production environments require that business-critical processes depend on timely, successful transmission of confidential information between participants. The architecture defined in this post combines EDC deployment best practices from the community with AWS recommendations to achieve fault tolerance, scalability, and security while keeping operational complexity at a minimum.

In part 3, you will learn about cost optimization strategies to run your production-ready connector efficiently and maximize the value it returns by supporting business use cases for data sharing along your supply network. In the meantime, explore the Dataspace Connector on AWS project and see how the patterns and best practices covered in this post come together in an end-to-end reference implementation.

References

About the authors

Optimizing Your IoT Devices for Environmental Sustainability

Post Syndicated from Jonas Bürkel original https://aws.amazon.com/blogs/architecture/optimizing-your-iot-devices-for-environmental-sustainability/

To become more environmentally sustainable, customers commonly introduce Internet of Things (IoT) devices. These connected devices collect and analyze data from commercial buildings, factories, homes, cars, and other locations to measure, understand, and improve operational efficiency. (There will be an estimated 24.1 billion active IoT devices by 2030 according to Transforma Insights.)

IoT devices offer several efficiencies. However, you must consider their environmental impact when using them. Devices must be manufactured, shipped, and installed; they consume energy during operations; and they must eventually be disposed of. They are also a challenge to maintain—an expert may need physical access to the device to diagnose issues and update it. This is especially true for smaller and cheaper devices, because extended device support and ongoing enhancements are often not economically feasible, which results in more frequent device replacements.

When architecting a solution to tackle operational efficiency challenges with IoT, consider the devices’ impact on environmental sustainability. Think critically about the impact of the devices you deploy and work to minimize their overall carbon footprint. This post considers device properties that influence an IoT device’s footprint throughout its lifecycle and shows you how Amazon Web Services (AWS) IoT services can help.

Architect for lean, efficient, and durable devices

So which device properties contribute towards minimizing environmental impact?

  • Lean devices use just the right amount of resources to do their job. They are designed, equipped, and built to use fewer resources, which reduces the impact of manufacturing and disposing them as well as their energy consumption. For example, electronic devices like smartphones use rare-earth metals in many of their components. These materials impact the environment when mined and disposed of. By reducing the amount of these materials used in your design, you can move towards being more sustainable.
  • Efficient devices lower their operational impact by using up-to-date and secure software and enhancements to code and data handling.
  • Durable devices remain in the field for a long time and still provide their intended function and value. They can adapt to changing business requirements and are able to recover from operational failure. The longer the device functions, the lower its carbon footprint will be. This is because device manufacturing, shipping, installing, and disposing will require relatively less effort.

In summary, deploy devices that efficiently use resources to bring business value for as long as possible. Finding the right tradeoff for your requirements allows you to improve operational efficiency while also maximizing your benefit on environmental sustainability.

High-level sustainable IoT architecture

Figure 1 shows building blocks that support sustainable device properties. Their main capabilities are:

  • Enabling remote device management
  • Allowing over-the-air (OTA) updates
  • Integrating with cloud services to access further processing capabilities while ensuring security of devices and data, at rest and in transit
Generic architecture for sustainable IoT devices

Figure 1. Generic architecture for sustainable IoT devices

Introducing AWS IoT Core and AWS IoT Greengrass to your architecture

Assuming you have an at least partially connected environment, the capabilities outlined in Figure 1 can be achieved by using mainly two AWS IoT services:

  • AWS IoT Core is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices.
  • AWS IoT Greengrass is an IoT open-source edge runtime and cloud service that helps you build, deploy, and manage device software.

Figure 2 shows how the building blocks introduced in Figure 1 translate to AWS IoT services.

AWS architecture for sustainable IoT devices

Figure 2. AWS architecture for sustainable IoT devices

Optimize your IoT devices for leanness and efficiency with AWS IoT Core

AWS IoT Core securely integrates IoT devices with other devices and the cloud. It allows devices to publish and subscribe to data in the cloud using device communication protocols. You can use this functionality to create event-driven data processing flows that can be integrated with additional services. For example, you can run machine learning inference, perform analytics, or interact with applications running on AWS.

According to a 451 Research report published in 2019, AWS can perform the same compute task with an 88% lower carbon footprint compared to the median of surveyed US enterprise data centers. More than two-thirds of this carbon reduction is attributable to more efficient servers and a higher server utilization. In 2021, 451 Research published similar reports for data centers in Asia Pacific and Europe.

AWS IoT Core offers this higher utilization and efficiency to edge devices in the following ways:

  • Non-latency critical, resource-intensive tasks can be run in the cloud where they can use managed services and be decommissioned when not in use.
  • Having less code on IoT devices also reduces maintenance efforts and attack surface while making it simpler to architect its software components for efficiency.
  • From a security perspective, AWS IoT Core protects and governs data exchange with the cloud in a central place. Each connected device must be credentialed to interact with AWS IoT. All traffic to and from AWS IoT is sent securely using Transport Layer Security (TLS) mutual authentication protocols. Services like AWS IoT Device Defender are available to analyze, audit, and monitor connected fleets of devices and cloud resources in AWS IoT at scale to detect abnormal behavior and mitigate security risks.

Customer Application:
Tibber, a Nordic energy startup, uses AWS IoT Core to securely exchange billions of messages per month about their clients’ real-time energy usage and aggregate data and perform analytics centrally. This allows them to keep their smart appliance lean and efficient while gaining access to scalable and more sustainable data processing capabilities.

Ensure device durability and longevity with AWS IoT Greengrass

Tasks like interacting with sensors or latency-critical computation must remain local. AWS IoT Greengrass, an edge runtime and cloud service, securely manages devices and device software, thereby enabling remote maintenance and secure OTA updates. It builds upon and extends the capabilities of AWS IoT Core and AWS IoT Device Management, which securely registers, organizes, monitors, and manages IoT devices.

AWS IoT Greengrass brings offline capabilities and simplifies the definition and distribution of business logic across Greengrass core devices. This allows for OTA updates of this business logic as well as the AWS IoT Greengrass Core software itself.

This is a distinctly different approach to what device manufacturers did in the past. Devices no longer need to be designed to run all code for one immutable purpose. Instead, they can be built to be flexible for potential future use cases, which ensures that business logic can be dynamically tweaked, maintained, and troubleshooted remotely when needed.

AWS IoT Greengrass does this using components. Components can represent applications, runtime installers, libraries, or any code that you would run on a device that are then distributed and managed through AWS IoT. Multiple AWS-provided components as well as the recently launched Greengrass Software Catalog extend the edge runtime’s default capabilities. The secure tunneling component, for example, establishes secure bidirectional communication with a Greengrass core device that is behind restricted firewalls, which can then be used for remote assistance and troubleshooting over SSH.

Conclusion

Historically, IoT devices were designed to stably and reliably serve one predefined purpose and were equipped for peak resource usage. However, as discussed in this post, to be sustainable, devices must now be lean, efficient, and durable. They must be manufactured, shipped, and installed once. From there, they should be able to be used flexibly for a long time. This way, they will consume less energy. Their smaller resource footprint and more efficient software allows organizations to improve operational efficiency but also fully realize their positive impact on emissions by minimizing devices’ carbon footprint throughout their lifecycle.

Ready to get started? Familiarize yourself with the topics of environmental sustainability and AWS IoT. Our AWS re:Invent 2021 Sustainability Attendee Guide covers this. When designing your IoT based solution, keep these device properties in mind. Follow the sustainability best practices described in the Sustainability Pillar of the AWS Well-Architected Framework.

Related information