Tag Archives: Amazon SageMaker Unified Studio

How to set up an air-gapped VPC for Amazon SageMaker Unified Studio

2026-04-02 Rohit Vashishtha

Post Syndicated from Rohit Vashishtha original https://aws.amazon.com/blogs/big-data/how-to-set-up-an-air-gapped-vpc-for-amazon-sagemaker-unified-studio/

Organizations are finding significant value using an integrated experience for all your data and AI with Amazon SageMaker Unified Studio. However, many organizations require strict network control to meet security and regulatory compliance requirements like HIPAA or FedRAMP for their data and AI initiatives, while maintaining operational efficiency.

In this post, we explore scenarios where customers need more control over their network infrastructure when building their unified data and analytics strategic layer. We’ll show how you can bring your own Amazon Virtual Private Cloud (Amazon VPC) and set up Amazon SageMaker Unified Studio for strict network control.

Solution overview

The solution covers complete technical know-how of a fully private network architecture using Amazon VPC with no public internet exposure. The approach leverages AWS PrivateLink through VPC endpoints to provide a secure communication between SageMaker Unified Studio and essential AWS services entirely over the AWS backbone network.

The architecture consists of three core components: a custom VPC named airgapped with multiple private subnets distributed across at least three Availability Zones for high availability, a comprehensive set of VPC interface and gateway endpoints for service connectivity, and the SageMaker Unified Studio domain configured to operate exclusively within this isolated environment. This design helps ensure that sensitive data never traverses the public internet while maintaining full functionality for data cataloging, query execution, and machine learning workflows.

By implementing this air-gapped configuration, organizations gain granular control over network traffic, simplified compliance auditing, and the ability to integrate SageMaker Unified Studio with existing private data sources through controlled network pathways. The solution supports both immediate operational needs and long-term scalability through careful IP address planning and modular endpoint architecture.

Prerequisites

The set up requires you to have an existing VPC (for this post, we’ll refer to the name as airgapped but in reality, it refers to the VPC you would like to securely set up SageMaker Unified Studio). If you don’t have an existing VPC, you can follow SageMaker Unified Studio domain quick create administrator guide to get started.

The high level steps to create a VPC meeting minimum requirements for SageMaker Unified Studio are as follows:

In the AWS Management Console, navigate to the VPC console.
Choose Create VPC.
Select the VPC and more radio button.
For Name tag auto-generation, enter airgapped or a name of your choice.
Keep the default values for IPv4 CIDR block, IPv6 CIDR block, Tenancy, NAT gateways, VPC endpoints, and DNS options.
Select 3 for Number of Availability Zones (AZs).
Select 0 for Number of public subnets.
Choose Create VPC.

This produces the following VPC resource map:

Figure 1 – VPC configuration

Set up SageMaker Unified Studio

Now, we will set up SageMaker Unified Studio in an existing VPC, named airgapped-vpc.

Navigate to the SageMaker console, choose Domains in the navigation pane.
Choose Create Domain.
For How do you want to set up your domain?, select Quick set up.
Expand the Quick set up settings
Provide a name for your domain, such as airgapped-domain.
For Virtual private cloud (VPC), select airgapped-vpc.
For subnets, select a minimum of two private subnets.
Choose Continue.
Enter an email address to create a user in AWS IAM Identity Center.
Choose Create domain.
Once the domain is created, choose Open unified studio or use SageMaker Unified Studio URL under Domain details to access SageMaker Unified Studio.

Figure 2 – Amazon SageMaker Unified Studio URL Welcome Page
After logging in to SageMaker Unified Studio, create a project using the guided wizard.
Once the project is created, we need to add the necessary VPC endpoints to allow traffic from the project to communicate to AWS services.
S3 Gateway VPC endpoint was already selected as part of VPC creation step 5 in prerequisites and thus created by default. Now we must add two more VPC endpoints for Amazon DataZone and AWS Security Token Service as illustrated in following step.

These are the minimum set of VPC endpoints to allow using the tooling within SageMaker Unified Studio. For a list of other mandatory and non-mandatory VPC endpoints refer to the tables in the latter part of this post.

Create an interface endpoint

To create an interface endpoint, complete following steps:

Go to the SageMaker Unified Studio Project details page and copy the Project ID.
Figure 3 – SageMaker Unifed Studio Project Details Page
Go to the VPC console and choose Endpoints.
Choose Create Endpoint.
Enter a name for the endpoint, for example, DataZone endpoint for SageMaker Unified Studio.
For AWS Services, enter DataZone.

Figure 4 – Interface Endpoint creation wizard for AWS Service datazone
Select Service Name = com.amazonaws.us-east-1.datazone from the available options.

Figure 5 – Interface Endpoint creation wizard network settings
Select the subnets in the airgapped-vpc that you created earlier.
Filter the Security Groups by pasting the copied Project ID.
Select the security group with Group Name datazone-<project-id>-dev.
Choose Create Endpoint.
Repeat the same steps to create a VPC endpoint for AWS STS.
Once the VPC endpoints are created, validate connectivity in the SageMaker project by running a SQL query or using a Jupyterlab notebook.

For a successful domain and project which does not get into any service level usage, the mandatory VPC endpoints to be created are: S3 Gateway, DataZone, and STS interface endpoints. For other service usage dependent operations like authentication, data preview and working with compute, you would require other mandatory service specific endpoints explained later in this post.

Best practices for VPC set up for various use cases

When setting up SageMaker Unified Studio domain and project profiles, you need to specify the VPC network, subnets, and security groups. Here are some best practices around IP allocation, usage volume and expected growth to consider for different use cases within enterprises.

Production and enterprise use cases

If your organization require strict network control to meet security and compliance requirements for data and AI initiatives, consider following best practices in your production environment.

Use the bring-your-own (BYO) VPC approach to comply with company-specific networking and security requirements.
Implement private networking using VPC endpoints to keep traffic within the AWS backbone.
Use at least two private subnets across different Availability Zones.
Enable DNS hostnames and DNS Support.
Disable auto-assign public IP on subnets.
Plan IP capacity for at least 5 years. A prescriptive guidance for SageMaker Unified Studio is shared in VPC and Networking details section later in this post. Consider the following:
- Number of users
- Number of apps per user
- Number of unique instance types per user
- Average number of training instances
- Expected growth percentage

Testing and non-production use cases

For development, testing, non-prod environment where use cases don’t have stringent security and compliance requirements, use automated setup for quick experiments. Use sample CloudFormation github templates as part of the SageMaker Unified Studio express set up, to automate domain and project creation. However, this includes an Internet Gateway which may not be suitable for security-sensitive environments.

Private networking use cases

VPCs with private subnets require essential service endpoints to allow client resources like Amazon EC2 instances to securely access AWS services. The traffic between your VPC and AWS services remains within AWS network avoiding public internet exposure.

Implement all mandatory VPC endpoints for core services (SageMaker, DataZone, Glue, and more).
Add optional endpoints based on specific service needs, like IPv4 endpoints, dual-stack endpoints, and FIPS endpoints to programmatically connect to an AWS service.
Work with network administrators for:
- Preinstalling needed resources through secure channels like private subnets and self-referencing inbound rules in security groups to enable limited access.
- Allowlisting only necessary external connections like NAT gateway IP and bastion host access in firewall rules.
- Setting up appropriate proxy configurations if required.

External data source access use cases

Consider the following when working with external systems like third-party SaaS platforms, on-premises databases, partner APIs, legacy systems, or external vendors.

Consult with network administrators for appropriate connection methods.
Consider AWS PrivateLink integration where available.
Implement appropriate security measures for non-AWS data your source documents.
For High Availability:
- Deploy across at least three different Availability Zones (at least two for AWS Regions with only two AZs).
- Verify there’s a minimum of three free IPs per subnet.
- Consider larger CIDR blocks (/16 recommended) for future scalability.

VPC and networking details

In this section, we provide details of each networking aspect starting with choice of VPCs, network connectivity details for integrated services to work, the basis of VPC and subnet requirements, and finally the VPC endpoints required for private service access.

VPC

At a high level, you have two options to supply VPCs and subnets:

Bring-your-own (BYO) VPC. This is typically the case for most customers, as most have company specific networking and security requirements to reuse an existing VPC, or to create a VPC that are compliant with those requirements.
Create VPC with the SageMaker quick set up template. When creating a SageMaker Unified Studio domain (DataZone V2 domain in CloudFormation) through the automated quick set up, you will be shown a Quick create stack wizard in CloudFormation which creates VPCs and subnets used to configure your domain.
Note: The quick create stack using template URL is not intended for production use. The template creates an Internet Gateway, which is not allowed in many enterprise settings. This is only appropriate if you are either trying out SageMaker Unified Studio or, running SageMaker Unified Studio for use cases that don’t have stringent security requirements.If you choose this option, you start with SageMaker console, navigate to domains and click Create domain button, followed by Create VPC button. You will navigate to CloudFormation and click on Create stack button to create a sample VPC named SageMakerUnifiedStudio-VPC with just one-click for trying out SageMaker Unified Studio.

Figure 6 – Create VPC button in SageMaker Unified Studio Create Domain Wizard

Cost estimation for recommended VPC set up

The exact cost depends on the configuration of your VPC. For more complex networking set ups (multi-VPC), you may need to use additional networking components such as a Transit Gateway, Network Firewall, and VPC Lattice. These components may incur charges, and cost depends on usage and AWS Region. Interface VPC endpoints are charged per availability zone. They also have a fixed and a variable component in the pricing structure. Use the AWS Pricing Calculator for a detailed estimate.

Network Connectivity

With regards to connectivity to the underlying AWS services integrated within SageMaker Unified Studio, there are two ways to enable connectivity (these are not Studio specific, these are standard ways to enable network connectivity within a VPC). This is an important security consideration that depends on your organization’s security policies.

Through the public Internet. Your traffic will traverse over the public Internet through an Internet Gateway in your VPC.
1. Your VPC must have an Internet Gateway attached to it.
2. Your public subnet must have a NAT Gateway. In addition, your public subnet’s route table must have a default route (0.0.0.0 for IPv4) to the Internet Gateway. This route is what makes the subnet public.
3. Your private subnets must have a default route to the public subnet’s NAT Gateway.
Through the AWS backbone. Your traffic will remain within the private AWS backbone through PrivateLink (by provisioning Interface and Gateway endpoints for the necessary AWS services in each Availability Zone).
1. A list of all the AWS services integrated into Studio and the VPC endpoints required can be found in section VPC Endpoints covered later in this post.
2. For non-AWS resources, certain external providers of these services may offer PrivateLink integration. Check with each provider’s documentation and your network administrator to understand the most suitable way to connect to these external providers.

In a private networking scenario, you will need to consider whether you need connectivity to non-AWS resources in a way that’s compliant with your organization’s security policies. A few examples include the following:

If you need to download software in your remote IDE host (for example, command line programs, such as Ping and Traceroute)
If you have code that connects to external APIs.
If you use software (such as JupyterLab or Code Editor extensions) that rely on external APIs.
If you depend on software dependencies hosted in the public domain (such as Maven, PyPi, npm)
If you need cross-Region access to certain resources (such as access to S3 buckets in a different Region)
If you need functionality whose underlying AWS services do not have VPC endpoints in all Regions or any Region.
1. Amazon Q (powers Q and code suggestions)
2. SQL Workbench (powers Query Editor)
3. IAM (powers Glue connections)

If you need to connect to data sources outside of AWS (such as Snowflake, Microsoft SQL Server, Google BigQuery)
Enterprise network administrators must also complete either of the following prerequisites to handle private networking scenarios:

Preinstall needed resources through secure channels if possible. An example would be to customize your SageMaker AI image by installing dependencies, after they are code scanned, vetted technically and legally by your organization.
If AWS PrivateLink integration is not available for external providers, allowlist network connections to these external sources. Allow firewall egress rules, directly or indirectly, through a proxy in your organization’s network. Check with your network administrator to understand the most appropriate option for your organization.

VPC Requirements

When setting up a new SageMaker Unified Studio Domain, it’s necessary to supply a VPC. It’s important to note that these VPC requirements are a union of all the requirements from the respective compute services integrated into Studio, some of which are reinforced by validation checks during the corresponding blueprint’s deployment. If these requirements that have validation checks are not fulfilled, the resource(s) contained in that blueprint may fail to create on project creation (on-create), or when creating the compute resource (on-demand). This section will present a summary of these requirements, as well as relevant documentation links from which they originate.

Subnet requirements for specific compute in a VPC

This section lists the compute services integrated in SageMaker Unified Studio that require VPC/subnets when provisioning the respective compute resources.

Compute Connections

Other Services

Requirements

Number of subnets: At least two private subnets. This requirement comes from Redshift Serverless.
Availability zones (AZs): At least two different AZs (for Regions with two AZs, two subnets are sufficient). This requirement comes from Redshift Serverless. For workgroups with Enhanced VPC Routing (EVR), you need three AZs.
Free IPs per subnet: At least three Ips per subnet. This requirement comes from Redshift Serverless without EVR. For detailed IP addresses requirement with EVR enabled workgroups, refer to Serverless usage considerations. Three is a minimum and may not be enough for your needs. For example, EMR cluster creation will fail if no subnets with enough IPs are found in the VPC. We recommend doing a forward-looking capacity planning exercise based on your use cases (for example, growth rate, users, compute needs) to project at least 5 years into the future. This helps to determine how many IPs are needed by the team using Studio and other services that use this VPC and come up with a ceiling for the CIDR block size.
Private or public subnets: We enforce that at least three private subnets be supplied, and recommend that only private subnets are chosen, with a few nuances. This requirement comes from SageMaker AI domain. A new SageMaker AI domain, when set up with VpcOnly mode, requires that all subnets in the VPC be private. This is the default networking mode in the Tooling blueprint. If you choose to use PublicInternetOnly mode, this restriction does not apply, you may choose public subnets from your VPC. To change the mode, modify the Tooling Blueprint parameter sagemakerDomainNetworkType.
Enable DNS hostname and DNS Support: Both must be enabled. This requirement comes from EMR. Without these VPC settings, enableDnsHostname and enableDnsSupport, connecting to the EMR Cluster using the private DNS name through the Livy Endpoint will fail. SSL Verification, which can only be done when connecting using the DNS name, not the IP.
Auto assign public IP: Disable. We recommend that this EC2 subnet setting (mapPublicIpOnLaunch) be disabled when using private subnets, because public IPs come at a cost and are a scarce resource in the total addressable IPv4 space.

VPC endpoints

If you choose to run SageMaker Unified Studio without public internet access, VPC endpoints are required for all services SageMaker Unified Studio needs to access. These endpoints provide secure, private connectivity between your VPC and AWS services without traversing the public internet. The following table lists the required endpoints, their types, and what each is used for.

Some endpoints may not show up directly in your browser’s network tab. The reason is that some of these services (such as CloudWatch) are transitively invoked by other services.

Mandatory endpoints

The following are required endpoints for SageMaker Unified Studio and supporting services to function properly. Gateway endpoints can be used where available, you can use interface endpoints for all other AWS services.

AWS service	Endpoint	Type	Purpose
Glue	`com.amazonaws.${region}.glue`	Interface	For Data Catalog and metadata management
STS	`com.amazonaws.${region}.sts`	Interface	Required for assuming IAM roles
S3	`com.amazonaws.${region}.s3`	Gateway	Required for datasets, Git backups, notebooks, and Git sync
SageMaker	`com.amazonaws.${region}.sagemaker.api`	Interface	Required for calling SageMaker APIs
SageMaker	`com.amazonaws.${region}.sagemaker.runtime`	Interface	For invoking deployed inference endpoints
DataZone	`com.amazonaws.${region}.datazone`	Interface	For data catalog and governance
Secrets Manager	`com.amazonaws.${region}.secretsmanager`	Interface	To securely access secrets
SSM	`com.amazonaws.${region}.ssm`	Interface	For secure command execution
SSM	`com.amazonaws.${region}.ssmmessages`	Interface	Enables live SSM sessions
KMS	`com.amazonaws.${region}.kms`	Interface	For decrypting data (volumes, S3, secrets)
EC2	`com.amazonaws.${region}.ec2`	Interface	For subnet and ENI management
EC2	`com.amazonaws.${region}.ec2messages`	Interface	Required for SSM messaging
Athena	`com.amazonaws.${region}.athena`	Interface	Required to run SQL queries
Amazon Q	`com.amazonaws.${region}.q`	Interface	Used by SageMaker Notebooks for enhanced productivity

Optional Endpoints

Only create these if the corresponding service is used in your environment.

AWS service	Endpoint	Type	Purpose
EMR	`com.amazonaws.${region}.emr-serverless`	Interface	Serverless Spark/Hive jobs
	`com.amazonaws.${region}.emr-serverless-services.livy`	Interface	Required for Livy job submission (EMR Serverless)
	`com.amazonaws.${region}.elasticmapreduce`	Interface	Classic EMR (EC2-based)
	`com.amazonaws.${region}.emr-containers`	Interface	EMR on EKS workloads
Redshift	`com.amazonaws.${region}.redshift`	Interface	For provisioned Redshift clusters
	`com.amazonaws.${region}.redshift-serverless`	Interface	For Redshift Serverless
	`com.amazonaws.${region}.redshift-data`	Interface	Required for running SQL against Redshift
Amazon Bedrock	`com.amazonaws.${region}.bedrock-runtime`	Interface	Invoke Bedrock models at runtime
	`com.amazonaws.${region}.bedrock-agent`	Interface	For Bedrock knowledge agents
	`com.amazonaws.${region}.bedrock-agent-runtime`	Interface	For running knowledge agent workloads
CloudWatch	`com.amazonaws.${region}.logs`	Interface	Application and notebook logs
RDS	`com.amazonaws.${region}.rds`	Interface	Connect to Amazon RDS and Aurora
CodeCommit	`com.amazonaws.${region}.codecommit`	Interface	Git integration with CodeCommit
CodeCommit	`com.amazonaws.${region}.git-codecommit`	Interface	Alternative endpoint for CodeCommit
CodeConnections and CodeStar	`com.amazonaws.${region}.codeconnections.api`	Interface	GitHub and GitLab repo integration
CodeConnections and CodeStar	`com.amazonaws.${region}.codestar-connections.api`	Interface	Alias of CodeConnections

Clean up

AWS resources provisioned in your AWS accounts may incur costs based on the resources consumed. Make sure you do not leave any unintended resources provisioned. If you created a VPC and subsequent resources as part of this post, make sure you delete them.

The following service resources provisioned during this blog post need to be deleted:

IAM Identity Center users and groups.
Resources provisioned within your project using tooling configuration and blueprints within your domain.
The airgapped VPC.

Conclusion

In this post, we walked through the process of using your own existing VPC when creating domains and projects in SageMaker Unified Studio. This approach benefits customers by giving them greater control over their network infrastructure while using the comprehensive data, analytics, and AI/ML capabilities of Amazon SageMaker. We also explored the critical role of VPC endpoints in this set up. You now understand when these become necessary components of your architecture, particularly in scenarios requiring enhanced security, compliance with data residency requirements, or improved network performance.

While using a custom VPC requires more initial set up than the Quick Create option, it provides the flexibility and control many organizations need for their data science and analytics workflows. This approach provides a mechanism for your SageMaker environment to integrate with your existing infrastructure and adheres to your organization’s networking policies. Custom VPC configurations are a powerful tool in your arsenal for building secure, compliant, and efficient data science environments.

To learn more, visit Amazon SageMaker Unified Studio – Administrator Guide and User Guide.

About the authors

Navigating multi-account deployments in Amazon SageMaker Unified Studio: a governance-first approach

2026-04-01 Ben Shafabakhsh

Post Syndicated from Ben Shafabakhsh original https://aws.amazon.com/blogs/big-data/navigating-multi-account-deployments-in-amazon-sagemaker-unified-studio-a-governance-first-approach/

Amazon SageMaker Unified Studio brings together data engineering, analytics, and machine learning (ML) workflows into a cohesive, governed environment. This unified approach reduces traditional silos between data teams and ML practitioners, so organizations can advance their AI and ML initiatives with greater collaboration and efficiency.

As enterprises begin their SageMaker Unified Studio adoption, they must determine the best practices for implementing data federation principles when using SageMaker Unified Studio across the organization. The way that you structure your SageMaker Unified Studio deployment is more than a technical decision. It directly impacts your governance framework, security posture, operational scalability, and day-to-day team collaboration.

In this post, we explore SageMaker Unified Studio multi-account deployments in depth: what they entail, why they matter, and how to implement them effectively. We examine architecture patterns, evaluate trade-offs across security boundaries, operational overhead, and team autonomy. We also provide practical guidance to help you design a deployment that balances centralized control with distributed ownership across your organization.

The multi-account challenge: why organizations struggle

If you’re working in a large enterprise, a multi-account AWS environment is often your starting position. If you’re starting from scratch, consider whether to use a single-account for all SageMaker Unified Studio components or dedicate separate accounts for governance and business units. A multi-account architecture aligns with AWS best practices and proves valuable if you have:

Distributed teams with independent operations: multi-account architecture accommodates multiple teams or business units that maintain separate operations so that each team can manage their projects autonomously within isolated environments. Each unit can deploy and manage resources independently, implement team-specific security controls, and scale infrastructure without impacting others. This is achieved through a shared, unified integrated development environment (IDE) for collaboration and standardized tools across the organization.
Compliance and data governance requirements: For regulatory mandates like GDPR, HIPAA, or data sovereignty needs, you will benefit from this setup, because sensitive data remains segregated in business-unit specific accounts. This reduces risk exposure, streamlines audits, and maintains compliance boundaries without compromising access to centralized collaboration tools.
Centralized governance: A multi-account architecture maintains visibility across all projects and business units from a single control plane. The Domain account can enforce security policies and compliance requirements across the entire organization and provide centralized monitoring, audit logging, and user access management.
Clear cost visibility and accountability: Multi-account architecture enables granular billing tracking, with each account generating separate bills that clearly attribute costs to specific teams or business units. This transparency streamlines budgeting and financial accountability, removing the complexity of cost allocation tags and manual reporting that’s typically required in single-account models where multiple teams share the same infrastructure and resources.

Overall, this approach improves efficiency, security, and scalability for you, whether you’re managing a few teams or coordinating across a larger organizational structure.

Understanding the core constructs of SageMaker Unified Studio

Before diving into multi-account strategies, it’s important to understand the foundational constructs of SageMaker Unified Studio. Each is elaborated in greater detail in the Administrator Guide.

Domain: The top-level administrative boundary where governance lives. In a multi-account setup, this is your centralized control plane for catalog, policies, and user access.
Project: A collaborative workspace for developing data, AI, and ML initiatives. In multi-account deployments, a Project’s metadata lives in the Domain account and compute and data resources deploy into associated business unit accounts. This separation is central to the pattern that we explore.
Project Profile: A template that standardizes how Projects are created. For multi-account setups, this is where administrators define which accounts and AWS Regions Projects can target.
Blueprints: Infrastructure as code (IaC) components that define what gets provisioned inside a Project. Each associated account must enable its relevant Blueprints before Projects can deploy there.

The following diagram illustrates how these key constructs interact. Within a Domain, users create Projects organized through a single Project Profile. The Project Profile defines and configures a collection of Blueprints. When a project is created, the infrastructure specified in those Blueprints is automatically provisioned and becomes available within the project workspace.

Figure 1: Amazon SageMaker Unified Studio Core Constructs

Multi-account setup in SageMaker Unified Studio

To illustrate these concepts in practice, we demonstrate with a sample enterprise organization that exemplifies enterprise environments with several AWS accounts belonging to different business units:

Central Data Governance team: Owns and manages governance and access control across the organization. They plan to build a data solution in a dedicated AWS account using SageMaker Unified Studio. The platform must provide an integrated development environment (IDE) to work with data and ML use cases and connect to multiple business unit’s AWS accounts (Finance and Marketing).
Finance Business Unit: Owns datasets for fraud analysis and churn prediction in their own AWS account.
Marketing Business Unit: Maintains customer sentiment data and campaign analytics in their own AWS account.

In the following diagram we show the Data platform constructs provided by SageMaker Unified Studio in each AWS account showing the clear separation between centralized governance and distributed resource deployment.

Figure 2: Sample organization architecture in Amazon SageMaker Unified Studio

The Central Data Governance Account contains the SageMaker Unified Studio Domain. This contains the shared platform resources (Catalog, shared infrastructure), governance constructs (Domain units, metadata forms), and governance policies (authorization policies, enforcement rules). These configuration elements define the standards and capabilities available across the organization. They’re the Service-level configuration data: Metadata, policies, and governance rules that define how resources should be provisioned.

In contrast, the Associated Accounts (Marketing and Finance) contain the actual AWS infrastructure, compute/storage (purple cubes) and data stores (cylinders), provisioned when Projects are created. The diagram shows how Marketing Projects and Finance Projects ultimately deploy their runtime resources into their respective business unit accounts. The separation keeps the governance centralized and consistent while allowing business unit dependent resources to be isolated, billed separately, and managed according to each business unit’s specific requirements.

To understand the core constructs of SageMaker Unified Studio, we listed the core components of SageMaker Unified Studio and explained how they relate to each other. Taking the same diagram as the basis, we will now represent how these constructs are created in our multi-account sample scenario.

Construct	Deployment Location	Deployed Resources
Domain	Central Data Governance Account	Portal Catalog Metadata Forms Authorization Policies
Project	Central Data Governance Account + Associated Account	In Central Data Governance Account: Project configuration and metadata In Associated Account: Project Infrastructure resources such as Compute Project Data
Project Profile	Central Data Governance Account	Project Profile Configuration
Blueprint	Associated Account	Blueprints are enabled in each associated account

Core construct deployment locations and resources in SageMaker Unified Studio

Implementing multi-account deployments

To enable production-ready data science and analytics workflows across multiple AWS accounts governed by a SageMaker Unified Studio Domain, organizations must establish a structured cross-account configuration. This setup allows each business unit (BU) to retain ownership of its Projects and AWS resources while using centralized governance provided by the Domain. The process involves four key steps: account association, Blueprint enablement, Project Profile configuration, and Project creation.

Note: The following steps provide a high-level overview of the multi-account deployment process. For a more detailed, step-by-step guide, refer to How to associate an account when using Amazon SageMaker Unified Studio.

Step 1: Account association to a domain

The Domain administrator associates each AWS account with the SageMaker Unified Studio Domain for seamless cross-account functionality by providing the AWS account number for the targeted accounts. This association lets the Domain publish and consume data from associated accounts, create resources within them, maintain cross-account access for the SageMaker Catalog, and deploy Projects directly into business unit accounts. Account association is a critical prerequisite for cross-account Project deployment. Behind the scenes, SageMaker Unified Studio uses AWS Resource Access Manager (AWS RAM) to make this cross-account functionality happen.

Step 2: Enabling blueprints

Each associate account administrator must enable the relevant Blueprints before creating Project Profiles. This important step verifies that Projects can provision the necessary tools and resources that users need to run their workloads. Blueprints serve as standardized infrastructure templates that administrators can use to enforce organizational standards, security controls, and best practices across all Projects. Through Blueprints, administrators configure essential resources including AWS Identity and Access Management (IAM) roles, AWS Key Management Service (AWS KMS) keys, Amazon Simple Storage Service (Amazon S3) buckets, Amazon Virtual Private Cloud (Amazon VPC) settings, and security groups. This centralized approach helps maintain consistency, compliance, and governance at scale while preventing users from creating Projects with misconfigured or non-compliant infrastructure.

Step 3: Configuring project profile

With the accounts successfully associated and the Blueprints enabled, the next step is to configure a Project Profile that determines where your Project resources will be deployed. Your choice of Project Profile strategy impacts both operational flexibility and governance.

Domain administrators control which Blueprints are included in each Project Profile and can specify the target AWS Regions and accounts for deployment, providing the governance foundation to standardize Project creation. Administrators can use pre-created Project Profiles like “All Capabilities” or “SQL Analytics”, or create custom Project Profiles tailored to specific organizational needs.

When configuring Project Profiles, you can choose between two deployment models:

Static (Pre-Defined): Profile specifies a fixed account and Region. Projects by default deploy to the same location. This is recommended for strict governance controls and compliance requirements where production resources must remain in designated accounts or Regions.
Dynamic (Parameterized): Users select from available associated accounts and Regions during Project creation (configured through Account Pools). This is recommended for multi-environment workflows (Dev/Test/Prod) and reducing administrative overhead by maintaining fewer profile templates.

Dynamic profiles balance governance with agility: Administrators define standards once, while users retain deployment flexibility aligned with their business needs.

Step 4: Project creation

With Project Profiles configured, you can now create a new Project from any of the associated accounts using the profile created in the previous step.

Defining project boundaries: when to create a new project

One of the common challenges that you will face is determining when to create a new Project. The answer significantly impacts collaboration effectiveness, resource isolation, cost tracking, and governance. Here’s a practical framework to guide your decision-making.

A Project should represent a distinct business initiative with a defined scope, a dedicated team, and measurable outcomes. Think of Projects as team workspaces organized around business outcomes, not technical components.

Create a new Project when you need clear separation across multiple dimensions: cost allocation, access control, and data governance. If Finance and Marketing teams require separate budget tracking, distinct data access policies, and different governance controls, they should have separate Projects. For example, “Customer Churn Prediction” and “Fraud Detection” might use similar tools, but if they have different stakeholders, budget owners, and data sensitivity requirements, so they warrant separation. Similarly, create separate Projects when facing different compliance or regulatory requirements (like HIPAA versus PCI-DSS) or when initiatives have independent deployment lifecycles. Experimental ML research Projects shouldn’t share workspaces with production recommendation engines that require stricter change controls and availability guarantees.

However, avoid fragmenting related work into unnecessary silos:

Don’t create separate Projects for individual workflows or pipelines, a single “Marketing Campaign Optimization” Project should contain audience segmentation, propensity modeling, and campaign attribution workflows together.
Don’t separate different data processing stages; keep data ingestion, transformation, and analysis within one Project to maintain clear lineage and enable seamless collaboration.
Projects are team workspaces, not personal sandboxes, so use shared Projects with role-based access control rather than creating individual Projects per team member.
Small proof of concepts (POC) or temporary experiments should be conducted within the parent Project, with successful initiatives promoted to dedicated production Projects only when they mature into full-scale capabilities requiring independent governance.

Conclusion

Throughout this post, we explored how the separation of governance and working accounts forms the foundation of a scalable, secure, and compliant data and AI platform.

With centralized governance in the Domain account, organizations can enforce consistent security policies, compliance requirements, and cost management, while allowing sub-accounts the autonomy over their own resources. This approach enhances security and compliance, and fosters collaboration and innovation within teams by providing them with the flexibility that they need to operate effectively. Ultimately, this governance-first strategy supports keeping data remains protected and accessible in a controlled manner, empowering teams to drive business outcomes efficiently. To implement a multi-account deployment for your organization, get started by creating your first SageMaker Unified Studio Domain and follow the step-by-step guidance to establish your governance-first architecture.

About the authors

Filter catalog assets using custom metadata search filters in Amazon SageMaker Unified Studio

2026-03-19 Ramesh H Singh

Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/filter-catalog-assets-using-custom-metadata-search-filters-in-amazon-sagemaker-unified-studio/

Finding the right data assets in large enterprise catalogs can be challenging, especially when thousands of datasets are cataloged with organization-specific metadata. Amazon SageMaker Unified Studio now supports custom metadata search filters. You can filter catalog assets using your own metadata form fields like therapeutic area, data sensitivity, or geographic region rather than relying only on free-text search. Custom metadata forms are structured templates that define additional attributes that can be attached to catalog assets.

In this post, you learn how to create custom metadata forms, publish assets with metadata values, and use structured filters to discover those assets. We explore a healthcare and life sciences use case. A research organization catalogs metrics in Amazon SageMaker Catalog using custom metadata forms with fields such as Therapeutic Area and Sample Size. Researchers building Machine learning models can now search datasets based on custom filters across hundreds of cataloged assets to identify the best datasets to train their models.

Key capabilities

Custom metadata search filters in SageMaker Unified Studio offer the following key capabilities:

Custom metadata form filters – You can filter search results using any custom metadata form fields defined in their catalog. For example, a researcher can filter by Therapeutic Area = Oncology and Data Sensitivity = Confidential to locate specific datasets.
Name and description filters – You can add filters that target asset names or descriptions using a text search operator, enabling targeted discovery without scanning full search results.
Date range filters – You can filter assets by date using on, before, after, and between operators, making it straightforward to locate recently updated or historically relevant assets.
Combinable filters – You can combine multiple filters to construct precise queries. For example, filtering by AWS Region = US AND Classification = PII AND Updated after 2026-01-01 returns only assets matching all three criteria.
Persistent filter selections – You can filter configurations stored in your browser and are not shared across devices or other users. You can later return to the catalog and find your previously defined filters.

Solution overview

In the following sections, we demonstrate how to set up custom metadata forms, publish assets with metadata values, and use custom metadata search filters to discover those assets.We complete the following three steps for the demonstration.

Create a custom metadata form
Create and publish assets with metadata
Use custom metadata search filters

Prerequisites

To follow along with this post, you should have:

An Amazon SageMaker Unified Studio domain setup
An existing project

For instructions on setting up a domain and project, see the Getting started guide.

To create a custom metadata form

Complete the following steps to create a custom metadata form with filterable fields:

In SageMaker Unified Studio, choose Project overview from the navigation pane.
Under Project catalog, choose Metadata entities.
Choose Create metadata form.
To create a new metadata form ‘research_metadata’ use the following details, then choose Create metadata form.
Define the form fields. For this demo, we add the following fields:
Create first field Therapeutic Area (String) – Mark as Searchable

Create second field Subject Count (Integer) – Mark as Filterable by range
Mark the form as ‘Enabled’ so the form is visible and can be used.

Create and publish with metadata

In this section, you create a custom asset and attach the research_metadata form created in the previous step.

Under Project catalog in the navigation pane, choose Metadata entities. Choose the ‘ASSET TYPES’ tab and select “CREATE ASSET TYPE’.
Create a new asset type and attach the metadata form that we created in the previous step.

A new asset type ‘metric’ is created.
Next, we will create two metrics. Under Project catalog in the navigation pane, choose Assets. On the Asset page, choose CREATE, and then choose Create asset from the menu.
In this demo, you create two metrics.

For the first metric ‘drug_1_treatment’, provide the following asset name and description.

Add the following values for the metadata form.

Validate all fields and choose CREATE.

Publish the asset to the catalog.

Next, we will create the second metric ‘drug_1_treatment’. Repeat the steps from the previous procedure and enter the values shown.

Subject Count = 450
Therapeutic Area = Oncology

Use custom metadata search filters

After publishing assets with custom metadata, go to the Browse Assets page to use the filters.

To browse assets and view filters

In SageMaker Unified Studio, choose Discover from the navigation bar, then select Catalog, Browse Assets.
The search page displays with the filter sidebar on the left. You can see the existing system filters (Data type, Glossary terms, Asset type, Owning project, Source Region, Source account, Domain unit) along with the new Date range and Add Filter sections.

Add a custom filter

Choose + Add Filter at the bottom of the filter sidebar. For Filter type, select Metadata form. For Metadata form, select research_metadata and add a filter as shown in the following image. Choose Apply when you’re done.

The search results update to show only assets where ‘subject_count’ is greater than 50.

To combine multiple filters

Choose + Add Filter again. For Filter type, select Metadata form. For Metadata form, select research_metadata and add a filter as shown in the following image. Choose Apply when you’re done.

Manage custom filters

Filter configurations are stored in the user’s browser and are not shared across devices or users.

To customize search, you could:

Toggle filters – Use the checkboxes next to each custom filter to enable or disable them without deleting.
Edit or delete – Choose the kebab menu (⋮) next to any custom filter to edit its values or delete it.
Clear all – Choose CLEAR next to the Custom filters header to deselect all custom filters at once.
Persistence – Your custom filters persist across browser sessions. When you return to the Browse Assets page, your previously defined filters are still listed in the sidebar, ready to be activated.

Using the SearchListings API

To search catalog assets programmatically, you can use the SearchListings API in Amazon DataZone, which supports the same filtering capabilities as the SageMaker Unified Studio UI. The following example filters assets where a custom string field contains a specific value and a numeric field is within a range:

aws datazone search-listings \
    --domain-identifier "dzd_your_domain_id" \
    --filters '{ "and": [
        { "filter": { "attribute": "research_metadata.TherapeuticArea", "value": "Oncology", "operator": "TEXT_SEARCH" } },
        { "filter": { "attribute": "research_metadata.SubjectCount", "intValue": 100, "operator": "GT" } }
    ] }'

For more details, see the SearchListings API documentation in the Amazon DataZone API Reference.

Best practices

Consider the following best practices when using custom metadata search filters:

Define your metadata forms before publishing assets at scale. If you publish assets before the forms are finalized, you might need to re-tag existing assets, which is a time-consuming process in large catalogs.
Define metadata forms aligned with your organization’s discovery needs (therapeutic areas, data classifications, geographic regions) before publishing assets at scale.
Use specific, consistent values in metadata fields to get precise filter results. For example, use standardized values (for example, use “Oncology” consistently rather than “oncology” or “Onc”) across all assets.
Combine multiple filters to narrow results efficiently rather than scanning through broad result sets.
Use the date range filter alongside custom metadata filters to locate assets within specific time windows.

Clean up resources

For instructions on deleting the added assets, see Delete an Amazon SageMaker Unified Studio asset.
For instructions on deleting the metadata forms, see Delete a metadata form in Amazon SageMaker Unified Studio.

Conclusion

Custom metadata search filters in Amazon SageMaker Unified Studio give data consumers the ability to find exact assets using structured filters based on their organization’s own metadata fields. By combining multiple filters across custom metadata forms, asset names, descriptions, and date ranges, data consumers can construct precise queries that surface the right datasets without scanning through broad search results. Filter persistence across browser sessions further streamlines repeated discovery workflows.

Custom metadata search filters are now available in AWS Regions where Amazon SageMaker is supported.

To learn more about Amazon SageMaker, see the Amazon SageMaker documentation. To get started with this capability, refer to the Amazon SageMaker Unified Studio User Guide.

About the authors

How Swiss Life Germany automated data governance and collaboration with Amazon SageMaker

2026-02-25 Tim Kopacz

Post Syndicated from Tim Kopacz original https://aws.amazon.com/blogs/big-data/how-swiss-life-germany-automated-data-governance-and-collaboration-with-amazon-sagemaker/

Data has become an indispensable strategic asset for the entire financial services industry, driving innovation and competitive advantage in an increasingly digital marketplace. At Swiss Life Germany, maximizing the value of this asset means empowering internal teams to derive actionable insights and deliver personalized financial solutions to diverse clientele. This led to the need to establish seamless data sharing workflows that enhance cross-departmental collaboration while maintaining strict security and compliance standards. To accomplish this, Swiss Life Germany decided to implement advanced data processing and governance capabilities using Amazon SageMaker.

Integrating SageMaker into a highly regulated enterprise environment required aligning the service’s agility with Swiss Life’s rigorous infrastructure as code (IaC) automation standards. This post demonstrates how Swiss Life Germany addressed these sophisticated deployment requirements by developing a custom Terraform pattern designed specifically for platform engineers and data architects.

Swiss Life Germany cloud journey

Swiss Life Germany is a leading provider of customized pension products and financial advice. Building on over 100 years of delivering insurance, retirement planning, and wealth management solutions, a key driver of the company’s recent evolution was the strategic transition from legacy on-premises data centers to a modern, cloud-centric architecture. After an extensive evaluation of various providers, Swiss Life Germany selected Amazon Web Services (AWS) as the strategic foundation to modernize their data operations. By using AWS, the organization was able to transition from capital-intensive data centers to a flexible pay-as-you-go model, significantly reducing the operational costs.

Following their comprehensive AWS cloud migration over the last two years—combining 30% re-platforming with 70% lift-and-shift strategies—Swiss Life Germany modernized infrastructure management through IaC. The company introduced the governance concept of an IT System. An IT System is a fundamental unit of management that defines a software component regardless of its origin. Whether a component is purchased from a vendor, self-developed or consumed as software as a service (SaaS), it’s integrated into this single governance structure. This ensures that off-the-shelf products and custom-coded applications are held to the same high standards of visibility and accountability. Every IT system is required to maintain specific attributes that allow for seamless oversight such as unique identifiers, assigned ownership and the associated AWS resources logically grouped under the IT System they support.

Where traditional approaches would store and expose this information in configuration management database (CMDB)-like systems to store static snapshots of asset data, Swiss Life adopted a more dynamic model. By using GraphQL API as a unified meta-model, the company queries application data directly from its primary source systems. This approach eliminates the delays common in batch-processed databases, ensuring maximum freshness. The API serves as a single entry point for infrastructure data, documentation, organizational metadata, and even inter-application dependencies. The transparency and automation gained through this everything-as-code and API-first approach provided a blueprint for the Swiss Life Data Platform: complete transparency, reproducibility, and end-to-end automation.

This robust technical foundation served as a catalyst and prerequisite for Swiss Life’s broader strategic goals and governed framework.

Defining the vision for a unified data solution

With the architectural foundations in place, the next challenge was to establish efficient data flows from production systems through data engineering teams to end users across various business divisions, with hundreds of specific use cases demanding attention.

For instance, Swiss Life’s customer portal specialists had to validate the effectiveness of campaign management and push notification systems in real-time, requiring secure and immediate access to interaction data.

Security requirements added another layer of complexity, because Swiss Life’s solution needed to incorporate robust compliance standards including two-factor authentication, session-based access controls, and granular row and column-level security protections.

To align with the overarching Swiss Life Germany cloud strategy, the company aimed to build a modern data solution atop their existing AWS data and analytics services. AWS introduced SageMaker to Swiss Life Germany following its announcement at AWS re:Invent 2024. A proof-of-concept quickly validated that this was the right tool to advance Swiss Life’s data journey. By deploying a fully automated framework, Swiss Life Germany sought to create a secure, compliant framework with SageMaker democratizing data access for authorized users, ultimately enabling faster business insights and more responsive customer experiences across the entire data environment.

Having met the infrastructure requirements, let’s look at what SageMaker looks like for end users and how data platform administrators can control access and resources at a granular level.

Users and their types of projects

A typical end user experience within Amazon SageMaker Unified Studio starts with creating a project. A project is a logical boundary within a domain where the data teams can collaborate and work on a business use case. Administrators would provision the blueprints and project profile templates for the data teams, as shown in the following figure.

However, at Swiss Life, they have extended the data platform administrator’s role to also create projects so they can maintain regulatory compliance and remove initial onboarding hurdles. The end user experience in SageMaker Unified Studio is simplified with data teams selecting their respective projects to work on a business initiative, as shown in the following figure.

To implement this solution effectively, Swiss Life identified different user groups:

A solution team developing an IT System that can act as producer or consumer of data assets.
A data scientist doing advanced data processing. They will most likely consume a lot of data assets and might produce some high aggregated data assets. The data processing software is also categorized as an IT System.
Business users who have some SQL skills and want to process data to get insights for their daily business.
A platform team administering the data platform. They provide core services to all users to make participation as straightforward as possible.
A data officer who wants to have a single point of interpretation for data.

Given this diverse set of user groups, the resulting data platform had to support a federated data organization with a centralized governance, decentralized data stores and data-processing organized at the IT System level. This architecture means the SageMaker management account—which orchestrates the data domain—contains no actual data, instead, data and compute resources reside in the individual IT System AWS accounts. Swiss Life’s implementation distinguishes between two fundamental project types:

IT System projects (for technical users)
Team projects (for non-technical users)

Swiss Life decided to align team projects with specific organizational units and operate them without staging environments, providing dedicated workspaces for departmental data initiatives. In contrast, IT System projects are associated with specific solutions such as customer portal or CRM systems. These follow a structured staging methodology, with each solution team managing dedicated DEV, TEST, and PROD environments to maintain proper development lifecycles and quality control.

This federated architecture is designed to handle the immense scale and diversity of Swiss Life’s data landscape. Swiss Life’s data platform would then aim to provide unified access to over 180 database servers with over 1,800 databases and 18 thousand tables across all stages (DEV, TEST and PROD).

In this post, we focus on the IT System projects.

How Swiss Life built the automation framework

Because Terraform is the preferred IaC tool across Swiss Life Germany, the team faced an interesting architectural challenge: while the existing infrastructure framework incorporates numerous AWS services that are readily supported by Terraform, SageMaker required a custom integration approach to align with Swiss Life’s advanced automation patterns.

Rather than adopting a manual ClickOps approach to infrastructure management, Swiss Life developed an innovative solution to keep the entire infrastructure—including SageMaker—within their Terraform automation, preserving key benefits like state management. The team accomplished this by using Terraform’s AWS Lambda invoke function resource with a create, read, update, delete (CRUD) lifecycle scope. By using this approach, the organization could maintain a single source of truth for infrastructure, while accommodating specific requirements of SageMaker. This component is called the Management Lambda and it serves as a bridge between Terraform’s declarative configuration and SageMaker, so that Swiss Life can provision, modify, and decommission Amazon SageMaker resources through established Terraform workflows.

The following is the snippet of a new domain creation using Terraform and Management Lambda:

resource"aws_lambda_invocation" "domain" {
  function_name = "management-lambda-function-name"
  lifecycle_scope = "CRUD"
  input = jsonencode({
  resource = "domain"
  domain_name = "SwissLife"
  domain_execution_role = "arn:aws:iam::012345678912:role/sus_domain_execution_role"
  domain_service_role = "arn:aws:iam::012345678912:role/sus_service_role"
  })
}

Using this approach, Swiss Life successfully automated every aspect of deploying a complete SageMaker domain installation within the Swiss Life cloud data platform. The automation encompasses the entire domain creation process, using the SageMaker domain unit feature as an organizational framework for diverse project portfolio.

Deployment architecture

Let’s dive deeper into the individual steps of the automation process itself. As said, all resources within SageMaker are controlled by the Terraform-invoked Management Lambda whereas other resources are directly managed by Terraform itself. The Management Lambda and SageMaker resources such as domains, metadata fields and others live in the central SageMaker account. Users of the data platform have their own AWS accounts. To start with, AWS Lake Formation had to be enabled across all AWS accounts, which could then act as consumer or provider to the platform. Using the established AWS Landing Zones mechanism, this was done by a single deployment to the management account. This early step also verified the management role being present in all accounts and assumable by the Management Lambda.

The following steps are used to set up Swiss Life’s data platform from scratch, as shown in the following diagram:

The Management Lambda is deployed to Swiss Life’s designated SageMaker account. This Lambda function uses the described CRUD pattern for all subsequent SageMaker-specific operations.
The domain provisioning begins by creating the service and domain execution roles, after which the Management Lambda creates the domain and uses these roles. During this step, administrative users and their associated permissions are also configured.
Upon successful domain creation, the Lambda function returns the domain identifier as output. This identifier is then used to let all AWS accounts of the company join this domain. These can now act as providers or consumers on the platform, resulting in a frictionless onboarding of teams.
Because Swiss Life decided to stage data products in a single domain, the DEV, TEST, and PROD domain units are then created, establishing the hierarchical structure under which IT System projects are subsequently created in the next implementation phase.

All projects and teams with the necessary prerequisites set up are then created automatically. This is done by using the enterprise GraphQL API mentioned to retrieve all IT products, their teams and roles. With that, each team already has their ready-to-use project in place upon singing into the platform. In detail this process looks like the following:

Continuing with the earlier example: the customer portal team needs to share their data with others in the organization and is using their dedicated project for this purpose. The process is shown in the following figure.

The deployment initiates with a cross-account role assumption by the Management Lambda to activate the blueprint configuration in the team’s AWS account. A standardized creation process was built to help facilitate all accounts are configured identically, maintaining consistency across the environment.
Next, a project profile specifically tailored for the customer portal project is created. This profile establishes the foundational settings and permissions framework that will govern the project’s operations.
With the profile in place, the actual project within this previously established project profile can now be provisioned, instantiating the working environment, where data sharing and collaboration will occur. This results in an identical amount of project profiles and projects in the SageMaker Unified Studio domain.
Finally, an automated membership management process is triggered. The system again queries Swiss Life’s Enterprise GraphQL API to identify all members of the solution team and automatically adds them as project members with appropriate permissions. This process executes daily, to help ensure that project access permissions remain current and accurately reflect team composition changes.

In the third and final deployment step, the user experience is enhanced by making the data platform immediately usable for teams in production. When teams and their members first access the domain URL, they find a project environment already populated with all necessary assets, so they can begin working without delay. This is accomplished through the following steps, shown in the following figure:

An automated discovery process is triggered that identifies all Amazon Simple Storage Service (Amazon S3) buckets and AWS Glue assets associated with the specific customer portal IT System. This inventory is created by using the AWS Resource Tagging API with specific filters targeting these asset types, so that all relevant resources for exactly that IT System are captured.
When identified, all discovered S3 buckets are registered as data lake locations within the platform. For each location, they create an AWS Identity and Access Management (IAM) role with precise access permissions, adhering to the least privilege security model.
Then grantable permissions are granted to the SageMaker project role for these assets, establishing a permission delegation framework that allows project members to manage access within their project scope—managing cross project access—while maintaining overall governance.
Finally, the AWS Glue databases are added as data sources within the project. These data sources are configured with daily synchronization schedules to automatically load new metadata into SageMaker, helping to ensure that catalog information remains current without manual intervention.

What a team needs to start with all of this

The overarching goal throughout this implementation has been to simplify the adoption process for the internal data teams. To ensure the data teams could immediately use the powerful capabilities of SageMaker without needing to manage its underlying architecture, Swiss Life Germany streamlined the experience by pre-packing the entire onboarding process into a high-level Terraform module. Teams can then use the module to deploy a complete, production-ready environment with minimal configuration, accelerating their path from setup to insight.

The following is an example of the code used by the module.

module "membership" {
	source = "<git-source>"
	it_system_labels = ["kundenportal"]
	domain_name = "SwissLife"
	vpc_id = "vpc_id"
	subnet_ids = ["subnet_a", "subnet_b", "subnet_c"]
}

To initiate this, the data teams define their basic parameters such as network configuration or their IT-System identifier as outlined previously and submit a pull request in the central Git repository. After the Swiss Life data platform team reviews and approves the request, the automated processes run in the background, preparing the complete environment. This automated approach has reduced deployment time for new environments from several weeks of manual coordination to under 20 minutes.

Rather than requiring users to understand the intricate deployment steps and managing the infrastructure, the automated deployment process empowers business units, like the customer portal team, to focus on deriving insights. At the same time, the Swiss Life Germany data platform team also maintains precise control over resource allocations, access rights and cost management.

Future enhancements

Looking ahead, Swiss Life plans to elevate its automation to a higher level of business abstraction. The next major enhancement focuses on removing the requirement for teams to request specific technical assets. Instead, the vision is to implement an intuitive interface where teams can specify the business terms or data domains they require. The system will automatically identify and provision the correct underlying technical assets associated with those business definitions.

This semantic layer will create a more natural interaction model, so that business users can think and work in familiar concepts rather than technical constructs. For example, rather than requesting access to specific S3 buckets or AWS Glue databases, a marketing analyst might indicate they need customer interaction data or campaign response metrics. An automated system will then map these business terms to the appropriate technical resources, provision access, and configure the environment accordingly.

By elevating automation to this business terminology level, Swiss Life aims to further reduce friction in the data access process while maintaining its robust security and governance framework. This evolution represents Swiss Life Germany’s commitment to continuously improving how data serves the business, making sophisticated data capabilities increasingly accessible to all parts of the organization.

Conclusion

Through the comprehensive automation of Amazon SageMaker, Swiss Life Germany has transformed their usage of data from a complex technical challenge into a streamlined business enabler. By using AWS services and their innovative Terraform-Lambda integration approach, Swiss Life created a secure, compliant data platform that maintains governance while democratizing access across the full organization. The automated deployment process helps ensure consistency across environments while dramatically reducing the technical knowledge required for teams to begin using advanced data capabilities. Business units, such as the customer portal team, can now focus on deriving insights rather than managing infrastructure, accelerating data-driven decision making throughout the company. This implementation represents a significant milestone in Swiss Life Germany’s cloud journey, demonstrating how thoughtful automation can simultaneously enhance security, improve operational efficiency, and accelerate business outcomes.

As of today, 5 organizational unit teams and 15 IT System teams were onboarded to the platform. To speed things up, Swiss Life has decided to onboard all 180 database clusters and consume data using SageMaker over the coming months. This expansion is designed to enable teams to use the data platform and enhance the efficiency of data discovery and data sharing processes across the organization.

About the authors

Amazon SageMaker AI now hosts NVIDIA Evo-2 NIM microservices

2026-02-24 Malvika Viswanathan

Post Syndicated from Malvika Viswanathan original https://aws.amazon.com/blogs/compute/amazon-sagemaker-ai-now-hosting-nvidia-evo-2-nim-microservices/

This post is co-written with Neel Patel, Abdullahi Olaoye, Kristopher Kersten, Aniket Deshpande from NVIDIA.

Today, we’re excited to announce that the NVIDIA Evo-2 NVIDIA NIM microservice are now listed in Amazon SageMaker JumpStart. You can use this launch to deploy accelerated and specialized NIM microservices to build, experiment, and responsibly scale your drug discovery workflows on Amazon Web Services (AWS).

In this post, we demonstrate how to get started with these models using Amazon SageMaker Studio.

NVIDIA NIM microservices on AWS

NVIDIA NIM integrates closely with AWS managed services, such as Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Kubernetes Service (Amazon EKS), and Amazon SageMaker AI, to support deployment of generative AI models at scale. As part of NVIDIA AI Enterprise, which is available in the AWS Marketplace, NVIDIA NIM is a set of microservices designed to accelerate the deployment of generative AI. These prebuilt containers support a broad spectrum of generative AI models, from open source community models, to NVIDIA Nemotron and custom models. NIM microservices are deployed with just a few lines of code, or with a few actions in the SageMaker Studio console. Engineered to facilitate seamless generative AI inferencing at scale, NIM ensures that generative AI applications can be deployed on various AWS services.

NVIDIA BioNeMo Evo 2 overview

NVIDIA BioNeMo is a platform of NIM microservices, developer tools, and AI models that accelerate building, adapting, and deploying biomolecular AI models for drug discovery. It packages curated training recipes, data loaders, and domain-optimized pretrained models for DNA, RNA, and proteins, alongside NVIDIA CUDA-X libraries such as NVIDIA cuEquivariance. These components power tasks such as 3D structure prediction, de novo design, virtual screening, docking, and property prediction with GPU-accelerated performance.

NVIDIA NIM microservices provide optimized, API-first inference that integrates directly into enterprise pipelines across on-premises and the cloud, providing scalable and secure deployment with faster time-to-market and lower Total Cost of Ownership (TCO). The Evo 2 NIM delivers a 40-billion parameter foundation model (FM) trained on a vast dataset of genomes that can be used to predict protein function, identify mutations, and accelerate bioengineering research. Furthermore, the Evo 2 NIM can be chained with other NIM microservices such as ESMFold to create end-to-end, containerized workflows that cut time-to-insight while streamlining deployment through consistent APIs.

SageMaker Studio overview

SageMaker Studio is a web-based integrated development environment (IDE) for machine learning (ML) that provides a unified visual interface for all of the tools that you need to complete each step of the ML development lifecycle. SageMaker Studio provides complete access, control, and visibility into each step of the ML workflow, from data preparation to model building, training, and deployment.

The key features of SageMaker Studio include:

Unified interface: Access all SageMaker capabilities through a single, web-based visual interface
Jupyter notebooks: Fully managed Jupyter notebooks with pre-configured kernels for popular ML frameworks
Model management: Browse, deploy, and manage models from AWS Marketplace and other sources through an intuitive interface
Collaboration: Share notebooks, experiments, and models with your team members
Built-in security: Integrated with AWS Identity and Access Management (IAM) for secure access control
Cost management: Monitor and control costs with built-in usage tracking and resource management tools

Amazon SageMaker JumpStart overview

SageMaker JumpStart is a fully managed service that offers state-of-the-art foundation models for various use cases such as content writing, code generation, question answering, copywriting, summarization, classification, and information retrieval. It provides a collection of pre-trained models that you can deploy quickly, accelerating the development and deployment of ML applications. One of the key components of SageMaker JumpStart is model hubs, which offer a vast catalog of pre-trained models, such as Mistral, for a variety of tasks. You can now discover and deploy Evo 2 NIM in Amazon SageMaker Studio or programmatically through the SageMaker Python SDK, so you can derive model performance and MLOps controls with Amazon SageMaker AI features such as Amazon SageMaker Pipelines, Amazon SageMaker Debugger, or container logs. The model is deployed in a secure AWS environment and in your VPC, helping to support data security for enterprise security needs.

Prerequisites

Before getting started with deployment, make sure that your IAM service role for SageMaker AI has the SageMakerFullAccess permission policy attached. To deploy the NVIDIA NIM microservices successfully, confirm one of the following:

Make sure that your IAM role has the following permissions, and that you have the authority to make AWS Marketplace subscriptions in the AWS account used:

aws-marketplace:ViewSubscriptions
aws-marketplace:Unsubscribe
aws-marketplace:Subscribe

If your account is already subscribed to the model, then you can skip to the following Deploy section. Otherwise, start by subscribing to the model package and move to the Deploy section after.

Subscribe to the model package

To subscribe to the model package, complete the following steps:

Open the SageMaker Jumpstart portal from the SageMaker AI page.
Search for Evo 2 NIM.
Choose View model, and on the Model details page choose Subscribe. This will take you to the AWS Marketplace listing for the Evo 2 NIM.
On the AWS Marketplace listing page, choose View purchase options, review the purchase terms and choose the Subscribe button if you and your organization agree with EULA, pricing, and support terms.
Choose Continue to with the configuration and choose an AWS Region where you have the service quota for the desired instance type.

A product Amazon Resource Name (ARN) is displayed. This is the model package ARN that you need to specify while creating a deployable model using the SageMaker SDK.

Option 1: Deploy the Evo 2 NIM using SageMaker Studio

The following section outlines how to deploy the EVO 2 NIM using SageMaker Studio.

Getting started with SageMaker Studio

Begin by accessing the AWS Management Console and navigating to the SageMaker AI service. When you’re in the SageMaker AI console, locate Studio in the left navigation panel and choose Open Studio next to your user profile. If you haven’t set up a SageMaker Studio domain yet, then you must create a new domain and user profile first. This launches the web-based SageMaker Studio interface where you can manage all aspects of your ML workflow.

Navigating to model packages

Within SageMaker Studio, look for Models in the left sidebar and choose JumpStart base models tab within the Models interface. This section contains all available model packages in SageMaker JumpStart, including those from the AWS Marketplace

Locating the Evo-2 NIM model

Use the search functionality to find the NVIDIA Evo-2 NIM model by searching for terms such as “Evo-2” or “NVIDIA”. When you locate the model package in the filtered results, choose it to view the Model overview page. This page provides an overview of the model and can have a Notebooks tab that will show a sample notebook that contains an example showing how to use the NIM. You can choose Open in JupyterLab to open the notebook in JupyterLab and use it as a starting point for using the NIM.

Configuring the model deployment

On the model package overview page, choose the Deploy button on the top right to begin the deployment process. You must configure several important settings: provide a unique endpoint name (such as “Evo-2-nim-endpoint”), choose an appropriate instance type (ml.g6e.12xlarge is recommended for optimal performance), set the initial instance count (typically 1 for initial testing), and specify an endpoint configuration name. Review all of these settings carefully before proceeding.

Initiating and monitoring the deployment

After verifying your configuration settings, choose Deploy to start the deployment process for creating a Real-time inferance endpoint. Navigate to the Deployments section and then the Endpoints section in the left sidebar to monitor the deployment progress. The endpoint status initially shows Creating and typically takes 5–10 minutes to complete. You can track the progress and should see the status change to InService once the deployment is successful.

Testing and validation

When your endpoint is deployed and shows the In Service status, you can optionally test it directly through the SageMaker Studio interface. Choose your deployed endpoint from the endpoints list to access the Endpoint summary page. Scroll down and select the Playground tab. If available, you will see two options: Test the sample request and Use Python SDK example code. You can use either option to validate the deployment by using a sample protein sequence. This validates the endpoint is working correctly before integrating it into your applications.

Option 2: Deploy Evo 2 using the SageMaker SDK

In this section we walk through deploying the Evo-2 NIM through the SageMaker SDK. Make sure that you have the account-level service limit for using ml.g6e.12xlarge for endpoint usage as one or more instances. Furthermore, NVIDIA provides a list of supported instance types that support deployment. Refer to the AWS Marketplace listing for the model to see the supported instance types. To request a service quota increase, go to the AWS service quotas.

import sagemaker
import boto3
from sagemaker import ModelPackage, get_execution_role
import json
# Initialize SageMaker session and role
role = get_execution_role()
sagemaker_session = sagemaker.Session()
# Model Package ARN from your AWS Marketplace subscription
# Replace this with your actual Model Package ARN after subscription
model_package_arn = "arn:aws:sagemaker:<region>:<account-id>:model-package/Evo-2-nim-model"
# Create model from AWS Marketplace Model Package
model = ModelPackage(
    role=role, 
    model_package_arn=model_package_arn,
    sagemaker_session=sagemaker_session
)
# Deploy the model to an endpoint
predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g6e.12xlarge",  # Using recommended NVIDIA GPU instance
    endpoint_name="Evo-2-endpoint",
    wait=True
)

Run Inference with Evo 2 SageMaker endpoint

When you have the model, you can use a sample text to do an inference request. NIM on SageMaker supports the OpenAI API inference protocol inference request format. For an explanation of the supported parameters, go to the Evo-2 API documentation.

Real-time inference example

sm_runtime = boto3.client("sagemaker-runtime", region_name=region)

generate_payload = {

 "sequence": "ACGTACGTACGT",

 "num_tokens": 100,

 "temperature": 0.7,

 "top_k": 3,

}

response = sm_runtime.invoke_endpoint(

EndpointName='Evo2-40b-2-1-0',

ContentType="application/json",

Body=json.dumps(generate_payload),

)

result = json.loads(response["Body"].read())

print("Generated DNA:", result["sequence"])
print("Elapsed (ms):", result.get("elapsed_ms"))

Example output:

Generated DNA: ACGTACATATGTTCGTACATTCGCACAGACGCCATTTTGAAAAATGCTTTAAATGGATTCAGAATTGGTCAAAATGCATAAATCCATCAAAATTTTTTTC
Elapsed (ms): 10770

Cleaning up

To avoid unwanted charges, complete the steps in this section to clean up your resources.

Deleting the endpoint from SageMaker Studio

In SageMaker Studio, navigate to the Endpoints section in the left sidebar under Inference to view all your active endpoints. Locate your Evo-2 NIM endpoint in the list and select it to open the endpoint details page. On this page, there is a Delete button. Choose Delete and confirm the deletion when prompted. The endpoint status changes to Deleting and disappears from your endpoints list when the deletion is complete. This process typically takes a few minutes, and when it’s deleted the endpoint stops incurring charges immediately.

Delete the SageMaker endpoint

The SageMaker endpoint that you deployed incurs costs if you leave it running. Use the following code to delete the endpoint if you want to stop incurring charges. For more details, go to Delete endpoints and resources.

# Delete endpoint when done (important for cost management)
predictor.delete_endpoint()

Conclusion

The availability of NVIDIA Evo-2 NIM microservices on Amazon SageMaker Jumpstart represents a significant advancement for researchers and organizations working in drug discovery. This solution provides GPU-accelerated multiple sequence alignments and dramatically speeds up structure prediction pipelines that are critical for protein design and antibody research. Users can implement the flexible deployment options—through SageMaker Studio, or SageMaker SDK—to choose the approach that best fits their workflow and technical expertise. The optimized performance of these NIM microservices, combined with the scalability and security of SageMaker, enables faster time-to-insight while streamlining the deployment of complex biomolecular AI models. We encourage you to try the Evo-2 NIM today and look out for future release of MSA-search and Boltz-2 NIMs to accelerate your drug discovery workflows and use the power of NVIDIA’s specialized microservices on AWS infrastructure.

Implement a data mesh pattern in Amazon SageMaker Catalog without changing applications

2026-02-23 Paolo Romagnoli

Post Syndicated from Paolo Romagnoli original https://aws.amazon.com/blogs/big-data/implement-a-data-mesh-pattern-with-amazon-sagemaker-catalog-without-making-changes-to-your-applications/

When creating a project in Amazon SageMaker Unified Studio, users select a project profile to define resources and tools to be provisioned in the project. These are used by Amazon SageMaker Catalog to implement a data mesh pattern. Some users don’t want to take advantage of resources provisioned along with the project for various reasons. For instance, they may want to avoid making changes to their existing applications and data products.

This post shows you how to implement a data mesh pattern by using Amazon SageMaker Catalog while keeping your current data repositories and consumer applications unchanged.

Solution overview

In this post, you will simulate a scenario based on data producer and data consumer that exists before Amazon SageMaker Catalog adoption. For this purpose, you will use a sample dataset to simulate existing data and simulate an existing application using an AWS Lambda function. You can apply the same solution to your real-life data and workloads.

The following diagram illustrates the solution architecture’s key configurations. In this architecture, the Amazon Simple Storage Service (Amazon S3) bucket and the AWS Glue Data Catalog in the producer account simulate the existing data repository. The Lambda function in the consumer account simulates the existing consumer application.

AWS cross-account data sharing via SageMaker & Lake Formation: Producer publishes to catalog, Consumer subscribes & accesses data

Here is a description of the key configurations highlighted in the architecture:

As part of an Amazon SageMaker domain, create a producer project (associated to a producer account) and a consumer project (associated to a consumer account). Among other resources, a project AWS Identity and Access Management (IAM) role is created for each project in the associated account.
In the producer account, use AWS Lake Formation to grant producer project’s IAM role permissions to access the existing data asset.
Publish the data asset in the Amazon SageMaker Catalog from the producer project.
Subscribe the data asset from the consumer project.
In the consumer account, configure your Lambda function to assume consumer project’s IAM role to access the subscribed data asset.

The solution architecture is based on the following Amazon Web Services (AWS) services and features:

Amazon SageMaker Catalog offers you a way to discover, govern, and collaborate on data and AI securely.
Amazon SageMaker Unified Studio provides a single data and AI development environment to discover and build with your data. Amazon SageMaker Unified Studio projects provide collaborative boundaries for users to accomplish data and AI tasks.
The lakehouse architecture of Amazon SageMaker is fully compatible with Apache Iceberg. It unifies data across Amazon S3 data lakes, Amazon Redshift data warehouses, and third-party and federated data sources.
AWS Lake Formation, which you can use centrally to govern, secure, and share data for analytics and machine learning.
AWS Glue Data Catalog is a persistent metadata store for your data assets. It contains table definitions, job definitions, schemas, and other control information to help you manage your AWS Glue environment.
Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance.

Setting up resources

In this section, you will prepare the resources and configurations you need for this solution.

Three AWS accounts

To follow this solution, you need three AWS accounts, and it’s better if they’re part of the same organization in AWS Organizations:

Producer account – Hosts the data asset to be published
Consumer account – Hosts the application that consumes the data published from the producer account
Governance account – Where the Amazon SageMaker Unified Studio domain is configured

Each account must have an Amazon Virtual Private Cloud (Amazon VPC) with at least two private subnets in two different Availability Zones. For instruction, refer to Create a VPC plus other VPC resources. Make sure to create both VPCs in the same Region you plan to apply this solution.

A governance account is used for the sake of convenience, but it’s not strictly needed because Amazon SageMaker can be configured and managed in producer or consumer accounts.If you don’t have access to three accounts, you can still use this post to understand the key configurations required to implement a data mesh pattern with Amazon SageMaker Catalog while keeping your current data repositories and consumer applications unchanged.

Create a data repository in the producer account

First, create a sample dataset by following these instructions:

Open a text editor.

Paste the following text in a new file:

name,stars
	oak,3
	maple,2
	birch,3
	willow,4
	pine,5
	mango,1
	neem,2
	banyan,5
	eucalyptus,3
	teak,2

Save the file as trees.csv. This is your sample data file.

After you create the sample dataset, create an S3 bucket and an AWS Glue database in the producer account, which will act as the data repository.

Create the S3 bucket and upload the trees.csv file in the producer account:

Access the S3 console in the producer account.
Create an S3 bucket. For instructions, refer to Creating a general purpose bucket.
Upload to the S3 bucket the trees.csv sample data file that you created. For instructions, refer to Uploading objects.

Create the AWS Glue database and table in the producer account:

Access the Glue console in the producer account.
In the navigation pane, under Data Catalog, choose Databases.
Choose Add database.
For Name, enter collections.
For Description, enter This database contains collections of statistics for natural resources.
Choose Create database.
In the navigation pane, under Data Catalog, choose Tables.
Choose Add table.
In the table creation guided procedure, enter the following input for Step 1: Set table properties:
1. For Name, enter trees.
2. For Database, select collections.
3. For Description, enter This table captures ratings data related to the characteristics of various tree species.
4. For Table format, select Standard AWS Glue table (default).
5. For Select the type of source, select S3.
6. For Data location is specified in, select my account.
7. For Include path, enter s3://<bucket-name>/<prefix>/ where <bucket-name> is the name of the S3 bucket you created earlier in this procedure and <prefix> is the optional prefix for the trees.csv file you uploaded.
8. For Data format, select CSV.
9. For Delimeter, select Comma (,).
Choose Next.
For Step 2: Choose or define schema, enter the following:
1. For Schema, select Define or upload a schema.
2. Choose Edit schema as JSON and enter the following schema in the pop-up:
```
[
  {
    "Name": "name",
    "Type": "string",
    "Parameters": {}
  },
  {
    "Name": "stars",
    "Type": "string",
    "Parameters": {}
  }
]
```
3. Choose Save.
4. Choose Next.
5. Choose Create.

Create a Lambda function in the consumer account

Create the Lambda function in the consumer account. This will simulate a data consumer application.First, in the consumer account create the IAM policy and the IAM role to be assigned to the Lambda function:

Access the IAM console in the consumer account.

Create an IAM policy and name it smus_consumer_athena_execution by using the following policy. Make sure to replace placeholders <AWS_Region> and <AWS_account_ID_number> with your Region and consumer account ID number. You will replace the <workgroup_id> placeholder later. For IAM policy creation instructions, refer to Create IAM policies (console).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AthenaExecution",
            "Action": [
                "athena:StartQueryExecution",
                "athena:GetQueryExecution",
                "athena:GetQueryResults"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:athena:<AWS_Region>:<AWS_account_ID_number>:workgroup/<workgroup_id>"
        }
    ]
}

Create an IAM role for AWS Lambda service and name it smus_consumer_lambda. Assign to it the AWS managed permission AWSLambdaBasicExecutionRole and the permission named smus_consumer_athena_execution that you just created. For instructions, refer to Create a role to delegate permissions to an AWS service.

After the IAM role for the Lambda function is in place, you can create the Lambda function in the consumer account:

Access the Lambda console in the consumer account.
In the navigation pane, choose Functions.
Choose Create function and enter the following information:
1. For Function name, enter consumer_function.
2. For Runtime, select Python 3.14.
3. Expand Change default execution role section.
4. For Execution role, select Use an existing role.
5. For Existing role, select smus_consumer_lambda.
Choose Create function.

Under the Code tab, in the Code source, replace the existing code with the following:

import boto3
import time
sts_client = boto3.client('sts')
role_arn = "<role_arn>"
session_name = "AthenaQuerySession"
catalog = "AwsDataCatalog"
database = "<database_name>"
workgroup = "<workgroup_id>"
query = "select * from "+catalog+"."+database+".trees"
def lambda_handler(event, context):
    # Assume SageMaker Unified Studio project role
    assumed_role_object = sts_client.assume_role(
        RoleArn=role_arn,
        RoleSessionName=session_name
    )
    # Get temporary credentials
    credentials = assumed_role_object['Credentials']
    # Create Athena client using temporary credentials
    athena = boto3.client(
        'athena',
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken'],
        region_name='eu-west-1'
    )
    # Execute Athena Query
    response = athena.start_query_execution(
        QueryString=query,
        QueryExecutionContext={
            'Database': database,
            'Catalog': catalog
        },
        WorkGroup=workgroup
    )
    query_execution_id = response['QueryExecutionId']
    # Polling with exponential backoff
    wait_time = 0.25  # Start with 0.25 seconds
    max_wait = 8      # Maximum wait time of 8 seconds
    
    while True:
        result = athena.get_query_execution(QueryExecutionId=query_execution_id)
        state = result['QueryExecution']['Status']['State']
        if state in ['FAILED', 'CANCELLED']:
            raise Exception(f"Query {state}")
        elif state == 'SUCCEEDED':
            break
        elif state in ['QUEUED', 'RUNNING']:
            time.sleep(wait_time)
            wait_time = min(wait_time * 2, max_wait)  # Double wait time, cap at max_wait
    # Retrieve results
    results = athena.get_query_results(QueryExecutionId=query_execution_id)
    return results

Choose Deploy.

The code provided for the Lambda function includes some placeholders that you will replace later, after you have the required information. Don’t test the Lambda function at this time because it will fail because of the presence of the placeholders.

Create a user with administrative access

Amazon SageMaker Unified Studio supports two distinct domain types: AWS IAM Identity Center based domains and IAM based domains. At the time of writing this post, only IAM Identity Center based domains support multi-accounts association, therefore in this post you work with this type of domain that requires IAM Identity Center.

In the governance account, you enable IAM Identity Center and create an administrative user to create and manage the Amazon SageMaker Unified Studio domain. Create a user with administrative access:

Enable IAM Identity Center in the governance account. For instructions, refer to Enable IAM Identity Center.
In IAM Identity Center in the governance account, grant administrative access to a user. For a tutorial about using the IAM Identity Center directory as your identity source, refer to Configure user access with the default IAM Identity Center directory.

To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user. For help signing in using an IAM Identity Center user, refer to Sign in to your AWS access portal.

Create a SageMaker Unified Studio domain

To create the Amazon SageMaker Unified Studio domain in the governance account refer to Create a Amazon SageMaker Unified Studio domain – quick setup.

After your domain is created, you can navigate to the Amazon SageMaker Unified Studio portal (a browser-based web application) where you can use your data and configured tools for analytics and AI. Save the Amazon SageMaker Unified Studio portal URL because you will use this URL later.

Solution steps

Now that you have the prerequisites in place, you can complete the following ten high-level steps to implement the solution.

Associate the producer and consumer accounts to the Amazon SageMaker Unified Studio domain

Start by associating the producer and consumer accounts to the newly created Amazon SageMaker Unified Studio domain. When you associate your producer and consumer accounts to the domain, make sure to select IAM users and roles can access APIs and IAM users can log in to Amazon SageMaker Unified Studio in the AWS RAM share managed permission section. For step-by-step instructions, refer to Associated accounts in Amazon SageMaker Unified Studio. If your AWS accounts are part of the same organization, your association requests are automatically accepted. However, if your AWS accounts aren’t part of the same organization, request association with the other AWS accounts in the governance account and then accept the association request in both the producer and consumer accounts.

Create two project profiles

Now, create two project profiles, one for the producer project and one for the consumer project.

In Amazon SageMaker Unified Studio, a project profile defines an uber template for projects in your Amazon SageMaker domain. A project profile is a collection of blueprints that provides reusable AWS CloudFormation templates used to create project resources.

A project profile is associated to a specific AWS account. This means, when a project is created the blueprints listed in the project profile are deployed in the associated AWS account. To use a project profile, you must enable its blueprints in the AWS account associated to the project profile.

Create the producer project profile

You’re going to create the producer project profile that is associated to the producer account. This project profile will be used to create the producer project. This profile includes by default the Tooling blueprint that creates resources for the project, including IAM user roles and security groups.

Before creating the project profile, you will enable the Tooling blueprint in the producer account using the following procedure:

Access the SageMaker console in the producer account.
In the navigation pane, choose Associated domains.
Select the domain you created while setting up.
On the Blueprints tab, choose Enable in the Tooling blueprint section as shown in the following image:

SageMaker Unified Studios Tooling blueprint config: disabled status with Enable button for IAM roles & AWS resource setup

For Virtual private cloud (VPC) select your account VPC.
For Subnets, select at least two subnets in different Availability Zones.
Choose Enable blueprint.

Proceed to creating the project profile in the governance account:

Access the SageMaker console in the governance account.
In the navigation pane, choose Domains.
Select the domain you created as part of prerequisites.
Under the Project profiles tab, choose Create and enter the following information:
1. For Project profile name, enter producer-project-profile.
2. For Project profile creation options, select Custom create.
3. DO NOT SELECT A BLUEPRINT for Blueprints because the Tooling blueprint is included by default in any project profile.
4. For Account, select Provide an account ID.
5. For Account ID, enter the producer account ID.
6. For Region, select Provide region name and then select the Region in which you’re working.
7. For Authorization, select Allow all users and groups.
8. For Project profile readiness, select Enable project profile on creation.
Choose Create project profile.

Create a consumer project profile

You also create a consumer project profile and associate it to the consumer account. This profile will be used to create the consumer project. The consumer project profile includes the LakeHouseDatabase blueprint, which is needed to create a lakehouse environment with an AWS Glue database for data management and an Amazon Athena workgroup for querying. The Tooling blueprint is included by default in the project profile.

Before creating the project profile, enable the Tooling and LakeHouseDatabase blueprints in the consumer account:

Access the SageMaker console in the consumer account.
In the navigation pane, choose Associated domains.
Select the domain you created as part of prerequisites.
On the Blueprints tab, choose Enable in the Tooling blueprint section.
For Virtual private cloud (VPC) select your account VPC.
For Subnets, select at least two subnets in different Availability Zones.
Choose Enable blueprint.
In the navigation pane, choose Associated domains.
Select the domain you created as part of prerequisites.
Under the Blueprints tab, select the LakeHouseDatabase blueprint.
Choose Enable.
Choose Enable blueprint.

After blueprints are enabled in the consumer account, you can proceed creating the project profile:

Access the SageMaker console in the governance account.
In the navigation pane, choose Domains.
Select the domain you created as part of prerequisites.
Under Project profiles tab choose Create and enter the following information:
1. For Project profile name, enter consumer-project-profile.
2. For Project profile creation options, select Custom create.
3. For Blueprints, select LakeHouseDatabase.
4. For Account, select Provide an account ID.
5. For Account ID, enter the consumer account ID.
6. For Region, select Provide region name and then select the Region you are working.
7. For Authorization, select Allow all users and groups.
8. For Project profile readiness, select Enable project profile on creation.
Choose Create project profile.

Create SageMaker Unified Studio producer and consumer projects

In Amazon SageMaker Unified Studio, a project is a boundary within a domain where you can collaborate with other users to work on a business use case. In projects, you can create and share data and resources.To create producer and consumer projects in Amazon SageMaker Unified Studio use the following instructions:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list.
Choose Create project and enter the following information:
1. For Project name, enter Producer.
2. For Project profile, select producer-project-profile.
Choose Continue.
Choose Continue.
Choose Create project.

After you’ve created the Producer project, note in a text file the Project role ARN that is displayed in the Project overview. The following image is shown for reference. The project role name is the string that follows arn:aws:iam::<account_ID>:role/ in the project role Amazon Resource Name (ARN). You will use both project role name and ARN later.

SageMaker Producer project overview: active status, files listed, S3 location & IAM role ARN displayed in project details tab

Repeat the preceding procedure to create the Consumer project. Be sure to enter Consumer for Project name and then select consumer-project-profile for Project profile. After it’s created, note the Project role ARN in a text file. The project role name is the string that follows arn:aws:iam::<account_ID>:role/ in the project role ARN. You will use both project role name and ARN later.

Bring your own data from the producer account

Bring your own data to the Amazon SageMaker Unified Studio Producer project. AWS provides several options to achieve this onboarding. The first option is automated onboarding in Amazon SageMaker lakehouse, in which you ingest the Amazon SageMaker lakehouse metadata of datasets into Amazon SageMaker Catalog. With this option, you can onboard your Amazon SageMaker lakehouse data as part of creating a new Amazon SageMaker Unified Studio domain or for an existing domain.

For more information about automated onboarding of Amazon SageMaker lakehouse data, refer to Onboarding data in Amazon SageMaker Unified Studio. As other options, you can bring in existing resources to your Amazon SageMaker Unified Studio project by using the Data and Compute pages in your project, or by using scripts provided in GitHub. For more information about using the Data and Compute pages or about using scripts, refer to Bringing existing resources into Amazon SageMaker Unified Studio. In this post, you will use Amazon SageMaker lakehouse capabilities to import your trees AWS Glue table into the Producer project.

Register the Amazon S3 location for the table

To use Lake Formation permissions for fine-grained access control to the trees table, you need to register in Lake Formation the Amazon S3 location of the trees table. To do that, complete the following actions:

Access the Lake Formation console in the producer account.
In the navigation pane under Administration, choose Data lake locations.
Choose Register location and enter the following information:
1. For S3 URI, enter s3://<bucket-name>/<prefix>/ where <bucket-name> is the name of the S3 bucket you created in the prerequisites and <prefix> is the optional prefix for the trees.csv file you uploaded as part of the prerequisite.
2. For IAM role, select AWSServiceRoleForLakeFormationDataAccess.
3. For Permission mode, select Lake Formation.
Choose Register location.

Grant `Producer` project role permissions on the database

Grant database access to the IAM role that is associated with your Producer project. This role is called the project role, and it was created in IAM upon project creation.

To access the AWS Glue Data Catalog collections database from the Producer project in the Amazon SageMaker Unified Studio, complete the following actions:

Access the Lake Formation console in the producer account.
In the navigation pane under Data Catalog, choose Databases.
Choose the collections database.
From the Actions menu, choose Grant and enter the following information:
1. For IAM users and roles, select your Producer project’s role name. This is the string starting with datazone_usr_role_ that is part of the Producer project role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
2. For Database permissions, select Describe.
Choose Grant.

Grant `Producer` project role permissions on the table

Grant trees table access to the IAM role that is associated with your Producer project. To grant these permissions use the following instructions:

Access the Lake Formation console in the producer account.
In the navigation pane under Data Catalog, choose Tables and MVs.
Select the trees table.
From the Actions menu, choose Grant and enter the following information:
1. For IAM users and roles, select your Producer project’s role. This is the string starting with datazone_usr_role_ that is part of the Producerproject role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
2. For Table permissions, select Select and Describe.
3. For Grantable permissions, select Select and Describe.
Choose Grant.

Revoke any existing permissions of `IAMAllowedPrincipals`

You must revoke the IAMAllowedPrincipals group permissions on both the database and table to enforce Lake Formation permission for access. For more information, refer to Revoking permission using the Lake Formation console.

Access the Lake Formation console in the producer account.
In the navigation pane under Permission, choose Data permissions.
Select the entries where Principal is set to IAMAllowedPrincipals and Resource is set to collections or trees as in the following image:

Data permissions table: 2 of 5 IAMAllowedPrincipals entries selected. All permissions granted for collections DB & trees table

Choose Revoke.
Enter revoke.
Choose Revoke again.

Verify that data is available in the `Producer` project

Verify that your collections database and trees table are accessible in the Producer project:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project drop-down menu and choose the Producer project.
In the navigation pane under Overview, choose Data.
Choose Lakehouse.
Choose AwsDataCatalog.
Choose collections.
Choose tables.
Choose the three-dot action menu next to your trees table and choose Preview data, as shown in the following image.
You’ll find data from the trees table as shown in the following image.

Create Amazon SageMaker Catalog asset

Even if it’s accessible in the project, to work with the trees table in Amazon SageMaker Catalog, you need to register the data source and create an Amazon SageMaker Catalog asset:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose the Producer project.
On the project page, under Project catalog in the navigation pane, choose Data sources.
Choose Create Data Source and make the following selections:
1. For Name, enter collections.
2. For Data source type, select AWS Glue (Lakehouse).
3. For Database name, select collections.
4. Choose Next.
5. Choose Next.
6. Choose Next.
7. Choose Create.
After the data source is created, you will be in the collections data source page, choose Run. This will import metadata and create the Amazon SageMaker Catalog asset.
In the collections data source, on the Data source runs tab, you’ll find your run marked as Completed and the trees asset Successfully created, as shown in the following image:

Publish the data asset in the Amazon SageMaker Catalog

Publishing a data asset manually is a one-time operation that you need to perform to allow others to access the data asset through the catalog:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose the Producer project.
On the project page under Project catalog, choose Assets.
Select your trees data asset that is available on the Inventory tab. The following image is shown for reference.
(Optional) If automated metadata generation is enabled when the data source is created, metadata for assets (such as the asset business name) is available to review and accept or reject. You can either choose Accept All or Reject All in the Automated Metadata Generation banner.
Choose Publish Asset. The following image is shown for reference.
Choose Publish Asset.

Subscribe to the data asset in the Amazon SageMaker Catalog

To consume data assets in the Consumer project, subscribe to the data asset by creating a subscription request:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose Consumer project.
On the Discover menu, choose Catalog.
Enter trees in the search box and then select the data asset returned from the search. If in step 7 “Publish the data asset in the Amazon SageMaker Catalog” you chose Accept All in the Automated Metadata Generation banner, your data asset will have a different business name generated by the automated metadata recommendations feature. The data asset technical name is trees. For reference, refer to the following image.
Choose Subscribe.
For Comment, enter a justification such as This data asset is needed for model training purposes.
Choose Subscribe again.

By default, asset subscription requests require manual approval by a data owner. However, if the requester in the Consumer project is also a member of the Producer project, the subscription request is automatically approved. For information about approving subscription requests, refer to Approve or reject a subscription request in Amazon SageMaker Unified Studio.

Configure your Lambda IAM role to access the subscribed data access

To enable your Lambda function access to the subscribed data asset, you need to allow the Lambda function to assume the Consumer project role. To do this, edit the Consumer project’s IAM role trust relationship:

Navigate to the IAM console in the consumer account.
In the navigation pane under Access management, choose Roles.
Select the Consumer project’s IAM role. This is the string starting with datazone_usr_role_ that is part of the Consumer project role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
Under the Trust relationships tab, choose Edit trust policy.
For backup reasons, make a copy of the existing trust policy in a text file.
In the Edit trust policy window, add the following statement to the existing trust policy without removing or overwriting other existing statements in the trust policy. Be sure to replace the placeholder <account_id> with your consumer AWS account ID.
```
{
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::<account_id>:role/smus_consumer_lambda"
    },
    "Action": [
        "sts:AssumeRole"
    ]
}	
```
Choose Update policy.

Test the Lambda function’s access to the subscribed data asset

Before you can test your Lambda function, you need to replace placeholders in the function code and in the IAM policy. There are three placeholders to be replaced: <role_arn>, <database_name> and <workgroup_id>. For <role_arn>, you already have the actual value, which is the Consumer project’s role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”. The next sections provide instructions to retrieve values for the other placeholders.

Retrieve the AWS Glue Data Catalog database name

You need to find the name of the AWS Glue Data Catalog database that was created along with the Consumer project. You will then use this value to replace the <database_name> placeholder in the consumer_function Lambda function code. To retrieve the AWS Glue Data Catalog database name, follow these instructions:

Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose Consumer project.
On the project page, under Overview, choose Data.
Choose Lakehouse.
Choose AwsDataCatalog.
Copy the name of the database. It should be an alphanumerical string starting with glue_db, as in the following image:

Copy the Workgroup ARN and save to a text file. The Athena workgroup ID is the string that follows arn:aws:athena:<region>:<account_ID>:workgroup/ in the Workgroup ARN.

Replace placeholder in the `smus_consumer_athena_execution` IAM policy

To replace the <workgroup_id> placeholder in the smus_consumer_athena_execution IAM policy, use the following procedure:

Access the IAM console in the consumer account.
In the navigation pane, choose Policies.
In the search field enter smus_consumer_athena_execution.
Select the smus_consumer_athena_execution policy.
Choose Edit.
Replace <workgroup_id> with the value you noted earlier.
Choose Next.
Choose Save changes.

Replace placeholders in the Lambda function code and test it

In this section, you will replace the <role_arn>, <database_name> and <workgroup_id> placeholders in the consumer_function Lambda function code, and then you can test the function ability to access data of the trees table.

Access the Lambda console in the consumer account.
In the navigation pane, choose Functions.
Select consumer_function.
Under the Code tab, replace <role_arn>, <database_name> and <workgroup_id> placeholders with the respective values you noted earlier.
Choose Deploy.
Under the Test tab, for Event name, enter mytest.
Choose Test.
Choose Details in the green banner titled Executing function that appears after the execution is completed.
The execution log reports the trees table content, as shown in the following image:

If your Lambda function execution fails due to timeout, change the function timeout setting as follows:

Access the Lambda console in the consumer account.
In the navigation pane, choose Functions.
Select consumer_function.
Under the Configuration tab, choose Edit.
For Timeout, enter 15 sec or a greater value.
Choose Save.

After increasing the timeout, test the function again.

Clean up

If you no longer need the resources you created as you followed this post, delete them to prevent incurring additional charges. Start by deleting your Amazon SageMaker Unified Studio domain in the governance account. For more information, refer to Delete domains.

To remove the AWS Glue collections database from the producer account, follow these steps:

Access the Glue console in the producer account.
In the navigation pane under Data Catalog, choose Databases.
Select the collections database.
Choose Delete.
Choose Delete.

To remove the S3 bucket from the producer account, empty the bucket and then you can delete the bucket. For information about emptying the bucket, refer to Emptying a general purpose bucket. For information about deleting the bucket, refer to Deleting a general purpose bucket.

To remove the Lambda function from the consumer account, follow these steps:

Access the Lambda console in the consumer account.
In the navigation pane, choose Functions.
Select the consumer_function Lambda function.
Choose the Actions menu and then choose Delete function.
Enter confirm.
Choose Delete.

To complete the cleanup, delete the IAM role named smus_consumer_lambda, then delete the IAM policy named smus_consumer_athena_execution in the consumer account. For information about removing a IAM role, refer to Delete roles or instance profiles. For information about removing an IAM policy, refer to Delete IAM policies.

Conclusion

In this post, we covered adopting Amazon SageMaker Catalog for data governance without rearchitecting your existing applications and data repositories. We walked through how to onboard existing data in Amazon SageMaker Unified Studio, then publish it in a catalog, and then subscribe and consume the data from resources deployed outside the context of an Amazon SageMaker Unified Studio project. This solution can help you accelerate your implementation of a data mesh pattern with Amazon SageMaker Catalog to publish, find, and access data securely in your organization.

For more information, refer to What is Amazon SageMaker? and work through the Amazon SageMaker Workshop to try the unified experience for data, analytics, and AI.

About the authors

Using Amazon SageMaker Unified Studio Identity center (IDC) and IAM-based domains together

2026-02-10 Praveen Kumar

Post Syndicated from Praveen Kumar original https://aws.amazon.com/blogs/big-data/using-amazon-sagemaker-unified-studio-identity-center-idc-and-iam-based-domains-together/

Amazon SageMaker Unified Studio now offers two domain configurations: Amazon SageMaker Unified Studio Identity Center(IDC)-based domains with comprehensive governance features, and Amazon SageMaker Unified Studio IAM-based domains with enhanced developer productivity tools.

In this post, we demonstrate how you can use both of these domain configurations of Amazon SageMaker Unified Studio using AWS Identity and Access Management (IAM) role reuse and attribute-based access control.

How authentication works in each configuration

Amazon SageMaker Unified Studio IDC-based domains authenticate users through AWS Identity and Access Management (IAM) Identity Center with Single Sign-On, preserving individual user identities throughout their sessions. These domains excel in governance with identity-based authorization, fine-grained access controls between users, and comprehensive catalog management featuring formal Publisher/Subscriber (Pub/Sub) data sharing workflows with approval processes—ideal for enterprise environments requiring strong identity management, compliance tracking, and identity-based audit trails.

Amazon SageMaker Unified Studio IAM-based domains authenticate through federated AWS Identity and Access Management (IAM) roles where all users accessing a project share the same role permissions. These domains prioritize developer productivity with modern tools including new serverless Notebooks, Athena Spark integration, the improved interface with vertical navigation, and built-in AI assistance, designed for development teams that need streamlined access and advanced analytics capabilities.

This solution facilitates organizations that are already using IDC-based domains to preserve their existing governance frameworks established in IDC-based domains while unlocking modern development capabilities for their teams through IAM-based domains. If you prefer to use the newly launched IAM-based domains, you can continue to do as well. The choice depends on your company’s needs.

Please note that at the time of writing this blog, IAM-based domains do not support Trusted identity propagation. This solution uses the project execution role to configure data access.

The challenge

Imagine a data steward (Sam) uses the IDC-based domain to define data access policies, manage the data catalog, and approve subscription requests to verify compliance and proper data governance.

On the other hand, a data engineer (Sarah), wants to use IDC-based domain for governance features such as SageMaker catalog and IAM-based domain for the new serverless Notebook to build data pipelines, perform advanced analytics, and accelerate development cycles. Sarah will request access to the data through IDC-based domain, and once access is approved by Sam, Sarah can access this data in serverless notebook available in IAM-based domain.

Solution overview

The integration leverages IAM role reuse, AWS Lake Formation Attribute-Based Access Control (ABAC) and Amazon SageMaker Catalog pub-sub model to automatically carry permissions from the IDC-based domain to the new IAM-based domain. When properly configured, data subscriptions managed through the IDC-based domain’s Pub/Sub model become immediately accessible in IAM-based domain projects, providing a unified data access experience.

The solution we will implement in the post involves creating an IAM-based domain project that is similar to your IDC consumer project (eg same team members, use case) , configuring execution roles, and enabling role reuse. This approach maintains the familiar subscription workflow while extending benefits to the IAM-based domain.The following diagram shows the high-level architecture of how this approach works.

AWS SageMaker data governance workflow diagram showing data engineer Sarah performing data discovery and exploration through SageMaker IDC and IAM domains, with data steward and owner Sam managing approvals via Business Data Catalog, connecting to Polyglot AI Notebook and SQL tools.

The solution architecture consists of:

Existing IDC-based domain: Contains producer and consumer projects with established data sharing via Pub/Sub model
IAM-based domain: New projects with federated and execution roles configured for modern development tools
IAM Identity Center: Manages federated access and permission sets
Attribute-Based Access Control: Tags on execution roles enable automatic permission inheritance

The solution provides 2 options: Option 1: IDC-Based Domain project role reuse provides the simplest integration path by directly reusing the existing consumer project IAM role from your IDC-based domain as the execution role in the IAM-based domain. The primary benefits include simplified setup requiring only policy changes (covered later in the blog), reduced administrative overhead with one less role to manage and lower risk of misconfiguration since you’re leveraging proven, existing roles. Choose Option 1 when you want the fastest implementation path, your organization prefers minimal role proliferation, you have well-established IDC-based domain roles that already have data access permissions, or your team has limited IAM expertise and wants to avoid complex tagging configurations.

Option 2: Creating a new execution role for the IAM-based domain project and use attribute-based access control (ABAC) through tagging with the IDC-based domain project ID. The key benefits include enhanced auditability with two distinct roles (one for IDC-based domain, one for IAM-based domain), clear separation showing which domain generated each request in CloudTrail logs, greater flexibility to customize permissions specific to IAM-based domain needs without affecting IDC-based domain operations, and better security isolation between the two domain types. The `AmazonDatazoneProject` tag enables attribute based access control, while maintaining distinct role identities. Choose Option 2 when: your organization requires detailed audit trails distinguishing between domain types, compliance policies mandate separation of concerns between governance and development environments, you want to track and attribute costs separately for each domain, or you need to provide evidence showing which domain (governance vs. development) accessed specific data resources for compliance reporting.

Here is the high-level view of how the identity and domain entities map to each other for both options:

AWS IAM Identity Center integration with Amazon SageMaker diagram showing access flow from IdC Groups through Permission Sets to AWS SSO IAM Roles, connecting to SageMaker domains with two implementation options: Option 1 using identical IAM roles, or Option 2 using project-tagged execution roles

Prerequisites

To follow along with this post, you should have:

An AWS account with Amazon SageMaker Unified Studio IDC-based domain already configured
Existing producer and consumer projects in the Amazon SageMaker Unified Studio IDC-based domain
Data assets published and subscribed through the Pub/Sub model
Administrative access to Amazon IAM Identity Center
Familiarity with Amazon SageMaker Unified Studio concepts including projects, environments, and IAM roles
Amazon SageMaker Unified Studio IAM-based domain created as an administrator

For this demonstration, we use a simplified setup with a sales producer project and a marketing consumer project that subscribes to these tables.

Understanding the current IDC-based domain setup

Our starting point includes a well-established Amazon SageMaker Unified Studio IDC-based domain structure:

Sales Producer Project

Contains a database with pipeline and sales tables
Managed by Sam, the data steward who creates and publishes data assets
Has its own project IAM role

Marketing Consumer Project

Managed by Sarah, the data engineer who subscribes to published data via IDC domain project
Has its own project IAM role
Successfully queries subscribed data through the IDC-based domain interface

Each project has an associated IAM role that governs access to data assets, and the Pub/Sub model manages subscription workflows and permissions.

Setting up federated role through permission sets

Federated roles through permission sets are used to authenticate and provide users with console access to IAM-based domains through AWS IAM Identity Center, where all users within a project share the same role permissions. When you assign a permission set, IAM Identity Center creates corresponding IAM Identity Center-controlled IAM role in AWS account, and attaches the policies specified in the permission set to that role.

IAM-based SMUS domains enable streamlined access to modern development tools (serverless Notebooks, Athena Spark, AI assistance) while maintaining governance, automatically propagating permissions across domains without requiring duplicate access approvals, and simplifying team member onboarding.You can use any IAM role to access IAM-based domain. For this post, we will use federated role option using AWS IAM Identity Center (IDC).

Grant access to Data engineer group for IAM-based domains in Identity Center

1) Set up federated role in AWS IAM Identity Center

Navigate to IAM Identity Center (IDC) in the AWS Management Console, then complete the following steps:

Go to permission set section in IDC. Create a new permission set called Marketing-federated-role and select Attach Policy.

AWS IAM Identity Center console screenshot displaying the marketing-federated-role permission set configuration page with provisioned status, 1-hour session duration, and empty AWS managed and customer managed policy sections with attach policy options.

Search for SageMakerStudioUserIAMConsolePolicy in the existing policy name from list and select SageMakerStudioUserIAMConsolePolicy from the list. Note that the managed policy SageMakerStudioUserIAMConsolePolicy must be attached or have the same permissions added via another policy to be able to access projects in a SageMaker IAM domain.

AWS IAM Identity Center console screenshot showing AWS managed policies section with one attached SageMakerStudioUserIAMConsolePolicy and empty customer managed policies section with detach and attach policy options available.

Go to the AWS account section of IDC.
Assign the created permission set to your AWS account.

AWS IAM Identity Center console screenshot showing AWS accounts page in hierarchy view with organization o-9svtz1aavh, displaying Root organizational unit containing AWS account n.com with marketing-federated-role permission set assigned and assign users or groups option.

For this post we assigned the permission set to marketing group, As a best practice, you should setup and grant access to groups rather than individual users.

AWS IAM Identity Center console screenshot showing marketing group details page with AWS accounts tab selected, displaying one AWS account access (management account amazon.com) with marketing-federated-role permission set applied.

Add Sarah to marketing group.

AWS IAM Identity Center console screenshot showing marketing group's Users tab with one enabled member (user sarah, Display name: Sarah M) who inherits permissions to AWS accounts and Identity Center enabled applications.

This creates a federated role that Sarah can use to access the IAM-based domain. The federated role appears as an IAM role within your account and serves as the entry point for console access.

Setting up IAM-based domain execution role

There are 2 options to setup execution role for IAM-based domain project. The execution role has a one-to-one mapping with the federated role.

Option 1 – IDC-based domain Project Role reuse

Instead of creating a new execution role and tagging it, you can configure the IAM-based domain project to directly reuse the consumer project IAM role from the IDC-based domain as the execution role. This option only needs policy changes to the consumer project IAM role. To find the IDC-based domain consumer project IAM role:

Navigate to the Amazon SageMaker Unified Studio IDC-based domain portal.
Open the Marketing Consumer Project.
Copy the project role ARN from the project overview page.

Amazon DataZone project overview page displaying marketing-project details with active status, project ID 4tcycvm4c684rt, domain ID dzd-47supbt0i3jysp, All capabilities profile, Corp domain unit, Amazon S3 location in us-east-2, and project role ARN with up-to-date status.

You will need to modify this execution role’s policy with detailed instructions provided later in the blog.

Setting up IAM-based domain project for option 1

To create an IAM-based domain project that will integrate with your existing IDC-based domain permissions, complete the following steps:

Log in to the AWS Console using IAM-based domain administrator.
Navigate to Amazon SageMaker page within console.
Choose Open.

Amazon SageMaker landing page displaying "The center for data, analytics, and AI" with tagline about next-generation integrated analytics experience, serverless notebooks with built-in AI Agent, Amazon DataZone integration note, and call-to-action panel featuring "Get started with Amazon SageMaker Unified Studio" with Open button and View existing domains

Once logged in to IAM-based domain as admin, choose Manage projects.

Amazon SageMaker admin-project dashboard displaying left navigation menu with data analytics and AI/ML sections, quick-start cards for exploring data, building in notebooks, and discovering ML models, plus four sample data project templates: Customer usage analysis (3 mins), Customer segmentation (8 mins), Customer churn prediction (5 mins), and Retail sales forecasting (20 mins).

Next, click on Create Project.

Amazon DataZone Domain Administration Projects page showing "Projects (3)" with description about enabling IAM role-based access to AWS Analytics and AI/ML tools, search functionality to find projects, last refreshed timestamp, and green Create project button.

Enter project name as “Marketing Consumer Project”.

Amazon DataZone Create project dialog showing Step 1 "Enter Details" with required Project name field containing "Marketing Consumer Project" (1-64 characters, a-z, A-Z, 0-9, spaces, dashes, underscores allowed) and optional Description field with 0/2048 character count, followed by Step 2 "Assign roles".

During project creation, select the following crucial roles and then choose Create Project:

Project IAM Role: The marketing federated role created in IAM Identity Center above. This is the role in the member account that has a role name with suffix AWSReservedSSO.
Project Role: – Choose project role for data engineer, copied from option 1.

Make policy changes to this project role as per the instruction on the SMUS UI page.

Option 2 – Bring your own execution role.

To create an IAM-based domain project that will integrate with your existing IDC-based domain permissions., you must tag the execution role for permission propagation. Amazon SageMaker Catalog and AWS Lake Formation use attribute-based access control, which means permissions can be inherited based on resource tags. For this option, you will need consumer project ID.To find the IDC-based domain consumer project ID:

Navigate to the Amazon SageMaker Unified Studio IDC-based domain portal.
Open the Marketing Consumer Project.
Copy the project ID from the project details.

Federated Role: The marketing federated role created in IAM Identity Center above.

Execution Role: – Choose execution role from option 2.

Make policy changes to this execution role as per the instruction.

Next, navigate to the IAM console and locate the execution role created for your IAM-based domain consumer project.
Add the following tag, this step relies on ABAC policies with projectId for subscriptions.

Key: AmazonDatazoneProject
Value: The project ID from your Amazon SageMaker Unified Studio IDC-based domain consumer project

This tag configuration results in data access grant from IDC-based domain consumer project to the IAM-based domain project execution role.

Verify data access in the IAM-based domain

After tagging the execution role, verify that permissions are set up correctly.Complete the following steps:

Use the SSO URL to log into the SSO Identity Center as Sarah.

Open the AWS console using federated role created earlier in setting federated role section.
Navigate to Amazon SageMaker.
Choose Amazon SageMaker Unified Studio IAM-based domain option (this will show up if project is already created with federated role).

In the Amazon SageMaker Unified Studio IAM-based domain project, navigate to the Data tab. If you created 2 projects with both option 1 and option 2 execution role, then 2 projects will show up and you can login to either to validate data access.

Verify that the consumer database and subscribed tables appear.

Create and use the new serverless notebooks

With permissions properly configured, you can now use IAM-based domain capabilities like serverless Notebooks. Complete the following steps:

In the Amazon SageMaker Unified Studio IAM-based domain project, select a table from the Data tab.
Choose Create notebook.
The Notebook opens with Athena SQL as the default cell type.
Write and run queries against your subscribed data.

The notebook runs with the execution role’s permissions, which now include access to all data subscribed through the IDC-based domain.

Key benefits of this integration

This integration approach delivers several important advantages:

Preserve existing investments

Continue using IDC-based domain governance and catalogs.
Maintain established Pub/Sub workflows.
No migration required for existing data assets.

Get modern capabilities

Provide developers with the new serverless Notebooks.
Access Athena Spark for advanced analytics.
Provides improved user experience and navigation.

Simplified permission management

Single subscription workflow manages access across both domains.
Consistent data access via role reuse and attribute-based access control.
No duplicate access requests or approvals needed.

Unified data experience

Developers access all subscribed data from one interface.
Consistent data catalog across domains.
Simplified onboarding for new team members.

Cleanup

Complete the following steps to delete the resources you created:

Delete the serverless Notebooks created in the IAM-based domain projects.
Delete the IAM-based domain projects (Marketing Consumer Project and Marketing Consumer Project 2).
Remove the permission set assignment from marketing group in IAM Identity Center.
Delete the Marketing-federated-role permission set in IAM Identity Center.
Remove the tags (AmazonDatazoneProject) from the execution role (if using Option 2).
Delete the execution role created for the IAM-based domain (if using Option 2 and not reusing the IDC-based domain project role).
Revert any policy changes made to the IDC-based domain consumer project IAM role (if using Option 1).
If you do not need the IAM-based domain anymore, delete it.
If you created any test data subscriptions in the IDC-based domain, remove them.

Conclusion

In this post, we demonstrated how to access Amazon SageMaker Unified Studio IDC-based domain with the new IAM-based domain using role reuse and attribute-based access control. This setup offers data engineers the best of both worlds: access to specialized modern development tools—including the new serverless Notebooks, Athena Spark integration, and built-in AI assistance , while maintaining proper governance that includes comprehensive catalog management and robust security controls established in the IDC-based domain.You can now confidently adopt Amazon SageMaker Unified Studio IAM-based domain capabilities knowing their established data governance, subscription workflows, and access controls remain intact and continue to function as expected.

Ready to get started with Amazon SageMaker Unified Studio and unlock the power of integrated governance and modern development tools for your organization? Visit the Amazon SageMaker Unified Studio documentation to learn more and begin your implementation today.

About the authors

Orchestrate end-to-end scalable ETL pipeline with Amazon SageMaker workflows

2026-02-10 Shubham Kumar

Post Syndicated from Shubham Kumar original https://aws.amazon.com/blogs/big-data/orchestrate-end-to-end-scalable-etl-pipeline-with-amazon-sagemaker-workflows/

Amazon SageMaker Unified Studio serves as a collaborative workspace where data engineers and scientists can work together on end-to-end data and machine learning (ML) workflows. SageMaker Unified Studio specializes in orchestrating complex data workflows across multiple AWS services through its integration with Amazon Managed Workflows for Apache Airflow (Amazon MWAA). Project owners can create shared environments where team members jointly develop and deploy workflows, while maintaining oversight of pipeline execution. This unified approach makes sure data pipelines run consistently and efficiently, with clear visibility into the entire process, making it seamless for teams to collaborate on sophisticated data and ML projects.

This post explores how to build and manage a comprehensive extract, transform, and load (ETL) pipeline using SageMaker Unified Studio workflows through a code-based approach. We demonstrate how to use a single, integrated interface to handle all aspects of data processing, from preparation to orchestration, by using AWS services including Amazon EMR, AWS Glue, Amazon Redshift, and Amazon MWAA. This solution streamlines the data pipeline through a single UI.

Example use case: Customer behavior analysis for an ecommerce platform

Let’s consider a real-world scenario: An e-commerce company wants to analyze customer transactions data to create a customer summary report. They have data coming from multiple sources:

Customer profile data stored in CSV files
Transaction history in JSON format
Website clickstream data in semi-structured log files

The company wants to do the following:

Extract data from these sources
Clean and transform the data
Perform quality checks
Load the processed data into a data warehouse
Schedule this pipeline to run daily

Solution overview

The following diagram illustrates the architecture that you implement in this post.

This architecture diagram illustrates a comprehensive, end-to-end data processing pipeline built on AWS services, orchestrated through Amazon SageMaker Unified Studio. The pipeline demonstrates best practices for data ingestion, transformation, quality validation, advanced processing, and analytics.

The workflow consists of the following steps:

Establish a data repository by creating an Amazon Simple Storage Service (Amazon S3) bucket with an organized folder structure for customer data, transaction history, and clickstream logs, and configure access policies for seamless integration with SageMaker Unified Studio.
Extract data from the S3 bucket using AWS Glue jobs.
Use AWS Glue and Amazon EMR Serverless to clean and transform the data.
Implement data quality validation using AWS Glue Data Quality.
Load the processed data into Amazon Redshift Serverless.
Create and manage the workflow environment using SageMaker Unified Studio with Identity Center–based domains.

Note: Amazon SageMaker Unified Studio supports two domain configuration models: IAM Identity Center (IdC)–based domains and IAM role–based domains. While IAM-based domains enable role-driven access management and visual workflows, this post specifically focuses on Identity Center–based domains, where users authenticate via IdC and projects access data and resources using project roles and identity-based authorization.

Prerequisites

Before beginning, ensure you have the following resources:

An AWS account.
AWS IAM Identity Center enabled in your AWS account with users and groups created. For detailed instructions, see Getting started with IAM Identity Center and Access Amazon SageMaker Unified Studio.

Configure Amazon SageMaker Unified Studio domain

This solution requires SageMaker Unified Studio domain in the us-east-1 AWS Region. Although SageMaker Unified Studio is available in multiple Regions, this post uses us-east-1 for consistency. For a complete list of supported Regions, refer to Regions where Amazon SageMaker Unified Studio is supported.

Complete the following steps to configure your domain:

Sign in to the AWS Management Console, navigate to Amazon SageMaker, and open the Domains section from the left navigation pane.
On the SageMaker console, choose Create domain, then choose Quick setup.
If the message “No VPC has been specifically set up for use with Amazon SageMaker Unified Studio” appears, select Create VPC. The process redirects to an AWS CloudFormation stack. Leave all settings at their default values and select Create stack.
Under Quick setup settings, for Name, enter a domain name (for example, etl-ecommerce-blog-demo). Review the selected configurations.
Choose Continue to proceed.
On the Create IAM Identity Center user page, create an SSO user (account with IAM Identity Center) or select an existing SSO user to log in to the Amazon SageMaker Unified Studio. The SSO selected here is used as the administrator in the Amazon SageMaker Unified Studio.
Choose Create domain.

For detailed instructions, see Create a SageMaker domain and Onboarding data in Amazon SageMaker Unified Studio.

After you have created a domain, popup will appear with the message: “Your domain has been created! You can now log in to Amazon SageMaker Unified Studio”. You can close the popup for now.

Create a project

In this section, we create a project to serve as a collaborative workspace for teams to work on business use cases. Complete the following steps:

Choose Open Unified Studio and sign in with your SSO credentials using the Sign in with SSO option.
Choose Create project.
Name the project (for example, ETL-Pipeline-Demo) and create it using the All capabilities project profile.
Choose Continue.
Keep the default values for the configuration parameters and choose Continue.
Choose Create project.

Project creation might take a few minutes. After the project is created, the environment will be configured for data access and processing.

Integrate S3 bucket with SageMaker Unified Studio

To enable external data processing within SageMaker Unified Studio, configure integration with an S3 bucket. This section walks through the steps to set up the S3 bucket, configure permissions, and integrate it with the project.

Create and configure S3 bucket

Complete the following steps to create your bucket:

In a new browser tab, open the AWS Management Console and search for S3.
On the Amazon S3 console, choose Create Bucket .
Create a bucket named ecommerce-raw-layer-bucket-demo-<Account-ID>-us-east-1. For detailed instructions, see create a general-purpose Amazon S3 bucket for storage.
Create the following folder structure in the bucket. For detailed instructions, see Creating a folder:
- raw/customers/
- raw/transactions/
- raw/clickstream/
- processed/
- analytics/

Upload sample data

In this section, we upload sample ecommerce data that represents a typical business scenario where customer behavior, transaction history, and website interactions need to be analyzed together.

The raw/customers/customers.csv file contains customer profile information, including registration details. This structured data will be processed first to establish the customer dimension for our analytics.

customer_id,name,email,registration_date
1,John Doe,[email protected],2022-01-15
2,Jane Smith,[email protected],2022-02-20
3,Robert Johnson,[email protected],2022-01-30
4,Emily Brown,[email protected],2022-03-05
5,Michael Wilson,[email protected],2022-02-10

The raw/transactions/transactions.json file contains purchase transactions with nested product arrays. This semi-structured data will be flattened and joined with customer data to analyze purchasing patterns and customer lifetime value.

[
{"transaction_id": "t1001", "customer_id": 1, "amount": 125.99, "date": "2023-01-10", "items": ["product1", "product2"]},
{"transaction_id": "t1002", "customer_id": 2, "amount": 89.50, "date": "2023-01-12", "items": ["product3"]},
{"transaction_id": "t1003", "customer_id": 1, "amount": 45.25, "date": "2023-01-15", "items": ["product2"]},
{"transaction_id": "t1004", "customer_id": 3, "amount": 210.75, "date": "2023-01-18", "items": ["product1", "product4", "product5"]},
{"transaction_id": "t1005", "customer_id": 4, "amount": 55.00, "date": "2023-01-20", "items": ["product3", "product6"]}
]

The raw/clickstream/clickstream.csv file captures user website interactions and behavior patterns. This time-series data will be processed to understand customer journey and conversion funnel analytics.

timestamp,customer_id,page,action
2023-01-10T10:15:23,1,homepage,view
2023-01-10T10:16:45,1,product_page,view
2023-01-10T10:18:12,1,product_page,add_to_cart
2023-01-10T10:20:30,1,checkout,view
2023-01-10T10:22:15,1,checkout,purchase
2023-01-12T14:30:10,2,homepage,view
2023-01-12T14:32:20,2,product_page,view
2023-01-12T14:35:45,2,product_page,add_to_cart
2023-01-12T14:40:12,2,checkout,view
2023-01-12T14:42:30,2,checkout,purchase

raw

For detailed instructions on uploading files to Amazon S3, refer to the Uploading objects.

Configure CORS policy

To allow access from the SageMaker Unified Studio domain portal, update the Cross-Origin Resource Sharing (CORS) configuration of the bucket:

On the bucket’s Permissions tab, choose Edit under Cross-origin resource sharing (CORS).

Enter the following CORS policy and replace domainUrl with the SageMaker Unified Studio domain URL (for example, https://<domain-id>.sagemaker.us-east-1.on.aws ). The URL can be found at the top of the domain details page on the SageMaker Unified Studio console.

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "PUT",
            "GET",
            "POST",
            "DELETE",
            "HEAD"
        ],
        "AllowedOrigins": [
            "domainUrl"
        ],
        "ExposeHeaders": [
            "x-amz-version-id"
        ]
    }
]

For detailed information, see Adding Amazon S3 data and gain access using the project role.

Grant Amazon S3 access to SageMaker project role

To enable SageMaker Unified Studio to access the external Amazon S3 location, the corresponding AWS Identity and Access Management (IAM) project role must be updated with the required permissions. Complete the following steps:

On the IAM console, choose Roles in the navigation pane.
Search for the project role using the last segment of the project role Amazon Resource Name (ARN). This information is located on the Project overview page in SageMaker Unified Studio (for example, datazone_usr_role_1a2b3c45de6789_abcd1efghij2kl).
Choose the project role to open the role details page.
On the Permissions tab, choose Add permissions, then choose Create inline policy.
Use the JSON editor to create a policy that grants the project role access to the Amazon S3 location
In the JSON policy below, replace the placeholder values with your actual environment details:
- Replace <BUCKET_PREFIX> with the prefix of S3 bucket name (for example, ecommerce-raw-layer)
- Replace <AWS_REGION> with the AWS Region where your AWS Glue Data Quality rulesets are created (for example, us-east-1)
- Replace <AWS_ACCOUNT_ID> with your AWS account ID

Paste the updated JSON policy into the JSON editor.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ETLBucketListAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::<BUCKET_PREFIX>-*"
        },
        {
            "Sid": "ETLObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::<BUCKET_PREFIX>-*/*"
        },
        {
            "Sid": "GlueDataQualityPublish",
            "Effect": "Allow",
            "Action": [
                "glue:PublishDataQuality"
            ],
 "Resource":"arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:dataQualityRuleset/*"
        }
    ]
}

Choose Next.
Enter a name for the policy (for example, etl-rawlayer-access), then choose Create policy.
Choose Add permissions again, then choose Create inline policy.

In the JSON editor, create a second policy to manage S3 Access Grants:Replace <BUCKET_PREFIX> with the prefix of S3 bucket name (for example, ecommerce-raw-layer) and paste this JSON policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3AGLocationManagement",
            "Effect": "Allow",
            "Action": [
                "s3:CreateAccessGrantsLocation",
                "s3:DeleteAccessGrantsLocation",
                "s3:GetAccessGrantsLocation"
            ],
            "Resource": [
                "arn:aws:s3:*:*:access-grants/default/*"
            ],
            "Condition": {
                "StringLike": {
                    "s3:accessGrantsLocationScope": "s3://<BUCKET_PREFIX>-*/*"
                }
            }
        },
        {
            "Sid": "S3AGPermissionManagement",
            "Effect": "Allow",
            "Action": [
                "s3:CreateAccessGrant",
                "s3:DeleteAccessGrant"
            ],
            "Resource": [
                "arn:aws:s3:*:*:access-grants/default/location/*",
                "arn:aws:s3:*:*:access-grants/default/grant/*"
            ],
            "Condition": {
                "StringLike": {
                    "s3:accessGrantScope": "s3://<BUCKET_PREFIX>-*/*"
                }
            }
        }
    ]
}

Choose Next.
Enter a name for the policy (for example, s3-access-grants-policy), then choose Create policy.

create policy

For detailed information about S3 Access Grants, see Adding Amazon S3 data.

Add S3 bucket to project

After you add policies to the project role for access to the Amazon S3 resources, complete the following steps to integrate the S3 bucket with the SageMaker Unified Studio project:

In SageMaker Unified Studio, open the project you created under Your projects.
Choose Data in the navigation pane.
Select Add and then Add S3 location.
Configure the S3 location:
1. For Name, enter a descriptive name (for example, E-commerce_Raw_Data).
2. For S3 URI, enter your bucket URI (for example, s3://ecommerce-raw-layer-bucket-demo-<Account-ID>-us-east-1/).
3. For AWS Region, enter your Region (for this example, us-east-1).
4. Leave Access role ARN blank.
5. Click Add S3 Location
Wait for the integration to complete.
Verify the S3 location appears in your project’s data catalog (on the Project overview page, on the Data tab, locate the Buckets pane to view the buckets and folders).

add

This process connects your S3 bucket to SageMaker Unified Studio, making your data ready for analysis.

Create notebook for job scripts

Before you can create the data processing jobs, you must set up a notebook to develop the scripts that will generate and process your data. Complete the following steps:

In SageMaker Unified Studio, on the top menu, under Build, choose JupyterLab.
Choose Configure Space and choose the instance type ml.t3.xlarge. This makes sure your JupyterLab instance has at least 4 vCPUs and 4 GiB of memory.
Choose Configure and Start Space or Save and Restart to launch your environment.
Wait a few moments for the instance to be ready.
Choose File, New, and Notebook to create a new notebook.
Set Kernel as Python 3, Connection type as PySpark, and Compute as Project.spark.compatibility.
In the notebook, enter the following script to use later for your AWS Glue job. This script processes raw data from three sources in the S3 data lake, standardizes dates, and converts data types before saving the cleaned data in Parquet format for optimal storage and querying.

Replace <Bucket-Name> with the name of actual S3 bucket in script:

import sys
from awsglue.transforms import *
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
from pyspark.sql import functions as F
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
# Customers
customer_df = (
    spark.read
    .option("header", "true")
    .csv("s3://<Bucket-Name>/raw/customers/")
    .withColumn("registration_date", F.to_date("registration_date"))
    .withColumn("processed_at", F.current_timestamp())
)
customer_df.write.mode("overwrite").parquet(
    "s3://<Bucket-Name>/processed/customers/"
)
# Transactions
transaction_df = (
    spark.read
    .json("s3://<Bucket-Name>/raw/transactions/")
    .withColumn("date", F.to_date("date"))
    .withColumn("customer_id", F.col("customer_id").cast("int"))
    .withColumn("processed_at", F.current_timestamp())
)
transaction_df.write.mode("overwrite").parquet(
    "s3://<Bucket-Name>/processed/transactions/"
)
# Clickstream
clickstream_df = (
    spark.read
    .option("header", "true")
    .csv("s3://<Bucket-Name>/raw/clickstream/")
    .withColumn("customer_id", F.col("customer_id").cast("int"))
    .withColumn("timestamp", F.to_timestamp("timestamp"))
    .withColumn("processed_at", F.current_timestamp())
)
clickstream_df.write.mode("overwrite").parquet(
    "s3://<Bucket-Name>/processed/clickstream/"
)
print("Data processing completed successfully")
job.commit()

This script processes customer, transaction, and clickstream data from the raw layer in Amazon S3 and saves it as Parquet files in the processed layer.

Choose File, Save Notebook As, and save the file as shared/etl_initial_processing_job.ipynb.

Create notebook for AWS Glue Data Quality

After you create the initial data processing script, the next step is to set up a notebook to perform data quality checks using AWS Glue. These checks help validate the integrity and completeness of your data before further processing. Complete the following steps:

Choose File, New, and Notebook to create a new notebook.
Set Kernel as Python 3, Connection type as PySpark, and Compute as Project.spark.compatibility.

In this new notebook, add the data quality check script using the AWS Glue EvaluateDataQuality method. Replace <Bucket-Name> with the name of actual S3 bucket in script:

from datetime import datetime
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsgluedq.transforms import EvaluateDataQuality
from awsglue.transforms import SelectFromCollection

# ---------------- Glue setup ----------------
sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
job = Job(glueContext)
job.init("GlueDQJob", {})

# ---------------- Constants ----------------
RUN_DATE = datetime.utcnow().strftime("%Y-%m-%d")
year, month, day = RUN_DATE.split("-")
OUTPUT_PATH = "s3://<Bucket-Name>/data-quality-results"

# ---------------- Tables and Rules ----------------
tables = {
    "customers": ["s3://<Bucket-Name>/processed/customers/",
                  ["IsComplete \"customer_id\"", "IsUnique \"customer_id\"", "IsComplete \"email\""]],
    "transactions": ["s3://<Bucket-Name>/processed/transactions/",
                     ["IsComplete \"transaction_id\"", "IsUnique \"transaction_id\""]],
    "clickstream": ["s3://<Bucket-Name>/processed/clickstream/",
                    ["IsComplete \"customer_id\"", "IsComplete \"action\""]]
}

# ---------------- Process Each Table ----------------
for table, (path, rules) in tables.items():
    df = glueContext.create_dynamic_frame.from_options("s3", {"paths":[path]}, "parquet")
    results = EvaluateDataQuality().process_rows(
        frame=df,
        ruleset=f"Rules = [{', '.join(rules)}]",
        publishing_options={"dataQualityEvaluationContext": table}
    )
    rows = SelectFromCollection.apply(results, key="rowLevelOutcomes", transformation_ctx="rows").toDF()
    rows = rows.drop("DataQualityRulesPass", "DataQualityRulesFail", "DataQualityRulesSkip")

    # Write passed/failed rows
    for status, colval in [("pass","Passed"), ("fail","Failed")]:
        tmp = rows.filter(rows.DataQualityEvaluationResult.contains(colval))
        if tmp.count() > 0:
            tmp.write.mode("append").parquet(
        f"{OUTPUT_PATH}/{table}/status=dq_{status}/Year={year}/Month={month}/Date={day}"
            )
print("Data Quality checks completed and written to S3")
job.commit()

Choose File, Save Notebook As, and save the file as shared/etl_data_quality_job.ipynb.

Create and test AWS Glue jobs

Jobs in SageMaker Unified Studio enable scalable, flexible ETL pipelines using AWS Glue. This section walks through creating and testing data processing jobs for efficient and governed data transformation.

Create initial data processing job

This job performs the first processing job in the ETL pipeline, transforming raw customer, transaction, and clickstream data and writing the cleaned output to Amazon S3 in Parquet format. Complete the following steps to create the job:

In SageMaker Unified Studio, go to your project.
On the top menu, choose Build, and under Data Analysis & Integration, choose Data processing jobs.
Choose Create job from notebooks.
Under Choose project files, choose Browse files.
Locate and select etl_initial_processing_job.ipynb (the notebook saved earlier in JupyterLab), then choose Select and Next.
Configure the job settings:
1. For Name, enter a name (for example, job-1).
2. For Description, enter a description (for example, Initial ETL job for customer data processing).
3. For IAM Role, choose the project role (default).
4. For Type, choose Spark.
5. For AWS Glue version, use version 5.0.
6. For Language, choose Python.
7. For Worker type, use G.1X.
8. For Number of Instances, set to 10.
9. For Number of retries, set to 0.
10. For Job timeout, set to 480.
11. For Compute connection, choose project.spark.compatibility.
12. Under Advanced settings, turn on Continuous logging.
Leave the remaining settings as default, then choose Submit.

After the job is created, a confirmation message will appear indicating that job-1 was created successfully.

Create AWS Glue Data Quality job

This job runs data quality checks on the transformed datasets using AWS Glue Data Quality. Rulesets validate completeness and uniqueness for key fields. Complete the following steps to create the job:

In SageMaker Unified Studio, go to your project.
On the top menu, choose Build, and under Data Analysis & Integration, choose Data processing jobs.
Choose Create job, Code-based job, and Create job from files.
Under Choose project files, choose Browse files.
Locate and select etl_glue_data_quality.ipynb, then choose Select and Next.
Configure the job settings:
For Name, enter a name (for example, job-2).
For Description, enter a description (for example, Data quality checks using AWS Glue Data Quality).
For IAM Role, choose the project role.
For Type, choose Spark.
For AWS Glue version, use version 5.0.
For Language, choose Python.
For Worker type, use G.1X.
For Number of Instances, set to 10.
For Number of retries, set to 0.
For Job timeout, set to 480.
For Compute connection, choose project.spark.compatibility.
Under Advanced settings, turn on Continuous logging.
Leave the remaining settings as default, then choose Submit.

After the job is created, a confirmation message will appear indicating that job-2 was created successfully.

Test AWS Glue jobs

Test both jobs to make sure they execute successfully:

In SageMaker Unified Studio, go to your project.
On the top menu, choose Build, and under Data Analysis & Integration, choose Data processing jobs.
Select job-1 and choose Run job.
Monitor the job execution and verify it completes successfully.
Similarly, select job-2 and choose Run job.
Monitor the job execution and verify it completes successfully.

Add EMR Serverless compute

In the ETL pipeline, we use EMR Serverless to perform compute-intensive transformations and aggregations on large datasets. It automatically scales resources based on workload, offering high performance with simplified operations. By integrating EMR Serverless with SageMaker Unified Studio, you can simplify the process of running Spark jobs interactively using Jupyter notebooks in a serverless environment.

This section walks through the steps to configure EMR Serverless compute within SageMaker Studio and use it for executing distributed data processing jobs.

Configure EMR Serverless in SageMaker Unified Studio

To use EMR Serverless for processing in the project, follow these steps:

In the navigation pane on Project Overview, choose Compute.
On the Data processing tab, choose Add compute and Create new compute resources.
Select EMR Serverless and choose Next.
Configure EMR Serverless settings:
For Compute name, enter a name (for example, etl-emr-serverless).
For Description, enter a description (for example, EMR Serverless for advanced data processing).
For Release label, choose emr-7.8.0.
For Permission mode, choose Compatibility.
Choose Add Compute to complete the setup.

After it’s configured, the EMR Serverless compute will be listed with the deployment status Active.

emr serverless

Create and run notebook with EMR Serverless

After you create the EMR Serverless compute, you can run PySpark-based data transformation jobs using a Jupyter notebook to perform large-scale data transformations. This job reads cleaned customer, transaction, and clickstream datasets from Amazon S3, performs aggregations and scoring, and writes the final analytics outputs back to Amazon S3 in both Parquet and CSV formats.Complete the following steps to create a notebook for EMR Serverless processing:

On the top menu, under Build, choose JupyterLab.
Choose File, New, and Notebook.
Set Kernel as Python 3, Connection type as PySpark, and Compute as emr-s.etl-emr-serverless.

Enter the following PySpark script to run your data transformation job on EMR Serverless. Provide the name of your S3 bucket:

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("CustomerAnalytics").getOrCreate()

customers = spark.read.parquet("s3://<bucket-name>/processed/customers/")
transactions = spark.read.parquet("s3://<bucket-name>/processed/transactions/")
clickstream = spark.read.parquet("s3://<bucket-name>/processed/clickstream/")

customer_spending = transactions.groupBy("customer_id").agg(
    F.count("transaction_id").alias("total_transactions"),
    F.sum("amount").alias("total_spent"),
    F.avg("amount").alias("avg_transaction_value"),
    F.datediff(F.current_date(), F.max("date")).alias("days_since_last_purchase")
)

customer_engagement = clickstream.groupBy("customer_id").agg(
    F.count("*").alias("total_clicks"),
    F.countDistinct("page").alias("unique_pages_visited"),
    F.count(F.when(F.col("action") == "purchase", 1)).alias("purchase_actions"),
    F.count(F.when(F.col("action") == "add_to_cart", 1)).alias("add_to_cart_actions")
)

customer_analytics = customers.join(customer_spending, on="customer_id", how="left").join(
    customer_engagement, on="customer_id", how="left")

customer_analytics = customer_analytics.na.fill(0, [
    "total_transactions", "total_spent", "total_clicks", 
    "unique_pages_visited", "purchase_actions", "add_to_cart_actions"
])

customer_analytics = customer_analytics.withColumn(
    "customer_value_score",
    (F.col("total_spent") * 0.5) + (F.col("total_transactions") * 0.3) + (F.col("purchase_actions") * 0.2)
)

customer_analytics.write.mode("overwrite").parquet("s3://<bucket-name>/analytics/customer_analytics/")

customer_summary = customer_analytics.select(
    "customer_id", "name", "email", "registration_date", 
    "total_transactions", "total_spent", "avg_transaction_value",
    "days_since_last_purchase", "total_clicks", "purchase_actions",
    "customer_value_score"
)

customer_summary.write.mode("overwrite").option("header", "true").csv("s3://<bucket-name>/analytics/customer_summary/")

print("EMR processing completed successfully")

Choose File, Save Notebook As, and save the file as shared/emr_data_transformation_job.ipynb.
Choose Run Cell to run the script.
Monitor the Script execution and verify it completes successfully.
Monitor the Spark job execution and ensure it completes without errors.

emr run

Add Redshift Serverless compute

With Redshift Serverless, users can run and scale data warehouse workloads without managing infrastructure. It is ideal for analytics use cases where data needs to be queried from Amazon S3 or integrated into a centralized warehouse. In this step, you add Redshift Serverless to the project for loading and querying processed customer analytics data generated in earlier stages of the pipeline. For more information about Redshift Serverless, see Amazon Redshift Serverless.

Set up Redshift Serverless compute in SageMaker Unified Studio

Complete the following steps to set up Redshift Serverless compute:

In SageMaker Unified Studio, choose the Compute tab within your project workspace (ETL-Pipeline-Demo).
On the SQL analytics tab, choose Add compute, then choose Create new compute resources to begin configuring your compute environment.
Select Amazon Redshift Serverless.
Configure the following:
1. For Compute name, enter a name (for example, ecommerce_data_warehouse).
2. For Description, enter a description (for example, Redshift Serverless for data warehouse).
3. For Workgroup name, enter a name (for example, redshift-serverless-workgroup).
4. For Maximum capacity, set to 512 RPUs.
5. For Database name, enter dev.
Choose Add Compute to create the Redshift Serverless resource.

After the compute is created, you can test the Amazon Redshift connection.

On the Data warehouse tab, confirm that redshift.ecommerce_data_warehouse is listed.
Choose the compute: redshift.ecommerce_data_warehouse.
On the Permissions tab, copy the IAM role ARN. You use this for the Redshift COPY command in the next step.

Create and execute querybook to load data into Amazon Redshift

In this step, you create a SQL script to load the processed customer summary data from Amazon S3 into a Redshift table. This enables centralized analytics for customer segmentation, lifetime value calculations, and marketing campaigns. Complete the following steps:

On the Build menu, under Data Analysis & Integration, choose Query editor.

Enter the following SQL into the querybook to create the customer_summary table in the public schema:

-- Create customer_summary table in public schema
CREATE TABLE IF NOT EXISTS public.customer_summary (
    customer_id INT PRIMARY KEY,
    name VARCHAR(100),
    email VARCHAR(100),
    registration_date DATE,
    total_transactions INT,
    total_spent DECIMAL(10, 2),
    avg_transaction_value DECIMAL(10, 2),
    days_since_last_purchase INT,
    total_clicks INT,
    purchase_actions INT,
    customer_value_score DECIMAL(10, 2)
);

Choose Add SQL to add a new SQL script.
Enter the following SQL into the querybook
```
TRUNCATE TABLE customer_summary;
```
Note: We truncate the customer_summary table to remove existing records and ensure a clean, duplicate-free reload of the latest aggregated data from S3 before running the COPY command.
Choose Add SQL to add a new SQL script.

Enter the following SQL to load the data into Redshift Serverless from your S3 bucket. Provide the name of your S3 bucket and IAM role ARN for Amazon Redshift:

-- Load data from S3 (replace with your bucket name and IAM role)
COPY public.customer_summary FROM 's3://<bucket-name>/analytics/customer_summary/'
IAM_ROLE 'arn:aws:iam::<Account-ID>:role/<your-redshift-role>'
FORMAT AS CSV
IGNOREHEADER 1
REGION 'us-east-1';

In the Query Editor, configure the following:
1. Connection: redshift.ecommerce_data_warehouse
2. Database: dev
3. Schema: public
Choose Choose to apply the connection settings.
Choose Run Cell for each cell to create the customer_summary table in the public schema and then load data from Amazon S3.
Choose Actions, Save, name the querybook final_data_product, and choose Save changes.

This completes the creation and execution of the Redshift data product using the querybook.

Create and manage the workflow environment

This section describes how to create a shared workflow environment and define a code-based workflow that automates a customer data pipeline using Apache Airflow within SageMaker Unified Studio. Shared environments facilitate collaboration among project members and centralized workflow management.

Create the workflow environment

Workflow environments must be created by project owners. After they’re created, members of the project can sync and use the workflows. Only project owners can update or delete workflow environments. Complete the following steps to create the workflow environment:

Choose Compute for your project.
On the Workflow environments tab, choose Create.
Review the configuration parameters and choose Create workflow environment.
Wait for the environment to be fully provisioned before proceeding It will take around 20 minutes to provision.

workflow

Create the code-based workflow

When the workflow environment is ready, define a code-based ETL pipeline using Airflow. This pipeline automates daily processing tasks across services like AWS Glue, EMR Serverless, and Redshift Serverless.

On the Build menu, under Orchestration, choose Workflows.
Choose Create new workflow, then choose Create workflow in code editor.
Configure Space and choose the instance type ml.t3.xlarge. This ensures your JupyterLab instance has at least 4 vCPUs and 4 GiB of memory.
Choose Configure and Restart Space to launch your environment.

sample_dag

The following script defines a daily scheduled ETL workflow that automates several actions:

Initial data transformation using AWS Glue
Data quality validation using AWS Glue (EvaluateDataQuality)
Advanced data processing with EMR Serverless using a Jupyter notebook
Loading transformed results into Redshift Serverless from a querybook

Replace the default DAG template with the following definition, ensuring that job names and input paths match the actual names used in your project:

from datetime import datetime
from airflow import DAG
from airflow.decorators import dag
from airflow.utils.dates import days_ago
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
from workflows.airflow.providers.amazon.aws.operators.sagemaker_workflows import NotebookOperator
from sagemaker_studio import Project
# Get SageMaker Studio project IAM role
project = Project()
default_args = {
    'owner': 'data_engineer',
    'depends_on_past': False,
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 1
}
@dag(
    dag_id='customer_etl_pipeline',
    default_args=default_args,
    schedule_interval='@daily',
    start_date=days_ago(1),
    is_paused_upon_creation=False,
    tags=['etl', 'customer-analytics'],
    catchup=False
)
def customer_etl_pipeline():
    # Step 1: Initial data transformation using Glue
    initial_transformation = GlueJobOperator(
        task_id='initial_transformation',
        job_name='job-1',
        iam_role_arn=project.iam_role,
    )
    # Step 2: Data quality checks using Glue DQ
    data_quality_check = GlueJobOperator(
        task_id='data_quality_check',
        job_name='job-6',
        iam_role_arn=project.iam_role,
    )
    # Step 3: EMR Serverless notebook processing
    emr_processing = NotebookOperator(
        task_id='emr_processing',
        input_config={
            "input_path": "emr_data_transformation_job.ipynb",
            "input_params": {}
        },
        output_config={"output_formats": ['NOTEBOOK']},
        poll_interval=10,
    )
    # Step 4: Load to Redshift notebook
    redshift_load = NotebookOperator(
        task_id='redshift_load',
        input_config={
            "input_path": "final_data_product.sqlnb",
            "input_params": {}
        },
        output_config={"output_formats": ['NOTEBOOK']},
        poll_interval=10,
    )
    # Task dependencies
    initial_transformation >> data_quality_check >> emr_processing >> redshift_load
# Instantiate DAG
customer_etl_dag = customer_etl_pipeline()

Choose File, Save python file, name the file shared/workflows/dags/customer_etl_pipeline.py, and choose Save.

Deploy and run the workflow

Complete the following steps to run the workflow:

On the Build menu, choose Workflows.
Choose the workflow customer_etl_pipeline and choose Run.

scheduled

Running a workflow puts tasks together to orchestrate Amazon SageMaker Unified Studio artifacts. You can view multiple runs for a workflow by navigating to the Workflows page and choosing the name of a workflow from the workflows list table.

To share your workflows with other project members in a workflow environment, refer to Share a code workflow with other project members in an Amazon SageMaker Unified Studio workflow environment.

Monitor and troubleshoot the workflow

After your Airflow workflows are deployed in SageMaker Unified Studio, monitoring becomes essential for maintaining reliable ETL operations. The integrated Amazon MWAA environment provides comprehensive observability into your data pipelines through the familiar Airflow web interface, enhanced with AWS monitoring capabilities. The Amazon MWAA integration with SageMaker Unified Studio offers real-time DAG execution tracking, detailed task logs, and performance metrics to help you quickly identify and resolve pipeline issues. Complete the following steps to monitor the workflow:

On the Build menu, choose Workflows.
Choose the workflow customer_etl_pipeline.
Choose View runs to see all executions.
Choose a specific run to view detailed task status.

workflows-run

For each task, you can view the status (Succeeded, Failed, Running), start and end times, duration, and logs and outputs. The workflow is also visible in the Airflow UI, accessible through the workflow environment, where you can view the DAG graph, monitor task execution in real time, access detailed logs, and view the status.

Go to Workflows and select the workflow named customer_etl_pipeline.
From the Actions menu, choose Open in Airflow UI.

airflow-ui-smus

After the workflow completes successfully, you can query the data product in the query editor.

On the Build menu, under Data Analysis & Integration, choose Query editor.
Run select * from "dev"."public"."customer_summary"

query-editor

Observe the contents of the customer_summary table, including aggregated customer metrics such as total transactions, total spent, average transaction value, clicks, and customer value scores. This allows verification that the ETL and data quality pipelines loaded and transformed the data correctly.

Clean up

To avoid unnecessary charges, complete the following steps:

Delete a workflow environment.
If you no longer need it, delete the project.
After you delete the project, delete the domain.

Conclusion

This post demonstrated how to build an end-to-end ETL pipeline using SageMaker Unified Studio workflows. We explored the complete development lifecycle, from setting up fundamental AWS infrastructure—including Amazon S3 CORS configuration and IAM permissions—to implementing sophisticated data processing workflows. The solution incorporates AWS Glue for initial data transformation and quality checks, EMR Serverless for advanced processing, and Redshift Serverless for data warehousing, all orchestrated through Airflow DAGs. This approach offers several key benefits: a unified interface that consolidates necessary tools, Python-based workflow flexibility, seamless AWS service integration, collaborative development through Git version control, cost-effective scaling through serverless computing, and comprehensive monitoring tools—all working together to create an efficient and maintainable data pipeline solution.

By using SageMaker Unified Studio workflows, you can accelerate your data pipeline development while maintaining enterprise-grade reliability and scalability. For more information about SageMaker Unified Studio and its capabilities, refer to the Amazon SageMaker Unified Studio documentation.

About the authors

Federate access to Amazon SageMaker Unified Studio with AWS IAM Identity Center and Ping Identity

2026-02-02 Raghavarao Sodabathina

Post Syndicated from Raghavarao Sodabathina original https://aws.amazon.com/blogs/big-data/federate-access-to-amazon-sagemaker-unified-studio-with-aws-iam-identity-center-and-ping-identity/

With an identity provider (IdP), you can manage your user identities outside of AWS and give these external user identities permissions to use AWS resources in your AWS accounts. External IdPs, such as Ping Identity, can integrate with AWS IAM Identity Center to be the source of truth for Amazon SageMaker Unified Studio. SageMaker Unified Studio also supports trusted identity propagation for SQL analytics, including Amazon Athena and Amazon Redshift.

SageMaker Unified Studio provides an integrated experience to use your data and tools for analytics and AI. You can use SageMaker Unified Studio to discover your data and put it to work using familiar AWS analytics and machine learning (ML) services for model development, generative AI, big data processing, and SQL analytics, assisted by Amazon Q Developer. By default, SageMaker domains support AWS Identity and Access Management (IAM) user credentials. You can also enable access to SageMaker domains in SageMaker Unified Studio for users with single sign-on (SSO) with IAM Identity Center and direct SAML integration with SageMaker Unified Studio.

Users can access SageMaker Unified Studio with their existing corporate credentials. With IAM Identity Center, administrators can connect their existing external IdPs and continue to manage users and groups in those existing identity systems, which can then be synchronized with IAM Identity Center using System for Cross-domain Identity Management (SCIM).In this post, we show how to set up workforce access with SageMaker Unified Studio using Ping Identity as an external IdP with IAM Identity Center.

In this post, we show how to set up workforce access with SageMaker Unified Studio using Ping Identity as an external IdP with IAM Identity Center.

Solution overview

We walk through the following high-level steps to implement this solution:

Enable IAM Identity Center.
Create a SageMaker Unified Studio domain.
Set up your IdP (for this example, Ping Identity).
Connect Ping Identity and IAM Identity Center.
Set up automatic provisioning of users and groups in IAM Identity Center.
Configure SageMaker Unified Studio SSO user access.

Prerequisites

For this walkthrough, you should have the following prerequisites:

An AWS account with IAM Identity Center enabled. It is recommended to use an organization-level IAM Identity Center instance for best practices and centralized identity management across your AWS organization.
A Ping Identity account.
A browser with network connectivity to Ping Identity and SageMaker Unified Studio.

Enable IAM Identity Center

To enable IAM Identity Center, follow the instructions in Enable IAM Identity Center.

Create a SageMaker Unified Studio domain

To create a SageMaker Unified Studio domain, refer to the instructions in Create a Amazon SageMaker Unified Studio domain – manual setup.

On the SageMaker console, go to the domain details and copy the Amazon Resource Name (ARN) under Domain ARN. You will use this value when you add your trust policy and when you connect your IAM IdP to your Ping Identity instance.

Create a SageMaker Unified Studio domain

Set up your IdP (Ping Identity)

In this section, we walk through the procedure to set up your IdP (for this example, Ping Identity).

Create an environment in Ping Identity

Complete the following steps to create an environment for Ping Identity:

Log in to your Ping Identity account.
Choose Create Environment.
Choose Create a Customer Solution.
In the Tailor your experiences pop-up, choose Skip.

Create a group in Ping Identity

Complete the following steps to create a group in Ping Identity:

On the Environments page, choose Manage Environments.
In the navigation pane, choose Directory, then choose Groups.
Choose the plus sign to add a group.
For Group Name, enter sagemaker
For Description, enter an optional description (for example, Amazon SageMaker Unified Studio).
For Population, choose Default.
Choose Save.
On the Roles tab for the sagemaker group, assign the Environment Admin role to the group.

Create a user in Ping Identity

Complete the following steps to create a user:

In the navigation pane, choose Directory, then choose Users.
Choose the plus sign to create a user.
Provide values for Given name, Family name, Username, and Email.
For Password, choose First time password.
Choose Save.

You can add more users as needed.

Assign group to user

Complete the following steps to assign your group to your user:

In the navigation pane, choose Directory, then choose Groups.
Choose the sagemaker group you created.
On the Users tab, choose the plus sign to add a user.
Add the user you created.

Connect Ping Identity and IAM Identity Center

To configure the integration between Ping Identity and IAM Identity Center, you need access to both management consoles. Although Ping Identity’s application catalog includes IAM Identity Center, we recommend configuring a standard SAML application for greater control over settings and attribute mappings.

Complete the following steps:

Go to the Ping Identity environment you created and choose Applications in the navigation pane.
Choose the plus sign to add an application:
1. For Application name, enter a name (for this example, we use unifiedstudio).
2. For Description, enter an optional description.
3. For Application Type, choose SAML Application.
4. Choose Configure.
Sign in to the IAM Identity Center console as a user with administrative privileges.
In the navigation pane, choose Settings to update your settings:
1. On the Identity source tab, choose Change identity source on the Actions dropdown menu.
2. For Choose identity source, select External identity provider, then choose Next.
3. In the Service provider metadata section, choose Download metadata file to download the IAM Identity Center metadata file.
  You will use this service provider metadata file in the next step when you connect Ping Identity with IAM Identity Center.
Return to the Ping Identity console and the SAML application page.
In the SAML Configuration section, select Import Metadata, upload the metadata file you downloaded, then choose Save.
On the Overview tab of the application page, choose Download Metadata under Connection details to download the Ping Identity IdP metadata.
You will use this for the SAML configuration in IAM Identity Center to set up Ping Identity as an IdP in the next step.
Return to the IAM Identity Center console and continue configuring your identity source:
1. In the Identity provider metadata section, choose Choose file under IdP SAML metadata, upload the metadata file you downloaded from Ping Identity, then choose Next.
2. Choose Accept to accept the disclaimer.
3. Choose Change identity source.
Return to the Ping Identity console to complete the SAML configuration.
On the Configuration tab, choose the edit icon to update the configuration:
1. For Sign, choose Sign Assertion & Response.
2. For Subject Name ID, enter urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress.
3. For Assertion Validity Duration, enter 300.
4. Leave the remaining values as default.
On the Attributes tab, choose the edit icon.
Choose +Add to add two attribute mappings:
1. Map the attribute saml-subject to Username, and leave Name format as default.
2. Map the attribute https://aws.amazon.com/SAML/Attributes/PrincipalTag:Email to Email Address, and set Name format to Unspecified.
3. Choose Save.
On the PingOne Policies tab, select Single Factor, then choose Save.
This post uses single-factor authentication for demonstration purposes only. In your environments, follow your organization’s security standards and governance framework.
On the Access tab, search for the sagemaker group under Group Membership Policy, and assign the unifiedstudio SAML application to the group.
Enable the application.

Set up automatic provisioning of users and groups from Ping Identity into IAM Identity Center

To configure the automatic provisioning of users and groups between Ping Identity and IAM Identity Center through SCIM, you must have access to both management consoles. Complete the following steps:

On the IAM Identity Center console, choose Settings in the navigation pane.
In the Automatic provisioning section, choose Enable.

This enables automatic provisioning in IAM Identity Center and displays the necessary SCIM endpoint and access token information.
In the Inbound automatic provisioning dialog box, copy the values for SCIM endpoint and Access token, then choose Close.
You will use these values to configure provisioning in Ping Identity in the next step.

This completes the setup process in IAM Identity Center.
Log in to the Ping Identity console.
In the navigation pane, choose Integrations, then choose Provisioning.
Choose the plus sign to add a new connection.
For Choose a connection type, choose Select next to Identity Store.
Provide a name (for this example, we use Identitycenter) and an optional description, then choose Next.
Under Configuration Authentication, provide the following configuration:
1. For SCIM BASE URL, enter the SCIM endpoint from IAM Identity Center.
2. For Authentication Method, choose OAuth 2 Bearer Token.
3. For Oauth Access Token, enter the access token from IAM Identity Center.
4. For Auth Type Header, choose Bearer (default option).
5. Choose Test Connection to validate the connection between Ping Identity and IAM Identity Center, then choose Next.
Under Configuration Preference, provide the following configuration:
1. For User Filter Expression, enter userName Eq “%s”.
2. For Group Membership Handling, select Merge.
3. Leave the remaining settings as default and choose Save.
On the Provisioning tab, choose the plus sign, then choose New Rule to create a rule for the SCIM connection.
Enter a name (for this example, unifiedstudio) and an optional description, then choose Create Rule.
Under the newly created rule, choose the plus sign next to Available Connections to add the connection identitycenter, then choose Save.
Edit the user filter:
1. For Attribute, choose Enabled.
2. For Operator, choose Equals.
3. For Value, choose true.
4. Choose Save.
Choose the edit icon next to Attribute Mapping and set the attribute mappings as shown in the following screenshot:
1. Delete the Primary Phone attribute mapping because it’s optional in AWS. Leaving this field blank can cause Ping Identity’s SCIM connector to generate errors during user provisioning.
2. Add a new attribute called Username under PingOne Directory and then map to displayName under Identitycenter.
Under Group Provisioning, choose the sagemaker group if you want to sync all sagemaker group users with auto provisioning.
1. In the pop-up, select I understand and want to continue, then choose Save.
On the Provisioning page, choose the Connections tab.
Enable the SCIM connection Identitycenter and rule unifiedstudio.

This completes the SCIM setup process between Ping Identity and IAM Identity Center.

Configure SageMaker Unified Studio SSO user access

Complete the following steps to configure SSO user access to SageMaker Unified Studio for your SageMaker domain:

On the SageMaker console, choose Domains in the navigation pane.
Choose the domain for which you want to configure SAML user access.
On the domain details page, you can find the SSO configuration in two locations:
1. From the main domain view, choose Configure next to Configure SSO user access.
2. Alternatively, scroll down to the User management tab and choose Configure SSO user access.
On the Choose user authentication method page, select IAM Identity Center, then choose Next.
For Choose user and group assignment method, choose from the following options, then choose Next:
1. Require assignments: Users and groups must be explicitly added to the domain to gain access. This provides more granular control over who can access the domain.
2. Do not require assignments: All authorized Ping Identity users and groups can access this domain if they have been assigned to the SAML application in Ping Identity.
For either option, users or groups must have access to the Ping Identity SAML application (unifiedstudio in this example) to authenticate successfully.
On the Review and save page, review your choices and choose Save. These settings can’t be changed after you save them.
If you’ve chosen to require assignments, use the Add users and groups section to add SAML users and groups to your domain.

Now, users will be able to access SageMaker Unified Studio using the domain URL with their SSO credentials.

You can explore different projects for your users and assign those projects based on your IdP user groups for fine-grained access controls. For example, you can create different SAML user groups based on their job function in Ping Identity, then assign those Ping Identity groups to the unifiedstudio SAML application in Ping Identity, and then assign those Ping Identity SAML groups to their respective project profiles in SageMaker Unified Studio. To assign project profiles for their respective groups, choose the Project profiles tab and choose your project profile. On the Authorized users and groups page, choose Add, then choose SSO groups. Choose Add users and groups button to complete the project profile assignment.

Assigning a project profile to Ping Identity group

Validate access with Ping Identity users

Complete the following steps to validate access:

On the SageMaker domain details page, choose the link for the SageMaker Unified Studio URL.
Log in with your user credentials.
After successful login, you will be redirected to the SageMaker Unified Studio home page. Here, you can explore different projects to your users and assign those projects based on your SAML user groups for fine-grained access control.
To assign an authorization policy, those Govern and then Domain units.
Choose your SageMaker domain, then choose a suitable authorization policy. For this example, we choose Project creation policy.
Choose Add policy grant to assign user groups or users to their respective project profiles.

You have successfully federated SageMaker Unified Studio with Ping Identity as an IdP with IAM Identity Center. You can connect to SageMaker Unified Studio by using your Ping Identity credentials.

Clean up

After you test out this solution, remember to delete the resources you created to avoid incurring future charges. For instructions to delete your SageMaker Unified Studio domain, refer to Delete domains. If you want to delete your Ping Identity account, reach out to Ping Identity for assistance.

Conclusion

In this post, we demonstrated how to set up Ping Identity as an IdP over SAML authentication for SageMaker Unified Studio access through IAM Identity Center federation. To learn more, refer to the Amazon SageMaker Unified Studio User Guide, which provides guidance on how to build data and AI applications using SageMaker.

About the authors

AWS Weekly Roundup: Amazon Bedrock agent workflows, Amazon SageMaker private connectivity, and more (February 2, 2026)

2026-02-02 Betty Zheng (郑予彬)

Post Syndicated from Betty Zheng (郑予彬) original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-bedrock-agent-workflows-amazon-sagemaker-private-connectivity-and-more-february-2-2026/

Over the past week, we passed Laba festival, a traditional marker in the Chinese calendar that signals the final stretch leading up to the Lunar New Year. For many in China, it’s a moment associated with reflection and preparation, wrapping up what the year has carried, and turning attention toward what lies ahead.

Looking forward, next week also brings Lichun, the beginning of spring and the first of the 24 solar terms. In Chinese tradition, spring is often seen as the season when growth begins and new cycles take shape. There’s a common saying that “a year’s plans begin in spring,” capturing the idea that this is a time to set one’s direction and start fresh.

Last week’s launches
Here are the launches that got my attention this week:

Amazon Bedrock enhances support for agent workflows with server-side tools and extended prompt caching – Amazon Bedrock introduced two updates that improve how developers build and operate AI agents. The Responses API now supports server-side tool use, so agents can perform actions such as web search, code execution, and database updates within AWS security boundaries. Bedrock also adds a 1-hour time-to-live (TTL) option for prompt caching, which helps improve performance and reduce the cost for long-running, multi-turn agent workflows. Server-side tools are available with OpenAI GPT OSS 20B and 120B models, and the 1-hour prompt caching TTL is generally available for select Claude models by Anthropic in Amazon Bedrock.
Amazon SageMaker Unified Studio adds private VPC connectivity with AWS PrivateLink – Amazon SageMaker Unified Studio now supports AWS PrivateLink, providing private connectivity between your VPC and SageMaker Unified Studio without routing customer data over the public internet. With SageMaker service endpoints onboarded into a VPC, data traffic remains within the AWS network and is governed by IAM policies, supporting stricter security and compliance requirements.
Amazon S3 adds support for changing object encryption without data movement – Amazon S3 now supports changing the server-side encryption type of existing encrypted objects without moving or re-uploading data. Using the UpdateObjectEncryption API, you can switch from SSE-S3 to SSE-KMS, rotate customer -managed AWS Key Management Service (AWS KMS) keys, or standardize encryption across buckets at scale with S3 Batch Operations while preserving object properties and lifecycle eligibility.
Amazon Keyspaces introduces table pre-warming for predictable high-throughput workloads – Amazon Keyspaces (for Apache Cassandra) now supports table pre-warming, which helps you proactively set warm throughput levels so tables can handle high read and write traffic instantly without cold-start delays. Pre-warming helps reduce throttling during sudden traffic spikes, such as product launches or sales events, and works with both on-demand and provisioned capacity modes, including multi-Region tables. The feature supports consistent, low-latency performance while giving you more control over throughput readiness.
Amazon DynamoDB MRSC global tables integrate with AWS Fault Injection Service – Amazon DynamoDB multi-Region strong consistency (MRSC) global tables now integrate with AWS Fault Injection Service. With this integration, you can simulate Regional failures, test replication behavior, and validate application resiliency for strongly consistent, multi-Region workloads.

Additional updates
Here are some additional projects, blog posts, and news items that I found interesting:

Building zero-trust access across multi-account AWS environments with AWS Verified Access – This post walks through how to implement AWS Verified Access in a centralized, shared-services architecture. It shows how to integrate with AWS IAM Identity Center and AWS Resource Access Manager (AWS RAM) to apply zero trust access controls at the application layer and reduce operational overhead across multi-account AWS environments.
Amazon EventBridge increases event payload size to 1 MB – Amazon EventBridge now supports event payloads up to 1 MB, an increase from the previous 256 KB limit. This update helps event-driven architectures carry richer context in a single event, including complex JSON structures, telemetry data, and machine learning (ML) or generative AI outputs, without splitting payloads or relying on external storage.
AWS MCP Server adds deployment agent SOPs (preview) – AWS introduced deployment standard operating procedures (SOPs) that AI agents can deploy web applications to AWS from a single natural language prompt in MCP -compatible integrated development environments (IDEs) and command line interfaces (CLIs) such as Kiro, Cursor, and Claude Code. The agent generates AWS Cloud Development Kit (AWS CDK) infrastructure, deploys AWS CloudFormation stacks, and sets up continuous integration and continuous delivery (CI/CD) workflows following AWS best practices. The preview supports frameworks including React, Vue.js, Angular, and Next.js.
AWS Network Firewall adds generation AI traffic visibility with web category filtering – AWS Network Firewall now provides visibility into generative AI application traffic through predefined web categories. You can use these categories directly in firewall rules to govern access to generative AI tools and other web services. When combined with TLS inspection, category-based filtering can be applied at the full URL level.
AWS Lambda adds enhanced observability for Kafka event source mappings – AWS Lambda introduced enhanced observability for Kafka event source mappings, providing Amazon CloudWatch Logs and metrics to monitor event polling configuration, scaling behavior, and event processing state. The update improves visibility into Kafka-based Lambda workloads, helping teams diagnose configuration issues, permission errors, and function failures more efficiently. The capability supports both Amazon Managed Streaming for Apache Kafka (Amazon MSK) and self-managed Apache Kafka event sources.
AWS CloudFormation 2025 year in review – This year-in-review post highlights CloudFormation updates delivered throughout 2025, with a focus on early validation, safer deployments, and improved developer workflows. It covers enhancements such as improved troubleshooting, drift-aware change sets, stack refactoring, StackSets updates, and new -IDE and AI -assisted tooling, including the CloudFormation language server and the Infrastructure as Code (IaC) MCP server.

Upcoming AWS events
Check your calendars so that you can sign up for this upcoming event:

AWS Community Day Romania (April 23–24, 2026) – This community-led AWS event brings together developers, architects, entrepreneurs, and students for more than 10 professional sessions delivered by AWS Heroes, Solutions Architects, and industry experts. Attendees can expect expert-led technical talks, insights from speakers with global conference experience, and opportunities to connect during dedicated networking breaks, all hosted at a premium venue designed to support collaboration and community engagement.

If you’re looking for more ways to stay connected beyond this event, join the AWS Builder Center to learn, build, and connect with builders in the AWS community.

Check back next Monday for another Weekly Roundup.

–betty

Build a trusted foundation for data and AI using Alation and Amazon SageMaker Unified Studio

2026-01-30 Anthony Lempelius, James Mesney

Post Syndicated from Anthony Lempelius, James Mesney original https://aws.amazon.com/blogs/big-data/build-a-trusted-foundation-for-data-and-ai-using-alation-and-amazon-sagemaker-unified-studio/

This post was co-written with Anthony Lempelius and James Mesney from Alation.

When a team wants to reuse a dataset, whether it is to build a new pipeline, launch a dashboard, run an analysis, or power an AI application, the first challenge is rarely the code. Data engineers need to understand lineage, transformations, and operational expectations. Data analysts and BI engineers need consistent definitions, metrics, and trusted sources. Data scientists and AI engineers need to know provenance, quality, access constraints, and how data or features were derived. In many organizations, that context is captured in different places by different teams, often across solutions like Alation and SageMaker Unified Studio, both of which can serve as a system of record for business context depending on who is doing the work and where they operate day to day. When those perspectives are not connected, people revalidate the same information, debate definitions, and duplicate documentation across tools. A unified metadata foundation brings these role specific views together so business context, technical metadata, and governance stay aligned across platforms, making data easier to trust, easier to find, and easier to use across analytics and AI.

The new Alation integration with Amazon SageMaker Unified Studio addresses these challenges by synchronizing catalog metadata between both systems. This synchronization creates a unified metadata experience where technical teams working in SageMaker Unified Studio and business teams working in Alation collaborate on top of the same metadata. You can verify how ML and analytics assets are created, understand dependencies, and maintain traceability across your data lifecycle regardless of which system your teams prefer to use.

In this post, we demonstrate who benefits from this integration, how it works, the specific metadata it synchronizes, and provide a complete deployment guide for your environment.

The value of unified metadata governance

Organizations managing large-scale analytics and ML workloads face critical challenges when metadata is fragmented across multiple systems. When metadata exists in silos, data scientists spend valuable time searching for the right datasets. Teams duplicate metadata management efforts, creating inconsistent definitions and conflicting metrics across the organization.

Regulatory requirements demand clear provenance. Without unified metadata governance, organizations struggle to demonstrate compliance, trace data origins, and maintain audit trails across their ML and analytics pipelines. Data discovery becomes a bottleneck when teams can’t quickly find, understand, and trust the data they need, delaying model development and reducing the overall business value of data investments.

Applying consistent governance policies across disparate systems is nearly impossible without a unified metadata layer. This creates security vulnerabilities, data quality issues, and compliance blind spots. A unified metadata governance approach alleviates these challenges by providing a single source of truth for metadata across ML and analytics systems, enabling faster data discovery, consistent governance, and confident compliance while reducing the operational burden on data and ML teams.

Solution overview

The Alation and SageMaker Unified Studio integration unifies the user experience, synchronizing metadata from cataloged assets between both systems.

This Phase 1 integration extracts metadata from Amazon SageMaker Catalog into Alation, giving you one place to discover assets.

The integration connects through AWS Identity and Access Management (IAM) authentication and synchronizes key metadata elements, including domains, projects, asset names, descriptions, owners, glossary terms, and custom metadata fields. Every metadata update includes provenance information: the originating service, the person who made the change, and the timestamp, creating comprehensive audit trails for compliance.

You can run metadata extractions on demand or schedule them to run automatically. The system performs an initial bulk extraction of your selected domains and projects, then keeps it up-to-date through incremental updates using either event-driven triggers or scheduled polling. Communication uses encrypted APIs with scoped IAM permissions following least-privilege principles.

This integration helps organizations in financial services, telecommunications, retail, manufacturing, and transportation that manage large numbers of analytics and ML workloads across many systems and teams. You can reduce metadata duplication, accelerate data discovery, and enable your data scientists, analysts, and engineers to find trusted data faster so they can focus on building insights rather than validating data quality.

The following diagram illustrates the solution architecture.

The following screenshot showcases the Alation catalog displaying the SageMaker Unified Studio project and its synchronized assets.

Metadata synchronization

This integration automatically synchronizes essential metadata between SageMaker Unified Studio and Alation, facilitating consistent information across both systems. The synchronization brings together the types of metadata you need for discovery, governance, and audit workflows, giving you clearer insight into how datasets, features, and models relate across your services.

The integration synchronizes catalog metadata, including domains, projects, asset names, descriptions, owners, glossary terms, and metadata forms. Additionally, the integration synchronizes provenance metadata, which includes information about the originating service, the actor who made the change, and the timestamp, to support traceability and audit workflows.

Integration mechanics

The integration connects SageMaker Unified Studio and Alation through a scoped IAM role that provides secure, encrypted communication. After you configure this connection within Alation, the system performs an initial extraction of your selected domains and projects, then keeps information current through incremental updates using either event-driven triggers or scheduled polling.

The integration synchronizes metadata forms from SageMaker Unified Studio into Alation through automated field mapping between both systems’ schemas. Metadata forms can capture various asset specific details like feature store references, training run identifiers, model versions, and evaluation metrics.

Every metadata update includes provenance information: the originating service, the person who made the change, and when it occurred. This supports audit and stewardship workflows. Access controls follow least-privilege principles through IAM while applying Alation’s role-based permissions, letting you limit synchronization by project, namespace, or tag as needed.

Security and compliance

Security and compliance are critical when synchronizing metadata across systems. This integration follows enterprise security practices to facilitate safe, controlled metadata synchronization. The connector uses least-privilege access, encrypted transport, and clear separation between metadata and data, so you can maintain governance without disrupting existing workflows.

You configure a scoped IAM role to define which accounts, projects, and namespaces the connector can access, making sure access follows your organization’s security policies. Metadata moves over TLS-protected APIs, and you control which domains and projects to include in Alation. By default, the integration synchronizes only metadata; your data files and artifacts remain in their original AWS locations unless you explicitly choose to export them.

Alation maintains a complete audit trail by recording extraction events, mapping changes, and stewardship activities. These security controls support compliant metadata governance while preserving your existing operational practices.

Prerequisites

Before setting up this integration, ensure you have the following:

An Alation Cloud Service (ACS) instance
Alation server admin access
An AWS account
A SageMaker Unified Studio domain and project with existing metadata

Configure authentication

Before configuring the Alation connector, you must set up the required AWS resources and permissions. The first step is to configure authentication. The Alation connector supports two authentication methods to access SageMaker Unified Studio. Choose the method that best fits your security requirements.

Option 1: IAM role (Recommended)

Create an IAM role that the Alation connector will assume to access SageMaker Unified Studio. For detailed instructions on creating IAM roles, see IAM role creation.

The following is an example IAM permission policy for SageMaker Catalog access:

{
   "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AlationSageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:ListDomains",
                "datazone:GetFormType",
                "datazone:Search",
                "datazone:ListProjects",
                "datazone:GetAsset"
            ],
            "Resource": "arn:aws:datazone:<region>:<account-id>:domain/*”
        }
    ]
}

The following is an example trust policy for the IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AlationSageMakerAccessAssumeRole",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<alation_provided_role_arn>"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Option 2: IAM user with access keys

Create an IAM user with programmatic access and attach the necessary permissions. For detailed instructions on creating IAM users, see Create an IAM user in your AWS account.

Create an IAM user with programmatic access enabled, attach the following policy, and generate access keys for use in Alation configuration:

{
   "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AlationSageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:ListDomains",
                "datazone:GetFormType",
                "datazone:Search",
                "datazone:ListProjects",
                "datazone:GetAsset"
            ],
            "Resource": "arn:aws:datazone:<region>:<account-id>:domain/*"
        }
    ]
}

Add IAM role or user to SageMaker Unified Studio domain

Add the IAM role or user you created to the SageMaker Unified Studio domain. For detailed instructions on adding users to a domain, see User management in Amazon SageMaker Unified Studio. The following screenshot shows an example of adding IAM users on the SageMaker dashboard.

Add IAM role or user to SageMaker Unified Studio projects

The IAM role or user must be added as a member to all SageMaker Unified Studio projects that contain metadata you want to synchronize with Alation. Projects without this member will not be included in the synchronization process.

Add the IAM role or user as a project member with Contributor or Owner permissions for each project you want to include in the sync, as illustrated in the following screenshot. For detailed instructions on adding project members, see Add project members.

Install SageMaker enhanced connector

After completing the AWS setup, you can configure the Alation connector to establish the integration. The connector is distributed as a .zip package for upload and installation in the Alation application. To obtain the connector, contact the Forward Deployed Engineering team or your Alation Account Manager.

When you have the .zip package, follow the installation procedures to add the connector.

Create and configure Alation’s data source

Navigate to the Data Sources section in Alation, create a new data source, and select SageMaker Catalog as the source type. Configure the connection settings with the authentication method chosen in the AWS setup.

For IAM role authentication, use the following configuration:

Connection Type: IAM Role
Role ARN: ARN of the IAM role created in AWS setup
External ID: External ID configured in the trust policy
AWS Region: Region where your SageMaker Unified Studio domain is located

For IAM user authentication, use the following configuration:

Connection Type: Access Keys
Access Key ID: Access key from AWS setup
Secret Access Key: Secret key from AWS setup
AWS Region: Region where your SageMaker Unified Studio domain is located

Test the connection to verify authentication and network connectivity, as shown in the following screenshot.

Configure metadata extraction settings

Configure the extraction scope by selecting the SageMaker domains and projects to synchronize, as shown in the following screenshot. Only projects where the IAM role or user is a member will be available for synchronization.

Run initial extraction

Execute the first metadata synchronization to import existing metadata from SageMaker Unified Studio into Alation. Monitor the extraction progress through Alation’s status indicators and validate that SageMaker assets appear correctly in the catalog.

The following screenshot shows the job history page with job status Running.

The following screenshot shows the job history page with job status Succeeded.

The following screenshot shows the Alation catalog displaying the SageMaker Unified Studio project and its synchronized assets.

Operate and tune

Configure ongoing operations by setting extraction cadence, configuring reconciliation alerts, and monitoring logs regularly. Add data stewards to synchronized assets, and consider enabling AI-generated descriptions or working with Alation Professional Services for advanced governance design.

Enhanced capabilities

The next phase of the integration introduces three key capabilities: bi-directional metadata synchronization, lineage replication, and data quality metadata replication. The bi-directional capability gives you the flexibility to control where metadata updates originate, either in Alation or in SageMaker Unified Studio, so you can manage metadata changes in the service that best aligns with your organizational workflows and governance processes.

The feature set is rolling out in phases. Phase 1 is available at the time of writing this post and provides extraction from SageMaker Unified Studio into Alation, including initial and incremental updates and audit logging. Phase 2 is coming soon and will offer configurable principal catalogs, advanced scoped syncs, and reconciliation workflows for Alation Cloud Service customers.

These enhancements will support governed, scalable ML operations with increasing depth and automation.

Conclusion

The Alation and SageMaker Unified Studio integration helps organizations bridge the gap between fast analytics and ML development and the governance requirements most enterprises face. By cataloging metadata from SageMaker Unified Studio in Alation, you gain a governed, discoverable view of how assets are created and used. This supports leaders, stewards, compliance teams, and ML practitioners who depend on accurate, well-documented data to scale analytics and AI responsibly.

To learn more about this integration and explore additional resources, refer to the Amazon SageMaker Unified Studio User Guide and Alation Documentation.

About the authors

Get started faster with one-click onboarding, serverless notebooks, and AI agents in Amazon SageMaker Unified Studio

2026-01-27 Siddharth Gupta

Post Syndicated from Siddharth Gupta original https://aws.amazon.com/blogs/big-data/get-started-faster-with-one-click-onboarding-serverless-notebooks-and-ai-agents-in-amazon-sagemaker-unified-studio/

Data teams today struggle with fragmented tools, complex infrastructure provisioning, and hours spent writing boilerplate code to connect to data sources. This forces analysts, data scientists, and engineers to work in separate environments, which slows collaboration and time to insight. Since our launch of Amazon SageMaker Unified Studio in March 2025, leading companies such as Bayer, NatWest, and Carrier have adopted it to bring their data teams into one collaborative workspace with unified tools, straightforward infrastructure provisioning, and fast connections to data sources.

Continuing our mission to provide faster time-to-value for customers, in November 2025, we announced Amazon SageMaker notebooks, a serverless workspace with a built-in AI agent in Amazon SageMaker Unified Studio. You can now launch a notebook in seconds, generate code from natural language prompts, and connect automatically to data across Amazon Simple Storage Service (Amazon S3), Amazon Redshift, third-party databases, and more from a single environment without needing to pre-provision or tune data processing infrastructure. Inside these serverless notebooks, analysts can perform SQL queries, data scientists can execute Python code, and data engineers can process large-scale data jobs in Spark within a single workspace. Together with the new one-click onboarding available for SageMaker Unified Studio, customers can go from their existing AWS data to running analytics and machine learning workloads much faster, spending their time on analysis rather than setup and configuration.

In this post, we walk you through how these new capabilities in SageMaker Unified Studio can help you consolidate your fragmented data tools, reduce time to insight, and collaborate across your data teams. Here’s a short demo of the new capabilities:

One-click onboarding of existing AWS datasets

Get started exploring your data with one-click onboarding that provisions and configures environments in minutes instead of weeks. The new onboarding experience can reuse existing AWS Identity and Access Management (IAM) roles to provide access to SageMaker Unified Studio, automatically connecting to data sources across S3 buckets, S3 Tables, AWS Glue Data Catalog, and AWS Lake Formation policies, removing the need for additional data permission setup. Under the covers, a new IAM-based domain and project are created with default notebook and compute resources preconfigured. When complete, you enter SageMaker Unified Studio with all your tools available in the left-side navigation along with built-in samples to accelerate first use, as seen in the following screenshot.

“New features with Amazon Sagemaker will unlock a new paradigm of innovation, allowing Codex to significantly accelerate time-to-value for our customers, and transform them from aging to agentic in weeks, not months.“

– Abhinav Sharma, Chief Data Officer, Codex

You can start directly from Amazon SageMaker, Amazon Athena, Amazon Redshift, or Amazon S3 Tables, giving them a fast path from their existing tools and data to the unified experience in SageMaker Unified Studio. After you choose Get Started and specify an IAM role, SageMaker automatically creates a project with the existing data permissions intact from Data Catalog, Lake Formation, and Amazon S3. As a result, teams can immediately discover and act on their data using the existing data permissions and infrastructure.

For more information, see New one-click onboarding and notebooks with a built-in AI agent in Amazon SageMaker Unified Studio

Serverless SageMaker notebooks

The fully managed, web-based notebooks in SageMaker Unified Studio support multiple programming languages, letting you write Python, SQL, and Spark code in the same notebook. The infrastructure adjusts automatically based on your workload, while built-in libraries create charts and insights directly in your workflow. When your analysis scales beyond interactive queries to large-scale data processing, Amazon Athena for Apache Spark engine delivers optimized performance, integrating with the serverless notebook experience to execute analytical workloads efficiently. This serverless approach eliminates the need to provision clusters or maintain servers, reducing the time from question to insight.

“The new SageMaker interface brings clarity and speed to the entire ML lifecycle. Its developer-friendly design has made our experimentation and delivery significantly faster,“

– Sachin Mittal, Product Manager at Deloitte.

As shown in the preceding image, the notebook gives data engineers, analysts, and data scientists one place to perform SQL queries, execute Python code, process large-scale data jobs, run machine learning workloads, and create visualizations without having to switch between tools.

AI-assisted development with Data Agent

To accelerate development further, the new SageMaker Data Agent helps create SQL, Python, or Spark code using natural language prompts. Instead of spending hours writing boilerplate code to connect to your data sources and understand schemas, you can describe what you want to accomplish. The agent analyzes data catalog metadata about your available datasets, schemas, and relationships to provide context-aware assistance.

In the preceding example image, if you prompt Build and analyze a complete sales forecast based on the sample retail data, the agent helps identify the relevant tables and suggests the appropriate joins and analysis approach, transforming what might take hours into minutes. To try this yourself, navigate to the Overview tab in your SageMaker Studio environment and look for the Retail Sales Forecasting with SageMaker XGBoost notebook in the sample notebooks collection—these examples are automatically available when you first set up SageMaker Studio. The agent breaks down complex analytical workflows into manageable, executable steps, so you can move from question to insight faster.

Learn more about SageMaker

In this post, we focused on three new SageMaker Unified Studio capabilities recently made available, but they’re a fraction of the more than 40 launches last year. Here’s a list of videos of re:Invent sessions and the measurable results from leading organizations adopting SageMaker Unified Studio, including:

Summary of 2025 launches: What’s new with Amazon SageMaker in the era of unified data and AI (ANT216)
NatWest Group plans to scale to 72,000 employees having federated data access using SageMaker Unified Studio. Watch their presentation.
Commonwealth Bank of Australia migrated 10 petabytes and 61,000 pipelines into AWS and has setup SageMaker Unified Studio to provide unified access to 40 different lines of business in their ongoing data transformation journey. Watch their presentation.
Carrier Global Corporation improved natural language to SQL agent accuracy by 38% through the SageMaker Catalog’s governed metadata and business glossary. Watch their presentation.
Bayer is now positioned to onboard over 300 TB of biomarker data and integrate siloed omics, clinical, and chemistry data repositories into a cohesive environment built on Amazon SageMaker. Read their story.

Conclusion

Using Amazon SageMaker Unified Studio serverless notebooks, AI-assisted development, and unified governance, you can speed up your data and AI workflows across data team functions while maintaining security and compliance. To learn more visit the SageMaker product page or get started in the SageMaker console.

About the authors

Power up your analytics with Amazon SageMaker Unified Studio integration with Tableau, Power BI, and more

2026-01-22 Narendra Gupta

Post Syndicated from Narendra Gupta original https://aws.amazon.com/blogs/big-data/power-up-your-analytics-with-amazon-sagemaker-unified-studio-integration-with-tableau-power-bi-and-more/

Organizations face challenges in accessing and analyzing governed data across multiple sources through their preferred business intelligence (BI) and analytics tools while maintaining security and governance. They need a seamless way to connect their familiar tools (like Tableau, Power BI, Excel) to Amazon SageMaker‘s data assets without compromising data governance and security protocols.

Amazon SageMaker supports authentication through the Amazon Athena JDBC driver, allowing data users to query their subscribed data lake assets via popular BI and analytics tools like Tableau, Power BI, Excel, SQL Workbench, DBeaver, and more. This integration empowers data users to access and analyze governed data within Amazon SageMaker using familiar tools, boosting both productivity and flexibility.

Customers use Amazon SageMaker Unified Studio to streamline data access and governance by enabling data users to locate and subscribe to data from multiple sources within a single project. Amazon SageMaker Unified Studio natively integrates with Amazon-specific options like Amazon Athena, Amazon Redshift, and Amazon SageMaker AI, allowing users to analyze their project governed data. With this launch of JDBC connectivity, Amazon SageMaker Unified Studio expands its support for data users, including analysts and scientists, allowing them to work in their preferred tools, whether it’s SQL Workbench, Domino, or Amazon-native solutions like Amazon Athena, while ensuring secure, governed access within Amazon SageMaker Unified Studio.

Getting Started

To get started, download and install the latest Athena JDBC driver for your tool of choice. After installation, copy the JDBC connection string from the Amazon SageMaker Unified Studio portal into the JDBC connection configuration to establish a connection from your tool. This directs you to authenticate using single sign-on (SSO) with your corporate credentials. After connecting, you can query, visualize, and share data—governed by Amazon SageMaker Unified Studio–within the tools you already know and trust.

In this post, we guide you through connecting various analytics tools to Amazon SageMaker Unified Studio using the Athena JDBC driver, enabling seamless access to your subscribed data within your Amazon SageMaker Unified Studio projects.

Solution overview

To demonstrate these capabilities, consider a use case where your marketing team wants to analyze sales data to understand patterns in sales by stores and sales representatives. To achieve this, your marketing team needs access to sales_performance_by_store, and sales_performance_by_rep data owned by the sales team. The sales team, acting as the data producer, publishes the necessary data assets to Amazon SageMaker Unified Studio, allowing the marketing team, as a consumer, to discover and subscribe to these assets.

After the subscription is approved, the data assets become available within the marketing team’s project environment in Amazon SageMaker Unified Studio. The marketing team can then use their preferred tool to perform data exploration. An example architecture of how this is done using DBeaver is shown in the following image:

SageMaker Unified Studio project architecture diagram showing data collaboration between Sales and Marketing teams with Amazon S3 storage and Athena integration

Prerequisites

To follow along with this post, you need the following prerequisites in place:

AWS account – If you don’t have an active AWS account, see How do I create and activate a new AWS account?.
Amazon SageMaker resources – You need a domain for Amazon SageMaker, and two Amazon SageMaker project.
Publish data assets – As the data producer from the sales team, you can now ingest individual data assets into Amazon SageMaker Unified Studio. For this use case, create a data source and import the technical metadata of two data assets – sales_performance_by_store, and sales_performance_by_rep – from AWS Glue Data Catalog. Ensure the data assets are enriched with business descriptions and published to the catalog.
Note: Here we are using tables which are in the Glue catalog but with Sagemaker Lakehouse you have the option to bring assets from other sources.
Subscribe data assets – As a data analyst from the marketing team, you can now discover and subscribe to the data assets. The data producer from the retail team reviews and approves your subscription. Upon successful fulfillment, the data assets are added to your SageMaker Unified project.

For detailed instructions for publishing and subscribing, see the Amazon SageMaker Unified Studio User Guide.

The following figure shows the subscribed assets added to the subscribed assets section in your marketing project catalog.

SageMaker Unified Studio Assets page displaying subscribed data assets with accessibility status indicators

In the following sections, we walk you through the steps to configure DBeaver to consume the subscribed assets from Amazon SageMaker Unified Studio.

Configuring DBeaver to access subscribed data assets

In this section, you configure DBeaver to access the subscribed assets from the Marketing project

To configure DBeaver:

Connect with JDBC: In the Amazon SageMaker Unified Studio, (1) open the Marketing project, (2) on the Project overview screen, (3) choose JDBC connection details tab.
Copy the JDBC connection URL into a text editor. The URL should have the following parameters needed for configuring the database connection in DBeaver – Domain ID, Environment ID, Region, and IDC Issuer URL.
Download and install the latest Athena driver:
- If DBeaver has the Athena driver pre-installed, it might be the older (v2) version. To ensure compatibility with Amazon SageMaker Unified Studio, you need the latest driver (v3), which includes the necessary authentication features.
- Download the latest JDBC driver—version 3.x.
- To install the latest driver:
  - Go to Database and then to Driver Manager in DBeaver.
  - Select the Athena driver and choose Edit.
  - Visit the Libraries tab.
  - Choose Download/Update to fetch the latest driver version.
  - If prompted, select the appropriate version and confirm the download.
In the DBeaver SQL client, create a new database connection and select the Athena driver.
Switch to the Driver Properties tab, enter the values of the following properties that are available in the JDBC connection URL you copied from Amazon SageMaker Unified Studio. If any of these properties are not already available, you can add them and provide their respective values.
- CredentialsProvider: The credentials provider to authenticate requests to AWS
- DataZoneDomainId: The ID of your Amazon DataZone domain
- DataZoneDomainRegion: The AWS Region where your domain is hosted
- DataZoneEnvironmentId: The ID of your DefaultDataLake environment
- IdentityCenterIssuerUrl: The issuer URL used by AWS Identity and Access Management (IAM) Identity Center for token issuance
- OutputLocation: Amazon S3 path for storing query results
- Region: The Region where the environment is created
- Workgroup: Amazon Athena workgroup of the environment
- ListenPort: Pick any four digits port number. This is the port number that listens for the IAM Identity Center response
Choose Test Connection….
You are redirected to the IAM Identity Center sign-in portal. Sign in with Marketing user credentials. If you’re already signed in through single sign-on (SSO), this step can be skipped.
After you sign in, if you are prompted to authorize the DataZoneAuthPlugin. Choose Allow access to authorize access to Amazon DataZone from DBeaver.
After sign in completes, you see the following message. You can close the window and go to the DBeaver.
After the connection is established, the following success message appears.
You can now view and query all subscribed assets directly within DBeaver.

These steps might also apply to other analytics tools and clients that support JDBC connections. If you’re using a different tool, you might need to adapt these instructions accordingly to ensure proper configuration and access to Amazon SageMaker Unified Studio data assets.

Integration with other applications

You can use similar steps for other BI and analytics tools that support standard database connections.

Connect to Tableau Desktop

Use the Athena JDBC driver to connect Tableau to Amazon SageMaker Unified Studio and visualize your subscribed data.To connect to Tableau Desktop:

Make sure that you’re using the latest Athena JDBC 3.x driver.
Copy the JDBC driver file and place it in the appropriate folders for your operating system
- For Mac OS: ~/Library/Tableau/Drivers
- For Windows: C:\Program Files\Tableau\Drivers
Open Tableau Desktop. From the To a Server connection menu, select Other Databases (JDBC) to connect to Amazon SageMaker Unified Studio.
Paste the JDBC connection URL you copied from the SageMaker Unified Studio portal into the URL. Leave other fields such as Dialect, Username, and Password blank and choose Sign in.
If you get a port is occupied error – add “;ListenPort=8055” to the URL to change the port. You can use any port number.
This redirects you to authenticate with IAM Identity Center. Enter the credentials of the Identity Center user that you used to sign in to the SageMaker Unified Studio portal. Authorize the DataZoneAuthPlugin to access Amazon DataZone from Tableau. Once the connection is established with the success message, you can view your project’s subscribed data directly within Tableau and build dashboards.

Connect to Microsoft Power BI

Now, we look at connecting Amazon SageMaker Unified Studio with Microsoft Power BI on Windows.While Amazon Athena provides a native ODBC driver for connecting to ODBC-compatible tools like Microsoft Power BI, it currently doesn’t support Amazon SageMaker Unified Studio authentication. Therefore, in this post, we use an ODBC-JDBC bridge to connect Amazon SageMaker Unified Studio with Microsoft Power BI using the Athena JDBC driver, which supports SageMaker Unified Studio authentication.

In this post, we’re using the ZappySys driver as the ODBC-JDBC bridge. This is a third-party solution that requires a separate licensing fee, which isn’t included in the AWS solution. You can choose to use any other solution for ODBC-JDBC bridge.To connect to Power BI:

Make sure that you have administrator privileges to run the ODBC Data Source Administrator.
From the Windows Start menu, run the ODBC Data Source Administrator (the 64-bit version) using run as Administrator.
Create a New Data Source with the ZappySys JDBC Bridge Driver. You are prompted to enter your connection details.
Paste the JDBC URL you copied from the SageMaker Unified Studio portal in the Connection String, along with the driver class and JDBC driver file. Make sure that you’re using the latest Athena JDBC 3.x driver.
Choose Test Connection. A new dialog window pops up after the connection is successful.
This redirects you to authenticate with IAM Identity Center. Enter the credentials of the Identity Center user that you used to sign in to the SageMaker Unified Studio portal. Authorize the DataZoneAuthPlugin.
Choose Preview tab on ZappySys JDBC Bridge Driver window and choose one of the subscribed tables to access data.
After configuring the data source, launch Power BI. Create a blank report or use an existing report to integrate the new visuals. Choose Get Data and select the name of the data source you created. This opens a new browser window to authenticate your credentials. Allow access to authorize the DataZone Auth plugin. After authorization is complete, you can build your reports in Microsoft Power BI with the subscribed data assets.

Connect to SQL Workbench

Discover how SQL Workbench can connect to Amazon SageMaker Unified Studio for users who prefer a SQL interface to query data lake tables and views subscribed through projects in Amazon SageMaker Unified Studio.

To connect to SQL Workbench:

Make sure that you’re using the latest Athena JDBC 3.x driver.
Open SQL Workbench/J and choose Manage Drivers.
Select the option to add a new driver. Enter a name for it, such as SMUSAthenaJDBC, and import the driver you downloaded in the previous steps.
Create a new connection profile and enter a name it, such as smus-profile. In the Driver dropdown, select the driver you configured. For the URL, enter the string jdbc:athena://region=us-east-1; (In the example, the Virginia Region is being used). Choose Extended Properties.
Under Extended Properties, add the following parameters that you copied from the SageMaker Unified Studio portal. You can also include these parameters in the JDBC (URL) connection string. Choose OK.
- Workgroup
- OutputLocation
- DataZoneDomainId
- IdentityCenterIssuerURL
- CredentialsProvider
- DatazoneEnvironmentId
- DataZoneDomainRegain
Alos add “ListenPort” with any port number.
This redirects you to authenticate with IAM Identity Center. Enter the credentials of the Identity Center user that you used to sign in to the SageMaker Unified Studio portal. Authorize the DataZoneAuthPlugin.
After successful connection, in SQL Workbench/J, under Database Explorer, select the database from the marketing project of SageMaker unified studio. Choose a subscribed table. Select the Data tab to see the data in the table.

Cleanup

To ensure no additional charges are incurred after testing, be sure to delete the Amazon SageMaker Unified Studio domain. See Delete domains for instructions.

Conclusion

Amazon SageMaker Unified Studio continues to expand its offerings, providing you with more flexibility to access, analyze, and visualize your subscribed data. With support for the Athena JDBC driver, you can now use a wide range of popular BI and analytics tools, making data accessed through Amazon SageMaker Unified Studio more accessible than ever before. Whether you’re using Tableau, Power BI, or other familiar tools, the integration with Amazon SageMaker Unified Studio ensures that your data remains secure and accessible to authorized users.

The feature is supported in all AWS commercial Regions where Amazon SageMaker Unified Studio is currently available. Get started with our technical documentation.

About the authors

Navigating architectural choices for a lakehouse using Amazon SageMaker

2026-01-12 Lakshmi Nair

Post Syndicated from Lakshmi Nair original https://aws.amazon.com/blogs/big-data/navigating-architectural-choices-for-a-lakehouse-using-amazon-sagemaker/

Organizations today are using data more than ever to drive decision-making and innovation. Because they work with petabytes of information, they have traditionally gravitated towards two distinct paradigms—data lakes and data warehouses. While each paradigm excels at specific use cases, they often create unintended barriers between the data assets.

Data lakes are often built on object storage such as Amazon Simple Storage Service (Amazon S3), which provide flexibility by supporting diverse data formats and schema-on-read capabilities. This enables multi-engine access where various processing frameworks (such as Apache Spark, Trino, and Presto) can query the same data. On the other hand, data warehouses (such as Amazon Redshift) excel in areas such as ACID (atomicity, consistency, isolation and durability) compliance, performance optimization, and straightforward deployment, making them suitable for structured and complex queries. As data volumes grow and analytics needs become more complex, organizations seek to bridge these silos and use the strengths of both paradigms. This is where the concept of lakehouse architecture is applied, offering a unified approach to data management and analytics.

Over time, several distinct lakehouse approaches have emerged. In this post, we show you how to evaluate and choose the right lakehouse pattern for your needs.

The data lake centric lakehouse approach begins with the scalability, cost-effectiveness, and flexibility of a traditional data lake built on object storage. The goal is to add a layer of transactional capabilities and data management traditionally found in databases, primarily through open table formats (such as Apache Hudi, Delta Lake, or Apache Iceberg). While open table formats have made significant strides by introducing ACID guarantees for single-table operations in data lakes, implementing multi-table transactions with complex referential integrity constraints and joins remains challenging. The fundamental nature of querying petabytes of files on object storage, often through distributed query engines, can result in slow interactive queries at high concurrency when compared to a highly optimized, indexed, and materialized data warehouse. Open table formats introduce compaction and indexing, but the full suite of intelligent storage optimizations found in highly mature, proprietary data warehouses is still evolving in data lake-centric architecture.

The data warehouse centric lakehouse approach offers robust analytical capabilities but has significant interoperability challenges. Though data warehouses provide JAVA Database Connectivity (JDBC) and Open Database Connectivity (ODBC) drivers for external access, the underlying data remains in proprietary formats, making it difficult for external tools or services to directly access it without complex extract, transform, and load (ETL) or API layers. This can lead to data duplication and latency. A data warehouse architecture might support reading open table formats, but its ability to write to them or participate in their transactional layers can be limited. This restricts true interoperability and can create shadow data silos.

On AWS, you can build a modern, open lakehouse architecture to achieve unified access to both data warehouses and data lakes. By using this approach, you can build sophisticated analytics, machine learning (ML), and generative AI applications while maintaining a single source of truth for their data. You don’t have to choose between a data lake or data warehouse. You can use existing investments and preserve the strengths of both paradigms while eliminating their respective weaknesses. The lakehouse architecture on AWS embraces open table formats such as Apache Hudi, Delta Lake, and Apache Iceberg.

You can accelerate your lakehouse journey with the next generation of Amazon SageMaker, which delivers an integrated experience for analytics and AI with unified access to data. SageMaker is built on an open lakehouse architecture that is fully compatible with Apache Iceberg. By extending support for Apache Iceberg REST APIs, SageMaker significantly adds interoperability and accessibility across various Apache Iceberg-compatible query engines and tools. At the core of this architecture is a metadata management layer built on AWS Glue Data Catalog and AWS Lake Formation, which provide unified governance and centralized access control.

Foundations of the Amazon SageMaker lakehouse architecture

The lakehouse architecture of Amazon SageMaker has four main components that work together to create a unified data platform.

Flexible storage to adapt to the workload patterns and requirements
Technical catalog that serves as a single source of truth for all metadata
Integrated permission management with fine-grained access control across all data assets
Open access framework built on Apache Iceberg REST APIs for universal compatibility

Catalogs and permissions

When building an open lakehouse, the catalog—your central repository of metadata—is a critical component for data discovery and governance. There are two types of catalogs in the lakehouse architecture of Amazon SageMaker: managed catalogs and federated catalogs.

Managed catalog refers to when the metadata is managed by the lakehouse, and the data is stored in a general purpose S3 bucket.
Federated catalog refers to mounting or connecting to external or existing data sources so you can query data from data sources such as Amazon Redshift, Snowflake, and Amazon DynamoDB without explicitly moving the data. For more information, see Data connections in the lakehouse architecture of Amazon SageMaker.

You can use an AWS Glue crawler to automatically discover and register this metadata in Data Catalog. Data Catalog stores the schema and table metadata of your data assets, effectively turning files into logical tables. After your data is cataloged, the next challenge is controlling who can access it. While you could use complex S3 bucket policies for every folder, this approach is difficult to manage and scale. Lake Formation provides a centralized database-style permissions model on the Data Catalog, giving you the flexibility to grant or revoke fine-grained access at row, column, and cell levels for individual users or roles.

Open access with Apache Iceberg REST APIs

The lakehouse architecture described in the preceding section and shown in the following figure also uses the AWS Glue Iceberg REST catalog through the service endpoint, which provides OSS compatibility, enabling increased interoperability for managing Iceberg table metadata across Spark and other open source analytics engines. You can choose the appropriate API based on table format and use case requirements.

In this post, we explore various lakehouse architecture patterns, focusing on how to optimally use data lake and data warehouse to create robust, scalable, and performance-driven data solutions.

Bringing data into your lakehouse on AWS

When building a lakehouse architecture, you can choose from three distinct patterns to access and integrate your data, each offering unique advantages for different use cases.

Traditional ETL is the classic method of extracting data, transforming it and loading it into your lakehouse.

When to use it:

- You need complex transformations and require highly curated and optimized data sets for downstream applications for better performance
- You need to perform historical data migrations
- You need data quality enforcement and standardization at scale
- You need highly governed curated data in a lakehouse

Zero-ETL is a modern architectural pattern where data automatically and continuously replicates from a source system to lakehouse with minimal or no manual intervention or custom code. Behind the scenes, the pattern uses change data capture (CDC) to automatically stream all new inserts, updates, and deletes from the source to the target. This architectural pattern is effective when the source system maintains a high degree of data cleanliness and structure, minimizing the need for heavy pre-load transformations, or when data refinement and aggregation can occur at the target end within lakehouse. Zero-ETL replicates data with minimal delay, and the transformation logic is performed on the target end closer to where the insights are generated by shifting it to a more efficient, post-load phase.

When to use it:

- You need to reduce operational complexity and gain flexible control over data replication for both near real-time and batch use cases.
- You need limited customization. While zero-ETL implies minimal work, some light transformations might still be required on the replicated data.
- You need to minimize the need for specialized ETL expertise.
- You need to maintain data freshness without processing delays and reduce risk of data inconsistencies. Zero-ETL facilitates faster time-to-insight.

Data federation (no-movement approach) is a method that enables querying and combining data from multiple disparate sources without physically moving or copying it into a single centralized location. This query-in-place approach allows the query engine to connect directly to the external source systems, delegate and execute queries, and combine results on the fly for presentation to the user. The effectiveness of this architecture pattern depends on three key factors: network latency between systems, source system performance capabilities, and the query engine’s ability to push down predicates to optimize query execution. This no-movement approach can significantly reduce data duplication and storage costs while providing real-time access to source data.

When to use it:

- You need to query the source system directly to use operational analytics.
- You don’t want to duplicate data to save on storage space and associated costs within your Lakehouse.
- You’re willing to trade some query performance and governance for immediate data availability and one-time analysis of live data.
- You don’t need to frequently query the data.

Understanding the storage layer of your lakehouse on AWS

Now that you’ve seen different ways to get data into a lakehouse, the next question is where to store the data. As shown in the following figure, you can architect a modern open lakehouse on AWS by storing the data in a data lake (Amazon S3 or Amazon S3 Tables) or data warehouse (Redshift Managed Storage), so you can optimize for both flexibility and performance based on your specific workload requirements.

A modern lakehouse isn’t a single storage technology but a strategic combination of them. The decision of where and how to store your data impacts everything from the speed of your dashboards to the efficiency of your ML models. You must consider not only the initial cost of storage but also the long-term costs of data retrieval, the latency required by your users, and the governance necessary to maintain a single source of truth. In this section, we delve into architectural patterns for the data lake and the data warehouse and provide a clear framework for when to use each storage pattern. While they have historically been seen as competing architectures, the modern and open lakehouse approach uses both to create a single, powerful data platform.

General purpose S3

A general purpose S3 bucket in Amazon S3 is the standard, foundational bucket type used for storing objects. It provides flexibility so that you can store your data in its native format without a rigid upfront schema. Because of the ability of an S3 bucket to decouple storage from compute, you can store the data in a highly scalable location, while a variety of query engines can access and process it independently. This means that you can choose the right tool for the job without having to move or duplicate the data. You can store petabytes of data without ever having to provision or manage storage capacity, and its tiered storage classes provide significant cost savings by automatically moving less-frequently accessed data to more affordable storage.

The existing Data Catalog functions as a managed catalog. It’s identified by the AWS account number, which means there is no migration needed for existing Data Catalogs; they’re already available in the lakehouse and become the default catalog for the new data, as shown in the following figure.

A foundational data lake on general purpose S3 is highly efficient for append-only workloads. However, its file-based nature lacks the transactional guarantees of a traditional database. This is where you can use the support of open-source transactional table formats such as Apache Hudi, Delta Lake, and Apache Iceberg. With these table formats, you can implement multi-version concurrency control, allowing multiple readers and writers to operate simultaneously without conflicts. They provide snapshot isolation, so that readers see consistent views of data even during write operations. A typical medallion architecture pattern with Apache Iceberg is depicted in the following figure. When building a lakehouse on AWS with Apache Iceberg, customers can choose between two primary approaches for storing their data on Amazon S3: General purpose S3 buckets with self-managed Iceberg or using the fully managed S3 Tables. Each path has distinct advantages, and the right choice depends on your specific needs for control, performance, and operational overhead.

General purpose S3 with Self-managed Iceberg

Using general purpose S3 buckets with self-managed Iceberg is a traditional approach where you store both data and Iceberg metadata files in standard S3 buckets. With this option, you maintain full control but are responsible for managing the complete Iceberg table lifecycle, including essential maintenance tasks such as compaction and garbage collection.

When to use it:

Maximum control: This approach provides complete control over the entire data life cycle. You can fine-tune every aspect of table maintenance, such as defining your own compaction schedules and strategies, which can be crucial for specific high-performance workloads or to optimize costs.
Flexibility and customization: It is ideal for organizations with strong in-house data engineering expertise that need to integrate with a wider range of open-source tools and custom scripts. You can use Amazon EMR or Apache Spark to manage the table operations.
Lower upfront costs: You pay only for Amazon S3 storage, API requests, and the compute resources you use for maintenance. This can be more cost-effective for smaller or less-frequent workloads where continuous, automated optimization isn’t necessary.

Note: The query performance depends entirely on your optimization strategy. Without continuous, scheduled jobs for compaction, performance can degrade over time as data gets fragmented. You must monitor these jobs to ensure efficient querying.

S3 Tables

S3 Tables provides S3 storage that’s optimized for analytic workloads and provides Apache Iceberg compatibility to store tabular data at scale. You can integrate S3 table buckets and tables with Data Catalog and register the catalog as a Lake Formation data location from the Lake Formation console or using service APIs, as shown in the following figure. This catalog will be registered and mounted as a federated lakehouse catalog.

When to use it:

Simplified operations: S3 Tables automatically handles table maintenance tasks such as compaction, snapshot management and orphan file cleanup in the background. This automation eliminates the need to build and manage custom maintenance jobs, significantly reducing your operational overhead.
Automated optimization: S3 Tables provides built-in automatic optimizations that improve query performance. These optimizations include background processes such as file compaction to address the small files problem and data layout optimizations specific to tabular data. However, this automation trades flexibility for convenience. Because you can’t control the timing or method of compaction operations, workloads with specific performance requirements might experience varying query performance.
Focus on data usage: S3 Tables reduces the engineering overhead and shifts the focus to data consumption, data governance and value creation.
Simplified entry to open table formats: It’s suitable for teams who are new to the concept of Apache Iceberg but want to use transactional capabilities on data lake.
No external catalog: Suitable for smaller teams who don’t want to manage an external catalog.

Redshift managed storage

While the data lake serves as the central source of truth for all your data, it’s not the most suitable data store for every job. For the most demanding business intelligence and reporting workloads, the data lake’s open and flexible nature can introduce performance unpredictability. To help ensure the desired performance, consider transitioning a curated subset of your data from the data lake to a data warehouse for the following reasons:

High concurrency BI and reporting: When hundreds of business users are concurrently running complex queries on live dashboards, a data warehouse is specifically optimized to handle these workloads with predictable, sub-second query latency.
Predictable performance SLAs:– For critical business processes that require data to be delivered at a guaranteed speed, such as financial reporting or end-of-day sales analysis, a data warehouse provides consistent performance.
Complex SQL workloads: While data lakes are powerful, they can struggle with highly complex queries involving numerous joins and massive aggregations. A data warehouse is purpose-built to run these relational workloads efficiently.

The lakehouse architecture on AWS supports Redshift Managed Storage (RMS), a storage option provided by Amazon Redshift, a fully managed, petabyte-scale data warehouse service in the cloud. RMS storage supports the automatic table optimization offered in Amazon Redshift such as built-in query optimizations for data warehousing workloads, automated materialized views, and AI-driven optimizations and scaling for frequently running workloads.

Federated RMS catalog: Onboard existing Amazon Redshift data warehouses to lakehouse

Implementing a federated catalog with existing Amazon Redshift data warehouses creates a metadata-only integration that requires no data movement. This approach lets you extend your established Amazon Redshift investments into a modern open lakehouse framework while maintaining compatibility with existing workflows. Amazon Redshift uses a hierarchical data organization structure:

Cluster level: Starts with a namespace
Database level: Contains multiple databases
Schema level: Organizes tables within databases

When you register your existing Amazon Redshift provisioned or serverless namespaces as a federated catalog in Data Catalog, this hierarchy maps directly into the lakehouse metadata layer. The lakehouse implementation on AWS supports multiple catalogs using a dynamic hierarchy to organize and map the underlying storage metadata.

After you register a namespace, the federated catalog automatically mounts across all Amazon Redshift data warehouses in your AWS Region and account. During this process, Amazon Redshift internally creates external databases that correspond to data shares. This mechanism remains completely abstracted from end users. By using federated catalogs, you can create and use immediate visibility and accessibility across your data ecosystem. Permissions on the federated catalogs can be managed by Lake Formation for both same account and cross account access.

The real capability of federated catalogs emerges when accessing Amazon Redshift-managed storage from external AWS engines such as Amazon Athena, Amazon EMR, or open source Spark. Because Amazon Redshift uses proprietary block-based storage that only Amazon Redshift engines can read natively, AWS automatically provisions a service-managed Amazon Redshift Serverless instance in the background. This service-managed instance acts as a translation layer between external engines and Amazon Redshift managed storage. AWS establishes automatic data shares between your registered federated catalog and the service-managed Amazon Redshift Serverless instance to enable secure, efficient data access. AWS also creates a service-managed Amazon S3 bucket in the background for data transfer.

When an external engine such as Athena submits queries against Amazon Redshift federated catalog, Lake Formation handles the credential vending by providing the temporary credentials to the requesting service. The query executes through the service-managed Amazon Redshift Serverless, which accesses data through automatically established data shares, processes results, offloads them to a service-managed Amazon S3 staging area, and then returns results to the original requesting engine.

To track the compute cost of the federated catalog of existing Amazon Redshift warehouse, use the following tag.

aws:redshift-serverless:LakehouseManagedWorkgroup value: "True"

To activate the AWS generated cost allocation tags for billing insight, follow the activation instructions. You can also view the computational cost of the resources in AWS Billing.

When to use it:

Existing Amazon Redshift investments: Federated catalogs are designed for organizations with existing Amazon Redshift deployments who want to use their data across multiple services without migration.
Cross-service data sharing:– Implement so teams can share existing data in an Amazon Redshift data warehouse across different warehouses and centralize their permissions.
Enterprise integration requirements: This approach is suitable for organizations that need to integrate with established data governance. It also maintains compatibility with current workflows while adding lakehouse capabilities.
Infrastructure control and pricing:– You can retain full control over compute capacity for their existing warehouses for predictable workloads. You can optimize compute capacity, choose between on-demand and reserved capacity pricing, and fine-tune performance parameters. This provides cost predictability and performance control for consistent workloads.

When implementing lakehouse architecture with multiple catalog types, selecting the appropriate query engine is crucial for both performance and cost optimization. This post focuses on the storage foundation of lakehouse, however for critical workloads involving extensive Amazon Redshift data operations, consider executing queries within Amazon Redshift or using Spark when possible. Complex joins spanning multiple Amazon Redshift tables through external engines might result in higher compute costs if the engines don’t support full predicate push-down.

Other use-cases

Build a multi-warehouse architecture

Amazon Redshift supports data sharing, which you can use to share live data between source and target Amazon Redshift clusters. By using data sharing, you can share live data without creating copies or moving data, enabling uses cases such as workload isolation (hub and spoke architecture) and cross group collaboration (data mesh architecture). Without a lakehouse architecture, you must create an explicit data share between source and target Amazon Redshift clusters. While managing these data shares in small deployments is relatively straightforward, it becomes complex in data mesh architectures.

The lakehouse architecture addresses this challenge so customers can publish their existing Amazon Redshift warehouses as federated catalogs. These federated catalogs are automatically mounted and made available as external databases in other consumer Amazon Redshift warehouses within the same account and Region. By using this approach, you can maintain a single copy of data and use multiple data warehouses to query it, eliminating the need to create and manage multiple data shares and scale with workload isolation. The permission management becomes centralized through Lake Formation, streamlining governance across the entire multi-warehouse environment.

Near real-time analytics on petabytes of transactional data with no pipeline management:

Zero-ETL integrations seamlessly replicate transactional data from OLTP data sources to Amazon Redshift, general purpose S3 (with self-managed Iceberg) or S3 Tables. This approach eliminates the need to maintain complex ETL pipelines, reducing the number of moving parts in your data architecture and potential points of failure. Business users can analyze fresh operational data immediately rather than working with stale data from the last ETL run.

See Aurora zero-ETL integrations for a list of OLTP data sources that can be replicated to an existing Amazon Redshift warehouse.

See Zero-ETL integrations for information about other supported data sources that can be replicated to an existing Amazon Redshift warehouse, general purpose S3 with self-managed Iceberg, and S3 Tables.

Conclusion

A lakehouse architecture isn’t about choosing between a data lake and a data warehouse. Instead, it’s an approach to interoperability where both frameworks coexist and serve different purposes within a unified data architecture. By understanding fundamental storage patterns, implementing effective catalog strategies, and using native storage capabilities, you can build scalable, high-performance data architectures that support both your current analytics needs and future innovation. For more information, see The lakehouse architecture of Amazon SageMaker.

About the authors

Use Amazon SageMaker custom tags for project resource governance and cost tracking

2026-01-09 David Victoria

Post Syndicated from David Victoria original https://aws.amazon.com/blogs/big-data/use-amazon-sagemaker-custom-tags-for-project-resource-governance-and-cost-tracking/

Amazon SageMaker announced a new feature that you can use to add custom tags to resources created through an Amazon SageMaker Unified Studio project. This helps you enforce tagging standards that conform to your organization’s service control policies (SCPs) and helps enable cost tracking reporting practices on resources created across the organization.

As a SageMaker administrator, you can configure a project profile with tag configurations that will be pushed down to projects that currently use or will use that project profile. The project profile is set up to pass either required key and value tag pairings or pass the key of the tag with a default value that can be modified during project creation. All tags passed to the project will result in the resources created by that project being tagged. This provides you with a governance mechanism that enforces that project resources have the expected tags across all projects of the domain.

The first release of custom tags for project resources is supported through an application programming interface (API), through Amazon DataZone SDKs. In this post, we look at use cases for custom tags and how to use the AWS Command Line Interface (AWS CLI) to add tags to project resources.

What we hear from customers

As customers continue to build and collaborate using AWS tools for model development, generative AI, data processing, and SQL analytics, they see the need to bring control and visibility into the resources being created. To support connectivity to these AWS tools from SageMaker Unified Studio projects, many different types of resources across AWS services need to be created. These resources are created through AWS CloudFormation stacks (through project environment deployment) by the Amazon SageMaker service. From customers we hear the following use cases:

Customers need to enforce that tagging practices conform to company policies through the use of AWS controls, such as SCPs, for resource creation. These controls block the creation of resources unless specific tags are placed on the resource.
Customers can also start with policies to enforce that the correct tags are placed when resources are created with the additional goal of standardizing on resource reporting. By placing identifiable information on resources when created, they enforce consistency and completeness when performing cost attribution reporting and observability.

Customer Swiss Life uses SageMaker as a single solution for cataloging, discovery, sharing, and governance of their enterprise data across business domains. They require all resources have a set of mandatory tags for their finance group to bill organizations across their company for the AWS resources created.

“The launch of project resource tags for Amazon SageMaker allows us to bring visibility to the costs incurred across our accounts. With this capability we are able to meet the resource tagging guidelines of our company and have confidence in attributing costs across our multi-account setup for the resources created by Amazon SageMaker projects.”

– Tim Kopacz, Software Developer at Swiss Life

Prerequisites

To get started with custom tags, you must have the following resources:

A SageMaker Unified Studio domain.
An AWS Identity and Access Management (IAM) entity with privileges to make AWS CLI calls to the domain.
An IAM entity authorized to make changes to the domain IAM provisioning role. If SageMaker created this for you, it will be called AmazonSageMakerProvisioning-<accountId>. The provisioning role provisions and manages resources defined in the selected blueprints in your account.

How to set up project resource tags

The following steps outline how you can configure custom tags for your SageMaker Unified Studio project resources:

(Optional) Update the SageMaker provisioning role to permit specific tag keys.
Create a new project profile with project resource tags configured.
Create a new project with project resource tags.
Update an existing project with project resource tags.
Validate that the resources are tagged.

(Optional) Update a SageMaker provisioning role to permit tag key values

The AmazonSageMakerProvisioning-<accountId> role has an AWS managed policy with condition aws:TagKeys allowing tags to be created by this role only if the tag key begins with AmazonDataZone. For this example, we will change the tag key to begin with different strings. Skip to Create a new project profile with project resource tags configured if you don’t need tag keys to have a different structure (such as begins with, contains, and so on)

Open the AWS Management Console and go to IAM.
In the navigation pane, choose Roles.
In the list, choose AmazonSageMakerProvisioning-<accountId>.
Choose the Permissions tab.
Choose Add permissions, and then choose Create inline policy.
Under Policy editor, select JSON.

Enter the following policy. Add the strings under the condition aws:TagKeys. In this example, tag keys beginning with ACME or tag keys with the exact match of CostCenter will be created by the role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CustomTagsUnTagPermissions",
            "Effect": "Allow",
            "Action": [
                "codecommit:UntagResource",
                "iam:UntagRole",
                "logs:UntagResource",
                "athena:UntagResource",
                "redshift-serverless:UntagResource",
                "scheduler:UntagResource",
                "bedrock:UntagResource",
                "neptune-graph:UntagResource",
                "quicksight:UntagResource",
                "glue:UntagResource",
                "airflow:UntagResource",
                "secretsmanager:UntagResource",
                "lambda:UntagResource",
                "emr-serverless:UntagResource",
                "elasticmapreduce:RemoveTags",
                "sagemaker:DeleteTags",
                "ec2:DeleteTags"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceAccount": "${aws:PrincipalAccount}"
                },
                "ForAllValues:StringLike": {
                    "aws:TagKeys": [
                        "AmazonDataZone*",
                        "ACME*",
                        "CostCenter"
                    ]
                },
                "Null": {
                    "aws:ResourceTag/AmazonDataZoneProject": "false"
                }
            }
        },
        {
            "Sid": "CustomTagsTaggingPermissions",
            "Effect": "Allow",
            "Action": [
                "cloudformation:TagResource",
                "codecommit:TagResource",
                "iam:TagRole",
                "glue:TagResource",
                "athena:TagResource",
                "lambda:TagResource",
                "redshift-serverless:TagResource",
                "logs:TagResource",
                "secretsmanager:TagResource",
                "sagemaker:AddTags",
                "emr-serverless:TagResource",
                "neptune-graph:TagResource",
                "bedrock:TagResource",
                "elasticmapreduce:AddTags",
                "airflow:TagResource",
                "scheduler:TagResource",
                "quicksight:TagResource",
                "emr-containers:TagResource",
                "logs:CreateLogGroup",
                "athena:CreateWorkGroup",
                "scheduler:CreateScheduleGroup",
                "cloudformation:CreateStack",
                "ec2:*"
            ],
            "Resource": "*",
            "Condition": {
                "ForAnyValue:StringLike": {
                    "aws:TagKeys": [
                        "AmazonDataZone*",
                        "ACME*",
                        "CostCenter"
                    ]
                },
                "StringEquals": {
                    "aws:ResourceAccount": "${aws:PrincipalAccount}"
                }
            }
        }
    ]
}

It’s possible to scope down the specific AWS service tag and un-tag permissions based on which blueprints or capabilities are being used.

Create a new project profile with project resource tags configured

Use the following steps to create a new SQL Analytics project profile with custom tags. The example uses AWS CLI commands.

Open the AWS CloudShell console.

Create a project profile using the following CLI command.

The project-resource-tags parameter consists of key (tag key), value (tag value), and isValueEditable (boolean indicating if the tag value can be modified during project creation or update).
The allow-custom-project-resource-tags parameter set to true permits the project creator to create additional key-value pairs. The key needs to conform to the inline policy of the AmazonSageMakerProvisioning-<accountId> role.
The project-resource-tags-description parameter is a description field for project resource tags. The max character limit is 2,048. The description needs to be passed in every time create-project-profile or update-project-profile is called.

aws datazone create-project-profile \
  --name "SQL Analytics with Project Resource Tags" \
  --description "Analyze your data in SageMaker Lakehouse using SQL" \
  --domain-identifier "$DOMAIN_ID" \
  --region "$REGION" \
  --status ENABLED \
  --project-resource-tags '[
    {
        "key": "ACME-Application",
        "value": "SageMaker",
        "isValueEditable": false
    },
    {
        "key": "CostCenter",
        "value": "123",
        "isValueEditable": true
    }
  ]' \
  --allow-custom-project-resource-tags \
  --environment-configurations '[
    {
        "name": "Tooling",
        "description": "Configuration for the Tooling Environment",
        "environmentBlueprintId": "",
        "deploymentMode": "ON_CREATE",
        "deploymentOrder": 0,
        "awsAccount": {
        "awsAccountId": "$ACCOUNT"
    },
    "awsRegion": {
        "regionName": "$REGION"
    },
        "configurationParameters": {
            "parameterOverrides": [
                {
                    "name": "enableSpaces",
                    "value": "false",
                    "isEditable": false
                },
                {
                    "name": "maxEbsVolumeSize",
                    "isEditable": false
                },
                {
                    "name": "idleTimeoutInMinutes",
                    "isEditable": false
                },
                {
                    "name": "lifecycleManagement",
                    "isEditable": false
                },
                {
                    "name": "enableNetworkIsolation",
                    "isEditable": false
                }
            ]
        }
    },
    {
        "name": "Lakehouse Database",
        "description": "Creates databases in Amazon SageMaker Lakehouse for storing tables in S3 and Amazon Athena resources for your SQL workloads",
        "environmentBlueprintId": "",
        "deploymentMode": "ON_CREATE",
        "deploymentOrder": 1,
        "awsAccount": {
            "awsAccountId": "$ACCOUNT"
        },
        "awsRegion": {
        "regionName": "$REGION"
        },
        "configurationParameters": {
            "parameterOverrides": [
                {
                    "name": "glueDbName",
                    "value": "glue_db",
                    "isEditable": true
                }
            ]
        }
    },
    {
        "name": "OnDemand RedshiftServerless",
        "description": "Enables you to create an additional Amazon Redshift Serverless workgroup for your SQL workloads",
        "environmentBlueprintId": "",
        "deploymentMode": "ON_DEMAND",
        "awsAccount": {
        "awsAccountId": "$ACCOUNT"
        },
        "awsRegion": {
            "regionName": "$REGION"
        },
        "configurationParameters": {
            "parameterOverrides": [
                {
                    "name": "redshiftDbName",
                    "value": "dev",
                    "isEditable": true
                    },
                    {
                    "name": "redshiftMaxCapacity",
                    "value": "512",
                    "isEditable": true
                    },
                    {
                    "name": "redshiftWorkgroupName",
                    "value": "redshift-serverless-workgroup",
                    "isEditable": true
                    },
                    {
                    "name": "redshiftBaseCapacity",
                    "value": "128",
                    "isEditable": true
                    },
                    {
                    "name": "connectionName",
                    "value": "redshift.serverless",
                    "isEditable": true
                    },
                    {
                    "name": "connectToRMSCatalog",
                    "value": "false",
                    "isEditable": false
                    }
                ]
            }
        },
        {
            "name": "OnDemand Catalog for Redshift Managed Storage",
            "description": "Enables you to create additional catalogs in Amazon SageMaker Lakehouse for storing data in Redshift Managed Storage",
            "environmentBlueprintId": "",
            "deploymentMode": "ON_DEMAND",
            "awsAccount": {
            "awsAccountId": "$ACCOUNT"
            },
            "awsRegion": {
                "regionName": "$REGION"
            },
            "configurationParameters": {
                "parameterOverrides": [
                    {
                        "name": "catalogName",
                        "isEditable": true
                    },
                    {
                        "name": "catalogDescription",
                        "value": "RMS catalog",
                        "isEditable": true
                    }
                ]
            }
        }
  ]'

This project profile will have the tag ACME-Application = SageMaker placed on all projects associated to the project profile and cannot be modified by the project creator. The tag CostCenter = 123 can have the value modified by the project creator because the isValueEditable property is set to true.

Grant permissions for users to use the project profile during project creation. In the Authorization section of the project profile set either Selected users or groups or Allow all users and groups.

The use of the allow-custom-project-resource-tags parameter means the project creator can add their own tags (key-value pair). The key must conform to the condition check in the policy of the provisioning role (AmazonSageMakerProvisioning-<accountId>). If the allow-custom-project-resource-tagsparameter is changed to false after a project created tags, tags created by the project will be removed during the next project update.

Updates to the project profile

Updates to project resource tags are possible through the update-project-profile command. The command will replace all values in the project-resource-tags section so be sure to include the exhaustive set of tags. Updates to the project profile are reflected in projects after running the update-project command or when a new project is created using the project profile. The following example adds a new tag, ACME-BusinessUnit = Retail.

There are three ways to work with the project-resource-tags parameter when updating the project profile.

Passing a non-empty list of project resource tags will replace the tags currently configured on the project profile.
Passing an empty list of project resource tags will clear out all previously configured tags:
- --project-resource-tags '[]'
Not including the project resource tag parameter will keep previously configured tags as-is.

aws datazone update-project-profile \
  --domain-identifier "$DOMAIN_ID" \
  --identifier "$PROJECT_PROFILE_ID" \
  --region "$REGION" \
  --project-resource-tags '[
    {
        "key": "ACME-Application",
        "value": "SageMaker",
        "isValueEditable": false
    },
    {
        "key": "CostCenter",
        "value": "123",
        "isValueEditable": true
    },
    {
        "key": "ACME-BusinessUnit",
        "value": "Retail",
        "isValueEditable": false
    }
  ]'

Create a new project with project resource tags

The following steps walk you through creating a new project that inherits tags from the project profile and lets the project creator modify one of the tag values.

Create a project using the following example CLI command.

Modify the CostCenter tag value using the --resource-tags parameter. Tags configured on the project profile where the isValueEditable attribute is false will be pushed to the project automatically.

aws datazone create-project \
  --domain-identifier "$DOMAIN_ID" \
  --region "$REGION" \
  --name "$PROJECT_NAME" \
  --description "New project with tags" \
  --project-profile-id "$PROJECT_PROFILE_ID" \
  --resource-tags '{
        "CostCenter": "456"
    }'

Update existing project with project resource tags

For existing projects associated to the project profile, you must update the project for the new tags to be applied.

Update the project using the following example CLI command.

In this scenario, an editable value needs to be updated and a new tag added. Tag CostCenter will have its default value overwritten as “789” and the new ACME-Department = Finance tag will be added.

aws datazone update-project \
  --domain-identifier "$DOMAIN_ID" \
  --identifier "$PROJECT_ID" \
  --project-profile-version "latest" \
  --region "$REGION" \
  --resource-tags '{
        "CostCenter": "789",
        "ACME-Department": "Finance"
    }'

Project level tags (those not configured from the project profile) need to be passed during project update to be preserved. For tags with isValueEditable = true configured from the project profile, any override previously set needs to be applied or the value will revert to the default from the project profile.

Validating resources are tagged

Validate that tags are placed correctly. An example resource that is created by the project is the project IAM role. Viewing the tags for this role should show the tags configured from the project profile.

Open SageMaker Unified Studio to get the project role from the Project details section of the project. The role name begins with datazone_usr_role_.
Open the IAM console.
In the navigation pane, choose Roles.
Search for the project IAM role.
Select the Tags tab.

Conclusion

In this post, we discussed tagging related use cases from customers and walked through getting started with custom tags in Amazon SageMaker to place tags on the resources created by the project. By giving administrators a way to configure project profiles with standardized tag configurations, you can now help ensure consistent tagging practices across all SageMaker Unified Studio projects while maintaining compliance with SCPs. This feature addresses two critical customer needs: enforcing organizational tagging standards through automated governance mechanisms and enabling accurate cost attribution reporting across multi-service deployments.

To learn more, visit Amazon SageMaker, then get started with Project resource tags.

About the authors

AWS analytics at re:Invent 2025: Unifying Data, AI, and governance at scale

2026-01-08 Larry Weber

Post Syndicated from Larry Weber original https://aws.amazon.com/blogs/big-data/aws-analytics-at-reinvent-2025-unifying-data-ai-and-governance-at-scale/

re:Invent 2025 showcased the bold Amazon Web Services (AWS) vision for the future of analytics, one where data warehouses, data lakes, and AI development converge into a seamless, open, intelligent platform, with Apache Iceberg compatibility at its core. Across over 18 major announcements spanning three weeks, AWS demonstrated how organizations can break down data silos, accelerate insights with AI, and maintain robust governance without sacrificing agility.

Amazon SageMaker: Your data platform, simplified

AWS introduced a faster, simpler approach to data platform onboarding for Amazon SageMaker Unified Studio. The new one-click onboarding experience eliminates weeks of setup, so teams can start working with existing datasets in minutes using their current AWS Identity and Access Management (IAM) roles and permissions. Accessible directly from Amazon SageMaker, Amazon Athena, Amazon Redshift, and Amazon S3 Tables consoles, this streamlined experience automatically creates SageMaker Unified Studio projects with existing data permissions intact. At its core is a powerful new serverless notebook that reimagines how data professionals work. This single interface combines SQL queries, Python code, Apache Spark processing, and natural language prompts, backed by Amazon Athena for Apache Spark to scale from interactive exploration to petabyte-scale jobs. Data engineers, analysts, and data scientists no longer need to context-switch between different tools based on workload—they can explore data with SQL, build models with Python, and use AI assistance, all in one place.

The introduction of Amazon SageMaker Data Agent in the new SageMaker notebooks marks a pivotal moment in AI-assisted development for data builders. This built-in agent doesn’t only generate code, it understands your data context, catalog information, and business metadata to create intelligent execution plans from natural language descriptions. When you describe an objective, the agent breaks down complex analytics and machine learning (ML) tasks into manageable steps, generates the required SQL and Python code, and maintains awareness of your notebook environment throughout the entire process. This capability transforms hours of manual coding into minutes of guided development, which means teams can focus on gleaning insights rather than repetitive boilerplate.

Embracing open data with Apache Iceberg

One significant theme across this year’s launches was the widespread adoption of Apache Iceberg across AWS analytics, transforming how organizations manage petabyte-scale data lakes. Catalog federation to remote Iceberg catalogs through the AWS Glue Data Catalog addresses a critical challenge in modern data architectures. You can now query remote Iceberg tables, stored in Amazon Simple Storage Service (Amazon S3) and catalogued in remote Iceberg catalogs, using preferred AWS analytics services such as Amazon Redshift, Amazon EMR, Amazon Athena, AWS Glue, and Amazon SageMaker, without moving or copying tables. Metadata synchronizes in real time, providing query results that reflect the current state. Catalog federation supports both coarse-grained access control and fine-grained access permissions through AWS Lake Formation enabling cross-account sharing and trusted identity propagation while maintaining consistent security across federated catalogs.

Amazon Redshift now writes directly to Apache Iceberg tables, enabling true open lakehouse architectures where analytics seamlessly span data warehouses and lakes. Apache Spark on Amazon EMR 7.12, AWS Glue, Amazon SageMaker notebooks, Amazon S3 Tables, and the AWS Glue Data Catalog now support Iceberg V3’s capabilities, including deletion vectors that mark deleted rows without expensive file rewrites, dramatically reducing pipeline costs and accelerating data modifications and row lineage. V3 automatically tracks every record’s history, creating audit trails essential for compliance and has table-level encryption that helps organizations meet stringent privacy regulations. These innovations mean faster writes, lower storage costs, comprehensive audit trails, and efficient incremental processing across your data architecture.

Governance that scales with your organization

Data governance received substantial attention at re:Invent with major enhancements to Amazon SageMaker Catalog. Organizations can now curate data at the column level with custom metadata forms and rich text descriptions, indexed in real time for immediate discoverability. New metadata enforcement rules require data producers to classify assets with approved business vocabulary before publication, providing consistency across the enterprise. The catalog uses Amazon Bedrock large language models (LLMs) to automatically suggest relevant business glossary terms by analyzing table metadata and schema information, bridging the gap between technical schemas and business language. Perhaps most importantly, SageMaker Catalog now exports its entire asset metadata as queryable Apache Iceberg tables through Amazon S3 Tables. This way, teams can analyze catalog inventory with standard SQL to answer questions like “which assets lack business descriptions?” or “how many confidential datasets were registered last month?” without building custom ETL infrastructure.

As organizations adopt multi-warehouse architectures to scale and isolate workloads, the new Amazon Redshift federated permissions capability eliminates governance complexity. Define data permissions one time from a Amazon Redshift warehouse, and they automatically enforce them across the warehouses in your account. Row-level, column-level, and masking controls apply consistently regardless of which warehouse queries originate from, and new warehouses automatically inherit permission policies. This horizontal scalability means organizations can add warehouses without increasing governance overhead, and analysts immediately see the databases from registered warehouses.

Accelerating AI innovation with Amazon OpenSearch Service

Amazon OpenSearch Service introduced powerful new capabilities to simplify and accelerate AI application development. With support for OpenSearch 3.3, agentic search enables precise results using natural language inputs without the need for complex queries, making it easier to build intelligent AI agents. The new Apache Calcite-powered PPL engine delivers query optimization and an extensive library of commands for more efficient data processing.

As seen in Matt Garman’s keynote, building large-scale vector databases is now dramatically faster with GPU acceleration and auto-optimization. Previously, creating large-scale vector indexes required days of building time and weeks of manual tuning by experts, which slowed innovation and prevented cost-performance optimizations. The new serverless auto-optimize jobs automatically evaluate index configurations—including k-nearest neighbors (k-NN) algorithms, quantization, and engine settings—based on your specified search latency and recall requirements. Combined with GPU acceleration, you can build optimized indexes up to ten times faster at 25% of the indexing cost, with serverless GPUs that activate dynamically and bill only when providing speed boosts. These advancements simplify scaling AI applications such as semantic search, recommendation engines, and agentic systems, so teams can innovate faster by dramatically reducing the time and effort needed to build large-scale, optimized vector databases.

Performance and cost optimization

Also announced in the keynote, Amazon EMR Serverless now eliminates local storage provisioning for Apache Spark workloads, introducing serverless storage that reduces data processing costs by up to 20% while preventing job failures from disk capacity constraints. The fully managed, auto scaling storage encrypts data in transit and at rest with job-level isolation, allowing Spark to release workers immediately when idle rather than keeping them active to preserve temporary data. Additionally, AWS Glue introduced materialized views based on Apache Iceberg, storing precomputed query results that automatically refresh as source data changes. Spark engines across Amazon Athena, Amazon EMR, and AWS Glue intelligently rewrite queries to use these views, accelerating performance by up to eight times while reducing compute costs. The service handles refresh schedules, change detection, incremental updates, and infrastructure management automatically.

The new Apache Spark upgrade agent for Amazon EMR transforms version upgrades from months-long projects into week-long initiatives. Using conversational interfaces, engineers express upgrade requirements in natural language while the agent automatically identifies API changes and behavioral modifications across PySpark and Scala applications. Engineers review and approve suggested changes before implementation, maintaining full control while the agent validates functional correctness through data quality checks. Currently supporting upgrades from Spark 2.4 to 3.5, this capability is available through SageMaker Unified Studio, Kiro CLI, or an integrated development environment (IDE) with Model Context Protocol compatibility.

For workflow optimization, AWS introduced a new Serverless deployment option for Amazon Managed Workflows for Apache Airflow (Amazon MWAA), which eliminates the operational overhead of managing Apache Airflow environments while optimizing costs through serverless scaling. This new offering addresses key challenges of operational scalability, cost optimization, and access management that data engineers and DevOps teams face when orchestrating workflows. With Amazon MWAA Serverless, data engineers can focus on defining their workflow logic rather than monitoring for provisioned capacity. They can now submit their Airflow workflows for execution on a schedule or on demand, paying only for the actual compute time used during each task’s execution.

Looking forward

These launches collectively represent more than incremental improvements. They signal a fundamental shift in how organizations are approaching analytics. By unifying data warehousing, data lakes, and ML under a common framework built on Apache Iceberg, simplifying access through intelligent interfaces powered by AI, and maintaining robust governance that scales effortlessly, AWS is giving organizations the tools to focus on insights rather than infrastructure. The emphasis on automation, from AI-assisted development to self-managing materialized views and serverless storage, reduces operational overhead while improving performance and cost efficiency. As data volumes continue to grow and AI becomes increasingly central to business operations, these capabilities position AWS customers to accelerate their data-driven initiatives with unprecedented simplicity and power. To view the Re:Invent 2025 Innovation Talk on analytics, visit Harnessing analytics for humans and AI on YouTube.

About the authors

Unifying governance and metadata across Amazon SageMaker Unified Studio and Atlan

2025-12-22 Karan Singh Thakur, Satabrata Paul

Post Syndicated from Karan Singh Thakur, Satabrata Paul original https://aws.amazon.com/blogs/big-data/unifying-governance-and-metadata-across-amazon-sagemaker-unified-studio-and-atlan/

This post was cowritten with Satabrata Paul and Karan Singh Thakur from Atlan

In this post, we show you how to unify governance and metadata across Amazon SageMaker Unified Studio and Atlan through a comprehensive bidirectional integration. You’ll learn how to deploy the necessary Amazon Web Services (AWS) infrastructure, configure secure connections, and set up automated synchronization to maintain consistent metadata across both platforms.

As organizations scale their data and AI programs, teams often work across distributed tools such as governance solutions for business users and analytics or machine learning (ML) environments for technical teams. Without tight integration between these systems, metadata becomes fragmented. A single asset can appear under different names, documentation might drift out of sync, and governance signals can become inconsistent across systems.

To address these challenges, Atlan, a modern data workspace that makes collaboration among diverse users like business, analysts, and engineers easier, increasing efficiency and agility in data projects, and AWS have built a bidirectional integration between Atlan and Amazon SageMaker Unified Studio. This integration creates a continuous connection between both environments so every team within the enterprise can work with a single, trusted, and synchronized view of metadata for their data and AI assets. By bridging the gap between diverse users collaborating in Atlan and technical teams working within Amazon SageMaker Unified Studio for analytics and ML, this integration maintains consistency across both platforms without requiring teams to switch contexts or manually reconcile metadata differences.

Why unified metadata governance matters

Enterprises today operate in hybrid environments. Business users rely on Atlan as an active metadata solution to manage, govern, and collaborate on data assets across the modern data stack. Atlan helps teams find, understand, and trust their data so they can use it effectively to drive business outcomes.

Organizations also use Amazon SageMaker Catalog to simplify the discovery, governance, and collaboration for both business and technical data across structured and unstructured sources. Teams can use the catalog to organize data products, capture context, and apply governance policies consistently within Amazon SageMaker Unified Studio.

This new integration synchronizes metadata between SageMaker Catalog and Atlan, maintaining consistency and keeping content current across both environments. With a unified view, every team within the enterprise can work confidently with a single, trusted representation of their data and AI assets.

Solution overview

The solution follows a phased rollout strategy to provide you with immediate value while progressively expanding toward comprehensive data and AI governance capabilities. The current phase focuses on establishing secure, scalable, and reliable metadata synchronization between Atlan and Amazon SageMaker Unified Studio.

The Phase 1 integration between Amazon SageMaker Catalog and Atlan enables both on-demand and scheduled bidirectional metadata synchronization across the two solutions. It uses the standard APIs of Amazon SageMaker Unified Studio and Atlan to create a scalable and configurable mechanism for metadata exchange. Key capabilities include:

Secure connection using IAM roles – The integration is established through a controlled AWS Identity and Access Management (IAM) based handshake. A predefined AWS CloudFormation template automatically provisions the IAM role and policies required to enable a secure, least-privilege connection between Amazon SageMaker Catalog and the Atlan application.
On-demand and scheduled synchronization – The integration supports both manual and automated metadata synchronization. API-driven workflows manage the exchange of glossary terms, asset descriptions, and classifications in both directions, keeping metadata consistent across systems.

After you’ve implemented Phase 1, you can perform bidirectional synchronization of glossary terms and descriptions between Amazon SageMaker Unified Studio and Atlan. This keeps your terminology consistent across both platforms, and your teams can maintain a single source of truth for business definitions. The integration also preserves your glossary structures, including parent-child relationships, so your carefully organized taxonomy remains intact during the sync process. Additionally, glossary terms are automatically associated with related data assets, saving you the manual effort of linking terms to the appropriate datasets and reducing the risk of inconsistencies.

Beyond glossary management, Phase 1 enables comprehensive ingestion of assets and metadata from Amazon SageMaker Unified Studio into Atlan. This includes your projects, both published and subscribed assets, domains and data products, glossaries and terms, metadata forms, and column descriptions. By bringing this information into Atlan, you create a unified view of your data landscape that makes it easier for data consumers to discover, understand, and trust the data they’re working with.

Prerequisites

To follow along with this integration setup, you must have the following resources already configured in your environment:

An Atlan tenant
A Node group IAM role
An Amazon SageMaker Unified Studio domain.
At least one Amazon SageMaker Unified Studio project with assets created and glossary terms defined.
Atlan API Token. You can generate this by navigating to API access under the Atlan’s Admin center.
Atlan top-level glossary. You can create this glossary container on Atlan to ingest SageMaker Unified Studio glossaries and terms.

The next section offers a step-by-step walkthrough of the integration, from initial setup to full operation. It demonstrates how you can establish the trust handshake between Amazon SageMaker Unified Studio and Atlan and how bidirectional synchronization functions in practice.

Setup on AWS

To begin the integration, you need Atlan’s Account Node Instance IAM role. This role allows the Atlan SageMaker Unified Studio application to securely assume the IAM role that you will create in your AWS account using an AWS CloudFormation template. The trust relationship between these two roles authorizes Atlan to publish metadata to Amazon SageMaker Catalog and to perform reverse synchronization from AWS back into Atlan.

The IAM policy follows the principle of least privilege, granting Atlan access only to the resources necessary for cataloging and governance. This approach maintains accurate metadata synchronization while preserving your existing cloud security and compliance controls.

Follow AWS best practices when configuring trust relationships. These cross-account access mechanisms require careful management and monitoring, particularly during security incidents. For comprehensive guidance on securing IAM roles and trust policies, refer to the Security best practices in IAM and Require workloads to use temporary credentials with IAM roles to access AWS.

Contact your Atlan administrator to obtain the Amazon Resource Name (ARN) of the Atlan Account Node Instance IAM role. You will need this value when configuring the CloudFormation stack in AWS.

The next step is to create an AWS IAM role using the provided CloudFormation template. This role establishes the trust relationship between your Amazon SageMaker Unified Studio environment and your Atlan tenant. Follow these steps:

Access the CloudFormation template. The CloudFormation template is currently available as a YAML file.
On the AWS Management Console, navigate to CloudFormation and choose Create stack, then choose With new resources (standard), as shown in the following screenshot.
Choose the provided CloudFormation template and choose Next.
Enter a name for the stack and complete the required parameters, as shown in the following screenshot:
1. AtlanNodeInstanceRoleArn – The ARN of the Atlan node instance role.
2. SMUSDomainId – The unique identifier for the SageMaker Unified Studio domain.
3. SMUSProjectsToSync – The project IDs where SageMaker Unified Studio and Atlan synchronization will be enabled. You can choose to either add the project IDs and keep updating this stack every time a Project is added or add the created IAM role to each project as owner.
Select the acknowledgement checkbox and choose Next, as shown in the following screenshot.
Choose Submit to start the stack deployment. When the process is complete, the stack status will update to CREATE_COMPLETE.
Note the IAM role ARN
After the CloudFormation stack has been deployed and the IAM role has been created, copy the IAM Role ARN from the CloudFormation output. You will need this value during the configuration process on the Atlan side to establish the secure connection between your Amazon SageMaker Unified Studio environment and your Atlan tenant.

Setup on Atlan

Now that you’ve deployed the necessary AWS resources, you’ll configure Atlan to establish the connection with Amazon SageMaker Unified Studio. This involves setting up the API token, configuring the IAM role, and creating the glossary container that will receive your synchronized metadata. Follow these steps:

Sign in to your Atlan tenant, as shown in the following screenshot.
On the New dropdown menu, choose New workflow.
On the Marketplace tab, search for and select the AWS SageMaker Unified Studio app, as shown in the following screenshot.
Enter credential details. Use the IAM role or user created by the CloudFormation template before, enter an API token, and choose your AWS Region, as shown in the following screenshot.
Enter connection details. In Connection name, enter a name. Under Connection Admins, choose the plus icon to add members (other users) to the connectors as admins. Assigning admin permissions to the connection allows these users to:
1. View and edit the assets in the connection.
2. Edit connection preferences.
3. Edit persona-based policies for the connection.
Choose metadata filters and preflight checks, as shown in the following screenshot:
- In the Select Glossary to enrich dropdown menu, choose the glossary container in Atlan to be enriched with glossaries and terms from Atlan.
- To check for necessary permissions required to run the workflow, select Quick test for necessary permissions before workflow run.
- To run the workflow, choose Run. To schedule it to run later, choose Schedule & Run.

Synchronization of metadata

Now that you’ve configured the integration between Atlan and Amazon SageMaker Unified Studio, let’s explore how metadata flows bidirectionally between both platforms to maintain consistency and governance across your data landscape.

The Atlan SageMaker Unified Studio connector uses a bidirectional synchronization model that keeps business context and technical metadata consistent across both solutions. The process delivers reliability, traceability, and governance-safe updates, regardless of where changes originate. The following diagram illustrates the solution architecture.

Sequential workflow for the SageMaker Unified Studio Atlan integration

The integration between SageMaker Unified Studio and Atlan follows a carefully orchestrated sequential workflow that enables seamless metadata synchronization across both platforms.

The process begins with connection setup through IAM, where authentication and authorization are configured to establish secure access between the customer’s AWS account and Atlan’s AWS environment. This foundational security layer allows subsequent data exchanges to occur within a trusted framework.

After the connection is established, the metadata sync workflow can be triggered either on a defined schedule or manually by the user, providing flexibility based on organizational needs. When triggered, the Atlan SageMaker Unified Studio app calls the SageMaker Unified Studio APIs to ingest assets and metadata from the source system.

The ingested assets then undergo processing and transformation within Atlan, where they are converted into Atlan’s metadata model. This processing step is crucial because it makes the assets discoverable, searchable, and governable inside the Atlan platform, which means teams can use Atlan’s full governance capabilities.

A key capability of this integration is its real-time reverse sync for metadata updates. When a user modifies metadata for the assets inside Atlan (such as adding tags or updating descriptions), Atlan’s real-time reverse sync pipelines immediately detect these changes and push the updates back to SageMaker Unified Studio. This keeps SageMaker Unified Studio reflecting the most up-to-date metadata entered by users in Atlan, eliminating the risk of metadata drift between systems.

This bidirectional sync creates a continuous loop where metadata flows from SageMaker Unified Studio to Atlan for ingestion and publication, simultaneously flowing back from Atlan to SageMaker Unified Studio through real-time reverse sync. The result is a consistent, bidirectional metadata flow that keeps both platforms synchronized. Teams can work confidently knowing that their metadata governance efforts are reflected across their data.

The following diagram illustrates this complete workflow, showing how metadata moves through each stage of the integration from initial IAM authentication through the continuous bidirectional sync loop that maintains metadata consistency across both platforms.

SageMaker Unified Studio to Atlan: Ingestion of metadata

The Atlan-SageMaker Unified Studio App periodically connects to SageMaker Unified Studio using secure API calls to ingest metadata. This metadata is transformed and mapped into Atlan’s metadata model, then published through the Atlan publish app as new or updated assets.

Each ingestion cycle is fully logged by Atlan’s audit service, which captures timestamps, correlation IDs, and the full change record. These logs support deduplication, troubleshooting, and replay in the event of partial failures.

Atlan to SageMaker Unified Studio: Synchronizing enriched business context

When users enrich assets inside Atlan, for example by updating descriptions or attaching glossary terms, the integration detects these changes and selectively pushes them back to SageMaker Unified Studio.

The reverse sync control plane is a pipeline that automatically detects changes made to assets and then triggers SageMaker Unified Studio Update API calls in the background to keep everything synchronized.

What’s next?

Phase 1 delivers core metadata synchronization and principal catalog selection for immediate consistency across your data governance platforms. Phase 2 will synchronize lineage and data quality, so teams see the same data flows and quality signals in both Atlan and SageMaker Catalog, enabling end-to-end visibility into how data moves through your pipelines and maintaining quality metrics consistently tracked across both systems. Phase 3 will add integrated approval workflows to streamline how access is requested and granted across solutions, reducing friction for data consumers while maintaining robust governance controls. These upcoming phases build toward a fully connected governance experience, keeping metadata, lineage, quality, and access policies aligned across the modern data stack.

Cleanup

If you no longer need the SageMaker Unified Studio connector integration, complete the following steps to clean up your environment and avoid unintended resource usage:

Delete the CloudFormation stack. Navigate to the AWS CloudFormation console, locate the stack deployed for this solution, and choose Delete. This action removes the AWS resources provisioned by the stack, including IAM roles, policies, and supporting components.
Remove the connection in Atlan. Visit Delete a connection to follow the steps outlined in Atlan’s documentation to delete the associated connection.

Cleaning up these components keeps your AWS and Atlan environments streamlined, secure, and cost-efficient.

Conclusion

In this post, you learned how to establish a bidirectional integration between Atlan and Amazon SageMaker Unified Studio that unifies metadata governance across your data and AI environments. You walked through deploying the necessary AWS infrastructure using CloudFormation, configuring the secure IAM based connection, and setting up bidirectional synchronization to keep glossary terms, descriptions, and governance context aligned across both platforms.

Organizations can use this integration to connect business and technical users within a single governance framework, creating a consistent, trusted view of data across the enterprise. With one secure configuration, teams can synchronize metadata between Atlan and Amazon SageMaker Unified Studio, establishing a reliable foundation for innovation, collaboration, and responsible AI at scale.

About the authors

How Bayer transforms Pharma R&D with a cloud-based data science ecosystem using Amazon SageMaker

2025-12-13 Avinash Erupaka

Post Syndicated from Avinash Erupaka original https://aws.amazon.com/blogs/big-data/how-bayer-transforms-pharma-rd-with-a-cloud-based-data-science-ecosystem-using-amazon-sagemaker/

This post was written with Avinash Erupaka from Bayer (IT PH, Drug Innovation platform)

How can pharmaceutical companies unlock the full potential of their data to drive breakthrough innovations? Bayer, a global leader in health and nutrition, is dedicated to tackling the pressing challenges of our time, including a growing and aging population and the strain on our planet’s ecosystems. Its mission of “Health for All, Hunger for None” drives its commitment to addressing societal and environmental needs through groundbreaking research. Bayer is focused on developing innovative solutions that make a tangible difference in the world and value for its customers, employees, and stakeholders. Headquartered in Leverkusen, Germany, Bayer operates across 80 countries and is pioneering a data science ecosystem that transforms how research teams access, analyze, and derive insights from complex scientific data.

By harnessing the power of data, analytics, artificial intelligence and machine learning (AI/ML), and generative AI, Bayer is creating a cloud-based Pharma R&D Data Science Ecosystem (DSE) on AWS that powers cutting-edge technologies and concepts with robust data management. In doing so, R&D teams can fully realize the potential of unified data and analytics.

In this post, we discuss how Bayer used the next generation of SageMaker to build a solution that unified data ingestion, storage, analytics, and AI/ML workflows. Built on data mesh principles, Bayer’s DSE integrates advanced data ingestion, storage, analytics, and ML workflows to enable agile experimentation and scalable insight generation. It democratizes access to analytics, fosters cross-Region collaboration, and provides flexible integration of structured, semi-structured, and unstructured data.

Challenges in pharmaceutical research

In pharmaceutical research, data has become the most critical asset for driving innovation. However, managing this data effectively presents unprecedented challenges and traditional data management approaches are becoming increasingly inadequate for complex, global research initiatives. Many pharma R&D organization face a complex ecosystem of data and analytics related obstacles that hinder scientific discovery and operational efficiency:

Siloed datasets – Research datasets are siloed across domains, limiting reuse and slowing discovery.
Multiple data modalities – Clinical trial data (structured), real-world evidence (semi-structured), and genomic files (unstructured) existed in isolation, complicating integration and analysis.
Inflexible ingestion capabilities – Systems that support batch processing (such as trial data), real-time data streams (for example, from lab equipment), and event-driven ingestion (such as regulatory updates).
Rising R&D costs – Disparate technologies and disconnected systems create operational inefficiencies and increased licensing and maintenance costs.
Inconsistent landscape to fully use ML – The absence of a unified data architecture and standardized, domain-agnostic MLOps workflows mean that data and analytics innovation is often ad hoc and non-repeatable. Teams lack a streamlined way to scale successful patterns, resulting in redundant efforts, longer development cycles, and missed opportunities for cross-domain synergy.
Disconnected architectures – Software solutions are not integrated into the wider unified ecosystem, resulting in silos, redundancies, and inefficiencies.

Recognizing these systemic challenges, Bayer embarked on a transformative journey. DSE is not just a technological solution, but a strategic reimagining of how research data and analytics could be used across a global organization. By bringing together cutting-edge technologies, standardized frameworks, a collaborative data mesh, and lakehouse architecture, Bayer set out to help researchers and engineers accelerate pharmaceutical innovation.

Finding a solution with the next generation of SageMaker

Bayer envisioned a unified data science ecosystem that would provide the following:

A unified collaborative development experience for all data scientists regardless of their location or specialization
Seamless access to both structured and unstructured data through a consistent interface
Built-in governance and compliance controls appropriate for pharmaceutical research
Scalable compute resources to handle the most complex analytical workloads

Bayer conducted a comprehensive evaluation of various solutions before selecting the next generation of SageMaker as the cornerstone of their new data science ecosystem. Although other options had merits, Bayer prioritized the following capabilities:

Access to multimodal data – Essential for genomics, proteomics, and advanced biomarker research
Centralized asset marketplace – Central hub to discover and reuse data, features, models, and other enterprise assets
Integrated tooling ecosystem – Streamlined access to key tools like Git, ETL, MLflow, and generative AI application builders in one place
Multi-domain and cross-Region support – Critical for global research collaboration
Price-performance – Necessary for sustainable, long-term scaling

The capabilities of Amazon SageMaker Unified Studio and Amazon SageMaker Catalog aligned with Bayer’s vision of decentralized mesh execution combined with centralized discovery and governance. They enabled teams to work with their preferred tools, such as Jupyter Notebooks or workflow builders, while maintaining discoverability and reusability of assets.

Solution overview

This section describes the key features and architecture of Bayer’s DSE built on SageMaker. The DSE solution addresses the identified challenges through a multi-layered architecture:

Breaking down data silos – Multimodal data ingestion capabilities of the solution break down data silos by enabling unified storage, processing of structured, semi-structured, and unstructured data through batch, streaming, and event-driven pipelines.
Handling diverse data modalities – A hybrid lakehouse architecture, built on Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and Amazon Redshift, provides a flexible foundation for handling diverse data modalities and maturities while providing data consistency and accessibility.
Reducing costs through standardization – To address rising R&D costs and operational inefficiencies, pre-wired analytical workbenches offer standardized templates and integrated development environments (IDEs) that reduce redundancy and accelerate workflow development.
Unlocking AI/ML with Amazon SageMaker AI and Amazon Bedrock – Advanced AI/ML capabilities, powered by Amazon SageMaker AI and Amazon Bedrock, create a standardized, domain-agnostic MLOps environment that enables repeatable innovation and cross-domain synergy.
Managing tools ecosystem with end-to-end observability – Robust governance and observability features provide compliance and system reliability while integrating previously disconnected tools into a unified, well-monitored ecosystem that breaks down architectural silos and promotes efficient resource utilization.

The DSE architecture implements data mesh principles where data domains (omics, regulatory, clinical trials) are treated as products, with ownership and management responsibilities assigned to domain experts. These domains are decentralized for execution but remain discoverable and reusable through SageMaker Catalog. At the core of the architecture is a hybrid mesh lakehouse architecture that combines Amazon S3 and Iceberg, providing the flexibility to handle both structured and unstructured data efficiently. SageMaker Unified Studio provides an analytical layer where researchers can access the full suite of tools needed for their work. The following diagram illustrates this architecture.

Impact

The first phase of Bayer’s DSE confirmed the next generation of SageMaker as a powerful foundation for their R&D DSE—designed to balance decentralized innovation with centralized governance through a scalable data mesh architecture. With this solution, Bayer can catalog and manage multimodal data assets—including structured and unstructured data, ML features, models, and custom scientific assets—with context-rich metadata across diverse Pharma R&D domains. Bayer is now positioned to onboard over 300 TB of biomarker data and integrate siloed omics, clinical, and chemistry data repositories into a cohesive environment. With integrated tools like JupyterLab Spaces, MLflow, and SageMaker AI Studio, the DSE platform is laying the groundwork for a comprehensive, GxP-aware ML workbench—paving the way to operationalize over 25 high-value ML use cases and support more than 100 data scientists across the organization.

“The Data Science Ecosystem is vital for developing our medicines,” says Daniel Gusenleitner, Mission Lead for the R&D Data Science Ecosystem. “It enhances our business workflows with advanced analytics, helping us accelerate the search for new treatments. By integrating data from the entire research and development process, we improve the chances of technical success and ensure our efforts are efficient. Unlocking our data also facilitates target discovery, leading to groundbreaking advancements in patient care.”

Next steps

Bayer has successfully begun their Data Science Ecosystem on the next generation of Amazon SageMaker and is working to onboard the first use case of advanced biomarker research. Building on the strong foundation, Bayer is also accelerating the evolution of the DSE solution with the following key enhancements:

Federated catalogs and cross-domain integration – Enabling search and reuse of data assets across therapeutic areas and business units
Advanced ontology and semantic layer – Enriching metadata with domain knowledge to support AI-based search, discovery, and reasoning
Adoption of generative and agentic AI workflows – Driving novel drug discovery and accelerating hypothesis generation

Conclusion

By leveraging the next generation of Amazon SageMaker to build their cloud-based Data Science Ecosystem, Bayer is creating a foundation for faster, more efficient research and discovery. Amazon SageMaker is unifying diverse data types, enabling global collaboration, and standardizing ML workflows to help position Bayer at the forefront of data-driven innovation.

To learn more and get started with the next generation of SageMaker, refer to Amazon SageMaker or the AWS console.

About the Authors

Accelerate AI development using Amazon SageMaker AI with serverless MLflow

2025-12-02 Donnie Prakoso

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/accelerate-ai-development-using-amazon-sagemaker-ai-with-serverless-mlflow/

Since we announced Amazon SageMaker AI with MLflow in June 2024, our customers have been using MLflow tracking servers to manage their machine learning (ML) and AI experimentation workflows. Building on this foundation, we’re continuing to evolve the MLflow experience to make experimentation even more accessible.

Today, I’m excited to announce that Amazon SageMaker AI with MLflow now includes a serverless capability that eliminates infrastructure management. This new MLflow capability transforms experiment tracking into an immediate, on-demand experience with automatic scaling that removes the need for capacity planning.

The shift to zero-infrastructure management fundamentally changes how teams approach AI experimentation—ideas can be tested immediately without infrastructure planning, enabling more iterative and exploratory development workflows.

Getting started with Amazon SageMaker AI and MLflow
Let me walk you through creating your first serverless MLflow instance.

I navigate to Amazon SageMaker AI Studio console and select the MLflow application. The term MLflow Apps replaces the previous MLflow tracking servers terminology, reflecting the simplified, application-focused approach.

Here, I can see there’s already a default MLflow App created. This simplified MLflow experience makes it more straightforward for me to start doing experiments.

I choose Create MLflow App, and enter a name. Here, I have both an AWS Identity and Access Management (IAM) role and Amazon Simple Service (Amazon S3) bucket are already been configured. I only need to modify them in Advanced settings if needed.

Here’s where the first major improvement becomes apparent—the creation process completes in approximately 2 minutes. This immediate availability enables rapid experimentation without infrastructure planning delays, eliminating the wait time that previously interrupted experimentation workflows.

After it’s created, I receive an MLflow Amazon Resource Name (ARN) for connecting from notebooks. The simplified management means no server sizing decisions or capacity planning required. I no longer need to choose between different configurations or manage infrastructure capacity, which means I can focus entirely on experimentation. You can learn how to use MLflow SDK at Integrate MLflow with your environment in the Amazon SageMaker Developer Guide.

With MLflow 3.4 support, I can now access new capabilities for generative AI development. MLflow Tracing captures detailed execution paths, inputs, outputs, and metadata throughout the development lifecycle, enabling efficient debugging across distributed AI systems.

This new capability also introduces cross-domain access and cross-account access through AWS Resource Access Manager (AWS RAM) share. This enhanced collaboration means that teams across different AWS domains and accounts can share MLflow instances securely, breaking down organizational silos.

Better together: Pipelines integration
Amazon SageMaker Pipelines is integrated with MLflow. SageMaker Pipelines is a serverless workflow orchestration service purpose-built for machine learning operations (MLOps) and large language model operations (LLMOps) automation—the practices of deploying, monitoring, and managing ML and LLM models in production. You can easily build, execute, and monitor repeatable end-to-end AI workflows with an intuitive drag-and-drop UI or the Python SDK.

From a pipeline, a default MLflow App will be created if one doesn’t already exist. The experiment name can be defined and metrics, parameters, and artifacts are logged to the MLflow App as defined in your code. SageMaker AI with MLflow is also integrated with familiar SageMaker AI model development capabilities like SageMaker AI JumpStart and Model Registry, enabling end-to-end workflow automation from data preparation through model fine-tuning.

Things to know
Here are key points to note:

Pricing – The new serverless MLflow capability is offered at no additional cost. Note there are service limits that apply.
Availability – This capability is available in the following AWS Regions: US East (N. Virginia, Ohio), US West (N.California, Oregon), Asia Pacific (Mumbai, Seoul, Singapore, Sydney, Tokyo), Canada (Central), Europe (Frankfurt, Ireland, London, Paris, Stockholm), South America (São Paulo).
Automatic upgrades: MLflow in-place version upgrades happen automatically, providing access to the latest features without manual migration work or compatibility concerns. The service currently supports MLflow 3.4, providing access to the latest capabilities including enhanced tracing features.
Migration support – You can use the open source MLflow export-import tool available at mlflow-export-import to help migrate from existing Tracking Servers, whether they’re from SageMaker AI, self-hosted, or otherwise to serverless MLflow (MLflow Apps).

Get started with serverless MLflow by visiting Amazon SageMaker AI Studio and creating your first MLflow App. Serverless MLflow is also supported in SageMaker Unified Studio for additional workflow flexibility.

Happy experimenting!
— Donnie

Orchestrating data processing tasks with a serverless visual workflow in Amazon SageMaker Unified Studio

2025-11-26 Suba Palanisamy

Post Syndicated from Suba Palanisamy original https://aws.amazon.com/blogs/big-data/orchestrating-data-processing-tasks-with-a-serverless-visual-workflow-in-amazon-sagemaker-unified-studio/

Automation of data processing and data integration tasks is essential for data engineers and analysts to maintain up-to-date data pipelines and reports. Amazon SageMaker Unified Studio is a single data and AI development environment where you can find and access the data in your organization and act on it using the ideal tools for your use case. SageMaker Unified Studio offers multiple ways to integrate with data through its editorial tools, including Visual ETL, Query Editor, and JupyterLab builders.

Recently, AWS launched the visual workflow experience in SageMaker Unified Studio IAM-based domains. With visual workflows, you don’t need to code Python DAGs manually or have deep expertise in Apache Airflow. Instead, you can visually define orchestration workflows through an intuitive drag-and-drop interface in SageMaker Unified Studio. The visual definition is automatically converted to workflow definitions that leverage Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless, providing enterprise-grade orchestration capabilities with a simplified user experience.

In this post, we show how to use the new visual workflow experience in SageMaker Unified Studio IAM-based domains to orchestrate an end-to-end machine learning workflow. The workflow ingests weather data, applies transformations, and generates predictions—all through a single, intuitive interface, without writing any orchestration code.

For more details on Amazon MWAA Serverless, see Introducing Amazon MWAA Serverless.

Example use case

To demonstrate how SageMaker Unified Studio simplifies end-to-end workflow orchestration, let’s walk through a real-world scenario from agricultural analytics. The following diagram shows a weather data processing workflow that we will orchestrate using the visual workflow experience in SageMaker Unified Studio.

A regional agricultural extension office collects hourly weather data from multiple stations across farming communities. Their goal is to analyze this data and provide farmers with actionable insights into weather patterns and their impact on crop conditions. To achieve this, the team built a ML–powered analytics workflow using SageMaker Unified Studio to automate the processing of incoming weather data and predict irrigation needs.

In this walkthrough, we demonstrate how the visual workflow experience in Unified Studio can orchestrate an end-to-end data pipeline that:

Monitors and ingests hourly weather data from Amazon Simple Storage Service (Amazon S3)
Transforms raw weather measurements using Visual ETL jobs (type casting, SQL operations, and data cleansing)
Generates seasonal irrigation predictions and crop impact insights using JupyterLab notebooks

Whenever new weather data arrives, the workflow automatically routes it through a series of transformation steps, and produces ready-to-use insights—all visually orchestrated in SageMaker Unified Studio with no custom orchestration code required.

Prerequisites

Before you begin, complete the following steps:

Signup for an AWS account and create a user with administrative access using the setup guide.
Setup your SageMaker Unified Studio IAM-based domain:
1. Navigate to the Amazon SageMaker console and use the Region selector in the top navigation bar to choose the appropriate AWS Region.
2. On the Amazon SageMaker home page, choose Get started.
3. For Project data access, choose to Auto-create a new role with admin permissions.
4. Select the checkbox for S3 table integration with AWS Analytics services, for Data encryption choose Use AWS owned key, and then Set up.
Go back to the Amazon SageMaker home page and choose Open to access the SageMaker Unified Studio experience.
From the SageMaker Studio UI you can access the project in the SageMaker Unified Studio IAM-based domain. This project curates all assets accessible through the designated Execution IAM role.

Workflow implementation steps

In this section, we use Amazon SageMaker Studio to create an end-to-end visual workflow in IAM-based domain.

Step 1: Set up data storage and import weather dataset

First, we’ll prepare the Amazon S3 storage locations for raw and processed data:

Download this weather dataset file to your local environment.
From the left menu of the project, choose Files. Under Shared, create two new folders raw_data and processed_data.
Upload the weather dataset file downloaded locally into raw_data folder.

Step 2: Create the weather data transformation job using Visual ETL

Next, create a Visual ETL job to transform the raw weather data through type casting, SQL transformations, and data cleansing:

From the left menu, under Data Analytics, choose Visual ETL and Create Visual Job.
Choose the + sign, and under Data sources, choose Amazon S3.
For the Amazon S3 node settings, choose the following:
- S3 URI: Choose Browse S3 and Select
- Delimiter: ,
- Multiline: Disabled
- Header: Enabled
- Infer schema: Disabled
- Recursive file lookup: Disabled
Choose the + sign next to the Amazon S3 box to add another node, under Transforms select Change columns.
Connect the Amazon S3 node to the change columns node.
Select the Change columns node to open the configuration window.
- Choose Add type cast. Select temperature_2m (¬∞C) as the source column and add temperature_celsius as the target column. Select float as the Type.
- Select precipitation (mm) as the source column and add Precipitation_mm as the target column. Select float as the Type.
- Select rain (mm) as the source column and add Rain_mm as the target column. Select float as the Type.
- Select windspeed_10m (km/h) as the source column and add windspeed as the target column. Select float as the Type.
- Close the configuration window.

Choose the + sign to add another node, under Transforms select SQL query . In the configuration window, paste in the following SQL statement:

SELECT 
    MAKE_TIMESTAMP(2016, 1, day_of_year, hour, 0, 0) as timestamp,
    (temperature_celsius * 9 / 5) + 32 as temp_f,
    rain_mm * 25.4 as rain_inches,
    windspeed
FROM {myDataSource}

Choose the + sign to add another node, under Data targets, choose Amazon S3 and provide the following options:
- S3 URI: Choose Browse S3 and select the processed_data folder created in Step 1.
- Format: CSV
- Update catalog: true
- Database: sagemaker_sample_db
- Table: weather_data
- Include header: true
- Ouput to a single file: false
Connect the nodes to create a complete job.
Save the Visual ETL and name it DataProcessing.

Step 3: Create the analysis and prediction notebook using JupyterLab

Now, we’ll set up the JupyterLab notebook that performs seasonal irrigation analysis and crop impact predictions based on temperature, rainfall, and wind speed patterns.Complete the following steps:

Download the Crop Irrigation Prediction Python notebook to your local environment.
In the SageMaker Unified Studio, from the left menu, choose JupyterLab. Wait for a few seconds for JupyterLab to be set up if you are trying for the first time.
Upload CropIrrigationPrediction.ipynb using the upload files option.
Review the notebook code to understand how it processes the weather data and generates irrigation predictions.

Step 4: Orchestrate the workflow

Finally, we will use the visual workflow to orchestrate tasks. With visual workflows, you can define a collection of tasks organized as a directed acyclic graph (DAG) that can run on a user-defined schedule.

Choose Workflows from the left menu.
Choose Create new Workflow.
Rename the workflow to WeatherDataProcessingOrchestration.
Create S3 task for monitoring and ingesting raw weather data:
1. Choose the + sign, then choose S3 Key Sensor.
2. Select S3-task to open the configuration window.
3. For Bucket key choose Browse S3 and choose the synthetic_weather_hourly_data.csv file from the shared/raw_data S3 folder.
Create a Glue task to transform the weather data:
1. Choose the + Sign and add Data Processing Job / Glue Job Operator.
2. Select Glue-task node to open the configuration window. For Operation type select Choose an existing Glue job.
3. For Job name, choose Browse Jobs and select DataProcessing (this is the visual ETL job we created in the previous step.
Choose the + sign and add SageMaker Unified Studio Jupyter Notebook Operator.
Select the Notebook-task to open the configuration window. For Source, choose Browse Files and choose CropIrrigationPrediction.ipynb.
Connect the tasks to create the complete workflow.
Review the Workflow settings and choose Save.
1. Provide a workflow description, “Workflow for Weather Data Processing”
2. For Trigger, choose Manual only, because in this example you will trigger the workflow manually. You can also configure the workflow to trigger automatically on a schedule or disable it from running

Step 5: Execute and monitor the workflow

To run your workflow, complete the following steps:

Choose Run to trigger workflow execution.
Choose View runs to see the running workflow.
Choose the Run ID for detailed logs on the execution.
When the run is complete, you can review the task logs by choosing the Task ID.

The model’s output is written to the S3 processed data output folder. You can review the crop irrigation prediction results to verify they reflect realistic weather patterns and field conditions. If any results appear unexpected or unclear, examine the upstream transformation steps or adjust the notebook logic to refine the outputs.

Clean up

To avoid incurring future charges, clean up the resources you created during this walkthrough. Leaving these resources running may result in ongoing costs for storage and compute.To clean up your resources:

On the workflows page, select your workflow, and under Actions, choose Delete workflow.
In Visual ETL, select your weather data transformation flow, and under Actions, choose Delete job.
In Query Editor, use the three dots next to the name of the table weather_data and choose Drop table.
In JupyterLab, in the File Browser sidebar, choose (right-click) your notebook and choose Delete.
In Files, choose the folder raw_data and under Actions, choose Delete. Repeat the steps for the folders processed_data and output.

Conclusion

In this post, you learned how you can use the visual workflow experience in Amazon SageMaker Unified Studio to build end-to-end data processing pipelines through an intuitive, no-code interface. This experience removes the need to write orchestration logic manually while still offering production-grade reliability and scalability powered by Amazon MWAA Serverless. Whether you’re processing weather data for agricultural insights or building more complex machine learning pipelines, the visual workflow experience accelerates development and makes workflow automation accessible to data engineers, analysts, and data scientists alike.As organizations increasingly rely on automated data pipelines to drive business decisions, the visual workflow experience provides the perfect balance of simplicity and power. We encourage you to explore this new capability in Amazon SageMaker Unified Studio and discover how it can transform your data processing workflows.

To learn more, visit the Amazon SageMaker Unified Studio page.