Tag Archives: Amazon SageMaker Unified Studio

Build a multi-team AI and analytics environment on Amazon SageMaker Unified Studio with Terraform

Post Syndicated from Harshi Surampalli original https://aws.amazon.com/blogs/big-data/build-a-multi-team-ai-and-analytics-environment-on-amazon-sagemaker-unified-studio-with-terraform/

Platform engineers who support data and AI teams face a recurring challenge: analytics, machine learning (ML), and AI developer teams each need different tools. When each team operates its own tools and infrastructure, governance becomes fragmented. Organizations face duplicated effort and limited collaboration. Data engineers, data scientists, and AI developers might all work with the same underlying data. Yet they navigate separate consoles, manage distinct AWS Identity and Access Management (IAM) configurations, and lack a shared view of each other’s work.

Amazon SageMaker Unified Studio brings these capabilities together in one governed workspace. Every practitioner gets access to the tools they need: SQL editors, JupyterLab notebooks, and the Amazon Bedrock IDE. At the same time, they share a common business data catalog, security model, and project structure, all governed centrally within Amazon SageMaker Unified Studio. Teams collaborate within projects, while access to data, compute, and development artifacts is managed centrally.

Setting up this kind of multi-team environment means coordinating IAM roles, blueprints, project profiles, and user onboarding on top of the initial domain configuration. New user roles and use cases might require enabling new blueprints and combining them into new project profiles to extend the functionality supported by the domain. To keep this repeatable and consistent, we use the Amazon SageMaker Unified Studio Terraform module to codify the entire setup. The module creates three purpose-built project profiles, one for each team, on a shared domain, plus an initial project to get you started. Creating the remaining per-team projects is a single additional sub-module block.

Consider a mid-size technology company with three teams that need cloud-based data and AI capabilities:

Team What they need Key AWS services
Data Analytics Query and transform data across Amazon Simple Storage Service (Amazon S3), Apache Spark jobs, and Amazon Redshift, build and orchestrate data pipelines, manage a data catalog Amazon SageMaker Lakehouse, Amazon Redshift Serverless, AWS Glue, Amazon Managed Workflows for Apache Airflow (MWAA)
Machine Learning Run experiments, train models, deploy endpoints Amazon SageMaker AI, MLflow, Amazon EMR Serverless
Generative AI Build chat agents, knowledge bases, prompt flows, and guardrails Amazon Bedrock

A common issue today is that customers don’t have a unified strategy to deploy and manage their infrastructure. The analytics team has a standalone Amazon Redshift cluster, the ML team provisions SageMaker notebooks ad hoc, and the generative AI team experiments with Bedrock through the console. Each team runs its own infrastructure. Because these environments are separate, there’s no shared data catalog, no consistent access control, and no central visibility into what’s running or what it costs.

The goal: a single unified environment where each team gets exactly the capabilities they need, with shared data access, consistent security, and infrastructure you define entirely as code.

Solution overview

We use the Amazon SageMaker Unified Studio Terraform module to create a single domain with three project profiles, one per team. A project profile is a collection of blueprints which are configurations used to create projects, while the domain provides shared governance, IAM roles, and networking.

Key module capabilities:

  • Dual provider: Uses both the AWS provider and AWS Cloud Control (AWSCC) provider for full Amazon SageMaker Unified Studio resource coverage.
  • Service role flexibility: All IAM roles are auto-created with least-privilege policies, or you can supply existing roles to be used by Amazon SageMaker Unified Studio.
  • Domain and project level access using AWS Identity and Access Management (IAM), single sign-on (SSO) with AWS IAM Identity Center or both.
  • KMS encryption: Optionally encrypt domain metadata with a customer-managed AWS Key Management Service (AWS KMS) key using kms_key_identifier.
  • Model governance: The domain module automatically provisions a model governance project for Generative AI administrators to be added to. Members of the project can enable or disable specific models accessible in the domain.
Single SageMaker Unified Studio domain with three project profiles sharing common IAM roles, VPC, and S3 storage

Figure 1: Multi-team Amazon SageMaker Unified Studio architecture

The Terraform module follows a layered architecture with four composable modules:

Layer 1: Domain module. Creates the domain, Tooling blueprint, IAM roles, and S3 bucket:

module "domain" {
  source     = "aws-ia/sagemaker-unified-studio/aws"
  domain_name = var.domain_name
  vpc_id      = vpc-xxx
  subnet_ids  =  ["subnet-xxx", "subnet-yyy", "subnet-zzz"]
  enable_sso  = var.enable_sso
}

Layer 2: Blueprint module. Enables individual blueprints on the domain using for_each:

module "blueprints" {
  source   = "aws-ia/sagemaker-unified-studio/aws//modules/blueprint"
  for_each = local.blueprint_configs

  domain_id      = module.domain.domain_id
  blueprint_name = each.value.blueprint_name
  regional_parameters = local.regional_parameters
  manage_access_role_arn = module.domain.manage_access_role_arn
  provisioning_role_arn  = module.domain.provisioning_role_arn
}

Layer 3: Project profile module. Composes blueprints into team-specific profiles:

module "sql_analytics_project_profile" {
  source    = "aws-ia/sagemaker-unified-studio/aws//modules/project-profile"
  domain_id = module.domain.domain_id
  name      = "SQL analytics"
  blueprints = local.sql_analytics_blueprint_config
  blueprint_dependencies = [for k, bp in module.blueprints : bp.entity_id]
  depends_on = [module.blueprints]
}

Layer 4: Project and Project membership module. Creates a project and assigns members:

module "project" {
  source             = "aws-ia/sagemaker-unified-studio/aws//modules/project"
  domain_id          = module.domain.domain_id
  project_name       = local.project_name
  project_profile_id =  concat(module.all_capabilities_project_profile, module.sql_analytics_project_profile, module.generative_ai_project_profile)[0].project_profile_id 
  depends_on = [module.create_project_from_project_profile_grant]
}
module "project_membership" {
  source = " aws-ia/sagemaker-unified-studio/aws//modules/project-membership"

  domain_id            = module.domain.domain_id
  project_id           = module.project[0].project_id
  project_owners       = var.project_owners
  project_contributors = var.project_contributors

  depends_on = [
    aws_datazone_user_profile.sso_users,
    aws_datazone_user_profile.iam_users,
    awscc_datazone_group_profile.sso_groups,
  ]
}

At a high level, the deployment mentioned in this post performs the following:

  1. Terraform creates the Amazon SageMaker Unified Studio domain with shared IAM roles and networking.
  2. Terraform enables all required blueprints on the domain (DataLake, RedshiftServerless, Bedrock services, EMR, and so on).
  3. Terraform composes three project profiles (one per team), selecting the appropriate blueprints and parameters.
  4. Terraform creates a project from the first available profile (All Capabilities profile).
  5. Project members (SSO users, SSO groups, IAM users, or IAM roles) are assigned as owners or contributors to the created project.

What each team gets

After deployment and teams are assigned to their respective projects, each team can:

  • Data Analytics team (SQL Analytics profile):
    • Query data across Amazon S3 and Amazon Redshift using SageMaker Lakehouse.
    • Manage a shared data catalog with AWS Glue.
    • Create and share SQL notebooks and dashboards.
  • Machine Learning team (All Capabilities profile):
    • Run ML experiments with MLflow integration.
    • Process data at scale with Amazon EMR Serverless.
    • Access all SQL analytics and generative AI capabilities.
    • Build end-to-end ML pipelines with Workflows.
  • Generative AI team (Generative AI profile):
    • Build chat agents and knowledge bases with Amazon Bedrock.
    • Create and test prompt flows and guardrails.
    • Evaluate model performance with Bedrock Evaluation.
    • Deploy serverless AI functions.

All three teams share the same governed domain, data catalog, IAM policies, and networking. This eliminates the silos that existed before. Team isolation is achieved by assigning users or groups to projects created from the appropriate project profile.

Prerequisites

Before you begin, verify the following:

  • An AWS account with permissions to create and manage Amazon DataZone, AWS Identity and Access Management (IAM), Amazon Simple Storage Service (Amazon S3), Amazon Elastic Compute Cloud (Amazon EC2), and Amazon Virtual Private Cloud (Amazon VPC) resources. See Changing permissions for an IAM user for guidance.
  • Terraform version >= 1.7.
  • AWS Command Line Interface (AWS CLI) configured with credentials that have access to the target account and AWS Region.
  • Amazon SageMaker Unified Studio available in your target Region.
  • AWS IAM Identity Center configured (required only if using SSO-based project membership). If using IAM-only membership, Identity Center isn’t required.

Solution walkthrough

We complete the setup in four steps:

  1. Clone the module repository.
  2. Configure the Terraform variables for each team.
  3. Apply the Terraform configuration.
  4. Verify the platform and onboard teams.

Step 1: Clone the repository

Clone the Amazon SageMaker Unified Studio Terraform module and navigate to the quick-setup example:

git clone https://github.com/aws-ia/terraform-aws-sagemaker-unified-studio.git
cd terraform-aws-sagemaker-unified-studio

The module directory structure:

Directory tree of the terraform-aws-sagemaker-unified-studio module repository

Figure 2: Directory structure of the Amazon SageMaker Unified Studio Terraform module

Change into the quick-setup example directory:

cd examples/quick-setup

Step 2: Configure the Terraform variables

Copy the example variables file and edit it:

cp terraform.tfvars.example terraform.tfvars

For our multi-team scenario, we enable all three project profiles:

# Domain configuration
domain_name        = "acme-ai-platform"
domain_description = "Multi-team AI and analytics domain"
aws_region         = "us-east-1"
environment        = "prod"

# Project configuration
project_name        = "analytics-pilot"
project_description = "Initial pilot project for the analytics team"

# Enable all three team profiles
enable_sql_analytics    = true   # Data Analytics team
enable_generative_ai    = true   # Generative AI team
enable_all_capabilities = true   # ML team (includes SQL + AI + ML + EMR)

# SSO user access
enable_sso = true
sso_users  = [
  "[email protected]"
]

# Project membership
project_owners = {
  sso_users = ["a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"]
}

project_contributors = {
  sso_groups = ["a1b2c3d4-5678-90ab-cdef-EXAMPLE22222"]
  iam_roles  = ["arn:aws:iam::123456789012:role/CI-CD-DeployRole"]
}

# Note: If your organization doesn't use Identity Center, set enable_sso = false and use iam_users and iam_roles instead. See "Adding IAM principals to projects" below. This is a one way door decision. If you change this later on the domain and all dependent resources need to be destroyed and recreated

# Optional: Use a specific VPC instead of the default
# vpc_id     = "vpc-0abc123def456"
# subnet_ids = ["subnet-0abc123", "subnet-0def456"]

Here’s what each profile creates:

Variable Team Blueprints enabled
enable_sql_analytics Data Analytics DataLake, LakehouseCatalog, RedshiftServerless
enable_generative_ai Generative AI BedrockChatAgent, BedrockEvaluation, BedrockFlow, BedrockFunction, BedrockGuardrail, BedrockKnowledgeBase, BedrockPrompt
enable_all_capabilities ML (full access) All of the preceding + EMR Serverless, EMR on EC2, MLExperiments, Workflows

Internally, the toggles build a dynamic map of blueprints that gets passed to the blueprint module’s for_each:

locals {
  blueprint_configs = merge(
    var.enable_generative_ai  ? { /* Bedrock blueprints */ } : {},
    var.enable_sql_analytics  ? { /* DataLake, Redshift, Lakehouse */ } : {},
    var.enable_all_capabilities ? { /* EMR, ML, Workflows + all above */ } : {},
  )
}

Network auto-discovery: If you don’t specify vpc_id and subnet_ids, the module automatically discovers and uses your default VPC and its subnets.

Customizing blueprint parameters: The quick-setup example uses recommended defaults. For production, you can customize Amazon Redshift capacity, Glue database names, or deployment modes in the sql_analytics_blueprint_config and generative_ai_blueprint_config locals in main.tf. For example, to increase Redshift Serverless max capacity:

"RedshiftServerless" = {
  deployment_mode = "ON_CREATE"
  parameter_overrides = {
    "redshiftMaxCapacity" = {
      value       = "1024"    # Increased from default 512
      is_editable = false
    }
  }
}

Step 3: Apply the Terraform configuration

Configure your AWS CLI credentials for the target account and Region, then:

  1. Run terraform init to download the modules and providers.
  2. Run terraform plan and review the output to verify:
    • An Amazon SageMaker Unified Studio domain will be created.
    • Three IAM roles will be created (domain execution, manage access, provisioning).
    • An S3 bucket for tooling storage will be created.
    • All selected blueprints will be enabled on the domain.
    • Three project profiles will be created (SQL Analytics, Generative AI, All Capabilities).
    • A project will be created from the first available profile.
    • Project memberships will be configured for SSO identities, IAM principals or both.
  3. Run terraform apply and allow the apply to complete.

Figure 3: Successful terraform apply output showing the domain URL, project ID, and project profile IDs

View the key outputs:

# Domain portal URL — share this with your teams
terraform output domain_url

# Project URL — direct link to the created project
terraform output project_url

# All enabled blueprints
terraform output blueprint_ids

# Project profile IDs (one per team)
terraform output project_profile_ids

# IAM role ARNs for auditing
terraform output domain_execution_role_arn
terraform output manage_access_role_arn
terraform output provisioning_role_arn

Step 4: Verify the platform and onboard teams

Verify the deployment. Confirm the environment is healthy before onboarding teams:

  1. Open the domain_url from the Terraform output and confirm the Amazon SageMaker Unified Studio portal loads, and you can sign in using a principal that was added to the project.
  2. In the portal, confirm the three project profiles (SQL Analytics, Generative AI, All Capabilities) appear under the domain.
  3. Confirm the initial project created by terraform apply is listed and project members can access the project interface.
  4. Cross-check the Terraform outputs (terraform output blueprint_ids, terraform output project_profile_ids) against the blueprints and profiles you expected.

Access the portal: Open the domain_url from the Terraform output. Any user added as an owner or member of the project can sign in immediately and access their project.

Create team-specific projects: The initial terraform apply creates one project. To create additional projects for each team, add more "project" sub-moduleblocks referencing different profiles:

module "genai_project" {
  source             = "aws-ia/sagemaker-unified-studio/aws//modules/project"
  domain_id          = module.domain.domain_id
  project_name       = "genai-team"
  project_profile_id = module.generative_ai_project_profile.project_profile_id
}

You can also create projects from the Amazon SageMaker Unified Studio portal using the available profiles.

Onboard additional users: To isolate teams, create separate projects using the appropriate project profiles as shown earlier, then add members (SSO users, SSO groups, IAM users, or IAM roles) to each project using the project_membership module. Each principal type can be used independently or in combination. This restricts each team to the capabilities defined by their project profile. For details on adding IAM principals, see the next section.

Adding IAM principals to projects

Amazon SageMaker Unified Studio uses project membership to control who can access a project’s resources (data, environments, and development artifacts). The Terraform module supports four principal types for project membership:

  • SSO users: Individual IAM Identity Center users.
  • SSO groups: IAM Identity Center groups (all members inherit access).
  • IAM users: Native IAM user ARNs.
  • IAM roles: IAM role ARNs (assumed by humans, services, or automation).

Project membership defines which principals are authorized to access a project. It also sets what actions they are allowed to perform as either project owners or contributors. When a principal is added to a project through project membership, they are automatically added to the domain membership list and are authorized to access the interface.

Why add IAM principals to projects?

While Identity Center users and groups cover most human access patterns, IAM principals unlock additional use cases:

Use Case Principal Type Example
Continuous integration and continuous delivery (CI/CD) pipeline deploys environments in a project IAM Role arn:aws:iam::123456789012:role/GitHubActionsDeployRole
Service account runs automated data pipelines IAM Role arn:aws:iam::123456789012:user/etl-service-account
Organization without Identity Center needs project access IAM User/Role Any IAM principal in the account

Configuration

The project_owners and project_contributors variables accept any combination of principal types:

project_owners = {
  sso_users  = ["[email protected]"]
  sso_groups = []
  iam_users  = ["arn:aws:iam::123456789012:user/platform-admin"]
  iam_roles  = ["arn:aws:iam::123456789012:role/DataPlatformLead"]
}

project_contributors = {
  sso_users  = []
  sso_groups = ["a1b2c3d4-5678-90ab-cdef-EXAMPLE22222"]
  iam_users  = ["arn:aws:iam::123456789012:user/data-engineer-1"]
  iam_roles  = ["arn:aws:iam::123456789012:role/CI-CD-DeploymentRole"]
}

Project members can be defined as owners or contributors. The AWS caller identity that runs this Terraform code is added as a project owner so that it has permission to manage project resources and update team membership. To know what permissions each of these roles have please refer to the full documentation here.

How the module registers principals

The module automatically creates the required domain-level profile for each principal before assigning membership. No manual pre-registration is needed. The module handles the full lifecycle.

IAM identity prerequisite: IAM roles and users must have the AWS managed SageMakerStudioUserIAMConsolePolicy (or equivalent) attached to access the Amazon SageMaker Unified Studio interface. This policy provides permissions for domain listing, project creation, and other portal operations. The module doesn’t automatically attach this policy to IAM roles.

Example: Granting a CI/CD pipeline access to a project

project_contributors = {
  iam_roles = ["arn:aws:iam::123456789012:role/GitHubActionsDeployRole"]
}

The pipeline role can now create and manage environments within the project scope, without domain-wide access.

Example: IAM-only project membership

Organizations that don’t use Identity Center can manage project membership entirely through IAM principals by setting enable_sso = false in the domain module.

enable_sso = false

project_owners = {
  iam_roles = ["arn:aws:iam::123456789012:role/PlatformAdmin"]
}

project_contributors = {
  iam_users = [
    "arn:aws:iam::123456789012:user/analyst-1",
    "arn:aws:iam::123456789012:user/analyst-2"
  ]
  iam_roles = [
    "arn:aws:iam::123456789012:role/DataScientistRole"
  ]
}

Note: Currently, only IAM principals within the same AWS account as the domain can access the Amazon SageMaker Unified Studio portal. Cross-account IAM role access might be limited. Verify cross-account support in your Region before relying on it for production access patterns.

Additional considerations

As you move this setup toward production, consider the following areas.

Scaling to multiple environments

For production readiness, deploy separate domains for development, staging, and production using Terraform workspaces or separate state files:

# Development
terraform workspace new dev
terraform apply -var="domain_name=acme-ai-dev" -var="environment=dev"

# Production
terraform workspace new prod
terraform apply -var="domain_name=acme-ai-prod" -var="environment=prod"

Cost management

The base infrastructure (domain, IAM roles, S3 bucket) has minimal ongoing costs. Primary charges occur when teams create and use environments:

  • Redshift Serverless: Pay per RPU-hour when queries are running.
  • EMR Serverless: Pay per vCPU and memory per second.
  • Bedrock: Pay per input and output token.
  • S3: Standard storage pricing.

Use the parameter_overrides in blueprint configurations to set capacity limits (for example, redshiftMaxCapacity) and control costs.

Security and governance

The module creates least-privilege service roles by default. For additional permissions, attach custom policies to the user role:

user_role_policy_arns = [
  "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess",
  "arn:aws:iam::123456789012:policy/CustomBedrockPolicy"
]

You can also configure VPC endpoints for private connectivity and enable AWS CloudTrail for audit logging of all Amazon DataZone API calls.

DevOps best practices

  1. Version control: Store configurations in a code repository with automated CI/CD pipelines.
  2. Module registry: Publish organization-specific module compositions to a private Terraform registry.

Cleanup

To destroy all resources:

# Empty the S3 bucket first
terraform output s3_bucket_name
aws s3 rm s3://BUCKET_NAME --recursive

# Destroy all resources
terraform destroy

Warning: This deletes all resources including the domain, projects, and any data stored in the S3 bucket. Additionally, extra resources created manually within the console won’t be affected by this terraform destroy.

Conclusion

In this post, we walked through a real-world scenario: building a multi-team AI and analytics unified environment where data analysts, ML engineers, and generative AI developers each get tailored capabilities on a shared, governed domain. Terraform provisions and manages the full set of Amazon SageMaker Unified Studio resources, including the domain, blueprints, project profiles, and an initial project. As your needs grow, you can extend the configuration by adding projects per team through the project sub-module.

Key benefits of this approach:

  • Team-specific profiles: Each team gets exactly the blueprints they need, nothing more.
  • Shared governance: One domain, one set of IAM roles, one data catalog, consistent policies.
  • Infrastructure as Code: Repeatable, auditable, version-controlled deployments.
  • Auto-discovery: Minimal configuration required; VPC, subnets, and instance details are discovered automatically.
  • Scalable: Add teams, environments, or accounts by extending the same Terraform configuration.
  • Flexible membership: Add SSO users, IAM users, IAM roles, or any combination as project members.

The Terraform module and examples are available in the Amazon SageMaker Unified Studio Terraform module repository.

Have questions or feedback? Leave us a comment or open an issue in the GitHub repository.

Additional resources


About the authors

Harshi Surampalli

Harshi Surampalli

Harshi is a Senior Technical Account Manager at AWS supporting Financial Services customers, with deep expertise in hybrid cloud architectures and infrastructure modernization. She helps enterprises optimize their AWS environments, strengthen operational resilience, and navigate complex migrations. Outside work, she enjoys reading, gardening, and spending time with family.

Samuel Ouvrard

Samuel Ouvrard

Samuel is a Solutions Architect at AWS Canada who works with enterprise and ISV customers across the financial services and adtech sectors. He has built deep expertise in Terraform by creating and maintaining open-source Terraform modules that help customers deploy AWS services through Infrastructure as Code, simplifying what were previously complex, manual setups into repeatable, version-controlled deployments.

Nathan Yee

Nathan Yee

Nathan is a Specialist Solutions Architect at Amazon Web Services (AWS) with a focus on Agentic Data & Analytics services. He helps customers architect scalable data platforms and optimize their data processing workloads.

Governance compliance beyond the catalog in Amazon SageMaker Unified Studio

Post Syndicated from Amom Luiz original https://aws.amazon.com/blogs/big-data/governance-compliance-beyond-the-catalog-in-amazon-sagemaker-unified-studio/

As your data catalog grows, how do you know which assets meet your organization’s governance standards? Amazon SageMaker Unified Studio provides the building blocks for governance such as metadata form definitions, data quality scores, glossary associations, subscription data, and a daily catalog export queryable through SQL.

A data catalog typically answers what exists as asset names, schemas, and ownership. SageMaker Unified Studio goes further, surfacing governance signals like metadata completeness, data quality scores, and glossary associations for each asset. This pattern brings them together into a single view you can query across all your assets, projects, and data products.

This post walks through how to build that enrichment pipeline and provide an asset scorecard example with an extensible set of dimensions and weights. However, the key takeaway is the pattern itself: after the signals are in one place, your team can define what “compliant” means for your context and manage assets and data products at scale. The pipeline is the foundation, and the business/compliance logic is yours to adapt and evolve.

The governance questions this pattern can answer include:

  • Data Owner: Which of my assets are missing a required metadata form?
  • Data Steward: Which assets have no glossary terms associated?
  • Data Platform team/Managers: Is compliance improving since the team published the new governance rules, within and between projects?
  • Chief Data Officer: Which departments need to work to provide more reliable data products to the organization?

This solution extends Analyzing your data catalog: Query SageMaker Catalog metadata with SQL, which introduced the asset metadata export feature. This post focuses on enriching the metadata export to support an enhanced and data-driven governance compliance strategy.

Solution overview

Architecture diagram showing the daily enrichment pipeline within an AWS account. Five enrichment data sources on the left (Amazon SageMaker Asset Export, Custom Policy Config, Search and GetListings API, GetAssetType API, and ListSubscriptions API) feed an Enrichment Lambda function triggered by an Amazon EventBridge Scheduler. The Lambda writes daily snapshot partitions to the governance_metadata.asset_compliance S3 Table, governed by AWS Lake Formation. Downstream consumers on the right include Amazon Athena for SQL queries, Amazon Quick Sight for the compliance dashboard used by data stewards, data owners, and CDOs, and Amazon EventBridge with SNS and SES for threshold alerts.

Figure 1: Daily governance enrichment pipeline, from the five data sources through the enrichment Lambda function to the Amazon Athena, AWS Quick Sight, and Amazon EventBridge consumers

The enriched compliance table is itself an Amazon Simple Storage Service (Amazon S3) Table asset in the SageMaker catalog queryable inside SageMaker Unified Studio alongside all domain assets.

The enrichment pipeline runs daily and consists of five data sources joined by an enrichment Lambda function:

  1. Amazon SageMaker Asset Metadata Export: the base inventory of all published assets, stored as an S3 Table.
  2. Form Policy Config: an S3 JSON file that defines custom organization-level required forms per project and asset type.
  3. Search / GetListing API: returns glossary terms, attached metadata forms, and a data quality summary per asset.
  4. GetAssetType API: returns the complete data quality rules count per asset.
  5. ListSubscriptions API: returns the number of active consumer subscriptions per published asset.

The pipeline reads from all five sources, computes a weighted compliance score per asset, and writes the result as a new daily snapshot partition to the governance_metadata.asset_compliance S3 Table. Downstream consumers such as Amazon Athena, AWS Quick Sight compliance dashboard, and Amazon EventBridge compliance alerts, read from that enriched table.

The following figure shows the end data product you will build: a compliance dashboard scoring every asset across your projects.

Amazon Quick Sight compliance dashboard showing four KPI cards at the top: Avg Compliance Score (36.59), Assets Without Glossary (27), Assets Without Business Description (28), and Assets Without Resource Description (13). Below, a donut chart titled Assets by Compliance Band shows 28 total assets distributed across Critical (majority), At Risk, and Compliant bands. To the right, a horizontal bar chart titled Assets by Compliance Score ranks assets from lowest to highest, with daily_revenue_report at the top.

Figure 2: Compliance dashboard scoring every asset across your projects, the end product you build

Prerequisites

Before you begin, verify that you have the following in place:

  • Python 3.12 and AWS Cloud Development Kit (AWS CDK) for deployment of the enrichment pipeline from the sample GitHub repository.
  • An Amazon SageMaker Unified Studio domain. This post demonstrates the solution using an AWS IAM Identity Center (IDC)-based domain. You need permission to enable the asset metadata export on the domain (activating the export is covered in Part 1).
  • Published assets in the SageMaker Unified Studio domain: the solution enriches information based on assets already published in the domain.
  • An AWS account with permissions to work with the services used in this solution: AWS Lambda, Amazon Athena, Amazon S3 including Amazon S3 Tables, AWS Glue Data Catalog, AWS Lake Formation, AWS Quick Sight, and Amazon EventBridge.
  • AWS Lake Formation access: The asset metadata export and the enriched table are governed by AWS Lake Formation, so you need to grant the Lambda execution role and the Amazon service role the appropriate DESCRIBE/SELECT permissions (detailed in the aws-samples repository).
  • The AWS Command Line Interface (AWS CLI): configured for your account and target AWS Region.
  • An AWS Quick Sight account in the same Region, required for the dashboard in Part 6.

Estimated time to complete is approximately 2 hours. Estimated cost is less than $5 per month for a small test domain (AWS Lambda invocations, Athena queries, and S3 storage). Costs scale with the number of assets and query volume.

Deploy the solution

The complete source code for this solution is available in the sample-smus-asset-scorecard repository on GitHub. The AWS Cloud Development Kit (AWS CDK) stack provisions all required infrastructure, including the enrichment Lambda function, Amazon S3 policy configuration bucket, Amazon EventBridge Scheduler rule, AWS Identity and Access Management (IAM) roles, and the S3 Table namespace. Follow the walkthrough available in the repository to deploy the solution.

Part 1: The asset metadata export

The asset metadata export serves as the primary data source for the enriched scorecard dataset. To activate the export once per domain, use the following AWS CLI command:

aws datazone put-data-export-configuration \
  --domain-identifier <domain-id> \
  --region <region> \
  --enable-export

Within 24 hours you can read the asset metadata export as an S3 Table in SageMaker Unified Studio query editor (table asset_metadata.asset). You can check the Exporting Asset Metadata official documentation to get more details about the Asset Metadata export. Three fields from the export are used by the enrichment pipeline:

  • asset_id: the identity of each asset in the catalog, and the join key the enrichment uses to map each row to its published listing (Part 3).
  • extended_metadata[‘owningEntityId’]: this single key from the metadata map identifies which project owns the asset, driving the per-project compliance policy lookup in Part 2.
  • business_description and resource_description: the documentation signals that contribute to the description dimension of the compliance score.

The following figure shows an example of the Asset Metadata Export:

Amazon SageMaker query editor showing a SELECT query result for the asset_metadata.asset table with 47 rows. Columns displayed are snapshot_time, asset_id, asset_name, extended_metadata, resource_type_enum, resource_id, and account_id. Sample rows include asset names such as churn, user_sessions_silver, ad_hoc_export_2025, raw_clickstream_events, daily_revenue_report, and monthly_active_users, all of type GlueTable. The account_id column is masked.

Figure 3: Example of the asset metadata export queried in the SageMaker query editor

Part 2: The policy layer: Defining mandatory metadata forms

The export tells you what metadata form fields are filled. But required is a two-level concept in this solution:

Level 1: Amazon SageMaker Unified Studio metadata enforcement rules. You can configure mandatory metadata forms for publishing and subscription flows. These enforcement rules are defined at the asset-type level. You can call the GetAssetType API and inspect the required boolean in formsOutput.

Level 2: Custom organization-defined requirements. This solution also accounts for required metadata forms that lack enforcement rules, but your organization still considers mandatory. For example, a project might require a DataClassificationForm on every Glue table, even though SageMaker Unified Studio itself doesn’t enforce it at the asset-type level. These project-specific rules don’t exist as a native API endpoint, they can be stored externally in an S3 configuration file.

The enrichment pipeline reads both layers and takes their union. The SageMaker Unified Studio enforcement rules come from GetAssetType API per asset type and the custom organization-specific overrides come from a policy file stored in Amazon S3. Change the following sample file with the project IDs and required forms for your use case and put it in the S3 bucket created with CDK (see the Walkthrough section in the Readme of sample repository):

{
  "default": {
    "required_forms": ["DataClassification"]
  },
  "projects": {
    "<project-id-1>": {
      "required_forms": ["DataClassification", "CriticalityForm"]
    },
    "<project-id-2>": {
      "required_forms": ["DataClassification", "CriticalityForm", "OwnershipForm"]
    }
  }
}

The S3 configuration file is updated whenever your governance policy evolves. The GetAssetType call ensures the pipeline always respects whatever enforcement rules are defined in SageMaker Unified Studio, including any new rules created in future revisions.

Part 3: The enrichment pipeline

The enrichment pipeline reads the latest export snapshot and joins it with governance facets from the published catalog through GetListing. It then merges the compliance policy from both enforcement rules (GetAssetType) and the custom organization-defined policy file. The complete implementation with annotated code is available in the aws-samples repository.

The enrichment Lambda reads the latest asset metadata export, calls the APIs for the dimensions not present in the export (data quality scores and glossary associations), and writes one enriched compliance record per asset.

3.1 The compliance score

The compliance score combines five independent dimensions, each contributing a weighted percentage as the following table shows:

Dimension Weight Signal
Form coverage 50% % of required forms present compared to policy
Data quality score 20% Data quality pass/fail percentage
Data quality rule coverage 10% Actual rules compared to minimum expected
Glossary presence 10% Has at least 1 glossary term
Description 10% Has a business or resource description

Weights and the minimum data quality rules threshold are configurable. Adjust them to reflect your organization’s governance priorities.

3.2 Run the enrichment pipeline

To run the pipeline, invoke the Lambda function as shown in the following example:

aws lambda invoke \
  --function-name gov-obs-enrichment \
  --region <region> \
  --cli-binary-format raw-in-base64-out \
  --payload '{}' /tmp/response.json

Part 4: The governance answers

With the enriched table available, the governance questions listed at the beginning of the post can be answered using SQL queries against the enriched table.

4.1 Assets missing required forms

The following query returns the list of assets missing required forms:

SELECT
    asset_name,
    project_name,
    missing_required_forms,
    dq_score,
    dq_rules_count,
    overall_compliance_score
FROM "s3tablescatalog/gov-obs-data"."governance_metadata"."asset_compliance"
WHERE snapshot_date = CAST(CURRENT_DATE AS VARCHAR)
AND json_array_length(missing_required_forms) > 0
ORDER BY overall_compliance_score ASC;
Amazon SageMaker query editor showing the enriched asset_compliance table result with 27 rows. Columns are asset_name, project_name, missing_required_forms, dq_score, dq_rules_count, and overall_compliance_score. Sample rows show fact_orders in the data-engineering project with a score of 16.67, raw_email_events in marketing-analytics with 25.0, and raw_clickstream_events in admin-project with 25.0. The project name containing the account ID is masked.

Figure 4: Query result listing the assets that are missing required metadata forms

4.2 Compliance scorecard per project

To explore the scorecard grouped by project, run the following query:

SELECT
    project_name,
    COUNT(*) AS total_assets,
    ROUND(AVG(overall_compliance_score), 1) AS avg_compliance_score,
    COUNT(CASE WHEN json_array_length(missing_required_forms) > 0 THEN 1 END) AS assets_with_form_gaps,
    COUNT(CASE WHEN dq_score >= 80 THEN 1 END) AS assets_passing_dq
FROM "s3tablescatalog/gov-obs-data"."governance_metadata"."asset_compliance"
WHERE snapshot_date = CAST(CURRENT_DATE AS VARCHAR)
GROUP BY project_name
ORDER BY avg_compliance_score ASC;
Amazon SageMaker query editor showing a project-level aggregation result with 3 rows. Columns are project_name, total_assets, avg_compliance_score, assets_with_form_gaps, and assets_passing_dq. Results show admin-project (22 assets, avg score 16.0, 20 form gaps, 2 passing data quality), data-engineering (4 assets, score 31.1, 4 form gaps, 0 passing data quality), and marketing-analytics (3 assets, score 40.0, 3 form gaps, 0 passing data quality). The project name containing the account ID is masked.

Figure 5: Per-project compliance scorecard aggregation

4.3 Compliance trend since rule publication

To answer how your assets are evolving since you published new rules you can run the following query:

SELECT
    snapshot_date,
    ROUND(AVG(overall_compliance_score), 1) AS avg_compliance_score,
    COUNT(*) AS total_assets,
    COUNT(CASE WHEN json_array_length(missing_required_forms) = 0 THEN 1 END)
    AS fully_compliant_assets
FROM "s3tablescatalog/gov-obs-data"."governance_metadata"."asset_compliance"
WHERE snapshot_date >= '2026-05-01' -- replace with your governance policy publish date
GROUP BY snapshot_date
ORDER BY snapshot_date;
Amazon SageMaker query editor showing a time-series compliance trend result with 3 rows. Columns are snapshot_date, avg_compliance_score, total_assets, and fully_compliant_assets. Rows show 2026-06-27 (score 36.6, 28 assets, 2 compliant), 2026-06-28 (score 36.6, 28 assets, 2 compliant), and 2026-06-29 (score 36.2, 29 assets, 2 compliant).

Figure 6: Compliance trend across daily snapshots since the governance policy was published

Part 5: Alerting on non-compliant assets

You can go further and use Amazon EventBridge Scheduler combined with AWS Lambda to send weekly reports and alert your team when compliance drops below a defined threshold:

def handler(event, context):
    # Query compliance for today
    df = wr.athena.read_sql_query(
        sql="""
        SELECT project_name,
        COUNT(*) FILTER (WHERE json_array_length(missing_required_forms) > 0) AS non_compliant
        FROM "s3tablescatalog/gov-obs-data"."governance_metadata"."asset_compliance"
        WHERE snapshot_date = CAST(CURRENT_DATE AS VARCHAR)
        GROUP BY project_name
        HAVING COUNT(*) FILTER (WHERE json_array_length(missing_required_forms) > 0) > 0
        """,
        database="governance_metadata"
    )
    if not df.empty:
        # Send alert using Amazon SNS, a Slack webhook, Amazon SES, and so on.
        send_governance_alert(df)

You can configure the Amazon EventBridge Scheduler rule to fire on a weekly schedule or daily if your organization requires tighter compliance monitoring. The send_governance_alert function is illustrative and can be routed to any channel your team prefers: Amazon Simple Notification Service (Amazon SNS) topics, Amazon Simple Email Service (Amazon SES), or a Slack webhook through an HTTP call.

Part 6: Building a governance dashboard with AWS Quick Sight

With the enriched governance table producing daily compliance scores, the final step is making these insights visible to stakeholders through an actionable and interactive dashboard. AWS Quick Sight connects directly to Amazon Athena, so you can visualize the compliance table directly.

6.1 Connecting and configuring AWS Quick Sight with the enriched table

Grant the AWS Quick Sight service role (aws-quicksight-service-role-v0) Lake Formation DESCRIBE and SELECT permissions on governance_metadata.asset_compliance. You can see details of grants needed in the aws-samples repository. In the repository you also can see a Python script which creates AWS Quick Sight data source, datasets and a complete dashboard with the compliance data, as the following figure shows:

Amazon Quick Sight compliance dashboard showing four KPI cards at the top: Avg Compliance Score (36.59), Assets Without Glossary (27), Assets Without Business Description (28), and Assets Without Resource Description (13). Below, a donut chart titled Assets by Compliance Band shows 28 total assets distributed across Critical (majority), At Risk, and Compliant bands. To the right, a horizontal bar chart titled Assets by Compliance Score ranks assets from lowest to highest, with daily_revenue_report at the top.

Figure 7: Compliance dashboard connected to the enriched table in AWS Quick Sight

Amazon Quick Sight Governance Dashboard showing four horizontal bar charts comparing three projects (marketing-analytics, admin-project, and data-engineering). Top left: Avg Score by Project: marketing-analytics leads at 40, followed by admin-project around 37, and data-engineering at 31. Top right: Avg data quality Score by Project: data-engineering leads at approximately 26. Bottom left: Avg data quality Rules by Project: admin-project and marketing-analytics near 1.8, data-engineering at 1.2. Bottom right: Assets With Form Gaps by Project: admin-project has the most at approximately 19.

Figure 8: Per-project comparison charts in the governance dashboard

6.2 Create visualization using natural language

Use the natural language Build feature in AWS Quick Sight to create each visualization with a plain-language prompt:

Visualization Natural language prompt Purpose
Compliance scorecard “Show average overall_compliance_score by project_name as a bar chart” Per-project accountability
Compliance band distribution “Show count of asset_name by compliance_band as a pie chart” Executive summary
Score trend over time “Show average overall_compliance_score by snapshot_date as a line chart” Is governance improving?
Form gap breakdown “Show count of asset_name where missing_required_forms is not empty by project_name as a stacked bar” Which teams have metadata form gaps?
Glossary coverage “Show count of asset_name where glossary_terms_count equals 0 as a KPI” How many assets have no terms?
Data Quality coverage “Show average dq_rules_count by project_name as a bar chart” Which projects lack data quality rules?
Amazon Quick Sight analysis editor showing the AutoGraph builder. On the left, the Asset Compliance dataset field list includes asset_name, compliance_band, dq_rules_count, dq_score, glossary_terms_count, missing_required_forms, overall_compliance_score, project_name, and snapshot_date. On the right, the Build a visual panel shows a natural language query typed as: Show average overall_compliance_score by project name, with a BUILD button and example prompts visible.

Figure 9: Building a visualization with a natural language prompt in the AWS Quick Sight analysis editor

Amazon Quick Sight: Build a visual panel showing the query interpreted as: Average of Overall Compliance Score by Project Name. The resulting horizontal bar chart displays three projects: marketing-analytics (40), admin-project-29 (37.15), and data-engineering (31.09), with an ADD TO ANALYSIS button at the bottom.

Figure 10: Natural language prompt interpreted as average compliance score by project

This produces an AWS Quick Sight dashboard built on top of the enriched table that delivers actionable governance, not raw metadata counts. Stakeholders see compliance scores, form gaps, and improvement trends without needing to understand the underlying API model.

Amazon Quick Sight horizontal bar chart titled Average of Overall Compliance Score by Project Name, part of the SageMaker Unified Studio Scorecard dashboard. Three bars are shown: marketing-analytics at 40, admin-project at approximately 37, and data-engineering at 31.09, on a scale from 0 to 40.

Figure 11: Average overall compliance score by project rendered as a bar chart

For a deeper walkthrough of the natural language Build approach applied to the raw export table, see Build governance dashboards for Amazon SageMaker Catalog with AWS Quick Sight. The same technique applies to your enriched compliance table with the richer signals added by this pipeline.

Part 7: Seeing the pipeline in action

To illustrate how to apply these concepts, consider a domain with 20 published AWS Glue table assets across three projects, each at a different governance maturity level. After the daily enrichment runs, querying the compliance table immediately surfaces the differences.

7.1 Per-project scorecard

Briefly, you can see which teams are investing in governance and which ones need attention. The marketing-analytics leads because its assets have metadata forms, glossary terms, and data quality rules configured, while the other two projects have only basic data quality checks in place.

Amazon Quick Sight: Amazon SageMaker Scorecard dashboard showing three visuals. Top left KPI card displays 16 unique assets. Bottom left horizontal bar chart titled Average of Overall Compliance Score by Project Name shows marketing-analytics at 40, admin-project at 37.15, and data-engineering at 31.09. Right side pie chart titled Unique number of Asset Name by Compliance Band shows Critical at 75% (15 assets), At Risk at 20% (4 assets), and Compliant at 5% (1 asset).

Figure 12: Per-project scorecard with compliance score and compliance band distribution

7.2 Drilling into individual assets

Based on compliance score defined previously as example, you can see the governance maturity into clear bands:

Score Band Typical profile
80–100 Fully governed All required forms present, glossary terms linked, data quality passing, description provided
60–79 Mostly governed Required forms present, but missing either glossary terms or data quality coverage
40–59 Partially governed Some required forms missing, or forms present but no glossary or data quality
20–39 Data quality only Data quality rules configured and running, but no governance forms, glossary, or description
1–19 Minimally documented Only a description present, but no forms, no data quality rules, no glossary
0 Ungoverned No forms, no glossary, no data quality rules, no description

The pipeline produces this output automatically on every scheduled run without manual intervention required. The Amazon EventBridge Scheduler fires daily, enriches the latest snapshot, and writes the compliance scorecard to the Iceberg table. A single Athena query then produces the per-project, per-asset view shown here.

Scalability note: in our test scenario with a domain of 29 published Glue Table assets across three projects, the enrichment Lambda completed in approximately 30 seconds. For organizations with larger domains and production environment, consider parallelizing asset enrichment calls and increasing Lambda memory allocation. For very large catalogs, a fan-out pattern using AWS Step Functions can distribute the workload across multiple Lambda invocations.

7.3 Consumption-weighted risk: who is using ungoverned data?

Compliance scores tell you what is below the required threshold. But not all compliance findings carry equal urgency. An asset with a score of 10 that five projects depend on might require immediate attention. The enrichment pipeline adds a subscription count per asset which represents the number of projects that have an approved subscription to consume it. This comes from the ListSubscriptions API, filtered by subscribedListingId (every published asset has a listing). Combined with the compliance score, it produces a risk matrix, as the following example shows:

Quadrant Compliance Subscriptions Recommended Action
Gold standard >= 80 High No action: well governed and well adopted
Over-invested >= 80 Zero Review: strong governance on unused assets
Backlog < 50 Zero Plan: low risk, improve over time
Critical < 50 >= 2 Act now: many consumers depend on ungoverned data

The risk-view query joins both columns:

SELECT
    asset_name,
    project_name,
    subscription_count,
    overall_compliance_score,
    CASE
        WHEN subscription_count >= 2 AND overall_compliance_score < 50 THEN 'CRITICAL'
        WHEN subscription_count >= 1 AND overall_compliance_score < 50 THEN 'MEDIUM_RISK'
        ELSE 'OK'
    END AS risk_level
FROM "s3tablescatalog/gov-obs-data"."governance_metadata"."asset_compliance"
WHERE snapshot_date = CAST(CURRENT_DATE AS VARCHAR)
AND subscription_count > 0
ORDER BY subscription_count DESC, overall_compliance_score ASC;
Amazon SageMaker query editor showing a subscribed asset risk query result with 4 rows. Columns are asset_name, project_name, subscription_count, overall_compliance_score, and risk_level. Results show daily_revenue_report (2 subscriptions, score 99.6, OK), user_sessions_silver (1 subscription, score 35.0, MEDIUM_RISK), churn (1 subscription, score 52.0, OK), and monthly_active_users (1 subscription, score 60.0, OK). The project name containing the account ID is masked.

Figure 13: Risk query joining compliance score and subscription count to prioritize ungoverned assets

This turns your governance dashboard from a passive scorecard into a prioritization engine and asset teams know exactly where to focus based on both compliance gaps and downstream impact.

7.4 Extending to data products

With SageMaker Unified Studio, data producers can group related assets into data products, which are versioned, self-contained packages aligned to a business use case. Data consumers subscribe to a data product in a single request and receive access to all its member assets at once.

With this structure, you can aggregate compliance at the data product level. Build an asset_id -> data_product_name mapping by iterating over all projects. The data product scorecard query then rolls up compliance metrics across all assets in each product:

SELECT
    COALESCE(NULLIF(data_product_name, ''), '(unassigned)') AS data_product,
    project_name,
    COUNT(*) AS total_assets,
    ROUND(AVG(overall_compliance_score), 1) AS product_compliance_score,
    SUM(subscription_count) AS total_subscriptions,
    COUNT(CASE WHEN JSON_ARRAY_LENGTH(missing_required_forms) > 0 THEN 1 END) AS assets_with_form_gaps
FROM "s3tablescatalog/gov-obs-data"."governance_metadata"."asset_compliance"
WHERE snapshot_date = date_format(current_date, '%Y-%m-%d')
GROUP BY
    COALESCE(NULLIF(data_product_name, ''), '(unassigned)'),
    project_name
ORDER BY product_compliance_score ASC;
Amazon SageMaker query editor showing a data product compliance query result with 5 rows. Columns are data_product, project_name, total_assets, product_compliance_score, total_subscriptions, and assets_with_form_gaps. Rows include entries across data-engineering (score 31.1), admin-project (score 31.3), and marketing-analytics (score 40.0), plus User Behavior (score 43.5, 6 form gaps) and Finance Analytics (score 79.8, 9 subscriptions, 0 form gaps). The project name containing the account ID is masked.

Figure 14: Data product compliance rollup across projects

This produces a compliance view at the level data consumers need: the products they subscribe to. A data product with a compliance score of 15 out of 100 and three subscriber projects is a potential governance incident, not a backlog item.

Cleaning up

If you deployed this pipeline in a test environment, delete the following resources to avoid incurring ongoing charges:

  • The enrichment Lambda function and its associated IAM role.
  • The Amazon EventBridge Scheduler rule.
  • The governance_metadata AWS Glue database and the asset_compliance S3 Table.
  • The S3 policy configuration file and any associated S3 bucket.
  • The AWS Quick Sight dataset and analysis (if created).
  • The Lake Formation permissions granted to the AWS Quick Sight service role.

The aws-samples repository has the commands needed to run the cleanup.

Conclusion

This post showed how to build a governance observability pipeline using Amazon SageMaker Unified Studio metadata exports and APIs, computes weighted compliance scores, and surfaces actionable governance insights through SageMaker Unified Studio queries and an AWS Quick Sight dashboard.

Amazon SageMaker Unified Studio provides the building blocks for governance such as asset type definitions, metadata form fields, data quality scores, glossary associations, subscription data, and a daily catalog export. This post demonstrates how to join those signals within Amazon SageMaker Unified Studio into a single enriched dataset to enhance data governance at scale.

The specific scoring model implemented here considering the five dimensions, their weights, and the risk classification by consumption is not prescriptive, but a starting point. Your organization’s governance policy might weight data product coverage more heavily, treat regulated project assets differently, or define “compliant” with a different threshold. The pipeline itself is the reusable part: after the signals are together, the scoring logic is yours to define and evolve.

Have questions about implementing this governance pipeline in your environment? Share your thoughts in the comments below or visit the aws-samples repository to contribute.


About the authors

Amom Luiz

Amom Luiz

Amom is a Senior Specialist Solutions Architect, Data & AI, at AWS. He is focused on helping customers to define and implement their Data & AI strategy at scale.

Brenda Medeiros

Brenda Medeiros de Florenco

Brenda is a DevOps Delivery Consultant in AWS Professional Services. She helps customers automate and operate cloud infrastructure and is expanding her focus toward data platforms and governance.

Guilherme Monte

Guilherme Monte

Guilherme is a Data Architect (Delivery Consultant) in AWS Professional Services. He specializes in data platforms and helps organizations build scalable, well-governed data ecosystems.

Introducing the Workday Data Cloud integration with AWS

Post Syndicated from Poornima Chand original https://aws.amazon.com/blogs/big-data/introducing-the-workday-data-cloud-integration-with-aws/

Connecting enterprise systems to modern analytics and AI platforms often requires building brittle pipelines, duplicating data, or sacrificing the governance and business context that makes the data trustworthy. Today we’re introducing the integration of Workday Data Cloud with AWS. With this integration you get bi-directional, zero-copy access to Workday’s governed people and money data from the AWS analytics and AI services that you already use.

Workday Data Cloud overview

Workday Data Cloud is an open, secure, and headless data platform that makes Workday’s people and money data available in the tools where developers build, with its semantic definitions, permission models, metric logic, and audit controls intact. It provides a curated, unified view of people and money data through the Workday Data Lake, bi-directional zero-copy access to that data through Workday Data Connect using Apache Iceberg, and near real-time SQL access for applications and AI agents through Workday Live Data Query.

AWS analytics overview

AWS provides a comprehensive set of analytics and AI services that work together for data engineering, analytics, and machine learning. Here’s how the key services come together in the context of the Workday Data Cloud integration:

Amazon SageMaker Unified Studio

Amazon SageMaker Unified Studio (SMUS) is a unified development environment for data analytics and AI. It brings together data access, querying, visualization, and notebook-based development in one place. With SMUS, developers can connect to Workday Data Cloud, explore datasets interactively, run SQL queries, and build machine learning models, all without switching between multiple tools or interfaces.

Amazon Athena

Amazon Athena is a serverless, interactive query service to analyze data directly in Amazon Simple Storage Service (Amazon S3) using standard SQL. With the Workday Data Cloud integration, analysts and developers can run ad hoc queries against Workday data stored as Iceberg tables without needing to set up or manage infrastructure.

Amazon Redshift

Amazon Redshift is a cloud data warehouse that delivers fast query performance on structured and semi-structured data using SQL. Organizations already using Amazon Redshift for their analytics workloads can now incorporate Workday people and money data through the Glue Data Catalog federation for cross-domain analytics without data movement.

AWS Glue Data Catalog

AWS Glue Data Catalog serves as the central metadata repository that makes Workday data discoverable across the supported AWS analytics services. Through the integration, Workday’s Iceberg REST Catalog is registered as a federated catalog in AWS Glue, which means any AWS service that reads from the AWS Glue Data Catalog such as Amazon Athena, Amazon Redshift, Amazon EMR, and Amazon SageMaker AI can automatically discover and query Workday datasets without additional configuration.

Glue Data Catalog also works in the other direction, letting Workday query data that lives in AWS. Customers can register the Iceberg REST Catalog endpoint of the AWS Glue Data Catalog as an external catalog in Workday Data Cloud. Workday users can then browse and query Apache Iceberg tables stored in Amazon S3 in near real-time, without copying the data into Workday.

AWS Lake Formation

With AWS Lake Formation, you can set up and manage a secure data lake. In this integration, AWS Lake Formation provides the fine-grained permission layer that governs who can access Workday data within the AWS environment, complementing the governance that Workday already applies at its data layer. Administrators can scope each analyst or application to specific databases, tables, rows, or columns.

Amazon S3

Amazon S3 is the foundational storage layer for data lakes on AWS. In the context of this integration, Amazon S3 stores Apache Iceberg tables that can be queried from Workday, so organizations can query and analyze data in AWS alongside their Workday data.

Workday Data Cloud and AWS integration overview

The integration between Workday Data Cloud and AWS enables bi-directional, zero-copy data access. This means data doesn’t need to be copied or replicated between Workday and AWS. Instead, each platform can query the other’s data in place, respecting existing governance and security controls on both sides.

Accessing data in AWS from Workday

Workday applications and AI agents can access data stored in AWS, enriching HR and finance workflows with external business data. Developers building agents or analytics within the Workday platform can query Apache Iceberg tables stored on Amazon S3 by registering the Iceberg REST Catalog endpoint of the AWS Glue Data Catalog as an external catalog in Workday Data Cloud.

Zero-copy federation

Workday Data Cloud federates to the AWS Glue Data Catalog using OAuth 2.0 authentication. After registration, Workday users can discover and query Iceberg tables backed by AWS directly from within Workday Data Cloud without copying data. AWS Lake Formation defines which databases and tables the Workday connection can read, whereas Workday’s Configurable Security controls which individual Workday principals can access the federated tables. This means an AI agent handling workforce planning in Workday can pull in revenue forecasts from an Iceberg table on AWS. Combining those financial projections with headcount data gives HR planners hiring recommendations to review, accounting for both business growth and budget constraints.

Accessing data in Workday from AWS

AWS analytics and AI services can access Workday’s governed people and money data. There are two primary patterns for AWS to query Workday data, each optimized for different use cases:

Live Data Query

Some use cases require near real-time access to Workday data, such as an AI agent checking current headcount or a dashboard showing today’s financial close status. For these, developers can use Workday Live Data Query. From Amazon SageMaker Unified Studio, Amazon Athena, Amazon EMR, or AWS Glue, users connect to Workday through the Live Data Query connector and execute SQL queries directly against live Workday datasets. The query runs within Workday’s governed environment and results flow back in near real-time. This is ideal for operational AI agents, near real-time dashboards, and any application where data freshness is critical.

Amazon SageMaker Unified Studio notebook querying live Workday supplier data with Workday Live Data Query

Figure 1: Querying live Workday supplier data from an Amazon SageMaker Unified Studio notebook with Workday Live Data Query

Zero-copy federation with Apache Iceberg

For near real-time analytics, machine learning (ML) training, and large-scale data exploration, developers access Workday data through Workday Data Connect’s Apache Iceberg integration. By registering the Workday Iceberg REST Catalog in AWS Glue Data Catalog or AWS Lake Formation, all Workday datasets appear as standard Iceberg tables discoverable by any AWS analytics service. The compute runs on the customer’s AWS environment (using Athena, Amazon Redshift, Amazon EMR, or SageMaker AI) while the data stays in Workday Data Cloud.

Amazon Athena query editor running standard SQL against a Workday Data Connect Iceberg table

Figure 2: Querying a Workday Data Connect Iceberg table with standard SQL in the Amazon Athena query editor

Querying a Workday Data Connect Iceberg table with standard SQL in the Amazon Athena query editor

Figure 3: Querying a Workday Data Connect Iceberg table with standard SQL in Amazon Redshift Query Editor v2.

SageMaker Unified Studio notebook using PySpark to browse Workday Iceberg tables through Workday Data Connect

Figure 4: Browsing Workday Iceberg tables with PySpark in a SageMaker Unified Studio notebook using Workday Data Connect

What this means for customers

This integration changes how organizations work with their people and money data for analytics and AI. Here’s what it unlocks:

  • Organizations no longer need to build and maintain complex extract, transform, and load (ETL) pipelines to get Workday data into their analytics environment, which removes data silos without duplicating data. The data stays where it lives, governed and accurate, while remaining fully accessible for analysis and AI.
  • When AI agents and models can access Workday data with its full business context, developers spend less time reverse-engineering business logic and rebuilding permission models. That context includes who has access to what, what a metric actually means, and how calculations are defined.
  • Workday’s permission model, audit controls, and business logic travel with the data, and AWS Lake Formation adds governance on the AWS side. The result is data access that is both open and controlled, which is exactly what regulated industries and enterprise security teams require.
  • Instead of building pipelines, monitoring data freshness, and reconciling duplicates, data teams can focus on building analytics and machine learning applications that drive business outcomes.

Use case walkthrough: AI-powered workforce analytics

Consider a Chief People Officer who wants to understand the relationship between employee engagement scores, compensation benchmarks, and attrition patterns across the organization. Today, this requires extracting data from Workday, loading it into a separate analytics platform, joining it with market data, and building custom reports. This process takes weeks and produces stale results.

With the Workday Data Cloud and AWS integration, a data team can perform the following steps:

  1. Connect to Workday Data Cloud from AWS Glue using the pre-built Workday connector.
  2. Explore workforce datasets such as compensation, tenure, engagement, and performance directly, with full semantic context (job levels, org hierarchies, and tenure calculations) already defined.
  3. Join Workday people data with market compensation benchmarks and recruiting pipeline data stored as Iceberg tables on Amazon S3, all through standard SQL in Athena or SageMaker AI notebooks.
  4. Build a machine learning model in SageMaker AI that predicts attrition risk based on the combined dataset.
  5. Deploy an AI agent that flags high-risk attrition patterns for HR business partners to review and act on. The agent uses Live Data Query for near real-time signals and batch Iceberg access for historical trend analysis.

The entire workflow runs without any data leaving its governed environment, without any custom pipelines to maintain, and with full auditability at every step.

Getting started

The Workday Data Cloud integration with AWS is generally available.

  • Amazon SageMaker Unified Studio users can connect to Workday Data Cloud directly through the Workday Live Data Query connector or the Workday Iceberg REST Catalog connector.
  • Workday Live Data Query connections can also be created in the AWS Glue console, and users can query live Workday data from Amazon Athena, Amazon EMR, and AWS Glue ETL jobs.
  • AWS Glue and AWS Lake Formation users can register the Workday Iceberg REST Catalog as a federated catalog to make Workday datasets discoverable across AWS analytics services.
  • Workday users seeking to query AWS data can register their Glue Data Catalog Iceberg REST endpoint as an external catalog in Workday Data Cloud.

For deeper technical walkthroughs and step-by-step configuration guides, refer to the following resources:

This integration represents a new model for enterprise data access, where governed data flows freely between platforms without compromising security, accuracy, or business context. We are excited to see what customers build with Workday Data Cloud and AWS together.


About the authors

Poornima Chand

Poornima Chand

Poornima is a Principal Solutions Architect at AWS, where she works with customers to solve their unique challenges using AWS technology solutions. She is passionate about driving innovation in agentic AI and helping enterprises build intelligent, scalable, and secure solutions. Outside of work, she enjoys traveling, reading, painting, and spending time in nature.

Archana Bansal

Archana Bansal

Archana is a Senior Manager of Product Management for Workday Data Cloud, helping organizations unlock more value from their enterprise data and AI. Over more than six years, she has shaped the platform’s strategy and product direction, with a focus on data interoperability. She brings 20+ years of product leadership across enterprise data, analytics, and financial platforms at some of the industry’s largest product companies.

Avijit Goswami

Avijit Goswami

Based in California, Avijit is a Principal Data & Applied AI Architect at AWS who partners with customers to design and build robust data foundations and agentic AI solutions. When he’s not working, he loves hitting new trails, discovering new destinations, cheering on his favorite teams, listening to music, and trying out new recipes in the kitchen.

Will Cavin

Will Cavin

Will Cavin is a Principal Technical Product Manager working across the Data and Analytics teams at AWS. He has a passion for helping customers scale their analytics workloads. Prior to AWS, Will served as a cryptologist in the United States Navy. Based in Northern Virginia, he likes to long-distance run and mountain bike with his wife and son in his free time.

Tej Nagabhatla

Tej Nagabhatla

Tej is a Senior Solutions Architect at AWS. He has years of experience guiding numerous Fortune 500 companies along their technology journeys. He specializes in providing architectural guidance across a wide range of topics around AI/ML, security, storage, containers, and serverless technologies. In his free time, Tej enjoys spending time with family, playing basketball, and traveling.

AWS Weekly Roundup: AgentCore Runtime V2, Amazon DynamoDB bulk put items, and more (November 9, 2026)

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-agentcore-runtime-v2-amazon-dynamodb-bulk-put-items-and-more-november-9-2026/

Most of the cost of adopting something new isn’t the new thing, it’s the move. Moving a Hive table to Apache Iceberg has typically meant weeks of engineering work. Last week, Amazon S3 Tables turned that into three steps: start, sync as often as you need, complete. The source stays writable except for a few minutes at cutover. The same week, Amazon DynamoDB started loading data from S3 into tables that already exist.

The pattern I like is that neither one asks you to stop what’s running. That’s also the way to read the agent news this week. The new Amazon Bedrock AgentCore Runtime is a single field on the runtime you already have, and Web Answer is a tool you attach to a gateway you already run. Now, let’s get into this week’s AWS news…

Headlines
The new AgentCore Runtime: elastic, optimized, and consistently fast starts – If you run agents on AgentCore and haven’t tried the new runtime since it became generally available, it’s worth a look. It changes two things. First, memory: sessions start small, page in more on demand, and give memory back when it goes cold instead of holding the peak, so you pay for what your agent uses. Second, cold starts: AgentCore prepares your environment once, snapshots it, and restores the snapshot for every new instance. In the team’s tests, that kept P75 cold starts at about 2 seconds for images from 200 MB up to 2 GB, while the original runtime rose from roughly 5.4 seconds to nearly 30 seconds as images grew. The per-unit rate is higher, but you use far fewer GB-hours, so the team expects most agents to see a lower bill, though that isn’t guaranteed.

To opt in, you set platformVersion to V2 when you create or update a runtime. Create and update take several minutes while the snapshot is prepared, and your container must report healthy on /ping within 120 seconds, so report health only after initialization finishes. The docs cover how to optimize your agent for V2. V2 is available in 10 Regions, and committed baseline pricing is coming soon.

Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg – You can now migrate Parquet tables registered in the AWS Glue Data Catalog into S3 Tables without building a pipeline or provisioning compute. S3 Tables validates the source and produces a compatibility report covering every object in scope. Each sync copies only what changed, so production keeps writing until cutover. Afterward you can query the table with Amazon Athena, Amazon SageMaker Unified Studio, or any engine that speaks the Iceberg REST API. You pay only for the data copied, with no charge for the conversion, and objects in S3 Intelligent-Tiering keep their tier. At launch it works within the same account and Region, in 10 Regions.

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

Amazon DynamoDB now supports bulk put items – You can now load line-delimited DynamoDB JSON or Amazon Ion files from Amazon S3 into existing tables as a managed, asynchronous job. Each record can carry its own condition expression, so one file can mix plain, insert-only, and version-guarded writes. Writes use dedicated system capacity and cost less than the equivalent per-request writes. An Accelerated tier handles deadline-bound jobs. The Database blog post gives illustrative Accelerated times of less than 1 hour for 100 million 1 KB items. One thing to know before you start: canceling doesn’t roll back items already written.

Announcing Web Answer on Amazon Bedrock AgentCore – You can now give an agent a single tool call that returns a synthesized, cited answer to a natural-language question, instead of building your own search, retrieval, and citation pipeline. Web Answer plans its own searches, runs several in parallel, and decides when it has enough evidence. You attach it as a target on AgentCore Gateway and call it over MCP, and you can limit it with domain include and deny lists and publish-date filters. It’s available in US East (N. Virginia). The getting started post explains when to use it instead of Web Search, which returns ranked results for your own model to synthesize.

AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event sources – You can now process records from a single Kafka partition across many parallel AWS Lambda invocations. Classic consumer groups cap parallelism at the partition count. Queue mode uses Kafka share groups and needs Kafka 4.2 or later with share groups enabled. It works with self-managed clusters and managed offerings such as Confluent Cloud. You give up ordering and get at-least-once delivery, so your functions need to be idempotent. The Compute blog walkthrough covers poller scaling and when to keep stream mode.

AWS announces AWS Interconnect – last mile preview with Equinix – You can now set up a private, managed connection from Equinix’s network of 282 data centers to AWS without ordering cross-connects or configuring routers. Capacity is quad-redundant across two locations and four links, with BGP and VLANs configured for you, a 99.99% availability SLA, and MACsec by default. You pick a Region, bandwidth, and AWS Direct Connect gateway, then paste the activation key into Equinix Fabric. The preview covers the San Jose metro and, as of last Thursday, Northern Virginia, with 1 Gbps connections at no charge.

AWS Security Hub now provides an SBOM inventory for your compute resources – You can now see every software package installed across your organization’s Amazon EC2 instances, Amazon ECR images, and Lambda functions in AWS Security Hub, with each package’s version and how many resources run it. That’s the list you want the next time a widely used library has a vulnerability. The inventory builds on the SBOMs that Amazon Inspector already generates, so it fills in automatically wherever Inspector is enabled. It’s included in Security Hub Essentials at no additional cost.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts and resources that you might find interesting:

Serverless ICYMI Q3 2026 – Julian Wood’s quarterly recap is a quick serverless catch-up before re:Invent: Lambda SnapStart for container images, a 90-minute timeout on Lambda Managed Instances, Lambda MicroVMs in 10 Regions, 1 MiB payloads in Amazon SNS, and the new AgentCore Runtime.

AWS contributes a large-transaction binary log optimization to MySQL – With binlog on, a huge transaction holds a global commit lock while its events are copied, blocking every other commit. The optimization, which Amazon Aurora MySQL has run in production since 2019, promotes the spill file into the binlog instead of copying it. In AWS’s tests, commit latency for a 50 GiB transaction dropped from about 7 minutes to about 2 to 2.5 seconds. It has now shipped to community MySQL.

How OrderServ achieved up to 64% cost savings with multi-tenant ECS – Cognizant’s food-ordering platform went from one Amazon ECS cluster per brand on AWS Fargate to a shared cluster on EC2 Auto Scaling groups with warm pools. The 64% applies to a 3-year Compute Savings Plan (47% with a 1-year plan). Scaling time also fell from 5–7 minutes to 30–60 seconds.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

AWS re:Invent – re:Invent returns to Las Vegas from November 30 to December 4. The event catalog is live and reserved seating is open, so register and claim your spots before they fill up.

AWS Community Days – Community-led conferences planned and delivered by community leaders, with a few more before the year ends: Belém, Brazil (November 14), JAWS FESTA 2026 in Akita, Japan (November 17), and Port Harcourt, Nigeria (December 12).

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Daniel Abib

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Implement least-privilege access with designations in Amazon SageMaker Unified Studio

Post Syndicated from Priyanka Chaudhary original https://aws.amazon.com/blogs/big-data/implement-least-privilege-access-with-designations-in-amazon-sagemaker-unified-studio/

When teams move data and AI work to production in a shared environment, one question quickly becomes unavoidable: who can do what? Amazon SageMaker Unified Studio now answers it with designations, which are named permission sets that give administrators granular control over the actions each member can perform within a project. Administrators define a designation once, then assign it to project members. A member might sign in through an AWS Identity and Access Management (IAM) role, an IAM user, or an AWS IAM Identity Center user or group. In each case, designations help you give each persona only the access its responsibilities require. Effective access also depends on identity and resource policies, service control policies, permissions boundaries, group memberships, and how the customer configures permission sets and assignments. Designations integrate with AWS CloudFormation, and there is no additional charge to use them beyond Amazon SageMaker Unified Studio itself.

In this post, we walk through why differentiated permissions matter, how designations work, and how to create and assign your first designation using the console and the AWS Command Line Interface (AWS CLI).

The challenge: One project, many roles

A single SageMaker Unified Studio project often brings together data engineers, data stewards, analysts, and machine learning (ML) engineers, each needing very different access. Consider a financial services team running its Lakehouse and ML workflows in one project. Engineers build and schedule extract, transform, and load (ETL) pipelines, and a steward approves subscription requests to keep sensitive data governed. Analysts query curated tables for dashboards, and ML engineers iterate on models in notebooks. They all work on the same data, but their access needs differ. The steward shouldn’t alter production pipelines, analysts shouldn’t launch expensive compute, and the scheduled ETL service account shouldn’t share permissions with someone running exploratory experiments. In the past, common approaches included giving everyone the same broad project role or splitting the work across separate projects.

“We automated everything about how teams get set up in SageMaker Unified Studio. Projects are created, members are synced from our directory daily, and data sources are registered without anyone clicking a button. The one thing we couldn’t automate was controlling what each person could do once they were inside. Solution teams, data scientists, and business analysts all arrived with the same access. With designations, we now define each user type’s permissions as part of the same automation that creates the project, so the right access is enforced from day one.”

— Tim Kopacz, Cloud Platform Architect, Swiss Life Germany

Without a purpose-built mechanism to scope permissions within a project, organizations face operational risk. A user might accidentally delete a connection that disrupts a production workload or modify catalog settings outside their area of responsibility. Teams often work around this by creating a separate project for each persona to isolate access. That approach fragments data across project boundaries and adds provisioning and member-management overhead. As more teams onboard, these workarounds compound to slow adoption and increase the governance burden.

Designations address several common scenarios:

  • Controlling access to Notebooks: Grant members full access, read-only access, or no access to notebooks within a project, so compute-intensive interactive work is available only to the roles responsible for it. This helps control costs by limiting who can run notebook workloads.
  • Separating exploration from production: Grant exploratory users read and explore access while production service accounts run scheduled workflows, all within the same project.
  • Supporting governance workflows: Assign specific personas responsibility for data stewardship, subscription approval, and metadata management, which helps prevent other members from bypassing your review process.

How designations work

Designations separate policy definition from member assignment, which is what makes them reusable across projects.

  • Define once: A domain administrator creates a designation by selecting granular permissions. For example, publish an asset, create data products, or approve subscriptions. Permissions are grouped into categories such as Catalog, Project, Subscriptions, Connections, and Notebooks, and are backed by the Cedar policy language.
  • Assign per project: Within a project, the project owner assigns one designation to each member. When a member performs an action through the portal or the API, SageMaker Unified Studio evaluates the assigned designation. It permits or denies the action based on the permissions the designation contains. Additionally, a member can also inherit designations through single sign-on (SSO) group membership. If a user is directly assigned one designation and also belongs to a group assigned to another, the user receives the union of both permission sets. Because effective permissions are the union of direct and group-inherited designations, review SSO group assignments regularly so members don’t receive more access than intended.

Because a designation is defined once and assigned per project, you get consistent permissions across every project without recreating policies each time. Data engineers, stewards, analysts, and automated pipelines can work within the same project with different permissions, which keeps data unified in one place rather than fragmented across duplicate projects.

Designations complement your existing IAM controls rather than replace them, and they provide one consistent authorization layer across every user type that SageMaker Unified Studio supports. This matters because SageMaker Unified Studio supports IAM roles, IAM users, SSO users, and SSO groups. You can’t attach IAM policies directly to SSO users or groups within SageMaker Unified Studio. Designations apply uniformly to all of these principals, so every user type is governed by the same permission model within a project.

Authorization happens in two layers. First, IAM evaluates whether a principal can invoke an API at the AWS API boundary. If IAM allows the call, the designation then evaluates whether the assigned permission set grants the specific action within the project. You don’t need to modify your existing IAM policies to adopt designations, but we still recommend that you continue to scope IAM roles to least privilege.

For example, the AmazonSageMakerDomainExecution role might hold the IAM permission to perform an API action such as datazone:CreateListingChangeSet, the permission to publish a data asset. The next layer is the designation: the action succeeds only if the project member’s designation includes the Manage Asset Publishing permission to publish assets in that project. Both layers must allow the action, and an IAM Deny blocks the call regardless of what the designation permits.

The following diagram shows how a project member’s request to publish a data set passes through both authorization layers.

Request to publish a data set passing through two layers: IAM at the API boundary, then the project designation

Figure 1: How a request to publish a data set passes through the IAM and designation authorization layers

The request first reaches the AWS API boundary, where IAM (layer 1) evaluates whether the principal can call the API. If IAM denies the call, the action is blocked. If IAM allows it, SageMaker Unified Studio (layer 2) evaluates the member’s designation. The action succeeds only when the designation also allows it. Otherwise, the action is blocked.

Managed designations

Every domain includes two AWS managed designations at the project level so you can start immediately:

  • Owner – Full control over all project resources, including project member management.
  • Contributor – Full access except for project member management and project deletion. Because Contributor is broad, we recommend custom designations for personas that don’t need full access.

You can view the permissions for these managed designations, but you can’t edit or delete them. To create your own permission sets, you build custom designations for the personas your organization needs.

Create a designation on the console

Designations are created and managed from the domain management portal. Only users whose profile is a domain administrator can access it. The domain management portal is separate from the Amazon SageMaker Unified Studio portal where members do their day-to-day work. In this example, we create an ML Engineer designation for members who build workloads through notebooks.

  1. Sign in to the SageMaker Unified Studio domain management portal.
  2. In the navigation pane, choose Designations. The Managed tab shows the AWS managed Owner and Contributor designations.
  3. Choose Create designation.
  4. For Name, enter ML Engineer, and add a Description.
  5. In the visual permission selector, expand the Notebook category and select only the permissions this persona requires, for example, Read-only actions for reviewers, or All Actions only for members who author notebooks.
  6. Choose Create.
Create designation page in the domain management portal with the visual permission selector expanded

Figure 2: Creating a designation in the Amazon SageMaker Unified Studio domain management portal, using the visual permission selector to choose the actions to include

The ML Engineer designation is now available to every project in the domain. You can also create a designation by duplicating an existing one, which pre-selects a set of permissions for you to customize. Review the pre-selected permissions and remove any that the new persona doesn’t need.

Assign a designation to a project member

After creation, a designation is available to every project in the domain. You can assign it from two places, depending on your domain type.

From the SageMaker Unified Studio portal (Identity Center-based domains), assign the designation from a project where you are an owner, or if your designation has the Manage Membership permission. Grant Manage Membership only to trusted project members. A member who has the Manage Membership permission can assign designations with more permissions than their own.

  1. Navigate to the project in the SageMaker Unified Studio portal.
  2. Choose the Members tab.
  3. To update an existing member, choose a designation from the Designation column for that user. To add a new member, choose Add member and select the member.
  4. For Designation, choose ML Engineer.
SageMaker Unified Studio portal Members tab assigning the ML Engineer designation to a project member

Figure 3: Assigning a designation to a project member from the Amazon SageMaker Unified Studio portal (Identity Center-based domains) using the Role list on the Members tab

From the domain management portal (both IAM and Identity Center-based domains):

  1. Sign in to the domain management portal and choose Projects in the navigation pane.
  2. Select the project, then choose the Members tab.
  3. To update an existing member, choose a designation from the Project designation column. To add a new member, choose Add members and select the type and member.
  4. For Designation, choose the designation to assign, then choose Save.
Domain management portal Members tab assigning a designation using the Project designation column

Figure 4: Assigning a designation from the domain management portal (both IAM and Identity Center-based domains) using the Project designation column on the Members tab

When the assigned user logs in to the project, they are permitted to create notebooks. Every project member is assigned a designation. Members without a valid designation have no permissions in the project.

Automate the full lifecycle with the AWS CLI and infrastructure as code (IaC)

You can manage the entire lifecycle, from creating a designation to assigning it, programmatically. This is useful for codifying governance and rolling it out at scale.

Prerequisites:

The IAM principal that runs these commands needs permissions (scoped to the domain) such as datazone:CreateDesignation, datazone:UpdateDesignation, and datazone:DeleteDesignation to manage designations. They need datazone:CreateProjectMembership to assign the designation to project members. Grant only the actions you need.

Create a designation with the AWS CLI:

aws datazone create-designation \
  --domain-identifier "dzd_example1234" \
  --name "Data Engineer" \
  --description "Runs data source and asset creation tasks" \
  --policy 'permit(principal, action in [
      DataZone::Action::"CreateDataSource",
      DataZone::Action::"UpdateDataSource",
      DataZone::Action::"CreateAsset"
    ], resource);'

Assign the designation when you add a member to a project:

aws datazone create-project-membership \
  --domain-identifier "dzd_example1234" \
  --project-identifier "proj_example5678" \
  --member '{"userIdentifier": "u-abc123def456"}' \
  --designation "USER_CREATED_DESIGNATION" \
  --user-created-designation-id "123456789"

Because designations are supported as resources in AWS CloudFormation, you can define them in your templates, version them in source control, and deploy them across domains through your existing infrastructure-as-code pipelines. You can also manage assignments with the CreateProjectMembership and UpdateProjectMembership API operations, so the full workflow can be automated without using the console.

Update designations as your needs evolve

When you update a designation’s permissions, the change takes effect immediately for all members assigned that designation across all projects, with no reassignment required. This means you can start with the minimum permissions each persona needs, and add permissions as requirements or new capabilities emerge. Because updates apply immediately across all projects, review designation changes through your change-management process and use AWS CloudTrail to audit them.

Delete a designation

Designations include an Allow delete property that is turned off by default, so a designation can’t be deleted until you explicitly allow it. When you create or edit a designation, set Allow delete to make the designation eligible for deletion. Once allowed, you can delete the designation even if project members are still assigned to it.

When you delete a designation that is still in use, those members remain in the project, but their assigned designation becomes invalid. They have no permissions in the project until the project owner reassigns them a valid designation. Plan deletions accordingly so members are not left without access.

You can also set this property programmatically. The CreateDesignation API and AWS CLI accept a deletion protection parameter, so you can codify the setting in your infrastructure as code templates.

aws datazone create-designation \
  --domain-identifier "dzd_example1234" \
  --name "Data Engineer" \
  --deletion-protection \
  --description "Can manage data sources and create assets" \
  --policy 'permit(principal, action in [DataZone::Action::"CreateDataSource", DataZone::Action::"UpdateDataSource", DataZone::Action::"CreateAsset"], resource);'

To create the designation without delete protection, use the --no-deletion-protection flag.

Conclusion

Designations give administrators control over who can do what within each Amazon SageMaker Unified Studio project. Define named permission sets once and assign them per project to help enforce separation of duties across data engineers, stewards, analysts, ML engineers, and automated pipelines. You don’t need to duplicate projects or build a permission system outside SageMaker Unified Studio. Designations require no additional infrastructure or cost, integrate with AWS CloudFormation, and layer cleanly on top of your existing IAM controls.

To get started, open the domain management portal for your Amazon SageMaker Unified Studio domain, choose Designations, and create your first permission set. To learn more, see the Amazon SageMaker Unified Studio documentation.


About the authors

Priyanka Chaudhary

Priyanka Chaudhary

Priyanka is a Senior Solutions Architect at AWS specializing in data lake and analytics services. She helps customers design scalable, secure cloud solutions and guides strategic customers through their cloud journey. Outside of work, she enjoys time with friends and family, movies, and travel.

David Victoria

David Victoria

David is a Senior Technical Product Manager with Amazon SageMaker at AWS. He focuses on administration and governance capabilities needed for customers to support their analytics systems. He is passionate about helping customers realize the most value from their data in a secure, governed manner. Outside of work, he enjoys hiking, traveling, and spending time with family.

Enforce IAM permissions boundaries for Amazon SageMaker Unified Studio Tooling blueprints

Post Syndicated from Sanjana Sekar original https://aws.amazon.com/blogs/big-data/enforce-iam-permissions-boundaries-for-amazon-sagemaker-unified-studio-tooling-blueprints/

Amazon SageMaker Unified Studio now supports custom permissions boundaries for IAM roles created by the Tooling blueprint. Organizations that enforce Service Control Policies (SCPs) requiring permissions boundaries on all AWS Identity and Access Management (IAM) roles can now adopt Amazon SageMaker Unified Studio without modifying their security posture.

Amazon SageMaker Unified Studio is a unified development environment that brings together data engineering, machine learning, and analytics tools into a single workspace. In Amazon SageMaker Unified Studio, a project is a collaborative workspace that bundles people, tools, and access permissions together. It builds every project from a project profile, which defines a list of blueprints. Blueprints are pre-configured infrastructure templates that provision AWS resources at project creation time or on demand, along with their default parameters. The Tooling blueprint is the only mandatory one. Amazon SageMaker Unified Studio deploys it with every project, creating foundational resources such as the project IAM role and security groups.

In this post, you learn how to create a permissions boundary that restricts AI agent capabilities. You then configure it on the Tooling blueprint using the AWS Command Line Interface (AWS CLI). Finally, you validate that the boundary is enforced on all provisioned roles.

The problem

Enterprises in regulated industries use SCPs to require that every IAM role in an account carries a permissions boundary. A well-scoped boundary prevents privilege escalation and verifies no role exceeds the maximum permissions defined by the organization’s security team. Before this feature, Amazon SageMaker Unified Studio Tooling blueprints created IAM roles without permissions boundaries. When an SCP enforced permissions boundaries, project creation failed with an explicit deny:

User: arn:aws:sts::<account-id>:assumed-role/AmazonSageMakerProvisioning-<account-id>/AmazonDataZoneEnvironmentDeployer-<account-id> is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::<account-id>:role/AmazonBedrockServiceRole-<project-id>-<env-id> with an explicit deny in a service control policy

Amazon SageMaker Unified Studio surfaces the blocked role creation as a Tooling environment provisioning failure, as shown in Figure 1.

SMUS project overview showing the Tooling environment in a failed state from a permissions boundary SCP denial

Figure 1: Project creation fails when the SCP requires a permissions boundary that is not attached

The project is marked as failed because its Tooling environment couldn’t deploy in the US East (N. Virginia) AWS Region (us-east-1). The details show a 403 permissions error, while the preceding IAM message identifies the underlying iam:CreateRole SCP denial. This blocked adoption for any organization with SCP-enforced permissions boundaries. The AWS CloudFormation event for the Tooling stack exposes the IAM failure behind the project-level error, as shown in Figure 2.

CloudFormation stack events showing the BedrockServiceRole in CREATE_FAILED from an iam:CreateRole SCP explicit deny

Figure 2: Detailed error showing the SCP denial in the Tooling blueprint AWS CloudFormation stack

The AmazonBedrockServiceRole resource entered CREATE_FAILED because iam:CreateRole was explicitly denied by the SCP, even though AWS CloudFormation surfaced the wrapper error as UnauthorizedTaggingOperation.

Granular control using a permissions boundary: Example use case

Beyond satisfying SCP requirements, permissions boundaries give administrators granular control over what the Tooling blueprint roles can do. For instance, some organizations have SecOps policies that require disabling Data Agent and Data Notebook capabilities across their accounts. These organizations want project members to access data connections and run SQL queries directly, but must block conversational AI, code generation, and notebook cell execution through the agent.

When PermissionsBoundaryArn is configured on the Tooling blueprint, SageMaker Unified Studio attaches the specified customer-managed permissions boundary to all IAM roles provisioned by that blueprint. If your governance requires a boundary, configure it explicitly and verify the resulting roles.

The following permissions boundary policy scopes the roles to the AWS services that Amazon SageMaker Unified Studio uses and then explicitly denies the Amazon DataZone actions that power the AI agent. This permissions boundary is provided for illustrative purposes only and isn’t a recommendation or reference for environment configuration. You should tailor your permissions boundaries to your specific workloads in accordance with the principle of least privilege.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSmusServiceScope",
      "Effect": "Allow",
      "Action": [
        "datazone:*",
        "sagemaker:*",
        "glue:*",
        "s3:*",
        "lakeformation:*",
        "redshift:*",
        "redshift-data:*",
        "redshift-serverless:*",
        "athena:*",
        "q:*",
        "elasticmapreduce:*",
        "bedrock:*",
        "lambda:*",
        "kms:*",
        "secretsmanager:*",
        "codecommit:*",
        "logs:*",
        "cloudwatch:*",
        "sts:AssumeRole",
        "iam:PassRole",
        "ec2:Describe*",
        "ec2:CreateNetworkInterface",
        "ec2:DeleteNetworkInterface",
        "ec2:CreateNetworkInterfacePermission",
        "ec2:DeleteNetworkInterfacePermission"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyDataNotebookAndDataAgent",
      "Effect": "Deny",
      "Action": [
        "datazone:*Notebook*",
        "datazone:*Cell*",
        "datazone:*Conversation*",
        "datazone:SendMessage",
        "datazone:GenerateCode",
        "datazone:CancelMessage"
      ],
      "Resource": "*"
    }
  ]
}

Warning: validate before using in production. This example scopes the roles to the service namespaces Amazon SageMaker Unified Studio uses, but it is still coarse (it allows each listed service in full) and is provided only for illustration. Because a permissions boundary is a ceiling, it must remain a superset of everything the three Tooling roles (datazone_usr_role, AmazonBedrockServiceRole, and AmazonBedrockLambdaExecutionRole) actually need. If Amazon SageMaker Unified Studio adds a dependency that isn’t listed, provisioning or in-console actions will fail with an access denied error. Validate in a non-production domain first.

With this boundary attached, the Tooling blueprint provisions normally, project members can access data connections and run SQL queries. However, any attempt to invoke the AI assistant or execute notebook cells through the agent returns an access denied error. The boundary acts as a ceiling that no policy attached to the role can override.

How it works

The custom permissions boundary feature operates at the blueprint configuration level. An administrator sets a PermissionsBoundaryArn in the Tooling blueprint’s regional parameters. When a user creates a new project that includes the Tooling blueprint, Amazon SageMaker Unified Studio provisions an AWS CloudFormation stack that creates three IAM roles and attaches the specified boundary to each:

  • datazone_usr_role – the role that all project members assume to access data and resources in that project.
  • AmazonBedrockServiceRole – for Amazon Bedrock operations.
  • AmazonBedrockLambdaExecutionRole – for Amazon Bedrock-related AWS Lambda functions.

Because the boundary is set at the blueprint level, it applies to every project created under that blueprint. No per-project configuration is needed.

Prerequisites

Before you begin, make sure that you have:

If your organization uses AWS Organizations with SCPs that require permissions boundaries, you will also need an organization with the target account as a member and permissions to create and attach SCPs in the management account.

Setting up the SCP (optional)

This section provides instructions to create an SCP and attach it to your AWS Organizations organizational unit or accounts. If your organization already enforces permissions boundaries through SCPs, skip this section. Otherwise, create an SCP in your AWS Organizations management account that denies IAM role creation unless an approved permissions boundary is attached. This also prevents the boundary from being removed, swapped, or weakened afterward:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRoleWithoutApprovedBoundary",
      "Effect": "Deny",
      "Action": [
        "iam:CreateRole",
        "iam:PutRolePermissionsBoundary"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "iam:PermissionsBoundary": "arn:aws:iam::${aws:PrincipalAccount}:policy/SMUSToolingBoundary"
        }
      }
    },
    {
      "Sid": "DenyRemovingBoundary",
      "Effect": "Deny",
      "Action": "iam:DeleteRolePermissionsBoundary",
      "Resource": "*"
    },
    {
      "Sid": "ProtectBoundaryPolicy",
      "Effect": "Deny",
      "Action": [
        "iam:DeletePolicy",
        "iam:CreatePolicyVersion",
        "iam:SetDefaultPolicyVersion"
      ],
      "Resource": "arn:aws:iam::${aws:PrincipalAccount}:policy/SMUSToolingBoundary"
    }
  ]
}

This policy does three things:

  • DenyRoleWithoutApprovedBoundary blocks creating a role, or attaching a boundary to an existing role, with anything other than the approved boundary ARN. Denying iam:PutRolePermissionsBoundary in addition to iam:CreateRole stops a privileged principal from swapping in a weaker boundary after the role exists.
  • DenyRemovingBoundary blocks iam:DeleteRolePermissionsBoundary outright, so the boundary cannot be stripped off. (This action doesn’t support the iam:PermissionsBoundary condition key, so it must be denied unconditionally.)
  • ProtectBoundaryPolicy prevents tampering with the boundary policy itself. Deleting it, or publishing and defaulting a new version that quietly widens what it allows.

Note: Scope these denies so you don’t lock yourself out. A broad deny on iam:PutRolePermissionsBoundary and iam:DeleteRolePermissionsBoundary also applies to your own administrators. Add an exception for a break-glass or IAM-admin role (for example, an aws:PrincipalArn StringNotLike condition) so a trusted principal can still manage boundaries.

To create the SCP, sign in to the AWS Organizations console with your management account and go to AWS Organizations → Policies → Service control policies. If SCPs aren’t enabled for your organization yet, choose Enable service control policies first. Choose Create policy, give it a name (for example, test_scp), and replace the default content in the policy editor with the JSON above substituting <account-id> with your account ID. Choose Create policy to save it.

After creating the SCP in the management account, verify its content before attaching it. Figure 3 shows the core create-role control. The full example above adds controls that prevent replacing or removing the boundary and modifying the protected policy.

Figure 3: Service Control Policy defined in the AWS Organizations management account

The AWS Organizations Content tab displays the customer-managed test_scp policy. Its visible statement denies iam:CreateRole unless the request uses the SMUSToolingBoundary policy.

Attach this SCP to the organizational unit or account where your Amazon SageMaker Unified Studio domain and domain-associated accounts reside. To do so, open the test_scp service control policy, choose the Targets tab, and choose Attach. The AWS organization structure appears; select the OU or account where the SCP should apply, then choose Attach policy.

Figure 4 identifies the member account that must inherit the SCP in this example organization. The target member account, datazone-account2, resides under OU2, while datazone-account1 is the organization’s management account. Attaching the SCP to the target account or a parent organizational unit enforces it there.

Figure 4: AWS Organizations account structure showing the management account and the target member account

After attaching the policy, verify the association on the SCP’s Targets tab, as shown in Figure 5.

SCP Targets tab listing datazone-account2 as an account target where test_scp is enforced

Figure 5: Service Control Policy attached to the target account where it should be enforced

The Targets tab lists datazone-account2 as an ACCOUNT target, confirming that test_scp is enforced directly on the intended member account.

Configuring the permissions boundary

In this section you will execute the required steps to create the permissions boundary and enable it in the Tooling blueprint. The example in this walkthrough uses us-east-1. Change it to the Region where your Amazon SageMaker Unified Studio domain is deployed. You must execute the configuration in the account where you plan to create your project. This can be your Amazon SageMaker Unified Studio domain account or accounts associated to your Amazon SageMaker Unified Studio domain.

Step 1: Create the permissions boundary policy

If you haven’t already created the boundary policy, save the following JSON document as a boundary-policy.json file on your workstation:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSmusServiceScope",
            "Effect": "Allow",
            "Action": [
                "datazone:*",
                "sagemaker:*",
                "glue:*",
                "s3:*",
                "lakeformation:*",
                "redshift:*",
                "redshift-data:*",
                "redshift-serverless:*",
                "athena:*",
                "q:*",
                "elasticmapreduce:*",
                "bedrock:*",
                "lambda:*",
                "kms:*",
                "secretsmanager:*",
                "codecommit:*",
                "logs:*",
                "cloudwatch:*",
                "sts:AssumeRole",
                "iam:PassRole",
                "ec2:Describe*",
                "ec2:CreateNetworkInterface",
                "ec2:DeleteNetworkInterface",
                "ec2:CreateNetworkInterfacePermission",
                "ec2:DeleteNetworkInterfacePermission",
                "iam:GetRole",
                "sqlworkbench:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "DenyDataNotebookAndDataAgent",
            "Effect": "Deny",
            "Action": [
                "datazone:*Notebook*",
                "datazone:*Cell*",
                "datazone:*Conversation*",
                "datazone:SendMessage",
                "datazone:GenerateCode",
                "datazone:CancelMessage"
            ],
            "Resource": "*"
        }
    ]
}

As noted previously, this illustrative policy is scoped to the services Amazon SageMaker Unified Studio uses but is still coarse, and its allow list must stay a superset of what all three Tooling roles need.

Then create the policy using the following command:

aws iam create-policy \
--policy-name SMUSToolingBoundary \
--policy-document file://boundary-policy.json \
--description "Permissions boundary for SMUS Tooling roles - denies Data Agent and Data Notebook capabilities"

Note the policy ARN from the output, because it will be used later in the procedure.

Step 2: Retrieve the ID of your domain

Retrieve the ID of your domain by running the following command. Replace <YOUR_DOMAIN_NAME> with the name of your SageMaker Unified Studio domain.

aws datazone list-domains \
--region us-east-1 \
--query "items[?name=='<YOUR_DOMAIN_NAME>'].id | [0]" \
--output text

Note the returned ID, because it will be used later in the procedure.

Step 3: Identify the Tooling blueprint

Retrieve the Tooling blueprint ID by running the following command. Replace <domain-id> with the ID you noted in Step 2.

aws datazone list-environment-blueprints \
  --domain-identifier <domain-id> \
  --managed \
  --region eu-west-1 \
  --query "items[?name=='Tooling'].id" \
  --output json | jq -r '.[0]'

Note the returned ID, because it will be used later in the procedure.

Step 4: Read the current configuration

Retrieve the current Tooling blueprint configuration by executing the following command. Replace <domain-id> with the ID from Step 2 and <tooling-bp-id> with the ID from Step 3.

aws datazone get-environment-blueprint-configuration \
--domain-identifier <domain-id> \
--environment-blueprint-identifier <tooling-bp-id> \
--region us-east-1 | tee tooling-bp-config-backup.json

Important: Back up the output of get-environment-blueprint-configuration before making any changes. The command above pipes the response to tooling-bp-config-backup.json so you have a restore point if you need to revert.

Note the values of provisioningRoleArn, manageAccessRoleArn, enabledRegions, and all fields inside regionalParameters (AZs, S3Location, Subnets, VpcId). You will need all of these in the next step.

Step 5: Set the permissions boundary

Update the blueprint configuration to include PermissionsBoundaryArn in the regional parameters using the following command.

Important: The put-environment-blueprint-configuration API operates in overwrite mode, it replaces the entire configuration with what you provide. You must include all existing values from the previous step’s output. The only new addition is PermissionsBoundaryArn inside the regional parameters. Omitting any existing parameter removes it.

Make sure to replace <domain-id> with the ID you noted in Step 2, <tooling-bp-id> with the ID you noted in Step 3, and all other <placeholder> values with the corresponding values from Step 4’s output.

aws datazone put-environment-blueprint-configuration \
--domain-identifier <domain-id> \
--environment-blueprint-identifier <tooling-bp-id> \
--enabled-regions '<enabledRegions>' \
--provisioning-role-arn "<provisioningRoleArn>" \
--manage-access-role-arn "<manageAccessRoleArn>" \
--regional-parameters '{
  "<region>": {
    "AZs": "<AZs>",
    "S3Location": "<S3Location>",
    "Subnets": "<Subnets>",
    "VpcId": "<VpcId>",
    "PermissionsBoundaryArn": "arn:aws:iam::<account-id>:policy/SMUSToolingBoundary"
  }
}' \
--region <region>

The following anonymized example is based on an existing Tooling blueprint configuration. Its S3Location reflects the bucket naming pattern used in that environment. Copy the exact S3Location returned in Step 4. Don’t use the following illustrative value. Here’s an example:

aws datazone put-environment-blueprint-configuration \
--domain-identifier <domain-id> \
--environment-blueprint-identifier <tooling-bp-id> \
--enabled-regions '["us-east-1"]' \
--provisioning-role-arn "arn:aws:iam::<account-id>:role/service-role/AmazonSageMakerProvisioning-<account-id>" \
--manage-access-role-arn "arn:aws:iam::<account-id>:role/service-role/AmazonSageMakerManageAccess-us-east-1-<domain-id>" \
--regional-parameters '{
  "us-east-1": {
    "AZs": "us-east-1a,us-east-1b,us-east-1c,us-east-1d",
    "S3Location": "s3://amazon-sagemaker-<account-id>-us-east-1-<suffix>",
    "Subnets": "<subnet-1>,<subnet-2>,<subnet-3>,<subnet-4>",
    "VpcId": "<vpc-id>",
    "PermissionsBoundaryArn": "arn:aws:iam::<account-id>:policy/SMUSToolingBoundary"
  }
}' \
--region us-east-1

Step 6: Verify the configuration was applied

Confirm the permissions boundary ARN is now set in the blueprint configuration using the following command. Make sure to replace <domain-id> with the ID you noted in Step 2 and <tooling-bp-id> with the ID you noted in Step 3.

aws datazone get-environment-blueprint-configuration \
--domain-identifier <domain-id> \
--environment-blueprint-identifier <tooling-bp-id> \
--region us-east-1 \
--query "regionalParameters.\"us-east-1\".PermissionsBoundaryArn"

The output should return your boundary policy ARN:

"arn:aws:iam::<account-id>:policy/SMUSToolingBoundary"

Validating the configuration

After configuring the permissions boundary, in this section you will get instructions to create a new project to verify it works end to end and that the IAM roles created with the project actually include the permissions boundary.

Step 1: Select a project profile in enabled state

Use the following command to list project profiles configured in your domain. Make sure to replace <domain-id> with the ID you noted in Step 2 of the “Configuring the permissions boundary” section.

aws datazone list-project-profiles \
--domain-identifier <domain-id> \
--region us-east-1

Choose a project profile that has "status": "ENABLED". Note the id of any project profile returned in the previous command.

Step 2: Create a test project

Create a new project using the following command. Make sure to replace <domain-id> with the ID you noted in Step 2 of the “Configuring the permissions boundary” section and to replace <profile-id> with the project profile ID noted in Step 1 of this section.

aws datazone create-project \
--domain-identifier <domain-id> \
--name "PB-Validation-$(date +%Y%m%d-%H%M%S)" \
--project-profile-id <profile-id> \
--region us-east-1

Note the id (project ID) returned in the response. Wait for the Tooling blueprint to provision. This typically takes a minute or two. After provisioning completes, confirm that the validation project reaches the Active state, as shown in Figure 6.

SMUS Projects list showing the timestamped PB-Validation project in Active status after successful creation

Figure 6: Project created successfully with the permissions boundary configured

The Projects list shows the timestamped PB-Validation-* project with an Active status, confirming that project creation succeeded with the custom boundary configured.

Step 3: Verify the roles have the boundary attached

In this section you check that the IAM roles created with the project have the permissions boundary attached. Use the following commands to get the configuration for the IAM roles created with the project you just created. Replace <domain-id> and <project-id> with the values from the previous steps.

# Get the environment ID
ENV_ID=$(aws datazone list-environments \
--domain-identifier <domain-id> \
--project-identifier <project-id> \
--region us-east-1 \
--query "items[?name=='Tooling'].id" --output text)

# List IAM roles in the AWS CloudFormation stack
aws cloudformation describe-stack-resources \
--stack-name "DataZone-Env-${ENV_ID}" \
--region us-east-1 \
--query "StackResources[?ResourceType=='AWS::IAM::Role'].PhysicalResourceId" \
--output table

# Verify each role has the boundary
aws iam get-role \
--role-name "<role-name>" \
--query 'Role.PermissionsBoundary'

All three roles should return a response showing the permissions boundary ARN:

{
  "PermissionsBoundaryType": "Policy",
  "PermissionsBoundaryArn": "arn:aws:iam::<account-id>:policy/SMUSToolingBoundary"
}

You can also verify each role in the IAM console. Figure 7 shows the permissions boundary for the project user role.

IAM console Permissions tab showing SMUSToolingBoundary as the permissions boundary on datazone_usr_role

Figure 7: IAM console showing the permissions boundary attached to the datazone_usr_role

The datazone_usr_role Permissions tab displays SMUSToolingBoundary as its customer-managed permissions boundary.

Figure 8 confirms that the same boundary is attached to the Amazon Bedrock service role.

IAM console showing SMUSToolingBoundary as the permissions boundary on AmazonBedrockServiceRole

Figure 8: IAM console showing the permissions boundary attached to the AmazonBedrockServiceRole

The AmazonBedrockServiceRole also displays SMUSToolingBoundary as its customer-managed permissions boundary.

Figure 9 verifies the boundary on the third Tooling role, the Bedrock Lambda execution role.

IAM console showing SMUSToolingBoundary as the permissions boundary on AmazonBedrockLambdaExecutionRole

Figure 9: IAM console showing the permissions boundary attached to the AmazonBedrockLambdaExecutionRole

The AmazonBedrockLambdaExecutionRole likewise displays SMUSToolingBoundary, confirming that all three provisioned roles carry the boundary.

Step 4: Verify the boundary denies AI agent actions

In this section you verify the boundary actually denies AI agent actions. If you configured the boundary from the use case section earlier, the boundary blocks Data Notebooks and messages to the Data Agent, such as the Query Editor assistant. Any such attempt returns an access denied error. The project user role has the boundary attached, so even if the role’s identity policies grant the relevant APIs, the boundary’s explicit deny takes precedence.

To confirm, navigate to your project in SageMaker Unified Studio and test the following actions:

  1. Attempt to create a notebook – In the left sidebar, select Notebooks. Select Create notebook. The operation will fail because the permissions boundary prevents the datazone:CreateNotebook action (Figure 10).

Figure 10: Permissions boundary preventing creation of Data Notebooks

After the create action, Amazon SageMaker Unified Studio reports Failed to create notebook and identifies datazone:CreateNotebook as explicitly denied by SMUSToolingBoundary.

  1. Attempt to use Data Agent in the Query Editor – In the left sidebar, select Query Editor, then select the Chat with AI icon. The agent chat will fail to load because the permissions boundary blocks the APIs required by Data Agent (Figure 11).

Figure 11: Permissions boundary preventing using Data Agent on Query Editor

The Query Editor remains available, but the Agent panel reports “You don’t have access to Data Agent“. In this configured test, that message is the user-visible result of denying the Data Agent APIs. The screenshot itself doesn’t display the denied API or boundary ARN.

Important considerations

  • Immutable after project creation – The permissions boundary is set at provisioning time. Changing the boundary ARN on the blueprint configuration only affects new projects. Existing projects retain their original boundary.
  • Applies to all Tooling-provisioned roles – When PermissionsBoundaryArn is configured on the Tooling blueprint, SageMaker Unified Studio attaches the specified customer-managed permissions boundary to all three IAM roles created by that blueprint. It’s applied uniformly — you can’t selectively apply it to individual roles. No boundary is attached unless you configure one, so if your governance requires a boundary, set it explicitly and verify the resulting roles rather than assuming one is present by default.
  • Policy must exist – The IAM policy referenced by PermissionsBoundaryArn must exist in the account before project creation. If the policy is deleted or the ARN is invalid, provisioning will fail.
  • Tooling blueprint only – Among Amazon SageMaker Unified Studio provided blueprints, only the Tooling blueprint supports custom permissions boundaries. Other provided blueprints that create IAM roles (for example, the EmrOnEc2 blueprint) don’t currently support this feature. If your organization requires permissions boundaries on roles created by additional blueprints, you can build custom blueprints that include a permissions boundary configuration so you can extend this security control across your entire project infrastructure.

Clean up

To remove test resources, delete the test project from the SageMaker Unified Studio UI. On the project’s Overview page, choose the ⋮ (more actions) menu in the top-right and choose Delete project.

Figure 12: Deleting the test project from the project Overview page.

In the Delete project dialog, type confirm in the text box to acknowledge that the action is final, then choose Delete project. This permanently deletes the project and its underlying resources, and triggers an asynchronous AWS CloudFormation stack deletion.

Figure 13: Confirming project deletion.

To remove the boundary from future projects, re-run the put-environment-blueprint-configuration command from Step 5: Set the permissions boundary, but omit the PermissionsBoundaryArn field from the regional parameters. Because you backed up the original configuration in Step 4: Read the current configuration (tooling-bp-config-backup.json), you can reuse the exact same provisioningRoleArn, manageAccessRoleArn, enabledRegions, and regionalParameters values (AZs, S3Location, Subnets, VpcId) — just without PermissionsBoundaryArn — so the blueprint returns to provisioning roles with no permissions boundary.

Conclusion

With the custom permissions boundary feature for Amazon SageMaker Unified Studio, organizations can adopt Amazon SageMaker Unified Studio Tooling blueprints without compromising their IAM governance posture. By configuring a single parameter on the Tooling blueprint, all IAM roles provisioned by future projects automatically carry the specified permissions boundary. This satisfies SCPs that mandate a boundary on every role and gives administrators granular control over what the Tooling roles can do, for example disabling AI agent and notebook capabilities. Remember that the example boundary in this post is illustrative, because it scopes to the services SageMaker Unified Studio uses but is still coarse.

“I just updated the EnvironmentBlueprintConfiguration for the Tooling blueprint to include the new PermissionsBoundaryArn param. After that the blueprint provisioned successfully with the required permissions boundary attached to all the IAM roles, in line with our security policies. In the end it was a one-line change.”

— Nat Noordanus, Data Tech Lead at Nexthink

To get started, create your permissions boundary policy, configure it on the Tooling blueprint using the CLI, and create a project to verify the boundary is attached.

For more information, see the documentation for Amazon SageMaker Unified Studio, IAM permissions boundaries, and Service Control Policies.


About the authors

Sanjana Sekar

Sanjana Sekar

Sanjana is a Software Development Engineer on the Amazon SageMaker Unified Studio team. She is focused on improving Data Agent capabilities and the compute blueprints experience within SageMaker Unified Studio. Outside of work, she enjoys hiking and biking.

Luca Perrozzi

Luca Perrozzi

Luca is a Solutions Architect at AWS, based in Switzerland. He focuses on innovation topics at AWS, especially in Artificial Intelligence. Luca holds a PhD in particle physics and has 15 years of hands-on experience as a research scientist and software engineer.

Ganesh Sambandan

Ganesh Sambandan

Ganesh is a Senior Technical Account Manager at AWS, helping organizations adopt best practices for running secure, reliable and well-architected workloads on AWS. He works closely with strategic customers to accelerate the adoption of AI-driven cloud operations, enabling more effective DevOps practices, automation and operational excellence.

Stefano Sandona

Stefano Sandona

Stefano is a Senior Worldwide Specialist Solutions Architect for Big Data at AWS, helping customers build efficient, secure, and scalable data solutions.

Paolo Romagnoli

Paolo Romagnoli

Paolo is a Senior Solutions Architect at AWS who helps global energy organizations design and build data and AI enterprise solutions at scale.

Configure domain-level VPC networking in Amazon SageMaker Unified Studio

Post Syndicated from Prasad Nadig original https://aws.amazon.com/blogs/big-data/configure-domain-level-vpc-networking-in-amazon-sagemaker-unified-studio/

Enterprise operations teams that run domain-level VPC networking in Amazon SageMaker Unified Studio often support dozens of projects spanning data engineering, analytics, and machine learning (ML) teams. Each project requires private connectivity to internal databases, Amazon Simple Storage Service (Amazon S3) buckets, and AWS services. Without a domain-level Amazon Virtual Private Cloud (Amazon VPC) configuration, project owners coordinate with the networking team individually. This piecemeal approach leads to inconsistent subnet choices, missing VPC endpoints, connectivity failures that are hard to troubleshoot, and a network posture that is difficult to audit.

With domain-level VPC networking, you configure the network once, and all new projects get the right network immediately upon creation. In this post, you learn how to:

  • Configure SageMaker Unified Studio domain-level VPC networking.
  • Select subnets and security groups that provide multi-Availability Zone (multi-AZ) resilience.
  • Update projects that have no VPC to inherit the domain VPC, and understand when a project must be recreated instead.
  • Validate network connectivity from within a project.

In this post, you learn how to configure VPC networking for a SageMaker Unified Studio domain that uses AWS Identity and Access Management (IAM)-based authentication. You see how network components map to domain and project resources, and how to plan a configuration that balances security, connectivity, and operational simplicity.

Solution overview

Domain-level VPC networking provides a single network configuration that applies to all new projects in the domain. Projects automatically inherit the VPC settings, including subnets, security groups, and connectivity to AWS services through VPC endpoints. Existing projects are an exception and are handled separately (see Step 3).

The following diagram shows a single VPC with private subnets across two Availability Zones configured at the domain level, with data engineering, analytics, and ML projects all inheriting that configuration.

Architecture diagram showing domain-level VPC configuration in Amazon SageMaker Unified Studio with private subnets across two Availability Zones.

Figure 1: Domain-level VPC configuration in Amazon SageMaker Unified Studio. A single VPC with private subnets across two Availability Zones is configured at the domain level. All projects (data engineering, analytics, ML) inherit this configuration automatically

Key benefits of this approach:

  • Configure once, apply across projects: New projects inherit the domain VPC without manual intervention.
  • Consistent security posture: A single network boundary covers all data, analytics, and ML workloads.
  • Simplified auditing: One VPC to audit rather than one per project. Turn on VPC Flow Logs and review AWS CloudTrail events for network-level auditing.
  • Reduced operational overhead: Project teams start working immediately without submitting networking requests.

The following AWS services are used in this solution:

Prerequisites

Before configuring domain-level VPC networking, verify you have the following:

  • Domain administrator permissions for Amazon SageMaker Unified Studio.
  • An existing VPC with the following requirements:
    • At least two private subnets in different Availability Zones.
    • DNS hostnames and DNS support enabled.
    • At least five available IP addresses per expected Amazon SageMaker Unified Studio project. This is a baseline minimum. Workloads using AWS Glue, Amazon EMR, or Amazon Redshift Serverless consume additional elastic network interfaces (ENIs) per worker or node. We recommend /24 or larger subnets for production domains and forward-looking capacity planning based on your expected users and compute types. For detailed guidance, see How to set up a network-isolated VPC for Amazon SageMaker Unified Studio.
  • VPC endpoints configured for the AWS services your projects access (for example, Amazon S3, AWS Glue, Amazon SageMaker AI).
  • Private DNS enabled on all interface VPC endpoints (you must enable this so that service DNS names resolve to private IPs). If you use centralized VPC endpoints through AWS Resource Access Manager (AWS RAM) or AWS Transit Gateway, configure Amazon Route 53 Resolver inbound endpoints instead.
  • S3 gateway endpoint route table associations configured for all selected private subnets (without this, S3 access fails in subnets whose route table lacks the prefix-list route).
  • A security group (optional), if not provided, SageMaker Unified Studio creates one automatically.
  • The SageMakerStudioAdminIAMConsolePolicy managed policy (or equivalent permissions including ec2:Describe*, ec2:CreateSecurityGroup, and datazone:* actions) attached to the domain administrator IAM role. See SageMakerStudioAdminIAMConsolePolicy in the AWS Managed Policy Reference for the full permission set.

Note: The VPC must be in the same AWS Region as the domain.

For detailed guidance on VPC networking configuration, see Configure VPC networking for IAM-based domains in the SageMaker Unified Studio Administrator Guide.

VPC endpoint requirements

Because your subnets are private (no internet gateway route), compute resources access AWS services through VPC endpoints. At a minimum, configure the following interface and gateway endpoints (add Amazon Athena, AWS Lake Formation, or Amazon Redshift endpoints if you use them in your projects):

Endpoint Type Purpose
com.amazonaws.region.s3 Gateway S3 access for data storage
com.amazonaws.region.glue Interface AWS Glue job connectivity
com.amazonaws.region.sagemaker.api Interface SageMaker API calls
com.amazonaws.region.sagemaker.runtime Interface Model inference
com.amazonaws.region.logs Interface Amazon CloudWatch Logs
com.amazonaws.region.monitoring Interface Amazon CloudWatch metrics
com.amazonaws.region.sts Interface IAM role assumption
com.amazonaws.region.datazone Interface Amazon SageMaker Unified Studio service connectivity
com.amazonaws.region.ecr.api Interface ECR API calls (container image metadata)
com.amazonaws.region.ecr.dkr Interface ECR image layer pulls (Docker registry)
com.amazonaws.region.kms Interface AWS Key Management Service (AWS KMS) encryption/decryption operations

Note: Interface endpoints incur an hourly charge per Availability Zone plus data processing fees. Gateway endpoints (such as S3) have no hourly charge. Factor endpoint count and AZ spread into your cost estimate.

For a comprehensive list of all mandatory and optional VPC endpoints for a fully network-isolated setup, see How to set up a network-isolated VPC for Amazon SageMaker Unified Studio. For current pricing details, see AWS PrivateLink pricing.

Note: Review your account’s service quotas for interface VPC endpoints per VPC (default 50) and ENIs per Region before scaling. Request increases through Service Quotas if needed.

Solution walkthrough

The following steps walk you through configuring the domain VPC and validating it, from signing in to the console through confirming private connectivity from a project.

Step 1: Sign in and navigate to networking settings

  1. Sign in to the AWS Management Console as your Amazon SageMaker Unified Studio domain administrator (the IAM role designated as the domain login role).
  2. Open the Amazon SageMaker console.
  3. Use the Region selector in the top navigation bar to select the Region where your domain exists.
  4. On the Amazon SageMaker Unified Studio landing page, choose Open to launch your IAM-based domain.

The following screenshot shows the Amazon SageMaker Unified Studio landing page, where you choose Open to launch the domain.

Amazon SageMaker Unified Studio landing page with the Open button to launch the IAM-based domain.

Figure 2: Amazon SageMaker Unified Studio landing page with the Open button to launch the IAM-based domain

  1. From the navigation pane, choose Domain management.

The following screenshot shows Domain management in the navigation pane.

Navigation pane showing Domain management link in Amazon SageMaker Unified Studio.

Figure 3: Domain management on navigation pane

Note: Access to the domain administration page is restricted to the IAM role specified as the domain login role during domain creation.

Step 2: Add VPC configuration

  1. In the navigation pane, choose Settings. In the Networking in this account section, choose Add VPC.

The following screenshot shows the Networking in this account section with the Add VPC button.

Domain management Settings page showing the Networking in this account section with Add VPC button.

Figure 4: Domain management Settings page showing the Networking in this account section to add a VPC

  1. For VPC, select the VPC with connectivity to your compute, database, and storage resources. If no VPC exists, choose Create VPC to provision one using AWS CloudFormation.
  2. For Subnets, select a minimum of two private subnets in different Availability Zones.
  3. (Optional) For Security group, select a security group to control inbound and outbound traffic. If you don’t choose one, SageMaker Unified Studio creates one automatically.
  4. Choose Save.
  5. Verify the VPC configuration status shows Ready in the Networking in this account section.

The following screenshots show the Add VPC dialog and the resulting Ready status in the Networking in this account section.

Add VPC dialog with fields for VPC, subnets, and security group selection.

Figure 5: Add VPC dialog with fields for VPC, subnets, and security group selection

VPC configuration status showing Ready in the Networking in this account section.

Figure 6: VPC configuration status showing Ready in the Networking in this account section

Note: IAM-based domains support only one VPC configuration at a time. AWS IAM Identity Center-based domains can have a VPC per Region. For details, see Configure VPC networking for IAM-based domains in the SageMaker Unified Studio Administrator Guide.

New projects created in the domain now automatically use the saved VPC configuration. Existing projects are an exception. See Step 3 to update them.

Step 3: Update existing projects

Existing projects don’t automatically inherit the domain VPC configuration. How you apply the new settings depends on the project’s current state:

Projects with no VPC configured – Update in place to adopt the domain VPC. See the following steps.

Projects that already have a VPC – These can’t be switched to a different VPC configuration. To adopt the domain VPC:

  1. Create a new project (which inherits the domain VPC automatically).
  2. Recreate connections in the new project.
  3. Migrate assets from the old project.
  4. Back up any data you need, then delete the original project.

Because recreation can disrupt in-progress work and doesn’t migrate project data automatically, schedule this as a planned maintenance window.

To update a project that currently has no VPC configured:

  1. From the domain administration page, choose Projects in the navigation pane.
  2. Choose the project you want to update.
  3. On the project detail page, a banner appears: “Configurations have changed. Please update this project to access the latest configuration.”
  4. In the banner, choose Update.
  5. Confirm the update when prompted.

Repeat this process for each existing project that should use the domain VPC. The following screenshot shows the project detail page with the configuration update banner.

Project detail page showing the update banner for VPC configuration changes.

Figure 7: Project detail page showing the configuration update banner

Step 4: Validate connectivity

After configuring the domain VPC and updating your projects, verify connectivity. Compute resources should have private connectivity to AWS services through the VPC, without any additional project-level network configuration.

Create a notebook in one of your projects as shown in the following figure and run the following code:

Creating a notebook in a SageMaker Unified Studio project to validate VPC connectivity.

Figure 8: Creating a notebook in a SageMaker Unified Studio project to validate VPC connectivity

Requirements: Python 3.8+, Boto3 1.26 or later. Run in a notebook within your SageMaker Unified Studio project.

import boto3
import socket
import ipaddress

def validate_vpc_connectivity():
    """Validate that the project has private connectivity to AWS services
    through the domain-level VPC configuration."""

    results = {}
    region = boto3.session.Session().region_name
    if not region:
        raise RuntimeError('Could not determine AWS Region. Run this notebook inside a SageMaker Unified Studio project.')

    # Test Amazon S3 access via VPC endpoint
    try:
        s3 = boto3.client('s3')
        response = s3.list_buckets()
        results['S3'] = f"[PASS] Accessible ({len(response['Buckets'])} buckets)"
    except Exception as e:
        results['S3'] = f"[FAIL] Failed: {e}"

    # Test AWS Glue access via VPC endpoint
    try:
        glue = boto3.client('glue')
        dbs = glue.get_databases()
        results['Glue'] = f"[PASS] Accessible ({len(dbs['DatabaseList'])} databases)"
    except Exception as e:
        results['Glue'] = f"[FAIL] Failed: {e}"

    # Test STS (role assumption through VPC endpoint)
    try:
        sts = boto3.client('sts')
        identity = sts.get_caller_identity()
        results['STS'] = f"[PASS] Accessible (Account: {identity['Account']})"
    except Exception as e:
        results['STS'] = f"[FAIL] Failed: {e}"

    # Verify interface endpoint resolves to private IP
    try:
        sts_endpoint = f"sts.{region}.amazonaws.com"
        addr_info = socket.getaddrinfo(sts_endpoint, 443, family=socket.AF_INET)
        ip = addr_info[0][4][0]
        is_private = ipaddress.ip_address(ip).is_private
        if is_private:
            results['DNS Resolution'] = f"[PASS] Private IP ({ip}) (traffic stays on AWS network)"
        else:
            results['DNS Resolution'] = f"[WARN] Public IP ({ip}) - check VPC endpoint config"
    except Exception as e:
        results['DNS Resolution'] = f"[FAIL] Failed: {e}"

    # Print results
    print("-" * 40)
    print("Domain VPC Connectivity Validation")
    print("-" * 40)
    for service, status in results.items():
        print(f" {service}: {status}")
    print("-" * 40)
    print(f"\n Region: {region}")

    # Check if all tests passed
    all_passed = all("[PASS]" in status for status in results.values())
    has_warn = any("[WARN]" in status for status in results.values())
    if all_passed:
        print(f"\n [PASS] All services accessible via private VPC endpoints.")
        print(f" This project inherited its network configuration")
        print(f" from the domain without per-project setup.")
    elif has_warn and all("[PASS]" in s or "[WARN]" in s for s in results.values()):
        print(f"\n [WARN] Services are reachable, but DNS resolves to public IPs.")
        print(f" Verify that Private DNS is enabled on your interface VPC endpoints.")
    else:
        print(f"\n [FAIL] Some services are not reachable.")
        print(f" Check that VPC endpoints are configured and security")
        print(f" groups allow outbound traffic on port 443.")

validate_vpc_connectivity()

Expected output when VPC is correctly configured:

Successful validation output showing all services accessible through private VPC endpoints.

Figure 9: Successful validation output showing all services accessible through private VPC endpoints

If any service shows a failure, one common cause is security groups preventing traffic on port 443 to the VPC endpoint. Other causes include missing VPC endpoints, incorrect route table entries, or DNS resolution issues. For more information, see Configure VPC networking for IAM-based domains in the SageMaker Unified Studio Administrator Guide.

Note: An AccessDenied error indicates the request reached the service. Connectivity is working, but IAM permissions need adjustment (for example, the S3 test requires s3:ListAllMyBuckets, which some project roles lack). A timeout or connection error points to a networking problem (missing endpoint, route, or security group rule). The following screenshot shows the validation output when VPC endpoints are missing, where the affected services report timeout errors.

Validation output when VPC endpoints are not configured showing timeout errors.

Figure 10: Validation output when VPC endpoints are not configured. Timeout errors indicate missing endpoints

The security group applied at the domain level controls network access for all projects. To review or tighten the rules:

  1. Navigate to the Amazon VPC console.
  2. Choose Security groups and choose the security group shown in your domain’s Networking settings.
  3. Review the Inbound rules and Outbound rules tabs.

By default, the auto-created security group allows all outbound traffic on port 443 (HTTPS) to reach AWS services through VPC endpoints. Consider restricting outbound rules to only the specific VPC endpoint security groups for least-privilege access. Additionally, make sure your VPC endpoint security groups allow inbound TCP 443 from the domain security group or subnet CIDRs. For distributed compute services (AWS Glue, Amazon EMR), add a self-referencing inbound rule to allow worker-to-worker communication.

Updating VPC configuration

After the initial setup, you can modify the VPC configuration to change the VPC, subnets, or security group:

  1. From the domain administration page, choose Settings in the navigation pane.
  2. In the Networking in this account section, under the Actions column, choose Update.
  3. Update the VPC, subnets, or security group as needed.
  4. Choose Update.

The following screenshot shows the Update VPC dialog, where you modify the VPC, subnets, or security group.

Settings page with the Actions menu showing Update and Remove options for VPC configuration.

Figure 11: Update VPC dialog showing the option to modify VPC, subnets, or security group for the domain

Important: Updating the VPC does not affect already provisioned resources. Newly created resources in projects use the updated VPC. Existing projects that already have a VPC keep their original settings and must be recreated to adopt the change. Projects with no VPC can be updated in place (see Step 3).

Clean up

To remove the VPC configuration from your domain:

  1. From the domain administration page, choose Settings in the navigation pane.
  2. In the Networking in this account section, choose the Actions menu (⋮) and choose Remove.

The following screenshot shows the Actions menu with the Remove option.

Actions menu in the Networking in this account section showing the Remove option.

Figure 12: Actions menu in the Networking in this account section showing the Remove option

If you created a dedicated VPC for this walkthrough and no longer need it:

  • Delete the VPC and associated resources (subnets, VPC endpoints, security groups) from the Amazon VPC console. Before deleting, remove the domain VPC configuration and make sure all project resources are terminated. Active projects create ENIs that block VPC and subnet deletion.
  • If you used an AWS CloudFormation template to create the VPC, delete the stack to remove all resources cleanly. Open the AWS CloudFormation console and delete the stack.

Note: Removing the domain VPC configuration does not retroactively change projects that already have VPC applied. Those projects retain their existing network configuration. New projects created after removal do not have a VPC configured.

Conclusion

In this post, we showed how to configure domain-level VPC networking in Amazon SageMaker Unified Studio. A single domain-level VPC eliminates per-project networking overhead, enforces a consistent security posture, and simplifies compliance auditing.

Key takeaways:

  • Domain-level VPC is a one-time configuration that automatically applies to all new projects.
  • Projects with no VPC can be updated in place. Projects that already have a VPC must be recreated to adopt a changed configuration.
  • Private subnets with VPC endpoints provide secure, private connectivity to AWS services without traversing the public internet.

As next steps, consider:

  • Reviewing your auto-created security group rules and tightening them for least-privilege access.
  • Adding VPC endpoints for additional AWS services as your projects’ needs evolve.
  • Monitoring subnet IP address utilization to plan capacity as you add more projects. Use the AvailableIpAddressCount Amazon CloudWatch metric for your subnets to track utilization and set alarms.

For more information, see Configure VPC networking for IAM-based domains in the Amazon SageMaker Unified Studio Administrator Guide.

 


About the authors

Prasad Nadig

Prasad Nadig

Prasad is a Senior Analytics Specialist Solutions Architect at Amazon Web Services (AWS), specializing in large-scale data analytics and AI. Prasad partners with customers to design, migrate, and modernize their analytics platforms on AWS into scalable, cost-effective solutions, with deep expertise in data lakes, data warehousing, distributed processing, and performance tuning at petabyte scale.

Amit Shyam Jaisinghani

Amit Shyam Jaisinghani

Amit is a Software Engineer on the SageMaker Studio team at Amazon Web Services, and he earned his Master’s degree in Computer Science from Rochester Institute of Technology. Since joining Amazon in 2019, he has built and enhanced several AWS services, including Amazon WorkSpaces and Amazon SageMaker Studio. Outside of work, he explores hiking trails, plays with his two cats, Missy and Minnie, and enjoys playing Age of Empire.

Arun Shanmugam

Arun Shanmugam

Arun is a Senior Analytics Solutions Architect at AWS, with a focus on building modern data architecture. He has been successfully delivering scalable data analytics solutions for customers across diverse industries. Outside of work, Arun is an avid outdoor enthusiast who actively engages in CrossFit, road biking, and cricket.

Query unstructured data in Amazon SageMaker Catalog using generative AI

Post Syndicated from Nishchai JM original https://aws.amazon.com/blogs/big-data/query-unstructured-data-in-amazon-sagemaker-catalog-using-generative-ai/

Each day, businesses generate massive amounts of unstructured data, such as PDFs, images, email, customer feedback, and medical reports. But knowing data exists isn’t enough. You need to find it, access it, and extract answers from it fast. In Part 1 of this series, you saw how to set up the producer side of the pipeline: using Amazon Textract and Anthropic Claude on Amazon Bedrock to extract and enrich metadata, and then publish those enriched assets to Amazon SageMaker Catalog so your organization can discover them.

In this post, you take the next step: the consumer side. You sign in as a data consumer, search for and subscribe to the enriched unstructured data assets, and then query them using two approaches. The first is a no-code chat agent for natural language queries. The second is Amazon Bedrock model inference for programmatic access. By the end of this post, you will know how to unlock the business knowledge inside your unstructured data and make it available to analysts and application engineers alike.

Solution overview

This post continues the two-part series architecture, where Amazon SageMaker Catalog acts as the central hub connecting data producers and consumers through a publish-subscribe model.

The consumer workflow picks up after the producer has enriched and published the unstructured data assets. As a consumer, you will:

  • Sign in to your SageMaker Unified Studio consumer project and search the catalog using keywords from the enriched metadata README.
  • Subscribe to the published Amazon Simple Storage Service (Amazon S3) asset and get the subscription approved by the producer.
  • Interact with the subscribed data through two options:
    • Option 1 – A no-code chat agent for natural language queries (NLQs), ideal for data analysts and business users.
    • Option 2 – Amazon Bedrock model inference for programmatic NLQ integration, suited for application engineers building data-driven applications.

The following diagram illustrates the consumer workflow in this solution. The consumer (1) signs in to SageMaker Unified Studio, (2) searches the Amazon SageMaker Catalog for enriched unstructured data assets using keywords from the AI-generated metadata, (3) subscribes to the S3 data asset and receives approval from the producer, and then (4) queries the data using either the Amazon Bedrock chat agent app (Option 1) or Amazon Bedrock model inference through a Jupyter notebook (Option 2).

With both a no-code and a programmatic path, consumers across different roles, from analysts to engineers, can query data in the way that fits their workflow, while the SageMaker Catalog approval workflow maintains governed access throughout.

Consumer workflow architecture: sign in to SageMaker Unified Studio, search the SageMaker Catalog, subscribe to the S3 asset with producer approval, then query with the Amazon Bedrock chat agent or model inference

Figure 1: Consumer workflow for the publish-subscribe solution

Prerequisites

Before you begin, make sure you have completed all steps in Part 1 of this series, including:

Consume published data from the consumer project

In this section, you sign in as a consumer user in the SageMaker Unified Studio consumer project. You then subscribe to the S3 bucket by searching for a keyword that is part of the README published in Part 1.

  1. Sign in to the consumer project and search for the keyword emergency, which was added to the README file during publishing. The search returns the enriched asset that the producer published in Part 1.

    SageMaker Unified Studio catalog search for the emergency keyword, returning the enriched asset published in Part 1

    Figure 2: Catalog search results for the emergency keyword

  2. Choose the asset from the results to view its details, including the AI-generated business metadata, glossary terms, and README content. Then choose Subscribe.

    Asset details page showing AI-generated business metadata, glossary terms, and README content, with the Subscribe button

    Figure 3: Asset details with AI-generated metadata and the Subscribe option

  3. Enter analysis as the Reason for request in the Comment section, then choose Request.

    Subscription request dialog with analysis entered as the reason for request in the Comment box

    Figure 4: Subscription request with the reason for request entered

  4. Sign back in to the producer project (unstructured-producer-project) to approve the subscription request.
  5. After approval, return to the consumer project and confirm that the subscribed asset now appears under Manage, Assets, Subscribed assets.

    Consumer project Subscribed assets list confirming the approved subscription

    Figure 5: Approved subscription under the Subscribed assets tab

With the subscription approved, you can now access the enriched unstructured data through two approaches.

Option 1: As a data or business analyst, you can use the Amazon Bedrock chat agent app for natural language queries.

Option 2: As an application engineer, you can use Amazon Bedrock model inference for programmatic natural language queries.

Let’s explore both options.

Option 1: Amazon Bedrock chat agent app

The Amazon Bedrock chat agent app gives you a no-code, conversational interface to query your enriched unstructured data using natural language. As a data analyst or business user, you can ask questions in plain English. You get answers grounded in the documents your organization has ingested, without writing any code. For production workloads, especially in sensitive domains such as healthcare, you can apply Amazon Bedrock Guardrails to add content filtering and grounding validation to your model responses.

Data scientists and application engineers can also extend these capabilities by integrating the chat agent app APIs into custom applications, so users can interact with unstructured Amazon S3 data programmatically.

To set up the Amazon Bedrock chat agent app on your subscribed dataset, complete the following steps.

Prerequisite: Add the S3 data location.

Before creating the chat agent app, you need to add the S3 location of your subscribed data as a registered location in your project.

  1. Choose the Data tab in Overview.
  2. Choose the S3 bucket, and then choose Add to add the S3 location.

    Data tab in the project Overview with the S3 bucket selected and the Add button to register the S3 location

    Figure 6: Adding the S3 location from the Data tab

  3. On the S3 location page, provide the following details:
    • Add a name: producerprojectdata.
    • Add the producer’s S3 path as a new S3 location: s3://amzn-sagemaker-bucket-<domain-id>-<project-id>/medical/.

    Note: You can get the S3 location details from the technical name of your subscribed asset.

    • Choose the AWS Region, and then choose Add data to add this as a new location.

    Note: Make sure the AWS Region you select supports the Amazon Bedrock foundation models used later in this post. For a list of available models by Region, see Supported Regions and models for Amazon Bedrock.

    S3 location page with the location name, producer S3 path, and AWS Region entered before choosing Add data

    Figure 7: S3 location details and AWS Region selection

    Note: Make sure to select only the PDF files within the S3 path for the data source.

    Data source selection showing only the PDF files within the S3 path selected

    Figure 8: Selecting the PDF files as the data source

After the location is added, it appears as a selectable S3 location when creating a knowledge base in AI Apps.

Complete the following steps to configure the chat agent app:

  1. In the left navigation pane, under Generative AI, choose AI Apps.
  2. In the Build section of the page, choose Chat agent.

    AI Apps Build section with Chat agent selected in the left navigation under Generative AI

    Figure 9: Choosing Chat agent in the AI Apps Build section

  3. Expand the Data tab to create a knowledge base with your S3 bucket. On the Create a new knowledge base page, enter the following:
    • Add a name: MedicalKB.
    • Add a description: Knowledge base built from subscribed medical S3 data assets. Contains medical documents used to provide grounded, context-aware responses to medical domain queries.
    • Choose the data source. You will see the S3 bucket that you added in the previous step.
    Create a new knowledge base page with the MedicalKB name, description, and the added S3 bucket as the data source

    Figure 10: Creating the MedicalKB knowledge base from the S3 data source

  4. Choose your embedding model. You can leave the default settings and choose Create. It might take 10–15 minutes to create the knowledge base, depending on file sizes.
  5. After the knowledge base is created, on the Chat agent page:
    • Choose your preferred model from the Model menu (you can switch between different large language models as needed).
    • Under Data, choose your published S3 bucket as the knowledge base.
    • Begin interacting with the agent by entering questions in the Enter prompt field.
    Chat agent page with a model selected and the MedicalKB knowledge base chosen, ready to enter a prompt

    Figure 11: Chat agent page with the model and knowledge base selected

For example, entering “Which age groups had the highest rates of emergency department visits for tooth disorders?” returns an answer grounded in the enriched dental dataset published in Part 1.

The chat agent uses the enriched README metadata along with the underlying documents to surface contextually relevant answers. Analysts can explore unstructured content without needing to know where the data lives or how it’s structured.

Option 2: Natural language queries using Amazon Bedrock model inference

This option demonstrates how to use Amazon Bedrock model inference to query subscribed data using natural language. You can integrate this capability with external chat applications so users can run natural language queries through Amazon Bedrock.

  1. In your consumer project, choose Manage, Assets from the bottom of the left navigation pane. On the Subscribed tab, choose your subscribed S3 asset. Under Actions, choose Open JupyterLab notebook.

    Subscribed S3 asset Actions menu with Open JupyterLab notebook selected in the consumer project

    Figure 12: Opening the JupyterLab notebook from the subscribed asset

  2. This opens the JupyterLab notebook environment. Upload the s3_document_consumer_v2.ipynb notebook and run all the cells. You can download the notebook from s3_document_consumer_v2.ipynb.Note: The project role requires permissions for Amazon S3, Amazon Textract, and Amazon Bedrock. If you followed Part 1, you might already have these policies attached. For details on the required policies and guidance, see the prerequisites in Part 1.
  3. Review the notebook cells.
    JupyterLab notebook cells with the final cell showing a sample question answered by Amazon Bedrock

    Figure 13: Sample question answered by Amazon Bedrock in the notebook

    In the final cell, you find a sample question that Amazon Bedrock answers: “Which primary payer types (Medicare, Medicaid, private insurance, and so on) account for the highest proportion of dental-related emergency department visits?”

    Amazon Bedrock processes the question against the enriched content in the S3 bucket and returns a grounded answer. You can replace this sample question with any query relevant to your documents.

The Amazon Bedrock model inference approach gives you programmatic control, making it possible to embed natural language query capabilities directly into your existing data applications and business intelligence tools.

Clean up

To avoid ongoing charges, make sure to delete the resources used in this solution immediately after completing the walkthrough. The primary cost drivers are SageMaker Unified Studio notebook instances, Amazon Bedrock model inference calls, and Amazon S3 storage.

  1. Stop SageMaker Unified Studio resources:
    • Close running notebooks.
    • Stop running notebook instances.
    • Shut down unused kernels.

    Note: Running notebook instances continue to incur charges even when not in use.

  2. Clean Amazon S3 storage:
    • Delete temporary files created during processing.
    • Remove uploaded test documents that are no longer needed.

    Note: Although Amazon S3 costs are minimal, large volumes of data can accumulate significant charges, so it’s best to remove unneeded data.

Conclusion

In this post, you saw how to consume and query the enriched unstructured data assets published in Part 1 of this series. By subscribing to assets through the Amazon SageMaker Catalog publish-subscribe model, you can discover, access, and interact with your organization’s unstructured data, whether through the no-code chat agent or Amazon Bedrock model inference.

Together, both parts of this series show you how to build a comprehensive pipeline that transforms raw unstructured documents into governed, queryable knowledge assets. The combination of Amazon Textract for extraction, Amazon Bedrock for intelligent summarization and NLQ, and Amazon SageMaker Catalog for governance and discoverability means your teams can focus on extracting business insights rather than managing infrastructure.

To continue your Amazon SageMaker journey, see the following resources:


About the authors

Nishchai JM

Nishchai JM

Nishchai is an Analytics and generative AI Specialist Solutions Architect at Amazon Web Services. He specializes in building larger scale distributed applications and helps customers modernize their workloads on AWS. He thinks Data is new oil and spends most of his time deriving insights from data.

KiKi Nwangwu

KiKi Nwangwu

KiKi is an Analytics and generative AI Specialist Solutions Architect at AWS. She specializes in helping customers architect, build, and modernize scalable data analytics and generative AI solutions. She enjoys traveling and exploring new cultures.

Narendra Gupta

Narendra Gupta

Narendra is a Sr. Specialist Solutions Architect for Data & AI (Analytics) at AWS. He works with customers to design data-driven solutions and has deep expertise in data governance and cataloging.

Aditya Edara

Aditya Edara

Aditya is a Support Engineer at AWS. He serves as a Subject Matter Expert in AWS Analytics services, specializing in Amazon EMR and AWS Glue. Aditya provides expert guidance and technical support to enterprise and strategic customers, helping them optimize data analytics solutions.

Discover and govern Snowflake data using SageMaker Unified Studio

Post Syndicated from Marco Duarte original https://aws.amazon.com/blogs/big-data/discover-and-govern-snowflake-data-using-sagemaker-unified-studio/

Many organizations operate in hybrid data environments where critical assets live in Snowflake while analytics workloads run on AWS, which can create governance gaps, discovery friction, and duplicated efforts when the two aren’t connected.

With Amazon SageMaker Unified Studio, you can govern data across Snowflake and AWS through its integrated catalog and AWS Glue Data Quality, a capability of AWS Glue. You connect directly to Snowflake tables without moving data, apply quality rules using AWS Glue Visual ETL, and publish validated assets to Amazon SageMaker Catalog, maintaining consistent governance across your entire distributed data estate.

Without this integration, cataloging Snowflake data requires building extraction pipelines, often taking days. With SageMaker Unified Studio connected to Snowflake, you can query, catalog, and validate the quality of federated data in 5–15 minutes. No data replication or custom ETL code required.

In this post, we show you how to connect Snowflake to Amazon SageMaker Unified Studio, register data assets in Amazon SageMaker Catalog, configure data quality validation using AWS Glue Visual ETL, and publish assets for unified collaboration. By following these steps, you enrich federated assets with data quality scores so that consumers across your organization can discover and trust the data, all while keeping it in Snowflake.

Solution overview

This solution integrates Snowflake with Amazon SageMaker Unified Studio for centralized data cataloging and quality validation.

The architecture uses an AWS Glue connection to federate the Snowflake catalog into Amazon SageMaker Unified Studio. Tables become available in the project catalog without complex storage configurations. You can query data directly using SQL analytics, publish datasets to Amazon SageMaker Catalog for organization-wide discovery, and apply data quality rules through AWS Glue Visual ETL pipelines.

The workflow consists of the following steps:

Architecture diagram: Snowflake federated into SageMaker Unified Studio through AWS Glue, with data quality validation and publishing to SageMaker Catalog

Figure 1: Architecture for federating Snowflake into SageMaker Unified Studio and validating data quality

  1. Snowflake connection creation on Amazon SageMaker Unified Studio — Amazon SageMaker Unified Studio uses an AWS Glue connection to federate Snowflake tables and views into its open data lakehouse architecture. The federated catalog entry is registered in AWS Glue Data Catalog and governed by AWS Lake Formation for centralized access control, without moving data out of Snowflake.
  2. Federate Snowflake tables into the Amazon SageMaker publisher project — The Amazon SageMaker publisher project discovers the federated Snowflake tables through the AWS Glue Data Catalog integration.
  3. Publish the dataset to Amazon SageMaker Catalog — The publisher project publishes the dataset as a governed asset to the Amazon SageMaker Catalog, making it discoverable for data consumers across the organization.
  4. Validate data quality — AWS Glue Data Quality runs validation rules against the federated Snowflake data and publishes the data quality results directly to the corresponding asset in Amazon SageMaker Catalog.
  5. Consume data — Users access Snowflake data through two paths:
    1. Publisher project users — Query data with SQL Analytics — Users in the publisher project can query the Snowflake data directly using Amazon SageMaker Unified Studio SQL Analytics for interactive exploration and analysis, without copying or moving data.
    2. Consumer project users — Discovery and subscription through SageMaker Catalog — Other Amazon SageMaker consumer projects discover the published asset in the Amazon SageMaker Catalog, subscribe to it, and consume the data for their analytics and machine learning workloads.

Prerequisites

To follow along, you need:

  • An active Snowflake account with administrator access.
  • Tables or views created within a schema inside a Snowflake database.
  • An Amazon SageMaker Unified Studio and project created.
  • An Amazon Simple Storage Service (Amazon S3) bucket for AWS Glue assets.
  • Appropriate AWS Identity and Access Management (IAM) permissions configured (Amazon SageMaker Catalog is built on Amazon DataZone, so the IAM actions use the datazone: prefix.)

Your AWS Glue job execution role requires specific permissions to interact with Amazon SageMaker Catalog.

Required IAM policies for the AWS Glue job role

1. Amazon SageMaker Catalog search and listing permissions: Attach a policy that allows the AWS Glue job to search and list assets in Amazon SageMaker Catalog.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "datazone:SearchListings",
        "datazone:GetListing",
        "datazone:ListDomains",
        "datazone:GetDomain"
      ],
      "Resource": "arn:aws:datazone:<REGION>:<ACCOUNT_ID>:domain/<DOMAIN_ID>"
    }
  ]
}

2. Amazon SageMaker Catalog time series data posting permissions: Add permissions to post data quality metrics:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "datazone:PostTimeSeriesDataPoints",
        "datazone:GetAsset",
        "datazone:ListAssetRevisions"
      ],
      "Resource": "arn:aws:datazone:<REGION>:<ACCOUNT_ID>:domain/<DOMAIN_ID>"
    }
  ]
}

Configure the AWS Glue job role as an Amazon SageMaker domain user

Configure the IAM role used by your AWS Glue job as a domain user. In the Amazon SageMaker console, navigate to your domain, choose Access management, and add the AWS Glue job execution IAM role as a domain user.

Project-level permissions

Add the AWS Glue job execution role as a project member with Owner permissions. Navigate to your project, go to Project settings > Members, and add the role.

For more information about IAM roles for AWS Glue, see the AWS Glue security documentation. For Amazon SageMaker Unified Studio permissions, refer to the Amazon SageMaker Unified Studio administrator guide.

Querying Snowflake datasets from Amazon SageMaker Unified Studio

The following sections walk you through connecting Snowflake to Amazon SageMaker Unified Studio and running data quality validation with results displayed in Amazon SageMaker Catalog.

Identifying information in Snowflake

First, gather your Snowflake connection details. You need a Snowflake account with tables or views created at the schema level within a database.

To obtain Snowflake connection information:

  1. Navigate to your Snowflake environment and sign in with administrator credentials.

    Snowflake sign-in screen for administrator credentials
  2. Choose your user account and choose Connect a tool to Snowflake.

  3. Note the Account/Server URL displayed on the screen.
  4. Choose the Config File tab, select values for Warehouse, Database, and Schema, and copy these values for use in the next section.

Creating the connection in Amazon SageMaker Unified Studio

The Add Connection feature stores Snowflake connectivity details including credentials, server, and database information. Amazon SageMaker Unified Studio uses this connection to federate the Snowflake catalog through AWS Glue, so you can query data within minutes of setup.

You need an Amazon SageMaker Unified Studio domain and a project, which acts as a data producer project.

To create the Snowflake connection:

  1. In your Amazon SageMaker Unified Studio project, go to Overview.

    SageMaker Unified Studio project Overview page
  2. Choose Data.

    Data option in the SageMaker Unified Studio project navigation
  3. Choose + Add, then choose Add Connection.

    Add menu in SageMaker Unified Studio with the Add Connection option
    Add Connection panel in SageMaker Unified Studio
  4. Choose Next.
  5. Select Snowflake and choose Next.

    Connection type selection showing Snowflake in SageMaker Unified Studio
  6. Complete the connection details:
    • Name: snowflake-connection.
    • Description (Optional): Enter a description for your connection.
    • Host: Your Snowflake account URL (for example, XXXXXXXXX-XXX000000.snowflakecomputing.com).
    • Port: 443.
    • Database: Your database name (for example, sm_demo).
    • Warehouse: Your warehouse name (for example, COMPUTE_WH).
    • Schema: Your schema name (for example, demo).
    • Additional Properties:
      • Register in AWS Glue Data Catalog: Turn on checkbox.
      • Case conflict handling: Select the option based on Snowflake naming syntax.
    • Authentication:
      • Username: Your Snowflake username.
      • Password: Your Snowflake password.
    Snowflake connection details form with name, host, port, database, warehouse, and schema fields
    Connection form showing authentication and AWS Glue Data Catalog registration options
  7. Choose Add Data.

After creating the connection, wait a few minutes for the federated connection to be established. Search within Amazon SageMaker Unified Studio for the database and created objects.

Federated Snowflake database and objects appearing in SageMaker Unified Studio search

Federated Snowflake tables registered in the AWS Glue Data Catalog

With the Snowflake connection established and the federated tables registered in AWS Glue Catalog, you’re now ready to query Snowflake data directly from Amazon SageMaker Unified Studio, without moving or replicating any data.

Query results from a federated Snowflake table in the SageMaker Unified Studio query editor

How federated queries work

When you run a query in the Amazon SageMaker Unified Studio query editor against a federated Snowflake table, Amazon Athena runs the request. Athena is the underlying query engine integrated into Amazon SageMaker Unified Studio. Athena reads the table definition from AWS Glue Catalog, connects to Snowflake through the established connection, and pushes the query down for execution. Athena returns results directly to the query editor while Snowflake processes the data in place, and only the query results travel across the connection. Amazon SageMaker Unified Studio doesn’t copy data to S3 or any intermediate storage.

After you’ve validated that queries return the expected results, the next step is to publish this dataset to Amazon SageMaker Catalog, making it discoverable and shareable across your organization.

Publishing Snowflake datasets to the SageMaker Catalog

Now that your Snowflake connection is configured, you can publish your datasets to the Amazon SageMaker Catalog, making them discoverable and shareable across your organization.

Creating data assets in SageMaker Catalog

Data assets in Amazon SageMaker Catalog are the cataloged representation of your data resources. They help teams discover, govern, and share data across your organization.

In this section, you create a data asset associated with a Snowflake table. This process transforms a technical Snowflake table into a cataloged resource enriched with business metadata.

To create a data source:

  1. In your Amazon SageMaker Unified Studio project, go to Manage.

    Manage tab in the SageMaker Unified Studio project
  2. Choose Data Sources.
  3. Choose Create Data Source.

  4. Select the AWS Glue option.

    Data source type selection showing the AWS Glue option
  5. Turn on the Import data lineage checkbox and select the connection: project.default_lakehouse.

    Data source configuration with Import data lineage and the project.default_lakehouse connection selected
  6. Complete the form and choose Next:
    • Catalog: Select Enter the catalog name and enter snowflake-connection.
    • Database name: Enter your database name (for example, movies).
    • Table selection criteria: Enter * for all tables in the database, or enter a specific table name.
    Data source form showing catalog name, database name, and table selection criteria
  7. Keep the default options and choose Next until you reach the summary screen.

    SageMaker Unified Studio data source configuration summary screen
    Data source review screen before creation
  8. Review your settings and choose Create.

To extract metadata and publish assets:

  1. Choose Run to start extracting metadata from AWS Glue Data Catalog.

    Data source detail page with the Run option to extract metadata from the AWS Glue Data Catalog
  2. Wait for the run to complete.
  3. Go to Assets to view the Asset Inventory.

    Asset inventory in SageMaker Catalog after the data source run completes

The following screenshot shows the asset inventory after the data source run completes.

  1. Choose an asset to view its details.

    Asset detail page in SageMaker Catalog showing the Snowflake table metadata

At this point, you can enrich the business context by choosing Generate Descriptions. Amazon SageMaker Catalog analyzes the asset’s technical structure and generate:

  • Business descriptions in natural language for the asset.
  • Contextual definitions for each field/column.
  • Suggested glossary terms that could be applied.
  1. After your asset has been enriched with the necessary business metadata, you can publish it to the Amazon SageMaker Catalog by choosing Publish Asset.

Publish Asset option on the enriched Snowflake asset in SageMaker Catalog

The Snowflake enriched asset is now available to data consumers across your organization. Other users can discover it, subscribe to it, and consume it without data replication.

Implementing data quality rules with AWS Glue Data Quality

This section explains how to apply data quality validations to Snowflake data using AWS Glue Data Quality and visualize results in Amazon SageMaker Catalog.

Setting up the custom transform

Upload two files to an Amazon S3 bucket in the same AWS account where you run AWS Glue:

Copy both files to your AWS Glue assets S3 bucket in the transforms folder (s3://aws-glue-assets-<account-id>-<region>/transforms). AWS Glue Studio reads all JSON files from this folder to register custom visual transforms.

Custom transform files uploaded to the transforms folder in the AWS Glue assets S3 bucket

In the following sections, we walk you through the steps of building an ETL pipeline for data quality validation using AWS Glue Studio.

Creating the AWS Glue Visual ETL job

AWS Glue for Spark provides built-in support for reading from Snowflake data sources.

To create a new visual ETL job:

  1. Open the AWS Glue console at https://console.aws.amazon.com/glue/. Choose ETL jobs, then Visual ETL.

    AWS Glue console showing ETL jobs and the Visual ETL option

Establishing the Snowflake connection

To add a Snowflake source:

  1. In the job pane, choose Snowflake as your source. For Snowflake connection, select the connection that you created earlier. Specify the relevant schema and table for data quality checks.

    Snowflake source node configured in the AWS Glue visual ETL job

The visual editor displays the Data source properties panel where you select your connection, database, and enter a custom query targeting your Snowflake table.

Applying data quality rules

After establishing the Snowflake connection, configure the data quality evaluation step using the Data Quality Definition Language (DQDL).

To add data quality validation:

  1. Choose Transform and choose Evaluate Data Quality.
  2. Define domain-specific data quality rules using DQDL. For more information, see the AWS DQDL documentation.

    Evaluate Data Quality transform with DQDL rules in AWS Glue Studio
  3. Choose to output the data quality results. Optionally, store outcomes in Amazon S3 or publish to Amazon CloudWatch with alert notifications.

The preview of the data quality results from the ruleOutcomes node shows the outcomes of each rule.

Preview of the data quality rule outcomes from the ruleOutcomes node

Post the data quality results to Amazon SageMaker Catalog

To configure the custom transform:

  1. Add the Datazone DQ Result Sink transform to your job.
  2. Connect the ruleOutcomes node output to this transform.
  3. Complete the parameters:
    • Role to assume (Optional): Only needed for associated accounts.
    • Domain ID: Your Amazon SageMaker Unified Studio domain ID (found in the Amazon SageMaker Unified Studio portal).
    • Table name and Schema name: Same values used when creating the Snowflake source transform.
    • Data quality ruleset name: The name you want to give to the ruleset in Amazon SageMaker Catalog.
    • Max results: Maximum number of assets to return in case of multiple matches.

The following image shows the complete job graph with the Datazone DQ Result Sink transform configured.

AWS Glue visual ETL job graph with Snowflake source, Evaluate Data Quality, ruleOutcomes, and Datazone DQ Result Sink nodes

The visual editor displays four nodes connected sequentially: the Snowflake data source, the Evaluate Data Quality transform, the ruleOutcomes SelectFromCollection transform, and the Datazone DQ Result Sink transform.

To configure job parameters:

  1. Choose Job details.
  2. In Job parameters, add the following key-value pair:
    • --additional-python-modules
    • boto3>=1.34.105
  3. Save and run the job.

AWS Glue job parameters with the additional-python-modules key set to boto3

Visualizing data quality results in the SageMaker Catalog

After the AWS Glue ETL job completes, you can view the data quality information directly in Amazon SageMaker Catalog. This is the key outcome of running data quality on a federated source: the asset gains quality scores and metadata without ever leaving Snowflake. This makes it trustworthy and ready for other teams across your organization to use. Data consumers can now discover this asset in Amazon SageMaker Catalog and evaluate its quality before subscribing, without needing direct access to Snowflake or running their own validation.

To view data quality results:

  1. Open the Amazon SageMaker Unified Studio console.
  2. Navigate to your project.
  3. Go to Assets.
  4. Choose the Snowflake data asset.
  5. View the data quality information displayed on the asset page.

The following image shows the asset page in Amazon SageMaker Catalog with the data quality score populated.

SageMaker Catalog asset page showing a populated data quality score for the Snowflake asset

Data Quality tab in SageMaker Catalog showing an overall score of 100 with the movies rule set passed

The Data Quality tab shows an overall score of 100 and lists the rule set movies with a Passed result (1/1). This confirms that the data quality checks from AWS Glue posted successfully to Amazon SageMaker Catalog.

Clean up

To avoid ongoing charges, remove the resources you created during this walkthrough:

  1. Delete the AWS Glue ETL job — Open the AWS Glue console, choose ETL jobs, select your job, and then choose Delete.
  2. Remove the AWS Glue connection — In the AWS Glue console, go to Connections, select the Snowflake connection, and then choose Delete.
  3. Delete the data source in SageMaker Catalog — In your Amazon SageMaker Unified Studio project, go to Data Sources, select the data source you created, and then choose Delete.
  4. Remove S3 assets — Delete the custom transform files from your s3://aws-glue-assets-<account-id>-<region>/transforms/ bucket.
  5. Remove IAM policies — Detach and delete the IAM policies you attached to the AWS Glue job execution role. Remove the role as a domain user and project member.

Conclusion

In this post, we showed you how to connect Snowflake to Amazon SageMaker Unified Studio for centralized data cataloging and quality validation. This approach maintains consistent governance without replicating data. Key benefits include:

  • Query without data movement: Access Snowflake data directly from Amazon SageMaker Unified Studio through federated queries, using the interoperable data architecture of AWS and eliminating time-consuming data replication.
  • Centralized governance: Maintain a single source of truth for data discovery, quality metrics, and governance policies across your distributed data estate.
  • Automated quality validation: Apply consistent data quality rules using AWS Glue Data Quality and visualize results directly in Amazon SageMaker Catalog.
  • Unified collaboration: Support data discovery and sharing across your organization through the publishing capabilities of Amazon SageMaker Catalog.

To get started, open the Amazon SageMaker Unified Studio console. To learn more about related topics, see Cross-account lakehouse governance with Amazon S3 Tables and SageMaker Catalog and Get started with AWS Glue Data Quality dynamic rules for ETL pipelines.


About the authors

Marco Duarte López

Marco Duarte López

Marco is a Data Specialist Solutions Architect at AWS, based in Santiago, Chile. He works with organizations across the region to design modern data architectures and governance frameworks that enable trusted, scalable data consumption. He is a member of the AWS Technical Field Community (TFC) for Analytics, where he specializes in Data & AI Governance, and has led data transformation programs for some of the largest enterprises in the region.

Diego Ortiz

Diego Ortiz

Diego is a Senior Data Strategy Solutions Architect for Latin America based in San Juan, Puerto Rico, with 14+ years of experience in technology roles. He supports organizations across countries and industries to develop data and AI strategies aligned with their business objectives, combining strategic vision with deep technical expertise in data and AI technologies. He is a core member of the Data Governance global community at AWS and leads the analytics technical community in the Spanish-speaking countries of Latin America.

Connect Amazon SageMaker Unified Studio to Microsoft Power BI – Part 1: IAM Identity Center (IDC)-based domains

Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/connect-amazon-sagemaker-unified-studio-to-microsoft-power-bi-part-1-iam-identity-center-idc-based-domains/

Connecting Power BI to your Amazon SageMaker Unified Studio data catalogs typically required third-party bridges. These bridges added complexity and licensing costs. In this post, you create a direct connection using new authentication modes in the Amazon Athena ODBC driver, removing those dependencies entirely. If your organization uses Power BI as its business intelligence (BI) tool, your analysts can configure access to governed data in Amazon SageMaker Unified Studio without changing their tools or workflows. As an AWS alternative, Amazon Quick Sight provides serverless BI integration with Amazon SageMaker Unified Studio at pay-per-session pricing.

A previous post showed the connection method using a third-party ODBC-JDBC bridge. The Amazon Athena ODBC driver (version 2.2.0 and later) now supports Amazon SageMaker Unified Studio authentication directly, eliminating the need for customers to configure third-party bridge components previously required for this connection. This bridge also created additional components and required ongoing maintenance. The native connection simplifies the architecture by reducing these requirements.

UC Irvine, a top-ten U.S. public research university, consolidates student data from systems across multiple departments into a single governed repository that supports reporting, research, and analytics for decision-making at the strategic, tactical, and operational levels. Many of their analysts rely on Power BI to explore and visualize this governed data.

“Our users rely on Power BI for data visualization and reporting, but connecting to governed data in AWS previously required workarounds. The ODBC connection feature gives a direct path from Power BI into our SageMaker Unified Studio projects—no bridge software, no extra licensing, just a connection string and we’re ready to go.”

— Bernadette Theologidy, Manager, Student Analytics, UC Irvine

The Athena ODBC driver introduces two new authentication modes for SageMaker Unified Studio:

  1. SageMakerBrowserIdc (for IDC-based domains): The driver opens a browser window and authenticates through AWS IAM Identity Center (and your external identity provider, if configured). No local AWS credentials are needed.
  2. SageMakerIam (for AWS Identity and Access Management (IAM)-based and IDC-based domains): The driver uses AWS credentials from the default credential provider chain. For this walkthrough, we use AWS IAM Identity Center to provide those credentials.

You connect Microsoft Power BI to Amazon SageMaker Unified Studio through Athena. The Athena ODBC driver supports using two connection methods that use these authentication modes:

Method 1: DSN-based (Athena Power BI connector): You configure an ODBC Data Source Name (DSN) and use the Athena connector in Power BI. This method supports DirectQuery and Import mode with both SageMakerBrowserIdc and SageMakerIam authentication.

Method 2: DSN-less (Power BI ODBC connector): You use the Power BI ODBC connector with a connection string, requiring no DSN configuration. This method supports Import mode only with SageMakerIam authentication. DirectQuery isn’t available because the Power BI ODBC connector doesn’t support it. The connection string in Power BI Desktop must match exactly the one on Power BI Service. Because the gateway runs as a Windows service without interactive browser access, both ends must use SageMakerIam.

Feature Method 1: DSN-based Method 2: DSN-less
Power BI Connector Amazon Athena connector ODBC connector
Data connectivity mode DirectQuery and Import Import only
Requires DSN configuration Yes No
Data freshness Real-time (DirectQuery) or scheduled (Import) Scheduled refresh only
Authentication types SageMakerIam and SageMakerBrowserIdc SageMakerIam only
Domain types supported IAM-based and IDC-based IAM-based and IDC-based
Best for Dashboards requiring live data Scenarios where DSN management is not possible or scheduled refresh is acceptable

This is Part 1 of a two-part series. This post covers IDC-based domains using both connection methods. Part 2 covers IAM-based domains.

Solution overview

In this walkthrough, you take the role of a data analyst at an energy company. You need to understand the current state and future direction of the U.S. power generation fleet using the Public Utility Data Liberation Project, available on the Registry of Open Data on AWS. Our goal is to analyze generation capacity and identify where new investment is flowing. We connect Power BI to Athena through Amazon SageMaker Unified Studio and query the EIA-860 generators dataset directly from our data catalog. The result is a single visualization that reveals the energy transition.

The following diagram illustrates the solution architecture for connecting Power BI to Amazon SageMaker Unified Studio through Amazon Athena.

Architecture diagram showing Power BI connecting to Amazon Athena through Amazon SageMaker Unified Studio, with a Microsoft on-premises data gateway on Amazon EC2

Figure 1: Architecture diagram

The following architecture demonstrates a six-step workflow.

  1. Data engineers and analysts connect Power BI Desktop to Athena as a data source.
  2. They build their reports locally.
  3. They then publish them to the Power BI Service.
  4. Microsoft On-Premises Data Gateway on an Amazon Elastic Compute Cloud (Amazon EC2) instance connects to Athena using the instance’s attached IAM role.
  5. The Power BI Service then uses this gateway connection.
  6. Report viewers access the published reports through Power BI Service to make data-driven decisions.

On the AWS side, Athena queries the data catalog managed by AWS Glue Data Catalog. The catalog references data stored in Amazon Simple Storage Service (Amazon S3). An Amazon SageMaker Unified Studio project governs all access.

In an IDC-based domain (covered in this post), Power BI Desktop uses SageMakerBrowserIdc for Method 1 and SageMakerIam for Method 2. Power BI Desktop can run on-premises or on an EC2 instance. The gateway always uses SageMakerIam (it runs as a Windows service without browser access) and authenticates using instance profile credentials, which rotate automatically. The gateway can only query data within projects where its IAM role has been added as a member. For IAM-based domains, see Part 2.

Prerequisites

Before connecting Power BI to Amazon SageMaker Unified Studio, verify that your environment meets these requirements:

  • Athena ODBC driver – The latest Amazon Athena ODBC driver (version 2.2.0 or more recent) for Windows 64-bit.
  • Microsoft Power BI Desktop – The latest version installed on your Windows machine.
  • Microsoft Power BI Pro License – Required for publishing reports and configuring the on-premises data gateway.
  • Microsoft Power BI on-premises data gateway – The latest version installed on the EC2 instance.
  • Amazon SageMaker Unified Studio – An Amazon SageMaker Unified Studio IDC-based domain.

You need an Amazon SageMaker Unified Studio project with data assets. For detailed instructions, refer to the Amazon SageMaker Unified Studio User Guide.

The following screenshot shows the Amazon SageMaker Unified Studio project Query Editor interface, which runs a preview query against the EIA-860 generators dataset.

SageMaker Unified Studio Query Editor previewing the EIA-860 generators dataset

Figure 2: SageMaker Unified Studio project with the EIA-860 generators dataset available in the data catalog

Method 1: DSN-based connection (Athena Power BI connector)

This method uses the Amazon Athena Power BI connector with an ODBC Data Source Name (DSN), supporting DirectQuery and Import mode.

You configure Power BI Desktop to connect to your data assets in Amazon SageMaker Unified Studio using the SageMakerBrowserIdc authentication mode. The driver opens a browser window and authenticates through IAM Identity Center (and your external identity provider, if configured).

Add your SSO user as a member of your SageMaker Unified Studio project

Your single sign-on (SSO) user needs project-level access to query data with Athena. Verify your user is listed as a project member or add it by following Add project members in the Amazon SageMaker Unified Studio User Guide.

The following screenshot shows the SageMaker Unified Studio project user management page, where project owners can add or remove project users and roles.

SageMaker Unified Studio project members page listing users and roles

Figure 3: Members of a SageMaker Unified Studio project

Gather configuration values to configure your Amazon Athena ODBC DSN

Gather the following values from your Amazon SageMaker Unified Studio project:

  1. Open your Amazon SageMaker Unified Studio project.
  2. In the top right, select the three dots.
  3. Choose Project details.
  4. Select JDBC and ODBC details.
  5. Under ODBC connection details copy the following information: IDC issuer URL, domain ID, project ID, Athena workgroup name and AWS Region.

The following screenshot shows the Amazon SageMaker Unified Studio project overview page, where you can copy these details.

SageMaker Unified Studio project overview showing ODBC connection details

Figure 4: ODBC connection details

Configure the ODBC DSN

Create a System DSN using the Amazon Athena ODBC driver. For the general DSN creation steps, see Configuring a data source name on Windows in the Amazon Athena User Guide.

Enter the following values:

Field Value
Data Source Name Name your datasource (for example, pbi-idcdomain)
Region The AWS Region where your Amazon SageMaker domain is provisioned (for example, us-east-1)
Catalog AwsDataCatalog
Database default
Workgroup Your Athena workgroup name (for example, workgroup-abcdefghij-klmexample)

In the Authentication Options, configure the following values:

Field Value
Authentication Type SageMakerBrowserIdc
SSO Start URL IAM Identity Center entry point (for example, https://identitycenter.amazonaws.com/ssoins-0example)
SSO Region Region of IAM Identity Center (for example, us-east-1)
SageMaker Domain ID dzd-123456example
SageMaker Project ID abcd12example
SageMaker Domain Region Region of your Amazon SageMaker Unified Studio project (for example, us-east-1)

Choose OK, then Test to verify the connection. Choose Allow Access when prompted by the browser.

The following screenshot shows the consent prompt.

Browser consent prompt requesting access approval during authentication

Figure 5: Browser consent prompt

The following screenshot shows the successful connection test.

ODBC DSN configuration showing a successful connection test with SageMakerBrowserIdc

Figure 6: Successful connection test in the ODBC DSN configuration with SageMakerBrowserIdc authentication

Connect Power BI Desktop to your data

With the DSN configured, you can connect Power BI Desktop to your data catalog and load the generators dataset.

  1. Open Power BI Desktop.
  2. Open the Get Data menu and select More.
  3. Search for and select Amazon Athena and choose Connect.
  4. For Data Source Name (DSN), enter pbi-idcdomain.
  5. Select DirectQuery.
  6. Choose OK.
  7. Choose Use Data Source Configuration and then Connect.
  8. In the AwsDataCatalog folder, navigate to your database.
  9. Select the core_eia860__scd_generators table.
  10. Choose Load.

The following screenshot shows Power BI Desktop successfully connected to the AWS data catalog.

Power BI Desktop connected to the data catalog with the generators table loaded

Figure 7: Power BI Desktop connected to the data catalog with the generators table loaded using SageMakerBrowserIdc authentication

Create your dashboard and publish it

You can create a dashboard to visualize U.S. power generation data. To create a visualization, complete the following steps:

  1. In the Visualizations pane, choose the Stacked bar chart.
  2. Assign the Y-Axis: Drag technology_description to the Y-Axis.
  3. Assign the X-Axis (Values): Drag capacity_mw to the X-Axis (automatically summed).
  4. Assign the Legend (Stack): Drag operational_status to the Legend field.
  5. Choose Publish.
  6. Give your report a name (for example, generation-idcdomain) and choose Save.
  7. Sign in and choose a destination workspace.
Power BI Desktop stacked bar chart of generation capacity by technology and operational status

Figure 8: Power BI Desktop report using the EIA-860 generators dataset

After publishing, the report structure is available on Power BI Service.

Method 2: DSN-less connection (Power BI ODBC connector)

In this method, you use the Power BI ODBC connector with a connection string (no DSN required). This method supports Import mode only and SageMakerIam authentication. Because the gateway cannot perform browser authentication, both Desktop and gateway must use SageMakerIam. If your workflow requires SageMakerBrowserIdc, use Method 1.

If your machine already has AWS credentials through another method in the default credential provider chain, skip the following setup.

Administrator setup

Create a custom permission set named SageMakerDataAnalyst in IAM Identity Center with the following inline policy. For detailed steps, see Create a permission set in the AWS IAM Identity Center User Guide.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:GetConnection",
                "datazone:ListConnections",
                "datazone:GetDomain",
                "datazone:GetProject"
            ],
            "Resource": "*"
        },
        {
            "Sid": "STSForDriver",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Assign your user to this permission set for the AWS account containing your SageMaker Unified Studio domain. Then configure your AWS Command Line Interface (AWS CLI) SSO profile by running aws configure sso. For the full CLI configuration walkthrough with detailed steps, see Part 2. After your profile is configured, run aws sso login to authenticate.

Add the IAM identity as a member of SageMaker Unified Studio project

The IAM identity providing credentials needs both domain-level and project-level access to query data through Athena.

  1. Add AWSReservedSSO_SageMakerDataAnalyst_1234example as a domain IAM user: see Managing users in the Amazon SageMaker Unified Studio Admin Guide. Choose Current account.
SageMaker Unified Studio domain users list including the IAM identity

Figure 9: List of users of your SageMaker Unified Studio domain including the IAM identity

  1. Add AWSReservedSSO_SageMakerDataAnalyst_1234example as a project member: see Add project members in the Amazon SageMaker Unified Studio User Guide.
SageMaker Unified Studio project members list including the IAM identity

Figure 10: Members of a SageMaker Unified Studio project including the IAM identity

Gather configuration values

Gather the following connection values from your Amazon SageMaker Unified Studio project:

  1. Open your Amazon SageMaker Unified Studio Project.
  2. On the navigation pane, choose Overview.
  3. Select JDBC and ODBC details.
  4. Select the Using IAM auth toggle.
  5. Copy the ODBC connection string.
SageMaker Unified Studio project overview showing the ODBC connection string for IAM auth

Figure 11: ODBC connection string on the SageMaker Unified Studio project overview

Connect Power BI Desktop to your data and publish

With the configuration parameters of your project, you can connect Power BI Desktop to your data catalog and load the generators dataset.

  1. Open Power BI Desktop.
  2. Open the Get Data menu and select More.
  3. Search for and select ODBC and choose Connect.
  4. For Data Source Name (DSN), select (None).
  5. Expand Advanced Options.
  6. In the Connection string field, enter your connection string. For example, Driver={Amazon Athena ODBC (x64)};AwsRegion=us-east-1;Catalog=AwsDataCatalog;Schema=default;Workgroup=workgroup-abcdefghij-klmexample;SageMakerDomainId= dzd-123456example;SageMakerProjectId= abcd12example;SageMakerDomainRegion=us-east-1;AuthenticationType=SageMakerIam;
  7. Choose OK.
  8. Choose Default or Custom and then Connect.
  9. In the AwsDataCatalog folder, navigate to your database.
  10. Select the core_eia860__scd_generators table.
  11. Choose Load.

When publishing, name your report generation-idcdomain-dsnless.

Configure the on-premises data gateway and view your report on Power BI Service

After creating your reports in Power BI Desktop, configure the on-premises data gateway to view your report on Power BI Service.

You can configure the gateway using either a DSN or a DSN-less connection string, matching the method you used in Power BI Desktop.

Create and attach an IAM role to the Power BI Gateway EC2 instance

Create an IAM role for the EC2 instance that will host your Power BI gateway. Name the role pbi-gateway-role (or a name of your choice). The role must use EC2 as the trusted entity and include the following inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:GetConnection",
                "datazone:ListConnections",
                "datazone:GetDomain",
                "datazone:GetProject"
            ],
            "Resource": "*"
        },
        {
            "Sid": "STSForDriver",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Attach this role to your Power BI Gateway EC2 instance. For detailed steps on creating and attaching an IAM role to an EC2 instance, refer to IAM roles for Amazon EC2 in the Amazon EC2 User Guide.

Add the Power BI Gateway IAM role as a member of SageMaker Unified Studio project

The gateway IAM role needs project-level access to query data through Athena.

  1. Add the IAM pbi-gateway-role role as a domain IAM user: see Managing users in the Amazon SageMaker Unified Studio Admin Guide. Choose Current account (or Associated account if your gateway is deployed in a different account).

The following screenshot, from the Amazon SageMaker page of the AWS Management Console, shows the list of users of your Amazon SageMaker Unified Studio domain, including the IAM gateway role.

SageMaker Unified Studio domain users list including the Power BI gateway IAM role

Figure 12: List of users of your SageMaker Unified Studio domain including the IAM gateway role

Add the IAM pbi-gateway-role role as a project member: see Add project members in the Amazon SageMaker Unified Studio User Guide.

The following screenshot shows the Amazon SageMaker Unified Studio project user management page listing the project members.

SageMaker Unified Studio project members list including the Power BI gateway IAM role

Figure 13: Members of a SageMaker Unified Studio project including the IAM gateway role

Configure the data source on Power BI Gateway

How you configure the data source depends on the method you used in Power BI Desktop.

Method 1 (DSN-based)

Configure a System DSN on the gateway EC2 instance following the same ODBC DSN steps described in Method 1. When configuring, make sure that:

  • You use the System DSN tab (not User DSN) because the gateway runs as a Windows service under a separate account.
  • The authentication type is set to SageMakerIam regardless of what you used on Desktop.
  • The DSN name matches exactly the one configured on Power BI Desktop (for example, pbi-idcdomain)

Method 2 (DSN-less)

No configuration is needed on the gateway machine itself. You configure the data source directly in Power BI Service.

Configure the data source and view your report on Power BI Service

To view your report, complete the following steps:

  1. Open the workspace where you saved your report.
  2. Search the Semantic Model which has the same name as your report (for example, generation-idcdomain) and choose the More options icon (three dots).
  3. Choose Settings.
  4. Expand Gateway and Cloud Connection.
  5. Choose View Datasources (play icon) on your gateway.
  6. Choose Manually add to gateway.
  7. Add a connection name (for example, pbi-idcdomain).

The next step depends on the method that you chose:

Method 1 (DSN-based)

  1. Add the DSN (for example, pbi-idcdomain) that matches exactly the one configured on Power BI Desktop.

Method 2 (DSN-less)

  1. In the Connection string field, enter the connection string that matches exactly the one used in Power BI Desktop.

Next, continue with the configuration:

  1. Select Anonymous as Authentication Method.
  2. Choose Create.
  3. Expand again Gateway and Cloud Connection.
  4. For Maps to, choose the connection that you created (for example, pbi-idcdomain).
  5. Choose Apply.
  6. Return to the workspace where you saved your report.
  7. On the Content section, choose your report (for example, generation-idcdomain).

The following screenshot shows a Power BI report on Power BI Service.

Published Power BI report rendering on Power BI Service

Figure 14: Power BI report on Power BI Service

You can now see your report online with the data from your Amazon SageMaker Unified Studio project.

Clean up

To avoid additional charges after testing, delete the Amazon SageMaker Unified Studio domain and EC2 instances. Refer to Delete domains and Terminate Instances for instructions.

Conclusion

In this post, you connected Microsoft Power BI to Amazon SageMaker Unified Studio using an IDC-based domain with both DSN-based and DSN-less methods. This provides a direct connection, with no third-party licensing, that maintains data governance. In Part 2, we cover IAM-based domains.

You can automate many steps of this process. For information about automating DSN creation on the Power BI Gateway or Service, refer to How ENGIE automates the deployment of Amazon Athena data sources on Microsoft Power BI. If you don’t want users adding the gateway IAM role directly, you can create a custom blueprint as a self-service tool for gateway role addition. The blueprint uses a ProjectMembership resource with a configurable parameter that project owners can activate at project creation, automatically adding the gateway role as a project contributor.

For additional best practices, refer to the Using Microsoft Power BI with the AWS Cloud Whitepaper. To learn more, visit Amazon SageMaker Unified Studio and Amazon Athena.


About the authors

Ramesh Singh

Ramesh Singh

Ramesh is a Senior Product Manager Technical (External Services) at AWS in Seattle, Washington, currently with the Amazon SageMaker team. He is passionate about building high-performance ML/AI and analytics products that help enterprise customers achieve their critical goals.

Armando Segnini

Armando Segnini

Armando is a Senior Analytics Specialist Solutions Architect at AWS, partnering with enterprise customers to architect scalable data, analytics, and AI platforms. He helps organizations turn complex data challenges into business value through expertise in streaming, BI integration, and generative AI. Outside of work, Armando enjoys traveling with his family, exploring new cultures, photography, and functional fitness competitions.

Gaurav Sharma

Gaurav is a Specialist Solutions Architect (Analytics) at AWS, supporting US public sector customers on their cloud journey. Outside of work, Gaurav enjoys spending time with his family and reading books.

Krishna Atluru

Krishna Atluru

Krishna is an Enterprise Support Lead TAM at AWS. He provides customers with in-depth guidance on improving security posture and operational excellence for their workloads, helping them build secure, resilient, and cost-effective solutions. His areas of expertise include building serverless architectures, and data and analytics solutions. Outside of work, Krishna enjoys cooking, swimming, and traveling.

Saushthav Saxena

Saushthav Saxena

Saushthav is a Software Development Engineer at AWS on the Amazon Athena team, where he has spent the past few years working on distributed systems and data analytics at scale. Based in the San Francisco Bay Area, his background spans full-stack development, high performance computing, and large-scale infrastructure. Outside of work, he enjoys reading sci-fi novels, swimming, and traveling with family and friends.

Connect Amazon SageMaker Unified Studio to Microsoft Power BI – Part 2: IAM-based domains

Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/connect-amazon-sagemaker-unified-studio-to-microsoft-power-bi-part-2-iam-based-domains/

In Part 1 of this series, we connected Microsoft Power BI to Amazon SageMaker Unified Studio using an IAM Identity Center (IDC)-based domain. The Amazon Athena ODBC driver (version 2.2.0 and later) supports Amazon SageMaker Unified Studio authentication natively, removing the third-party ODBC-JDBC bridge previously required. We walked through both the DSN-based connection and the DSN-less connection, from Power BI Desktop through the on-premises data gateway to Power BI Service, where report viewers access published dashboards.

In this post, you create the same direct connection using an AWS Identity and Access Management (IAM)-based domain. The walkthrough covers the same two connection methods. The differences are the Amazon SageMaker Unified Studio console navigation paths, the configuration values, and an additional administrator setup that provides AWS credentials through AWS IAM Identity Center. This is Part 2 of a two-part series. For a detailed comparison of the two connection methods, see Part 1.

Solution overview

The architecture is the same as the previous post (see the architecture diagram and walkthrough scenario in Part 1). Power BI Desktop connects to Amazon Athena through the ODBC driver and the Amazon SageMaker Unified Studio project governs all data access. At the same time, the on-premises data gateway on an Amazon Elastic Compute Cloud (Amazon EC2) instance bridges the connection to Power BI Service so report viewers can access published dashboards.

The difference is in authentication: An IAM-based domain uses SageMakerIam authentication for both connection methods. The driver retrieves credentials from the AWS default credential provider chain. For this walkthrough, AWS IAM Identity Center provides those credentials through a custom permission set. Power BI Desktop can run on-premises or on an EC2 instance in the AWS Cloud. The gateway EC2 instance authenticates using its attached IAM role.

Prerequisites

Complete the prerequisites from Part 1. Additionally, you need:

  • AWS Command Line Interface (AWS CLI) – The latest version of the AWS CLI installed on your Windows machine. In this post series, the ODBC driver uses the AWS IAM Identity Center profile configured through the CLI for authentication.
  • Amazon SageMaker Unified Studio – An Amazon SageMaker Unified Studio IAM-based domain with AWS IAM Identity Center single sign-on (SSO) enabled.

The following screenshot shows the Amazon SageMaker Unified Studio (IAM-based domain) project Query Editor interface. It runs a preview query on the EIA-860 generators dataset.

SageMaker Unified Studio Query Editor previewing the EIA-860 generators dataset in an IAM-based domain

Figure 1: SageMaker Unified Studio (IAM-based domain) project with the EIA-860 generators dataset available in the data catalog

Administrator setup

This section configures AWS IAM Identity Center to provide credentials for the SageMakerIam authentication mode. It applies to Method 1 (IAM-based domain) and Method 2 (both domain types). If your machine already has AWS credentials available through another method in the default credential provider chain, you can skip this section and proceed directly to the method of your choice. For the full list of credential sources, refer to Credential providers in the AWS SDKs and Tools Reference Guide.

Create a permission set in IAM Identity Center

Create a custom permission set named SageMakerDataAnalyst in IAM Identity Center with the following inline policy. For detailed steps, see Create a permission set in the AWS IAM Identity Center User Guide.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:GetConnection",
                "datazone:ListConnections",
                "datazone:GetDomain",
                "datazone:GetProject"
            ],
            "Resource": "*"
        },
        {
            "Sid": "STSForDriver",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

The "Resource": "*" is required because these API actions do not support resource-level permissions. For more information, see Actions, resources, and condition keys for Amazon DataZone.

This doesn’t grant broad access to your data. These are read-only metadata actions that allow the ODBC driver to discover connection details and retrieve temporary Athena credentials. The actual data access is governed by Amazon SageMaker Unified Studio project membership: Users can only query data within projects where they have been explicitly added as members. The Amazon SageMaker Unified Studio project IAM role provides Athena and Amazon S3 permissions separately.

Assign users to the permission set

To assign users or groups to the target AWS account, complete the following steps:

  1. In the IAM Identity Center console, choose AWS accounts.
  2. Select the target account where your Amazon SageMaker Unified Studio IAM-based domain is deployed.
  3. Choose Assign users or groups.
  4. Select the SSO users or groups that need access.
  5. Select the SageMakerDataAnalyst permission set.
  6. Choose Submit.

Configure AWS IAM Identity Center profile

To configure the AWS IAM Identity Center profile, run the following command in your terminal on Windows:

aws configure sso

When prompted, enter the following values:

Prompt Value
SSO session name For example, smus
SSO start URL The IDC issuer URL. For example, https://identitycenter.amazonaws.com/ssoins-0example
SSO region The SSO Region. For example, us-east-1
SSO registration scopes sso:account:access

A browser window opens for authentication. After authentication, select your account and the SageMakerDataAnalyst role.

The following screenshots show the consent window and the successful authentication message.

Browser consent prompt requesting access approval during AWS CLI SSO authentication

Figure 2: Browser consent prompt

Browser page confirming successful AWS CLI SSO authentication

Figure 3: Browser authentication successful message

When prompted, enter the following values:

Prompt Value
Default client Region None
CLI default output format None
Profile Name Change value by default

The resulting ~/.aws/config file should look like the following:

[default]
sso_session = smus
sso_account_id = 1234example
sso_role_name = SageMakerDataAnalyst

[sso-session smus]
sso_start_url = https://identitycenter.amazonaws.com/ssoins-0example
sso_region = us-east-1
sso_registration_scopes = sso:account:access

Verify authentication and daily use

To verify that your SSO profile is working correctly, run the following command:

aws sts get-caller-identity

You should receive a response like the following:

{
    "UserId": "AROARHJJNFBQD6EXAMPLE:[email protected]",
    "Account": "111122223333",
    "Arn": "arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_SageMakerDataAnalyst_1234example/[email protected]"
}

For daily use, no passwords or EC2 instance roles are required. When your SSO session expires, run the following command to quickly refresh it:

aws sso login

Add your IAM identity as a member of your Amazon SageMaker Unified Studio project

The IAM identity providing credentials to the ODBC driver needs project-level access to query data through Athena. If you completed the administrator setup, this is the SSO role associated with your permission set (for example, AWSReservedSSO_SageMakerDataAnalyst_1234example). If you’re using another credential source, add the IAM role or user that provides those credentials. For detailed steps, see Managing users for IAM-based domains in the Amazon SageMaker Unified Studio Administrator Guide.

The following screenshot shows the Amazon SageMaker Unified Studio domain management page, which lists the members in a project.

SageMaker Unified Studio project members list

Figure 4: List of members of your SageMaker Unified Studio project

Gather the information to authenticate

To get the parameters that you need to authenticate, complete these steps:

  1. Open your Amazon SageMaker Unified Studio Project.
  2. Open Domain Management.
  3. Choose Users.
  4. Choose View SSO connection.
  5. Copy the end of the Instance ARN, so we can build the Instance URL like https://identitycenter.amazonaws.com/ssoins-0example

The following screenshot shows the Amazon SageMaker Unified Studio domain management page with SSO connection details.

SageMaker Unified Studio domain SSO connection details showing the IAM Identity Center instance ARN

Figure 5: AWS IAM Identity Center information

  1. Choose the user icon and copy the Region as shown in the following screenshot.
SageMaker Unified Studio user menu showing the Region

Figure 6: User icon with the Region information

Method 1: DSN-based connection (Athena Power BI connector)

In this method, you configure an ODBC Data Source Name (DSN) and use the Amazon Athena connector in Power BI. This method uses SageMakerIam authentication mode and supports both DirectQuery and Import mode.

This section covers IAM-based domains. For IDC-based domains, see Part 1.

Gather configuration values to configure your Amazon Athena ODBC DSN

Before configuring the ODBC DSN, gather the following connection values from your Amazon SageMaker Unified Studio project:

  1. Open your Amazon SageMaker Unified Studio Project.
  2. Top right, select the three dots.
  3. Choose Project details.
  4. Select JDBC and ODBC details.
  5. Copy the following values: domain ID, Amazon SageMaker project ID, AWS Region, and Athena workgroup.

The following screenshot shows the Amazon SageMaker Unified Studio project overview page, which provides the project details to copy.

SageMaker Unified Studio project details showing domain ID, project ID, Region, and Athena workgroup

Figure 7: Project details with SageMaker domain ID, SageMaker project ID, Region, and Athena workgroup

Configure the ODBC DSN

Create a System DSN using the Amazon Athena ODBC driver. For the general DSN creation steps, see Configuring a data source name on Windows in the Amazon Athena User Guide. Enter the following values:

Field Value
Data Source Name Name your datasource (for example, pbi-iamdomain)
Region The AWS Region where your Amazon SageMaker domain is provisioned (for example, us-east-1)
Catalog AwsDataCatalog
Database default
Workgroup Your Athena workgroup name (for example, workgroup-abcdefghij-klmexample)

In the Authentication Options, configure the following values:

Field Value
Authentication Type SageMakerIam
SageMaker Domain ID dzd-123456example
SageMaker Project ID abcd12example
SageMaker Region Region of your SageMaker Unified Studio project (for example, us-east-1)

Choose OK, then Test to verify the connection. Choose Allow Access when prompted by the browser.

The following screenshot shows the successful connection test.

ODBC DSN configuration showing a successful connection test with SageMakerIam

Figure 8: Successful connection test in the ODBC DSN configuration with SageMakerIam authentication

Connect Power BI Desktop to your data

With the DSN configured, you can connect Power BI Desktop to your data catalog and load the generators dataset.

  1. Open Microsoft Power BI Desktop.
  2. Open the Get Data menu and select More.
  3. Search for and select Amazon Athena and choose Connect.
  4. For Data Source Name (DSN), enter pbi-iamdomain.
  5. Select DirectQuery.
  6. Choose OK.
  7. Choose Use Data Source Configuration and then Connect.
  8. In the AwsDataCatalog folder, navigate to your database.
  9. Select the core_eia860__scd_generators table.
  10. Choose Load.

The following screenshot shows Power BI Desktop successfully connected to the data catalog.

Power BI Desktop connected to the data catalog with the generators table loaded

Figure 9: Power BI Desktop connected to the data catalog with the generators table loaded using SageMakerIam authentication

Create your dashboard and publish it

You can create a dashboard to visualize U.S. power generation data. To create a visualization, complete the following steps:

  1. In the Visualizations pane, choose the Stacked bar chart.
  2. Assign the Y-Axis: Drag technology_description to the Y-Axis.
  3. Assign the X-Axis (Values): Drag capacity_mw to the X-Axis (automatically summed).
  4. Assign the Legend (Stack): Drag operational_status to the Legend field.
  5. Choose Publish.
  6. Give your report a name (for example, generation-iamdomain) and choose Save.
  7. Sign in and choose a destination workspace.

The following screenshot shows the Power BI dashboard with U.S. power generation data.

Power BI stacked bar chart of U.S. generation capacity by technology and operational status

Figure 10: Power BI dashboard with U.S. power generation data

After you publish, the report structure becomes available on Microsoft Power BI Service.

Method 2: DSN-less connection (Power BI ODBC connector)

In this method, you use the Power BI ODBC connector with a connection string (no DSN required). This method supports Import mode only and SageMakerIam authentication. Because the gateway can’t perform browser authentication and connection strings need to match, both Desktop and gateway must use SageMakerIam.

This section covers IAM-based domains. For IDC-based domains, see Part 1.

Gather configuration values to configure your DSN-less connection

Gather the following connection values from your Amazon SageMaker Unified Studio project:

  1. Open your Amazon SageMaker Unified Studio Project.
  2. Top right, select the three dots.
  3. Choose Project details.
  4. Select JDBC and ODBC details.
  5. Copy the ODBC connection string.

The following screenshot shows the Amazon SageMaker Unified Studio project overview page with the ODBC connection string to copy.

SageMaker Unified Studio project overview showing the ODBC connection string

Figure 11: Project details with ODBC connection string

Connect Power BI Desktop to your data and publish

With the configuration parameters of your project, you can connect Power BI Desktop to your data catalog and load the generators dataset.

  1. Open Power BI Desktop.
  2. Open the Get Data menu and select More.
  3. Search for and select ODBC and choose Connect.
  4. For Data Source Name (DSN), select (None).
  5. Expand Advanced Options.
  6. In the Connection string field, enter your connection string. For example, Driver={Amazon Athena ODBC (x64)};AwsRegion=us-east-1;Catalog=AwsDataCatalog;Schema=default;Workgroup=workgroup-abcdefghij-klmexample;SageMakerDomainId= dzd-123456example;SageMakerProjectId= abcd12example;SageMakerDomainRegion=us-east-1;AuthenticationType=SageMakerIam;
  7. Choose OK.
  8. Choose Default or Custom and then Connect.
  9. In the AwsDataCatalog folder, navigate to your database.
  10. Select the core_eia860__scd_generators table.
  11. Choose Load.

When publishing, name your report generation-iamdomain-dsnless.

Configure the gateway and view your report on Power BI Service

After creating your reports in Power BI Desktop, configure the on-premises data gateway to view your report on Power BI Service.

You can configure the gateway using either a DSN or a DSN-less connection string, matching the method you used in Power BI Desktop.

Create and attach an IAM role to the Power BI Gateway EC2 instance

Create an IAM role for the EC2 instance that will host your Power BI gateway. Name the role pbi-gateway-role (or a name of your choice). The role must use EC2 as the trusted entity and include the following inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:GetConnection",
                "datazone:ListConnections",
                "datazone:GetDomain",
                "datazone:GetProject"
            ],
            "Resource": "*"
        },
        {
            "Sid": "STSForDriver",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Attach this role to your Power BI Gateway EC2 instance. For detailed steps on creating and attaching an IAM role to an EC2 instance, refer to IAM roles for Amazon EC2 in the Amazon EC2 User Guide.

Add the Power BI Gateway IAM role as a member of SageMaker Unified Studio project

The gateway IAM role needs project-level access to query data through Athena. The steps to add the role differ depending on your domain type.

IAM-based domain

  1. Open your Amazon SageMaker Unified Studio Project.
  2. Open Domain Management.
  3. Choose your Project Name.
  4. Choose Members.
  5. Choose Add members.
  6. Select the IAM role of your Power BI gateway (for example, pbi-gateway-role).
  7. Choose Add.

The following screenshot shows the Amazon SageMaker Unified Studio project domain management page with options to add members to a project.

SageMaker Unified Studio project members list including the Power BI gateway IAM role

Figure 12: List of members of a SageMaker Unified Studio project with the IAM gateway role

Configure the data source on Power BI Gateway

How you configure the data source depends on the method you used in Power BI Desktop.

Method 1 (DSN-based)

Configure a System DSN on the gateway EC2 instance following the same ODBC DSN steps described in Method 1. When configuring, make sure that:

  • You use the System DSN tab (not User DSN) because the gateway runs as a Windows service under a separate account.
  • The authentication type is set to SageMakerIam.
  • The DSN name matches exactly the one configured on Power BI Desktop (for example, pbi-iamdomain).

Method 2 (DSN-less)

No configuration is needed on the gateway machine itself. You configure the data source directly in Power BI Service.

Configure the data source and view your report on Power BI Service

To view your report, complete the following steps:

  1. Open the workspace where you saved your report.
  2. Search the Semantic Model which has the same name as your report (for example, generation-iamdomain) and choose the More options icon (three dots).
  3. Choose Settings.
  4. Expand Gateway and Cloud Connection.
  5. Choose View Datasources (play icon) on your gateway.
  6. Choose Manually add to gateway.
  7. Add a connection name (for example, pbi-iamdomain).

The next step depends on the method that you chose:

Method 1 (DSN-based)

  1. Add the DSN (for example, pbi-iamdomain) that matches exactly the one configured on Power BI Desktop.

Method 2 (DSN-less)

  1. In the Connection string field, enter the connection string that matches exactly the one used in Power BI Desktop.

Next, continue with the configuration:

  1. Select Anonymous as Authentication Method.
  2. Choose Create.
  3. Expand again Gateway and Cloud Connection.
  4. For Maps to, choose the connection that you created (for example, pbi-iamdomain).
  5. Choose Apply.
  6. Return to the workspace where you saved your report.
  7. On the Content section, choose your report (for example, generation-iamdomain).

The following screenshot shows a report on Power BI Service.

Published Power BI report rendering on Power BI Service

Figure 13: Power BI report on Power BI Service

You can now see your report online with the data from your Amazon SageMaker Unified Studio project.

Clean up

To avoid additional charges after testing, delete the Amazon SageMaker Unified Studio domain and EC2 instances. Refer to Delete domains and Terminate Instances for instructions.

Conclusion

In this two-part series, you connected Power BI to Amazon SageMaker Unified Studio through Amazon Athena. Part 1 covered IDC-based domains. This post covered IAM-based domains using SageMakerIam authentication. This provides a direct connection path, with no third-party licensing, while maintaining data governance and security.

You can automate many steps of this process. For information about automating DSN creation on the Power BI Gateway or Service, refer to How ENGIE automates the deployment of Amazon Athena data sources on Microsoft Power BI. If you don’t want users adding the gateway IAM role directly, you can create a custom blueprint as a self-service tool for gateway role addition. The blueprint uses a ProjectMembership resource with a configurable parameter that project owners can activate at project creation, automatically adding the gateway role as a project contributor.

For additional best practices, refer to the Using Microsoft Power BI with the AWS Cloud Whitepaper. To learn more, visit Amazon SageMaker Unified Studio and Amazon Athena.


About the authors

Ramesh H Singh

Ramesh H Singh

Ramesh is a Senior Product Manager Technical at AWS in Seattle, focused on Amazon SageMaker. He’s passionate about building analytics and AI products that help enterprise customers unlock real value from their data. Away from work, he spends his time hiking with family and exploring spirituality. Connect with him on LinkedIn.

Armando Segnini

Armando Segnini

Armando is a Senior Analytics Specialist Solutions Architect at AWS, partnering with enterprise customers to architect scalable data, analytics, and AI platforms. He helps organizations turn complex data challenges into business value through expertise in streaming, BI integration, and generative AI. Outside of work, Armando enjoys traveling with his family, exploring new cultures, photography, and functional fitness competitions.

Gaurav Sharma

Gaurav is a Specialist Solutions Architect (Analytics) at AWS, supporting US public sector customers on their cloud journey. Outside of work, Gaurav enjoys spending time with his family and reading books.

Krishna Atluru

Krishna Atluru

Krishna is an Enterprise Support Lead TAM at AWS. He provides customers with in-depth guidance on improving security posture and operational excellence for their workloads, helping them build secure, resilient, and cost-effective solutions. His areas of expertise include building serverless architectures, and data and analytics solutions. Outside of work, Krishna enjoys cooking, swimming, and traveling.

Saushthav Saxena

Saushthav Saxena

Saushthav is a Software Development Engineer at AWS on the Amazon Athena team, where he has spent the past few years working on distributed systems and data analytics at scale. Based in the San Francisco Bay Area, his background spans full-stack development, high-performance computing, and large-scale infrastructure. Outside of work, he enjoys reading sci-fi novels, swimming, and traveling with family and friends.

Track SageMaker Unified Studio project costs with custom tags and AWS CUR

Post Syndicated from Nisha Gambhir original https://aws.amazon.com/blogs/big-data/track-sagemaker-unified-studio-project-costs-with-custom-tags-and-aws-cur/

Organizations running machine learning (ML), analytics, and generative AI workloads on Amazon SageMaker Unified Studio domains and projects face a common cost governance challenge. System tags (AmazonDataZoneDomainId and AmazonDataZoneProject) are automatically propagated to all underlying project resources. However, custom tags such as CostCenter, Team, or Environment are not propagated to dynamic resources created through the Studio UI. This creates a gap when you need to report project costs grouped by custom tags.

In this post, we walk through a serverless solution that bridges this gap by enriching AWS Cost and Usage Report (CUR) data with custom project tags. By the end of this post, you can build an Amazon Quick Sight dashboard to filter and analyze Amazon SageMaker Unified Studio project costs by any custom tag dimension that you define. This gives your team the visibility to make informed spending decisions.

Solution overview

The solution consists of three automated subsystems:

  1. Event-driven tag lookup management – An Amazon EventBridge rule captures Amazon DataZone project lifecycle events (Create, Update, Delete) and triggers an AWS Lambda function. The function maintains an Amazon DynamoDB lookup table that maps each project’s DomainId and ProjectId to its custom tags.
  2. CUR enrichment pipeline – An AWS Glue extract, transform, and load (ETL) job reads CUR 2.0 Parquet data from Amazon Simple Storage Service (Amazon S3). The job joins each billing line item with the DynamoDB lookup table using the system tags (DomainId, ProjectId), appends the custom tag values as new columns, and writes the enriched data back to Amazon S3.
  3. Cost visualization – An Amazon Quick Sight dashboard backed by a custom SQL dataset over Amazon Athena provides interactive cost and consumption analytics filtered by custom tags.

Architecture

The following diagram shows the end-to-end architecture:

Figure 1: SageMaker Unified Studio project custom tag cost reporting

The workflow is as follows:

  • An Amazon SageMaker Unified Studio administrator creates or updates a project with custom tags.
  • AWS CloudTrail captures the API call.
  • Amazon EventBridge matches the event.
  • The Lambda orchestrator writes the tag mapping to DynamoDB.
  • Separately, AWS Data Exports delivers CUR data to Amazon S3.
  • The AWS Glue ETL job enriches CUR line items with custom tags from DynamoDB.
  • The AWS Glue Crawler catalogs the enriched data.
  • Amazon Quick Sight visualizes costs by custom tags.

Prerequisites

Before deploying this solution, you need:

  • An Amazon SageMaker Unified Studio domain (you create projects after deployment).
  • AWS Cloud Development Kit (AWS CDK) CLI installed.
  • Python 3.12+.
  • Amazon Quick Sight Enterprise edition enabled in your account.
  • An AWS Identity and Access Management (IAM) user or role with permissions to deploy AWS CloudFormation stacks.

Step 1: Configure custom tags on your project profile

You configure custom tags on project profiles through the Amazon DataZone API. First, enable custom tags on your project profile:

aws datazone update-project-profile \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_PROFILE_ID \
  --region $REGION \
  --allow-custom-project-resource-tags \
  --project-resource-tags '[
  {"key": "CostCenter", "value": "default", "isValueEditable": true},
  {"key": "Team", "value": "default", "isValueEditable": true},
  {"key": "Environment", "value": "default", "isValueEditable": true}
]'

When creating or updating a project, set the tag values:

aws datazone update-project \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_ID \
  --project-profile-version latest \
  --region $REGION \
  --resource-tags '{"CostCenter": "CC-100", "Team": "ML-Platform", "Environment": "Production"}'

Important: The AmazonSageMakerProvisioning-<domainAccountId> role needs an inline policy that permits your custom tag keys. Without this, project environment deployment fails.

The following is the inline policy that’s used for the custom tags shared in this post:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCustomTagKeys",
      "Effect": "Allow",
      "Action": [
        "sagemaker:AddTags",
        "sagemaker:DeleteTags",
        "cloudformation:TagResource",
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack"
      ],
      "Resource": "*",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:TagKeys": [
            "AmazonDataZone*",
            "CostCenter",
            "Team",
            "Environment"
          ]
        }
      }
    }
  ]
}

Step 2: Activate cost allocation tags

Activate the SageMaker Unified Studio system tags as cost allocation tags so they appear in CUR data:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Active"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Active"}
]'

These tags take up to 24 hours to start appearing in CUR reports after activation.

Step 3: Deploy the infrastructure

The solution is packaged as a CDK application. Clone the GitHub repository and deploy:

# Install dependencies
pip install -r requirements.txt

# Bootstrap CDK (first time only)
cdk bootstrap aws://$ACCOUNT_ID/$REGION

# Deploy
cdk deploy

This creates the following resources:

  • DynamoDB table (smus-project-tag-lookup) – stores project-to-tag mappings.
  • Lambda function (smus-orchestrator) – processes project lifecycle events.
  • Amazon EventBridge rule – matches Amazon DataZone CreateProject/UpdateProject/DeleteProject events.
  • S3 buckets – for raw CUR and enriched CUR data.
  • AWS Glue ETL job (smus-cur-enrichment) – enriches CUR with custom tags.
  • AWS Glue Crawler – catalogs enriched data.
  • Amazon Simple Notification Service (Amazon SNS) topic – pipeline failure alerts.

Note: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight), so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Step 4: Configure CUR delivery

Create a CUR 2.0 export through AWS Data Exports that delivers Parquet files to the CUR S3 bucket created by the stack. The following screenshots show the complete configuration process in the AWS Billing and Cost Management console.

To create the export, follow these steps:

  1. Go to AWS Billing and Cost Management and then choose Data Exports.
  2. Choose Create in the upper right corner of the Exports and dashboards page. The Data Exports console shows any existing exports, their status, export type, data table, and last refresh date.
  3. On the Create export page, under Export details, select Standard data export and enter an export name. Under Data table content settings, select CUR 2.0.
  4. Under Data table configurations, set Time granularity to Hourly. The configuration page also lets you choose additional export content options such as including resource IDs, split cost allocation data, caller identity allocation data, and capacity reservation columns.
  5. Under Data export delivery options, set Compression type and file format to Parquet. Under Data export storage settings, configure the S3 bucket to: smus-cur-report-{account-id}-{region} and set the S3 path prefix as needed. Choose Create to finish.
Data Exports console listing existing exports with status, type, and last refresh date

Figure 2: Data Exports page listing existing exports

Create export page with Standard data export selected and CUR 2.0 chosen

Figure 3: Create export page with Standard data export and CUR 2.0 selected

Data table configurations with time granularity set to Hourly

Figure 4: Data table configurations with time granularity set to Hourly

Data export delivery options with Parquet format and the S3 storage destination configured

Figure 5: Data export delivery options with Parquet format and S3 storage settings

Step 5: How the event-driven tag capture works

When a project is created or updated in Amazon SageMaker Unified Studio (through the Studio UI or API), the following happens automatically:

  1. CloudTrail logs the Amazon DataZone API call.
  2. Amazon EventBridge matches the event.
  3. Amazon EventBridge invokes the Lambda function.
  4. The Lambda extracts custom tags from the CloudTrail event payload.
  5. The Lambda writes a record to DynamoDB with the DomainId, ProjectId, and all custom tag key-value pairs.

The Lambda function reads tags directly from the responseElements.resourceTags field of the CloudTrail event rather than making a separate GetProject API call. This avoids a race condition where GetProject might return empty tags while the project is in the UPDATING state.

def _extract_tags_from_event(detail):
    tags = {}
    response_elements = detail.get("responseElements") or {}
    for tag_entry in response_elements.get("resourceTags", []):
        if isinstance(tag_entry, dict) and "key" in tag_entry:
            tags[tag_entry["key"]] = tag_entry["value"]
    request_params = detail.get("requestParameters") or {}
    req_tags = request_params.get("resourceTags", {})
    if isinstance(req_tags, dict):
        tags.update(req_tags)
    return tags

Step 6: How the CUR enrichment works

The AWS Glue ETL job runs on a schedule (after each CUR delivery):

  1. Reads CUR Parquet files from the CUR S3 bucket.
  2. Reads all records from the DynamoDB lookup table.
  3. Performs a left outer join on DomainId and ProjectId.
  4. Appends custom tag columns (CostCenter, Team, Environment, and so on) to each CUR line item.
  5. Writes enriched Parquet to the enriched S3 bucket.

Line items without a matching project in the lookup table retain all original columns with NULL custom tag values. No data is dropped.

joined_df = cur_df.join(
    lookup_df,
    on=(
        (cur_df[DOMAIN_COL] == lookup_df["domainId"])
        & (cur_df[PROJECT_COL] == lookup_df["projectId"])
    ),
    how="left_outer",
)

Step 7: Set up the Amazon Quick Sight dashboard

After the first ETL run and crawler execution, set up the Amazon Quick Sight dashboard:

python scripts/setup_quicksight.py \
  --account-id $ACCOUNT_ID \
  --region $REGION \
  --quicksight-user $QUICKSIGHT_USER_ARN

This creates a dashboard with five visuals:

  • Cost by Custom Tag (CostCenter) – horizontal bar chart.
  • Cost by Project – horizontal bar chart.
  • Daily Cost Trend – line chart.
  • Cost by Service per Project – stacked bar chart.
  • Usage by Project & Service – summary table.

And six interactive list filters: Domain, Project, CostCenter, Team, Environment, Service.

The custom SQL includes a CASE statement for service categorization:

SELECT
  line_item_usage_start_date,
  line_item_product_code,
  line_item_usage_amount,
  line_item_unblended_cost,
  resource_tags_user_amazondatazone_domain_id AS domain_id,
  resource_tags_user_amazondatazone_project AS project_id,
  costcenter, team, environment,
  CASE
    WHEN line_item_product_code = 'AmazonSageMaker' THEN 'SageMaker'
    WHEN line_item_product_code = 'AmazonS3' THEN 'S3'
    WHEN line_item_product_code = 'AWSGlue' THEN 'Glue'
    ELSE line_item_product_code
  END AS service_category
FROM "smus_cost_reporting"."enriched_cur"
WHERE line_item_unblended_cost > 0

Step 8: Verifying the solution

After deploying the infrastructure and setting up the dashboard, verify that each component of the pipeline is functioning correctly.

8.1 Verify Amazon EventBridge is capturing project events

  1. Open the Amazon EventBridge console.
  2. In the navigation pane, choose Rules.
  3. Select the rule created by the CDK stack (for example, SmusCostReporting-ProjectTagRule).
  4. Choose the Monitoring tab.
  5. Confirm that the invocations are being recorded in the metrics.
  6. Create or update an Amazon SageMaker Unified Studio project with custom tags using the following command:
    aws datazone update-project \
      --domain-identifier <domain-id> \
      --identifier <project-id> \
      --custom-tags CostCenter=Engineering Team=DataPlatform Environment=Production

  7. Within a few seconds, the Amazon EventBridge rule should show a new invocation in its metrics.

8.2 Verify DynamoDB schema and tag mappings

The DynamoDB lookup table uses a simple key schema:

Attribute Type Role
domainId String Partition Key
projectId String Sort Key
CostCenter String Custom tag
Team String Custom tag
Environment String Custom tag

Custom tags are stored as dynamic attributes. Any tag key set on a project becomes a column in the table.

8.2.1 Verify DynamoDB table contains tag mappings

  1. Open the DynamoDB console.
  2. Navigate to the table created by the stack (for example, SmusCostReporting-ProjectTagsTable).
  3. Choose Explore table items.
  4. Scan for your project with the following keys:
    Partition key (domainId): <your-domain-id>
    Sort key (projectId): <your-project-id>

  5. Confirm the item contains the expected custom tag attributes (CostCenter, Team, Environment) with the values you assigned.
  6. Alternatively, use the AWS CLI:
    aws dynamodb get-item \
      --table-name SmusCostReporting-ProjectTagsTable \
      --key '{"domainId": {"S": "<domain-id>"}, "projectId": {"S": "<project-id>"}}'

8.3 Verify the AWS Glue ETL job enriches CUR data

  1. Wait for the next CUR delivery (hourly if configured as described in Step 4).
  2. Wait for the subsequent AWS Glue job execution.
  3. Open the AWS Glue console.
  4. In the navigation pane, choose ETL Jobs.
  5. Confirm the job completed successfully (status: Succeeded).
  6. Query the enriched data in Amazon Athena to confirm custom tag columns are populated:
    SELECT
      line_item_usage_start_date,
      line_item_product_code,
      line_item_unblended_cost,
      costcenter,
      team,
      environment
    FROM "smus_cost_reporting"."enriched_cur"
    WHERE costcenter IS NOT NULL
    LIMIT 10;

You should see rows with your custom tag values populated in the costcenter, team, and environment columns.

8.4 Verify the Amazon Quick Sight dashboard displays enriched data

  1. Open the Amazon Quick Sight console and navigate to the dashboard created by the setup script.
  2. Confirm that:
    • The Cost by Custom Tag (CostCenter) bar chart displays cost data grouped by your CostCenter values.
    • The list filters for CostCenter, Team, and Environment contain selectable values.
    • Selecting a filter value correctly narrows the displayed data.
  3. If the dashboard shows no data, verify that:
    • The AWS Glue Crawler has run after the ETL job (check the crawler’s last run status in the AWS Glue console).
    • The SPICE dataset has been refreshed. In the Amazon Quick Sight console, navigate to Datasets, select the dataset, and then choose Refresh now.

Figure 6 shows the Amazon Quick Sight dashboard with two side-by-side horizontal bar charts: Cost by Cost Center and Cost by Project. Domain Name and Project Name list filters appear at the top.

Amazon Quick Sight dashboard with Cost by Cost Center and Cost by Project bar charts and Domain and Project filters

Figure 6: Amazon Quick Sight dashboard showing cost data by custom tags, including Cost by Cost Center and Cost by Project bar charts with Domain Name and Project Name filters

Note: The first end-to-end cycle can take up to 48 hours depending on CUR delivery timing. After the initial cycle completes, subsequent updates will flow automatically on the configured schedule.

Operational considerations

Monitoring: The Amazon SNS topic smus-cost-reporting-alerts receives notifications when the AWS Glue ETL job fails or the Lambda orchestrator encounters repeated errors. Subscribe an email address or Slack webhook to stay informed. For instructions on how to create a subscription, see Subscribing to an Amazon SNS topic.

Cost: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight, SPICE) so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Scaling: The DynamoDB table uses on-demand capacity and can scale to accommodate your projects. You can scale the AWS Glue ETL job by increasing the number of workers for larger CUR datasets. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.

New tag keys: When you add new custom tag keys to projects, the ETL automatically picks them up as new columns. The AWS Glue Crawler’s UPDATE_IN_DATABASE policy adds new columns to the catalog table without manual intervention.

Cleanup

Warning: The following cleanup steps will permanently delete all CUR data, project tag mappings, and Amazon Quick Sight dashboards.

To remove all resources:

# Delete Amazon Quick Sight resources
python scripts/setup_quicksight.py --account-id $ACCOUNT_ID --region $REGION --quicksight-user $QS_USER --clean

# Delete CDK stack
cdk destroy

Go to AWS Billing and Cost Management, and then choose Data Exports and delete the CUR 2.0 export created in Step 4.

Deactivate the cost allocation tags that were activated in Step 2:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Inactive"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Inactive"}
]'

Conclusion

In this post, we showed how to build an end-to-end cost reporting solution for Amazon SageMaker Unified Studio projects using custom tags. This solution combines tag capture driven by Amazon EventBridge, CUR enrichment through AWS Glue ETL, and visualization in Amazon Quick Sight. With it, organizations can track and attribute costs by CostCenter, Team, Environment, or any custom dimension. This works even for resources created through the Studio UI that don’t receive custom tag propagation.

This solution serves as an extension to the custom tag propagation feature and reports cost for all project resources. The architecture is fully serverless, automated, and can be deployed to any AWS account using the provided CDK application.

To start building your custom tag cost reporting pipeline, visit the GitHub repository. To learn more about the underlying services, visit the Amazon SageMaker Unified Studio service page. For a related approach to custom tag governance, see Use Amazon SageMaker custom tags for project resource governance and cost tracking

References


About the authors

Nisha Gambhir

Nisha Gambhir

Nisha is a Senior AI/ML & Cloud Architect based out of India. She is passionate about helping customers design, architect and develop secure, scalable and reliable applications using AI/ML and Agentic AI. She loves working on latest technologies, providing simple and scalable solutions that drive positive business outcomes.

Dr Anil Giri

Dr Anil Giri

Anil is a Solutions Architect at AWS, based in London, UK, where he helps ISV customers design and deploy agentic AI systems in production. He specializes in multi-agent orchestration, retrieval-augmented generation, and event-driven serverless architectures on Amazon Bedrock, with a focus on building reliable, secure, and scalable solutions that deliver measurable business outcomes.

Satish Sarapuri

Satish Sarapuri

Satish is a Sr. Data Architect, Data Mesh / Data Lake/Gen AI at AWS. He helps enterprise-level customers build high-performance, highly available, cost-effective, resilient, and secure generative AI, data mesh, data lake, and analytics platform solutions on AWS, through which customers can make data-driven decisions to gain impactful outcomes for their business and help them on their digital and data transformation journey. In his spare time, he enjoys trail running and spending quality time with his family.

Ram Vittal

Ram Vittal

Ram is a Principal GenAI/ML Specialist at AWS. He has over 3 decades of experience building distributed, hybrid, and cloud applications. He is passionate about building secure, scalable, reliable AI/ML and big data solutions to help customers with their cloud adoption and optimization journey. In his spare time, he rides motorcycle and enjoys the nature with his family.

Secure SageMaker Unified Studio access with SAML and conditional policies

Post Syndicated from Manos Samatas original https://aws.amazon.com/blogs/big-data/secure-sagemaker-unified-studio-access-with-saml-and-conditional-policies/

Amazon SageMaker Unified Studio is a single data and AI development environment that brings together data preparation, analytics, and machine learning (ML) development in one place. By unifying these workflows, it saves teams from managing multiple tools and makes it straightforward for data scientists, analysts, and developers to build, train, and deploy ML models while collaborating. In Amazon SageMaker Unified Studio, a domain is the organizing entity for connecting your assets, users, and their projects. With Amazon SageMaker unified domains, you have the flexibility to reflect the data and analytics needs of your organizational structure. You can create a single unified domain for your enterprise or multiple domains for different business units.

Some enterprises, especially those in regulated industries, might require limiting access to trusted networks (such as VPN CIDRs) or to managed devices that meet compliance standards through device attestation.

In this post, we demonstrate how to integrate SageMaker Unified Studio as a custom SAML application and apply conditional access policies for enforcing device compliance, IP-based restrictions, or multi-factor authentication (MFA). For this post, we use Okta as the identity provider (IdP).

Solution overview

This solution demonstrates how to integrate Amazon SageMaker Unified Studio (SMUS) with external SAML identity providers such as Okta. The integration enforces enterprise security controls, including trusted network access, device compliance, and multi-factor authentication. With this integration, organizations in regulated industries can maintain strict access controls while providing single sign-on for their data science and AI development teams. By using SAML 2.0 federation with conditional access policies, you can help make sure that only authenticated users on compliant devices from trusted networks gain access. This access applies to your SageMaker Unified Studio domains and the associated data and AI workloads.

SAML authentication flow from a corporate device through the identity provider and AWS STS to Amazon SageMaker Unified Studio

Authentication flow for accessing SageMaker Unified Studio through SAML

The architecture diagram illustrates the secure authentication flow for accessing SageMaker Unified Studio through SAML integration:

  1. Users typically initiate access from corporate-managed devices through VPN or trusted network connections.
  2. The IdP authenticates the user and evaluates conditional access policies defined by your organization. Based on these policies, it checks for trusted devices, approved source IP ranges, and MFA completion. If any policy fails, the login is rejected. Otherwise, authentication proceeds.
  3. Upon successful authentication and policy validation, the IdP generates a digitally signed SAML assertion containing user attributes and group memberships, securely delivering it to the user’s browser through HTTP POST binding.
  4. The client browser automatically posts the SAML assertion to the AWS Security Token Service (AWS STS) sign-in endpoint. There, the AWS IAM Identity Provider validates the trust relationship with your corporate IdP through pre-configured SAML federation settings.
  5. AWS STS validates the SAML assertion signature and authenticity. It then maps the user attributes to a specifically configured IAM role with SageMaker Unified Studio permissions, including the datazone:GetIamPortalLoginUrl permission required for domain access.
  6. AWS STS confirms successful role assumption and generates temporary AWS credentials with a defined session duration. It then issues an HTTP redirect that returns the browser to the SageMaker Unified Studio domain with authenticated session tokens.
  7. Users gain access to the unified environment for data preparation, analytics, and machine learning development. All activities are governed by the assumed IAM role permissions and logged for comprehensive audit trails.

Walkthrough

In this walkthrough, you create a SAML application in Okta, connect it to AWS, and configure a SageMaker Unified Studio domain to use it for authentication.

Prerequisites

Before you get started, make sure you have the following:

  1. Familiarity with Amazon SageMaker Unified Studio.
  2. A basic understanding of SAML 2.0.
  3. AWS Identity and Access Management (IAM) permissions to create a domain in Amazon SageMaker Unified Studio.
  4. Access to your SAML IdP (such as Okta or Entra ID) to create and configure a SAML application.

Step 1: Create an application in Okta

The first step is to set up a new SAML application in Okta that manages authentication for SMUS.

  1. In Okta, go to Applications → Create App Integration, and choose SAML 2.0.
  2. Provide an App name.
  3. Set the Single sign-on URL to https://signin.aws.amazon.com/saml.
  4. Set Name ID format to Persistent.
  5. Set the Audience URI (SP Entity ID) to https://signin.aws.amazon.com/saml.
  6. Choose Next, and finish creating the application.
  7. Once created, copy the Metadata URL and Sign On URL. You need these in later steps.

Step 2: Create an identity provider in IAM

Now, let’s connect Okta to AWS by creating an IAM identity provider. This allows AWS to trust authentication responses from Okta.

  1. Open the IAM console.
  2. Go to Identity providers → Add provider.
  3. Select SAML as the provider type.
  4. Provide a Provider name.
  5. In Okta, go to your application’s Sign On tab, choose Identity Provider metadata, and save the XML file. Upload it here.
  6. Choose Add provider.
  7. Copy the ARN of this provider. You need it when you create the role.

Step 3: Create an IAM role for Okta

Next, create an IAM role that Okta can assume. This role defines what access users have when they sign in through Okta.

  1. In IAM, go to Roles → Create role.
  2. Use the following trust policy (replace both instances of “{Replace with Identity provider ARN}” with the ARN you copied in Step 2):
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "{Replace with Identity provider ARN}"
            },
            "Action": "sts:AssumeRoleWithSAML",
            "Condition": {
                "StringEquals": {
                    "SAML:aud": "https://signin.aws.amazon.com/saml"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "{Replace with Identity provider ARN}"
            },
            "Action": "sts:TagSession",
            "Condition": {
                "StringLike": {
                    "aws:RequestTag/Email": "*"
                }
            }
        }
    ]
}
  1. Attach a permission policy. For example:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "datazone:GetIamPortalLoginUrl",
            "Resource": "arn:aws:datazone:<REGION>:<ACCOUNT-ID>:domain/<DOMAIN-ID>"
        }
    ]
}

Replace <REGION>, <ACCOUNT-ID>, and <DOMAIN-ID> with the corresponding values from your SageMaker Unified Studio domain ARN (arn:aws:sagemaker:<REGION>:<ACCOUNT-ID>:domain/<DOMAIN-ID>). You can find the domain ARN in the SageMaker console under Domains.

Step 4: Configure SAML assertions

To make sure AWS understands who is signing in, configure the SAML assertions in Okta.

  1. Open your application in Okta.
  2. Go to General → SAML Settings → Edit.
  3. Choose Next until you reach Attribute Statements.
  4. Add the following mappings:
    • https://aws.amazon.com/SAML/Attributes/PrincipalTag:Email → user.email.
    • https://aws.amazon.com/SAML/Attributes/Role → {IAMROLEARN,IdentityProviderARN}.
    • https://aws.amazon.com/SAML/Attributes/RoleSessionName → user.email.

Step 5: Create an SMUS domain

Finally, let’s set up the SMUS domain and tie it all together.

Note: Creating a SageMaker Unified Studio domain incurs charges. For pricing details, see the Amazon SageMaker pricing page.

  1. Open the Amazon SageMaker console.
  2. Choose Create domain.
  3. Choose Manual setup (this allows for SAML integration).
  4. Enter a domain name, then choose Create.
  5. In Configure SSO user access, select SAML, then choose Next.
  6. Set the IdP SSO URL to the Sign On URL from Step 1.
  7. Select Do not require assignments. (Access is instead managed by your IdP team through Okta or Entra.)
  8. Choose Next, then choose Save.

To verify the integration works, open your SMUS domain and choose Sign in with SSO. You are redirected to Okta, and conditional access policies such as VPN, device attestation, or MFA apply automatically.

  1. Open your SMUS domain URL in a browser.
  2. Choose Sign in with SSO.
  3. Confirm that you are redirected to Okta for authentication.
  4. Sign in with your Okta credentials.
  5. Verify that you are redirected back to the SMUS domain with access to your projects.

Step 6: Assign users to the Okta application

Before users can authenticate through Okta to access SMUS, you must assign them to the application.

  1. In Okta, navigate to your SAML application.
  2. Go to the Assignments tab.
  3. Choose Assign, and select Assign to People or Assign to Groups.
  4. Select the users or groups who need access to SMUS.
  5. Choose Save and Go Back, then choose Done.

Step 7: Apply conditional access policies

Up to Step 5, we configured SMUS with an external SAML IdP. At this point, anyone assigned to the new application in your IdP can sign in and access the SMUS domain.

This is where conditional access policies come into play. Based on your organization’s governance model, you can add policies in your IdP to further control how and when users gain access. For example:

  • Restricting access to specific corporate IP address ranges (for example, only through VPN).
  • Enforcing device compliance so that only managed or secure devices can connect.
  • Adding MFA requirements for sensitive actions.
  • Applying device attestation to help assess whether the endpoint conforms to security baselines.

Most major IdPs, including Okta and Entra ID, support conditional access. You can find more details in their documentation:

These policies allow you to enforce the right level of protection, from something as simple as requiring users to connect through corporate networks to something as advanced as verifying device attestation across your fleet.

Clean up

To avoid incurring ongoing charges, delete the resources you created during this walkthrough:

  1. Delete the Amazon SageMaker Unified Studio domain from the SageMaker console.
  2. Delete the IAM role you created for Okta.
  3. Delete the IAM identity provider.
  4. Delete the SAML application in Okta.

Important: Deleting the SMUS domain permanently removes all projects, assets, and data within it. Back up any important work before proceeding.

Conclusion

By integrating SMUS with an external IdP through SAML, you can help enforce modern access controls based on your organization’s security requirements. This post walked through how to configure SMUS with a custom SAML application and pointed you toward resources for setting up conditional access policies.

With conditional access in place, you can decide, based on your organization’s needs, whether access should be limited to trusted users on trusted networks, trusted devices, or both. This approach can help provide a more secure and compliant login experience that aligns SMUS access with your company’s broader identity and security strategy.


About the authors

Amit Samal

Amit Samal

Amit is a Sr. Delivery Consultant in World Wide Public Sector, Professional Services at AWS working with UKGI Customers. Amit has been with AWS for about 4 years and has been helping customers across the UKGI to design & implement secure, resilient and cost-effective workloads on AWS. Amit is passionate about all areas of technology, but has focus areas in Networking, Migrations, and Application Modernizations.

Manos Samatas

Manos Samatas

Manos is a Principal Solutions Architect in Data and AI with Amazon Web Services. He works with government, non-profit, education and healthcare customers in the UK on data and AI projects, helping build solutions using AWS. Manos lives and works in London. In his spare time, he enjoys reading, watching sports, playing video games and socialising with friends.

Scaling fine-grained access control for enterprise lakehouse using SageMaker Unified Studio and AWS Lake Formation

Post Syndicated from Chintan Agrawal original https://aws.amazon.com/blogs/big-data/scaling-fine-grained-access-control-for-enterprise-lakehouse-using-sagemaker-unified-studio-and-aws-lake-formation/

As enterprise lakehouses grow to thousands of tables across multiple business domains and regions, scaling fine-grained access control becomes a critical governance challenge. Data governance teams spend significant time manually granting table-level permissions, only to face permission drift, inconsistent enforcement, and limited auditability. Without a scalable approach, each new dataset requires manual policy updates, increasing the risk of unauthorized access and slowing time-to-insight for analysts and data scientists.

In this post, we show you how to solve this problem by combining AWS IAM Identity Center, AWS Lake Formation tag-based access control (TBAC), and trusted identity propagation in Amazon SageMaker Unified Studio. You deploy a complete governance architecture using AWS Cloud Development Kit (AWS CDK) that classifies data with LF-Tags, maps IAM Identity Center groups to tag-based policies, and enforces permissions at query time across analytics engines. The solution uses Apache Iceberg tables stored in Amazon Simple Storage Service (Amazon S3) and registered in the AWS Glue Data Catalog.

The core governance challenge

As organizations mature their lakehouse environments, governance complexity increases with each new dataset. Several challenges commonly emerge:

  • Explosive dataset growth: Iceberg-based lakehouses often contain thousands of tables distributed across raw, curated, and conformed zones. Each new dataset introduces additional governance requirements, making table-level permission grants operationally expensive.
  • Multi-domain data ownership: Enterprise lakehouses typically serve multiple business domains such as commercial analytics, clinical research, and regulatory reporting. These domains require strict isolation while still supporting controlled data sharing.
  • Regional data sovereignty: Organizations operating globally must enforce geographic boundaries for sensitive datasets. EU clinical trial data might be restricted by GDPR regulations, whereas US commercial datasets follow different compliance frameworks.
  • Sensitivity-based access controls: Within each domain, datasets vary in sensitivity. Pricing strategies, drug discovery research, and patient-related datasets require stricter access controls than standard operational data.
  • Role explosion: Pure RBAC approaches attempt to encode these dimensions into roles, leading to role proliferation. Manual Lake Formation grants at the table level create permission drift and limited scalability.

To address these challenges, enterprise lakehouse governance must satisfy several criteria:

  • Least-privilege access.
  • Dynamic scalability as new datasets are onboarded.
  • Multi-dimensional enforcement across domain, region, and sensitivity.
  • Auditability traceable to individual users.
  • Automation-ready, configuration-driven workflows.

TBAC addresses each of these challenges directly. Instead of granting permissions on individual tables, you define tag-based policies that automatically apply to any resource matching the tag expression. New datasets inherit access rules through tag inheritance, eliminating manual policy updates (solving explosive dataset growth). Domain and region tags enforce strict isolation between business units (solving multi-domain ownership and regional sovereignty). Sensitivity tags control access within domains without role proliferation (solving sensitivity-based controls and role explosion). The following sections describe the architecture that implements this model and walk you through deploying it end to end.

Reference architecture overview

The governance model integrates identity, metadata, and lakehouse services into a unified access architecture that enforces fine-grained permissions consistently across analytics and machine learning (ML) workloads. The architecture consists of five layers, each handling a distinct responsibility in the access control flow.

The following diagram illustrates the end-to-end architecture, showing how user identity flows from IAM Identity Center through SageMaker Unified Studio to Lake Formation for tag-based policy evaluation against the AWS Glue Data Catalog and Amazon S3 storage layer.

Architecture linking IAM Identity Center, SageMaker Unified Studio, Lake Formation, the Glue Data Catalog, and Amazon S3

Figure 1: End-to-end governance architecture for the enterprise lakehouse

1. Identity and authentication layer: IAM Identity Center manages user identities and group memberships, integrates with corporate identity providers, and provides centralized lifecycle management for enterprise users. IAM Identity Center groups represent business roles and serve as the principals that receive Lake Formation permissions.

2. Unified analytics and ML access layer: Amazon SageMaker Unified Studio serves as the primary interface where analysts, data scientists, and ML engineers discover datasets, run queries, and build ML workflows. Because SageMaker Unified Studio integrates with multiple compute engines, including Amazon Athena, AWS Glue, Amazon EMR, and Amazon Redshift, users can access data using their preferred analytics tools while maintaining consistent governance.

3. Governance and authorization layer: AWS Lake Formation provides fine-grained access control across AWS Glue catalog resources using LF-Tags. Instead of granting permissions directly on databases and tables, Lake Formation evaluates LF-Tag policies dynamically and grants or denies access at query time. Governance teams define access rules once, and Lake Formation automatically applies them to new datasets as they are onboarded.

4. Governance automation layer: Two AWS Lambda functions automate tag assignment and permission provisioning. JSON metadata configuration files drive both pipelines, so governance teams manage access control through configuration rather than manual console operations.

5. Metadata and storage layer: Apache Iceberg tables stored in Amazon S3 form the foundation of the lakehouse. You register these tables in the AWS Glue Data Catalog, which provides centralized metadata management and interoperability across analytics services. Lake Formation evaluates governance decisions at the catalog level rather than independently by each analytics engine.

End-to-end access flow

When a user queries a dataset from SageMaker Unified Studio, the following sequence occurs:

  1. The user authenticates through IAM Identity Center and accesses SageMaker Unified Studio.
  2. SageMaker passes the user’s identity context to downstream analytics services using trusted identity propagation.
  3. The analytics engine requests data access from Lake Formation.
  4. Lake Formation evaluates LF-Tag policies against the user’s IAM Identity Center group membership.
  5. Access is granted or denied dynamically at query time.

Because authorization decisions are centralized in Lake Formation, governance remains consistent regardless of which analytics engine the user employs.

Hybrid RBAC + ABAC governance model

The governance model combines identity context from IAM Identity Center with metadata-driven classification using LF-Tags. The following table summarizes how each layer contributes to the overall governance workflow.

Governance capability IAM Identity Center contribution Lake Formation LF-Tag contribution Governance outcome
Identity context Organizes users into groups aligned with business roles Evaluates permissions using group membership Role-aligned access boundaries
Data classification Provides role eligibility for data access Classifies datasets by domain, region, sensitivity, and layer Attribute-aware authorization
Scalability Simplifies user lifecycle management Automatically applies policies to newly tagged datasets Governance that scales with dataset growth
Operational model Centralizes role lifecycle operations Enables metadata-driven policy automation Reduced administrative overhead

IAM Identity Center defines who can request access, LF-Tags define what datasets are eligible, and Lake Formation enforces policies dynamically at query time.

Enterprise LF-Tag data model

A structured tagging strategy is the foundation of scalable Lake Formation governance. In this solution, the solution classifies datasets across four governance dimensions.

Tag Key Tag Values Purpose Example Usage
region us, eu, global Geographic data location Enforce GDPR compliance for EU data
domain commercial, clinical_research, regulatory Business domain Separate commercial from clinical data
data_class standard, sensitive, regulated Data sensitivity level Restrict access to sensitive pricing data
layer raw, curated, conformed Data processing stage Grant analysts access to curated data only

Together, these dimensions enable multi-dimensional authorization policies that reflect both organizational structure and regulatory requirements.

Tag inheritance and evaluation

LF-Tags can be applied at three resource levels within the Glue Data Catalog: database, table, and column. In this implementation, database-level tags define broad governance attributes (domain, region, layer), table-level tags capture dataset-specific sensitivity (data_class), and column-level tags can further restrict access to individual fields. Lake Formation evaluates the effective tag set at query time by combining inherited and explicitly assigned tags.

For example, a database tagged domain=commercial, region=us, layer=raw automatically applies those tags to all tables within it. A table-level data_class=sensitive tag supplements the inherited tags to distinguish sensitive pricing data from standard sales data. This inheritance model means new tables automatically receive governance coverage without manual tag assignment. To learn more, refer to Lake Formation tag-based access control best practices.

Prerequisites

Before deploying the solution, complete the following setup in the us-east-1 Region. Use the same AWS Region throughout all steps.

  1. AWS account and IAM Identity Center: Enable IAM Identity Center and create test users. Note your Identity Store ID from the IAM Identity Center console under Settings. For setup guidance, see Getting started with IAM Identity Center.
  2. Lake Formation configuration: Complete the following setup in the Lake Formation console:2.1. Change Data Catalog default permissions. In the navigation pane under Administration, choose Data Catalog settings. Uncheck Use only IAM access control for new databases and uncheck Use only IAM access control for new tables in new databases. Choose Save. This makes sure Lake Formation permissions govern access to databases and tables created by the CDK stacks.
    Lake Formation Data Catalog settings with both IAM-only access control checkboxes cleared

    Figure 2: Lake Formation Data Catalog settings with both IAM-only access control checkboxes unchecked

    2.2. Integrate with IAM Identity Center. Complete the prerequisites for IAM Identity Center integration with Lake Formation, including enabling trusted identity propagation.You don’t need to manually create a Lake Formation administrator. The CDK deployment in Step 2: Deploy all stacks automatically registers the required administrators via the LfAdminStack (see lf-admin-stack.ts). S3 data location registration is a post-deployment console step covered after the CDK creates the buckets.

  3. SageMaker Unified Studio: Create a SageMaker Unified Studio domain, select your IAM Identity Center instance for authentication, and enable trusted identity propagation. For a detailed walkthrough, see Accelerate your analytics with Amazon S3 Tables and Amazon SageMaker Lakehouse and enable trusted identity propagation for the domain.
  4. Local tooling: Install AWS Command Line Interface (AWS CLI), Python 3.x, Node.js 18+, AWS CDK CLI (npm install -g aws-cdk), and Git.

Solution overview

Now that you understand the governance model and tag taxonomy, the following section walks you through deploying the complete infrastructure and configuring access control.

The deployment uses AWS CDK (TypeScript) and consists of seven stacks that create the complete governance infrastructure. The CDK app manages stack dependencies automatically, so a single cdk deploy --all command deploys everything in the correct order.

The architecture uses a two-layer data lake pattern. The raw layer stores data as CSV files in Amazon S3, registered as external tables in the AWS Glue Data Catalog. The curated layer uses Apache Iceberg v2 tables for ACID transactions and schema evolution. Three business domains (US Commercial, EU Clinical Research, and Global Regulatory) each have one representative table per layer, giving six tables total.

Lake Formation tag-based access control (TBAC) governs all access using four tag dimensions:

Tag Key Values Purpose
domain commercial, clinical_research, regulatory Business domain isolation
region us, eu Geographic data boundary
data_class standard, sensitive, regulated Sensitivity classification
layer raw, curated Data layer identification

Step 1: Clone the repository and install dependencies

Clone the accompanying repository and install the CDK project dependencies:

git clone https://github.com/aws-samples/sample-aws-smus-governance-automation
cd aws-smus-governance-automation/cdk
npm install

The CDK project is written in TypeScript and uses aws-cdk-lib v2. The lib/ directory contains seven stack definitions, and bin/app.ts wires them together with explicit dependency ordering.

If this is your first CDK deployment in this account and Region, bootstrap the CDK environment. Bootstrapping provisions an S3 bucket and IAM roles that CDK uses to deploy assets:

cdk bootstrap aws://<ACCOUNT_ID>/us-east-1

Step 2: Deploy all stacks

Deploy the entire infrastructure with a single command. Pass your IAM Identity Center Identity Store ID as a CDK context variable:

cdk deploy --all -c identityStoreId=d-xxxxxxxxxx --require-approval never --region us-east-1

CDK will prompt for IAM permission changes on each stack. The --require-approval never flag auto-approves these so the deployment runs unattended.

CDK deploys the seven stacks in dependency order:

  1. LfSetupStack: Lake Formation admin registration + LF-Tags (domain, region, data_class, layer)
  2. GlueRawTablesStack: S3 bucket + three Glue databases + three CSV-backed tables.
  3. GlueCuratedTablesStack: S3 bucket + three Glue databases + three Iceberg v2 tables.
  4. SsoGroupsStack: three IAM Identity Center groups (DataLake-US-Commercial, DataLake-EU-Clinical-Research-Sensitive, DataLake-Regulatory)The three groups map to specific tag combinations that control data access:
    • DataLake-US-Commercial: domain=commercial, region=us, data_class=standard.
    • DataLake-EU-Clinical-Research-Sensitive: domain=clinical_research, region=eu, data_class=sensitive,regulated.
    • DataLake-Regulatory: domain=regulatory (all regions, all data classes within regulatory).

    The following table summarizes the user personas, their group assignments, and the data access each group provides:

  5. AssetTaggingAutomationStack: Tag automation Lambda.
  6. SsoPermissionAutomationStack: Permission automation Lambda.
  7. LfAdminStack: Registers CDK + Lambda roles as Lake Formation admins.

After deployment completes, review the CloudFormation stack outputs. They include S3 bucket names, database names, SSO group IDs, and Lambda function ARNs.

The following figure shows all seven CDK stacks deployed successfully in the CloudFormation console.

CloudFormation console showing all seven CDK stacks in CREATE_COMPLETE status

Figure 3: CloudFormation console showing all seven CDK stacks in CREATE_COMPLETE status

Register S3 data locations with Lake Formation: Now that the S3 buckets exist, register them with Lake Formation. In the Lake Formation console, under Administration, choose Data lake locations, then choose Register location. Register both buckets from the stack outputs (for example, s3://datalake-raw-data-<ACCOUNT_ID>-us-east-1 and s3://datalake-curated-data-<ACCOUNT_ID>-us-east-1). For IAM role, use the default AWSServiceRoleForLakeFormationDataAccess and choose Lake Formation as the permission mode. See Registering an Amazon S3 location for step-by-step instructions.

The following figure shows both data lake S3 locations registered in the Lake Formation console.

Lake Formation Data lake locations page listing the registered raw and curated S3 buckets

Figure 4: Lake Formation Data lake locations page with raw and curated S3 buckets registered

Step 3: Populate sample datasets

The scripts use Amazon Athena to insert sample data. Athena stores query results under the athena-results/ prefix in the shared governance metadata bucket (lf-governance-metadata-<ACCOUNT_ID>-<REGION>) created by the CDK deployment.

Populate the raw and curated tables:

cd ../scripts
python3 populate_raw_layer.py
python3 populate_curated_layer.py

Each script executes INSERT INTO statements through the Athena StartQueryExecution API and waits for completion. You should see success messages for all six tables (three raw, three curated).

After populating the tables, you can verify the data in the Glue Data Catalog. The following figure shows the six tables across the three raw and three curated databases.

AWS Glue Data Catalog showing the six databases and tables created by the deployment

Figure 5: AWS Glue Data Catalog showing the six databases and tables created by the CDK deployment

You can also preview the data by querying a table. The following figure shows sample data from the us_sales_summary table.

Athena query results showing sample commercial rows from the us_sales_summary table

Figure 6: Query results for the us_sales_summary table with sample commercial data

Step 4: Apply LF-Tags to data assets

The following diagram illustrates the governance automation flow, showing how metadata JSON configuration files drive the two Lambda pipelines for asset tagging and SSO permission management.

Governance automation flow with the asset tagging and SSO permission Lambda pipelines

Figure 7: Governance automation flow showing the asset tagging and SSO permission Lambda pipelines

The diagram shows two parallel pipelines, each following three steps:

Asset tagging pipeline (left):

  1. Metadata upload – A data governance administrator uploads metadata JSON files (metadata-raw-tables.json and metadata-curated-tables.json) to the asset-tagging/ prefix in the shared S3 governance metadata bucket. These files define which LF-Tags to assign to each AWS Glue database and table.
  2. Lambda processing – The S3 upload triggers the LakeFormationTagAutomation Lambda function, which reads the metadata and calls the Lake Formation API.
  3. Tag operations – The Lambda creates or updates LF-Tags, then assigns them to the target databases and tables in the AWS Glue Data Catalog.

SSO permission pipeline (right):

  1. Permission upload – Three permission JSON files (one per IAM Identity Center group) are uploaded to the sso-permissions/ prefix. These files define the LF-Tag policy expressions that control data access.
  2. Lambda processing – The upload triggers the LakeFormationSSOPermissionAutomation Lambda function.
  3. Permission operations – The Lambda grants tag-based permissions to the corresponding IAM Identity Center groups through the Lake Formation API.

Both pipelines log execution details to Amazon CloudWatch for monitoring and troubleshooting.

Two metadata JSON configuration files drive the asset tagging Lambda that declaratively define which LF-Tags to apply to each AWS Glue resource:

  • metadata-raw-tables.json: Tag definitions for the three raw layer databases and tables.
  • metadata-curated-tables.json: Tag definitions for the three curated layer databases and tables.

Each entry in these files specifies the following fields:

Field Description Example
catalog_id Your AWS account ID (Glue Data Catalog ID) 123456789012
resource_type DATABASE or TABLE DATABASE
database_name AWS Glue database name raw_us_commercial_db
table_name AWS Glue table name (only for TABLE entries) us_sales_summary
lf_tags Array of LF-Tag key/value pairs to assign [{“TagKey”:“domain”,“TagValues”:[“commercial”]}]
access_type Action to perform (GRANT) GRANT

Parameters you must update before invoking: Replace the catalog_id value in every entry of both files with your own AWS account ID. The database and table names match the resources created by the CDK stacks, so those should not be changed unless you customized the stack parameters.

The following snippet from metadata-raw-tables.json shows a database-level entry and a table-level entry:

[
  {
    "comment": "DATABASE LEVEL TAGS - US Commercial RAW Domain",
    "access_type": "GRANT",
    "resource_type": "DATABASE",
    "catalog_id": "<YOUR_ACCOUNT_ID>",
    "database_name": "raw_us_commercial_db",
    "lf_tags": [
      { "TagKey": "region", "TagValues": ["us"] },
      { "TagKey": "domain", "TagValues": ["commercial"] },
      { "TagKey": "layer", "TagValues": ["raw"] }
    ]
  },
  {
    "comment": "TABLE LEVEL TAGS - US Commercial RAW Table (Standard Access)",
    "access_type": "GRANT",
    "resource_type": "TABLE",
    "catalog_id": "<YOUR_ACCOUNT_ID>",
    "database_name": "raw_us_commercial_db",
    "table_name": "us_sales_summary",
    "lf_tags": [
      { "TagKey": "data_class", "TagValues": ["standard"] }
    ]
  }
]

The Lambda applies tags at two levels: database-level entries assign domain, region, and layer tags, while table-level entries assign the data_class tag (standard, sensitive, or regulated). Because of two-level tagging, new tables added to a tagged database automatically inherit the database-level tags. Only the table-specific data_class tag needs explicit assignment. To learn more about this pattern, refer to Lake Formation tag-based access control best practices.

Invoke the Lambda for both layers:

cd ../lf-asset-tagging-automation
aws lambda invoke \
    --function-name LakeFormationTagAutomation \
    --payload fileb://metadata-raw-tables.json \
    --cli-binary-format raw-in-base64-out \
    response.json

aws lambda invoke \
    --function-name LakeFormationTagAutomation \
    --payload fileb://metadata-curated-tables.json \
    --cli-binary-format raw-in-base64-out \
    response.json

Verify tag assignment using the GetResourceLFTags API:

aws lakeformation get-resource-lf-tags \
    --resource '{"Table":{"DatabaseName":"raw_us_commercial_db","Name":"us_sales_summary"}}' \
    --region us-east-1

You should see domain=commercial, region=us, layer=raw, and data_class=standard in the response.

The following figure shows the LF-Tags assigned to the us_sales_summary table in the Lake Formation console, confirming that both database-level inherited tags and table-level tags are applied correctly.

Lake Formation console showing inherited and table-level LF-Tags on the us_sales_summary table

Figure 8: LF-Tags on the us_sales_summary table showing inherited and table-level tags

Step 5: Provision SSO group permissions

Three permission JSON files (one per IAM Identity Center group) define the LF-Tag policy expressions. Update sso_group with the group UUID from the SsoGroupsStack outputs and identity_center_account_id with your AWS account ID. For detailed configuration, see the repository README.

[
  {
    "sso_name": "DataLake-US-Commercial",
    "sso_group": "<GROUP_UUID_FROM_CDK_OUTPUT>",
    "identity_center_account_id": "<YOUR_ACCOUNT_ID>",
    "resources": [
      {
        "resource_type": "DATABASE",
        "permissions": ["DESCRIBE"],
        "lf_tag_expression": [
          { "TagKey": "domain", "TagValues": ["commercial"] },
          { "TagKey": "region", "TagValues": ["us"] },
          { "TagKey": "layer", "TagValues": ["curated", "raw"] }
        ]
      },
      {
        "resource_type": "TABLE",
        "permissions": ["SELECT", "DESCRIBE"],
        "lf_tag_expression": [
          { "TagKey": "domain", "TagValues": ["commercial"] },
          { "TagKey": "region", "TagValues": ["us"] },
          { "TagKey": "data_class", "TagValues": ["standard"] },
          { "TagKey": "layer", "TagValues": ["curated", "raw"] }
        ]
      }
    ]
  }
]

Apply permissions for each group:

cd ../lf-sso-permission-automation
python3 lambda_function.py us-commercial-permissions.json
python3 lambda_function.py eu-clinical-research-sensitive-permissions.json
python3 lambda_function.py regulatory-permissions.json
aws lakeformation list-permissions \
    --principal '{"DataLakePrincipalIdentifier":"arn:aws:identitystore:::group/<GROUP_ID>"}' \
    --region us-east-1

Step 6: Validate fine-grained access control

With all permissions in place, validate that Lake Formation TBAC enforces the correct access boundaries by signing in to SageMaker Unified Studio as different IAM Identity Center users.

Test as Sarah (US Commercial Analyst) — Sarah belongs to DataLake-US-Commercial, which grants access to standard commercial data only.

SELECT * FROM raw_us_commercial_db.us_sales_summary LIMIT 10;

Sarah sees all rows and columns successfully:

SageMaker Unified Studio results: Sarah’s successful query on us_sales_summary

Figure 9: Sarah’s successful query on us_sales_summary in SageMaker Unified Studio

Querying outside her authorized domain returns an access denied error:

SELECT * FROM raw_eu_clinical_research_db.eu_drug_discovery LIMIT 10;
Access denied error when Sarah queries eu_drug_discovery outside her domain

Figure 10: Access denied when Sarah queries eu_drug_discovery, confirming TBAC enforcement

Test as Dr. Chen (EU Clinical Research Lead) — Dr. Chen can access sensitive and regulated EU clinical research data (eu_drug_discovery) but is denied access to US commercial data (us_sales_summary), confirming regional and domain isolation.

Query results showing Dr. Chen’s successful query on eu_drug_discovery

Figure 11: Dr. Chen’s successful query on eu_drug_discovery

Access denied error when Dr. Chen queries us_sales_summary

Figure 12: Access denied when Dr. Chen queries us_sales_summary

Test as Alex (Regulatory Affairs Specialist) — Alex’s tag expression uses only domain=regulatory without a region constraint, granting cross-regional access to regulatory data while maintaining strict isolation from commercial and clinical research domains.

Query results showing Alex’s successful query on fda_submissions

Figure 13: Alex’s successful query on fda_submissions

Access denied error when Alex queries us_sales_summary

Figure 14: Access denied when Alex queries us_sales_summary

These tests demonstrate that TBAC enforces fine-grained permissions based on user identity, data classification, regional boundaries, and domain separation, without per-table permission grants. As new tables are added and tagged, existing groups automatically gain or are denied access based on their tag expressions. This is the core advantage of TBAC over named resource permissions.

Audit user access with CloudTrail

A key benefit of integrating Lake Formation with IAM Identity Center is the detailed audit trail available through AWS CloudTrail. Filter Event history by Event name GetDataAccess to see every data access event. Each record includes the IAM Identity Center user UUID (userIdentity.onBehalfOf.userId), the specific table accessed (requestParameters.tableArn), and confirmation that trusted identity propagation was used (additionalEventData.LakeFormationTrustedCallerInvocation: true).

CloudTrail GetDataAccess event showing Identity Center user identity and table access details

Figure 15: CloudTrail GetDataAccess event showing Identity Center user identity and table access details

To resolve the user UUID to a human-readable name, query the Identity Store:

aws identitystore describe-user \
    --identity-store-id d-xxxxxxxxxx \
    --user-id <USER_UUID_FROM_EVENT> \
    --region us-east-1

This audit capability provides the detailed access logs required for HIPAA, GDPR, and FDA compliance, showing exactly which users accessed which data and when. Learn about configuring CloudTrail for Lake Formation in Logging Lake Formation API calls with CloudTrail.

Cleanup

Run cdk destroy --all to remove all stacks. Manually delete the retained S3 data buckets (datalake-raw-data-* and datalake-curated-data-*) and revoke any remaining Lake Formation permissions. For detailed cleanup steps, see the repository README.

Conclusion

In this post, we showed you how to implement scalable fine-grained access control for an enterprise lakehouse by combining AWS Lake Formation tag-based access control, IAM Identity Center, and trusted identity propagation in SageMaker Unified Studio. The four-dimension LF-Tag taxonomy, hybrid RBAC + ABAC governance model, and metadata-driven Lambda automation together create a governance architecture where new datasets automatically inherit access policies through tag inheritance, permissions scale without per-table grants, and every data access event is auditable to the individual user through CloudTrail.

To extend this solution, consider adding new business domains, implementing column-level security with LF-Tags, scaling to multi-account architectures with Lake Formation cross-account sharing, or integrating additional analytics services such as Amazon Redshift Spectrum or Amazon EMR.

Get started by deploying the CDK stacks from the accompanying repository. To learn more:


About the authors

Chintan Agrawal

Chintan Agrawal

Chintan is a Solutions Architect with over 7 years of experience, with a specialization in Analytics and Healthcare domain. He possesses a strong enthusiasm for assisting clients in discovering valuable insights from their data. Through his expertise, he constructs innovative solutions that empower businesses to arrive at informed, data-driven choices.

Chaitanya Vejendla

Chaitanya Vejendla

Chaitanya is a Senior Solutions Architect and part of Global Healthcare and Life Sciences industry division at AWS. He focuses on developing strategic plans for building an end-to-end analytical strategy for large biopharma, healthcare, and life sciences organizations. His expertise spans across data analytics, data governance, AI, ML, big data, and healthcare-related technologies.

Govern Amazon Redshift Data Warehouses Data Across Accounts using Amazon SageMaker Unified Studio

Post Syndicated from Bandana Das original https://aws.amazon.com/blogs/big-data/govern-amazon-redshift-data-across-accounts-with-sagemaker-unified-studio/

Managing data governance across multiple Amazon Redshift clusters in different AWS accounts presents significant challenges. Organizations operating multiple Amazon Redshift clusters across AWS accounts often rely on manual processes for secure data sharing, which increases operational overhead and governance requirements. In this post, we show you how to use Amazon SageMaker Unified Studio to implement cross-account data sharing in Amazon Redshift using data mesh principles. We demonstrate how to build a scalable data mesh architecture that supports secure, auditable data sharing across AWS accounts while reducing operational burden.

Amazon SageMaker Unified Studio as the backbone of our data mesh

Amazon SageMaker Unified Studio is a data and AI development service which brings together functionality and tools from existing AWS Analytics and AI and machine learning (ML) services, including Amazon EMR, AWS Glue, Amazon Athena, Amazon Redshift, Amazon Bedrock and Amazon SageMaker AI. With the service, organizations can catalog, discover, share, and govern data stored across Amazon Web Services (AWS) without relying on manual coordination between AWS accounts.

A data mesh is an architectural approach that treats data as a product, with decentralized ownership by data producers while maintaining centralized governance. This architecture separates source systems, data producers (data publishers), data consumers (data subscribers), and central governance. The solution we present is tailored for cross-AWS account usage, creating a foundation for data governance so you can share data across Amazon Redshift clusters in different AWS accounts.

Our proposed solution addresses the following common challenges that organizations face when sharing data across AWS accounts:

  • Manual, ad-hoc data sharing processes are replaced with automated, event-driven data publishing to the SageMaker Unified Studio catalog.
  • Inconsistent governance across different use cases is resolved through a consistent governance framework with proper access controls.
  • High load on producer Amazon Redshift clusters is reduced through decoupled publishing that lowers the operational burden on data producers.
  • Complex credential management is simplified using AWS Secrets Manager and AWS KMS encryption.
  • Lack of auditable data publishing is addressed with full traceability of access and permissions supported by the SageMaker Unified Studio service.

With this approach, you can help reduce the time and effort required for cross-account data sharing while maintaining security and governance standards.

Architectural overview

The architecture spans three AWS accounts, each with a distinct role in the data mesh:

Central Data Governance Account (Account A) hosts the Amazon SageMaker Unified Studio domain, which serves as the unified catalog and governance layer for data discovery, access control, and subscription management across accounts.

Data Producer (Account B) hosts the source of data and processing workflows. Raw data lands in an Amazon Simple Storage Service (Amazon S3) source bucket and is processed through AWS Glue extract, transform, and load (ETL) jobs or Amazon Redshift auto copy into the Amazon Redshift source database. Amazon Redshift credentials are securely stored in AWS Secrets Manager.

Data Consumer (Account C) hosts the target Amazon Redshift database and analytics workflows. After access is granted, consumers can query shared data and connect downstream visualization tools.

While this diagram shows a single producer and consumer for simplicity, in a real-world deployment there might be hundreds of producer and consumer accounts connecting through the central governance layer. Amazon SageMaker Unified Studio scales to support this by providing a single place for managing data products regardless of the number of participating accounts.

The data sharing workflow is driven by Amazon SageMaker Unified Studio. The data owner publishes data to the catalog, where it becomes discoverable by consumers across accounts. Consumers browse the catalog, subscribe to data products, and the data owner approves the request. After approval, Amazon SageMaker Unified Studio handles the cross-account sharing, granting the consumer access without requiring direct connectivity between producer and consumer Amazon Redshift clusters.

Publishing Amazon Redshift data assets to the data mesh

In a data mesh architecture, data producers need to make their data products discoverable and accessible across the organization. Amazon SageMaker Unified Studio provides a centralized catalog where data assets can be published for consumer subscription.

In practice, this means registering your data sources with the catalog so they can be discovered, governed, and subscribed to by consuming teams. This section walks through the steps required to register Amazon Redshift data sources with SageMaker Unified Studio.

Before you can publish data assets from your producer account, you need to complete several configuration steps across your Amazon Redshift cluster, AWS Secrets Manager, and Amazon SageMaker Unified Studio.

Prerequisites

  • Install the AWS Command Line Interface (AWS CLI) (v2.15+ recommended).
  • Obtain temporary credentials with permissions to administer each account (producer, consumer, and domain account)
  • IAM permissions required: redshift:* on the relevant clusters, secretsmanager:CreateSecret / PutResourcePolicy / TagResource, kms:CreateKey / PutKeyPolicy / TagResource, datazone:* for subscription-target creation, and iam:PassRole for the Amazon Redshift cluster role.
  • Amazon Redshift clusters must use RA3 node types (ra3.xlplus, ra3.4xlarge, or ra3.16xlarge). Data sharing is not supported on other node types.
  • Amazon SageMaker Unified Studio domain must already be created in Account A with the Tooling and LakeHouseCatalog blueprints available.
  • All resources must be in an AWS Region where Amazon SageMaker Unified Studio is available.

Step 1: Account association and blueprint enablement

To implement the data mesh architecture described in the previous section, you need to set up the following accounts and enable the required blueprints. This ensures that the central governance layer can discover and manage data assets across your producer and consumer accounts.

This post uses three separate AWS accounts to illustrate the cross-account data sharing pattern. However, Amazon SageMaker Unified Studio also supports publishing and subscribing to data within a single account or across any number of accounts depending on your organizational setup. Additionally, this walkthrough uses a provisioned Amazon Redshift cluster, but Amazon SageMaker Unified Studio also supports Amazon Redshift Serverless for both publishing and subscribing to data assets.

Step 2: Configure your Amazon Redshift cluster and credentials

  • In the producer account (Account B), the data to be shared resides in an Amazon Redshift cluster.
  • Verify that your Amazon Redshift cluster uses node types from the RA3 family.
  • Add the following tags to your Amazon Redshift cluster.

Amazon Redshift console showing tags added to the cluster

  • Create a superuser in Amazon Redshift for Amazon SageMaker Unified Studio. For the Amazon Redshift cluster, the database user you provide in AWS Secrets Manager must have superuser permissions. With superuser permission, your Amazon Redshift cluster can publish data and subscribe from the data mesh created with Amazon SageMaker Unified Studio, and it manages the subscriptions (access) on your behalf. For reference, see the note section in this QuickStart guide with sample Amazon Redshift data.

Tag key-value pairs configured on the Amazon Redshift cluster

  • Store the user’s credentials in Secrets Manager. Select the credential type, enter the credential values, and choose the AWS Key Management Service (AWS KMS) key with which to encrypt the secret

QuickStart guide note about providing superuser credentials for Amazon SageMaker Unified Studio

Tags on the AWS Secrets Manager secret including the Amazon Redshift cluster ARN

Resource policy added to the AWS Secrets Manager secret for Amazon SageMaker Unified Studio access

  • If your secret is encrypted with a customer managed AWS KMS key, append the key policy with the following statement and add a tag to the key: AmazonDataZoneEnvironment = All. You can skip this step if you’re using an AWS managed KMS key.
{
    "Sid": "AllowSMUSRolesSecretsAccess",
    "Effect": "Allow",
    "Principal": {
        "AWS": "*"
    },
    "Action": [
        "kms:Decrypt",
        "kms:DescribeKey",
        "kms:GenerateDataKey"
    ],
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "kms:ViaService": "secretsmanager.<<AWS_Region>>.amazonaws.com"
        },
        "StringLike": {
            "aws:PrincipalArn": [
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift",
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/<<Redshift_Cluster_IAM_Role_Name>>",
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/datazone*",
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/service-role/AmazonSageMaker*"
            ]
        }
    }
},
{
    "Sid": "AllowSMUSRolesCreateGrant",
    "Effect": "Allow",
    "Principal": {
        "AWS": "*"
    },
    "Action": "kms:CreateGrant",
    "Resource": "*",
    "Condition": {
        "Bool": {
            "kms:GrantIsForAWSResource": "true"
        },
        "StringEquals": {
            "kms:ViaService": "secretsmanager.<<AWS_Region>>.amazonaws.com"
        },
        "StringLike": {
            "aws:PrincipalArn": [
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift",
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/<<Redshift_Cluster_IAM_Role_Name>>"
            ]
        }
    }
}

Note: Enable automatic rotation. Configure Secrets Manager automatic rotation for this secret with a rotation interval appropriate to your security policy (for example, every 30 days). When implementing rotation, verify that the rotation Lambda function updates the credentials in both Secrets Manager and Amazon Redshift database users simultaneously. Note that Amazon SageMaker Unified Studio retrieves the secret at connection time, so rotation must produce credentials that are valid immediately upon storage: use the alternating-users rotation strategy if you need to avoid downtime during rotation. See the Secrets Manager rotation documentation for setup instructions.

Using Amazon Redshift Serverless?

  • Add the following Tags to the Amazon Redshift Serverless namespace and workgroup.

Tags added to the Amazon Redshift Serverless namespace and workgroup

  • In the Secrets Manager secret, verify the host points to your Serverless endpoint.

AWS Secrets Manager secret showing the host pointing to the Serverless endpoint

  • Add the following tags to the AWS Secrets Manager secret.

Tags added to the AWS Secrets Manager secret for the Redshift Serverless credentials

Publish Amazon Redshift data to the data mesh

With prerequisites complete, you can now register your Amazon Redshift cluster as a data source in Amazon SageMaker Unified Studio.

Step 1: Create an Amazon Redshift type connection

  • Sign in to Account B, navigate to your Amazon SageMaker Unified Studio associated domain, and open the Amazon SageMaker Unified Studio URL.

Amazon SageMaker Unified Studio associated domain sign-in page

Add an Amazon Redshift connection form in Amazon SageMaker Unified Studio

  • The newly created Amazon Redshift connection appears here.

Newly created Amazon Redshift connection listed in Amazon SageMaker Unified Studio

Step 2: Create the data source for your Amazon Redshift data warehouse

Add an Amazon Redshift data source form in Amazon SageMaker Unified Studio

Amazon Redshift data source configuration in Amazon SageMaker Unified Studio

  • For Publishing settings, choose whether assets are immediately discoverable in Amazon SageMaker Catalog.

Publishing settings controlling asset discoverability in the Amazon SageMaker catalog

Using Amazon Redshift Serverless?

When creating the connection and data source, use your workgroupName instead of clusterName. The rest of the data source configuration remains the same.

Step 3: Run the data source and publish the data asset to the data mesh

Data source run configuration in Amazon SageMaker Unified Studio

Data source run results in Amazon SageMaker Unified Studio

  • During creation of data source if you choose Publishing settings such as assets are immediately discoverable, the Amazon Redshift tables and views appear in the catalog as Published, ready for discovery and subscription by data consumers.

Published Amazon Redshift tables and views in the Amazon SageMaker catalog

Data discovery view in the Amazon SageMaker Unified Studio portal

Subscribe Amazon Redshift data through the data mesh

To complete the end-to-end test, you need to set up a consumer Amazon Redshift cluster in Account C.

Step 1: Setting up the consumer cluster

  • Follow the prerequisites from Steps 1 and 2 in the previous section, make sure the cluster and secret are properly tagged as in the following screenshots:
  • Amazon Redshift cluster tags:

Tags applied to the consumer Amazon Redshift cluster

  • Tags for the AWS Secrets Manager secret that stores the user credentials for the Amazon Redshift cluster:

Tags on the AWS Secrets Manager secret storing the consumer cluster credentials

Step 2: Connect the consumer cluster to the data mesh

  • Log into Amazon SageMaker Unified Studio and navigate to your consumer project.
  • In the Compute section of your project, choose Add compute, then choose Connect to existing compute resources.
  • Choose Amazon Redshift Provisioned.
  • Select your consumer Amazon Redshift cluster from the dropdown list and enter the Secrets Manager name.
  • Choose Add compute.
  • Your newly added Amazon Redshift cluster should now show as available.

Consumer Amazon Redshift cluster added as compute in Amazon SageMaker Unified Studio

  • The newly added Amazon Redshift cluster shows an Available state.

Consumer Amazon Redshift cluster showing an Available state

  • In the Data section you can see that objects (table/views) from Amazon Redshift cluster are visible and you can query them.

Data section showing Amazon Redshift tables and views available to query

Step 3: Creating a subscription target

  • Find the tooling environment ID: in your local terminal after obtaining correct credentials as a project member, run this command to find the tooling environment ID.
export REGION='<your-region>'
export SUBSCRIBER_PROJECT_ID='<your-project-id>'
export DOMAIN_ID='dzd-xxxxxxx'

aws datazone list-environments \
  --domain-identifier $DOMAIN_ID \
  --project-identifier $SUBSCRIBER_PROJECT_ID \
  --region $REGION
  • In the response, find and copy the tooling environment ID as shown in the following example.
{
    "items": [
        {
            "projectId": "<PROJECT_ID>",
            "id": "<ENVIRONMENT_ID>",
            "createdBy": "SYSTEM",
            "createdAt": "<TIMESTAMP>",
            "updatedAt": "<TIMESTAMP>",
            "name": "Tooling",
            "awsAccountId": "<AWS_ACCOUNT_ID>",
            "awsAccountRegion": "eu-west-1",
            "provider": "Amazon SageMaker",
            "status": "ACTIVE",
            "environmentConfigurationId": "<ENVIRONMENT_CONFIGURATION_ID>"
        }
    ]
}
  • Locate the Manage Access Role: In Account C, navigate to SageMaker Unified Studio and find the Tooling blueprint. In the Provisioning Tab you will find the Manage Access role and copy the value, as it is needed for the next CLI call.

Provisioning tab showing the Manage Access role for the Tooling blueprint

  • Create the Subscription Target.

With all the information collected, you can create the subscription target for the Amazon Redshift cluster as shown by the CLI call.

export TOOLING_ENV_ID='<tooling-environment-id>'
export AUTHORIZED_PRINCIPAL='datazone_env_<tooling-env-id>'
export MANAGE_ACCESS_ROLE='arn:aws:iam::<account-id>:role/service-role/AmazonSageMakerManageAccess-<domain-id>'

aws datazone create-subscription-target \
  --domain-identifier $DOMAIN_ID \
  --environment-identifier $TOOLING_ENV_ID \
  --name "RedshiftCluster-default-target" \
  --subscription-target-config '[{
    "formName": "RedshiftSubscriptionTargetConfigForm",
    "content": "{\"databaseName\":\"<db-name>\",\"secretManagerArn\":\"arn:aws:secretsmanager:<region>:<account>:secret:<secret-name>\",\"clusterIdentifier\":\"<cluster-id>\",\"schemaName\":\"<schema-name>"}"
  }]' \
  --applicable-asset-types RedshiftViewAssetType RedshiftTableAssetType \
  --manage-access-role $MANAGE_ACCESS_ROLE \
  --provider "Amazon SageMaker" \
  --type RedshiftSubscriptionTargetType \
  --authorized-principals $AUTHORIZED_PRINCIPAL

Using Amazon Redshift Serverless?

Use RedshiftServerlessSubscriptionTargetType as the --type and RedshiftServerlessSubscriptionTargetConfigForm as the formName in the subscription target config. Replace clusterIdentifier with workgroupName in the content JSON.

  • Verify the Subscription Target.

To verify that the subscription target was created successfully, make a last CLI call. You should find in the return a new subscription target with the name RedshiftCluster-default-target.

aws datazone list-subscription-targets \
  --environment-identifier $TOOLING_ENV_ID \
  --domain-identifier $DOMAIN_ID \
  --region $REGION

Step 4: Subscribing to data assets

  • Open the data catalog inside SageMaker Unified Studio and search for the assets you want to subscribe to.

Data catalog search for assets to subscribe to in Amazon SageMaker Unified Studio

Adding multiple databases and schemas

To publish assets from entirely different databases on the same Amazon Redshift cluster, you need to create a separate data source for each database, meaning repeating the steps mentioned in the section before. Each data source points to the same cluster connection but specifies a different database name. This approach gives you independent control over scheduling, publishing settings, and metadata generation for each database’s assets.

On the consumer side, each subscription target is bound to a specific database and schema combination. This is the target location where SageMaker Unified Studio will create views that give the consumer access to subscribed assets. To receive subscribed data in multiple databases or schemas, you create one subscription target per database-schema combination. For example, different teams within the consumer account might want the data materialized in their own schema. The following example shows this pattern:

# Subscription target for the sales schema
aws datazone create-subscription-target \
  --domain-identifier $DOMAIN_ID \
  --environment-identifier $TOOLING_ENV_ID \
  --name "RedshiftCluster-sales-target" \
  --subscription-target-config '[{ "formName": "RedshiftSubscriptionTargetConfigForm", "content": "{\"databaseName\":\"consumer_db\",\"secretManagerArn\":\"arn:aws:secretsmanager:<region>:<account>:secret:<secret-name>\",\"host\":\"<endpoint>\",\"port\":\"5439\",\"schemaName\":\"sales\"}" }]' \
  --applicable-asset-types RedshiftViewAssetType RedshiftTableAssetType \
  --manage-access-role $MANAGE_ACCESS_ROLE \
  --provider "Amazon SageMaker" \
  --type RedshiftSubscriptionTargetType \
  --authorized-principals $AUTHORIZED_PRINCIPAL

# Subscription target for the marketing schema
aws datazone create-subscription-target \
  --domain-identifier $DOMAIN_ID \
  --environment-identifier $TOOLING_ENV_ID \
  --name "RedshiftCluster-marketing-target" \
  --subscription-target-config '[{ "formName": "RedshiftSubscriptionTargetConfigForm", "content": "{\"databaseName\":\"consumer_db\",\"secretManagerArn\":\"arn:aws:secretsmanager:<region>:<account>:secret:<secret-name>\",\"host\":\"<endpoint>\",\"port\":\"5439\",\"schemaName\":\"marketing\"}" }]' \
  --applicable-asset-types RedshiftViewAssetType RedshiftTableAssetType \
  --manage-access-role $MANAGE_ACCESS_ROLE \
  --provider "Amazon SageMaker" \
  --type RedshiftSubscriptionTargetType \
  --authorized-principals $AUTHORIZED_PRINCIPAL

Verifying the audit trail

To substantiate the governance and traceability claims in this architecture, enable AWS CloudTrail in all three accounts with data events for Secrets Manager and KMS. Enable Amazon Redshift audit logging on clusters to capture connection and query activity through STL_CONNECTION_LOG and STL_QUERY. Subscription approvals and rejections are recorded by SageMaker Unified Studio and emitted to CloudTrail under the datazone.amazonaws.com event source. Look for CreateSubscriptionRequest, AcceptSubscriptionRequest, and RejectSubscriptionRequest events.

Clean up

If you deployed this solution for testing or evaluation purposes and no longer need the resources, we recommend cleaning up to avoid unnecessary costs. Amazon Redshift clusters, Secrets Manager secrets, and SageMaker Unified Studio projects all incur charges when left running. The following steps guide you through a structured teardown in the correct order: subscriptions first, then data assets, and finally the infrastructure itself. This order verifies that no orphaned resources remain.

  • Remove all subscriptions
  • Delete your data assets
  • Delete the projects
    • Delete the project within your SageMaker Unified Studio Domain after all subscriptions are removed. Make sure to delete both the consumer and producer projects.
  • Delete the SageMaker Unified Studio Domain in Account A.

Conclusion

In this post, we demonstrated how Amazon SageMaker Unified Studio simplifies cross-account data governance for Amazon Redshift. By implementing this solution, organizations can move away from ad-hoc, non-auditable data sharing processes to a secure, scalable, and fully governed approach. Amazon SageMaker Unified Studio serves as the central governance layer that data producers and consumers use to publish, discover, and subscribe to data products across AWS accounts. This turns a fragmented data landscape into a well-governed data mesh without the need for custom tooling or manual coordination.

With cross-account data sharing and governance in place, the natural next step is to use this well-governed data for machine learning and generative AI workloads. Because Amazon SageMaker Unified Studio brings together data, analytics, and AI capabilities in a single environment, teams can more efficiently transition from discovering and subscribing to data products to building ML models and generative AI applications, all within the same environment. This reduces the traditional friction between data engineering and data science, accelerating time to value. To get started with establishing your organization’s data mesh using Amazon SageMaker Unified Studio, follow the guidance for Setting up Amazon SageMaker Unified Studio.


About the authors

Bandana Das

Bandana Das is a senior Data Architect in Amazon Web Services and specializes in Data and Analytics. She builds event-driven data architectures to support customers in Data management and data-driven decision making. She is also passionate about enabling customers on their Data management journey to the cloud.

Sindi Cali

Sindi Cali is a ProServe Consultant with AWS Professional Services. She supports customers in building data driven applications in AWS.

Anirban Saha

Anirban Saha is a DevOps Architect at AWS, specializing in architecting and implementation of solutions for customer challenges. He is passionate about well-architected infrastructures, automation, data-driven solutions and helping make the customer’s cloud journey as smooth as possible.

Stoyan Stoyanov

Stoyan Stoyanov works for AWS as a DevOps Engineer. He has more than 10 years of experience in software engineering, cloud technologies, DevOps, data engineering, and security.

Viral Thakkar

Viral Thakkar is a Software Engineer at AWS, working on Amazon DataZone and Amazon SageMaker Unified Studio with a primary focus on distributed systems and data governance with deep expertise in building large-scale data analytics and pipelining solutions. He is passionate about tackling complex distributed systems challenges while also creating tools and automated scripts that simplify day-to-day workflows and improve productivity.

Introducing Apache Spark Connect support in AWS Glue interactive sessions

Post Syndicated from Zach Mitchell original https://aws.amazon.com/blogs/big-data/introducing-apache-spark-connect-support-in-aws-glue-interactive-sessions/

When we built AWS Glue interactive sessions, our goal was to make AWS Glue as interactive as running local Python from a notebook. We mostly succeeded. With a straightforward Python package and a Jupyter notebook, you could execute remotely against the AWS Glue ephemeral Spark backend. The Livy-based approach was ahead of its time, but it had limitations from its REST-based protocol. Running local PySpark unlocked powerful integrated development environment (IDE) features such as debugging and linting, so your environment could understand the code and help you develop Spark applications more quickly. Customers would often split their development work. They used local Spark (or Docker containers) to develop in an IDE on a small amount of data, then switched to AWS Glue interactive sessions to validate scaling and tuning against the full dataset.

With modern PySpark releases came a new protocol: Apache Spark Connect. Spark Connect bridges the gap between these two worlds: you develop in local Python, but execute on AWS Glue against actual data. Today, AWS Glue interactive sessions support Spark Connect natively. You can connect from any environment that supports the PySpark remote() API, including VS Code, PyCharm, Amazon SageMaker Unified Studio notebooks, and standalone Python applications. You don’t need to install specialized kernels or manage cluster infrastructure.

What Spark Connect changes

Spark Connect, introduced in Spark 3.4, decouples the Spark client from the server through a lightweight gRPC protocol. Instead of running your driver program on the cluster, your IDE communicates with a remote Spark server through a thin client layer. This architecture unlocks the key workflow improvement: you develop locally and execute remotely.

Spark Connect architecture diagram showing a thin client communicating with a remote Apache Spark server

Spark Connect architecture — thin client with the full power of Apache Spark

With Spark Connect support in AWS Glue interactive sessions, you get:

  • IDE freedom – Use VS Code, PyCharm, JupyterLab, or any Python environment. No kernel installation required.
  • Programmatic access – Build Spark into your Python applications and automation scripts with a standard SparkSession.builder.remote() call.
  • Serverless execution – AWS Glue provisions and manages the Spark cluster. You pay only for the data processing units (DPUs) consumed while your session is active.
  • Spark Connect monitoring – The Spark Live UI now includes a dedicated Connect tab showing active Spark Connect sessions and operations alongside the existing Jobs, Stages, and Executors views.

Getting started with SageMaker Unified Studio

Amazon SageMaker Unified Studio provides the most direct path to Spark Connect on AWS Glue. The notebook environment handles session creation, endpoint retrieval, and token refresh automatically, so no connection boilerplate is required.

Prerequisite: You need an Amazon SageMaker Unified Studio project to use this workflow. If you don’t have one, create a project in your SageMaker Unified Studio domain first.

To connect to an AWS Glue Spark Connect session:

  1. Sign in to SageMaker Unified Studio, choose your project, and create or open a Notebook.

A notebook open in SageMaker Unified Studio

A notebook open in SageMaker Unified Studio

  1. Choose the compute icon in the left toolbar to open the Compute environment panel. Expand the Spark section.

Compute environment panel in SageMaker Unified Studio with the Spark section expanded

The Compute environment panel with the Spark dropdown list

  1. Select a Glue Spark connection. Depending on your SageMaker domain configuration, you will see either default.spark or named connections such as project.spark.compatibility. Select the appropriate Glue (Spark) connection and choose Apply.

Notebook cell showing spark.version returns 3.5.6-amzn-1 after connecting to Glue Spark Connect

Connected to Glue Spark Connect — running spark.version returns ‘3.5.6-amzn-1’

After you make your selection, you’re connected. The spark session object is available natively. No imports or configuration are needed. Start running PySpark immediately:

spark.sql("SHOW DATABASES").show()

The session manages itself in the background, including automatic token refresh.

Using the sagemaker_studio SDK

The sagemaker-studio Python package extends the Spark Connect experience beyond SageMaker Unified Studio notebooks into local IDEs, continuous integration and continuous delivery (CI/CD) pipelines, and any Python environment. The sparkutils module handles session initialization and connection configuration in a single call. You get the same streamlined experience as in the notebook, anywhere you run Python:

from sagemaker_studio import sparkutils

# Initialize a Glue Spark Connect session using your project connection
spark = sparkutils.init(connection_name="default.spark")

# Run queries immediately
spark.sql("SHOW DATABASES").show()

You can also use sparkutils.get_spark_options() to retrieve pre-configured Java Database Connectivity (JDBC) options for reading and writing to data sources through your project connections. Supported sources include Amazon Redshift, Amazon Aurora, and Amazon DocumentDB (with MongoDB compatibility):

# Get connection options for a Redshift connection in your project
options = sparkutils.get_spark_options("my_redshift_connection")

# Read from Redshift via Spark Connect
df = spark.read.format("jdbc").options(**options).option("dbtable", "analytics.orders").load()
df.show()

Within SageMaker Unified Studio, the sagemaker-studio SDK is native to the environment. The spark session and sparkutils are available without installation. For local IDE use, install it with pip install sagemaker-studio and configure credentials through an AWS named profile or boto3 session.

How it works

Spark Connect sessions in AWS Glue use a three-step workflow:

  1. Create a session – Call the CreateSession API with SessionType set to SPARK_CONNECT. The session provisions in approximately 30 seconds.
  2. Retrieve the endpoint – Call GetSessionEndpoint to receive a sc:// gRPC endpoint URL and a time-limited authentication token.
  3. Connect with PySpark – Pass the endpoint and token to SparkSession.builder.remote() and start running Spark operations.

Spark Connect protocol flow from the DataFrame API to a logical plan, sent over gRPC and protobuf, with results streamed back over gRPC and Arrow

Spark Connect protocol flow — DataFrame API translated to logical plan, sent via gRPC/protobuf, results streamed back via gRPC/Arrow

Connecting with the low-level API

Some environments don’t have the sagemaker-studio SDK, such as custom containers, AWS Lambda functions, or non-Python toolchains. In these environments, or if you’re not using SageMaker Unified Studio, you can use the AWS SDK (Boto3) to manage sessions directly. The following example demonstrates the full workflow:

import time, boto3, urllib.parse
from pyspark.sql import SparkSession

glue = boto3.client("glue", region_name="us-east-1")

# 1. Create a Spark Connect session
session_id = "my-spark-connect-session"
glue.create_session(
    Id=session_id,
    Role="arn:aws:iam::123456789012:role/GlueServiceRole",
    Command={"Name": "glueetl"},
    GlueVersion="5.1",
    SessionType="SPARK_CONNECT",
    DefaultArguments={"--enable-spark-live-ui": "true"},
)

# 2. Wait for the session to reach READY
while True:
    status = glue.get_session(Id=session_id)["Session"]["Status"]
    if status == "READY":
        break
    time.sleep(5)

# 3. Get the Spark Connect endpoint
sc = glue.get_session_endpoint(SessionId=session_id)["SparkConnect"]
endpoint_url = sc["Url"]
auth_token = sc["AuthToken"]

# 4. Connect with PySpark
encoded_token = urllib.parse.quote(auth_token, safe="")
connection_string = f"{endpoint_url}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
spark = SparkSession.builder.remote(connection_string).getOrCreate()
spark.sql("SELECT 1 + 1 AS result").show()

Monitoring with Spark Live UI

When you enable the Spark Live UI at session creation, you gain access to a real-time dashboard showing:

  • Jobs and Stages – Track active, completed, and failed jobs with stage-level metrics.
  • Executors – Monitor memory usage, shuffle data, and executor health.
  • SQL – Inspect query plans and execution details.
  • Connect tab – View active Spark Connect sessions and operations (specific to Spark Connect).

Access the dashboard through the GetDashboardUrl API or directly from the AWS Glue console.

import boto3, webbrowser

glue = boto3.client("glue", region_name="us-east-1")
dashboard = glue.get_dashboard_url(
    ResourceId="my-spark-connect-session",
    ResourceType="SESSION",
)
webbrowser.open(dashboard["Url"])

In SageMaker Unified Studio, no API call is needed. Choose Ready in the notebook status bar to open the kernel info popover. From there, open the Spark UI link for the live dashboard or Spark Driver Logs for real-time log output.

Notebook status bar Ready button that opens the Spark UI and Spark Driver Logs links

Image showing “Ready” in the status bar to access Spark UI and Driver Logs directly from the notebook

Token refresh

Authentication tokens expire after 30 minutes. In SageMaker Unified Studio, this is handled automatically. For programmatic use, you can use a background thread to keep the connection alive. The following helper reconnects transparently before the token expires:

import threading, time, boto3, urllib.parse
from pyspark.sql import SparkSession

class GlueSparkConnect:
    """Maintains a SparkSession with automatic token refresh."""

    def __init__(self, session_id, region="us-east-1", refresh_margin=300):
        self.session_id = session_id
        self.glue = boto3.client("glue", region_name=region)
        self.refresh_margin = refresh_margin  # seconds before expiry to refresh
        self._lock = threading.Lock()
        self.spark = self._connect()
        self._start_refresh_loop()

    def _connect(self):
        sc = self.glue.get_session_endpoint(SessionId=self.session_id)["SparkConnect"]
        encoded_token = urllib.parse.quote(sc["AuthToken"], safe="")
        remote_url = f"{sc['Url']}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
        self._token_expiry = sc["AuthTokenExpirationTime"].timestamp()
        return SparkSession.builder.remote(remote_url).getOrCreate()

    def _start_refresh_loop(self):
        def _loop():
            while True:
                sleep_for = max(self._token_expiry - time.time() - self.refresh_margin, 30)
                time.sleep(sleep_for)
                with self._lock:
                    self.spark = self._connect()
        t = threading.Thread(target=_loop, daemon=True)
        t.start()

# Usage
session = GlueSparkConnect("my-spark-connect-session")
session.spark.sql("SELECT 1 + 1 AS result").show()

The background thread sleeps until 5 minutes before token expiry, then transparently reconnects. Because the daemon thread exits when your script ends, there is no cleanup required.

Getting started

To start using Spark Connect with AWS Glue interactive sessions:

  1. Use AWS Glue version 5.1 (Apache Spark 3.5.6).
  2. Install PySpark 3.5.6 locally: pip install pyspark==3.5.6.
  3. Grant your AWS Identity and Access Management (IAM) identity permissions for glue:CreateSession, glue:GetSession, and glue:GetSessionEndpoint.
  4. Create a session with --session-type SPARK_CONNECT and connect from your preferred environment.

VPC note: If you connect to AWS Glue interactive sessions through a virtual private cloud (VPC) endpoint, add the new Spark Connect endpoint (com.amazonaws.{region}.glue.sessions) to your VPC configuration. Existing AWS Glue VPC endpoints don’t cover Spark Connect traffic.

For detailed instructions, see Connecting to a Spark Connect session in the AWS Glue Developer Guide.


About the authors

Zach Mitchell

Zach Mitchell

Zach is a Senior Big Data Architect at AWS Worldwide Specialist Organization for Analytics. He works with customers to design and build data applications on AWS, with a focus on SageMaker Unified Studio, AWS Glue, and AWS Lake Formation. Outside of work, he enjoys building things with code and occasionally writing about it.

Shrey Malpani

Shrey Malpani

Shrey is a Senior Technical Product Manager at AWS Analytics. He is focused on building and scaling data processing, data integration, and data management capabilities across services like AWS Glue, Amazon EMR, and Amazon Redshift that help customers build AI-ready data platforms for their analytics or machine learning workflows.

Vaibhav Naik

Vaibhav Naik

Vaibhav is a Software Engineer at AWS Glue, where he leads the development of enterprise Generative AI managed services and Agentic data systems. He has over a decade of experience designing massive-scale cloud infrastructure and distributed computing platforms.

Tom Olson

Tom Olson

Tom is a Software Development Engineer on the AWS Glue team, focused on Interactive Sessions and operational excellence. He brings over 20 years of software development experience, including government contracting and EC2 Networking at AWS. Outside of work, he enjoys running and playing board games.

Gaurav Krishnan

Gaurav Krishnan

Gaurav is a Software Development Engineer at AWS Glue. He has a deep interest in distributed systems and creating low-friction developer experiences for interactive data workloads on Apache Spark. In his spare time, he enjoys running and trying new restaurants.

Modernizing financial analytics with Amazon SageMaker Unified Studio

Post Syndicated from Umang Aggarwal original https://aws.amazon.com/blogs/architecture/modernizing-financial-analytics-with-amazon-sagemaker-unified-studio/

Avanse Financial Services is one of India’s leading education loan providers. Their Data Engineering Team had built a data lake on AWS using Amazon Simple Storage Service (Amazon S3), Amazon Athena, and AWS Glue for data ingestion and processing. However, their analytics and reporting layer ran on an external analytics application that wasn’t integrated with AWS. Data had to be copied from Amazon S3 into this external application before analysts could run any report, its license consumed a significant portion of their budget despite low utilization, and every integration with AWS services required custom-built pipelines.

After evaluating their options, Avanse migrated to a cloud-native lakehouse architecture using Amazon SageMaker Unified Studio, which unified their data engineering, analytics, and artificial intelligence (AI) workflows in a single governed environment on AWS. In this post, we walk through their migration journey so you can adapt their approach to your own environment.

Why Avanse chose to modernize

The separation between their AWS data lake and their external analytics application created five problems:

  1. Daily data synchronization bottleneck. Every report required a 4-hour batch copy from Amazon S3 into the external analytics application before analysts could query it. Business decisions were based on data that was at least a day old.
  2. Fixed licensing costs disconnected from usage. The external analytics application charged an annual fee regardless of how many queries analysts ran. Avanse needed usage-based pricing that matched what they actually consumed, not a fixed fee for capacity they weren’t using.
  3. Limited auditability. The external analytics application ran on a shared server where different business units (risk, collections, portfolio management) shared the same resources. It lacked granular audit trails, making it difficult to trace who accessed what data and when, or to allocate costs per team.
  4. No centralized data discovery. Although AWS Glue Data Catalog managed schema metadata for the data lake, the external analytics application couldn’t access it. Analysts working in that application relied on folder structures and manual documentation to find the right datasets, slowing onboarding and increasing the risk of using outdated data.
  5. Disconnected from AWS services. The external analytics application couldn’t query data in Amazon S3 or use AWS Glue catalogs natively. Every data flow required connectors and custom-built pipelines, adding maintenance overhead.

Additionally, some datasets were stored on Network File System (NFS) storage outside of Amazon S3, creating another data silo that needed to be consolidated.

Avanse chose Amazon SageMaker Unified Studio because it addressed all five challenges: direct querying of data in Amazon S3 avoiding synchronization, usage-based compute through Amazon Athena and Amazon EMR Serverless, project-based isolation with per-project billing, lineage tracking with AWS IAM Identity Center, and native integration with their existing AWS services.

Solution overview

The core architectural change was moving from a two-application model to a single integrated stack:

Previous architecture
Avanse’s data ingestion and processing ran on AWS (Amazon S3, AWS Glue, Athena), but analytics and reporting ran on an external analytics application. Data had to be batch-copied from Amazon S3 into this external application daily before analysts could query it. Each system had its own access controls, and there was no shared catalog or lineage tracking between them.
New architecture
Analytics now run directly against data in Amazon S3 through Amazon SageMaker Unified Studio. There’s no data copy step. Analysts query the same data that the ingestion pipelines produce, using Athena for SQL and EMR Serverless for large-scale processing. Governance, access control, and lineage are centralized through IAM Identity Center and SageMaker Catalog.

The following diagram illustrates the target architecture. It follows a lakehouse pattern, storing data in open formats on Amazon S3 while maintaining ACID transaction support for the consistency financial regulators expect.

Three-layer lakehouse architecture for Avanse on AWS, showing the data layer with Amazon S3 and AWS Glue Data Catalog, the compute layer with Amazon SageMaker Unified Studio, AWS Glue ETL, AWS Lambda, Amazon EMR Serverless, Amazon SageMaker AI, and Amazon Bedrock, and the governance layer with AWS IAM Identity Center, SageMaker Catalog, and Amazon DataZone

The architecture has three layers:

  1. Data layer – Amazon S3 stores data in open formats (Parquet, Delta Lake) with S3 Intelligent-Tiering for automatic cost optimization. AWS Glue Data Catalog maintains schema metadata, making data discoverable across tools.
  2. Compute layer – Amazon SageMaker Unified Studio provides project-based workspaces organized by business function. Collections uses the built-in SQL Query Editor powered by Athena, Risk Reporting uses JupyterLab for interactive analysis, and MIS runs large-scale Spark jobs through Amazon EMR Serverless. AWS Glue ETL handles data transformations and AWS Lambda provides event-driven triggers for report generation. For machine learning (ML) workloads, Amazon SageMaker AI supports model training and deployment, with Amazon Bedrock available for generative AI capabilities such as enhancing risk narratives.
  3. Governance layer – IAM Identity Center provides SSO and audit logging across workspaces. SageMaker Catalog serves as the business glossary with data lineage tracking and access controls. Amazon DataZone connects components through a common metadata layer.

Migration journey

Avanse followed a five-phase approach. The timelines can be adapted to your environment, but the systematic progression from validation through production deployment is key.

Phase 1: Technical validation (72-hour workshop)

Avanse started with a focused 72-hour workshop using isolated SageMaker environments where developers could experiment without impacting production. Their team tested SQL analytics against existing Athena tables and validated that Python and PySpark could replicate their existing analytics workflows.

The team confirmed that querying data directly in Amazon S3 addressed their synchronization bottleneck entirely. The 4-hour daily data copy was no longer necessary, which validated the migration approach.

Phase 2: Data migration and storage optimization

Avanse migrated datasets from NFS storage and legacy analytics formats into Amazon S3, consolidating the data into a single location. They implemented S3 Intelligent-Tiering, which automatically moves data between access tiers based on usage patterns, optimizing costs without impacting retrieval performance.

They replaced legacy analytics connectors with native Athena workgroups within SageMaker Unified Studio, avoiding data synchronization entirely. Source data remained in Amazon S3, queryable by both Athena SQL and SageMaker notebooks, establishing a single source of truth.

Phase 3: Compute modernization

Avanse moved from a shared analytics server to project-based isolation in SageMaker Unified Studio. Each business function (Risk Reporting, Collections, MIS) received its own project with dedicated compute spaces running JupyterLab. Project-specific IAM execution roles provided access controls and cost allocation per business unit.

A single browser-based URL with multi-factor authentication (MFA) now provides access to SQL analytics using the built-in query editor, ML development in JupyterLab notebooks, and big data processing through Amazon EMR Serverless. This replaced the need for local analytics client installations.

Phase 4: Governance implementation

Avanse deployed SageMaker Catalog as their central business data catalog. Analysts now discover approved datasets through semantic search rather than navigating folder structures or relying on manual documentation. They mapped technical Athena table names to business terms. For example, analysts search for “collection efficiency” and find the relevant tables with descriptions, schemas, and lineage.

Lineage capture traces each metric in risk reports back to source tables, transformations, and intermediate datasets. Every action (notebook execution, SQL query, data access) is tied to IAM Identity Center users, creating the comprehensive audit trail their compliance team needed.

Phase 5: Use case migration

Rather than attempting a big-bang migration, Avanse moved critical workflows one at a time:

Portfolio MIS (Monthly/Fortnightly)
Previously required the daily 4-hour data copy from Amazon S3 into the external analytics application before report generation could begin. Avanse avoided the data synchronization step entirely and now generates MIS reports by querying existing Athena tables directly in Amazon S3. Because the source data was already on AWS, there was no need to involve the external application for this activity. Report generation dropped from hours to under 30 minutes.
Collection Efficiency and Bounce Calculation
Ported complex legacy analytics procedures for calculating metrics like collection efficiency and bounce rates to event-driven processing using AWS Glue ETL, AWS Lambda, and PySpark jobs for high-volume data aggregation. The serverless execution model charges only for compute time consumed.
EDW Risk Reporting
Large-scale regulatory joins of Enterprise Data Warehouse assets previously ran as legacy scheduled procedures. These now run as SQL queries in the SageMaker Unified Studio query editor, where analysts execute them on-demand or schedule them through Athena workgroups. The distributed query engine handles complex multi-table joins spanning millions of rows.
Scorecard Generation
Model building shifted from the external analytics application to SageMaker AI workflows. Data scientists use JupyterLab with Python libraries and deploy models directly to SageMaker endpoints, avoiding data movement between separate environments.

Overcoming technical challenges

One technical challenge was code migration. Avanse’s analytics code base contained years of accumulated proprietary scripts and procedures. Direct line-by-line translation was not practical. Instead, they took a pragmatic approach: basic data transformations moved to SQL in Athena, complex business logic was rewritten in PySpark for scalability, and statistical procedures were replaced with Python libraries like pandas and scikit-learn. The approach was to focus on what the code accomplishes, then implement it using cloud-native patterns.

The other technical challenge was performance validation. The team needed to confirm that querying data in Amazon S3 would deliver acceptable performance compared to the external analytics application’s in-memory processing. Queries against Parquet-formatted data in Amazon S3 using Athena delivered comparable performance for standard reporting workloads, while avoiding the 4-hour daily data synchronization step entirely. For large-scale regulatory joins spanning millions of rows, Amazon EMR Serverless provided distributed Spark processing that completed in minutes rather than the hours required in the external application.

Key outcomes

Area Result
Licensing costs Avoided external analytics application fees entirely
Storage costs Reduced through S3 Intelligent-Tiering, which automatically moves data between access tiers based on usage patterns
Report generation From over 4 hours (including data synchronization from Amazon S3 to the external analytics application) to under 30 minutes with direct Amazon S3 querying
Compliance audits From weeks of manual investigation to days with automated lineage reports
Compute costs Usage-based serverless model replaced always-on external analytics infrastructure
Collaboration Unified browser-based environment for data scientists, analysts, and engineers

“By adopting SageMaker Unified Studio, we as the Data Team eliminated legacy licensing costs, reduced storage and compute expenses with a serverless, usage-based model, and accelerated our periodic report generation. At the same time, we transformed compliance and collaboration by cutting audit timelines while unifying our teams in a single, efficient data environment.” – Komal Thakkar, AVP – Lead, Data Engineering, Avanse Financial Services

Best practices

Based on their experience, Avanse recommends:

  • Start with a workshop. Validate your specific use cases in a 72-hour technical validation before committing to full migration.
  • Migrate use cases, not code. Focus on what your analytics accomplish, then implement using cloud-native patterns rather than translating legacy scripts line by line.
  • Invest in governance early. Implement the data catalog and lineage tracking from day one.
  • Embrace project-based isolation. Organize around business functions for clear cost allocation and security boundaries.
  • Document business logic. Use migration as an opportunity to capture undocumented knowledge in the business glossary and dataset descriptions.

Conclusion

Avanse’s migration from an external analytics application to Amazon SageMaker Unified Studio consolidated their analytics stack into a single integrated environment on AWS. By querying data directly in Amazon S3 instead of copying it into the external application, they alleviated their biggest operational bottleneck. Project-based isolation replaced a shared server model, giving each business unit independent compute and clear cost visibility. And centralized governance through SageMaker Catalog and IAM Identity Center gave their compliance team the audit trails they had been missing.

The serverless, usage-based model means Avanse no longer pays for idle capacity. The lakehouse architecture supports new analytics patterns as they emerge, and native integration with AWS services, including generative AI through Amazon Bedrock, positions them to adopt new capabilities as their needs evolve.

Next steps

Start your analytics modernization journey by scheduling a 72-hour technical validation workshop. Contact your AWS account team to discuss your migration approach.

For more information, see:

Detecting fraud patterns across Snowflake and AWS using SageMaker Data Agent

Post Syndicated from Akash Gupta original https://aws.amazon.com/blogs/big-data/detecting-fraud-patterns-across-snowflake-and-aws-using-sagemaker-data-agent/

Financial services organizations increasingly run analytical workloads across multiple systems. For example, customers typically store transaction records in Snowflake for its concurrency handling during peak volumes, while they store risk scores, customer profiles, and behavioral signals on AWS. To bridge that divide, practitioners have had to stitch together manual exports, custom extract, transform, and load (ETL) code, and external business intelligence (BI) tools to query both sources, cache expensive aggregations, and visualize results.

Amazon SageMaker Data Agent now closes these gaps with three new capabilities in Amazon SageMaker Unified Studio notebooks: SQL analytics on Snowflake data sources, materialized view management, and interactive charting. Practitioners can use them together to query Snowflake alongside AWS data, pre-compute and schedule repeated aggregations, and create interactive visualizations from natural language prompts in a single notebook, without writing boilerplate code or switching tools.

In this post, we describe the challenges these capabilities address, introduce each one, and walk through a fraud analytics scenario that demonstrates them working together in an end-to-end investigation workflow.

Challenges with fraud detection

Fraud analytics teams working in SageMaker Unified Studio notebooks encounter several recurring friction points that slow their path from alert to insight:

  • Querying across AWS and third-party warehouses. Customers store transaction data in Snowflake and maintain risk scores and customer profiles on AWS. SageMaker Data Agent supported SQL generation for AWS-native engines: Amazon Athena, Amazon Redshift, Apache Spark, and DuckDB. However, it didn’t yet generate Snowflake-dialect SQL. This created a gap for customers working with data distributed across both AWS services and Snowflake. Analysts had to write Snowflake SQL manually and export results as CSV files to join with AWS data. The process consumed 1–2 hours before any actual investigation could begin.
  • Rich visualization requires coding expertise. When analysts want to plot query results, they must write Python code using packages like matplotlib, seaborn, or plotly. They must choose the right chart type, format axes, handle data transformations, and debug rendering issues. For fraud teams whose expertise is in investigation rather than data visualization code, each chart becomes a detour: either learn the package interface, ask an engineer for help, or export to an external BI tool. This slows the exploratory cycle that fraud investigations depend on, where every new angle (time-of-day patterns, category breakdowns, geographic clusters) ideally takes seconds, not minutes of code iteration.
  • Expensive repeated queries with no caching. Fraud signal queries flag transactions that exceed a customer’s historical average and compute risk-score distributions by merchant category. These queries re-scan entire tables on each execution. A team running the same aggregation every morning over millions of rows pays the full compute cost each time, with no mechanism to pre-compute results or schedule automatic refreshes. For fraud teams, this means investigations start with a 30-minute wait for queries that ran identically yesterday.

These three friction points (accessing data across platforms, visualizing it interactively, and operationalizing repeated analyses) are what the new Data Agent capabilities address together.

What’s new in Data Agent

Snowflake connectivity

SageMaker Data Agent can now connect to Snowflake data warehouses through connections registered in Amazon SageMaker Unified Studio. The agent discovers available Snowflake databases, browses schemas progressively (databases → schemas → tables → columns), and generates Snowflake-dialect SQL, including Snowflake-specific syntax like FLATTEN, VARIANT column access, and semi-structured data handling. Analysts query Snowflake tables alongside AWS data sources from a single notebook conversation, and the agent handles dialect differences automatically: Snowflake SQL for extraction, Spark SQL for Amazon Simple Storage Service (Amazon S3) Tables operations, with no manual translation required.

Materialized view management

Data Agent now creates and manages materialized views through natural language prompts. Analysts describe the aggregation they want, for example, “create a materialized view that flags transactions where risk_score is above 0.7, refreshed every 6 hours,” and the agent generates the Spark SQL DDL, including SCHEDULE REFRESH syntax. Materialized views store pre-computed results in Apache Iceberg format for fast repeated access, turning expensive full-table scans into sub-second queries. Supported operations include create, refresh, drop, describe, and scheduled refresh. When asked, Data Agent can also analyze notebook query patterns and recommend which queries would benefit from materialization.

Interactive charting

Instead of generating matplotlib code that produces static images, Data Agent now creates native interactive chart cells powered by Vega-Lite. Supported chart types include bar, line, scatter, pie, area, heatmap, and more. Charts render inline in the notebook with hover tooltips, zoom, and filtering. Analysts can reconfigure them through the sidebar or by typing inline instructions like “change this to a heatmap showing volume by hour and category.” This removes the cycle of modifying Python plotting code or exporting to an external BI tool every time the analysis needs a different view.

Detecting fraud patterns across Snowflake and AWS: a walkthrough

Solution overview

In this section, we walk through how these three capabilities work together in a realistic fraud investigation. A fraud analytics lead at a mid-size fintech processes a high volume of card transactions daily. Customers store transaction data in Snowflake and maintain customer risk profiles on AWS.

This morning, the real-time alerting system flagged an unusual spike in declined transactions from a cluster of new accounts, all purchasing high-value electronics. The analyst suspects a fraud ring using synthetic identities, fabricated customer profiles that pass initial verification but share telltale patterns like similar device fingerprints or overlapping IP ranges. The analyst has three goals:

  • Confirm the fraud ring hypothesis. Determine whether the flagged accounts share device fingerprints, IP ranges, or behavioral patterns indicating coordinated fraud.
  • Quantify the exposure. Calculate total fraudulent transaction volume and identify all affected accounts, not only the ones that triggered today’s alert.
  • Set up ongoing monitoring. Create a reusable, auto-refreshing query so the team catches the next ring faster.

The analyst wants to do all of this without leaving the SageMaker notebook, without writing boilerplate data-engineering code, and within a single morning standup cycle so the investigations team can be briefed by noon.

How Data Agent approaches this analysis

Data Agent is context-aware. It discovers your actual table names, column schemas, and data source connections through Amazon SageMaker Unified Studio rather than requiring you to specify them manually. It generates SQL in the correct dialect for each source (Snowflake SQL for Snowflake, Spark SQL for S3 Tables) and operates within your existing AWS Identity and Access Management (IAM) permissions boundaries.

You interact with Data Agent in two modes: the Agent Panel for multi-step investigations like the example walkthrough that follows, where each prompt builds on previous context, and inline interactions for quick adjustments like “change this to a heatmap” directly on a chart cell.

Prerequisites

Before starting this walkthrough, verify that you have:

  • An Amazon SageMaker Unified Studio domain with a project configured.
  • A Snowflake account with a warehouse and USAGE grants on the database and schemas you want to query.
  • A Snowflake connection registered in your SageMaker Unified Studio project.
  • An S3 Tables catalog in your project containing customer data (or equivalent AWS-hosted tables for joining with Snowflake data).
  • A notebook open in SageMaker Unified Studio with Data Agent available in the chat panel.

Step 1: Explore Snowflake transaction data

What the analyst wants: Before investigating the fraud ring, the analyst must understand what data is available in Snowflake and verify recent transactions are accessible. The schema isn’t memorized (the payments team manages these tables), so Data Agent needs to discover the structure.

In the SageMaker notebook Agent Panel, the analyst types:

“Show me a preview of transactions over $500 for the last 24 hours. I’m looking for repeated high-value purchases that might indicate synthetic identity fraud.”

What Data Agent does for you: Data Agent discovers the Snowflake connection through SageMaker Unified Studio, browses the available databases, and locates PAYMENTS_DB → CARD_TRANSACTIONS schema → transactions table. It surfaces the column structure (transaction_id, customer_id, amount, merchant_category, transaction_timestamp, device_fingerprint, ip_address) so the analyst can confirm the right data is available without writing a single DESCRIBE TABLE statement.

Data Agent then generates a Snowflake-dialect SQL query to preview the last 24 hours of high-value transactions (amount > $500), returning hundreds of results. The preview immediately reveals what was suspected: alongside legitimate high-value purchases (mortgage payments, business supplies), there are clusters of electronics purchases at similar price points from different customer_id values but the same device_fingerprint, a classic synthetic identity pattern.

Data Agent querying Snowflake transaction data and generating equivalent code in the cell

Figure 1: Data Agent querying Snowflake transaction data and generating equivalent code in the cell.

Notebook cell results showing high-value Snowflake transactions

Figure 2: Displaying results when the notebook cell runs.

Step 2: Land Snowflake data into S3 Tables and join with risk profiles

What the analyst wants: Pulling historical high-value transactions into S3 Tables makes this data available for downstream analysis, including the materialized view that will cross-reference risk profiles automatically.

“Load the last 90 days of transactions where amount is greater than 500 into S3 Tables.”

What Data Agent does for you: Data Agent queries Snowflake to extract a large volume of high-value transactions from the last 90 days, converts the result to a PySpark DataFrame, creates an Apache Iceberg table at payments.fraud_analytics.high_value_transactions, and writes all the rows. Data Agent stores the transaction data (transaction_id, customer_id, amount, merchant_category, transaction_timestamp, device_fingerprint, ip_address) as Iceberg in S3 Tables, allowing you to query it entirely on AWS.

Data Agent handles the cross-source complexity: Snowflake-dialect SQL for extraction, automatic schema inference for the Iceberg table, and PySpark for the write. The analyst didn’t write a single line of ETL code.

Prompt sent to Data Agent to land Snowflake transactions into an S3 Tables

Figure 3: Sending a prompt to land Snowflake transactions into an S3 Tables catalog.

Generated PySpark code that reads transaction data from Snowflake

Figure 4: Reading data from Snowflake using code Data Agent generated.

Generated cell creating an S3 Tables Iceberg table populated with Snowflake data

Figure 5: Data Agent creating a new cell to create an S3 Tables Iceberg table and populate it with the Snowflake data.

Step 3: Create a materialized view for ongoing fraud monitoring

What the analyst wants: The pattern is confirmed, but re-running this expensive join across two tables every morning isn’t sustainable. A pre-computed view that automatically refreshes and surfaces transactions from high-risk customers means tomorrow’s investigation starts with answers instead of queries (goal #3, ongoing monitoring).

“Create a materialized view called mv_fraud_signals that joins high_value_transactions with customer_risk_profiles, flagging transactions where risk_score is above 0.7. Refresh it every 6 hours.”

What Data Agent does for you: Data Agent browses the S3 Tables catalog to discover both tables and their schemas, generates the Spark SQL DDL with SCHEDULE REFRESH EVERY 6 HOURS, and creates an INNER JOIN on customer_id with a risk_score > 0.7 filter. The resulting materialized view contains only the high-risk subset of transactions, and subsequent queries against it return significantly faster compared to a full table scan.

Data Agent can also recommend materialized views when asked. If the analyst prompts “analyze my notebook and suggest which queries would benefit from materialized views,” Data Agent examines query patterns and suggests candidates. This is useful when a team runs the same expensive aggregations repeatedly without realizing a materialized view would help.

New cell created by Data Agent to create the mv_fraud_signals materialized view

Figure 6: Data Agent creates a new cell to create the materialized view.

Generated query against the newly created materialized view

Figure 7: Data Agent adds code to query the newly created materialized view.

Step 4: Visualize fraud patterns with interactive charting

What the analyst wants: The data is ready, but the investigations team needs a clear visual story by noon to see which merchant categories are targeted and what time of day the fraud occurs, so they can build detection rules. The team needs interactive charts that can be explored on the fly, not static matplotlib images that need regenerating every time someone asks “what about category X?”

“Show me a scatter plot of flagged transactions: amount vs risk_score, colored by merchant_category.”

What Data Agent does for you: Data Agent queries the materialized view, generates a Vega-Lite specification, and renders an interactive scatter plot directly in the notebook cell, with no matplotlib code and no BI tool export. Hovering over any point reveals the transaction details. A dense cluster immediately stands out: Electronics & Computers transactions with risk scores between 0.75–0.95, all in the $950–$1,000 range.

Generated scatter plot of flagged transactions colored by merchant category

Detail view of the scatter plot highlighting the Electronics cluster

Figures 8, 9, and 10: Data Agent creates a scatter plot showing a dense cluster of Electronics transactions in the $950–$1,000 range with risk scores between 0.75–.95.

The analyst follows up with a second prompt to explore temporal patterns:

“Change this to a heatmap showing transaction volume by hour of day and merchant category.”

What Data Agent does for you: Data Agent generates a new heatmap visualization from the same materialized view. The heatmap reveals that Business Supplies and Mortgage Payments maintain steady transaction volumes throughout the day. However, Electronics shows a distinctly uneven temporal distribution, with noticeable volume dips during early morning hours (midnight to 5 AM) and late evening. This variability, absent in legitimate purchase categories, is a signal the detection rules team can act on immediately.

Heatmap of transaction volume by hour and merchant category

Detail view of the heatmap showing off-hours dips in the Electronics row

Figures 11 and 12: Data Agent creates a heat map to show transaction volume by hour of day and merchant category, revealing uneven temporal distribution in high-risk categories.

From insight to action

This investigation, from Snowflake connection to visual evidence, streamlined a workflow that previously required significant time across multiple tools. The analyst shares the notebook link with the investigations team, who confirm a fraud ring of dozens of synthetic identities responsible for significant fraudulent purchases. The temporal pattern, uneven Electronics transaction distribution with off-hours variability, is added to the company’s real-time detection rules that same afternoon.

The materialized view continues refreshing every 6 hours. The next morning, it flags three new accounts matching the same pattern, caught within hours of their first transaction instead of days.

Why SageMaker Data Agent for fraud analytics

This walkthrough demonstrates three new capabilities working together:

  • SQL analytics on Snowflake data sources removed the CSV export and manual ETL that consumed half of the investigation time.
  • Materialized view management turned a one-time query into persistent, auto-refreshing monitoring, transforming reactive investigations into proactive detection.
  • Interactive charting kept the entire analysis in the notebook, removing the BI tool context switch and making the inline exploration that revealed the Electronics temporal anomaly possible.

For the team, the combined effect is a reduction in time-to-insight, allowing faster fraud pattern analysis. This means daily fraud pattern reviews instead of weekly, and an investigation workflow that’s reproducible. The notebook itself serves as documentation for compliance and audit purposes.

Cleanup

The walkthrough creates notebook cells, SQL queries, and materialized views in your SageMaker Unified Studio session. To remove the generated cells, delete them from your notebook or delete the notebook itself.

If you created resources specifically for this walkthrough, remove the following to avoid ongoing charges:

  • Materialized view. In the notebook Agent Panel, prompt: “Drop the materialized view mv_fraud_signals.” This removes the Iceberg table from S3 Tables and cancels the scheduled refresh. Alternatively, run the Spark SQL statement DROP MATERIALIZED VIEW payments.fraud_analytics.mv_fraud_signals directly.
  • Landed Iceberg tables. Drop any tables created during the data landing step (for example, payments.fraud_analytics.high_value_transactions) by prompting Data Agent or running DROP TABLE in a Spark SQL cell. This removes the data from S3 Tables and the underlying Amazon Simple Storage Service (Amazon S3) storage.
  • SageMaker Unified Studio domain. If you created a domain solely for this walkthrough, delete it to stop incurring charges. Refer to the SageMaker Unified Studio administration guide for deletion steps.
  • Amazon S3 storage. Verify that dropping the materialized view and Iceberg tables removed the associated S3 objects. If residual Iceberg metadata files remain in your S3 Tables bucket, delete them manually.
  • Snowflake compute. No persistent Snowflake resources are created. Queries use your existing warehouse. Review your Snowflake query history to estimate the compute credits consumed during the walkthrough.

Conclusion

In this post, we walked through three new capabilities in Amazon SageMaker Data Agent for notebooks: Snowflake connectivity, materialized views, and native interactive charting. Using a fraud analytics scenario, we demonstrated how these features work together. We connected to a Snowflake warehouse to explore transaction data, landed results into S3 Tables and joined them with AWS-hosted risk profiles, created a materialized view for ongoing fraud monitoring, and visualized patterns with interactive charts that revealed temporal anomalies in Electronics transactions linked to dozens of synthetic identities.

These capabilities are available now in Amazon SageMaker Unified Studio. To get started, open a notebook in your SageMaker Unified Studio domain and begin a conversation with Data Agent in the chat panel.

To learn more, see the following resources:


About the authors

Akash Gupta

Akash Gupta

Akash is a Software Development Engineer on the Amazon SageMaker Unified Studio team, where he builds integrated tools and agentic experiences. An alumnus of Santa Clara University, he is passionate about building scalable solutions that simplify how customers interact with their data. In his spare time, he enjoys singing and cooking.

Mukesh Sahay

Mukesh Sahay

Mukesh Sahay is a Software Development Engineer at Amazon SageMaker, focused on building the SageMaker Data Agent. The agent provides intelligent assistance for code generation, error diagnosis, and data analysis recommendations for data engineers, analysts, and scientists. His work spans agentic AI architectures that transform natural language prompts into executable code and analysis plans across diverse data sources. An alumnus of San Jose State University, Mukesh brings over a decade and a half of experience in building scalable, intelligent data systems.

Eason Ma

Eason Ma

Eason is a Software Development Engineer within SageMaker’s Agentic AI Experiences. His focus is on building agentic infrastructure and intelligent data experiences that help users seamlessly interact with their data across multiple sources. He holds a Master’s in Computer Science from the University of Illinois at Urbana-Champaign and a Bachelor’s in Computer Science from the University of Tennessee, Knoxville. A proud Vol, he brings that same volunteer energy to everything he builds.

Anagha Barve

Anagha Barve

Anagha is a Software Development Manager on the Amazon SageMaker Unified Studio team. Her team is focused on building tools and integrated experiences for the developers using Amazon SageMaker Unified Studio. In her spare time, she enjoys cooking, gardening and traveling.

Siddharth Gupta

Siddharth Gupta

Siddharth is heading Generative AI within SageMaker’s Unified Experiences. His focus is on driving agentic experiences, where AI systems act autonomously on behalf of users to accomplish complex tasks. An alumnus of the University of Illinois at Urbana-Champaign, he brings extensive experience from his roles at Yahoo, Glassdoor, and Twitch.

AI-assisted data development with Kiro and SageMaker Unified Studio

Post Syndicated from Zach Mitchell original https://aws.amazon.com/blogs/big-data/ai-assisted-data-development-with-kiro-and-sagemaker-unified-studio/

AI coding assistants are transforming software development, but data engineering presents unique challenges: governed data access, shared compute environments, and compliance controls that are designed to remain in place. How do you bring the power of agentic AI development into a governed data environment? With the AWS Toolkit for Visual Studio Code, you can connect Kiro, VS Code, or Cursor directly to Amazon SageMaker Unified Studio.

When you connect your editor to a SageMaker Unified Studio Space (a cloud-based compute environment inside your project), you get AI-assisted development with your preferred tools while your data governance, project permissions, and compute are managed by SageMaker Unified Studio. Additionally, SageMaker Unified Studio automatically generates steering files (like AGENTS.md) that provide your AI assistant with context about your project environment, so it understands your data and project configuration from the first prompt.

This post demonstrates the integration using Kiro. The same Remote Access connection works with VS Code and Cursor. The post starts by showing what you can do with this integration: using natural language to explore and analyze data in a governed environment. We then walk through the setup so you can try it yourself.

What’s new

With the AWS Toolkit, you can connect Kiro, VS Code, and Cursor to your SageMaker Space over a secure SSH tunnel. No additional extensions or SSH key management required. After the connection is established, your IDE has full access to your Space’s file system, compute, and data services.

Two capabilities make this especially powerful for data work:

  • Automatic AI steering – When connecting Kiro to SageMaker Unified Studio,  Kiro generates AGENTS.md and smus-context.md files that provide your AI assistant with context about your environment, including project configuration, environment details, and utilities for discovering your data catalog and project structure. Kiro detects these files automatically; other editors can use them as context for their own AI features.
  • MCP server support – have Kiro discover and configure itself for the Model Context Protocol servers on your remote SageMaker space ( like smus_local and aws-dataprocessing) to give your agent direct access to your AWS Glue Data Catalog, Amazon Athena queries, and SageMaker Unified Studio project metadata.

The following diagram shows how the components connect:

Architecture diagram: How the components connect

See it in action: AI-assisted development with governed data

Before walking through the setup, we explain what you can do with this integration. This walkthrough uses Kiro as the editor. With Kiro connected to a SageMaker Unified Studio Space, MCP servers configured, and steering documents in place, we can use natural language to explore data and build analytics. The AI assistant has all the context it needs to do this well.

Note: Agentic AI output is nondeterministic. The exact code, tool choices, and responses Kiro produces will vary between sessions, even with the same prompt. The following walkthrough shows one representative session. Your experience will differ in the specifics, but the patterns and capabilities demonstrated here are consistent.

Step 1: Explore the data

Start with a simple prompt:

show my databases and the tables I have access to

Even with native MCP tools available, Kiro often prefers the AWS Command Line Interface (AWS CLI) and bash to retrieve information. This is expected and typically does not affect the outcome. If you prefer MCP tools for every operation, you can add that preference to a steering document.

Kiro used the sagemaker_studio SDK to discover the catalog:

python3 -c "
from sagemaker_studio import Project
project = Project()
conn = project.connection()
catalog = conn.catalog()
print('Databases:')
for db in catalog.databases:
    print(f'  - {db.name}')
"
Databases:
  - default
  - sagemaker_sample_db

Then it drilled into the table schema:

python3 -c "
from sagemaker_studio import Project
project = Project()
conn = project.connection()
catalog = conn.catalog()
db = catalog.database('sagemaker_sample_db')
print('Tables in sagemaker_sample_db:')
for t in db.tables:
    print(f'  - {t.name}')
    print(f'    Columns: {[(c.name, c.type) for c in t.columns]}')
"
Tables in sagemaker_sample_db:
  - churn
    Columns: [('state', 'string'), ('account_length', 'int'),
              ('area_code', 'string'), ('phone', 'string'),
              ('intl_plan', 'string'), ('vmail_plan', 'string'),
              ('vmail_message', 'int'), ('day_mins', 'double'),
              ('day_calls', 'int'), ('day_charge', 'double'),
              ('eve_mins', 'double'), ('eve_calls', 'int'),
              ('eve_charge', 'double'), ('night_mins', 'double'),
              ('night_calls', 'int'), ('night_charge', 'double'),
              ('intl_mins', 'double'), ('intl_calls', 'int'),
              ('intl_charge', 'double'), ('custserv_calls', 'int'),
              ('churn', 'boolean')]

Kiro discovered the sagemaker_sample_db.churn dataset, a sample dataset that ships with SageMaker Unified Studio containing 10,000 rows and 21 columns of customer churn data (state, account length, call minutes, service calls, churn flag, and more). Notice that we did not write any of this code. We asked a question in natural language, and Kiro chose the right SDK calls, explored the catalog, and surfaced the results.

Another, more natural way to get the same answer is to ask directly. Prompting “Let us sample the churn table.” yields the same catalog paths and schema output, along with additional metrics like row count and a data sample, all from a single conversational prompt:

SageMaker Unified Studio console showing the sagemaker_sample_db.churn dataset listed in the catalog

Figure 1 — The sagemaker_sample_db.churn dataset in the catalog

Schema view showing the 21 columns of the churn table including state, account_length, call minutes, and the churn boolean

Figure 2 — Churn dataset schema with 21 columns

from sagemaker_studio import sqlutils
result = sqlutils.sql(
    'SELECT COUNT(*) AS total_rows FROM sagemaker_sample_db.churn',
    connection_name='default.sql'
)
print('=== Total Row Count ===')
print(result)
=== Total Row Count ===
   total_rows
0       10000

With the schema and row count in hand, Kiro sampled the data to round out its understanding of the dataset:

Comprehensive data sample showing 10 rows from the churn table with all 21 columns populated

Figure 3 — Comprehensive data sample after Kiro catalog exploration

Step 2: Run analytics with full context

With the data explored, ask Kiro to run a data quality evaluation:

Can we run basic statistical evaluations for data quality?

Because Kiro had already explored the catalog and sampled the data, it made smart choices about how to run the analysis. Instead of using PySpark for this 10,000-row table, Kiro used Athena using sqlutils to run the evaluation directly. It produced a thorough data quality report:

  • 10,000 rows, 21 columns, zero nulls across all columns. Clean on that front.
  • 5,000 duplicate rows (50 percent). Significant, worth investigating before modeling.
  • Outliers minimal. Most columns have less than 1 percent outlier rate by IQR.
  • Churn is nearly 50/50 split (50.04 percent False, 49.96 percent True). Unusually balanced, indicating synthetic data.
  • Clear signal in key features. Churners and non-churners show differences in day_mins (7.52 vs. 3.52), eve_mins (5.95 vs. 4.11), and vmail_message (175 vs. 278).
  • State distribution roughly uniform (~2% each), intl_plan and vmail_plan near 50/50.

The key insight here is what Kiro did not do. It did not default to PySpark because the environment supports Spark. Having explored the data first, understanding the table size, column types, and that churn is a proper Boolean (not a string), Kiro independently chose the right engine for the workload and produced correct analytics on the first pass.

Best practice: Explore first, code second

Start every AI-assisted development session with data exploration. Ask your AI assistant to discover your catalog, sample your tables, and understand the schema before asking it to build anything. This single step helps reduce a common source of errors in AI-assisted data work: the LLM making assumptions about data it has not seen.

Exploring your data gives the large language model (LLM) the context it needs to properly help with your project. It saves hallucinations and rework, results in faster development time, and reduces token costs.

Ready to try it yourself? The following sections walk through the full setup: prerequisites, connecting your editor to your SageMaker Space, configuring MCP servers, and working with notebooks.

Prerequisites

Before you begin, make sure you have the following:

  • A SageMaker Unified Studio domain and project with at least one project that has a compute environment provisioned (Tooling or ToolingLight). These should come standard with every SageMaker project except those provisioned with the SQL & Gen AI blueprints. If you need to set up SageMaker Unified Studio, see Getting started with Amazon SageMaker Unified Studio.
  • A Space with Remote Access enabled. Either a JupyterLab or Code Editor Space works. The instance must have at least 8 GiB of memory (for example, ml.t3.large or larger). The default ml.t3.medium (4 GiB) can’t enable Remote Access. You must upgrade the instance type first, then toggle Remote Access to Enabled in the Configure Space dialog.
  • A VS Code-compatible editor. Kiro, VS Code, Cursor, or another VS Code-based IDE installed on your local machine. This walkthrough uses Kiro, but the Remote Access connection has been tested with VS Code and Cursor as well.
  • AWS Toolkit v4.1.0 or later. Kiro ships with the AWS Toolkit pre-installed. For VS Code and Cursor, install the AWS Toolkit extension and verify your version is 4.1.0 or later (Cmd+Shift+X and search for “AWS Toolkit”).
  • AWS credentials. You must be authenticated in the SageMaker Unified Studio panel of the AWS Toolkit with the same identity (AWS IAM Identity Center or AWS Identity and Access Management (IAM)) that you use to access SageMaker Unified Studio in the browser.
  • Network connectivity. Your Space must have internet access (PublicInternetOnly mode, or virtual private cloud (VPC) with a NAT gateway or HTTP proxy that allows VS Code and Open VSX endpoints).

The following screenshots show the SageMaker Unified Studio portal and the Configure Space dialog. Navigate to your project, select your Space, and verify the configuration. Remote Access is disabled when the instance has less than 8 GiB of memory. Select an instance with at least 8 GiB, such as ml.t3.large, then enable Remote Access. This is a one-time configuration per Space.

SageMaker Unified Studio portal showing the Spaces list for a project

Figure 4 — SMUS project Spaces overview in the portal

Configure Space dialog with the instance type selector open and ml.t3.large highlighted

Figure 5 — Configure Space dialog showing instance type selection

Configure Space dialog with the Remote Access toggle set to Enabled on an 8 GiB instance

Figure 6 — Enabling Remote Access on a Space with 8 GiB or more

Connecting your editor to your SageMaker Space

There are two ways to connect: directly from the SageMaker Unified Studio portal, or from your local IDE using the AWS Toolkit.

Method 1: Connect from the SageMaker Unified Studio portal

To launch your IDE directly from the portal, navigate to your project’s Code Spaces page, find your Space, and choose Open in to select your editor (Kiro, VS Code, or Cursor):

Code Spaces list with the Open in menu showing options for Kiro, VS Code, and Cursor

Figure 7 — Open in Local IDE from the Code Spaces list

You can also launch from within a Space’s details page:

Space details page with the Open in menu expanded

Figure 8 — Open in Local IDE from the Space details page

Or from within the JupyterLab or Code Editor browser environment:

JupyterLab toolbar with the Open in Local IDE option visible

Figure 9 — Open in Local IDE from JupyterLab

Your browser will prompt you to allow opening the IDE. Confirm, and the editor launches with an SSH connection to your Space already established via the AWS Toolkit. No additional configuration is typically required.

Method 2: Connect from your IDE via the AWS Toolkit

  1. Open your editor on your local machine. Then, in the AWS Toolkit panel, choose Sign in. Authenticate with your IAM Identity Center or IAM credentials, the same identity you use to access SageMaker Unified Studio in the browser. The following screenshots show Kiro, but the steps are the same in VS Code and Cursor.Figure 10 — AWS Toolkit button in Kiro
    Figure 10 — AWS Toolkit button in KiroAWS Toolkit panel expanded in Kiro showing the Sign in option

    Figure 11 — AWS Toolkit panel expanded

    AWS Toolkit Sign in dialog with profile selection

    Figure 12 — AWS Toolkit Sign in dialog

  2. Choose your AWS profile. You must have a profile configured in the AWS CLI with the correct account and AWS Region set.
  3. In the Toolkit panel, browse your SageMaker Unified Studio domains and projects. Select the project that you want to work in.

Kiro AWS Toolkit panel showing SageMaker Unified Studio domains and projects in a tree view

Figure 13 — Browsing SMUS domains and projects in Kiro

Important: The credentials that you use in the AWS Toolkit must match the identity that you use in the SageMaker Unified Studio portal. The Toolkit validates that your identity has access to the Space.

AI steering: How SageMaker Unified Studio pre-seeds AI context

The real value of the feature comes from what you don’t need to do. When connected to Kiro SageMaker Unified Studio automatically generates steering files that guide your AI assistant with project context, so you can focus on building analytics rather than configuring connections. When you open a SageMaker Unified Studio project, SageMaker Unified Studio presents a prompt to create steering files: an AGENTS.md file that references a newly created smus-context.md. These files provide context about your project environment, such as project configuration, environment details, and utilities for discovering your data catalog and project structure. Kiro detects and applies these files automatically; in other editors, you can reference them as context for your AI features.

SageMaker Unified Studio popup offering to create AGENTS.md and smus-context.md steering files

Figure 14 — SMUS popup offering to create steering files

Kiro file explorer showing the generated AGENTS.md and smus-context.md files at the project root

Figure 15 — Generated AGENTS.md and smus-context.md steering files

Without these steering files, your AI assistant would need several back-and-forth prompts to discover what data you have and how to access it. With them, the assistant understands your project from the first prompt: how to discover your databases, how your environment is configured, and what tools are available. The steering files also help properly configure MCP servers, which you set up in the next section.

Exploring your project

After you’re connected, the project structure expands into Data and Compute sections in the sidebar, as it would in the SageMaker Unified Studio portal.

Kiro sidebar showing the Data and Compute sections expanded under a SageMaker Unified Studio project

Figure 16 — Project Data and Compute sections in the Kiro sidebar

You can explore your data catalog and S3 buckets directly from the sidebar:

Kiro sidebar with the data catalog tree and S3 buckets expanded under the project

Figure 17 — Exploring the data catalog and S3 buckets from the sidebar

You can also remote into a compatible Space for direct development. Hover over a Space and select the remote icon on the right:

Kiro sidebar showing the remote connection icon next to a compatible Space

Figure 18 — Remote connection icon on a compatible Space

After a moment, the Space opens in a new Kiro window:

New Kiro window opened with a remote connection to the SageMaker Unified Studio Space

Figure 19 — Space opened in a new Kiro window

You must sign in again, and then trust the authors of the files in the Space:

Trust authors dialog asking to confirm trust for files in the remote Space

Figure 20 — Trust authors dialog for the Space files

You’re now connected to your Space. The Toolkit works on the Space the way it does locally, except the resources are scoped to the project’s permissions.

Kiro window connected to a SageMaker Unified Studio Space with the AWS Toolkit panel active

Figure 21 — Connected to the SMUS Space with the Toolkit active

Setting up MCP servers

Before you can use AI-assisted development effectively, you must give Kiro access to your data services through Model Context Protocol (MCP) servers. MCP servers extend the Kiro agent with tools: the ability to query catalogs, run SQL, manage credentials, and more.

Out of the box, Kiro has no MCP servers configured:

Kiro MCP servers panel with no servers configured

Figure 22 — Kiro MCP servers panel with no servers configured

Prompt Kiro to find and configure the MCP servers that ship pre-installed on your SageMaker Space. Using the steering file context, Kiro located the servers and generated the configuration. If a server fails to connect, select the failed entry and Kiro will suggest fixes. You might need additional prompts to get the smus_spark_upgrade server (a pre-installed MCP server for managing Spark session upgrades) working correctly.

Kiro chat panel showing the agent discovering and configuring SageMaker Unified Studio MCP servers

Figure 23 — Kiro discovering and configuring SMUS MCP servers

MCP servers panel after iterating on configuration fixes, showing servers connected

Figure 24 — MCP servers after iterating on configuration fixes

For more deterministic results, you can also configure the MCP servers manually. Here is a sample configuration:

{
    "mcpServers": {
        "smus_local": {
            "command": "python3",
            "args": ["-m", "sagemaker_studio.mcp_server"],
            "env": {}
        },
        "aws-dataprocessing": {
            "command": "uvx",
            "args": ["awslabs.aws-dataprocessing-mcp-server@latest"],
            "env": {
                "AWS_REGION": "us-east-1",
                "FASTMCP_LOG_LEVEL": "ERROR"
            },
            "disabled": ["emr_*"]
        }
    }
}

Note: Your MCP configuration might vary depending on your SageMaker Unified Studio environment. Use the preceding configuration as a starting point and let your editor adjust if a server fails to connect.

Next, add the AWS Data Processing MCP server to get catalog information and Athena query capabilities. This isn’t strictly required (Kiro can use Python or AWS CLI for the same tasks), but it gives the agent native tools for catalog and query operations.

AWS Data Processing MCP server tools listed in Kiro with the Amazon EMR tool group disabled

Figure 25 — AWS Data Processing MCP server tools with Amazon EMR tools disabled

You can list the tools that each MCP server provides. Because the AWS Data Processing MCP server includes tools for many services, we recommend disabling tools that you don’t need for a given project to save model context. For this walkthrough, disable the Amazon EMR tools to focus on AWS Glue and Amazon Athena.

Exploring data with notebooks

Kiro supports Jupyter notebooks in your SageMaker Space with the same language and connection selectors that you would find in SageMaker JupyterLab or Code Editor. Open the command palette (Cmd+Shift+P) and create a new Jupyter notebook:

Kiro command palette filtered to the Create New Jupyter Notebook command

Figure 26 — Command palette to create a new Jupyter notebook

New Jupyter notebook open in Kiro showing language and connection selectors at the bottom-right of a cell

Figure 27 — New Jupyter notebook opened in Kiro with language and connection selectors in a notebook cell

As in SageMaker JupyterLab, you get language and connection selectors in the bottom right of each cell. Choose the connection selector to see your available connections:

SageMaker connection selector dropdown showing the available connections for the project

Figure 28 — SageMaker connection selector

Select PySpark to fill in the magic commands for your cell. Write your code (in this case, enter spark and press Shift+Enter) to verify the session starts:

Notebook cell prefilled with the PySpark magic command and a spark verification statement

Figure 29 — PySpark magic command and spark verification code

PySpark cell running in the Kiro notebook

Figure 30 — Running the PySpark cell

If this is your first time using Jupyter with Kiro, you’re prompted to install the Jupyter extension. After it’s installed, select the kernel from Python Environments → Base:

Jupyter kernel selection prompt in Kiro after installing the Jupyter extension

Figure 31 — Jupyter kernel selection prompt

Kernel picker showing the Python kernel selected from the Base environment

Figure 32 — Selecting the Python kernel from the Base environment

Re-run your cell. After a few moments, AWS Glue provisions a PySpark session:

AWS Glue provisioning a PySpark session in a Jupyter notebook in Kiro

Figure 33 — AWS Glue provisioning a PySpark session in a Jupyter notebook in Kiro

You see results the way you would in JupyterLab in the SageMaker Unified Studio portal:

PySpark code running in a Jupyter notebook in Kiro with output cells populated

Figure 34 — PySpark code running in a Jupyter notebook in Kiro

The notebook generate button

You will notice a Generate button underneath notebook cells. Let’s test it with a simple prompt:

looking at the above cell for reference, show me the accounts where state = california
using pyspark prefixing the cell with `%%pyspark default.spark` and sorting by
account_length

Notebook cell showing the Generate button populated with a natural language prompt

Figure 35 — Using the Generate button with a natural language prompt

Generated PySpark code populating a notebook cell after using the Generate button

Figure 36 — Generated PySpark code from the prompt

This prompt builder, like other notebook generation features, doesn’t have good context on the surrounding cells. You must be explicit about what you want because it won’t read other code or cells as input.

While the Kiro notebook generate button works for straightforward edits, for serious code generation, we recommend that you use Kiro agent mode. This mode has full project and SageMaker context, as demonstrated in the “See it in action” walkthrough earlier in this post.

What’s happening under the hood

When you connect your editor to a SageMaker Unified Studio Space, the AWS Toolkit extension establishes a secure SSH tunnel between your local IDE and your cloud-based Space.

Key details:

  • SSH tunnel. The connection is managed entirely by the AWS Toolkit (v4.1.0+) or VS Code’s built-in SSH extension. No separate Remote SSH extension is needed; the capability is built in.
  • File system access. Your editor sees the Space’s persistent storage at /home/sagemaker-user/, including shared project files and notebooks or scripts you create.
  • SageMaker Unified Studio steering context. The integration generates AGENTS.md and smus-context.md files that provide your AI assistant with context about your project environment and utilities for understanding your data. This is what makes the assistant effective from the first prompt.
  • MCP server integration. MCP servers like smus_local (for project metadata and environment utilities) and aws-dataprocessing (for AWS Glue Data Catalog and Amazon Athena) extend your editor’s AI with direct access to your data services. Your own MCP servers will be equally valuable here.
  • Credential flow. The Toolkit uses your existing AWS identity (IAM Identity Center or IAM) to authenticate to the Space. No separate SSH keys to manage. The aws_context_provider tool from the smus_local MCP server handles credential discovery for agent operations.

Best practices

To work effectively with your IDE and SageMaker Unified Studio:

  • Explore your data before building. Start every session by asking your AI assistant to discover your catalog, sample your data, and understand the schema. This single step helps reduce the most common source of errors in AI-assisted data work: the LLM making assumptions about data it has not seen. See the “See it in action” walkthrough earlier in this post for a concrete example of the difference this makes.
  • Use the SageMaker Unified Studio steering files. When prompted to create AGENTS.md and smus-context.md, accept. These files are the foundation that makes everything else work: environment context, MCP server configuration, and project understanding. Without them, your AI assistant starts from zero on every prompt. Kiro detects these automatically; in other editors, add them as context.
  • Disable unused MCP tools. The AWS Data Processing MCP server includes tools for AWS Glue, Amazon EMR, Amazon Athena, and more. Disable the services that you’re not using for a given project to save model context and reduce noise.
  • Be specific in your prompts. The more detail you give your AI (column names, query patterns you prefer, output formats), the closer the first pass will be. “Run data quality evaluation using Athena SQL” gets you better code than “check my data.”
  • Always test interactively first. Whether in notebooks or the terminal, validate code before deploying it. AI agents can iterate quickly, but catching issues in an interactive session is faster than debugging a failed AWS Glue job. Athena PySpark and the SageMaker sqlutils and sparkutils packages are great for this.
  • Stop your Space when idle. Your Space runs on compute (the same instance types as Code Editor and JupyterLab). If idle, the Space will terminate after 60 minutes and close your remote connection. Close the remote window and reconnect to continue.

Things to know

  • Notebook agent mode. For notebook-heavy analytics workflows where you want agentic AI to generate and run cells directly, SageMaker Notebooks with Data Agent in SageMaker Unified Studio is the recommended option today. Current notebook support in local editors covers editing, running, and generating code in individual cells.
  • MCP setup takes iteration. Configuring MCP servers may require iteration, especially for servers with complex authentication. Many AI-enabled editors can self-correct when a server fails. For more deterministic results, use the preceding MCP configuration JSON as a starting point rather than relying solely on auto-discovery.
  • CLI preference. AI agents often prefer the AWS CLI and bash even when MCP tools are available. This doesn’t affect outcomes, but you can steer your assistant toward MCP tools using a steering document if you prefer consistency.

Security and governance boundaries

A core benefit of this integration is that your existing security and governance controls remain enforced. Your editor connects to your SageMaker Space through a secure SSH tunnel managed by the AWS Toolkit. It does not bypass your organization’s access controls. Data access is governed by the same AWS Lake Formation permissions and IAM Identity Center authentication that apply when you work in the SageMaker Unified Studio portal directly. Your project-level permissions, database grants, and column-level security policies apply consistently whether a query originates from an AI agent, a notebook cell, or the SageMaker console. Data access is governed by the boundaries you define in your SageMaker Unified Studio domain and project configuration.

Clean up

To avoid ongoing charges from billable resources (SageMaker Space compute charges per hour, AWS Glue sessions charge per DPU-hour, Amazon Athena queries charge per TB scanned):

  1. Stop your Space – In the SageMaker Unified Studio portal, navigate to your project’s Spaces and stop the Space you used for this walkthrough.
  2. Disconnect: Close the remote connection in your editor (File → Close Remote Connection).
  3. Verify AWS Glue sessions are terminated – If you ran PySpark queries during this walkthrough, verify that the sessions are stopped. In the SageMaker Unified Studio portal, navigate to Data processing and confirm no active AWS Glue sessions remain. Sessions auto-terminate when the Space stops, but verify to avoid unexpected charges.
  4. Delete demo resources (optional) – File deletion is permanent and cannot be undone. Back up any work that you want to retain before proceeding. If you created scripts or files during this walkthrough that you no longer need, delete them from /home/sagemaker-user/. For example, delete any test notebooks, Python scripts, or generated data files. The sample sagemaker_sample_db.churn dataset is read-only and doesn’t need cleanup.

Conclusion

This post showed what happens when agentic AI meets governed data, and walked through how to set it up yourself.

Three key insights emerged from this hands-on experience:

  1. SageMaker Unified Studio steering files transform the developer experience. Your AI assistant is project-aware from the first prompt, understanding your environment and available data without manual setup.
  2. MCP servers bridge “AI that writes code” with “AI that queries your data”. The smus_local and aws-dataprocessing servers are essential for effective agentic data work.
  3. The “explore first” pattern pays immediate dividends. When your AI assistant understands your data before writing code, it makes smarter engine choices and produces correct analytics on the first pass.

This integration brings together two capabilities that are stronger together: your IDE handles the AI-assisted coding and iteration, while SageMaker Unified Studio handles data governance, access control, and compute management. You get the productivity of an agentic AI coding assistant without compromising on the controls your organization requires.

To get started, download Kiro, install VS Code or Cursor, and add the AWS Toolkit for Visual Studio Code (v4.1.0 or later). Then visit the Amazon SageMaker Unified Studio documentation and the AWS Data Processing MCP Server to set up your first Space. For related reading, see Speed up delivery of ML workloads using Code Editor in Amazon SageMaker Unified Studio.


About the authors

Zach Mitchell

Zach Mitchell

Zach is a Senior Big Data Architect in AWS Worldwide Specialist Organization for Analytics. He works with customers to design and build data applications on AWS, with a focus on SageMaker Unified Studio, AWS Glue, and AWS Lake Formation. Outside of work, he enjoys building things with code and occasionally writing about it.

Anchit Gupta

Anchit Gupta

Anchit is a Senior Product Manager on the Amazon SageMaker Unified Studio team at AWS.

Leah Wagner

Leah Wagner

Leah is a Senior Solutions Architect in AWS Worldwide Specialist Organization for Analytics.

Bhargava Varadharajan

Bhargava Varadharajan

Bhargava is a Senior Software Engineer on the Amazon SageMaker Unified Studio team at AWS.

Majisha Namath Parambath

Majisha Namath Parambath

Majisha is a Software Development Engineer on the Amazon SageMaker Unified Studio team at AWS.

Build governance dashboards for Amazon SageMaker Catalog with Amazon Quick

Post Syndicated from Steve Phillips original https://aws.amazon.com/blogs/big-data/build-governance-dashboards-for-amazon-sagemaker-catalog-with-amazon-quick/

Maintaining visibility into your data catalog’s health requires more than ad-hoc queries. Data stewards and compliance teams need automated dashboards that surface governance metrics and alert them when issues arise. These issues include undocumented assets, missing ownership, and stale metadata.

In a previous post, we showed you how to query Amazon SageMaker Catalog metadata using SQL by using the metadata export feature. This post builds on that foundation by demonstrating how to create governance dashboards with Amazon Quick.

Amazon Quick is an agentic AI-powered digital workspace that provides integrated analytics, automation, and research capabilities. With Amazon Quick Sight, a component of Amazon Quick, you can create interactive dashboards and visualizations with automatic chart suggestions and machine learning (ML) insights.

We walk through how to connect Amazon Quick Sight to your Amazon SageMaker Catalog metadata and build governance dashboards using natural language prompts.

Solution overview

This solution extends the metadata export architecture by adding a visualization layer:

  1. Amazon SageMaker Catalog exports asset metadata daily to Amazon Simple Storage Service (Amazon S3) Tables
  2. Amazon Athena queries the metadata using standard SQL
  3. Amazon Quick Sight connects to Athena for interactive dashboards
  4. Amazon Quick uses natural language to build visualizations

AWS Cloud architecture diagram showing the data flow for SageMaker Catalog metadata visualization. Amazon SageMaker Catalog exports asset metadata and daily exports to Amazon S3 Tables in a bucket named aws-sagemaker-catalog. Amazon Athena queries the S3 Tables data using SQL queries. Amazon Quick connects to Athena to provide interactive dashboards, natural language queries, and executive summaries. Arrows indicate the left-to-right data flow from SageMaker Catalog to S3 Tables to Athena, with Athena connecting down to Amazon Quick.

Figure 1 – Amazon SageMaker Catalog governance dashboard architecture

Prerequisites

Before you begin, complete the following steps from Analyzing your data catalog: Query SageMaker Catalog metadata with SQL. You must also have the following:

  • Amazon SageMaker Catalog metadata export enabled
  • Amazon Athena configured with query results S3 bucket
  • AWS Lake Formation permissions configured for AWS Identity and Access Management (IAM)-based access
  • Verified that the asset_metadata.asset table contains data

Additionally, you need:

Building a governance dashboard with Amazon Quick Sight

To visualize catalog health metrics, connect Amazon Quick Sight to your Athena metadata tables.

Configure Amazon Quick Sight permissions

  1. Grant permissions to the Amazon Quick Sight service role.

The Amazon Quick Sight service role (default name: aws-quicksight-service-role-v0) needs permissions to access Amazon S3 Tables and AWS Glue catalog:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTableBucket",
        "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation"
      ],
      "Resource": "arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/aws-sagemaker-catalog/*"
    },
    {
      "Effect": "Allow",
      "Action": "glue:GetCatalog",
      "Resource": "arn:aws:glue:REGION:ACCOUNT_ID:catalog"
    }
  ]
}

Add this as an inline policy to the Amazon Quick Sight service role in the IAM console.

  1. Grant AWS Lake Formation permissions:

Both the Amazon Quick Sight service role and your Amazon Quick Sight admin user need AWS Lake Formation permissions on the S3 Tables catalog. First, find your Amazon Quick Sight admin user ARN by running this AWS Command Line Interface (AWS CLI) command:

aws quicksight list-users \
  --aws-account-id ACCOUNT_ID \
  --namespace default \
  --region us-east-1

Amazon Quick Sight users are managed in the Amazon Quick Sight home AWS Region (us-east-1).To grant permissions, use the Lake Formation console.

  1. Navigate to AWS Lake Formation in the AWS Management Console.
  2. Select Data permissions and Grant.
  3. For Principals, choose SAML users and groups.
  4. Enter your Amazon Quick Sight admin user ARN (from the preceding command).
  5. Under LF-Tags or catalog resources, choose Named Data Catalog resources.
  6. For Catalogs, choose the S3 Tables catalog: ACCOUNT_ID:s3tablescatalog/aws-sagemaker-catalog.
  7. For Databases, choose asset_metadata.
  8. Under Tables, choose asset.
  9. For Table permissions, choose Select and Describe.
  10. Select Grant.

Screenshot of AWS Lake Formation Grant permissions page showing the complete permission configuration workflow. At the top, the resource selection shows the 'asset_metadata' database and 'asset' table from the s3tablescatalog/aws-sagemaker-catalog catalog. Below that are optional sections for Views and Data filters, both unselected. The main content area displays three permission configuration sections. First, the 'Table permissions' section shows two subsections: 'Table permissions' with checkboxes for Select (checked, highlighted with orange box), Describe (checked, highlighted with orange box), Insert, Alter, Delete, Drop, and Super; and 'Grantable permissions' with the same permission options all unchecked. The Super permission includes explanatory text stating it is the union of all individual permissions and supersedes them. The Grantable permissions section explains that this allows the principal to grant any of the permissions to others and supersedes grantable permissions. At the bottom, the 'Data permissions' section displays two radio button options: 'All data access' (selected) which grants access to all data without restrictions, and 'Column-based access' which grants data access to specific columns only. An orange arrow points from the right side down to the bottom right corner where 'Cancel' and 'Grant' buttons are located, with the Grant button highlighted in orange.

Figure 2 – Grant access to Amazon SageMaker Catalog resources

  1. Repeat steps 1–9 for the Amazon Quick Sight service role, but in step 2 choose IAM users and roles instead.

When choosing the catalog in the Lake Formation console, you must choose the full S3 Tables catalog identifier (ACCOUNT_ID:s3tablescatalog/aws-sagemaker-catalog) to see the asset_metadata database.

Create an Amazon Quick Sight dataset.

Access S3 Tables data by creating a Quick Sight dataset using an Amazon Athena data source and the custom SQL option. An S3 Tables data source is also available but requires additional permissions. See Introducing new data source with S3 Tables in Amazon Quick for using S3 Tables as an Amazon Quick data source.

  1. Open Amazon Quick Sight in the AWS Management Console.
  2. Select Analyses and Create analysis.

Amazon QuickSight Analyses page showing the left navigation menu with Analyses selected under the Quick Sight section. The main content area displays a promotional banner for creating insightful and interactive visualizations with sample chart previews. Below the banner, an orange arrow points to the Create analysis button in the upper right. A table lists an existing analysis named New custom SQL analysis owned by Me and last updated a month ago.

Figure 3 – Create Amazon Quick Sight analysis

  1. Choose Create dataset and Create data source.

Amazon QuickSight Create Analysis dialog prompting the user to choose a dataset. A search field for datasets is shown at the top left. An orange arrow points to the Create dataset button in the upper right. A table below lists one available dataset named New custom SQL with a data source of New custom SQL, owned by Me, and last modified on March 5, 2026.

Figure 4 – Create dataset

  1. Select Amazon Athena as the data source and select Next.
  2. Enter a Data source name (for example, “SageMaker Catalog Metadata”) and choose Create data source.

Amazon QuickSight New Amazon Athena data source configuration dialog. The Data source name field is highlighted with an orange box and contains the value SageMaker Catalog Metadata. The Athena workgroup dropdown is set to primary. A Validate connection button and SSL is enabled label appear at the bottom left. An orange box highlights the Create data source button at the bottom right.

Figure 5 – Create data source

  1. Select Use custom SQL and enter a custom SQL query that references the S3 Tables catalog using the full three-part name.

Amazon QuickSight Choose your table dialog for the SageMaker Catalog Metadata data source. The Catalog dropdown is set to AwsDataCatalog and the Database dropdown shows a Select prompt. An instructional message explains to choose Prepare data to create a SQL query or choose Select table. An orange arrow points down to the Use custom SQL button highlighted with a blue box at the bottom center. The Select button is highlighted with an orange box at the bottom right.

Figure 6 – Use custom SQL

Amazon QuickSight Enter custom SQL query dialog. The query name field shows New custom SQL. The SQL editor contains a query reading SELECT FROM s3tablescatalog/aws-sagemaker-catalog with the query text underlined in orange. An orange box highlights the Confirm query button at the bottom right. An Edit/Preview data button appears at the bottom left.

Figure 7 – Enter custom SQL

SELECT * FROM "s3tablescatalog/aws-sagemaker-catalog".asset_metadata.asset

  1. Select Confirm query.
  2. Choose Directly query your data (SPICE import may fail with S3 Tables catalogs)
    Amazon QuickSight Finish dataset creation dialog showing the custom SQL dataset named New custom SQL with the SageMaker Catalog Metadata data source. Two radio button options are displayed: Import to SPICE for quicker analytics with 100 GB available shown in green, and Directly query your data which is selected and highlighted with an orange box. An orange box highlights the Visualize button at the bottom right. Edit/Preview data and Augment with SageMaker buttons appear at the bottom left and center.

Figure 8 – Directly query your data

  1. Choose Visualize and Create to start building your dashboard.

Create visualizations with Amazon Quick.

With Amazon Quick, you can build governance dashboards using natural language prompts. This removes the need for manual field configuration. This approach is faster and more intuitive than traditional dashboard building.The Amazon Quick Sight user must have AdminPro or AuthorPro subscription (the Build feature isn’t available for Reader users).Start building your dashboard with the following steps:

  1. Select Build in the top toolbar to open the natural language builder.

Amazon QuickSight analysis editor for New custom SQL analysis. The left Data panel shows the dataset fields including accountid, asset_created_time, asset_id, asset_name, asset_updated_time, business_description, catalog, extended_metadata, namespace, region, resource_description, resource_id, resource_name, resource_type_enum, and snapshot_time. The center Visuals panel shows the Build button highlighted with an orange box and a grid of available chart types. The right canvas area displays an empty AutoGraph placeholder with the message Add 1 or more fields to build a visual. An Add Data section with a dashed border prompts to add a dimension or measure.

Figure 9 – Amazon Quick build dashboard

  1. You will see a text box where you can describe the visualization that you want to create.

Amazon QuickSight analysis editor with the Build a visual panel open on the right side. An orange arrow points to the natural language input field where the user has typed a prompt requesting asset distribution by resource type as a pie chart, with a Build button next to it. Below the input field, a tooltip explains to describe the visual you would like to build with examples including map showing the top 5 cities by sales, MoM profit in 2026, and average revenue by quarter. The left Data panel shows dataset fields and the center Visuals panel displays available chart types.

Create each visualization using natural language. For each of the six recommended visualizations, enter the corresponding natural language prompt, select Build, then choose ADD TO ANALYSIS.

Amazon QuickSight analysis editor with the Build a visual panel open on the right side. The natural language prompt reads Show asset distribution by resource type as a pie chart with a Build button. Below, the system shows the interpretation as Unique number of Asset Id by Resource Type Enum using the New custom SQL dataset. A pie chart preview is displayed showing the distribution with a large segment labeled GlueTable. An orange arrow points to the Add to Analysis button at the bottom of the panel.

Figure 11 – Add to analysis

Visualization 1: Asset inventory by type

Show count of asset_id by resource_type_enum as a pie chart

After the pie chart is created, choose ADD TO ANALYSIS.

Visualization 2: Documentation completeness

Show count of asset_id where business_description is not null asa KPI

After the KPI is created, choose ADD TO ANALYSIS.

Visualization 3: Monthly registration trends

Show count of asset_id by asset_created_time month as a line chart

After the line chart is created, choose ADD TO ANALYSIS.

Visualization 4: Asset count by account

Show count of asset_id by account_id as a bar chart

After the bar chart is created, choose ADD TO ANALYSIS.

Visualization 5: Namespace distribution

Show count of asset_id by namespace as a treemap

After the treemap is created, choose ADD TO ANALYSIS.

Visualization 6: Resource type by namespace

Show count of asset_id by resource_type_enum and namespace as a heat map

Choose ADD TO ANALYSIS

  1. Arrange and publish your governance dashboard with the following steps:
  2. Delete any empty or unwanted visualizations by choosing the three dots menu and choosing Delete.
  3. Arrange visualizations by dragging them into your preferred layout.
  4. Resize visualizations to emphasize key metrics.
  5. Add titles to each visualization for clarity.
  6. Choose PUBLISH in the top right corner.
  7. Enter a dashboard name: “SageMaker Catalog Governance Dashboard”.
  8. Verify these options are selected:
    1. Allow executive summary.
    2. Allow sharing stories.
    3. Allow sharing scenarios.
  9. Choose Publish dashboard.

Amazon QuickSight SageMaker Catalog Governance Dashboard showing five visualizations. Top left is a pie chart titled Unique number of Asset Id by Resource Type showing all assets as GlueTable type. Top center is a key performance indicator displaying a total of 500 unique assets. Top right is a horizontal bar chart titled Unique number of Asset Id by Account Id showing five AWS account IDs with values of 109, 105, 104, 103, and 79 assets respectively. Middle left is a stacked bar chart titled Unique number of Asset Id by Resource Type Enum and Namespace showing GlueTable assets distributed across namespaces with values ranging from 33 to 52. Middle right is a treemap titled Unique number of Asset Id by Namespace with trading_analytics at 52, compliance_reporting at 51, treasury_ops at 50, market_data at 44, fraud_detection at 42, customer_analytics at 40, credit_scoring at 40, risk_management at 39, portfolio_mgmt at 37, regulatory at 37, loan_origination at 35, and payments at 33. Bottom is a line chart titled Unique number of Asset Id by Asset Created Time month showing asset creation trends from April 2025 through March 2026 with values fluctuating between approximately 30 and 50 assets per month.

Figure 12 – Amazon SageMaker Catalog governance dashboard

    1. Analyze your dashboard with natural language.

After you publish, you can ask questions about your governance data:

    1. On the dashboard, choose Analyze this dashboard in a Scenario in the top center.
    2. In the Data to Insights panel, enter natural language questions such as:
      1. “Which resource types have the lowest documentation rates?”
      2. “How many assets were registered last month compared to this month?”
      3. “What percentage of assets lack ownership information?”
    3. Choose Submit to generate AI-powered insights.

Amazon Quick analyzes your data and provides insights with supporting visualizations.

    1. Generate executive summaries

Create automated governance reports for data stewards and compliance teams:

    1. Choose the Amazon Quick logo in the top left to return to the home page
    2. Select Dashboards from the left panel
    3. Choose your “SageMaker Catalog Governance Dashboard”
    4. Choose the Create dropdown menu in the top right
    5. Select Executive Summary

Amazon Quick will automatically generate a summary with key governance insights, including Total asset counts and growth trends, Documentation completeness metrics, Ownership coverage statistics, and Classification distribution analysis.

    1. Create governance stories.

Build governance reports that combine multiple dashboards:

    1. From the Create dropdown, select Story.
    2. Enter a prompt: “Write a summary of catalog governance metrics and data quality trends”.
    3. Choose Add to select dashboards to include in the report.
    4. Choose Build (this might take a few minutes to complete).

Amazon Quick will generate a narrative report combining your visualizations with AI-generated insights. Share the reports with leadership or compliance teams.

Governance dashboards contain metadata such as ownership and classification details. Restrict access to users who need it. In the Amazon Quick Sight console, open the dashboard, choose Share, and grant access to named users or a dedicated Quick Sight group (for example, data-stewards) instead of selecting Everyone in this account. Review the dashboard’s permissions periodically and remove entries that are no longer needed.

Cleaning up

To avoid ongoing charges, clean up the resources created in this walkthrough. Delete Amazon Quick Sight resources including the dashboard, analyses, and dataset.

Conclusion

In this post, you connected Amazon Quick Sight to your Amazon SageMaker Catalog metadata export, built governance dashboards using the Amazon Quick natural language prompts. This approach gives data stewards and compliance teams visibility into catalog health through six key visualizations covering asset inventory, documentation completeness, registration trends, account distribution, classification coverage, and stale asset detection.

Together with the metadata export and SQL query capabilities covered in the Analyzing your data catalog: Query SageMaker Catalog metadata with SQL post, this solution provides a complete, low-overhead governance monitoring pipeline from raw catalog metadata to executive-ready.

To learn more about Amazon SageMaker Catalogs, see Amazon SageMaker Catalog documentation. To expand the work done with Amazon Quick, review Amazon Quick Sight documentation.


About the authors

Steve Phillips

Steve is a Principal Technical Account Manager and Analytics specialist at AWS in the North America region. Steve currently focuses on data warehouse architectural design, data lakes, data ingestion pipelines, and cloud distributed architectures.

Ramesh Singh

Ramesh is a Senior Product Manager Technical (External Services) at AWS in Seattle, Washington, currently with the Amazon SageMaker team. He is passionate about building high-performance ML/AI and analytics products that help enterprise customers achieve their critical goals using cutting-edge technology.

Pradeep Misra

Pradeep is a Principal Analytics and Applied AI Solutions Architect at AWS. He is passionate about solving customer challenges using data, analytics, and Applied AI. Outside of work, he likes exploring new places and playing badminton with his family. He also likes doing science experiments, building LEGOs, and watching anime with his daughters.

Rohith Kayathi

Rohith is a Senior Software Engineer at Amazon Web Services (AWS) working with Amazon SageMaker team. He leads business data catalog, generative AI–powered metadata curation, and lineage solutions. He is passionate about building large-scale distributed systems, solving complex problems, and setting the bar for engineering excellence for his team.