When you develop SQL against Amazon Redshift and Amazon Athena, you spend time finding the right tables across hundreds of databases, writing complex joins and aggregations, debugging failed queries without context from previous attempts, and re-specifying filters for every new question. Amazon SageMaker Data Agent in Query Editor takes a different approach. You describe what you need in natural language, and the Data Agent generates the SQL. It references your actual tables through AWS Glue Data Catalog, proposes step-by-step plans for complex questions, retains context across your session, and offers one-click error recovery with Fix with AI. In this post, you learn how to use Data Agent in Query Editor to explore data, build multi-step analyses, recover from errors, and summarize results using a public education dataset.
Solution overview
You can go from a natural language question to executable SQL in seconds. Data Agent in Query Editor provides a conversational interface with direct access to your AWS data environment, so you spend less time on query mechanics and more time on analysis. Data Agent in Query Editor focuses specifically on SQL development against Amazon Redshift and Amazon Athena. (For Python, SQL, and PySpark across broader analytical and machine learning (ML) workloads, use Data Agent in notebooks.)
Data Agent provides four key capabilities:
Catalog-aware SQL generation. You don’t need to browse catalog structures or memorize schema details. Data Agent reads your table metadata directly.
Querybook and session context. You build on previous work. Data Agent uses context from your earlier queries and results.
Step-by-step planning. You review and approve a structured plan before Data Agent generates SQL.
Fix with AI. You recover from failed queries with one click.
Data Agent integrates with AWS Glue Data Catalog and reads your actual table names, column types, descriptions, and relationships, so generated SQL references your real tables. Each follow-up question builds on your current Query Editor session—the SQL cells in your querybook, the active connection, your selected cell, and execution results from previously run cells. For complex requests, Data Agent produces a structured plan that specifies which data to retrieve, how to aggregate it, and what filters to apply. You review and approve each step before Data Agent proceeds. When a query fails, choose Fix with AI to get a corrected query based on the error and the failed cell’s context.
[Figure 1: The Query Editor Fix with AI panel, showing a corrected SQL query ready for your review.]
Walkthrough: Education data analysis
In this section, you use Data Agent in Query Editor to analyze California schools data and identify where SAT improvement investment has the most impact. The walkthrough covers four tasks:
Explore available data.
Build a multi-step analysis plan.
Summarize insights from your queries.
Recover from a failed query.
The same workflow applies to your own data, whether you are analyzing sales figures, operational metrics, or financial records.
The California schools dataset contains SAT score results, school demographic information, and county-level data for public schools across California. The dataset includes tables that organize SAT scores by subject (reading, writing, math), school details (name, address, county, district), and enrollment figures. After you upload the data into your project database, you directly access the tables from Query Editor through your Amazon Athena or Amazon Redshift Lakehouse connection.
Prerequisites
To complete this walkthrough, you need intermediate SQL knowledge and basic familiarity with the AWS Management Console. You don’t need prior AWS Glue experience, but familiarity with data catalogs (centralized metadata repositories) helps.
You can choose one of two setup paths:
Quick start (5 minutes). SageMaker Unified Studio provides a sample database (sagemaker_sample_db) with pre-loaded data. To explore it, choose Data in the navigation pane, expand AwsDataCatalog, and select sagemaker_sample_db.
Full setup (30–45 minutes). Upload the California schools dataset into your project’s Lakehouse database. This dataset is publicly available from the California Department of Education. Download the SAT scores, school information, and county-level data files, then upload them through the SageMaker Unified Studio UI. In your project, go to Build, choose Query editor, right-click your project database in the Data explorer, and choose Create table. Drag and drop each CSV file to create the tables. SageMaker Unified Studio stores the data in the project-managed Amazon Simple Storage Service (Amazon S3) location, registers it in AWS Glue Data Catalog, and applies AWS Lake Formation governance automatically.
Running queries against Amazon Athena or Amazon Redshift might incur costs. For pricing details, refer to Amazon Athena pricing and Amazon Redshift pricing. For detailed setup instructions, refer to AWS Identity and Access Management (IAM)-based domains and projects. Before starting the walkthrough, you must have a SageMaker Unified Studio IAM-based domain with a project using the SQL analytics or All Capabilities project profile. The project automatically provisions an AWS Glue database, the required IAM role, and Athena or Redshift Lakehouse connections.
[Figure 2: The Data Explorer panel in Query Editor, showing the california_schools_db and sagemaker_sample_db tables.]
Explore available data. To start, enter the following prompt in the Data Agent panel:
Query my SAT scores from my california_schools_db
Data Agent searches AWS Glue Data Catalog, locates the relevant tables, and generates an initial exploratory query that retrieves SAT score records. It adds a SQL cell directly to your querybook.
Review the generated SQL in the comparison view, which highlights the proposed code.
Choose Accept, Reject, or Accept and run.
After you run the cell, the results appear inline, giving you a view of the data (column names, score ranges, and the number of records) before you write SQL.
[Figure 3: Data Agent returns an exploratory query for the california_schools_db tables, ready for your review.]
[Figure 4: The SQL query results appear beneath the cell after you choose Accept and run.]
Build a multi-step analysis plan. With the data explored, enter a more complex analytical question:
Identify which subjects need investment to improve SAT scores in the lowest-performing counties. Include school-level details with addresses.
Data Agent proposes a step-by-step plan before generating SQL. For this request, Data Agent breaks the question into three steps:
Aggregate SAT scores by county and subject to find performance patterns.
Filter to counties with a sufficient number of schools and rank the lowest performers.
Join school address data to produce a final detailed list.
Review the plan in the Data Agent panel and choose Run step-by-step to proceed.
[Figure 5: Data Agent proposes a multi-step plan with options to Cancel plan or Run step-by-step.]
Data Agent generates SQL for each step and adds it as a separate querybook cell. Review each cell’s SQL in the comparison view, then choose Accept and run to execute it. The results from each step are visible inline, so you can verify the intermediate output (county-level aggregations, the filtered ranking, and the final school list) before moving to the next step. When the steps are complete, your querybook contains the full analytical progression from raw scores to a detailed investment list.
[Figure 6: Each plan step produces a querybook cell that you can review and run independently.]
Summarize insights from your queries. After running the analysis, enter the following prompt:
Summarize the insights from my queries
Data Agent has context on your querybook, including the SQL and the query results from each cell. It generates a natural language summary: which counties are underperforming, which subjects (reading, writing, or math) need the most attention in each county, and how many schools appear on the investment list. This summary provides a starting point for a report or presentation.
[Figure 7: Data Agent summarizes insights from the accumulated query results in the querybook.]
Recover from a failed query. During the analysis, a generated query might produce an error, for example, referencing a column name that doesn’t match the schema or a join condition that returns unexpected results. When a cell fails, Query Editor displays the error message and a Fix with AI option.
Choose Fix with AI, and Data Agent reads the error in the context of the failed cell, then generates corrected SQL and updates the querybook cell. Run the corrected cell to verify the fix.
[Figure 8: After you choose Fix with AI, Data Agent is prompted to generate a corrected query for the failed cell.]
[Figure 9: Data Agent returns corrected SQL for you to review.]
Security and governance
Data Agent operates within your AWS environment and only accesses data that your IAM policies explicitly permit. Your existing IAM access controls and AWS Lake Formation permissions determine what data Data Agent can reach. To use Data Agent, your project role must have permissions to invoke specific Amazon DataZone APIs. For more information, refer to Actions, resources, and condition keys for Amazon DataZone.
Data Agent includes content filtering that prevents it from responding to off-topic requests, requests to reveal its system prompt, and requests for internal technical implementation details. Data Agent is restricted to AWS-related topics and English-language output.
Amazon SageMaker stores your natural language prompts and generated SQL in the AWS Region where you created your SageMaker Unified Studio domain. Data Agent doesn’t store your data, querybook context, or catalog metadata.
The walkthrough creates querybook cells in your Query Editor session but doesn’t provision standalone infrastructure. To remove the generated SQL cells, delete them from your querybook or delete the querybook itself.
If you uploaded the California schools dataset specifically for this walkthrough, remove the following resources to avoid ongoing charges:
SageMaker Unified Studio domain. If you created a domain solely for this walkthrough, delete it to stop incurring charges. Refer to the SageMaker Unified Studio administration guide for deletion steps.
Uploaded tables. In the Data explorer, right-click each table you created and choose Delete table to remove the data from your project database and the underlying S3 storage.
Amazon Athena query results. Amazon Athena stores query results in an S3 output location. Delete the query result files from that bucket, or delete the bucket if you created it solely for this walkthrough.
Amazon CloudWatch logs. If Amazon Athena queries generated CloudWatch log groups, delete those log groups to avoid storage charges.
Conclusion
Data Agent in Query Editor brings conversational, catalog-aware SQL development to your Amazon Redshift and Amazon Athena workloads. In this post, you explored unfamiliar data, built a multi-step investment analysis, recovered from query errors, and summarized findings through natural language prompts.
Data Agent works within your existing IAM and AWS Lake Formation security controls, keeps your data within your AWS environment, and retains context across your analytical workflow so each question builds on the last.
Get started with these next steps:
Run your first prompt. Open Query Editor in your SageMaker Unified Studio domain and enter Show me the top 10 tables in my catalog with the most columns. For setup, refer to the SageMaker Unified Studio getting started guide.
Add descriptions to your AWS Glue Data Catalog. Table descriptions and column-level business metadata improve the quality of generated SQL. For best practices, refer to Populating the AWS Glue Data Catalog.
Try a multi-step analysis. Enter Which product categories had declining revenue quarter-over-quarter, and which regions drove the decline? and review Data Agent’s plan step by step.
If you build notebooks for recurring tasks such as daily customer analysis, weekly report generation, or data quality checks in Amazon SageMaker Unified Studio, you’ve likely wanted to run them automatically on a schedule. Until now, there wasn’t a native way to do this. Teams had to manage orchestration separately, even though the interactive notebook experience was already in place. Now, notebook scheduling is available, so you can configure your production workloads to run automatically with minimal manual intervention.
In this post, we walk you through the new scheduling and orchestrating capabilities for notebooks in Amazon SageMaker Unified Studio. You will learn how to:
Trigger on-demand background runs, such as a model re-training job, without waiting at your desk.
Create recurring schedules for tasks such as nightly data freshness checks or weekly business reviews.
Parameterize notebooks so a single template can generate reports across different AWS Regions or customer segments.
Orchestrate multi-notebook workflows where one notebook’s output feeds into the next. For example, an extract, transform, and load (ETL) pipeline followed by a summary dashboard refresh.
Debug failed runs with AI-assisted troubleshooting.
Sample use case overview
In this walkthrough, you will take on the role of a logistics analyst who monitors shipping performance across carriers. The notebook loads shipping data from the ShippingLogs.csv dataset, identifies late deliveries, and generates a performance summary. You want to run this notebook every morning without manual intervention, reuse it across different carriers, and know when something goes wrong.
You will start by running a notebook in the background and viewing the results. Next, you will create a recurring schedule for daily runs, then parameterize the notebook to generate reports for different carriers. You will also orchestrate the notebook in a multi-step workflow and debug a failed run using AI-assisted troubleshooting.
Prerequisites
Before you begin, you need:
An Amazon SageMaker Unified Studio project with Notebooks enabled. See Set up IAM-based domains for permission requirements.
A sample dataset. We use the ShippingLogs.csv dataset, which contains shipping data including estimated and actual delivery times, carriers, and origins. You can download it from the Workshop Studio (the file is named ShippingLogs.csv on the linked page).
Setting up the notebook
Start by creating a new notebook in your SageMaker Unified Studio project. If you haven’t already, upload the ShippingLogs.csv file under the Shared tab in the Files panel.
In the first cell, we load and explore the dataset. To reference the file in code, select the file in the Shared tab and copy the Amazon Simple Storage Service (Amazon S3) URI shown in the file details. Alternatively, you can reference it with this code:
import pandas as pd
from sagemaker_studio import Project
# Initialize the project
proj = Project()
# Get the S3 root path
s3_root = proj.s3.root
df = pd.read_csv(s3_root + '/ShippingLogs.csv')
df.head()
The dataset contains columns including Carrier, ActualShippingDays, ExpectedShippingDays, ShippingOrigin, ShippingPriority, and OnTimeDelivery. Add a second cell to analyze shipping performance for a single carrier:
import matplotlib.pyplot as plt
carrier_data = df[df['Carrier'] == 'GlobalFreight']
# Flag late deliveries
carrier_data['is_late'] = carrier_data['ActualShippingDays'] > carrier_data['ExpectedShippingDays']
late_pct = carrier_data['is_late'].mean() * 100
# Visualize actual vs expected shipping days
plt.figure(figsize=(12, 4))
plt.hist(carrier_data['ActualShippingDays'] - carrier_data['ExpectedShippingDays'], bins=20, edgecolor='black')
plt.axvline(x=0, color='red', linestyle='--', label='On time')
plt.title(f'Shipping Delay Distribution - GlobalFreight ({late_pct:.1f}% late)')
plt.xlabel('Days Over Expected')
plt.ylabel('Number of Shipments')
plt.legend()
plt.show()
With the notebook working interactively, you’re ready to automate it.
Running a notebook asynchronously
To trigger an asynchronous run, open your notebook. In the notebook header, choose the menu on the Run all button, and then choose Run in background.
This captures a snapshot of the notebook in its current state and starts a run on a separate dedicated compute. You can continue working on other tasks or close the browser entirely. Your interactive session isn’t affected.
You will see a notification at the bottom of your screen confirming that the run started. To check the status of your run, choose View Run in the notification. This opens a view showing every background and scheduled run with its status, duration, and a link to view the full output.
You can choose to view the run details at any point to view results as cells run. The run details include three tabs:
Output: The notebook in read-only mode with cell results rendered, including dataframe outputs, visualizations, and print statements.
Parameters: The parameter values used for this run.
Logs: Run logs for debugging.
You can also access past runs by selecting the View Runs option in the notebook header.
Stopping an in-progress run
If you need to cancel a run, open the run, and choose Stop. The run terminates, and its status updates to reflect the cancellation.
What to know about background runs
Compute: Each background run uses its own dedicated compute, separate from your interactive session. Your interactive work isn’t interrupted.
Packages: The packages that you install through the notebook’s package manager will be available in your background runs. When you use !pip install in code cells, the asynchronous run installs those packages as well.
Local files: Background runs can’t access files stored locally in your notebook environment. Reference data from your project’s shared storage (Amazon S3) or connected data sources instead.
Startup time: Expect a few minutes of startup time while compute is provisioned and your environment is prepared.
Creating a recurring schedule
Now that you’ve confirmed asynchronous runs work correctly, you can automate the notebook on a schedule. Choose the schedule icon in the notebook header to open the schedule creation form.
Configure the following settings:
Schedule name: Enter a descriptive name, such as Daily Shipping Report.
Schedule type: Choose Recurring for repeated runs or One-time for a single future run.
Frequency: Define how often the notebook runs using a rate (for example, every one day) or a cron expression. Set the time zone and the start and end dates for the schedule. For example, set the schedule to run every day at 7:00 AM UTC starting tomorrow.
Flexible time window (optional): The number of minutes after the scheduled start time within which the run can be invoked. For example, with a 5-minute window, the notebook runs within 5 minutes of the start time.
Advanced settings:
Compute Instance: Keep the current settings or override with a different instance type for the asynchronous run to use.
Timeout: Set a maximum run duration to help prevent notebooks from running indefinitely. If left blank, it defaults to 60 minutes.
Choose Create.
The schedule appears in the Schedules tab of the activity panel. SageMaker Unified Studio creates an Amazon EventBridge Scheduler schedule for each schedule you configure.
Viewing schedule run history
To view past runs for a schedule, choose the schedule name in the Schedules activity panel. This opens the schedule details view, where you can see the list of runs triggered by that schedule, the duration of each run, and a link to open the notebook output for an individual run.
Editing and deleting schedules
To modify a schedule, choose Edit next to it in the Schedules panel. You can change the frequency, instance type, timeout, and other configuration fields. To pause or resume a schedule, choose Pause or Resume from the same menu. To remove a schedule, choose Delete from that menu. Deleting a schedule stops future runs but preserves historical run outputs in Amazon S3 for auditing purposes.
Parameterizing notebooks
With parameters, you can reuse a single notebook across different inputs without duplicating code. For example, you can run the same shipping performance report for each carrier by passing a different carrier name to each run.
Defining parameters
Open the Parameters activity panel and choose Add. Set the parameter name to carrier and the default value to GlobalFreight.
Using parameters in code
In your notebook, replace the second cell with the following code. This retrieves the carrier parameter value using the SageMaker Unified Studio Python SDK instead of the hardcoded value:
Creating schedules with different parameter values
Now create three schedules for the same notebook, each targeting a different carrier:
“daily-shipping-gf” with carrier = GlobalFreight.
“daily-shipping-mc” with carrier = MicroCarrier.
“daily-shipping-shipper” with carrier = Shipper.
When you view a historical run, a separate Parameters tab in the run output displays the parameter values that were active for that run.
You can also override parameter values when triggering an on-demand background run. Choose the menu on the Run all button, then choose Run with settings. You can keep the defaults or provide custom values for that run.
Orchestrating with Workflows
To combine notebooks into a multi-step pipeline, such as running a data calculation notebook before the shipping log notebook, you can use the Notebook Operator in the Workflows tool to orchestrate them.
To do this, choose the Add to workflows button under the options menu of the notebook header.
This takes you to the Workflows tool, adding a new Notebook Operator task with prefilled properties from your notebook. When configuring the Operator task:
Select the target notebook from the notebook menu.
Use the Parameters widget to pass notebook parameters into the run of the notebook.
Specify optional arguments such as the compute instance and timeout configuration for the run.
Workflows also supports polling for the status of a notebook run for a particular notebook using Notebook Sensor. In Workflows, you can add a new Sensor task by hovering on the edge of the existing Operator task, where a plus (+) button is displayed.
You can then search for and add the Notebook Sensor to the canvas.
When configuring the Sensor task, specify the notebook run ID within the text field. The Operator’s form field contains Jinja templating to retrieve the notebook run. If the Sensor is used within the same workflow as the Operator, this template can be copied to use within a Sensor to poll the notebook run. Select the target notebook from the notebook menu.
Within Workflows, you can configure notebook runs to emit outputs and use those outputs as inputs for subsequent notebook runs.
Building off of the previous shipping log notebook example, we will pass the carrier parameter from an upstream notebook’s output. Your shipping-logs-analysis notebook should be already set up.
Because the notebook depends on the carrier parameter, you can specify it in the Parameters panel.
Now, define a second notebook, calculate-best-carrier, which performs a calculation to determine our best carrier to use for shipping:
import pandas as pd
from sagemaker_studio import Project
# Initialize the project
proj = Project()
# Get the S3 root path
s3_root = proj.s3.root
df = pd.read_csv(s3_root + '/ShippingLogs.csv')
df.head()
carrier_stats = df.groupby('Carrier').agg(
total=('OrderID', 'count'),
late=('OnTimeDelivery', lambda x: (x == 'Late').sum())
).reset_index()
carrier_stats['late_pct'] = carrier_stats['late'] / carrier_stats['total'] * 100
best = carrier_stats.sort_values('late_pct', ascending=True).iloc[0]
best_carrier = best['Carrier']
print("Late % by carrier:")
print(carrier_stats.to_string(index=False))
print(f"\nBest carrier: {best_carrier} ({best['late_pct']:.1f}% late)")
To configure the calculate-best-carrier notebook’s outputs, you can choose the Variables panel. A new selector is available at the bottom of this panel which allows you to select variables to mark as outputs.
We want this notebook to emit the best_carrier variable.
Now, use the Add to workflows button as previously demonstrated to quickly add this notebook within a workflow. Chain a second Notebook Operator that points to our shipping-logs-analysis notebook. Because we specified a parameter dependency on carrier for this notebook, it’s available as an option in the Parameters widget menu.
When they’re chained, the notebook tasks detect the outputs set in upstream notebook runs. These outputs can be selected as keys within the Parameters widget of the Operator to pass into the run. This can be done recursively for an arbitrary number of Operator tasks. We can select the emitted best_carrier output from the calculate-best-carrier notebook.
You can now choose the Save button on the top left of the visual canvas and the Run button to start the workflow. When the workflow is completed, the specified notebook outputs are available in the Task Output panel and the notebook run result can be viewed in the Notebooks tool.
In a similar manner, the Notebook Sensor will also emit the notebook outputs from a particular notebook’s run which can be used within other tasks. This is useful when you want to retrieve outputs from a notebook run in another workflow.
Debugging a failed run with AI assistance
When viewing your past runs, you notice that a run from earlier today has a Failed status. Choose the failed run to open the notebook output in read-only mode.
In this example, suppose you incorrectly referred to column name ActualShippingDays as DeliveryDays. The run would fail with a KeyError: 'DeliveryDays' in the cell that computes late deliveries.
At the top of the failed run output, choose Troubleshoot with AI. Choosing the Troubleshoot with AI button lands you in the notebook with the Agent chat panel open.
The data agent analyzes the cell outputs, identifies the cell that errored, explains the root cause, and suggests a fix. In this case, it identifies that the column DeliveryDays doesn’t exist in the dataframe and suggests updating the code reference. You can review the change, then verify the fix by choosing Run in background from the Run all menu to trigger a test run before the next scheduled run.
Note: You can also use the Data Agent to create schedules and start notebook runs using natural language, without having to navigate.
Cleaning up
To avoid incurring future charges, delete the resources that you created in this walkthrough:
Delete any schedules that you created from the Schedules panel in your notebook.
Delete test notebooks if you don’t need them.
Navigate to the Workflows page and delete any workflows that you created during this walkthrough.
Your project’s Amazon S3 storage retains historical run outputs until you manually remove them.
Conclusion
In this post, we showed how to run notebooks in the background in Amazon SageMaker Unified Studio using background runs, schedules, parameterization, workflow orchestration, and AI-assisted debugging. Using a shipping logistics dataset, we demonstrated how a single notebook can be parameterized to generate performance reports for different carriers on independent schedules, all without duplicating code or managing extensive infrastructure.
To get started, open a notebook in your SageMaker Unified Studio project, choose the menu on the Run all button in the notebook header, and choose Run in background. For more advanced use cases, explore workflows in Amazon SageMaker Unified Studio to build multi-step data pipelines, or review the Amazon SageMaker Unified Studio User Guide for additional configuration options.
Enterprises face challenges when teams create data assets outside of central data catalogs. It adds overhead for discovery, and limits collaboration. Amazon’s Business Data Technologies (BDT) team has built an enterprise data catalog (Andes) for sharing datasets under well-defined policies. However, teams created catalog of local datasets and other non-tabular assets such as dashboards and metrics, outside Andes. This made it difficult to discover all assets in a consolidated way.
In this post, we share how Amazon.com is working to integrate catalogs by extending enterprise data catalog Andes with Amazon SageMaker.
Need for expanding catalog and governance from datasets to data assets
Without a single solution, users had to search multiple catalogs depending upon the asset type. Teams spent considerable time indexing the different catalogs and identifying the right one for their task. This slowed them down and took time away from solving the business problems.
To address these challenges, BDT team identified four critical capabilities needed:
Multimodal catalog – Data consumers required the ability to blend enterprise data with local datasets and use them together for specific use cases. Teams sought to discover not only datasets, but also assets such as metrics, dashboards, and business files, to obtain a complete view of available resources. This necessitated a catalog that consolidates datasets and data assets in one location.
Uniform governance and enforcement – To maintain best data protection practices and support business goals, teams need consistent enterprise-wide data governance where they request access once and the system enforces that access uniformly across all compute engines, alleviating fragmented or redundant access management. For internal systems, there was need for trusted identity propagation so user identity is preserved and used across AWS and internal systems for consistent enforcing.
Multi-approval workflows – The solution supports multiple approval workflows within a single system, using Andes for dataset approvals and a custom workflow for dashboard approvals to maintain total governance and visibility across data assets.
Delegated ownership – While enterprise teams retain overarching governance responsibility, business-specific data stewards required the ability to modify select attributes and apply appropriate tags to assets produced by their respective producers and consumers.
Solution: Unify datasets and data assets with Amazon SageMaker
Amazon chose to extend Andes with Amazon SageMaker to enhance the discovery experience. SageMaker offers native support for multimodal catalogs, and integrated with enterprise identity management, making it the ideal foundation for extending Andes’ governance model.
Rather than broadcasting assets across multiple domains, a single enterprise-wide domain standardizes and synchronizes data assets in one place. This domain is associated with AWS IAM Identity Center, which is connected to Amazon’s corporate identity system to maintain best data protection practices by limiting direct permissions and using corporate identity and group-based permissions.
This integrated architecture directly addresses the identified challenges:
Single-pane asset discovery – Datasets and data assets are accessible through a single, consolidated view, avoiding the need to navigate across disparate systems or domains. This simplifies discovery and reduces the time to insight for teams across the organization.
Extended governance – Governance of both enterprise-wide and local datasets is orchestrated through a single system.
Extended observability – Trusted Identity Propagation (TIP) through AWS IAM Identity Center allows human users to access data interactively using their corporate identities. This provides audit-trail visibility into who is accessing what data for audits and organization’s observability requirements.
Amazon tool integration – Integration with Git and other internal systems automates management of accounts, permissions, and approvals. This reduces manual overhead and helps maintain that access controls remain tightly aligned with existing business workflows.
Design overview
This section describes the key features and design of the Amazon SageMaker integration. The technical implementation consists of three core components:
1) Catalog connectors
Amazon built connectors and ingestion paths to bring data assets into Amazon SageMaker while maintaining business continuity and preserving existing governance:
Andes integration: SageMaker provides APIs to synchronize assets from external catalogs. BDT extended this to bring Andes datasets (with their sophisticated metadata, business context) into the integrated experience. The integration preserves Andes’ permission model and governance workflows, to maintain existing security standards and best practices intact.
Account onboarding: Teams self-serve onboard their AWS accounts through an AWS Lambda-based integration. When creating projects, SageMaker queries this service to determine which accounts a user’s identity can access.
2) Delegated ownership
When data systems scale across business units, centralized governance teams need to delegate permissions for catalog enrichment, policy enforcement, and metadata management.
Catalog enhancement allows business teams to define and publish their own business glossaries, curated vocabularies of domain-specific terms, definitions, and relationships, directly within the catalog. Allowing business owners to author and maintain these glossaries increased accuracy and discoverability of catalog assets. Data consumers across the enterprise benefit from clearer, more consistent terminology.
3) Integration with consumption and access tooling
Teams discover data in SageMaker Unified Studio and consume it through both SageMaker Unified Studio and internal tooling:
Data discovery: SageMaker Unified Studio integrates with Amazon-wide Identity Center allowing almost all Amazon users to authenticate and search for cataloged assets. This integration addresses the data discovery problem by providing enterprise-wide visibility into available data resources.
Integrated development environment: SageMaker Unified Studio provides built-in tooling out of the box including a Query Editor for SQL analytics and Amazon SageMaker AI for machine learning (ML), which helps teams access data, build models, and collaborate across organizational boundaries.
Code repository integration: Manage code with full Git operations supported from SageMaker Unified Studio. Query code and notebook code persist to GitFarm (Amazon’s internal Git system), allowing teams to view and manage their work through Amazon’s standard version control system.
Native analytics integration: Projects directly connect to AWS analytics engines including Amazon Athena for SQL, AWS Glue and Amazon EMR for Apache Spark, and Amazon Redshift for data warehousing. User-authored jobs use Andes governance and permissions across engines for consistent access control.
SageMaker implementation results
SageMaker catalog now encompasses various types of data assets from across the organization, representing an expansion from datasets alone to a complete inventory of data, dashboards, metrics, models, and other data assets, all while maintaining best practices and appropriate access and use guardrails.
“SageMaker provides a unified catalog that makes discovery and sharing of data assets, metrics and dashboards across teams straightforward, with direct integration to Andes datasets. SageMaker delivers deep integration through Git repository connections and enterprise identity management that aligns with existing Amazon workflows.”
– Gerry Moses, Sr. Principal TPM, Amazon
Faster data discovery – Data consumers can go to one place to locate trusted, high-quality assets with significantly less friction, which reduces the time from question to insight. By surfacing well-documented, governed assets through an enriched catalog, teams can confidently identify the right data for their use cases without navigating sprawling, inconsistent inventories or relying on tribal knowledge.
Improved collaboration – Breaks down data silos by making curated assets discoverable and reusable across Amazon. When teams can build on shared, authoritative datasets rather than creating redundant copies, data proliferation is reduced.
Conclusion
By integrating their existing governance tooling with Amazon SageMaker to build a centralized data catalog, BDT is creating a foundation for faster, more efficient data discovery across teams. Amazon SageMaker helped unify diverse data types with their existing catalog and enabled collaboration across teams to help them find the right data. By integrating with existing governance frameworks, BDT demonstrates how organizations can expand their catalog capabilities while preserving existing enterprise investments.
Data teams understand which services their applications need but lack continuous integration and continuous delivery (CI/CD) expertise, while DevOps teams understand deployment automation but must learn each AWS service’s provisioning requirements.
The CI/CD CLI for Amazon SageMaker Unified Studio (aws-smus-cicd-cli) is an open source command line tool that automates deployment of multi-service data and AI applications across pipeline stages. Data teams define their application once in a YAML manifest, DevOps teams deploy with a single command, and the CLI handles configuration substitution, dependency ordering, and resource provisioning automatically. For details, see the CI/CD CLI documentation.
In this post, we walk through how the CI/CD CLI works, show you how to deploy a real application across environments, and demonstrate how it fits into your existing CI/CD workflows.
Customer spotlight
Bureau Veritas, a global leader in testing, inspection, and certification, operates across multiple SageMaker Unified Studio environments to support its data and AI teams. With their data and DevOps teams working on different parts of the application lifecycle, Bureau Veritas needed a controlled way to promote workloads from development through test to production while preserving clear ownership boundaries between the two teams.
“We need to promote data and AI applications across SageMaker Unified Studio environments in a controlled way that respects the boundaries between our data teams and our DevOps teams. The CI/CD CLI does exactly that — a single manifest from the data team, a single deploy command from DevOps, and full control over what goes to production.”
— Gilles Kempf, Architecture Manager, Bureau Veritas
How the CI/CD CLI works
The CI/CD CLI introduces a clean separation of concerns between data teams and DevOps teams.
Data teams define what to deploy in a declarative YAML manifest (manifest.yaml). The manifest describes the application’s resources, including AWS Glue extract, transform, and load (ETL) jobs, Athena queries, Airflow directed acyclic graphs (DAGs), Quick Sight dashboards, and SageMaker training jobs, along with stage-specific configurations for each environment.
DevOps teams define how and when to deploy using their existing CI/CD systems. They retain full control over their deployment methodology. They choose whether to promote content through git branches, a bundle artifactory, or both; they decide the shape of the pipeline, including which stages to include (dev, staging, pre-prod, prod) and which manual approvals or security gates are required. They run aws-smus-cicd-cli deploy inside GitHub Actions, Jenkins, or GitLab CI workflows without needing to understand which AWS services the application uses or how SageMaker Unified Studio projects are structured. The CLI is a utility for AWS analytics service deployment, not a CI/CD methodology. Your team’s existing conventions for branches, approvals, and pipeline shape stay exactly as they are.
The CLI is the abstraction layer between the two. It reads the manifest, substitutes stage-specific configurations (S3 paths, AWS Identity and Access Management (IAM) roles, account IDs, and connection strings), provisions resources in dependency order, and handles all AWS service interactions.The following diagram illustrates this separation:
Key concepts
Application manifest
Each stage maps to a dedicated SageMaker Unified Studio project. This one-stage-to-one-project mapping is the foundation of CI/CD isolation: each project has its own domain, IAM boundaries, connections, and data, so changes in dev can never affect prod. For stronger isolation, projects can span different AWS accounts and AWS Regions. For example, dev in a sandbox account and prod in a production account in a different Region. Because each stage is a real SageMaker Unified Studio project, teams can open it in the console at any time to observe workflows, inspect resources, and troubleshoot deployments. Project membership is managed per project, so you control exactly who has access to each stage. For example, developers in dev and a release team in prod.The manifest file is the single source of truth for your application. It declares:
Content: application code from git repositories, data files from S3, Quick Sight dashboards, and workflow definitions.
Stages: environment-specific project mappings (dev, test, prod, etc.), each isolated as described earlier.
Configuration: stage-specific settings that are substituted automatically at deploy time.
Here is an example manifest for an analytics application with AWS Glue ETL and Quick Sight: applicationName: SalesAnalyticsDashboard
Each stage must map to a separate SageMaker Unified Studio project, providing full isolation between environments. The CLI substitutes variables like ${AWS_ACCOUNT_ID} and ${AWS_REGION} at deploy time based on the target environment.
Bundles
A bundle is an immutable, versioned archive of your application. The bundle command reads from a source stage (typically dev) and packages the application code, workflow definitions, and resolved configurations into a self-contained artifact. The deploy command then applies that artifact to one or more target stages (test or prod).
This stage-to-bundle-to-stage promotion model supports controlled rollout through quality gates:
# Package from dev
aws-smus-cicd-cli bundle --manifest manifest.yaml
# Deploy to test
aws-smus-cicd-cli deploy --manifest app.tar.gz --targets test
# Validate the test deployment
aws-smus-cicd-cli test --manifest manifest.yaml --targets test
# Promote the same bundle to prod
aws-smus-cicd-cli deploy --manifest app.tar.gz --targets prod
The same artifact is deployed at every stage without rebuilding, providing audit trails and reproducible deployments for regulated industries.
SageMaker Catalog integration
The CLI manages Amazon SageMaker Catalog resources as part of the deployment process. You can define catalog assets, glossaries, glossary terms, form types, asset types, and metadata forms, in your manifest. During deployment, the CLI searches for assets in the catalog, creates subscription requests for required data access, and waits for approval before proceeding. This automates the data governance workflow that teams previously handled manually.
CLI commands
The CI/CD CLI provides commands that cover the full deployment lifecycle:
Command
Description
describe
Validates the manifest, checks that target projects exist, and confirms the execution role has required permissions. Use –connect to validate against live AWS environments.
bundle
Reads from a source stage and packages application code, workflow definitions, and configurations into an immutable, versioned archive.
deploy
Applies bundle contents to one or more target stages. Provisions resources in dependency order.
test
Runs post-deployment validation to confirm services are running and ready for workloads.
create
Generates a starter manifest from an existing SageMaker Unified Studio project.
run
Triggers Airflow workflow execution on MWAA or Airflow Serverless connections.
monitor
Monitors workflow execution status in real time.
logs
Fetches and streams workflow execution logs.
destroy
Removes deployed resources and projects for cleanup or failure recovery.
Walkthrough: deploying a Quick Sight dashboard with AWS Glue ETL
In this section, we walk through deploying an analytics application that uses AWS Glue for ETL, Athena for queries, and Quick Sight for dashboards. This example is available in the GitHub repository.
Use case
An analytics team owns a Sales Analytics Dashboard built on AWS Glue ETL, Athena, and Quick Sight. They want to promote changes from a development environment to production with reproducible builds, automated validation, and a clear approval gate between stages, without writing custom deployment scripts or exposing data engineers to AWS provisioning details.
Solution overview
We use a sample application from the CI/CD CLI GitHub repository that includes AWS Glue ETL scripts, an Airflow workflow definition, a Quick Sight dashboard bundle, and integration tests. A single manifest.yaml describes the application and its dev and prod stages. The CLI handles the full lifecycle: bundle the app from dev, deploy it to test, run validation, and promote the same immutable artifact to prod.
Prerequisites
Before you begin, make sure you have the following:
Existing SageMaker Unified Studio projects for your target stages.
Solution architecture
Each stage in the manifest maps to a dedicated SageMaker Unified Studio project (see the separation-of-concerns diagram in “How the CI/CD CLI works” earlier in this post). At deploy time, the CLI uploads ETL scripts and workflow definitions to the project’s S3 storage connection, provisions the Airflow workflow in MWAA Serverless, runs the workflow to create AWS Glue jobs and databases, and imports the Quick Sight dashboard. The same bundle artifact is applied to every downstream stage, ensuring dev, test, and prod stay in sync while remaining fully isolated.
Solution implementation
Step 1: Install the CLI
Install the CLI from PyPI:
pip install aws-smus-cicd-cli
Step 2: Create or customize a manifest
Clone the repository and start from the analytics example:
The example includes AWS Glue ETL scripts, an Airflow workflow definition, a Quick Sight dashboard bundle, and integration tests. Open manifest.yaml and update the project, domain, and deployment_configuration values under each stage so they match your own SageMaker Unified Studio projects and connection names.Alternatively, generate a manifest from an existing project: aws-smus-cicd-cli create --domain-id <your-domain-id> --dev-project-id <your-project-id>
Step 3: Validate your configuration
Run the describe command with --connect to verify your environment is ready. This connects to your AWS environment and validates that target projects exist, the execution role has the required permissions, and connections are reachable. Fix any issues before deploying.
aws-smus-cicd-cli deploy --targets test --manifest manifest
During deployment, the CLI:
Uploads ETL scripts and workflow definitions to S3 using the project’s storage connection.
Creates the Airflow workflow in MWAA Serverless.
Runs the workflow, which provisions AWS Glue jobs, creates databases, and runs ETL transformations.
Imports the Quick Sight dashboard and refreshes datasets with the latest data.
Processes any catalog asset subscriptions defined in the manifest.
Step 5: Validate
Run post-deployment validation to confirm services are running and ready for workloads:
aws-smus-cicd-cli test --manifest manifest.yaml --targets test
Step 6: Promote to production
Promote the same bundle artifact that was validated in the test stage to production. This guarantees the exact same artifact runs in prod:
# Promote the same bundle that was validated in test to prod
aws-smus-cicd-cli deploy --manifest app.tar.gz --targets prod
Integrating with GitHub Actions
The CLI works with existing CI/CD solutions. The GitHub repository includes reusable workflow templates that DevOps teams can adopt directly.The following is an example of a GitHub Actions workflow that implements a full bundle-based deployment pipeline:
In the next section, we cover which AWS services and workload types the CLI supports.
Supported workloads
The CLI deploys applications that span the following AWS services through Airflow workflow definitions:
Analytics and BI: AWS Glue ETL jobs and crawlers, Amazon Athena queries, Amazon Quick Sight dashboards, Amazon EMR jobs, Amazon Redshift queries.
Machine learning: SageMaker training jobs, ML model endpoints, SageMaker AI Pipelines.
Code and workflows: Jupyter notebooks, Python scripts, Airflow DAGs (MWAA and MWAA Serverless).
Data and storage: S3 data files, Git repositories, SageMaker Catalog resources (glossaries, glossary terms, form types, asset types, assets, data products, metadata forms).
The examples directory includes working applications for each of these patterns, with manifests, workflow definitions, and integration tests.
Failure recovery
If a deployment fails, the CLI stops at the point of failure and reports the error with a detailed stack trace. To recover:
Run aws-smus-cicd-cli describe --connect to check which resources exist and which permissions are missing.
Fix the issue and rerun aws-smus-cicd-cli deploy.
For bundle-based deployments, redeploy a previous bundle version.
Use aws-smus-cicd-cli destroy --targets <target> --force to clean up a failed deployment.
In this post, you learned how the Amazon SageMaker Unified Studio CI/CD CLI gives data and DevOps teams a clean separation of concerns: data teams describe their application once in a YAML manifest, and DevOps teams deploy it with a single command through their existing CI/CD pipelines. You saw how stages map to isolated SageMaker Unified Studio projects (optionally spanning AWS accounts and Regions), how bundles provide immutable, reproducible promotion through test and production, and how the CLI integrates with GitHub Actions, Jenkins, GitLab CI, and Azure DevOps. You also walked through deploying a Glue-and-Quick-Sight analytics application from dev through to prod.
Get started
The CI/CD CLI is available at no additional cost in all AWS Regions where Amazon SageMaker Unified Studio is available. You pay only for the underlying AWS resources provisioned during deployment.
Use the following steps to try it out:
Install the CLI:
pip install aws-smus-cicd-cli
Browse the example applications for analytics and ML patterns.
Data engineers running Apache Spark jobs on Amazon EMR face a persistent challenge: understanding how data moves through Spark pipelines as it’s transformed, joined, and written to downstream tables . Tracking these transformations manually requires examining job logs, reviewing code, and piecing together transformation logic across multiple sources. As pipelines scale, this process becomes complex. The visibility gap affects key business activities: troubleshooting data quality issues takes longer – impact analysis for schema changes requires more effort – and compliance audits need extensive documentation of data provenance.
Amazon SageMaker is the center for all your data and analytics where you can find and access all the data in your organization and act on it using tools across various use case. This unified platform addresses the data visibility challenge by bringing together data governance, collaboration, and discovery into a single interface. At the heart of this platform is Amazon SageMaker Catalog, a centralized hub that enables organizations to catalog, govern, and discover all their data assets with complete visibility into lineage. By capturing data lineage across your entire data ecosystem from raw sources through transformations to final outputs, SageMaker Catalog enables you to track data provenance across your entire platform, enable collaboration with clear visibility into data ownership and quality metrics, build trust through comprehensive data lineage that supports compliance and confident decision-making, and accelerate discovery of trustworthy, governance-ready data assets. You can access and visualize this lineage directly in Amazon SageMaker Unified Studio, which serves as the unified interface to explore data relationships and collaborate across your analytics workflows.
Amazon EMR, starting from version 7.11, now includes native OpenLineage support that automates lineage capture. OpenLineage is an open-source framework for data lineage that automatically emits lineage metadata from your data transformation jobs directly into Amazon SageMaker Catalog, or other data governance solutions, without requiring customizations.
This EMR native support of OpenLineage is part of a growing set of integrations across AWS analytics services including AWS Glue, Amazon EMR Serverless, and Amazon Redshift. The complete list of services with native OpenLineage integration can be found in the data lineage support matrix.
In this post, you’ll walk through a practical, step-by-step example that shows how to capture and track data lineage from Spark jobs running on Amazon EMR directly into Amazon SageMaker Catalog using OpenLineage. You’ll see how lineage metadata flows automatically and explore data relationships and dependencies across your workflows in Amazon SageMaker Unified Studio.
Solution overview
Imagine you’re part of a large enterprise that relies on HR analytics to optimize workforce planning, compensation strategies, and talent retention practices. Your data engineering team owns the delivery of these analytical products by processing raw HR datasets (including employee records, attendance logs, and compensation details), with Spark jobs running on your Amazon EMR infrastructure.
With time, Spark jobs have grown in complexity. Your team now struggles to maintain visibility into how data moves through pipelines, who modified it, and how to map dependencies between datasets and final analytical products.
The following solution demonstrates how you can address these challenges by automatically capturing data lineage end-to-end from Spark jobs running on your EMR infrastructure and visualizing it in Amazon SageMaker Unified Studio so that you and the business understand data provenance of the final analytical products.
The architecture includes a Data Layer with CSV files containing employee, attendance, salary, and bonus data stored in Amazon S3 (Simple Storage Service), representing typical HR and payroll source systems.
The Processing Layer uses Amazon EMR cluster running Apache Spark jobs that transform raw data into analytical tables. The first Spark job joins employee and attendance data while the second Spark job combines attendance with compensation data. Both jobs use Apache Iceberg table format to provide ACID (Atomic, Consistent, Isolated, and Durable) transactions and time travel capabilities.
The Metadata Layer uses AWS Glue Data Catalog to store Iceberg table metadata, making tables discoverable and accessible across AWS analytics services. A Lineage Layer uses the OpenLineage integration in EMR to automatically track input/output datasets (CSV files and Iceberg tables), transformation logic at column level (joins, filters, aggregations), and job execution metadata.
Finally, the Data Governance Layer uses Amazon SageMaker Catalog to capture and process OpenLineage events posted by the EMR Spark jobs and automatically build a comprehensive lineage graph that shows complete data provenance from CSV source files through Spark transformations to Iceberg analytical tables.
Before you deploy this solution, make sure you have the following resources in place.
Prerequisites
For this walkthrough, you should have the following prerequisites:
A Amazon VPC (Virtual Private Cloud) with at least one subnet with internet access. You can provision this VPC as you create the Amazon SageMaker domain next.
An existing Amazon SageMaker Unified Studio domain and project. To get started, use the quick setup option as explained here. To create a project, follow the instructions here.
An S3 bucket with the sample data files and Spark scripts uploaded (see Prepare Your Source Data below)
Default EMR service roles — if this is your first time using EMR in this account, run `aws emr create-default-roles` from the AWS CLI or CloudShell to create them.
With these prerequisites in place, let’s examine what the AWS CloudFormation template will deploy to your AWS environment.
Architecture components
The deployment creates several interconnected components that work together to capture and visualize lineage:
An S3 bucket to store all data and artifacts for the solution.
An EMR cluster (v 7.12.0) with Apache Iceberg support enabled and OpenLineage integration pre-installed, ready to run Spark jobs with lineage tracking.
A set of IAM policies that grant the necessary permissions to the EMR cluster to post lineage events to your SageMaker Unified Studio domain.
A set of AWS Lake Formation permissions that grant the EMR cluster to create, alter, and drop Iceberg tables in your specified Glue database.
With an understanding of what will be deployed, you’re ready to launch the CloudFormation stack.
Deploy the solution
Note: While this walkthrough uses the AWS EMR console and AWS CLI to verify the cluster and run Spark jobs, you can also perform these steps directly from Amazon SageMaker Unified Studio. SMUS provides a unified interface to create and manage EMR clusters, submit Spark jobs, and monitor execution — all within the same environment where you’ll later explore the lineage captured in Amazon SageMaker Catalog.
Prepare your source data
Before deploying the CloudFormation stack, clone or download the following git repository.PutHereGitRepo
Upload the CSV files downloaded from git to the input/ prefix and the spark scripts in scripts/ prefix. You can run the following command to upload the files:
To deploy the solution, complete the following steps in CloudFormation console:
Create new stack by specifying the CloudFormation yaml file previously download from git repository PutHereThe YMLFileName
Enter a stack name (e.g., emr-lineage-demo) and provide the following parameters:
SourceS3BucketName: S3 bucket containing your CSV files and Spark scripts
SourceCSVPrefix: S3 prefix where CSV files are located
SourceScriptsPrefix: S3 prefix where Spark scripts are located
GlueDatabaseName: The name of the Glue database associated to your Amazon SageMaker Unified Studio project.
DataZoneDomainId: Your SageMaker Unified Studio domain ID.
VpcId: The id of the VPC that was deployed as part of the prerequisites.
For EMRReleaseLabel, MasterInstanceType, CoreInstanceType and CoreInstanceCount, keep the default values.
Acknowledge IAM resource creation, choose Next and then Submit. The CloudFormation stack takes approximately 10 to 15 minutes to complete.
In the EMR console, wait for the cluster status to show as WAITING before moving to the next step.
Now that the EMR cluster is running with OpenLineage enabled, let’s examine how the Spark jobs are configured to capture lineage metadata.
Explore data lineage configuration in EMR
When submitting Spark jobs to EMR, specific configurations enable OpenLineage to create and post lineage events to SageMaker Unified Studio as the job runs:
spark.hadoop.hive.metastore.client.factory.class – Configures Spark to use AWS Glue as the Hive metastore.
spark.jars – Path to the pre-installed OpenLineage library (available on EMR 7.11+).
spark.extraListeners – Registers an OpenLineage listener to capture metadata of input / output datasets and transformations.
spark.openlineage.transport.type – Uses the OpenLineage DataZone transport option to send lineage events directly into SageMaker Catalog.
spark.openlineage.transport.domainId – The ID of your SageMaker Unified Studio domain, that serves as the target for lineage events.
spark.glue.accountId – Your AWS account ID for Glue data catalog operations.
Now that you understand the configuration that enables automatic lineage capture, you’re ready to run the data pipeline.
When running this two-step pipeline, you will calculate the total employee compensation by combining salary adjustments, bonuses, and attendance data. The final analytical asset will serve payroll processing and budgeting.
Run employee attendance analysis job
The first job reads employee details (in employees.csv dataset) and attendance records (in attendance.csv dataset), joins the datasets on EmployeeID and creates a unified dataset (employee_attendance Iceberg table) in your Glue database.
Copy the value of the Job1SubmitCommand output key. Note that this is the command you’ll use to submit the first job in EMR with the right configuration.
Run the command in your terminal or AWS CloudShell.
Monitor the job in the Amazon EMR console under Steps.
Run employee compensation analysis job
Now, you will calculate the total employee compensation (Iceberg table) by combining salary adjustments (salary_adjustments.csv dataset), bonuses (bonus_payments.csv dataset), and attendance (calculated in the last step):
Navigate to Data Catalog, then Tables and select your SageMaker project’s database.
Confirm that employee_attendance and employee_compensation tables are listed.
With both Spark jobs complete, you can now visualize the complete data lineage graph in Amazon SageMaker Unified Studio.
Visualizing lineage in SageMaker Unified Studio
SageMaker Unified Studio provides a graph-based data lineage visualization that helps data engineers, analysts, and data scientists clearly understand which source datasets (files or tables) feed into each dataset, what transformations and logic are applied at every step, which downstream analytics assets consume the data, and how changes to upstream data or transformations may impact the rest of the data pipeline.
Now that the data pipeline run successfully, let’s review the captured lineage for the HR data in SageMaker Unified Studio:
Navigate to Assets and Click on employee_compensation. Under the LINEAGE tab you’ll find the lineage graph view that SageMaker builds based on the OpenLineage metadata captured from the EMR Spark jobs as they run.
You’ll first see three lineage nodes from left to right: one representing the EMR Spark job that created the final Iceberg table, a second one representing the actual Iceberg table in the Glue catalog, and a third one representing the data asset in the SageMaker Catalog inventory that maps to the Glue table.
Click on any lineage node to view its underlying metadata in the details pane, including dataset names, S3 locations, schema, data types, job execution details and more.
Expand the lineage to the left by clicking on the double arrow next to the first lineage node. Keep expanding until you hit the originating datasets.
Expanding the graph to the left reveals the complete data pipeline back to original CSV source files. You can see how compensation data depends on upstream attendance analytics.
Note how each lineage node represents an element in the data pipeline you run, including both Spark jobs and even the intermediate employee_attendance Iceberg table that connects them.
You can expand column-level lineage by clicking on the column section of a lineage node of a dataset or data asset. This allows you to understand how data changes at a column level as it goes downstream your data pipeline.
Cleanup
To avoid ongoing charges, clean up the resources:
First, empty the destination bucket by running the following command in your terminal or with AWS CloudShell.
Choose the stack you created, then choose Delete and then Delete stack when prompted.
Conclusion
In this post, you explore how to capture data lineage from Spark jobs in Amazon EMR (v7.11+) directly into Amazon SageMaker Unified Studio. You learned how to set up an Amazon EMR cluster with native OpenLineage support to automatically track lineage metadata from Spark jobs processing your data. You also configured the integration between EMR and Amazon SageMaker Catalog to ensure lineage information flows seamlessly into your governance platform. Finally, you explored the resulting lineage graph in SageMaker Unified Studio and saw how it provides comprehensive visibility into data transformations, from source CSV files through Spark processing jobs to final analytical tables using Apache Iceberg format.
We encourage you to now test these capabilities with your own data pipelines running on EMR. By implementing automated lineage tracking, many customers have strengthened their governance frameworks while gaining valuable insights into data dependencies, impact analysis, and compliance requirements. This approach enables data teams to build trust in their analytics outputs while maintaining the agility needed to derive business value from their data assets.
As your data and machine learning (ML) assets grow, tracking which assets lack documentation or monitoring asset registration trends becomes challenging without custom reporting infrastructure. You need visibility into your catalog’s health, without the overhead of managing ETL jobs. The metadata feature of Amazon SageMaker provides this capability to users. Converting catalog asset metadata into Apache Iceberg tables stored in Amazon S3 Tables removes the need to build and maintain custom ETL pipelines. Your team can then query asset metadata directly using standard SQL tools. You can now answer governance questions like asset registration trends, classification status, and metadata completeness using standard SQL queries through tools like Amazon Athena, Amazon SageMaker Unified Studio notebooks, and BIsystems.
This automated approach reduces ETL development time and gives your team visibility into catalog health, compliance gaps, and asset lifecycle patterns. The exported tables include technical metadata, business metadata, project ownership details, and timestamps, partitioned by snapshot date to enable time travel queries and historical analysis. Teams can use this capability to proactively monitor catalog health, identify gaps in documentation, track asset lifecycle patterns, and make sure that governance policies are consistently applied.
How metadata export works
After you enable the metadata export feature, it runs automatically on a daily schedule:
SageMaker Catalog creates the infrastructure — An Amazon Simple Storage Service (Amazon S3) table bucket named aws-sagemaker-catalog is created with an asset_metadata namespace and an empty asset table.
Daily snapshots are captured — A scheduled job runs once per day around midnight (local time per AWS Region) to export updated asset metadata.
Metadata is structured and partitioned — The export captures technical metadata (resource_id, resource_type), business metadata (asset_name, business_description), project ownership details, and timestamps, partitioned by snapshot_date for query performance.
Data becomes queryable — Within 24 hours, the asset table appears in Amazon SageMaker Unified Studio under the aws-sagemaker-catalog bucket and becomes accessible through Amazon Athena, Studio notebooks, or external BI tools.
Teams query using standard SQL — Data teams can now answer questions like “How many assets were registered last month?” or “Which assets lack business descriptions?” without building custom ETL pipelines.
The export evaluates catalog assets and their metadata properties in the domain, converting them into Apache Iceberg table format. The data flows into downstream analytics operations immediately, with no separate ETL or batch processes to maintain. The exported metadata becomes part of a queryable data lake that supports time-travel queries and historical analysis.
In this post, we demonstrate how to use the metadata export capability in Amazon SageMaker Catalog and perform analytics on these tables. We explore the following specific use-cases.
Audit historical changes to investigate what an asset looked like at a specific point in time.
Monitor asset growth view how the data catalog has grown over the last 30 days.
Track metadata improvements to see which assets gained descriptions or ownership over time.
Solution overview
Figure 1 – SageMaker catalog export to S3 Tables
The architecture consists of three key components:
Amazon SageMaker Catalog exports asset metadata daily to Amazon S3.
S3 Tables stores metadata as Apache Iceberg tables in the aws-sagemaker-catalog bucket with ACID compliance and time travel.
Query engines (Amazon Athena, Amazon Redshift, and Apache Spark) access metadata using standard SQL from the asset_metadata.asset table.
What metadata is exposed?
SageMaker Catalog exports metadata in the asset_metadata.asset table:
Metadata Type
Fields
Description
Technical metadata
resource_id, resource_type_enum, account_id, region
Resource identifiers (ARN), types (GlueTable, RedshiftTable, S3Collection), and location
An Amazon SageMaker project for publishing assets.
For SageMaker Unified Studio domain setup instructions, refer to the SageMaker Unified Studio Getting started guide.
After you complete the prerequisites, complete the following steps.
Add this policy to our IAM user or role to enable metadata export. If using SageMaker Unified Studio to query the catalog, add this policy to the AmazonSageMakerAdminIAMExecutionRole managed role.
Grant describe and select permissions for SageMaker Catalog with AWS Lake Formation. This step can be performed in the AWS Lake Formation console.
Select Permissions -> Data permissions and choose Grant.
Figure 2 – AWS Lake Formation grant permission
Under Principal type, select Principals, IAM users and roles and the AWS managed AmazonSageMakerAdminIAMExecutionRole execution role.
Choose Named Data Catalog resources.
Under Catalogs, search for and select <account-id>:s3tablecatalog/aws-sagemaker-catalog.
Under Databases, select asset_metadata database.
Figure 3 – AWS Lake Formation catalog, database, and table
Figure 4 – AWS Lake Formation grant permission
For Table, select asset.
Under Table permissions, check Select and Describe.
Choose Grant to save the permissions.
Enable data export using the AWS CLI
Configure metadata export using the PutDataExportConfiguration API. The Amazon DataZone service automatically creates an S3 table bucket named aws-sagemaker-catalog with an asset_metadata namespace, and schedules a daily export job. Asset metadata is exported once daily around midnight local time per AWS Region.
The SageMaker Domain identifier is available on domain detail page in the AWS Management Console. Accessing the asset table through the S3 Tables console or the Data tab in SageMaker Unified Studio can require up to 24 hours.
AWS CLI command to enable SageMaker catalog export:
Navigate to Amazon SageMaker Domains in the AWS Management Console.
Select your domain and select Open.
Figure 5 – Open Amazon SageMaker Unified Studio
In SageMaker Unified Studio, choose a project from the Select a project dropdown list.
To query SageMaker catalog data, select Build in the menu bar and then choose Query Editor. To create a new project, follow the instructions in the Amazon SageMaker Unified Studio User Guide.
Figure 6 – Open SageMaker Unified Studio Query Editor
The asset_metadata.asset table is available in Data explorer. Use Data explorer to view the schema and query data to perform analytics from.
Expand Catalogs in Data explorer. Then, select and expand s3tablecatalog, aws-sagemaker-catalog, asset_metadata, and asset.
Test querying the catalog with SELECT * FROM asset_metadata.asset LIMIT 10;.
Figure 7 – Query SageMaker catalog
Queries for observability and analytics
With setup complete, execute queries to gain insights on catalog usage and changes. To monitor asset growth, and view how the data catalog has grown over the last five days:
SELECT
DATE (snapshot_time) as date,
COUNT (*) as total_assets
FROM asset_metadata.asset
WHERE
DATE (snapshot_time) >= CURRENT_DATE - INTERVAL '5' DAY
GROUP BY DATE (snapshot_time)
ORDER BY date DESC;
Figure 8 – Query asset growth
Use the catalog to track metadata changes to determine which assets gained descriptions or ownership over time. Use this query to identify assets that gained business descriptions over the past five days by comparing today’s snapshot with the earlier snapshot.
SELECT
t.asset_id,
t.resource_name,
p.business_description as description_before,
t.business_description as description_now
FROM asset_metadata.asset t
JOIN asset_metadata.asset p ON t.asset_id = p.asset_id
WHERE DATE(t.snapshot_time) = CURRENT_DATE
AND DATE(p.snapshot_time) = CURRENT_DATE - INTERVAL '5' DAY
AND p.business_description IS NULL
AND t.business_description IS NOT NULL;
Investigate asset values at a specific point in time using this query to retrieve metadata from any snapshot date.
SELECT
asset_id,
resource_name,
business_description,
extended_metadata['owningEntityId'] as owner,
snapshot_time
FROM asset_metadata.asset
WHERE asset_id = 'your-asset-id'
AND DATE(snapshot_time) = DATE('2025-11-26');
Clean up resources
To avoid ongoing charges, clean up the resources created in this walkthrough:
Disable metadata export:
Disable the daily metadata export to stop new snapshots:
Optionally, delete the S3 Tables namespace containing the exported metadata to remove historical snapshots and stop storage charges. For instructions on how to delete S3 tables, see Deleting an Amazon S3 table in the Amazon Simple Storage Service User Guide.
Conclusion
In this post, you enabled the metadata export feature of SageMaker Catalog and used SQL queries to gain visibility into your asset inventory. The feature converts asset metadata into Apache Iceberg tables partitioned by snapshot date, so you can perform time-travel queries, monitor catalog growth, track metadata completeness, and audit historical asset states. This provides a repeatable, low-overhead way to maintain catalog health and meet governance requirements over time.
In this post, we demonstrate how Notebooks in Amazon SageMaker Unified Studio help you get to insights faster by simplifying infrastructure configuration. You’ll see how to analyze housing price data, create scalable data tables, run distributed profiling, and train machine learning (ML) models within a single notebook environment.
Data scientists and analysts often spend days configuring infrastructure and managing authentication across multiple data sources before they can begin analysis. When working with data across Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Snowflake, and local files, teams face repeated authentication setup, manual compute scaling decisions, and tool-switching overhead that delays insights.
Notebooks in Amazon SageMaker Unified Studio provide instant access to 12+ data sources, compute scaling from local to distributed processing, and AI-powered code generation within a single browser-based environment. You’ll learn to use polyglot programming, multi-engine compute, and AI-assisted development to accelerate your path from question to insight.
What are Notebooks in Amazon SageMaker Unified Studio?
Notebooks in Amazon SageMaker Unified Studio provide an interactive environment for data analysis, exploration, engineering, and machine learning workflows. It delivers five integrated capabilities:
Polyglot programming: Write code in Python and SQL interchangeably within the same notebook environment
Unified data access: Connect instantly to data stored in Amazon S3, AWS Glue Data Catalog, Apache Iceberg tables, and third-party sources like Snowflake and BigQuery
Native visualization: Create charts directly from Python and SQL results for immersive data analytics
AI-powered development: Generate code through natural language prompts using SageMaker Data Agent, with an intelligent chat interface for data analytics, data science, and ML tasks
Flexible compute: Scale from basic instances to GPU-powered environments as your needs grow
Architecture
This section covers the architecture of Notebooks, which delivers enterprise-scale analytics with browser-based simplicity through a cloud-native architecture that integrates multiple compute engines, diverse data sources, and AI-powered assistance.
Presentation layer
You access the notebook interface through Amazon SageMaker Unified Studio, interacting with a familiar interface featuring code cells for execution, markdown cells for documentation, and visualization cells for charts and tables.
Compute layer
A dedicated notebook server manages your kernel lifecycle and session state. Key components include a Language Server for code completion, a Python 3.11 runtime with pre-loaded data science libraries, and a Polyglot Kernel that handles your Python, PySpark, and SQL execution within the same notebook. Persistent Amazon Elastic Block Store (Amazon EBS) storage backs each notebook you create.
Execution layer
Notebooks support multiple execution engines, automatically routing your code to the optimal processing engine. In-memory execution handles your smaller datasets and rapid prototyping. Apache Spark via Amazon Athena provides distributed processing for your large-scale analytics via Spark Connect. Native connectivity to Amazon Athena (Trino), Amazon Redshift, Snowflake, and BigQuery processes your SQL queries.
Data Integration
You get unified access to 12+ data sources including AWS-native (Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift) and third-party (Snowflake, BigQuery, PostgreSQL, MySQL) data sources. For the latest supported data sources, see Connect to data sources .
Security is embedded throughout the architecture to protect your work. Data access respects your AWS Identity and Access Management (AWS IAM) permissions. The notebook and the agent can only access data sources you’re authorized to use. Communication between components uses encrypted channels, and your notebook storage is encrypted at rest. The AI agent includes built-in guardrails to help prevent destructive operations and logs interactions for your compliance and auditing purposes.
Prerequisites
Before you begin, you need:
An AWS account with appropriate permissions to create Amazon SageMaker Unified Studio resources. See Set up IAM-based domains for complete permission requirements.
Basic familiarity with Python programming and SQL queries
Understanding of data analysis concepts and ML workflows
Access to the sample housing dataset (provided in the walkthrough)
You will be prompted either to select an existing AWS Identity and Access Management (AWS IAM) role that has access to your data and compute, or to create a new role. For this walkthrough, choose Create a new role and leave the other options at their defaults.
Choose Set up. It takes a few minutes to complete your environment.
Use case
In this post, you’ll use a Notebook and the SageMaker Data Agent to perform the following:
Working with dataset: Upload sample dataset housing.csv and explore with data explorer
Polyglot programming: Query dataframes with SQL via DuckDB
Multi-engine access via AWS Glue: Create an AWS Glue table to unlock Athena SQL/Spark engines for distributed processing
Advanced analytics: Use Athena Spark for data profiling
AI-assisted development: Generate profiling and ML code with Data Agent
ML workflow: Train Random Forest model and evaluate results
First, let’s walk through the interface and explore its core capabilities.
Understanding the interface
The Notebooks interface follows familiar notebook conventions with cells for code execution and markdown for documentation. Within the notebook, you’ll see your current programming environment (such as Python 3.11) and compute profile specifications. The interface allows you to:
Access your data by browsing files, exploring data catalogs, and managing third-party connections
Monitor variables created within your notebook context
Scale compute resources on demand by adjusting virtual CPUs and RAM based on your workload requirements, even scaling up to GPU instances
Manage packages by installing and configuring Python packages as needed
Working with the dataset
For this walkthrough, you’ll use the housing.csv sample dataset which you can download from this page. (the file is named canvas-sample-housing.csv on the linked page). Choose the Files icon in the left panel and choose the Local tab. Upload the CSV file to the notebook on the Local tab.
Notebooks provide you with instant access to your data assets. Using the data explorer, you can browse your AWS Glue Data Catalog, Amazon S3 table catalogs, Amazon S3 buckets, and configured third-party connections.
Choose the three-dot options menu.
Choose Read as dataframe, then run the inserted cell in the notebook to view the results.
import pandas as pd
<<df_csv_xxxx>> = pd.read_csv('housing.csv')
<<df_csv_xxxx>>
When you return a dataframe, Notebooks render it in a rich table format with automatic data profiling.
Polyglot programming: Python and SQL together
One of the most powerful features in Notebooks is the interoperability between Python and SQL. After you load data into a Python dataframe, you can immediately query it using SQL. For example, to calculate total population and household by ocean proximity, you can run:
select sum(population) ,sum(households),ocean_proximity
from<<df_csv_xxxx>>
group byocean_proximity
The notebook’s autocomplete functionality recognizes dataframes in your context, making SQL queries intuitive.
This SQL query runs on DuckDB (an in-memory SQL database engine), which requires no separate installation or server maintenance on your part. DuckDB’s lightweight design integrates into Python, Java, and other environments, making it ideal for your rapid interactive data analysis. For distributed processing needs, you can use engines such as Apache Spark or Trino after creating an AWS Glue table for this dataset.
Create an AWS Glue table for the dataset
After you create an AWS Glue table, you can query the dataset using various AWS Glue catalog-compatible engines, including Amazon Athena SQL (Trino) and Amazon Athena Spark. These engines deliver optimal price-performance for your specific workload requirements.
Start by creating an AWS Glue database. To do that, create a new cell in the notebook by choosing SQL and selecting Amazon Athena (SQL).
Run this SQL to create a database: create database demo;
Next, go to data explorer and choose +Add on the top left, then choose Create table. Choose the database you created earlier and enter a name for the table. Upload the housing.csv dataset file used earlier. Continue by choosing Next in the side panel to create the table.
Next, let’s run a sample SQL query in a new cell using Amazon Athena SQL:
select sum(population) , sum(households), ocean_proximity
fromdemo.housing
group by ocean_proximity
Advanced capabilities with Athena Spark
Before you can build an ML model to predict house prices, let’s analyze the dataset further and run data profiling for additional insights. For advanced exploration, you can use Amazon Athena Spark within your notebook.To do that, you’ll create a new Python cell which has a built-in Spark session. Run the following code to check the Spark version:
# Verify Spark version
spark.version
Using the SageMaker Data Agent for data profiling
Instead of writing boilerplate code manually, you can use the built-in generative AI capability.
Prompt: “Perform data profiling and create visualization for housing table”
The AI assistant generates comprehensive profiling code for you, including basic statistics calculation, column-level profiling, data type analysis, and missing value detection.
The agent accessed your AWS Glue Data Catalog, understood your housing table structure, and generated profiling code tailored to your specific columns and data types. This context awareness reduces the trial-and-error cycle you’d normally face when adapting generic code snippets to your environment. Review the generated code and run it. The fast response times help you iterate on your analysis efficiently.
If you encounter an error, you can resolve it using Fix with AI as shown in the following figure. When errors occur during execution, the “Fix with AI” feature analyzes the traceback, diagnoses the root cause, and generates corrected code, so you can keep your analysis moving forward.
Training ML models
Next, you’ll use the data agent to generate code for training a model that predicts housing prices.
Prompt: “Generate code to train a model that predicts housing prices. Use table housing.”
The AI assistant generates end-to-end code for you that:
Reads housing data from AWS Glue catalog using Amazon Athena Spark and converts to pandas
Converts string columns to numeric, encodes using one-hot encoding and removes missing values
Trains a Random Forest model to predict median house values
Evaluates model performance (RMSE, MAE, R-square)
Displays top 10 most important features for predictions
This multi-step orchestration saves you hours of development time by handling the entire workflow from data access to model evaluation.
If you encounter an error, you can resolve it using Fix with AI available in the results traceback section.
This workflow showcased Notebooks’ unified capabilities: you uploaded files locally, created AWS Glue tables for multi-engine access, used Amazon Athena Spark for distributed profiling, and used AI-assisted ML development to predict housing prices. All of this happened within a single notebook environment without switching tools.
Key benefits and best practices
Notebooks in Amazon SageMaker Unified Studio deliver several advantages:
Faster time to insights: With traditional environments, you might spend hours on configuration before analysis begins. Notebooks bypass this overhead, so you can start work immediately.
Improved collaboration: You can share notebooks with consistent environments, supporting reproducibility and reducing “works on my machine” issues.
Reduced complexity: You can access multiple data sources and compute engines from one interface rather than navigating separate tools for each data source or processing engine.
AI-accelerated development: Generate task-specific code and receive intelligent suggestions, reducing time spent on repetitive coding tasks.
Scalable performance: Handle datasets from megabytes to petabytes with appropriate compute resources. The system scales automatically as data volumes grow.
Best practices
Start with appropriate compute profiles by beginning with smaller instances and scaling up as your needs grow.
Use AI assistance with natural language prompts for your repetitive tasks and complex operations.
Combine engines strategically by using Amazon Athena Spark for your large-scale processing, Amazon Redshift for data warehousing and other specialized engines for your specific workloads.
Document your work using markdown cells to create living documentation alongside your code.
Organize using multiple cells by breaking the complex workflows into logical steps for better readability and debugging.
Cleaning up
To avoid incurring future charges, delete the resources you created in this walkthrough:
In the Amazon SageMaker Unified Studio console, navigate to the Notebook page
Delete the notebook
Delete the demo database and housing table from the AWS Glue Data Catalog
Delete Amazon SageMaker Unified Studio domain created during this walkthrough
If you created a new IAM role specifically for this walkthrough, delete it from the IAM console
Conclusion
In this post, we demonstrated how Notebooks in Amazon SageMaker Unified Studio help you work more efficiently and deliver insights more quickly. By combining familiar notebook interfaces with enterprise-scale compute, multi-engine support, and generative AI assistance, teams can streamline data and AI workflows.
The integration of Python and SQL, instant access to diverse data sources, and intelligent code generation capabilities make Notebooks a valuable tool for modern data teams. Teams can perform exploratory data analysis, build complex data pipelines, or train ML models with the flexibility and power needed within a single, intuitive environment.
Amazon Athena recently added support for reading Parquet Column Indexes in Apache Iceberg tables on November 21, 2025. With this optimization, Athena can perform page-level data pruning to skip unnecessary data within Parquet row groups, potentially reducing the amount of data scanned and improving query runtime for queries with selective filters. For data teams, this may help enable faster insights and help reduce costs when analyzing large-scale data lakes.
Data teams building data lakes often choose Apache Iceberg for its ACID transactions, schema evolution, and metadata management capabilities. Athena is a serverless query engine that allows you to query Amazon S3-based data lakes using SQL, and you don’t need to manage infrastructure. Based on the type of data and query logic, Athena can apply multiple query optimizations to improve performance and reduce costs.
In this blog post, we use Athena and Amazon SageMaker Unified Studio to explore Parquet Column Indexes and demonstrate how they can improve Iceberg query performance. We explain what Parquet Column Indexes are, demonstrate their performance benefits, and show you how to use them in your applications.
Overview of Parquet Column Indexes
Parquet Column Indexes store metadata that query engines can use to skip irrelevant data with greater precision than row group statistics alone. To understand how they work, consider how data is structured within Parquet files and how engines like Athena process them.
Parquet files organize data hierarchically by dividing data into row groups (typically 128-512 MB each) and further subdividing them into pages (typically 1 MB each). Traditionally, Parquet maintains metadata on the contents of each row group level in the form of min/max statistics, allowing engines like Athena to skip row groups that don’t satisfy query predicates. Although this approach reduces the bytes scanned and query runtime, it has limitations. If even a single page within a row group overlaps with the values you are searching for, Athena scans all pages within the row group.
Parquet Column Indexes help address this problem by storing page-level min/max statistics in the Parquet file footer. Row group statistics provide coarse-grained filtering, but Parquet Column Indexes enable finer-grained filtering by allowing query engines like Athena to skip individual pages within a row group. Consider a Parquet file with a single row group containing 5 pages for a column. The row group has min/max statistics of (1, 20), and each page for that column has the following min/max statistics.
When Athena runs a query filtering for values equal to 2, it first checks the row group statistics and confirms that 2 falls within the range (1, 20). Athena will then plan to scan the pages within that row group. Without Parquet Column Indexes, Athena scans each of the 5 pages in the row group. With Parquet Column Indexes, Athena examines the page-level statistics and determines that only page-0 and page-1 need to be read, skipping the remaining 3 pages.
How to use Parquet Column Indexes with Athena
Athena uses Parquet Column Indexes based on table type:
Amazon S3 Tables: Athena automatically uses Parquet Column Indexes by default when they are present.
Iceberg tables in S3 general purpose buckets: Athena does not use Parquet Column Indexes by default. To allow Athena to use Parquet Column Indexes, add an AWS Glue table property named use_iceberg_parquet_column_index and set it to true. Use the AWS Glue console or AWS Glue UpdateTable API to perform these actions.
Measuring Athena performance gains when using Parquet Column Indexes
Now that we understand what Parquet Column Indexes are, we’ll demonstrate the performance benefits of using Parquet Column Indexes by analyzing the catalog_sales table from a 3TB TPC-DS dataset. This table contains ecommerce transaction data including order dates, sales amounts, customer IDs, and product information. This dataset is a good proxy for the types of business analysis that you might perform on your own data, such as identifying sales trends, analyzing customer purchasing patterns, and calculating revenue metrics. We compare query execution statistics with and without Parquet Column Indexes to quantify the performance improvement.
Prerequisites
Before you begin, you must have the following resources:
A SageMaker Unified Studio IAM-based domain.
An Execution IAM Role configured within the SageMaker Unified Studio IAM-based domain with access to S3, AWS Glue Data Catalog, and Athena.
An S3 bucket in your account to store Iceberg table data and Athena query results.
Create catalog_sales Iceberg table
Complete the following steps using SageMaker Unified Studio notebooks. There, you can use SageMaker Unified Studio’s multi-dialect notebook functionality to work with your data using the Athena SQL and Spark engines. To create a catalog_sales Iceberg table in your account, follow these steps:
Navigate to Amazon SageMaker in the AWS Management Console and choose Open under Get started with Amazon SageMaker Unified Studio.
From the side navigation, select Notebooks and choose Create Notebook. The subsequent steps in this post will execute scripts in this notebook.
Create a new SQL cell in the notebook and set the connection type to Athena (Spark). Execute the following query to create a database for the tables in this post.
CREATE DATABASE parquet_column_index_blog;
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the following query to create a Hive table pointing to the location of the TPC-DS catalog_sales table data at the public S3 bucket.
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the following query to add the Hive partitions to the AWS Glue metadata.
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Replace s3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/catalog_sales/ with the S3 URI where you want to store your Iceberg table data, then execute the following query to create the catalog_sales Iceberg table from the Hive table.
CREATE TABLE parquet_column_index_blog.catalog_sales
USING iceberg
PARTITIONED BY (cs_sold_date_sk)
LOCATION 's3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/catalog_sales/'
AS
SELECT * FROM parquet_column_index_blog.catalog_sales_hive;
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the following query to delete the catalog_sales_hive table, which was only needed to create the catalog_sales Iceberg table.
DROP TABLE parquet_column_index_blog.catalog_sales_hive;
Run an Athena query without Parquet Column Indexes
After creating the catalog_sales Iceberg table in the preceding steps, we run a simple query that analyzes shipping delays of the top 10 most ordered items. This type of analysis could be critical for ecommerce and retail operations. By identifying which popular items experience the greatest delays, fulfillment teams can focus resources where they matter most. For example, you can adjust inventory placement, change warehouse assignments, or address carrier issues. Additionally, popular items with significant shipping delays are more likely to result in order cancellations or returns, so proactively identifying these issues helps protect revenue.
SELECT cs_item_sk,
SUM(cs_quantity) as total_orders,
AVG(cs_ship_date_sk - cs_sold_date_sk) as avg_ship_delay_days,
MIN(cs_ship_date_sk - cs_sold_date_sk) as min_ship_delay,
MAX(cs_ship_date_sk - cs_sold_date_sk) as max_ship_delay,
SUM(
CASE
WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
END
) as late_shipments,
SUM(
CASE
WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
END
) * 100.0 / COUNT(*) as late_shipment_pct,
AVG(cs_ext_ship_cost) as avg_shipping_cost
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IN (
SELECT cs_item_sk
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY SUM(cs_quantity) DESC
LIMIT 10
)
AND cs_ship_date_sk IS NOT NULL
AND cs_sold_date_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY avg_ship_delay_days DESC;
Additionally, this query is a good candidate for demonstrating the effectiveness of using Parquet Column Indexes because it has a selective filter predicate on a single column cs_item_sk. When Athena executes this query, it first identifies row groups whose min/max ranges overlap with the top 10 most ordered items. Without using Parquet Column Indexes, Athena has to scan every page of data within those matched row groups. However, when using Parquet Column Indexes, Athena can prune data further by skipping individual pages within those row groups whose min/max ranges do not overlap with the ids. Complete the following steps to establish baseline query performance when Athena does not use Parquet Column Indexes during the query.
Create a new Python cell in the notebook. Replace s3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/query_results/ with the S3 URI where you want to store your Athena query results, then execute the following script. Note the runtime and bytes scanned that will be printed. The script will run the query five times with query result reuse disabled and chooses the minimum runtime and the corresponding bytes scanned among those iterations. See our numbers in the Run Athena query with Parquet Column Indexes section.
import boto3
import time
# Configuration
DATABASE = "parquet_column_index_blog"
OUTPUT_LOCATION = "s3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/query_results/"
def run_athena_query(query: str, database: str, output_location: str):
athena_client = boto3.client('athena')
response = athena_client.start_query_execution(
QueryString=query,
QueryExecutionContext={'Database': database},
ResultConfiguration={'OutputLocation': output_location}
)
query_execution_id = response['QueryExecutionId']
while True:
result = athena_client.get_query_execution(QueryExecutionId=query_execution_id)
state = result['QueryExecution']['Status']['State']
if state in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
break
time.sleep(5)
if state != 'SUCCEEDED':
raise Exception(f"Query failed with state: {state}")
stats = result['QueryExecution']['Statistics']
return {
'execution_time_sec': stats['EngineExecutionTimeInMillis'] / 1000,
'data_scanned_gb': stats['DataScannedInBytes'] / (1024 ** 3)
}
def benchmark_query(query: str, database: str, output_location: str, num_runs: int = 5):
results = []
for i in range(num_runs):
stats = run_athena_query(query, database, output_location)
results.append(stats)
best_run = min(results, key=lambda r: r['execution_time_sec'])
execution_time = round(best_run['execution_time_sec'], 1)
data_scanned = round(best_run['data_scanned_gb'], 1)
print(f"Execution time: {execution_time} sec")
print(f"Data scanned: {data_scanned} GB")
QUERY = """
SELECT cs_item_sk,
SUM(cs_quantity) as total_orders,
AVG(cs_ship_date_sk - cs_sold_date_sk) as avg_ship_delay_days,
MIN(cs_ship_date_sk - cs_sold_date_sk) as min_ship_delay,
MAX(cs_ship_date_sk - cs_sold_date_sk) as max_ship_delay,
SUM(
CASE
WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
END
) as late_shipments,
SUM(
CASE
WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
END
) * 100.0 / COUNT(*) as late_shipment_pct,
AVG(cs_ext_ship_cost) as avg_shipping_cost
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IN (
SELECT cs_item_sk
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY SUM(cs_quantity) DESC
LIMIT 10
)
AND cs_ship_date_sk IS NOT NULL
AND cs_sold_date_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY avg_ship_delay_days DESC;
"""
# Run benchmark
benchmark_query(QUERY, DATABASE, OUTPUT_LOCATION, num_runs=5)
Sort the catalog_sales table
Before rerunning the query with Athena using Parquet Column Indexes, you need to sort the catalog_sales table by the cs_item_sk column. In the preceding query, there is a dynamic filter as a subquery on the cs_item_sk column:
cs_item_sk IN (
SELECT cs_item_sk
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY SUM(cs_quantity) DESC
LIMIT 10
)
When executing this query, Athena pushes down the filter predicate to the data source level, fetching only rows that match the top 10 most ordered items. To maximize page pruning with Parquet Column Indexes, rows with the same cs_item_sk values should be stored near each other in the Parquet file. Without sorting, matching values could be scattered across many pages, forcing Athena to read more data. Sorting the table by cs_item_sk clusters similar values together, enabling Athena to read fewer pages.
Let’s examine the Parquet Column Indexes in one of the Parquet files to understand how the data in the catalog_sales table is currently organized. First, download the Parquet file from the cs_sold_date_sk = 2450815 partition and install the open-source parquet-cli tool on your local machine. Replace <local-path-to-parquet-file> with the path to the downloaded Parquet file, then run the following command on your local machine:
parquet column-index <local-path-to-parquet-file>
This displays Parquet Column Indexes for all columns. For brevity, only the first 11 pages of the cs_item_sk column from the first row group are shown in the following example:
Notice that nearly every page contains a wide range of values. This overlap means Athena cannot eliminate pages when filtering with Parquet Column Indexes on cs_item_sk. For example, searching for cs_item_sk = 100 requires scanning each of the 11 pages because the value 100 falls within every page’s min/max range. With this overlap, enabling Athena to use Parquet Column Indexes would provide no performance benefit. Sorting the data by cs_item_sk eliminates this overlap, creating distinct, non-overlapping ranges for each page. To make Parquet Column Indexes more effective, sort the table by completing the following step:
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the query to sort the cs_item_sk column values of the catalog_sales table in ascending order and to put all the null values in the last few Parquet pages. New Iceberg data files will be generated from this query.
Running the parquet column-index command on the sorted data file from the cs_sold_date_sk = 2450815 partition shows that the Parquet Column Indexes are now sorted and have non-overlapping ranges. The first 11 pages of the cs_item_sk column from the first row group are shown in the following example:
Now when searching for cs_item_sk = 100, Athena only needs to read page-0, skipping the remaining 10 pages entirely.
Run Athena query with Parquet Column Indexes
Now that the data is sorted to eliminate overlapping pages within the row groups for the cs_item_sk column, we run two experiments on the sorted data. The first measures the impact of sorting alone, and the second measures the combined effect of sorting with Parquet Column Indexes.
Create a new Python cell in the notebook. Execute the same script in the section Run Athena query without Parquet Column Indexes and take note of the query runtime and bytes scanned results. This measures the performance of querying sorted data without using Parquet Column Indexes.
Create a new Python cell in the notebook. Execute the following Python script to set the use_iceberg_parquet_column_index table property to true for the catalog_sales table in the AWS Glue Data Catalog.
import boto3
def add_iceberg_parquet_column_index(database_name: str, table_name: str):
glue_client = boto3.client('glue')
# Get current table definition
response = glue_client.get_table(DatabaseName=database_name, Name=table_name)
table = response['Table']
# Build TableInput with only allowed fields
table_input = {'Name': table['Name']}
allowed_fields = [
'Description', 'Owner', 'LastAccessTime', 'LastAnalyzedTime',
'Retention', 'StorageDescriptor', 'PartitionKeys', 'ViewOriginalText',
'ViewExpandedText', 'TableType', 'Parameters', 'TargetTable'
]
for field in allowed_fields:
if field in table:
table_input[field] = table[field]
# Add the property
if 'Parameters' not in table_input:
table_input['Parameters'] = {}
table_input['Parameters']['use_iceberg_parquet_column_index'] = 'true'
# Update the table
glue_client.update_table(DatabaseName=database_name, TableInput=table_input)
# Usage
add_iceberg_parquet_column_index("parquet_column_index_blog", "catalog_sales")
Create a new Python cell in the notebook. Execute the same script in the section Run Athena query without Parquet Column Indexes and take note of the query runtime and bytes scanned results. This measures the performance of querying sorted data using Parquet Column Indexes.
Athena query time and bytes scanned improvement
The following table summarizes the results from each experiment. The percentage improvements for the sorted experiments are measured against the unsorted baseline.
Experiment
Runtime (sec)
Bytes Scanned (GB)
Unsorted without Parquet Column Indexes
20.6
45.2
Sorted without Parquet Column Indexes
15.4 (25.2% faster)
27.8 (38.5% fewer bytes)
Sorted with Parquet Column Indexes
10.3 (50.0% faster)
13.0 (71.2% fewer bytes)
Recommendations
To maximize Athena’s ability to use Parquet Column Indexes and achieve optimal query performance, we recommend the following.
Sort data by frequently filtered columns. This allows Athena to efficiently read Parquet Column Indexes and skip irrelevant pages, potentially reducing scan time. When data is sorted by a filter column, similar values are clustered together within pages. Because Parquet Column Indexes store min/max values for each page, Athena can quickly determine which pages contain matching values and skip the rest.
Sort data by high-cardinality columns. This creates distinct value ranges between pages, maximizing the opportunity for Athena to skip pages during query execution. High-cardinality (many distinct values) columns produce non-overlapping min/max ranges across pages, allowing Athena to more effectively filter out irrelevant pages. In contrast, low-cardinality columns such as boolean or status fields result in overlapping ranges across many pages, reducing the number of skipped pages.
Clean up
When you have finished the steps in this post, complete the following cleanup actions to avoid incurring ongoing charges:
Create a new SQL cell in the notebook and set the connection type to Athena (Spark). Execute the following command to drop the parquet_column_index_blog database and the catalog_sales table.
DROP DATABASE parquet_column_index_blog CASCADE;
Delete the Iceberg table data and the Athena query results from your S3 bucket.
Delete the SageMaker Unified Studio IAM-based domain if it is no longer needed.
Conclusion
In this post, we showed you how Athena uses Parquet Column Indexes to speed up queries and reduce the number of bytes scanned. By using Parquet Column Indexes, Athena can skip irrelevant data pages to improve query performance, especially for queries with selective filters on sorted data. Refer to Optimize Iceberg tables to learn more about this feature and try it out on your own queries.
Organizations are finding significant value using an integrated experience for all your data and AI with Amazon SageMaker Unified Studio. However, many organizations require strict network control to meet security and regulatory compliance requirements like HIPAA or FedRAMP for their data and AI initiatives, while maintaining operational efficiency.
In this post, we explore scenarios where customers need more control over their network infrastructure when building their unified data and analytics strategic layer. We’ll show how you can bring your own Amazon Virtual Private Cloud (Amazon VPC) and set up Amazon SageMaker Unified Studio for strict network control.
Solution overview
The solution covers complete technical know-how of a fully private network architecture using Amazon VPC with no public internet exposure. The approach leverages AWS PrivateLink through VPC endpoints to provide a secure communication between SageMaker Unified Studio and essential AWS services entirely over the AWS backbone network.
The architecture consists of three core components: a custom VPC named airgapped with multiple private subnets distributed across at least three Availability Zones for high availability, a comprehensive set of VPC interface and gateway endpoints for service connectivity, and the SageMaker Unified Studio domain configured to operate exclusively within this isolated environment. This design helps ensure that sensitive data never traverses the public internet while maintaining full functionality for data cataloging, query execution, and machine learning workflows.
By implementing this air-gapped configuration, organizations gain granular control over network traffic, simplified compliance auditing, and the ability to integrate SageMaker Unified Studio with existing private data sources through controlled network pathways. The solution supports both immediate operational needs and long-term scalability through careful IP address planning and modular endpoint architecture.
Prerequisites
The set up requires you to have an existing VPC (for this post, we’ll refer to the name as airgapped but in reality, it refers to the VPC you would like to securely set up SageMaker Unified Studio). If you don’t have an existing VPC, you can follow SageMaker Unified Studio domain quick create administrator guide to get started.
The high level steps to create a VPC meeting minimum requirements for SageMaker Unified Studio are as follows:
Once the domain is created, choose Open unified studio or use SageMaker Unified Studio URL under Domain details to access SageMaker Unified Studio.
Figure 2 – Amazon SageMaker Unified Studio URL Welcome Page
After logging in to SageMaker Unified Studio, create a project using the guided wizard.
Once the project is created, we need to add the necessary VPC endpoints to allow traffic from the project to communicate to AWS services.
S3 Gateway VPC endpoint was already selected as part of VPC creation step 5 in prerequisites and thus created by default. Now we must add two more VPC endpoints for Amazon DataZone and AWS Security Token Service as illustrated in following step.
These are the minimum set of VPC endpoints to allow using the tooling within SageMaker Unified Studio. For a list of other mandatory and non-mandatory VPC endpoints refer to the tables in the latter part of this post.
Create an interface endpoint
To create an interface endpoint, complete following steps:
Go to the SageMaker Unified Studio Project details page and copy the Project ID. Figure 3 – SageMaker Unifed Studio Project Details Page
Go to the VPC console and choose Endpoints.
Choose Create Endpoint.
Enter a name for the endpoint, for example, DataZone endpoint for SageMaker Unified Studio.
For AWS Services, enter DataZone.
Figure 4 – Interface Endpoint creation wizard for AWS Service datazone
Select Service Name = com.amazonaws.us-east-1.datazone from the available options.
Select the subnets in the airgapped-vpc that you created earlier.
Filter the Security Groups by pasting the copied Project ID.
Select the security group with Group Namedatazone-<project-id>-dev.
Choose Create Endpoint.
Repeat the same steps to create a VPC endpoint for AWS STS.
Once the VPC endpoints are created, validate connectivity in the SageMaker project by running a SQL query or using a Jupyterlab notebook.
For a successful domain and project which does not get into any service level usage, the mandatory VPC endpoints to be created are: S3 Gateway, DataZone, and STS interface endpoints. For other service usage dependent operations like authentication, data preview and working with compute, you would require other mandatory service specific endpoints explained later in this post.
Best practices for VPC set up for various use cases
When setting up SageMaker Unified Studio domain and project profiles, you need to specify the VPC network, subnets, and security groups. Here are some best practices around IP allocation, usage volume and expected growth to consider for different use cases within enterprises.
Production and enterprise use cases
If your organization require strict network control to meet security and compliance requirements for data and AI initiatives, consider following best practices in your production environment.
Use the bring-your-own (BYO) VPC approach to comply with company-specific networking and security requirements.
Implement private networking using VPC endpoints to keep traffic within the AWS backbone.
Use at least two private subnets across different Availability Zones.
Enable DNS hostnames and DNS Support.
Disable auto-assign public IP on subnets.
Plan IP capacity for at least 5 years. A prescriptive guidance for SageMaker Unified Studio is shared in VPC and Networking details section later in this post. Consider the following:
Number of users
Number of apps per user
Number of unique instance types per user
Average number of training instances
Expected growth percentage
Testing and non-production use cases
For development, testing, non-prod environment where use cases don’t have stringent security and compliance requirements, use automated setup for quick experiments. Use sample CloudFormation github templates as part of the SageMaker Unified Studio express set up, to automate domain and project creation. However, this includes an Internet Gateway which may not be suitable for security-sensitive environments.
Private networking use cases
VPCs with private subnets require essential service endpoints to allow client resources like Amazon EC2 instances to securely access AWS services. The traffic between your VPC and AWS services remains within AWS network avoiding public internet exposure.
Implement all mandatory VPC endpoints for core services (SageMaker, DataZone, Glue, and more).
Add optional endpoints based on specific service needs, like IPv4 endpoints, dual-stack endpoints, and FIPS endpoints to programmatically connect to an AWS service.
Work with network administrators for:
Preinstalling needed resources through secure channels like private subnets and self-referencing inbound rules in security groups to enable limited access.
Allowlisting only necessary external connections like NAT gateway IP and bastion host access in firewall rules.
Setting up appropriate proxy configurations if required.
External data source access use cases
Consider the following when working with external systems like third-party SaaS platforms, on-premises databases, partner APIs, legacy systems, or external vendors.
Consult with network administrators for appropriate connection methods.
Consider AWS PrivateLink integration where available.
Implement appropriate security measures for non-AWS data your source documents.
For High Availability:
Deploy across at least three different Availability Zones (at least two for AWS Regions with only two AZs).
Verify there’s a minimum of three free IPs per subnet.
Consider larger CIDR blocks (/16 recommended) for future scalability.
VPC and networking details
In this section, we provide details of each networking aspect starting with choice of VPCs, network connectivity details for integrated services to work, the basis of VPC and subnet requirements, and finally the VPC endpoints required for private service access.
VPC
At a high level, you have two options to supply VPCs and subnets:
Bring-your-own (BYO) VPC. This is typically the case for most customers, as most have company specific networking and security requirements to reuse an existing VPC, or to create a VPC that are compliant with those requirements.
Create VPC with the SageMaker quick set up template. When creating a SageMaker Unified Studio domain (DataZone V2 domain in CloudFormation) through the automated quick set up, you will be shown a Quick create stack wizard in CloudFormation which creates VPCs and subnets used to configure your domain.
Note: The quick create stack using template URL is not intended for production use. The template creates an Internet Gateway, which is not allowed in many enterprise settings. This is only appropriate if you are either trying out SageMaker Unified Studio or, running SageMaker Unified Studio for use cases that don’t have stringent security requirements.If you choose this option, you start with SageMaker console, navigate to domains and click Create domain button, followed by Create VPC button. You will navigate to CloudFormation and click on Create stack button to create a sample VPC named SageMakerUnifiedStudio-VPC with just one-click for trying out SageMaker Unified Studio.
Figure 6 – Create VPC button in SageMaker Unified Studio Create Domain Wizard
Cost estimation for recommended VPC set up
The exact cost depends on the configuration of your VPC. For more complex networking set ups (multi-VPC), you may need to use additional networking components such as a Transit Gateway, Network Firewall, and VPC Lattice. These components may incur charges, and cost depends on usage and AWS Region. Interface VPC endpoints are charged per availability zone. They also have a fixed and a variable component in the pricing structure. Use the AWS Pricing Calculator for a detailed estimate.
Network Connectivity
With regards to connectivity to the underlying AWS services integrated within SageMaker Unified Studio, there are two ways to enable connectivity (these are not Studio specific, these are standard ways to enable network connectivity within a VPC). This is an important security consideration that depends on your organization’s security policies.
Through the public Internet. Your traffic will traverse over the public Internet through an Internet Gateway in your VPC.
Your public subnet must have a NAT Gateway. In addition, your public subnet’s route table must have a default route (0.0.0.0 for IPv4) to the Internet Gateway. This route is what makes the subnet public.
Your private subnets must have a default route to the public subnet’s NAT Gateway.
Through the AWS backbone. Your traffic will remain within the private AWS backbone through PrivateLink (by provisioning Interface and Gateway endpoints for the necessary AWS services in each Availability Zone).
A list of all the AWS services integrated into Studio and the VPC endpoints required can be found in section VPC Endpoints covered later in this post.
For non-AWS resources, certain external providers of these services may offer PrivateLink integration. Check with each provider’s documentation and your network administrator to understand the most suitable way to connect to these external providers.
In a private networking scenario, you will need to consider whether you need connectivity to non-AWS resources in a way that’s compliant with your organization’s security policies. A few examples include the following:
If you need to download software in your remote IDE host (for example, command line programs, such as Ping and Traceroute)
If you have code that connects to external APIs.
If you use software (such as JupyterLab or Code Editor extensions) that rely on external APIs.
If you depend on software dependencies hosted in the public domain (such as Maven, PyPi, npm)
If you need cross-Region access to certain resources (such as access to S3 buckets in a different Region)
If you need functionality whose underlying AWS services do not have VPC endpoints in all Regions or any Region.
If you need to connect to data sources outside of AWS (such as Snowflake, Microsoft SQL Server, Google BigQuery) Enterprise network administrators must also complete either of the following prerequisites to handle private networking scenarios:
Preinstall needed resources through secure channels if possible. An example would be to customize your SageMaker AI image by installing dependencies, after they are code scanned, vetted technically and legally by your organization.
If AWS PrivateLink integration is not available for external providers, allowlist network connections to these external sources. Allow firewall egress rules, directly or indirectly, through a proxy in your organization’s network. Check with your network administrator to understand the most appropriate option for your organization.
VPC Requirements
When setting up a new SageMaker Unified Studio Domain, it’s necessary to supply a VPC. It’s important to note that these VPC requirements are a union of all the requirements from the respective compute services integrated into Studio, some of which are reinforced by validation checks during the corresponding blueprint’s deployment. If these requirements that have validation checks are not fulfilled, the resource(s) contained in that blueprint may fail to create on project creation (on-create), or when creating the compute resource (on-demand). This section will present a summary of these requirements, as well as relevant documentation links from which they originate.
Subnet requirements for specific compute in a VPC
This section lists the compute services integrated in SageMaker Unified Studio that require VPC/subnets when provisioning the respective compute resources.
Number of subnets: At least two private subnets. This requirement comes from Redshift Serverless.
Availability zones (AZs): At least two different AZs (for Regions with two AZs, two subnets are sufficient). This requirement comes from Redshift Serverless. For workgroups with Enhanced VPC Routing (EVR), you need three AZs.
Free IPs per subnet: At least three Ips per subnet. This requirement comes from Redshift Serverless without EVR. For detailed IP addresses requirement with EVR enabled workgroups, refer to Serverless usage considerations. Three is a minimum and may not be enough for your needs. For example, EMR cluster creation will fail if no subnets with enough IPs are found in the VPC. We recommend doing a forward-looking capacity planning exercise based on your use cases (for example, growth rate, users, compute needs) to project at least 5 years into the future. This helps to determine how many IPs are needed by the team using Studio and other services that use this VPC and come up with a ceiling for the CIDR block size.
Private or public subnets: We enforce that at least three private subnets be supplied, and recommend that only private subnets are chosen, with a few nuances. This requirement comes from SageMaker AI domain. A new SageMaker AI domain, when set up with VpcOnly mode, requires that all subnets in the VPC be private. This is the default networking mode in the Tooling blueprint. If you choose to use PublicInternetOnly mode, this restriction does not apply, you may choose public subnets from your VPC. To change the mode, modify the Tooling Blueprint parameter sagemakerDomainNetworkType.
Enable DNS hostname and DNS Support: Both must be enabled. This requirement comes from EMR. Without these VPC settings, enableDnsHostname and enableDnsSupport, connecting to the EMR Cluster using the private DNS name through the Livy Endpoint will fail. SSL Verification, which can only be done when connecting using the DNS name, not the IP.
Auto assign public IP: Disable. We recommend that this EC2 subnet setting (mapPublicIpOnLaunch) be disabled when using private subnets, because public IPs come at a cost and are a scarce resource in the total addressable IPv4 space.
VPC endpoints
If you choose to run SageMaker Unified Studio without public internet access, VPC endpoints are required for all services SageMaker Unified Studio needs to access. These endpoints provide secure, private connectivity between your VPC and AWS services without traversing the public internet. The following table lists the required endpoints, their types, and what each is used for.
Some endpoints may not show up directly in your browser’s network tab. The reason is that some of these services (such as CloudWatch) are transitively invoked by other services.
Mandatory endpoints
The following are required endpoints for SageMaker Unified Studio and supporting services to function properly. Gateway endpoints can be used where available, you can use interface endpoints for all other AWS services.
AWS service
Endpoint
Type
Purpose
Glue
com.amazonaws.${region}.glue
Interface
For Data Catalog and metadata management
STS
com.amazonaws.${region}.sts
Interface
Required for assuming IAM roles
S3
com.amazonaws.${region}.s3
Gateway
Required for datasets, Git backups, notebooks, and Git sync
SageMaker
com.amazonaws.${region}.sagemaker.api
Interface
Required for calling SageMaker APIs
com.amazonaws.${region}.sagemaker.runtime
Interface
For invoking deployed inference endpoints
DataZone
com.amazonaws.${region}.datazone
Interface
For data catalog and governance
Secrets Manager
com.amazonaws.${region}.secretsmanager
Interface
To securely access secrets
SSM
com.amazonaws.${region}.ssm
Interface
For secure command execution
com.amazonaws.${region}.ssmmessages
Interface
Enables live SSM sessions
KMS
com.amazonaws.${region}.kms
Interface
For decrypting data (volumes, S3, secrets)
EC2
com.amazonaws.${region}.ec2
Interface
For subnet and ENI management
com.amazonaws.${region}.ec2messages
Interface
Required for SSM messaging
Athena
com.amazonaws.${region}.athena
Interface
Required to run SQL queries
Amazon Q
com.amazonaws.${region}.q
Interface
Used by SageMaker Notebooks for enhanced productivity
Optional Endpoints
Only create these if the corresponding service is used in your environment.
AWS resources provisioned in your AWS accounts may incur costs based on the resources consumed. Make sure you do not leave any unintended resources provisioned. If you created a VPC and subsequent resources as part of this post, make sure you delete them.
The following service resources provisioned during this blog post need to be deleted:
IAM Identity Center users and groups.
Resources provisioned within your project using tooling configuration and blueprints within your domain.
The airgapped VPC.
Conclusion
In this post, we walked through the process of using your own existing VPC when creating domains and projects in SageMaker Unified Studio. This approach benefits customers by giving them greater control over their network infrastructure while using the comprehensive data, analytics, and AI/ML capabilities of Amazon SageMaker. We also explored the critical role of VPC endpoints in this set up. You now understand when these become necessary components of your architecture, particularly in scenarios requiring enhanced security, compliance with data residency requirements, or improved network performance.
While using a custom VPC requires more initial set up than the Quick Create option, it provides the flexibility and control many organizations need for their data science and analytics workflows. This approach provides a mechanism for your SageMaker environment to integrate with your existing infrastructure and adheres to your organization’s networking policies. Custom VPC configurations are a powerful tool in your arsenal for building secure, compliant, and efficient data science environments.
Amazon SageMaker Unified Studio brings together data engineering, analytics, and machine learning (ML) workflows into a cohesive, governed environment. This unified approach reduces traditional silos between data teams and ML practitioners, so organizations can advance their AI and ML initiatives with greater collaboration and efficiency.
As enterprises begin their SageMaker Unified Studio adoption, they must determine the best practices for implementing data federation principles when using SageMaker Unified Studio across the organization. The way that you structure your SageMaker Unified Studio deployment is more than a technical decision. It directly impacts your governance framework, security posture, operational scalability, and day-to-day team collaboration.
In this post, we explore SageMaker Unified Studio multi-account deployments in depth: what they entail, why they matter, and how to implement them effectively. We examine architecture patterns, evaluate trade-offs across security boundaries, operational overhead, and team autonomy. We also provide practical guidance to help you design a deployment that balances centralized control with distributed ownership across your organization.
The multi-account challenge: why organizations struggle
If you’re working in a large enterprise, a multi-account AWS environment is often your starting position. If you’re starting from scratch, consider whether to use a single-account for all SageMaker Unified Studio components or dedicate separate accounts for governance and business units. A multi-account architecture aligns with AWS best practices and proves valuable if you have:
Distributed teams with independent operations: multi-account architecture accommodates multiple teams or business units that maintain separate operations so that each team can manage their projects autonomously within isolated environments. Each unit can deploy and manage resources independently, implement team-specific security controls, and scale infrastructure without impacting others. This is achieved through a shared, unified integrated development environment (IDE) for collaboration and standardized tools across the organization.
Compliance and data governance requirements: For regulatory mandates like GDPR, HIPAA, or data sovereignty needs, you will benefit from this setup, because sensitive data remains segregated in business-unit specific accounts. This reduces risk exposure, streamlines audits, and maintains compliance boundaries without compromising access to centralized collaboration tools.
Centralized governance: A multi-account architecture maintains visibility across all projects and business units from a single control plane. The Domain account can enforce security policies and compliance requirements across the entire organization and provide centralized monitoring, audit logging, and user access management.
Clear cost visibility and accountability: Multi-account architecture enables granular billing tracking, with each account generating separate bills that clearly attribute costs to specific teams or business units. This transparency streamlines budgeting and financial accountability, removing the complexity of cost allocation tags and manual reporting that’s typically required in single-account models where multiple teams share the same infrastructure and resources.
Overall, this approach improves efficiency, security, and scalability for you, whether you’re managing a few teams or coordinating across a larger organizational structure.
Understanding the core constructs of SageMaker Unified Studio
Before diving into multi-account strategies, it’s important to understand the foundational constructs of SageMaker Unified Studio. Each is elaborated in greater detail in the Administrator Guide.
Domain: The top-level administrative boundary where governance lives. In a multi-account setup, this is your centralized control plane for catalog, policies, and user access.
Project: A collaborative workspace for developing data, AI, and ML initiatives. In multi-account deployments, a Project’s metadata lives in the Domain account and compute and data resources deploy into associated business unit accounts. This separation is central to the pattern that we explore.
Project Profile: A template that standardizes how Projects are created. For multi-account setups, this is where administrators define which accounts and AWS Regions Projects can target.
Blueprints: Infrastructure as code (IaC) components that define what gets provisioned inside a Project. Each associated account must enable its relevant Blueprints before Projects can deploy there.
The following diagram illustrates how these key constructs interact. Within a Domain, users create Projects organized through a single Project Profile. The Project Profile defines and configures a collection of Blueprints. When a project is created, the infrastructure specified in those Blueprints is automatically provisioned and becomes available within the project workspace.
Figure 1: Amazon SageMaker Unified Studio Core Constructs
Multi-account setup in SageMaker Unified Studio
To illustrate these concepts in practice, we demonstrate with a sample enterprise organization that exemplifies enterprise environments with several AWS accounts belonging to different business units:
Central Data Governance team: Owns and manages governance and access control across the organization. They plan to build a data solution in a dedicated AWS account using SageMaker Unified Studio. The platform must provide an integrated development environment (IDE) to work with data and ML use cases and connect to multiple business unit’s AWS accounts (Finance and Marketing).
Finance Business Unit: Owns datasets for fraud analysis and churn prediction in their own AWS account.
Marketing Business Unit: Maintains customer sentiment data and campaign analytics in their own AWS account.
In the following diagram we show the Data platform constructs provided by SageMaker Unified Studio in each AWS account showing the clear separation between centralized governance and distributed resource deployment.
Figure 2: Sample organization architecture in Amazon SageMaker Unified Studio
The Central Data Governance Account contains the SageMaker Unified Studio Domain. This contains the shared platform resources (Catalog, shared infrastructure), governance constructs (Domain units, metadata forms), and governance policies (authorization policies, enforcement rules). These configuration elements define the standards and capabilities available across the organization. They’re the Service-level configuration data: Metadata, policies, and governance rules that define how resources should be provisioned.
In contrast, the Associated Accounts (Marketing and Finance) contain the actual AWS infrastructure, compute/storage (purple cubes) and data stores (cylinders), provisioned when Projects are created. The diagram shows how Marketing Projects and Finance Projects ultimately deploy their runtime resources into their respective business unit accounts. The separation keeps the governance centralized and consistent while allowing business unit dependent resources to be isolated, billed separately, and managed according to each business unit’s specific requirements.
To understand the core constructs of SageMaker Unified Studio, we listed the core components of SageMaker Unified Studio and explained how they relate to each other. Taking the same diagram as the basis, we will now represent how these constructs are created in our multi-account sample scenario.
Core construct deployment locations and resources in SageMaker Unified Studio
Implementing multi-account deployments
To enable production-ready data science and analytics workflows across multiple AWS accounts governed by a SageMaker Unified Studio Domain, organizations must establish a structured cross-account configuration. This setup allows each business unit (BU) to retain ownership of its Projects and AWS resources while using centralized governance provided by the Domain. The process involves four key steps: account association, Blueprint enablement, Project Profile configuration, and Project creation.
The Domain administrator associates each AWS account with the SageMaker Unified Studio Domain for seamless cross-account functionality by providing the AWS account number for the targeted accounts. This association lets the Domain publish and consume data from associated accounts, create resources within them, maintain cross-account access for the SageMaker Catalog, and deploy Projects directly into business unit accounts. Account association is a critical prerequisite for cross-account Project deployment. Behind the scenes, SageMaker Unified Studio uses AWS Resource Access Manager (AWS RAM) to make this cross-account functionality happen.
Step 2: Enabling blueprints
Each associate account administrator must enable the relevant Blueprints before creating Project Profiles. This important step verifies that Projects can provision the necessary tools and resources that users need to run their workloads. Blueprints serve as standardized infrastructure templates that administrators can use to enforce organizational standards, security controls, and best practices across all Projects. Through Blueprints, administrators configure essential resources including AWS Identity and Access Management (IAM) roles, AWS Key Management Service (AWS KMS) keys, Amazon Simple Storage Service (Amazon S3) buckets, Amazon Virtual Private Cloud (Amazon VPC) settings, and security groups. This centralized approach helps maintain consistency, compliance, and governance at scale while preventing users from creating Projects with misconfigured or non-compliant infrastructure.
Step 3: Configuring project profile
With the accounts successfully associated and the Blueprints enabled, the next step is to configure a Project Profile that determines where your Project resources will be deployed. Your choice of Project Profile strategy impacts both operational flexibility and governance.
Domain administrators control which Blueprints are included in each Project Profile and can specify the target AWS Regions and accounts for deployment, providing the governance foundation to standardize Project creation. Administrators can use pre-created Project Profiles like “All Capabilities” or “SQL Analytics”, or create custom Project Profiles tailored to specific organizational needs.
When configuring Project Profiles, you can choose between two deployment models:
Static (Pre-Defined): Profile specifies a fixed account and Region. Projects by default deploy to the same location. This is recommended for strict governance controls and compliance requirements where production resources must remain in designated accounts or Regions.
Dynamic (Parameterized): Users select from available associated accounts and Regions during Project creation (configured through Account Pools). This is recommended for multi-environment workflows (Dev/Test/Prod) and reducing administrative overhead by maintaining fewer profile templates.
Dynamic profiles balance governance with agility: Administrators define standards once, while users retain deployment flexibility aligned with their business needs.
Step 4: Project creation
With Project Profiles configured, you can now create a new Project from any of the associated accounts using the profile created in the previous step.
Defining project boundaries: when to create a new project
One of the common challenges that you will face is determining when to create a new Project. The answer significantly impacts collaboration effectiveness, resource isolation, cost tracking, and governance. Here’s a practical framework to guide your decision-making.
A Project should represent a distinct business initiative with a defined scope, a dedicated team, and measurable outcomes.Think of Projects as team workspaces organized around business outcomes, not technical components.
Create a new Project when you need clear separation across multiple dimensions: cost allocation, access control, and data governance. If Finance and Marketing teams require separate budget tracking, distinct data access policies, and different governance controls, they should have separate Projects. For example, “Customer Churn Prediction” and “Fraud Detection” might use similar tools, but if they have different stakeholders, budget owners, and data sensitivity requirements, so they warrant separation. Similarly, create separate Projects when facing different compliance or regulatory requirements (like HIPAA versus PCI-DSS) or when initiatives have independent deployment lifecycles. Experimental ML research Projects shouldn’t share workspaces with production recommendation engines that require stricter change controls and availability guarantees.
However, avoid fragmenting related work into unnecessary silos:
Don’t create separate Projects for individual workflows or pipelines, a single “Marketing Campaign Optimization” Project should contain audience segmentation, propensity modeling, and campaign attribution workflows together.
Don’t separate different data processing stages; keep data ingestion, transformation, and analysis within one Project to maintain clear lineage and enable seamless collaboration.
Projects are team workspaces, not personal sandboxes, so use shared Projects with role-based access control rather than creating individual Projects per team member.
Small proof of concepts (POC) or temporary experiments should be conducted within the parent Project, with successful initiatives promoted to dedicated production Projects only when they mature into full-scale capabilities requiring independent governance.
Conclusion
Throughout this post, we explored how the separation of governance and working accounts forms the foundation of a scalable, secure, and compliant data and AI platform.
With centralized governance in the Domain account, organizations can enforce consistent security policies, compliance requirements, and cost management, while allowing sub-accounts the autonomy over their own resources. This approach enhances security and compliance, and fosters collaboration and innovation within teams by providing them with the flexibility that they need to operate effectively. Ultimately, this governance-first strategy supports keeping data remains protected and accessible in a controlled manner, empowering teams to drive business outcomes efficiently. To implement a multi-account deployment for your organization, get started by creating your first SageMaker Unified Studio Domain and follow the step-by-step guidance to establish your governance-first architecture.
Finding the right data assets in large enterprise catalogs can be challenging, especially when thousands of datasets are cataloged with organization-specific metadata. Amazon SageMaker Unified Studio now supports custom metadata search filters. You can filter catalog assets using your own metadata form fields like therapeutic area, data sensitivity, or geographic region rather than relying only on free-text search. Custom metadata forms are structured templates that define additional attributes that can be attached to catalog assets.
In this post, you learn how to create custom metadata forms, publish assets with metadata values, and use structured filters to discover those assets. We explore a healthcare and life sciences use case. A research organization catalogs metrics in Amazon SageMaker Catalog using custom metadata forms with fields such as Therapeutic Area and Sample Size. Researchers building Machine learning models can now search datasets based on custom filters across hundreds of cataloged assets to identify the best datasets to train their models.
Key capabilities
Custom metadata search filters in SageMaker Unified Studio offer the following key capabilities:
Custom metadata form filters – You can filter search results using any custom metadata form fields defined in their catalog. For example, a researcher can filter by Therapeutic Area = Oncology and Data Sensitivity = Confidential to locate specific datasets.
Name and description filters – You can add filters that target asset names or descriptions using a text search operator, enabling targeted discovery without scanning full search results.
Date range filters – You can filter assets by date using on, before, after, and between operators, making it straightforward to locate recently updated or historically relevant assets.
Combinable filters – You can combine multiple filters to construct precise queries. For example, filtering by AWS Region = US AND Classification = PII AND Updated after 2026-01-01 returns only assets matching all three criteria.
Persistent filter selections – You can filter configurations stored in your browser and are not shared across devices or other users. You can later return to the catalog and find your previously defined filters.
Solution overview
In the following sections, we demonstrate how to set up custom metadata forms, publish assets with metadata values, and use custom metadata search filters to discover those assets.We complete the following three steps for the demonstration.
For instructions on setting up a domain and project, see the Getting started guide.
To create a custom metadata form
Complete the following steps to create a custom metadata form with filterable fields:
In SageMaker Unified Studio, choose Project overview from the navigation pane.
Under Project catalog, choose Metadata entities.
Choose Create metadata form.
To create a new metadata form ‘research_metadata’ use the following details, then choose Create metadata form.
Define the form fields. For this demo, we add the following fields:
Create first field Therapeutic Area (String) – Mark as Searchable
Create second field Subject Count (Integer) – Mark as Filterable by range
Mark the form as ‘Enabled’ so the form is visible and can be used.
Create and publish with metadata
In this section, you create a custom asset and attach the research_metadata form created in the previous step.
Under Project catalog in the navigation pane, choose Metadata entities. Choose the ‘ASSET TYPES’ tab and select “CREATE ASSET TYPE’.
Create a new asset type and attach the metadata form that we created in the previous step. A new asset type ‘metric’ is created.
Next, we will create two metrics. Under Project catalog in the navigation pane, choose Assets. On the Asset page, choose CREATE, and then choose Create asset from the menu.
In this demo, you create two metrics.
For the first metric ‘drug_1_treatment’, provide the following asset name and description.
Add the following values for the metadata form.
Validate all fields and choose CREATE.
Publish the asset to the catalog.
Next, we will create the second metric ‘drug_1_treatment’. Repeat the steps from the previous procedure and enter the values shown.
Subject Count = 450
Therapeutic Area = Oncology
Use custom metadata search filters
After publishing assets with custom metadata, go to the Browse Assets page to use the filters.
To browse assets and view filters
In SageMaker Unified Studio, choose Discover from the navigation bar, then select Catalog, Browse Assets.
The search page displays with the filter sidebar on the left. You can see the existing system filters (Data type, Glossary terms, Asset type, Owning project, Source Region, Source account, Domain unit) along with the new Date range and Add Filter sections.
Add a custom filter
Choose + Add Filter at the bottom of the filter sidebar. For Filter type, select Metadata form. For Metadata form, select research_metadata and add a filter as shown in the following image. Choose Apply when you’re done. The search results update to show only assets where ‘subject_count’ is greater than 50.
To combine multiple filters
Choose + Add Filter again. For Filter type, select Metadata form. For Metadata form, select research_metadata and add a filter as shown in the following image. Choose Apply when you’re done.
Manage custom filters
Filter configurations are stored in the user’s browser and are not shared across devices or users.
To customize search, you could:
Toggle filters – Use the checkboxes next to each custom filter to enable or disable them without deleting.
Edit or delete – Choose the kebab menu (⋮) next to any custom filter to edit its values or delete it.
Clear all – Choose CLEAR next to the Custom filters header to deselect all custom filters at once.
Persistence – Your custom filters persist across browser sessions. When you return to the Browse Assets page, your previously defined filters are still listed in the sidebar, ready to be activated.
Using the SearchListings API
To search catalog assets programmatically, you can use the SearchListings API in Amazon DataZone, which supports the same filtering capabilities as the SageMaker Unified Studio UI. The following example filters assets where a custom string field contains a specific value and a numeric field is within a range:
Consider the following best practices when using custom metadata search filters:
Define your metadata forms before publishing assets at scale. If you publish assets before the forms are finalized, you might need to re-tag existing assets, which is a time-consuming process in large catalogs.
Define metadata forms aligned with your organization’s discovery needs (therapeutic areas, data classifications, geographic regions) before publishing assets at scale.
Use specific, consistent values in metadata fields to get precise filter results. For example, use standardized values (for example, use “Oncology” consistently rather than “oncology” or “Onc”) across all assets.
Combine multiple filters to narrow results efficiently rather than scanning through broad result sets.
Use the date range filter alongside custom metadata filters to locate assets within specific time windows.
Custom metadata search filters in Amazon SageMaker Unified Studio give data consumers the ability to find exact assets using structured filters based on their organization’s own metadata fields. By combining multiple filters across custom metadata forms, asset names, descriptions, and date ranges, data consumers can construct precise queries that surface the right datasets without scanning through broad search results. Filter persistence across browser sessions further streamlines repeated discovery workflows.
Custom metadata search filters are now available in AWS Regions where Amazon SageMaker is supported.
Data has become an indispensable strategic asset for the entire financial services industry, driving innovation and competitive advantage in an increasingly digital marketplace. At Swiss Life Germany, maximizing the value of this asset means empowering internal teams to derive actionable insights and deliver personalized financial solutions to diverse clientele. This led to the need to establish seamless data sharing workflows that enhance cross-departmental collaboration while maintaining strict security and compliance standards. To accomplish this, Swiss Life Germany decided to implement advanced data processing and governance capabilities using Amazon SageMaker.
Integrating SageMaker into a highly regulated enterprise environment required aligning the service’s agility with Swiss Life’s rigorous infrastructure as code (IaC) automation standards. This post demonstrates how Swiss Life Germany addressed these sophisticated deployment requirements by developing a custom Terraform pattern designed specifically for platform engineers and data architects.
Swiss Life Germany cloud journey
Swiss Life Germany is a leading provider of customized pension products and financial advice. Building on over 100 years of delivering insurance, retirement planning, and wealth management solutions, a key driver of the company’s recent evolution was the strategic transition from legacy on-premises data centers to a modern, cloud-centric architecture. After an extensive evaluation of various providers, Swiss Life Germany selected Amazon Web Services (AWS) as the strategic foundation to modernize their data operations. By using AWS, the organization was able to transition from capital-intensive data centers to a flexible pay-as-you-go model, significantly reducing the operational costs.
Following their comprehensive AWS cloud migration over the last two years—combining 30% re-platforming with 70% lift-and-shift strategies—Swiss Life Germany modernized infrastructure management through IaC. The company introduced the governance concept of an IT System. An IT System is a fundamental unit of management that defines a software component regardless of its origin. Whether a component is purchased from a vendor, self-developed or consumed as software as a service (SaaS), it’s integrated into this single governance structure. This ensures that off-the-shelf products and custom-coded applications are held to the same high standards of visibility and accountability. Every IT system is required to maintain specific attributes that allow for seamless oversight such as unique identifiers, assigned ownership and the associated AWS resources logically grouped under the IT System they support.
Where traditional approaches would store and expose this information in configuration management database (CMDB)-like systems to store static snapshots of asset data, Swiss Life adopted a more dynamic model. By using GraphQL API as a unified meta-model, the company queries application data directly from its primary source systems. This approach eliminates the delays common in batch-processed databases, ensuring maximum freshness. The API serves as a single entry point for infrastructure data, documentation, organizational metadata, and even inter-application dependencies. The transparency and automation gained through this everything-as-code and API-first approach provided a blueprint for the Swiss Life Data Platform: complete transparency, reproducibility, and end-to-end automation.
This robust technical foundation served as a catalyst and prerequisite for Swiss Life’s broader strategic goals and governed framework.
Defining the vision for a unified data solution
With the architectural foundations in place, the next challenge was to establish efficient data flows from production systems through data engineering teams to end users across various business divisions, with hundreds of specific use cases demanding attention.
For instance, Swiss Life’s customer portal specialists had to validate the effectiveness of campaign management and push notification systems in real-time, requiring secure and immediate access to interaction data.
Security requirements added another layer of complexity, because Swiss Life’s solution needed to incorporate robust compliance standards including two-factor authentication, session-based access controls, and granular row and column-level security protections.
To align with the overarching Swiss Life Germany cloud strategy, the company aimed to build a modern data solution atop their existing AWS data and analytics services. AWS introduced SageMaker to Swiss Life Germany following its announcement at AWS re:Invent 2024. A proof-of-concept quickly validated that this was the right tool to advance Swiss Life’s data journey. By deploying a fully automated framework, Swiss Life Germany sought to create a secure, compliant framework with SageMaker democratizing data access for authorized users, ultimately enabling faster business insights and more responsive customer experiences across the entire data environment.
Having met the infrastructure requirements, let’s look at what SageMaker looks like for end users and how data platform administrators can control access and resources at a granular level.
Users and their types of projects
A typical end user experience within Amazon SageMaker Unified Studio starts with creating a project. A project is a logical boundary within a domain where the data teams can collaborate and work on a business use case. Administrators would provision the blueprints and project profile templates for the data teams, as shown in the following figure.
However, at Swiss Life, they have extended the data platform administrator’s role to also create projects so they can maintain regulatory compliance and remove initial onboarding hurdles. The end user experience in SageMaker Unified Studio is simplified with data teams selecting their respective projects to work on a business initiative, as shown in the following figure.
To implement this solution effectively, Swiss Life identified different user groups:
A solution team developing an IT System that can act as producer or consumer of data assets.
A data scientist doing advanced data processing. They will most likely consume a lot of data assets and might produce some high aggregated data assets. The data processing software is also categorized as an IT System.
Business users who have some SQL skills and want to process data to get insights for their daily business.
A platform team administering the data platform. They provide core services to all users to make participation as straightforward as possible.
A data officer who wants to have a single point of interpretation for data.
Given this diverse set of user groups, the resulting data platform had to support a federated data organization with a centralized governance, decentralized data stores and data-processing organized at the IT System level. This architecture means the SageMaker management account—which orchestrates the data domain—contains no actual data, instead, data and compute resources reside in the individual IT System AWS accounts. Swiss Life’s implementation distinguishes between two fundamental project types:
IT System projects (for technical users)
Team projects (for non-technical users)
Swiss Life decided to align team projects with specific organizational units and operate them without staging environments, providing dedicated workspaces for departmental data initiatives. In contrast, IT System projects are associated with specific solutions such as customer portal or CRM systems. These follow a structured staging methodology, with each solution team managing dedicated DEV, TEST, and PROD environments to maintain proper development lifecycles and quality control.
This federated architecture is designed to handle the immense scale and diversity of Swiss Life’s data landscape. Swiss Life’s data platform would then aim to provide unified access to over 180 database servers with over 1,800 databases and 18 thousand tables across all stages (DEV, TEST and PROD).
In this post, we focus on the IT System projects.
How Swiss Life built the automation framework
Because Terraform is the preferred IaC tool across Swiss Life Germany, the team faced an interesting architectural challenge: while the existing infrastructure framework incorporates numerous AWS services that are readily supported by Terraform, SageMaker required a custom integration approach to align with Swiss Life’s advanced automation patterns.
Rather than adopting a manual ClickOps approach to infrastructure management, Swiss Life developed an innovative solution to keep the entire infrastructure—including SageMaker—within their Terraform automation, preserving key benefits like state management. The team accomplished this by using Terraform’s AWS Lambda invoke function resource with a create, read, update, delete (CRUD) lifecycle scope. By using this approach, the organization could maintain a single source of truth for infrastructure, while accommodating specific requirements of SageMaker. This component is called the Management Lambda and it serves as a bridge between Terraform’s declarative configuration and SageMaker, so that Swiss Life can provision, modify, and decommission Amazon SageMaker resources through established Terraform workflows.
The following is the snippet of a new domain creation using Terraform and Management Lambda:
Using this approach, Swiss Life successfully automated every aspect of deploying a complete SageMaker domain installation within the Swiss Life cloud data platform. The automation encompasses the entire domain creation process, using the SageMaker domain unit feature as an organizational framework for diverse project portfolio.
Deployment architecture
Let’s dive deeper into the individual steps of the automation process itself. As said, all resources within SageMaker are controlled by the Terraform-invoked Management Lambda whereas other resources are directly managed by Terraform itself. The Management Lambda and SageMaker resources such as domains, metadata fields and others live in the central SageMaker account. Users of the data platform have their own AWS accounts. To start with, AWS Lake Formation had to be enabled across all AWS accounts, which could then act as consumer or provider to the platform. Using the established AWS Landing Zones mechanism, this was done by a single deployment to the management account. This early step also verified the management role being present in all accounts and assumable by the Management Lambda.
The following steps are used to set up Swiss Life’s data platform from scratch, as shown in the following diagram:
The Management Lambda is deployed to Swiss Life’s designated SageMaker account. This Lambda function uses the described CRUD pattern for all subsequent SageMaker-specific operations.
The domain provisioning begins by creating the service and domain execution roles, after which the Management Lambda creates the domain and uses these roles. During this step, administrative users and their associated permissions are also configured.
Upon successful domain creation, the Lambda function returns the domain identifier as output. This identifier is then used to let all AWS accounts of the company join this domain. These can now act as providers or consumers on the platform, resulting in a frictionless onboarding of teams.
Because Swiss Life decided to stage data products in a single domain, the DEV, TEST, and PROD domain units are then created, establishing the hierarchical structure under which IT System projects are subsequently created in the next implementation phase.
All projects and teams with the necessary prerequisites set up are then created automatically. This is done by using the enterprise GraphQL API mentioned to retrieve all IT products, their teams and roles. With that, each team already has their ready-to-use project in place upon singing into the platform. In detail this process looks like the following:
Continuing with the earlier example: the customer portal team needs to share their data with others in the organization and is using their dedicated project for this purpose. The process is shown in the following figure.
The deployment initiates with a cross-account role assumption by the Management Lambda to activate the blueprint configuration in the team’s AWS account. A standardized creation process was built to help facilitate all accounts are configured identically, maintaining consistency across the environment.
Next, a project profile specifically tailored for the customer portal project is created. This profile establishes the foundational settings and permissions framework that will govern the project’s operations.
With the profile in place, the actual project within this previously established project profile can now be provisioned, instantiating the working environment, where data sharing and collaboration will occur. This results in an identical amount of project profiles and projects in the SageMaker Unified Studio domain.
Finally, an automated membership management process is triggered. The system again queries Swiss Life’s Enterprise GraphQL API to identify all members of the solution team and automatically adds them as project members with appropriate permissions. This process executes daily, to help ensure that project access permissions remain current and accurately reflect team composition changes.
In the third and final deployment step, the user experience is enhanced by making the data platform immediately usable for teams in production. When teams and their members first access the domain URL, they find a project environment already populated with all necessary assets, so they can begin working without delay. This is accomplished through the following steps, shown in the following figure:
An automated discovery process is triggered that identifies all Amazon Simple Storage Service (Amazon S3) buckets and AWS Glue assets associated with the specific customer portal IT System. This inventory is created by using the AWS Resource Tagging API with specific filters targeting these asset types, so that all relevant resources for exactly that IT System are captured.
When identified, all discovered S3 buckets are registered as data lake locations within the platform. For each location, they create an AWS Identity and Access Management (IAM) role with precise access permissions, adhering to the least privilege security model.
Then grantable permissions are granted to the SageMaker project role for these assets, establishing a permission delegation framework that allows project members to manage access within their project scope—managing cross project access—while maintaining overall governance.
Finally, the AWS Glue databases are added as data sources within the project. These data sources are configured with daily synchronization schedules to automatically load new metadata into SageMaker, helping to ensure that catalog information remains current without manual intervention.
What a team needs to start with all of this
The overarching goal throughout this implementation has been to simplify the adoption process for the internal data teams. To ensure the data teams could immediately use the powerful capabilities of SageMaker without needing to manage its underlying architecture, Swiss Life Germany streamlined the experience by pre-packing the entire onboarding process into a high-level Terraform module. Teams can then use the module to deploy a complete, production-ready environment with minimal configuration, accelerating their path from setup to insight.
The following is an example of the code used by the module.
To initiate this, the data teams define their basic parameters such as network configuration or their IT-System identifier as outlined previously and submit a pull request in the central Git repository. After the Swiss Life data platform team reviews and approves the request, the automated processes run in the background, preparing the complete environment. This automated approach has reduced deployment time for new environments from several weeks of manual coordination to under 20 minutes.
Rather than requiring users to understand the intricate deployment steps and managing the infrastructure, the automated deployment process empowers business units, like the customer portal team, to focus on deriving insights. At the same time, the Swiss Life Germany data platform team also maintains precise control over resource allocations, access rights and cost management.
Future enhancements
Looking ahead, Swiss Life plans to elevate its automation to a higher level of business abstraction. The next major enhancement focuses on removing the requirement for teams to request specific technical assets. Instead, the vision is to implement an intuitive interface where teams can specify the business terms or data domains they require. The system will automatically identify and provision the correct underlying technical assets associated with those business definitions.
This semantic layer will create a more natural interaction model, so that business users can think and work in familiar concepts rather than technical constructs. For example, rather than requesting access to specific S3 buckets or AWS Glue databases, a marketing analyst might indicate they need customer interaction data or campaign response metrics. An automated system will then map these business terms to the appropriate technical resources, provision access, and configure the environment accordingly.
By elevating automation to this business terminology level, Swiss Life aims to further reduce friction in the data access process while maintaining its robust security and governance framework. This evolution represents Swiss Life Germany’s commitment to continuously improving how data serves the business, making sophisticated data capabilities increasingly accessible to all parts of the organization.
Conclusion
Through the comprehensive automation of Amazon SageMaker, Swiss Life Germany has transformed their usage of data from a complex technical challenge into a streamlined business enabler. By using AWS services and their innovative Terraform-Lambda integration approach, Swiss Life created a secure, compliant data platform that maintains governance while democratizing access across the full organization. The automated deployment process helps ensure consistency across environments while dramatically reducing the technical knowledge required for teams to begin using advanced data capabilities. Business units, such as the customer portal team, can now focus on deriving insights rather than managing infrastructure, accelerating data-driven decision making throughout the company. This implementation represents a significant milestone in Swiss Life Germany’s cloud journey, demonstrating how thoughtful automation can simultaneously enhance security, improve operational efficiency, and accelerate business outcomes.
As of today, 5 organizational unit teams and 15 IT System teams were onboarded to the platform. To speed things up, Swiss Life has decided to onboard all 180 database clusters and consume data using SageMaker over the coming months. This expansion is designed to enable teams to use the data platform and enhance the efficiency of data discovery and data sharing processes across the organization.
This post is co-written with Neel Patel, Abdullahi Olaoye, Kristopher Kersten, Aniket Deshpande from NVIDIA.
Today, we’re excited to announce that the NVIDIA Evo-2 NVIDIA NIM microservice are now listed in Amazon SageMaker JumpStart. You can use this launch to deploy accelerated and specialized NIM microservices to build, experiment, and responsibly scale your drug discovery workflows on Amazon Web Services (AWS).
In this post, we demonstrate how to get started with these models using Amazon SageMaker Studio.
NVIDIA NIM microservices on AWS
NVIDIA NIM integrates closely with AWS managed services, such as Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Kubernetes Service (Amazon EKS), and Amazon SageMaker AI, to support deployment of generative AI models at scale. As part of NVIDIA AI Enterprise, which is available in the AWS Marketplace, NVIDIA NIM is a set of microservices designed to accelerate the deployment of generative AI. These prebuilt containers support a broad spectrum of generative AI models, from open source community models, to NVIDIA Nemotron and custom models. NIM microservices are deployed with just a few lines of code, or with a few actions in the SageMaker Studio console. Engineered to facilitate seamless generative AI inferencing at scale, NIM ensures that generative AI applications can be deployed on various AWS services.
NVIDIA BioNeMo Evo 2 overview
NVIDIA BioNeMo is a platform of NIM microservices, developer tools, and AI models that accelerate building, adapting, and deploying biomolecular AI models for drug discovery. It packages curated training recipes, data loaders, and domain-optimized pretrained models for DNA, RNA, and proteins, alongside NVIDIA CUDA-X libraries such as NVIDIA cuEquivariance. These components power tasks such as 3D structure prediction, de novo design, virtual screening, docking, and property prediction with GPU-accelerated performance.
NVIDIA NIM microservices provide optimized, API-first inference that integrates directly into enterprise pipelines across on-premises and the cloud, providing scalable and secure deployment with faster time-to-market and lower Total Cost of Ownership (TCO). The Evo 2 NIM delivers a 40-billion parameter foundation model (FM) trained on a vast dataset of genomes that can be used to predict protein function, identify mutations, and accelerate bioengineering research. Furthermore, the Evo 2 NIM can be chained with other NIM microservices such as ESMFold to create end-to-end, containerized workflows that cut time-to-insight while streamlining deployment through consistent APIs.
SageMaker Studio overview
SageMaker Studio is a web-based integrated development environment (IDE) for machine learning (ML) that provides a unified visual interface for all of the tools that you need to complete each step of the ML development lifecycle. SageMaker Studio provides complete access, control, and visibility into each step of the ML workflow, from data preparation to model building, training, and deployment.
The key features of SageMaker Studio include:
Unified interface: Access all SageMaker capabilities through a single, web-based visual interface
Jupyter notebooks: Fully managed Jupyter notebooks with pre-configured kernels for popular ML frameworks
Model management: Browse, deploy, and manage models from AWS Marketplace and other sources through an intuitive interface
Collaboration: Share notebooks, experiments, and models with your team members
Cost management: Monitor and control costs with built-in usage tracking and resource management tools
Amazon SageMaker JumpStart overview
SageMaker JumpStart is a fully managed service that offers state-of-the-art foundation models for various use cases such as content writing, code generation, question answering, copywriting, summarization, classification, and information retrieval. It provides a collection of pre-trained models that you can deploy quickly, accelerating the development and deployment of ML applications. One of the key components of SageMaker JumpStart is model hubs, which offer a vast catalog of pre-trained models, such as Mistral, for a variety of tasks. You can now discover and deploy Evo 2 NIM in Amazon SageMaker Studio or programmatically through the SageMaker Python SDK, so you can derive model performance and MLOps controls with Amazon SageMaker AI features such as Amazon SageMaker Pipelines, Amazon SageMaker Debugger, or container logs. The model is deployed in a secure AWS environment and in your VPC, helping to support data security for enterprise security needs.
Prerequisites
Before getting started with deployment, make sure that your IAM service role for SageMaker AI has the SageMakerFullAccess permission policy attached. To deploy the NVIDIA NIM microservices successfully, confirm one of the following:
Make sure that your IAM role has the following permissions, and that you have the authority to make AWS Marketplace subscriptions in the AWS account used:
aws-marketplace:ViewSubscriptions
aws-marketplace:Unsubscribe
aws-marketplace:Subscribe
If your account is already subscribed to the model, then you can skip to the following Deploy section. Otherwise, start by subscribing to the model package and move to the Deploy section after.
Subscribe to the model package
To subscribe to the model package, complete the following steps:
Open the SageMaker Jumpstart portal from the SageMaker AI page.
Search for Evo 2 NIM.
Choose View model, and on the Model details page choose Subscribe. This will take you to the AWS Marketplace listing for the Evo 2 NIM.
On the AWS Marketplace listing page, choose View purchase options, review the purchase terms and choose the Subscribe button if you and your organization agree with EULA, pricing, and support terms.
Choose Continue to with the configuration and choose an AWS Region where you have the service quota for the desired instance type.
A product Amazon Resource Name (ARN) is displayed. This is the model package ARN that you need to specify while creating a deployable model using the SageMaker SDK.
Option 1: Deploy the Evo 2 NIM using SageMaker Studio
The following section outlines how to deploy the EVO 2 NIM using SageMaker Studio.
Getting started with SageMaker Studio
Begin by accessing the AWS Management Console and navigating to the SageMaker AI service. When you’re in the SageMaker AI console, locate Studio in the left navigation panel and choose Open Studio next to your user profile. If you haven’t set up a SageMaker Studio domain yet, then you must create a new domain and user profile first. This launches the web-based SageMaker Studio interface where you can manage all aspects of your ML workflow.
Navigating to model packages
Within SageMaker Studio, look for Models in the left sidebar and choose JumpStart base models tab within the Models interface. This section contains all available model packages in SageMaker JumpStart, including those from the AWS Marketplace
Locating the Evo-2 NIM model
Use the search functionality to find the NVIDIA Evo-2 NIM model by searching for terms such as “Evo-2” or “NVIDIA”. When you locate the model package in the filtered results, choose it to view the Model overview page. This page provides an overview of the model and can have a Notebooks tab that will show a sample notebook that contains an example showing how to use the NIM. You can choose Open in JupyterLab to open the notebook in JupyterLab and use it as a starting point for using the NIM.
Configuring the model deployment
On the model package overview page, choose the Deploy button on the top right to begin the deployment process. You must configure several important settings: provide a unique endpoint name (such as “Evo-2-nim-endpoint”), choose an appropriate instance type (ml.g6e.12xlarge is recommended for optimal performance), set the initial instance count (typically 1 for initial testing), and specify an endpoint configuration name. Review all of these settings carefully before proceeding.
Initiating and monitoring the deployment
After verifying your configuration settings, choose Deploy to start the deployment process for creating a Real-time inferance endpoint. Navigate to the Deployments section and then the Endpoints section in the left sidebar to monitor the deployment progress. The endpoint status initially shows Creating and typically takes 5–10 minutes to complete. You can track the progress and should see the status change to InService once the deployment is successful.
Testing and validation
When your endpoint is deployed and shows the In Service status, you can optionally test it directly through the SageMaker Studio interface. Choose your deployed endpoint from the endpoints list to access the Endpoint summary page. Scroll down and select the Playground tab. If available, you will see two options: Test the sample request and Use Python SDK example code. You can use either option to validate the deployment by using a sample protein sequence. This validates the endpoint is working correctly before integrating it into your applications.
Option 2: Deploy Evo 2 using the SageMaker SDK
In this section we walk through deploying the Evo-2 NIM through the SageMaker SDK. Make sure that you have the account-level service limit for using ml.g6e.12xlarge for endpoint usage as one or more instances. Furthermore, NVIDIA provides a list of supported instance types that support deployment. Refer to the AWS Marketplace listing for the model to see the supported instance types. To request a service quota increase, go to the AWS service quotas.
import sagemaker
import boto3
from sagemaker import ModelPackage, get_execution_role
import json
# Initialize SageMaker session and role
role = get_execution_role()
sagemaker_session = sagemaker.Session()
# Model Package ARN from your AWS Marketplace subscription
# Replace this with your actual Model Package ARN after subscription
model_package_arn = "arn:aws:sagemaker:<region>:<account-id>:model-package/Evo-2-nim-model"
# Create model from AWS Marketplace Model Package
model = ModelPackage(
role=role,
model_package_arn=model_package_arn,
sagemaker_session=sagemaker_session
)
# Deploy the model to an endpoint
predictor = model.deploy(
initial_instance_count=1,
instance_type="ml.g6e.12xlarge", # Using recommended NVIDIA GPU instance
endpoint_name="Evo-2-endpoint",
wait=True
)
Run Inference with Evo 2 SageMaker endpoint
When you have the model, you can use a sample text to do an inference request. NIM on SageMaker supports the OpenAI API inference protocol inference request format. For an explanation of the supported parameters, go to the Evo-2 API documentation.
To avoid unwanted charges, complete the steps in this section to clean up your resources.
Deleting the endpoint from SageMaker Studio
In SageMaker Studio, navigate to the Endpoints section in the left sidebar under Inference to view all your active endpoints. Locate your Evo-2 NIM endpoint in the list and select it to open the endpoint details page. On this page, there is a Delete button. Choose Delete and confirm the deletion when prompted. The endpoint status changes to Deleting and disappears from your endpoints list when the deletion is complete. This process typically takes a few minutes, and when it’s deleted the endpoint stops incurring charges immediately.
Delete the SageMaker endpoint
The SageMaker endpoint that you deployed incurs costs if you leave it running. Use the following code to delete the endpoint if you want to stop incurring charges. For more details, go to Delete endpoints and resources.
# Delete endpoint when done (important for cost management)
predictor.delete_endpoint()
Conclusion
The availability of NVIDIA Evo-2 NIM microservices on Amazon SageMaker Jumpstart represents a significant advancement for researchers and organizations working in drug discovery. This solution provides GPU-accelerated multiple sequence alignments and dramatically speeds up structure prediction pipelines that are critical for protein design and antibody research. Users can implement the flexible deployment options—through SageMaker Studio, or SageMaker SDK—to choose the approach that best fits their workflow and technical expertise. The optimized performance of these NIM microservices, combined with the scalability and security of SageMaker, enables faster time-to-insight while streamlining the deployment of complex biomolecular AI models. We encourage you to try the Evo-2 NIM today and look out for future release of MSA-search and Boltz-2 NIMs to accelerate your drug discovery workflows and use the power of NVIDIA’s specialized microservices on AWS infrastructure.
When creating a project in Amazon SageMaker Unified Studio, users select a project profile to define resources and tools to be provisioned in the project. These are used by Amazon SageMaker Catalog to implement a data mesh pattern. Some users don’t want to take advantage of resources provisioned along with the project for various reasons. For instance, they may want to avoid making changes to their existing applications and data products.
This post shows you how to implement a data mesh pattern by using Amazon SageMaker Catalog while keeping your current data repositories and consumer applications unchanged.
Solution overview
In this post, you will simulate a scenario based on data producer and data consumer that exists before Amazon SageMaker Catalog adoption. For this purpose, you will use a sample dataset to simulate existing data and simulate an existing application using an AWS Lambda function. You can apply the same solution to your real-life data and workloads.
The following diagram illustrates the solution architecture’s key configurations. In this architecture, the Amazon Simple Storage Service (Amazon S3) bucket and the AWS Glue Data Catalog in the producer account simulate the existing data repository. The Lambda function in the consumer account simulates the existing consumer application.
Here is a description of the key configurations highlighted in the architecture:
As part of an Amazon SageMaker domain, create a producer project (associated to a producer account) and a consumer project (associated to a consumer account). Among other resources, a project AWS Identity and Access Management (IAM) role is created for each project in the associated account.
In the producer account, use AWS Lake Formation to grant producer project’s IAM role permissions to access the existing data asset.
Publish the data asset in the Amazon SageMaker Catalog from the producer project.
Subscribe the data asset from the consumer project.
In the consumer account, configure your Lambda function to assume consumer project’s IAM role to access the subscribed data asset.
The solution architecture is based on the following Amazon Web Services (AWS) services and features:
Amazon SageMaker Catalog offers you a way to discover, govern, and collaborate on data and AI securely.
Amazon SageMaker Unified Studio provides a single data and AI development environment to discover and build with your data. Amazon SageMaker Unified Studio projects provide collaborative boundaries for users to accomplish data and AI tasks.
AWS Lake Formation, which you can use centrally to govern, secure, and share data for analytics and machine learning.
AWS Glue Data Catalog is a persistent metadata store for your data assets. It contains table definitions, job definitions, schemas, and other control information to help you manage your AWS Glue environment.
Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance.
Setting up resources
In this section, you will prepare the resources and configurations you need for this solution.
Three AWS accounts
To follow this solution, you need three AWS accounts, and it’s better if they’re part of the same organization in AWS Organizations:
Producer account – Hosts the data asset to be published
Consumer account – Hosts the application that consumes the data published from the producer account
Governance account – Where the Amazon SageMaker Unified Studio domain is configured
Each account must have an Amazon Virtual Private Cloud (Amazon VPC) with at least two private subnets in two different Availability Zones. For instruction, refer to Create a VPC plus other VPC resources. Make sure to create both VPCs in the same Region you plan to apply this solution.
A governance account is used for the sake of convenience, but it’s not strictly needed because Amazon SageMaker can be configured and managed in producer or consumer accounts.If you don’t have access to three accounts, you can still use this post to understand the key configurations required to implement a data mesh pattern with Amazon SageMaker Catalog while keeping your current data repositories and consumer applications unchanged.
Create a data repository in the producer account
First, create a sample dataset by following these instructions:
In the navigation pane, under Data Catalog, choose Databases.
Choose Add database.
For Name, enter collections.
For Description, enter This database contains collections of statistics for natural resources.
Choose Create database.
In the navigation pane, under Data Catalog, choose Tables.
Choose Add table.
In the table creation guided procedure, enter the following input for Step 1: Set table properties:
For Name, enter trees.
For Database, select collections.
For Description, enter This table captures ratings data related to the characteristics of various tree species.
For Table format, select Standard AWS Glue table (default).
For Select the type of source, select S3.
For Data location is specified in, select my account.
For Include path, enter s3://<bucket-name>/<prefix>/ where <bucket-name> is the name of the S3 bucket you created earlier in this procedure and <prefix> is the optional prefix for the trees.csv file you uploaded.
For Data format, select CSV.
For Delimeter, select Comma (,).
Choose Next.
For Step 2: Choose or define schema, enter the following:
For Schema, select Define or upload a schema.
Choose Edit schema as JSON and enter the following schema in the pop-up:
Create the Lambda function in the consumer account. This will simulate a data consumer application.First, in the consumer account create the IAM policy and the IAM role to be assigned to the Lambda function:
Create an IAM policy and name it smus_consumer_athena_execution by using the following policy. Make sure to replace placeholders <AWS_Region> and <AWS_account_ID_number> with your Region and consumer account ID number. You will replace the <workgroup_id> placeholder later. For IAM policy creation instructions, refer to Create IAM policies (console).
Create an IAM role for AWS Lambda service and name it smus_consumer_lambda. Assign to it the AWS managed permission AWSLambdaBasicExecutionRole and the permission named smus_consumer_athena_execution that you just created. For instructions, refer to Create a role to delegate permissions to an AWS service.
After the IAM role for the Lambda function is in place, you can create the Lambda function in the consumer account:
Choose Create function and enter the following information:
For Function name, enter consumer_function.
For Runtime, select Python 3.14.
Expand Change default execution role section.
For Execution role, select Use an existing role.
For Existing role, select smus_consumer_lambda.
Choose Create function.
Under the Code tab, in the Code source, replace the existing code with the following:
import boto3
import time
sts_client = boto3.client('sts')
role_arn = "<role_arn>"
session_name = "AthenaQuerySession"
catalog = "AwsDataCatalog"
database = "<database_name>"
workgroup = "<workgroup_id>"
query = "select * from "+catalog+"."+database+".trees"
def lambda_handler(event, context):
# Assume SageMaker Unified Studio project role
assumed_role_object = sts_client.assume_role(
RoleArn=role_arn,
RoleSessionName=session_name
)
# Get temporary credentials
credentials = assumed_role_object['Credentials']
# Create Athena client using temporary credentials
athena = boto3.client(
'athena',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
region_name='eu-west-1'
)
# Execute Athena Query
response = athena.start_query_execution(
QueryString=query,
QueryExecutionContext={
'Database': database,
'Catalog': catalog
},
WorkGroup=workgroup
)
query_execution_id = response['QueryExecutionId']
# Polling with exponential backoff
wait_time = 0.25 # Start with 0.25 seconds
max_wait = 8 # Maximum wait time of 8 seconds
while True:
result = athena.get_query_execution(QueryExecutionId=query_execution_id)
state = result['QueryExecution']['Status']['State']
if state in ['FAILED', 'CANCELLED']:
raise Exception(f"Query {state}")
elif state == 'SUCCEEDED':
break
elif state in ['QUEUED', 'RUNNING']:
time.sleep(wait_time)
wait_time = min(wait_time * 2, max_wait) # Double wait time, cap at max_wait
# Retrieve results
results = athena.get_query_results(QueryExecutionId=query_execution_id)
return results
Choose Deploy.
The code provided for the Lambda function includes some placeholders that you will replace later, after you have the required information. Don’t test the Lambda function at this time because it will fail because of the presence of the placeholders.
Create a user with administrative access
Amazon SageMaker Unified Studio supports two distinct domain types: AWS IAM Identity Center based domains and IAM based domains. At the time of writing this post, only IAM Identity Center based domains support multi-accounts association, therefore in this post you work with this type of domain that requires IAM Identity Center.
In the governance account, you enable IAM Identity Center and create an administrative user to create and manage the Amazon SageMaker Unified Studio domain. Create a user with administrative access:
Enable IAM Identity Center in the governance account. For instructions, refer to Enable IAM Identity Center.
To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user. For help signing in using an IAM Identity Center user, refer to Sign in to your AWS access portal.
After your domain is created, you can navigate to the Amazon SageMaker Unified Studio portal (a browser-based web application) where you can use your data and configured tools for analytics and AI. Save the Amazon SageMaker Unified Studio portal URL because you will use this URL later.
Solution steps
Now that you have the prerequisites in place, you can complete the following ten high-level steps to implement the solution.
Associate the producer and consumer accounts to the Amazon SageMaker Unified Studio domain
Start by associating the producer and consumer accounts to the newly created Amazon SageMaker Unified Studio domain. When you associate your producer and consumer accounts to the domain, make sure to select IAM users and roles can access APIs and IAM users can log in to Amazon SageMaker Unified Studio in the AWS RAM share managed permission section. For step-by-step instructions, refer to Associated accounts in Amazon SageMaker Unified Studio. If your AWS accounts are part of the same organization, your association requests are automatically accepted. However, if your AWS accounts aren’t part of the same organization, request association with the other AWS accounts in the governance account and then accept the association request in both the producer and consumer accounts.
Create two project profiles
Now, create two project profiles, one for the producer project and one for the consumer project.
In Amazon SageMaker Unified Studio, a project profile defines an uber template for projects in your Amazon SageMaker domain. A project profile is a collection of blueprints that provides reusable AWS CloudFormation templates used to create project resources.
A project profile is associated to a specific AWS account. This means, when a project is created the blueprints listed in the project profile are deployed in the associated AWS account. To use a project profile, you must enable its blueprints in the AWS account associated to the project profile.
Create the producer project profile
You’re going to create the producer project profile that is associated to the producer account. This project profile will be used to create the producer project. This profile includes by default the Tooling blueprint that creates resources for the project, including IAM user roles and security groups.
Before creating the project profile, you will enable the Tooling blueprint in the producer account using the following procedure:
Select the domain you created as part of prerequisites.
Under the Project profiles tab, choose Create and enter the following information:
For Project profile name, enter producer-project-profile.
For Project profile creation options, select Custom create.
DO NOT SELECT A BLUEPRINT for Blueprints because the Tooling blueprint is included by default in any project profile.
For Account, select Provide an account ID.
For Account ID, enter the producer account ID.
For Region, select Provide region name and then select the Region in which you’re working.
For Authorization, select Allow all users and groups.
For Project profile readiness, select Enable project profile on creation.
Choose Create project profile.
Create a consumer project profile
You also create a consumer project profile and associate it to the consumer account. This profile will be used to create the consumer project. The consumer project profile includes the LakeHouseDatabase blueprint, which is needed to create a lakehouse environment with an AWS Glue database for data management and an Amazon Athena workgroup for querying. The Tooling blueprint is included by default in the project profile.
Before creating the project profile, enable the Tooling and LakeHouseDatabase blueprints in the consumer account:
Select the domain you created as part of prerequisites.
Under Project profiles tab choose Create and enter the following information:
For Project profile name, enter consumer-project-profile.
For Project profile creation options, select Custom create.
For Blueprints, select LakeHouseDatabase.
For Account, select Provide an account ID.
For Account ID, enter the consumer account ID.
For Region, select Provide region name and then select the Region you are working.
For Authorization, select Allow all users and groups.
For Project profile readiness, select Enable project profile on creation.
Choose Create project profile.
Create SageMaker Unified Studio producer and consumer projects
In Amazon SageMaker Unified Studio, a project is a boundary within a domain where you can collaborate with other users to work on a business use case. In projects, you can create and share data and resources.To create producer and consumer projects in Amazon SageMaker Unified Studio use the following instructions:
Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list.
Choose Create project and enter the following information:
For Project name, enter Producer.
For Project profile, select producer-project-profile.
Choose Continue.
Choose Continue.
Choose Create project.
After you’ve created the Producer project, note in a text file the Project role ARN that is displayed in the Project overview. The following image is shown for reference. The project role name is the string that follows arn:aws:iam::<account_ID>:role/ in the project role Amazon Resource Name (ARN). You will use both project role name and ARN later.
Repeat the preceding procedure to create the Consumer project. Be sure to enter Consumer for Project name and then select consumer-project-profile for Project profile. After it’s created, note the Project role ARN in a text file. The project role name is the string that follows arn:aws:iam::<account_ID>:role/ in the project role ARN. You will use both project role name and ARN later.
Bring your own data from the producer account
Bring your own data to the Amazon SageMaker Unified Studio Producer project. AWS provides several options to achieve this onboarding. The first option is automated onboarding in Amazon SageMaker lakehouse, in which you ingest the Amazon SageMaker lakehouse metadata of datasets into Amazon SageMaker Catalog. With this option, you can onboard your Amazon SageMaker lakehouse data as part of creating a new Amazon SageMaker Unified Studio domain or for an existing domain.
For more information about automated onboarding of Amazon SageMaker lakehouse data, refer to Onboarding data in Amazon SageMaker Unified Studio. As other options, you can bring in existing resources to your Amazon SageMaker Unified Studio project by using the Data and Compute pages in your project, or by using scripts provided in GitHub. For more information about using the Data and Compute pages or about using scripts, refer to Bringing existing resources into Amazon SageMaker Unified Studio. In this post, you will use Amazon SageMaker lakehouse capabilities to import your trees AWS Glue table into the Producer project.
Register the Amazon S3 location for the table
To use Lake Formation permissions for fine-grained access control to the trees table, you need to register in Lake Formation the Amazon S3 location of the trees table. To do that, complete the following actions:
In the navigation pane under Administration, choose Data lake locations.
Choose Register location and enter the following information:
For S3 URI, enter s3://<bucket-name>/<prefix>/ where <bucket-name> is the name of the S3 bucket you created in the prerequisites and <prefix> is the optional prefix for the trees.csv file you uploaded as part of the prerequisite.
For IAM role, select AWSServiceRoleForLakeFormationDataAccess.
For Permission mode, select Lake Formation.
Choose Register location.
Grant Producer project role permissions on the database
Grant database access to the IAM role that is associated with your Producer project. This role is called the project role, and it was created in IAM upon project creation.
To access the AWS Glue Data Catalog collections database from the Producer project in the Amazon SageMaker Unified Studio, complete the following actions:
In the navigation pane under Data Catalog, choose Databases.
Choose the collections database.
From the Actions menu, choose Grant and enter the following information:
For IAM users and roles, select your Producer project’s role name. This is the string starting with datazone_usr_role_ that is part of the Producer project role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
For Database permissions, select Describe.
Choose Grant.
Grant Producer project role permissions on the table
Grant trees table access to the IAM role that is associated with your Producer project. To grant these permissions use the following instructions:
In the navigation pane under Data Catalog, choose Tables and MVs.
Select the trees table.
From the Actions menu, choose Grant and enter the following information:
For IAM users and roles, select your Producer project’s role. This is the string starting with datazone_usr_role_ that is part of the Producerproject role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
For Table permissions, select Select and Describe.
For Grantable permissions, select Select and Describe.
Choose Grant.
Revoke any existing permissions of IAMAllowedPrincipals
You must revoke the IAMAllowedPrincipals group permissions on both the database and table to enforce Lake Formation permission for access. For more information, refer to Revoking permission using the Lake Formation console.
In the navigation pane under Permission, choose Data permissions.
Select the entries where Principal is set to IAMAllowedPrincipals and Resource is set to collections or trees as in the following image:
Choose Revoke.
Enter revoke.
Choose Revoke again.
Verify that data is available in the Producer project
Verify that your collections database and trees table are accessible in the Producer project:
Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project drop-down menu and choose the Producer project.
In the navigation pane under Overview, choose Data.
Choose Lakehouse.
Choose AwsDataCatalog.
Choose collections.
Choose tables.
Choose the three-dot action menu next to your trees table and choose Preview data, as shown in the following image.
You’ll find data from the trees table as shown in the following image.
Create Amazon SageMaker Catalog asset
Even if it’s accessible in the project, to work with the trees table in Amazon SageMaker Catalog, you need to register the data source and create an Amazon SageMaker Catalog asset:
Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose the Producer project.
On the project page, under Project catalog in the navigation pane, choose Data sources.
Choose Create Data Source and make the following selections:
For Name, enter collections.
For Data source type, select AWS Glue (Lakehouse).
For Database name, select collections.
Choose Next.
Choose Next.
Choose Next.
Choose Create.
After the data source is created, you will be in the collections data source page, choose Run. This will import metadata and create the Amazon SageMaker Catalog asset.
In the collections data source, on the Data source runs tab, you’ll find your run marked as Completed and the trees asset Successfully created, as shown in the following image:
Publish the data asset in the Amazon SageMaker Catalog
Publishing a data asset manually is a one-time operation that you need to perform to allow others to access the data asset through the catalog:
Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose the Producer project.
On the project page under Project catalog, choose Assets.
Select your trees data asset that is available on the Inventory tab. The following image is shown for reference.
(Optional) If automated metadata generation is enabled when the data source is created, metadata for assets (such as the asset business name) is available to review and accept or reject. You can either choose Accept All or Reject All in the Automated Metadata Generation banner.
Choose Publish Asset. The following image is shown for reference.
Choose Publish Asset.
Subscribe to the data asset in the Amazon SageMaker Catalog
To consume data assets in the Consumer project, subscribe to the data asset by creating a subscription request:
Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose Consumer project.
On the Discover menu, choose Catalog.
Enter trees in the search box and then select the data asset returned from the search. If in step 7 “Publish the data asset in the Amazon SageMaker Catalog” you chose Accept All in the Automated Metadata Generation banner, your data asset will have a different business name generated by the automated metadata recommendations feature. The data asset technical name is trees. For reference, refer to the following image.
Choose Subscribe.
For Comment, enter a justification such as This data asset is needed for model training purposes.
Choose Subscribe again.
By default, asset subscription requests require manual approval by a data owner. However, if the requester in the Consumer project is also a member of the Producer project, the subscription request is automatically approved. For information about approving subscription requests, refer to Approve or reject a subscription request in Amazon SageMaker Unified Studio.
Configure your Lambda IAM role to access the subscribed data access
To enable your Lambda function access to the subscribed data asset, you need to allow the Lambda function to assume the Consumer project role. To do this, edit the Consumer project’s IAM role trust relationship:
Navigate to the IAM console in the consumer account.
In the navigation pane under Access management, choose Roles.
Select the Consumer project’s IAM role. This is the string starting with datazone_usr_role_ that is part of the Consumer project role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
Under the Trust relationships tab, choose Edit trust policy.
For backup reasons, make a copy of the existing trust policy in a text file.
In the Edit trust policy window, add the following statement to the existing trust policy without removing or overwriting other existing statements in the trust policy. Be sure to replace the placeholder <account_id> with your consumer AWS account ID.
Test the Lambda function’s access to the subscribed data asset
Before you can test your Lambda function, you need to replace placeholders in the function code and in the IAM policy. There are three placeholders to be replaced: <role_arn>, <database_name> and <workgroup_id>. For <role_arn>, you already have the actual value, which is the Consumer project’s role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”. The next sections provide instructions to retrieve values for the other placeholders.
Retrieve the AWS Glue Data Catalog database name
You need to find the name of the AWS Glue Data Catalog database that was created along with the Consumer project. You will then use this value to replace the <database_name> placeholder in the consumer_function Lambda function code. To retrieve the AWS Glue Data Catalog database name, follow these instructions:
Access the Amazon SageMaker Unified Studio portal.
Choose the Select a project dropdown list and choose Consumer project.
On the project page, under Overview, choose Data.
Choose Lakehouse.
Choose AwsDataCatalog.
Copy the name of the database. It should be an alphanumerical string starting with glue_db, as in the following image:
Copy the Workgroup ARN and save to a text file. The Athena workgroup ID is the string that follows arn:aws:athena:<region>:<account_ID>:workgroup/ in the Workgroup ARN.
Replace placeholder in the smus_consumer_athena_execution IAM policy
To replace the <workgroup_id> placeholder in the smus_consumer_athena_execution IAM policy, use the following procedure:
In the search field enter smus_consumer_athena_execution.
Select the smus_consumer_athena_execution policy.
Choose Edit.
Replace <workgroup_id> with the value you noted earlier.
Choose Next.
Choose Save changes.
Replace placeholders in the Lambda function code and test it
In this section, you will replace the <role_arn>, <database_name> and <workgroup_id> placeholders in the consumer_function Lambda function code, and then you can test the function ability to access data of the trees table.
After increasing the timeout, test the function again.
Clean up
If you no longer need the resources you created as you followed this post, delete them to prevent incurring additional charges. Start by deleting your Amazon SageMaker Unified Studio domain in the governance account. For more information, refer to Delete domains.
To remove the AWS Glue collections database from the producer account, follow these steps:
In the navigation pane under Data Catalog, choose Databases.
Select the collections database.
Choose Delete.
Choose Delete.
To remove the S3 bucket from the producer account, empty the bucket and then you can delete the bucket. For information about emptying the bucket, refer to Emptying a general purpose bucket. For information about deleting the bucket, refer to Deleting a general purpose bucket.
To remove the Lambda function from the consumer account, follow these steps:
Choose the Actions menu and then choose Delete function.
Enter confirm.
Choose Delete.
To complete the cleanup, delete the IAM role named smus_consumer_lambda, then delete the IAM policy named smus_consumer_athena_execution in the consumer account. For information about removing a IAM role, refer to Delete roles or instance profiles. For information about removing an IAM policy, refer to Delete IAM policies.
Conclusion
In this post, we covered adopting Amazon SageMaker Catalog for data governance without rearchitecting your existing applications and data repositories. We walked through how to onboard existing data in Amazon SageMaker Unified Studio, then publish it in a catalog, and then subscribe and consume the data from resources deployed outside the context of an Amazon SageMaker Unified Studio project. This solution can help you accelerate your implementation of a data mesh pattern with Amazon SageMaker Catalog to publish, find, and access data securely in your organization.
Amazon SageMaker Unified Studio now offers two domain configurations: Amazon SageMaker Unified Studio Identity Center(IDC)-based domains with comprehensive governance features, and Amazon SageMaker Unified Studio IAM-based domains with enhanced developer productivity tools.
In this post, we demonstrate how you can use both of these domain configurations of Amazon SageMaker Unified Studio using AWS Identity and Access Management (IAM) role reuse and attribute-based access control.
How authentication works in each configuration
Amazon SageMaker Unified Studio IDC-based domains authenticate users through AWS Identity and Access Management (IAM) Identity Center with Single Sign-On, preserving individual user identities throughout their sessions. These domains excel in governance with identity-based authorization, fine-grained access controls between users, and comprehensive catalog management featuring formal Publisher/Subscriber (Pub/Sub) data sharing workflows with approval processes—ideal for enterprise environments requiring strong identity management, compliance tracking, and identity-based audit trails.
Amazon SageMaker Unified Studio IAM-based domains authenticate through federated AWS Identity and Access Management (IAM) roles where all users accessing a project share the same role permissions. These domains prioritize developer productivity with modern tools including new serverless Notebooks, Athena Spark integration, the improved interface with vertical navigation, and built-in AI assistance, designed for development teams that need streamlined access and advanced analytics capabilities.
This solution facilitates organizations that are already using IDC-based domains to preserve their existing governance frameworks established in IDC-based domains while unlocking modern development capabilities for their teams through IAM-based domains. If you prefer to use the newly launched IAM-based domains, you can continue to do as well. The choice depends on your company’s needs.
Imagine a data steward (Sam) uses the IDC-based domain to define data access policies, manage the data catalog, and approve subscription requests to verify compliance and proper data governance.
On the other hand, a data engineer (Sarah), wants to use IDC-based domain for governance features such as SageMaker catalog and IAM-based domain for the new serverless Notebook to build data pipelines, perform advanced analytics, and accelerate development cycles. Sarah will request access to the data through IDC-based domain, and once access is approved by Sam, Sarah can access this data in serverless notebook available in IAM-based domain.
Solution overview
The integration leverages IAM role reuse, AWS Lake Formation Attribute-Based Access Control (ABAC) and Amazon SageMaker Catalog pub-sub model to automatically carry permissions from the IDC-based domain to the new IAM-based domain. When properly configured, data subscriptions managed through the IDC-based domain’s Pub/Sub model become immediately accessible in IAM-based domain projects, providing a unified data access experience.
The solution we will implement in the post involves creating an IAM-based domain project that is similar to your IDC consumer project (eg same team members, use case) , configuring execution roles, and enabling role reuse. This approach maintains the familiar subscription workflow while extending benefits to the IAM-based domain.The following diagram shows the high-level architecture of how this approach works.
The solution architecture consists of:
Existing IDC-based domain: Contains producer and consumer projects with established data sharing via Pub/Sub model
IAM-based domain: New projects with federated and execution roles configured for modern development tools
IAM Identity Center: Manages federated access and permission sets
The solution provides 2 options: Option 1: IDC-Based Domain project role reuse provides the simplest integration path by directly reusing the existing consumer project IAM role from your IDC-based domain as the execution role in the IAM-based domain. The primary benefits include simplified setup requiring only policy changes (covered later in the blog), reduced administrative overhead with one less role to manage and lower risk of misconfiguration since you’re leveraging proven, existing roles. Choose Option 1 when you want the fastest implementation path, your organization prefers minimal role proliferation, you have well-established IDC-based domain roles that already have data access permissions, or your team has limited IAM expertise and wants to avoid complex tagging configurations.
Option 2: Creating a new execution role for the IAM-based domain project and use attribute-based access control (ABAC) through tagging with the IDC-based domain project ID. The key benefits include enhanced auditability with two distinct roles (one for IDC-based domain, one for IAM-based domain), clear separation showing which domain generated each request in CloudTrail logs, greater flexibility to customize permissions specific to IAM-based domain needs without affecting IDC-based domain operations, and better security isolation between the two domain types. The `AmazonDatazoneProject` tag enables attribute based access control, while maintaining distinct role identities. Choose Option 2 when: your organization requires detailed audit trails distinguishing between domain types, compliance policies mandate separation of concerns between governance and development environments, you want to track and attribute costs separately for each domain, or you need to provide evidence showing which domain (governance vs. development) accessed specific data resources for compliance reporting.
Here is the high-level view of how the identity and domain entities map to each other for both options:
For this demonstration, we use a simplified setup with a sales producer project and a marketing consumer project that subscribes to these tables.
Understanding the current IDC-based domain setup
Our starting point includes a well-established Amazon SageMaker Unified Studio IDC-based domain structure:
Sales Producer Project
Contains a database with pipeline and sales tables
Managed by Sam, the data steward who creates and publishes data assets
Has its own project IAM role
Marketing Consumer Project
Managed by Sarah, the data engineer who subscribes to published data via IDC domain project
Has its own project IAM role
Successfully queries subscribed data through the IDC-based domain interface
Each project has an associated IAM role that governs access to data assets, and the Pub/Sub model manages subscription workflows and permissions.
Setting up federated role through permission sets
Federated roles through permission sets are used to authenticate and provide users with console access to IAM-based domains through AWS IAM Identity Center, where all users within a project share the same role permissions. When you assign a permission set, IAM Identity Center creates corresponding IAM Identity Center-controlled IAM role in AWS account, and attaches the policies specified in the permission set to that role.
IAM-based SMUS domains enable streamlined access to modern development tools (serverless Notebooks, Athena Spark, AI assistance) while maintaining governance, automatically propagating permissions across domains without requiring duplicate access approvals, and simplifying team member onboarding.You can use any IAM role to access IAM-based domain. For this post, we will use federated role option using AWS IAM Identity Center (IDC).
Grant access to Data engineer group for IAM-based domains in Identity Center
1) Set up federated role in AWS IAM Identity Center
Navigate to IAM Identity Center (IDC) in the AWS Management Console, then complete the following steps:
Go to permission set section in IDC. Create a new permission set called Marketing-federated-role and select Attach Policy.
Search for SageMakerStudioUserIAMConsolePolicy in the existing policy name from list and select SageMakerStudioUserIAMConsolePolicy from the list. Note that the managed policy SageMakerStudioUserIAMConsolePolicy must be attached or have the same permissions added via another policy to be able to access projects in a SageMaker IAM domain.
Go to the AWS account section of IDC.
Assign the created permission set to your AWS account.
For this post we assigned the permission set to marketing group, As a best practice, you should setup and grant access to groups rather than individual users.
Add Sarah to marketing group.
This creates a federated role that Sarah can use to access the IAM-based domain. The federated role appears as an IAM role within your account and serves as the entry point for console access.
Setting up IAM-based domain execution role
There are 2 options to setup execution role for IAM-based domain project. The execution role has a one-to-one mapping with the federated role.
Option 1 – IDC-based domain Project Role reuse
Instead of creating a new execution role and tagging it, you can configure the IAM-based domain project to directly reuse the consumer project IAM role from the IDC-based domain as the execution role. This option only needs policy changes to the consumer project IAM role. To find the IDC-based domain consumer project IAM role:
Navigate to the Amazon SageMaker Unified Studio IDC-based domain portal.
Open the Marketing Consumer Project.
Copy the project role ARN from the project overview page.
You will need to modify this execution role’s policy with detailed instructions provided later in the blog.
Setting up IAM-based domain project for option 1
To create an IAM-based domain project that will integrate with your existing IDC-based domain permissions, complete the following steps:
Log in to the AWS Console using IAM-based domain administrator.
Navigate to Amazon SageMaker page within console.
Choose Open.
Once logged in to IAM-based domain as admin, choose Manage projects.
Next, click on Create Project.
Enter project name as “Marketing Consumer Project”.
During project creation, select the following crucial roles and then choose Create Project:
Project IAM Role: The marketing federated role created in IAM Identity Center above. This is the role in the member account that has a role name with suffix AWSReservedSSO.
Project Role: – Choose project role for data engineer, copied from option 1.
Make policy changes to this project role as per the instruction on the SMUS UI page.
Option 2 – Bring your own execution role.
To create an IAM-based domain project that will integrate with your existing IDC-based domain permissions., you must tag the execution role for permission propagation. Amazon SageMaker Catalog and AWS Lake Formation use attribute-based access control, which means permissions can be inherited based on resource tags. For this option, you will need consumer project ID.To find the IDC-based domain consumer project ID:
Navigate to the Amazon SageMaker Unified Studio IDC-based domain portal.
Open the Marketing Consumer Project.
Copy the project ID from the project details.
Federated Role: The marketing federated role created in IAM Identity Center above.
Execution Role: – Choose execution role from option 2.
Make policy changes to this execution role as per the instruction.
Next, navigate to the IAM console and locate the execution role created for your IAM-based domain consumer project.
Add the following tag, this step relies on ABAC policies with projectId for subscriptions.
Key: AmazonDatazoneProject
Value: The project ID from your Amazon SageMaker Unified Studio IDC-based domain consumer project
This tag configuration results in data access grant from IDC-based domain consumer project to the IAM-based domain project execution role.
Verify data access in the IAM-based domain
After tagging the execution role, verify that permissions are set up correctly.Complete the following steps:
Use the SSO URL to log into the SSO Identity Center as Sarah.
Open the AWS console using federated role created earlier in setting federated role section.
Navigate to Amazon SageMaker.
Choose Amazon SageMaker Unified Studio IAM-based domain option (this will show up if project is already created with federated role).
In the Amazon SageMaker Unified Studio IAM-based domain project, navigate to the Data tab. If you created 2 projects with both option 1 and option 2 execution role, then 2 projects will show up and you can login to either to validate data access.
Verify that the consumer database and subscribed tables appear.
Create and use the new serverless notebooks
With permissions properly configured, you can now use IAM-based domain capabilities like serverless Notebooks. Complete the following steps:
In the Amazon SageMaker Unified Studio IAM-based domain project, select a table from the Data tab.
Choose Create notebook.
The Notebook opens with Athena SQL as the default cell type.
Write and run queries against your subscribed data.
The notebook runs with the execution role’s permissions, which now include access to all data subscribed through the IDC-based domain.
Key benefits of this integration
This integration approach delivers several important advantages:
Preserve existing investments
Continue using IDC-based domain governance and catalogs.
Maintain established Pub/Sub workflows.
No migration required for existing data assets.
Get modern capabilities
Provide developers with the new serverless Notebooks.
Single subscription workflow manages access across both domains.
Consistent data access via role reuse and attribute-based access control.
No duplicate access requests or approvals needed.
Unified data experience
Developers access all subscribed data from one interface.
Consistent data catalog across domains.
Simplified onboarding for new team members.
Cleanup
Complete the following steps to delete the resources you created:
Delete the serverless Notebooks created in the IAM-based domain projects.
Delete the IAM-based domain projects (Marketing Consumer Project and Marketing Consumer Project 2).
Remove the permission set assignment from marketing group in IAM Identity Center.
Delete the Marketing-federated-role permission set in IAM Identity Center.
Remove the tags (AmazonDatazoneProject) from the execution role (if using Option 2).
Delete the execution role created for the IAM-based domain (if using Option 2 and not reusing the IDC-based domain project role).
Revert any policy changes made to the IDC-based domain consumer project IAM role (if using Option 1).
If you do not need the IAM-based domain anymore, delete it.
If you created any test data subscriptions in the IDC-based domain, remove them.
Conclusion
In this post, we demonstrated how to access Amazon SageMaker Unified Studio IDC-based domain with the new IAM-based domain using role reuse and attribute-based access control. This setup offers data engineers the best of both worlds: access to specialized modern development tools—including the new serverless Notebooks, Athena Spark integration, and built-in AI assistance , while maintaining proper governance that includes comprehensive catalog management and robust security controls established in the IDC-based domain.You can now confidently adopt Amazon SageMaker Unified Studio IAM-based domain capabilities knowing their established data governance, subscription workflows, and access controls remain intact and continue to function as expected.
Ready to get started with Amazon SageMaker Unified Studio and unlock the power of integrated governance and modern development tools for your organization? Visit the Amazon SageMaker Unified Studio documentation to learn more and begin your implementation today.
Amazon SageMaker Unified Studio serves as a collaborative workspace where data engineers and scientists can work together on end-to-end data and machine learning (ML) workflows. SageMaker Unified Studio specializes in orchestrating complex data workflows across multiple AWS services through its integration with Amazon Managed Workflows for Apache Airflow (Amazon MWAA). Project owners can create shared environments where team members jointly develop and deploy workflows, while maintaining oversight of pipeline execution. This unified approach makes sure data pipelines run consistently and efficiently, with clear visibility into the entire process, making it seamless for teams to collaborate on sophisticated data and ML projects.
This post explores how to build and manage a comprehensive extract, transform, and load (ETL) pipeline using SageMaker Unified Studio workflows through a code-based approach. We demonstrate how to use a single, integrated interface to handle all aspects of data processing, from preparation to orchestration, by using AWS services including Amazon EMR, AWS Glue, Amazon Redshift, and Amazon MWAA. This solution streamlines the data pipeline through a single UI.
Example use case: Customer behavior analysis for an ecommerce platform
Let’s consider a real-world scenario: An e-commerce company wants to analyze customer transactions data to create a customer summary report. They have data coming from multiple sources:
Customer profile data stored in CSV files
Transaction history in JSON format
Website clickstream data in semi-structured log files
The company wants to do the following:
Extract data from these sources
Clean and transform the data
Perform quality checks
Load the processed data into a data warehouse
Schedule this pipeline to run daily
Solution overview
The following diagram illustrates the architecture that you implement in this post.
The workflow consists of the following steps:
Establish a data repository by creating an Amazon Simple Storage Service (Amazon S3) bucket with an organized folder structure for customer data, transaction history, and clickstream logs, and configure access policies for seamless integration with SageMaker Unified Studio.
Extract data from the S3 bucket using AWS Glue jobs.
Use AWS Glue and Amazon EMR Serverless to clean and transform the data.
Create and manage the workflow environment using SageMaker Unified Studio with Identity Center–based domains.
Note: Amazon SageMaker Unified Studio supports two domain configuration models: IAM Identity Center (IdC)–based domains and IAM role–based domains. While IAM-based domains enable role-driven access management and visual workflows, this post specifically focuses on Identity Center–based domains, where users authenticate via IdC and projects access data and resources using project roles and identity-based authorization.
Prerequisites
Before beginning, ensure you have the following resources:
This solution requires SageMaker Unified Studio domain in the us-east-1 AWS Region. Although SageMaker Unified Studio is available in multiple Regions, this post uses us-east-1 for consistency. For a complete list of supported Regions, refer to Regions where Amazon SageMaker Unified Studio is supported.
Complete the following steps to configure your domain:
Sign in to the AWS Management Console, navigate to Amazon SageMaker, and open the Domains section from the left navigation pane.
On the SageMaker console, choose Create domain, then choose Quick setup.
If the message “No VPC has been specifically set up for use with Amazon SageMaker Unified Studio” appears, select Create VPC. The process redirects to an AWS CloudFormation stack. Leave all settings at their default values and select Create stack.
Under Quick setup settings, for Name, enter a domain name (for example, etl-ecommerce-blog-demo). Review the selected configurations.
Choose Continue to proceed.
On the Create IAM Identity Center user page, create an SSO user (account with IAM Identity Center) or select an existing SSO user to log in to the Amazon SageMaker Unified Studio. The SSO selected here is used as the administrator in the Amazon SageMaker Unified Studio.
After you have created a domain, popup will appear with the message: “Your domain has been created! You can now log in to Amazon SageMaker Unified Studio”. You can close the popup for now.
Create a project
In this section, we create a project to serve as a collaborative workspace for teams to work on business use cases. Complete the following steps:
Choose Open Unified Studio and sign in with your SSO credentials using the Sign in with SSO option.
Choose Create project.
Name the project (for example, ETL-Pipeline-Demo) and create it using the All capabilities project profile.
Choose Continue.
Keep the default values for the configuration parameters and choose Continue.
Choose Create project.
Project creation might take a few minutes. After the project is created, the environment will be configured for data access and processing.
Integrate S3 bucket with SageMaker Unified Studio
To enable external data processing within SageMaker Unified Studio, configure integration with an S3 bucket. This section walks through the steps to set up the S3 bucket, configure permissions, and integrate it with the project.
Create and configure S3 bucket
Complete the following steps to create your bucket:
In a new browser tab, open the AWS Management Console and search for S3.
Create the following folder structure in the bucket. For detailed instructions, see Creating a folder:
raw/customers/
raw/transactions/
raw/clickstream/
processed/
analytics/
Upload sample data
In this section, we upload sample ecommerce data that represents a typical business scenario where customer behavior, transaction history, and website interactions need to be analyzed together.
The raw/customers/customers.csv file contains customer profile information, including registration details. This structured data will be processed first to establish the customer dimension for our analytics.
The raw/transactions/transactions.json file contains purchase transactions with nested product arrays. This semi-structured data will be flattened and joined with customer data to analyze purchasing patterns and customer lifetime value.
The raw/clickstream/clickstream.csv file captures user website interactions and behavior patterns. This time-series data will be processed to understand customer journey and conversion funnel analytics.
For detailed instructions on uploading files to Amazon S3, refer to the Uploading objects.
Configure CORS policy
To allow access from the SageMaker Unified Studio domain portal, update the Cross-Origin Resource Sharing (CORS) configuration of the bucket:
On the bucket’s Permissions tab, choose Edit under Cross-origin resource sharing (CORS).
Enter the following CORS policy and replace domainUrl with the SageMaker Unified Studio domain URL (for example, https://<domain-id>.sagemaker.us-east-1.on.aws ). The URL can be found at the top of the domain details page on the SageMaker Unified Studio console.
To enable SageMaker Unified Studio to access the external Amazon S3 location, the corresponding AWS Identity and Access Management (IAM) project role must be updated with the required permissions. Complete the following steps:
On the IAM console, choose Roles in the navigation pane.
Search for the project role using the last segment of the project role Amazon Resource Name (ARN). This information is located on the Project overview page in SageMaker Unified Studio (for example, datazone_usr_role_1a2b3c45de6789_abcd1efghij2kl).
Choose the project role to open the role details page.
On the Permissions tab, choose Add permissions, then choose Create inline policy.
Use the JSON editor to create a policy that grants the project role access to the Amazon S3 location
In the JSON policy below, replace the placeholder values with your actual environment details:
Replace <BUCKET_PREFIX> with the prefix of S3 bucket name (for example, ecommerce-raw-layer)
Replace <AWS_REGION> with the AWS Region where your AWS Glue Data Quality rulesets are created (for example, us-east-1)
Replace <AWS_ACCOUNT_ID> with your AWS account ID
Paste the updated JSON policy into the JSON editor.
Enter a name for the policy (for example, etl-rawlayer-access), then choose Create policy.
Choose Add permissions again, then choose Create inline policy.
In the JSON editor, create a second policy to manage S3 Access Grants:Replace <BUCKET_PREFIX> with the prefix of S3 bucket name (for example, ecommerce-raw-layer) and paste this JSON policy.
After you add policies to the project role for access to the Amazon S3 resources, complete the following steps to integrate the S3 bucket with the SageMaker Unified Studio project:
In SageMaker Unified Studio, open the project you created under Your projects.
Choose Data in the navigation pane.
Select Add and then Add S3 location.
Configure the S3 location:
For Name, enter a descriptive name (for example, E-commerce_Raw_Data).
For S3 URI, enter your bucket URI (for example, s3://ecommerce-raw-layer-bucket-demo-<Account-ID>-us-east-1/).
For AWS Region, enter your Region (for this example, us-east-1).
Leave Access role ARN blank.
Click Add S3 Location
Wait for the integration to complete.
Verify the S3 location appears in your project’s data catalog (on the Project overview page, on the Data tab, locate the Buckets pane to view the buckets and folders).
This process connects your S3 bucket to SageMaker Unified Studio, making your data ready for analysis.
Create notebook for job scripts
Before you can create the data processing jobs, you must set up a notebook to develop the scripts that will generate and process your data. Complete the following steps:
In SageMaker Unified Studio, on the top menu, under Build, choose JupyterLab.
Choose Configure Space and choose the instance type ml.t3.xlarge. This makes sure your JupyterLab instance has at least 4 vCPUs and 4 GiB of memory.
Choose Configureand Start Space or Save and Restart to launch your environment.
Wait a few moments for the instance to be ready.
Choose File, New, and Notebook to create a new notebook.
Set Kernel as Python 3, Connection type as PySpark, and Compute as Project.spark.compatibility.
In the notebook, enter the following script to use later for your AWS Glue job. This script processes raw data from three sources in the S3 data lake, standardizes dates, and converts data types before saving the cleaned data in Parquet format for optimal storage and querying.
Replace <Bucket-Name> with the name of actual S3 bucket in script:
This script processes customer, transaction, and clickstream data from the raw layer in Amazon S3 and saves it as Parquet files in the processed layer.
Choose File, Save Notebook As, and save the file as shared/etl_initial_processing_job.ipynb.
Create notebook for AWS Glue Data Quality
After you create the initial data processing script, the next step is to set up a notebook to perform data quality checks using AWS Glue. These checks help validate the integrity and completeness of your data before further processing. Complete the following steps:
Choose File, New, and Notebook to create a new notebook.
Set Kernel as Python 3, Connection type as PySpark, and Compute as Project.spark.compatibility.
In this new notebook, add the data quality check script using the AWS Glue EvaluateDataQuality method. Replace <Bucket-Name> with the name of actual S3 bucket in script:
from datetime import datetime
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsgluedq.transforms import EvaluateDataQuality
from awsglue.transforms import SelectFromCollection
# ---------------- Glue setup ----------------
sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
job = Job(glueContext)
job.init("GlueDQJob", {})
# ---------------- Constants ----------------
RUN_DATE = datetime.utcnow().strftime("%Y-%m-%d")
year, month, day = RUN_DATE.split("-")
OUTPUT_PATH = "s3://<Bucket-Name>/data-quality-results"
# ---------------- Tables and Rules ----------------
tables = {
"customers": ["s3://<Bucket-Name>/processed/customers/",
["IsComplete \"customer_id\"", "IsUnique \"customer_id\"", "IsComplete \"email\""]],
"transactions": ["s3://<Bucket-Name>/processed/transactions/",
["IsComplete \"transaction_id\"", "IsUnique \"transaction_id\""]],
"clickstream": ["s3://<Bucket-Name>/processed/clickstream/",
["IsComplete \"customer_id\"", "IsComplete \"action\""]]
}
# ---------------- Process Each Table ----------------
for table, (path, rules) in tables.items():
df = glueContext.create_dynamic_frame.from_options("s3", {"paths":[path]}, "parquet")
results = EvaluateDataQuality().process_rows(
frame=df,
ruleset=f"Rules = [{', '.join(rules)}]",
publishing_options={"dataQualityEvaluationContext": table}
)
rows = SelectFromCollection.apply(results, key="rowLevelOutcomes", transformation_ctx="rows").toDF()
rows = rows.drop("DataQualityRulesPass", "DataQualityRulesFail", "DataQualityRulesSkip")
# Write passed/failed rows
for status, colval in [("pass","Passed"), ("fail","Failed")]:
tmp = rows.filter(rows.DataQualityEvaluationResult.contains(colval))
if tmp.count() > 0:
tmp.write.mode("append").parquet(
f"{OUTPUT_PATH}/{table}/status=dq_{status}/Year={year}/Month={month}/Date={day}"
)
print("Data Quality checks completed and written to S3")
job.commit()
Choose File, Save Notebook As, and save the file as shared/etl_data_quality_job.ipynb.
Create and test AWS Glue jobs
Jobs in SageMaker Unified Studio enable scalable, flexible ETL pipelines using AWS Glue. This section walks through creating and testing data processing jobs for efficient and governed data transformation.
Create initial data processing job
This job performs the first processing job in the ETL pipeline, transforming raw customer, transaction, and clickstream data and writing the cleaned output to Amazon S3 in Parquet format. Complete the following steps to create the job:
In SageMaker Unified Studio, go to your project.
On the top menu, choose Build, and under Data Analysis & Integration, choose Data processing jobs.
Choose Create job from notebooks.
Under Choose project files, choose Browse files.
Locate and select etl_initial_processing_job.ipynb (the notebook saved earlier in JupyterLab), then choose Select and Next.
Configure the job settings:
For Name, enter a name (for example, job-1).
For Description, enter a description (for example, Initial ETL job for customer data processing).
For IAM Role, choose the project role (default).
For Type, choose Spark.
For AWS Glue version, use version 5.0.
For Language, choose Python.
For Worker type, use G.1X.
For Number of Instances, set to 10.
For Number of retries, set to 0.
For Job timeout, set to 480.
For Compute connection, choose project.spark.compatibility.
Under Advanced settings, turn on Continuous logging.
Leave the remaining settings as default, then choose Submit.
After the job is created, a confirmation message will appear indicating that job-1 was created successfully.
Create AWS Glue Data Quality job
This job runs data quality checks on the transformed datasets using AWS Glue Data Quality. Rulesets validate completeness and uniqueness for key fields. Complete the following steps to create the job:
In SageMaker Unified Studio, go to your project.
On the top menu, choose Build, and under Data Analysis & Integration, choose Data processing jobs.
Choose Create job, Code-based job, and Create job from files.
Under Choose project files, choose Browse files.
Locate and select etl_glue_data_quality.ipynb, then choose Select and Next.
Configure the job settings:
For Name, enter a name (for example, job-2).
For Description, enter a description (for example, Data quality checks using AWS Glue Data Quality).
For IAM Role, choose the project role.
For Type, choose Spark.
For AWS Glue version, use version 5.0.
For Language, choose Python.
For Worker type, use G.1X.
For Number of Instances, set to 10.
For Number of retries, set to 0.
For Job timeout, set to 480.
For Compute connection, choose project.spark.compatibility.
Under Advanced settings, turn on Continuous logging.
Leave the remaining settings as default, then choose Submit.
After the job is created, a confirmation message will appear indicating that job-2 was created successfully.
Test AWS Glue jobs
Test both jobs to make sure they execute successfully:
In SageMaker Unified Studio, go to your project.
On the top menu, choose Build, and under Data Analysis & Integration, choose Data processing jobs.
Select job-1 and choose Run job.
Monitor the job execution and verify it completes successfully.
Similarly, select job-2 and choose Run job.
Monitor the job execution and verify it completes successfully.
Add EMR Serverless compute
In the ETL pipeline, we use EMR Serverless to perform compute-intensive transformations and aggregations on large datasets. It automatically scales resources based on workload, offering high performance with simplified operations. By integrating EMR Serverless with SageMaker Unified Studio, you can simplify the process of running Spark jobs interactively using Jupyter notebooks in a serverless environment.
This section walks through the steps to configure EMR Serverless compute within SageMaker Studio and use it for executing distributed data processing jobs.
Configure EMR Serverless in SageMaker Unified Studio
To use EMR Serverless for processing in the project, follow these steps:
In the navigation pane on Project Overview, choose Compute.
On the Data processing tab, choose Add compute and Create new compute resources.
Select EMR Serverless and choose Next.
Configure EMR Serverless settings:
For Compute name, enter a name (for example, etl-emr-serverless).
For Description, enter a description (for example, EMR Serverless for advanced data processing).
For Release label, choose emr-7.8.0.
For Permission mode, choose Compatibility.
Choose Add Compute to complete the setup.
After it’s configured, the EMR Serverless compute will be listed with the deployment status Active.
Create and run notebook with EMR Serverless
After you create the EMR Serverless compute, you can run PySpark-based data transformation jobs using a Jupyter notebook to perform large-scale data transformations. This job reads cleaned customer, transaction, and clickstream datasets from Amazon S3, performs aggregations and scoring, and writes the final analytics outputs back to Amazon S3 in both Parquet and CSV formats.Complete the following steps to create a notebook for EMR Serverless processing:
On the top menu, under Build, choose JupyterLab.
Choose File, New, and Notebook.
Set Kernel as Python 3, Connection type as PySpark, and Compute as emr-s.etl-emr-serverless.
Enter the following PySpark script to run your data transformation job on EMR Serverless. Provide the name of your S3 bucket:
Choose File, Save Notebook As, and save the file as shared/emr_data_transformation_job.ipynb.
Choose Run Cell to run the script.
Monitor the Script execution and verify it completes successfully.
Monitor the Spark job execution and ensure it completes without errors.
Add Redshift Serverless compute
With Redshift Serverless, users can run and scale data warehouse workloads without managing infrastructure. It is ideal for analytics use cases where data needs to be queried from Amazon S3 or integrated into a centralized warehouse. In this step, you add Redshift Serverless to the project for loading and querying processed customer analytics data generated in earlier stages of the pipeline. For more information about Redshift Serverless, see Amazon Redshift Serverless.
Set up Redshift Serverless compute in SageMaker Unified Studio
Complete the following steps to set up Redshift Serverless compute:
In SageMaker Unified Studio, choose the Compute tab within your project workspace (ETL-Pipeline-Demo).
On the SQL analytics tab, choose Add compute, then choose Create new compute resources to begin configuring your compute environment.
Select Amazon Redshift Serverless.
Configure the following:
For Compute name, enter a name (for example, ecommerce_data_warehouse).
For Description, enter a description (for example, Redshift Serverless for data warehouse).
For Workgroup name, enter a name (for example, redshift-serverless-workgroup).
For Maximum capacity, set to 512 RPUs.
For Database name, enter dev.
Choose Add Compute to create the Redshift Serverless resource.
After the compute is created, you can test the Amazon Redshift connection.
On the Data warehouse tab, confirm that redshift.ecommerce_data_warehouse is listed.
Choose the compute: redshift.ecommerce_data_warehouse.
On the Permissions tab, copy the IAM role ARN. You use this for the Redshift COPY command in the next step.
Create and execute querybook to load data into Amazon Redshift
In this step, you create a SQL script to load the processed customer summary data from Amazon S3 into a Redshift table. This enables centralized analytics for customer segmentation, lifetime value calculations, and marketing campaigns. Complete the following steps:
On the Build menu, under Data Analysis & Integration, choose Query editor.
Enter the following SQL into the querybook to create the customer_summary table in the public schema:
-- Create customer_summary table in public schema
CREATE TABLE IF NOT EXISTS public.customer_summary (
customer_id INT PRIMARY KEY,
name VARCHAR(100),
email VARCHAR(100),
registration_date DATE,
total_transactions INT,
total_spent DECIMAL(10, 2),
avg_transaction_value DECIMAL(10, 2),
days_since_last_purchase INT,
total_clicks INT,
purchase_actions INT,
customer_value_score DECIMAL(10, 2)
);
Choose Add SQL to add a new SQL script.
Enter the following SQL into the querybook
TRUNCATE TABLE customer_summary;
Note: We truncate the customer_summary table to remove existing records and ensure a clean, duplicate-free reload of the latest aggregated data from S3 before running the COPY command.
Choose Add SQL to add a new SQL script.
Enter the following SQL to load the data into Redshift Serverless from your S3 bucket. Provide the name of your S3 bucket and IAM role ARN for Amazon Redshift:
-- Load data from S3 (replace with your bucket name and IAM role)
COPY public.customer_summary FROM 's3://<bucket-name>/analytics/customer_summary/'
IAM_ROLE 'arn:aws:iam::<Account-ID>:role/<your-redshift-role>'
FORMAT AS CSV
IGNOREHEADER 1
REGION 'us-east-1';
In the Query Editor, configure the following:
Connection: redshift.ecommerce_data_warehouse
Database: dev
Schema: public
Choose Choose to apply the connection settings.
Choose Run Cell for each cell to create the customer_summary table in the public schema and then load data from Amazon S3.
Choose Actions, Save, name the querybook final_data_product, and choose Save changes.
This completes the creation and execution of the Redshift data product using the querybook.
Create and manage the workflow environment
This section describes how to create a shared workflow environment and define a code-based workflow that automates a customer data pipeline using Apache Airflow within SageMaker Unified Studio. Shared environments facilitate collaboration among project members and centralized workflow management.
Create the workflow environment
Workflow environments must be created by project owners. After they’re created, members of the project can sync and use the workflows. Only project owners can update or delete workflow environments. Complete the following steps to create the workflow environment:
Choose Compute for your project.
On the Workflow environments tab, choose Create.
Review the configuration parameters and choose Create workflow environment.
Wait for the environment to be fully provisioned before proceeding It will take around 20 minutes to provision.
Create the code-based workflow
When the workflow environment is ready, define a code-based ETL pipeline using Airflow. This pipeline automates daily processing tasks across services like AWS Glue, EMR Serverless, and Redshift Serverless.
On the Build menu, under Orchestration, choose Workflows.
Choose Create new workflow, then choose Create workflow in code editor.
Configure Space and choose the instance type ml.t3.xlarge. This ensures your JupyterLab instance has at least 4 vCPUs and 4 GiB of memory.
Choose Configureand Restart Space to launch your environment.
The following script defines a daily scheduled ETL workflow that automates several actions:
Initial data transformation using AWS Glue
Data quality validation using AWS Glue (EvaluateDataQuality)
Advanced data processing with EMR Serverless using a Jupyter notebook
Loading transformed results into Redshift Serverless from a querybook
Replace the default DAG template with the following definition, ensuring that job names and input paths match the actual names used in your project:
from datetime import datetime
from airflow import DAG
from airflow.decorators import dag
from airflow.utils.dates import days_ago
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
from workflows.airflow.providers.amazon.aws.operators.sagemaker_workflows import NotebookOperator
from sagemaker_studio import Project
# Get SageMaker Studio project IAM role
project = Project()
default_args = {
'owner': 'data_engineer',
'depends_on_past': False,
'email_on_failure': True,
'email_on_retry': False,
'retries': 1
}
@dag(
dag_id='customer_etl_pipeline',
default_args=default_args,
schedule_interval='@daily',
start_date=days_ago(1),
is_paused_upon_creation=False,
tags=['etl', 'customer-analytics'],
catchup=False
)
def customer_etl_pipeline():
# Step 1: Initial data transformation using Glue
initial_transformation = GlueJobOperator(
task_id='initial_transformation',
job_name='job-1',
iam_role_arn=project.iam_role,
)
# Step 2: Data quality checks using Glue DQ
data_quality_check = GlueJobOperator(
task_id='data_quality_check',
job_name='job-6',
iam_role_arn=project.iam_role,
)
# Step 3: EMR Serverless notebook processing
emr_processing = NotebookOperator(
task_id='emr_processing',
input_config={
"input_path": "emr_data_transformation_job.ipynb",
"input_params": {}
},
output_config={"output_formats": ['NOTEBOOK']},
poll_interval=10,
)
# Step 4: Load to Redshift notebook
redshift_load = NotebookOperator(
task_id='redshift_load',
input_config={
"input_path": "final_data_product.sqlnb",
"input_params": {}
},
output_config={"output_formats": ['NOTEBOOK']},
poll_interval=10,
)
# Task dependencies
initial_transformation >> data_quality_check >> emr_processing >> redshift_load
# Instantiate DAG
customer_etl_dag = customer_etl_pipeline()
Choose File, Save python file, name the file shared/workflows/dags/customer_etl_pipeline.py, and choose Save.
Deploy and run the workflow
Complete the following steps to run the workflow:
On the Build menu, choose Workflows.
Choose the workflow customer_etl_pipeline and choose Run.
Running a workflow puts tasks together to orchestrate Amazon SageMaker Unified Studio artifacts. You can view multiple runs for a workflow by navigating to the Workflows page and choosing the name of a workflow from the workflows list table.
After your Airflow workflows are deployed in SageMaker Unified Studio, monitoring becomes essential for maintaining reliable ETL operations. The integrated Amazon MWAA environment provides comprehensive observability into your data pipelines through the familiar Airflow web interface, enhanced with AWS monitoring capabilities. The Amazon MWAA integration with SageMaker Unified Studio offers real-time DAG execution tracking, detailed task logs, and performance metrics to help you quickly identify and resolve pipeline issues. Complete the following steps to monitor the workflow:
On the Build menu, choose Workflows.
Choose the workflow customer_etl_pipeline.
Choose View runs to see all executions.
Choose a specific run to view detailed task status.
For each task, you can view the status (Succeeded, Failed, Running), start and end times, duration, and logs and outputs. The workflow is also visible in the Airflow UI, accessible through the workflow environment, where you can view the DAG graph, monitor task execution in real time, access detailed logs, and view the status.
Go to Workflows and select the workflow named customer_etl_pipeline.
From the Actions menu, choose Open in Airflow UI.
After the workflow completes successfully, you can query the data product in the query editor.
On the Build menu, under Data Analysis & Integration, choose Query editor.
Run select * from "dev"."public"."customer_summary"
Observe the contents of the customer_summary table, including aggregated customer metrics such as total transactions, total spent, average transaction value, clicks, and customer value scores. This allows verification that the ETL and data quality pipelines loaded and transformed the data correctly.
Clean up
To avoid unnecessary charges, complete the following steps:
This post demonstrated how to build an end-to-end ETL pipeline using SageMaker Unified Studio workflows. We explored the complete development lifecycle, from setting up fundamental AWS infrastructure—including Amazon S3 CORS configuration and IAM permissions—to implementing sophisticated data processing workflows. The solution incorporates AWS Glue for initial data transformation and quality checks, EMR Serverless for advanced processing, and Redshift Serverless for data warehousing, all orchestrated through Airflow DAGs. This approach offers several key benefits: a unified interface that consolidates necessary tools, Python-based workflow flexibility, seamless AWS service integration, collaborative development through Git version control, cost-effective scaling through serverless computing, and comprehensive monitoring tools—all working together to create an efficient and maintainable data pipeline solution.
By using SageMaker Unified Studio workflows, you can accelerate your data pipeline development while maintaining enterprise-grade reliability and scalability. For more information about SageMaker Unified Studio and its capabilities, refer to the Amazon SageMaker Unified Studio documentation.
SageMaker Unified Studio provides an integrated experience to use your data and tools for analytics and AI. You can use SageMaker Unified Studio to discover your data and put it to work using familiar AWS analytics and machine learning (ML) services for model development, generative AI, big data processing, and SQL analytics, assisted by Amazon Q Developer. By default, SageMaker domains support AWS Identity and Access Management (IAM) user credentials. You can also enable access to SageMaker domains in SageMaker Unified Studio for users with single sign-on (SSO) with IAM Identity Center and direct SAML integration with SageMaker Unified Studio.
Users can access SageMaker Unified Studio with their existing corporate credentials. With IAM Identity Center, administrators can connect their existing external IdPs and continue to manage users and groups in those existing identity systems, which can then be synchronized with IAM Identity Center using System for Cross-domain Identity Management (SCIM).In this post, we show how to set up workforce access with SageMaker Unified Studio using Ping Identity as an external IdP with IAM Identity Center.
In this post, we show how to set up workforce access with SageMaker Unified Studio using Ping Identity as an external IdP with IAM Identity Center.
Solution overview
We walk through the following high-level steps to implement this solution:
Enable IAM Identity Center.
Create a SageMaker Unified Studio domain.
Set up your IdP (for this example, Ping Identity).
Connect Ping Identity and IAM Identity Center.
Set up automatic provisioning of users and groups in IAM Identity Center.
Configure SageMaker Unified Studio SSO user access.
Prerequisites
For this walkthrough, you should have the following prerequisites:
An AWS account with IAM Identity Center enabled. It is recommended to use an organization-level IAM Identity Center instance for best practices and centralized identity management across your AWS organization.
On the SageMaker console, go to the domain details and copy the Amazon Resource Name (ARN) under Domain ARN. You will use this value when you add your trust policy and when you connect your IAM IdP to your Ping Identity instance.
Set up your IdP (Ping Identity)
In this section, we walk through the procedure to set up your IdP (for this example, Ping Identity).
Create an environment in Ping Identity
Complete the following steps to create an environment for Ping Identity:
In the Tailor your experiences pop-up, choose Skip.
Create a group in Ping Identity
Complete the following steps to create a group in Ping Identity:
On the Environments page, choose Manage Environments.
In the navigation pane, choose Directory, then choose Groups.
Choose the plus sign to add a group.
For Group Name, enter sagemaker
For Description, enter an optional description (for example, Amazon SageMaker Unified Studio).
For Population, choose Default.
Choose Save.
On the Roles tab for the sagemaker group, assign the Environment Admin role to the group.
Create a user in Ping Identity
Complete the following steps to create a user:
In the navigation pane, choose Directory, then choose Users.
Choose the plus sign to create a user.
Provide values for Given name, Family name, Username, and Email.
For Password, choose First time password.
Choose Save.
You can add more users as needed.
Assign group to user
Complete the following steps to assign your group to your user:
In the navigation pane, choose Directory, then choose Groups.
Choose the sagemaker group you created.
On the Users tab, choose the plus sign to add a user.
Add the user you created.
Connect Ping Identity and IAM Identity Center
To configure the integration between Ping Identity and IAM Identity Center, you need access to both management consoles. Although Ping Identity’s application catalog includes IAM Identity Center, we recommend configuring a standard SAML application for greater control over settings and attribute mappings.
Complete the following steps:
Go to the Ping Identity environment you created and choose Applications in the navigation pane.
Choose the plus sign to add an application:
For Application name, enter a name (for this example, we use unifiedstudio).
For Description, enter an optional description.
For Application Type, choose SAML Application.
Choose Configure.
Sign in to the IAM Identity Center console as a user with administrative privileges.
In the navigation pane, choose Settings to update your settings:
On the Identity source tab, choose Change identity source on the Actions dropdown menu.
For Choose identity source, select External identity provider, then choose Next.
In the Service provider metadata section, choose Download metadata file to download the IAM Identity Center metadata file.
You will use this service provider metadata file in the next step when you connect Ping Identity with IAM Identity Center.
Return to the Ping Identity console and the SAML application page.
In the SAML Configuration section, select Import Metadata, upload the metadata file you downloaded, then choose Save.
On the Overview tab of the application page, choose Download Metadata under Connection details to download the Ping Identity IdP metadata. You will use this for the SAML configuration in IAM Identity Center to set up Ping Identity as an IdP in the next step.
Return to the IAM Identity Center console and continue configuring your identity source:
In the Identity provider metadata section, choose Choose file under IdP SAML metadata, upload the metadata file you downloaded from Ping Identity, then choose Next.
Choose Accept to accept the disclaimer.
Choose Change identity source.
Return to the Ping Identity console to complete the SAML configuration.
On the Configuration tab, choose the edit icon to update the configuration:
For Sign, choose Sign Assertion & Response.
For Subject Name ID, enter urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress.
For Assertion Validity Duration, enter 300.
Leave the remaining values as default.
On the Attributes tab, choose the edit icon.
Choose +Add to add two attribute mappings:
Map the attribute saml-subject to Username, and leave Name format as default.
Map the attribute https://aws.amazon.com/SAML/Attributes/PrincipalTag:Email to Email Address, and set Name format to Unspecified.
Choose Save.
On the PingOnePolicies tab, select Single Factor, then choose Save. This post uses single-factor authentication for demonstration purposes only. In your environments, follow your organization’s security standards and governance framework.
On the Access tab, search for the sagemaker group under Group Membership Policy, and assign the unifiedstudio SAML application to the group.
Enable the application.
Set up automatic provisioning of users and groups from Ping Identity into IAM Identity Center
To configure the automatic provisioning of users and groups between Ping Identity and IAM Identity Center through SCIM, you must have access to both management consoles. Complete the following steps:
On the IAM Identity Center console, choose Settings in the navigation pane.
In the Automatic provisioning section, choose Enable.
This enables automatic provisioning in IAM Identity Center and displays the necessary SCIM endpoint and access token information.
In the Inbound automatic provisioning dialog box, copy the values for SCIM endpoint and Access token, then choose Close. You will use these values to configure provisioning in Ping Identity in the next step.
This completes the setup process in IAM Identity Center.
Log in to the Ping Identity console.
In the navigation pane, choose Integrations, then choose Provisioning.
Choose the plus sign to add a new connection.
For Choose a connection type, choose Select next to Identity Store.
Provide a name (for this example, we use Identitycenter) and an optional description, then choose Next.
Under Configuration Authentication, provide the following configuration:
For SCIM BASE URL, enter the SCIM endpoint from IAM Identity Center.
For Authentication Method, choose OAuth 2 Bearer Token.
For Oauth Access Token, enter the access token from IAM Identity Center.
For Auth Type Header, choose Bearer (default option).
Choose Test Connection to validate the connection between Ping Identity and IAM Identity Center, then choose Next.
Under Configuration Preference, provide the following configuration:
For User Filter Expression, enter userName Eq “%s”.
For Group Membership Handling, select Merge.
Leave the remaining settings as default and choose Save.
On the Provisioning tab, choose the plus sign, then choose New Rule to create a rule for the SCIM connection.
Enter a name (for this example, unifiedstudio) and an optional description, then choose Create Rule.
Under the newly created rule, choose the plus sign next to Available Connections to add the connection identitycenter, then choose Save.
Edit the user filter:
For Attribute, choose Enabled.
For Operator, choose Equals.
For Value, choose true.
Choose Save.
Choose the edit icon next to Attribute Mapping and set the attribute mappings as shown in the following screenshot:
Delete the Primary Phone attribute mapping because it’s optional in AWS. Leaving this field blank can cause Ping Identity’s SCIM connector to generate errors during user provisioning.
Add a new attribute called Username under PingOne Directory and then map to displayName under Identitycenter.
Under Group Provisioning, choose the sagemaker group if you want to sync all sagemaker group users with auto provisioning.
In the pop-up, select I understand and want to continue, then choose Save.
On the Provisioning page, choose the Connections tab.
Enable the SCIM connection Identitycenter and rule unifiedstudio.
This completes the SCIM setup process between Ping Identity and IAM Identity Center.
Configure SageMaker Unified Studio SSO user access
Complete the following steps to configure SSO user access to SageMaker Unified Studio for your SageMaker domain:
On the SageMaker console, choose Domains in the navigation pane.
Choose the domain for which you want to configure SAML user access.
On the domain details page, you can find the SSO configuration in two locations:
From the main domain view, choose Configure next to Configure SSO user access.
Alternatively, scroll down to the User management tab and choose Configure SSO user access.
On the Choose user authentication method page, select IAM Identity Center, then choose Next.
For Choose user and group assignment method, choose from the following options, then choose Next:
Require assignments: Users and groups must be explicitly added to the domain to gain access. This provides more granular control over who can access the domain.
Do not require assignments: All authorized Ping Identity users and groups can access this domain if they have been assigned to the SAML application in Ping Identity.
For either option, users or groups must have access to the Ping Identity SAML application (unifiedstudio in this example) to authenticate successfully.
On the Review and save page, review your choices and choose Save. These settings can’t be changed after you save them.
If you’ve chosen to require assignments, use the Add users and groups section to add SAML users and groups to your domain.
Now, users will be able to access SageMaker Unified Studio using the domain URL with their SSO credentials.
You can explore different projects for your users and assign those projects based on your IdP user groups for fine-grained access controls. For example, you can create different SAML user groups based on their job function in Ping Identity, then assign those Ping Identity groups to the unifiedstudio SAML application in Ping Identity, and then assign those Ping Identity SAML groups to their respective project profiles in SageMaker Unified Studio. To assign project profiles for their respective groups, choose the Project profiles tab and choose your project profile. On the Authorized users and groups page, choose Add, then choose SSO groups. Choose Add users and groups button to complete the project profile assignment.
Validate access with Ping Identity users
Complete the following steps to validate access:
On the SageMaker domain details page, choose the link for the SageMaker Unified Studio URL.
Log in with your user credentials. After successful login, you will be redirected to the SageMaker Unified Studio home page. Here, you can explore different projects to your users and assign those projects based on your SAML user groups for fine-grained access control.
To assign an authorization policy, those Govern and then Domain units.
Choose your SageMaker domain, then choose a suitable authorization policy. For this example, we choose Project creation policy.
Choose Add policy grant to assign user groups or users to their respective project profiles.
You have successfully federated SageMaker Unified Studio with Ping Identity as an IdP with IAM Identity Center. You can connect to SageMaker Unified Studio by using your Ping Identity credentials.
Clean up
After you test out this solution, remember to delete the resources you created to avoid incurring future charges. For instructions to delete your SageMaker Unified Studio domain, refer to Delete domains. If you want to delete your Ping Identity account, reach out to Ping Identity for assistance.
Conclusion
In this post, we demonstrated how to set up Ping Identity as an IdP over SAML authentication for SageMaker Unified Studio access through IAM Identity Center federation. To learn more, refer to the Amazon SageMaker Unified Studio User Guide, which provides guidance on how to build data and AI applications using SageMaker.
Over the past week, we passed Laba festival, a traditional marker in the Chinese calendar that signals the final stretch leading up to the Lunar New Year. For many in China, it’s a moment associated with reflection and preparation, wrapping up what the year has carried, and turning attention toward what lies ahead.
Looking forward, next week also brings Lichun, the beginning of spring and the first of the 24 solar terms. In Chinese tradition, spring is often seen as the season when growth begins and new cycles take shape. There’s a common saying that “a year’s plans begin in spring,” capturing the idea that this is a time to set one’s direction and start fresh.
Last week’s launches Here are the launches that got my attention this week:
Amazon Bedrock enhances support for agent workflows with server-side tools and extended prompt caching – Amazon Bedrock introduced two updates that improve how developers build and operate AI agents. The Responses API now supports server-side tool use, so agents can perform actions such as web search, code execution, and database updates within AWS security boundaries. Bedrock also adds a 1-hour time-to-live (TTL) option for prompt caching, which helps improve performance and reduce the cost for long-running, multi-turn agent workflows. Server-side tools are available with OpenAI GPT OSS 20B and 120B models, and the 1-hour prompt caching TTL is generally available for select Claude models by Anthropic in Amazon Bedrock.
Amazon SageMaker Unified Studio adds private VPC connectivity with AWS PrivateLink – Amazon SageMaker Unified Studio now supports AWS PrivateLink, providing private connectivity between your VPC and SageMaker Unified Studio without routing customer data over the public internet. With SageMaker service endpoints onboarded into a VPC, data traffic remains within the AWS network and is governed by IAM policies, supporting stricter security and compliance requirements.
Amazon S3 adds support for changing object encryption without data movement – Amazon S3 now supports changing the server-side encryption type of existing encrypted objects without moving or re-uploading data. Using the UpdateObjectEncryption API, you can switch from SSE-S3 to SSE-KMS, rotate customer -managed AWS Key Management Service (AWS KMS) keys, or standardize encryption across buckets at scale with S3 Batch Operations while preserving object properties and lifecycle eligibility.
Amazon Keyspaces introduces table pre-warming for predictable high-throughput workloads – Amazon Keyspaces (for Apache Cassandra) now supports table pre-warming, which helps you proactively set warm throughput levels so tables can handle high read and write traffic instantly without cold-start delays. Pre-warming helps reduce throttling during sudden traffic spikes, such as product launches or sales events, and works with both on-demand and provisioned capacity modes, including multi-Region tables. The feature supports consistent, low-latency performance while giving you more control over throughput readiness.
Additional updates Here are some additional projects, blog posts, and news items that I found interesting:
Building zero-trust access across multi-account AWS environments with AWS Verified Access – This post walks through how to implement AWS Verified Access in a centralized, shared-services architecture. It shows how to integrate with AWS IAM Identity Center and AWS Resource Access Manager (AWS RAM) to apply zero trust access controls at the application layer and reduce operational overhead across multi-account AWS environments.
Amazon EventBridge increases event payload size to 1 MB – Amazon EventBridge now supports event payloads up to 1 MB, an increase from the previous 256 KB limit. This update helps event-driven architectures carry richer context in a single event, including complex JSON structures, telemetry data, and machine learning (ML) or generative AI outputs, without splitting payloads or relying on external storage.
AWS MCP Server adds deployment agent SOPs (preview) – AWS introduced deployment standard operating procedures (SOPs) that AI agents can deploy web applications to AWS from a single natural language prompt in MCP -compatible integrated development environments (IDEs) and command line interfaces (CLIs) such as Kiro, Cursor, and Claude Code. The agent generates AWS Cloud Development Kit (AWS CDK) infrastructure, deploys AWS CloudFormation stacks, and sets up continuous integration and continuous delivery (CI/CD) workflows following AWS best practices. The preview supports frameworks including React, Vue.js, Angular, and Next.js.
AWS Network Firewall adds generation AI traffic visibility with web category filtering – AWS Network Firewall now provides visibility into generative AI application traffic through predefined web categories. You can use these categories directly in firewall rules to govern access to generative AI tools and other web services. When combined with TLS inspection, category-based filtering can be applied at the full URL level.
AWS Lambda adds enhanced observability for Kafka event source mappings – AWS Lambda introduced enhanced observability for Kafka event source mappings, providing Amazon CloudWatch Logs and metrics to monitor event polling configuration, scaling behavior, and event processing state. The update improves visibility into Kafka-based Lambda workloads, helping teams diagnose configuration issues, permission errors, and function failures more efficiently. The capability supports both Amazon Managed Streaming for Apache Kafka (Amazon MSK) and self-managed Apache Kafka event sources.
AWS CloudFormation 2025 year in review – This year-in-review post highlights CloudFormation updates delivered throughout 2025, with a focus on early validation, safer deployments, and improved developer workflows. It covers enhancements such as improved troubleshooting, drift-aware change sets, stack refactoring, StackSets updates, and new -IDE and AI -assisted tooling, including the CloudFormation language server and the Infrastructure as Code (IaC) MCP server.
Upcoming AWS events Check your calendars so that you can sign up for this upcoming event:
AWS Community Day Romania (April 23–24, 2026) – This community-led AWS event brings together developers, architects, entrepreneurs, and students for more than 10 professional sessions delivered by AWS Heroes, Solutions Architects, and industry experts. Attendees can expect expert-led technical talks, insights from speakers with global conference experience, and opportunities to connect during dedicated networking breaks, all hosted at a premium venue designed to support collaboration and community engagement.
If you’re looking for more ways to stay connected beyond this event, join the AWS Builder Center to learn, build, and connect with builders in the AWS community.
This post was co-written with Anthony Lempelius and James Mesney from Alation.
When a team wants to reuse a dataset, whether it is to build a new pipeline, launch a dashboard, run an analysis, or power an AI application, the first challenge is rarely the code. Data engineers need to understand lineage, transformations, and operational expectations. Data analysts and BI engineers need consistent definitions, metrics, and trusted sources. Data scientists and AI engineers need to know provenance, quality, access constraints, and how data or features were derived. In many organizations, that context is captured in different places by different teams, often across solutions like Alation and SageMaker Unified Studio, both of which can serve as a system of record for business context depending on who is doing the work and where they operate day to day. When those perspectives are not connected, people revalidate the same information, debate definitions, and duplicate documentation across tools. A unified metadata foundation brings these role specific views together so business context, technical metadata, and governance stay aligned across platforms, making data easier to trust, easier to find, and easier to use across analytics and AI.
The new Alation integration with Amazon SageMaker Unified Studio addresses these challenges by synchronizing catalog metadata between both systems. This synchronization creates a unified metadata experience where technical teams working in SageMaker Unified Studio and business teams working in Alation collaborate on top of the same metadata. You can verify how ML and analytics assets are created, understand dependencies, and maintain traceability across your data lifecycle regardless of which system your teams prefer to use.
In this post, we demonstrate who benefits from this integration, how it works, the specific metadata it synchronizes, and provide a complete deployment guide for your environment.
The value of unified metadata governance
Organizations managing large-scale analytics and ML workloads face critical challenges when metadata is fragmented across multiple systems. When metadata exists in silos, data scientists spend valuable time searching for the right datasets. Teams duplicate metadata management efforts, creating inconsistent definitions and conflicting metrics across the organization.
Regulatory requirements demand clear provenance. Without unified metadata governance, organizations struggle to demonstrate compliance, trace data origins, and maintain audit trails across their ML and analytics pipelines. Data discovery becomes a bottleneck when teams can’t quickly find, understand, and trust the data they need, delaying model development and reducing the overall business value of data investments.
Applying consistent governance policies across disparate systems is nearly impossible without a unified metadata layer. This creates security vulnerabilities, data quality issues, and compliance blind spots. A unified metadata governance approach alleviates these challenges by providing a single source of truth for metadata across ML and analytics systems, enabling faster data discovery, consistent governance, and confident compliance while reducing the operational burden on data and ML teams.
Solution overview
The Alation and SageMaker Unified Studio integration unifies the user experience, synchronizing metadata from cataloged assets between both systems.
This Phase 1 integration extracts metadata from Amazon SageMaker Catalog into Alation, giving you one place to discover assets.
The integration connects through AWS Identity and Access Management (IAM) authentication and synchronizes key metadata elements, including domains, projects, asset names, descriptions, owners, glossary terms, and custom metadata fields. Every metadata update includes provenance information: the originating service, the person who made the change, and the timestamp, creating comprehensive audit trails for compliance.
You can run metadata extractions on demand or schedule them to run automatically. The system performs an initial bulk extraction of your selected domains and projects, then keeps it up-to-date through incremental updates using either event-driven triggers or scheduled polling. Communication uses encrypted APIs with scoped IAM permissions following least-privilege principles.
This integration helps organizations in financial services, telecommunications, retail, manufacturing, and transportation that manage large numbers of analytics and ML workloads across many systems and teams. You can reduce metadata duplication, accelerate data discovery, and enable your data scientists, analysts, and engineers to find trusted data faster so they can focus on building insights rather than validating data quality.
The following diagram illustrates the solution architecture.
The following screenshot showcases the Alation catalog displaying the SageMaker Unified Studio project and its synchronized assets.
Metadata synchronization
This integration automatically synchronizes essential metadata between SageMaker Unified Studio and Alation, facilitating consistent information across both systems. The synchronization brings together the types of metadata you need for discovery, governance, and audit workflows, giving you clearer insight into how datasets, features, and models relate across your services.
The integration synchronizes catalog metadata, including domains, projects, asset names, descriptions, owners, glossary terms, and metadata forms. Additionally, the integration synchronizes provenance metadata, which includes information about the originating service, the actor who made the change, and the timestamp, to support traceability and audit workflows.
Integration mechanics
The integration connects SageMaker Unified Studio and Alation through a scoped IAM role that provides secure, encrypted communication. After you configure this connection within Alation, the system performs an initial extraction of your selected domains and projects, then keeps information current through incremental updates using either event-driven triggers or scheduled polling.
The integration synchronizes metadata forms from SageMaker Unified Studio into Alation through automated field mapping between both systems’ schemas. Metadata forms can capture various asset specific details like feature store references, training run identifiers, model versions, and evaluation metrics.
Every metadata update includes provenance information: the originating service, the person who made the change, and when it occurred. This supports audit and stewardship workflows. Access controls follow least-privilege principles through IAM while applying Alation’s role-based permissions, letting you limit synchronization by project, namespace, or tag as needed.
Security and compliance
Security and compliance are critical when synchronizing metadata across systems. This integration follows enterprise security practices to facilitate safe, controlled metadata synchronization. The connector uses least-privilege access, encrypted transport, and clear separation between metadata and data, so you can maintain governance without disrupting existing workflows.
You configure a scoped IAM role to define which accounts, projects, and namespaces the connector can access, making sure access follows your organization’s security policies. Metadata moves over TLS-protected APIs, and you control which domains and projects to include in Alation. By default, the integration synchronizes only metadata; your data files and artifacts remain in their original AWS locations unless you explicitly choose to export them.
Alation maintains a complete audit trail by recording extraction events, mapping changes, and stewardship activities. These security controls support compliant metadata governance while preserving your existing operational practices.
Prerequisites
Before setting up this integration, ensure you have the following:
An Alation Cloud Service (ACS) instance
Alation server admin access
An AWS account
A SageMaker Unified Studio domain and project with existing metadata
Configure authentication
Before configuring the Alation connector, you must set up the required AWS resources and permissions. The first step is to configure authentication. The Alation connector supports two authentication methods to access SageMaker Unified Studio. Choose the method that best fits your security requirements.
Option 1: IAM role (Recommended)
Create an IAM role that the Alation connector will assume to access SageMaker Unified Studio. For detailed instructions on creating IAM roles, see IAM role creation.
The following is an example IAM permission policy for SageMaker Catalog access:
Create an IAM user with programmatic access and attach the necessary permissions. For detailed instructions on creating IAM users, see Create an IAM user in your AWS account.
Create an IAM user with programmatic access enabled, attach the following policy, and generate access keys for use in Alation configuration:
Add IAM role or user to SageMaker Unified Studio domain
Add the IAM role or user you created to the SageMaker Unified Studio domain. For detailed instructions on adding users to a domain, see User management in Amazon SageMaker Unified Studio. The following screenshot shows an example of adding IAM users on the SageMaker dashboard.
Add IAM role or user to SageMaker Unified Studio projects
The IAM role or user must be added as a member to all SageMaker Unified Studio projects that contain metadata you want to synchronize with Alation. Projects without this member will not be included in the synchronization process.
Add the IAM role or user as a project member with Contributor or Owner permissions for each project you want to include in the sync, as illustrated in the following screenshot. For detailed instructions on adding project members, see Add project members.
Install SageMaker enhanced connector
After completing the AWS setup, you can configure the Alation connector to establish the integration. The connector is distributed as a .zip package for upload and installation in the Alation application. To obtain the connector, contact the Forward Deployed Engineering team or your Alation Account Manager.
Navigate to the Data Sources section in Alation, create a new data source, and select SageMaker Catalog as the source type. Configure the connection settings with the authentication method chosen in the AWS setup.
For IAM role authentication, use the following configuration:
Connection Type: IAM Role
Role ARN: ARN of the IAM role created in AWS setup
External ID: External ID configured in the trust policy
AWS Region: Region where your SageMaker Unified Studio domain is located
For IAM user authentication, use the following configuration:
Connection Type: Access Keys
Access Key ID: Access key from AWS setup
Secret Access Key: Secret key from AWS setup
AWS Region: Region where your SageMaker Unified Studio domain is located
Test the connection to verify authentication and network connectivity, as shown in the following screenshot.
Configure metadata extraction settings
Configure the extraction scope by selecting the SageMaker domains and projects to synchronize, as shown in the following screenshot. Only projects where the IAM role or user is a member will be available for synchronization.
Run initial extraction
Execute the first metadata synchronization to import existing metadata from SageMaker Unified Studio into Alation. Monitor the extraction progress through Alation’s status indicators and validate that SageMaker assets appear correctly in the catalog.
The following screenshot shows the job history page with job status Running.
The following screenshot shows the job history page with job status Succeeded.
The following screenshot shows the Alation catalog displaying the SageMaker Unified Studio project and its synchronized assets.
Operate and tune
Configure ongoing operations by setting extraction cadence, configuring reconciliation alerts, and monitoring logs regularly. Add data stewards to synchronized assets, and consider enabling AI-generated descriptions or working with Alation Professional Services for advanced governance design.
Enhanced capabilities
The next phase of the integration introduces three key capabilities: bi-directional metadata synchronization, lineage replication, and data quality metadata replication. The bi-directional capability gives you the flexibility to control where metadata updates originate, either in Alation or in SageMaker Unified Studio, so you can manage metadata changes in the service that best aligns with your organizational workflows and governance processes.
The feature set is rolling out in phases. Phase 1 is available at the time of writing this post and provides extraction from SageMaker Unified Studio into Alation, including initial and incremental updates and audit logging. Phase 2 is coming soon and will offer configurable principal catalogs, advanced scoped syncs, and reconciliation workflows for Alation Cloud Service customers.
These enhancements will support governed, scalable ML operations with increasing depth and automation.
Conclusion
The Alation and SageMaker Unified Studio integration helps organizations bridge the gap between fast analytics and ML development and the governance requirements most enterprises face. By cataloging metadata from SageMaker Unified Studio in Alation, you gain a governed, discoverable view of how assets are created and used. This supports leaders, stewards, compliance teams, and ML practitioners who depend on accurate, well-documented data to scale analytics and AI responsibly.
Data teams today struggle with fragmented tools, complex infrastructure provisioning, and hours spent writing boilerplate code to connect to data sources. This forces analysts, data scientists, and engineers to work in separate environments, which slows collaboration and time to insight. Since our launch of Amazon SageMaker Unified Studio in March 2025, leading companies such as Bayer, NatWest, and Carrier have adopted it to bring their data teams into one collaborative workspace with unified tools, straightforward infrastructure provisioning, and fast connections to data sources.
Continuing our mission to provide faster time-to-value for customers, in November 2025, we announced Amazon SageMaker notebooks, a serverless workspace with a built-in AI agent in Amazon SageMaker Unified Studio. You can now launch a notebook in seconds, generate code from natural language prompts, and connect automatically to data across Amazon Simple Storage Service (Amazon S3), Amazon Redshift, third-party databases, and more from a single environment without needing to pre-provision or tune data processing infrastructure. Inside these serverless notebooks, analysts can perform SQL queries, data scientists can execute Python code, and data engineers can process large-scale data jobs in Spark within a single workspace. Together with the new one-click onboarding available for SageMaker Unified Studio, customers can go from their existing AWS data to running analytics and machine learning workloads much faster, spending their time on analysis rather than setup and configuration.
In this post, we walk you through how these new capabilities in SageMaker Unified Studio can help you consolidate your fragmented data tools, reduce time to insight, and collaborate across your data teams. Here’s a short demo of the new capabilities:
One-click onboarding of existing AWS datasets
Get started exploring your data with one-click onboarding that provisions and configures environments in minutes instead of weeks. The new onboarding experience can reuse existing AWS Identity and Access Management (IAM) roles to provide access to SageMaker Unified Studio, automatically connecting to data sources across S3 buckets, S3 Tables, AWS Glue Data Catalog, and AWS Lake Formation policies, removing the need for additional data permission setup. Under the covers, a new IAM-based domain and project are created with default notebook and compute resources preconfigured. When complete, you enter SageMaker Unified Studio with all your tools available in the left-side navigation along with built-in samples to accelerate first use, as seen in the following screenshot.
“New features with Amazon Sagemaker will unlock a new paradigm of innovation, allowing Codex to significantly accelerate time-to-value for our customers, and transform them from aging to agentic in weeks, not months.“
– Abhinav Sharma, Chief Data Officer, Codex
You can start directly from Amazon SageMaker, Amazon Athena, Amazon Redshift, or Amazon S3 Tables, giving them a fast path from their existing tools and data to the unified experience in SageMaker Unified Studio. After you choose Get Started and specify an IAM role, SageMaker automatically creates a project with the existing data permissions intact from Data Catalog, Lake Formation, and Amazon S3. As a result, teams can immediately discover and act on their data using the existing data permissions and infrastructure.
The fully managed, web-based notebooks in SageMaker Unified Studio support multiple programming languages, letting you write Python, SQL, and Spark code in the same notebook. The infrastructure adjusts automatically based on your workload, while built-in libraries create charts and insights directly in your workflow. When your analysis scales beyond interactive queries to large-scale data processing, Amazon Athena for Apache Spark engine delivers optimized performance, integrating with the serverless notebook experience to execute analytical workloads efficiently. This serverless approach eliminates the need to provision clusters or maintain servers, reducing the time from question to insight.
“The new SageMaker interface brings clarity and speed to the entire ML lifecycle. Its developer-friendly design has made our experimentation and delivery significantly faster,“
– Sachin Mittal, Product Manager at Deloitte.
As shown in the preceding image, the notebook gives data engineers, analysts, and data scientists one place to perform SQL queries, execute Python code, process large-scale data jobs, run machine learning workloads, and create visualizations without having to switch between tools.
AI-assisted development with Data Agent
To accelerate development further, the new SageMaker Data Agent helps create SQL, Python, or Spark code using natural language prompts. Instead of spending hours writing boilerplate code to connect to your data sources and understand schemas, you can describe what you want to accomplish. The agent analyzes data catalog metadata about your available datasets, schemas, and relationships to provide context-aware assistance.
In the preceding example image, if you prompt Build and analyze a complete sales forecast based on the sample retail data, the agent helps identify the relevant tables and suggests the appropriate joins and analysis approach, transforming what might take hours into minutes. To try this yourself, navigate to the Overview tab in your SageMaker Studio environment and look for the Retail Sales Forecasting with SageMaker XGBoost notebook in the sample notebooks collection—these examples are automatically available when you first set up SageMaker Studio. The agent breaks down complex analytical workflows into manageable, executable steps, so you can move from question to insight faster.
Learn more about SageMaker
In this post, we focused on three new SageMaker Unified Studio capabilities recently made available, but they’re a fraction of the more than 40 launches last year. Here’s a list of videos of re:Invent sessions and the measurable results from leading organizations adopting SageMaker Unified Studio, including:
NatWest Group plans to scale to 72,000 employees having federated data access using SageMaker Unified Studio. Watch their presentation.
Commonwealth Bank of Australia migrated 10 petabytes and 61,000 pipelines into AWS and has setup SageMaker Unified Studio to provide unified access to 40 different lines of business in their ongoing data transformation journey. Watch their presentation.
Carrier Global Corporation improved natural language to SQL agent accuracy by 38% through the SageMaker Catalog’s governed metadata and business glossary. Watch their presentation.
Bayer is now positioned to onboard over 300 TB of biomarker data and integrate siloed omics, clinical, and chemistry data repositories into a cohesive environment built on Amazon SageMaker. Read their story.
Conclusion
Using Amazon SageMaker Unified Studio serverless notebooks, AI-assisted development, and unified governance, you can speed up your data and AI workflows across data team functions while maintaining security and compliance. To learn more visit the SageMaker product page or get started in the SageMaker console.
About the authors
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.