Tag Archives: AWS Lake Formation

How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito

Post Syndicated from Balaji Selva Rajan original https://aws.amazon.com/blogs/big-data/how-samsung-ads-federated-snowflake-iceberg-tables-using-amazon-cognito/

This post is co-written with Ajay Gunasekaran, ML Platform Engineer at Samsung Ads.

Samsung Ads, the advertising division of Samsung Electronics, keeps its feature data in Snowflake as Apache Iceberg tables. Its machine learning (ML) team trains large-scale models on Amazon SageMaker AI, serving programmatic buyers who reach Samsung Smart TV and connected TV audiences. Connecting the two usually means copying the data into Amazon Simple Storage Service (Amazon S3) on a schedule. That adds a second pipeline to operate, a second copy to store, and two sets of permissions to keep in agreement.

Instead of copying the data, a team can write the final pipeline stage as Snowflake-managed Iceberg tables in Amazon S3 and federate them into AWS Lake Formation. Amazon Cognito supplies the OAuth2 tokens that Snowflake trusts. Both applications then read one copy of the data in Amazon S3, with Snowflake managing the tables and the team using Lake Formation to control which principals can read them. Model engineers can choose the query engine that fits each training job, without duplicating data or maintaining a second pipeline.

In this post, we show you how to build the pattern Samsung Ads used. You federate Snowflake Iceberg tables into AWS Lake Formation with Amazon Cognito, then query them from Amazon Athena.

Solution overview

This solution federates the Snowflake Horizon Catalog into AWS Lake Formation, using Amazon Cognito to issue OAuth2 tokens for machine-to-machine authentication. AWS Glue connection runs inside an Amazon Virtual Private Cloud (Amazon VPC), reaching Snowflake through AWS PrivateLink and the Cognito token endpoint through a NAT gateway. AWS analytics engines then query Snowflake-managed Iceberg tables under Lake Formation governance, with no second copy of the tables to maintain. The following diagram shows the query path, from Amazon Athena through AWS Lake Formation and the AWS Glue connection to Snowflake and Amazon S3.

Query path from Amazon Athena through AWS Lake Formation and an AWS Glue connection to Snowflake and Amazon S3

Figure 1: Architecture for Snowflake Horizon Catalog federation to AWS Lake Formation using Amazon Cognito

Here’s how the workflow runs:

  1. A user or application submits a SQL query in Amazon Athena that references a federated table in the Snowflake catalog.
  2. Athena delegates catalog resolution and permission checks to Lake Formation, which evaluates fine-grained access controls on the federated catalog entry.
  3. Lake Formation resolves the table reference through the AWS Glue Data Catalog, which holds the connection configuration for Snowflake and invokes the AWS Glue connection.
  4. AWS Glue connection runs in a private subnet. It sends a client_credentials request to the Amazon Cognito token endpoint with the client ID, client secret, and scope. The request leaves through the NAT gateway and internet gateway because the machine-to-machine token endpoint is public.
  5. Amazon Cognito invokes the Pre Token Generation Lambda trigger (V3_0), which adds the aud claim that Snowflake External OAuth validation requires. Amazon Cognito returns the access token to the connection.
  6. Using that token, AWS Glue connection calls the Snowflake Horizon Catalog Iceberg REST endpoint for table metadata. The response includes the schema and the location of the data files in Amazon S3. An Amazon Route 53 private hosted zone resolves the hostname, because the Snowflake endpoint service doesn’t provide private DNS. The request travels through the VPC endpoint over AWS PrivateLink.
  7. The connection returns the metadata to Lake Formation, which generates temporary, scoped Amazon S3 credentials and passes them to Athena.
  8. Athena uses those credentials to read the Iceberg metadata and Parquet data files from the registered Amazon S3 location, runs the query, and returns results.

Why Amazon Cognito

AWS Glue connection authenticates with the OAuth2 client_credentials grant, so providers Snowflake trusts that support that grant can issue the token. Samsung Ads chose Amazon Cognito for three reasons:

  • Each table lookup against Snowflake depends on a token from the provider, so the team wanted to own it alongside the rest of the solution.
  • An enterprise identity provider is typically administered by a central security team, so each new machine-to-machine client is a cross-team request. Owning the user pool makes it self-service.
  • Each app client is a distinct machine identity, and the security integration trusts the user pool rather than an individual client. You can therefore serve several workloads from one integration, each with its own read-only Snowflake role.

Prerequisites

Before you begin, verify that you have the following:

  • Snowflake account hosted on AWS in the same AWS Region where you create the AWS resources in this post, on the Business-Critical edition or higher, which AWS PrivateLink requires. Confirm the Region with SELECT CURRENT_REGION();. You need ACCOUNTADMIN to create the security integration and to authorize PrivateLink.
    • Snowflake-managed Iceberg tables created in Snowflake Horizon Catalog using CATALOG = 'SNOWFLAKE' with an external volume pointing to the S3 bucket. If you don’t have one, see Configure an external volume for Amazon S3 in the Snowflake documentation.
  • AWS account with an AWS Identity and Access Management (IAM) principal that has data lake administrator permissions in AWS Lake Formation. The principal also needs permissions to create IAM roles and policies, Amazon Cognito user pools, AWS Lambda functions, VPC resources, and AWS CloudShell. We use AWS Command Line Interface (AWS CLI) commands throughout this post.
  • Amazon S3 bucket that holds your Snowflake-managed Iceberg data files.
  • An AWS user with long-term access keys that you delete immediately after Step 4:
    • SYSTEM$AUTHORIZE_PRIVATELINK needs a federation token from the long-term keys of an IAM user, because sts:GetFederationToken rejects role-session credentials. Snowflake persists the authorization, so you can delete the key and the user afterwards and the federation keeps working.

Walkthrough

Two AWS CloudFormation stacks cover the setup that is identical in every deployment. The Snowflake and Lake Formation steps stay as explicit SQL and AWS CLI commands, so you can adapt them to your existing Snowflake account and VPC.

Step What you configure How you run it
1 & 2 Cognito user pool, machine-to-machine app client, and the Lambda function that adds the aud claim Launch Stack, snowflake-federation-oauth
3 Snowflake security integration, read-only role, service user, and table grants Snowflake SQL worksheet
4 PrivateLink authorization for your AWS account AWS CLI and Snowflake SQL worksheet
5 VPC, PrivateLink endpoint, Route 53 private hosted zone, and the Lake Formation data access role Launch Stack, snowflake-federation-network
6 S3 location registration, AWS Glue connection, federated catalog, and Lake Formation permissions AWS CLI
7 Query validation Athena console

Step 1: Deploy the Cognito user pool and token customization Lambda function

Snowflake External OAuth validation requires an aud claim in each access token. The OAuth2 client_credentials grant doesn’t define that claim, so the Pre Token Generation trigger adds it. This stack creates the user pool, the machine-to-machine app client, and a Pre Token Generation Lambda function that adds the claim. The stack also writes the app client secret to AWS Secrets Manager in the format the AWS Glue connection expects. It then generates the Snowflake SQL statements you run in Step 3.

  1. Choose Launch Stack.

Launch Stack

  1. In the Region selector, confirm the Region matches your Snowflake account Region, then choose Next.
  2. For Stack name, keep snowflake-federation-oauth. Leave all parameters at their default values, then choose Next.
  3. On Configure stack options, keep the defaults and choose Next.
  4. On Review and create, select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Submit.
  5. When the stack status is CREATE_COMPLETE, choose the Outputs tab.

You use six of the outputs in later steps.

Output Used in
VerifyAudClaimCommand Step 2
SnowflakeSecurityIntegrationSQL Step 3
OAuthClientSecretArn Step 5 and Step 6
CognitoTokenEndpoint, CognitoClientId and CognitoScope Step 3 and Step 6
CloudFormation Outputs tab listing the six stack outputs used in later steps

Figure 2: CloudFormation Outputs tab showing the values used in later steps

Note: After the connection validates in Step 6, AWS Glue caches the token it mints by writing it back into this secret. Treat read access to the secret as read access to the federated tables.

Step 2: Verify the access token

Confirm that the Lambda function is adding the claim before you configure Snowflake. A missing claim doesn’t fail here. It surfaces several steps later as an AWS Glue connection that can’t authenticate, which is harder to attribute.

  1. On the Outputs tab, copy the value of VerifyAudClaimCommand.
  2. Choose the AWS CloudShell icon on the console navigation bar, as shown in the following screenshot.
  3. Paste the command and run it. Confirm two values in the output. The aud claim matches your Cognito client ID, and scope is snowflake/catalog.read.
Decoded access token payload showing the injected aud claim and the snowflake/catalog.read scope

Figure 3: Decoded access token payload showing the injected aud claim and the catalog.read scope

Step 3: Configure Snowflake

In this step you configure Snowflake to trust Cognito tokens, create the service user that Lake Formation authenticates as, and grant it read access to your tables.

  1. Run the following statement in a Snowflake worksheet, replacing the three placeholders with the CognitoIssuerUri, CognitoJwksUrl, and CognitoClientId outputs from Step 1. The SnowflakeSecurityIntegrationSQL output contains the same statement with your values filled in. CREATE omits OR REPLACE, so it fails rather than overwriting an existing integration.
USE ROLE ACCOUNTADMIN;

-- Return a row if the integration exists. If so, use ALTER SECURITY INTEGRATION instead of CREATE
SHOW INTEGRATIONS LIKE 'lakeformation_cognito_oauth';

CREATE SECURITY INTEGRATION lakeformation_cognito_oauth
  TYPE = EXTERNAL_OAUTH
  ENABLED = TRUE
  EXTERNAL_OAUTH_TYPE = CUSTOM
  EXTERNAL_OAUTH_ISSUER = '<CognitoIssuerUri>'
  EXTERNAL_OAUTH_JWS_KEYS_URL = '<CognitoJwksUrl>'
  EXTERNAL_OAUTH_TOKEN_USER_MAPPING_CLAIM = 'sub'
  EXTERNAL_OAUTH_SNOWFLAKE_USER_MAPPING_ATTRIBUTE = 'login_name'
  EXTERNAL_OAUTH_AUDIENCE_LIST = ('<CognitoClientId>')
  EXTERNAL_OAUTH_ANY_ROLE_MODE = 'ENABLE'
  EXTERNAL_OAUTH_BLOCKED_ROLES_LIST = ('ACCOUNTADMIN', 'SECURITYADMIN', 'USERADMIN', 'SYSADMIN');

Snowflake normally takes the role from a session:role:<ROLE> scope, which Amazon Cognito custom scopes can’t carry because they use the form resourceServer/scope. The statement therefore sets EXTERNAL_OAUTH_ANY_ROLE_MODE to ENABLE, which lets the session use any role granted to the service user, so keep that user to a single role. The blocked roles list also covers SYSADMIN and USERADMIN, which the Snowflake default list doesn’t.

  1. Create the read-only role and the service user. Set LOGIN_NAME to the CognitoClientId output, because the token’s sub claim carries the client ID and the integration maps sub to login_name.
CREATE ROLE lakeformation_catalog_reader
  COMMENT = 'Read-only access for AWS Lake Formation catalog federation';

CREATE USER lakeformation_service_user_cognito
  LOGIN_NAME = '<CognitoClientId from Step 1>'
  DEFAULT_ROLE = lakeformation_catalog_reader
  TYPE = SERVICE;

GRANT ROLE lakeformation_catalog_reader TO USER lakeformation_service_user_cognito;
  1. Grant the role read access to the tables you want to federate. For the privileges the Iceberg REST Catalog API requires, see Configure access control in the Snowflake documentation.
GRANT USAGE ON DATABASE <database>
  TO ROLE lakeformation_catalog_reader;

GRANT USAGE ON SCHEMA <database>.<schema>
  TO ROLE lakeformation_catalog_reader;

GRANT SELECT ON ALL ICEBERG TABLES IN SCHEMA <database>.<schema>
  TO ROLE lakeformation_catalog_reader;

GRANT SELECT ON FUTURE ICEBERG TABLES IN SCHEMA <database>.<schema>
  TO ROLE lakeformation_catalog_reader;

Note: Federate a schema that contains only Snowflake-managed Iceberg tables. An externally managed table can fail the entire table listing. Converted tables, tables loaded from Iceberg-compatible Parquet files and clones whose source table the role can’t access aren’t supported. Tables with mixed-case quoted names aren’t federated, because the connection you create in Step 6 sets CATALOG_CASING_FILTER to UPPERCASE_ONLY.

Snowflake masking and row access policies don’t apply when Athena reads a federated table, because Lake Formation vends the Amazon S3 credentials and Athena reads the Parquet files directly. Keep tables that rely on them out of the federated schema or recreate the controls in Lake Formation and test them.

  1. Next, find the Amazon S3 bucket and prefix your external volume writes to. You need both in Step 5.
DESC EXTERNAL VOLUME <your_external_volume>;

STORAGE_BASE_URL holds the bucket and an optional prefix. For s3://my-bucket/iceberg/, the bucket is my-bucket and the prefix is iceberg. If nothing follows the bucket name, leave the prefix blank in Step 5. A prefix is better, because it limits the data access role to that path. If ENCRYPTION_TYPE is AWS_SSE_KMS, note the key in ENCRYPTION_KMS_KEY_ID for Step 5. If it is NONE, check the bucket’s default encryption in the Amazon S3 console. The Snowflake role that owns the Iceberg tables must keep the USAGE privilege on the external volume, or operations that request vended credentials fail.

If the bucket uses a customer managed KMS key, set KMS_KEY_ID on the external volume to the full key ARN. Then add the following statement to the key policy, so the session Snowflake uses for the role in STORAGE_AWS_ROLE_ARN can read the encrypted files. Without it, the AWS Glue connection fails with an error that says no session policy allows kms:Decrypt.

{
  "Sid": "AllowSnowflakeVolumeSession",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:sts::<account-id>:assumed-role/<external-volume-role>/snowflake"
  },
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey",
    "kms:GenerateDataKey"
  ],
  "Resource": "*"
}

Snowflake requires proof that you own the AWS account before it accepts a PrivateLink connection. You provide that proof as an AWS Security Token Service (AWS STS) federation token. You must call get-federation-token with the long-term credentials of an IAM user, because AWS STS doesn’t support calling it with temporary credentials from an assumed role.

  1. Using an IAM user with long-term access keys, generate a federation token:
aws sts get-federation-token --name snowflake-privatelink --duration-seconds 900 --profile <iam-user-profile>

The token appears in the SQL statement, so Snowflake keeps it in query history. The 15-minute duration limits how long it stays valid.

  1. In your Snowflake worksheet, pass your AWS account ID and the full JSON output from the previous command:
USE ROLE ACCOUNTADMIN;
SELECT SYSTEM$AUTHORIZE_PRIVATELINK('<aws-account-id>', '<full-federation-token-json>');

The output confirms that Snowflake authorized PrivateLink for your account.

Snowflake worksheet output confirming PrivateLink authorization for the AWS account

Figure 4: Snowflake output confirming PrivateLink authorization

  1. Retrieve the two values you need for the stack in Step 5:
WITH cfg AS (SELECT PARSE_JSON(SYSTEM$GET_PRIVATELINK_CONFIG()) AS j)
SELECT j:"regionless-privatelink-account-url"::string AS privatelink_hostname,
       j:"privatelink-vpce-id"::string AS vpce_service_name
FROM cfg;

Record both values. privatelink_hostname is the Snowflake PrivateLink hostname, and vpce_service_name is the endpoint service name.

After authorization succeeds, delete the temporary IAM user and its access key. Revoking PrivateLink access later requires a new federation token, so recreate a similar user at that point. For more information, see How to retrieve a Federation Token from AWS for PrivateLink Self-Service.

This stack creates the interface VPC endpoint to the Snowflake PrivateLink service and the Amazon Route 53 private hosted zone that resolves your Snowflake hostname to it. It also creates the security group that the endpoint shares with the AWS Glue connection. Keep Existing VPC ID blank and it also builds a VPC with public and private subnets and a NAT gateway.

To use an existing VPC, supply Existing VPC ID and two or more private subnet IDs in different Availability Zones. The stack then creates no subnets and no NAT gateway, so confirm three things first:

  • You have enabled DNS resolution and DNS hostnames on the VPC.
  • The private subnets have a 0.0.0.0/0 route to a NAT gateway, which the AWS Glue connection uses to reach the Amazon Cognito token endpoint. Amazon Cognito doesn’t support AWS PrivateLink, but the traffic stays on the AWS network because both sit in the same Region. Without this route, the stack still succeeds and the connection fails in Step 6.
  • No Amazon Route 53 private hosted zone for privatelink.snowflakecomputing.com is already associated with the VPC. A second zone for the same domain makes name resolution unpredictable.

To deploy the stack, complete the following steps:

  1. Choose Launch Stack.

Launch Stack

  1. Confirm the Region selector matches your Snowflake account Region, then choose Next.
  2. For Stack name, keep snowflake-federation-network, then enter the following parameters.
Parameter Value
Snowflake PrivateLink hostname privatelink_hostname from Step 4
Snowflake PrivateLink service name vpce_service_name from Step 4
Iceberg S3 bucket name The bucket in the STORAGE_BASE_URL output from Step 3
Iceberg data key prefix The prefix in the STORAGE_BASE_URL output from Step 3. Leave blank if you have none
Iceberg bucket KMS key ARN KMS key ARN. Leave blank if the bucket does’t use KMS
OAuth client secret ARN OAuthClientSecretArn from Step 1

If the bucket uses SSE-KMS, the stack grants the data access role kms:Decrypt on the key you enter. The default key policy already allows this. If your key policy doesn’t delegate access to IAM, add the role as a key user.

  1. Choose Next, keep the defaults on Configure stack options, and choose Next.
  2. On Review and create, select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Submit.
  3. When the stack status is CREATE_COMPLETE, which takes about 5 minutes, choose the Outputs tab and record PrivateSubnetIds, GlueConnectionSecurityGroupId, LakeFormationDataRoleArn, and RegisteredS3Location. You use all four values in Step 6.

NAT gateway and the interface VPC endpoint carry hourly and data processing charges. For more information, see Amazon VPC pricing.

Step 6: Register with Lake Formation and create the federated catalog

The remaining setup is a sequence of AWS CLI commands. Run them in AWS CloudShell in the same Region as both stacks.

  1. Set the values used by the commands that follow. Fill in the Region, and the stack outputs from Step 1 and Step 5.
REGION='<your Region>'
ACCT=$(aws sts get-caller-identity --query Account --output text)
PREFIX=snowflake-federation
CONN=$PREFIX-iceberg-connection
CATALOG=$PREFIX-catalog
S3_LOCATION='<RegisteredS3Location from Step 5>'
LFROLE='<LakeFormationDataRoleArn from Step 5>'
SECRET_ARN='<OAuthClientSecretArn from Step 1>'
TOKEN_URL='<CognitoTokenEndpoint from Step 1>'
CLIENT_ID='<CognitoClientId from Step 1>'
SCOPE='<CognitoScope from Step 1>'
SUBNET='<one subnet ID from PrivateSubnetIds in Step 5>'
SG='<GlueConnectionSecurityGroupId from Step 5>'
WORKSPACE_URL='https://<privatelink_hostname from Step 4>/polaris/api/catalog'
REMOTE_DB='<Snowflake database name, uppercase>'
SCHEMA='<Snowflake schema name, lowercase>'
QUERY_PRINCIPAL='<IAM role or user ARN that will run Athena queries>'
  1. Register the Amazon S3 location that holds your Iceberg data. This is what lets Lake Formation vend scoped credentials to Athena for the underlying data files.
aws lakeformation register-resource \
  --resource-arn "$S3_LOCATION" \
  --role-arn "$LFROLE" --with-federation
  1. Create the AWS Glue connection. This writes the definition to a file, then creates it.
cat > conn.json <<JSON
{
  "Name": "$CONN",
  "Description": "Snowflake Iceberg REST catalog through AWS Cognito.",
  "ConnectionType": "ICEBERGRESTCATALOG",
  "ConnectionProperties": {
    "INSTANCE_URL": "$WORKSPACE_URL",
    "ROLE_ARN": "$LFROLE",
    "CATALOG_CASING_FILTER": "UPPERCASE_ONLY"
  },
  "AuthenticationConfiguration": {
    "AuthenticationType": "OAUTH2",
    "SecretArn": "$SECRET_ARN",
    "OAuth2Properties": {
      "OAuth2GrantType": "CLIENT_CREDENTIALS",
      "TokenUrl": "$TOKEN_URL",
      "OAuth2ClientApplication": { "UserManagedClientApplicationClientId": "$CLIENT_ID" },
      "TokenUrlParametersMap": { "scope": "$SCOPE" }
    }
  },
  "PhysicalConnectionRequirements": {
    "SubnetId": "$SUBNET",
    "SecurityGroupIdList": ["$SG"]
  }
}
JSON

aws glue create-connection --connection-input file://conn.json
  1. Check the connection status. AWS Glue validates asynchronously. It creates the network interface, completes the OAuth2 exchange with Amazon Cognito, and calls the Snowflake Iceberg REST API over AWS PrivateLink. Repeat this command until it returns READY, which takes approximately 5–10 minutes.
aws glue get-connection --name $CONN --query Connection.Status --output text

If it returns FAILED, read the reason and correct the Snowflake configuration from Step 3. A missing security integration, service user, or grant is the usual cause.

aws glue get-connection --name $CONN --query Connection.StatusReason --output text

Then delete the connection before you retry, because the name stays in use.

aws glue delete-connection --connection-name $CONN
  1. Register the connection with Lake Formation using privileged access. A federated catalog can only use a connection registered this way.
aws lakeformation register-resource \
  --resource-arn arn:aws:glue:$REGION:$ACCT:connection/$CONN \
  --role-arn "$LFROLE" --with-federation --with-privileged-access
  1. Create the federated catalog, then confirm the tables surface through it.
aws glue create-catalog --name $CATALOG --catalog-input '{
  "FederatedCatalog": {"Identifier": "'"$REMOTE_DB"'", "ConnectionName": "'"$CONN"'"},
  "CreateDatabaseDefaultPermissions": [],
  "CreateTableDefaultPermissions": []
}'

aws glue get-tables --catalog-id $ACCT:$CATALOG --database-name $SCHEMA --query 'TableList[].Name'
  1. Grant Lake Formation permissions to the data teams that will query the tables. The principal that created the catalog can already query it. Every other principal needs an explicit grant, including other Lake Formation administrators.
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier="$QUERY_PRINCIPAL" \
  --resource '{"Database":{"CatalogId":"'"$ACCT:$CATALOG"'","Name":"'"$SCHEMA"'"}}' \
  --permissions DESCRIBE

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier="$QUERY_PRINCIPAL" \
  --resource '{"Table":{"CatalogId":"'"$ACCT:$CATALOG"'","DatabaseName":"'"$SCHEMA"'","TableWildcard":{}}}' \
  --permissions SELECT DESCRIBE

Note: The catalog ID for a federated catalog is :, not the bare account ID. Using the bare account ID grants permissions on the local AWS Glue Data Catalog instead, which succeeds and then leaves Athena returning an access denied error.

Step 7: Query the federated tables from Amazon Athena

  1. Open the Amazon Athena console and choose Query editor. If this is the first time you use Athena in this Region, choose Settings, then Manage, and set an Amazon S3 location for query results.
  2. In the left panel, select the following.
Field Value
Data source AwsDataCatalog
Catalog Your federated catalog, for example snowflake-federation-catalog
Database Your Snowflake schema name, in lowercase
  1. Run the following query:
SELECT * FROM "<snowflake_schema>"."<snowflake_tablename>" LIMIT 10;

Rows in the results pane confirm the federation works end to end, as shown in the following screenshot.

Amazon Athena query results returning 10 rows from a Snowflake-managed Iceberg table

Figure 5: Query results in Amazon Athena from a Snowflake-managed Iceberg table

To confirm that Athena reads current Snowflake data rather than a copy, insert a row in Snowflake and run the query again. The new row appears with no additional data movement into your AWS account.

Dropping a table in Snowflake doesn’t revoke the Lake Formation permissions granted on it. Revoke those permissions explicitly when you remove a federated table.

Clean up

To avoid incurring future charges, delete the resources you created while following this post. Complete the following steps in both your AWS account and your Snowflake account.

AWS account

  1. On the AWS Lake Formation console, choose Catalogs, select your federated catalog, then choose Actions and Delete catalog.
  2. Record the network interfaces the AWS Glue connection created, while the connection still exists. Filtering on the Step 5 security group and the ICEBERGRESTCATALOG description keeps the list to this deployment.
SG='<GlueConnectionSecurityGroupId from Step 5>'
aws ec2 describe-network-interfaces \
  --filters "Name=group-id,Values=$SG" "Name=description,Values=*ICEBERGRESTCATALOG" \
  --query 'NetworkInterfaces[].NetworkInterfaceId' --output text | tr '\t' '\n' > glue-enis.txt

cat glue-enis.txt
  1. Deregister and delete the AWS Glue connection. Do this before you delete the stacks, because the stack deletion fails while the connection’s network interfaces still hold the security group. Run the following in AWS CloudShell, replacing the Region, account ID, and connection name with your values.
aws lakeformation deregister-resource --resource-arn arn:aws:glue:<region>:<account-id>:connection/<glue-connection-name>
aws glue delete-connection --connection-name <glue-connection-name>
  1. Deleting the connection releases its network interfaces, but they stay in the in-use state for several minutes and AWS Glue doesn’t remove them. Wait for each one to reach available, because the Amazon Elastic Compute Cloud (Amazon EC2) console Detach action fails even with Force detachment selected. The following script checks each interface you recorded and deletes it when ready.
#!/bin/bash
while read -r eni; do
  [ -z "$eni" ] && continue
  for attempt in $(seq 1 20); do
    status=$(aws ec2 describe-network-interfaces \
      --network-interface-ids "$eni" \
      --query 'NetworkInterfaces[0].Status' \
      --output text 2>/dev/null || echo deleted)
    if [ "$status" = "available" ]; then
      aws ec2 delete-network-interface --network-interface-id "$eni"
      echo "$eni deleted"
      break
    fi
    if [ "$status" = "deleted" ]; then
      echo "$eni no longer exists"
      break
    fi
    echo "$eni is $status, checking again in 30 seconds"
    sleep 30
  done
done < glue-enis.txt
  1. On the Lake Formation console, under Administration, choose Data lake locations, select the registered Amazon S3 location, then choose Actions and Remove.
  1. On the AWS CloudFormation console, delete snowflake-federation-network first, then snowflake-federation-oauth. This removes the VPC endpoint, the Amazon Route 53 private hosted zone, the security group, and the IAM role. It also removes the Amazon Cognito user pool, the AWS Lambda function, and the client secret. If the stack created the VPC, subnets, and NAT gateway, it removes those too.

Snowflake account

  1. Open a SQL worksheet and run the following:
USE ROLE ACCOUNTADMIN;
DROP SECURITY INTEGRATION lakeformation_cognito_oauth;

USE ROLE USERADMIN;
DROP USER lakeformation_service_user_cognito;
DROP ROLE lakeformation_catalog_reader;
  1. Revoke the PrivateLink authorization for your AWS account. This requires a fresh federation token, generated the same way as during setup, from an IAM user with long-term access keys:
aws sts get-federation-token --name snowflake-privatelink --duration-seconds 900 --profile <iam-user-profile>
USE ROLE ACCOUNTADMIN;
SELECT SYSTEM$REVOKE_PRIVATELINK('<aws-account-id>', '<full-federation-token-json>');
  1. Confirm the account is no longer authorized:
SELECT SYSTEM$GET_PRIVATELINK_AUTHORIZED_ENDPOINTS();

Deleting the federated catalog and connection doesn’t delete data in Snowflake or Amazon S3. It only removes the federation configuration in your AWS account.

Conclusion

In this post, you learned how to federate Snowflake Horizon Catalog Iceberg tables into AWS Lake Formation and query them from Amazon Athena. Amazon Cognito handled the OAuth2 tokens for machine-to-machine authentication. To extend this pattern to additional workloads, create a new app client in your Cognito user pool and a corresponding read-only Snowflake role.

To get started, see Catalog federation to remote Iceberg catalogs and Federate to Snowflake Iceberg Catalog in the AWS Lake Formation Developer Guide, and External OAuth overview in the Snowflake documentation. If you have feedback about this post, submit comments in the Comments section.

 


About the authors

Ajay Gunasekaran

Ajay Gunasekaran

Ajay is an ML Platform Engineer at Samsung Ads. He has been instrumental in building the ML platform across the Snowflake and AWS ecosystems and is passionate about making complex infrastructure simple, reliable, and easy to use. Curious by nature, he enjoys learning, sharing ideas, and solving challenging problems.

Balaji Selva Rajan

Balaji Selva Rajan

Balaji is a Technical Account Manager at AWS, specializing in Cloud Operations, Storage, and Data Analytics. He helps customers to run their workloads efficiently on AWS and is particularly interested in turning recurring customer problems into repeatable, reusable patterns.

David Zhang

David Zhang

David is a Senior Specialist Solutions Architect specializing in Data & AI at AWS, where he focuses on designing and implementing open lakehouse architectures and agentic AI solutions. He is an active speaker at AWS events and contributes to technical publications and open-source initiatives. He is also the co-author of Apache Iceberg on AWS book. Outside of work, David enjoys staying active through volleyball, tennis, and basketball.

Steven Alyekhin

Steven Alyekhin

Steven is a Senior Solutions Architect on the AWS Canada ISV team, where he helps software companies build generative AI, machine learning, and data solutions on AWS.

Deliver multi-tenant AI analytics at scale with AgentCore and Lake Formation

Post Syndicated from Fortune Hui original https://aws.amazon.com/blogs/big-data/deliver-multi-tenant-ai-analytics-at-scale-with-agentcore-and-lake-formation/

Governed, multi-tenant data access is a critical requirement for software and business-to-consumer (B2C) products that embed AI-powered analytics. A data agent adds significant value by answering natural-language queries over structured data, yet when that agent serves external users, the hardest problem isn’t natural language to SQL. It’s making sure each tenant can only access their authorized rows, columns, and tables.

The challenge of fine-grained data access for external users

Modern-day data agents are often built with an orchestrating framework, connected to a Model Context Protocol (MCP) server for data queries. The data agent translates validated natural-language input into SQL and executes queries on behalf of users across tenant boundaries. Because the agent operates with broad data reach, it needs a robust, fine-grained access control mechanism that makes sure each user can access only their authorized tables, columns, rows, or cells.

AWS offers various data authorization mechanisms to fit different use cases.

  1. AWS IAM Identity Center (IDC) + Trusted Identity Propagation (TIP) + AWS Lake Formation (see the blog post Simplify data lake access control for your enterprise users with trusted identity propagation in AWS IAM Identity Center, AWS Lake Formation, and Amazon S3 Access Grants).
  2. Amazon Cognito + Lake Formation.

Choosing the right identity model for external users

IAM Identity Center with Trusted Identity Propagation (TIP) is designed for workforce identity, propagating enterprise identities across AWS analytics services through SAML 2.0 and SCIM federation, making it well suited for enterprise federation scenarios.

For external-facing B2C or multitenant applications, Amazon Cognito is a better fit because it offers:

  • Mobile and single page applications (SPAs): Cognito User Pools provide OpenID Connect (OIDC) integration with hosted UI and native SDKs for browser and mobile apps.
  • Social identity federation: Cognito federates with social providers such as Google, Facebook, Apple, and other OIDC-compatible providers for frictionless consumer sign-up.
  • Self-service user management: Cognito offers built-in flows for self-registration, email/phone verification, password recovery, and multi-factor authentication (MFA), reducing operational overhead for large external user bases.

For these reasons, Cognito User Pools combined with Identity Pools provide a natural fit for external users at scale.

Who this is for

  • Software as a service (SaaS) application architects building customer-facing analytics where each tenant sees only their data.
  • B2C data product teams serving hundreds of thousands of external users.
  • Partner portal developers providing scoped data access to network partners.
  • ISVs migrating to AWS that need agentic data capabilities with tenant isolation.

Solution overview

This post shows how to apply this traditional Cognito and Lake Formation approach on AgentCore Gateway, a capability of Amazon Bedrock AgentCore. The result is a data agent that enforces row- and column-level security based on the authenticated user’s identity. Amazon Bedrock AgentCore provides a managed environment for deploying and operating agents at scale. AgentCore Gateway is a fully managed gateway that provides a single, secure entry point for agentic traffic, connecting agents to tools and to other agents.

Architecture and flow

The following diagram illustrates the end-to-end architecture.

End-to-end architecture of the multi-tenant data agent

Figure 1: End-to-end architecture of the multi-tenant data agent

  1. User signs in through Cognito Hosted UI and receives OAuth tokens. The ID token contains the custom:team claim.
  2. User sends a natural language query in the frontend.
  3. The frontend invokes AgentCore runtime with the access token (Authorization header) and ID token (X-ID-Token header).
  4. AgentCore runtime forwards both tokens through the agent code to the MCP Gateway.
  5. MCP Gateway validates the access token with JWT authorizer and propagates X-ID-Token header to the AWS Lambda Target.
  6. Target Lambda (with minimal permissions) receives the ID token from X-ID-Token.
  7. Cognito Identity Pool exchanges the ID token with GetCredentialsForIdentity operation for temporary AWS credentials scoped to the user’s AWS Identity and Access Management (IAM) role through rule-based mapping.
  8. Lambda executes the Amazon Athena query using those user-scoped credentials.
  9. Lake Formation evaluates the IAM role’s LF-Tag permissions and data cell filters to enforce row- and column-level security.
  10. User receives only their authorized data.

Technical design: defining permissions

The following diagram shows how the custom:team attribute flows through the permission chain.

Permission chain mapping the team tag into Lake Formation permissions

Figure 2: Data access permission chain mapping the team tag into Lake Formation permissions

  1. Custom attribute (Cognito User Pool): Each user is tagged with custom:team. The value appears as a claim in the ID token.
  2. Rule-based mapping (Cognito Identity Pool): The team claim maps to a corresponding IAM role ARN through rule-based mapping.
  3. Data permission for IAM role (Lake Formation): The IAM role is granted access to AWS Glue Data Catalog tables through the Lake Formation permission system, which supports tag-based access control (TBAC) and role-based access control (RBAC).
  4. Tag-based access control (Lake Formation): A Lake Formation tag (LF-Tag) applied at the table or row level (through data cell filters) determines what data each role can access.

Custom attribute (Cognito User Pool)

The User Pool holds your external users with custom attributes that drive access control:

# User Pool with custom team attribute
self.user_pool = cognito.UserPool(
    self, "UserPool",
    custom_attributes={
        "team": cognito.StringAttribute(mutable=True),
    },
)

Users authenticate and receive an ID token containing their claims:

{
    "sub": "abc-123",
    "email": "[email protected]",
    "custom:team": "Team A"
}

Here’s how the user attribute looks in the Cognito User Pool.

The custom:team attribute defined in the Cognito User Pool console

Figure 3: The custom:team attribute defined as a custom attribute in the Cognito User Pool

Rule-based mapping (Cognito Identity Pool)

The Identity Pool maps these claims to IAM roles using rules-based role mapping:

# Rules: custom:team claim → IAM role
cognito.CfnIdentityPoolRoleAttachment(
    self, "RoleAttachment",
    identity_pool_id=identity_pool.ref,
    role_mappings= {
        "cognito": {
            "type": "Rules",
            "rules_configuration": {
                "rules": [
                    {
                        "claim": "custom:team",
                        "match_type": "Equals",
                        "value": "Team A",
                        "role_arn": team_a_role.role_arn,
                    },
                    {
                        "claim": "custom:team",
                        "match_type": "Equals",
                        "value": "Team B",
                        "role_arn": team_b_role.role_arn,
                    },
                ],
            },
        },
    },
)

When the Target Lambda exchanges the ID token, Identity Pool reads custom:team, matches it to a rule, and returns temporary credentials for the corresponding IAM role.

Role mapping rules in the Cognito Identity Pool that map the team attribute

Figure 4: The team attribute mapped by role mapping rules defined in the Cognito Identity Pool

Data permission granted to IAM role (Lake Formation)

With Lake Formation TBAC, you can control access declaratively with tags instead of writing filtering logic in your application. The two IAM roles are granted access to resources assigned to an LF-Tag.

Define an LF-Tag:

Tag Key: team
Tag Values: TeamA, TeamB, shared

The following figure shows the LF-Tag definition in the Lake Formation console, with the tag key “team” and its three allowed values.

Lake Formation LF-Tags for team-level data access

Figure 5: Lake Formation LF-Tags for team-level data access

Grant tag-based permissions to each role:

# TeamA role can access TeamA + shared tagged resources
aws lakeformation grant-permissions \
    --principal '{"DataLakePrincipalIdentifier": "arn:aws:iam::ACCOUNT:role/TeamA"}' \
    --resource '{"LFTagPolicy": {"ResourceType": "TABLE", "Expression": [{"TagKey": "team", "TagValues": ["TeamA", "shared"]}]}}' \
    --permissions "SELECT" "DESCRIBE"

The following two figures show the resulting Lake Formation data permissions for the TeamA and TeamB IAM roles, including table-level access through LF-Tags and row-level access through data cell filters.

Lake Formation permissions granting the TeamA IAM role table and row-level access

Figure 6: Lake Formation data permissions granting the TeamA IAM role table access through LF-Tags and row-level access through a data cell filter

Lake Formation permissions granting the TeamB IAM role table and row-level access

Figure 7: Lake Formation data permissions granting the TeamB IAM role table access through LF-Tags and row-level access through a data cell filter

Lake Formation data access control (TBAC)

AWS Glue Data Catalog resources are assigned to an LF-Tag. Assignment can be at the database, table, or row level. This example includes table-level and row-level assignment (through the use of a data cell filter).

Assign tags to tables:

Table LF Tag Who Can Access
venue team=TeamA Team A only
category team=TeamB Team B only
users team=shared Both teams
sales_extended Row filter Per-team rows

The three figures that follow show the LF-Tag assignment for each table in the Lake Formation console: “venue” tagged as TeamA, “category” tagged as TeamB, and “users” tagged as shared.

LF-Tag assignment on the venue table with team value TeamA

Figure 8: LF-Tag assignment on the venue table for the team tag with value TeamA

LF-Tag assignment on the category table with team value TeamB

Figure 9: LF-Tag assignment on the category table for the team tag with value TeamB

LF-Tag assignment on the users table with team value shared

Figure 10: LF-Tag assignment on the users table for the team tag with value shared

For row-level security, create data cell filters:

# TeamA only sees rows where team = 'Team A'
aws lakeformation create-data-cells-filter \
    --table-data '{
    "DatabaseName": "tickit2",
    "TableName": "sales_extended",
    "Name": "TeamAFilter",
    "RowFilter": {"FilterExpression": "team = '\''Team A'\''"},
    "ColumnWildcard": {}
}'

The following three figures show the data cell filter configuration in the Lake Formation console: an overview of the two filters applied to the “sales_extended” table, followed by the row filter expression for Team A and the row filter expression for Team B.

Two data cell filters applied to the sales_extended table

Figure 11: Two data cell filters applied to the sales_extended table, one per team

Data cell filter with the row filter expression for Team A

Figure 12: Data cell filter defined with the row filter expression team = ‘Team A’

Data cell filter with the row filter expression for Team B

Figure 13: Data cell filter defined with the row filter expression team = ‘Team B’

Technical design: enforcing permissions

A key design consideration in this solution is how role mapping runs securely without tampering. The design uses the Lambda target as a credential broker that exchanges the user-scoped ID token for an IAM data role through Identity Pool. The Lambda execution role has minimal IAM permissions to the Cognito Identity Pool and no data permissions. It only assumes the IAM data roles. The ID token propagated through AgentCore Gateway is validated intact with an inbound JWT authorizer.

AgentCore Gateway: Header propagation

The AgentCore Gateway propagates the ID token to the Target Lambda using metadataConfiguration on the gateway target:

{
    "metadataConfiguration": {
        "allowedRequestHeaders": ["X-ID-Token"],
        "allowedResponseHeaders": ["X-ID-Token"]
    }
}

The following figure shows this metadata configuration in the AgentCore Gateway console, with “X-ID-Token” listed under both the propagated request headers and propagated response headers.

X-ID-Token configured as a propagated request and response header on the AgentCore Lambda target

Figure 14: X-ID-Token configured as a propagated request header and response header on the AgentCore Lambda target

The agent code extracts both tokens from the AgentCore runtime headers and passes them to the Gateway:

@app.entrypoint
async def invoke(payload, context):
    headers = context.request_headers
    # Runtime lowercases custom headers
    access_token = headers.get('Authorization')
    id_token = headers.get(
        'x-amzn-bedrock-agentcore-runtime-custom-x-id-token')
    with get_mcp_client(access_token, id_token) as client:
        tools = client.list_tools_sync()
        agent = Agent(..., tools=tools)
        # ...

The Target Lambda receives the propagated header in:

propagated = context.client_context.custom.get(
    'bedrockAgentCorePropagatedHeaders', {}
)
id_token = propagated.get('X-ID-Token')

The Target Lambda: A credential broker, not a data accessor

This is the key architectural decision. The Target Lambda execution role has only two permissions:

self.target_fn.add_to_role_policy(iam.PolicyStatement(
    actions=[
        "cognito-identity:GetId",
        "cognito-identity:GetCredentialsForIdentity",
    ],
    resources=["*"],
))

No Athena permissions. No S3 permissions. No Glue permissions. The Lambda exchanges the user’s ID token for scoped credentials and uses those to query:

def get_identity_pool_session(id_token: str):
    """Exchange ID token for user-scoped IAM credentials."""
    cognito_identity = boto3.client('cognito-identity',
        region_name=COGNITO_REGION)
    provider = f'cognito-idp.{COGNITO_REGION}.amazonaws.com/{USER_POOL_ID}'

    # Step 1: Get identity ID from token
    identity = cognito_identity.get_id(
        IdentityPoolId=IDENTITY_POOL_ID,
        Logins={provider: id_token}
    )

    # Step 2: Get credentials (role determined by Identity Pool rules)
    creds = cognito_identity.get_credentials_for_identity(
        IdentityId=identity['IdentityId'],
        Logins={provider: id_token}
    )

    # Step 3: Create session with user-scoped credentials
    return boto3.Session(
        aws_access_key_id=creds['Credentials']['AccessKeyId'],
        aws_secret_access_key=creds['Credentials']['SecretKey'],
        aws_session_token=creds['Credentials']['SessionToken']
    )

The authenticated role: Trust and permissions

Each team role trusts the Cognito Identity Pool and has Lake Formation + Athena permissions:

{
    "Effect": "Allow",
    "Principal": {
        "Federated": "cognito-identity.amazonaws.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
        "StringEquals": {
            "cognito-identity.amazonaws.com:aud": "<IDENTITY_POOL_ID>"
        },
        "ForAnyValue:StringLike": {
            "cognito-identity.amazonaws.com:amr": "authenticated"
        }
    }
}

The role’s IAM policy grants Amazon Athena, AWS Glue, Amazon Simple Storage Service (Amazon S3), and lakeformation:GetDataAccess. But Lake Formation TBAC further restricts what data the role can actually see based on its LF-Tag permissions.

The Athena query then runs under the user’s assumed role:

def execute_query(session, sql):
    # This Athena client uses the user's credentials, not Lambda's
    athena = session.client('athena', region_name=ATHENA_REGION)
    response = athena.start_query_execution(
        QueryString=sql,
        QueryExecutionContext = {'Database': 'tickit2'},
        ResultConfiguration = { 'OutputLocation': ATHENA_OUTPUT }
    )
    # Lake Formation enforces TBAC transparently
    ...

Deployment

Our solution is implemented and wrapped within AWS Cloud Development Kit (AWS CDK) stacks, which are available in the GitHub repository.

Prerequisites

To deploy the solution, you need the following prerequisites:

  1. An AWS account.
  2. AWS Command Line Interface (AWS CLI).
  3. Valid AWS credentials configured for the AWS CLI. For more information, see Configuring settings for the AWS CLI.
  4. Python 3.10+.
  5. uv Python package manager. For more information, see Installing uv.
  6. Docker.
  7. Node.js and npm installed for the frontend.
  8. AWS CDK. For more information, see Getting started with the AWS CDK.

Manual preparation before deployment

Before deploying the CDK stacks, complete the following manual steps:

  1. Register the CDK CloudFormation execution role as a Lake Formation administrator.

The FgacLakeFormationStack creates LF-Tags and grants Lake Formation permissions. This requires the CloudFormation execution role to be a Lake Formation Data Lake Administrator. Follow Create a data lake administrator for detailed instructions.

Adding an IAM role as a Data Lake Administrator in Lake Formation

Figure 15: Adding an IAM role identity as a Data Lake Administrator in Lake Formation

Run the following AWS CLI command (replace [<YOUR_PROFILE>], [<ACCOUNT_ID>], [<REGION>] with your values):

aws lakeformation put-data-lake-settings \
    --profile [<YOUR_PROFILE>] \
    --region [<REGION>] \
    --data-lake-settings '{
    "DataLakeAdmins": [
        {"DataLakePrincipalIdentifier": "arn:aws:iam:: [<ACCOUNT_ID>]:role/cdk-hnb659fds-cfn-exec-role-[<ACCOUNT_ID>]-[<REGION>]"}
    ]
}'

To verify, run the following AWS CLI command:

aws lakeformation get-data-lake-settings \
    --profile [<YOUR_PROFILE>] \
    --region [<REGION>] \
    --query "DataLakeSettings.DataLakeAdmins"

Note: If you already have existing Lake Formation admins, include them in the DataLakeAdmins array to avoid removing them. Use get-data-lake-settings first to check.

  1. Download seed data.

The data lake stack deploys seed CSV data to S3. Run below python script to download it first on your environment:

python backend/infra/seed_data/download_tickit.py
  1. Log in to Amazon Elastic Container Registry (Amazon ECR) before deploying.

Run this AWS CLI command (replace [<YOUR_PROFILE>], [<ACCOUNT_ID>], [<REGION>] with your values)

aws ecr get-login-password --region [<REGION>] --profile [<YOUR_PROFILE>] | docker login --username AWS --password-stdin [<ACCOUNT_ID>].dkr.ecr. [<REGION>].amazonaws.com

Deploying the solutions

  1. Deploy the backend CDK stack.
cd backend/infra
uv sync
uv run cdk deploy --region [<REGION>] --profile [<YOUR_PROFILE>] --all

Enter “y” when asked to deploy security-sensitive updates on various stacks.

"--require-approval" is enabled and stack includes security-sensitive updates: Do you wish to deploy these changes? (y/n)y
  1. Install dependencies and start frontend.
cd ../..
cd frontend/data-chatbot
npm install

# Download backend configurations that powers the frontend
bash scripts/load-config.sh --region [<REGION>] --profile [<YOUR_PROFILE>]
#
# Loading config from SSM Parameter Store (region: [<REGION>], profile: [<YOUR_PROFILE>])...
# user_pool_id: us-east-1_2*******4
# user_pool_client_id: nkp9p****************ebv
# cognito_region: us-east-1
# agent_arn: arn:aws:bedrock-agentcore: [<REGION>]:[<ACCOUNT_ID>]:runtime/fgacdataagent-D6******hQ
agentcore_region: us-east-1
# Generated .env.local
# Generated amplify_outputs.json
# Done.

npm run dev
# > [email protected] dev
# > next dev
#
# ▲ Next.js 14.2.10
# - Local: http://localhost:3000
# - Environments: .env.local
#
# ✓ Starting...
# ✓ Ready in 2.4s

Frontend should start at http://localhost:3000

The sample data chat frontend application

Figure 16: The sample data chat frontend application

Scenarios walkthrough

This section walks through four scenarios that verify tenant-scoped data access. You log in as Team A, then Team B, and confirm each user sees only their authorized data. You then grant Team A additional access through Lake Formation and verify the updated permissions take effect.

View per-team data access

Each user’s custom:team attribute maps to a team-specific IAM role that scopes their data access. The following steps verify isolation for both Team A and Team B.

Expected access:

 

Team A user Team B user
sales_extended rows team = ‘Team A’ only team = ‘Team B’ only
Accessible tables venue (TeamA), users (shared) category (TeamB), users (shared)
Blocked table Category venue

Steps (repeat for each team user):

  1. Navigate to http://localhost:3000 and log in. You can find the username and password in the cdk.json file.
The Cognito hosted UI in the chat application

Figure 17: The Cognito hosted UI in the chat application

  1. Enter “Show me all the tables available in the tickit2 database” (or choose Show all tables). Confirm the table list matches the expected access.
Chatbot showing the table list for Team A

Figure 18: Chatbot returning the table list from natural-language input for Team A

Chatbot showing the table list for Team B

Figure 19: Chatbot returning the table list from natural-language input for Team B

  1. Enter “Query the sales_extended table and count the number of rows grouped by team.” (or choose Count sales by team). Confirm only the user’s own team rows appear.
Chatbot showing sales_extended rows for Team A only

Figure 20: Chatbot returning sales_extended records for Team A only from natural-language input

Chatbot showing sales_extended rows for Team B only

Figure 21: Chatbot returning sales_extended records for Team B only from natural-language input

  1. Choose Sign out in the top-right corner before switching users.

Figure 22: Signing out of the Team A user

Figure 23: Signing out of the Team B user

Grant extra access to Team A user

In Lake Formation, grant the Team A user access to the Team B records in the sales_extended table and to the category table. Use the following AWS CLI commands or follow the equivalent console steps shown in Figures 24–29.

Step 1: Grant ASSOCIATE permission on the team LF-Tag (replace [<ADMIN_ROLE_OR_USER_ARN>] with the ARN of the IAM identity you use to manage LF-Tags):

aws lakeformation grant-permissions \
    --principal '{"DataLakePrincipalIdentifier": "[<ADMIN_ROLE_OR_USER_ARN>]"}' \
    --resource '{"LFTag": {"TagKey": "team", "TagValues": ["TeamA", "TeamB", "shared"]}}' \
    --permissions "ASSOCIATE" \
    --region [<REGION>] --profile [<YOUR_PROFILE>]

Alternatively, in the Lake Formation console, navigate to LF-Tags and permissions, open the LF-Tag permissions tab, choose Grant permissions, select LF-Tag key-value pair permissions, pick your admin identity as the principal, select the team key with all values, check Associate, and choose Grant (Figures 24–26).

Choosing LF-Tag key-value pair permissions

Figure 24: Choosing LF-Tag key-value pair permissions

Choosing the identity used to manage the LF-Tag

Figure 25: Choosing the identity used to manage the LF-Tag

Assigning ASSOCIATE permission on all values of the team LF-Tag

Figure 26: Assigning the ASSOCIATE permission for the identity on all values of the team LF-Tag

Step 2: Change the category table tag from TeamB to shared:

# Remove the existing TeamB tag
aws lakeformation remove-lf-tags-from-resource \
    --resource '{"Table": {"DatabaseName": "tickit2", "Name": "category"}}' \
    --lf-tags '[{"TagKey": "team", "TagValues": ["TeamB"]}]' \
    --region [<REGION>] --profile [<YOUR_PROFILE>]

# Assign the shared tag
aws lakeformation add-lf-tags-to-resource \
    --resource '{"Table": {"DatabaseName": "tickit2", "Name": "category"}}' \
    --lf-tags '[{"TagKey": "team", "TagValues": ["shared"]}]' \
    --region [<REGION>] --profile [<YOUR_PROFILE>]

Alternatively, in the Lake Formation console, navigate to Tables and materialized views, find category in database tickit2, open the LF-Tags tab, and change the value from TeamB to shared.

Editing the LF-Tag on the category table

Figure 27: Editing the LF-Tag on the category table

Editing the LF-Tag value from TeamB to shared

Figure 28: Editing the LF-Tag value from TeamB to shared to grant table access to Team A as well

Modifying the row filter expression in the data cell filter

Figure 29: Modifying the row filter expression in the data cell filter to grant Team A users access to Team B records

Step 3: Update the data cell filter to include Team B rows for Team A:

aws lakeformation update-data-cells-filter \
    --region [<REGION>] --profile [<YOUR_PROFILE>] \
    --table-data '{
    "DatabaseName": "tickit2",
    "TableName": "sales_extended",
    "Name": "TeamAFilter",
    "RowFilter": {"FilterExpression": "team = '\''Team A'\'' OR team = '\''Team B'\''"},
    "ColumnWildcard": {}
}'

Alternatively, in the Lake Formation console, navigate to Data filters, find TeamAFilter, and modify the row filter expression to team = 'Team A' OR team = 'Team B'.

Verify updated Team A user permissions

Note: It may take several minutes for Lake Formation permissions to propagate.

Log in again as the Team A user. Run the same two queries from the View per-team data access section:

  1. Show all tables: You should now also see the category table.
Team A user can now also see the category table

Figure 30: Team A user can now also see the category table

  1. Count sales by team: You should now see rows for both Team A and Team B.
Team A user can now see Team B records in the sales_extended table

Figure 31: Team A user can now see records in the sales_extended table that belong to Team B

Clean up

To remove all components generated, run the following command:

uv run cdk destroy --all --profile <YOUR_PROFILE> --region <REGION>

Security considerations

  • Principle of least privilege: The Lambda has no data permissions and can only broker credential exchange.
  • Scoped data access: Data access flows through user-scoped credentials with Lake Formation enforcement.
  • OAuth token validated at Cognito level: IAM authentication at AgentCore Gateway level.
  • Audit trail: AWS CloudTrail and Lake Formation audit logs trace data access to individual users.
  • No tag injection risk: Session tags derived from validated token claims through Identity Pool.

Conclusion

If you’re building a SaaS application, partner portal, or B2C product where external users query a shared data lake, the hardest part isn’t the AI. It’s making sure every user sees only the data they’re authorized to access.

By combining Cognito User Pools (authentication), Cognito Identity Pools (token-to-credential exchange), Amazon Bedrock AgentCore (agentic platform), and Lake Formation TBAC (declarative access control), you can build a governed multi-tenant data agent that delivers on both fronts:

  • Zero-trust data isolation: Lake Formation scopes each query to the authenticated user’s role, enforcing access in the authorization chain rather than in prompt logic.
  • Minimal-privilege architecture: The Target Lambda acts only as a credential broker, without holding broad data permissions.
  • Declarative governance at scale: LF-Tags enforce row- and column-level security centrally, with no filtering logic in application code.
  • Audit readiness: CloudTrail and Lake Formation audit logs trace data access from authentication through query execution.

This pattern applies wherever external users need scoped data lake access: SaaS analytics, partner portals, customer-facing dashboards, or multi-tenant data products. To get started, review the Lake Formation tag-based access control documentation and the Amazon Bedrock AgentCore documentation, then deploy the sample solution to try it in your own account.


About the author

Fortune Hui

Fortune Hui

Fortune is a Solutions Architect at AWS Hong Kong, working with conglomerate customers. He helps customers and partners build big data platform and Agentic AI applications. In his free time, he plays badminton and enjoys whisky.

Enable cross-Region fine-grained data access with AWS Lake Formation and Trusted Identity Propagation

Post Syndicated from Maneesh Sharma original https://aws.amazon.com/blogs/big-data/enable-cross-region-fine-grained-data-access-with-aws-lake-formation-and-trusted-identity-propagation/

Organizations that store analytics data on AWS often need two things at once. The first is fine-grained access control that limits each person to only the data they are authorized to see. The second is auditability down to the individual user level so that every query is traceable to the person who ran it rather than to a shared AWS Identity and Access Management (IAM) role.

AWS IAM Identity Center Trusted Identity Propagation (TIP) meets both requirements. A user signs in once, and that identity propagates from the client application through the query engine to Lake Formation and the storage layer, so that every request is authorized as the actual user and every query is logged in AWS CloudTrail.

AWS now extends TIP across Regions: IAM Identity Center replicates its identity objects such as users, groups, and session information to every AWS Region that you choose, so services across the stack can recognize the same user identity even when they span different Regions.

This is Part 1 of a three-part series. This post covers the foundation and querying from Amazon SageMaker Unified Studio (Amazon Athena and Amazon Redshift). Part 2 covers Amazon Quick, and Part 3 covers Amazon EMR.

Overview

In this post, the producer shares data from the Asia Pacific (Sydney) Region (ap-southeast-2) and the consumer queries it from the South America (São Paulo) Region (sa-east-1) as themselves. We cover two cases that follow the same pattern. In Case 1, you share Amazon Simple Storage Service (Amazon S3) data governed by Lake Formation, queried from Amazon SageMaker Unified Studio (Athena). In Case 2, you share Amazon Redshift data through a datashare registered to the AWS Glue Data Catalog, queried from Amazon Redshift Serverless.

Cross-Region, cross-account architecture: a Sydney producer governs data with Lake Formation and a São Paulo consumer queries it

Figure 1: Cross-Region and cross-account architecture

Starting on the left of the diagram, your identity provider syncs users and groups into AWS IAM Identity Center through the System for Cross-domain Identity Management (SCIM) protocol. Running as an organization instance in the management account, IAM Identity Center replicates those identities to every Region in use. In the center, the producer account in Sydney uses AWS Lake Formation to centrally govern both Amazon S3 and Amazon Redshift data. When the producer shares a table or database, that share flows to the consumer account in São Paulo, shown on the right, where a resource link makes the shared data queryable as if it were local. On the right, the data analyst signs in to Amazon SageMaker Unified Studio with corporate credentials. With Trusted Identity Propagation enabled, the analyst’s identity travels with every request to Lake Formation. Lake Formation authorizes each query based on who the user is rather than which IAM role they assumed.

The example uses three accounts across three Regions. The producer account runs Amazon S3, AWS Glue Data Catalog, Lake Formation, and Amazon Redshift Serverless in ap-southeast-2 (Sydney). The consumer account queries from sa-east-1 (São Paulo) using SageMaker Unified Studio or Amazon Redshift. IAM Identity Center runs in us-east-1 (Virginia) in the management account.

Cross-account sharing through Lake Formation can span AWS Organizations, but the TIP integration adds a tighter requirement. For the user’s identity to propagate, the producer and consumer accounts must be in the same organization and share the same organization instance of IAM Identity Center. You can repeat the consumer steps to add more accounts or Regions. With this launch, you can query that shared data from a different Region while Lake Formation authorizes every request as the actual user.

How the pieces fit together

Four capabilities underpin this architecture. IAM Identity Center provides a single identity plane across Regions, so the same users and groups are available in every account. Each service in the stack supports Trusted Identity Propagation, so the user’s identity flows from the client application to Lake Formation without falling back to IAM roles. Corporate sign-in through IAM Identity Center makes that identity available across Regions and accounts within the organization. Finally, AWS Resource Access Manager (AWS RAM) shares and resource links bridge the Region gap so the consumer can query data cataloged in the producer’s Region.

Prerequisites

Before you begin, make sure that you have the following in place:

  • The producer account and consumer account are in the same organization.
  • An organization instance of IAM Identity Center, with users and groups provisioned.
  • IAM Identity Center replicated to every Region where you operate. See Deploy AWS applications and access AWS accounts across multiple Regions with IAM Identity Center.
  • You have permissions to administer AWS Lake Formation, AWS Glue, AWS RAM, Amazon Simple Storage Service (Amazon S3), Amazon SageMaker Unified Studio, and Amazon Redshift in the relevant accounts.
  • In the producer Lake Formation Data catalog settings, the Use only IAM access control options should be cleared.
  • The AWS services used in this post are integrated with AWS IAM Identity Center in both the producer and consumer accounts, across the Regions used in this walkthrough. See Trusted identity propagation use cases for more information.

Case 1: Share S3 data (Lake Formation)

In this case, the S3 data and its catalog stay in the producer’s account and Region the whole time, with Lake Formation governing them in Sydney (ap-southeast-2). What crosses Regions is the query itself. The consumer, in a different account, connects from São Paulo (sa-east-1) and queries that same data through Amazon Athena (through SageMaker Unified Studio). This is a federated query pattern: the data stays in place, and only the query crosses Regions. That split (catalog and data in one Region, client and query in another) is what makes this genuinely cross-Region, not just cross-account.

Part 1: Producer account (Sydney, ap-southeast-2)

Region note: the walkthrough uses ap-southeast-2 (producer) and sa-east-1 (consumer and query). Substitute your own Regions and account IDs where you see placeholders. All cross-account Lake Formation grants must be made in the producer Region. The only exception is the resource link and its Describe grant, which are created in the consumer’s query Region.

Step 1.1: Create the sample data file

Create a file named customers.csv with the following content.

customer_id,first_name,last_name,email,city,state,signup_date
1,Martha,Rivera,[email protected],Seattle,WA,2024-01-15
2,Jane,Doe,[email protected],Portland,OR,2024-02-20
3,Mateo,Jackson,[email protected],Denver,CO,2024-03-10
4,Saanvi,Sarkar,[email protected],Austin,TX,2024-04-05
5,Mary,Major,[email protected],Chicago,IL,2024-05-12

Step 1.2: Create an S3 bucket and upload the data

  1. Open the Amazon S3 console in the producer account (ap-southeast-2).
  2. Choose Create bucket, enter a unique name (for example, amzn-s3-demo-sydney-lakehouse-bucket), and choose Create bucket.
  3. Open the bucket, create a folder named customers, and upload customers.csv into it.

Step 1.3: Create the Lake Formation registration role (with sts:SetContext)

Create an IAM role that Lake Formation assumes to read the S3 bucket. This role is used when registering the data lake location and for vending credentials, including for TIP. Don’t use the Lake Formation service-linked role. It doesn’t include sts:SetContext.

  1. Open the IAM console, choose Roles, then Create role.
  2. For Trusted entity type, choose Custom trust policy and paste the following trust policy.
  3. Choose Next, skip attaching managed policies, and choose Next.
  4. Name the role (for example, LakeFormation-DataLocation-role) and choose Create role.
  5. Open the role, choose Add permissions, then Create inline policy, and paste the following permissions policy.
  6. Name the inline policy (for example, LF-S3Access) and save it.

Trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "lakeformation.amazonaws.com" },
      "Action": [
        "sts:AssumeRole",
        "sts:SetContext"
      ]
    }
  ]
}

Permissions policy (replace the bucket name):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::<your-s3-bucket>",
        "arn:aws:s3:::<your-s3-bucket>/*"
      ]
    }
  ]
}

Note: The sts:SetContext action in the trust policy is required for TIP. The Lake Formation service-linked role doesn’t include it, which is why a custom role is required here.

Step 1.4: Register the S3 location in Lake Formation

  1. Open the Lake Formation console in ap-southeast-2.
  2. Under Administration, choose Data lake locations, then Register location.
  3. For Amazon S3 path, enter your bucket path (for example, s3://amzn-s3-demo-sydney-lakehouse-bucket).
  4. For IAM role, choose the LakeFormation-DataLocation-role created in Step 1.3.
  5. For Permission mode, choose Lake Formation, then choose Register location.

Step 1.5: Create the AWS Glue database

  1. In the Lake Formation console, under Data catalog, choose Databases.
  2. Choose Create database.
  3. Enter the database name, for this post sydney_lakehouse, and choose Create database.

Step 1.6: Create the table manually

  1. Open the AWS Glue console in ap-southeast-2.
  2. Under Data Catalog, choose Tables, Add table, then Add table manually.
  3. For Table name, enter customers.
  4. For Database, choose sydney_lakehouse.
  5. For the data store, choose Amazon S3 and set the location to s3://amzn-s3-demo-sydney-lakehouse-bucket/customers/.
  6. For the data format, choose CSV with comma (,) as the delimiter.
  7. Define the schema using the following columns.
  8. Add the table property skip.header.line.count with value 1 to skip the CSV header row.
  9. Review and choose Create.
Column name Data type
customer_id string
first_name string
last_name string
email string
city string
state string
signup_date string

When you create the table as CSV on the console, define all columns as string. The CSV serializer/deserializer used for this table doesn’t parse typed columns such as date or bigint reliably. Cast values in your queries as needed (for example, CAST(signup_date AS date)).

Step 1.7: Grant DATABASE permission to the external (consumer) account

  1. In the Lake Formation console, choose Permissions, Data lake permissions, then Grant.
  2. Under Principals, choose External accounts and enter the consumer account ID (<CONSUMER_ACCOUNT_ID>).
  3. Choose Named Data Catalog resources. For Databases, choose your database, in this post sydney_lakehouse.
  4. For Database permissions, select Describe, and select Describe under Grantable permissions.
  5. Choose Grant.

Step 1.8: Grant TABLE permission to the external (consumer) account

  1. Choose Permissions, Data lake permissions, then Grant.
  2. Under Principals, choose External accounts and enter <CONSUMER_ACCOUNT_ID>.
  3. Choose Named Data Catalog resources. For Databases choose sydney_lakehouse. For Tables choose customers.
  4. For Table permissions, select Select and Describe, and select the same under Grantable permissions.
  5. Choose Grant.

Step 1.9: Grant DATABASE permission to the IAM Identity Center group/user

  1. Choose Permissions, Data lake permissions, then Grant.
  2. Under Principals, choose IAM Identity Center and select the group or user (for example, admin).
  3. Choose Named Data Catalog resources. For Databases choose your database, in this post sydney_lakehouse.
  4. For Database permissions, select Describe.
  5. Choose Grant.

Step 1.10: Grant TABLE permission to the IAM Identity Center group/user

  1. Choose Permissions, Data lake permissions, then Grant.
  2. Under Principals, choose IAM Identity Center and select the group or user (for example, admin).
  3. Choose Named Data Catalog resources. For Databases choose your database and table, in this post sydney_lakehouse and for Tables choose customers.
  4. For Table permissions, select Select and Describe.
  5. Choose Grant.

Note: Because the accounts share one organization instance of IAM Identity Center, the producer can grant the IAM Identity Center group directly on the underlying table. The consumer only grants Describe on the local resource link (Step 2.3).

At this point, the producer side is complete. You have the necessary Lake Formation permissions for both the consumer account and your users. To verify, open Data lake permissions in the Lake Formation console and confirm you see four grants covering the database and table.

Part 2: Consumer account

In this part, you switch to the consumer account to accept the AWS RAM share, create a resource link that points back to the shared database, and grant your users access to it. After these steps, the consumer account is ready to query the producer’s data from a different Region.

Step 2.1: Accept the AWS RAM share (Sydney Region)

  1. Open the AWS RAM console and switch to ap-southeast-2 (Sydney).
  2. Under Shared with me, choose Resource shares.
  3. Select the resource share from the producer account and choose Accept resource share.
  1. Open the Lake Formation console and switch to sa-east-1.
  2. Under Data catalog, choose Databases, then Create resource link.
  3. For Resource link name, enter a name, for example rl_sydney_s3_datalake.
  4. For Shared database’s Region, choose Asia Pacific (Sydney) ap-southeast-2.
  5. For Shared database, choose your database. In this post the database is called sydney_lakehouse.
  6. Shared database’s catalog ID populates automatically (the producer account).
  7. Choose Create.
  1. In the Lake Formation console (sa-east-1), choose Permissions, Data lake permissions, then Grant.
  2. Under Principals, choose IAM Identity Center and select the group or user (for example, admin).
  3. Choose Named Data Catalog resources. For Databases choose rl_sydney_s3_datalake.
  4. For Resource link permissions, select Describe.
  5. Choose Grant.

Part 3: SageMaker Unified Studio (sa-east-1)

With the producer and consumer accounts configured, you now set up SageMaker Unified Studio in the query Region and run the cross-Region query.

Step 3.1: Create the SageMaker Unified Studio domain

Open the Amazon SageMaker console in sa-east-1, choose Unified Studio, then Create domain. See the SageMaker Unified Studio administration guide for the quick setup. Because your IAM Identity Center integration is already complete, you can specify an IAM Identity Center user in the domain configuration settings.

SageMaker Unified Studio domain configuration with an IAM Identity Center user added

Figure 2: Adding an SSO user in the Amazon SageMaker Unified Studio domain

To enable Trusted Identity Propagation (TIP) in SageMaker Unified Studio, refer to the following sections in this post:

  1. Enable TIP in SageMaker Unified Studio.
  2. Enable user access for SageMaker Unified Studio domain.
  3. Add project members.
  4. Create a SQL analytics project in Unified Studio. For this post, we use the project name my_project_idc.

Step 3.2: Enable TIP

  1. Sign in to the SageMaker Unified Studio portal by choosing Sign in with SSO as an IAM Identity Center user.
IAM Identity Center sign-in page for the SageMaker Unified Studio portal

Figure 3: Sign in with SSO in Amazon SageMaker Unified Studio

  1. Open the project (my_project_idc) and go to the Query Editor.
  2. Under Catalogs, choose AwsDataCatalog.
  3. Choose rl_sydney_s3_datalake, then tables, and then select customers.
  4. Choose the three dots next to the customers table and choose Preview data.
Query Editor previewing the customers table through the rl_sydney_s3_datalake resource link

Figure 4: Access Amazon S3 data in Amazon SageMaker Unified Studio

The query auto-runs using the Athena compute, as shown in the following screenshot:

SELECT * FROM "AwsDataCatalog"."rl_sydney_s3_datalake"."customers" LIMIT 10;
Query results returning customer rows from the Sydney S3 bucket through the resource link

Figure 5: Preview Amazon S3 data in Amazon SageMaker Unified Studio

The customer data from the producer S3 bucket in Sydney is returned, governed by Lake Formation and accessed through the resource link in sa-east-1, with the user identity propagated through TIP.

Case 2: Share Amazon Redshift data using AWS Glue federated catalog

This case follows the same centralized-catalog pattern as Case 1, but the consumer queries the data with Amazon Redshift Serverless instead of Athena. The analyst runs these queries from SageMaker Unified Studio using its Amazon Redshift query experience.

Part 1: Producer account (Sydney, ap-southeast-2)

Make sure that your Amazon Redshift consumer cluster is configured with IAM Identity Center in both ap-southeast-2 and sa-east-1 Regions. If you need to review the procedure, see Integrate Identity Provider (IdP) with Amazon Redshift Query Editor V2 using AWS IAM Identity Center for seamless Single Sign-On.

Step 1.1: Register a serverless namespace to the AWS Glue Data Catalog

  1. Open the Amazon Redshift console.
  2. On the navigation menu, choose Redshift Serverless. The Namespaces / Workgroups section shows your namespaces and workgroups for the current Region.
  3. Choose the name of the namespace you want to register.
  4. From Actions, choose Register to AWS Glue Data Catalog.
Amazon Redshift Serverless namespace selected to register with the AWS Glue Data Catalog

Figure 6: Amazon Redshift Serverless namespace to register with the AWS Glue Data Catalog

  1. Choose Do not register with Amazon Redshift federated permissions.
  2. Select Use AWS Lake Formation permissions.
  3. For Destination account ID, enter the producer account ID (<PRODUCER_ACCOUNT_ID>), the account that will hold the catalog.
  4. Choose Register. You are taken to the AWS Lake Formation console.
Registering the Amazon Redshift Serverless namespace with the AWS Glue Data Catalog

Figure 7: Register the Amazon Redshift Serverless namespace with the AWS Glue Data Catalog

  1. Follow the catalog creation process in Lake Formation. See Creating Amazon Redshift federated catalogs and follow these steps:
    1. Review a namespace invitation and accept it.
    2. Create a federated catalog.

Step 1.2: Grant DATABASE permission to the external (consumer) account

  1. In the Lake Formation console (ap-southeast-2), choose Permissions, Data lake permissions, then Grant.
  2. Under Principals, choose External accounts and enter <CONSUMER_ACCOUNT_ID>.
  3. Choose Named Data Catalog resources. For Catalogs/Databases, choose the federated database (for example, ns-redshift/dev, schema sales_schema).
  4. For Database permissions, select Describe (and Describe under Grantable permissions).
  5. Choose Grant.

Step 1.3: Grant TABLE permission to the external (consumer) account

  1. Choose Permissions, Data lake permissions, then Grant.
  2. Under Principals, choose External accounts and enter <CONSUMER_ACCOUNT_ID>.
  3. Choose Named Data Catalog resources and select the shared table (for example, store_sales).
  4. For Table permissions, select Select and Describe (and the same under Grantable permissions).
  5. Choose Grant.

Step 1.4: Grant DATABASE permission to the IAM Identity Center group/user

  1. Choose Permissions, Data lake permissions, then Grant.
  2. Under Principals, choose IAM Identity Center and select the group or user.
  3. Choose Named Data Catalog resources and select the federated database (schema).
  4. For Database permissions, select Describe.
  5. Choose Grant.

Step 1.5: Grant TABLE permission to the IAM Identity Center group/user

  1. Choose Permissions, Data lake permissions, then Grant.
  2. Under Principals, choose IAM Identity Center and select the group or user.
  3. Choose Named Data Catalog resources and select the shared table (for example, store_sales).
  4. For Table permissions, select Select and Describe.
  5. Choose Grant.

Part 2: Consumer account

At this point, the producer side for Case 2 is complete. To verify, open Data lake permissions in the Lake Formation console and confirm you see four grants covering the federated database and shared table.

In this part, you switch to the consumer account to accept the AWS RAM share, configure the IAM role for cross-Region Amazon Redshift access, and set up TIP so that users can query the producer’s Amazon Redshift data from a different Region.

Step 2.1: Accept the AWS RAM share (Sydney Region)

  1. Open the AWS RAM console and switch to ap-southeast-2.
  2. Under Shared with me, choose Resource shares.
  3. Select the resource share from the producer account and choose Accept resource share.

Step 2.2: Update the trust relationship for the IAM role

Navigate to the IAM role (IAMIDCRedshiftRole) created in this post.

Make sure the trust policy includes Amazon Redshift and Amazon Redshift Serverless for sts:AssumeRole and sts:SetContext, as shown in the following policy:

Trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "redshift.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:SetContext"]
    },
    {
      "Effect": "Allow",
      "Principal": { "Service": "redshift-serverless.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:SetContext"]
    }
  ]
}

Step 2.3: Add a new policy to the IAM role

Navigate to the IAM role (IAMIDCRedshiftRole) created in this post and add the following policy for AWS Glue and Lake Formation.

Permissions policy 1 (AWS Glue read and Lake Formation data access):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueGetTable",
      "Effect": "Allow",
      "Action": [
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetPartition",
        "glue:GetPartitions"
      ],
      "Resource": "*"
    },
    {
      "Sid": "LakeFormationDataAccess",
      "Effect": "Allow",
      "Action": [
        "lakeformation:GetDataAccess",
        "lakeformation:GetTemporaryGlueTableCredentials"
      ],
      "Resource": "*"
    }
  ]
}

Note: These actions require "Resource": "*" because they are used for catalog-level discovery, which does not support resource-scoped permissions. Lake Formation column-level and row-level grants still enforce least-privilege access to the underlying data.

Step 2.4: Set up cross-Region Trusted Identity Propagation (TIP) access in Amazon Redshift

  1. Sign in to the Amazon Redshift console using the admin role.
  2. Navigate to Query Editor V2.
  3. Choose the options menu (three dots) next to the cluster and choose Create connection.
  4. Connect as the admin user and run the following command to create an external schema that points to your cross-Region AWS Glue Data Catalog:

Syntax:

CREATE EXTERNAL SCHEMA <schema_name>
FROM DATA CATALOG
DATABASE '<glue_db_name>'
REGION '<producer region where glue Data Catalog lives>'
CATALOG_ID '<catalog_name>';

Example:

CREATE EXTERNAL SCHEMA redshift_cross_region_tip
FROM DATA CATALOG
DATABASE 'sales_schema'
REGION 'ap-southeast-2'
CATALOG_ID '123456789012:ns-producer-redshift-syd/dev';

You can get this catalog_name information from the producer account Lake Formation in the Sydney Region. Navigate to Databases on the left and choose the respective Amazon Redshift catalog ID from the drop-down.

Lake Formation Databases list showing the Amazon Redshift federated catalog ID

Figure 8: AWS Lake Formation showing the catalog ID of the Amazon Redshift namespace registered with the AWS Glue Data Catalog

  1. Grant the IAM Identity Center group access to the external schema.
GRANT USAGE ON schema redshift_cross_region_tip TO ROLE "AWSIDC:admin";

Step 2.5: Query the data from Amazon Redshift Query Editor V2

  1. In Query Editor V2, choose the options menu (three dots) next to the cluster and choose Edit connection, and for authentication, select IAM Identity Center.
  2. A pop-up redirects you to your IdP login page. Enter your IdP credentials.
  3. After successful authentication, you are redirected to Query Editor V2 as an IAM Identity Center user.
  4. Run the following query.
SELECT * FROM "dev"."redshift_cross_region_tip"."store_sales";
Amazon Redshift Query Editor V2 returning cross-Region store_sales rows

Figure 9: Query cross-Region, cross-account data using Amazon Redshift Query Editor V2

Step 2.6: Query the data from SageMaker Unified Studio

For a consumer cluster to use in SageMaker Unified Studio, you need to tag the Amazon Redshift cluster or workgroup with one of the following options. Choose one before proceeding.

Option 1 – Add a tag to allow only a specific SageMaker Unified Studio project to access it: AmazonDataZoneProject=<projectID>

Option 2 – Add a tag to allow all SageMaker Unified Studio projects in this account to access it: for-use-with-all-datazone-projects=true

After the Amazon Redshift cluster or workgroup has a tag, continue with the following steps.

  1. Sign in to the SageMaker Unified Studio portal by choosing Sign in with SSO as an IAM Identity Center user.
  2. In SageMaker Unified Studio, navigate to Query Editor and expand Connections.
  3. Choose redshift_cross_region_tip, then tables, and then select store_sales.
  4. Choose the three dots next to the store_sales table and choose Preview data. The query auto-runs using the Amazon Redshift compute, as shown in the following screenshot:
SageMaker Unified Studio previewing store_sales using Amazon Redshift compute

Figure 10: Query data using Amazon Redshift compute in Amazon SageMaker Unified Studio

The Amazon Redshift data shared from the producer in Sydney is returned. It is governed and centralized in one account and Region (ap-southeast-2) using Lake Formation and accessed from the consumer account in a second Region (sa-east-1), with the user identity propagated through TIP.

Both cases use the same Region split. The catalog and the data stay in the producer Region (Sydney), and only the client and the query run from the consumer Region (São Paulo). They differ only in the query engine. Case 1 uses Amazon Athena, and Case 2 uses Amazon Redshift Serverless. That difference in engine is also what changes their cross-Region behavior. With Athena (Case 1), queries work cross-Region without additional constraints. However, if you use an Amazon Redshift datashare, the Amazon Redshift query engine must be in the same Region as the Amazon Redshift namespace.

Clean up

To avoid ongoing charges, remove the resources that you created.

Consumer account:

  1. Delete the SageMaker Unified Studio project and domain (Case 1).
  2. Delete the Amazon Redshift Serverless workgroup and namespace (Case 2).
  3. In Lake Formation (sa-east-1), delete the resource links (rl_sydney_s3_datalake, rl_redshift_cross_db_tip).
  4. Revoke any Lake Formation grants you created and, if applicable, leave/deny the AWS RAM share.

Producer account:

  1. Revoke the Lake Formation grants to the consumer account and to the IAM Identity Center group.
  2. Revoke the AWS RAM share.
  3. Deregister the S3 data lake location, and delete the customers table and the sydney_lakehouse database.
  4. Empty and delete the S3 bucket.
  5. Deregister the Amazon Redshift namespace from the AWS Glue Data Catalog and delete the federated catalog (Case 2).
  6. Delete the IAM roles you created (LakeFormation-DataLocation-role, IAMIDCRedshiftRole).

Conclusion

In this post, you configured AWS Lake Formation Trusted Identity Propagation so that data governed and shared in one Region (Sydney) can be queried from another Region (São Paulo). Lake Formation cross-account grants are made in the producer Region, and the consumer reaches the shared data from the query Region in the way that fits each engine. For Amazon S3 data queried through Amazon Athena (Case 1), the consumer creates a resource link in the query Region that points back to the shared database. For Amazon Redshift data shared through a datashare and AWS Glue federated catalog (Case 2), the consumer instead creates an external schema in Amazon Redshift that references the producer’s Glue Data Catalog directly in the producer Region. No resource link is required. In both cases, because access is authorized against the user’s IAM Identity Center identity rather than a shared role, you get least-privilege access, per-user auditability in AWS CloudTrail, and a single place to manage permissions.

To add more consumer accounts or query Regions, repeat the consumer-side steps for the relevant case: accept the AWS RAM share in the producer Region, then either create a resource link in the query Region and grant Describe on it to your IAM Identity Center principals (Amazon S3 and Athena), or create an external schema that references the producer’s Glue catalog and grant usage to your IAM Identity Center principals for Amazon Redshift.

As a next step, review the Trusted Identity Propagation supported applications and services to plan which services in your stack to enable with TIP, using the Lake Formation cross-account sharing documentation as a reference.

 


About the authors

Maneesh Sharma

Maneesh Sharma

Maneesh is a Senior Analytics Specialist Solutions Architect at AWS with more than 15 years of experience designing and implementing large-scale data warehouse and analytics solutions. He works with FSI and Enterprise customers to implement modern analytics architectures using Amazon Redshift, Amazon SageMaker Unified Studio, Amazon S3 Tables, a capability of Amazon S3, AWS Glue, AWS Lake Formation, and AWS IAM Identity Center.

Laura Reith

Laura Reith

Laura is an Identity Solutions Architect at AWS, where she thrives on helping customers overcome security and identity challenges. In her free time, she enjoys wreck diving and traveling around the world.

Sri Pooja Ravipati

Sri Pooja Ravipati

Sri is a Technical Product Manager for AWS IAM Identity Center with eight years of experience across product and technology. She works with enterprise customers to simplify how they manage workforce identity and access, building capabilities that make secure access easier to manage at scale.

Songzhi Liu

Songzhi Liu

Songzhi is a Principal Big Data Architect with the AWS Identity Solutions team. In this role, he collaborates closely with AWS customers and cross-functional teams to design and implement scalable data architectures, focusing on integrating big data and machine learning solutions to enhance identity awareness within the AWS environment.

Identity-aware AI data agents with AWS Lake Formation and Trusted Identity Propagation

Post Syndicated from Thiyagarajan Mani original https://aws.amazon.com/blogs/security/identity-aware-ai-data-agents-with-aws-lake-formation-and-trusted-identity-propagation/

You’re building a data agent that lets business users ask questions about lakehouse data in natural language. You’ve already built governance policies that control who can access which datasets. The challenge is making the agent respect those rules without rebuilding them in your application code.

When a user asks a question, the agent maps it to data and constructs a query. The tool runs under its own AWS Identity and Access Management (IAM) role, so AWS Lake Formation sees the tool’s credentials, not the person behind the request. This leaves you with two inadequate options: restrict tool access (limiting self-service analytics) or rebuild access controls in application code (moving governance out of the data layer).

In this post, we show you a different approach: identity-aware AI data agents that propagate each user’s identity through every hop, from the user, through the agent, into the tool, so Lake Formation evaluates the user’s grants. Your application code makes no authorization decisions, existing Lake Formation policies work without modifications, and AWS CloudTrail records the actual person who accessed the data.

In this post, you learn how to:

  • Set up per-user data access controls for AI agents querying your lakehouse without rewriting your existing Lake Formation governance policies by configuring Amazon Bedrock AgentCore to carry each user’s identity through trusted identity propagation (TIP).
  • Ensure auditability and compliance by performing a server-side token exchange inside AWS Lambda that converts the identity token into Lake Formation credentials scoped to the real user, with CloudTrail recording every query.
  • Validate the pattern end-to-end by testing with multiple users and confirming per-user query results and CloudTrail audit evidence.

The building blocks in this post, OAuth 2.0 token delegation, AWS IAM Identity Center, Lake Formation, and Lambda are well-documented individually. The new constraint is that a foundation model (FM) now sits in the middle of the propagation chain. The FM is the agent’s brain, it decides which tools to call and what arguments to pass and anything that enters its context (prompts, tool schemas, arguments) is accessible within that trust boundary. So, the user’s identity token must reach the tool, but it must do so on the HTTP transport layer, bypassing the FM’s reasoning layer entirely. This post demonstrates the three Amazon Bedrock AgentCore configurations that achieve that.

Whether you’re building agents on Strands, LangGraph, or a similar framework, this post gives you a deployable pattern on Amazon Bedrock AgentCore. Security engineers will find the identity-transport and audit properties relevant, and data platform owners will see how existing Lake Formation grants extend to AI workloads with no changes.

Solution overview

With this pattern (shown in Figure 1), your data agent can query Lake Formation governed data with per-user access controls, full CloudTrail audit trails, and no changes to your existing governance policies. Here’s how it works.

  1. Two tokens travel through the system. An access_token authenticates the request at each trust boundary (Bedrock AgentCore Runtime, Bedrock AgentCore Gateway). An id_token carries your identity and is exchanged, server-side, for the identity context that Lake Formation evaluates.
  2. A separate TIP role carries only the IAM permissions needed to call service APIs, it has no Lake Formation data grants.
  3. Lake Formation evaluates only the propagated user identity, not the TIP role.

The request flow is:

  1. User to UI: The user authenticates with an OpenID Connect (OIDC) identity provider (IdP). This post uses Amazon Cognito, but you can use any OIDC provider, such as Okta. The UI receives an id_token and an access_token.
  2. UI to Bedrock AgentCore Runtime: The UI calls the agent running in AgentCore Runtime over HTTPS. The access_token goes in the standard Authorization header. The id_token goes in a custom header: X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken. The HTTP body contains only the user’s prompt.
  3. AgentCore Runtime to agent code: The runtime validates the access_token against the configured JSON Web Token (JWT) authorizer, then passes the request to the agent container with both headers accessible through context.request_headers.
  4. Agent to Bedrock AgentCore Gateway: The agent opens a Model Context Protocol (MCP) connection to an AgentCore gateway, including both headers on the connection. The AgentCore gateway validates the access_token and forwards the custom header to its Lambda target.
  5. AgentCore Gateway to Lambda: The propagated headers arrive in context.client_context.custom[‘bedrockAgentCorePropagatedHeaders’].
  6. Lambda to the data layer: Lambda validates the id_token, exchanges it for an identity context, assumes a role with that context, and runs the Amazon Athena query under the user’s identity.

Lake Formation evaluates grants against the real user. Athena returns only the rows and columns that the user is entitled to see. CloudTrail records the assumed role with an onBehalfOf entry identifying the human.

Now that you’ve seen the end-to-end flow, the following sections walk through each piece, starting with what you need to have in place before you build.

Prerequisites

This post assumes you have the working knowledge of OAuth 2.0, IAM, and Lake Formation grants.

  • An IAM Identity Center instance with a trusted token issuer (TTI) configured to accept your OIDC IdP’s tokens.
  • An OAuth Application in IAM Identity Center configured for CreateTokenWithIAM with the JWT Bearer grant.
  • Lake Formation governing your AWS Glue Data Catalog, with grants already assigned to users or groups.
  • An Athena workgroup and an Amazon Simple Storage Service (Amazon S3) bucket for query results.
  • An OIDC IdP. The reference implementation uses Amazon Cognito as the demo IdP, but the pattern is IdP-agnostic. Auth0, Microsoft Entra ID, Okta, Ping, or any OIDC-compliant provider works identically, if your TTI accepts its tokens.

This post doesn’t walk through setting up any of these components. The existing AWS documentation covers each one. For more information, see the links in the preceding list and the related resources at the end of this post.

The three Bedrock AgentCore configuration steps

Three Bedrock AgentCore features make this pattern work. Together they form the chain of custody for the user’s id_token from the moment it arrives at the Bedrock AgentCore Runtime to the moment Lambda uses it.

Configure the runtime request header allow list

Bedrock AgentCore Runtime doesn’t pass request headers into the agent container by default. You opt in by declaring an allow list, either through agentcore configure or directly in the runtime configuration:

# .bedrock_agentcore.yaml

request_header_configuration:
	requestHeaderAllowlist:
		- Authorization
		- X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken

AgentCore Runtime supports two types of forwarded headers:

  • The standard Authorization header for OAuth inbound JWT authentication (access_token), and
  • Custom headers prefixed with X-Amzn-Bedrock-AgentCore-Runtime-Custom-

The id_token in this pattern uses a custom header. Inside the agent, the allow listed headers arrive as a dictionary object on the request context.

The following Python code runs in the agent container:

from bedrock_agentcore.runtime import BedrockAgentCoreApp
 
ID_TOKEN_HEADER = "X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken"
app = BedrockAgentCoreApp()
 
@app.entrypoint
def invoke(payload, context):
    request_headers = getattr(context, "request_headers", None) or {}
 
    id_token = ""
    access_token = ""
    for key, value in request_headers.items():
        if key.lower() == ID_TOKEN_HEADER.lower():
            id_token = value
        elif key.lower() == "authorization":
            access_token = value.replace("Bearer ", "")
 
    # ... build MCP headers and call the Gateway

The agent code reads the id_token from the HTTP transport and forwards it (also on the HTTP transport) to the next hop. It doesn’t treat the token as a tool argument and doesn’t inject it into a prompt.

Key takeaway: The runtime allow list is the first gate. Without it, the id_token doesn’t reach your agent code.

Configure AgentCore Gateway metadata for header propagation

An AgentCore Gateway is the Model Context Protocol (MCP) endpoint the agent talks to. When AgentCore Gateway invokes a Lambda target, it doesn’t forward arbitrary request headers by default. You configure which headers to propagate using metadataConfiguration.allowedRequestHeaders on the target:

from aws_cdk import aws_bedrockagentcore as agentcore
 
self.gateway_target = agentcore.CfnGatewayTarget(
    self, "AthenaTarget",
    name="athena-executor",
    gateway_identifier=self.gateway.attr_gateway_identifier,
    target_configuration=agentcore.CfnGatewayTarget.TargetConfigurationProperty(
        mcp=agentcore.CfnGatewayTarget.McpTargetConfigurationProperty(
            lambda_=agentcore.CfnGatewayTarget.McpLambdaTargetConfigurationProperty(
                lambda_arn=lambda_function_arn,
                tool_schema=...  # execute_athena_query
            ),
        ),
    ),
    metadata_configuration=agentcore.CfnGatewayTarget.MetadataConfigurationProperty(
        allowed_request_headers=["X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken"],
    ),
    ...
)

Notice that the tool schema doesn’t include an id_token parameter; there’s no token parameter on any tool. The FM doesn’t see, select, or pass an id_token because the token isn’t part of the tool’s contract. It travels parallel to the tool call, on the HTTP connection, through metadataConfiguration.

This is a critical property for security. If you put the id_token in the tool schema instead, the FM becomes responsible for passing it, which means the token lands in prompts, traces, memory, and logs. Keeping the token off the tool contract keeps it out of the FM entirely.

Key takeaway: The metadataConfiguration of the AgentCore gateway is the second gate. It controls which headers cross from the agent into the Lambda function without touching the tool schema.

Read propagated headers in Lambda

On the Lambda side, the propagated header arrives not in the event body but in the client context, under a specific key.

The following Python code runs in the Lambda function:

ID_TOKEN_HEADER = "X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken"
 
def lambda_handler(event, context):
    client_ctx = getattr(context.client_context, "custom", {}) or {}
    propagated_headers = client_ctx.get("bedrockAgentCorePropagatedHeaders", {})
    id_token = propagated_headers.get(ID_TOKEN_HEADER)
 
    if not id_token:
        return {"statusCode": 400, "body": {"error": "missing id_token"}}
 
    # Tool arguments from the agent's call (no id_token here)
    query = event["query"]
    database = event["database"]
    workgroup = event["workgroup"]
    output_location = event["output_location"]
 
    # ... validate token, exchange, run query

The event dictionary contains the tool’s declared parameters and nothing else. You reach the id_token only through context.client_context.custom[‘bedrockAgentCorePropagatedHeaders’]. That’s the handoff point.

Key takeaway: The id_token arrives through the client context rather than tool arguments; the FM has no access to it. The Lambda is the only component that reads the token.

Perform the server-side token exchange

After Lambda has the id_token, it validates the token and exchanges it for an identityContext. This step uses standard IAM Identity Center TIP mechanics. That it happens inside Lambda rather than anywhere else is what keeps the identity context from crossing a process boundary.

import boto3, jwt, time
 
def exchange_and_assume(id_token: str, user_sub: str):
    # 1. Validate id_token against the OIDC IdP's JWKS.
    claims = validate_id_token(id_token)  # signature, exp, aud, iss, token_use
 
    # 2. Exchange for an Identity Center identityContext.
    sso_oidc = boto3.client("sso-oidc")
    resp = sso_oidc.create_token_with_iam(
        clientId=OAUTH_APPLICATION_ARN,
        grantType="urn:ietf:params:oauth:grant-type:jwt-bearer",
        assertion=id_token,
    )
    identity_context = resp["awsAdditionalDetails"]["identityContext"]
 
    # 3. AssumeRole with ProvidedContexts
    sts = boto3.client("sts")
    assumed = sts.assume_role(
        RoleArn=TIP_ROLE_ARN,
        RoleSessionName=f"agent-tip-{user_sub[:8]}-{int(time.time())}",
        ProvidedContexts=[{
            "ProviderArn": "arn:aws:iam::aws:contextProvider/IdentityCenter",
            "ContextAssertion": identity_context,
        }],
    )
    creds = assumed["Credentials"]
    return boto3.Session(
        aws_access_key_id=creds["AccessKeyId"],
        aws_secret_access_key=creds["SecretAccessKey"],
        aws_session_token=creds["SessionToken"],
    )

Two properties come out of this exchange:

  • The identityContext is created and consumed inside a single Lambda invocation. It doesn’t get returned to the agent, the gateway, or the UI.
  • The resulting boto3.Session holds short-lived credentials whose underlying identity assertion is the real user. When the session calls Athena, the query runs with the user’s identity propagated. Lake Formation sees the user, not the Lambda function’s role.

Everything after this, including the Athena query and result formatting, is standard boto3.

Configure Lake Formation grants

You need one grant to the IAM Identity Center user or group. That’s the whole story at the Lake Formation layer.

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:identitystore:::user/<USER_ID> \
  --resource '{"Table":{"DatabaseName":"iceberg_db","Name":"trip_details"}}' \
  --permissions SELECT

This is the grant Lake Formation evaluates at query time. You add column-level and row-level filters to the same user or group the same way. Nothing here is aware of or specific to AI agents. If you already have a Lake Formation grants model for human users, you already have the grants this pattern needs.

Understanding the TIP role: The TIP role that Lambda assumes has no Lake Formation data grants. It holds only IAM permissions to call the service APIs: athena:* for query runs, glue:* for catalog reads, lakeformation:GetDataAccess for the query plan handshake, and Amazon S3 access for the Athena output bucket. When Lambda assumes this role with an identityContext attached through ProvidedContexts, Lake Formation evaluates only the propagated user identity against its grants. The role itself is transparent to the authorization decision.

In the more common agent runs as a role pattern, the role carries the grants, which is why per-user governance breaks. Here the role carries no grants; it’s a session vehicle, not an authorization subject.

Test the pattern end-to-end

Two users, same question, different outcomes.

User A has SELECT on trip_details. They ask the agent for five records from the table.

Figure 2: Five records are requested and returned by the agent

Figure 2: Five records are requested and returned by the agent

User B has no grant on trip_details. They ask the same question.

Figure 3: User doesn’t have access to the data

Figure 3: User doesn’t have access to the data

No code changed between the two interactions. No parameter was toggled. Lake Formation made the decision based on the propagated identity.

The CloudTrail record for the AssumeRole call shows the delegation:

{
  "eventSource": "sts.amazonaws.com",
  "eventName": "AssumeRole",
  "requestParameters": {
    "roleArn": "arn:aws:iam::<ACCOUNT_ID>:role/<TIP_ROLE_NAME>",
    "roleSessionName": "agent-tip-<user_sub_prefix>-<timestamp>",
    "providedContexts": [{
      "providerArn": "arn:aws:iam::aws:contextProvider/IdentityCenter",
      "contextAssertion": "<identity-context-assertion>"
    }]
  },
  "userIdentity": {
    "type": "AssumedRole",
    "onBehalfOf": {
      "userId": "<IDENTITY_STORE_USER_ID>",
      "identityStoreArn": "arn:aws:identitystore::<ACCOUNT_ID>:identitystore/<ID>"
    }
  }
}

The onBehalfOf block closes the audit loop. Each query the agent runs on a user’s behalf has a CloudTrail record naming that user, with no additional instrumentation in your code.

Security properties

Four properties follow from this architecture. These are the core value propositions of the identity-aware pattern:

  1. The id_token never reaches the foundation model: It travels on HTTP headers at every hop, and Lambda reads it from context.client_context.custom. It’s not a parameter on any tool. The FM has no path to it: not in tool arguments, not in prompts, not in memory, not in traces.
  2. The identity context stays inside a single Lambda invocation: It’s derived from the id_token, used immediately in an AssumeRole call, and discarded. It doesn’t go back to the agent, the gateway, or the UI.
  3. Authorization decisions live in Lake Formation, against the real user only: The TIP role the Lambda function assumes has no data grants. Lake Formation evaluates the propagated user identity. No code path in the agent, gateway, or Lambda function performs authorization logic.
  4. The audit trail requires no extra work: The CloudTrail AssumeRole event with onBehalfOf identifies the human user for every query. You get the same audit fidelity you would have for human users accessing data directly.

Deploy the pattern

To deploy this pattern, you need to configure four things:

  1. An OIDC IdP with a TTI in IAM Identity Center accepting its tokens, and an Identity Center OAuth Application configured for CreateTokenWithIAM.
  2. Bedrock AgentCore Runtime running your agent container with requestHeaderAllowlist covering Authorization and your custom id_token header.
  3. Bedrock AgentCore Gateway with a Lambda target whose metadataConfiguration.allowedRequestHeaders includes the id_token header. The Lambda target’s tool schema has no id_token parameter.
  4. Lambda reading the id_token from context.client_context.custom[‘bedrockAgentCorePropagatedHeaders’], performing the token exchange through sso-oidc:CreateTokenWithIAM, and calling sts:AssumeRole with ProvidedContexts to get the TIP-bearing session.

Conclusion

When an AI agent queries a governed lakehouse, the data layer needs to know who’s asking, not which role the agent is running under. This post showed you how to resolve that by treating the agent as an OAuth delegated actor. The user’s token travels alongside the agent’s HTTP transport but doesn’t enter the model’s context, and the token exchange that produces query-time credentials happens server-side inside Lambda, scoped to a single invocation.

The three Bedrock AgentCore features that make this composable (requestHeaderAllowlist on runtime, metadataConfiguration.allowedRequestHeaders on gateway, and bedrockAgentCorePropagatedHeaders on Lambda) are specific to building on Bedrock AgentCore. Everything downstream of Lambda is IAM Identity Center and Lake Formation functionality.

If you’re building agents that read governed data, you don’t have to choose between a single over-permissioned service role and per-user code paths. The identity the data layer evaluates can be the real user, the audit trail can name the real user, and the foundation model doesn’t need to know the user’s token exists.

The result is a clean separation: the user’s identity travels end-to-end, the model never sees it, and the data layer enforces it exactly as if the user queried directly.

Related resources

If you have feedback about this post, submit comments in the Comments section below.


Thiyagarajan Mani

Thiyagarajan Mani

Thiyagarajan is a Sr. Delivery Consultant at AWS. With over 20 years of experience, he architects modern data platforms spanning lakehouses, ingestion pipelines, and generative and agentic AI applications. He specializes in transforming legacy data ecosystems into scalable, cloud-centered architectures that unlock advanced analytics and intelligent automation for customers. Outside work, he enjoys time with family and riding bikes.

Mihir Borkar

Mihir Borkar

Mihir is a Senior Solutions Architect at AWS, partnering with ISVs to build and scale their platforms. With a decade of experience across data architecture and delivery, his background in enterprise-scale engagements gives him a builder’s perspective, bridging architecture strategy with practical implementation. Outside work, he explores the latest developments in cloud technologies and AI/ML.

Philippe-Duplessis-Guindon

Philippe Duplessis-Guindon

Philippe is a Delivery Consultant at AWS, where he has worked on a wide range of generative AI projects, touching on most aspects from infrastructure and DevOps to software development and AI/ML. After earning his bachelor’s in software engineering and a master’s in computer vision and machine learning from Polytechnique Montréal, he joined AWS to help customers accelerate their generative AI journeys.

Umang Khambhalikar

Umang Khambhalikar

Umang is a Senior Consultant at AWS Professional Services, specializing in data and AI. He helps enterprise customers design and implement secure, governed data platforms and agentic AI solutions using services like Amazon Bedrock AgentCore, AWS Lake Formation, and AWS Glue. Umang brings over 19 years of IT consulting experience to solving complex challenges at the intersection of security, data governance, and generative AI.

Enable cross-cloud analytics with Amazon S3 Tables and Google BigQuery, Part 2: access control with Lake Formation

Post Syndicated from Lakshmi Nair original https://aws.amazon.com/blogs/big-data/enable-cross-cloud-analytics-with-amazon-s3-tables-and-google-bigquery-part-2-access-control-with-lake-formation/

In Part 1, we showed how to connect Google BigQuery to Amazon Simple Storage Service (Amazon S3) Tables, a capability of Amazon S3, using access control based on AWS Identity and Access Management (IAM). A single IAM policy governs both table metadata and data access. We also walked through common cross-cloud analytics scenarios where this pattern adds value. This post covers the approach using AWS Lake Formation. Instead of relying solely on IAM policies for data access, Lake Formation manages fine-grained permissions and vends temporary, scoped credentials to the requesting engine. This is a better fit when multiple engines need different levels of access to the same tables, or when you want to manage grants centrally without touching IAM policies every time a new consumer comes along.

Solution overview

You use the AWS Glue Iceberg REST Catalog (IRC) as the bridge between BigQuery and S3 Tables. BigQuery’s cross-cloud Lakehouse creates a federated catalog that syncs metadata from the Glue IRC, then uses the synced metadata to read Iceberg data files directly.

Architecture diagram showing BigQuery connecting to Amazon S3 Tables through the AWS Glue Iceberg REST Catalog

Figure 1: Architecture diagram showing BigQuery connecting to Amazon S3 Tables through the AWS Glue Iceberg REST Catalog

The key components in this architecture:

  1. Amazon S3 Tables: With Amazon S3 Tables, data is stored in table buckets, specifically designed for storing tables in the Apache Iceberg format. Table metadata is registered on AWS Glue Data Catalog for discovery and governance.
  2. AWS Glue Data Catalog: With AWS Glue Data Catalog, you can access the federated s3tablescatalog catalog that maps S3 Tables resources (table buckets, namespaces, tables) into a catalog hierarchy from supported analytics engines. The standard Iceberg REST endpoint of Glue Data Catalog serves table metadata to external engines. BigQuery connects through this endpoint.
  3. AWS Lake Formation: With AWS Lake Formation, you define access permissions at the catalog, database, and table level. Instead of granting broad IAM permissions for data access, Lake Formation evaluates permissions at query time and issues short-lived credentials limited to the resources the caller is authorized to read.
  4. Google Cross-Cloud Lakehouse: With Google Cross-Cloud Lakehouse, you can connect BigQuery to external Iceberg catalogs. It assumes an IAM role using OpenID Connect (OIDC), calls the AWS Glue Iceberg REST endpoint, and syncs metadata on a configurable refresh interval.

Prerequisites

Before you begin, you need:

  • An AWS account with Amazon S3 Tables available in your AWS Region.
  • A Google Cloud project with billing enabled and the BigLake API activated.
  • AWS Command Line Interface (AWS CLI) and gcloud CLI installed and configured.
  • An S3 table bucket with at least one namespace and table containing data.

Setting up Amazon S3 Tables

If you already have S3 Tables with data, skip to the next section. Otherwise, create a table bucket, namespace, and populate a table.

Create a table bucket and namespace

Use AWS CLI to create resources as follows:

# Create a Table bucket
aws s3tables create-table-bucket \
    --name <TABLE_BUCKET_NAME> \
    --region <REGION>

# Create a Namespace (Database)
aws s3tables create-namespace \
    --table-bucket-arn "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET_NAME>" \
    --namespace <NAMESPACE> \
    --region <REGION>

Set up S3 Tables integration with the Glue Data Catalog using Lake Formation mode

Lake Formation needs its own service role to interact with S3 Tables on your behalf. This is the role Lake Formation assumes internally when it reads or writes data on behalf of authorized callers.

Create a Lake Formation service IAM role named LakeFormationS3TablesServiceRole with the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LakeFormationPermissionsForS3ListTableBucket",
      "Effect": "Allow",
      "Action": ["s3tables:ListTableBuckets"],
      "Resource": ["*"]
    },
    {
      "Sid": "LakeFormationDataAccessPermissionsForS3TableBucket",
      "Effect": "Allow",
      "Action": [
        "s3tables:CreateTableBucket", "s3tables:GetTableBucket",
        "s3tables:CreateNamespace", "s3tables:GetNamespace",
        "s3tables:ListNamespaces", "s3tables:DeleteNamespace",
        "s3tables:DeleteTableBucket", "s3tables:CreateTable",
        "s3tables:DeleteTable", "s3tables:GetTable",
        "s3tables:ListTables", "s3tables:RenameTable",
        "s3tables:UpdateTableMetadataLocation", "s3tables:GetTableMetadataLocation",
        "s3tables:GetTableData", "s3tables:PutTableData"
      ],
      "Resource": ["arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/*"]
    }
  ]
}

Attach the following trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LakeFormationDataAccessPolicy",
      "Effect": "Allow",
      "Principal": { "Service": "lakeformation.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:SetContext", "sts:SetSourceIdentity"],
      "Condition": { "StringEquals": { "aws:SourceAccount": "<AWS_ACCOUNT_ID>" } }
    }
  ]
}

In the Lake Formation console, in the navigation pane, choose Catalogs, and then choose Enable S3 Table Integration.

The Enable S3 Table Integration option on the Catalogs page of the Lake Formation console

Figure 2: Enabling the S3 Tables integration in the Lake Formation console

Choose the role you created earlier when prompted for an IAM role, and select Allow external engines to access data in Amazon S3 locations with full table access.

S3 Tables integration performs the following:

  1. Registers the S3 Tables data location with Lake Formation.
  2. Creates the s3tablescatalog federated catalog in Glue.

Important: Before enabling the integration, verify your Lake Formation data lake settings have empty default permissions to prevent IAMAllowedPrincipals from being auto-granted on the catalog:

aws lakeformation put-data-lake-settings \
    --data-lake-settings '{"DataLakeAdmins":[{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ADMIN_ROLE>"}],"CreateDatabaseDefaultPermissions":[],"CreateTableDefaultPermissions":[]}' \
    --region <AWS_REGION>
The S3 Tables integration dialog in Lake Formation with full table access selected

Figure 3: Selecting full table access for external engines during S3 Tables integration

When you select this option, you  allow external engines to access data in Amazon S3 locations with full table access, and Lake Formation grants full table-level access to external engines. Column-level and row-level filtering are not enforced for external engine connections. Access is granted at the whole-table level.

Verify the integration by confirming the catalog in Lake Formation console.

Create a table and insert data

Now, to create the table and insert data, open the Amazon Athena console. In the query editor, select s3tablescatalog/<TABLE_BUCKET_NAME> as your data source and <NAMESPACE> as the database. Then run the following SQL statements one by one:

CREATE TABLE `<NAMESPACE>`.orders (
    order_id STRING,
    customer_id STRING,
    amount BIGINT,
    order_date DATE,
    region STRING
)
TBLPROPERTIES ('table_type' = 'iceberg');

INSERT INTO orders
VALUES
    ('ORD-001', 'C100', 4500, DATE '2024-06-01', 'EMEA'),
    ('ORD-002', 'C200', 8900, DATE '2024-06-01', 'EMEA'),
    ('ORD-003', 'C100', 3200, DATE '2024-06-02', 'NAMER'),
    ('ORD-004', 'C300', 12000, DATE '2024-06-02', 'NAMER'),
    ('ORD-005', 'C400', 6700, DATE '2024-06-03', 'APJ'),
    ('ORD-006', 'C200', 4100, DATE '2024-06-03', 'APJ'),
    ('ORD-007', 'C500', 9500, DATE '2024-06-04', 'EMEA'),
    ('ORD-008', 'C100', 2800, DATE '2024-06-04', 'LATAM'),
    ('ORD-009', 'C600', 15000, DATE '2024-06-05', 'NAMER'),
    ('ORD-010', 'C300', 7200, DATE '2024-06-05', 'LATAM');

Configuring cross-cloud access

BigQuery assumes an AWS IAM role using OIDC federation to access the AWS Glue IRC. This section walks through creating the role, OIDC provider, and permissions.

Create the OIDC identity provider

Register Google as an OIDC identity provider in your AWS account. This allows AWS to validate tokens issued by Google’s identity service:

aws iam create-open-id-connect-provider \
    --url https://accounts.google.com \
    --client-id-list accounts.google.com \
    --thumbprint-list 08745487e891c19e3078c1f2a07e452950ef36f6

The –thumbprint-list parameter is optional. When omitted, IAM automatically retrieves the thumbprint from the OIDC provider’s certificate. See AWS documentation for details.

Create the cross-cloud IAM role on AWS

Sign in to the AWS Management Console. Create the role with a placeholder trust policy. You will update it with the actual BigLake service account ID after you create the federated catalog in Google Cloud.

aws iam create-role \
    --role-name bigquery-cross-cloud-role \
    --max-session-duration 43200 \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["PLACEHOLDER"],
            "accounts.google.com:aud": ["PLACEHOLDER"]
          }
        }
      }]
    }'

The --max-session-duration 43200 allows sessions up to 12 hours, which is needed for long-running BigQuery queries.

Attach permissions

The permissions policy differs based on your access control approach. For the Lake Formation approach, attach the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueRead",
      "Effect": "Allow",
      "Action": [
        "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases",
        "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions"
      ],
      "Resource": [
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog/<TABLE_BUCKET>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>/*"
      ]
    },
    {
      "Sid": "S3TablesRead",
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTableBucket", "s3tables:ListTableBuckets",
        "s3tables:ListNamespaces", "s3tables:GetNamespace",
        "s3tables:ListTables", "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation", "s3tables:GetTableData"
      ],
      "Resource": [
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>",
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>/*"
      ]
    },
    {
      "Sid": "LakeFormationCredentialVending",
      "Effect": "Allow",
      "Action": ["lakeformation:GetDataAccess"],
      "Resource": "*"
    }
  ]
}

Grant Lake Formation permissions

Lake Formation permissions work as a layered grant model: you grant access at each level of the catalog hierarchy, from catalog down to table. The cross-cloud role needs DESCRIBE on the catalog and database so it can discover what exists, and SELECT plus DESCRIBE on the table so it can read the actual data. Without grants at every level, Lake Formation denies access even if the IAM policy allows it.

If using Lake Formation, grant the bigquery-cross-cloud-role access to your tables:

  • Grant catalog permission: DESCRIBE.
  • Grant database permission: DESCRIBE.
  • Grant table permission: SELECT, DESCRIBE.

Grant Lake Formation permissions on the cross-cloud role (one-time).

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Catalog":{"Id":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>"}}'     --permissions '["DESCRIBE"]'     --region <AWS_REGION>

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Database":{"CatalogId":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>","Name":"<NAMESPACE>"}}'     --permissions '["DESCRIBE"]'     --region <AWS_REGION>

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Table":{"CatalogId":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>","DatabaseName":"<NAMESPACE>","Name":"orders"}}'     --permissions '["SELECT","DESCRIBE"]'     --region <AWS_REGION>

Before granting Lake Formation permissions, revoke the default IAMAllowedPrincipals access. By default, Lake Formation grants IAMAllowedPrincipals full access to all databases and tables, so you first need to revoke this to enforce fine grain access. IAMAllowedPrincipals provides backward compatibility when you start using Lake Formation permissions to secure the Data Catalog resources that were earlier protected by IAM policies for AWS Glue.

Set up Lake Formation for external engines

For table metadata to sync from Glue to BigLake/BigQuery, the following Lake Formation settings are required. You might notice that a similar setting also appeared during the S3 Table integration setup. The first one registers the data location and enables external access at the catalog level, while this one enables the Lake Formation credential vending mechanism at the account level for all external engines. For a clean cross-cloud setup, we recommend that you enable both.

In the Lake Formation console, choose Administration, then Application integration settings, and then select Allow external engines to access data in Amazon S3 locations with full table access.

Application integration settings in the Lake Formation console with external-engine access enabled

Figure 4: Enabling external-engine access in Lake Formation application integration settings

Connecting BigQuery to S3 Tables

With the AWS side configured, create the federated catalog in Google Cloud that connects BigQuery to the AWS Glue IRC.

Create the federated catalog

Authenticate to Google Cloud using gcloud auth login, or use Cloud Shell, which is pre-authenticated. Verify the BigLake API is enabled:

gcloud services enable biglake.googleapis.com --project="<GCP_PROJECT_ID>"

For Lake Formation mode (with credential vending):

gcloud alpha biglake iceberg catalogs create <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --catalog-type=federated \
    --federated-catalog-type=glue \
    --glue-aws-region=<AWS_REGION> \
    --glue-aws-role-arn=arn:aws:iam::<ACCOUNT_ID>:role/bigquery-cross-cloud-role \
    --glue-warehouse=<ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET> \
    --primary-location=<GCP_REGION> \
    --credential-mode=vended-credentials

The --glue-warehouse parameter uses the format <AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>. This tells the AWS Glue IRC to scope requests to your specific S3 Tables bucket within the federated catalog hierarchy.

The --credential-mode=vended-credentials flag (Lake Formation mode) instructs BigQuery Lakehouse to request scoped temporary credentials from Lake Formation rather than using the role’s IAM permissions directly for data access.

The --primary-location refers to the Google Cloud region where the federated catalog metadata is stored. Use the AWS to Google Cloud region mapping to find the corresponding GCP region for your AWS Region. For example, AWS us-east-1 maps to GCP us-east4.

Retrieve the BigLake service account ID

After catalog creation, Google provisions a dedicated service account for your federated catalog. Retrieve its numeric ID:

BIGLAKE_SA_ID=$(gcloud alpha biglake iceberg catalogs describe <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --format="value(biglake-service-account-id)")
echo $BIGLAKE_SA_ID

Update the AWS trust policy

Back on AWS, replace the placeholder in the IAM role’s trust policy with the actual service account ID:

aws iam update-assume-role-policy \
    --role-name bigquery-cross-cloud-role \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["<BIGLAKE_SA_ID>"],
            "accounts.google.com:aud": ["<BIGLAKE_SA_ID>"]
          }
        }
      }]
    }'

Register the service account ID in the OIDC provider’s audience list. Without this step, AWS rejects the token because the aud claim doesn’t match any registered client:

aws iam add-client-id-to-open-id-connect-provider \
    --open-id-connect-provider-arn "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com" \
    --client-id "<BIGLAKE_SA_ID>"

Set up metadata sync

Wait 3–5 minutes for IAM changes to propagate globally, then set up background refresh:

gcloud alpha biglake iceberg catalogs update <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --refresh-interval=300s

The --refresh-interval (300 seconds in this example) determines how often BigQuery syncs metadata from the AWS Glue IRC. New tables and schema changes appear in BigQuery within this interval.

Querying from BigQuery

After the catalog refresh completes, BigQuery automatically creates external datasets corresponding to the synced namespaces. No manual CREATE SCHEMA is required.

Verify the sync:

gcloud alpha biglake iceberg namespaces list \
    --catalog="<FEDERATED_CATALOG_NAME>" \
    --project="<GCP_PROJECT_ID>"

Run a query in BigQuery:

SELECT * FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders` LIMIT 1000

Sample Query Output:

SELECT
    customer_id,
    COUNT(*) as order_count,
    SUM(amount) as total_spend
FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders`
GROUP BY customer_id
ORDER BY total_spend DESC
BigQuery query results showing order count and total spend per customer from the Amazon S3 Tables data

Figure 5: BigQuery query results returned through Lake Formation credential vending

BigQuery reads the Iceberg metadata to identify which Parquet data files contain relevant data. It also applies partition pruning where applicable, and fetches only the necessary files from S3 Tables managed storage.

Schema evolution

When new columns are added to an Iceberg table on the AWS side (through Spark, Athena, or the AWS Glue IRC), the schema change is captured in Iceberg’s metadata. On the next Lakehouse refresh cycle, BigQuery picks up the new columns automatically. No DDL changes are needed in BigQuery.

Metadata freshness

The s3tablescatalog catalog in AWS Glue is a federated catalog that resolves table metadata live from the S3 Tables service on each request. When a streaming job commits new data to an S3 Table, the latest metadata is immediately available through the AWS Glue IRC. BigQuery sees the update on its next refresh cycle (as configured by --refresh-interval).

OIDC identity federation

The trust relationship between Google Cloud and AWS uses OpenID Connect. When BigQuery Lakehouse needs to access your data, it presents a signed JWT token containing:

  • iss: accounts.google.com (the issuer)
  • sub: The BigLake service account ID (identifies which catalog is making the request)
  • aud: The same service account ID (the intended audience)

AWS validates this token against the registered OIDC provider and trust policy conditions before issuing temporary credentials. Each federated catalog receives a unique service account ID, providing per-catalog isolation and auditability through AWS CloudTrail.

Network path

By default, traffic between BigQuery and AWS travels over the public internet. For workloads requiring private connectivity, Google Cloud supports Cross-Cloud Interconnect or Partner Interconnect. This helps routing queries over a dedicated network path. Refer to the Google Cloud documentation for private interconnect configuration.

Clean up

To avoid ongoing charges, remove the resources created in this walkthrough.

On AWS:

# Delete the table (if created for this walkthrough)
aws s3tables delete-table \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace analytics --name orders --region <AWS_REGION>

# Delete namespace and table bucket
aws s3tables delete-namespace \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace <NAMESPACE> --region <AWS_REGION>

aws s3tables delete-table-bucket --name <TABLE_BUCKET> --region <AWS_REGION>

# Delete IAM role and OIDC provider (if no longer needed)
aws iam delete-role --role-name bigquery-cross-cloud-role

On Google Cloud:

gcloud alpha biglake iceberg catalogs delete <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" --location=<GCP_REGION>

Conclusion

This post demonstrated how to query Amazon S3 Tables from Google BigQuery using AWS Lake Formation credential vending, where Lake Formation manages the permissions and issues temporary, scoped credentials for data access. With the open Iceberg format, you can write data once on AWS and read it from supported engines that speak Iceberg, including BigQuery.

Together with the IAM approach covered in Part 1, two access control modes provide flexibility: IAM for teams who want a straightforward setup and Lake Formation for organizations with complex governance requirements where multiple engines need centrally managed access to the same data.

To get started with this pattern in your environment:


About the authors

Lakshmi Nair

Lakshmi Nair

Lakshmi is a Principal Analytics Specialist Solutions Architect at AWS. She specializes in designing advanced analytics systems across industries. She focuses on crafting cloud-based data platforms, enabling real-time streaming, big data processing, and robust data governance.

Srividya Parthasarathy

Srividya Parthasarathy

Srividya was a Senior Big Data Architect on the AWS Lake Formation team. She works with product team and customer to build robust features and solutions for their analytical data platform. She enjoys building data mesh solutions and sharing them with the community.

How a global payment processor preserved AWS RAM shares and Lake Formation permissions during an AWS Organizations migration

Post Syndicated from Sam Mukherjee original https://aws.amazon.com/blogs/architecture/how-a-global-payment-processor-preserved-aws-ram-shares-and-lake-formation-permissions-during-an-aws-organizations-migration/

Accounts move between organizations in AWS Organizations whenever a business changes shape. A merger folds one estate into another. A divestiture carves one out, and some companies run more than one organization by design.

The moves take more care when AWS Resource Access Manager (AWS RAM) resource shares are involved. An organization-bound share trusts an account through its organization membership, so when the account leaves, AWS RAM removes that association. Anything in production that depends on a shared resource needs a continuity plan before the first account moves.

A leading worldwide provider of payment technology and software solutions, based out of the United States, used temporary AWS RAM resource shares to preserve AWS Lake Formation permissions during an AWS Organizations migration of 382 AWS accounts. The payment processor serves merchants and financial institutions around the world. The program separated them from their former parent company, a US-headquartered financial technology provider serving banking and capital markets clients globally, before a Transitional Service Agreement (TSA) expired in April 2026.

The migration unfolded alongside wider corporate change. In January 2026, the payment processor was acquired by a leading payments technology company headquartered in the United States. The transaction transforms the acquirer into a pure-play commerce solutions provider, serving the full spectrum of clients from small businesses to global enterprises worldwide. The migrating estate also included the payment processor’s embedded payments platform, a US-based provider of embedded payment and automated onboarding tools for software-as-a-service (SaaS) platforms.

Most workloads kept running when the original organization-bound shares broke, but the control plane lost access. They needed a migration pattern that preserved service continuity without leaving temporary permissions behind.

AWS partnered with them to design and validate that pattern in two weeks. It uses retained bridge shares for the move, then restores the original shares as the durable permission objects.

In this post, we explain why the bridge works, why the original share must return, and how the company applied the pattern at enterprise scale. For command-level implementation, see Transfer AWS accounts between AWS Organizations while preserving AWS Lake Formation permissions and aws-samples/sample-aws-ram-org-migration.

Solution overview

An AWS account can consume a resource to share automatically while it belongs to the same organization as the producer. When the account leaves, AWS RAM removes that organization-bound principal association. Creating a retained bridge share as an external association before the move preserves access through the organization’s change.

After the move, the company restores the migrated account to the original share, verifies access, and removes the bridge. The original share remains the AWS Lake Formation-managed source of truth. New grants and resource changes continue to attach to it, not to the point-in-time bridge copy. Keeping both would create duplicate permission state and drift.

The following diagram shows the migration wave structure.

Migration wave structure. Stage one covers fourteen non-production waves across eight months, none of which crossed an organization boundary. Stage two covers sixteen production waves: one pilot wave, twelve scheduled waves on a weekly cadence, and three contingency waves. The transitional service agreement expires inside the contingency window, leaving only the first contingency week usable.

The company’s cloud engineering team ran 14 non-production waves, a production pilot, 12 weekly production waves, and three contingency waves. The TSA expired inside the contingency window, leaving about one week of usable slack.

Challenge

The risk surfaced in a production wave in February 2026. A terraform apply against a shared AWS Transit Gateway failed with a permission error, although traffic kept flowing and no alarm fired.

This was a control plane failure. Existing attachments, DNS paths, and certificates continued to work, but engineers could not change shared resources. The dedicated AWS Transit Gateway, Amazon Route 53 Resolver, and the embedded payments platform’s AWS Glue Data Catalog waves were still ahead.

Why the issue stayed hidden

Most services retain their data plane when an AWS RAM association breaks. For example, an Amazon Elastic Compute Cloud (Amazon EC2) instance in a shared Amazon Virtual Private Cloud (Amazon VPC) keeps running, but the account cannot launch a new instance. Infrastructure-as-code exposed the problem because it needed control-plane access.

The following table summarizes the affected resource types confirmed by the customer.

Resource AWS service Effect of losing the share
AWS Transit Gateway ec2:TransitGateway Loses control plane access, keeps the data plane
Amazon Route 53 Resolver rules route53resolver:ResolverRule Risk of DNS resolution disruption
AWS Private Certificate Authority (AWS Private CA) acm-pca:CertificateAuthority Loses the share, issued certificates keep working
Amazon EC2 prefix lists ec2:PrefixList Keeps the data plane, blocks new resource creation
AWS Glue Data Catalog databases and tables AWS Glue and AWS Lake Formation Requires bridge-share validation before migration

Some services require additional handling. Depending on its resource-cleanup configuration, an AWS Firewall Manager policy can remove AWS Network Firewall rules, which you must then redeploy, and organization-integrated AWS CloudFormation StackSets can delete stacks unless you set them to retain.

The embedded payments platform initially entered the estate through an acquisition by the former parent company. Its embedded payment and automated onboarding capabilities were subsequently integrated into the payment processor’s ecosystem to support a broader platform-focused offering for SaaS providers. The platform shared databases and tables across 10 accounts with account IDs as principals, and the workstream was paused rather than testing the migration against production.

Why the original shares broke

The company had enabled sharing with AWS Organizations in the producer account. AWS RAM therefore trusted each in-organization principal through organization membership, even when a share named an account ID. When an account left the source organization, AWS RAM removed that organization-bound association.

A share created for a principal outside the organization behaves differently. AWS RAM sends an invitation, and the accepted association is external. Because it does not depend on organization membership, it survives the account move. The bridge-share pattern uses this behavior.

Why non-production testing missed it

The company’s non-production accounts were already in a separate organization. They never crossed the boundary that caused production associations to break.

Fourteen clean waves validated the migration process but not the production-only condition. Each validation environment must cross the same trust boundary as production.

Applying the bridge-share pattern

This failure was raised with the AWS account team, which brought AWS RAM, AWS Glue, AWS Lake Formation, and AWS Organizations service teams into the response. The company first used a manual recovery path for 21 accounts while the teams automated a scalable approach.

On February 27, 2026, AWS released RetainSharingOnAccountLeaveOrganization for new AWS RAM resource shares. The setting marks principals as external after they accept the invitation. The customer confirmed that the setting does not retrofit existing shares, so those shares needed a temporary parallel share.

Retaining access during the move

AWS RAM allows a resource to belong to more than one resource share, so a parallel share can exist alongside the original. A second retained share was created alongside each original, targeting the same consumer account.

The consumer accepted the invitation before migration, creating an external association. During the move, AWS RAM removed the original organization-bound association while the bridge continued to grant access. AWS Organizations also support transferring an account directly between organizations, so the move itself does not require an intermediate standalone period.

Why restore the original share?

The bridge is a migration-only continuity copy. The original AWS Lake Formation-created share remains the durable, service-managed permission object. If a team adds a grant or changes a shared resource during the migration window, that change applies to the original share, not automatically to the bridge.

The company therefore restored migrated principals to the original share before deleting the bridge. Leaving both in place would create two permission paths that can diverge, complicate audits, and conceal which share is authoritative.

Automation deletes a bridge only after it finds a non-bridge original whose resources, principals, and permissions cover the bridge and whose associations are all ASSOCIATED. This check confirms the original share’s associations are active before removing the temporary path. Access and connectivity were validated separately, as described in the Outcome section.

Validated workflow

AWS validated the pattern across three test accounts and two organizations before it was used in production. Testing confirmed that allowExternalPrincipals alone was not enough. The bridge also required retainSharingOnAccountLeaveOrganization.

The following diagram shows the bridge before and after the account move.

Bridge share behavior before and after an account moves between AWS Organizations. Before the move, the original organization-scoped share and the accepted bridge share both grant access. After the move, the original share is revoked and the accepted bridge share continues to grant access.

Each production wave used five steps:

  1. Inventory. Map each original share, resource, principal, permission, and Region. AWS RAM is Regional, so repeat the inventory in every in-scope Region.
  2. Create and accept bridges. Create a retained share for the same resource and principals, then accept its invitation from each consumer account before migration.
  3. Migrate. Move the account. AWS RAM removes the organization-bound association, while the accepted bridge keeps access active.
  4. Restore originals. Add the migrated account IDs back to the original shares as external principals. This reactivates the durable shares and includes grants created during the migration window.
  5. Validate and remove bridges. Confirm resources, principals, permissions, and association status, then delete only bridge shares fully covered by active originals.

This workflow ran for every remaining production wave and kept the weekly cadence.

Validating AWS Glue and Lake Formation permissions

The embedded payments platform shared AWS Glue Data Catalog databases and tables across 10 accounts, with account IDs as principals. The configuration was reproduced in disposable accounts, and the resource policy was recorded through a cross-organization move.

The validated automation records principal-to-share mappings, supports dry-run and execute modes, restores principals to the original shares, and deletes bridges only after validation. The embedded payments platform completed its production migration on July 21, 2026.

Outcome

  • The global payment processor migrated 378 of the 382 accounts into its landing zone. The final four awaited approvals from external stakeholders.

The TSA with the former parent company ended on schedule in April 2026. No customer-facing workload lost availability, and the company recorded no network drops across the production waves. The company and AWS moved from discovery to a validated bridge-share pattern in two weeks.

After each migration, the cloud engineering team restored the original shares, verified access and connectivity, and removed the bridges. Deleting the temporary copies confirmed that the original service-managed permission path was active and authoritative.

Production, non-production, and the embedded payments platform now run in one landing zone. They control their own guardrails, security posture, provisioning, and change process.

AWS has published the validated pattern and automation, so other organizations can start with a tested procedure.

Lessons learned

This program produced three lessons for organizations planning similar migrations.

Match validation boundaries to production

A test organization cannot expose this failure unless it crosses the same organization boundary as production. Map each production risk to an environment that can reproduce it before the first wave.

Monitor control plane changes

AWS RAM emits resource share state-change events directly to Amazon EventBridge, and AWS CloudTrail records DisassociateResourceShare API calls for audit. A weekly post-migration sweep provided a periodic reconciliation check to catch stale shares.

Inventory dependencies and destination guardrails

The Account Assessment for AWS Organizations tool inventories AWS RAM dependencies before teams set the wave plan. The company’s cloud engineering team reviewed destination guardrails at the same time. A service control policy that blocked ram:AcceptResourceShareInvitation during migration windows was temporarily adjusted.

Conclusion

The migration of this leading payment technology and software company shows how a retained bridge share can protect access while an account moves between AWS Organizations. The bridge is temporary: restoring the original share keeps AWS Lake Formation permissions aligned with future grants and avoids two sources of permission state. Inventory, dry-run-first automation, and post-move validation helped the company meet its deadline without customer disruption.

Next steps

To apply this pattern, read Transfer AWS accounts between AWS Organizations while preserving AWS Lake Formation permissions and review aws-samples/sample-aws-ram-org-migration. Run the scripts in dry-run mode, validate each Region and account, and engage your AWS account team early when AWS Glue Data Catalog or AWS Lake Formation resources are in scope.


About the authors

Scaling fine-grained access control for enterprise lakehouse using SageMaker Unified Studio and AWS Lake Formation

Post Syndicated from Chintan Agrawal original https://aws.amazon.com/blogs/big-data/scaling-fine-grained-access-control-for-enterprise-lakehouse-using-sagemaker-unified-studio-and-aws-lake-formation/

As enterprise lakehouses grow to thousands of tables across multiple business domains and regions, scaling fine-grained access control becomes a critical governance challenge. Data governance teams spend significant time manually granting table-level permissions, only to face permission drift, inconsistent enforcement, and limited auditability. Without a scalable approach, each new dataset requires manual policy updates, increasing the risk of unauthorized access and slowing time-to-insight for analysts and data scientists.

In this post, we show you how to solve this problem by combining AWS IAM Identity Center, AWS Lake Formation tag-based access control (TBAC), and trusted identity propagation in Amazon SageMaker Unified Studio. You deploy a complete governance architecture using AWS Cloud Development Kit (AWS CDK) that classifies data with LF-Tags, maps IAM Identity Center groups to tag-based policies, and enforces permissions at query time across analytics engines. The solution uses Apache Iceberg tables stored in Amazon Simple Storage Service (Amazon S3) and registered in the AWS Glue Data Catalog.

The core governance challenge

As organizations mature their lakehouse environments, governance complexity increases with each new dataset. Several challenges commonly emerge:

  • Explosive dataset growth: Iceberg-based lakehouses often contain thousands of tables distributed across raw, curated, and conformed zones. Each new dataset introduces additional governance requirements, making table-level permission grants operationally expensive.
  • Multi-domain data ownership: Enterprise lakehouses typically serve multiple business domains such as commercial analytics, clinical research, and regulatory reporting. These domains require strict isolation while still supporting controlled data sharing.
  • Regional data sovereignty: Organizations operating globally must enforce geographic boundaries for sensitive datasets. EU clinical trial data might be restricted by GDPR regulations, whereas US commercial datasets follow different compliance frameworks.
  • Sensitivity-based access controls: Within each domain, datasets vary in sensitivity. Pricing strategies, drug discovery research, and patient-related datasets require stricter access controls than standard operational data.
  • Role explosion: Pure RBAC approaches attempt to encode these dimensions into roles, leading to role proliferation. Manual Lake Formation grants at the table level create permission drift and limited scalability.

To address these challenges, enterprise lakehouse governance must satisfy several criteria:

  • Least-privilege access.
  • Dynamic scalability as new datasets are onboarded.
  • Multi-dimensional enforcement across domain, region, and sensitivity.
  • Auditability traceable to individual users.
  • Automation-ready, configuration-driven workflows.

TBAC addresses each of these challenges directly. Instead of granting permissions on individual tables, you define tag-based policies that automatically apply to any resource matching the tag expression. New datasets inherit access rules through tag inheritance, eliminating manual policy updates (solving explosive dataset growth). Domain and region tags enforce strict isolation between business units (solving multi-domain ownership and regional sovereignty). Sensitivity tags control access within domains without role proliferation (solving sensitivity-based controls and role explosion). The following sections describe the architecture that implements this model and walk you through deploying it end to end.

Reference architecture overview

The governance model integrates identity, metadata, and lakehouse services into a unified access architecture that enforces fine-grained permissions consistently across analytics and machine learning (ML) workloads. The architecture consists of five layers, each handling a distinct responsibility in the access control flow.

The following diagram illustrates the end-to-end architecture, showing how user identity flows from IAM Identity Center through SageMaker Unified Studio to Lake Formation for tag-based policy evaluation against the AWS Glue Data Catalog and Amazon S3 storage layer.

Architecture linking IAM Identity Center, SageMaker Unified Studio, Lake Formation, the Glue Data Catalog, and Amazon S3

Figure 1: End-to-end governance architecture for the enterprise lakehouse

1. Identity and authentication layer: IAM Identity Center manages user identities and group memberships, integrates with corporate identity providers, and provides centralized lifecycle management for enterprise users. IAM Identity Center groups represent business roles and serve as the principals that receive Lake Formation permissions.

2. Unified analytics and ML access layer: Amazon SageMaker Unified Studio serves as the primary interface where analysts, data scientists, and ML engineers discover datasets, run queries, and build ML workflows. Because SageMaker Unified Studio integrates with multiple compute engines, including Amazon Athena, AWS Glue, Amazon EMR, and Amazon Redshift, users can access data using their preferred analytics tools while maintaining consistent governance.

3. Governance and authorization layer: AWS Lake Formation provides fine-grained access control across AWS Glue catalog resources using LF-Tags. Instead of granting permissions directly on databases and tables, Lake Formation evaluates LF-Tag policies dynamically and grants or denies access at query time. Governance teams define access rules once, and Lake Formation automatically applies them to new datasets as they are onboarded.

4. Governance automation layer: Two AWS Lambda functions automate tag assignment and permission provisioning. JSON metadata configuration files drive both pipelines, so governance teams manage access control through configuration rather than manual console operations.

5. Metadata and storage layer: Apache Iceberg tables stored in Amazon S3 form the foundation of the lakehouse. You register these tables in the AWS Glue Data Catalog, which provides centralized metadata management and interoperability across analytics services. Lake Formation evaluates governance decisions at the catalog level rather than independently by each analytics engine.

End-to-end access flow

When a user queries a dataset from SageMaker Unified Studio, the following sequence occurs:

  1. The user authenticates through IAM Identity Center and accesses SageMaker Unified Studio.
  2. SageMaker passes the user’s identity context to downstream analytics services using trusted identity propagation.
  3. The analytics engine requests data access from Lake Formation.
  4. Lake Formation evaluates LF-Tag policies against the user’s IAM Identity Center group membership.
  5. Access is granted or denied dynamically at query time.

Because authorization decisions are centralized in Lake Formation, governance remains consistent regardless of which analytics engine the user employs.

Hybrid RBAC + ABAC governance model

The governance model combines identity context from IAM Identity Center with metadata-driven classification using LF-Tags. The following table summarizes how each layer contributes to the overall governance workflow.

Governance capability IAM Identity Center contribution Lake Formation LF-Tag contribution Governance outcome
Identity context Organizes users into groups aligned with business roles Evaluates permissions using group membership Role-aligned access boundaries
Data classification Provides role eligibility for data access Classifies datasets by domain, region, sensitivity, and layer Attribute-aware authorization
Scalability Simplifies user lifecycle management Automatically applies policies to newly tagged datasets Governance that scales with dataset growth
Operational model Centralizes role lifecycle operations Enables metadata-driven policy automation Reduced administrative overhead

IAM Identity Center defines who can request access, LF-Tags define what datasets are eligible, and Lake Formation enforces policies dynamically at query time.

Enterprise LF-Tag data model

A structured tagging strategy is the foundation of scalable Lake Formation governance. In this solution, the solution classifies datasets across four governance dimensions.

Tag Key Tag Values Purpose Example Usage
region us, eu, global Geographic data location Enforce GDPR compliance for EU data
domain commercial, clinical_research, regulatory Business domain Separate commercial from clinical data
data_class standard, sensitive, regulated Data sensitivity level Restrict access to sensitive pricing data
layer raw, curated, conformed Data processing stage Grant analysts access to curated data only

Together, these dimensions enable multi-dimensional authorization policies that reflect both organizational structure and regulatory requirements.

Tag inheritance and evaluation

LF-Tags can be applied at three resource levels within the Glue Data Catalog: database, table, and column. In this implementation, database-level tags define broad governance attributes (domain, region, layer), table-level tags capture dataset-specific sensitivity (data_class), and column-level tags can further restrict access to individual fields. Lake Formation evaluates the effective tag set at query time by combining inherited and explicitly assigned tags.

For example, a database tagged domain=commercial, region=us, layer=raw automatically applies those tags to all tables within it. A table-level data_class=sensitive tag supplements the inherited tags to distinguish sensitive pricing data from standard sales data. This inheritance model means new tables automatically receive governance coverage without manual tag assignment. To learn more, refer to Lake Formation tag-based access control best practices.

Prerequisites

Before deploying the solution, complete the following setup in the us-east-1 Region. Use the same AWS Region throughout all steps.

  1. AWS account and IAM Identity Center: Enable IAM Identity Center and create test users. Note your Identity Store ID from the IAM Identity Center console under Settings. For setup guidance, see Getting started with IAM Identity Center.
  2. Lake Formation configuration: Complete the following setup in the Lake Formation console:2.1. Change Data Catalog default permissions. In the navigation pane under Administration, choose Data Catalog settings. Uncheck Use only IAM access control for new databases and uncheck Use only IAM access control for new tables in new databases. Choose Save. This makes sure Lake Formation permissions govern access to databases and tables created by the CDK stacks.
    Lake Formation Data Catalog settings with both IAM-only access control checkboxes cleared

    Figure 2: Lake Formation Data Catalog settings with both IAM-only access control checkboxes unchecked

    2.2. Integrate with IAM Identity Center. Complete the prerequisites for IAM Identity Center integration with Lake Formation, including enabling trusted identity propagation.You don’t need to manually create a Lake Formation administrator. The CDK deployment in Step 2: Deploy all stacks automatically registers the required administrators via the LfAdminStack (see lf-admin-stack.ts). S3 data location registration is a post-deployment console step covered after the CDK creates the buckets.

  3. SageMaker Unified Studio: Create a SageMaker Unified Studio domain, select your IAM Identity Center instance for authentication, and enable trusted identity propagation. For a detailed walkthrough, see Accelerate your analytics with Amazon S3 Tables and Amazon SageMaker Lakehouse and enable trusted identity propagation for the domain.
  4. Local tooling: Install AWS Command Line Interface (AWS CLI), Python 3.x, Node.js 18+, AWS CDK CLI (npm install -g aws-cdk), and Git.

Solution overview

Now that you understand the governance model and tag taxonomy, the following section walks you through deploying the complete infrastructure and configuring access control.

The deployment uses AWS CDK (TypeScript) and consists of seven stacks that create the complete governance infrastructure. The CDK app manages stack dependencies automatically, so a single cdk deploy --all command deploys everything in the correct order.

The architecture uses a two-layer data lake pattern. The raw layer stores data as CSV files in Amazon S3, registered as external tables in the AWS Glue Data Catalog. The curated layer uses Apache Iceberg v2 tables for ACID transactions and schema evolution. Three business domains (US Commercial, EU Clinical Research, and Global Regulatory) each have one representative table per layer, giving six tables total.

Lake Formation tag-based access control (TBAC) governs all access using four tag dimensions:

Tag Key Values Purpose
domain commercial, clinical_research, regulatory Business domain isolation
region us, eu Geographic data boundary
data_class standard, sensitive, regulated Sensitivity classification
layer raw, curated Data layer identification

Step 1: Clone the repository and install dependencies

Clone the accompanying repository and install the CDK project dependencies:

git clone https://github.com/aws-samples/sample-aws-smus-governance-automation
cd aws-smus-governance-automation/cdk
npm install

The CDK project is written in TypeScript and uses aws-cdk-lib v2. The lib/ directory contains seven stack definitions, and bin/app.ts wires them together with explicit dependency ordering.

If this is your first CDK deployment in this account and Region, bootstrap the CDK environment. Bootstrapping provisions an S3 bucket and IAM roles that CDK uses to deploy assets:

cdk bootstrap aws://<ACCOUNT_ID>/us-east-1

Step 2: Deploy all stacks

Deploy the entire infrastructure with a single command. Pass your IAM Identity Center Identity Store ID as a CDK context variable:

cdk deploy --all -c identityStoreId=d-xxxxxxxxxx --require-approval never --region us-east-1

CDK will prompt for IAM permission changes on each stack. The --require-approval never flag auto-approves these so the deployment runs unattended.

CDK deploys the seven stacks in dependency order:

  1. LfSetupStack: Lake Formation admin registration + LF-Tags (domain, region, data_class, layer)
  2. GlueRawTablesStack: S3 bucket + three Glue databases + three CSV-backed tables.
  3. GlueCuratedTablesStack: S3 bucket + three Glue databases + three Iceberg v2 tables.
  4. SsoGroupsStack: three IAM Identity Center groups (DataLake-US-Commercial, DataLake-EU-Clinical-Research-Sensitive, DataLake-Regulatory)The three groups map to specific tag combinations that control data access:
    • DataLake-US-Commercial: domain=commercial, region=us, data_class=standard.
    • DataLake-EU-Clinical-Research-Sensitive: domain=clinical_research, region=eu, data_class=sensitive,regulated.
    • DataLake-Regulatory: domain=regulatory (all regions, all data classes within regulatory).

    The following table summarizes the user personas, their group assignments, and the data access each group provides:

  5. AssetTaggingAutomationStack: Tag automation Lambda.
  6. SsoPermissionAutomationStack: Permission automation Lambda.
  7. LfAdminStack: Registers CDK + Lambda roles as Lake Formation admins.

After deployment completes, review the CloudFormation stack outputs. They include S3 bucket names, database names, SSO group IDs, and Lambda function ARNs.

The following figure shows all seven CDK stacks deployed successfully in the CloudFormation console.

CloudFormation console showing all seven CDK stacks in CREATE_COMPLETE status

Figure 3: CloudFormation console showing all seven CDK stacks in CREATE_COMPLETE status

Register S3 data locations with Lake Formation: Now that the S3 buckets exist, register them with Lake Formation. In the Lake Formation console, under Administration, choose Data lake locations, then choose Register location. Register both buckets from the stack outputs (for example, s3://datalake-raw-data-<ACCOUNT_ID>-us-east-1 and s3://datalake-curated-data-<ACCOUNT_ID>-us-east-1). For IAM role, use the default AWSServiceRoleForLakeFormationDataAccess and choose Lake Formation as the permission mode. See Registering an Amazon S3 location for step-by-step instructions.

The following figure shows both data lake S3 locations registered in the Lake Formation console.

Lake Formation Data lake locations page listing the registered raw and curated S3 buckets

Figure 4: Lake Formation Data lake locations page with raw and curated S3 buckets registered

Step 3: Populate sample datasets

The scripts use Amazon Athena to insert sample data. Athena stores query results under the athena-results/ prefix in the shared governance metadata bucket (lf-governance-metadata-<ACCOUNT_ID>-<REGION>) created by the CDK deployment.

Populate the raw and curated tables:

cd ../scripts
python3 populate_raw_layer.py
python3 populate_curated_layer.py

Each script executes INSERT INTO statements through the Athena StartQueryExecution API and waits for completion. You should see success messages for all six tables (three raw, three curated).

After populating the tables, you can verify the data in the Glue Data Catalog. The following figure shows the six tables across the three raw and three curated databases.

AWS Glue Data Catalog showing the six databases and tables created by the deployment

Figure 5: AWS Glue Data Catalog showing the six databases and tables created by the CDK deployment

You can also preview the data by querying a table. The following figure shows sample data from the us_sales_summary table.

Athena query results showing sample commercial rows from the us_sales_summary table

Figure 6: Query results for the us_sales_summary table with sample commercial data

Step 4: Apply LF-Tags to data assets

The following diagram illustrates the governance automation flow, showing how metadata JSON configuration files drive the two Lambda pipelines for asset tagging and SSO permission management.

Governance automation flow with the asset tagging and SSO permission Lambda pipelines

Figure 7: Governance automation flow showing the asset tagging and SSO permission Lambda pipelines

The diagram shows two parallel pipelines, each following three steps:

Asset tagging pipeline (left):

  1. Metadata upload – A data governance administrator uploads metadata JSON files (metadata-raw-tables.json and metadata-curated-tables.json) to the asset-tagging/ prefix in the shared S3 governance metadata bucket. These files define which LF-Tags to assign to each AWS Glue database and table.
  2. Lambda processing – The S3 upload triggers the LakeFormationTagAutomation Lambda function, which reads the metadata and calls the Lake Formation API.
  3. Tag operations – The Lambda creates or updates LF-Tags, then assigns them to the target databases and tables in the AWS Glue Data Catalog.

SSO permission pipeline (right):

  1. Permission upload – Three permission JSON files (one per IAM Identity Center group) are uploaded to the sso-permissions/ prefix. These files define the LF-Tag policy expressions that control data access.
  2. Lambda processing – The upload triggers the LakeFormationSSOPermissionAutomation Lambda function.
  3. Permission operations – The Lambda grants tag-based permissions to the corresponding IAM Identity Center groups through the Lake Formation API.

Both pipelines log execution details to Amazon CloudWatch for monitoring and troubleshooting.

Two metadata JSON configuration files drive the asset tagging Lambda that declaratively define which LF-Tags to apply to each AWS Glue resource:

  • metadata-raw-tables.json: Tag definitions for the three raw layer databases and tables.
  • metadata-curated-tables.json: Tag definitions for the three curated layer databases and tables.

Each entry in these files specifies the following fields:

Field Description Example
catalog_id Your AWS account ID (Glue Data Catalog ID) 123456789012
resource_type DATABASE or TABLE DATABASE
database_name AWS Glue database name raw_us_commercial_db
table_name AWS Glue table name (only for TABLE entries) us_sales_summary
lf_tags Array of LF-Tag key/value pairs to assign [{“TagKey”:“domain”,“TagValues”:[“commercial”]}]
access_type Action to perform (GRANT) GRANT

Parameters you must update before invoking: Replace the catalog_id value in every entry of both files with your own AWS account ID. The database and table names match the resources created by the CDK stacks, so those should not be changed unless you customized the stack parameters.

The following snippet from metadata-raw-tables.json shows a database-level entry and a table-level entry:

[
  {
    "comment": "DATABASE LEVEL TAGS - US Commercial RAW Domain",
    "access_type": "GRANT",
    "resource_type": "DATABASE",
    "catalog_id": "<YOUR_ACCOUNT_ID>",
    "database_name": "raw_us_commercial_db",
    "lf_tags": [
      { "TagKey": "region", "TagValues": ["us"] },
      { "TagKey": "domain", "TagValues": ["commercial"] },
      { "TagKey": "layer", "TagValues": ["raw"] }
    ]
  },
  {
    "comment": "TABLE LEVEL TAGS - US Commercial RAW Table (Standard Access)",
    "access_type": "GRANT",
    "resource_type": "TABLE",
    "catalog_id": "<YOUR_ACCOUNT_ID>",
    "database_name": "raw_us_commercial_db",
    "table_name": "us_sales_summary",
    "lf_tags": [
      { "TagKey": "data_class", "TagValues": ["standard"] }
    ]
  }
]

The Lambda applies tags at two levels: database-level entries assign domain, region, and layer tags, while table-level entries assign the data_class tag (standard, sensitive, or regulated). Because of two-level tagging, new tables added to a tagged database automatically inherit the database-level tags. Only the table-specific data_class tag needs explicit assignment. To learn more about this pattern, refer to Lake Formation tag-based access control best practices.

Invoke the Lambda for both layers:

cd ../lf-asset-tagging-automation
aws lambda invoke \
    --function-name LakeFormationTagAutomation \
    --payload fileb://metadata-raw-tables.json \
    --cli-binary-format raw-in-base64-out \
    response.json

aws lambda invoke \
    --function-name LakeFormationTagAutomation \
    --payload fileb://metadata-curated-tables.json \
    --cli-binary-format raw-in-base64-out \
    response.json

Verify tag assignment using the GetResourceLFTags API:

aws lakeformation get-resource-lf-tags \
    --resource '{"Table":{"DatabaseName":"raw_us_commercial_db","Name":"us_sales_summary"}}' \
    --region us-east-1

You should see domain=commercial, region=us, layer=raw, and data_class=standard in the response.

The following figure shows the LF-Tags assigned to the us_sales_summary table in the Lake Formation console, confirming that both database-level inherited tags and table-level tags are applied correctly.

Lake Formation console showing inherited and table-level LF-Tags on the us_sales_summary table

Figure 8: LF-Tags on the us_sales_summary table showing inherited and table-level tags

Step 5: Provision SSO group permissions

Three permission JSON files (one per IAM Identity Center group) define the LF-Tag policy expressions. Update sso_group with the group UUID from the SsoGroupsStack outputs and identity_center_account_id with your AWS account ID. For detailed configuration, see the repository README.

[
  {
    "sso_name": "DataLake-US-Commercial",
    "sso_group": "<GROUP_UUID_FROM_CDK_OUTPUT>",
    "identity_center_account_id": "<YOUR_ACCOUNT_ID>",
    "resources": [
      {
        "resource_type": "DATABASE",
        "permissions": ["DESCRIBE"],
        "lf_tag_expression": [
          { "TagKey": "domain", "TagValues": ["commercial"] },
          { "TagKey": "region", "TagValues": ["us"] },
          { "TagKey": "layer", "TagValues": ["curated", "raw"] }
        ]
      },
      {
        "resource_type": "TABLE",
        "permissions": ["SELECT", "DESCRIBE"],
        "lf_tag_expression": [
          { "TagKey": "domain", "TagValues": ["commercial"] },
          { "TagKey": "region", "TagValues": ["us"] },
          { "TagKey": "data_class", "TagValues": ["standard"] },
          { "TagKey": "layer", "TagValues": ["curated", "raw"] }
        ]
      }
    ]
  }
]

Apply permissions for each group:

cd ../lf-sso-permission-automation
python3 lambda_function.py us-commercial-permissions.json
python3 lambda_function.py eu-clinical-research-sensitive-permissions.json
python3 lambda_function.py regulatory-permissions.json
aws lakeformation list-permissions \
    --principal '{"DataLakePrincipalIdentifier":"arn:aws:identitystore:::group/<GROUP_ID>"}' \
    --region us-east-1

Step 6: Validate fine-grained access control

With all permissions in place, validate that Lake Formation TBAC enforces the correct access boundaries by signing in to SageMaker Unified Studio as different IAM Identity Center users.

Test as Sarah (US Commercial Analyst) — Sarah belongs to DataLake-US-Commercial, which grants access to standard commercial data only.

SELECT * FROM raw_us_commercial_db.us_sales_summary LIMIT 10;

Sarah sees all rows and columns successfully:

SageMaker Unified Studio results: Sarah’s successful query on us_sales_summary

Figure 9: Sarah’s successful query on us_sales_summary in SageMaker Unified Studio

Querying outside her authorized domain returns an access denied error:

SELECT * FROM raw_eu_clinical_research_db.eu_drug_discovery LIMIT 10;
Access denied error when Sarah queries eu_drug_discovery outside her domain

Figure 10: Access denied when Sarah queries eu_drug_discovery, confirming TBAC enforcement

Test as Dr. Chen (EU Clinical Research Lead) — Dr. Chen can access sensitive and regulated EU clinical research data (eu_drug_discovery) but is denied access to US commercial data (us_sales_summary), confirming regional and domain isolation.

Query results showing Dr. Chen’s successful query on eu_drug_discovery

Figure 11: Dr. Chen’s successful query on eu_drug_discovery

Access denied error when Dr. Chen queries us_sales_summary

Figure 12: Access denied when Dr. Chen queries us_sales_summary

Test as Alex (Regulatory Affairs Specialist) — Alex’s tag expression uses only domain=regulatory without a region constraint, granting cross-regional access to regulatory data while maintaining strict isolation from commercial and clinical research domains.

Query results showing Alex’s successful query on fda_submissions

Figure 13: Alex’s successful query on fda_submissions

Access denied error when Alex queries us_sales_summary

Figure 14: Access denied when Alex queries us_sales_summary

These tests demonstrate that TBAC enforces fine-grained permissions based on user identity, data classification, regional boundaries, and domain separation, without per-table permission grants. As new tables are added and tagged, existing groups automatically gain or are denied access based on their tag expressions. This is the core advantage of TBAC over named resource permissions.

Audit user access with CloudTrail

A key benefit of integrating Lake Formation with IAM Identity Center is the detailed audit trail available through AWS CloudTrail. Filter Event history by Event name GetDataAccess to see every data access event. Each record includes the IAM Identity Center user UUID (userIdentity.onBehalfOf.userId), the specific table accessed (requestParameters.tableArn), and confirmation that trusted identity propagation was used (additionalEventData.LakeFormationTrustedCallerInvocation: true).

CloudTrail GetDataAccess event showing Identity Center user identity and table access details

Figure 15: CloudTrail GetDataAccess event showing Identity Center user identity and table access details

To resolve the user UUID to a human-readable name, query the Identity Store:

aws identitystore describe-user \
    --identity-store-id d-xxxxxxxxxx \
    --user-id <USER_UUID_FROM_EVENT> \
    --region us-east-1

This audit capability provides the detailed access logs required for HIPAA, GDPR, and FDA compliance, showing exactly which users accessed which data and when. Learn about configuring CloudTrail for Lake Formation in Logging Lake Formation API calls with CloudTrail.

Cleanup

Run cdk destroy --all to remove all stacks. Manually delete the retained S3 data buckets (datalake-raw-data-* and datalake-curated-data-*) and revoke any remaining Lake Formation permissions. For detailed cleanup steps, see the repository README.

Conclusion

In this post, we showed you how to implement scalable fine-grained access control for an enterprise lakehouse by combining AWS Lake Formation tag-based access control, IAM Identity Center, and trusted identity propagation in SageMaker Unified Studio. The four-dimension LF-Tag taxonomy, hybrid RBAC + ABAC governance model, and metadata-driven Lambda automation together create a governance architecture where new datasets automatically inherit access policies through tag inheritance, permissions scale without per-table grants, and every data access event is auditable to the individual user through CloudTrail.

To extend this solution, consider adding new business domains, implementing column-level security with LF-Tags, scaling to multi-account architectures with Lake Formation cross-account sharing, or integrating additional analytics services such as Amazon Redshift Spectrum or Amazon EMR.

Get started by deploying the CDK stacks from the accompanying repository. To learn more:


About the authors

Chintan Agrawal

Chintan Agrawal

Chintan is a Solutions Architect with over 7 years of experience, with a specialization in Analytics and Healthcare domain. He possesses a strong enthusiasm for assisting clients in discovering valuable insights from their data. Through his expertise, he constructs innovative solutions that empower businesses to arrive at informed, data-driven choices.

Chaitanya Vejendla

Chaitanya Vejendla

Chaitanya is a Senior Solutions Architect and part of Global Healthcare and Life Sciences industry division at AWS. He focuses on developing strategic plans for building an end-to-end analytical strategy for large biopharma, healthcare, and life sciences organizations. His expertise spans across data analytics, data governance, AI, ML, big data, and healthcare-related technologies.

Automate creating AWS Glue Data Catalog views with AWS SDK for data mesh use case

Post Syndicated from Aarthi Srinivasan original https://aws.amazon.com/blogs/big-data/automate-creating-aws-glue-data-catalog-views-with-aws-sdk-for-data-mesh-use-case/

AWS Glue Data Catalog view is a multi-dialect view that supports querying from multiple SQL query engines, such as Amazon Athena, Amazon Redshift Spectrum, Apache Spark in Amazon EMR and AWS Glue. You can create a Data Catalog view in one account, using an AWS Identity and Access Management (IAM) definer role in the same or different account and use AWS Lake Formation to share the view across multiple accounts. The definer role has the required full SELECT on the base tables to create the view and share it with other users for querying. The Data Catalog assumes the definer role and manages access of the base tables when the view is queried, thus allowing to share a subset of data without sharing the underlying base tables.

AWS Glue now adds AWS SDK support for creating and updating the ATHENA dialect of Glue views. With this addition, you can now create ATHENA and SPARK dialects of Glue views simultaneously, using a cross account IAM definer role. This feature enhances the automation to create and update Glue views, like that of Data Catalog tables. In our earlier blog Create AWS Glue Data Catalog views using cross-account definer roles, we had introduced IAM definer roles in a cross-account use case to create Data Catalog views with SPARK dialects using the APIs – CreateTable() and UpdateTable() – while creating and adding ATHENA dialects using Athena query editor. As a continuation to it, this post shows you how to use the Catalog objects API CreateTable() to programmatically create ATHENA and SPARK dialects using cross-account IAM definer roles, and how to add the ATHENA dialect programmatically for the views that were created earlier with only SPARK dialect.

Cross account definer roles enable enterprise data mesh architectures where multiple accounts are interconnected in a central governance and multiple producers and consumers. The central governance account hosts the database, tables and permissions, while the producer accounts maintain CI/CD pipelines to create and manage those data assets. Having the definer role in producer accounts allows those CI/CD pipelines to be fully managed by IAM roles in the individual accounts.

Key points on creating multi-dialect views using cross-account definer roles

  • ATHENA dialects are validated and asynchronously created. Hence, a cross-account Glue connection is required for validation for every producer account-central governance account pair. This is a one-time setup.
  • SPARK dialects are not validated. Hence SPARK dialect’s create syntax requires SubObjects list of the base tables and StorageDescriptor fields for the columns of the view.
  • Though queries on cross account views can be run using database resource link names, the view definition SQL query for creating the view requires the original database and base table names from the central governance account.
  • If a view has SPARK and ATHENA dialects available, we recommend updating both the dialects of the view simultaneously using update_table() API/SDK, for any changes in the SQL definition of the view or the base table. This will keep both the dialects queryable.
  • Creating and updating both SPARK and ATHENA dialects using cross account definer role is supported using AWS CloudFormation.
  • The Data Catalog view that can be created using cross account IAM definer roles are available in SPARK and ATHENA dialects and currently not supported for Redshift Spectrum dialect.

Prerequisites

We use the same setup used in Create AWS Glue Data Catalog views using cross-account definer roles for the sample database, tables, definer role, resource link, IAM and Lake Formation permissions on those resources and principals between the two AWS accounts. Summarizing the requirements as below.

  • The setup includes a central governance account with Data Catalog database bankdata_icebergdb and two tables transaction_table1 and transaction_table2, a producer account with a Data-Analyst role used as view definer role.
  • Lake Formation permissions on the central account’s database and tables are granted to the producer account Data-Analyst role as per the earlier blog. The definer role in producer account should have database DESCRIBE and CREATE_TABLE permissions, table SELECT and DESCRIBE permission on all columns and rows of the base tables. The IAM permissions required on the definer role are detailed in Prerequisites for creating views. Similarly, follow the earlier blog to create resource link for the shared database and grant Lake Formation permissions on the resource link to the Data-Analyst
  • An Athena data source named centraladmin in the producer account, pointing to the Data Catalog of the central governance account.

Creating ATHENA and SPARK dialects at the same time

Creating both ATHENA and SPARK dialects of a Glue catalog view simultaneously is now supported by the AWS SDK. In the producer account, create a new Glue connection, required for the Athena dialect validation. This is a prerequisite for creating the ATHENA dialect of the Glue catalog view using cross account definer role. Then we create a Glue view with both dialects.

  1. Sign in to the producer account as the Lake Formation admin role, or any role with permission to create AWS Glue connections.
  2. Using an AWS Command Line Interface (AWS CLI) environment, such as AWS CloudShell, create an AWS Glue connection as follows.
    aws glue create-connection --cli-input-json file://athena-validation-connection.json

    The content of athena-validation-connection.json is as follows.

    {
        "CatalogId": "<producer-account-id>",
        "ConnectionInput": {
            "Name": "glue-view-validation-connection",
            "Description": "Glue view Athena cross-account validation connection",
            "ConnectionType": "VIEW_VALIDATION_ATHENA",
            "ConnectionProperties": {
                "WORKGROUP_NAME": "primary",
                "DATA_SOURCE": "centraladmin"
            }
        }
    }

    Note: If you are using Athena for the first time in your account or using Primary workgroup, setup the query results location bucket using Specify a query result location.

  3. Sign out as the Lake Formation admin and sign back in to the producer account as the definer IAM role, Data-Analyst.
  4. Create an AWS Glue view using the create-table CLI command and JSON file, or using the AWS SDK for Python (Boto3) script.
    aws glue create-table --cli-input-json file://create_multipledialects.json

    The content of create_multipledialects.json is as follows.

     {
       "DatabaseName": "rl_bank_iceberg",
       "TableInput": {
         "Name": "view_2dialects_2basetables_fromcli",
         "StorageDescriptor": {
           "Columns": [
             {
               "Name": "transaction_id",
               "Type": "string"
             },
             {
               "Name": "transaction_type",
               "Type": "string"
             },
             {
               "Name": "transaction_amount",
               "Type": "double"
             },
             {
               "Name": "transaction_location",
               "Type": "string"
             },
             {
               "Name": "transaction_date",
               "Type": "date"
             }
         },
         "ViewDefinition": {
           "SubObjects": [
             "arn:aws:glue:us-west-2:<central-account-id>:table/bankdata_icebergdb/transaction_table1",
             "arn:aws:glue:us-west-2:<central-account-id>:table/bankdata_icebergdb/transaction_table2"
            ],
           "IsProtected": true,
           "Representations": [
             {
               "Dialect": "SPARK",
               "DialectVersion": "1.0",
               "ViewOriginalText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id",
               "ViewExpandedText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id"
             },
             {
                "Dialect": "ATHENA",
                "DialectVersion": "3",
                "ViewOriginalText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id",
                "ValidationConnection": "glue-view-validation-connection"
             }
           ]
         }
       }
    }

    Notes about fields in the above CLI input JSON (applies to all SDK):

    • The definer is by default the API caller, but a Definer field can be set to explicitly specify a different IAM role.
    • In the ViewDefinition, database qualifiers are required for SPARK dialect. That is, the SQL definition provided for ViewOriginalText and ViewExpandedText should be in <source_database_name>.<source_table_name> format.
  5. After the view is created, you can inspect the details on the Lake Formation console. The SQL definitions show both ATHENA and SPARK as shown in the following screenshot.

Lake Formation console showing the SQL definitions tab for the new Data Catalog view, with both ATHENA and SPARK dialects listed

If your view creation fails for any of the dialects, you can use the AWS Glue get-table CLI command with --include-status-details to see what the error is and rectify it.

aws glue get-table --database-name <rl_database_name> --name <view_name> --include-status-details

Glue PySpark script

The PySpark script for creating a view with ATHENA and SPARK dialects are provided below. Download and edit the Pyspark script with your bucket name, producer and central account ids, region and relevant Glue resource names: bdb_5773_createview_bothdialects.py

Provide the following settings to run the script in your Glue Studio. For details on running a Spark job in Glue, refer Working with Spark jobs in AWS Glue.

  • Choose Data-Analyst as the job execution IAM role.
  • Choose Glue 5.1 for Glue version.
  • For the Requested number of workers, provide >=4. This is an FGAC Spark driver requirement, which is needed for Glue catalog views. Below screenshot shows these settings.
  • Add the following 2 properties as additional job parameters. A screenshot is shown for reference.
    --datalake-formats = iceberg
    --enable-lakeformation-fine-grained-access=true

    AWS Glue ETL job configuration page showing the additional job parameters set for the multi-dialect view creation script

  • Save and run the Glue job. Check the stdout logs to review the query on the newly created view.

A sample update_table script is also provided below, to illustrate changing the view definition with additional columns. Note the REPLACE keyword:

bdb_5773_updateview_bothdialects.py

Adding ATHENA dialect using SDK to an existing AWS Glue view

You can update an existing AWS Glue view that was created with the SPARK dialect and add the ATHENA dialect using the SDK. The following example uses the update-table CLI command.

aws glue update-table --cli-input-json file://add-athena-dialect.json

The content of add-athena-dialect.json is as follows.

{
    "DatabaseName": "rl_bank_iceberg",
    "ViewUpdateAction": "ADD",
    "TableInput": {
        "Name": "view_sparkfirst_athenanext",
        "ViewDefinition": {
            "Representations": [
                {
                    "Dialect": "ATHENA",
                    "DialectVersion": "3",
                    "ViewOriginalText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id",
                    "ValidationConnection": "glue-view-validation-connection"
                }
            ]
        }
    }
}

Verify the added dialect on the view by reviewing the SQL definitions of the view in Lake Formation console or using GetTable(). If you want to edit the SQL definition or change the base tables of an existing view that has both SPARK and ATHENA dialects, you can do so using the update_table API (using SDK or CLI), with "ViewUpdateAction": “REPLACE” and provide both the dialect definition under ViewDefinition.

You can run queries on the view from the producer account as Data-Analyst. The view can be shared using Lake Formation Tags or named method, just like sharing tables, to additional consumer accounts from the central governance account. The consumer accounts will create a resource link and query the views.

Cleanup

To avoid incurring ongoing costs, clean up the resources you used for this post:

  1. Revoke the Lake Formation permissions granted to the Data-Analyst role and the producer account from the central governance account.
  2. Drop the Data Catalog tables, views, and the database.
  3. Delete the Athena query results from your Amazon Simple Storage Service (Amazon S3) bucket.
  4. Delete the Data-Analyst role from IAM.
  5. Delete the AWS Glue connection and the Athena data source.
  6. Delete the AWS Glue job, if you tried the Python script as an AWS Glue job.

Conclusion

In this post, I demonstrated how to use cross-account IAM definer roles with AWS Glue Data Catalog views, how to create and update ATHENA and SPARK dialects using the Data Catalog CreateTable() and UpdateTable() APIs. The multi-dialect Data Catalog views allow sharing a subset of data from different tables using Lake Formation permissions, including LF-Tags based access control. The cross-account definer roles support multi-account data mesh architectures so that the producer IAM roles can run the CI/CD pipelines in its account. We encourage you to try the feature and share your feedback in the comments.

Acknowledgements: I would like to thank all the team members who worked to add AWS SDK support for creating ATHENA and SPARK dialects together for AWS Glue views – Daniil Arushanov, Wyatt Hawes, Yuxi Wu, Santhosh Padmanabhan and Karthik Devaraj.


About the author

Aarthi Srinivasan

Aarthi Srinivasan

Aarthi is a Senior Big Data Architect working on data, analytics and GenAI topics with the worldwide Specialists Org at AWS. She works with AWS customers and partners to architect data lake solutions, enhance product features, and establish best practices for data governance and analytics services adoption.

Zero Copy access to Apache Iceberg tables in Amazon S3 from Salesforce Data 360 using the Iceberg REST endpoint from AWS Glue Data Catalog

Post Syndicated from Avijit Goswami original https://aws.amazon.com/blogs/big-data/zero-copy-access-to-apache-iceberg-tables-in-amazon-s3-from-salesforce-data-360-using-the-iceberg-rest-endpoint-from-aws-glue-data-catalog/

Companies increasingly need to query and analyze data across platforms without the cost and complexity of moving it. Salesforce and AWS have collaborated to make this possible by providing Zero Copy access to Apache Iceberg tables stored in Amazon Simple Storage Service (Amazon S3) directly from Salesforce Data 360, using the Iceberg REST endpoint from AWS Glue Data Catalog with data access managed by AWS Lake Formation. This integration helps customers federate their Amazon S3 data lakes with Data 360, preserving data governance, freshness, and business semantics without replication.

Zero Copy file federation plays an important role in activating applications and experiences. By removing the need to physically move or copy data, and connecting to data at the storage level, it addresses key challenges including:

  • Cost efficiency – Reduce storage duplication costs and minimize the compute resources required for data pipelines.
  • High scale – Access data with near-native performance at scale through in-Region access.
  • Enhanced agility – Access and analyze data in real time, accelerating time-to-insight and supporting faster response to evolving business needs.
  • Streamlined operations – Remove the complexity of building and maintaining intricate data pipelines, clearing up valuable data engineering resources.

In this post, we demonstrate how AWS and Salesforce customers can access their enterprise data lakes on AWS from Data 360 using Zero Copy file federation.

What is Data 360?

Data 360 is the real-time data engine that activates trusted context across the entire Salesforce platform. It connects all your enterprise data — data warehouses, data lakes, third-party signals, and more — to the business context, logic, and governance that already live in Salesforce, without moving or copying it. With Zero Copy federation, your teams and AI agents always operate from a complete, current, and trusted picture of your business in the moment it’s needed. It serves as the essential system of context for Agentforce, enabling agents to reliably get real work done.

What is Apache Iceberg?

Apache Iceberg is a high-performance, open table format for huge analytic datasets that brings the reliability and simplicity of SQL tables to big data. It’s a thriving open source project under the Apache Software Foundation. Data engineers use Apache Iceberg because it’s fast, efficient, and reliable at any scale and keeps records of how datasets change over time. Apache Iceberg offers integrations with popular data processing frameworks such as Apache Spark, Apache Flink, Apache Hive, Presto, and more.

Why Amazon S3 for Apache Iceberg data lakes?

Amazon S3 is regarded as the best place to build data lakes because of its durability, availability, scalability, security, compliance, and audit capabilities, and its ability to integrate with a broad portfolio of AWS and third-party tools for data ingestion and processing. Apache Iceberg was designed and built to interact with Amazon S3, and provides support for many Amazon S3 features as listed in the Iceberg documentation.

What is Zero Copy file federation?

File federation, also termed catalog federation, uses the Data Catalog to communicate with remote catalog systems to discover catalog objects and to authorize access to their data in Amazon S3. When you query a remote Iceberg table, the Data Catalog discovers the latest table information in the remote catalog at query runtime, getting the table’s Amazon S3 location, current schema, and partition information. Your analytics engine then uses this information to access Iceberg data files directly from Amazon S3, and Lake Formation manages access to the table and data by vending scoped credentials to the table data stored in Amazon S3. This approach avoids metadata and data duplication while providing real-time access to remote Iceberg tables through your preferred AWS analytics engines.

Solution overview

Apache Iceberg file federation lets Data 360 directly query data stored in Amazon S3 without copying or moving the data. This Zero Copy approach provides several benefits:

  • Real-time access to Amazon S3 data from Salesforce.
  • Reduced data movement and storage costs.
  • Simplified data architecture.
  • Improved data freshness.

The following diagram illustrates the architecture of the integration between Data 360 and Amazon S3 using Apache Iceberg file federation.

Key components:

  1. Amazon S3 stores the source data in Apache Iceberg format.
  2. AWS Glue Data Catalog maintains the metadata for Iceberg tables.
  3. AWS Glue Iceberg REST endpoint provides RESTful access to Iceberg tables.
  4. AWS Lake Formation manages metadata and underlying data access for Amazon S3-based data lakes.
  5. Data 360 processes and analyzes the data.
  6. Apache Iceberg connector provides direct access to query Amazon S3 data from Salesforce.

Walkthrough

The following walkthrough shows you how to set up Zero Copy file federation.

Prerequisites

Before you begin, you need the following:

Configure your AWS environment

Set up an Amazon S3 bucket and Iceberg table

Sign in as the data lake admin and complete the following steps:

  1. Open the Amazon S3 console.
  2. Choose Create bucket to create a bucket.
  3. For Bucket type, choose General purpose, provide a Bucket name, and choose Create bucket.
  4. In the bucket, create two prefixes by choosing Create folder.
  5. Name the prefixes athena_iceberg and athena_results.
  6. Inside the athena_iceberg prefix, create another prefix named customer_iceberg.

Create an Iceberg table using Athena

  1. Open the Amazon Athena console.
  2. Choose Query your data in Athena console, then choose Launch query editor.
  3. In Athena, choose Edit settings.
  4. Set s3://<your-bucket-name>/athena_results/ as the Location of query result, then choose Save. Replace <your-bucket-name> with your bucket name.
  5. Choose Editor to return to the query editor page.
  6. To create the database, copy the following query into the query editor and choose Run. You need to be in the Athena Query Editor to run the following commands.
    create database iceberg_db;

  7. To create the Iceberg table, copy the following query into the query editor, replace <s3 bucket location> with your Amazon S3 bucket location hosting the Iceberg table, and choose Run.
    CREATE TABLE iceberg_db.churn (
        state string,
        account_length int,
        area_code string,
        phone string,
        intl_plan string,
        vmail_plan string,
        vmail_message int,
        day_mins double,
        day_calls int,
        day_charge double,
        eve_mins double,
        eve_calls int,
        eve_charge double,
        night_mins double,
        night_calls int,
        night_charge double,
        intl_mins double,
        intl_calls int,
        intl_charge double,
        custserv_calls int,
        churn boolean)
    LOCATION 's3://<s3 bucket location>/iceberg/churn'
    TBLPROPERTIES (
        'table_type'='iceberg',
        'compression_level'='3',
        'format'='PARQUET',
        'write_compression'='ZSTD'
    );

  8. Insert some records into the table.
    -- Sample data insert for "iceberg_db"."churn"
    -- Execute this in the Athena console
    INSERT INTO "iceberg_db"."churn" VALUES
    ('KS', 128, '415', '382-4657', 'no', 'yes', 25, 265.1, 110, 45.07, 197.4, 99, 16.78, 244.7, 91, 11.01, 10.0, 3, 2.70, 1, false),
    ('OH', 107, '415', '371-7191', 'no', 'yes', 26, 161.6, 123, 27.47, 195.5, 103, 16.62, 254.4, 103, 11.45, 13.7, 3, 3.70, 1, false),
    ('NJ', 137, '415', '358-1921', 'no', 'no', 0, 243.4, 114, 41.38, 121.2, 110, 10.30, 162.6, 104, 7.32, 12.2, 5, 3.29, 0, false),
    ('OH', 84, '408', '375-9999', 'yes', 'no', 0, 299.4, 71, 50.90, 61.9, 88, 5.26, 196.9, 89, 8.86, 6.6, 7, 1.78, 2, false),
    ('OK', 75, '415', '330-6626', 'yes', 'no', 0, 166.7, 113, 28.34, 148.3, 122, 12.61, 186.9, 121, 8.41, 10.1, 3, 2.73, 3, false),
    ('AL', 118, '510', '391-8027', 'yes', 'no', 0, 223.4, 98, 37.98, 220.6, 101, 18.75, 203.9, 118, 9.18, 6.3, 6, 1.70, 0, false),
    ('MA', 121, '510', '355-9993', 'no', 'yes', 24, 218.2, 88, 37.09, 348.5, 108, 29.62, 212.6, 118, 9.57, 7.5, 7, 2.03, 3, false),
    ('MO', 147, '415', '329-9001', 'yes', 'no', 0, 157.0, 79, 26.69, 103.1, 94, 8.76, 211.8, 96, 9.53, 7.1, 4, 1.92, 0, false),
    ('WV', 141, '415', '330-8173', 'yes', 'yes', 37, 258.6, 84, 43.96, 222.0, 111, 18.87, 326.4, 97, 14.69, 11.2, 5, 3.02, 0, false),
    ('IN', 65, '415', '329-6603', 'no', 'no', 0, 129.1, 137, 21.95, 228.5, 83, 19.42, 208.8, 111, 9.40, 12.7, 6, 3.43, 4, true),
    ('RI', 74, '415', '344-9230', 'no', 'no', 0, 187.7, 127, 31.91, 163.4, 148, 13.89, 196.0, 94, 8.82, 9.1, 5, 2.46, 0, false),
    ('IA', 168, '408', '363-1107', 'no', 'no', 0, 275.8, 90, 46.89, 230.0, 73, 19.55, 191.3, 57, 8.61, 9.9, 3, 2.67, 4, true),
    ('MT', 95, '510', '394-8006', 'no', 'no', 0, 113.2, 96, 19.24, 269.9, 107, 22.94, 229.1, 87, 10.31, 7.1, 4, 1.92, 1, false),
    ('NY', 62, '415', '371-5765', 'no', 'no', 0, 236.5, 127, 40.21, 145.3, 101, 12.35, 225.0, 103, 10.13, 12.0, 1, 3.24, 5, true),
    ('TX', 109, '408', '356-2992', 'no', 'yes', 33, 190.7, 114, 32.42, 218.2, 111, 18.55, 156.5, 122, 7.04, 11.6, 5, 3.13, 1, false),
    ('CA', 155, '510', '328-8230', 'no', 'no', 0, 197.3, 78, 33.54, 160.2, 86, 13.62, 280.1, 90, 12.60, 8.8, 2, 2.38, 2, false),
    ('WA', 132, '415', '382-1011', 'yes', 'no', 0, 302.7, 67, 51.46, 212.0, 105, 18.02, 265.5, 82, 11.95, 10.3, 4, 2.78, 3, true),
    ('FL', 88, '408', '344-5678', 'no', 'yes', 18, 145.3, 95, 24.70, 187.6, 92, 15.95, 198.2, 108, 8.92, 9.4, 3, 2.54, 0, false),
    ('CO', 201, '510', '367-4321', 'no', 'no', 0, 312.5, 142, 53.13, 178.9, 76, 15.21, 145.7, 95, 6.56, 14.2, 8, 3.83, 6, true),
    ('GA', 56, '415', '390-2244', 'no', 'yes', 12, 178.4, 101, 30.33, 205.1, 119, 17.43, 230.8, 100, 10.39, 8.0, 2, 2.16, 1, false);

Register the bucket with Lake Formation in Lake Formation mode

To use Lake Formation permissions for access control to the churn table, you must register the location. To do that, complete the following actions:

  1. Open the AWS Lake Formation console.
  2. In the navigation pane under Administration, choose Data lake locations.
  3. Choose Register location and enter the following information:
    1. For S3 URI, enter s3://<s3 bucket location>/iceberg/churn. Replace <s3 bucket location> with your Amazon S3 bucket location hosting the Iceberg table.
    2. For IAM role, choose the user-defined IAM role that you created in the prerequisites.
    3. For Permission mode, choose Lake Formation.
  4. Choose Register location.

Enable third-party integration in Lake Formation

From the Lake Formation console, enable full table access for external engines.

  1. Open the AWS Lake Formation console.
  2. On the left pane, expand the Administration section.
  3. Choose Application integration settings and select Allow external engines to access data in Amazon S3 locations with full table access.
  4. Choose Save.

Application integration settings page in the Lake Formation console with full table access enabled for external engines

Set up an IAM user for third-party access

  1. Open the IAM console.
  2. From the left navigation menu, choose Policies, then choose Create policy. Choose JSON and paste the following policy:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "VisualEditor0",
                "Effect": "Allow",
                "Action": "lakeformation:GetDataAccess",
                "Resource": "*"
            }
        ]
    }

  3. Choose next, provide a name for the policy, and choose Create policy.
  4. From the left navigation menu, choose Users, then choose Create user.
  5. For username, enter data_cloud_user, choose next, and choose Attach policies directly.
  6. Choose AWSGlueServiceRole and the policy that you created in step 3. Choose next and Create user.
  7. Choose the user, then choose Security credentials to create an access key.
  8. Scroll down and choose Create access key, choose Applications running outside AWS, and choose Create access key.
  9. Copy the access key and secret access key, and save them securely. You need these to configure the connector in Data 360.

Set up Lake Formation resource permissions for third-party data access

  1. Open the Lake Formation console.
  2. From the left navigation under Data Catalog, choose Databases, then choose the athena_iceberg_db database.
  3. From the Actions menu, choose Permissions, Grant.
  4. In Principals, choose IAM users and roles, and from the menu choose data_cloud_user, which you just created in IAM.
  5. Scroll down to grant permissions by choosing All tables, then choose Select and Describe permissions for the tables.
  6. Choose Grant to apply the permissions.

Set up Apache Iceberg file federation in Data 360

Create and configure the connection

  1. Navigate to Salesforce Setup. For instructions, see Set Up the AWS Glue Data Catalog Connection.
  2. In Data Cloud, choose Setup, then choose Data Cloud Setup.
    Data Cloud Setup page in Salesforce showing options to configure connections
  3. Under External Integrations, choose Other Connectors.
  4. Choose New.
  5. On the Source tab, choose AWS Glue Data Catalog, then choose Next.
    New connector page in Salesforce Data Cloud with AWS Glue Data Catalog selected as the source
  6. Complete the following information shown in the following screen:
    1. In the Authentication Details section, enter the AWS access key ID and AWS secret access key for the IAM user. Make sure that the IAM user has a policy that grants the user read-only access to AWS Glue Data Catalog. Use Lake Formation to configure storage credential vending. This approach is for AWS Glue Data Catalog to vend temporary credentials at run time so that Data 360 can access the underlying storage bucket.
    2. For Catalog URL, enter the URL of AWS Glue Data Catalog. See Connecting to the Data Catalog by using AWS Glue Iceberg REST endpoint.
    3. For Catalog ID, enter the 12-digit AWS account ID linked to AWS Glue Data Catalog.
    4. For Signing Region, enter the host AWS Region where AWS Glue Data Catalog is located.
    5. For Signing Service, enter glue. Data 360 requires the Signing Service, in addition to the AWS access key ID, secret access key, and Signing Region, to sign requests to AWS Glue Data Catalog by using AWS Signature Version 4.
    6. Test the connection and check for the success message.
    7. Save the connection details.

    AWS Glue Data Catalog connection configuration form in Salesforce Data Cloud showing authentication details, catalog URL, catalog ID, signing Region, and signing service fields

  7. After the configuration is complete and saved, the new AWS Glue Data Catalog connection shows up with “Active” status in the Connectors screen.Connectors screen in Salesforce Data Cloud showing the new AWS Glue Data Catalog connection with Active status

Create and configure the data stream

  1. In Data Cloud, on the Data Streams tab, choose New.
  2. Under Other Sources, choose the AWS Glue Data Catalog source, then choose Next.
  3. From the menus, choose the connection that you just set up, choose a database in your AWS Glue catalog where you have an Iceberg table, choose the table that you want to stream, and choose Next.Data stream configuration page showing AWS Glue Data Catalog source with connection, database, and Iceberg table selection menus
  4. Enter the object name and object API name. For more information, see Data Lake Object Naming Standards.
  5. Choose the category to specify the type of data to ingest. For more information, see Category.
  6. Choose a primary key to uniquely identify the incoming records. For more information, see Primary Key.
  7. Choose the source fields you want to ingest, then choose Next. Fields with convertible data types are listed under Supported Fields.Source fields selection page showing supported fields for the Iceberg table data stream
  8. Choose the relevant data space. Choose “Default” if you don’t have any other data space provisioned in your org. For more information, see Data Spaces.
  9. Choose Deploy.
    Data stream deployment confirmation page in Salesforce Data Cloud
  10. After the setup is complete, the new data stream appears in your Data Cloud environment.
    Data Cloud environment showing the newly deployed data stream for the Iceberg table
  11. The data stream is ready. You can now go to the Data Explorer in your Data Cloud environment and start viewing the Iceberg tables that reside in your external AWS account.Data Explorer in Salesforce Data Cloud showing Iceberg tables from the external AWS account

Best practices and considerations

  • Use IAM roles with least-privilege access. Grant only the specific permissions each service or user needs.
  • Implement appropriate Amazon S3 bucket policies. Define bucket-level policies that restrict access by AWS account, VPC endpoint, or IP range.
  • Monitor access patterns. Enable Amazon S3 server access logging or AWS CloudTrail data events to track who reads from and writes to your table buckets.
  • Optimize Iceberg table partitioning. Choose partition keys that align with your most common query filters.
  • Consider data access patterns. Design your table layout around how data is actually queried.
  • Implement lifecycle policies for Amazon S3 objects. Configure Amazon S3 lifecycle rules to transition older data files to other storage classes.
  • Use appropriate Iceberg file compaction strategies. Run compaction regularly to merge small files produced by streaming or frequent batch appends.
  • Monitor data transfer costs. Track cross-Region and internet egress charges using AWS Cost Explorer as applicable.

Clean up

After you finish testing, clean up all the resources in your AWS account that you created (including the Amazon S3 bucket, Athena tables, and other AWS services) to avoid recurring costs.

Conclusion

By implementing Apache Iceberg file federation between Data 360 and Amazon S3, you can create a more efficient and streamlined data architecture. This solution gives you real-time access to Amazon S3 data while using the analytics capabilities of Data 360. As businesses continue to prioritize data-driven decision-making, Zero Copy data sharing plays an important role in unlocking the full potential of customer data across platforms.

To learn more, review the following resources:


About the authors

Avijit Goswami

Avijit Goswami

Avijit is a principal specialist solutions architect at AWS specializing in data and analytics. He helps customers design and implement robust data lake solutions. Outside the office, you can find Avijit exploring new trails, discovering new destinations, cheering on his favorite teams, enjoying music, or testing out new recipes in the kitchen.

Srividya Parthasarathy

Srividya Parthasarathy

Srividya is a Senior Big Data Architect on the AWS Lake Formation team. She works with the product team and customers to build robust features and solutions for their analytical data platform. She enjoys building data mesh solutions and sharing them with the community.

Pratik Das

Pratik Das

Pratik is a Senior Product Manager with AWS Lake Formation. He is passionate about all things data and works with customers to understand their requirements and build delightful experiences. He has a background in building data-driven solutions and machine learning systems

Bill Tarr

Bill Tarr

From software builder to architecture, Bill has 20+ years of experience shaping best-in-class SaaS technology strategies for organizations from startup to enterprise. He’s also an AWS SaaS community leader and a producer of the “Building SaaS on AWS” show on twitch.com/aws, as well as an experienced public speaker with experience at top tier AWS events such as re:Invent, and publisher of SaaS best practices.

How MAPFRE USA modernized fraud claims with Amazon EMR Serverless

Post Syndicated from Lijan Kuniyil original https://aws.amazon.com/blogs/architecture/how-mapfre-usa-modernized-fraud-claims-with-amazon-emr-serverless/

Insurance fraud remains a significant challenge for the insurance industry. Fraudulent claims can increase loss costs, reduce trust, and consume investigation capacity that could otherwise be focused on serving customers. Traditional fraud detection approaches typically rely on rules-based controls, manual investigation triggers, historical claim patterns, and structured-data-only analysis. These approaches are useful for known fraud patterns, but they can struggle to detect sophisticated fraud rings or hidden relationships across claimants, policies, vehicles, providers, addresses, and prior suspicious activities.

MAPFRE USA is a top-rated auto and home insurer in Massachusetts, serving customers in 11 states nationwide. Our coverage includes auto, home, motorcycle, watercraft, business insurance, and more. As part of MAPFRE Group, we’re a worldwide leader serving over 31.1 million customers in more than 100 countries with a team of 31,000 employees. In collaboration with AWS and Neo4j, MAPFRE USA modernized its fraud prevention capabilities by combining graph-based features with machine learning (ML) models deployed on AWS. This initiative focused initially on Massachusetts auto insurance and later expanded to home insurance. It has delivered significant business impact, exceeding $5 million in net present value (NPV) over five years, with realized savings already outperforming projections.

In this post, we share how MAPFRE USA designed and implemented this solution, highlight the technical architecture running on AWS, specifically the MAPFRE data platform called Atenea, and explore lessons learned that can apply to other industries facing complex fraud challenges.

Business challenge

Fraudulent claims aren’t always isolated events. They often involve hidden networks of policyholders, vehicles, providers, and prior suspicious activities. Detecting these complex relationships requires going beyond traditional structured data analysis.

MAPFRE set out with a clear goal:

  • Goal: Improve fraud detection accuracy and claims handling efficiency.
  • Key performance indicator (KPI): Identify fraudulent claims missed by traditional methods.
  • Approach: Develop several ML models using both traditional structured data and 54 graph-based features derived from claim relationships.
  • Deployment: Integrate with Guidewire Claims, so front-line adjusters automatically receive fraud alerts with explanations.

Each flagged claim exposure generates a Guidewire activity showing the top three model drivers, helping investigators understand why the claim was flagged and act quickly.

Technical solution on AWS (Atenea data platform)

The fraud detection platform is built on a modern data architecture on AWS, designed to scale efficiently and support long-term governance.

At its core, the solution uses Apache Iceberg tables stored on Amazon Simple Storage Service (Amazon S3), with metadata managed through the AWS Glue Data Catalog and access governed through AWS Lake Formation as part of the Atenea lakehouse governance model. The platform feature store is implemented through feature-store-managed Iceberg tables that manage model features, predictions, and Guidewire activities. The implementation is structured across three logical layers:

  • Silver layer: Iceberg tables that contain source data from each of the sources. Used as the initial consumption point of the platform.
  • Gold layer: Iceberg tables storing intermediate data, such as unified Guidewire activity logs, Auto features, and Home features.
  • Platinum layer: Feature Store-managed Iceberg tables containing encoded features and model predictions, making them reusable across models and ensuring strong metadata governance.

Processing pipelines are executed on Amazon EMR Serverless, with orchestration managed by Apache Airflow operators running on Amazon Managed Workflows for Apache Airflow (MWAA). This provides elastic, cost-efficient compute for both batch processing and fast-time scoring, while keeping orchestration, monitoring, and recovery centralized.

For graph enrichment, the platform connects to Neo4j using a dedicated driver, enabling advanced network-based features like suspicious claim linkages, provider fraud ratios, and centrality metrics.

This architecture supports efficient, reliable, and transparent production execution. It uses repeatable Airflow orchestration, environment-based continuous integration and continuous delivery (CI/CD) promotion, centralized monitoring, failure notifications, retry mechanisms, dead-letter queue handling for Guidewire integration, and controlled secret management. At the same time, the layered lakehouse design keeps the platform flexible enough to evolve with new business needs and fraud detection use cases.

Fraud detection architecture on AWS showing data ingestion to Amazon S3, the Silver, Gold, and Platinum Iceberg layers, Neo4j graph enrichment, Amazon EMR Serverless processing, and Guidewire integration

The data sources here are policy, claims, vehicles, and notes (from AS400 and Guidewire), which are structured data. Derived features that capture entity relationships make up the graph data.

Let’s go through the architecture overview:

  1. Data ingestion – Claim batch data is uploaded to Amazon S3. The data is standardized and materialized in Iceberg tables within the Silver layer.
  2. Graph enrichment – Data processed to update Neo4j graph database hosted on AWS.
  3. Model training and scoring – Batch scoring for several ML models.
  4. Model orchestration – Unified orchestration for ingestion, training, and inference using Apache Airflow operators. CI/CD pipelines for promotion across environments.
  5. Execution platform – Amazon EMR Serverless for cost-efficient Spark processing. Migration to Apache Iceberg plus AWS Glue Data Catalog for scalable metadata handling.
  6. Integration with claims systems – Fraud predictions automatically create Guidewire activities, enriched with a description for investigators.
  7. Secrets and security – AWS Secrets Manager securely stores credentials and tokens for Guidewire API integration, with environment-specific and region-specific access controls.
  8. Monitoring and reliability – Amazon CloudWatch and Amazon Simple Notification Service (Amazon SNS) provide visibility into pipeline health and notify teams on failures. Data quality checks are executed at key stages of the pipeline to validate data availability, schema consistency, completeness, and business-rule expectations before outputs are consumed by models or sent to Guidewire.

Guidewire integration with MLOps on AWS

One of the most important parts of MAPFRE’s solution was closing the loop between ML predictions and the claims handling system. This required a resilient integration between the Atenea data platform on AWS and Guidewire Claims.

Integration flow:

  1. When an ML use case finishes scoring, the results are written as JSON files into the S3 path: <bucket_name>/guidewire/.
  2. An S3 event notification triggers the AWS Lambda function LambdaXXXInvokeGuidewireAPI.
  3. This Lambda function:
    • Reads the JSON file.
    • Calls the Guidewire Predictive Model API.
    • Because Guidewire doesn’t support batch requests, the Lambda function sends each JSON payload individually. This keeps the integration compatible with Guidewire and isolates failures at the individual activity level, but it increases the number of API calls and makes retry, throttling, DLQ handling, and monitoring controls important.
  4. If successful, the API responds with HTTP 201 (activity created).
    • If not, the Lambda retries up to two times.
    • Failed requests are sent to an SQS Dead-Letter Queue (DLQ) and an SNS notification is published to an SNS queue for monitoring.
  5. Secrets are stored in AWS Secrets Manager and injected as Lambda environment variables, along with AWS Region-specific URLs for token retrieval and API endpoints.
  6. Example JSON structure for Guidewire integration:
    {
      "method": "createPredictiveActivity",
      "params": [
        {
          "claimNumber": "AUXXXXXXX",
          "exposureNumber": 1,
          "subject": "Fraud alert from ML model",
          "description": "Claim flagged as potential fraud based on graph + ML features",
          "shortSubject": "ML_Fraud_Flag",
          "priority": "high",
          "availableForClosedClaim": true,
          "autoCloseOnExposureClosure": false,
          "targetDays": 4,
          "escalationDays": 6
        }
      ]
    }

Guidewire integration flow from Amazon S3 to an AWS Lambda function that calls the Guidewire API, with an SQS dead-letter queue and Amazon SNS for failures

Key benefits of this integration:

  • Real-time actionability – Fraud predictions automatically create Guidewire activities for front-line adjusters.
  • Resilience – Built-in retries, DLQ handling, and SNS alerts keep failed events from being lost.
  • Security – Secrets and tokens are managed using AWS Secrets Manager, with strict environment separation (dev, pre, pro).
  • Scalability – Any new MLOps use case writes results into the S3 output path, automatically flowing into Guidewire.

This integration shows that fraud models don’t just exist in isolation but actively augment daily claim workflows in production. It connects Atenea’s MLOps pipelines on AWS directly with business decisioning systems, which is critical to realizing the fraud savings impact.

Data quality and resilience

For robustness, we apply data quality checks on ingestion pipelines and graph features. Automated validation detects anomalies early, monitoring dashboards track KPIs and model performance, and standardized recovery and promotion processes run across environments.

Visualization and investigative tools

Neo4j Bloom supports Special Investigations Unit (SIU) workflows by visually exploring entity relationships, such as a provider linked across multiple suspicious claims, accelerating fraud ring identification.

Neo4j Bloom graph visualization showing a provider node linked across multiple suspicious insurance claims

Conclusion

The fraud detection model for auto claims has enhanced MAPFRE USA’s ability to identify fraudulent activity, driving significant savings and improving overall claims efficiency.

During the pilot phase alone, savings exceeded projections by over half a million dollars, and in production the initiative has proven an NPV of more than $5M at current business volumes. These results confirm the business case and highlight the strength of combining structured data with graph-based features to uncover fraud networks that traditional approaches miss.

The results have been compelling:

  • Accuracy gains – detection improved by 50–135 percent compared to baseline methods.
  • Realized value – In 2025, MA Auto and MA Home claim savings reached a combined total of $6.81M, with $6.59M from MA Auto and $225K from MA Home.
  • Proven return on investment (ROI) – the project delivered an NPV of $4.7M at approval, and results are already exceeding expectations.
  • Cross-functional success – the initiative brought together Claims, IT Data, Advanced Analytics, and Neo4j teams in an agile, collaborative model.

Beyond the financial outcomes, several lessons emerged. First, cross-functional collaboration between groups like Claims, Data Engineering, Advanced Analytics, and technology partners like AWS and Neo4j was critical to success. Second, explainability proved essential. By presenting adjusters with the top model drivers directly in Guidewire, we increased trust and adoption of the system substantially. Finally, building resilience into the architecture through monitoring, retries, and data quality processes helped the models operate reliably in production.

Looking ahead, the platform is well-positioned to expand beyond fraud detection. New use cases such as underwriting anomaly detection, customer entity resolution, and retention modeling are already on the roadmap. With a robust architecture built on AWS using Amazon EMR Serverless, Apache Iceberg on Amazon S3 supported by AWS Glue Data Catalog and AWS Lake Formation, a custom-built Feature Store, and Neo4j, MAPFRE now has a scalable foundation to continue driving innovation and business impact.

To start building a similar solution, open the Amazon EMR console and review the AWS Architecture Center for reference patterns you can adapt to your own fraud detection and analytics workloads.


About the authors

How BigBasket uses the Iceberg based lakehouse architecture on AWS to power lightning-fast grocery delivery across India

Post Syndicated from Annie Mattoo original https://aws.amazon.com/blogs/big-data/how-bigbasket-uses-the-iceberg-based-lakehouse-architecture-on-aws-to-power-lightning-fast-grocery-delivery-across-india/

Delivering fresh groceries to millions of customers across India in a few minutes demands a radically modern data architecture and resilient processes to help the business make faster decisions. This is what BigBasket was able to achieve by building a lakehouse architecture on AWS.

In this post, we demonstrate how BigBasket implemented the lakehouse architecture on AWS, including their architecture decisions, implementation approach, and the measurable business results you can expect from a similar modernization. Whether you’re facing scalability challenges or planning your own lakehouse implementation, this blueprint provides actionable insights you can adapt for your organization.

About BigBasket

BigBasket (Innovative Retail Concepts Private Limited) is India’s largest online supermarket, serving millions of customers across over 60 cities. Founded in 2011, the company offers groceries, fresh produce, household items, and personal care products through its mobile app and website, operating subscription services (BBDaily) and quick commerce (bbnow). For BigBasket, the ability to deliver groceries on time isn’t only a competitive advantage. It’s the foundation of customer trust, where every minute counts.

However, rapid business growth brought significant operational challenges:

  • Inability to consistently meet on-time delivery adherence because of high order volumes, extended travel times, and more, directly impacting key metrics like on-time rate (OTR)-10 mins and OTR-15 mins.
  • Struggling to meet on-time delivery targets because of picking inefficiency, high order volumes, and extended travel times, directly impacting key metrics like OTR-10 mins and OTR-15 mins.
  • Delays in stock availability impacting vendor fill-rates, inter-distribution center orders, and warehouse operations.
  • Inaccurate stock forecasting for top-selling stock keeping units (SKUs), assortment variety, event SKUs, store capacity, and buying cycles.
  • Lower dark store productivity across picking, stacking, order processing, and goods receipt notes (GRN).

Behind these business challenges lay a fundamental technology problem: the existing data infrastructure couldn’t keep pace. The company experienced rapid store growth, expanding 4x in a short timeframe, which exposed several limitations within their existing data architecture that needed attention.

Understanding the technical bottlenecks

BigBasket’s initial architecture relied heavily on a single data warehouse built on Amazon Redshift to meet all reporting and dashboarding needs. While this traditional approach had served them well initially, several important limitations emerged:

  • Stale data: Extract, transform, load (ETL) pipelines delivered only day-old (D-1) data, making near real-time analysis impossible for dashboard requirements.
  • Extended recovery times: Pipeline failure recovery processes took several hours, causing significant delays in data availability for business users.
  • Schema rigidity: Schema changes in source databases frequently triggered pipeline failures because of a lack of schema evolution support.
  • Scalability constraints: The infrastructure struggled to handle the sudden load increase from 13,000 to over 35,000 transactions for reports and dashboards with more than 1,000 dataset refreshes.
  • Cost implications: Increasing data volumes demanded additional compute resources, driving up costs.

Diagram of the scalability and cost limitations of BigBasket’s legacy Amazon Redshift data warehouse

It became clear that the existing data infrastructure wasn’t able to meet the evolving business requirements and a redesign of their data architecture is needed.

Why lakehouse architecture?

A modern data lakehouse architecture addresses these issues with near real-time data processing, flexible schema evolution, and scalable analytics, capabilities necessary for fast-moving commerce operations. The lakehouse approach combines the flexibility and cost-effectiveness of data lakes with the performance and governance features of data warehouses, combining the strengths of both. The design of a data lakehouse provides interoperability across storage systems for combined analytics activities.

Solution overview

BigBasket partnered with AWS to implement a comprehensive lakehouse architecture using a combination of AWS native services and open-source technologies.

The following diagram shows an elaborated view of Bigbasket’s modernized architecture on AWS.

Detailed lakehouse data flow across bronze, silver, and gold medallion layers on AWS

Data ingestion: Enabling continuous replication

AWS Database Migration Service (AWS DMS) ingests data from online transaction processing (OLTP) databases running on Amazon Relational Database Service (Amazon RDS) into the lakehouse on AWS.

This method continuously replicates data with minimal latency, so your analytics reflect near real-time business operations.

Storage and governance: Building a solid foundation

The lakehouse is built on Amazon Simple Storage Service (Amazon S3) and Amazon Redshift, which serve as the centralized data lake and warehouse following a medallion architecture.

The architecture persists all analytical data using Apache Iceberg as the open table format. Iceberg provides a robust foundation for large-scale analytics with the following capabilities:

  • ACID transactions: Guarantees data consistency and correctness across concurrent read and write operations.
  • Time travel: Supports querying historical table versions for auditing, troubleshooting, and recovery.
  • Schema evolution: Allows schema changes without disrupting existing queries or downstream pipelines.

The medallion architecture structures data across three logical layers within the lakehouse:

  • Bronze layer: Implements change data capture (CDC)-based source replication using AWS DMS. Raw change events flow into Amazon S3 as Apache Parquet files in their original format from source systems, preserving the complete change history. The data pipeline processes and deduplicates these events using Apache Spark on Amazon EMR to create and maintain Apache Iceberg tables that act as replicated source tables.
  • Silver layer: Represents the conformed data model, where data is cleansed, standardized, and validated with enforced quality checks. This layer contains core dimension and fact tables, modeled for analytical consistency and reuse across domains. Data is stored as Apache Iceberg tables on Amazon S3, making it reliable and performant for downstream analytics and transformations.
  • Gold layer: Provides business-ready data marts and wide tables optimized for reporting, dashboarding, and domain-specific use cases. These datasets are curated to align with business metrics and key performance indicators (KPIs) and are served from Amazon Redshift, using Iceberg-backed tables to deliver fast, scalable analytics for business intelligence (BI) tools and end users.

This layered approach maintains a clear separation of concerns across raw ingestion, analytical modeling, and business consumption, while supporting scalability and flexibility across the organization. AWS Lake Formation enforces fine-grained data access controls, and the AWS Glue Data Catalog centrally manages metadata across Amazon S3 and Amazon Redshift, ensuring consistent data discovery and governance across the analytics ecosystem.

Data processing: Flexibility and performance

For data processing and transformations, BigBasket uses Amazon EMR with Apache Spark and dbt, orchestrated by Apache Airflow running on Amazon Elastic Kubernetes Service (Amazon EKS) as the core compute layer of the lakehouse. Apache Spark on Amazon EMR handles large-scale distributed processing, including CDC deduplication, incremental transformations, and complex data reshaping. Apache Iceberg serves as the open table format, which provides several critical capabilities.

dbt is used to define and execute transformation logic using SQL, managing the build of data models such as staging, intermediate, and final tables on top of the raw data. dbt uses the dbt-Trino adapter to run these transformations using the Trino engine, materializing the results as Apache Iceberg tables in Amazon S3. This approach provides a simple, modular, and governed way to manage transformations while taking advantage of Iceberg’s transactional guarantees.

These features are necessary for production lakehouse implementations and help you avoid vendor lock-in while maintaining enterprise reliability.

Online analytical processing (OLAP) and analytics: Hybrid approach for cost optimization

The analytics layer uses a hybrid approach that you can adapt based on your query patterns:

  • Amazon Redshift: For querying of active, frequently accessed data from the Gold layer.
  • Amazon Athena: For ad-hoc queries on historical data.
  • Apache Trino: For federated queries across multiple data sources while powering dbt-driven transformations directly on Apache Iceberg tables.

This hybrid strategy optimizes costs by keeping frequently accessed data in Amazon Redshift while querying historical data directly from Iceberg tables in Amazon S3. Amazon Redshift data sharing supports a multi-warehouse architecture for cross-team collaboration, allowing different teams to access shared datasets without data duplication.

Orchestration: Managing complex workflows

Apache Airflow running on Amazon EKS orchestrates and schedules data pipelines across the entire environment, providing visibility and control over complex workflows. This gives you a unified view for monitoring and managing your data operations.

Machine learning integration

Amazon SageMaker AI powers machine learning workloads for predictive analytics and model training directly on lakehouse data, from demand forecasting to delivery optimization. This tight integration means your data scientists can work with the same governed data that powers your analytics.

Visualization: Making insights accessible

Amazon Quick Sight provides data visualization and business intelligence reporting capabilities, making insights accessible to business users across the organization without requiring technical expertise.

Special focus: Clickstream data processing

BigBasket implemented a sophisticated dual-path architecture for processing clickstream data from mobile apps and web interactions:

  • Real-time path: Data flows through Scala stream collectors on Amazon Elastic Compute Cloud (Amazon EC2) (behind Elastic Load Balancing) to Amazon Kinesis Data Streams and Amazon OpenSearch Service for immediate insights into customer behavior. This path is necessary when you need to react to user actions within seconds, for example detecting fraud or personalizing experiences in real time.
  • Batch path: The batch path validates data, stores it in Amazon S3, processes it through Amazon EMR, and loads it into Amazon Redshift for comprehensive historical analysis. This path handles data quality checks, enrichment, and aggregation for long-term analytics.

The trade-off between these approaches is latency versus completeness. Real-time processing gives you speed but may sacrifice some data quality checks, while batch processing provides accuracy but introduces delay. This dual approach achieves both immediate operational insights and deep analytical capabilities, letting you optimize for different use cases.

The following diagram shows how the clickstream data is handled and effectively processed today.

BigBasket’s dual-path clickstream processing architecture with real-time and batch paths on AWS

The results: measurable business impact

The data platform transformation achieved significant results across multiple dimensions:

Technical improvements

  • Near real-time data: Achieved near real-time data availability for dashboards within 3–5 minutes, replacing previously day-old data.
  • Rapid failure recovery: Pipeline failure re-runs now complete in minutes instead of hours.
  • Comprehensive governance: Full control over data governance with robust observability, lineage, data accuracy, and consistency.
  • Enhanced scalability: Successfully handling over 35,000 reports and dashboards with over 1,000 dataset refreshes.

Business outcomes

  • On-time delivery: Improved monitoring with real-time insights on low-performing stores.
  • Stock availability: Reduced operational issues with visibility into key bottlenecks.
  • Stock forecasting: Improved accuracy and availability of top-selling SKUs.
  • Dark store productivity: Enhanced productivity of warehouse executives across all operations.

Key takeaways: lessons for modern data platforms

BigBasket’s journey offers valuable insights for organizations facing similar challenges:

  1. Quick commerce needs quick observability. In the fast-paced world of quick commerce, faster decision-making directly improves business metrics. Real-time data isn’t a luxury. It’s a necessity.
  2. Embrace ELT for real-time needs. Shifting from traditional ETL to an extract, load, transform (ELT) pattern within a lakehouse architecture is important to unlock near real-time analytics capabilities.
  3. A lakehouse delivers speed and governance. Modern lakehouse architectures don’t force trade-offs. You can achieve both fast data availability and comprehensive control, lineage, and accuracy.
  4. Focus on operational resilience. Designing for rapid failure recovery (re-runs in minutes, not hours) is necessary for maintaining data availability and business trust, especially in customer-facing operations.
  5. Incremental migration. You don’t need to rebuild everything. Evolve your current Amazon S3 data lake or reuse your existing investments in Amazon Redshift to build the data lakehouse capabilities.

The road ahead

BigBasket continues to innovate, now moving to adopt Amazon SageMaker Unified Studio to access all lakehouse components in a simplified manner across the enterprise. This next evolution will further streamline data access and accelerate insights across teams.

The company’s transformation demonstrates that with the right architecture and AWS services, organizations can turn data infrastructure challenges into competitive advantages, delivering not only better analytics but better customer experiences.

As you plan your own lakehouse implementation, use these patterns and lessons learned to accelerate your journey and avoid common pitfalls.


About the authors

Naga Sandeep Grandhi

Naga Sandeep Grandhi

Sandeep is an engineering leader at BigBasket, driving data platform and cloud architecture initiatives, including the next-gen data lake built for scale, reliability, and real-time insights.

Vikram Kumar

Vikram Kumar

Vikram is a Principal Engineer at BigBasket, where he leads the data engineering team. He specializes in designing and scaling modern data platforms on AWS, enabling BigBasket to process large-scale data efficiently and power data-driven decision-making across the organization.

Annie Mattoo

Annie Mattoo

Annie is a Sr. Analytics Specialist at AWS, bringing over 15+ years of expertise in helping customers with their DATA & AI journeys. She has successfully led customer teams to seamlessly adopt AWS Data & AI services and has worked with Fortune 500 customers across the globe in her previous roles.

Vineet Thapliyal

Vineet Thapliyal

Vineet is an Enterprise Account Manager at Amazon Web Services (AWS) in Bengaluru, India, where he manages strategic cloud and generative AI engagements across some of India’s largest conglomerates spanning energy, retail, and technology. He is passionate about helping enterprises unlock business value through AI/ML, cloud modernization, and industry-specific innovation — from renewable energy analytics to retail transformation at scale.

Anirudh Chawla

Anirudh Chawla

Anirudh is an Analytics Solution Architect at AWS. He helps organization empowers businesses to harness their data effectively through AWS’s analytics platform. His interest lies in building highly available distributed systems.

Deploy modern data platforms in minutes with MDAA

Post Syndicated from Sudeshna Dash original https://aws.amazon.com/blogs/big-data/deploy-modern-data-platforms-in-minutes-with-mdaa/

Modern Data Architecture Accelerator (MDAA) is an open source framework that replaces infrastructure code with concise YAML configuration, so your team can deploy a governed, production-ready data architecture, reducing deployment time from months to weeks (depending on complexity and team experience).

Organizations building modern data architecture on AWS face a critical challenge: deploying production-ready, governed infrastructure traditionally requires 6–12 months of custom development, thousands of lines of infrastructure code, and continuous remediation cycles to maintain security and compliance. Governance is often added incrementally, treated as an afterthought that creates compliance gaps and engineering rework.

MDAA addresses this by replacing infrastructure code with concise YAML configuration, achieving up to 97.6 percent code reduction (from approximately 1,800 lines of AWS CloudFormation to 45 lines of MDAA YAML) while embedding governance from the start. The complete Governed Lakehouse Starter Kit deploys 491 AWS resources across 12 stacks from approximately 450 lines of YAML configuration, representing a 66x verbosity ratio where each line automatically expands into production-ready infrastructure.

In this post, we explore how MDAA transforms data architecture development from months of manual coding to production-ready deployment through configuration-driven infrastructure and embedded governance, examine a real customer transformation, and provide a clear implementation pathway for your own data modernization journey.

Customer use case and challenge

A university system office needed to modernize its analytics architecture across 17 campuses while managing sensitive educational data. Their third-party dependency created bottlenecks that slowed feature implementation from weeks to months, and their IT team lacked the cloud skillsets to build modern infrastructure independently.

With MDAA, they achieved:

  • 95 percent reduction in time-to-value for dashboard and feature implementation (from weeks to hours).
  • 17 campuses integrated into a unified, secure architecture.
  • 7.2TB of data and over 8,000 dashboards migrated successfully.
  • Significant cost savings by removing third-party dependencies and reducing license costs.
  • Enhanced security posture for external stakeholders accessing sensitive educational data.

The team used MDAA to implement a modernization strategy with continuous integration and continuous delivery (CI/CD) for automated deployment. The architecture now supports rapid response to stakeholder requests while maintaining strict data governance through AWS Lake Formation.

Their transformation demonstrates what becomes possible when governance is embedded from launch rather than added incrementally, moving from months-long manual development to weeks of production-ready deployment through configuration-driven infrastructure.

Solution: MDAA and its value propositions

MDAA’s capabilities stem from its modular, composable architecture. The accelerator provides over 40 pre-built modules that encapsulate AWS best practices for security, governance, and operational excellence. Organizations describe the outcomes they want in MDAA-specific YAML configuration files (not CloudFormation or Terraform YAML) and the accelerator automatically translates these configurations into AWS Cloud Development Kit (AWS CDK) constructs, which then deploy via CloudFormation with embedded governance.

Configuration over code. The MDAA framework takes a fundamentally different approach: describe the outcomes you want in YAML, and the accelerator deploys production-ready infrastructure with embedded governance. Consider deploying a governed data lake where fraud detection teams need write access to transaction data, while marketing analytics teams require read-only access to customer behavior data. Traditional approaches require over 1,800 lines of CloudFormation across Amazon Simple Storage Service (Amazon S3) buckets, AWS Key Management Service (AWS KMS) keys, AWS Identity and Access Management (IAM) policies, and Lake Formation permissions. With MDAA, the same governed data lake is expressed in 45 lines of configuration, a 97.6 percent reduction, while helping you apply encryption, least-privilege access, and cross-account governance as built-in defaults.

The configuration deploys multi-zone S3 storage with KMS encryption, Lake Formation permissions with tag-based access control (TBAC) enabled, Amazon SageMaker Unified Studio for data product discovery, and encrypted AWS Glue Data Catalog with automated crawlers. All permissions flow through Lake Formation rather than individual IAM policies.

Embedded governance from day one. Governance is declared in YAML and deployed alongside infrastructure from the first run. Fine-grained access controls, encrypted data catalogs, data quality validation, audit trails, and sensitive data classification are all part of the same configuration. MDAA’s Governed Lakehouse starter kit defines an entire governed data architecture in roughly 450 lines of YAML, which produces approximately 29,700 lines of CloudFormation across 12 stacks (a 98.5 percent reduction in infrastructure code).

Modular, composable architecture. Each module is purpose-built to handle a specific capability within the data architecture. Modules communicate through AWS Systems Manager Parameter Store, passing resource identifiers (Amazon Resource Names (ARNs), IDs, and names) between stacks. This approach removes hardcoded dependencies. A KMS key created in one module can be referenced by another through parameter resolution, with all dependencies resolved automatically at deployment time.

The diagram illustrates the deployed architecture and team-level access flow that MDAA generates from the 45-line configuration.

Progressive architecture patterns. MDAA provides four reference architecture patterns that align to progressive stages of data infrastructure maturity:

  • Basic Data Lake deploys a governed data lake with built-in security controls, data quality checks, centralized metadata management using AWS Lake Formation and AWS Glue.
  • Data Science Platform extends the data lake with Amazon SageMaker notebooks, feature stores, and machine learning (ML) pipelines so data science teams can experiment and train models on governed data.
  • SageMaker Unified Studio adds a single interface for analytics and ML collaboration, connecting data engineers, analysts, and data scientists in one workspace.
  • Generative AI Platform layers Amazon Bedrock and Retrieval Augmented Generation (RAG) capabilities on top of your existing data foundation, so teams can build generative AI applications grounded in enterprise data.

Each pattern builds the one before it. You can start with the Basic Data Lake and adopt additional patterns as your team’s needs grow. MDAA’s modular design means you add capabilities without rearchitecting what you already deployed.

The infrastructure is versioned through GitHub, repeatable across environments, and auditable through comprehensive AWS CloudTrail logging. Data engineers focus on data pipelines and business logic while MDAA manages infrastructure complexity and governance integration. This represents the fundamental shift: from writing infrastructure code to describing the outcomes you want through configuration, with governance embedded from the start.

Use case of MDAA: Governed data architecture

DataOps teams spend significant time on governance tasks, including permissions management, compliance validation, and access control, rather than building pipelines and analytics. These aren’t data problems, they’re governance problems that consume engineering capacity meant for higher-value work. MDAA addresses this at the architectural level. Governance is declared in YAML and deployed alongside infrastructure from the first run.

The following sections walk through how each governance module works in practice.

Publish, discover, subscribe, and consume data products between business units: SageMaker Unified Studio

Amazon SageMaker Unified Studio provides a governed data catalog where data producers publish data products, and consumers discover and subscribe to them. Your deployment with MDAA includes a pre-configured domain, blueprints (managed and custom), projects, and environment profiles, all defined in a single configuration file:

# sagemaker.yaml --- 16 lines that deploy 114 CloudFormation resources
domains:
  domain1:
    dataAdminRole:
      id: ssm:/{{org}}/govern1/generated-role/data-admin/id
    description: SMUS Domain 1
    userAssignment: MANUAL

    tooling:
      vpcId: '{{context:vpc_id}}'
      subnetIds:
        - '{{context:private_subnet_id1}}'
        - '{{context:private_subnet_id2}}'

    groups:
      team1:
        ssoId: '{{context:team1-group-sso-id}}'
      team2:
        ssoId: '{{context:team2-group-sso-id}}'

Behind this configuration, MDAA deploys an Amazon SageMaker Unified Studio domain with dedicated KMS keys, execution and provisioning roles, and single sign-on group profiles for team access. Data producers tag and publish assets with metadata, ownership, and classification. Consumers browse a searchable catalog, see only authorized assets, and request access through a governed workflow. Cross-account and cross-business-unit data sharing flows through a subscription model, ensuring every access grant is tracked, auditable, and revocable.

Use case of MDAA: Restricting access to cardholder data using Lake Formation

AWS Lake Formation provides fine-grained access control at database and table levels, removing manual IAM policy management. MDAA deploys AWS Lake Formation with pre-configured settings that disable IAMAllowedPrincipals, the critical governance setting that ensures all permissions flow through centralized governance:

# lakeformation-settings.yaml --- 6 lines that deploy 25 CloudFormation resources
lakeFormationAdminRoles:
  - id: generated-role-id:data-admin
createCdkLFAdmin: true
createDataZoneAdminRole: true
iamAllowedPrincipalsDefault: false

That last flag is the single most important governance setting in the platform. Without it, an IAM principal with glue:GetTable can read tables in the catalog, bypassing the entire access control model. Most manual setups miss this or defer it.

With the data lake configuration, you declare roles and access policies in YAML where admins get full control, engineers get read access to curated data, extract, transform, and load (ETL) roles get scoped write access, and MDAA compiles them into the correct S3 bucket policies and Lake Formation registrations.

Use case of MDAA: Ensuring data integrity with AWS Glue Data Quality

AWS Glue Data Quality runs automated validation rulesets continuously as part of the pipeline, not as periodic batch checks. MDAA’s data quality module supports over 15 built-in rule types, from completeness and uniqueness checks to statistical thresholds and data freshness validation:

# data-quality.yaml
projectName: example-project

rulesets:
  customer-data-quality:
    description: Validate customer data completeness and uniqueness
    targetTable:
      databaseName: project:databaseName/customer-data
      tableName: customers
    ruleset:
      - ruleType: IsComplete
        column: customer_id
      - ruleType: Uniqueness
        column: email
        comparisonOperator: ">"
        threshold: 0.95
      - ruleType: RowCount
        comparisonOperator: ">"
        value: 100

Quality metrics flow into Amazon CloudWatch for real-time alerting. If anomalies are detected, automated workflows quarantine affected records and alert data engineering teams before issues reach downstream consumers.

Protecting metadata at rest: AWS Glue Data Catalog encryption

Table schemas, column names, and partition structures can reveal sensitive information about an organization’s data architecture, even without access to the underlying data. AWS Glue Catalog Encryption secures metadata at rest using AWS KMS-managed keys. MDAA configures catalog encryption by default, so schema definitions and connection passwords are encrypted from initial deployment without requiring manual key management setup. Access to catalog metadata follows the same Lake Formation governance controls applied to the data itself, so teams see only the schemas that they’re authorized to query.

Auditing every data access event: CloudTrail integration

Every data access event must be logged and attributable to a specific identity. Without a complete audit trail, demonstrating compliance during a regulatory review becomes a manual, error-prone process. AWS CloudTrail captures API-level activity across the data infrastructure, recording who accesses what data, when, and from which service. MDAA configures CloudTrail integration by default, so audit logging is active from initial deployment rather than added retroactively. Log data flows into a centralized, tamper-resistant store, giving compliance teams a single location to query access history across all business units and accounts.

Identifying sensitive data automatically: Macie integration

In large environments, sensitive information spreads across dozens of S3 buckets through pipelines, transforms, and ad hoc data drops, and self-reporting data owners consistently produce gaps. Amazon Macie uses machine learning to automatically discover and classify sensitive data in S3, surfacing findings at the object level without manual tagging. MDAA configures Macie across your S3 buckets during deployment, routing findings to Amazon EventBridge where automated workflows can alert owners or trigger remediation.

Together, these controls form a layered defense: Lake Formation governs access to cataloged data, Glue Data Quality validates integrity on arrival, and Macie identifies sensitive data that lands outside governed pipelines to reduce compliance risk.

Multi-account data mesh

MDAA provides extensive support for multi-account data mesh setups, with decentralized data ownership across business units and centralized governance. The data mesh starter kit supports cross-account data product publishing and consumption, allowing organizations to scale data sharing while maintaining consistent security and compliance controls.

Technical implementation

Ready to deploy your modern data architecture? Here are the resources to get started:

MDAA Implementation Guide provides detailed instructions for deploying all starter packages, including architecture patterns, configuration examples, security best practices, and troubleshooting guidance.

MDAA Hands-on Workshop offers step-by-step guided implementation with AWS experts. The workshop covers configuration management best practices, implementation patterns, hands-on labs with real-world scenarios, and cleanup instructions.

GitHub Repository and Documentation provide source code, module reference, and comprehensive documentation.

Organizations approach MDAA from different starting points. Some modernize existing data architectures, migrating from on-premises infrastructure or legacy cloud architectures. Others build new architectures for artificial intelligence and machine learning (AI/ML) initiatives or generative AI applications. Financial services organizations require PCI-DSS compliance from day one. Healthcare organizations need controls that can help support HIPAA. Each journey benefits from MDAA’s configuration-driven approach and embedded governance.

Conclusion

MDAA transforms data architecture development from months of manual coding to production-ready deployment. Configuration-driven infrastructure reduces development time by 40–60 percent while embedding governance from the start. The university system’s 95 percent reduction in time-to-value demonstrates the outcome: organizations deploy secure, compliant, governed data architectures in weeks rather than months.

Financial services organizations can deploy architectures to help them align with PCI-DSS compliance requirements using Lake Formation access controls, Glue Data Quality validation, SageMaker Unified Studio data discovery, comprehensive CloudTrail audit trails, and automated Macie data classification, all inherited from configuration rather than built manually.

Data architecture journeys need not follow six-month timelines with governance added incrementally. MDAA provides an alternative: describe the outcomes you want through YAML configuration, inherit pre-validated security controls, and deploy production-ready infrastructure with comprehensive governance from initial deployment.

Security and compliance is a shared responsibility between AWS and the customer. For more information, see the AWS Shared Responsibility Model.

Need help or have questions? Contact AWS ProServe for personalized guidance on selecting the right package and deployment strategy for your organization.


About the author

Sudeshna Dash

Sudeshna Dash

Sudeshna is a Data Scientist at AWS Professional Services based in Berlin, Germany. She specializes in data architecture, generative AI, and agentic AI systems on AWS. Sudeshna is a contributor to the Modern Data Architecture Accelerator (MDAA) open-source project and helps customers design and deploy governed, production-ready data and AI/ML architectures on AWS.

John Reynolds

John Reynolds is a Principal Engineer with AWS Professional Services based in Seattle, Washington. He leads the architecture and development of Modern Data Architecture Accelerator (MDAA), focusing on turning proven delivery patterns into reusable, production-ready foundations that customers can adopt and extend at scale.

Access Amazon S3 data files directly using AWS Lake Formation permissions

Post Syndicated from Aarthi Srinivasan original https://aws.amazon.com/blogs/big-data/access-amazon-s3-data-files-directly-using-aws-lake-formation-permissions/

Data scientists and ML engineers often need to access raw data files in Amazon Simple Storage Service (Amazon S3) for machine learning training, data exploration, and generative AI workflows. However, when table-level access is governed by AWS Lake Formation, accessing the underlying S3 files has required maintaining separate permission mechanisms. S3 bucket policies or AWS Identity and Access Management (IAM) role policies create operational overhead and risk of permission drift.

Lake Formation now supports direct access to S3 data file locations for tables whose permissions it manages. Previously, data scientists with Lake Formation permissions on AWS Glue Data Catalog tables could query them using spark.sql(). Now, they can also read and write the underlying S3 data files using spark.read.parquet() or spark.read.csv() from Amazon EMR Spark jobs, Amazon SageMaker Unified Studio notebooks with EMR compute, and custom applications. All access is governed by the same Lake Formation permissions.

This capability is powered by the new GetTemporaryDataLocationCredentials() API, which vends temporary credentials scoped to registered S3 locations when callers have appropriate Lake Formation permissions on the corresponding Data Catalog tables. This eliminates the need to manage separate S3 bucket policies for file-level access while maintaining fine-grained access control in Lake Formation for table-based access. It enables your data scientists to explore S3 datasets securely, accelerate machine learning pipelines, and build generative AI workflows without compromising governance.

In this post, we demonstrate reading from and writing to Lake Formation-managed S3 locations using Apache Spark jobs from EMR. Lake Formation credential vending for S3 location access is available in EMR release label 7.13 and later, Boto3 1.42.29 and later, AWS Java SDK 2.41.32 and later, and AWS Command Line Interface (AWS CLI) version 2.33.1 and later.

Key use cases for Lake Formation permissions to S3 locations

  • Unified permissions for Analytics and Machine Learning pipelines – Data scientists can access both structured tables through SQL queries and underlying data files through programmatic APIs for machine learning and AI workloads. They are empowered to use tools of their choice – for example, use Amazon Athena for SQL analytics with the table names while read and write to the underlying files in their SageMaker notebook or Spark application with spark.read.parquet(“s3://bucket/database_path/table_files/).
  • Enable AI ready data lakes – Machine learning pipelines can read training data directly from governed data lakes. Generative AI applications can access foundation model training datasets, and data exploration workflows to use native file APIs while maintaining centralized governance and compliance.
  • Reduced operational complexity – Operations teams don’t need to maintain separate permission policies – one in Lake Formation for table access and another in S3 bucket policies or AWS Identity and Access Management (IAM) roles for file access. This reduces the risk of permission mismatches and avoids inconsistent access control.
  • Unified audit capability – Auditors do not need to examine multiple log sources, such as S3 Access Logs, AWS CloudTrail events from different services, to understand who accessed what data and when. With this feature, you get a unified CloudTrail audit trail showing both table access through SQL engines and file access through direct APIs, with each access event linked to the Lake Formation permission grant.

What customers are saying

“Through our close collaboration with AWS, Lake Formation’s new S3 location-based permissions have transformed how we manage data governance at Intuit. By unifying two separate access mechanisms for the same data into one unified permission model, we’ve dramatically reduced complexity and streamlined our auditing process. This is exactly the kind of simplification that lets our teams move faster without compromising security, ensuring we maintain the strict compliance and governance standards our regulators expect.”

— Tapan Upadhyay, Group Engineering Manager, Intuit

Lake Formation Credential Vending Plugin for AWS SDK v2 for Java

Lake Formation has made available a specialized library AWS Lake Formation Credential Vending Plugin for AWS SDK V2 for Java. The Java plugin intercepts S3 requests for data, checks Lake Formation permissions for the requested location, and provides temporary scoped credentials to the client if permissions are granted in Lake Formation. If the S3 location access permissions are not managed by Lake Formation, the plugin checks for access in Amazon S3 Access Grants and lastly falls back to IAM permissions. The plugin is supported independently of Spark and comes as an enhancement to EMR Spark Full Table Access (FTA) mode, starting in EMR 7.13 and later. The plugin is integrated at the S3A level. Therefore, any client of S3A can enable it by setting the S3A configurations, in addition to the EMR Lake Formation Full Table Access (FTA) configuration as follows:

fs.s3a.lakeformation.access.grants.enabled = true
fs.s3a.lakeformation.access.grants.fallback.to.iam = true

With the Java plugin, you can enable governance for data lake resources in your custom applications with Lake Formation permissions – managing both fine grained access for users requiring restricted access on Data Catalog tables while providing direct S3 object level access to use-cases that require them.

Note: (1) The principal that will be accessing direct S3 locations of the tables will require full table access. That is, Lake Formation SELECT permission on all columns and rows of the table is required. (2) The Spark cluster needs FTA configuration. (3) Currently, Apache Iceberg table format is not supported with this plugin.

Solution overview

A financial services company runs daily ETL jobs using Spark in EMR. They process raw transaction records in S3 and store the processed records in another S3 location. The transformed Parquet data is registered with Lake Formation and cataloged as a table in Data Catalog. The ETL job will have direct IAM access to the raw data location, while it uses Lake Formation permissions to write to and read from the curated table location. Downstream, a data-analyst role will query the curated table, with restricted column access. The solution is shown in Figure 1.

Figure 1 – Architecture shows EMR Spark writing curated records to the S3 location of a table using Lake Formation permissions while Data-Analyst queries the same table with Lake Formation fine grained access control in Athena.

Architecture diagram showing EMR Spark writing curated records to the S3 location of a table using Lake Formation permissions while Data-Analyst queries the same table with Lake Formation fine-grained access control in Athena

Prerequisites

To get started exploring this feature, we recommend you have the following setup.

Solution walkthrough

First, we will get the setup ready with S3, sample database, table, and data. We will add a raw data set to S3 location, create a table with parquet data in another S3 location that represents the curated dataset for further downstream consumption. We will register the table data location with Lake Formation and grant permissions for the EMR run time role and Data-Analyst role.

Your S3 bucket will have the following structure.

Raw data – s3://<your-bucket-name>/raw/transactions/dt=2024-03-21/

Process data for table – s3://<your-bucket-name>/processed/transactions/

Spark script – s3://<your-bucket-name>/scripts/

Logs for the EMR cluster – s3://<your-bucket-name>/logs/

Step 1 – Create a parquet table in Data Catalog

From the Athena console query editor, create a table in Data Catalog.

-- Create a database
CREATE DATABASE finance_db;

-- Create an external table pointing to the S3 location
CREATE EXTERNAL TABLE IF NOT EXISTS finance_db.transactions_processed (
    transaction_id STRING,
    merchant_name STRING,
    amount DECIMAL(18,2),
    currency STRING,
    account_number STRING,
    card_type STRING,
    status STRING,
    region STRING
)
PARTITIONED BY (transaction_date DATE)
STORED AS PARQUET
LOCATION 's3:///processed/transactions/'
TBLPROPERTIES (
    'parquet.compress'='SNAPPY'
);

Step 2 – Register S3 location and grant table permission to IAM roles in Lake Formation

2.1 Register the table data location s3://<your-bucket-name>/processed/transactions/ with Lake Formation in Lake Formation mode using the custom S3 registration IAM role. For details on how to register locations with Lake Formation, refer Adding an Amazon S3 location to your data lake.

2.2 Grant DESCRIBE permission on the database finance_db and ALL permission on the table transactions_processed to your EMR runtime role.

2.3 Grant Data location permission to EMR runtime role on the curated table’s location. This is to allow writing to that location.

2.4 Grant DESCRIBE permission on the database finance_db and SELECT permission on the table transactions_processed to your Data-Analyst role. Exclude the columns transaction_id and account_number while granting SELECT permissions on the table to the Data-Analyst role.

For details on how to grant Lake Formation permissions, refer Granting database permissions using the named resource method; Granting table permissions using the named resource method and Granting data location permissions.

Step 3 – Run ETL script in EMR

3.1 Download the script bdb-5860-script.py.

3.2 Edit the S3 bucket name placeholder in the script (RAW_PATH and TABLE_PATH) to your resource names and upload to your S3 path s3://<your-bucket-name>/scripts/.

3.3 Make sure your EMR runtime role has access to the script location in its IAM policy permissions.

3.4 Submit and run the script as a step to the EMR cluster, following instructions at Add a Spark step.

What does the script do?

It populates raw records of transaction data into a Spark data frame, writes to the raw data bucket location using IAM permissions on the EMR runtime role. We apply some transformations and write directly to the S3 location of the table that is registered with Lake Formation, from the data frame using Spark’s native Parquet writer.

The following figure shows the stdout of the step.

EMR step stdout showing successful Spark job execution with data written to the Lake Formation-managed S3 location

The Java plugin integrated into EMR 7.13 automatically handles the access for the table’s data location registered with Lake Formation, so you don’t need to manually call the GetTemporaryDataLocationCredentials() API. In this example, the table data location s3://<your-bucket-name>/processed/transactions/ is registered with Lake Formation, for which EMR runtime role is granted ALL permissions. The direct S3 location access support by Lake Formation allows reading and writing to the location directly using Spark data frame.

Step 4 – Run query as Data-Analyst using Athena

Log in as the Data-Analyst role to the Athena console. Run a select query on the table as follows.

SELECT * FROM finance_db.transactions_processed WHERE status = 'DECLINED' AND transaction_date=DATE '2024-03-21';

The Data-Analyst role should see all but two columns of the table.

Athena query results showing the Data-Analyst role can access all columns except transaction_id and account_number

With these steps complete, we’ve read from and written to direct S3 locations using Spark data frames with the syntax s3://bucketname/prefix/, and accessed the same data using database_name.table_name syntax with Lake Formation permissions. This shows fine-grained access at table level and coarse-grained access at the file path level.

Clean up

To avoid incurring costs, clean up the resources you created for this post.

  1. Delete the Data Catalog database and tables. This removes the related Lake Formation permissions too. Remove the S3 bucket registration from Lake Formation.
  2. Delete the data files, logs, and the PySpark script of this post from your S3 bucket.
  3. Terminate the EMR cluster.

Conclusion

In this post, we showed how to use Lake Formation’s direct S3 location access to read and write data files using Spark data frames from Amazon EMR, while maintaining unified governance through Lake Formation permissions. We walked through the GetTemporaryDataLocationCredentials() API and the AWS Lake Formation Credential Vending Plugin for AWS SDK v2 for Java, which is integrated into EMR release labels 7.13 and later.

This capability unifies permission management for both fine-grained table-based access and direct S3 file path access in Lake Formation. Your data scientists can now use spark.read.parquet() and spark.write alongside spark.sql(), governed by the same permissions, audited in the same CloudTrail logs, and managed from a single console.

To get started, launch an EMR 7.13 cluster and start exploring the feature. Here are some additional resources:

Acknowledgements: We would like to thank all the team members who worked to launch this feature successfully – Rajas Bhate, Akhil Yendluri, Kunal Parikh, Sharda Khubchandani, Dhananjay Badaya, Santhosh Padmanabhan, Nitin Agrawal and Sandeep Adwankar.


About the authors

Aarthi Srinivasan

Aarthi Srinivasan

Aarthi is a Senior Big Data Architect at Amazon Web Services (AWS). She works with AWS customers and partners to architect data lake solutions, enhance product features, and establish best practices for data governance.

Archana Inapudi

Archana Inapudi

Archana is a Senior Solutions Architect at Amazon Web Services (AWS). She works with strategic enterprise customers to drive cloud data modernization, architect data lake and analytics solutions, and establish best practices for data governance and security. With over 15 years of experience in cloud, data engineering, and AI/ML, Archana is passionate about using technology to accelerate growth and deliver business outcomes.

Srinivasan Krishnasamy

Srinivasan Krishnasamy

Srinivasan is a Principal Delivery Consultant at AWS with 25+ years of experience architecting data and analytics solutions at scale. He partners with enterprise customers to modernize data platforms, build robust data governance frameworks, and drive measurable business outcomes on AWS, using the full spectrum of data engineering, AI/ML, and generative AI. Outside of work, he enjoys hiking, swimming, and gardening.

Anandkumar Kaliaperumal

Anandkumar Kaliaperumal

Anandkumar is a Senior Delivery Consultant at AWS, bringing over 23 years of deep expertise in data and analytics. A specialist in architecting scalable data analytics, AI/ML, and generative AI solutions, he thrives on tackling complex data challenges spanning data engineering, analytics, machine learning, and generative AI workloads.

Mitali Sheth

Mitali Sheth

Mitali is a Streaming Data Engineer at Amazon Web Services (AWS) Professional Services. She works with strategic software customers to architect real-time analytics solutions, design event-driven architectures, and modernize streaming infrastructure using Amazon MSK, Amazon Managed Flink, AWS Glue, and AWS Lake Formation. She holds an M.S. in Computer Science from the University of Florida.

Securing client confidentiality at scale: Automated data discovery and governed analytics for legal workloads

Post Syndicated from Rohan Kamat original https://aws.amazon.com/blogs/big-data/securing-client-confidentiality-at-scale-automated-data-discovery-and-governed-analytics-for-legal-workloads/

Automating data security and analytics for legal documents presents a unique challenge when your legal team stores documents with strong access controls, organized by client and matter, encrypted at rest, and governed by well-defined policies. But what happens when you want to run analytics across those repositories? The typical path is extracting content into separate data pipelines or third-party tools, which fragments your governance model and introduces new risks. Law firms and corporate legal departments operate under distinct obligations that make data governance non-negotiable. Attorney-client privilege, work product doctrine, and professional conduct rules impose strict duties around how client information is handled, accessed, and disclosed. Governance failure in this context isn’t just a compliance gap, it can result in privilege waiver, disqualification from representation, or disciplinary action.

Legal professionals use ethical walls, also called information barriers, as structural safeguards that prevent the flow of confidential information between teams within a firm that represent adverse or potentially conflicting interests. Professional conduct rules mandate these barriers, and failure to maintain them can result in firm disqualification, malpractice liability, or regulatory sanctions.

Privilege boundaries are equally critical. Attorney-client privilege and work product protection apply only when you properly control access to the underlying material. If you expose privileged documents or metadata about their contents to unauthorized individuals, you risk losing your privilege protection. When organizations fail to maintain reasonable controls over privileged material, courts might find that they have waived their privilege. You should therefore actively manage your access governance, not only as a security concern but as a legal preservation requirement.When you extract content into separate analytics systems or grant broader access than your matter structures support, you create pressure on both protections. You gain visibility but lose confidence in your controls.

In this post, we show you a reference architecture that automates sensitive data discovery across legal document repositories on Amazon Web Services (AWS), demonstrate how to capture structured findings as a compliance dataset, and guide you through building a governed analytics workspace that maintains your security boundaries. You walk away with a practical model for building security and analytics into the same lifecycle, without moving documents outside their system of record.

Analytics shouldn’t weaken governance

Most legal organizations have invested heavily in securing their document repositories. You store documents in structured storage, organized by client and matter. You access controls map to matter boundaries (the organizational and access structures that separate one client engagement from another). You establish retention and hold policies.The difficulty starts when teams want to analyze what’s inside those repositories. Running analytics typically means copying content into a separate system, standing up a new data pipeline, or granting broader access than existing matter structures support. Each of these steps introduces governance gaps. Manual reporting fills some of the void, but it doesn’t scale and can’t provide continuous visibility. What’s missing is a model where security controls and analytics reinforce each other, where the act of discovering sensitive data also produces the dataset that you use for reporting, and where governance applies once and carries through every downstream operation.

Automation addresses this by combining continuous sensitive data discovery with governed analytics, built on discovery metadata rather than document copies. This automated approach delivers four key advantages:

  • No document movement. Your files stay in their system of record. Analytics runs against structured discovery metadata, not document content, so governance boundaries remain intact.
  • Continuous discovery instead of manual scanning. Automated classification identifies regulated and sensitive information on an ongoing basis, replacing periodic manual reviews with on demand visibility.
  • Unified governance. You define matter-aligned access policies once, and they carry through from document storage to findings analytics and compliance reporting.
  • Built-in audit readiness. A durable record of discovery findings and remediation actions accumulates automatically over time, giving you structured evidence for client reviews and regulatory inquiries.

Reference Architecture

The following architecture shows how continuous discovery, governance, and compliance operations can work together without copying legal documents into analytics systems.

This reference architecture illustrates how law firms and corporate legal departments can automate sensitive data discovery and compliance analytics on AWS without moving documents outside their system of record

Architecture walkthrough

Store and protect documents in Amazon Simple Storage Service (Amazon S3)

Store your legal documents in Amazon S3, which serves as the system of record for document content. Align your buckets and prefixes to client and matter structures so that access controls map directly to matter boundaries. Where your retention or legal hold requirements demand it, apply S3 Object Lock to enforce immutability. You can encrypt your data using AWS Key Management Service (AWS KMS), which gives you centralized control over encryption keys and policies.

Discover and classify sensitive data with Amazon Macie

You will configure Amazon Macie to continuously analyze your document repositories. Macie identifies regulated information such as personally identifiable information (PII), financial data, and other sensitive content and produces structured findings that describe what Macie identified and where it exists. This provides ongoing visibility into data exposure without requiring document movement or manual scanning.

Catalog and govern findings with AWS Glue and AWS Lake Formation

You will use AWS Glue to catalog the findings dataset and maintain its schema so it stays query-ready. Apply AWS Lake Formation tag-based policies to govern access, aligning tags to client, matter, and confidentiality tier. This approach enforces ethical walls and least-privilege access consistently across analytics and reporting activities.

AI-powered chat agent using Amazon Quick Suite

You can create custom chat agents to tailor conversational interfaces for specific legal business needs. These agents can be configured with legal-specific knowledge bases, connected to relevant document repositories, and customized with instructions appropriate for legal workflows. You can use this chat agent to interact with your legal documents through natural language conversation for capabilities like:

  • E-Discovery:Search and analyze large volumes of legal documents to quickly find relevant information across your document repository.
  • Contract Analysis:Review contracts and automatically extract key terms, clauses, and obligations to streamline your contract review process.

The chat agent can help you navigate complex document sets through conversational queries, making legal research and document review more efficient and accessible.

Analyze and report with Amazon Quick Sight

You will use Amazon Quick as your compliance operations workspace. Quick provides a unified environment where your teams can query findings, generate dashboards, track remediation actions, and produce audit-ready reports. The agentic AI capabilities of Amazon Quick can autonomously build analyses, surface anomalies across matters, generate executive summaries for client reviews, and proactively recommend remediation priorities based on finding severity and trends. Combined with built-in data stories for automated narrative generation and pixel-perfect paginated reports for regulatory submissions, Quick reduces the time from discovery to action while keeping your teams within a governed interface aligned to matter-based permissions. Rather than switching between separate visualization, workflow, and reporting tools, your legal and compliance teams can review findings, manage response activities, and collaborate all within a single workspace that respects ethical walls and privilege boundaries.

Escalate high-severity findings

For high-severity findings that demand immediate attention, route alerts through AWS Security Hub or Amazon Simple Notification Service (Amazon SNS) to trigger escalation workflows. This connects visibility directly to action when your teams identify sensitive data risks.

Why this approach works for legal

Documents stay where they belong. Your files remain in Amazon S3, aligned to client and matter boundaries. No content moves into separate analytics pipelines.Ethical walls remain intact. Because analytics is built on discovery findings and not document copies, you can govern access to findings using the same matter-aligned controls that apply to documents. Compliance and security teams gain visibility without expanding document access.Discovery runs continuously, not periodically. Rather than scheduling quarterly or annual scans, you maintain a current view of sensitive data across your repositories.

Governance applies once and carries through. Lake Formation tag-based policies govern findings access at the catalog level. You define your matter and confidentiality mappings once, and they carry through to every dashboard, query, and report.Audit readiness is built in. Instead of assembling reports manually before a client review or regulatory inquiry, you maintain a historical record of discovery findings and remediation actions. You can demonstrate your posture over time with consistent, structured evidence.

Security and analytics reinforce each other. Your analytics capability is built on top of your security controls, not alongside them. Strengthening one strengthens the other.

Cost considerations

The primary cost drivers for this architecture include:

  • Amazon Macie: You pay based on the number of S3 buckets evaluated and the volume of data inspected for sensitive data discovery. Review Amazon Macie pricing for current rates.
  • Amazon S3: Storage costs for both your document repositories and the compliance intelligence bucket. Consider S3 lifecycle policies to tier older findings into lower-cost storage classes.
  • AWS Glue and AWS Lake Formation: Charges for crawlers and catalog storage. For most implementations, these costs are modest.
  • Amazon QuickSight: Per-user pricing based on the edition that you select (Standard or Enterprise). Enterprise edition supports row-level and column-level security, which aligns well with matter-based governance.
  • Amazon EventBridge, AWS Security Hub, and Amazon SNS: Charges based on event volume and notifications delivered. For findings-based workflows, these costs are generally low.

Use the AWS Pricing Calculator to estimate costs based on your repository size, user count, and discovery frequency.

Getting started

Start by identifying a representative set of document repositories in Amazon S3. We recommend that you start with two or three matters that span different practice areas and confidentiality tiers.

  1. Turn on Amazon Macie for those repositories and configure automated sensitive data discovery.
  2. Catalog the findings dataset with AWS Glue and apply Lake Formation tag-based access policies aligned to your matter structure.
  3. Build your first Amazon Quick Sight dashboard to visualize findings by matter, sensitivity type, and severity.
  4. Define escalation rules in AWS Security Hub or Amazon SNS for high-severity findings.

After you validate this workflow against your initial repositories, expand gradually. Add more repositories to Macie discovery. Refine your governance tags to reflect practice areas and confidentiality tiers. Extend your dashboards from basic posture visibility to trend analysis and remediation tracking.The goal isn’t to build a comprehensive analytics solution all at once. Start with a secure foundation where discovery findings, governance, and reporting operate together in a way that aligns with your legal workflows, and then expand from there.

Conclusion

You don’t have to choose between protecting client data and understanding it. By building analytics on top of governed discovery findings and using a unified compliance workspace, you gain visibility into your data posture without weakening confidentiality boundaries.This approach brings security, governance, and analytics together in a way that reflects how legal work is actually structured. It provides continuous visibility, supports audit readiness, and delivers insight without requiring documents to move outside their system of record.

Next steps

Review the Amazon Macie User Guide to understand sensitive data discovery configuration options and Amazon Quick Sight documentation to evaluate dashboard and row-level security capabilities.

Contact your AWS account team to discuss implementation support for legal and compliance workloads.


About the authors

Photo of Author - Rohan Kamat

Rohan Kamat

Rohan Kamat is a Solutions Architecture Leader within HCLS with extensive experience in cloud architecture, cybersecurity, Identity and Access Management, and enterprise networking. Rohan focuses on helping architects build both depth in cloud technologies and strength in executive communication, making sure they can confidently guide organizations through business and technical transformation. Outside of his professional work, Rohan enjoys time with his family, organizing community cricket events, and exploring fitness and wellness activities like pickleball and ping pong. He also enjoys planning travel experiences that bring people together and create lasting shared memories.

Photo of Author- Miguel Lopez Luis

Miguel Lopez Luis

Miguel Lopez Luis is an AWS Solutions Architect who works with small and medium businesses across the United States. He graduated with a Bachelor’s degree in Cybersecurity from Bellevue University in Nebraska and is a member of the Omega Nu Lambda Honor Society. Leveraging his extensive expertise in business management, Miguel is passionate about planning strategic initiatives, leading cross-functional teams, and mentoring others. In his personal time, he enjoys activities that involve travel, sports, and cooking.

Photo of Author - Pranali Khose

Pranali Khose

Pranali Khose is an AWS Solutions Architect based in Seattle. She works directly with small and medium business (SMB) customers across the United States, to design and implement cloud solutions that address their unique business challenges and accelerate digital transformation. Pranali holds a Master of Science in Computer Science from the University of Texas at Arlington.

How Twilio secured their multi-engine query platform with AWS Lake Formation

Post Syndicated from Aakash Pradeep, Venkatram Bondugula original https://aws.amazon.com/blogs/big-data/how-twilio-secured-their-multi-engine-query-platform-with-aws-lake-formation/

This is a guest post by Aakash Pradeep, Principal Software Engineer, and Venkatram Bondugula, Software Engineer at Twilio, in partnership with AWS.

Twilio is a cloud communications platform that provides programmable APIs and tools for developers to easily integrate voice, messaging, email, video, and other communication features into their applications and customer engagement workflows.

In this blog series we discuss how we built a multi-engine query platform at Twilio. The first part introduces the use case that led us to build a new platform and why we selected Amazon Athena alongside our open-source Presto implementation. This second part discusses how Twilio’s query infrastructure platform integrates with AWS Lake Formation to provide fine-grained access control to all their data.

At Twilio, we faced critical challenges in managing our multi-engine query platform across a complex data mesh architecture spanning multiple AWS accounts and Lines of Business. We needed a unified permissions model that could work consistently across different query engines like OSS Presto and Amazon Athena, eliminating the fragmented authentication experiences in our infrastructure. The growing demand for secure cross-account data sharing required moving beyond manual, multi-step provisioning processes that depended heavily on human intervention. Additionally, Twilio’s compliance and data stewardship requirements demanded fine-grained access controls at row, column, and cell levels, necessitating a scalable and flexible approach to permission management. By adopting the AWS Glue Data Catalog as our managed metastore and AWS Lake Formation for governance, we implemented Tag-Based Access Control (LF-TBAC) to simplify access management, enabled data sharing through automated workflows, and established a centralized governance framework that provided uniform permissions management across all AWS services.

Transitioning to a managed metastore and governance solutions

We discussed in part 1, how we were looking to move to managed services to alleviate us of the burden of managing the underlying infrastructure of a query platform. Along with our decision to adopt Amazon Athena, we also began to evaluate the adoption of Amazon EMR Serverless for our Spark workloads, which made us aware of the fact that we needed to migrate to a managed solution for our Apache Hive metastore.

We selected the AWS Glue Data Catalog as our managed metastore repository to support our enterprise-wide data mesh architecture. For managing permissions to the Data Catalog assets, we chose AWS Lake Formation, a service that enables data governance and security at scale using familiar database-like permissions. Lake Formation provides a unified permissions model as well as support for enabling data mesh architecture that we were seeking.

Lake Formation’s support for row, column, and cell-level access controls provides the fine-grained access control (FGAC) capabilities required by our compliance and data stewardship policies. Additionally, Lake Formation’s tag-based access control (LF-TBAC) feature allows us to define FGAC permissions based on tags attached to the Data Catalog resources, enabling flexible and scalable permission management.

Integrating Odin with AWS Lake Formation

Odin, our Presto-based gateway, serves as a central hub for query processing, managing authentication, routing, and the complete workflow throughout a query’s lifecycle. As the primary interface, Odin enables users to connect through JDBC or APIs from various BI tools, SQL IDEs, and other applications.

Beyond its core routing capabilities, Odin utilizes local caches implemented using Google’s Guava caching library to optimize performance across the platform. Guava delivers efficient in-memory caching for Java applications by storing data locally within the application instance, resulting in significantly faster retrieval times. Odin employs multiple Guava caching layers across various modules to ensure optimal response times for frequently accessed data and metadata.

Building on this performance foundation, Odin implements authentication and authorization layers to ensure secure and controlled access to data across multiple query engines. These security components work together to verify user identities and enforce data access policies, providing a unified security framework that abstracts away the complexities of individual engine implementations while maintaining strict governance standards.

The authentication layer

Different query engines like OSS Presto and Amazon Athena each implement their own authentication mechanisms. To create a consistent user experience, Odin provides a unified authentication layer that shields users from these underlying differences. Currently, Odin’s pluggable authentication system supports LDAP integration, with plans to expand this capability to include Okta authentication using IAM Identity center in the future.

The authorization layer

For data consumers using AWS Analytics services such as AWS Glue, Amazon EMR, and Athena through an IAM federated role-based access, AWS Lake Formation provided critical authorization capabilities for data governance through their existing integrations. However, we needed to extend its capabilities to integrate with OSS Presto. Additionally, our users for the query infrastructure platform were not mapped to an IAM user so would need to build a custom authorization layer in Odin to verify permissions and integrate with Lake Formation. Our challenge was creating a consistent way to control data access across all our query engines.

When a user runs a query, Odin’s authorization layer checks three key pieces of information:

  • Table details: which database and table the query is accessing
  • User permissions: what data tags the user has access to
  • Resource tags: what security tags are attached to the requested table

We store user permissions in Amazon DynamoDB, which allows us to quickly look up what each user can access. By matching the user’s tags with the table’s Lake Formation tags, we can determine if the query should be allowed. To keep things fast, we cache this information temporarily, allowing us to expedite authorization for recent requests.

How the authorization works:

  1. Initial check: First, we see if this user recently ran a similar successful query (within the last 5 minutes).
  2. Gather information: We collect the table details, user permissions, and security tags—first checking our cache, then fetching from AWS Glue Data Catalog and Lake Formation if needed.
  3. Match permissions: We compare the user’s access tags stored in a DynamoDB table against the table’s security tags in Lake Formation.
  4. Make decision: If the user’s permissions match what’s required for their query action (like SELECT or INSERT), access is granted.

This approach allows us to make use of Lake Formation tag-based access control while keeping our authorization logic separate from the individual query engines. By using smart caching and efficient lookups, we can verify permissions in just milliseconds.

Building a data mesh

At Twilio, we have multiple line of business (LoBs) each managing their own data platform infrastructure. The individual platforms are spread across multiple AWS accounts, and primarily store data on Amazon S3 in variety of open table formats, such as Apache Hudi, Apache Iceberg, and Delta Lake. Each platform independently supports analytics and machine learning use cases, however, there was a growing need for secure sharing of data across LoBs. Additionally, we needed to enable self-service discovery and provisioning of access to the data with a centralized governance framework.

Data consumers bring their own AWS accounts and choice of tools, which include not only AWS services such as Amazon Athena, AWS Glue ETL jobs (Spark), and Amazon EMR, but also AWS partner solutions. To improve the process of access fulfillment, data auditability and lowering the operational overhead involved, we needed an automated framework in place that had minimal human intervention and oversight.

Implementing a data subscription workflow

Previously, consumers requiring access to specific data sets would need to go through multiple steps to secure access, which involved several dependencies and manual actions. To simplify this process and provide a self-service capability, we decided to build a custom integration solution between ServiceNow and AWS Lake Formation. At Twilio, ServiceNow is used extensively to automate workflows and build custom applications to connect disparate systems and improve operational efficiency.

We automated key parts of the data access process using Twilio’s standard tools: Git for version control, Terraform for infrastructure management, and custom scripts to execute the necessary AWS actions.

We automated three main use cases:

1. Sharing data between accounts

When one team needs to share data with another team or with our central governance account, the process starts with a Git pull request (PR). This triggers our custom Lake Formation automation tool, which:

  • Connects to the source AWS account with admin permissions
  • Sets up data sharing using the security tags (LF-Tags) specified in a YAML configuration file
  • Completes the share using AWS Resource Access Manager (RAM)
  • Creates resource links in the target account so the data appears in their catalog
  • Updates ServiceNow with the newly shared database and table information

2. Granting permissions to user roles

When users request access to data, our automation tool grants tag-based permissions directly to their IAM roles in Lake Formation. This happens after approval of either a Git PR or ServiceNow ticket.

3. Granting access to individual users

For individual user access requests:

  • Users submit a request in ServiceNow for specific tables
  • After approval, ServiceNow calls our internal API that checks relevant Lake Formation tags
  • The request is validated and sent to an Amazon Simple Queue Service (Amazon SQS) queue
  • A consumer service processes the request, updates the user’s permissions in our DynamoDB table (which Odin uses for authorization checks), and includes retry logic for reliability
  • Once complete, the service updates the ServiceNow ticket to notify the user

The overall subscription and authorization flow is as shown in the diagram below:

Diagram of Twilio's AWS data query platform showing user access requests flowing through ServiceNow and LF-Tag validation before queries reach Amazon Athena via Odin EC2 instances.

  1. Users submit a request in ServiceNow for access to a database, table, or LF-Tag
  2. The system retrieves the relevant LF-Tags from Lake Formation through our API integration
  3. Upon approval, the automation procedure adds the user to the User-To-Tag DynamoDB table, grants IAM role permissions in Lake Formation, and sets up cross-account sharing via RAM as needed
  4. Users submit SQL query to the Odin presto gateway
  5. Odin authorizes the user through LDAP
  6. Odin parsers the SQL query to identify the tables involved and the action being performed (SELECT, DDL, and more)
  7. Odin validates permissions using the User to LF-Tag mapping and Lake formation grants to authorize the SQL query based on granted permissions
  8. If authorized, Odin routes the query to Amazon Athena or Presto

Using standardized tools and processes to provide self-service capabilities to the users helped us scale the governance framework and support broader use cases. Important capabilities in Lake Formation, such as Tag-based access control (TBAC) and cross-account sharing of data, simplified developing automations and our overall approach to governance.

Lessons learned- Cache is king

“By adopting AWS Glue Data Catalog as our managed metastore and AWS Lake Formation for Tag-Based Access Control, we simplified access management and enabled data sharing by reducing auth overhead to just 6-10 milliseconds through caching and targeted scaling.”

As Odin began handling queries at scale, we encountered performance bottlenecks in our customized authorization process as we had to retrieve information from multiple services, particularly with complex queries spanning multiple tables. The authorization checks involved in the performance bottleneck frequently caused query timeouts which impacted overall system reliability. The root of the problem lay in our sequential authorization workflow: our system first had to parse each query to identify all tables requiring identity verification, then make separate API calls to the AWS Glue Data Catalog and Lake Formation for each table’s permissions. It became clear that we needed to optimize this authentication process to reduce response times and improve the overall query experience.

We also recognized there were different caching needs between our POST operations and GET/DELETE HTTP calls, so we decided to separate them into two different Application Load Balancer (ALB) target groups. For POST requests, which required Lake Formation authentication, we found that concentrating traffic through just 2-3 target instances distributed across multiple Availability Zones (AZ) was more efficient. This approach allowed authentication information to be effectively cached locally on these dedicated instances, dramatically reducing the volume of API calls to the Lake Formation service.

GET and DELETE requests follow a more simplified workflow. Since users have already completed initial authorization, there is no need to continue to perform authorization checks. Although they follow a simpler workflow, these requests have much higher volume with requests numbering into the 10s of millions per hour. Due to this scale, we opted to implement horizontal scaling to scale the target ALB to 10 Amazon EC2 instances to fetch the query history from the DynamoDB table. These EC2 instances make use of local LRU caching with a 5-minute expiration policy for authentication data.

By implementing authentication caching and adopting specialized approaches for different HTTP request types with targeted scaling groups, we successfully reduced Odin’s overall overhead to a maximum of 6-10 milliseconds for both authentication and authorization.

Conclusion and what’s next

In this post, we explored how we enhanced Odin, our unified multi-engine query platform, with authentication and authorization capabilities using AWS Lake Formation and a custom authorization workflow. By using AWS services including Lake Formation, AWS Glue Data Catalog, and Amazon DynamoDB alongside Twilio’s existing infrastructure, we created a scalable self-service governance framework that streamlines user access management, simplifies auditing, and enables seamless data sharing across our complex cloud environment. With this workflow automation, we eliminated operational overhead while building a secure, robust platform that serves as the foundation for Twilio’s data mesh architecture.

Going forward, we are focusing on strengthening our authentication and authorization framework by enabling trusted federation with an identity provider(IdP) through AWS IAM Identity Center, which integrates directly with Lake Formation. Using Trusted Identity Propagation capabilities supported by IAM IDC will allow us to establish a consistent governance flow based on a user identity and will allow us to unlock the full capabilities of AWS Lake Formation such as fine-grained access control with data filters.

To learn more and get started with building with AWS Lake Formation, see Getting started with Lake Formation, and How to build a data mesh architecture at scale using AWS Lake Formation tag-based access control.


About the authors

Aakash Pradeep

Aakash Pradeep

Aakash is a Principal Software Engineer with over 15 years of experience across ingestion, compute, storage, and query platforms. Aakash is a PrestoCon speaker, holds multiple patents in real-time analytics, and is passionate about building high-performance distributed systems.

Venkatram Bondugula

Venkatram Bondugula

Venkatram is a seasoned backend engineer with over a decade of experience specializing in the design and development of scalable data platforms for big data and distributed systems. With a strong background in backend architecture and data engineering, he has built and optimized high-performance systems that power data-driven decision-making at scale.

Aneesh Chandra PN

Aneesh Chandra PN

Aneesh is a Principal Analytics Solutions Architect at AWS working with Strategic customers. He is passionate about using technology advancements to solve customers’ data challenges. He uses his strong expertise on analytics, distributed systems and open source frameworks to be a trusted technical advisor for AWS customers.

Amber Runnels

Amber Runnels

Amber is a Senior Analytics Specialist Solutions Architect at AWS specializing in big data and distributed systems. She helps customers optimize workloads in the AWS data ecosystem to achieve a scalable, performant, and cost-effective architecture. Aside from technology, she is passionate about exploring the many places and cultures this world has to offer, reading novels, and building terrariums.

Implement a data mesh pattern in Amazon SageMaker Catalog without changing applications

Post Syndicated from Paolo Romagnoli original https://aws.amazon.com/blogs/big-data/implement-a-data-mesh-pattern-with-amazon-sagemaker-catalog-without-making-changes-to-your-applications/

When creating a project in Amazon SageMaker Unified Studio, users select a project profile to define resources and tools to be provisioned in the project. These are used by Amazon SageMaker Catalog to implement a data mesh pattern. Some users don’t want to take advantage of resources provisioned along with the project for various reasons. For instance, they may want to avoid making changes to their existing applications and data products.

This post shows you how to implement a data mesh pattern by using Amazon SageMaker Catalog while keeping your current data repositories and consumer applications unchanged.

Solution overview

In this post, you will simulate a scenario based on data producer and data consumer that exists before Amazon SageMaker Catalog adoption. For this purpose, you will use a sample dataset to simulate existing data and simulate an existing application using an AWS Lambda function. You can apply the same solution to your real-life data and workloads.

The following diagram illustrates the solution architecture’s key configurations. In this architecture, the Amazon Simple Storage Service (Amazon S3) bucket and the AWS Glue Data Catalog in the producer account simulate the existing data repository. The Lambda function in the consumer account simulates the existing consumer application.

AWS cross-account data sharing via SageMaker & Lake Formation: Producer publishes to catalog, Consumer subscribes & accesses data

Here is a description of the key configurations highlighted in the architecture:

  1. As part of an Amazon SageMaker domain, create a producer project (associated to a producer account) and a consumer project (associated to a consumer account). Among other resources, a project AWS Identity and Access Management (IAM) role is created for each project in the associated account.
  2. In the producer account, use AWS Lake Formation to grant producer project’s IAM role permissions to access the existing data asset.
  3. Publish the data asset in the Amazon SageMaker Catalog from the producer project.
  4. Subscribe the data asset from the consumer project.
  5. In the consumer account, configure your Lambda function to assume consumer project’s IAM role to access the subscribed data asset.

The solution architecture is based on the following Amazon Web Services (AWS) services and features:

  • Amazon SageMaker Catalog offers you a way to discover, govern, and collaborate on data and AI securely.
  • Amazon SageMaker Unified Studio provides a single data and AI development environment to discover and build with your data. Amazon SageMaker Unified Studio projects provide collaborative boundaries for users to accomplish data and AI tasks.
  • The lakehouse architecture of Amazon SageMaker is fully compatible with Apache Iceberg. It unifies data across Amazon S3 data lakes, Amazon Redshift data warehouses, and third-party and federated data sources.
  • AWS Lake Formation, which you can use centrally to govern, secure, and share data for analytics and machine learning.
  • AWS Glue Data Catalog is a persistent metadata store for your data assets. It contains table definitions, job definitions, schemas, and other control information to help you manage your AWS Glue environment.
  • Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance.

Setting up resources

In this section, you will prepare the resources and configurations you need for this solution.

Three AWS accounts

To follow this solution, you need three AWS accounts, and it’s better if they’re part of the same organization in AWS Organizations:

  • Producer account – Hosts the data asset to be published
  • Consumer account – Hosts the application that consumes the data published from the producer account
  • Governance account – Where the Amazon SageMaker Unified Studio domain is configured

Each account must have an Amazon Virtual Private Cloud (Amazon VPC) with at least two private subnets in two different Availability Zones. For instruction, refer to Create a VPC plus other VPC resources. Make sure to create both VPCs in the same Region you plan to apply this solution.

A governance account is used for the sake of convenience, but it’s not strictly needed because Amazon SageMaker can be configured and managed in producer or consumer accounts.If you don’t have access to three accounts, you can still use this post to understand the key configurations required to implement a data mesh pattern with Amazon SageMaker Catalog while keeping your current data repositories and consumer applications unchanged.

Create a data repository in the producer account

First, create a sample dataset by following these instructions:

  1. Open a text editor.
  2. Paste the following text in a new file:
    name,stars
    	oak,3
    	maple,2
    	birch,3
    	willow,4
    	pine,5
    	mango,1
    	neem,2
    	banyan,5
    	eucalyptus,3
    	teak,2

  3. Save the file as trees.csv. This is your sample data file.

After you create the sample dataset, create an S3 bucket and an AWS Glue database in the producer account, which will act as the data repository.

Create the S3 bucket and upload the trees.csv file in the producer account:

  1. Access the S3 console in the producer account.
  2. Create an S3 bucket. For instructions, refer to Creating a general purpose bucket.
  3. Upload to the S3 bucket the trees.csv sample data file that you created. For instructions, refer to Uploading objects.

Create the AWS Glue database and table in the producer account:

  1. Access the Glue console in the producer account.
  2. In the navigation pane, under Data Catalog, choose Databases.
  3. Choose Add database.
  4. For Name, enter collections.
  5. For Description, enter This database contains collections of statistics for natural resources.
  6. Choose Create database.
  7. In the navigation pane, under Data Catalog, choose Tables.
  8. Choose Add table.
  9. In the table creation guided procedure, enter the following input for Step 1: Set table properties:
    1. For Name, enter trees.
    2. For Database, select collections.
    3. For Description, enter This table captures ratings data related to the characteristics of various tree species.
    4. For Table format, select Standard AWS Glue table (default).
    5. For Select the type of source, select S3.
    6. For Data location is specified in, select my account.
    7. For Include path, enter s3://<bucket-name>/<prefix>/ where <bucket-name> is the name of the S3 bucket you created earlier in this procedure and <prefix> is the optional prefix for the trees.csv file you uploaded.
    8. For Data format, select CSV.
    9. For Delimeter, select Comma (,).
  10. Choose Next.
  11. For Step 2: Choose or define schema, enter the following:
    1. For Schema, select Define or upload a schema.
    2. Choose Edit schema as JSON and enter the following schema in the pop-up:
      [
        {
          "Name": "name",
          "Type": "string",
          "Parameters": {}
        },
        {
          "Name": "stars",
          "Type": "string",
          "Parameters": {}
        }
      ]

    3. Choose Save.
    4. Choose Next.
    5. Choose Create.

Create a Lambda function in the consumer account

Create the Lambda function in the consumer account. This will simulate a data consumer application.First, in the consumer account create the IAM policy and the IAM role to be assigned to the Lambda function:

  1. Access the IAM console in the consumer account.
  2. Create an IAM policy and name it smus_consumer_athena_execution by using the following policy. Make sure to replace placeholders <AWS_Region> and <AWS_account_ID_number> with your Region and consumer account ID number. You will replace the <workgroup_id> placeholder later. For IAM policy creation instructions, refer to Create IAM policies (console).
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AthenaExecution",
                "Action": [
                    "athena:StartQueryExecution",
                    "athena:GetQueryExecution",
                    "athena:GetQueryResults"
                ],
                "Effect": "Allow",
                "Resource": "arn:aws:athena:<AWS_Region>:<AWS_account_ID_number>:workgroup/<workgroup_id>"
            }
        ]
    }

  3. Create an IAM role for AWS Lambda service and name it smus_consumer_lambda. Assign to it the AWS managed permission AWSLambdaBasicExecutionRole and the permission named smus_consumer_athena_execution that you just created. For instructions, refer to Create a role to delegate permissions to an AWS service.

After the IAM role for the Lambda function is in place, you can create the Lambda function in the consumer account:

  1. Access the Lambda console in the consumer account.
  2. In the navigation pane, choose Functions.
  3. Choose Create function and enter the following information:
    1. For Function name, enter consumer_function.
    2. For Runtime, select Python 3.14.
    3. Expand Change default execution role section.
    4. For Execution role, select Use an existing role.
    5. For Existing role, select smus_consumer_lambda.
  4. Choose Create function.
  5. Under the Code tab, in the Code source, replace the existing code with the following:
    import boto3
    import time
    sts_client = boto3.client('sts')
    role_arn = "<role_arn>"
    session_name = "AthenaQuerySession"
    catalog = "AwsDataCatalog"
    database = "<database_name>"
    workgroup = "<workgroup_id>"
    query = "select * from "+catalog+"."+database+".trees"
    def lambda_handler(event, context):
        # Assume SageMaker Unified Studio project role
        assumed_role_object = sts_client.assume_role(
            RoleArn=role_arn,
            RoleSessionName=session_name
        )
        # Get temporary credentials
        credentials = assumed_role_object['Credentials']
        # Create Athena client using temporary credentials
        athena = boto3.client(
            'athena',
            aws_access_key_id=credentials['AccessKeyId'],
            aws_secret_access_key=credentials['SecretAccessKey'],
            aws_session_token=credentials['SessionToken'],
            region_name='eu-west-1'
        )
        # Execute Athena Query
        response = athena.start_query_execution(
            QueryString=query,
            QueryExecutionContext={
                'Database': database,
                'Catalog': catalog
            },
            WorkGroup=workgroup
        )
        query_execution_id = response['QueryExecutionId']
        # Polling with exponential backoff
        wait_time = 0.25  # Start with 0.25 seconds
        max_wait = 8      # Maximum wait time of 8 seconds
        
        while True:
            result = athena.get_query_execution(QueryExecutionId=query_execution_id)
            state = result['QueryExecution']['Status']['State']
            if state in ['FAILED', 'CANCELLED']:
                raise Exception(f"Query {state}")
            elif state == 'SUCCEEDED':
                break
            elif state in ['QUEUED', 'RUNNING']:
                time.sleep(wait_time)
                wait_time = min(wait_time * 2, max_wait)  # Double wait time, cap at max_wait
        # Retrieve results
        results = athena.get_query_results(QueryExecutionId=query_execution_id)
        return results

  6. Choose Deploy.

The code provided for the Lambda function includes some placeholders that you will replace later, after you have the required information. Don’t test the Lambda function at this time because it will fail because of the presence of the placeholders.

Create a user with administrative access

Amazon SageMaker Unified Studio supports two distinct domain types: AWS IAM Identity Center based domains and IAM based domains. At the time of writing this post, only IAM Identity Center based domains support multi-accounts association, therefore in this post you work with this type of domain that requires IAM Identity Center.

In the governance account, you enable IAM Identity Center and create an administrative user to create and manage the Amazon SageMaker Unified Studio domain. Create a user with administrative access:

  1. Enable IAM Identity Center in the governance account. For instructions, refer to Enable IAM Identity Center.
  2. In IAM Identity Center in the governance account, grant administrative access to a user. For a tutorial about using the IAM Identity Center directory as your identity source, refer to Configure user access with the default IAM Identity Center directory.

Sign in as the user with administrative access:

  • To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user. For help signing in using an IAM Identity Center user, refer to Sign in to your AWS access portal.

Create a SageMaker Unified Studio domain

To create the Amazon SageMaker Unified Studio domain in the governance account refer to Create a Amazon SageMaker Unified Studio domain – quick setup.

After your domain is created, you can navigate to the Amazon SageMaker Unified Studio portal (a browser-based web application) where you can use your data and configured tools for analytics and AI. Save the Amazon SageMaker Unified Studio portal URL because you will use this URL later.

Solution steps

Now that you have the prerequisites in place, you can complete the following ten high-level steps to implement the solution.

Associate the producer and consumer accounts to the Amazon SageMaker Unified Studio domain

Start by associating the producer and consumer accounts to the newly created Amazon SageMaker Unified Studio domain. When you associate your producer and consumer accounts to the domain, make sure to select IAM users and roles can access APIs and IAM users can log in to Amazon SageMaker Unified Studio in the AWS RAM share managed permission section. For step-by-step instructions, refer to Associated accounts in Amazon SageMaker Unified Studio. If your AWS accounts are part of the same organization, your association requests are automatically accepted. However, if your AWS accounts aren’t part of the same organization, request association with the other AWS accounts in the governance account and then accept the association request in both the producer and consumer accounts.

Create two project profiles

Now, create two project profiles, one for the producer project and one for the consumer project.

In Amazon SageMaker Unified Studio, a project profile defines an uber template for projects in your Amazon SageMaker domain. A project profile is a collection of blueprints that provides reusable AWS CloudFormation templates used to create project resources.

A project profile is associated to a specific AWS account. This means, when a project is created the blueprints listed in the project profile are deployed in the associated AWS account. To use a project profile, you must enable its blueprints in the AWS account associated to the project profile.

Create the producer project profile

You’re going to create the producer project profile that is associated to the producer account. This project profile will be used to create the producer project. This profile includes by default the Tooling blueprint that creates resources for the project, including IAM user roles and security groups.

Before creating the project profile, you will enable the Tooling blueprint in the producer account using the following procedure:

  1. Access the SageMaker console in the producer account.
  2. In the navigation pane, choose Associated domains.
  3. Select the domain you created while setting up.
  4. On the Blueprints tab, choose Enable in the Tooling blueprint section as shown in the following image:
  5. SageMaker Unified Studios Tooling blueprint config: disabled status with Enable button for IAM roles & AWS resource setup

  6. For Virtual private cloud (VPC) select your account VPC.
  7. For Subnets, select at least two subnets in different Availability Zones.
  8. Choose Enable blueprint.

Proceed to creating the project profile in the governance account:

  1. Access the SageMaker console in the governance account.
  2. In the navigation pane, choose Domains.
  3. Select the domain you created as part of prerequisites.
  4. Under the Project profiles tab, choose Create and enter the following information:
    1. For Project profile name, enter producer-project-profile.
    2. For Project profile creation options, select Custom create.
    3. DO NOT SELECT A BLUEPRINT for Blueprints because the Tooling blueprint is included by default in any project profile.
    4. For Account, select Provide an account ID.
    5. For Account ID, enter the producer account ID.
    6. For Region, select Provide region name and then select the Region in which you’re working.
    7. For Authorization, select Allow all users and groups.
    8. For Project profile readiness, select Enable project profile on creation.
  5. Choose Create project profile.

Create a consumer project profile

You also create a consumer project profile and associate it to the consumer account. This profile will be used to create the consumer project. The consumer project profile includes the LakeHouseDatabase blueprint, which is needed to create a lakehouse environment with an AWS Glue database for data management and an Amazon Athena workgroup for querying. The Tooling blueprint is included by default in the project profile.

Before creating the project profile, enable the Tooling and LakeHouseDatabase blueprints in the consumer account:

  1. Access the SageMaker console in the consumer account.
  2. In the navigation pane, choose Associated domains.
  3. Select the domain you created as part of prerequisites.
  4. On the Blueprints tab, choose Enable in the Tooling blueprint section.
  5. For Virtual private cloud (VPC) select your account VPC.
  6. For Subnets, select at least two subnets in different Availability Zones.
  7. Choose Enable blueprint.
  8. In the navigation pane, choose Associated domains.
  9. Select the domain you created as part of prerequisites.
  10. Under the Blueprints tab, select the LakeHouseDatabase blueprint.
  11. Choose Enable.
  12. Choose Enable blueprint.

After blueprints are enabled in the consumer account, you can proceed creating the project profile:

  1. Access the SageMaker console in the governance account.
  2. In the navigation pane, choose Domains.
  3. Select the domain you created as part of prerequisites.
  4. Under Project profiles tab choose Create and enter the following information:
    1. For Project profile name, enter consumer-project-profile.
    2. For Project profile creation options, select Custom create.
    3. For Blueprints, select LakeHouseDatabase.
    4. For Account, select Provide an account ID.
    5. For Account ID, enter the consumer account ID.
    6. For Region, select Provide region name and then select the Region you are working.
    7. For Authorization, select Allow all users and groups.
    8. For Project profile readiness, select Enable project profile on creation.
  5. Choose Create project profile.

Create SageMaker Unified Studio producer and consumer projects

In Amazon SageMaker Unified Studio, a project is a boundary within a domain where you can collaborate with other users to work on a business use case. In projects, you can create and share data and resources.To create producer and consumer projects in Amazon SageMaker Unified Studio use the following instructions:

  1. Access the Amazon SageMaker Unified Studio portal.
  2. Choose the Select a project dropdown list.
  3. Choose Create project and enter the following information:
    1. For Project name, enter Producer.
    2. For Project profile, select producer-project-profile.
  4. Choose Continue.
  5. Choose Continue.
  6. Choose Create project.

After you’ve created the Producer project, note in a text file the Project role ARN that is displayed in the Project overview. The following image is shown for reference. The project role name is the string that follows arn:aws:iam::<account_ID>:role/ in the project role Amazon Resource Name (ARN). You will use both project role name and ARN later.

SageMaker Producer project overview: active status, files listed, S3 location & IAM role ARN displayed in project details tab

Repeat the preceding procedure to create the Consumer project. Be sure to enter Consumer for Project name and then select consumer-project-profile for Project profile. After it’s created, note the Project role ARN in a text file. The project role name is the string that follows arn:aws:iam::<account_ID>:role/ in the project role ARN. You will use both project role name and ARN later.

Bring your own data from the producer account

Bring your own data to the Amazon SageMaker Unified Studio Producer project. AWS provides several options to achieve this onboarding. The first option is automated onboarding in Amazon SageMaker lakehouse, in which you ingest the Amazon SageMaker lakehouse metadata of datasets into Amazon SageMaker Catalog. With this option, you can onboard your Amazon SageMaker lakehouse data as part of creating a new Amazon SageMaker Unified Studio domain or for an existing domain.

For more information about automated onboarding of Amazon SageMaker lakehouse data, refer to Onboarding data in Amazon SageMaker Unified Studio. As other options, you can bring in existing resources to your Amazon SageMaker Unified Studio project by using the Data and Compute pages in your project, or by using scripts provided in GitHub. For more information about using the Data and Compute pages or about using scripts, refer to Bringing existing resources into Amazon SageMaker Unified Studio. In this post, you will use Amazon SageMaker lakehouse capabilities to import your trees AWS Glue table into the Producer project.

Register the Amazon S3 location for the table

To use Lake Formation permissions for fine-grained access control to the trees table, you need to register in Lake Formation the Amazon S3 location of the trees table. To do that, complete the following actions:

  1. Access the Lake Formation console in the producer account.
  2. In the navigation pane under Administration, choose Data lake locations.
  3. Choose Register location and enter the following information:
    1. For S3 URI, enter s3://<bucket-name>/<prefix>/ where <bucket-name> is the name of the S3 bucket you created in the prerequisites and <prefix> is the optional prefix for the trees.csv file you uploaded as part of the prerequisite.
    2. For IAM role, select AWSServiceRoleForLakeFormationDataAccess.
    3. For Permission mode, select Lake Formation.
  4. Choose Register location.

Grant Producer project role permissions on the database

Grant database access to the IAM role that is associated with your Producer project. This role is called the project role, and it was created in IAM upon project creation.

To access the AWS Glue Data Catalog collections database from the Producer project in the Amazon SageMaker Unified Studio, complete the following actions:

  1. Access the Lake Formation console in the producer account.
  2. In the navigation pane under Data Catalog, choose Databases.
  3. Choose the collections database.
  4. From the Actions menu, choose Grant and enter the following information:
    1. For IAM users and roles, select your Producer project’s role name. This is the string starting with datazone_usr_role_ that is part of the Producer project role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
    2. For Database permissions, select Describe.
  5. Choose Grant.

Grant Producer project role permissions on the table

Grant trees table access to the IAM role that is associated with your Producer project. To grant these permissions use the following instructions:

  1. Access the Lake Formation console in the producer account.
  2. In the navigation pane under Data Catalog, choose Tables and MVs.
  3. Select the trees table.
  4. From the Actions menu, choose Grant and enter the following information:
    1. For IAM users and roles, select your Producer project’s role. This is the string starting with datazone_usr_role_ that is part of the Producerproject role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
    2. For Table permissions, select Select and Describe.
    3. For Grantable permissions, select Select and Describe.
  5. Choose Grant.

Revoke any existing permissions of IAMAllowedPrincipals

You must revoke the IAMAllowedPrincipals group permissions on both the database and table to enforce Lake Formation permission for access. For more information, refer to Revoking permission using the Lake Formation console.

  1. Access the Lake Formation console in the producer account.
  2. In the navigation pane under Permission, choose Data permissions.
  3. Select the entries where Principal is set to IAMAllowedPrincipals and Resource is set to collections or trees as in the following image:
  4. Data permissions table: 2 of 5 IAMAllowedPrincipals entries selected. All permissions granted for collections DB & trees table

  5. Choose Revoke.
  6. Enter revoke.
  7. Choose Revoke again.

Verify that data is available in the Producer project

Verify that your collections database and trees table are accessible in the Producer project:

  1. Access the Amazon SageMaker Unified Studio portal.
  2. Choose the Select a project drop-down menu and choose the Producer project.
  3. In the navigation pane under Overview, choose Data.
  4. Choose Lakehouse.
  5. Choose AwsDataCatalog.
  6. Choose collections.
  7. Choose tables.
  8. Choose the three-dot action menu next to your trees table and choose Preview data, as shown in the following image.
    AWS Data Catalog interface: collections database in Lakehouse with trees table, presenting preview/notebook/drop options
  9. You’ll find data from the trees table as shown in the following image.
    Query Editor showing SQL query on trees table with results: oak (3 stars), maple (2), birch (3). Red arrow highlights output

Create Amazon SageMaker Catalog asset

Even if it’s accessible in the project, to work with the trees table in Amazon SageMaker Catalog, you need to register the data source and create an Amazon SageMaker Catalog asset:

  1. Access the Amazon SageMaker Unified Studio portal.
  2. Choose the Select a project dropdown list and choose the Producer project.
  3. On the project page, under Project catalog in the navigation pane, choose Data sources.
  4. Choose Create Data Source and make the following selections:
    1. For Name, enter collections.
    2. For Data source type, select AWS Glue (Lakehouse).
    3. For Database name, select collections.
    4. Choose Next.
    5. Choose Next.
    6. Choose Next.
    7. Choose Create.
  5. After the data source is created, you will be in the collections data source page, choose Run. This will import metadata and create the Amazon SageMaker Catalog asset.
  6. In the collections data source, on the Data source runs tab, you’ll find your run marked as Completed and the trees asset Successfully created, as shown in the following image:
    Producer project Assets page: Inventory tab presenting trees Glue Table asset with red arrows highlighting navigation & selection

Publish the data asset in the Amazon SageMaker Catalog

Publishing a data asset manually is a one-time operation that you need to perform to allow others to access the data asset through the catalog:

  1. Access the Amazon SageMaker Unified Studio portal.
  2. Choose the Select a project dropdown list and choose the Producer project.
  3. On the project page under Project catalog, choose Assets.
  4. Select your trees data asset that is available on the Inventory tab. The following image is shown for reference.
    Assets Inventory page: trees Glue Table listed in Producer project with navigation arrows highlighting menu selection
  5. (Optional) If automated metadata generation is enabled when the data source is created, metadata for assets (such as the asset business name) is available to review and accept or reject. You can either choose Accept All or Reject All in the Automated Metadata Generation banner.
  6. Choose Publish Asset. The following image is shown for reference.
    Asset overview: Agricultural Crop Yield dataset with automated metadata banner, ACCEPT ALL & PUBLISH ASSET buttons highlighted
  7. Choose Publish Asset.

Subscribe to the data asset in the Amazon SageMaker Catalog

To consume data assets in the Consumer project, subscribe to the data asset by creating a subscription request:

  1. Access the Amazon SageMaker Unified Studio portal.
  2. Choose the Select a project dropdown list and choose Consumer project.
  3. On the Discover menu, choose Catalog.
  4. Enter trees in the search box and then select the data asset returned from the search. If in step 7 “Publish the data asset in the Amazon SageMaker Catalog” you chose Accept All in the Automated Metadata Generation banner, your data asset will have a different business name generated by the automated metadata recommendations feature. The data asset technical name is trees. For reference, refer to the following image.
    Data Catalog search: 'trees' query shows Agricultural Crop Yield dataset with browse assets & data products options
  5. Choose Subscribe.
  6. For Comment, enter a justification such as This data asset is needed for model training purposes.
  7. Choose Subscribe again.

By default, asset subscription requests require manual approval by a data owner. However, if the requester in the Consumer project is also a member of the Producer project, the subscription request is automatically approved. For information about approving subscription requests, refer to Approve or reject a subscription request in Amazon SageMaker Unified Studio.

Configure your Lambda IAM role to access the subscribed data access

To enable your Lambda function access to the subscribed data asset, you need to allow the Lambda function to assume the Consumer project role. To do this, edit the Consumer project’s IAM role trust relationship:

  1. Navigate to the IAM console in the consumer account.
  2. In the navigation pane under Access management, choose Roles.
  3. Select the Consumer project’s IAM role. This is the string starting with datazone_usr_role_ that is part of the Consumer project role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”.
  4. Under the Trust relationships tab, choose Edit trust policy.
  5. For backup reasons, make a copy of the existing trust policy in a text file.
  6. In the Edit trust policy window, add the following statement to the existing trust policy without removing or overwriting other existing statements in the trust policy. Be sure to replace the placeholder <account_id> with your consumer AWS account ID.
    {
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::<account_id>:role/smus_consumer_lambda"
        },
        "Action": [
            "sts:AssumeRole"
        ]
    }	

    IAM trust policy editor: JSON code with red arrow highlighting AWS principal ARN for smus_consumer_lambda role

  7. Choose Update policy.

Test the Lambda function’s access to the subscribed data asset

Before you can test your Lambda function, you need to replace placeholders in the function code and in the IAM policy. There are three placeholders to be replaced: <role_arn>, <database_name> and <workgroup_id>. For <role_arn>, you already have the actual value, which is the Consumer project’s role ARN that you noted in step 3 “Create SageMaker Unified Studio producer and consumer projects”. The next sections provide instructions to retrieve values for the other placeholders.

Retrieve the AWS Glue Data Catalog database name

You need to find the name of the AWS Glue Data Catalog database that was created along with the Consumer project. You will then use this value to replace the <database_name> placeholder in the consumer_function Lambda function code. To retrieve the AWS Glue Data Catalog database name, follow these instructions:

  1. Access the Amazon SageMaker Unified Studio portal.
  2. Choose the Select a project dropdown list and choose Consumer project.
  3. On the project page, under Overview, choose Data.
  4. Choose Lakehouse.
  5. Choose AwsDataCatalog.
  6. Copy the name of the database. It should be an alphanumerical string starting with glue_db, as in the following image:
  7. Consumer project Data page: Lakehouse > AwsDataCatalog > glue_db database navigation with tables & views expandable sections” width=”1084″ height=”294″> </p>
</ol>
<h4>Retrieve the Athena workgroup ID</h4>
<p>You need to find the ID of the Athena workgroup that was created along with the <code>Consumer</code> project. You will then use this value to replace the <code><workgroup_id></code> placeholder in the <code>consumer_function</code> Lambda function code and in the <code>smus_consumer_athena_execution</code> IAM policy. Use the following instructions to retrieve the Athena workgroup ID:</p>
<ol>
<li>Access the Amazon SageMaker Unified Studio portal.</li>
<li>Choose the <strong>Select a project</strong> dropdown list and choose <code>Consumer</code> project.</li>
<li>On the project page, under <strong>Overview</strong>, choose <strong>Compute</strong>.</li>
<li>Under the <strong>SQL analytics</strong> tab, select <strong>project.athena</strong>, as in the following image:<br /> <img decoding=

  8. Copy the Workgroup ARN and save to a text file. The Athena workgroup ID is the string that follows arn:aws:athena:<region>:<account_ID>:workgroup/ in the Workgroup ARN.

Replace placeholder in the smus_consumer_athena_execution IAM policy

To replace the <workgroup_id> placeholder in the smus_consumer_athena_execution IAM policy, use the following procedure:

  1. Access the IAM console in the consumer account.
  2. In the navigation pane, choose Policies.
  3. In the search field enter smus_consumer_athena_execution.
  4. Select the smus_consumer_athena_execution policy.
  5. Choose Edit.
  6. Replace <workgroup_id> with the value you noted earlier.
  7. Choose Next.
  8. Choose Save changes.

Replace placeholders in the Lambda function code and test it

In this section, you will replace the <role_arn>, <database_name> and <workgroup_id> placeholders in the consumer_function Lambda function code, and then you can test the function ability to access data of the trees table.

  1. Access the Lambda console in the consumer account.
  2. In the navigation pane, choose Functions.
  3. Select consumer_function.
  4. Under the Code tab, replace <role_arn>, <database_name> and <workgroup_id> placeholders with the respective values you noted earlier.
  5. Choose Deploy.
  6. Under the Test tab, for Event name, enter mytest.
  7. Choose Test.
  8. Choose Details in the green banner titled Executing function that appears after the execution is completed.
  9. The execution log reports the trees table content, as shown in the following image:
    Lambda test results: consumer_function succeeded with JSON output showing VarCharValue 'ok' and '3', execution details available

If your Lambda function execution fails due to timeout, change the function timeout setting as follows:

  1. Access the Lambda console in the consumer account.
  2. In the navigation pane, choose Functions.
  3. Select consumer_function.
  4. Under the Configuration tab, choose Edit.
  5. For Timeout, enter 15 sec or a greater value.
  6. Choose Save.

After increasing the timeout, test the function again.

Clean up

If you no longer need the resources you created as you followed this post, delete them to prevent incurring additional charges. Start by deleting your Amazon SageMaker Unified Studio domain in the governance account. For more information, refer to Delete domains.

To remove the AWS Glue collections database from the producer account, follow these steps:

  1. Access the Glue console in the producer account.
  2. In the navigation pane under Data Catalog, choose Databases.
  3. Select the collections database.
  4. Choose Delete.
  5. Choose Delete.

To remove the S3 bucket from the producer account, empty the bucket and then you can delete the bucket. For information about emptying the bucket, refer to Emptying a general purpose bucket. For information about deleting the bucket, refer to Deleting a general purpose bucket.

To remove the Lambda function from the consumer account, follow these steps:

  1. Access the Lambda console in the consumer account.
  2. In the navigation pane, choose Functions.
  3. Select the consumer_function Lambda function.
  4. Choose the Actions menu and then choose Delete function.
  5. Enter confirm.
  6. Choose Delete.

To complete the cleanup, delete the IAM role named smus_consumer_lambda, then delete the IAM policy named smus_consumer_athena_execution in the consumer account. For information about removing a IAM role, refer to Delete roles or instance profiles. For information about removing an IAM policy, refer to Delete IAM policies.

Conclusion

In this post, we covered adopting Amazon SageMaker Catalog for data governance without rearchitecting your existing applications and data repositories. We walked through how to onboard existing data in Amazon SageMaker Unified Studio, then publish it in a catalog, and then subscribe and consume the data from resources deployed outside the context of an Amazon SageMaker Unified Studio project. This solution can help you accelerate your implementation of a data mesh pattern with Amazon SageMaker Catalog to publish, find, and access data securely in your organization.

For more information, refer to What is Amazon SageMaker? and work through the Amazon SageMaker Workshop to try the unified experience for data, analytics, and AI.


About the authors

Paolo Romagnoli

Paolo is a Senior Solutions Architect at AWS for Energy and Utilities. With 20+ years of experience in designing and building enterprise solutions, he works with global energy customers to design solutions to address customers’ business and technical needs. He is passionate about technology and enjoys running.

Joel Farvault

Joel is a Principal Specialist SA Analytics for AWS with 25 years’ experience working on enterprise architecture, data governance and analytics. He uses his experience to advise customers on their data strategy and technology foundations.

Access Snowflake Horizon Catalog data using catalog federation in the AWS Glue Data Catalog

Post Syndicated from Andries Engelbrecht original https://aws.amazon.com/blogs/big-data/access-snowflake-horizon-catalog-data-using-catalog-federation-in-the-aws-glue-data-catalog/

This is a guest post by Andries Engelbrecht, Principal Partner Solutions Engineer at Snowflake, in partnership with AWS.

AWS announced a new catalog federation feature that allows you to directly access data from Snowflake Horizon Catalog through the AWS Glue Data Catalog. This integration enables you to discover and query Horizon Catalog data in Iceberg format through REST endpoints while applying fine-grained access controls using AWS Lake Formation. The new catalog federation combined with Snowflake’s catalog-linked database feature means users can access data stored across AWS and Snowflake from a single point of entry, reducing data movement and associated costs by eliminating the need to duplicate data across platforms.

In this post, we show you how to connect the AWS Glue Data Catalog to Snowflake Horizon Catalog and query the data using AWS analytics services. We cover how to set up catalogs in Horizon Catalog and configure required permissions, create and configure the federation connection in AWS Glue, implement fine-grained access controls using AWS Lake Formation, and finally, query federated tables using Amazon Athena. This step-by-step approach guides you through the complete process of establishing a integration between your Snowflake and AWS data environments.

Business examples and key benefits

Catalog federation enables several critical business scenarios while delivering key operational and strategic benefits.

Common examples

This federation capability addresses several key business scenarios:

  • Governed, cross-platform analytics: Query data across AWS and Snowflake environments to improve data-driven decision making without data movement or duplication
  • Data mesh implementation: Enable secure and federated data discovery while maintaining domain-oriented ownership
  • Compliance management: Implement consistent access controls and auditing across platforms

Key benefits

  • Operational efficiency: Eliminate data duplication and reduce Extract Transform Load (ETL) workloads
  • Enhanced security: Centralize access control through AWS Lake Formation with fine-grained permissions
  • Cost optimization: Minimize data transfer and storage costs across platforms
  • Improved agility: Enable faster time to insights with direct query access
  • Simplified governance: Maintain unified compliance and audit framework

Solution overview

The solution uses catalog federation in the AWS Glue Data Catalog to integrate with Snowflake Horizon Catalog. This integration supports both Snowflake Horizon, where the catalog is internal to Snowflake, and external catalogs such as Apache Polaris, Snowflake Open Catalog (a managed service that hosts Apache Polaris), and others.

The following diagram illustrates how AWS Glue Data Catalog federates with Snowflake Horizon Catalog, enabling customers to directly access Iceberg-format data managed by Snowflake Horizon Catalog through the Glue Data Catalog.

Architecture diagram showing integration between AWS services and Snowflake using federated catalog connections through Apache Iceberg REST API.

The integration works through three main components:

  1. Authentication: Uses OAuth2 credentials of Snowflake principal
  2. Access Control: AWS Lake Formation manages fine-grained permissions
  3. Query Access: AWS Analytics services like Amazon Athena can directly query the federated tables

Now, we walk through the step-by-step process of setting up this integration.

Prerequisites

Before you begin, confirm you have the following:

Configure Snowflake Horizon Catalog for Iceberg external access

Snowflake Horizon Catalog already supports managing Iceberg tables. For this walkthrough, you need to create Snowflake-managed Iceberg tables with data stored in Amazon S3.

Follow these steps in order:

  1. Create an external volume for S3: First, create an external volume that points to your S3 bucket where Iceberg table data is stored. Follow the instructions in Create External Volume(s) for the Iceberg Tables on S3.
  2. Create a database: Create a database to organize your tables. Refer to the Snowflake database creation documentation.
  3. Create a schema: Create a schema within your database following the Snowflake schema creation guide.
  4. Create an Iceberg table: Create your Iceberg table using the external volume. Follow the instructions to Create Iceberg Table.

After completing these steps, your Snowflake-managed Iceberg tables are ready to federate with AWS Glue Data Catalog.

Configure access control and authentication

To enable AWS Glue to access your Snowflake-managed Iceberg tables, you need to configure access control and obtain authentication credentials.

Step 1: Configure access control

Create a dedicated Snowflake role for external engine access to establish clear governance boundaries. Follow the instructions in Configure Access Control for external engines and set up the appropriate permissions for your Iceberg tables.

Step 2: Obtain an access token

Generate an access token for authenticating AWS Glue to Snowflake Horizon Catalog. Snowflake supports three authentication mechanisms:

  • External OAuth
  • Key-pair authentication
  • Programmatic Access Token (PAT)

Choose the authentication method that best fits your security requirements and follow the corresponding Snowflake documentation to generate your credentials.

Catalog Federation supports OAuth or custom authentication. For details on using OAuth refer to Federate to Snowflake Iceberg Catalog.

For this post, we use custom authentication and generate access token using PAT. Replace role_name with the principal role and token_value with the principal’s Programmatic Access Token.

curl --location 'https://<accountidentifier>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=client_credentials' \
--data-urlencode 'scope=session:role:<role_name>' \
--data-urlencode 'client_secret=<token_value>'

Note down the access token that is generated.

Step 3: Enable catalog federation

With access control configured and authentication credentials in hand, AWS Glue Catalog Federation can now connect to and access Snowflake’s Horizon Catalog.

Optional: Snowflake Open Catalog configuration

If you prefer to use Snowflake Open Catalog for Iceberg external access instead, refer to Sync a Snowflake-managed table with Snowflake Open Catalog for alternative setup instructions.

Setup Glue Catalog federation with Snowflake Horizon Catalog

Create a secret on AWS Secrets Manager

Log in to AWS console using the IAM role that has access to AWS Secrets Manager. Open Secrets Manager:

  • Choose Store a new secret and select Other type of secret for the secret type.
  • Set the key-value pair:
    • Key: BEARER_TOKEN
    • Value: The access token noted earlier
  • Choose Next and provide the secret name as horizon-secret.
  • Complete the setup by choosing Store.

Alternatively, you can use the CLI to create the secret by running the following command.

Replace your-access-token and your-region with your actual values:

aws secretsmanager create-secret \
    --name horizon-secret \
    --description "Snowflake Horizon access token" \
    --secret-string '{
        "BEARER_TOKEN": "your-access-token"
    }' \
    --region your-region

Create IAM role for catalog federation

As the catalog owner of a federated catalog in AWS Glue Data Catalog, you can use Lake Formation to implement comprehensive access controls for your data teams:

Access control options

You can implement access controls at different granularity levels depending on your governance needs:

  • Coarse-grained: Table-level permissions
  • Fine-grained: Column-level, row-level, and cell-level filtering
  • Tag-based: Dynamic access based on data classification tags

Lake Formation requires an IAM role with permissions to access the underlying S3 locations of your external catalog.

Create an IAM role that enables the Glue Connection to access AWS Secrets Manager, VPC configurations (optional) and Lake formation to manage credential vending for S3 bucket/prefix.

Required permissions

  1. Secrets Manager access: The Glue connection requires permissions to retrieve secret values from Secrets Manager for OAuth tokens stored for your Snowflake service connection.
  2. Amazon Virtual Private Cloud (VPC) Access (optional): When using VPC endpoints to restrict connectivity to your Snowflake Open Catalog account, the Glue connection needs permissions to describe and use VPC network interfaces. This configuration ensures secure, controlled access to both your stored credentials and network resources while maintaining proper isolation through VPC endpoints.
  3. S3 bucket and AWS Key Management Service (KMS) key permission: The Glue connection requires S3 permissions to read certificates if used in the connection setup. Additionally, Lake Formation requires read permissions on the bucket/prefix where the remote catalog table data resides. If the data is encrypted using a KMS key, additional KMS permissions are required.

Setup steps:

Run the following command using AWS CLI by replacing the placeholder with your setup information:

Create a JSON file (e.g., trust-policy.json) with the following structure:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": ["glue.amazonaws.com","lakeformation.amazonaws.com"]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Use the aws iam create-role command, referencing the trust policy file:

aws iam create-role \
    --role-name LFDataAccessRole \
    --assume-role-policy-document file://<path_file_downloaded>/trust-policy.json 

First, create a JSON file (such as, permissions-policy.json) for the permissions:


{
"Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": [
                "<secrets manager ARN>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateNetworkInterface",
                "ec2:DeleteNetworkInterface",
                "ec2:DescribeNetworkInterfaces"
            ],
            "Resource": "*",
            "Condition": {
                "ArnEquals": {
                    "ec2:Vpc": "arn:aws:ec2:region:account-id:vpc/<vpc-id>", 
                    "ec2:Subnet": [ 
                        "arn:aws:ec2:region:account-id:subnet/<subnet-id>"
                    ]
                }
            }
        },
        {
           # Required when using custom cert to sign requests.
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucketname>/<certpath>"
            ]
        },
        { # Required when using customer managed encryption key for s3 
            "Effect": "Allow",
            "Action": [
                "kms:decrypt",
                "kms:encrypt"
            ],
            "Resource": [
                "<kmsKey>"
            ]
        }
    ]
}

Then, attach it to the role:

aws iam put-role-policy \
--role-name LFDataAccessRole \
--policy-name myaccesspolicies \
--policy-document file://<path_file_downloaded>/permissions- policy.json

Create federated catalog in Glue Data Catalog

AWS Glue supports the SNOWFLAKEICEBERGRESTCATALOG connection type for connecting Glue Data Catalog with Snowflake Horizon Catalog and Snowflake Open Catalog. This Glue connector supports OAuth2 authentication and includes additional configuration parameters like CASING_TYPE to customize how AWS Glue Data Catalog discovers metadata in the Snowflake Horizon Catalog accounts.

Log in to your AWS console as a data lake admin and open the AWS Lake Formation console.

  1. Choose Catalog in the left navigation pane and select Create catalog.
  2. Choose the data source as Snowflake Horizon Catalog.
    AWS Lake Formation console screenshot showing Step 1 of catalog creation wizard with five federation type options, Snowflake Horizon Catalog selected.
  3. Provide the following information:
    • Name: Name of the federated catalog in Glue Catalog. For this post, we use federated_lakehousedb
    • Catalog name in Snowflake: Catalog name existing in Snowflake Horizon Catalog, this should match exact name in Horizon catalog. For this post, we use LAKEHOUSEDB
    • For Connection details, choose New connection configurations:
      • Connection name: Name for the glue connection. For this post, we use federatedconnection1.
      • Workspace URL: Horizon IRC url (format: https://<account_identifier>.snowflakecomputing.com)
      • Casing type: choose Uppercase only
      • Authentication:
        • Authentication type: choose Custom. Alternatively, you can select OAuth2 authentication. For Custom authentication, an access token is created, refreshed, and managed by the customer’s application or system and stored using AWS Secrets Manager.
        • OAuth Secret: Provide the secret manager ARN that was created in the previous step.
  • If you have AWS PrivateLink setup and/or a proxy setup, you can provide network details under Settings for network configurations (optional).
  • For Register Glue connection with Lake Formation:
    • Choose the IAM role created earlier(LFDataAccessRole) to manage data access using Lake Formation.

To test the connection, choose Run test. After the connection information is validated, it shows as successful.

Green success banner displaying "Connection test successful" with checkmark icon, confirming valid AWS configuration.

You can now create the catalog by selecting Create catalog.

Alternatively, you can use AWS CLI to create connection and catalog using example commands:

aws glue create-connection \
--connection-input '{
"Name": "federatedconnection1",
"ConnectionType": "SNOWFLAKEICEBERGRESTCATALOG",
"ConnectionProperties": {
    "INSTANCE_URL": "<your-snowflake-account-URL>",
    "ROLE_ARN": "< ARN_of_LFDataAccessRole>",
    "CATALOG_CASING_FILTER": "UPPERCASE_ONLY"
},
"AuthenticationConfiguration": {
    "AuthenticationType": "CUSTOM",
    "SecretArn": "arn:aws:secretsmanager:<your-aws-region>:<your-aws-account-id>:secret:horizon-secret"
}
}' \
--region <your-aws-region>
aws lakeformation register-resource \
    --resource-arn <ARN_of_federatedconnection1_connection> \
    --role-arn <ARN_of_LFDataAccessRole> \
    --with-federation \
    --with-privileged-access \
    --region <your-aws-region>
aws glue create-catalog \
    --name federated_lakehousedb \
    --catalog-input '{
    "FederatedCatalog": {
        "Identifier": "LAKEHOUSEDB",
        "ConnectionName": “federatedconnection1 "
    },
    "CreateTableDefaultPermissions": [],
    "CreateDatabaseDefaultPermissions": []
}'

After the catalog is created, the Horizon databases and tables are listed under the federated catalog.

You can implement fine grained access control on the tables by applying row/column filter using Lake Formation.

Query the data using Athena query editor:

Open the Amazon Athena console and run the following query to access the federated Horizon table:

SELECT * FROM "public"."customer" limit 10;

Clean up

To clean up your resources, complete the following steps:

  1. Drop the Snowflake Database with Cascade.
  2. Drop External Volume created for Iceberg Tables on S3.
  3. Drop the resources in Glue Data Catalog and Lake Formation created for this post.
  4. Delete the IAM roles and S3 buckets used for this post.
  5. Delete any VPC, KMS keys if used for this post setup.

Conclusion

In this post, we demonstrated how to establish a secure connection between AWS Analytics services and Snowflake Horizon Catalog, enabling you to access your data from a single connected and governed view. You learned how to:

  • Configure catalog federation between AWS Glue Data Catalog and Snowflake Horizon Catalog
  • Set up OAuth2 authentication for secure access
  • Grant access to Iceberg table in Snowflake Horizon Catalog using AWS Lake Formation
  • Query federated tables using Amazon Athena

You can follow the same steps to establish a secure connection with open-source catalog options such as Snowflake Open Catalog, a managed service for Apache Iceberg. Remember to clean up any resources you created while following this tutorial to avoid ongoing charges.

To further explore this solution in your environment, consider the following resources:

These resources can help you to implement and optimize this integration pattern for your specific use case. As you begin this journey, remember to start small, validate your architecture with test data, and gradually scale your implementation based on your organization’s needs. Stay tuned for future workshops and resources.


About the authors

 

Andries Engelbrecht

Andries Engelbrecht

Andries is a Principal Partner Solutions Engineer at Snowflake working with AWS. He supports product and service integrations, as well the development of joint solutions with AWS. Andries has over 25 years of experience in the field of data and analytics.

Nidhi Gupta

Nidhi Gupta

Nidhi is a Senior Partner Solutions Architect at AWS, specializing in data analytics and AI. She helps customers and partners build and optimize Snowflake workloads on AWS. Nidhi has extensive experience leading development, production releases and deployments, with focus on Data, AI, ML, generative AI, and Advanced Analytics.

Srividya Parthasarathy

Srividya Parthasarathy

Srividya is a Senior Big Data Architect on the AWS Lake Formation team. She works with the product team and customers to build robust features and solutions for their analytical data platform. She enjoys building data mesh solutions and sharing them with the community.

Pratik Das

Pratik Das

Pratik is a Senior Product Manager with AWS Lake Formation. He is passionate about all things data and works with customers to understand their requirements and build delightful experiences. He has a background in building data-driven solutions and machine learning systems.

 

Access Databricks Unity Catalog data using catalog federation in the AWS Glue Data Catalog

Post Syndicated from Srividya Parthasarathy original https://aws.amazon.com/blogs/big-data/access-databricks-unity-catalog-data-using-catalog-federation-in-the-aws-glue-data-catalog/

AWS has launched the catalog federation capability, enabling direct access to Apache Iceberg tables managed in Databricks Unity Catalog through the AWS Glue Data Catalog. With this integration, you can discover and query Unity Catalog data in Iceberg format using an Iceberg REST API endpoint, while maintaining granular access controls through AWS Lake Formation. This approach significantly reduces operational overhead for managing catalog synchronization and associated costs by alleviating the need to replicate or duplicate datasets between platforms.

In this post, we demonstrate how to set up catalog federation between the Glue Data Catalog and Databricks Unity Catalog, enabling data querying using AWS analytics services.

Use cases and key benefits

This federation capability is particularly valuable if you run multiple data platforms, because you can maintain your existing Iceberg catalog investments while using AWS analytics services. Catalog federation supports read operations and provides the following benefits:

  • Interoperability – You can enable interoperability across different data platforms and tools through Iceberg REST APIs while preserving the value of your established technology investments.
  • Cross-platform analytics – You can connect AWS analytics tools (Amazon Athena, Amazon Redshift, Apache Spark) to query Iceberg and UniForm tables stored in Databricks Unity Catalog. It supports Databricks on AWS integration with the AWS Glue Iceberg REST Catalog for metadata retrieval, while using Lake Formation for permission management.
  • Metadata management – The solution avoids manual catalog synchronization by making Databricks Unity Catalog databases and tables discoverable within the Data Catalog. You can implement unified governance through Lake Formation for fine-grained access control across federated catalog resources.

Solution overview

The solution uses catalog federation in the Data Catalog to integrate with Databricks Unity Catalog. The federated catalog created in AWS Glue mirrors the catalog objects in Databricks Unity Catalog and supports OAuth-based authentication. The solution is represented in the following diagram.

The integration involves three high-level steps:

  1. Set up an integration principal in Databricks Unity Catalog and provide required read access on catalog resources to this principal. Enable OAuth-based authentication for the integration principal.
  2. Set up catalog federation to Databricks Unity Catalog in the Glue Data Catalog:
    1. Create a federated catalog in the Data Catalog using an AWS Glue connection.
    2. Create an AWS Glue connection that uses the credentials of the integration principal (in Step 1) to connect to Databricks Unity Catalog. Configure an AWS Identity and Access Management (IAM) role with permission to Amazon Simple Storage Service (Amazon S3) locations where the Iceberg table data resides. In a cross-account scenario, make sure the bucket policy grants required access to this IAM role.
  3. Discover Iceberg tables in federated catalogs using Lake Formation or AWS Glue APIs. During query operations, Lake Formation manages fine-grained permissions on federated resources and credential vending for access to the underlying data.

In the following sections, we walk through the steps to integrate the Glue Data Catalog with Databricks Unity Catalog on AWS.

Prerequisites

To follow along with the solution presented in this post, you must have the following prerequisites:

  • Databricks Workspace (on AWS) with Databricks Unity Catalog configured.
  • An IAM role that is a Lake Formation data lake administrator in your AWS account. A data lake administrator is an IAM principal that can register S3 locations, access the Data Catalog, grant Lake Formation permissions to other users, and view AWS CloudTrail logs. See Create a data lake administrator for more information.

Configure Databricks Unity Catalog for external access

Catalog federation to a Databricks Unity Catalog uses the OAuth2 credentials of a Databricks service principal configured in the workspace admin settings. This authentication mechanism allows the Data Catalog to access the metadata of various objects (such as catalogs, databases, and tables) within Databricks Unity Catalog, based on the privileges associated with the service principal. For proper functionality, grant the service principal with the necessary permissions (read permission on catalog, schema, and tables) to read the metadata of these objects and allow access from external engines.

Next, catalog federation enables discovery and query of Iceberg tables in your Databricks Unity Catalog. For reading delta tables, enable UniForm on a Delta Lake table in Databricks to generate Iceberg metadata. For more information, refer to Read Delta tables with Iceberg clients.

Follow the Databricks tutorial and documentation to create the service principal and associated privileges in your Databricks workspace. For this post, we use a service principal named integrationprincipal that is configured with required permissions (SELECT, USE CATALOG, USE SCHEMA) on Databricks Unity Catalog objects and will be used for authentication to catalog instance.

Catalog federation supports OAuth2 authentication, so enable OAuth for the service principal and note down the client_id and client_secret for later use.

Set up Data Catalog federation with Databricks Unity Catalog

Now that you have service principal access for Databricks Unity Catalog, you can set up catalog federation in the Data Catalog. To do so, you create an AWS Secrets Manager secret and create an IAM role for catalog federation.

Create secret

Complete the following steps to create a secret:

  1. Sign in to the AWS Management Console using an IAM role with access to Secrets Manager.
  2. On the Secrets Manager console, choose Store a new secret and Other type of secret.
  3. Set the key-value pair:
    1. Key: USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET
    2. Value: The client secret noted earlier
  4. Choose Next.
  5. Enter a name for your secret (for this post, we use dbx).
  6. Choose Store.

Create IAM role for catalog federation

As the catalog owner of a federated catalog in the Data Catalog, you can use Lake Formation to implement comprehensive access controls, including table filters, column filters, and row filters, as well as tag-based access for your data teams.

Lake Formation requires an IAM role with permissions to access the underlying S3 locations of your external catalog.

In this step, you create an IAM role that enables the AWS Glue connection to access Secrets Manager, optional virtual private cloud (VPC) configurations, and Lake Formation to manage credential vending for the S3 bucket and prefix:

  • Secrets Manager access – The AWS Glue connection requires permissions to retrieve secret values from Secrets Manager for OAuth tokens stored for your Databricks Unity service connection.
  • VPC access (optional) – When using VPC endpoints to restrict connectivity to your Databricks Unity account, the AWS Glue connection needs permissions to describe and utilize VPC network interfaces. This configuration provides secure, controlled access to both your stored credentials and network resources while maintaining proper isolation through VPC endpoints.
  • S3 bucket and AWS KMS key permission – The AWS Glue connection requires Amazon S3 permissions to read certificates if used in the connection setup. Additionally, Lake Formation requires read permissions on the bucket and prefix where the remote catalog table data resides. If the data is encrypted using an AWS Key Management Service (AWS KMS) key, additional AWS KMS permissions are required.

Complete the following steps:

  1. Create an IAM role called LFDataAccessRole with the following policies:
    {
     "Version": "2012-10-17",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
                     "secretsmanager:GetSecretValue",
                     "secretsmanager:DescribeSecret"
                 ],
                 "Resource": [
                     "<secrets manager ARN>"
                 ]
             },
             {
                 "Effect": "Allow",
                 "Action": [
                     "ec2:CreateNetworkInterface",
                     "ec2:DeleteNetworkInterface",
                     "ec2:DescribeNetworkInterfaces"
                 ],
                 "Resource": "*",
                 "Condition": {
                     "ArnEquals": {
                         "ec2:Vpc": "arn:aws:ec2:region:account-id:vpc/<vpc-id>", 
                         "ec2:Subnet": [ 
                             "arn:aws:ec2:region:account-id:subnet/<subnet-id>" 
                         ]
                     }
                 }
             },
             {
                # Required when using custom cert to sign requests.
                 "Effect": "Allow",
                 "Action": [
                     "s3:GetObject"
                 ],
                 "Resource": [
                     "arn:aws:s3
    :::<bucketname>/<certpath>"
                 ]
             },
             { # Required when using customer managed encryption key for s3 
                 "Effect": "Allow",
                 "Action": [
                     "kms:decrypt",
                     "kms:encrypt"
                 ],
                 "Resource": [
                     "<kmsKey>"
                 ]
             }
         ]
     }

  2. Configure the role with the following trust policy:
    {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect":  "Allow",
                  "Principal": {
                       "Service": ["glue.amazonaws.com","lakeformation.amazonaws.com"]
                  },
                  "Action":  "sts:AssumeRole"
              }
          ]
      }

Create federated catalog in Data Catalog

AWS Glue supports the DATABRICKSICEBERGRESTCATALOG connection type for connecting the Data Catalog with managed Databricks Unity Catalog. This AWS Glue connector supports OAuth2 authentication for discovering metadata in Databricks Unity Catalog.

Complete the following steps to create the federated catalog:

  1. Sign in to the console as a data lake admin.
  2. On the Lake Formation console, choose Catalogs in the navigation pane.
  3. Choose Create catalog.
  4. For Name, enter a name for your catalog.
  5. For Catalog name in Databricks, enter the name of a catalog existing in Databricks Unity Catalog.
  6. For Connection name, enter a name for the AWS Glue connection.
  7. For Workspace URL, enter the Unity Iceberg REST API URL (in format https://<workspace-url>/cloud.databricks.com).
  8. For Authentication, provide the following information:
    1. For Authentication type, choose OAuth2. Alternatively, you can choose Custom authentication. For Custom authentication, an access token is created, refreshed, and managed by the customer’s application or system and stored using Secrets Manager.
    2. For Token URL, enter the token authentication server URL.
    3. For OAuth Client ID, enter the client_id for integrationprincipal.
    4. For OAuth Secret, enter the secret ARN that you created in the previous step. Alternatively, you can provide the client_secret directly.
    5. For Token URL parameter map scope, provide the API scope supported.
  9. If you have AWS PrivateLink set up or a proxy set up, you can provide network details under Settings for network configurations.
  10. For Register Glue connection with Lake Formation, choose the IAM role (LFDataAccessRole) created earlier to manage data access using Lake Formation.

When the setup is done using AWS Command Line Interface (AWS CLI) commands, you have options to create two separate IAM roles:

  • IAM role with policies to access network and secrets, which AWS Glue assumes to manage authentication
  • IAM role with access to the S3 bucket, which Lake Formation assumes to manage credential vending for data access

On the console, this setup is simplified with a single role having combined policies. For more details, refer to Federate to Databricks Unity Catalog.

  1. To test the connection, choose Run test.
  2. You can proceed to create the catalog.

After you create the catalog, you can see the databases and tables in Databricks Unity Catalog listed under the federated catalog. You can implement fine-grained access control on the tables by applying row and column filters using Lake Formation. The following video shows the catalog federation setup with Databricks Unity Catalog.

Discover and query the data using Athena

In this post, we show how to use the Athena query editor to discover and query the Databricks Unity Catalog tables. On the Athena console, run the following query to access the federated table:SELECT * FROM "customerschema"."person" limit 10;The following video demonstrates querying the federated table from Athena.

If you use the Amazon Redshift query engine, you must create a resource link on the federated database and grant permission on the resource link to the user or role. This database resource link is automounted under awsdatacatalog based on the permission granted for the user or role and available for querying. For instructions, refer to Creating resource links.

Clean up

To clean up your resources, complete the following steps:

  1. Delete the catalog and namespace in Databricks Unity Catalog for this post.
  2. Drop the resources in the Data Catalog and Lake Formation created for this post.
  3. Delete the IAM roles and S3 buckets used for this post.
  4. Delete any VPC and KMS keys if used for this post.

Conclusion

In this post, we explored the key elements of catalog federation and its architectural design, illustrating the interaction between the AWS Glue Data Catalog and Databricks Unity Catalog through centralized authorization and credential distribution for protected data access. By removing the requirement for complicated synchronization workflows, catalog federation makes it possible to query Iceberg data on Amazon S3 directly at its source using AWS analytics services with data governance across multi-catalog platforms. Try out the solution for your own use case, and share your feedback and questions in the comments.


About the Authors

Srividya Parthasarathy

Srividya Parthasarathy

Srividya is a Senior Big Data Architect on the AWS Lake Formation team. She works with the product team and customers to build robust features and solutions for their analytical data platform. She enjoys building data mesh solutions and sharing them with the community.

Venkatavaradhan (Venkat) Viswanathan

Venkatavaradhan (Venkat) Viswanathan

Venkat” is a Global Partner Solutions Architect at Amazon Web Services. Venkat is a Technology Strategy Leader in Data, AI, ML, Generative AI, and Advanced Analytics. Venkat is a Global SME for Databricks and helps AWS customers design, build, secure, and optimize Databricks workloads on AWS.

AWS analytics at re:Invent 2025: Unifying Data, AI, and governance at scale

Post Syndicated from Larry Weber original https://aws.amazon.com/blogs/big-data/aws-analytics-at-reinvent-2025-unifying-data-ai-and-governance-at-scale/

re:Invent 2025 showcased the bold Amazon Web Services (AWS) vision for the future of analytics, one where data warehouses, data lakes, and AI development converge into a seamless, open, intelligent platform, with Apache Iceberg compatibility at its core. Across over 18 major announcements spanning three weeks, AWS demonstrated how organizations can break down data silos, accelerate insights with AI, and maintain robust governance without sacrificing agility.

Amazon SageMaker: Your data platform, simplified

AWS introduced a faster, simpler approach to data platform onboarding for Amazon SageMaker Unified Studio. The new one-click onboarding experience eliminates weeks of setup, so teams can start working with existing datasets in minutes using their current AWS Identity and Access Management (IAM) roles and permissions. Accessible directly from Amazon SageMaker, Amazon Athena, Amazon Redshift, and Amazon S3 Tables consoles, this streamlined experience automatically creates SageMaker Unified Studio projects with existing data permissions intact. At its core is a powerful new serverless notebook that reimagines how data professionals work. This single interface combines SQL queries, Python code, Apache Spark processing, and natural language prompts, backed by Amazon Athena for Apache Spark to scale from interactive exploration to petabyte-scale jobs. Data engineers, analysts, and data scientists no longer need to context-switch between different tools based on workload—they can explore data with SQL, build models with Python, and use AI assistance, all in one place.

The introduction of Amazon SageMaker Data Agent in the new SageMaker notebooks marks a pivotal moment in AI-assisted development for data builders. This built-in agent doesn’t only generate code, it understands your data context, catalog information, and business metadata to create intelligent execution plans from natural language descriptions. When you describe an objective, the agent breaks down complex analytics and machine learning (ML) tasks into manageable steps, generates the required SQL and Python code, and maintains awareness of your notebook environment throughout the entire process. This capability transforms hours of manual coding into minutes of guided development, which means teams can focus on gleaning insights rather than repetitive boilerplate.

Embracing open data with Apache Iceberg

One significant theme across this year’s launches was the widespread adoption of Apache Iceberg across AWS analytics, transforming how organizations manage petabyte-scale data lakes. Catalog federation to remote Iceberg catalogs through the AWS Glue Data Catalog addresses a critical challenge in modern data architectures. You can now query remote Iceberg tables, stored in Amazon Simple Storage Service (Amazon S3) and catalogued in remote Iceberg catalogs, using preferred AWS analytics services such as Amazon Redshift, Amazon EMR, Amazon Athena, AWS Glue, and Amazon SageMaker, without moving or copying tables. Metadata synchronizes in real time, providing query results that reflect the current state. Catalog federation supports both coarse-grained access control and fine-grained access permissions through AWS Lake Formation enabling cross-account sharing and trusted identity propagation while maintaining consistent security across federated catalogs.

Amazon Redshift now writes directly to Apache Iceberg tables, enabling true open lakehouse architectures where analytics seamlessly span data warehouses and lakes. Apache Spark on Amazon EMR 7.12, AWS Glue, Amazon SageMaker notebooks, Amazon S3 Tables, and the AWS Glue Data Catalog now support Iceberg V3’s capabilities, including deletion vectors that mark deleted rows without expensive file rewrites, dramatically reducing pipeline costs and accelerating data modifications and row lineage. V3 automatically tracks every record’s history, creating audit trails essential for compliance and has table-level encryption that helps organizations meet stringent privacy regulations. These innovations mean faster writes, lower storage costs, comprehensive audit trails, and efficient incremental processing across your data architecture.

Governance that scales with your organization

Data governance received substantial attention at re:Invent with major enhancements to Amazon SageMaker Catalog. Organizations can now curate data at the column level with custom metadata forms and rich text descriptions, indexed in real time for immediate discoverability. New metadata enforcement rules require data producers to classify assets with approved business vocabulary before publication, providing consistency across the enterprise. The catalog uses Amazon Bedrock large language models (LLMs) to automatically suggest relevant business glossary terms by analyzing table metadata and schema information, bridging the gap between technical schemas and business language. Perhaps most importantly, SageMaker Catalog now exports its entire asset metadata as queryable Apache Iceberg tables through Amazon S3 Tables. This way, teams can analyze catalog inventory with standard SQL to answer questions like “which assets lack business descriptions?” or “how many confidential datasets were registered last month?” without building custom ETL infrastructure.

As organizations adopt multi-warehouse architectures to scale and isolate workloads, the new Amazon Redshift federated permissions capability eliminates governance complexity. Define data permissions one time from a Amazon Redshift warehouse, and they automatically enforce them across the warehouses in your account. Row-level, column-level, and masking controls apply consistently regardless of which warehouse queries originate from, and new warehouses automatically inherit permission policies. This horizontal scalability means organizations can add warehouses without increasing governance overhead, and analysts immediately see the databases from registered warehouses.

Accelerating AI innovation with Amazon OpenSearch Service

Amazon OpenSearch Service introduced powerful new capabilities to simplify and accelerate AI application development. With support for OpenSearch 3.3, agentic search enables precise results using natural language inputs without the need for complex queries, making it easier to build intelligent AI agents. The new Apache Calcite-powered PPL engine delivers query optimization and an extensive library of commands for more efficient data processing.

As seen in Matt Garman’s keynote, building large-scale vector databases is now dramatically faster with GPU acceleration and auto-optimization. Previously, creating large-scale vector indexes required days of building time and weeks of manual tuning by experts, which slowed innovation and prevented cost-performance optimizations. The new serverless auto-optimize jobs automatically evaluate index configurations—including k-nearest neighbors (k-NN) algorithms, quantization, and engine settings—based on your specified search latency and recall requirements. Combined with GPU acceleration, you can build optimized indexes up to ten times faster at 25% of the indexing cost, with serverless GPUs that activate dynamically and bill only when providing speed boosts. These advancements simplify scaling AI applications such as semantic search, recommendation engines, and agentic systems, so teams can innovate faster by dramatically reducing the time and effort needed to build large-scale, optimized vector databases.

Performance and cost optimization

Also announced in the keynote, Amazon EMR Serverless now eliminates local storage provisioning for Apache Spark workloads, introducing serverless storage that reduces data processing costs by up to 20% while preventing job failures from disk capacity constraints. The fully managed, auto scaling storage encrypts data in transit and at rest with job-level isolation, allowing Spark to release workers immediately when idle rather than keeping them active to preserve temporary data. Additionally, AWS Glue introduced materialized views based on Apache Iceberg, storing precomputed query results that automatically refresh as source data changes. Spark engines across Amazon Athena, Amazon EMR, and AWS Glue intelligently rewrite queries to use these views, accelerating performance by up to eight times while reducing compute costs. The service handles refresh schedules, change detection, incremental updates, and infrastructure management automatically.

The new Apache Spark upgrade agent for Amazon EMR transforms version upgrades from months-long projects into week-long initiatives. Using conversational interfaces, engineers express upgrade requirements in natural language while the agent automatically identifies API changes and behavioral modifications across PySpark and Scala applications. Engineers review and approve suggested changes before implementation, maintaining full control while the agent validates functional correctness through data quality checks. Currently supporting upgrades from Spark 2.4 to 3.5, this capability is available through SageMaker Unified Studio, Kiro CLI, or an integrated development environment (IDE) with Model Context Protocol compatibility.

For workflow optimization, AWS introduced a new Serverless deployment option for Amazon Managed Workflows for Apache Airflow (Amazon MWAA), which eliminates the operational overhead of managing Apache Airflow environments while optimizing costs through serverless scaling. This new offering addresses key challenges of operational scalability, cost optimization, and access management that data engineers and DevOps teams face when orchestrating workflows. With Amazon MWAA Serverless, data engineers can focus on defining their workflow logic rather than monitoring for provisioned capacity. They can now submit their Airflow workflows for execution on a schedule or on demand, paying only for the actual compute time used during each task’s execution.

Looking forward

These launches collectively represent more than incremental improvements. They signal a fundamental shift in how organizations are approaching analytics. By unifying data warehousing, data lakes, and ML under a common framework built on Apache Iceberg, simplifying access through intelligent interfaces powered by AI, and maintaining robust governance that scales effortlessly, AWS is giving organizations the tools to focus on insights rather than infrastructure. The emphasis on automation, from AI-assisted development to self-managing materialized views and serverless storage, reduces operational overhead while improving performance and cost efficiency. As data volumes continue to grow and AI becomes increasingly central to business operations, these capabilities position AWS customers to accelerate their data-driven initiatives with unprecedented simplicity and power. To view the Re:Invent 2025 Innovation Talk on analytics, visit Harnessing analytics for humans and AI on YouTube.


About the authors

Larry Weber

Larry Weber

Larry leads product marketing for the analytics portfolio at AWS.

Building scalable AWS Lake Formation governed data lakes with dbt and Amazon Managed Workflows for Apache Airflow

Post Syndicated from Abhilasha Agarwal original https://aws.amazon.com/blogs/big-data/building-scalable-aws-lake-formation-governed-data-lakes-with-dbt-and-amazon-managed-workflows-for-apache-airflow/

Organizations often struggle with building scalable and maintainable data lakes—especially when handling complex data transformations, enforcing data quality, and monitoring compliance with established governance. Traditional approaches typically involve custom scripts and disparate tools, which can increase operational overhead and complicate access control. A scalable, integrated approach is needed to simplify these processes, improve data reliability, and support enterprise-grade governance.

Apache Airflow has emerged as a powerful solution for orchestrating complex data pipelines in the cloud. Amazon Managed Workflows for Apache Airflow (MWAA) extends this capability by providing a fully managed service that eliminates infrastructure management overhead. This service enables teams to focus on building and scaling their data workflows while AWS handles the underlying infrastructure, security, and maintenance requirements.

dbt enhances data transformation workflows by bringing software engineering best practices to analytics. It enables analytics engineers to transform warehouse data using familiar SQL select statements while providing essential features like version control, testing, and documentation. As part of the ELT (Extract, Load, Transform) process, dbt handles the transformation phase, working directly within a data warehouse to enable efficient and reliable data processing. This approach allows teams to maintain a single source of truth for metrics and business definitions while enabling data quality through built-in testing capabilities.

In this post, we show how to build a governed data lake that uses modern data tools and AWS services.

Solution overview

We explore a comprehensive solution that includes:

  • A metadata-driven framework in MWAA that dynamically generates directed acyclic graphs (DAGs), significantly improving pipeline scalability and reducing maintenance overhead.
  • dbt with Amazon Athena adapter to implement modular, SQL-based data transformations directly on a data lake, enabling well-structured, and thoroughly tested transformations.
  • An automated framework that proactively identifies and segregates problematic records, maintaining the integrity of data assets.
  • AWS Lake Formation to implement fine-grained access controls for Athena tables, ensuring proper data governance and security throughout a data lake environment.

Together, these components create a robust, maintainable, and secure data management solution suitable for enterprise-scale deployments.

The following architecture illustrates the components of the solution.

The workflow contains the following steps:

  1. Multiple data sources (PostgreSQL, MySQL, SFTP) push data to an Amazon S3 raw bucket
  2. S3 event triggers AWS Lambda Function
  3. Lambda function triggers the MWAA DAG to convert file formats to parquet
  4. Data is stored in Amazon S3 formatted bucket under formatted_stg prefix
  5. Crawler crawls the data in formatted_stg prefix in the formatted bucket and creates catalog tables
  6. dbt using Athena adapter processes the data and puts the processed data after data quality checks under formatted prefix in Formatted bucket
  7. dbt using Athena adapter can perform further transformations on the formatted data and put the transformed data in Published bucket

Prerequisites

To implement this solution, the following prerequisites need to be met.

Deploy the solution

For this solution, we provide an AWS CloudFormation (CFN) template that sets up the services included in the architecture, to enable repeatable deployments.

Note:

  • US-EAST-1 Region is required for the deployment.
  • Deploying this solution will involve costs associated with AWS services.

To deploy the solution, complete the following steps:

  1. Before deploying the stack, open the AWS Lake Formation console. Add your console role as a Data Lake Administrator and choose Confirm to save the changes.
  2. Download the CloudFormation template.
    After the file is downloaded to the local machine, follow the steps below to deploy the stack using this template:

    1. Open the AWS CloudFormation Console.
    2. Choose Create stack and choose With new resources (standard).
    3. Under Specify template, select Upload a template file.
    4. Select Choose file and upload the CFN template that was downloaded earlier.
    5. Choose Next to proceed.

  3. Enter a stack name (for example, bdb4834-data-lake-blog-stack) and configure the parameters (bdb4834-MWAAClusterName can be left as the default value and update SNSEmailEndpoints with your email address), then choose Next.
  4. Select “I acknowledge that AWS CloudFormation might create IAM resources with custom names” and choose Next

  5. Review all the configuration details on the next page, then choose Submit.
  6. Wait for the stack creation to complete in the AWS CloudFormation console. The process typically takes approximately 35 to 40 minutes to provision all required resources.

    The following table shows resources available in the AWS Account after CloudFormation template deployment is successfully completed:

    Resource Type Description Example Resource Name
    S3 Buckets For storing raw, processed data and assets bdb4834-mwaa-bucket-<AWS_ACCOUNT>-<AWS_REGION>,bdb4834-raw-bucket-<AWS_ACCOUNT>-<AWS_REGION>,bdb4834-formatted-bucket-<AWS_ACCOUNT>-<AWS_REGION>,bdb4834-published-bucket-<AWS_ACCOUNT>-<AWS_REGION>
    IAM Role Role assumed by MWAA for permissions bdb4834-mwaa-role
    MWAA Environment Managed Airflow environment for orchestration bdb4834-MyMWAACluster
    VPC Network setup required by MWAA bdb4834-MyVPC
    Glue Catalog Databases Logical grouping of metadata for tables bdb4834_formatted_stg,bdb4834_formatted_exception, bdb4834_formatted, bdb4834_published
    Glue Crawlers Automatically catalog metadata from S3 bdb4834-formatted-stg-crawler
    Lambda Lambda to Trigger MWAA DAG on file arrival and to setup Lake Formation Permissions bdb4834_mwaa_trigger_process_s3_files,bdb4834-lf-tags-automation
    Lake Formation Setup Centralized governance and permissions LF-Setup for the above Resources
    Airflow DAGs Airflow DAGs are stored in the S3 bucket named mwaa-bucket-<AWS_ACCOUNT>-<AWS_REGION> under the dags/ prefix. These DAGs are responsible for triggering data pipelines based on either file arrival events or scheduled intervals. The exact functionality of each DAG is explained in the following sections. blog-test-data-processingcrawler-daily-runcreate-audit-tableprocess_raw_to_formatted_stage
  7. When the stack is complete perform the below steps:
    1. Open the Amazon Managed Workflows for Apache Airflow (MWAA) console, choose on Open Airflow UI
    2. In the DAGs console, locate the following DAGs and unpause them by unchecking the toggle switch (radio button) next to each DAG.

Add sample data to raw S3 bucket and create catalog tables

In this section, we upload sample data to raw S3 bucket (bucket name starting with bdb4834-raw-bucket) and convert the file formats to parquet and run AWS Glue crawler to create catalog tables that are used by dbt in the ELT Process. Glue Crawler automatically scans the data in S3 and creates or updates tables in the Glue Data Catalog, making the data queryable and accessible for transformation.

  1. Download the sample data.
  2. Zip folder contains two sample data files, cards.json and customers.json
    Schema for cards.json

    Field Data Type Description
    cust_id String Unique customer identifier
    cc_number String Credit card number
    cc_expiry_date String Credit card expiry date

    Schema for customers.json

    Field Data Type Description
    cust_id String Unique customer identifier
    fname String First name
    lname String Last name
    gender String Gender
    address String Full address
    dob String Date of birth (YYYY/MM/DD)
    phone String Phone number
    email String Email address
  3. Open S3 console, choose General purpose buckets in the navigation pane.
  4. Locate the S3 bucket with a name starting with bdb4834-raw-bucket. This bucket is created by the CloudFormation stack and can also be found under the stack’s Resources tab in the CloudFormation console.
  5. Choose the bucket name to open it, and follow these steps to create the required prefix:
    1. Choose Create folder.
    2. Enter the folder name as mwaa/blog/partition_dt=YYYY-MM-DD/, replacing YYYY-MM-DD with the actual date to be used for the partition.
    3. Choose Create folder to confirm.
  6. Upload the sample data files from the location to the s3 raw bucket prefix.
  7. As soon as the files are uploaded, the on_put object event on the raw bucket invokes thebdb4834_mwaa_trigger_process_s3_files lambda which triggers the process_raw_to_formatted_stg MWAA DAG.
    1. In the Airflow UI, choose the process_raw_to_formatted_stg DAG to view execution status. This DAG converts the file formats to parquet and typically completes within a few seconds.
    2. (Optional) To check the Lambda execution details:
      1. On the AWS Lambda Console, choose Functions in the navigation pane.
      2. Select the function named bdb4834_mwaa_trigger_process_s3_files.
  8. Validate the parquet files are created in formatted bucket (bucket name starting with bdb4834-formatted) under the respective data object prefix.
  9. Before proceeding further, re-upload the Lake Formation metadata file in MWAA bucket.
    1. Open the S3 console, choose General purpose buckets in the navigation pane.
    2. Search for the bucket starting with bdb4834-mwaa-bucket
    3. Choose the bucket name and go to the lakeformation prefix. Download the file named lf_tags_metadata.json. Now, re-upload the same file to the same location.
      Note: This re-upload is necessary because the Lambda function is configured to trigger on file arrival. When the resources were initially created by the CloudFormation stack, the files were simply moved to S3 and did not trigger the Lambda. Re-uploading the file ensures the Lambda function is executed as intended.
    4. As soon as the file is uploaded, the on_put object event on the MWAA bucket invokes the lf_tags_automation lambda, which creates the Lake Formation (LF) tags as defined in the metadata file and grants access to the specified AWS Identity and Access Management (IAM) roles for read/write.
    5. Validate that the LF-Tags have been created by visiting the Lake Formation Console. In the left navigation pane, choose Permissions, and then select LF-Tags and permissions.
  10. Now, run the crawler DAG to create/update the catalog tables: crawler-daily-run
    1. In the Airflow UI select the crawler-daily-run DAG and choose Trigger DAG to execute it.
    2. This DAG is configured to trigger Glue Crawler which crawls the formatted_stg prefix under the bdb4834-formatted s3 bucket to create catalog tables as per the prefixes available under the formatted_stg prefix.
      bdb4834-formatted-bucket-<aws-account-id>-<region>/formatted_stg/
      

    3. Monitor the execution of the crawler-daily-run DAG until it completes, which typically takes 2 to 3 minutes. The crawler run status can be verified in the AWS Glue Console by following these steps:
      1. Open the AWS Glue Console.
      2. In the left navigation pane, choose Crawlers.
      3. Search for the crawler named bdb4834-formatted-stg-crawler.
      4. Check the Last run status column to confirm the crawler executed successfully.
      5. Choose the crawler name to view additional run details and logs if needed.

    4. Once the crawler has completed successfully, in the left-hand panel, choose Databases and select the bdb4834_formatted_stg database to view the created tables, which should appear as showing in the following image. Optionally, select the table’s name to view its schema, and then select Table data to open Athena for data analysis. (An error may appear when querying data using Athena due to Lake Formation permissions. Review the Governance using Lake Formation section in this post to resolve the issue.)

Note: If this is the first time Athena is being used, a query result location must be configured by specifying an S3 bucket. Follow the instructions in the AWS Athena documentation to set up the S3 staging bucket for storing query results.

Run model through DAG in MWAA

In this section, we cover how dbt models run in MWAA using Athena adapter to create Glue-catalogued tables and how auditing is done for each run.

  1. After creating the tables in the Glue database using the AWS Glue Crawler in the previous steps, we can now proceed to run the dbt models in MWAA. These models are stored in S3 in the form of SQL files, located at the S3 prefix: bdb4834-mwaa-bucket-<account_id>-us-east-1/dags/dbt/models/
    The following are the dbt models and their functionality:

    • mwaa_blog_cards_exception.sql This model reads data from the mwaa_blog_cards table in the bdb4834_formatted_stg database and writes records with data quality issues to the mwaa_blog_cards_exception table in the bdb4834_formatted_exception database.
    • mwaa_blog_customers_exception.sql This model reads data from the mwaa_blog_customers table in the bdb4834_formatted_stg database and writes records with data quality issues to the mwaa_blog_customers_exception table in the bdb4834_formatted_exception database.
    • mwaa_blog_cards.sql This model reads data from the mwaa_blog_cards table in the bdb4834_formatted_stg database and loads it into the mwaa_blog_cards table in the bdb4834_formatted database. If the target table does not exist, dbt automatically creates it.
    • mwaa_blog_customers.sql This model reads data from the mwaa_blog_customers table in the bdb4834_formatted_stg database and loads it into the mwaa_blog_customers table in the bdb4834_formatted database. If the target table does not exist, dbt automatically creates it.
  2. The mwaa_blog_cards.sql model processes credit card data and depends on the mwaa_blog_customers.sql model to complete successfully before it runs. This dependency is necessary because certain data quality checks—such as referential integrity validations between customer and card records—must be performed beforehand.
    • These relationships and checks are defined in the schema.yml file located in the same S3 path: bdb4834-mwaa-bucket-<account_id>-us-east-1/dags/dbt/models/. The schema.yml file provides metadata for dbt models, including model dependencies, column definitions, and data quality tests. It utilizes macros like get_dq_macro.sql and dq_referentialcheck.sql (found under the macros/ directory) to enforce these validations.

    As a result, dbt automatically generates a lineage graph based on the declared dependencies. This visual graph helps orchestrate model execution order—ensuring models like mwaa_blog_customers.sql run before dependent models such as mwaa_blog_cards.sql, and identifies which models can execute in parallel to optimize the pipeline.

  3. As a pre-step before running models, choose the trigger DAG button for create-audit-table to create audit table for storing run details for each model.
  4. Trigger the blog-test-data-processing DAG in the Airflow UI to start the Model run.
  5. Choose blog-test-data-processing to see the execution status. This DAG runs the models in order and creates Glue catalogued iceberg tables. The flow diagram of a DAG from Airflow UI can be found by choosing Graph after choosing DAG.

    1. The exception models puts the failed records under exception prefix in S3:
      bdb4834-formatted-bucket-<aws-account-id>-<region>/formatted_exception/

      Records that failed are found in an added column, tests_failed, where all the data quality checks that failed for that particular row are added, separated by a pipe (‘|’). (For the mwaa_blog_customers_exception two exception records are found in the table.)

    2. The passed records are put under formatted prefix in S3.
      bdb4834-formatted-bucket-<aws-account-id>-<region>/formatted/

    3. For each run, a run audit is captured in the audit table with execution details like model_nm, process_nm, execution_start_date, execution_end_date, execution_status, execution_failure_reason, rows_affected.
      Find the data in S3 under the prefix bdb4834-formatted-bucket-<aws-account-id>-<region>/audit_control/
    4. Monitor the execution until the DAG completes, which can take up to 2-3 mins. The execution status of the DAG can be seen in the left panel after opening the DAG.
    5. Once the DAG has completed successfully, open the AWS Glue console and select Databases. Select the bdb4834_formatted database, which should create three tables, as shown in the following image.
      Optionally, choose Table data to access Athena for data analysis.
    6. Choose bdb4834_formatted_exception database from under Databases in AWS Glue console, which should create two tables as shown in the following image.
    7. Each model is assigned LF tags through the config block of model itself. Therefore, when the iceberg tables are created through dbt, LF tags are attached to the tables after the run completes.

      Validate the LF tags attached to the tables by visiting the AWS Lake Formation console. In the left navigation pane, choose Tables and look for mwaa_blog_customers or mwaa_blog_cards table under bdb4834_formatted database. Select any table among the two and under Actions, choose Edit LF tags and the tags are attached, as shown in the following screen shot.

    8. Similarly, for the bdb4834_formatted_exception database, select any one of the exception tables under the bdb4834_formatted_exception database and the LF tags are attached.
    9. Run SQL queries on the tables created by opening the Athena console and running Analytical queries on the tables created above.Sample SQL queries:
      SELECT * FROM bdb4834_formatted.mwaa_blog_cards;
      Output: Total 30 rows

      SELECT * FROM bdb4834_formatted_exception.mwaa_blog_customers_exception;
      Output: Total 2 records

Governance using Lake Formation

In this section, we show how assigning Lake Formation permissions and creating LF tags is automated using the metadata file.Below is a metadata file structure, which is needed for reference when uploading the metadata file for Lake Formation in Airflow S3 bucket, inside the Lake Formation prefix.

Metadata file structure-
{
    "role_arn": "<<IAM_ROLE_ARN>>",
    "access_type": "GRANT",
    "lf_tags": [
      {
        "TagKey": "<<LF_tag_key>>",
        "TagValues": ["<<LF_tag_values>>"]
      }
    ],
	  "named_data_catalog": [
      {
        "Database": "<<Database_Name>>",
        "Table": ""<<Table_Name>>"
      }
    ],
    "table_permissions": ["SELECT", "DESCRIBE"]
  }

Components of the metadata file

  • role_arn: The IAM role that the Lambda function assumes to perform operations.
  • access_type: Specifies whether the action is to grant or revoke permissions (GRANT, REVOKE).
  • lf_tags: Tags used for tag-based access control (TBAC) in Lake Formation.
  • named_data_catalog: A list of databases and tables on which Lake Formation permissions or tags are applied to.
  • table_permissions: Lake Formation-specific permissions (e.g., SELECT, DESCRIBE, ALTER, etc.).

Lambda function bdb4834-lf-tags-automation parses this JSON and grants the required LF tags to the role with given table permissions.

  1. To update the metadata file, download it from the MWAA bucket (lakeformation prefix)
    bdb4834-mwaa-bucket-<<ACCOUNT_NO>>-<<REGION>>/lakeformation/lf_tags_metadata.json

  2. Add a JSON object with the metadata structure defined above, mentioning the IAM role ARN and the tags and tables to which access needs to be granted.
    Example:Let’s assume below is how the metadata file initially looks like:

    
    	[
    	{
        "role_arn": "arn:aws:iam::XXX:role/aws-reserved/sso.amazonaws.com/XX ",
        "access_type": "GRANT",
        "lf_tags": [
          {
            "TagKey": " blog",
            "TagValues": ["bdb-4834"]
          }
        ],
        "named_data_catalog": [],
        "table_permissions": ["SELECT", "DESCRIBE"]
      }
    ]

    Below is the json object that has to be added in the above metadata file:

    
    {
              "role_arn": "arn:aws:iam::XXX:role/aws-reserved/sso.amazonaws.com/XX ",
              "access_type": "GRANT",
              "lf_tags": [],
              "named_data_catalog": [
              {
                "Database": " bdb4834_formatted",
                "Table": "audit_control"
              },
              {
                "Database": " bdb4834_formatted_stg",
                "Table": "*"
              }
             ],
             "table_permissions": ["SELECT", "DESCRIBE"]}
    
    
    

    So now, the final metadata file should look like:

    
    [
      {
        "role_arn": "arn:aws:iam::XXX:role/aws-reserved/sso.amazonaws.com/XX ",
        "access_type": "GRANT",
        "lf_tags": [
          {
            "TagKey": "blog",
            "TagValues": ["bdb-4834"]
          }
        ],
        "named_data_catalog": [],
        "table_permissions": ["SELECT", "DESCRIBE"]
      },
      {
        "role_arn": "arn:aws:iam::XXX:role/aws-reserved/sso.amazonaws.com/XX ",
        "access_type": "GRANT",
        "lf_tags": [],
        "named_data_catalog": [
          {
            "Database": " bdb4834_formatted",
            "Table": "audit_control"
          },
          {
            "Database": " bdb4834_formatted_stg",
            "Table": "*"
          }
        ],
        "table_permissions": ["SELECT", "DESCRIBE"]
      }
    ]

  3. Upon uploading this file at the same location (bdb4834-mwaa-bucket-<<ACCOUNT_NO>>-<<REGION>>/lakeformation/) in S3, the lf_tags_automation lambda is triggered to create LF tags if they don’t exist and then it assigns those tags to the IAM role ARN and also grants permission to the IAM role ARN using named_data_catalog as defined.

    To verify the permissions, go to the Lake Formation console and choose Tables under Data Catalog and search for the table name.

To check LF-Tags, choose the table name and under the LF tags section, all the tags are found attached to this table.

This metadata file used as a structured input to an AWS Lambda function automates the following to perform automated, consistent, and scalable data access governance across the AWS Lake Formation environments:

  • Granting AWS Lake Formation (LF) permissions on Glue Data Catalog resources (like databases and tables).
  • Creating Lake Formation Tags and Applying Lake Formation tags (LF-Tags) for tag-based access control (TBAC).

Explore more on dbt

Now that the deployment includes a bdb4834-published S3 bucket and a published Catalog database, robust dbt models can be built for data transformation and curation.

Here’s how to implement a complete dbt workflow:

  • Start by developing models that follow this pattern:
    • Read from the formatted tables in the staging area
    • Apply business logic, joins, and aggregations
    • Write clean, analysis-ready data to the published schema
  • Tagging for automation: Use consistent dbt tags to enable automatic DAG generation. These tags trigger MWAA orchestration to automatically include new models in the execution pipeline.
  • Adding new models: When working with new datasets, refer to existing models for guidance. Apply appropriate LF tags for data access control. The new LF tags can also now be used for permissions.
  • Enable DAG execution: For new datasets, update the MWAA metadata file to include a new JSON entry. This step is necessary to generate a DAG that executes the new dbt models.

This approach ensures the dbt implementation scales systematically while maintaining automated orchestration and proper data governance.

Clean up

1. Open the S3 console and delete all objects from below buckets:

  • bdb4834-raw-bucket-<aws-account-id>-<region>
  • bdb4834-formatted -bucket-<aws-account-id>-<region>
  • bdb4834-mwaa-bucket-<aws-account-id>-<region>
  • bdb4834-published-bucket-<aws-account-id>-<region>

To delete all objects, choose the bucket name, select all objects and choose Delete.

After that, type ‘permanently delete’ in the text box and choose Delete Objects.

Do this for all three buckets mentioned above.

2. Go to the AWS Cloudformation console, choose you’re the stack name and select Delete. It may take approximately 40 mins for the deletion to complete.

Recommendations

When using dbt with MWAA, some typical challenges include worker resource exhaustion, dependency management issues, and in some rare cases, issues like DAGs disappearing and re-appearing when there are a large number of dynamic DAGs being created from a single python script.

To mitigate these issues, follow these best practices:

1. Scale the MWAA environment appropriately by upgrading the environment class as required.

2. Use custom requirements.txt and proper dbt adapter configuration to ensure consistent environments.

3. Set airflow configuration parameters to tune the performance of MWAA.

Conclusion

In this post, we explored the end-to-end setup of a governed data lake using MWAA and dbt which improved data quality, security, and compliance, leading to better decision-making and increased operational efficiency. We also covered how to build custom dbt frameworks for auditing and data quality, automate Lake Formation access control, and dynamically generate MWAA DAGs based on dbt tags. These capabilities enable a scalable, secure, and automated data lake architecture, streamlining data governance and orchestration.

For further exploring, refer to From data lakes to insights: dbt adapter for Amazon Athena now supported in dbt Cloud


About the authors

Muralidhar Reddy

Muralidhar Reddy

Muralidhar is a Delivery Consultant at Amazon Web Services (AWS), helping customers build and implement data analytics solution. When he’s not working, Murali is an avid bike rider and loves exploring new places.

Abhilasha Agarwal

Abhilasha Agarwal

Abhilasha is an Associate Delivery Consultant at Amazon Web Services (AWS), support customers in building robust data analytics solutions. Apart from work, she loves cooking and trying out fun outdoor experiences.