Post Syndicated from Avijit Goswami original https://aws.amazon.com/blogs/big-data/zero-copy-access-to-apache-iceberg-tables-in-amazon-s3-from-salesforce-data-360-using-the-iceberg-rest-endpoint-from-aws-glue-data-catalog/

Companies increasingly need to query and analyze data across platforms without the cost and complexity of moving it. Salesforce and AWS have collaborated to make this possible by providing Zero Copy access to Apache Iceberg tables stored in Amazon Simple Storage Service (Amazon S3) directly from Salesforce Data 360, using the Iceberg REST endpoint from AWS Glue Data Catalog with data access managed by AWS Lake Formation. This integration helps customers federate their Amazon S3 data lakes with Data 360, preserving data governance, freshness, and business semantics without replication.

Zero Copy file federation plays an important role in activating applications and experiences. By removing the need to physically move or copy data, and connecting to data at the storage level, it addresses key challenges including:

• Cost efficiency – Reduce storage duplication costs and minimize the compute resources required for data pipelines.
• High scale – Access data with near-native performance at scale through in-Region access.
• Enhanced agility – Access and analyze data in real time, accelerating time-to-insight and supporting faster response to evolving business needs.
• Streamlined operations – Remove the complexity of building and maintaining intricate data pipelines, clearing up valuable data engineering resources.

In this post, we demonstrate how AWS and Salesforce customers can access their enterprise data lakes on AWS from Data 360 using Zero Copy file federation.

What is Data 360?

Data 360 is the real-time data engine that activates trusted context across the entire Salesforce platform. It connects all your enterprise data — data warehouses, data lakes, third-party signals, and more — to the business context, logic, and governance that already live in Salesforce, without moving or copying it. With Zero Copy federation, your teams and AI agents always operate from a complete, current, and trusted picture of your business in the moment it’s needed. It serves as the essential system of context for Agentforce, enabling agents to reliably get real work done.

What is Apache Iceberg?

Apache Iceberg is a high-performance, open table format for huge analytic datasets that brings the reliability and simplicity of SQL tables to big data. It’s a thriving open source project under the Apache Software Foundation. Data engineers use Apache Iceberg because it’s fast, efficient, and reliable at any scale and keeps records of how datasets change over time. Apache Iceberg offers integrations with popular data processing frameworks such as Apache Spark, Apache Flink, Apache Hive, Presto, and more.

Why Amazon S3 for Apache Iceberg data lakes?

Amazon S3 is regarded as the best place to build data lakes because of its durability, availability, scalability, security, compliance, and audit capabilities, and its ability to integrate with a broad portfolio of AWS and third-party tools for data ingestion and processing. Apache Iceberg was designed and built to interact with Amazon S3, and provides support for many Amazon S3 features as listed in the Iceberg documentation.

What is Zero Copy file federation?

File federation, also termed catalog federation, uses the Data Catalog to communicate with remote catalog systems to discover catalog objects and to authorize access to their data in Amazon S3. When you query a remote Iceberg table, the Data Catalog discovers the latest table information in the remote catalog at query runtime, getting the table’s Amazon S3 location, current schema, and partition information. Your analytics engine then uses this information to access Iceberg data files directly from Amazon S3, and Lake Formation manages access to the table and data by vending scoped credentials to the table data stored in Amazon S3. This approach avoids metadata and data duplication while providing real-time access to remote Iceberg tables through your preferred AWS analytics engines.

Solution overview

Apache Iceberg file federation lets Data 360 directly query data stored in Amazon S3 without copying or moving the data. This Zero Copy approach provides several benefits:

• Real-time access to Amazon S3 data from Salesforce.
• Reduced data movement and storage costs.
• Simplified data architecture.
• Improved data freshness.

The following diagram illustrates the architecture of the integration between Data 360 and Amazon S3 using Apache Iceberg file federation.

Key components:

1. Amazon S3 stores the source data in Apache Iceberg format.
2. AWS Glue Data Catalog maintains the metadata for Iceberg tables.
3. AWS Glue Iceberg REST endpoint provides RESTful access to Iceberg tables.
4. AWS Lake Formation manages metadata and underlying data access for Amazon S3-based data lakes.
5. Data 360 processes and analyzes the data.
6. Apache Iceberg connector provides direct access to query Amazon S3 data from Salesforce.

Walkthrough

The following walkthrough shows you how to set up Zero Copy file federation.

Prerequisites

Before you begin, you need the following:

• An AWS account.
• A data-lake-admin role following the steps in Lake Formation personas and IAM permissions reference, with access to AWS Identity and Access Management (IAM), AWS Glue Data Catalog, AWS Lake Formation, Amazon S3, and Amazon Athena.
• A Data 360 instance.
• Familiarity with AWS and Data 360.
• A user-defined IAM role that Lake Formation will use to register the bucket. To create one, see Requirements for roles used to register locations.

Configure your AWS environment

Set up an Amazon S3 bucket and Iceberg table

Sign in as the data lake admin and complete the following steps:

1. Open the Amazon S3 console.
2. Choose Create bucket to create a bucket.
3. For Bucket type, choose General purpose, provide a Bucket name, and choose Create bucket.
4. In the bucket, create two prefixes by choosing Create folder.
5. Name the prefixes athena_iceberg and athena_results.
6. Inside the athena_iceberg prefix, create another prefix named customer_iceberg.

Create an Iceberg table using Athena

1. Open the Amazon Athena console.
2. Choose Query your data in Athena console, then choose Launch query editor.
3. In Athena, choose Edit settings.
4. Set s3://<your-bucket-name>/athena_results/ as the Location of query result, then choose Save. Replace <your-bucket-name> with your bucket name.
5. Choose Editor to return to the query editor page.
6. To create the database, copy the following query into the query editor and choose Run. You need to be in the Athena Query Editor to run the following commands.

   create database iceberg_db;

7. To create the Iceberg table, copy the following query into the query editor, replace <s3 bucket location> with your Amazon S3 bucket location hosting the Iceberg table, and choose Run.

   CREATE TABLE iceberg_db.churn (
       state string,
       account_length int,
       area_code string,
       phone string,
       intl_plan string,
       vmail_plan string,
       vmail_message int,
       day_mins double,
       day_calls int,
       day_charge double,
       eve_mins double,
       eve_calls int,
       eve_charge double,
       night_mins double,
       night_calls int,
       night_charge double,
       intl_mins double,
       intl_calls int,
       intl_charge double,
       custserv_calls int,
       churn boolean)
   LOCATION 's3://<s3 bucket location>/iceberg/churn'
   TBLPROPERTIES (
       'table_type'='iceberg',
       'compression_level'='3',
       'format'='PARQUET',
       'write_compression'='ZSTD'
   );

8. Insert some records into the table.

   -- Sample data insert for "iceberg_db"."churn"
   -- Execute this in the Athena console
   INSERT INTO "iceberg_db"."churn" VALUES
   ('KS', 128, '415', '382-4657', 'no', 'yes', 25, 265.1, 110, 45.07, 197.4, 99, 16.78, 244.7, 91, 11.01, 10.0, 3, 2.70, 1, false),
   ('OH', 107, '415', '371-7191', 'no', 'yes', 26, 161.6, 123, 27.47, 195.5, 103, 16.62, 254.4, 103, 11.45, 13.7, 3, 3.70, 1, false),
   ('NJ', 137, '415', '358-1921', 'no', 'no', 0, 243.4, 114, 41.38, 121.2, 110, 10.30, 162.6, 104, 7.32, 12.2, 5, 3.29, 0, false),
   ('OH', 84, '408', '375-9999', 'yes', 'no', 0, 299.4, 71, 50.90, 61.9, 88, 5.26, 196.9, 89, 8.86, 6.6, 7, 1.78, 2, false),
   ('OK', 75, '415', '330-6626', 'yes', 'no', 0, 166.7, 113, 28.34, 148.3, 122, 12.61, 186.9, 121, 8.41, 10.1, 3, 2.73, 3, false),
   ('AL', 118, '510', '391-8027', 'yes', 'no', 0, 223.4, 98, 37.98, 220.6, 101, 18.75, 203.9, 118, 9.18, 6.3, 6, 1.70, 0, false),
   ('MA', 121, '510', '355-9993', 'no', 'yes', 24, 218.2, 88, 37.09, 348.5, 108, 29.62, 212.6, 118, 9.57, 7.5, 7, 2.03, 3, false),
   ('MO', 147, '415', '329-9001', 'yes', 'no', 0, 157.0, 79, 26.69, 103.1, 94, 8.76, 211.8, 96, 9.53, 7.1, 4, 1.92, 0, false),
   ('WV', 141, '415', '330-8173', 'yes', 'yes', 37, 258.6, 84, 43.96, 222.0, 111, 18.87, 326.4, 97, 14.69, 11.2, 5, 3.02, 0, false),
   ('IN', 65, '415', '329-6603', 'no', 'no', 0, 129.1, 137, 21.95, 228.5, 83, 19.42, 208.8, 111, 9.40, 12.7, 6, 3.43, 4, true),
   ('RI', 74, '415', '344-9230', 'no', 'no', 0, 187.7, 127, 31.91, 163.4, 148, 13.89, 196.0, 94, 8.82, 9.1, 5, 2.46, 0, false),
   ('IA', 168, '408', '363-1107', 'no', 'no', 0, 275.8, 90, 46.89, 230.0, 73, 19.55, 191.3, 57, 8.61, 9.9, 3, 2.67, 4, true),
   ('MT', 95, '510', '394-8006', 'no', 'no', 0, 113.2, 96, 19.24, 269.9, 107, 22.94, 229.1, 87, 10.31, 7.1, 4, 1.92, 1, false),
   ('NY', 62, '415', '371-5765', 'no', 'no', 0, 236.5, 127, 40.21, 145.3, 101, 12.35, 225.0, 103, 10.13, 12.0, 1, 3.24, 5, true),
   ('TX', 109, '408', '356-2992', 'no', 'yes', 33, 190.7, 114, 32.42, 218.2, 111, 18.55, 156.5, 122, 7.04, 11.6, 5, 3.13, 1, false),
   ('CA', 155, '510', '328-8230', 'no', 'no', 0, 197.3, 78, 33.54, 160.2, 86, 13.62, 280.1, 90, 12.60, 8.8, 2, 2.38, 2, false),
   ('WA', 132, '415', '382-1011', 'yes', 'no', 0, 302.7, 67, 51.46, 212.0, 105, 18.02, 265.5, 82, 11.95, 10.3, 4, 2.78, 3, true),
   ('FL', 88, '408', '344-5678', 'no', 'yes', 18, 145.3, 95, 24.70, 187.6, 92, 15.95, 198.2, 108, 8.92, 9.4, 3, 2.54, 0, false),
   ('CO', 201, '510', '367-4321', 'no', 'no', 0, 312.5, 142, 53.13, 178.9, 76, 15.21, 145.7, 95, 6.56, 14.2, 8, 3.83, 6, true),
   ('GA', 56, '415', '390-2244', 'no', 'yes', 12, 178.4, 101, 30.33, 205.1, 119, 17.43, 230.8, 100, 10.39, 8.0, 2, 2.16, 1, false);

Register the bucket with Lake Formation in Lake Formation mode

To use Lake Formation permissions for access control to the churn table, you must register the location. To do that, complete the following actions:

1. Open the AWS Lake Formation console.
2. In the navigation pane under Administration, choose Data lake locations.
3. Choose Register location and enter the following information:
   1. For S3 URI, enter s3://<s3 bucket location>/iceberg/churn. Replace <s3 bucket location> with your Amazon S3 bucket location hosting the Iceberg table.
   2. For IAM role, choose the user-defined IAM role that you created in the prerequisites.
   3. For Permission mode, choose Lake Formation.
4. Choose Register location.

Enable third-party integration in Lake Formation

From the Lake Formation console, enable full table access for external engines.

1. Open the AWS Lake Formation console.
2. On the left pane, expand the Administration section.
3. Choose Application integration settings and select Allow external engines to access data in Amazon S3 locations with full table access.
4. Choose Save.

Set up an IAM user for third-party access

1. Open the IAM console.
2. From the left navigation menu, choose Policies, then choose Create policy. Choose JSON and paste the following policy:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": "lakeformation:GetDataAccess",
               "Resource": "*"
           }
       ]
   }

3. Choose next, provide a name for the policy, and choose Create policy.
4. From the left navigation menu, choose Users, then choose Create user.
5. For username, enter data_cloud_user, choose next, and choose Attach policies directly.
6. Choose AWSGlueServiceRole and the policy that you created in step 3. Choose next and Create user.
7. Choose the user, then choose Security credentials to create an access key.
8. Scroll down and choose Create access key, choose Applications running outside AWS, and choose Create access key.
9. Copy the access key and secret access key, and save them securely. You need these to configure the connector in Data 360.

Set up Lake Formation resource permissions for third-party data access

1. Open the Lake Formation console.
2. From the left navigation under Data Catalog, choose Databases, then choose the athena_iceberg_db database.
3. From the Actions menu, choose Permissions, Grant.
4. In Principals, choose IAM users and roles, and from the menu choose data_cloud_user, which you just created in IAM.
5. Scroll down to grant permissions by choosing All tables, then choose Select and Describe permissions for the tables.
6. Choose Grant to apply the permissions.

Set up Apache Iceberg file federation in Data 360

Create and configure the connection

1. Navigate to Salesforce Setup. For instructions, see Set Up the AWS Glue Data Catalog Connection.
2. In Data Cloud, choose Setup, then choose Data Cloud Setup.
   
3. Under External Integrations, choose Other Connectors.
4. Choose New.
5. On the Source tab, choose AWS Glue Data Catalog, then choose Next.
   
6. Complete the following information shown in the following screen:
   1. In the Authentication Details section, enter the AWS access key ID and AWS secret access key for the IAM user. Make sure that the IAM user has a policy that grants the user read-only access to AWS Glue Data Catalog. Use Lake Formation to configure storage credential vending. This approach is for AWS Glue Data Catalog to vend temporary credentials at run time so that Data 360 can access the underlying storage bucket.
   2. For Catalog URL, enter the URL of AWS Glue Data Catalog. See Connecting to the Data Catalog by using AWS Glue Iceberg REST endpoint.
   3. For Catalog ID, enter the 12-digit AWS account ID linked to AWS Glue Data Catalog.
   4. For Signing Region, enter the host AWS Region where AWS Glue Data Catalog is located.
   5. For Signing Service, enter glue. Data 360 requires the Signing Service, in addition to the AWS access key ID, secret access key, and Signing Region, to sign requests to AWS Glue Data Catalog by using AWS Signature Version 4.
   6. Test the connection and check for the success message.
   7. Save the connection details.

   

7. After the configuration is complete and saved, the new AWS Glue Data Catalog connection shows up with “Active” status in the Connectors screen.

Create and configure the data stream

1. In Data Cloud, on the Data Streams tab, choose New.
2. Under Other Sources, choose the AWS Glue Data Catalog source, then choose Next.
3. From the menus, choose the connection that you just set up, choose a database in your AWS Glue catalog where you have an Iceberg table, choose the table that you want to stream, and choose Next.
4. Enter the object name and object API name. For more information, see Data Lake Object Naming Standards.
5. Choose the category to specify the type of data to ingest. For more information, see Category.
6. Choose a primary key to uniquely identify the incoming records. For more information, see Primary Key.
7. Choose the source fields you want to ingest, then choose Next. Fields with convertible data types are listed under Supported Fields.
8. Choose the relevant data space. Choose “Default” if you don’t have any other data space provisioned in your org. For more information, see Data Spaces.
9. Choose Deploy.
   
10. After the setup is complete, the new data stream appears in your Data Cloud environment.
    
11. The data stream is ready. You can now go to the Data Explorer in your Data Cloud environment and start viewing the Iceberg tables that reside in your external AWS account.

Best practices and considerations

• Use IAM roles with least-privilege access. Grant only the specific permissions each service or user needs.
• Implement appropriate Amazon S3 bucket policies. Define bucket-level policies that restrict access by AWS account, VPC endpoint, or IP range.
• Monitor access patterns. Enable Amazon S3 server access logging or AWS CloudTrail data events to track who reads from and writes to your table buckets.
• Optimize Iceberg table partitioning. Choose partition keys that align with your most common query filters.
• Consider data access patterns. Design your table layout around how data is actually queried.
• Implement lifecycle policies for Amazon S3 objects. Configure Amazon S3 lifecycle rules to transition older data files to other storage classes.
• Use appropriate Iceberg file compaction strategies. Run compaction regularly to merge small files produced by streaming or frequent batch appends.
• Monitor data transfer costs. Track cross-Region and internet egress charges using AWS Cost Explorer as applicable.

Clean up

After you finish testing, clean up all the resources in your AWS account that you created (including the Amazon S3 bucket, Athena tables, and other AWS services) to avoid recurring costs.

Conclusion

By implementing Apache Iceberg file federation between Data 360 and Amazon S3, you can create a more efficient and streamlined data architecture. This solution gives you real-time access to Amazon S3 data while using the analytics capabilities of Data 360. As businesses continue to prioritize data-driven decision-making, Zero Copy data sharing plays an important role in unlocking the full potential of customer data across platforms.

To learn more, review the following resources:

• Apache Iceberg documentation
• AWS Glue Iceberg capabilities
• Salesforce Data Cloud documentation
• AWS Big Data Blog

---

About the authors