All posts by Nitin Kumar

Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake

Post Syndicated from Nitin Kumar original https://aws.amazon.com/blogs/big-data/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake/

A cloud data lake is a scalable, low-cost place to store data from many different sources. Over the years, data lakes built on Amazon Simple Storage Service (Amazon S3) have become a common home for enterprise data, used by teams who query it for analytics and machine learning. However, data lakes have historically made it hard to update and delete records, keep transactions isolated, and handle concurrent changes safely. Open table formats such as Apache Hudi, Delta Lake, and Apache Iceberg were created to solve these problems. They add ACID transactions and row-level updates and delete, along with snapshot history and schema enforcement that make data lakes easier to audit and govern.

As organizations’ data architectures mature and evolve, the need to migrate between these table formats becomes increasingly common. This transition is often driven by changing requirements, performance considerations, or the desire to use newer features. Among these migrations, the shift from Apache Hudi to Apache Iceberg has gained particular attention, as organizations seek to capitalize on Iceberg’s modern architecture and advanced capabilities.

In this post, we show how to convert Apache Hudi tables to Apache Iceberg, using a Jupyter notebook running on AWS Glue 5.1 interactive sessions.

Benefits of migration to Iceberg

Enhanced partition evolution

Iceberg supports dynamic changes to partitioning schemes without requiring complete data rewrites. This capability is valuable in evolving data environments where query patterns change over time. You can adjust your partitioning strategy to optimize performance, reducing the operational overhead traditionally associated with such changes. Your data lake can adapt to changing business requirements without costly reorganization efforts.

Hidden partitioning

In addition to standard partitioning on an existing column, Iceberg supports hidden partitioning, which decouples the physical partition layout from the columns that queries filter on. The table specification declares a partition transform, which is a rule that tells Iceberg how to derive the partition value from a source column, such as a time bucket or a hash of the column value. The transform is defined once at the table level, and Iceberg applies it automatically on every write. This removes a long-standing source of friction on both sides of the table: writers no longer need to extract and maintain a separate partition column alongside the real one at ingestion time, and queries no longer need to carry an extra predicate on that derived column to get good performance.

Openness with the Iceberg REST Catalog

The Iceberg REST Catalog specification defines a standard HTTP protocol for catalog operations. Any engine that speaks the protocol can talk to any catalog that implements it, without a catalog-specific client library or implementation JAR on the class path. Integration moves from a code and dependency problem to a configuration problem, making Iceberg a strong fit for multi-engine data architectures.

Deep integration with AWS services

Iceberg has native integration across the AWS analytics ecosystem. Amazon Athena, Amazon Redshift, Amazon EMR, and AWS Glue all read and write Iceberg tables directly. The AWS Glue Data Catalog provides catalog and metadata management for these tables, and Amazon SageMaker builds on the same open Lakehouse foundation, letting analytics and artificial intelligence and machine learning (AI/ML) workloads share a single copy of data across Amazon S3 data lakes (including Amazon S3 Tables), with SageMaker Feature Store supporting an Iceberg offline store for faster training dataset extraction. On the ingestion side, Amazon Data Firehose delivers data into Iceberg tables from sources such as Direct PUT, Amazon Kinesis Data Streams, Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon CloudWatch Logs, and AWS IoT, with optional on-the-fly record transformation by using AWS Lambda. No-code Streaming Tables map a single Kinesis Data Streams stream or a single Amazon MSK (Express brokers) Kafka topic to a single Amazon S3 Tables Iceberg table, with the target table auto-created and partitioning configured on the delivery channel.

AWS managed features

AWS provides managed capabilities specifically designed for Iceberg. These managed features reduce operational overhead while maintaining performance and cost efficiency, so teams can focus on deriving insights rather than managing infrastructure.

  • AWS Glue Iceberg table optimizer – Automated compaction, snapshot management, and orphan file cleanup without manual intervention on Iceberg tables in General S3 buckets.
  • Amazon S3 Tables – Fully managed Iceberg tables with automatic optimization and maintenance.
  • Zero-ETL integration – Managed, incremental replication from Amazon DynamoDB and third-party enterprise applications such as Salesforce, SAP, ServiceNow, and Zendesk into Iceberg tables in Amazon S3 (self-managed or Amazon S3 Tables), with no custom pipeline or connector code to maintain. Source changes are applied incrementally, so the target stays in sync without full reloads.
  • Iceberg materialized views – SQL-defined views materialized as native Iceberg tables, refreshed on an automatic schedule and incrementally when the source tables allow it, with change detection handled for you. Well suited to consumption-layer tables that feed dashboards, APIs, and downstream pipelines, since the work shifts from query time to the managed refresh.

These benefits make Apache Iceberg a strong choice for modern data lake implementations, offering advantages in performance, manageability, and scalability compared to traditional table formats.

In the context of migrating data to Apache Iceberg, two primary approaches exist: in-place migration and full table migration. We previously covered both in the blog post “Migrate an existing data lake to a transactional data lake using Apache Iceberg” which walks through the methodology for traditional Hive-style tables, meaning partitioned Apache Parquet, Apache ORC, or Apache Avro datasets registered in a Hive-compatible catalog. The same two approaches apply when the starting point is an Apache Hudi data lake, which comes in two flavors, Copy-on-Write (CoW) and Merge-on-Read (MoR).

In-place migration

This approach generates fresh Iceberg metadata on top of the existing data files of the source table, without rewriting the data itself. The Parquet files of a Hive, Hudi, or Delta Lake table stay in place, and a new or existing Iceberg table is pointed at them with its own metadata layer. The main advantage is cost and speed: no data rewrite, so the migration is fast and cheap on both compute and storage. The main limitation is that the Iceberg table inherits the source table’s schema, partitioning, and physical file shape (file sizes, row groups, sort order), and none of these can be changed during the migration itself.

Full table migration

In this approach, the data itself is rewritten into a new Iceberg table, typically using a writeTo() call or a CREATE TABLE AS SELECT (CTAS) statement. It works for any source an engine can read from (Hive, Hudi, Delta Lake, relational databases, and others), and because the data is being rewritten anyway, the migration step is also where you can reshape it: change the schema, apply Iceberg hidden partitioning, resort the data for pruning, and target larger file sizes than the source has. The trade-offs are cost, time, and storage: higher compute spend, longer migration, and the dataset is effectively stored twice (source and target) until the migration completes, and the source is retired. For large historical tables, a single CTAS is usually impractical, so the common pattern is to seed the table with a small CTAS and then run INSERT INTO ... SELECT partition by partition, parallelizing and checkpointing across batches.

In this post we work through both approaches end-to-end. We first create a sample Hudi event table in each flavor, CoW and MoR, then migrate them in place to Iceberg using two techniques, the Spark Iceberg add_files procedure and the Apache XTable sync tool, covering the considerations and prerequisites of each. Finally, we walk through the full table migration path, where the source data is rewritten into a new Iceberg table from scratch.

Solution overview

The following diagram illustrates the architecture that you implement through this post.

Architecture diagram of the Hudi-to-Iceberg migration workflow on AWS

Figure 1: Migration architecture showing the in-place paths (add_files and Apache XTable) and the full-table paths (CTAS and INSERT) to Amazon S3 and Amazon S3 Tables, all registered in the AWS Glue Data Catalog and queried with Amazon Athena

The workflow consists of the following steps:

  1. An AWS Glue notebook writes the source data as Apache Hudi tables to an Amazon S3 bucket and registers them in the AWS Glue Data Catalog. This creates the existing Hudi data lake that the migration starts from.
  2. The first in-place migration approach uses an AWS Glue notebook to run the Apache Iceberg add_files procedure. The procedure creates Iceberg metadata on top of the Parquet data files already in the Hudi table. The new Iceberg metadata points to the same Parquet files, so no data is copied or rewritten. This makes it a fast, low-cost migration path.
  3. The second in-place migration approach runs Apache XTable on an Amazon Elastic Compute Cloud (Amazon EC2) instance to convert the Hudi table metadata into Iceberg metadata. XTable reads the Hudi commit timeline and creates matching Iceberg metadata that points to the same Parquet files. Both table formats can then share a single copy of the data.
  4. For a full migration, an AWS Glue notebook uses CREATE TABLE AS SELECT (CTAS) or INSERT INTO statements to read the Hudi data and rewrite it as new Iceberg tables. The tables can go in a general purpose S3 bucket or in Amazon S3 Tables. This approach fully separates the new tables from the Hudi source. S3 Tables also handle table maintenance for you, such as compaction and snapshot management.
  5. The AWS Glue Data Catalog registers the Iceberg tables from all three approaches. Users can then use Amazon Athena, a serverless query service, to run SQL queries on these tables and validate the migrated data. Athena enables interactive SQL-based exploration of data directly in Amazon S3 without the need for complex infrastructure management.

Prerequisites

Before getting started, make sure you have the following:

Launch an AWS Glue Studio notebook for processing

For this post, we use an AWS Glue Studio notebook. Follow the steps in Getting started with notebooks in AWS Glue Studio to set up the notebook environment.

  1. Sign in to the AWS Management Console and open the AWS Glue console.
  2. In the Region selector (top right), choose your preferred working Region.
  3. Choose ETL Jobs in the left pane.
  4. Choose Notebook under Author using an interactive code notebook, select Start fresh, choose your IAM role, and choose Create notebook.
  5. After a minute, the Jupyter notebook editor appears. Clear all the default cells.

Create Hudi Copy-on-Write (CoW) and Merge-on-Read (MoR) sample event tables

This notebook creates Hudi tables using both Copy-on-Write (CoW) and Merge-on-Read (MoR) table types with sample events. We perform inserts, updates, and deletes, then inspect the underlying file structures to understand the key differences:

  • CoW: an update or delete fully rewrites every base Parquet file that contains an impacted row, producing a new file slice in the file group. Reads are cheap, and writes amplify with the fraction of files touched.
  • MoR: an update or delete is appended to an Avro log file alongside the base file instead of rewriting the base, and base + logs are merged at read time (or compacted into a new base file later). Cheaper writes. Costlier reads until compaction.
  1. Execute the first cell with the following configuration to set up the Spark session with Hudi and Iceberg support:

Code:

{
"glue_version": "5.1",
"worker_type": "G.1X",
"number_of_workers": 5,
"idle_timeout": 30,
"--datalake-formats" : "hudi,iceberg",
"--conf" : "spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.glue_iceberg=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_iceberg.warehouse=s3://YOUR_BUCKET/warehouse/ --conf spark.sql.catalog.glue_iceberg.type=glue --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar"
}
  1. Initialize the session and configuration. Update the bucket_name, db_name and region variables with your configurations. Reusable helper – list_table_files(table_name, label="", verbose=False) renders a table’s S3 layout as a directory tree so you can SEE how CLEAN, COMPACT and add_files change the files underneath. It is reused before/after each of those steps instead of re-coding the listing.

Code:

# ──────────────────────────────────────────────
# Source Hudi tables
# ──────────────────────────────────────────────
hudi_events_cow_table = "events_hudi_cow" # Copy-on-Write source
hudi_events_mor_table = "events_hudi_mor" # Merge-on-Read source
# ──────────────────────────────────────────────
# In-place migration targets (add_files/xtable)
# ──────────────────────────────────────────────
iceberg_cow_inplace = "events_iceberg_cow_inplace"
iceberg_mor_inplace = "events_iceberg_mor_inplace"
iceberg_cow_xtable = "events_hudi_cow_iceberg_xtable"
iceberg_mor_xtable = "events_hudi_mor_iceberg_xtable"
# ──────────────────────────────────────────────
# Full table (shadow) migration targets
# ──────────────────────────────────────────────
iceberg_cow_full_ctas = "events_iceberg_cow_full_ctas"
iceberg_cow_full_create_insert = "events_iceberg_cow_full_create_insert"
iceberg_mor_full_ctas = "events_iceberg_mor_full_ctas"
iceberg_mor_full_create_insert = "events_iceberg_mor_full_create_insert"
# ──────────────────────────────────────────────
# Infrastructure
# ──────────────────────────────────────────────
bucket_name = "demoblogiceberg-bdb5001" # S3 bucket for data
db_name = "hudi_to_iceberg_blog" # Glue Data Catalog database
region = "us-east-1"
# Create the Glue Data Catalog database
spark.sql(f"""
CREATE DATABASE IF NOT EXISTS {db_name}
LOCATION 's3://{bucket_name}/{db_name}'
""")
# Reusable helper —list_table_files()(S3 layout tree)
import boto3
s3 = boto3.client('s3', region_name=region)
# File/folder annotations keyed by classifier result
_NOTES = {
"hoodie_dir": "<-- Hudi metadata (timeline, schema, indexes)",
"hoodie_mdt": "<-- Hudi metadata table (internal index & stats, MoR)",
"hoodie_files": "<-- file listing index",
"hoodie_cstats":"<-- column-level min/max/null stats",
"hoodie_ridx": "<-- record-key -> file-group mapping",
"ice_meta_dir": "<-- Iceberg metadata layer (metadata.json, manifest lists, manifests)",
"ice_data_dir": "<-- Iceberg data layer (base Parquet files)",
"base": "<-- base Parquet file",
"log": "<-- Hudi delta log file (MoR update, uncompacted)",
"part_meta": "<-- Hudi partition marker",
"metadata_json":"<-- Iceberg table metadata (schema, snapshots)",
"manifest_list":"<-- Iceberg manifest list (snapshot -> manifests)",
"manifest": "<-- Iceberg manifest (lists data files)",
"props": "<-- table configuration & schema",
}
# Fixed-path annotations for metadata directories
_DIR_KIND = {
".hoodie": "hoodie_dir", ".hoodie/metadata": "hoodie_mdt",
".hoodie/metadata/files": "hoodie_files",
".hoodie/metadata/column_stats": "hoodie_cstats",
".hoodie/metadata/record_index": "hoodie_ridx",
"metadata": "ice_meta_dir", "data": "ice_data_dir",
}
def _fmt_size(n):
"""Human-readable byte size."""
for u in ("B", "KB", "MB", "GB", "TB"):
if n < 1024 or u == "TB":
return f"{n:,.0f} {u}" if u == "B" else f"{n:,.1f} {u}"
n /= 1024.0
def _leaf_kind(name):
"""Classify a leaf file by name -> annotation key / count bucket."""
if ".log." in name: return "log"
if name.startswith(".hoodie_partition_metadata"): return "part_meta"
if name == "hoodie.properties": return "props"
if name.endswith(".metadata.json"): return "metadata_json"
if name.startswith("snap-") and name.endswith(".avro"): return "manifest_list"
if name.endswith(".avro"): return "manifest"
if name.endswith(".parquet"): return "base"
return ""
def list_table_files(table_name, label="", verbose=False, max_depth=2, max_children=2):
"""Render a Hudi/Iceberg table's S3 layout as an annotated directory tree.
Teaching view for CLEAN/COMPACT/add_files: multiple base files in a partition
= stale file slices; .log.* = uncompacted MoR updates. The large Hudi .hoodie/
is summarized (depth/width capped); Iceberg metadata/ and data partitions show
in full. verbose=True expands everything.
"""
prefix = f"{db_name}/{table_name}/"
header = f"s3://{bucket_name}/{prefix}"
if label:
header += f" [{label}]"
print(header)
in_meta = lambda p: (p.split("/", 1)[0] if p else "") in (".hoodie", "metadata")
capped_meta = lambda p: (p.split("/", 1)[0] if p else "") == ".hoodie"
# ---- gather objects into a nested tree + tally honest counts -----------
root = {"d": {}, "f": []}
n_base = n_log = n_meta = n_other = 0
s_base = s_log = s_meta = 0
has_hudi = has_ice = False
for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket_name, Prefix=prefix):
for obj in page.get("Contents", []):
key = obj["Key"].replace(prefix, "")
size = obj["Size"]
if key == "" or key.endswith("/"):
continue # S3 directory marker
parts = key.split("/")
fname, top = parts[-1], parts[0]
if top == ".hoodie": has_hudi = True
if top == "metadata": has_ice = True
# files inside a metadata folder count ONLY as metadata
if top in (".hoodie", "metadata"):
n_meta += 1; s_meta += size
elif ".log." in fname:
n_log += 1; s_log += size
elif fname.endswith(".parquet"):
n_base += 1; s_base += size
elif fname.startswith(".hoodie"):
n_meta += 1; s_meta += size; has_hudi = True
else:
n_other += 1
node = root
for part in parts[:-1]:
node = node["d"].setdefault(part, {"d": {}, "f": []})
node["f"].append((fname, size))
# ---- helpers -----------------------------------------------------------
def count_all(node):
return len(node["f"]) + sum(count_all(c) for c in node["d"].values())
def collapse(node, name, rel):
"""Merge single-child, file-less dir chains (year=/month=/day=)."""
while (len(node["d"]) == 1 and not node["f"] and not in_meta(rel)):
cname, child = next(iter(node["d"].items()))
crel = f"{rel}/{cname}"
if in_meta(crel): break
name, rel, node = f"{name}/{cname}", crel, child
return name, rel, node
def line(indent, branch, text, note, width):
s = f"{indent}{branch}{text}"
return f"{s:<{width}} {note}" if note else s
def render(node, rel, indent, depth):
entries = ([("d", n, c) for n, c in sorted(node["d"].items())]
+ [("f", n, sz) for n, sz in sorted(node["f"], key=lambda f: (".log." in f[0], f[0]))])
cap = (not verbose) and capped_meta(rel) and len(entries) > max_children
shown = entries[:max_children] if cap else entries
hidden = len(entries) - len(shown)
for i, item in enumerate(shown):
last = (i == len(shown) - 1) and hidden == 0
branch = "└── " if last else "├── "
sub = indent + (" " if last else "│ ")
if item[0] == "d":
_, name, child = item
disp, erel, enode = collapse(child, name, f"{rel}/{name}" if rel else name)
note = _NOTES.get(_DIR_KIND.get(erel, ""), "")
cnt = f" ({count_all(enode)} obj)" if (in_meta(erel) or erel in _DIR_KIND) else ""
print(line(indent, branch, f"{disp}/{cnt}", note, 60))
if (not verbose) and capped_meta(erel) and depth + 1 > max_depth:
inner = count_all(enode)
if inner:
print(f"{sub}└── … ({inner} objects, use verbose=True)")
else:
render(enode, erel, sub, depth + 1)
else:
_, name, size = item
note = _NOTES.get(_leaf_kind(name), "")
print(line(indent, branch, f"{name} ({_fmt_size(size)})", note, 72))
if hidden:
print(f"{indent}└── … (+{hidden} more)")
render(root, "", "", 0)
# ---- per-partition slice note (Hudi concept; skip metadata & Iceberg data/) ----
def slices(node, rel):
top = rel.split("/", 1)[0] if rel else ""
if top in (".hoodie", "metadata", "data"):
return
nb = sum(1 for f, _ in node["f"] if f.endswith(".parquet"))
nl = sum(1 for f, _ in node["f"] if ".log." in f)
if nb > 1:
print(f" note: {rel}/ has {nb} base files -> multiple file slices (stale versions)")
elif nb == 1 and nl:
print(f" note: {rel}/ has 1 base + {nl} log file(s) -> uncompacted MoR updates")
for n, c in node["d"].items():
slices(c, f"{rel}/{n}" if rel else n)
slices(root, "")
# ---- in-place migration note -------------------------------------------
if has_ice and "data" not in root["d"] and n_base == 0:
print(" note: Iceberg metadata/ present but no local data/ folder")
print(" -> in-place migration: data files remain registered in place at the source (Hudi) location; no data copied")
# ---- summary -----------------------------------------------------------
total = n_base + n_log + n_meta + n_other
print(
f"SUMMARY: {n_base} base · {n_log} log · {n_meta} meta"
+ (f" · {n_other} other" if n_other else "")
+ f" · {total} files · {_fmt_size(s_base + s_log + s_meta)}"
)
print(f" .hoodie/: {'present' if has_hudi else 'none'} metadata/ (Iceberg): {'present' if has_ice else 'none'}")
print()

Sample event data

We use a website clickstream events dataset, partitioned by year/month/day – a realistic pattern for event-driven architectures.

Note: In this example, we create the table with a preCombineField, which flips the default write operation of INSERT INTO from insert to upsert. You can override it explicitly with hoodie.spark.sql.insert.into.operation when you need a different behavior.

CoW table: create, upsert, and delete

We create a Hudi CoW table, insert records, apply an upsert, and delete a record, then inspect how the files change.

Create the Hudi CoW events table

Code:

spark.sql(f"""
CREATE TABLE {db_name}.{hudi_events_cow_table} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING,
year INT,
month INT,
day INT
)
USING HUDI
OPTIONS (
type = 'cow',
primaryKey = 'event_id',
preCombineField = 'event_timestamp'
)
PARTITIONED BY (year, month, day)
LOCATION 's3://{bucket_name}/{db_name}/{hudi_events_cow_table}/'
""")

Insert event records

Code:

spark.sql(f"""
INSERT INTO {db_name}.{hudi_events_cow_table}
VALUES
('EVT_001', 'USER_101', 'page_view', timestamp('2024-04-01 08:15:00'), '/home', 2024, 4, 1),
('EVT_002', 'USER_102', 'click', timestamp('2024-04-01 09:30:00'), '/products', 2024, 4, 1),
('EVT_003', 'USER_103', 'purchase', timestamp('2024-04-02 10:45:00'), '/checkout', 2024, 4, 2),
('EVT_004', 'USER_101', 'page_view', timestamp('2024-04-02 14:20:00'), '/blog', 2024, 4, 2),
('EVT_005', 'USER_104', 'signup', timestamp('2024-04-03 11:00:00'), '/register', 2024, 4, 3),
('EVT_006', 'USER_102', 'click', timestamp('2024-04-03 16:45:00'), '/deals', 2024, 4, 3)
""")

Upsert: simulate late-arriving corrections

Code:

spark.sql(f"""
INSERT INTO {db_name}.{hudi_events_cow_table}
VALUES
('EVT_002', 'USER_102', 'add_to_cart', timestamp('2024-04-01 09:31:00'), '/products/item-42', 2024, 4, 1),
('EVT_005', 'USER_104', 'signup', timestamp('2024-04-03 11:01:00'), '/welcome', 2024, 4, 3)
""")

Delete: remove a record

Code:

spark.sql(f"""
DELETE FROM {db_name}.{hudi_events_cow_table}
WHERE event_id = 'EVT_006'
""")

Inspect the CoW file structure

CoW tables rewrite the entire Parquet file on every upsert/delete. You will see one base Parquet file per file group (no log files).

Code:

list_table_files(hudi_events_cow_table, "COW --- initial file structure")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_cow/ [COW — initial file structure]
├── .hoodie/ (50 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=4/
├── day=1/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ ├── 7d187e9a.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
│ └── 7d187e9a.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
├── day=2/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── e583fe17.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
└── day=3/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
├── eedc1b38.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
├── eedc1b38.....20261011152539102.parquet (426.2 KB) <-- base Parquet file
└── eedc1b38.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
note: year=2024/month=4/day=1/ has 2 base files -> multiple file slices (stale versions)
note: year=2024/month=4/day=3/ has 3 base files -> multiple file slices (stale versions)
SUMMARY: 6 base · 0 log · 53 meta · 59 files · 2.8 MB
.hoodie/: present metadata/ (Iceberg): none

MoR table: create, upsert, and delete

We run the same create, upsert, and delete sequence on a Hudi MoR table.

Create the Hudi MoR events table

Code:

spark.sql(f"""
CREATE TABLE {db_name}.{hudi_events_mor_table} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING,
year INT,
month INT,
day INT
)
USING HUDI
OPTIONS (
type = 'mor',
primaryKey = 'event_id',
preCombineField = 'event_timestamp'
)
PARTITIONED BY (year, month, day)
LOCATION 's3://{bucket_name}/{db_name}/{hudi_events_mor_table}/'
""")

Insert initial event records

Code:

spark.sql(f"""
INSERT INTO {db_name}.{hudi_events_mor_table}
VALUES
('EVT_001', 'USER_201', 'page_view', timestamp('2024-05-10 07:00:00'), '/home', 2024, 5, 10),
('EVT_002', 'USER_202', 'click', timestamp('2024-05-10 08:15:00'), '/products', 2024, 5, 10),
('EVT_003', 'USER_203', 'purchase', timestamp('2024-05-11 09:30:00'), '/checkout', 2024, 5, 11),
('EVT_004', 'USER_201', 'page_view', timestamp('2024-05-11 12:00:00'), '/account', 2024, 5, 11),
('EVT_005', 'USER_204', 'signup', timestamp('2024-05-12 10:30:00'), '/register', 2024, 5, 12),
('EVT_006', 'USER_202', 'click', timestamp('2024-05-12 15:45:00'), '/deals', 2024, 5, 12)
""")

Upsert: simulate late-arriving corrections (creates delta log files)

Code:

spark.sql(f"""
INSERT INTO {db_name}.{hudi_events_mor_table}
VALUES
('EVT_002', 'USER_202', 'add_to_cart', timestamp('2024-05-10 08:16:00'), '/products/item-99', 2024, 5, 10),
('EVT_005', 'USER_204', 'signup', timestamp('2024-05-12 10:31:00'), '/welcome', 2024, 5, 12)
""")

Delete: remove a record

Code:

spark.sql(f"""
DELETE FROM {db_name}.{hudi_events_mor_table}
WHERE event_id = 'EVT_006'
""")

Inspect MoR: base Parquet and log files

MoR tables write updates as delta log files (.log.*) alongside the base Parquet files. This avoids full file rewrites for faster writes.

Code:

list_table_files(hudi_events_mor_table, "MOR --- base parquet + log files")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_mor/ [MOR — base parquet + log files]
├── .hoodie/ (50 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=5/
├── day=10/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ ├── 7d5ce247.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
│ └── .7d5ce247.....20261011153145862.log.1_1-164-788 (1.3 KB) <-- Hudi delta log file (MoR update, uncompacted)
├── day=11/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── f9b8147e.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
└── day=12/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
├── 3f0b5ad4.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
├── .3f0b5ad4.....20261011153145862.log.1_0-164-787 (1.3 KB) <-- Hudi delta log file (MoR update, uncompacted)
└── .3f0b5ad4.....20261011153215018.log.1_0-192-901 (1.1 KB) <-- Hudi delta log file (MoR update, uncompacted)
note: year=2024/month=5/day=10/ has 1 base + 1 log file(s) -> uncompacted MoR updates
note: year=2024/month=5/day=12/ has 1 base + 2 log file(s) -> uncompacted MoR updates
SUMMARY: 3 base · 3 log · 53 meta · 59 files · 1.5 MB
.hoodie/: present metadata/ (Iceberg): none

Query both Hudi tables

Code:

print("=== COW Table (events_hudi_cow) ===")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url, year, month, day FROM {db_name}.{hudi_events_cow_table} ORDER BY event_id").show(truncate=False)
print("\n=== MOR Table (events_hudi_mor) ===")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url, year, month, day FROM {db_name}.{hudi_events_mor_table} ORDER BY event_id").show(truncate=False)

Output:

=== COW Table (events_hudi_cow) ===
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
|event_id|user_id |event_type |event_timestamp |page_url |year|month|day|
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |2024|4 |1 |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|2024|4 |1 |
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |2024|4 |2 |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |2024|4 |2 |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |2024|4 |3 |
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
=== MOR Table (events_hudi_mor) ===
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
|event_id|user_id |event_type |event_timestamp |page_url |year|month|day|
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
|EVT_001 |USER_201|page_view |2024-05-10 07:00:00|/home |2024|5 |10 |
|EVT_002 |USER_202|add_to_cart|2024-05-10 08:16:00|/products/item-99|2024|5 |10 |
|EVT_003 |USER_203|purchase |2024-05-11 09:30:00|/checkout |2024|5 |11 |
|EVT_004 |USER_201|page_view |2024-05-11 12:00:00|/account |2024|5 |11 |
|EVT_005 |USER_204|signup |2024-05-12 10:31:00|/welcome |2024|5 |12 |
+--------+--------+-----------+-------------------+-----------------+----+-----+---+

Migrate Hudi tables to Iceberg with in-place migration

In-place migration keeps the original Parquet data files where they are and only builds Iceberg metadata on top of them. There is no rewrite of the underlying data, which makes the migration fast and inexpensive even on large tables. Two tools cover the common cases on AWS:

  • add_files (Iceberg built-in procedure). A Spark procedure shipped with Apache Iceberg that registers existing Parquet files into an Iceberg table by writing the corresponding manifests, without rewriting the data. It runs as a Spark job in which the source table’s partitions are distributed across Spark tasks, so file listing and column-statistics gathering happen in parallel on the cluster.
  • Apache XTable (incubating). An open-source cross-format translation tool that can read the metadata of one lakehouse format (Hudi, Delta, or Iceberg) and write out the equivalent metadata for another, pointing at the same underlying Parquet files. XTable can run once to produce the initial Iceberg metadata, or run on a schedule to keep the Iceberg view continuously in sync with ongoing writes to the Hudi table.

Choosing between add_files and Apache XTable for in-place migration

When planning an in-place migration, it is important to understand the strengths of each approach:

  • add_files is best suited for a one-time migration. It registers existing Hudi Parquet files into an Iceberg table in a single operation. After the migration is complete, the source Hudi table and target Iceberg table are independent, and any subsequent changes to the Hudi table are not reflected in Iceberg.
  • Apache XTable supports both an initial one-time conversion and ongoing incremental synchronization. This makes it ideal for transition periods where you need to keep both the source Hudi tables and the target Iceberg tables in sync while downstream consumers gradually migrate to Iceberg. After all workloads have been cut over, you can stop the sync process and decommission the Hudi tables.

Important notes

add_files/XTable register existing Parquet files as-is, so they rely on the column names baked into each file. Added and dropped columns are safe, but if the source table was renamed any column during its lifecycle, files written before the rename still hold the old name and reads against the new name return NULL for them. For tables with a history of column renames, prefer a full migration instead.

Iceberg version and column statistics (in-place add_files / snapshot): In Iceberg 1.9.0 through 1.10.0, the add_files and snapshot procedures don’t generate column stats in the Iceberg metadata. The cause is a NameMapping loss in ParquetUtil.footerMetrics. Without those column-level lower/upper bounds and null counts, the migrated Iceberg table loses most of its file-skipping ability at query planning time, which translates into more Parquet files being inspected, higher scan volume, and slower queries, especially for selective filters on non-partition columns. The issue is fixed in the Iceberg 1.10.1 release (#14617). Affected runtimes are AWS Glue 5.1 and Amazon EMR 7.11–7.13. We recommend AWS Glue 6.0+ or Amazon EMR 7.14+ for production migrations.

Shared data files during in-place migration: With both approaches, the in-place migration doesn’t copy data. The resulting Iceberg table points to the same underlying Parquet files as the source Hudi table. This means that until you perform a full cutover to Iceberg, you must avoid destructive operations on the Iceberg table that could delete or modify the shared data files. Specifically:

  • Don’t drop the Iceberg table with the PURGE option.
  • Don’t run Iceberg maintenance tasks such as expire_snapshots or remove_orphan_files.

Running any of these before cutover can remove files that the source Hudi table still depends on, leading to data loss. Iceberg exposes a built-in guardrail against this: the gc.enabled table property. When set to false, Iceberg blocks the preceding destructive operations. How you apply this depends on which migration path you choose.

With add_files. Disable garbage collection immediately after the procedure finishes and keep it disabled until you have fully cut over to Iceberg.

Right after add_files:

spark.sql(f"""
ALTER TABLE {db_name}.{table_name}
SET TBLPROPERTIES ('gc.enabled' = 'false');
""")

After Hudi source is retired and the Iceberg table owns its own data, re-enable GC so maintenance can run normally:

spark.sql(f"""
ALTER TABLE {db_name}.{table_name}
SET TBLPROPERTIES ('gc.enabled' = 'true');
""")

With Apache XTable (incremental sync). The gc.enabled = false guardrail is only partially usable here, because XTable’s incremental sync needs GC to be enabled on the Iceberg table to operate normally. A practical pattern is:

  1. Run the initial full XTable sync.
  2. Temporarily set gc.enabled = false to run post-migration validation and consistency checks on the Iceberg table in a safe state.
  3. Set gc.enabled = true again before starting incremental sync and leave it on for the duration of the sync period.

Because the automatic guardrail is off during incremental sync, the burden shifts to operational discipline: as long as XTable is syncing and data files are shared with the Hudi table, don’t run any Iceberg operation that can delete source data files (no DROP TABLE ... PURGE, no expire_snapshots, no remove_orphan_files, no manual object deletion against the shared S3 prefix). Treat the Iceberg table as read-only in terms of file lifecycle until cutover is complete, and Hudi is decommissioned, at which point normal Iceberg maintenance can resume.

Don’t enable the AWS Glue Data Catalog table optimizer before cutover.

Glue-managed Iceberg maintenance (compaction, snapshot retention, orphan file cleanup) will rewrite or delete the Parquet files that the source Hudi table still depends on. Verify that no catalog-level optimizer applies to the migrated table and leave table-level optimization off until cutover is complete.

Prerequisites for in-place migration with add_files. The add_files procedure registers every Parquet file it finds under the source location into the target Iceberg table, with no awareness of Hudi’s file-group versioning. If more than one file version of the same record is present at migration time, each version is registered as a separate row, and the Iceberg table will contain duplicates. The source Hudi table must therefore expose exactly one file version per record before add_files runs:

  • CoW tables: Run CLEAN to remove old file versions, keeping only the latest Parquet files.
  • MoR tables: Run COMPACTION (merge delta logs into base Parquet) + CLEAN before calling add_files, since add_files can only register base Parquet files and cannot process Hudi delta log files.

Option 1: Migrate Hudi tables to Iceberg with the add_files Spark procedure

This option uses the Iceberg add_files procedure to register the existing Hudi Parquet files into new Iceberg tables. Run it once per table, after preparing the Hudi table so that only the current file versions remain.

Migrate CoW to Iceberg

We create the target Iceberg table, prepare the Hudi source so that only the current file versions remain, register those files, and validate the result.

Create the target Iceberg table with the same schema

Code:

spark.sql(f"""
CREATE TABLE glue_iceberg.{db_name}.{iceberg_cow_inplace} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING,
year INT,
month INT,
day INT
)
USING iceberg
PARTITIONED BY (year, month, day)
LOCATION 's3://{bucket_name}/{db_name}/{iceberg_cow_inplace}/'
""")

Run Hudi CLEAN to keep only the latest file versions

Check files listing for the Hudi table before the CLEAN operation:

Code:

list_table_files(hudi_events_cow_table, "COW --- BEFORE CLEAN")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_cow/ [COW — BEFORE CLEAN]
├── .hoodie/ (50 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=4/
├── day=1/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ ├── 7d187e9a.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
│ └── 7d187e9a.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
├── day=2/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── e583fe17.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
└── day=3/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
├── eedc1b38.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
├── eedc1b38.....20261011152539102.parquet (426.2 KB) <-- base Parquet file
└── eedc1b38.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
note: year=2024/month=4/day=1/ has 2 base files -> multiple file slices (stale versions)
note: year=2024/month=4/day=3/ has 3 base files -> multiple file slices (stale versions)
SUMMARY: 6 base · 0 log · 53 meta · 59 files · 2.8 MB
.hoodie/: present metadata/ (Iceberg): none

Run the CLEAN operation specifying to keep only the latest file slices:

Code:

spark.sql(f"""
CALL spark_catalog.system.run_clean(
table => '{db_name}.{hudi_events_cow_table}',
clean_policy => 'KEEP_LATEST_FILE_VERSIONS',
file_versions_retained => 1
)
""").show(truncate=False)

Output:

+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+
|start_clean_time |time_taken_in_millis|total_files_deleted|earliest_commit_to_retain|bootstrap_part_metadata|version|
+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+
|20261011154407413|382 |3 | |{} |2 |
+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+

Check files listing for the Hudi table after the CLEAN operation:

Code:

list_table_files(hudi_events_cow_table, "COW --- AFTER CLEAN")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_cow/ [COW — AFTER CLEAN]
├── .hoodie/ (59 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=4/
├── day=1/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── 7d187e9a.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
├── day=2/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── e583fe17.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
└── day=3/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
└── eedc1b38.....20261011152539102.parquet (426.2 KB) <-- base Parquet file
SUMMARY: 3 base · 0 log · 62 meta · 65 files · 1.6 MB
.hoodie/: present metadata/ (Iceberg): none

Register Hudi Parquet files into Iceberg with add_files

Code:

spark.sql(f"""
CALL glue_iceberg.system.add_files(
source_table => 'parquet.`s3://{bucket_name}/{db_name}/{hudi_events_cow_table}/`',
table => 'glue_iceberg.{db_name}.{iceberg_cow_inplace}'
)
""").show(truncate=False)

Output:

+-----------------+-----------------------+
|added_files_count|changed_partition_count|
+-----------------+-----------------------+
|3 |NULL |
+-----------------+-----------------------+

Validate: compare Hudi CoW and Iceberg (in-place)

We can see the Hudi and Iceberg tables reflect the same content:

Code:

print("Source: Hudi COW")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url FROM {db_name}.{hudi_events_cow_table} ORDER BY event_id").show(truncate=False)
print("Target: Iceberg (in-place)")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url FROM glue_iceberg.{db_name}.{iceberg_cow_inplace} ORDER BY event_id").show(truncate=False)

Output:

Source: Hudi COW
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+
Target: Iceberg (in-place)
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

We can see under the Iceberg table location only Iceberg metadata has been generated. These point to the source Hudi table data files.

Code:

list_table_files(iceberg_cow_inplace, "COW Iceberg --- AFTER add_files (note metadata/ folder)", verbose=True)

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_iceberg_cow_inplace/ [COW Iceberg — AFTER add_files (note metadata/ folder)]
└── metadata/ (5 obj) <-- Iceberg metadata layer (metadata.json, manifest lists, manifests)
├── 00000.....metadata.json (1.3 KB) <-- Iceberg table metadata (schema, snapshots)
├── 00001.....metadata.json (2.0 KB) <-- Iceberg table metadata (schema, snapshots)
├── 00002.....metadata.json (3.0 KB) <-- Iceberg table metadata (schema, snapshots)
├── snap-7577.....avro (4.4 KB) <-- Iceberg manifest list (snapshot -> manifests)
└── stage-223...-manifest.....avro (6.7 KB) <-- Iceberg manifest (lists data files)
note: Iceberg metadata/ present but no local data/ folder
-> in-place migration: data files remain registered in place at the source (Hudi) location; no data copied
SUMMARY: 0 base · 0 log · 5 meta · 5 files · 17.4 KB
.hoodie/: none metadata/ (Iceberg): present

Migrate MoR to Iceberg

For the MoR table, we compact and clean the Hudi source first, then register the compacted files and validate the result.

Create the target Iceberg table

Code:

spark.sql(f"""
CREATE TABLE glue_iceberg.{db_name}.{iceberg_mor_inplace} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING,
year INT,
month INT,
day INT
)
USING iceberg
PARTITIONED BY (year, month, day)
LOCATION 's3://{bucket_name}/{db_name}/{iceberg_mor_inplace}/'
""")

Compact the MoR table: merge delta logs into base Parquet files

Check files listing for the Hudi table before the CLEAN operation:

Code:

list_table_files(hudi_events_mor_table, "MOR --- BEFORE COMPACT+CLEAN")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_mor/ [MOR — BEFORE COMPACT+CLEAN]
├── .hoodie/ (50 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=5/
├── day=10/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ ├── 7d5ce247.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
│ └── .7d5ce247.....20261011153145862.log.1_1-164-788 (1.3 KB) <-- Hudi delta log file (MoR update, uncompacted)
├── day=11/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── f9b8147e.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
└── day=12/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
├── 3f0b5ad4.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
├── .3f0b5ad4.....20261011153145862.log.1_0-164-787 (1.3 KB)
└── .3f0b5ad4.....20261011153215018.log.1_0-192-901 (1.1 KB) <-- Hudi delta log file (MoR update, uncompacted)
note: year=2024/month=5/day=10/ has 1 base + 1 log file(s) -> uncompacted MoR updates
note: year=2024/month=5/day=12/ has 1 base + 2 log file(s) -> uncompacted MoR updates
SUMMARY: 3 base · 3 log · 53 meta · 59 files · 1.5 MB
.hoodie/: present metadata/ (Iceberg): none

Trigger COMPACTION scheduling on the MoR table with the delta-commit and age thresholds set to their minimum, so any file slice with pending log files is included in the plan and merged back into its base Parquet file.

Code:

# Schedule compaction
spark.sql(f"""
CALL spark_catalog.system.run_compaction(
table => '{db_name}.{hudi_events_mor_table}',
op => 'schedule',
options =>
'hoodie.compact.inline.max.delta.commits=1,
hoodie.compact.inline.max.delta.seconds=0'
)
""").show(truncate=False)

Output:

+-----------------+--------------+---------+
|timestamp |operation_size|state |
+-----------------+--------------+---------+
|20261011154522819|2 |REQUESTED|
+-----------------+--------------+---------+

Execute the previously scheduled compaction plan to merge the pending log files into their base Parquet files. The REQUESTED instant produced by the schedule step transitions to COMPLETED, and the table is left with base Parquet files only.

Code:

# Execute compaction
spark.sql(f"""
CALL spark_catalog.system.run_compaction(
table => '{db_name}.{hudi_events_mor_table}',
op => 'run'
)
""").show(truncate=False)

Output:

+-----------------+--------------+---------+
|timestamp |operation_size|state |
+-----------------+--------------+---------+
|20261011154522819|2 |COMPLETED|
+-----------------+--------------+---------+

Run Hudi CLEAN to remove old file versions and log files

Code:

spark.sql(f"""
CALL spark_catalog.system.run_clean(
table => '{db_name}.{hudi_events_mor_table}',
clean_policy => 'KEEP_LATEST_FILE_VERSIONS',
file_versions_retained => 1
)
""").show(truncate=False)

Output:

+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+
|start_clean_time |time_taken_in_millis|total_files_deleted|earliest_commit_to_retain|bootstrap_part_metadata|version|
+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+
|20261011154552026|345 |5 | |{} |2 |
+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+

Code:

list_table_files(hudi_events_mor_table, "MOR --- AFTER COMPACT+CLEAN")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_mor/ [MOR — AFTER COMPACT+CLEAN]
├── .hoodie/ (69 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=5/
├── day=10/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── 7d5ce247.....20261011154522819.parquet (426.3 KB) <-- base Parquet file
├── day=11/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── f9b8147e.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
└── day=12/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
└── 3f0b5ad4.....20261011154522819.parquet (426.1 KB) <-- base Parquet file
SUMMARY: 3 base · 0 log · 72 meta · 75 files · 1.7 MB
.hoodie/: present metadata/ (Iceberg): none

Register compacted Parquet files into Iceberg with add_files

Code:

spark.sql(f"""
CALL glue_iceberg.system.add_files(
source_table => 'parquet.`s3://{bucket_name}/{db_name}/{hudi_events_mor_table}/`',
table => 'glue_iceberg.{db_name}.{iceberg_mor_inplace}'
)
""").show(truncate=False)

Output:

+-----------------+-----------------------+
|added_files_count|changed_partition_count|
+-----------------+-----------------------+
|3 |NULL |
+-----------------+-----------------------+

Refresh the Hudi table to invalidate Spark’s cached metadata and file listings, so subsequent queries re-read the post-compaction state from the catalog and storage instead of returning stale pre-compaction results.

Code:

spark.sql(f"REFRESH TABLE {db_name}.{hudi_events_mor_table}")

Validate: compare Hudi MoR and Iceberg (in-place)

Code:

print("Source: Hudi MOR")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url FROM {db_name}.{hudi_events_mor_table} ORDER BY event_id").show(truncate=False)
print("Target: Iceberg (in-place)")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url FROM glue_iceberg.{db_name}.{iceberg_mor_inplace} ORDER BY event_id").show(truncate=False)

Output:

Source: Hudi MOR
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_201|page_view |2024-05-10 07:00:00|/home |
|EVT_002 |USER_202|add_to_cart|2024-05-10 08:16:00|/products/item-99|
|EVT_003 |USER_203|purchase |2024-05-11 09:30:00|/checkout |
|EVT_004 |USER_201|page_view |2024-05-11 12:00:00|/account |
|EVT_005 |USER_204|signup |2024-05-12 10:31:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+
Target: Iceberg (in-place)
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_201|page_view |2024-05-10 07:00:00|/home |
|EVT_002 |USER_202|add_to_cart|2024-05-10 08:16:00|/products/item-99|
|EVT_003 |USER_203|purchase |2024-05-11 09:30:00|/checkout |
|EVT_004 |USER_201|page_view |2024-05-11 12:00:00|/account |
|EVT_005 |USER_204|signup |2024-05-12 10:31:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

Code:

list_table_files(iceberg_mor_inplace, "MOR Iceberg --- AFTER add_files (note metadata/ folder)", verbose=True)

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_iceberg_mor_inplace/ [MOR Iceberg — AFTER add_files (note metadata/ folder)]
└── metadata/ (5 obj) <-- Iceberg metadata layer (metadata.json, manifest lists, manifests)
├── 00000.....metadata.json (1.3 KB) <-- Iceberg table metadata (schema, snapshots)
├── 00001.....metadata.json (2.0 KB) <-- Iceberg table metadata (schema, snapshots)
├── 00002.....metadata.json (3.0 KB) <-- Iceberg table metadata (schema, snapshots)
├── snap-5097.....avro (4.4 KB) <-- Iceberg manifest list (snapshot -> manifests)
└── stage-254...-manifest.....avro (6.7 KB) <-- Iceberg manifest (lists data files)
note: Iceberg metadata/ present but no local data/ folder
-> in-place migration: data files remain registered in place at the source (Hudi) location; no data copied
SUMMARY: 0 base · 0 log · 5 meta · 5 files · 17.4 KB
.hoodie/: none metadata/ (Iceberg): present

Option 2: Migrate Hudi tables to Iceberg with the Apache XTable sync tool

Apache XTable (Incubating) is an open-source interoperability layer that translates metadata between Apache Hudi, Apache Iceberg, and Delta Lake without copying data. Its sync utility supports both one-time conversion and ongoing incremental synchronization, enabling parallel access to source and target formats during migration. Because it ships as a self-contained Java process with no managed-service dependency, you can run it anywhere a JVM is available: a developer laptop for one-off conversions, an Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS) container, or an AWS Lambda function, triggered on a schedule through Amazon EventBridge or on an event such as a new Hudi commit.

XTable’s sync utility does two things:

  1. First run – performs the initial in-place migration of a Hudi table to Iceberg.
  2. Subsequent runs – keep the Iceberg metadata in sync with the source table as it continues to receive writes.

Compile xtable-utilities following the instructions in the Apache XTable GitHub repository and copy xtable-utilities/target/xtable-utilities_2.12-0.5.0-SNAPSHOT-bundled.jar into the current working directory.

Migrate MoR to Iceberg

Note: When running the XTable sync tool on a MoR table, the utility only reads the base Parquet files and doesn’t process the delta log files. As a result, the generated Iceberg metadata reflects only the Hudi read-optimized view, meaning any recent updates or deletes stored in delta logs will be missing. To avoid data loss, always run compaction on the MoR table before executing the sync tool (for example, similar run_compaction commands as for the add_files example). Unlike add_files, however, there is no need to run CLEAN beforehand: XTable doesn’t scan the table location file by file, but builds the Iceberg manifests from Hudi’s own metadata, so it already knows which Parquet files belong to the current file slices and ignores superseded versions automatically.

Configure Apache XTables Configuration files.

Create the datasetConfig.yaml file containing indications about the dataset(s) to convert with the following content:

sourceFormat: HUDI
targetFormats:
  - ICEBERG
datasets:
  - tableBasePath: s3://YOUR_BUCKET/hudi_to_iceberg_blog/events_hudi_mor/
    tableName: events_hudi_mor_iceberg_xtable
    namespace: hudi_to_iceberg_blog
    partitionSpec: year:VALUE,month:VALUE,day:VALUE

Create the glueDataCatalog.yaml file containing Iceberg catalog (AWS Glue Data Catalog) settings with the following content:

catalogImpl: org.apache.iceberg.aws.glue.GlueCatalog
catalogName: glue
catalogOptions:
  warehouse: s3://YOUR_BUCKET/warehouse/
  catalog-impl: org.apache.iceberg.aws.glue.GlueCatalog
  io-impl: org.apache.iceberg.aws.s3.S3FileIO

Run the conversion

export AWS_DEFAULT_REGION=us-east-1

java -jar incubator-xtable/xtable-utilities/target/xtable-utilities_2.12-0.5.0-SNAPSHOT-bundled.jar \
-d datasetConfig.yaml \
-i glueDataCatalog.yaml

CLI options (from --help)

Flag Long form Description
-d --datasetConfig Path to the dataset configuration YAML (required)
-i --icebergCatalogConfig Path to the Iceberg catalog config YAML
-c --convertersConfig Override converter configs
-p --hadoopConfig Extra Hadoop config XML
-m --continuousMode Run on a scheduled loop
-t --continuousModeInterval Loop interval in seconds (with -m)

Verify the result

% aws glue get-table --database-name hudi_to_iceberg_blog \
--name events_hudi_mor_iceberg_xtable \
--query 'Table.{Name:Name,Type:Parameters.table_type,MetaLoc:Parameters.metadata_location}'

{
  "Name": "events_hudi_mor_iceberg_xtable",
  "Type": "ICEBERG",
  "MetaLoc": "s3://YOUR_BUCKET/hudi_to_iceberg_blog/events_hudi_mor/metadata/00002-a88318ba-ad13-4a0c-85a8-d595b53e9a89.metadata.json"
}

% aws s3 ls s3://YOUR_BUCKET/hudi_to_iceberg_blog/events_hudi_mor/metadata/
2026-09-22 16:26:18        648 00000-c71bcac6-3eee-4b98-875f-7562770db80b.metadata.json
2026-09-22 16:26:19       1940 00001-941c5622-95e3-41d7-942a-dabd69e561d1.metadata.json
2026-09-22 16:26:20       4293 00002-a88318ba-ad13-4a0c-85a8-d595b53e9a89.metadata.json
2026-09-22 16:26:20       8845 11adf6a4-58b9-4cc2-8cf0-317a7b43a8f7-m0.avro
2026-09-22 16:26:20       4493 snap-8409429281635613515-1-11adf6a4-58b9-4cc2-8cf0-317a7b43a8f7.avro
Amazon Athena console query returning five rows from the migrated Iceberg table

Figure 2: Validating the Apache XTable conversion by querying the migrated Iceberg table in Amazon Athena

Note: If you keep ingesting into the source Hudi table and want to keep the Iceberg table in sync, re-run the same conversion command. XTable automatically detects the previous execution and runs in incremental mode.

Migrate CoW to Iceberg

For the CoW table, update the datasetConfig.yaml with the CoW table path and name.

sourceFormat: HUDI
targetFormats:
  - ICEBERG
datasets:
  - tableBasePath: s3://YOUR_BUCKET/hudi_to_iceberg_blog/events_hudi_cow/
    tableName: events_hudi_mor_iceberg_xtable
    namespace: hudi_to_iceberg_blog
    partitionSpec: year:VALUE,month:VALUE,day:VALUE

The glueDataCatalog.yaml, conversion command, and verification steps remain the same as described in the MOR example.

Migrate Hudi tables to Iceberg with full table migration

Full table migration (also called shadow migration) reads all data from Hudi and writes it into a brand-new Iceberg table. This approach:

  • Allows schema and partition evolution (for example, switching to Iceberg hidden partitioning).
  • Produces a clean Iceberg table with no Hudi metadata dependencies.
  • Requires additional storage (data is copied)

Two options are demonstrated:

  • Option A: CTAS – a single-statement CREATE TABLE ... AS SELECT.
  • Option B: CREATE + INSERT – explicit table creation followed by data insertion.

Note: When performing a full table migration, the process is the same regardless of whether your source is a Hudi CoW or MoR table. This is because Spark reads from the Hudi real-time view by default, which reflects the latest table state. For CoW tables, this means reading the current base Parquet files. For MoR tables, the real-time view automatically merges the base Parquet files with any pending delta log files, ensuring all recent inserts, updates, and deletes are captured. As a result, the migrated Iceberg table will contain a complete and up-to-date snapshot of your data, with no additional compaction or cleanup steps required before migration.

We have taken a Hudi CoW table example to demonstrate the migration from Hudi CoW table to Iceberg table.

Prepare the column list for migration

Filter out Hudi internal metadata columns (_hoodie_*) and explicit partition columns to use Iceberg hidden partitioning.

Code:

# Build column list excluding Hudi metadata and explicit partition columns
all_columns_cow = spark.sql(f"SELECT * FROM {db_name}.{hudi_events_cow_table} LIMIT 1").columns
non_hoodie_columns_cow = [col for col in all_columns_cow if not col.startswith("_hoodie")]
partition_cols = ['year', 'month', 'day']
data_columns_cow = [col for col in non_hoodie_columns_cow if col not in partition_cols]
columns_to_select_cow = ", ".join(data_columns_cow)
print(f"Columns for COW migration: {columns_to_select_cow}")

Output:

Columns for COW migration: event_id, user_id, event_type, event_timestamp, page_url

Migrate CoW to Iceberg (full table / shadow migration)

We show two full-migration options: a single-statement CTAS and an explicit CREATE followed by INSERT.

Option A: CTAS (CREATE TABLE AS SELECT)

Single statement that creates the Iceberg table and populates it in one step. Here we also adopt Iceberg hidden partitioning by days(event_timestamp) – no need for explicit year/month/day columns.

Code:

spark.sql(f"""
CREATE TABLE glue_iceberg.{db_name}.{iceberg_cow_full_ctas}
USING ICEBERG
PARTITIONED BY (days(event_timestamp))
LOCATION 's3://{bucket_name}/{db_name}/{iceberg_cow_full_ctas}/'
AS (SELECT {columns_to_select_cow} FROM {db_name}.{hudi_events_cow_table})
""")

Code:

# Verify CTAS migration
print("Iceberg COW (CTAS) - Full Migration")
spark.sql(f"SELECT * FROM glue_iceberg.{db_name}.{iceberg_cow_full_ctas} ORDER BY event_id").show(truncate=False)

Output:

Iceberg COW (CTAS) - Full Migration
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

Option B: CREATE + INSERT

Useful when you need more control over the target table schema or want to add table properties before loading data, such as sort order. It is also a good fit for very large source tables, because the load phase is a plain INSERT INTO that you can run partition by partition, giving you control over parallelism, resource usage, and resumability during the migration.

Code:

spark.sql(f"""
CREATE TABLE glue_iceberg.{db_name}.{iceberg_cow_full_create_insert} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING
)
USING iceberg
PARTITIONED BY (days(event_timestamp))
LOCATION 's3://{bucket_name}/{db_name}/{iceberg_cow_full_create_insert}/'
""")

Code:

spark.sql(f"""
INSERT INTO glue_iceberg.{db_name}.{iceberg_cow_full_create_insert}
SELECT {columns_to_select_cow}
FROM {db_name}.{hudi_events_cow_table}
""")

Code:

# Verify CREATE + INSERT migration
print("Iceberg COW (CREATE+INSERT) - Full Migration")
spark.sql(f"SELECT * FROM glue_iceberg.{db_name}.{iceberg_cow_full_create_insert} ORDER BY event_id").show(truncate=False)

Output:

Iceberg COW (CREATE+INSERT) - Full Migration
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

Full table migration of Hudi to Amazon S3 Tables

Create a new notebook following the same procedure described earlier, updating the configuration as shown in the following steps.

Configure the Spark session

Change <ACCOUNT_ID>, <REGION>, and <TABLE_BUCKET_NAME> to your S3 Tables bucket details.

Code:

%%configure
{
"glue_version": "5.1",
"worker_type": "G.1X",
"number_of_workers": 5,
"idle_timeout": 30,
"--datalake-formats": "hudi,iceberg",
"--conf": "spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.s3tablesbucket=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.s3tablesbucket.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.s3tablesbucket.glue.id=<ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET_NAME> --conf spark.sql.catalog.s3tablesbucket.warehouse=arn:aws:s3tables:<REGION>:<ACCOUNT_ID>:bucket/<TABLE_BUCKET_NAME>"
}

Configuration variables

Code:

# ──────────────────────────────────────────────
# Source Hudi tables - created in previous notebook
# ──────────────────────────────────────────────
hudi_events_cow_table = "events_hudi_cow" # Copy-on-Write source
# ──────────────────────────────────────────────
# s3tables targets - Full table migration
# ──────────────────────────────────────────────
s3tables_cow_full = "events_s3tables_cow_full"
# ──────────────────────────────────────────────
# Infrastructure
# ──────────────────────────────────────────────
db_name = "hudi_to_iceberg_blog" # Glue Data Catalog database
namespace_name = "hudi_to_s3tables" # S3tables Catalog database

Create the S3 Tables namespace

Code:

spark.sql(f"""
CREATE NAMESPACE IF NOT EXISTS s3tablesbucket.{namespace_name}
""")

Full table migration with CREATE + INSERT

Let’s perform full migration to S3tables. We also implement Iceberg hidden partitioning during the migration.

Hudi CoW table example

Code:

spark.sql(f"""
CREATE TABLE s3tablesbucket.{namespace_name}.{s3tables_cow_full} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING
)
USING iceberg
PARTITIONED BY (days(event_timestamp))
""")

Code:

spark.sql(f"""
INSERT INTO s3tablesbucket.{namespace_name}.{s3tables_cow_full}
SELECT
event_id,
user_id,
event_type,
event_timestamp,
page_url
FROM {db_name}.{hudi_events_cow_table}
""")

Query the S3 Tables Iceberg table

Code:

spark.sql(f"""
SELECT * FROM s3tablesbucket.{namespace_name}.{s3tables_cow_full} ORDER BY event_id
""").show(truncate=False)

Output:

+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

Clean up

To avoid incurring future charges and to clean up unused roles and policies, delete the resources you created.

  1. Clean up the S3 Tables tables and namespace (same Spark session configuration as for the S3 Tables section):
    spark.sql(f"DROP TABLE IF EXISTS s3tablesbucket.{namespace_name}.{s3tables_cow_full} PURGE")
    spark.sql(f"DROP NAMESPACE IF EXISTS s3tablesbucket.{namespace_name}")

  2. Clean up the tables and namespace created in the general S3 bucket (same Spark session configuration as for the S3 General buckets section):
    in_place_iceberg = [
        iceberg_cow_inplace,
        iceberg_mor_inplace,
        iceberg_cow_xtable,
        iceberg_mor_xtable,
    ]
    for t in in_place_iceberg:
        spark.sql(f"DROP TABLE IF EXISTS glue_iceberg.{db_name}.{t}")
    
    owned_iceberg = [
        iceberg_cow_full_ctas,
        iceberg_cow_full_create_insert,
        iceberg_mor_full_ctas,
        iceberg_mor_full_create_insert,
    ]
    for t in owned_iceberg:
        spark.sql(f"DROP TABLE IF EXISTS glue_iceberg.{db_name}.{t}")
    
    # --- Hudi tables ---
    hudi_tables = [
        hudi_events_cow_table,
        hudi_events_mor_table,
    ]
    for t in hudi_tables:
        spark.sql(f"DROP TABLE IF EXISTS {db_name}.{t}")
    
    spark.sql(f"DROP DATABASE IF EXISTS {db_name} CASCADE")

  3. On the Amazon S3 console, navigate to the database folder and delete it.
  4. In the AWS Glue console, under ETL jobs, select the created notebook and choose Actions, Delete job(s).
  5. In the IAM console, delete the role you created for the notebook.

Conclusion

This post outlined strategies for migrating Apache Hudi tables to Apache Iceberg on Amazon S3 and Amazon S3 Tables, enabling organizations to build a modern transactional data lake. Most of the migrations were performed from AWS Glue 5.1 notebooks with interactive sessions, with the exception of the Apache XTable path, which was executed as a standalone Java process from a local workstation to show that cross-format conversion doesn’t require a managed Spark environment and can run anywhere a JVM is available.

The migration process, whether through an in-place migration, a full table migration, or a combination of both, should be chosen based on a thorough analysis of the data architecture and the integration pipelines that read from and write to the table. Factors to weigh include performance requirements, cost, data freshness needs, and the acceptable downtime window during the cutover. The migration should follow well-defined phases that identify the data integration patterns and use cases in play, so the selected strategy aligns with the specific requirements of each table.

To get started, open the AWS Glue console and try the migration path that fits your tables. For more information about the procedures used in this post, see the Apache Iceberg Spark procedures documentation and the AWS Glue Developer Guide.

Note: Always run the migration end to end in a non-production environment first and validate the migrated Iceberg table against the source Hudi table before cutting production workloads over. We recommend preparing a validation runbook with queries that compare row counts per partition, aggregate values on key metrics, latest-record checks against the preCombine field, and sample point lookups on primary keys, and running it both immediately after the migration and again after each downstream pipeline is repointed at the Iceberg table. Keep the source Hudi table available until the validation runbook has passed on production traffic.


About the authors

Nitin Kumar

Nitin Kumar

Nitin is a Solutions Architect at AWS Industries. He partners with customers to transform their cloud journey through innovative, scalable solutions. In his free time, he likes to watch movies and spend time with his family.

Shubham Purwar

Shubham Purwar

Shubham is an AWS Analytics Specialist Solution Architect. He helps organizations unlock the full potential of their data by designing and implementing scalable, secure, and high-performance analytics solutions on the AWS platform. In his free time, Shubham loves to spend time with his family and travel around the world.

Anirudh Chawla

Anirudh Chawla

Anirudh is an AWS Analytics Specialist Solution Architect. He helps organizations empower businesses to harness their data effectively through AWS’s analytics platform. His interest lies in building highly available distributed systems.

Stefano Sandonà

Stefano Sandonà

Stefano is a Senior Big Data Specialist Solution Architect at AWS. Passionate about data, distributed systems, and security, he helps customers worldwide architect scalable and secure data platforms, with a particular focus on open table formats to build performant and optimized transactional data lakes.

Unlock real-time data insights with schema evolution using Amazon MSK Serverless, Iceberg, and AWS Glue streaming

Post Syndicated from Nitin Kumar original https://aws.amazon.com/blogs/big-data/unlock-real-time-data-insights-with-schema-evolution-using-amazon-msk-serverless-iceberg-and-aws-glue-streaming/

Efficient real-time synchronization of data within data lakes present challenges. Any data inaccuracies or latency issues can significantly compromise analytical insights and subsequent business strategies. Organizations increasingly require synchronized data in near real-time to extract actionable intelligence and respond promptly to evolving market dynamics. Additionally, scalability remains a concern for data lake implementations, which must accommodate expanding volumes of streaming data and maintain optimal performance without incurring high operational costs.

Schema evolution is the process of modifying the structure (schema) of a data table to accommodate changes in the data over time, such as adding or removing columns, without disrupting ongoing operations or requiring a complete data rewrite. Schema evolution is vital in streaming data environments for several reasons. Unlike batch processing, streaming pipelines operate continuously, ingesting data in real time from sources that are actively serving production applications. Source systems naturally evolve over time as businesses add new features, refine data models, or respond to changing requirements. Without proper schema evolution capabilities, even minor changes to source schemas can force streaming pipeline shutdowns, requiring developers to manually reconcile schema differences and rebuild tables.

Such disruptions reduce the core value proposition of streaming architectures—continuous, low-latency data processing. Organizations can maintain uninterrupted data flows and keep source systems evolving independently by using the seamless schema evolution provided by Apache Iceberg. This reduces operational friction and maintains the availability of real-time analytics and applications even as underlying data structures change.

Apache Iceberg is an open table format, delivering essential capabilities for streaming workloads, including robust schema evolution support. This critical feature enables table schemas to adapt dynamically as source database structures evolve, maintaining operational continuity. Consequently, when database columns undergo additions, removals, or modifications, the data lake accommodates these changes seamlessly without requiring manual intervention or risking data inconsistencies.

Our comprehensive solution showcases an end-to-end real-time CDC pipeline that enables immediate processing of data modifications from Amazon Relational Database Service (Amazon RDS) for MySQL, streaming altered records directly to AWS Glue streaming jobs using Amazon Managed Streaming for Apache Kafka (Amazon MSK) Serverless. These jobs continually process incoming changes and update Iceberg tables on Amazon Simple Storage Service (Amazon S3) so that the data lake reflects the current state of the operational database environment in real time. By using Apache Iceberg’s comprehensive schema evolution support, our ETL pipeline automatically adapts to database schema modifications, providing data lake consistency and currentness without manual intervention. This approach combines complete process control with instantaneous analytics on operational data, eliminating traditional latency, and future-proofs the solution to address evolving organizational data needs. The architecture’s inherent flexibility facilitates adaptation to diverse use cases requiring immediate data insights.

Solution overview

To effectively address streaming challenges, we propose an architecture using Amazon MSK Serverless, a comprehensive managed Apache Kafka service that autonomously provisions and scales computational and storage resources. This solution offers a frictionless mechanism for ingesting and processing streaming data without the complexity of capacity management. Our implementation uses Amazon MSK Connect with the Debezium MySQL connector to capture and stream database modifications in real time. Rather than employing traditional batch processing methodologies, we implement an AWS Glue streaming job that directly consumes data from Kafka topics, processes CDC events as they occur, and writes transformed data to Apache Iceberg tables on Amazon S3.

The workflow consists of the following:

  1. Data flows from Amazon RDS through Amazon MSK Connect using the Debezium MySQL connector to Amazon MSK Serverless. This represents a CDC pipeline that captures database changes from the relational database and streams them to Kafka.
  2. From Amazon MSK Serverless, the data then moves to AWS Glue job, which processes the data and stores it in Amazon S3 as Iceberg tables. The AWS Glue job interacts with the AWS Glue Data Catalog to maintain metadata about the datasets.
  3. Analyze the data using the serverless interactive query service Amazon Athena, which can be used to query the iceberg table created in Data Catalog. This allows for interactive data analysis without managing infrastructure.

The following diagram illustrates the architecture that we implement through this post. Each number corresponds to the preceding list and shows major components that you implement.

Prerequisites

Before getting started, make sure you have the following:

  • An active AWS account with billing enabled
  • An AWS Identity and Access Management (IAM) user with specific permissions to create and manage resources, such as a virtual private cloud (VPC), subnet, security group, IAM roles, NAT gateway, internet gateway, Amazon Elastic Compute Cloud (Amazon EC2) client, MSK Serverless, MSK Connector and its plugin AWS Glue job, and S3 buckets.
  • Sufficient VPC capacity in your chosen AWS Region.

For this post, we create the solution resources in the US East (N. Virginia) – us-east-1 Region using AWS CloudFormation templates. In the following sections, we show you how to configure your resources and implement the solution.

Configuring CDC and processing using AWS CloudFormation

In this post, you use the CloudFormation template vpc-msk-mskconnect-rds-client-gluejob.yaml. This template sets up the streaming CDC pipeline resources such as a VPC, subnet, security group, IAM roles, NAT, internet gateway, EC2 client, MSK Serverless, MSK Connect, Amazon RDS, S3 buckets, and AWS Glue job.

To create the solution resources for the CDC pipeline, complete the following steps:

  1. Launch the stack vpc-msk-mskconnect-rds-client-gluejob.yaml using the CloudFormation template:
  2. Provide the parameter values as listed in the following table.

    A B C
    1 Parameters Description Sample value
    2 EnvironmentName An environment name that is prefixed to resource names. msk-iceberg-cdc-pipeline
    3 DatabasePassword Database admin account password. ****
    4 InstanceType MSK client EC2 instance type. t2.micro
    5 LatestAmiId Latest AMI ID of Amazon Linux 3 for ec2 instance. You can use the default value. /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64
    6 VpcCIDR IP range (CIDR notation) for this VPC. 10.192.0.0/16
    7 PublicSubnet1CIDR IP range (CIDR notation) for the public subnet in the first Availability Zone. 10.192.10.0/24
    8 PublicSubnet2CIDR IP range (CIDR notation) for the public subnet in the second Availability Zone. 10.192.11.0/24
    9 PrivateSubnet1CIDR IP range (CIDR notation) for the private subnet in the first Availability Zone. 10.192.20.0/24
    10 PrivateSubnet2CIDR IP range (CIDR notation) for the private subnet in the second Availability Zone. 10.192.21.0/24
    11 NumberOfWorkers Number of workers for AWS Glue streaming job. 3
    12 GlueWorkerType Worker type for AWS Glue streaming job. For example, G.1X. G.1X
    13 GlueDatabaseName Name of the AWS Glue Data Catalog database. glue_cdc_blogdb
    14 GlueTableName Name of the AWS Glue Data Catalog table. iceberg_cdc_tbl

The stack creation process can take approximately 25 minutes to complete. You can check the Outputs tab for the stack after the stack is created, as shown in the following screenshot.

Following the successful deployment of the CloudFormation stack, you now have a fully operational Amazon RDS database environment. The database instance contains the salesdb database with the customer table populated with 30 data records.

These records have been streamed to the Kafka topic through the Debezium MySQL connector implementation, establishing a reliable CDC pipeline. With this foundation in place, proceed to the next phase of the data architecture: near real-time data processing using the AWS Glue streaming job.

Run the AWS Glue streaming job

To transfer the data load from the Kafka topic (created by the Debezium MySQL connector for database table customer) to the Iceberg table, run the AWS Glue streaming job configured by the CloudFormation setup. This process will migrate all existing customer data from the source database table to the Iceberg table. Complete the following steps:

  1. On the CloudFormation console, choose the stack vpc-msk-mskconnect-rds-client-gluejob.yaml
  2. On the Outputs tab, retrieve the name of the AWS Glue streaming job from the GlueJobName row. In the following screenshot, the name is IcebergCDC-msk-iceberg-cdc-pipeline.
  3. On the AWS Glue console, choose ETL jobs in the navigation pane.
  4. Search for the AWS Glue job named IcebergCDC-msk-iceberg-cdc-pipeline.
  5. Choose the job name to open its details page.
  6. Choose Run to start the job. On the Runs tab, confirm if the job ran without failure.

You need to wait approximately 2 minutes for the job to process before continuing. This pause allows the jobrun to fully process records from the Kafka topic (initial load) and create the Iceberg table.

Query the Iceberg table using Athena

After the AWS Glue streaming job has successfully started and the Iceberg table has been created in the Data Catalog, follow these steps to validate the data using Athena:

  1. On the Athena console, navigate to the query editor.
  2. Choose the Data Catalog as the data source.
  3. Choose the database glue_cdc_blogdb.
  4. To validate the data, enter the following query to preview the data and find the total count:
    SELECT id, name, mktsegment FROM "glue_cdc_blogdb"."iceberg_cdc_tbl" order by id desc limit 40;
    
    SELECT count(*) as total_rows FROM "glue_cdc_blogdb"."iceberg_cdc_tbl";

    The following screenshot shows the output of the example query.

After performing the preceding steps, you’ve established a complete near real-time data processing pipeline by running an AWS Glue streaming job that transfers data from Kafka topics to an Apache Iceberg table, then verified the successful data migration by querying the results through Amazon Athena.

Upload incremental (CDC) data for further processing

Now that you’ve successfully completed the initial full data load, it’s time to focus on the dynamic aspects of the data pipeline. In this section, we explore how the system handles ongoing data modifications such as insertions, updates, and deletions in Amazon RDS for MySQL database. These changes won’t go unnoticed. Our Debezium MySQL connector stands ready to capture each modification event, transforming database changes into a continuous stream of data. Working in tandem with our AWS Glue streaming job, this architecture is designed to promptly process and propagate every change in our source database through our data pipeline.Let’s see this real-time data synchronization mechanism in action, demonstrating how our modern data infrastructure maintains consistency across systems with minimal latency. Follow these steps:

  1. On the Amazon EC2 console, access the EC2 instance that you created using the CloudFormation template named as KafkaClientInstance.
  2. Log in to the EC2 instance using AWS Systems Manager Agent (SSM Agent). Select the instance named as KafkaClientInstance and then choose Connect.
  3. Enter the following commands to insert the data into the RDS table. Use the same database password you entered when you created the CloudFormation stack.
    sudo su - ec2-user
    RDS_AURORA_ENDPOINT=`aws rds describe-db-instances --region us-east-1 | jq -r '.DBInstances[] | select(.DBName == "salesdb") | .Endpoint.Address'`
    mysql -f -u master -h $RDS_AURORA_ENDPOINT  --password

  4. Now perform the insert, update, and delete in the CUSTOMER table.
    use salesdb;
    
    INSERT INTO customer VALUES(31, 'Customer Name 31', 'Market segment 31');
    INSERT INTO customer VALUES(32, 'Customer Name 32', 'Market segment 32');
    
    UPDATE customer SET name='Customer Name update 29', mktsegment='Market segment update 29' WHERE id = 29;
    UPDATE customer SET name='Customer Name update 30', mktsegment='Market segment update 30' WHERE id = 30;
    
    DELETE FROM customer WHERE id = 27;
    DELETE FROM customer WHERE id = 28;
    

  5. Validate the data to verify the insert, update, and delete records in the Iceberg table from Athena, as shown in the following screenshot.

After performing the preceding steps, you’ve learned how our CDC pipeline handles ongoing data modifications by performing insertions, updates, and deletions in the MySQL database and verifying how these changes are automatically captured by Debezium MySQL connector, streamed through Kafka, and reflected in the Iceberg table in near real time.

Schema evolution: Adding new columns to the Iceberg table

The schema evolution mechanism in this implementation provides an automated approach to detecting and adding new columns from incoming data to existing Iceberg tables. Although Iceberg inherently supports robust schema evolution capabilities (including adding, dropping, and renaming columns, updating types, and reordering), this code specifically automates the column addition process for streaming environments. This automation uses Iceberg’s underlying schema evolution capabilities, which guarantee correctness through unique column IDs that ensure new columns never read existing values from another column. By handling column additions programmatically, the system reduces operational overhead in streaming pipelines where manual schema management would create bottlenecks. However, dropping and renaming columns, updating types, and reordering still required manual intervention.

When new data arrives through Kafka streams, the handle_schema_evolution() function orchestrates a four-step process to ensure seamless table schema updates.

  1. It analyzes the incoming batch DataFrame to infer its schema structure, cataloging all column names and their corresponding data types.
  2. It retrieves the existing Iceberg table’s schema from the AWS Glue catalog to establish a baseline for comparison.
  3. The system then performs a schema comparison using method compare_schemas() between batch schema with existing table schema.
    1. If the incoming frame contains fewer columns than the catalog table, no action is taken.
    2. It identifies any new columns present in the incoming data that don’t exist in the current table structure and returns a list of new columns that need to be added.
    3. New columns will be added at the last.
    4. Handle type evolution isn’t supported. If needed, you can handle the same at comment # Handle type evolution in the compare_schemas() method.
    5. If the destination table has columns that are dropped in the source table, it doesn’t drop those columns. If that is required for your use case, you can use drop column manually using ALTER TABLE ... DROP COLUMN.
    6. Renaming the column isn’t supported. To rename the column use case, manually evolve the schema using ALTER TABLE … RENAME COLUMN.
  4. Finally, if new columns are discovered, the function executes ALTER TABLE … ADD COLUMN statements to evolve the Iceberg table schema, adding the new columns with their appropriate data types.

This approach eliminates the need for manual schema management and prevents data pipeline failures that would typically occur when encountering unexpected fields in streaming data. The implementation also includes proper error handling and logging to track schema evolution events, making it particularly valuable for environments where data structures frequently change.

def infer_schema_from_batch(batch_df):
    """
    Infer schema from the batch DataFrame
    Returns a dictionary with column names and their inferred types
    """
    schema_dict = {}
    for field in batch_df.schema.fields:
        schema_dict[field.name] = field.dataType
    return schema_dict

def get_existing_table_schema(spark, table_identifier):
    """
    Read the existing table schema from the Iceberg table
    Returns a dictionary with column names and their types
    """
    try:
        existing_df = spark.table(table_identifier)
        schema_dict = {}
        for field in existing_df.schema.fields:
            schema_dict[field.name] = field.dataType
        return schema_dict
    except Exception as e:
        print(f"Error reading existing table schema: {e}")
        return {}

def compare_schemas(batch_schema, existing_schema):
    """
    Compare batch schema with existing table schema
    Returns a list of new columns that need to be added
    """
    new_columns = []
    
    for col_name, col_type in batch_schema.items():
        if col_name not in existing_schema:
            new_columns.append((col_name, col_type))
        elif existing_schema[col_name] != col_type:
            # Handle type evolution if needed
            print(f"Warning: Column {col_name} type mismatch - existing: {existing_schema[col_name]}, new: {col_type}")
    
    return new_columns

def spark_type_to_sql_string(spark_type):
    """
    Convert Spark DataType to SQL string representation for ALTER TABLE
    """
    type_mapping = {
        'IntegerType': 'INT',
        'LongType': 'BIGINT',
        'StringType': 'STRING',
        'BooleanType': 'BOOLEAN',
        'DoubleType': 'DOUBLE',
        'FloatType': 'FLOAT',
        'TimestampType': 'TIMESTAMP',
        'DateType': 'DATE'
    }
    
    type_name = type(spark_type).__name__
    return type_mapping.get(type_name, 'STRING')

def evolve_table_schema(spark, table_identifier, new_columns):
    """
    Alter the Iceberg table to add new columns
    """
    if not new_columns:
        return
    
    try:
        for col_name, col_type in new_columns:
            sql_type = spark_type_to_sql_string(col_type)
            alter_sql = f"ALTER TABLE {table_identifier} ADD COLUMN {col_name} {sql_type}"
            print(f"Executing schema evolution: {alter_sql}")
            spark.sql(alter_sql)
            print(f"Successfully added column {col_name} with type {sql_type}")
    except Exception as e:
        print(f"Error during schema evolution: {e}")
        raise e

def handle_schema_evolution(spark, batch_df, table_identifier):
    """
    schema evolution steps
    1. Infer schema from batch DataFrame
    2. Read existing table schema
    3. Compare schemas and identify new columns
    4. Alter table if schema evolved
    """
    # Step 1: Infer schema from batch DataFrame
    batch_schema = infer_schema_from_batch(batch_df)
    print(f"Batch schema: {batch_schema}")
    
    # Step 2: Read existing table schema
    existing_schema = get_existing_table_schema(spark, table_identifier)
    print(f"Existing table schema: {existing_schema}")
    
    # Step 3: Compare schemas
    new_columns = compare_schemas(batch_schema, existing_schema)
    
    # Step 4: Evolve schema if needed
    if new_columns:
        print(f"Schema evolution detected. New columns: {new_columns}")
        evolve_table_schema(spark, table_identifier, new_columns)
        return True
    else:
        print("No schema evolution needed")
        return False

In this section, we demonstrate how our system handles structural changes to the underlying data model by adding a new status column to the customer table and populating it with default values. Our architecture is designed to seamlessly propagate these schema modifications throughout the pipeline so that downstream analytics and processing capabilities remain uninterrupted while accommodating the enhanced data model. This flexibility is essential for maintaining a responsive, business-aligned data infrastructure that can evolve alongside changing organizational needs.

  1. Add a new status column to the customer table and populate it with default values as Green.
    use salesdb;
    
    ALTER TABLE customer ADD COLUMN status VARCHAR(20) NOT NULL;
    
    UPDATE customer SET status = 'Green';
    

  2. Use the Athena console to validate the data and schema evolution, as shown in the following screenshot.

When schema evolution occurs in an Iceberg table, the metadata.json file undergoes specific updates to track and manage these changes. In job when schema evolution detected, it ran the following query to evolve the schema for the Iceberg table.

ALTER TABLE glue_catalog.glue_cdc_blogdb.iceberg_cdc_tbl ADD COLUMN status string

We checked the metadata.json file in Amazon S3 for iceberg table location, and the following screenshot shows how the schema evolved.

We now explain how our implementation handles schema evolution by automatically detecting and adding new columns from incoming data streams to existing Iceberg tables. The system employs a four-step process that analyzes incoming data schemas, compares them with existing table structures, identifies new columns, and executes the necessary ALTER TABLE statements to evolve the schema without manual intervention, though certain schema changes still require manual handling.

Clean up

To clean up your resources, complete the following steps:

  1. Stop the running AWS Glue streaming job:
    1. On the AWS Glue console, choose ETL jobs in the navigation pane.
    2. Search for the AWS Glue job named IcebergCDC-msk-iceberg-cdc-pipeline.
    3. Choose the job name to open its details page.
    4. On the Runs tab, select running jobrun and choose Stop job run. Confirm that the job stopped successfully.
  2. Remove the AWS Glue database and table:
    1. On the AWS Glue console, choose Tables in the navigation pane, select iceberg_cdc_tbl, and choose Delete.
    2. Choose Databases in the navigation pane, select glue_cdc_blogdb, and choose Delete.
  3. Delete the CloudFormation stack vpc-msk-mskconnect-rds-client-gluejob.yaml.

Conclusion

This post showcases a solution that businesses can use to access real-time data insights without the traditional delays between data creation and analysis. By combining Amazon MSK Serverless, Debezium MySQL connector, AWS Glue streaming, and Apache Iceberg tables, the architecture captures database changes instantly and makes them immediately available for analytics through Amazon Athena. A standout feature is the system’s ability to automatically adapt when database structures change—such as adding new columns—without disrupting operations or requiring manual intervention. This eliminates the technical complexity typically associated with real-time data pipelines and provides business users with the most current information for decision-making, effectively bridging the gap between operational databases and analytical systems in a cost-effective, scalable way.


About the Authors

Nitin Kumar

Nitin Kumar

Nitin is a Cloud Engineer (ETL) at AWS, specializing in AWS Glue. With a decade of experience, he excels in aiding customers with their big data workloads, focusing on data processing and analytics. In his free time, he likes to watch movies and spend time with his family.

Shubham Purwar

Shubham Purwar

Shubham is an Analytics Specialist Solutions Architect at AWS. He helps organizations unlock the full potential of their data by designing and implementing scalable, secure, and high-performance analytics solutions on AWS. In his free time, Shubham loves to spend time with his family and travel around the world.

Noritaka Sekiyama

Noritaka Sekiyama

Noritaka is a Principal Big Data Architect on the AWS Glue team. He works based in Tokyo, Japan. He is responsible for building software artifacts to help customers. In his spare time, he enjoys cycling with his road bike.

Migrate Delta tables from Azure Data Lake Storage to Amazon S3 using AWS Glue

Post Syndicated from Nitin Kumar original https://aws.amazon.com/blogs/big-data/migrate-delta-tables-from-azure-data-lake-storage-to-amazon-s3-using-aws-glue/

Organizations are increasingly using a multi-cloud strategy to run their production workloads. We often see requests from customers who have started their data journey by building data lakes on Microsoft Azure, to extend access to the data to AWS services. Customers want to use a variety of AWS analytics, data, AI, and machine learning (ML) services like AWS Glue, Amazon Redshift, and Amazon SageMaker to build more cost-efficient, performant data solutions harnessing the strength of individual cloud service providers for their business use cases.

In such scenarios, data engineers face challenges in connecting and extracting data from storage containers on Microsoft Azure. Customers typically use Azure Data Lake Storage Gen2 (ADLS Gen2) as their data lake storage medium and store the data in open table formats like Delta tables, and want to use AWS analytics services like AWS Glue to read the delta tables. AWS Glue, with its ability to process data using Apache Spark and connect to various data sources, is a suitable solution for addressing the challenges of accessing data across multiple cloud environments.

AWS Glue is a serverless data integration service that makes it straightforward to discover, prepare, and combine data for analytics, ML, and application development. AWS Glue custom connectors allow you to discover and integrate additional data sources, such as software as a service (SaaS) applications and your custom data sources. With just a few clicks, you can search for and subscribe to connectors from AWS Marketplace and begin your data preparation workflow in minutes.

In this post, we explain how you can extract data from ADLS Gen2 using the Azure Data Lake Storage Connector for AWS Glue. We specifically demonstrate how to import data stored in Delta tables in ADLS Gen2. We provide step-by-step guidance on how to configure the connector, author an AWS Glue ETL (extract, transform, and load) script, and load the extracted data into Amazon Simple Storage Service (Amazon S3).

Azure Data Lake Storage Connector for AWS Glue

The Azure Data Lake Storage Connector for AWS Glue simplifies the process of connecting AWS Glue jobs to extract data from ADLS Gen2. It uses the Hadoop’s FileSystem interface and the ADLS Gen2 connector for Hadoop. The Azure Data Lake Storage Connector for AWS Glue also includes the hadoop-azure module, which lets you run Apache Hadoop or Apache Spark jobs directly with data in ADLS. When the connector is added to the AWS Glue environment, AWS Glue loads the library from the Amazon Elastic Container Registry (Amazon ECR) repository during initialization (as a connector). When AWS Glue has internet access, the Spark job in AWS Glue can read from and write to ADLS.

With the availability of the Azure Data Lake Storage Connector for AWS Glue in AWS Marketplace, an AWS Glue connection makes sure you have the required packages to use in your AWS Glue job.

For this post, we use the Shared Key authentication method.

Solution overview

In this post, our objective is to migrate a product table named sample_delta_table, which currently resides in ADLS Gen2, to Amazon S3. To accomplish this, we use AWS Glue, the Azure Data Lake Storage Connector for AWS Glue, and AWS Secrets Manager to securely store the Azure shared key. We employed an AWS Glue serverless ETL job, configured with the connector, to establish a connection to ADLS using shared key authentication over the public internet. After the table is migrated to Amazon S3, we use Amazon Athena to query Delta Lake tables.

The following architecture diagram illustrates how AWS Glue facilitates data ingestion from ADLS.

Prerequisites

You need the following prerequisites:

Configure your ADLS Gen2 account in Secrets Manager

Complete the following steps to create a secret in Secrets Manager to store the ADLS credentials:

  1. On the Secrets Manager console, choose Store a new secret.
  2. For Secret type, select Other type of secret.
  3. Enter the key accountName for the ADLS Gen2 storage account name.
  4. Enter the key accountKey for the ADLS Gen2 storage account key.
  5. Enter the key container for the ADLS Gen2 container.
  6. Leave the rest of the options as default and choose Next.

  1. Enter a name for the secret (for example, adlstorage_credentials).
  2. Choose Next.
  3. Complete the rest of the steps to store the secret.

Subscribe to the Azure Data Lake Storage Connector for AWS Glue

The Azure Data Lake Storage Connector for AWS Glue simplifies the process of connecting AWS Glue jobs to extract data from ADLS Gen2. The connector is available as an AWS Marketplace offering.

Complete the following steps to subscribe to the connector:

  1. Log in to your AWS account with the necessary permissions.
  2. Navigate to the AWS Marketplace page for the Azure Data Lake Storage Connector for AWS Glue.
  3. Choose Continue to Subscribe.
  4. Choose Continue to Configuration after reading the EULA.

  1. For Fulfilment option, choose Glue 4.0.
  2. For Software version, choose the latest software version.
  3. Choose Continue to Launch.

Create a custom connection in AWS Glue

After you’re subscribed to the connector, complete the following steps to create an AWS Glue connection based on it. This connection will be added to the AWS Glue job to make sure the connector is available and the data store connection information is accessible to establish a network pathway.

To create the AWS Glue connection, you need to activate the Azure Data Lake Storage Connector for AWS Glue on the AWS Glue Studio console. After you choose Continue to Launch in the previous steps, you’re redirected to the connector landing page.

  1. In the Configuration details section, choose Usage instructions.
  2. Choose Activate the Glue connector from AWS Glue Studio.

The AWS Glue Studio console allows the option to either activate the connector or activate it and create the connection in one step. For this post, we choose the second option.

  1. For Connector, confirm Azure ADLS Connector for AWS Glue 4.0 is selected.
  2. For Name, enter a name for the connection (for example, AzureADLSStorageGen2Connection).
  3. Enter an optional description.
  4. Choose Create connection and activate connector.

The connection is now ready for use. The connector and connection information is visible on the Data connections page of the AWS Glue console.


Read Delta tables from ADLS Gen2 using the connector in an AWS Glue ETL job

Complete the following steps to create an AWS Glue job and configure the AWS Glue connection and job parameter options:

  1. On the AWS Glue console, choose ETL jobs in the navigation pane.
  2. Choose Author code with a script editor and choose Script editor.
  3. Choose Create script and go to the Job details section.
  4. Update the settings for Name and IAM role.
  5. Under Advanced properties, add the AWS Glue connection AzureADLSStorageGen2Connection created in previous steps.
  1. For Job parameters, add the key --datalake-formats with the value as delta.
  1. Use the following script to read the Delta table from ADLS. Provide the path to where you have Delta table files in your Azure storage account container and the S3 bucket for writing delta files to the output S3 location.
from pyspark.sql import SparkSession
from delta.tables import *
import boto3
import json

spark = SparkSession.builder.getOrCreate()

sm = boto3.client('secretsmanager')
response = sm.get_secret_value(SecretId="adlstorage_credentials")
value = json.loads(response['SecretString'])
account_name_sparkconfig = f"fs.azure.account.key.{value['accountName']}.dfs.core.windows.net"
account_name = value['accountName']
account_key = value['accountKey']
container_name = value['container']
path = f"abfss://{container_name}@{account_name}.dfs.core.windows.net/path-to-delta-table-files/"
s3DeltaTablePath="s3://yourdatalakebucketname/deltatablepath/"

# Method: Shared Key  
spark.conf.set(account_name_sparkconfig, account_key)

# Read delta table from ADLS gen2 storage
df = spark.read.format("delta").load(path)

# Write delta table to S3 path.
if DeltaTable.isDeltaTable(spark,s3DeltaTablePath):
    s3deltaTable = DeltaTable.forPath(spark,s3DeltaTablePath)
    print("Merge to existing s3 delta table")
    (s3deltaTable.alias("target")
        .merge(df.alias("source"), "target.product_id = source.product_id")
        .whenMatchedUpdateAll()
        .whenNotMatchedInsertAll()
        .execute()
    )
else:
    print("Create delta table to S3.")
    df.write.format("delta").save(s3DeltaTablePath)
  1. Choose Run to start the job.
  2. On the Runs tab, confirm the job ran successfully.
  3. On the Amazon S3 console, verify the delta files in the S3 bucket (Delta table path).
  4. Create a database and table in Athena to query the migrated Delta table in Amazon S3.

You can accomplish this step using an AWS Glue crawler. The crawler can automatically crawl your Delta table stored in Amazon S3 and create the necessary metadata in the AWS Glue Data Catalog. Athena can then use this metadata to query and analyze the Delta table seamlessly. For more information, see Crawl Delta Lake tables using AWS Glue crawlers.

CREATE DATABASE deltadb;
CREATE EXTERNAL TABLE deltadb.sample_delta_table
LOCATION 's3://yourdatalakebucketname/deltatablepath/'
TBLPROPERTIES ('table_type'='DELTA');

12. Query the Delta table:

SELECT * FROM "deltadb"."sample_delta_table" limit 10;

By following the steps outlined in the post, you have successfully migrated a Delta table from ADLS Gen2 to Amazon S3 using an AWS Glue ETL job.

Read the Delta table in an AWS Glue notebook

The following are optional steps if you want to read the Delta table from ADLS Gen2 in an AWS Glue notebook:

  1. Create a notebook and run the following code in the first notebook cell to configure the AWS Glue connection and --datalake-formats in an interactive session:
%idle_timeout 30
%glue_version 4.0
%worker_type G.1X
%number_of_workers 5
%connections AzureADLSStorageGen2Connection
%%configure
{
   "--datalake-formats": "delta"
}

  1. Run the following code in a new cell to read the Delta table stored in ADLS Gen 2. Provide the path to where you have delta files in an Azure storage account container and the S3 bucket for writing delta files to Amazon S3.
from pyspark.sql import SparkSession
from delta.tables import *
import boto3
import json

spark = SparkSession.builder.getOrCreate()

sm = boto3.client('secretsmanager')
response = sm.get_secret_value(SecretId="adlstorage_credentials")
value = json.loads(response['SecretString'])
account_name_sparkconfig = f"fs.azure.account.key.{value['accountName']}.dfs.core.windows.net"
account_name = value['accountName']
account_key = value['accountKey']
container_name = value['container']
path = f"abfss://{container_name}@{account_name}.dfs.core.windows.net/path-to-delta-table-files/"
s3DeltaTablePath="s3://yourdatalakebucketname/deltatablepath/"

# Method: Shared Key  
spark.conf.set(account_name_sparkconfig, account_key)

# Read delta table from ADLS gen2 storage
df = spark.read.format("delta").load(path)

# Write delta table to S3 path.
if DeltaTable.isDeltaTable(spark,s3DeltaTablePath):
    s3deltaTable = DeltaTable.forPath(spark,s3DeltaTablePath)
    print("Merge to existing s3 delta table")
    (s3deltaTable.alias("target")
        .merge(df.alias("source"), "target.product_id = source.product_id")
        .whenMatchedUpdateAll()
        .whenNotMatchedInsertAll()
        .execute()
    )
else:
    print("Create delta table to S3.")
    df.write.format("delta").save(s3DeltaTablePath)

Clean up

To clean up your resources, complete the following steps:

  1. Remove the AWS Glue job, database, table, and connection:
    1. On the AWS Glue console, choose Tables in the navigation pane, select sample_delta_table, and choose Delete.
    2. Choose Databases in the navigation pane, select deltadb, and choose Delete.
    3. Choose Connections in the navigation pane, select AzureADLSStorageGen2Connection, and on the Actions menu, choose Delete.
  2. On the Secrets Manager console, choose Secrets in the navigation pane, select adlstorage_credentials, and on the Actions menu, choose Delete secret.
  3. If you are no longer going to use this connector, you can cancel the subscription to the connector:
    1. On the AWS Marketplace console, choose Manage subscriptions.
    2. Select the subscription for the product that you want to cancel, and on the Actions menu, choose Cancel subscription.
    3. Read the information provided and select the acknowledgement check box.
    4. Choose Yes, cancel subscription.
  4. On the Amazon S3 console, delete the data in the S3 bucket that you used in the previous steps. 

You can also use the AWS Command Line Interface (AWS CLI) to remove the AWS Glue and Secrets Manager resources. Remove the AWS Glue job, database, table, connection, and Secrets Manager secret with the following command:

aws glue delete-job —job-name <your_job_name>
aws glue delete-connection —connection-name <your_connection_name>
aws secretsmanager delete-secret —secret-id <your_secretsmanager_id>
aws glue delete-table --database-name deltadb --name sample_delta_table
aws glue delete-database --name deltadb

Conclusion

In this post, we demonstrated a real-world example of migrating a Delta table from Azure Delta Lake Storage Gen2 to Amazon S3 using AWS Glue. We used an AWS Glue serverless ETL job, configured with an AWS Marketplace connector, to establish a connection to ADLS using shared key authentication over the public internet. Additionally, we used Secrets Manager to securely store the shared key and seamlessly integrate it within the AWS Glue ETL job, providing a secure and efficient migration process. Lastly, we provided guidance on querying the Delta Lake table from Athena.

Try out the solution for your own use case, and let us know your feedback and questions in the comments.


About the Authors

Nitin Kumar is a Cloud Engineer (ETL) at Amazon Web Services, specialized in AWS Glue. With a decade of experience, he excels in aiding customers with their big data workloads, focusing on data processing and analytics. He is committed to helping customers overcome ETL challenges and develop scalable data processing and analytics pipelines on AWS. In his free time, he likes to watch movies and spend time with his family.

Shubham Purwar is a Cloud Engineer (ETL) at AWS Bengaluru, specialized in AWS Glue and Amazon Athena. He is passionate about helping customers solve issues related to their ETL workload and implement scalable data processing and analytics pipelines on AWS. In his free time, Shubham loves to spend time with his family and travel around the world.

Pramod Kumar P is a Solutions Architect at Amazon Web Services. With 19 years of technology experience and close to a decade of designing and architecting connectivity solutions (IoT) on AWS, he guides customers to build solutions with the right architectural tenets to meet their business outcomes.

Madhavi Watve is a Senior Solutions Architect at Amazon Web Services, providing help and guidance to a broad range of customers to build highly secure, scalable, reliable, and cost-efficient applications on the cloud. She brings over 20 years of technology experience in software development and architecture and is data analytics specialist.

Swathi S is a Technical Account Manager with the Enterprise Support team in Amazon Web Services. She has over 6 years of experience with AWS on big data technologies and specializes in analytics frameworks. She is passionate about helping AWS customers navigate the cloud space and enjoys assisting with design and optimization of analytics workloads on AWS.