A coordination framework for formal proof development with Kiro

Post Syndicated from Haoyu Li original https://aws.amazon.com/blogs/devops/a-coordination-framework-for-formal-proof-development-with-kiro/

Large language models (LLMs) and coding agents are increasingly capable of proving difficult mathematical theorems and completing formally verified Lean programs. Yet, existing agentic provers rely on the backend models’ capabilities while lacking the ability to explore when they get stuck: when a proof resembles patterns the model already knows, it might succeed quickly. When it doesn’t, the model can spend thousands of tokens repeating failed tactics, lose track of intermediate results, or abandon the proof altogether.

We develop a prover harness, MAESTRO, in Kiro CLI that gives LLMs a systematic framework for exploring, decomposing, and making structured progress on difficult problems. The prover constructs a shared proof skeleton and organizes the work across specialized agents that explore independent proof obligations concurrently. A persistent coordinator evaluates their feedback, repairs broken proof structure, and decomposes difficult goals into smaller lemmas when direct attempts fail. This turns proof generation into an iterative process of planning, proving, diagnosing, and revising.

We test on a series of challenging benchmark datasets, including ArxivLean (research-level mathematical theorem proving) and FormalQualBench (graduate-level mathematical theorem proving). Across these, Kiro CLI with GPT and Claude models consistently solves more hard problems than Codex and Claude Code with the corresponding models. We apply Kiro CLI to prove the six problems from the 2026 International Mathematical Olympiad (IMO, held July 15–16, 2026). Kiro CLI solves all six problems in 1 hour 46 minutes. Previously reported total times were 8 hours 18 minutes for Humanfia and 24 hours 56 minutes for AxiomProver. We also find that Kiro CLI can discover and test novel proof strategies, suggesting potential applications in mathematical research.

Kiro CLI strengthens base LLMs’ proving capabilities

We evaluate Kiro CLI across several established benchmarks and compare it against plain agent harnesses, Codex and Claude Code, using the same underlying base models. These benchmarks are:

  • PutnamBench, formalized undergraduate Putnam competition problems.
  • FormalQualBench, graduate qualifying-examination problems.
  • ArxivLean, theorem statements drawn from recent research papers.
  • Neural Theorem Proving for Verification Conditions (NTP4VC), verification conditions extracted from real-world software-verification projects.

Together, they span diverse levels and domains of formal reasoning. Across this spectrum, Kiro CLI consistently achieves higher solve rates, with the gap becoming particularly pronounced on the harder research-level problems. This suggests that the gains are not simply because of stronger base models. The specialized coordination machinery in MAESTRO makes a substantial difference in how effectively those models can solve difficult mathematics.

Benchmark Number of tasks GPT 5.6 Sol Claude Opus 5
Kiro CLI Codex Improvement Kiro CLI Claude Code Improvement
PutnamBench 672 690 656 +14 671 645 +26
NTP4VC 600 599 588 +11 600 589 +11
ArxivLean – June 48 33 24 +9 37 17 +20
FormalQualBench 23 14 11 +3 16 9 +7

We evaluate Kiro CLI across five reasoning-effort levels for GPT 5.6 Sol and Claude Opus 5 on a dataset comprising six challenging IMO 2026 problems, formalized by AxiomMath. With GPT 5.6 Sol, Kiro CLI solves five of the six problems at low effort and all six at medium effort or higher. By comparison, Codex solves none at low effort, three at medium, five at high, and all six at the two highest levels. With Claude Opus 5, Kiro CLI solves all six problems at every effort level. Claude Code solves three at low and medium effort and five at each higher level. These results demonstrate that our harness substantially strengthens the base models’ theorem-proving capabilities.

Solve rates for Kiro CLI, Codex, and Claude Code across five reasoning-effort levels on the IMO 2026 problems

Figure 1: Solve rates across five reasoning-effort levels on the IMO 2026 problems

Kiro CLI outperforms existing custom solutions

We compare Kiro CLI with existing proving systems on the IMO 2026 dataset. Across all six IMO 2026 problems, Kiro CLI completes the proofs in 1 hour 46 minutes. Compared with 8 hours 18 minutes for Humanfia, 8 hours 57 minutes for Pramaana, and 24 hours 56 minutes for AxiomProver, Kiro CLI is 4.7×, 5.1×, and 14.1× faster, respectively. The advantage is consistent across the individual problems, suggesting that the coordination framework improves not only success rate, but also the efficiency with which difficult proofs are found.

Proof-completion times for Kiro CLI compared with Humanfia, Pramaana, and AxiomProver on the IMO 2026 problems

Figure 2: Proof-completion times for Kiro CLI vs. existing custom solutions on IMO 2026

The core of the Kiro CLI prover: Coordination

The core of the Kiro CLI prover harness is coordination. Rather than treating theorem proving as a collection of independent lemma-solving jobs, it maintains a single evolving proof plan. Prover agents work on individual obligations and return checked lemmas, counterexamples, partial proofs, and structured reports about what is or isn’t working. A persistent coordinator reads this feedback together with the current Lean development and continuously revises the global proof architecture. It can preserve useful results from a failed branch and repair broken interfaces between lemmas. When a helper is false, it retracts both the helper and the downstream structure that depends on it. The coordinator can also introduce new abstractions, redirect workers, or leave productive branches alone. This allows local discoveries to influence the entire proof rather than being lost inside isolated attempts. In particular, the prover can recognize a flawed decomposition and restructure the proof around new evidence. Moreover, it can discover novel mathematical arguments.

Example 1: Repairing the proof architecture

We see this behavior in an Opus 5 run on the Quillen–Suslin theorem from FormalQualBench. At one point, a worker proposed simplifying the proof by keeping track of only part of the algebraic structure after a localization step. A later worker found a concrete counterexample showing that this simplification had thrown away information that was essential to completing the proof. In fact, the proposed intermediate lemma was stronger than what the original theorem required and would imply a substantially harder open statement.

Rather than repeatedly trying to prove the bad lemma, the coordinator recognized that the proof had been decomposed incorrectly. It recorded the counterexample, rejected later proposals that made the same mistake, and redesigned the intermediate interface so that both complementary pieces of the structure were preserved. After this information was carried through the remaining steps, the two pieces could be recombined into a global basis and the proof completed.

Example 2: Discovering novel proof strategies

Coordination can also lead Kiro CLI to discover new mathematical constructions. One example comes from the ArxivLean theorem Real Matrices as Linear Combinations of Three Orthogonal Matrices.

Both the original paper and our prover begin with the same standard simplification: reduce the matrix problem to a problem about its singular values. From there, however, the proofs diverge. The original paper first solves the easier case in which all singular values are distinct. When some values are repeated, it slightly perturbs the matrix to remove these repetitions, applies the distinct-value construction, and then uses a limiting argument to recover the original matrix. Our prover instead found a direct construction that works even when singular values are repeated. It groups the singular values into overlapping two-dimensional blocks and chooses intermediate values so that neighboring blocks fit together consistently. These local constructions then telescope into three global orthogonal matrices whose linear combination gives the original matrix.

Thus, it doesn’t merely translate the argument from the paper into Lean. It introduces different intermediate objects and reorganizes the proof around them, producing a route that directly handles the case for which the original proof uses perturbation and compactness.

Conclusions

Proof development isn’t only a collection of frontier model workers working individually. What matters is how we organize the work. MAESTRO in Kiro CLI is built around this observation: individual provers explore locally, while a persistent coordinator accumulates evidence and continuously reshapes the global proof. Through this coordination-heavy design, the prover has reached nearly saturating performance on several benchmarks of competition-level formal mathematics, while also showing promising results on substantially harder, research-level problems. You can find more technical details about MAESTRO in our paper, Maestro: Agentic Formal Proof Development by Coordination.


About the authors

Haoyu Li

Haoyu Li

Haoyu is a Ph.D. student in Computer Science at the University of Illinois Urbana-Champaign, where he researches techniques and algorithms for formal verification. His research focuses on trustworthy machine learning, AI safety, and formal methods. Haoyu was an intern at AWS AI Labs. Learn more at haoyuli02.github.io.

Zeren Shui

Zeren Shui

Zeren is an Applied Scientist at AWS AI Labs. His research interests span a broad spectrum of deep learning, large language models, agentic AI with a focus on the development of coding agents. Zeren holds a Ph.D. in computer science from University of Minnesota – Twin Cities.

Yuhao Zhang

Yuhao Zhang

Yuhao is a Senior Applied Scientist at AWS AI Labs, where he researches techniques and algorithms that increase the trustworthiness of AI coding agent. His primary research focuses on the fields of software engineering, programming languages, and artificial intelligence. Learn more at: https://foreverzyh.github.io/.

Ferhat Erata

Ferhat Erata

Ferhat is an Applied Scientist at AWS AI Labs. He combines large language models with formal reasoning to improve the safety and security of generative and agentic AI. His work includes autoformalization, verified code generation, and training reasoning models with verifier feedback. His offensive security research includes reverse engineering quantum circuits from power traces and detecting side-channel vulnerabilities in post-quantum cryptographic code. Over more than a decade in industry before his doctoral studies, he co-founded UNIT Research & Development, led the development of ModelWriter, a platform connecting text documents with formal models, and coordinated national consortia for EUREKA/ITEA industrial research projects. Ferhat holds a PhD in computer science from Yale University. His website is https://ferhat.ai/.

Murali Krishna Ramanathan

Murali Krishna Ramanathan

Murali is a Senior Principal Scientist at Amazon Web Services (AWS), where he works on AI-powered developer tools. Previously at Uber, he was the architect of Piranha, an open-source tool that automates the removal of stale feature-flag code, built before large language models made automated code transformation mainstream. Earlier, as a faculty member at the Indian Institute of Science (IISc), Bangalore, he conducted research in program analysis and automated testing, work recognized with a Google Faculty Research Award and ISSTA Distinguished Paper Awards. He holds a Ph.D. in Computer Science from Purdue University.

Hao Zhou

Hao Zhou

Hao is a Senior Applied Scientist at AWS AI Labs. His research interest is in machine learning, optimization, and big data analysis. He currently works on LLM training, inference, and agents, with a focus on repository-level coding tasks. He holds a PhD in Electrical Engineering from Northwestern University.

Glasswing overwhelms maintainers with underwhelming findings (VulnCheck)

Post Syndicated from jzb original https://lwn.net/Articles/1107173/

Patrick Garrity of VulnCheck has written a blog
post
that looks at vulnerabilities found by Anthropic’s Project Glasswing since it
launched on April 7, and finds it wanting:

Of the 37,966 findings Anthropic reports, only 7,349 (19.4%) have been made
public. Just 217 of those have been marked fixed. The more glaring number is
that 4.4% (1,654) of the project’s findings have already been marked as
duplicates, 7.5x more duplicates than fixes. On top of that, 419 findings have
been withdrawn. It seems odd that the ledger now claims a 94.2% true positive
rate, up from 90.8% at launch, when more findings have been withdrawn or marked
as duplicates than fixed, likely a feedback loop missing to the triage rates
listed as “True Positives”.

[…] AI is a valuable tool for finding vulnerabilities, and I’m not
discounting that. But seven months in, the evidence still doesn’t match the
launch narrative. The CVEs are a sliver of this year’s total, almost none are
being exploited, fixes have stalled while duplicates and withdrawals pile up,
and the triage burden is shifting onto the open source maintainers Glasswing
promised to help. The receipts keep coming in. They still don’t reconcile.

Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake

Post Syndicated from Nitin Kumar original https://aws.amazon.com/blogs/big-data/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake/

A cloud data lake is a scalable, low-cost place to store data from many different sources. Over the years, data lakes built on Amazon Simple Storage Service (Amazon S3) have become a common home for enterprise data, used by teams who query it for analytics and machine learning. However, data lakes have historically made it hard to update and delete records, keep transactions isolated, and handle concurrent changes safely. Open table formats such as Apache Hudi, Delta Lake, and Apache Iceberg were created to solve these problems. They add ACID transactions and row-level updates and delete, along with snapshot history and schema enforcement that make data lakes easier to audit and govern.

As organizations’ data architectures mature and evolve, the need to migrate between these table formats becomes increasingly common. This transition is often driven by changing requirements, performance considerations, or the desire to use newer features. Among these migrations, the shift from Apache Hudi to Apache Iceberg has gained particular attention, as organizations seek to capitalize on Iceberg’s modern architecture and advanced capabilities.

In this post, we show how to convert Apache Hudi tables to Apache Iceberg, using a Jupyter notebook running on AWS Glue 5.1 interactive sessions.

Benefits of migration to Iceberg

Enhanced partition evolution

Iceberg supports dynamic changes to partitioning schemes without requiring complete data rewrites. This capability is valuable in evolving data environments where query patterns change over time. You can adjust your partitioning strategy to optimize performance, reducing the operational overhead traditionally associated with such changes. Your data lake can adapt to changing business requirements without costly reorganization efforts.

Hidden partitioning

In addition to standard partitioning on an existing column, Iceberg supports hidden partitioning, which decouples the physical partition layout from the columns that queries filter on. The table specification declares a partition transform, which is a rule that tells Iceberg how to derive the partition value from a source column, such as a time bucket or a hash of the column value. The transform is defined once at the table level, and Iceberg applies it automatically on every write. This removes a long-standing source of friction on both sides of the table: writers no longer need to extract and maintain a separate partition column alongside the real one at ingestion time, and queries no longer need to carry an extra predicate on that derived column to get good performance.

Openness with the Iceberg REST Catalog

The Iceberg REST Catalog specification defines a standard HTTP protocol for catalog operations. Any engine that speaks the protocol can talk to any catalog that implements it, without a catalog-specific client library or implementation JAR on the class path. Integration moves from a code and dependency problem to a configuration problem, making Iceberg a strong fit for multi-engine data architectures.

Deep integration with AWS services

Iceberg has native integration across the AWS analytics ecosystem. Amazon Athena, Amazon Redshift, Amazon EMR, and AWS Glue all read and write Iceberg tables directly. The AWS Glue Data Catalog provides catalog and metadata management for these tables, and Amazon SageMaker builds on the same open Lakehouse foundation, letting analytics and artificial intelligence and machine learning (AI/ML) workloads share a single copy of data across Amazon S3 data lakes (including Amazon S3 Tables), with SageMaker Feature Store supporting an Iceberg offline store for faster training dataset extraction. On the ingestion side, Amazon Data Firehose delivers data into Iceberg tables from sources such as Direct PUT, Amazon Kinesis Data Streams, Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon CloudWatch Logs, and AWS IoT, with optional on-the-fly record transformation by using AWS Lambda. No-code Streaming Tables map a single Kinesis Data Streams stream or a single Amazon MSK (Express brokers) Kafka topic to a single Amazon S3 Tables Iceberg table, with the target table auto-created and partitioning configured on the delivery channel.

AWS managed features

AWS provides managed capabilities specifically designed for Iceberg. These managed features reduce operational overhead while maintaining performance and cost efficiency, so teams can focus on deriving insights rather than managing infrastructure.

  • AWS Glue Iceberg table optimizer – Automated compaction, snapshot management, and orphan file cleanup without manual intervention on Iceberg tables in General S3 buckets.
  • Amazon S3 Tables – Fully managed Iceberg tables with automatic optimization and maintenance.
  • Zero-ETL integration – Managed, incremental replication from Amazon DynamoDB and third-party enterprise applications such as Salesforce, SAP, ServiceNow, and Zendesk into Iceberg tables in Amazon S3 (self-managed or Amazon S3 Tables), with no custom pipeline or connector code to maintain. Source changes are applied incrementally, so the target stays in sync without full reloads.
  • Iceberg materialized views – SQL-defined views materialized as native Iceberg tables, refreshed on an automatic schedule and incrementally when the source tables allow it, with change detection handled for you. Well suited to consumption-layer tables that feed dashboards, APIs, and downstream pipelines, since the work shifts from query time to the managed refresh.

These benefits make Apache Iceberg a strong choice for modern data lake implementations, offering advantages in performance, manageability, and scalability compared to traditional table formats.

In the context of migrating data to Apache Iceberg, two primary approaches exist: in-place migration and full table migration. We previously covered both in the blog post “Migrate an existing data lake to a transactional data lake using Apache Iceberg” which walks through the methodology for traditional Hive-style tables, meaning partitioned Apache Parquet, Apache ORC, or Apache Avro datasets registered in a Hive-compatible catalog. The same two approaches apply when the starting point is an Apache Hudi data lake, which comes in two flavors, Copy-on-Write (CoW) and Merge-on-Read (MoR).

In-place migration

This approach generates fresh Iceberg metadata on top of the existing data files of the source table, without rewriting the data itself. The Parquet files of a Hive, Hudi, or Delta Lake table stay in place, and a new or existing Iceberg table is pointed at them with its own metadata layer. The main advantage is cost and speed: no data rewrite, so the migration is fast and cheap on both compute and storage. The main limitation is that the Iceberg table inherits the source table’s schema, partitioning, and physical file shape (file sizes, row groups, sort order), and none of these can be changed during the migration itself.

Full table migration

In this approach, the data itself is rewritten into a new Iceberg table, typically using a writeTo() call or a CREATE TABLE AS SELECT (CTAS) statement. It works for any source an engine can read from (Hive, Hudi, Delta Lake, relational databases, and others), and because the data is being rewritten anyway, the migration step is also where you can reshape it: change the schema, apply Iceberg hidden partitioning, resort the data for pruning, and target larger file sizes than the source has. The trade-offs are cost, time, and storage: higher compute spend, longer migration, and the dataset is effectively stored twice (source and target) until the migration completes, and the source is retired. For large historical tables, a single CTAS is usually impractical, so the common pattern is to seed the table with a small CTAS and then run INSERT INTO ... SELECT partition by partition, parallelizing and checkpointing across batches.

In this post we work through both approaches end-to-end. We first create a sample Hudi event table in each flavor, CoW and MoR, then migrate them in place to Iceberg using two techniques, the Spark Iceberg add_files procedure and the Apache XTable sync tool, covering the considerations and prerequisites of each. Finally, we walk through the full table migration path, where the source data is rewritten into a new Iceberg table from scratch.

Solution overview

The following diagram illustrates the architecture that you implement through this post.

Architecture diagram of the Hudi-to-Iceberg migration workflow on AWS

Figure 1: Migration architecture showing the in-place paths (add_files and Apache XTable) and the full-table paths (CTAS and INSERT) to Amazon S3 and Amazon S3 Tables, all registered in the AWS Glue Data Catalog and queried with Amazon Athena

The workflow consists of the following steps:

  1. An AWS Glue notebook writes the source data as Apache Hudi tables to an Amazon S3 bucket and registers them in the AWS Glue Data Catalog. This creates the existing Hudi data lake that the migration starts from.
  2. The first in-place migration approach uses an AWS Glue notebook to run the Apache Iceberg add_files procedure. The procedure creates Iceberg metadata on top of the Parquet data files already in the Hudi table. The new Iceberg metadata points to the same Parquet files, so no data is copied or rewritten. This makes it a fast, low-cost migration path.
  3. The second in-place migration approach runs Apache XTable on an Amazon Elastic Compute Cloud (Amazon EC2) instance to convert the Hudi table metadata into Iceberg metadata. XTable reads the Hudi commit timeline and creates matching Iceberg metadata that points to the same Parquet files. Both table formats can then share a single copy of the data.
  4. For a full migration, an AWS Glue notebook uses CREATE TABLE AS SELECT (CTAS) or INSERT INTO statements to read the Hudi data and rewrite it as new Iceberg tables. The tables can go in a general purpose S3 bucket or in Amazon S3 Tables. This approach fully separates the new tables from the Hudi source. S3 Tables also handle table maintenance for you, such as compaction and snapshot management.
  5. The AWS Glue Data Catalog registers the Iceberg tables from all three approaches. Users can then use Amazon Athena, a serverless query service, to run SQL queries on these tables and validate the migrated data. Athena enables interactive SQL-based exploration of data directly in Amazon S3 without the need for complex infrastructure management.

Prerequisites

Before getting started, make sure you have the following:

Launch an AWS Glue Studio notebook for processing

For this post, we use an AWS Glue Studio notebook. Follow the steps in Getting started with notebooks in AWS Glue Studio to set up the notebook environment.

  1. Sign in to the AWS Management Console and open the AWS Glue console.
  2. In the Region selector (top right), choose your preferred working Region.
  3. Choose ETL Jobs in the left pane.
  4. Choose Notebook under Author using an interactive code notebook, select Start fresh, choose your IAM role, and choose Create notebook.
  5. After a minute, the Jupyter notebook editor appears. Clear all the default cells.

Create Hudi Copy-on-Write (CoW) and Merge-on-Read (MoR) sample event tables

This notebook creates Hudi tables using both Copy-on-Write (CoW) and Merge-on-Read (MoR) table types with sample events. We perform inserts, updates, and deletes, then inspect the underlying file structures to understand the key differences:

  • CoW: an update or delete fully rewrites every base Parquet file that contains an impacted row, producing a new file slice in the file group. Reads are cheap, and writes amplify with the fraction of files touched.
  • MoR: an update or delete is appended to an Avro log file alongside the base file instead of rewriting the base, and base + logs are merged at read time (or compacted into a new base file later). Cheaper writes. Costlier reads until compaction.
  1. Execute the first cell with the following configuration to set up the Spark session with Hudi and Iceberg support:

Code:

{
"glue_version": "5.1",
"worker_type": "G.1X",
"number_of_workers": 5,
"idle_timeout": 30,
"--datalake-formats" : "hudi,iceberg",
"--conf" : "spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.glue_iceberg=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_iceberg.warehouse=s3://YOUR_BUCKET/warehouse/ --conf spark.sql.catalog.glue_iceberg.type=glue --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar"
}
  1. Initialize the session and configuration. Update the bucket_name, db_name and region variables with your configurations. Reusable helper – list_table_files(table_name, label="", verbose=False) renders a table’s S3 layout as a directory tree so you can SEE how CLEAN, COMPACT and add_files change the files underneath. It is reused before/after each of those steps instead of re-coding the listing.

Code:

# ──────────────────────────────────────────────
# Source Hudi tables
# ──────────────────────────────────────────────
hudi_events_cow_table = "events_hudi_cow" # Copy-on-Write source
hudi_events_mor_table = "events_hudi_mor" # Merge-on-Read source
# ──────────────────────────────────────────────
# In-place migration targets (add_files/xtable)
# ──────────────────────────────────────────────
iceberg_cow_inplace = "events_iceberg_cow_inplace"
iceberg_mor_inplace = "events_iceberg_mor_inplace"
iceberg_cow_xtable = "events_hudi_cow_iceberg_xtable"
iceberg_mor_xtable = "events_hudi_mor_iceberg_xtable"
# ──────────────────────────────────────────────
# Full table (shadow) migration targets
# ──────────────────────────────────────────────
iceberg_cow_full_ctas = "events_iceberg_cow_full_ctas"
iceberg_cow_full_create_insert = "events_iceberg_cow_full_create_insert"
iceberg_mor_full_ctas = "events_iceberg_mor_full_ctas"
iceberg_mor_full_create_insert = "events_iceberg_mor_full_create_insert"
# ──────────────────────────────────────────────
# Infrastructure
# ──────────────────────────────────────────────
bucket_name = "demoblogiceberg-bdb5001" # S3 bucket for data
db_name = "hudi_to_iceberg_blog" # Glue Data Catalog database
region = "us-east-1"
# Create the Glue Data Catalog database
spark.sql(f"""
CREATE DATABASE IF NOT EXISTS {db_name}
LOCATION 's3://{bucket_name}/{db_name}'
""")
# Reusable helper —list_table_files()(S3 layout tree)
import boto3
s3 = boto3.client('s3', region_name=region)
# File/folder annotations keyed by classifier result
_NOTES = {
"hoodie_dir": "<-- Hudi metadata (timeline, schema, indexes)",
"hoodie_mdt": "<-- Hudi metadata table (internal index & stats, MoR)",
"hoodie_files": "<-- file listing index",
"hoodie_cstats":"<-- column-level min/max/null stats",
"hoodie_ridx": "<-- record-key -> file-group mapping",
"ice_meta_dir": "<-- Iceberg metadata layer (metadata.json, manifest lists, manifests)",
"ice_data_dir": "<-- Iceberg data layer (base Parquet files)",
"base": "<-- base Parquet file",
"log": "<-- Hudi delta log file (MoR update, uncompacted)",
"part_meta": "<-- Hudi partition marker",
"metadata_json":"<-- Iceberg table metadata (schema, snapshots)",
"manifest_list":"<-- Iceberg manifest list (snapshot -> manifests)",
"manifest": "<-- Iceberg manifest (lists data files)",
"props": "<-- table configuration & schema",
}
# Fixed-path annotations for metadata directories
_DIR_KIND = {
".hoodie": "hoodie_dir", ".hoodie/metadata": "hoodie_mdt",
".hoodie/metadata/files": "hoodie_files",
".hoodie/metadata/column_stats": "hoodie_cstats",
".hoodie/metadata/record_index": "hoodie_ridx",
"metadata": "ice_meta_dir", "data": "ice_data_dir",
}
def _fmt_size(n):
"""Human-readable byte size."""
for u in ("B", "KB", "MB", "GB", "TB"):
if n < 1024 or u == "TB":
return f"{n:,.0f} {u}" if u == "B" else f"{n:,.1f} {u}"
n /= 1024.0
def _leaf_kind(name):
"""Classify a leaf file by name -> annotation key / count bucket."""
if ".log." in name: return "log"
if name.startswith(".hoodie_partition_metadata"): return "part_meta"
if name == "hoodie.properties": return "props"
if name.endswith(".metadata.json"): return "metadata_json"
if name.startswith("snap-") and name.endswith(".avro"): return "manifest_list"
if name.endswith(".avro"): return "manifest"
if name.endswith(".parquet"): return "base"
return ""
def list_table_files(table_name, label="", verbose=False, max_depth=2, max_children=2):
"""Render a Hudi/Iceberg table's S3 layout as an annotated directory tree.
Teaching view for CLEAN/COMPACT/add_files: multiple base files in a partition
= stale file slices; .log.* = uncompacted MoR updates. The large Hudi .hoodie/
is summarized (depth/width capped); Iceberg metadata/ and data partitions show
in full. verbose=True expands everything.
"""
prefix = f"{db_name}/{table_name}/"
header = f"s3://{bucket_name}/{prefix}"
if label:
header += f" [{label}]"
print(header)
in_meta = lambda p: (p.split("/", 1)[0] if p else "") in (".hoodie", "metadata")
capped_meta = lambda p: (p.split("/", 1)[0] if p else "") == ".hoodie"
# ---- gather objects into a nested tree + tally honest counts -----------
root = {"d": {}, "f": []}
n_base = n_log = n_meta = n_other = 0
s_base = s_log = s_meta = 0
has_hudi = has_ice = False
for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket_name, Prefix=prefix):
for obj in page.get("Contents", []):
key = obj["Key"].replace(prefix, "")
size = obj["Size"]
if key == "" or key.endswith("/"):
continue # S3 directory marker
parts = key.split("/")
fname, top = parts[-1], parts[0]
if top == ".hoodie": has_hudi = True
if top == "metadata": has_ice = True
# files inside a metadata folder count ONLY as metadata
if top in (".hoodie", "metadata"):
n_meta += 1; s_meta += size
elif ".log." in fname:
n_log += 1; s_log += size
elif fname.endswith(".parquet"):
n_base += 1; s_base += size
elif fname.startswith(".hoodie"):
n_meta += 1; s_meta += size; has_hudi = True
else:
n_other += 1
node = root
for part in parts[:-1]:
node = node["d"].setdefault(part, {"d": {}, "f": []})
node["f"].append((fname, size))
# ---- helpers -----------------------------------------------------------
def count_all(node):
return len(node["f"]) + sum(count_all(c) for c in node["d"].values())
def collapse(node, name, rel):
"""Merge single-child, file-less dir chains (year=/month=/day=)."""
while (len(node["d"]) == 1 and not node["f"] and not in_meta(rel)):
cname, child = next(iter(node["d"].items()))
crel = f"{rel}/{cname}"
if in_meta(crel): break
name, rel, node = f"{name}/{cname}", crel, child
return name, rel, node
def line(indent, branch, text, note, width):
s = f"{indent}{branch}{text}"
return f"{s:<{width}} {note}" if note else s
def render(node, rel, indent, depth):
entries = ([("d", n, c) for n, c in sorted(node["d"].items())]
+ [("f", n, sz) for n, sz in sorted(node["f"], key=lambda f: (".log." in f[0], f[0]))])
cap = (not verbose) and capped_meta(rel) and len(entries) > max_children
shown = entries[:max_children] if cap else entries
hidden = len(entries) - len(shown)
for i, item in enumerate(shown):
last = (i == len(shown) - 1) and hidden == 0
branch = "└── " if last else "├── "
sub = indent + (" " if last else "│ ")
if item[0] == "d":
_, name, child = item
disp, erel, enode = collapse(child, name, f"{rel}/{name}" if rel else name)
note = _NOTES.get(_DIR_KIND.get(erel, ""), "")
cnt = f" ({count_all(enode)} obj)" if (in_meta(erel) or erel in _DIR_KIND) else ""
print(line(indent, branch, f"{disp}/{cnt}", note, 60))
if (not verbose) and capped_meta(erel) and depth + 1 > max_depth:
inner = count_all(enode)
if inner:
print(f"{sub}└── … ({inner} objects, use verbose=True)")
else:
render(enode, erel, sub, depth + 1)
else:
_, name, size = item
note = _NOTES.get(_leaf_kind(name), "")
print(line(indent, branch, f"{name} ({_fmt_size(size)})", note, 72))
if hidden:
print(f"{indent}└── … (+{hidden} more)")
render(root, "", "", 0)
# ---- per-partition slice note (Hudi concept; skip metadata & Iceberg data/) ----
def slices(node, rel):
top = rel.split("/", 1)[0] if rel else ""
if top in (".hoodie", "metadata", "data"):
return
nb = sum(1 for f, _ in node["f"] if f.endswith(".parquet"))
nl = sum(1 for f, _ in node["f"] if ".log." in f)
if nb > 1:
print(f" note: {rel}/ has {nb} base files -> multiple file slices (stale versions)")
elif nb == 1 and nl:
print(f" note: {rel}/ has 1 base + {nl} log file(s) -> uncompacted MoR updates")
for n, c in node["d"].items():
slices(c, f"{rel}/{n}" if rel else n)
slices(root, "")
# ---- in-place migration note -------------------------------------------
if has_ice and "data" not in root["d"] and n_base == 0:
print(" note: Iceberg metadata/ present but no local data/ folder")
print(" -> in-place migration: data files remain registered in place at the source (Hudi) location; no data copied")
# ---- summary -----------------------------------------------------------
total = n_base + n_log + n_meta + n_other
print(
f"SUMMARY: {n_base} base · {n_log} log · {n_meta} meta"
+ (f" · {n_other} other" if n_other else "")
+ f" · {total} files · {_fmt_size(s_base + s_log + s_meta)}"
)
print(f" .hoodie/: {'present' if has_hudi else 'none'} metadata/ (Iceberg): {'present' if has_ice else 'none'}")
print()

Sample event data

We use a website clickstream events dataset, partitioned by year/month/day – a realistic pattern for event-driven architectures.

Note: In this example, we create the table with a preCombineField, which flips the default write operation of INSERT INTO from insert to upsert. You can override it explicitly with hoodie.spark.sql.insert.into.operation when you need a different behavior.

CoW table: create, upsert, and delete

We create a Hudi CoW table, insert records, apply an upsert, and delete a record, then inspect how the files change.

Create the Hudi CoW events table

Code:

spark.sql(f"""
CREATE TABLE {db_name}.{hudi_events_cow_table} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING,
year INT,
month INT,
day INT
)
USING HUDI
OPTIONS (
type = 'cow',
primaryKey = 'event_id',
preCombineField = 'event_timestamp'
)
PARTITIONED BY (year, month, day)
LOCATION 's3://{bucket_name}/{db_name}/{hudi_events_cow_table}/'
""")

Insert event records

Code:

spark.sql(f"""
INSERT INTO {db_name}.{hudi_events_cow_table}
VALUES
('EVT_001', 'USER_101', 'page_view', timestamp('2024-04-01 08:15:00'), '/home', 2024, 4, 1),
('EVT_002', 'USER_102', 'click', timestamp('2024-04-01 09:30:00'), '/products', 2024, 4, 1),
('EVT_003', 'USER_103', 'purchase', timestamp('2024-04-02 10:45:00'), '/checkout', 2024, 4, 2),
('EVT_004', 'USER_101', 'page_view', timestamp('2024-04-02 14:20:00'), '/blog', 2024, 4, 2),
('EVT_005', 'USER_104', 'signup', timestamp('2024-04-03 11:00:00'), '/register', 2024, 4, 3),
('EVT_006', 'USER_102', 'click', timestamp('2024-04-03 16:45:00'), '/deals', 2024, 4, 3)
""")

Upsert: simulate late-arriving corrections

Code:

spark.sql(f"""
INSERT INTO {db_name}.{hudi_events_cow_table}
VALUES
('EVT_002', 'USER_102', 'add_to_cart', timestamp('2024-04-01 09:31:00'), '/products/item-42', 2024, 4, 1),
('EVT_005', 'USER_104', 'signup', timestamp('2024-04-03 11:01:00'), '/welcome', 2024, 4, 3)
""")

Delete: remove a record

Code:

spark.sql(f"""
DELETE FROM {db_name}.{hudi_events_cow_table}
WHERE event_id = 'EVT_006'
""")

Inspect the CoW file structure

CoW tables rewrite the entire Parquet file on every upsert/delete. You will see one base Parquet file per file group (no log files).

Code:

list_table_files(hudi_events_cow_table, "COW --- initial file structure")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_cow/ [COW — initial file structure]
├── .hoodie/ (50 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=4/
├── day=1/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ ├── 7d187e9a.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
│ └── 7d187e9a.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
├── day=2/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── e583fe17.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
└── day=3/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
├── eedc1b38.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
├── eedc1b38.....20261011152539102.parquet (426.2 KB) <-- base Parquet file
└── eedc1b38.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
note: year=2024/month=4/day=1/ has 2 base files -> multiple file slices (stale versions)
note: year=2024/month=4/day=3/ has 3 base files -> multiple file slices (stale versions)
SUMMARY: 6 base · 0 log · 53 meta · 59 files · 2.8 MB
.hoodie/: present metadata/ (Iceberg): none

MoR table: create, upsert, and delete

We run the same create, upsert, and delete sequence on a Hudi MoR table.

Create the Hudi MoR events table

Code:

spark.sql(f"""
CREATE TABLE {db_name}.{hudi_events_mor_table} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING,
year INT,
month INT,
day INT
)
USING HUDI
OPTIONS (
type = 'mor',
primaryKey = 'event_id',
preCombineField = 'event_timestamp'
)
PARTITIONED BY (year, month, day)
LOCATION 's3://{bucket_name}/{db_name}/{hudi_events_mor_table}/'
""")

Insert initial event records

Code:

spark.sql(f"""
INSERT INTO {db_name}.{hudi_events_mor_table}
VALUES
('EVT_001', 'USER_201', 'page_view', timestamp('2024-05-10 07:00:00'), '/home', 2024, 5, 10),
('EVT_002', 'USER_202', 'click', timestamp('2024-05-10 08:15:00'), '/products', 2024, 5, 10),
('EVT_003', 'USER_203', 'purchase', timestamp('2024-05-11 09:30:00'), '/checkout', 2024, 5, 11),
('EVT_004', 'USER_201', 'page_view', timestamp('2024-05-11 12:00:00'), '/account', 2024, 5, 11),
('EVT_005', 'USER_204', 'signup', timestamp('2024-05-12 10:30:00'), '/register', 2024, 5, 12),
('EVT_006', 'USER_202', 'click', timestamp('2024-05-12 15:45:00'), '/deals', 2024, 5, 12)
""")

Upsert: simulate late-arriving corrections (creates delta log files)

Code:

spark.sql(f"""
INSERT INTO {db_name}.{hudi_events_mor_table}
VALUES
('EVT_002', 'USER_202', 'add_to_cart', timestamp('2024-05-10 08:16:00'), '/products/item-99', 2024, 5, 10),
('EVT_005', 'USER_204', 'signup', timestamp('2024-05-12 10:31:00'), '/welcome', 2024, 5, 12)
""")

Delete: remove a record

Code:

spark.sql(f"""
DELETE FROM {db_name}.{hudi_events_mor_table}
WHERE event_id = 'EVT_006'
""")

Inspect MoR: base Parquet and log files

MoR tables write updates as delta log files (.log.*) alongside the base Parquet files. This avoids full file rewrites for faster writes.

Code:

list_table_files(hudi_events_mor_table, "MOR --- base parquet + log files")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_mor/ [MOR — base parquet + log files]
├── .hoodie/ (50 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=5/
├── day=10/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ ├── 7d5ce247.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
│ └── .7d5ce247.....20261011153145862.log.1_1-164-788 (1.3 KB) <-- Hudi delta log file (MoR update, uncompacted)
├── day=11/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── f9b8147e.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
└── day=12/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
├── 3f0b5ad4.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
├── .3f0b5ad4.....20261011153145862.log.1_0-164-787 (1.3 KB) <-- Hudi delta log file (MoR update, uncompacted)
└── .3f0b5ad4.....20261011153215018.log.1_0-192-901 (1.1 KB) <-- Hudi delta log file (MoR update, uncompacted)
note: year=2024/month=5/day=10/ has 1 base + 1 log file(s) -> uncompacted MoR updates
note: year=2024/month=5/day=12/ has 1 base + 2 log file(s) -> uncompacted MoR updates
SUMMARY: 3 base · 3 log · 53 meta · 59 files · 1.5 MB
.hoodie/: present metadata/ (Iceberg): none

Query both Hudi tables

Code:

print("=== COW Table (events_hudi_cow) ===")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url, year, month, day FROM {db_name}.{hudi_events_cow_table} ORDER BY event_id").show(truncate=False)
print("\n=== MOR Table (events_hudi_mor) ===")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url, year, month, day FROM {db_name}.{hudi_events_mor_table} ORDER BY event_id").show(truncate=False)

Output:

=== COW Table (events_hudi_cow) ===
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
|event_id|user_id |event_type |event_timestamp |page_url |year|month|day|
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |2024|4 |1 |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|2024|4 |1 |
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |2024|4 |2 |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |2024|4 |2 |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |2024|4 |3 |
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
=== MOR Table (events_hudi_mor) ===
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
|event_id|user_id |event_type |event_timestamp |page_url |year|month|day|
+--------+--------+-----------+-------------------+-----------------+----+-----+---+
|EVT_001 |USER_201|page_view |2024-05-10 07:00:00|/home |2024|5 |10 |
|EVT_002 |USER_202|add_to_cart|2024-05-10 08:16:00|/products/item-99|2024|5 |10 |
|EVT_003 |USER_203|purchase |2024-05-11 09:30:00|/checkout |2024|5 |11 |
|EVT_004 |USER_201|page_view |2024-05-11 12:00:00|/account |2024|5 |11 |
|EVT_005 |USER_204|signup |2024-05-12 10:31:00|/welcome |2024|5 |12 |
+--------+--------+-----------+-------------------+-----------------+----+-----+---+

Migrate Hudi tables to Iceberg with in-place migration

In-place migration keeps the original Parquet data files where they are and only builds Iceberg metadata on top of them. There is no rewrite of the underlying data, which makes the migration fast and inexpensive even on large tables. Two tools cover the common cases on AWS:

  • add_files (Iceberg built-in procedure). A Spark procedure shipped with Apache Iceberg that registers existing Parquet files into an Iceberg table by writing the corresponding manifests, without rewriting the data. It runs as a Spark job in which the source table’s partitions are distributed across Spark tasks, so file listing and column-statistics gathering happen in parallel on the cluster.
  • Apache XTable (incubating). An open-source cross-format translation tool that can read the metadata of one lakehouse format (Hudi, Delta, or Iceberg) and write out the equivalent metadata for another, pointing at the same underlying Parquet files. XTable can run once to produce the initial Iceberg metadata, or run on a schedule to keep the Iceberg view continuously in sync with ongoing writes to the Hudi table.

Choosing between add_files and Apache XTable for in-place migration

When planning an in-place migration, it is important to understand the strengths of each approach:

  • add_files is best suited for a one-time migration. It registers existing Hudi Parquet files into an Iceberg table in a single operation. After the migration is complete, the source Hudi table and target Iceberg table are independent, and any subsequent changes to the Hudi table are not reflected in Iceberg.
  • Apache XTable supports both an initial one-time conversion and ongoing incremental synchronization. This makes it ideal for transition periods where you need to keep both the source Hudi tables and the target Iceberg tables in sync while downstream consumers gradually migrate to Iceberg. After all workloads have been cut over, you can stop the sync process and decommission the Hudi tables.

Important notes

add_files/XTable register existing Parquet files as-is, so they rely on the column names baked into each file. Added and dropped columns are safe, but if the source table was renamed any column during its lifecycle, files written before the rename still hold the old name and reads against the new name return NULL for them. For tables with a history of column renames, prefer a full migration instead.

Iceberg version and column statistics (in-place add_files / snapshot): In Iceberg 1.9.0 through 1.10.0, the add_files and snapshot procedures don’t generate column stats in the Iceberg metadata. The cause is a NameMapping loss in ParquetUtil.footerMetrics. Without those column-level lower/upper bounds and null counts, the migrated Iceberg table loses most of its file-skipping ability at query planning time, which translates into more Parquet files being inspected, higher scan volume, and slower queries, especially for selective filters on non-partition columns. The issue is fixed in the Iceberg 1.10.1 release (#14617). Affected runtimes are AWS Glue 5.1 and Amazon EMR 7.11–7.13. We recommend AWS Glue 6.0+ or Amazon EMR 7.14+ for production migrations.

Shared data files during in-place migration: With both approaches, the in-place migration doesn’t copy data. The resulting Iceberg table points to the same underlying Parquet files as the source Hudi table. This means that until you perform a full cutover to Iceberg, you must avoid destructive operations on the Iceberg table that could delete or modify the shared data files. Specifically:

  • Don’t drop the Iceberg table with the PURGE option.
  • Don’t run Iceberg maintenance tasks such as expire_snapshots or remove_orphan_files.

Running any of these before cutover can remove files that the source Hudi table still depends on, leading to data loss. Iceberg exposes a built-in guardrail against this: the gc.enabled table property. When set to false, Iceberg blocks the preceding destructive operations. How you apply this depends on which migration path you choose.

With add_files. Disable garbage collection immediately after the procedure finishes and keep it disabled until you have fully cut over to Iceberg.

Right after add_files:

spark.sql(f"""
ALTER TABLE {db_name}.{table_name}
SET TBLPROPERTIES ('gc.enabled' = 'false');
""")

After Hudi source is retired and the Iceberg table owns its own data, re-enable GC so maintenance can run normally:

spark.sql(f"""
ALTER TABLE {db_name}.{table_name}
SET TBLPROPERTIES ('gc.enabled' = 'true');
""")

With Apache XTable (incremental sync). The gc.enabled = false guardrail is only partially usable here, because XTable’s incremental sync needs GC to be enabled on the Iceberg table to operate normally. A practical pattern is:

  1. Run the initial full XTable sync.
  2. Temporarily set gc.enabled = false to run post-migration validation and consistency checks on the Iceberg table in a safe state.
  3. Set gc.enabled = true again before starting incremental sync and leave it on for the duration of the sync period.

Because the automatic guardrail is off during incremental sync, the burden shifts to operational discipline: as long as XTable is syncing and data files are shared with the Hudi table, don’t run any Iceberg operation that can delete source data files (no DROP TABLE ... PURGE, no expire_snapshots, no remove_orphan_files, no manual object deletion against the shared S3 prefix). Treat the Iceberg table as read-only in terms of file lifecycle until cutover is complete, and Hudi is decommissioned, at which point normal Iceberg maintenance can resume.

Don’t enable the AWS Glue Data Catalog table optimizer before cutover.

Glue-managed Iceberg maintenance (compaction, snapshot retention, orphan file cleanup) will rewrite or delete the Parquet files that the source Hudi table still depends on. Verify that no catalog-level optimizer applies to the migrated table and leave table-level optimization off until cutover is complete.

Prerequisites for in-place migration with add_files. The add_files procedure registers every Parquet file it finds under the source location into the target Iceberg table, with no awareness of Hudi’s file-group versioning. If more than one file version of the same record is present at migration time, each version is registered as a separate row, and the Iceberg table will contain duplicates. The source Hudi table must therefore expose exactly one file version per record before add_files runs:

  • CoW tables: Run CLEAN to remove old file versions, keeping only the latest Parquet files.
  • MoR tables: Run COMPACTION (merge delta logs into base Parquet) + CLEAN before calling add_files, since add_files can only register base Parquet files and cannot process Hudi delta log files.

Option 1: Migrate Hudi tables to Iceberg with the add_files Spark procedure

This option uses the Iceberg add_files procedure to register the existing Hudi Parquet files into new Iceberg tables. Run it once per table, after preparing the Hudi table so that only the current file versions remain.

Migrate CoW to Iceberg

We create the target Iceberg table, prepare the Hudi source so that only the current file versions remain, register those files, and validate the result.

Create the target Iceberg table with the same schema

Code:

spark.sql(f"""
CREATE TABLE glue_iceberg.{db_name}.{iceberg_cow_inplace} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING,
year INT,
month INT,
day INT
)
USING iceberg
PARTITIONED BY (year, month, day)
LOCATION 's3://{bucket_name}/{db_name}/{iceberg_cow_inplace}/'
""")

Run Hudi CLEAN to keep only the latest file versions

Check files listing for the Hudi table before the CLEAN operation:

Code:

list_table_files(hudi_events_cow_table, "COW --- BEFORE CLEAN")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_cow/ [COW — BEFORE CLEAN]
├── .hoodie/ (50 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=4/
├── day=1/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ ├── 7d187e9a.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
│ └── 7d187e9a.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
├── day=2/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── e583fe17.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
└── day=3/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
├── eedc1b38.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
├── eedc1b38.....20261011152539102.parquet (426.2 KB) <-- base Parquet file
└── eedc1b38.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
note: year=2024/month=4/day=1/ has 2 base files -> multiple file slices (stale versions)
note: year=2024/month=4/day=3/ has 3 base files -> multiple file slices (stale versions)
SUMMARY: 6 base · 0 log · 53 meta · 59 files · 2.8 MB
.hoodie/: present metadata/ (Iceberg): none

Run the CLEAN operation specifying to keep only the latest file slices:

Code:

spark.sql(f"""
CALL spark_catalog.system.run_clean(
table => '{db_name}.{hudi_events_cow_table}',
clean_policy => 'KEEP_LATEST_FILE_VERSIONS',
file_versions_retained => 1
)
""").show(truncate=False)

Output:

+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+
|start_clean_time |time_taken_in_millis|total_files_deleted|earliest_commit_to_retain|bootstrap_part_metadata|version|
+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+
|20261011154407413|382 |3 | |{} |2 |
+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+

Check files listing for the Hudi table after the CLEAN operation:

Code:

list_table_files(hudi_events_cow_table, "COW --- AFTER CLEAN")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_cow/ [COW — AFTER CLEAN]
├── .hoodie/ (59 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=4/
├── day=1/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── 7d187e9a.....20261011152509633.parquet (426.3 KB) <-- base Parquet file
├── day=2/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── e583fe17.....20261011152403779.parquet (426.4 KB) <-- base Parquet file
└── day=3/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
└── eedc1b38.....20261011152539102.parquet (426.2 KB) <-- base Parquet file
SUMMARY: 3 base · 0 log · 62 meta · 65 files · 1.6 MB
.hoodie/: present metadata/ (Iceberg): none

Register Hudi Parquet files into Iceberg with add_files

Code:

spark.sql(f"""
CALL glue_iceberg.system.add_files(
source_table => 'parquet.`s3://{bucket_name}/{db_name}/{hudi_events_cow_table}/`',
table => 'glue_iceberg.{db_name}.{iceberg_cow_inplace}'
)
""").show(truncate=False)

Output:

+-----------------+-----------------------+
|added_files_count|changed_partition_count|
+-----------------+-----------------------+
|3 |NULL |
+-----------------+-----------------------+

Validate: compare Hudi CoW and Iceberg (in-place)

We can see the Hudi and Iceberg tables reflect the same content:

Code:

print("Source: Hudi COW")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url FROM {db_name}.{hudi_events_cow_table} ORDER BY event_id").show(truncate=False)
print("Target: Iceberg (in-place)")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url FROM glue_iceberg.{db_name}.{iceberg_cow_inplace} ORDER BY event_id").show(truncate=False)

Output:

Source: Hudi COW
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+
Target: Iceberg (in-place)
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

We can see under the Iceberg table location only Iceberg metadata has been generated. These point to the source Hudi table data files.

Code:

list_table_files(iceberg_cow_inplace, "COW Iceberg --- AFTER add_files (note metadata/ folder)", verbose=True)

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_iceberg_cow_inplace/ [COW Iceberg — AFTER add_files (note metadata/ folder)]
└── metadata/ (5 obj) <-- Iceberg metadata layer (metadata.json, manifest lists, manifests)
├── 00000.....metadata.json (1.3 KB) <-- Iceberg table metadata (schema, snapshots)
├── 00001.....metadata.json (2.0 KB) <-- Iceberg table metadata (schema, snapshots)
├── 00002.....metadata.json (3.0 KB) <-- Iceberg table metadata (schema, snapshots)
├── snap-7577.....avro (4.4 KB) <-- Iceberg manifest list (snapshot -> manifests)
└── stage-223...-manifest.....avro (6.7 KB) <-- Iceberg manifest (lists data files)
note: Iceberg metadata/ present but no local data/ folder
-> in-place migration: data files remain registered in place at the source (Hudi) location; no data copied
SUMMARY: 0 base · 0 log · 5 meta · 5 files · 17.4 KB
.hoodie/: none metadata/ (Iceberg): present

Migrate MoR to Iceberg

For the MoR table, we compact and clean the Hudi source first, then register the compacted files and validate the result.

Create the target Iceberg table

Code:

spark.sql(f"""
CREATE TABLE glue_iceberg.{db_name}.{iceberg_mor_inplace} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING,
year INT,
month INT,
day INT
)
USING iceberg
PARTITIONED BY (year, month, day)
LOCATION 's3://{bucket_name}/{db_name}/{iceberg_mor_inplace}/'
""")

Compact the MoR table: merge delta logs into base Parquet files

Check files listing for the Hudi table before the CLEAN operation:

Code:

list_table_files(hudi_events_mor_table, "MOR --- BEFORE COMPACT+CLEAN")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_mor/ [MOR — BEFORE COMPACT+CLEAN]
├── .hoodie/ (50 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=5/
├── day=10/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ ├── 7d5ce247.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
│ └── .7d5ce247.....20261011153145862.log.1_1-164-788 (1.3 KB) <-- Hudi delta log file (MoR update, uncompacted)
├── day=11/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── f9b8147e.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
└── day=12/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
├── 3f0b5ad4.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
├── .3f0b5ad4.....20261011153145862.log.1_0-164-787 (1.3 KB)
└── .3f0b5ad4.....20261011153215018.log.1_0-192-901 (1.1 KB) <-- Hudi delta log file (MoR update, uncompacted)
note: year=2024/month=5/day=10/ has 1 base + 1 log file(s) -> uncompacted MoR updates
note: year=2024/month=5/day=12/ has 1 base + 2 log file(s) -> uncompacted MoR updates
SUMMARY: 3 base · 3 log · 53 meta · 59 files · 1.5 MB
.hoodie/: present metadata/ (Iceberg): none

Trigger COMPACTION scheduling on the MoR table with the delta-commit and age thresholds set to their minimum, so any file slice with pending log files is included in the plan and merged back into its base Parquet file.

Code:

# Schedule compaction
spark.sql(f"""
CALL spark_catalog.system.run_compaction(
table => '{db_name}.{hudi_events_mor_table}',
op => 'schedule',
options =>
'hoodie.compact.inline.max.delta.commits=1,
hoodie.compact.inline.max.delta.seconds=0'
)
""").show(truncate=False)

Output:

+-----------------+--------------+---------+
|timestamp |operation_size|state |
+-----------------+--------------+---------+
|20261011154522819|2 |REQUESTED|
+-----------------+--------------+---------+

Execute the previously scheduled compaction plan to merge the pending log files into their base Parquet files. The REQUESTED instant produced by the schedule step transitions to COMPLETED, and the table is left with base Parquet files only.

Code:

# Execute compaction
spark.sql(f"""
CALL spark_catalog.system.run_compaction(
table => '{db_name}.{hudi_events_mor_table}',
op => 'run'
)
""").show(truncate=False)

Output:

+-----------------+--------------+---------+
|timestamp |operation_size|state |
+-----------------+--------------+---------+
|20261011154522819|2 |COMPLETED|
+-----------------+--------------+---------+

Run Hudi CLEAN to remove old file versions and log files

Code:

spark.sql(f"""
CALL spark_catalog.system.run_clean(
table => '{db_name}.{hudi_events_mor_table}',
clean_policy => 'KEEP_LATEST_FILE_VERSIONS',
file_versions_retained => 1
)
""").show(truncate=False)

Output:

+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+
|start_clean_time |time_taken_in_millis|total_files_deleted|earliest_commit_to_retain|bootstrap_part_metadata|version|
+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+
|20261011154552026|345 |5 | |{} |2 |
+-----------------+--------------------+-------------------+-------------------------+-----------------------+-------+

Code:

list_table_files(hudi_events_mor_table, "MOR --- AFTER COMPACT+CLEAN")

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_hudi_mor/ [MOR — AFTER COMPACT+CLEAN]
├── .hoodie/ (69 obj) <-- Hudi metadata (timeline, schema, indexes)
│ ├── .......trimmed
└── year=2024/month=5/
├── day=10/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── 7d5ce247.....20261011154522819.parquet (426.3 KB) <-- base Parquet file
├── day=11/
│ ├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
│ └── f9b8147e.....20261011153110535.parquet (426.4 KB) <-- base Parquet file
└── day=12/
├── .hoodie_partition_metadata (96 B) <-- Hudi partition marker
└── 3f0b5ad4.....20261011154522819.parquet (426.1 KB) <-- base Parquet file
SUMMARY: 3 base · 0 log · 72 meta · 75 files · 1.7 MB
.hoodie/: present metadata/ (Iceberg): none

Register compacted Parquet files into Iceberg with add_files

Code:

spark.sql(f"""
CALL glue_iceberg.system.add_files(
source_table => 'parquet.`s3://{bucket_name}/{db_name}/{hudi_events_mor_table}/`',
table => 'glue_iceberg.{db_name}.{iceberg_mor_inplace}'
)
""").show(truncate=False)

Output:

+-----------------+-----------------------+
|added_files_count|changed_partition_count|
+-----------------+-----------------------+
|3 |NULL |
+-----------------+-----------------------+

Refresh the Hudi table to invalidate Spark’s cached metadata and file listings, so subsequent queries re-read the post-compaction state from the catalog and storage instead of returning stale pre-compaction results.

Code:

spark.sql(f"REFRESH TABLE {db_name}.{hudi_events_mor_table}")

Validate: compare Hudi MoR and Iceberg (in-place)

Code:

print("Source: Hudi MOR")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url FROM {db_name}.{hudi_events_mor_table} ORDER BY event_id").show(truncate=False)
print("Target: Iceberg (in-place)")
spark.sql(f"SELECT event_id, user_id, event_type, event_timestamp, page_url FROM glue_iceberg.{db_name}.{iceberg_mor_inplace} ORDER BY event_id").show(truncate=False)

Output:

Source: Hudi MOR
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_201|page_view |2024-05-10 07:00:00|/home |
|EVT_002 |USER_202|add_to_cart|2024-05-10 08:16:00|/products/item-99|
|EVT_003 |USER_203|purchase |2024-05-11 09:30:00|/checkout |
|EVT_004 |USER_201|page_view |2024-05-11 12:00:00|/account |
|EVT_005 |USER_204|signup |2024-05-12 10:31:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+
Target: Iceberg (in-place)
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_201|page_view |2024-05-10 07:00:00|/home |
|EVT_002 |USER_202|add_to_cart|2024-05-10 08:16:00|/products/item-99|
|EVT_003 |USER_203|purchase |2024-05-11 09:30:00|/checkout |
|EVT_004 |USER_201|page_view |2024-05-11 12:00:00|/account |
|EVT_005 |USER_204|signup |2024-05-12 10:31:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

Code:

list_table_files(iceberg_mor_inplace, "MOR Iceberg --- AFTER add_files (note metadata/ folder)", verbose=True)

Output:

s3://demoblogiceberg-bdb5001/hudi_to_iceberg_blog/events_iceberg_mor_inplace/ [MOR Iceberg — AFTER add_files (note metadata/ folder)]
└── metadata/ (5 obj) <-- Iceberg metadata layer (metadata.json, manifest lists, manifests)
├── 00000.....metadata.json (1.3 KB) <-- Iceberg table metadata (schema, snapshots)
├── 00001.....metadata.json (2.0 KB) <-- Iceberg table metadata (schema, snapshots)
├── 00002.....metadata.json (3.0 KB) <-- Iceberg table metadata (schema, snapshots)
├── snap-5097.....avro (4.4 KB) <-- Iceberg manifest list (snapshot -> manifests)
└── stage-254...-manifest.....avro (6.7 KB) <-- Iceberg manifest (lists data files)
note: Iceberg metadata/ present but no local data/ folder
-> in-place migration: data files remain registered in place at the source (Hudi) location; no data copied
SUMMARY: 0 base · 0 log · 5 meta · 5 files · 17.4 KB
.hoodie/: none metadata/ (Iceberg): present

Option 2: Migrate Hudi tables to Iceberg with the Apache XTable sync tool

Apache XTable (Incubating) is an open-source interoperability layer that translates metadata between Apache Hudi, Apache Iceberg, and Delta Lake without copying data. Its sync utility supports both one-time conversion and ongoing incremental synchronization, enabling parallel access to source and target formats during migration. Because it ships as a self-contained Java process with no managed-service dependency, you can run it anywhere a JVM is available: a developer laptop for one-off conversions, an Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS) container, or an AWS Lambda function, triggered on a schedule through Amazon EventBridge or on an event such as a new Hudi commit.

XTable’s sync utility does two things:

  1. First run – performs the initial in-place migration of a Hudi table to Iceberg.
  2. Subsequent runs – keep the Iceberg metadata in sync with the source table as it continues to receive writes.

Compile xtable-utilities following the instructions in the Apache XTable GitHub repository and copy xtable-utilities/target/xtable-utilities_2.12-0.5.0-SNAPSHOT-bundled.jar into the current working directory.

Migrate MoR to Iceberg

Note: When running the XTable sync tool on a MoR table, the utility only reads the base Parquet files and doesn’t process the delta log files. As a result, the generated Iceberg metadata reflects only the Hudi read-optimized view, meaning any recent updates or deletes stored in delta logs will be missing. To avoid data loss, always run compaction on the MoR table before executing the sync tool (for example, similar run_compaction commands as for the add_files example). Unlike add_files, however, there is no need to run CLEAN beforehand: XTable doesn’t scan the table location file by file, but builds the Iceberg manifests from Hudi’s own metadata, so it already knows which Parquet files belong to the current file slices and ignores superseded versions automatically.

Configure Apache XTables Configuration files.

Create the datasetConfig.yaml file containing indications about the dataset(s) to convert with the following content:

sourceFormat: HUDI
targetFormats:
  - ICEBERG
datasets:
  - tableBasePath: s3://YOUR_BUCKET/hudi_to_iceberg_blog/events_hudi_mor/
    tableName: events_hudi_mor_iceberg_xtable
    namespace: hudi_to_iceberg_blog
    partitionSpec: year:VALUE,month:VALUE,day:VALUE

Create the glueDataCatalog.yaml file containing Iceberg catalog (AWS Glue Data Catalog) settings with the following content:

catalogImpl: org.apache.iceberg.aws.glue.GlueCatalog
catalogName: glue
catalogOptions:
  warehouse: s3://YOUR_BUCKET/warehouse/
  catalog-impl: org.apache.iceberg.aws.glue.GlueCatalog
  io-impl: org.apache.iceberg.aws.s3.S3FileIO

Run the conversion

export AWS_DEFAULT_REGION=us-east-1

java -jar incubator-xtable/xtable-utilities/target/xtable-utilities_2.12-0.5.0-SNAPSHOT-bundled.jar \
-d datasetConfig.yaml \
-i glueDataCatalog.yaml

CLI options (from --help)

Flag Long form Description
-d --datasetConfig Path to the dataset configuration YAML (required)
-i --icebergCatalogConfig Path to the Iceberg catalog config YAML
-c --convertersConfig Override converter configs
-p --hadoopConfig Extra Hadoop config XML
-m --continuousMode Run on a scheduled loop
-t --continuousModeInterval Loop interval in seconds (with -m)

Verify the result

% aws glue get-table --database-name hudi_to_iceberg_blog \
--name events_hudi_mor_iceberg_xtable \
--query 'Table.{Name:Name,Type:Parameters.table_type,MetaLoc:Parameters.metadata_location}'

{
  "Name": "events_hudi_mor_iceberg_xtable",
  "Type": "ICEBERG",
  "MetaLoc": "s3://YOUR_BUCKET/hudi_to_iceberg_blog/events_hudi_mor/metadata/00002-a88318ba-ad13-4a0c-85a8-d595b53e9a89.metadata.json"
}

% aws s3 ls s3://YOUR_BUCKET/hudi_to_iceberg_blog/events_hudi_mor/metadata/
2026-09-22 16:26:18        648 00000-c71bcac6-3eee-4b98-875f-7562770db80b.metadata.json
2026-09-22 16:26:19       1940 00001-941c5622-95e3-41d7-942a-dabd69e561d1.metadata.json
2026-09-22 16:26:20       4293 00002-a88318ba-ad13-4a0c-85a8-d595b53e9a89.metadata.json
2026-09-22 16:26:20       8845 11adf6a4-58b9-4cc2-8cf0-317a7b43a8f7-m0.avro
2026-09-22 16:26:20       4493 snap-8409429281635613515-1-11adf6a4-58b9-4cc2-8cf0-317a7b43a8f7.avro
Amazon Athena console query returning five rows from the migrated Iceberg table

Figure 2: Validating the Apache XTable conversion by querying the migrated Iceberg table in Amazon Athena

Note: If you keep ingesting into the source Hudi table and want to keep the Iceberg table in sync, re-run the same conversion command. XTable automatically detects the previous execution and runs in incremental mode.

Migrate CoW to Iceberg

For the CoW table, update the datasetConfig.yaml with the CoW table path and name.

sourceFormat: HUDI
targetFormats:
  - ICEBERG
datasets:
  - tableBasePath: s3://YOUR_BUCKET/hudi_to_iceberg_blog/events_hudi_cow/
    tableName: events_hudi_mor_iceberg_xtable
    namespace: hudi_to_iceberg_blog
    partitionSpec: year:VALUE,month:VALUE,day:VALUE

The glueDataCatalog.yaml, conversion command, and verification steps remain the same as described in the MOR example.

Migrate Hudi tables to Iceberg with full table migration

Full table migration (also called shadow migration) reads all data from Hudi and writes it into a brand-new Iceberg table. This approach:

  • Allows schema and partition evolution (for example, switching to Iceberg hidden partitioning).
  • Produces a clean Iceberg table with no Hudi metadata dependencies.
  • Requires additional storage (data is copied)

Two options are demonstrated:

  • Option A: CTAS – a single-statement CREATE TABLE ... AS SELECT.
  • Option B: CREATE + INSERT – explicit table creation followed by data insertion.

Note: When performing a full table migration, the process is the same regardless of whether your source is a Hudi CoW or MoR table. This is because Spark reads from the Hudi real-time view by default, which reflects the latest table state. For CoW tables, this means reading the current base Parquet files. For MoR tables, the real-time view automatically merges the base Parquet files with any pending delta log files, ensuring all recent inserts, updates, and deletes are captured. As a result, the migrated Iceberg table will contain a complete and up-to-date snapshot of your data, with no additional compaction or cleanup steps required before migration.

We have taken a Hudi CoW table example to demonstrate the migration from Hudi CoW table to Iceberg table.

Prepare the column list for migration

Filter out Hudi internal metadata columns (_hoodie_*) and explicit partition columns to use Iceberg hidden partitioning.

Code:

# Build column list excluding Hudi metadata and explicit partition columns
all_columns_cow = spark.sql(f"SELECT * FROM {db_name}.{hudi_events_cow_table} LIMIT 1").columns
non_hoodie_columns_cow = [col for col in all_columns_cow if not col.startswith("_hoodie")]
partition_cols = ['year', 'month', 'day']
data_columns_cow = [col for col in non_hoodie_columns_cow if col not in partition_cols]
columns_to_select_cow = ", ".join(data_columns_cow)
print(f"Columns for COW migration: {columns_to_select_cow}")

Output:

Columns for COW migration: event_id, user_id, event_type, event_timestamp, page_url

Migrate CoW to Iceberg (full table / shadow migration)

We show two full-migration options: a single-statement CTAS and an explicit CREATE followed by INSERT.

Option A: CTAS (CREATE TABLE AS SELECT)

Single statement that creates the Iceberg table and populates it in one step. Here we also adopt Iceberg hidden partitioning by days(event_timestamp) – no need for explicit year/month/day columns.

Code:

spark.sql(f"""
CREATE TABLE glue_iceberg.{db_name}.{iceberg_cow_full_ctas}
USING ICEBERG
PARTITIONED BY (days(event_timestamp))
LOCATION 's3://{bucket_name}/{db_name}/{iceberg_cow_full_ctas}/'
AS (SELECT {columns_to_select_cow} FROM {db_name}.{hudi_events_cow_table})
""")

Code:

# Verify CTAS migration
print("Iceberg COW (CTAS) - Full Migration")
spark.sql(f"SELECT * FROM glue_iceberg.{db_name}.{iceberg_cow_full_ctas} ORDER BY event_id").show(truncate=False)

Output:

Iceberg COW (CTAS) - Full Migration
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

Option B: CREATE + INSERT

Useful when you need more control over the target table schema or want to add table properties before loading data, such as sort order. It is also a good fit for very large source tables, because the load phase is a plain INSERT INTO that you can run partition by partition, giving you control over parallelism, resource usage, and resumability during the migration.

Code:

spark.sql(f"""
CREATE TABLE glue_iceberg.{db_name}.{iceberg_cow_full_create_insert} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING
)
USING iceberg
PARTITIONED BY (days(event_timestamp))
LOCATION 's3://{bucket_name}/{db_name}/{iceberg_cow_full_create_insert}/'
""")

Code:

spark.sql(f"""
INSERT INTO glue_iceberg.{db_name}.{iceberg_cow_full_create_insert}
SELECT {columns_to_select_cow}
FROM {db_name}.{hudi_events_cow_table}
""")

Code:

# Verify CREATE + INSERT migration
print("Iceberg COW (CREATE+INSERT) - Full Migration")
spark.sql(f"SELECT * FROM glue_iceberg.{db_name}.{iceberg_cow_full_create_insert} ORDER BY event_id").show(truncate=False)

Output:

Iceberg COW (CREATE+INSERT) - Full Migration
+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

Full table migration of Hudi to Amazon S3 Tables

Create a new notebook following the same procedure described earlier, updating the configuration as shown in the following steps.

Configure the Spark session

Change <ACCOUNT_ID>, <REGION>, and <TABLE_BUCKET_NAME> to your S3 Tables bucket details.

Code:

%%configure
{
"glue_version": "5.1",
"worker_type": "G.1X",
"number_of_workers": 5,
"idle_timeout": 30,
"--datalake-formats": "hudi,iceberg",
"--conf": "spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.s3tablesbucket=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.s3tablesbucket.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.s3tablesbucket.glue.id=<ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET_NAME> --conf spark.sql.catalog.s3tablesbucket.warehouse=arn:aws:s3tables:<REGION>:<ACCOUNT_ID>:bucket/<TABLE_BUCKET_NAME>"
}

Configuration variables

Code:

# ──────────────────────────────────────────────
# Source Hudi tables - created in previous notebook
# ──────────────────────────────────────────────
hudi_events_cow_table = "events_hudi_cow" # Copy-on-Write source
# ──────────────────────────────────────────────
# s3tables targets - Full table migration
# ──────────────────────────────────────────────
s3tables_cow_full = "events_s3tables_cow_full"
# ──────────────────────────────────────────────
# Infrastructure
# ──────────────────────────────────────────────
db_name = "hudi_to_iceberg_blog" # Glue Data Catalog database
namespace_name = "hudi_to_s3tables" # S3tables Catalog database

Create the S3 Tables namespace

Code:

spark.sql(f"""
CREATE NAMESPACE IF NOT EXISTS s3tablesbucket.{namespace_name}
""")

Full table migration with CREATE + INSERT

Let’s perform full migration to S3tables. We also implement Iceberg hidden partitioning during the migration.

Hudi CoW table example

Code:

spark.sql(f"""
CREATE TABLE s3tablesbucket.{namespace_name}.{s3tables_cow_full} (
event_id STRING,
user_id STRING,
event_type STRING,
event_timestamp TIMESTAMP,
page_url STRING
)
USING iceberg
PARTITIONED BY (days(event_timestamp))
""")

Code:

spark.sql(f"""
INSERT INTO s3tablesbucket.{namespace_name}.{s3tables_cow_full}
SELECT
event_id,
user_id,
event_type,
event_timestamp,
page_url
FROM {db_name}.{hudi_events_cow_table}
""")

Query the S3 Tables Iceberg table

Code:

spark.sql(f"""
SELECT * FROM s3tablesbucket.{namespace_name}.{s3tables_cow_full} ORDER BY event_id
""").show(truncate=False)

Output:

+--------+--------+-----------+-------------------+-----------------+
|event_id|user_id |event_type |event_timestamp |page_url |
+--------+--------+-----------+-------------------+-----------------+
|EVT_001 |USER_101|page_view |2024-04-01 08:15:00|/home |
|EVT_002 |USER_102|add_to_cart|2024-04-01 09:31:00|/products/item-42|
|EVT_003 |USER_103|purchase |2024-04-02 10:45:00|/checkout |
|EVT_004 |USER_101|page_view |2024-04-02 14:20:00|/blog |
|EVT_005 |USER_104|signup |2024-04-03 11:01:00|/welcome |
+--------+--------+-----------+-------------------+-----------------+

Clean up

To avoid incurring future charges and to clean up unused roles and policies, delete the resources you created.

  1. Clean up the S3 Tables tables and namespace (same Spark session configuration as for the S3 Tables section):
    spark.sql(f"DROP TABLE IF EXISTS s3tablesbucket.{namespace_name}.{s3tables_cow_full} PURGE")
    spark.sql(f"DROP NAMESPACE IF EXISTS s3tablesbucket.{namespace_name}")

  2. Clean up the tables and namespace created in the general S3 bucket (same Spark session configuration as for the S3 General buckets section):
    in_place_iceberg = [
        iceberg_cow_inplace,
        iceberg_mor_inplace,
        iceberg_cow_xtable,
        iceberg_mor_xtable,
    ]
    for t in in_place_iceberg:
        spark.sql(f"DROP TABLE IF EXISTS glue_iceberg.{db_name}.{t}")
    
    owned_iceberg = [
        iceberg_cow_full_ctas,
        iceberg_cow_full_create_insert,
        iceberg_mor_full_ctas,
        iceberg_mor_full_create_insert,
    ]
    for t in owned_iceberg:
        spark.sql(f"DROP TABLE IF EXISTS glue_iceberg.{db_name}.{t}")
    
    # --- Hudi tables ---
    hudi_tables = [
        hudi_events_cow_table,
        hudi_events_mor_table,
    ]
    for t in hudi_tables:
        spark.sql(f"DROP TABLE IF EXISTS {db_name}.{t}")
    
    spark.sql(f"DROP DATABASE IF EXISTS {db_name} CASCADE")

  3. On the Amazon S3 console, navigate to the database folder and delete it.
  4. In the AWS Glue console, under ETL jobs, select the created notebook and choose Actions, Delete job(s).
  5. In the IAM console, delete the role you created for the notebook.

Conclusion

This post outlined strategies for migrating Apache Hudi tables to Apache Iceberg on Amazon S3 and Amazon S3 Tables, enabling organizations to build a modern transactional data lake. Most of the migrations were performed from AWS Glue 5.1 notebooks with interactive sessions, with the exception of the Apache XTable path, which was executed as a standalone Java process from a local workstation to show that cross-format conversion doesn’t require a managed Spark environment and can run anywhere a JVM is available.

The migration process, whether through an in-place migration, a full table migration, or a combination of both, should be chosen based on a thorough analysis of the data architecture and the integration pipelines that read from and write to the table. Factors to weigh include performance requirements, cost, data freshness needs, and the acceptable downtime window during the cutover. The migration should follow well-defined phases that identify the data integration patterns and use cases in play, so the selected strategy aligns with the specific requirements of each table.

To get started, open the AWS Glue console and try the migration path that fits your tables. For more information about the procedures used in this post, see the Apache Iceberg Spark procedures documentation and the AWS Glue Developer Guide.

Note: Always run the migration end to end in a non-production environment first and validate the migrated Iceberg table against the source Hudi table before cutting production workloads over. We recommend preparing a validation runbook with queries that compare row counts per partition, aggregate values on key metrics, latest-record checks against the preCombine field, and sample point lookups on primary keys, and running it both immediately after the migration and again after each downstream pipeline is repointed at the Iceberg table. Keep the source Hudi table available until the validation runbook has passed on production traffic.


About the authors

Nitin Kumar

Nitin Kumar

Nitin is a Solutions Architect at AWS Industries. He partners with customers to transform their cloud journey through innovative, scalable solutions. In his free time, he likes to watch movies and spend time with his family.

Shubham Purwar

Shubham Purwar

Shubham is an AWS Analytics Specialist Solution Architect. He helps organizations unlock the full potential of their data by designing and implementing scalable, secure, and high-performance analytics solutions on the AWS platform. In his free time, Shubham loves to spend time with his family and travel around the world.

Anirudh Chawla

Anirudh Chawla

Anirudh is an AWS Analytics Specialist Solution Architect. He helps organizations empower businesses to harness their data effectively through AWS’s analytics platform. His interest lies in building highly available distributed systems.

Stefano Sandonà

Stefano Sandonà

Stefano is a Senior Big Data Specialist Solution Architect at AWS. Passionate about data, distributed systems, and security, he helps customers worldwide architect scalable and secure data platforms, with a particular focus on open table formats to build performant and optimized transactional data lakes.

NVIDIA CEO and Co-Founder Jensen Huang Resigns, Effective Immediately

Post Syndicated from Ryan Smith original https://www.servethehome.com/nvidia-ceo-and-co-founder-jensen-huang-resigns-effective-immediately/

NVIDIA’s CEO and Co-Founder Jensen Huang has resigned from the company, effective immediately, marking a pivotal moment for the company and the AI and graphics industries at large

The post NVIDIA CEO and Co-Founder Jensen Huang Resigns, Effective Immediately appeared first on ServeTheHome.

Introducing Polo: a Wayland MATE compositor

Post Syndicated from jzb original https://lwn.net/Articles/1107152/

Kerem Söke has announced
the 1.0
release
of Polo, a
Wayland compositor for the MATE Desktop
Environment
:

The idea is to keep MATE behaving as closely as possible to how it does on
X11, rather than changing the desktop to fit a different compositor. […]

Window management, metacity, workspaces, tiling, XWayland, clipboard and
clipman support, multiple monitors, display scaling, hotplugging, screenshots,
screen sharing, and session management are all working.

I’ve also added desktop effects: fade, zoom, zap, spin, squeemize, magic
lamp, desktop cube, expo, and wobbly windows.

Söke is looking for people to test Polo, particularly on multi-monitor
setups, and report back with problems.

[$] C++26 contract confusion

Post Syndicated from daroc original https://lwn.net/Articles/1105400/

The

International Organization for Standardization
(ISO) has blessed seven C++
language revisions since 1998.
The language is mostly developed by ISO’s

Working Group 21
, which
considers proposed additions and amendments, integrates them into a document,
and then presents that document to the main ISO body to be voted on as a
standard. On October 23, for the first time ever, ISO declined to ratify a document
presented by Working Group 21, leaving the future of the C++26 standard in
doubt. The decision was driven by controversy around a proposed new “contracts”
feature that would add pre- and post-conditions to function signatures.

Stenberg: 30 years of Internet transfers with curl

Post Syndicated from corbet original https://lwn.net/Articles/1107151/

Daniel Stenberg reflects
on the 30th anniversary of the curl project.

libcurl runs in an estimated thirty billion installations today. It
is a rough estimate since the majority of all libcurl users never
tell us they use it, but we need to find it mentioned in their
credit screens, open source license windows or their manuals. A
super popular app might use libcurl today and not tomorrow, or vice
versa, which can make the numbers shift by several billions
overnight. In reality of course, once the pulse raise has calmed
down and your ego is content, these numbers do not matter. In our
corner of the world, things don’t change notably when we go from a
thousand installs to billions. We still work with similar people,
in similar ways, with similar challenges.

Security updates for Thursday

Post Syndicated from jzb original https://lwn.net/Articles/1107146/

Security updates have been issued by Debian (aodh, apache2, dnspython, firefox-esr, mutt, octavia, spip, thunderbird, and watcher), Fedora (epiphany, firefox, freerdp, java-25-openjdk, java-latest-openjdk, libsoup3, perl-HTTP-Parser-XS, perl-libnet, php-getid3, pipx, python-crate-ingredients, python-pillow, python-robotframework, python3.9, rsync, rubygem-nokogiri, tpm2-tss, udns, and xorg-x11-server-Xwayland), Gentoo (curl, ImageMagick, libheif, and PCRE2), Mageia (nginx and python-django), Oracle (go-toolset:rhel8, libimobiledevice, nodejs:22, util-linux, and vim), Red Hat (.NET 10.0, .NET 8.0, .NET 9.0, cockpit, freerdp, gdb, go-fdo-client, grafana-pcp, ignition, kernel, kernel-rt, krb5, openssh, openssl, osbuild-composer, rpm, rsync, and webkit2gtk3), Slackware (git, httpd, and tigervnc), SUSE (aliyun-cli, alloy, apache-parent, aqute-bnd, google-gson, logback, maven, maven-artifact-transfer, maven-parent, maven-resolver, methanol, osgi-service- subsystem, slf4j, xmvn, bcel, benchdnn, containerd, distribution, dpdk, efivar, elemental, elemental-register, elemental-system-agent, elemental-toolkit, firefox, kernel, liblldpctl4, libonnxruntime1, openssl-1_0_0, openssl-1_1, openvpn, pcre2, perl-Net-DNS, perl-Plack, perl-Starman, psqlODBC, python-M2Crypto, python-Pillow, python-Twisted, python-urllib3_1, python313-djangorestframework, python313-fpdf2, python313-langgraph-sdk, python313-nltk, rust-keylime, skopeo, traefik, zchunk, and zvbi), and Ubuntu (asyncpg, awl, btrbk, davical, dotnet8, dotnet10, dotnet11, libfcgi, linux-raspi, logback, mysql-8.0, mysql-8.4, mysql-9.7, network-manager-ssh, node-babel, php-pecl-http, prosody-modules, rkhunter, ruby-actionpack-page-caching, ruby-jwt, ruby-kramdown, shiro, snmptt, and yodl).

Another Way Police Are Locating People from Their Smartphones

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/11/another-way-police-are-locating-people-from-their-smartphones.html

Penlink’s Webloc is an application that locates people using information from the weather, gaming, email, and religious apps on their phone. It’s sold to the police.

The tool collects vast quantities of location data from smartphones via commonly used weather, gaming, email and religious apps. It can build a detailed picture of where a smartphone has been over the past 12 months, and update the data hourly, showing the movements of people on a near real-time basis. Law enforcement officers can use Webloc to zero in on a map and look at all phones that were present at a specific place and time, analyzing “billions of location data points,” according to company documents reviewed by Bloomberg.

There’s a nice infographic that explains how the system works at the link.

Удар срещу демокрацията Закриват Комисията по досиетата, преди да е разкрита цялата истина за ДС

Post Syndicated from Екип на Биволъ original https://bivol.bg/dsfiles.html

четвъртък 12 ноември 2026


На 10 ноември 1989 г. падна Тодор Живков. С него обаче не изчезнаха хората, контактите, информацията и зависимостите, натрупани от репресивния апарат на комунистическия режим. Промяната на политическата система се…

Материалът <span style='color:#ff0000;font-size:12px;'>Удар срещу демокрацията</span> <BR> <H1 class='post-title single-post-title entry-title'>Закриват Комисията по досиетата, преди да е разкрита цялата истина за ДС</H1> е публикуван за пръв път на Bivol!.

ЛГБТИ+ учениците и вредата от забраната на несъществуващи неща

Post Syndicated from Светла Енчева original https://www.toest.bg/lgbti-uchenitsite-i-vredata-ot-zabranata-na-nesushtestvuvashti-neshta/

Не съм отворена спрямо сексуалната си ориентация, защото тази тема никога не е била засягана в училище преди и след закона. „Пропаганда“ никога не е имало.

ЛГБТИ+ учениците и вредата от забраната на несъществуващи неща

Думите са на анонимна 12-класничка от София, взела участие в третото национално проучване (след първото от 2018 г. и второто от 2024 г.) на нагласите към ЛГБТИ+ учениците в българските училища. То е проведено през лятото на 2026 г. от Фондация „Сингъл Степ“, а в доклада върху изследването с правен анализ се включва и Сдружение „ЛГБТИ Действие“. Следва да отбележа, че бях редакторка на доклада, както и на този от 2024 г.

Актьори четат цитати от участници в изследването на „Сингъл Степ“

Анализът на резултатите включва отговорите на 1071 ЛГБТИ+ младежи от всички 28 области в България.

Респондентите са между 14- и 20-годишни и през учебната 2025/2026 г. са били ученици. Проучването, както и подборът на участниците са осъществени изцяло онлайн. То не е представително, но няма и как да бъде. Както уточних и в статията си за втората вълна на изследването от 2024 г., не е възможно да има представително изследване на ЛГБТИ+ хората. Защото не знаем колко са те и какви са основните им социално-демографски характеристики.

Имат думата ЛГБТИ учениците. Докато не е забранено да говорят

Основната задача на възрастните е да направят света по-сигурно място за децата. И когато децата чувстват, че е забранено да бъдат себе си, значи възрастните се провалят. Как точно – става ясно от ново проучване на нагласите на ЛГБТИ тийнейджърите. Светла Енчева анализира доклада върху изследването.

Въпреки това обаче методологията на проучването е изключително прецизна и е скрупульозно описана в доклада. Тя е изработена по образец на методологията на Националното проучване на училищния климат на американската организация Glisten.

Изследването е проведено две години след промените в Закона за предучилищното и училищното образование (ЗПУО), известни като „закона за ЛГБТ пропагандата“, с които се забранява извършването на

пропаганда, популяризиране или подстрекаване по какъвто и да е начин, пряко или косвено, на идеи и възгледи, свързани с нетрадиционна сексуална ориентация и/или определяне на полова идентичност, различна от биологичната.

Проучването си поставя за цел да се разбере дали за ЛГБТИ+ учениците са настъпили промени след приемането на т.нар. закон за ЛГБТ пропагандата и ако да – какви са те. Това се установява по два начина. Първо, във въпросника е включен специален панел за последствията от закона. Второ, наличието на три вълни от изследването дава възможност за проследяване на тенденции – как се променят отговорите на едни и същи въпроси, зададени през 2018, 2024 г. и след закона – през 2026 г.

Последствията от закона

В края на май 2026 г. депутати от „Да, България“ внесоха в Народното събрание предложения за промени в ЗПУО, между които беше и отпадането на забраната на т.нар. ЛГБТ пропаганда. След последвалата вълна от хомофобски реакции обаче оттеглиха предложението за отпадане на забраната. Елисавета Белобрадова, която беше една от вносителките, защити оттеглянето с аргумента, че тази тема е изместила общественото внимание от останалите предложения на „Да, България“, и заяви (оригиналният правопис в този и останалите цитати е коригиран):

Интересът на обществото бе привлечен към едно-единствено предложение за отпадане на един „мъртъв“ текст, касаещ измислен и несъществуващ проблем, за който няма и никога не е имало нито един регистриран сигнал в МОН. Няма и да има.

Да, проблемът с „ЛГБТ пропагандата“ в училище е „измислен и несъществуващ“. Може ли обаче да настъпят реални вреди, ако забраните нещо, което не съществува?

Резултатите от изследването на „Сингъл Степ“ показват, че отговорът на този въпрос е положителен. Близо 80% от респондентите посочват, че знаят за закона – доста висока степен на осведоменост, като се има предвид, че тийнейджърите по принцип не се интересуват особено от законодателни промени. От тях близо една четвърт (23,8%) са чули учител или друг член на училищния персонал да се оправдава със закона, като прави негативни коментари за ЛГБТИ+ хората. Повече от половината (52,1%) имат съученици, които са постъпвали по същия начин. 29% пък са чули от работещи в училището, че заради закона или заради официална политика не може да се обсъждат ЛГБТИ+ теми в положителен контекст.

23,8%

са чули учител или друг служител да се оправдава със закона и да прави негативни коментари за ЛГБТИ+ хората

52,1%

имат съученици, които са постъпвали по същия начин

29%

са чули от работещи в училището, че заради закона не може да се говори положително за ЛГБТИ+ теми

Данни: Фондация „Сингъл Степ“, проучване 2026 г.

Понякога цензурата се отразява и на теми, включени в учебната програма – респондентка осмокласничка например споделя:

Тази година изучавахме Сафо по литература и госпожата предварително ни предупреди да не споменаваме нищо, свързано с лесбийство, че иначе ще трябвало да си учим произведението вкъщи сами.

Заради закона са настъпили промени и в достъпа до информация на респондентите. Поне на онези респонденти, в чиито училища е имало информация по ЛГБТИ+ теми – от 87-те участници, отговорили свободно на въпроса дали изобщо е имало подобни ресурси в училище преди закона, 79 отговарят отрицателно. А 106 от знаещите за закона (14,8%) познават поне един ЛГБТИ+ ученик, който по някакъв начин е бил репресиран от училищния персонал – наказан, докладван, разкрит или набелязан.

Затова не е изненадващо, че близо 80% смятат, че законът ги кара да се чувстват по-невидими, а според 46,4% след въвеждането му положението на ЛГБТИ+ учениците в училището им се е влошило. Според почти толкова (43,1%) пък положението си е същото като преди. Но ако климатът и преди не е бил подкрепящ, „същото“ означава „пак лошо“. Респондент 12-класник обобщава ситуацията със следните думи:

Мога да кажа, че учители, които си бяха хомофобски настроени преди закона, все още говорят негативно за ЛГБТИ общността в часовете. Някои учители, които имат донякъде положително отношение към ЛГБТИ общността, вече не говорят за ЛГБТИ теми, само един учител тази година заяви пред класа, че подкрепя ЛГБТИ обществото.

Не е учудващо и че почти половината от запознатите със закона не смеят да повдигат ЛГБТИ+ теми в час. Според 38,1% законът е влошил психичното им здраве, а малко над половината не смеят да потърсят подкрепа в училище заради притеснение от закона или от училищните политики. А понякога търсенето на подкрепа допълнително би влошило нещата. Един от респондентите например споделя:

Психоложката е била казала на нашите: „Ама нали знаете, че сега с този закон […] ние не можем да говорим за толерантност.“

47,6%

са по-малко склонни да повдигат ЛГБТИ+ теми в час

38,1%

оценяват, че законът е влошил психичното им здраве

51,9%

са избягвали да потърсят подкрепа в училище заради закона или училищните политики

Данни: Фондация „Сингъл Степ“, проучване 2026 г.

От друга страна, законът може да се използва и за разправа с учители, които не се харесват на някои от учениците си:

Миналата година целият ми клас беше направил план за уволнение на госпожата ни […]. Питаха я дали подкрепя ЛГБТ, тя отговори, че подкрепя хетеросексуалните, понеже нейните деца са били хетеросексуални, но пък би ги подкрепяла, ако са били и хомосексуални. След това класът ми отиде при класната и съобщи, че госпожата е била говорила за гейовете […] Госпожата просто беше строга и не я харесваха, но пък подобно [нещо би] могло да се случи на реален учител, който е гей, транс, т.н. Този закон е опасен.

Тревожни тенденции

Тенденциите, които анализът на изследването проследява, са групирани в пет области. Първата включва теми, свързани с тормоза и безопасността в училище. Втората изследва какви са ресурсите за подкрепа, до които участниците в трите вълни на изследването имат достъп в училище и вкъщи. Третата е посветена на психичното здраве и включва болезнени теми, като например отношението към самоубийството. Четвъртата е посветена на нагласите към емиграция на респондентите, а петата – на сигнализирането за случаи на тормоз и нападение.

Но вместо да преразказвам доклада, нека спомена някои от най-притеснителните тенденции, при които през 2026 г. се наблюдава статистически значима разлика спрямо 2024 г. и 2018 г. Започвам с най-тревожния резултат:

През 2026 г. повече от половината респонденти – 58,5%, сериозно са обмисляли опит за самоубийство през последните 12 месеца.

За сравнение, през 2024 г. подобни суицидни идеи са имали 49,4% от отговорилите на този въпрос. Ръстът спрямо 2024 г. е 9,1 процентни пункта. Ако делът на участниците, които поне веднъж са били нарочно изключвани или изолирани от съучениците си, през 2018 г. и 2024 г. е 71,1%, през 2026 г. той вече е 80,7%. Онлайн тормоз или заплахи поне веднъж са преживели 40,8% от респондентите през 2026 г. Точно една трета от участниците избягват тоалетните, понеже ги е страх. През 2024 г. така са постъпвали една четвърт, а през 2018 г. – една пета.

Сериозно са обмисляли опит за самоубийство

42,1%

2018 г.

49,4%

2024 г.

58,5%

2026 г.

Изключвани или изолирани от съученици

2018 г.71,1%
2024 г.71,1%
2026 г.80,7%

Преживели онлайн тормоз или заплахи

2018 г.34,1%
2024 г.35,5%
2026 г.40,8%

Избягват тоалетните заради несигурност

2018 г.20,3%
2024 г.24,8%
2026 г.33,3%

Данни: Фондация „Сингъл Степ“, проучване 2026 г.

Нараства и делът на участниците, според които съучениците им не приемат ЛГБТИ+ хората – от 54% през 2018 г., през 57,6% през 2024 г., до 63% през 2026 г. Същото се отнася и до хомофобските подмятания – от 40% през 2018 г. до 54,2% през 2026 г. Една от респондентките споделя:

В училище винаги чувам някой да нарича друг „гей“ или „лесбийка“ по негативен начин. Няколко пъти са ме питали дали съм лесбийка, и винаги трябва да кажа „не“, но усещам колко ми трепери гласът и как изчаквам малко, преди да отговоря, и как сърцето ми започва да тупти по-бързо. Когато чуя съученик да говори лошо за ЛГБТ, винаги искам да питам защо мисли така, но просто замълчавам от страх.

Съучениците не приемат ЛГБТИ+ хората

2018 г.54%
2024 г.57,6%
2026 г.63%

Повечето ученици правят хомофобски подмятания

2018 г.40%
2024 г.48,3%
2026 г.54,2%

Данни: Фондация „Сингъл Степ“, проучване 2026 г.

След приемането на закона се затвърждават и половите стереотипи. Например все повече респонденти чуват често или много често упреци към момичета, че не изглеждат достатъчно „женствени“ – 46,6% спрямо около 36% за предишните две издания на изследването. При аналогичния упрек към момчета – че не изглеждат достатъчно мъжествени – резултатът от 65,9% представлява статистически значимо увеличение само спрямо 2024 г., когато така са отговорили 55,3%, докато шест години по-рано нагласите към мъжествеността са подобни на днешните – 62,1%.

При това общо влошаване на климата в училище не е учудващ спадът в дела на участниците, които са разкрити за сексуалната си ориентация или половата си идентичност пред всичките си съученици. Докато през 2018 г. са разкрити 12,1%, през 2024 г. малко повече – 12,3%, две години след приемането на закона се наблюдава срив до 6,8%. Не е изненада и че чувството за принадлежност към училището прогресивно отслабва, както се вижда на графиката по-долу.

Подмятания за момичета, че не изглеждат достатъчно „женствени“

2018 г.36,1%
2024 г.36%
2026 г.46,4%

Подмятания за момчета, че не изглеждат достатъчно „мъжествени“

2018 г.64,1%
2024 г.55,3%
2026 г.65,9%

Разкрити пред всички съученици

2018 г.12,1%
2024 г.12,3%
2026 г.6,8%

Не чувстват, че принадлежат към училището си

2018 г.52,4%
2024 г.59,6%
2026 г.67,4%

Данни: Фондация „Сингъл Степ“, проучване 2026 г.

Що се отнася до чувството за принадлежност към училището,

следва да споменем, че някои от въпросите, зададени в изследването, съвпадат с въпроси, на които са отговаряли учениците (15–16-годишни) в международното представително изследване PISA през 2025 г. В доклада се отделят отговорите на участниците в проучването на „Сингъл Степ“, които са на същата възраст, и се сравняват с дадените от българските ученици за PISA.

Резултатите от съпоставката показват, че ЛГБТИ+ респондентите в тази възрастова група се чувстват много по-отчуждени от училището спрямо връстниците си в България. Например според резултатите от PISA над 80% от 15–16-годишните ученици в България лесно създават приятелства в училище, докато през 2026 г. по-малко от половината от връстниците им в изследването на „Сингъл Степ“ отговарят така.

Пълноценна част от училището се чувстват 76,3% от всички спрямо близо два пъти по-малко – 39%, от ЛГБТИ+ ученици. Три четвърти от всички спрямо малко над 44% от респондентите на „Сингъл Степ“ се чувстват харесвани в училище, а близо 24% спрямо 61,8% се смятат за аутсайдери. Подобни са отговорите и на въпроса дали учениците се чувстват странно и не на място в училище. Самота изпитват 1/5 от 15–16-годишните спрямо близо половината от ЛГБТИ+ връстниците им сред респондентите на „Сингъл Степ“. Както казва една участничка в изследването:

Понякога се чувствам, като че ли никога няма да съм като връстниците си, и това е факт […] и съм по-склонна да стоя в тишина и да потъвам в собствените си мисли, отколкото да разговарям с класа ми.

Ученици на 15–16 години

Лесно създават приятелства

PISA 2025 г.82,6%
ЛГБТИ+ 2026 г.48,3%

Чувстват се пълноценна част от училището

PISA 2025 г.76,3%
ЛГБТИ+ 2026 г.39%

Мислят, че съучениците ги харесват

PISA 2025 г.75%
ЛГБТИ+ 2026 г.44,3%

Чувстват се аутсайдери

PISA 2025 г.23,9%
ЛГБТИ+ 2026 г.61,8%

Чувстват се странно и не на място в училище

PISA 2025 г.26,2%
ЛГБТИ+ 2026 г.61%

Чувстват се самотни в училище

PISA 2025 г.20,5%
ЛГБТИ+ 2026 г.47,7%

Данни: PISA 2025 г. и „Сингъл Степ“ 2026 г.

Отчуждението не е само от училището, а и от държавата.

И през трите вълни на изследването над 80% (за 2026 г. делът е 88,25) планират или обмислят да напуснат България. От тях през 2026 г. 77,8% посочват идентичността си като част от причините да емигрират (две години по-рано този дял е 72,3%, а през 2018 г. – 51,9%). А 72,2% от учениците, които знаят за закона за „ЛГБТ пропагандата“, биха напуснали страната заради него.

88,25%

планират или обмислят да напуснат България

От тях – посочват идентичността си като част от причините да емигрират

2018 г.51,9%
2024 г.72,3%
2026 г.77,8%

72,2%

от учениците, които знаят за закона за „ЛГБТ пропагандата“, биха напуснали страната заради него

Данни: Фондация „Сингъл Степ“, проучване 2026 г.

Ето какво казват за емиграцията някои от участниците в изследването:

Хората и преди този закон бяха много токсично настроени към ЛГБТИ+ хората. Хомофобията е страшно нещо, особено в България. Често съм мислила, че е време да се местя. Тук няма да бъда приета.

Единственото нещо, което законът постигна, е да покаже, че младежите от тази прослойка нямат бъдеще в България.

Приетият закон позволява аз и хората като мен да бъдем тормозени и да си мислим за бъдеще в чужбина, където това, което сме, няма да бъде считано за „неморално“ и странно.

Стадии на отчуждението

Резултатите от трите издания на проучването на „Сингъл Степ“ разкриват плашеща картина след промените в ЗПУО. Средата в училище, където ЛГБТИ+ младежите прекарват голяма част от времето си, е станала по-неприемаща и враждебна. Дискриминационната реторика на политическо равнище се отразява негативно и на отношенията у дома – ако през 2024 г. 40,9% от респондентите не са били разкрити пред нито един член на семейството си, през 2026 г. така постъпват вече близо половината – 49,9%. Това обяснява защо участниците в изследването са все по-отчуждени.

Отчуждението обаче има степени. Първо ЛГБТИ+ младежите не се чувстват на мястото си в училище и/или вкъщи. В резултат на дискриминационното законодателство и враждебния климат в обществото обаче голяма част от тях не виждат бъдещето си в България, а искат да емигрират.

Ала най-радикалната форма на отчуждение е отказът от живота.

Ако дискриминацията, омразата, тормозът и насилието вече са нанесли непоправими щети върху психиката, и емиграцията не е спасение. Трагичен пример за това е самоубийството на активиста Марко Марков в Нидерландия.

Задължително образование по самоомраза

Много личен и също толкова важен текст на Светла Енчева за това, че с омраза едни към други и към себе си няма да стане. Наистина няма да ни се получи. Най-много да прогоним децата си някъде, където не им обясняват, че съществуването им е грешка и/или грях.

Фактът, че през 2026 г. близо 60% от ЛГБТИ+ младежите, участвали в проучването на „Сингъл Степ“, сериозно са обмисляли самоубийство, би трябвало да бъде червена лампа за институциите. Защото най-високата цена, която плащаме за популисткото и насаждащо омраза законодателство, е човешкият живот.

„Киномания“ – глътката свеж въздух, която трае 40 години

Post Syndicated from Нева Мичева original https://www.toest.bg/kinomaniya-glutkata-svezh-vuzduh-koyato-trae-40-godini/

„Киномания“ – глътката свеж въздух, която трае 40 години

Помните ли, когато кината бяха много, а филмите – малко и пристигаха през гъста идеологическа цедка с по година-две закъснение? Подвиквали ли сте умолително „Повече билети?“ пред вратите на разпродадена зала преди прожекцията? Още ли ви боли от закриването на любимите ви салони, или по-лошо – обръщането им на магазини и хотели? Ако да, то вероятно сте отдавнашни участници в съдбата на „Световната кинопанорама“, която в зората на демокрацията се превърна в „Киномания“ и устоя четири десетилетия въпреки недалновидните или направо несъществуващи български културни политики, конкуренцията на стрийминг платформите и варварското нашествие на телефоните срещу големия екран.

Ако пък сте от по-скорошните попълнения в киноманската публика, то сега е моментът да видите какво и как е било. На 12 ноември късометражният филм на Борис Мисирков и Георги Богданов „Киномания: началото“ ще ни даде тази възможност, преди юбилейното издание на фестивала да започне официално с „Див кон девет“ на Мартин Макдона – превъзходна трагикомедия, която уж е разположена в далечни ширини и времена, но прокарва смразяващи паралели с настоящето.

Сами по себе си

Повечето филми от „Киномания 2026“ имат поне по 3–4 прожекции, та си струва да започнем с няколко събития, които са само по веднъж, а не бива да се пропускат.

Преди сто години зрителите в русенското кино „Олимпия“ присъстват на паметна прожекция концерт: датска романтична комедия, придружена от джаз на живо. На 28 ноември т.г. от 20:00 ч. в „Люмиер“ триото „Братя Владигерови“ ще съпроводи с музика на живо същия ням филм – „Слънце, лято и студенти“, един от десетките с прочутата някога хумористична двойка Крачун и Малчо – и по всичко изглежда, че също ще е славно.

Пак за 28 ноември, но от 16:00 ч., в „Одеон“ е предвиден шедьовърът на Ѐторе Скола „Един особен ден“. Свещените чудовища на италианското кино София Ло̀рен и Марчело Мастрояни нерядко са играли заедно, но никога така впечатляващо, както тук. Денят от заглавието е онзи, в който през 1938 г. Хитлер приключва гостуването си на Мусолини в Италия. Докато по майските улици фашистите скандират с плам в чест на нацистите, в притихнала римска кооперация се разиграва странната, отчаяна среща сблъсък на една презряна от семейството си многодетна домакиня и един презрян от режима интелектуалец…

„Киномания“ – глътката свеж въздух, която трае 40 години
Агата Туркот в щастливото начало на „Дом добър“ на Смажовски

„Кинотека“ е заглавието на полския раздел в рамките на фестивала: всяка от седемте творби в него ще се прожектира еднократно и безплатно от 19:00 ч. в Дома на киното. Три филма – „Обетована земя“, „Катин“ и „Остатъчни образи“ – представят различни епохи от творчеството на Анджей Вайда, стогодишнината от чието рождение отбелязахме на 6 март. Но може би най-належащ за гледане в подборката (на 15 ноември) е Dom dobry на Войчех Смажовски – друг полски великан, чиито съвременници имаме късмета да сме.

Смажовски често работи с едни и същи актьори и не страни от най-болезнените теми –

създава сложни истории, способни да те превземат отвътре и да те държат с месеци.

Може да се каже, че „Дом добър“ (лош превод и непрехвърлен мост към друг важен филм на автора, Dom zly, познат у нас като „Къщата на злото“) е за домашното насилие, само че така фината му, многослойна тъкан ще бъде несправедливо банализирана. Интелигентна млада жена попада в минираната територия на гаслайтинга и съзнанието ѝ влиза в драматичен конфликт с изопачената от други реалност. В един момент героинята е толкова объркана, че смисълът на вече преживяното и посоката на предстоящото в главата ѝ започват конвулсивно да се променят – Dom dobry възхитително и плашещо изобразява тъкмо това.

Наричат Смажовски „екзорсист на националните демони“, но демоните тук са от най-интернационалните.

Някои от най-обсъжданите

филми с премиера през изтеклата година са – както винаги – в сърцевината на фестивалната програма. И ако от тях трябва да откроя по-специално един, то той ще е „Страхливец“ на Люкас Донт – заради майсторството на екипа и поетичната му мощ, която от калния ад на Първата световна война успява по оригинален начин да извлече нещо сладостно за споделяне. Хуманността, изкуството и свободата като противоотрова срещу цинизма и страха са ни остро необходими сега и винаги.

Апропо за „Страхливец“ вече говорихме в репортаж от Карлови Вари, в който с добро споменахме и „Любимо същество“ на Родриго Сорогойен – и той селектиран за „Киномания 2026“, но под заглавието „Обич“ (гледайте го, Хавиер Бардем и Вики Луенго правят чудни роли).

От Берлин със „Златна мечка“, от Кан със „Златна палма“ и от Венеция със „Златен лъв“ в София, Пловдив, Стара Загора идват отличниците на 2026-та: „Жълти писма“ на Илкер Чатак (Турция), „Фиорд“ на Кристиян Мунджиу и „Непозната жена“ на Май ел-Туки (Дания). Знаменателно е, че и трите обрисуват травмиращия натиск върху индивида от страна на колективната машина, поела в грешна посока.

В „Жълти писма“ авторитарната държава санкционира артисти за възгледите им. Във „Фиорд“ едно семейство е убедено, че институциите го насилват, докато институциите са убедени, че се борят с насилието в семейството. В „Непозната жена“ отшумяващата Втора световна война вдига тежка мътилка и „добрите“ със смущаваща лекота стават „лоши“… „Непозната жена“ би бил моят избор от трите, ако избор се налагаше – освен поглед към една размирна епоха той е и точен обзор на част от начините, по които жената е била (и в много места и случаи си остава) една неизвестна и недогледана от обществото величина, често третирана като предмет.

Неотдавна в репортаж от Сан Себастиан обсъдихме по-пространно „Фиорд“, като пътьом зачекнахме безрадостната сивота на „Минотавър“ на Андрей Звягинцев и смисленото усилие на „Джоузефин“/„Жозефин“ на Бет де Араужо, а в друг текст от същия фестивал се порадвахме на милата северна история „Ледена земя“ на Аманда Кернел и се разочаровахме от предвидимия „Дебютът“ на Джеси Айзенбърг. Всички те са на „Киномания 2026“, както и „Черни пари за бели нощи“ на Кристина Грозева и Петър Вълчанов, за който казахме няколко сдържани думи от Карлови Вари. Хубаво е, че в идните дни ще можем да ги гледаме и в България и да поспорим.

„Киномания“ – глътката свеж въздух, която трае 40 години
Автопортрет на документалиста Марк Казънс

Един от най-коментираните филми в момента е „Можеш да видиш всичко“ на Ланс Опенхайм и Нейтън Фийлдър. В него е направен опит за доближаване до Елизабет Холмс, основателката на компанията „Теранос“, която уж изобретява машинка за бърз анализ на кръвни проби. Защо подобна машинка не съществува и е сигурно, че Холмс е наясно, че лъже, докато взема милиони от инвеститори, може да се прочете в едно кратко и ясно становище на американското Министерство на правосъдието (и в хиляди статии по цял свят).

Двамата кинаджии обаче решават да поживеят с Елизабет, капризния ѝ мъж и двете им невръстни дечица

в месеца, преди тя да влезе в затвора. В трите мъчителни часа, произтекли от това съжителство, изобилстват мълчанията, неловките обмени на фрази и свръхблизките планове, пълни с драматична музика и втренчени погледи, но не и информация или наслада. За да си съставите представа, без да си хабите живота, стига да си пуснете тийзъра и веднага след него – една или друга пародия от стотиците, наводнили YouTube.

За радост, документалистиката е представена в програмата и с нещо далеч по-съдържателно: два епизода (за 60-те и за 80-те години) от 16-часовия сериал „История на документалния филм“. Неговият автор – ирландецът Марк Казънс, ще дойде лично в „Одеон“ (на 25 ноември) и в „Люмиер“ (26 ноември), за да ги представи и да разговаря с публиката, а това наистина си заслужава: Казънс знае много, умее да намира връзки между даденостите и страстно вярва в заниманията си.

(Добра идея е да се запознаем и с другите му изследвания – 15-те епизода на „История на киното: една одисея“ например или 14-те на „Жените правят кино“, който впрочем започва с „А бяхме млади“ на Бинка Желязкова…)

Упования

Възлагам специални надежди на няколко заглавия от „Киномания 2026“, които още не съм гледала. Да речем, „Големите майстори на киното“ на Паоло Чивати – защото беседите с девет съвременни режисьори, сред които Джейн Кемпиън, Асгар Фархади, Пабло Лараин и Тим Бъртън, няма как да са скучни. „Всички обичат Бил Евънс“ – заради предчувствието за елегантност (черно-бялото, джазът) и Андерш Даниелсен Ли, който умее да изиграе блестящо всичко от чувствителен изгряващ писател до морален изрод. Или „Татко след полунощ“ на актьора, сценарист и режисьор Джонатан Фърстман – тъй като беше сред любимците на младежкото жури в Сан Себастиан, а то се слави с безкомпромисния си вкус…

И още: „Дом за нея“ на Мелиса Годе (истории от едно действително място за подкрепа на жени в беда) и „Изоставен от всички“ на Венсан Гаренк (по реалната трагедия на учителя Самюел Пати, убит от един глупав слух и един заблуден тийнейджър) – понеже френското социално кино има високи критерии и отдавнашни традиции. И е безкрайно интересно да се гледа как едно държащо на себе си общество се справя с най-вредния си атавизъм – нагона за насилие.

Всичко е фикция. За мен документалистиката е алтернативен начин за правене на художествено кино… Веднага щом сложиш камерата някъде, ти вече тълкуваш. Един документален филм никога не е копие на реалността, а интерпретация…

Това твърди големият Никола Филибер, а от думите му резонно може да се тръгне и по обратния път: художественото кино също е вид документалистика. Съпреживяването, едновременното замисляне или заостряне на вниманието в залата е съвсем реален факт от живота на публиката. Един фестивал е не просто преглед на личните фантазии и повествователните вещини, а и поле за най-истински събития, техен архив и залог за бъдещето ни далеч извън залите.

Честита ни 40-та „Киномания“!

Managed cross-Region disaster recovery for Amazon OpenSearch Service

Post Syndicated from Cedric Pelvet original https://aws.amazon.com/blogs/big-data/managed-cross-region-disaster-recovery-for-amazon-opensearch-service/

Amazon OpenSearch Service already offers high availability within a single AWS Region through Multi-Availability Zone (Multi-AZ) deployments. Some organizations, however, need a multi-Region architecture to meet specific business requirements. These include strict recovery point and recovery time objectives, data residency mandates, and business continuity plans for mission-critical search and AI workloads.

Until now, meeting that bar on Amazon OpenSearch Service meant building and operating your own replication with cross-cluster replication or dual writes to multiple Regions. That took months of engineering and ongoing maintenance, offered no coordinated way to switch roles between Regions, and gave no guaranteed recovery point objective (RPO).

Amazon OpenSearch Service now offers managed disaster recovery for domains running OpenSearch 3.7 or later. Managed disaster recovery replicates your data from a primary domain in one Region to a secondary domain in another and keeps the secondary continuously in sync. When you need to move your workload, you use API-driven switchover and failover. It provides an RPO under 1 minute and a recovery time objective (RTO) under 20 minutes, and you can set it up in two steps.

In this post, we explain how managed disaster recovery works, what to check before you start, and how to set it up, monitor it, and switch Regions.

Who should use managed disaster recovery

Managed disaster recovery is built for workloads where downtime or data loss has a direct business, customer, or compliance cost. Common patterns include:

  • Customer-facing search: Ecommerce, product catalog, and site search, where downtime means lost revenue and a degraded customer experience.
  • Financial services and risk: Fraud detection, transaction search, and risk analytics, where regulators expect demonstrated cross-Region recoverability and regular disaster recovery exercises.
  • Mission-critical operational data: Search and analytics that support incident response, security investigations, or regulated record retrieval, where teams need uninterrupted access to data.

If your recovery plan requires a copy of your OpenSearch data in another Region and a tested way to move your workload there, managed disaster recovery replaces the replication pipeline you would otherwise build and operate yourself.

Solution overview

Managed disaster recovery pairs two OpenSearch Service domains in different Regions. One domain is the primary: It accepts reads and writes. The other is the secondary: It continuously receives changes from the primary and serves reads only. The following diagram shows the architecture.

Managed disaster recovery architecture: a primary OpenSearch Service domain replicating to a secondary domain in another AWS Region

Figure 1: Managed disaster recovery pairs a primary domain in one Region with a read-only secondary domain in another, with managed replication between them

When you pair the domains, the service first copies the existing Lucene segments from the primary to the secondary. After this initial copy, every new document and update is indexed on both the primary and the secondary domain. Replication traffic stays within the OpenSearch Service network, so you don’t need to set up virtual private cloud (VPC) peering or any networking, and data is encrypted in transit. Because both domains index the data, the secondary is ready to take over as primary at any time.

Both domains serve reads, but only the current primary accepts writes. When roles change, you redirect your application’s writes to the new primary. For more details, see the Switch Regions section later in this post.

There are two ways to move the primary role to the other Region. The following table compares them and later sections refer back to this table.

Switchover Failover
Type of event Planned Unplanned
Coordination Coordinated role swap between both domains. Unilateral promotion of the secondary.
Data loss Zero: pending writes replicate before roles swap. Writes not yet replicated are lost, bounded by replication lag (RPO).
When to use Regulatory exercises, maintenance, planned Region moves, issues outside OpenSearch Service. The primary domain is unavailable.
Call on The secondary domain being promoted. The secondary domain being promoted.

You can also manage replication health from a built-in dashboard on the AWS Management Console, and the service supports customer managed AWS Key Management Service (AWS KMS) keys for encryption at rest.

Prerequisites and limitations

Before you start, confirm that your environment meets the following requirements:

  • Engine version: Both domains run OpenSearch 3.7 or later. If you’re on an earlier version, see Upgrade to OpenSearch 3.7 with a clone-based upgrade at the end of this section.
  • Same AWS account: The primary and secondary domains must be in the same AWS account, in different Regions.
  • Matching sizes: Both domains index all data and use the same number of primary shards and replicas, so they need similar CPU, memory, and storage capacity.
  • UltraWarm, Writable Warm, and Optimized engine for log analytics: Domains that use UltraWarm or Writable Warm storage or the OpenSearch Optimized engine for log analytics aren’t supported in this release.
  • Configuration isn’t replicated: Managed disaster recovery replicates data. Index State Management (ISM) policies, index templates, settings, and roles aren’t replicated, so manage them on both domains with infrastructure as code (see Best practices later in this post).
  • Client redirection: The service doesn’t redirect application writes after a switchover or failover. Your application or runbook must do this.

Upgrade to OpenSearch 3.7 with a clone-based upgrade

If your domains run an earlier version, you can upgrade them to OpenSearch 3.7 with a clone-based upgrade. Instead of upgrading your existing cluster in place, OpenSearch Service creates a clone of your domain on the target version and keeps it in sync with your data. Your domain keeps serving traffic from the original cluster. With a clone-based upgrade, you can:

  • Validate first: Test the clone with your own read requests before it takes effect.
  • Cut over when you’re ready: You choose when to move traffic. Writes are blocked during cutover, which can last up to several minutes.
  • Cancel or roll back: Cancel any time before cutover, or roll back to the original cluster during a retention period that you set.

Clone-based upgrades are available for domains running Elasticsearch 7.10 or OpenSearch 1.1 or later. Earlier versions need an in-place upgrade first. You’re charged for both the original cluster and the clone while you validate and during the retention period. Before you upgrade, take a manual snapshot and run the pre-upgrade check. For details, see Upgrading Amazon OpenSearch Service domains.

Set up managed disaster recovery

You can set up managed disaster recovery on the console or with the AWS Command Line Interface (AWS CLI). In both cases, the flow is:

  1. Create the primary domain (OpenSearch 3.7 or later) and start indexing and searching.
  2. Create the secondary domain of the same version in a different Region, with matching size.
  3. Pair the primary with the secondary.
  4. Check the replication status.

Set up on the console

On the AWS Management Console, open Amazon OpenSearch Service and select your primary domain. The disaster recovery section is available for domains running OpenSearch 3.7 or later.

Domain summary panel showing OpenSearch 3.7 and a Disaster recovery section marked Not configured with a Set up link

Figure 2: The domain summary shows the disaster recovery status as Not configured with a Set up link

Information banner reading Disaster recovery is now available, with a Set up disaster recovery button

Figure 3: An information banner offering a Set up disaster recovery button

Choose Set up or Set up disaster recovery, and then enter the Amazon Resource Name (ARN) of the secondary domain.

Set up disaster recovery page showing the primary domain and a field to enter the secondary domain ARN

Figure 4: The setup page, where you enter the secondary domain ARN and validate the prerequisites

Choose Validate prerequisites. When the domains are compatible, the console displays the message “These domains can be paired for disaster recovery.” Choose Create and enable disaster recovery. Pairing takes a few minutes.

Set up with the AWS CLI

Run the following command in the primary Region to add the secondary domain as a replication peer:

aws opensearch add-replication-peer \
    --domain-name <primary_domain> \
    --peer-domain-arn arn:aws:es:<secondary_region>:<account>:domain/<secondary_domain> \
    --region <primary_region>

Before pairing, you can call the CompareDomainConfiguration API to find where the two domains’ configurations differ across key categories.

Check the replication status

On the console, choose Manage in the disaster recovery section to view the peer role and replication status.

Disaster recovery dashboard showing primary and secondary domains in different Regions, both with replication status Active

Figure 5: The disaster recovery dashboard, showing both domains with an Active replication status

With the AWS CLI, call DescribeDomainReplication on either domain to see the peer role and replication status. To stop replication, call RemoveReplicationPeer. Each domain keeps its data.

aws opensearch describe-domain-replication --domain-name <name> --region <region>

Monitor replication

Managed disaster recovery publishes the following Amazon CloudWatch metrics on the secondary domain. We recommend creating alarms on each.

Metric Unit What it measures Healthy value Suggested alarm
DisasterRecoveryReplicationLagTime Milliseconds How far behind the primary the secondary is Up to 10 seconds, even when caught up Sustained above your RPO budget (for example, > 30,000 ms for 5 minutes)
DisasterRecoveryReplicationLagOps Count Changes the secondary hasn’t applied yet A few seconds under active writes, workload dependent. Growing continuously over several periods
DisasterRecoveryBootstrappingIndices Count Indices still doing their first full copy 0 (non-zero right after setup or new index creation) Non-zero longer than your expected initial sync time
DisasterRecoveryDegradedIndices Count Indices replicating but not fully healthy 0 > 0 for 15 minutes
DisasterRecoveryFailedIndices Count Indices where replication stopped with an error and no longer receive changes 0 > 0 (any datapoint)

For more information, see Monitoring Amazon OpenSearch Service domains.

Switch Regions with switchover or failover

Use switchover for planned events and failover when the primary is unavailable (see the comparison table in Solution overview). In both cases, call the API on the secondary domain that you’re promoting.

Perform a planned switchover

SwitchoverReplication performs a coordinated role swap with zero data loss. Run it on the secondary domain:

aws opensearch switchover-replication --region <secondary_region> \
    --domain-name <secondary_domain> \
    --peer-domain-arn arn:aws:es:<primary_region>:<account>:domain/<primary_domain>

Perform an unplanned failover

FailoverReplication promotes the secondary to primary unilaterally. Data written to the old primary but not yet replicated is lost. The loss is bounded by the replication lag. Because of this, the command requires the --allow-data-loss flag:

aws opensearch failover-replication --region <secondary_region> \
    --domain-name <secondary_domain> \
    --peer-domain-arn arn:aws:es:<primary_region>:<account>:domain/<primary_domain> \
    --allow-data-loss

Redirect client writes

Managed disaster recovery doesn’t redirect your applications. Both domains stay readable, but only the primary is writable, so after a switchover or failover your applications must send writes to the newly promoted primary. Make the write-path update part of the same runbook step as the API call, for example by updating a DNS record, a configuration parameter, or a service discovery entry.

After a switchover or failover

After a switchover, the former primary becomes the secondary and replication continues in the reverse direction with no further action.

After a failover, when the original domain is available again, you can fail back. The original domain needs to be re-seeded before it can be used again for replication. After the re-seed, you can switch over to restore the original primary Region.

Performance results

We measured setup, initial sync, steady-state replication lag, planned switchover, and unplanned failover across multiple r7g instance sizes and data footprints. The tests used synthetic Apache log data with a sustained write load of 240 MBps.

Domain details:

  • R7g.4xlarge domain: 6 data nodes + 3× m7g.large cluster managers, 12 TiB gp3 per node.
  • R7g.8xlarge domain: 6 data nodes + 3× m7g.large cluster managers, 12 TiB gp3 per node.
  • Indices use 12 primary shards + 1 replica, rollover at 50 GB per primary shard.

Note: Results can vary by workload, data size, and configuration.

Setup and initial sync

Pairing is one API call and completed in 1 to 3 minutes in our tests, from 1 TiB to 10 TiB. Initial sync time depends on data size.

Data Instance Turn on disaster recovery Full initial sync
1 TiB r7g.4xlarge ~1 min 20min
10 TiB r7g.4xlarge ~1 min 2 hr 31 min
10 TiB r7g.8xlarge ~1 min 2 hr 18 min

Steady-state replication lag

Replication lag is how far behind the secondary is under sustained write load, measured with DisasterRecoveryReplicationLagTime (p95 over the test window).

Data Instance Write load Replication lag (p95)
1 TiB r7g.4xlarge 240 MBps ~10 s
10 TiB r7g.4xlarge 240 MBps ~10 s
10 TiB r7g.8xlarge 240 MBps ~10 s

Planned switchover

Pending writes replicate to the new primary before the role swap completes, so data loss is zero. RTO is how long clients see writes fail before the new primary accepts them.

Data Instance Write load Writes down (RTO) Data loss
1 TiB r7g.4xlarge 240 MBps 36.8 s 0
10 TiB r7g.4xlarge 240 MBps 38 s 0
10 TiB r7g.8xlarge 240 MBps 57 s 0

Unplanned failover

RPO is how many seconds of writes were lost. Role flip confirmed is how long until the control plane confirms the promotion. Writes down is how long until clients receive HTTP 201 from the new primary.

Data Instance Write load Role flip confirmed Writes down (RTO) Data loss (RPO)
1 TiB r7g.4xlarge 240 MBps 6.4 s 69 s ~8 s (~2.0M docs)
10 TiB r7g.4xlarge 240 MBps 8.1 s 61 s ~10 s (~2.46M docs)
10 TiB r7g.8xlarge 240 MBps 5.8 s 60 s ~10 s (~2.46M docs)

Key takeaways

In our tests, we saw the following results:

  • RPO: Unplanned failover lost about 8 to 10 seconds of writes, well within the under-1-minute RPO target. Planned switchover lost no data.
  • RTO: Writes resumed in about 1 minute after failover and under 1 minute after switchover, well within the under-20-minute RTO target.
  • Scale: Pairing time, lag, and recovery times stayed roughly constant from 1 TiB to 10 TiB. Only the one-time initial sync grows with data size.

Best practices

  • Manage configuration with infrastructure as code – Keep index templates, ISM policies, settings, and roles in a Git repository and apply them to both domains, for example with the OpenSearch Terraform provider.
  • Upgrade the secondary first – When upgrading engine versions, upgrade the secondary domain first and the active primary last. We recommend testing the performance of the new version on the secondary, then upgrading the primary after the secondary domain upgrade is validated. Don’t run mixed versions permanently.
  • Size domains to match – Keep instance types, counts, storage, and shard configuration aligned, and use CompareDomainConfiguration to detect drift.
  • Decide your failover automation – Failover is manual by design, because the right trigger differs by business. You can build event-driven automation using CloudWatch alarms and the failover and switchover APIs.
  • Exercise your runbook – Use switchover regularly to rehearse Region moves, including the client write redirection step.

Pricing

There is no additional charge for managed disaster recovery. You pay for the secondary domain’s instances and for cross-Region data transfer. For instance pricing, see Amazon OpenSearch Service pricing.

Conclusion and next steps

Managed disaster recovery gives Amazon OpenSearch Service domains continuous cross-Region replication, a zero-data-loss planned switchover, and a bounded-RPO failover, without building and operating your own replication pipeline. It serves regulated industries and business-critical workloads with strict availability, recovery, and business continuity requirements.

To get started:

  1. Create or upgrade domains to OpenSearch 3.7 or later in two Regions in the same AWS account.
  2. Pair them on the console or with the AddReplicationPeer API.
  3. Set CloudWatch alarms on the replication metrics, and build and rehearse your switchover and failover runbook.

For more information, see the Amazon OpenSearch Service disaster recovery documentation and the API reference, open the OpenSearch Service console, or view a demo video.


About the authors

Cédric Pelvet

Cédric Pelvet

Cédric is a Principal AWS Specialist Solutions Architect. He helps customers design scalable solutions for real-time data and search workloads. In his free time, his activities are learning new languages and practicing the violin.

Michael Torio

Michael Torio

Michael is an OpenSearch Specialist Solutions Architect based in Mountain View, CA. He works with customers across every stage of their data analytics journey, helping them design, scale, and operate OpenSearch in production.

The collective thoughts of the interwebz