How to migrate from Amazon CloudSearch to Amazon OpenSearch Serverless

2026-09-10 Prasad Nadig

Post Syndicated from Prasad Nadig original https://aws.amazon.com/blogs/big-data/how-to-migrate-from-amazon-cloudsearch-to-amazon-opensearch-serverless/

If you run search on Amazon CloudSearch, now is the time to plan your migration to Amazon OpenSearch Serverless. Modern search has moved on to capabilities beyond what CloudSearch provides: semantic and hybrid search, Retrieval Augmented Generation (RAG), and agentic search. OpenSearch Serverless gives you all of these with automatic scaling on a pay-for-what-you-use basis. You don’t need to choose or maintain infrastructure. OpenSearch Serverless maintains the hands-off, operational simplicity of CloudSearch.

This post shows you how to migrate your CloudSearch domain to an Amazon OpenSearch Serverless collection. We walk you through assessing your CloudSearch configuration, creating an OpenSearch Serverless collection with explicit index mappings, converting your documents and queries, configuring security policies, loading your data with Amazon OpenSearch Ingestion, and validating the migration before cutting over.

Key differences to note

Independent scaling – OpenSearch Serverless scales indexing and search compute independently and can scale compute to zero when a collection is idle (you still pay for storage). For the cost structure, see Managing capacity limits for Amazon OpenSearch Serverless.
Layered security model – OpenSearch Serverless applies encryption, network, and data access policies at separate layers. For the security model, see Security in Amazon OpenSearch Serverless.

Prerequisites

To follow along with this post, you need the following:

An AWS account.
An existing Amazon CloudSearch domain with indexed data.
Source data available in a durable store such as Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB (CloudSearch doesn’t provide a built-in export or backup feature, so your original source data is required to re-ingest into OpenSearch).
AWS Identity and Access Management (IAM) permissions to create and manage Amazon OpenSearch Serverless collections, encryption policies, network policies, and data access policies.
An Amazon OpenSearch Ingestion pipeline (or alternative ingestion method) for loading data.

Plan the migration

Planning is where you decide what success means: minimal downtime, no data loss, current functionality preserved, and custom configurations carried over. You don’t need to plan for infrastructure because OpenSearch Serverless provisions and scales compute for you. Your main planning task is to assess your current CloudSearch configuration so you can reproduce its behavior on the target.

Document your existing setup from the Amazon CloudSearch console. Record the current instance type, the partition count, and the replication count. Capture the total document count and overall data size, and record every field definition, including field types and the search, facet, and sort settings for each field. Note any analyzers, synonyms, stopwords, or custom rank expressions. Note whether you use the 2011 or the 2013 CloudSearch API version, because the 2013 API added faceting and filtering features that change how you model the target.

OpenSearch Serverless is the right target for most CloudSearch workloads, but not all of them. If your workload needs very low read-after-write latency (a short refresh interval), tight and predictable query response times, or direct control over instance configuration, choose an Amazon OpenSearch Service managed clusters deployment instead and size it from your workload profile.

The migration involves four main concerns: your source data format, your queries, your field definitions, and your access policies. Before you plan the details, it helps to see the whole migration at once. The following diagram maps the migration across four phases: your source CloudSearch environment, the migration pipeline that converts and moves your data, the OpenSearch Serverless target, and cutover and operations.

Migration workflow across four phases: source CloudSearch, migration pipeline, OpenSearch Serverless target, and cutover and operations

Figure 1: The migration workflow across four phases

In the source environment, you assess your CloudSearch configuration and back up your source data (Amazon S3, Amazon DynamoDB, or another store). Note the Source Data Format (SDF), the URL-based query syntax, and the IAM access policies you need to carry over. In the migration pipeline, you map field types, convert the data format from CloudSearch JSON to OpenSearch-compatible JSON, convert your queries to the OpenSearch query domain-specific language (DSL), configure security, bulk-ingest the data, and validate the result. The OpenSearch Serverless target holds the collection, index mappings, ingested documents, and the encryption, network, and data access policies, and it scales with your workload on a pay-per-use basis. In cutover and operations, you update your application to the new endpoint and clients, monitor with Amazon CloudWatch, and decommission CloudSearch once no traffic remains.

Model your data in OpenSearch Service

OpenSearch Service uses index mappings to define the fields and data types in an index. Because you know your CloudSearch schema, define the target mapping explicitly when you create the index. Create the index and set its mapping in a single request, and set dynamic to strict so OpenSearch rejects any document that contains a field you did not define. Strict mapping catches schema drift at ingest time, avoiding the default OpenSearch behavior of creating new mappings for undefined fields.

PUT /imdb_movies
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "title": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword" }
        }
      },
      "genres": { "type": "keyword" },
      "rating": { "type": "float" },
      "release_date": { "type": "date" }
      ...
    }
  }
}

Field type mapping

The following table maps CloudSearch field types to their OpenSearch Service equivalents.

CloudSearch	OpenSearch Service equivalent	Notes
text	text	Text is tokenized. Stemming, synonyms, and stopwords apply. Good for matching user terms.
literal	keyword	Not tokenized. Good for exact-match search.
int	integer	Use for ranking, faceting, and narrowing.
double	float or double	.
date	date	.
boolean	boolean	.
latlon	geo_point	.
text-array	text	OpenSearch handles arrays natively, so map to the base text type.
literal-array	keyword	OpenSearch handles arrays natively, so map to the base keyword type.
multi-value	nested or object	.
long	long	.
binary	binary	.

Two mapping details deserve attention. First, pick the smallest numeric type that fits your data rather than copying the widths CloudSearch uses. CloudSearch stores integers as 64-bit values, but few datasets hold numbers that large. A long or a double consumes more disk than an integer, a short, or a float with no benefit when the values are small. Evaluate the actual range of each field and choose the narrowest type that holds it. Reserve long for values that genuinely exceed the roughly 2.1 billion ceiling of integer, and use float instead of double unless you need double precision. Smaller types shrink your index and speed up queries.

Second, if you sort or aggregate on a text field, add a keyword sub-field. The preceding example mapping has a keyword subfield for the title field. You access the field using dot notation: title.keyword. OpenSearch doesn’t sort or aggregate analyzed text fields by default.

As noted earlier, if you run several CloudSearch domains, model each one as a separate index within a single OpenSearch Serverless collection to consolidate them.

Move your data

Migrating to OpenSearch Service is a re-ingestion: you convert your source documents and index them into the collection you created. CloudSearch doesn’t provide a built-in backup or snapshot feature. It relies on the documents you send through the indexing process, so before you migrate, make sure your source data is available in a durable store such as Amazon S3, Amazon DynamoDB, or another database.

The conversion is a format translation. CloudSearch accepts data in SDF as JSON or XML, where a document batch is a collection of add and delete operations. The JSON that CloudSearch uses differs from the JSON that OpenSearch Service expects, so you must transform each source document into an OpenSearch document whose fields match the index mapping you defined earlier. Handle the same details the mapping calls out: emit each numeric value so it fits the narrow type you chose for its field rather than a wide long or double, format dates to match your date mapping, and drop or rename any field that your strict mapping doesn’t define.

CloudSearch batch format showing add and delete operations in JSON

OpenSearch bulk batch format showing index operations in JSON

Figure 2: CloudSearch batch format (left) compared to OpenSearch batch format (right)

You can write a small conversion script. Have the script write its output to an Amazon S3 bucket so the converted documents live in a durable store you can re-ingest from as many times as you need.

With your converted documents in Amazon S3, use Amazon OpenSearch Ingestion to load them. Amazon OpenSearch Ingestion is a feature of Amazon OpenSearch Service that you can use to ingest, filter, transform, enrich, and route data to an Amazon OpenSearch Service domain or an OpenSearch Serverless collection. Configure an OpenSearch Ingestion pipeline with an Amazon S3 source (you can use an OpenSearch Ingestion blueprint to get started) that reads your converted documents. Let its built-in processors apply any final transformation before the pipeline writes to your collection. A managed pipeline reading from Amazon S3 gives you a repeatable, restartable load without operating ingestion infrastructure, which makes it the recommended path for most migrations.

If you prefer to load data directly, OpenSearch Service exposes a REST API, so you can index documents with a standard client such as curl or with the OpenSearch client libraries for many languages. Direct indexing is convenient for a small dataset or a quick test, but an Amazon S3 source with OpenSearch Ingestion is the better choice for a production migration.

Convert your queries

CloudSearch uses a URL-based query format. You pass a query parameter in the URL and submit either a simple string search or a JSON-formatted query. OpenSearch Service uses a REST API and the OpenSearch query DSL in the request body, which gives you compound queries, function scoring, and richer relevance control. You can use generative AI coding assistants to help with this translation. Provide your CloudSearch query patterns, and the model generates the equivalent OpenSearch query DSL, which you then validate against your test cases.

Query syntax changes

CloudSearch appends parameters such as sort to the query URL, while OpenSearch expresses sorting, filtering, and boosting as explicit elements of the request body. For example, a title search for “shakespeare” in CloudSearch looks like the following.

https://my-cloudsearch-domain.us-east-1.cloudsearch.amazonaws.com/2013-01-01/search?q=shakespeare&size=10

The equivalent query in OpenSearch Service uses the query DSL.

GET /imdb_movies/_search
{
  "query": {
    "match": { "title": "shakespeare" }
  }
}

To keep result sets consistent after migration, set the default operator to AND in OpenSearch to match the default query behavior of CloudSearch. The following table shows common CloudSearch query patterns and their OpenSearch Service equivalents, using a sample IMDB movies dataset.

Query type	CloudSearch (Lucene syntax)	OpenSearch Service query DSL
Compound AND	`title:"Inception" AND genres:"Sci-Fi"`	`{"query":{"bool":{"must":[{"match":{"title":"Inception"}},{"match":{"genres":"Sci-Fi"}}]}}}`
Compound NOT	`title:"Star Wars" AND NOT genres:"Comedy"`	`{"query":{"bool":{"must":[{"match":{"title":"Star Wars"}}],"must_not":[{"match":{"genres":"Comedy"}}]}}}`
Wildcard	`title:Batman*`	`{"query":{"wildcard":{"title":{"value":"batman*"}}}}`
Numeric range	`rating:[7 TO 9]`	`{"query":{"range":{"rating":{"gte":7,"lte":9}}}}`
Date range (after)	`release_date:[2015-01-01T00:00:00Z TO *]`	`{"query":{"range":{"release_date":{"gte":"2015-01-01T00:00:00Z"}}}}`
Boosting	`title:"The Matrix"^6 OR genres:"Sci-Fi"^4`	`{"query":{"bool":{"should":[{"query_string":{"query":"title": \"The Matrix\"^6","fields":["title"]}},{"query_string":{"query":"genres:\"Sci-Fi\"^4","fields":["genres"]}}]}}}`
Sorting	`title:"Batman" sort=release_date desc`	`{"query":{"match":{"title":"Batman"}},"sort":[{"release_date":{"order":"desc"}}]}`

Sorting and boosting

Boosting is useful when you want certain fields or terms to carry more weight in relevance scoring. A higher boost value means the term contributes more to the score. OpenSearch also supports sorting by _score (relevance), which is the default when you specify no sort. For the full query language, see the OpenSearch query DSL documentation.

Configure security

CloudSearch uses AWS Identity and Access Management policies to control access to its configuration and domain service APIs. You attach user-based policies to an IAM role, user, or group, and the document, search, and suggest actions in those policies control access to the CloudSearch APIs.

OpenSearch Serverless applies security through policies at several layers.

Collections: Encrypted at rest by default, using either an AWS owned key or a customer managed key defined in an encryption policy.
Network policies: Define whether a collection is reachable privately through a virtual private cloud (VPC) endpoint or over the internet.
Data access policies: Control which IAM principals and Security Assertion Markup Language (SAML) identities can create indexes and read or write data in the collection.

Amazon OpenSearch Service provisioned domains also offer fine-grained access control, with role-based access control and security at the index, document, and field level. For OpenSearch Serverless, data access policies provide collection-level and index-level permissions, controlling which IAM principals and SAML identities can create, read, or write data within a collection.

Validate the migration

Validation confirms that the migration is complete and correct before you send production traffic to OpenSearch Serverless. Work through five kinds of validation.

Documents: Check your document count. Your OpenSearch Serverless indexes should have the same count as your CloudSearch indexes.
Queries: Translate your most important queries and run them manually against your collection. Spot check the output for the presence of important results.
Ranking: Check the order of results, especially for queries with custom rank functions or field weighting. Results might not match exactly, so look for anything that’s incorrect.
Latency: Ideally you should tee your production traffic to your Serverless collection to get real latency metrics. Worst case, generate at least 100,000 synthetic queries across all your query types and run them. Monitor OpenSearch Compute Unit (OCU) consumption with Amazon CloudWatch to understand your cost profile.

To validate search functionality, run the same query against both systems and compare the results. Reuse the query pairs from the conversion step so you exercise the syntax differences directly. For example, to check a numeric range against the sample IMDB movies dataset, run the following query in CloudSearch.

https://my-cloudsearch-domain.us-east-1.cloudsearch.amazonaws.com/2013-01-01/search?q=rating: [7 TO 9]&size=10

Run the equivalent query DSL against your OpenSearch Serverless collection.

GET /imdb_movies/_search
{
  "query": {
    "range": { "rating": { "gte": 7, "lte": 9 } }
  }
}

Confirm that both queries return the same set of movies. Then repeat the comparison for a query that exercises relevance, such as the boosted query from the conversion step, and confirm the top results appear in the same order.

GET /imdb_movies/_search
{
  "query": {
    "bool": {
      "should": [
        { "match": { "title": { "query": "The Matrix", "boost": 6 } } },
        { "match": { "genres": { "query": "Sci-Fi", "boost": 4 } } }
      ]
    }
  }
}

Cut over and operate

When validation passes, update your application to use the OpenSearch Serverless endpoint and the query DSL, and switch from the CloudSearch SDK to the OpenSearch client libraries. After cutover, confirm that no application still points to a CloudSearch endpoint, retain your source data backups in Amazon S3 for rollback, and then delete the CloudSearch domain.

Operating OpenSearch Serverless in production is lighter than operating a domain, because OpenSearch Serverless scales compute for you and you do not tune shards, instance types, or capacity. Your focus shifts to cost and search quality. Monitor OCU consumption and search latency with Amazon CloudWatch, and set alarms on the thresholds that matter to you. Review OCU usage patterns to understand cost and find optimization opportunities, and set capacity limits on the collection to cap the maximum OCUs it can consume. For guidance, see Managing capacity limits for Amazon OpenSearch Serverless and Monitoring Amazon OpenSearch Serverless.

Cost considerations

With OpenSearch Serverless, you pay only for the compute and storage your workload consumes, and OpenSearch Serverless charges for compute and storage separately. OpenSearch Serverless scales indexing compute and search compute independently, so a write-heavy or a read-heavy workload scales only the dimension it needs, and compute can scale to zero when a collection is idle, in which case you pay only for storage. To share hardware across workloads, place collections in a collection group so they draw from the same compute rather than each provisioning its own. For pricing and unit details, see Amazon OpenSearch Service pricing.

Clean up

Because you’re migrating to OpenSearch Serverless, the resources that you’ve created will likely become your production resources. If not, delete any OpenSearch Serverless collections and S3 buckets you created to avoid incurring ongoing cost.

Conclusion

In this post, you saw how Amazon CloudSearch and Amazon OpenSearch Serverless compare, and how the concepts you rely on in CloudSearch (field types, query syntax, autoscaling, and access control) translate into OpenSearch Service. You assess your CloudSearch configuration, model your data with explicit OpenSearch mappings, move your converted documents into the collection with OpenSearch Ingestion, convert your URL-based queries into the OpenSearch query DSL, configure security, and validate before cutover. OpenSearch Serverless gives you the hands-off operational model you have with CloudSearch, and adds richer query capabilities, granular data access policies, and automatic scaling. To get started, create an OpenSearch Serverless collection on the AWS Management Console and follow the steps in this post.

To learn more, see the following resources:

About the authors

Accelerating Spark queries with Iceberg materialized views

2026-09-10 Yuzhou Sun

Post Syndicated from Yuzhou Sun original https://aws.amazon.com/blogs/big-data/accelerating-spark-queries-with-iceberg-materialized-views/

In this post, you learn how to reduce Apache Spark query execution time with Apache Iceberg materialized views without changing a single SQL query.

Organizations running analytical workloads on their data lakes often hit a common wall: queries that are slow and costly, yet difficult to rewrite by hand. Multi-table joins, heavy aggregations, and window functions over large fact tables all drive up execution times, but the SQL behind them often can’t be changed. It might come from business intelligence (BI) dashboards, packaged independent software vendor (ISV) applications, or legacy reports, where editing the source introduces regression risk that outweighs the performance gain.

Starting with Amazon EMR 7.12.0 and AWS Glue 5.1, you can accelerate these queries without rewriting them. Automatic query rewrite analyzes the logical plan of each incoming query and compares it against a metadata cache of available MVs. When the optimizer finds a materialized view (MV) that satisfies all or part of a query, it rewrites the plan to read from that MV instead of the base tables. Matches can be structural (aggregations and joins) or exact (more complex patterns like window functions). If no MV matches, the original query runs unchanged with no impact on correctness.

If you have previously tried to speed up slow analytical queries, you might have considered one of the following alternatives. Here is how automatic query rewrite compares:

Query modification approach	Stored results	Refreshes	Modification to existing queries
Standard views in AWS Glue	No (re-runs each time)	n/a	Required
Custom ETL pipeline	Yes	Manual	Required
Hand-rolled rewrite	Yes	Manual	Required
Materialized views with automatic rewrite enabled	Yes	Automatically through AWS Glue Data Catalog on a schedule when configured	Not required when supported

In this post, we:

Give a high-level overview of how automatic query rewrite works in Apache Spark.
Walk through a concrete example, showing how the same query can benefit from MVs at different levels of coverage.
Discuss the trade-offs so you can choose the right MV shape for your workload.

Prerequisites

To use automatic query rewrite with Iceberg materialized views, you need:

Amazon EMR release 7.12.0 or later, or AWS Glue 5.1 or later.
Source tables in Apache Iceberg or Parquet format, registered in the AWS Glue Data Catalog, in the same AWS Region and account as the materialized view. Parquet source tables are supported for automatic query rewrite starting with Amazon EMR 7.14.0 and AWS Glue 8.1.
An Amazon Simple Storage Service (Amazon S3) Tables (a capability of Amazon S3) bucket, or an S3 general purpose bucket, for the materialized view data.
Permissions for the definer role. You can use AWS Identity and Access Management (IAM) policies or AWS Lake Formation.
Automatic query rewrite turned on in your Spark session: --conf spark.sql.optimizer.answerQueriesWithMVs.enabled=true.
For Parquet source tables, set spark.sql.materializedView.v1SourceTables.enabled=true and spark.sql.materializedView.v1ETagVersioning.enabled=true.

For more Spark configurations, see Introducing Apache Iceberg materialized views in AWS Glue Data Catalog.

How it works

Here is how MVs and automatic query rewrite work together:

You define a SQL query with aggregations, joins, or filters across your supported source tables.
AWS Glue Data Catalog stores the precomputed results as an Apache Iceberg table in your Amazon S3 bucket. You can store it in a general purpose S3 bucket or in Amazon S3 Tables. Any Apache Iceberg-compatible query engine can read the materialized view, including Amazon Athena, Amazon EMR, AWS Glue, Amazon Redshift, and Iceberg-compatible third-party query engines. Automatic query rewrite is available on the AWS optimized Spark runtime in Amazon Athena, Amazon EMR, and AWS Glue. Other engines can query the materialized view directly, but they don’t rewrite queries to use it automatically.
Automatic refresh keeps the MV current on a schedule that you define, for example SCHEDULE REFRESH EVERY 1 DAY. You set it at creation time or later with ALTER MATERIALIZED VIEW ... ADD SCHEDULE REFRESH. At that scheduled time, the refresh process checks the current Apache Iceberg snapshot ID or Parquet file ETags and refreshes the MV when it detects source-table changes.
Automatic query rewrite redirects matching queries to the MV at query optimization time. Automatic query rewrite in Apache Spark uses two matching strategies:
- Structural rewrite (adapted from Amazon Redshift) handles an MV defined as a single SELECT-FROM-WHERE-GROUP-BY block over INNER joins. The optimizer can roll up an MV’s aggregates to a coarser grain and pull extra query predicates up onto the MV scan.
- Exact-match rewrite handles MVs defined as other shapes, such as window functions and outer joins, by matching a canonicalized form of the MV body against subtrees of the query plan.

When the optimizer evaluates a query, it consults a metadata cache of MVs from the configured catalogs and chooses the best match. It also checks MV staleness during optimization. It skips stale MVs, so rewrite won’t return stale results. If no MV matches, the original query runs unchanged.

Note that automatic query rewrite is opt-in: set spark.sql.optimizer.answerQueriesWithMVs.enabled=true when creating the Apache Spark session.

Example: One query with three potential MVs

An MV doesn’t need to cover an entire query to help it. Automatic query rewrite in Apache Spark operates on subtrees: when an MV matches a portion of your query plan, the rewriter substitutes that subtree and lets the rest of the query run on the rewrite output unchanged. The same query can therefore be served by many possible MV designs, each making a different trade-off between per-query speedup, storage cost, and reuse across other queries.

To make this concrete, consider a typical analytics query: “Top 100 preferred US customers by total store spending.” It joins fact and dimension tables, applies two selective filters on the customer dimension, aggregates per customer, ranks the result with a window function, and keeps only the top 100:

SELECT c_customer_id, total_revenue, num_transactions, avg_purchase, revenue_rank
FROM (
    SELECT cust.c_customer_id,
        SUM(sales.ss_quantity * sales.ss_sales_price) AS total_revenue,
        COUNT(*) AS num_transactions,
        AVG(sales.ss_quantity * sales.ss_sales_price) AS avg_purchase,
        RANK() OVER (ORDER BY SUM(sales.ss_quantity * sales.ss_sales_price) DESC) AS revenue_rank
    FROM base_catalog.base_db.store_sales sales
    INNER JOIN base_catalog.base_db.customer cust
        ON sales.ss_customer_sk = cust.c_customer_sk
    WHERE cust.c_birth_country = 'UNITED STATES'
        AND cust.c_preferred_cust_flag = 'Y'
    GROUP BY cust.c_customer_id
) ranked
WHERE revenue_rank <= 100
ORDER BY revenue_rank;

Query 1: The original query. Top 100 preferred US customers by total store spending, before any materialized view.

Three MV designs cover progressively more of this query, from a single-table pre-aggregate to the full query body itself:

Tier 1: Pre-aggregate store_sales only, no join, no filter. This tier is a single-table aggregate of store_sales at customer-surrogate-key grain. The query still must join the customer table, apply both filters, re-aggregate at c_customer_id grain, and run the window function.

CREATE MATERIALIZED VIEW mv_catalog.mv_db.customer_tier_1 AS
SELECT ss_customer_sk,
    SUM(ss_quantity * ss_sales_price) AS sum_revenue,
    COUNT(ss_quantity * ss_sales_price) AS count_revenue,
    COUNT(*) AS num
FROM base_catalog.base_db.store_sales
GROUP BY ss_customer_sk;

Tier 1 MV: Single-table pre-aggregate of store_sales by customer surrogate key (no join, no filter).

The following plans compare the original query plan to the rewritten plan:

Window, filter, Sort
+- Aggregate by c_customer_id
:  total_revenue = SUM(ss_quantity * ss_sales_price)
:  num_transactions = COUNT(*)
:  avg_purchase = AVG(ss_quantity * ss_sales_price)
+- Project
   +- Join Inner ON ss_customer_sk = c_customer_sk
      :- BatchScan store_sales <- reads the large store_sales table
      +- Filter c_birth_country='UNITED STATES' AND c_preferred_cust_flag='Y'
         +- BatchScan customer

Plan 1: Original plan. Scans the large store_sales table.

Window, filter, Sort
+- Aggregate by c_customer_id <- rolls up pre-aggregated sums
:  total_revenue = SUM(sum_revenue) <- sum of sum_revenue
:  num_transactions = SUM(num) <- sum of num
:  avg_purchase = SUM(sum_revenue) / SUM(count_revenue) <- sum of sum_revenue / sum of count_revenue
+- Project
   +- Join Inner ON ss_customer_sk = c_customer_sk
      :- BatchScan customer_tier_1 <- reads pre-aggregated MV
      +- Filter c_birth_country='UNITED STATES' AND c_preferred_cust_flag='Y'
         +- BatchScan customer

Plan 2: Rewritten plan (Tier 1). Reads the pre-aggregated customer_tier_1 MV.

Tier 2: Pre-join store_sales x customer, pre-apply one filter (c_preferred_cust_flag = ‘Y’). The middle tier pre-joins both tables and bakes in the preferred-customer filter. The query still must apply the country filter as a residual on the MV scan and run the RANK() window.

CREATE MATERIALIZED VIEW mv_catalog.mv_db.customer_tier_2 AS
SELECT cust.c_customer_id, cust.c_birth_country,
    SUM(sales.ss_quantity * sales.ss_sales_price) AS sum_revenue,
    COUNT(sales.ss_quantity * sales.ss_sales_price) AS count_revenue,
    COUNT(*) AS num
FROM base_catalog.base_db.store_sales sales
INNER JOIN base_catalog.base_db.customer cust
    ON sales.ss_customer_sk = cust.c_customer_sk
WHERE cust.c_preferred_cust_flag = 'Y'
GROUP BY cust.c_customer_id, cust.c_birth_country;

Tier 2 MV: Pre-joins store_sales and customer, with the preferred-customer filter applied.

Rewritten query plan:

Window, filter, Sort
+- Aggregate by c_customer_id <- rolls up pre-aggregated sums
:  total_revenue = SUM(sum_revenue) <- sum of sum_revenue
:  num_transactions = SUM(num) <- sum of num
:  avg_purchase = SUM(sum_revenue) / SUM(count_revenue) <- reads pre-aggregated MV
+- Filter c_birth_country='UNITED STATES' [residual filter on MV scan]
   +- BatchScan customer_tier_2 <- reads pre-aggregated MV

Plan 3: Rewritten plan (Tier 2). Country filter applied as a residual on the MV scan.

Tier 3: Match the entire query, including the window function and top N filter. This is the most specific tier. The MV body is the target query verbatim (minus the top-level ORDER BY, which is meaningless for a stored set). The MV stores the top-ranked rows the query asks for (rank ≤ 100).

CREATE MATERIALIZED VIEW mv_catalog.mv_db.customer_tier_3 AS
SELECT c_customer_id, total_revenue, num_transactions, avg_purchase, revenue_rank
FROM (
    SELECT cust.c_customer_id,
        SUM(sales.ss_quantity * sales.ss_sales_price) AS total_revenue,
        COUNT(*) AS num_transactions,
        AVG(sales.ss_quantity * sales.ss_sales_price) AS avg_purchase,
        RANK() OVER (ORDER BY SUM(sales.ss_quantity * sales.ss_sales_price) DESC) AS revenue_rank
    FROM base_catalog.base_db.store_sales sales
    INNER JOIN base_catalog.base_db.customer cust
        ON sales.ss_customer_sk = cust.c_customer_sk
    WHERE cust.c_birth_country = 'UNITED STATES'
        AND cust.c_preferred_cust_flag = 'Y'
    GROUP BY cust.c_customer_id
) ranked
WHERE revenue_rank <= 100;

Tier 3 MV: Stores the exact ranked output of the query (exact-match path).

This tier exercises the exact-match rewrite path: the rewriter canonicalizes the MV body and matches it against the query’s logical plan.

Rewritten plan:

Sort revenue_rank ASC
+- BatchScan customer_tier_3 <- reads around 100 stored rows

Plan 4: Rewritten plan (Tier 3). Reads around 100 stored rows.

The trade-off

The three tiers trade per-query speedup against reuse and storage. In our testing on TPC-DS 3 TB, we observed the following:

MV design	Pre-computed	Reuse	Per-query speedup	MV size
Baseline (no MV)	nothing	n/a	1x	n/a
Tier 1: store_sales agg by customer surrogate key	aggregate of all sales per customer	broadest: any per-customer aggregation	~5x faster	0.07% of store_sales for TPC-DS 3 TB
Tier 2: store_sales x customer agg, one filter pre-applied	join + aggregate, preferred customers only	medium: any country filter, preferred customers	~10x faster	0.04% of store_sales for TPC-DS 3 TB
Tier 3: entire query body verbatim (exact-match)	exact ranked output of this query	narrowest: only this exact query shape	20x+ faster	negligible (only 100 rows)

Performance measured on TPC-DS 3 TB. Speedup is the ratio of baseline execution time to MV-accelerated execution time. Results might vary based on data characteristics, cluster size, and query complexity.

In addition, MVs incur additional cost. Each one runs a query against your source tables once and stores the result. The more pre-computation it does (joining more tables, applying more filters), the more time it takes.

The following chart plots per-query speedup and creation time for the three tiers in our testing on TPC-DS 3 TB. Per-query speedup rises steadily, from about 5x at Tier 1 to over 20x at Tier 3. Creation time doesn’t follow the same pattern: it peaks at Tier 2. Tier 2 pre-joins and aggregates all preferred customers across every country, so it materializes the most data work. Tier 3 applies both filters, so it processes far fewer rows and costs less to create.

Chart comparing three materialized view designs. In our testing with TPC-DS 3 TB, we observed per-query speedup rises from about 5x (Tier 1) to over 20x (Tier 3), while creation time peaks at Tier 2, which materializes the most data work. Stacked bars show creation time split into catalog setup, data work, and commit.

Figure 1: Per-query speedup and creation time across the three materialized view tiers, measured on TPC-DS 3 TB

Start by identifying one expensive query that runs repeatedly with stable filters. It is likely a good candidate for an exact-match MV.

Validating automatic query rewrite

To confirm that your query benefited from automatic rewrite:

Query plan inspection: Check the query’s optimized logical plan or physical plan for a leaf scan node referencing the MV (for example, BatchScan mv_catalog.mv_db.your_mv_name). If the MV appears as a scan source, rewrite succeeded.
Log confirmation (Amazon EMR 7.14.0+): Look for INFO-level log entries such as AQMV outcome: rewritten=true, mvs=[mv_name], duration=12ms.
No-rewrite diagnostics (Amazon EMR 7.14.0+): If rewrite didn’t occur, check the MVRewriteMetricsEvent in the Apache Spark Event Log for the specific reason the optimizer skipped the MV.

If you have set spark.sql.optimizer.answerQueriesWithMVs.enabled=true but your query still runs against the base tables, check the following common causes:

Write commands block rewrite by default. INSERT and MERGE statements don’t trigger rewrite. Set spark.sql.optimizer.answerQueriesWithMVs.commandBlockingEnabled=false to turn on rewrite within write command subqueries.
The MV is stale. Rewrite skips the MV when one or more source tables have changed since its last refresh. Wait for the next scheduled refresh, or force an immediate refresh with REFRESH MATERIALIZED VIEW <mv_name>.
Heuristic candidate filtering. The optimizer uses heuristic checks to narrow the set of MV candidates before attempting a full match. In some cases, an MV that could benefit the query might be filtered out early by these heuristics.
Spark version mismatch (Amazon EMR 7.13.0+). Automatic query rewrite skips MVs whose stored IMV_sparkVersion does not match the cluster’s current Apache Spark version. To bypass this check, set spark.sql.materializedView.sparkVersionCompatibilityCheck.enabled=false.
MV metadata cache not loaded. The metadata cache loads lazily during optimization of the first rewritable query in a Spark session. If your critical query fires before the cache is warm, the MV will not be available. Run a small warm-up query (for example, SELECT 1 FROM <some_iceberg_table>) at session start to pay this cost off the critical path.
MV metadata cache memory limit reached. If the cache was disabled or stopped loading MVs because of reaching its memory limit, increase spark.driver.memory.
Too many tables in configured catalogs. If there are many tables or MVs in the configured catalogs, the cache might not finish loading before your query starts. Place MVs in a dedicated catalog, add it to spark.sql.materializedViews.additionalCatalogs, and set spark.sql.materializedViews.scanCurrentCatalog=false to skip scanning the current catalog.
Parquet base tables have additional limitations and configuration requirements. For automatic query rewrite with Parquet base tables, set spark.sql.materializedView.v1SourceTables.enabled=true and spark.sql.materializedView.v1ETagVersioning.enabled=true. Without ETag versioning, Spark can’t determine a usable source-table version and skips the MV. Partitioned Parquet base tables are also subject to additional validation limits.

Performance considerations

Turning on automatic query rewrite has overhead: it introduces trade-offs that might affect some queries negatively:

Optimization overhead. Enabling rewrite adds processing time during query optimization as the optimizer evaluates MV candidates against the query plan. This overhead applies to every query in the session, including those that ultimately don’t match any MV.
Reduced task parallelism. Reading from an MV instead of the original base table might produce fewer tasks or introduce data skew, depending on the MV’s data layout. This reduces parallelism compared to a direct scan of the larger, more evenly distributed source table.

Conclusion

In this post, we showed how automatic query rewrite can accelerate your existing Apache Spark workloads. It uses Apache Iceberg materialized views in the AWS Glue Data Catalog, without changing a single line of SQL. By storing precomputed results as managed Apache Iceberg tables, the AWS Glue Data Catalog lets the Apache Spark optimizer transparently substitute matching query plans. You get the performance benefit of pre-aggregation without the application-level rewiring. BI dashboards, ISV-generated reports, and legacy pipelines all benefit the moment a matching MV exists.

We walked through three MV designs for the same analytical query, each striking a different balance between per-query speedup, storage footprint, and reuse across your workload. As the trade-off table shows, our testing found that a narrow, exact-match MV delivered 20x+ acceleration for a single query shape. A broader pre-aggregate served an entire family of queries at a more modest ~5x gain. The right choice depends on how many queries share the same join-and-aggregate pattern and how frequently your source data changes.

To get started:

Launch an Amazon EMR 7.12.0+ cluster or an AWS Glue 5.1+ job.
Create an MV over your most expensive repeating query using CREATE MATERIALIZED VIEW in the AWS Glue Data Catalog.
Turn on automatic query rewrite by setting spark.sql.optimizer.answerQueriesWithMVs.enabled=true in your Spark session configuration.
Verify the rewrite by inspecting the optimized query plan for an MV scan node, or by checking INFO-level logs on Amazon EMR 7.14.0+.

Queries with multi-table joins, heavy aggregations, or window functions over large fact tables are strong initial candidates. Start with one high-cost, frequently executed query. Validate the speedup, then expand to broader MVs as you identify shared patterns across your workload.

Special thanks to everyone who contributed to the automatic query rewrite feature and this blog: Andre Hernich, Leon Lin, Yiyang Chen, Geeta Krishna Panda, Ashok Chintalapati, Muhammad Malik, Rishabh Bhatia, and Giovanni Fumarola.

References

For more detail, see the following resources:

About the authors

Every team is a data team — bring Amazon Redshift analytics to ChatGPT Work

2026-09-10 Naresh Chainani

Post Syndicated from Naresh Chainani original https://aws.amazon.com/blogs/big-data/every-team-is-a-data-team-bring-amazon-redshift-analytics-to-chatgpt-work/

Today, AWS is announcing the AWS Data Analytics plugin for the new Data agent in ChatGPT Work. The plugin helps teams across an organization ask questions in natural language, analyze governed data across their Amazon Redshift data warehouse and data lakes, and create shareable dashboards. All this happens from a conversation in ChatGPT Work.

Tens of thousands of customers choose Amazon Redshift every day to run their most demanding workloads, because it delivers analytics at scale with industry-leading price performance. They love how Amazon Redshift provides access to their data warehouses and data lakes together in one place. Teams can combine curated business data with the broader operational, historical, and third-party data stored in open formats like Apache Iceberg in their data lakes. This gives them a complete picture to make business-critical decisions across their data.

Customers have asked AWS for a way to put that trusted data in the hands of more of their people. That means not only the analysts and engineers who write SQL, but also the sales leaders, operations managers, and finance teams who depend on the results. A sales leader wants to know how the customer pipeline has changed this quarter. An operations manager wants to understand why fulfillment times changed over the past month. That’s why we built the AWS Data Analytics plugin, bringing the power of Amazon Redshift and AWS analytics to ChatGPT Work.

“Business teams can make decisions faster when they can source their own analytics and build the dashboards they need. Our work with AWS gives more people that ability, helping them understand changes in performance and decide where to focus. The AWS Data Analytics plugin connects Amazon Redshift to the Data agent in ChatGPT Work, so employees can analyze trusted company data simply by asking, with their organization’s existing access controls in place.”

— Arpan Shah, General Manager, Technology at OpenAI

The new plugin helps shorten the path from question to decision for everyone. Using the Data agent in ChatGPT Work, employees can explore the data they are authorized to access in Amazon Redshift by asking questions in everyday language. They can then refine the analysis, investigate changes, and turn the results into a dashboard without leaving ChatGPT Work. The plugin works with both Amazon Redshift provisioned clusters and Serverless workgroups. Customers can integrate it into their existing multi-cluster or multi-workgroup environments and benefit from the cost and security controls they’ve already set up.

Consider Maya, a business analyst supporting a revenue operations team. She wants to understand the revenue performance across various segments and regions.

Maya starts by loading the AWS Data Analytics plugin in ChatGPT Work, and then asking:

What are the revenue metrics for the past 30 days compared to the previous 30-day period?

ChatGPT Work conversation asking for revenue metrics over the past 30 days compared to the previous 30-day period

Figure 1: Asking for revenue metrics in ChatGPT Work using the AWS Data Analytics plugin

The plugin translates her question into SQL, or a sequence of queries if needed, and runs them against the relevant data in Amazon Redshift. It returns key revenue performance metrics based on the same curated revenue data that her analytics team maintains.

Table of revenue performance metrics the plugin returned from Amazon Redshift

Figure 2: Revenue performance metrics returned from Amazon Redshift

Maya notices that gross margin is declining and asks a follow-up question:

What is my revenue breakdown by product category and region for the past 90 days?

Revenue results segmented by product category and region for the past 90 days in ChatGPT Work

Figure 3: Revenue breakdown by product category and region for the past 90 days

The plugin carries the context forward, segments the results, and helps Maya understand each segment’s performance for the past 90 days. She can inspect the analysis and ask additional questions to drill down even further to understand why certain regions are lagging or why certain segments are outperforming others.

This conversational workflow doesn’t replace the data models, metric definitions, or governance practices that the analytics team has established. It helps more employees use that data directly, giving analysts more time for high-value work.

The AWS Data Analytics plugin connects ChatGPT Work to Amazon Redshift and uses the context of the connected analytics environment to help answer questions with the Data agent. During a conversation, it can:

Discover the schemas, tables, columns, and data types available to the user.
Translate a natural-language question into Amazon Redshift SQL.
Run the query against the customer’s Amazon Redshift environment.
Present the results in a table or concise explanation.
Use follow-up questions to filter, compare, or drill into the results.
Turn an analysis into an interactive dashboard that teams can share and explore.

Because the analysis runs against the customer’s existing data, teams can continue to use the curated datasets and business definitions they already maintain in Amazon Redshift. Customers whose Amazon Redshift environments query data in both a warehouse and a data lake can also make that data available through the governed datasets exposed to the plugin. The AWS Data Analytics plugin also supports our broader AWS data and analytics services. This includes the ability to work with AWS Glue Data Catalog, Amazon S3 Tables (a capability of Amazon Simple Storage Service (Amazon S3)), Amazon Athena, and vector search on AWS.

Natural-language analytics requires more than passing a prompt to a database. The agent needs to understand SQL specific to Amazon Redshift, discover metadata, choose the right tables and columns, and construct queries that follow service best practices. The plugin was built using Amazon Redshift skills from the Agent Toolkit for AWS. These skills provide tested procedures and service-specific guidance that agents can use when working with Amazon Redshift.

To get started, install the AWS Data Analytics plugin in ChatGPT Work to connect it to Amazon Redshift. Give your teams a conversational path to governed insights across your data warehouse and data lake today.

To learn more, see the following resources:

About the author

I Tested 12 Soundbars Under $100: There’s a New Budget King

2026-09-10 The Hook Up

Post Syndicated from The Hook Up original https://www.youtube.com/watch?v=MCYAnapaUN4

Forever Young. Да поговорим за възхода на „Алтернатива за Германия“

2026-09-10 Светла Енчева

Post Syndicated from Светла Енчева original https://www.toest.bg/forever-young-da-pogovorim-za-vuzhoda-na-alternativa-za-germaniya/

Forever Young. Да поговорим за възхода на „Алтернатива за Германия“

В навечерието на изборите в германската федерална провинция Саксония-Анхалт на 6 септември 2026 г., спечелени от крайнодясната партия „Алтернатива за Германия“ (АзГ), един 42-годишен хит преживя ренесанс. Става въпрос за Forever Young на немската група Alphaville. Няма как да не сте чували тази песен, ако имате съзнателни спомени от 80-те, а е вероятно да ви говори нещо, дори да сте родени по-късно.

Макар привидно да възпява мечтата за вечна младост, песента всъщност е политическа – в нея става дума за превъоръжаването по време на Студената война:

Надяваме се на най-доброто,
но очакваме най-лошото,
ще пуснеш ли бомбата, или не?

Иронично от днешна гледна точка, в песента е цензуриран пасаж, в който става дума за фашизма.

Forever Young се превърна в своеобразен химн на съпротивата срещу АзГ „благодарение“ на недалновидността на организаторите на Фестивала на щастието (Glücksgefühle-Festival) в град Хокенхайм, Баден-Вюртемберг. Те оттеглиха поканата за участие към Alphaville с аргумента, че не искат политически послания на фестивала, а вокалът на групата Мариан Голд е известен с критичното си отношение към АзГ.

В Германия обаче правото на изразяване на демократични ценности се цени високо и за разлика от България, редовно се практикува. Логично, последва скандал. Организаторите се извиниха и „оттеглиха оттеглянето“ на поканата, но Alphaville вече не искаха да се включат.

За сметка на това редица участници изпълниха от сцената на фестивала Forever Young в знак на протест, като не пропуснаха да подчертаят важността на демокрацията. И/или да наругаят АзГ, както направи например мъжки хор от Кьолн.

Какво се случи в Саксония-Анхалт?

Хем беше очаквано, хем настана масова изненада. Така може да се обобщят реакциите по отношение на изборите в Саксония-Анхалт. АзГ получи подкрепата на близо 44% от гласувалите и едва три места я делят от пълно мнозинство в местния парламент. Избирателната активност беше най-високата в тази източногерманска провинция от 1990 г., тоест откакто в нея се провеждат свободни избори – 77,8%. Саксония-Анхалт е провинция с едва около два милиона души население, но отзвукът от резултатите е огромен.

Възходът на крайната десница в Германия – как и защо?

Тайна сбирка на дяснорадикални лидери, на която се е обсъждала „ремиграция“, стана причината за многохилядни протести в Германия. Защо дясната реторика и формации като „Алтернатива за Германия“ стават все по-силни? От Марина Лякова.

Какви са причините за този отзвук?

За първи път АзГ е толкова близо до вземането на властта, а участието ѝ в управлението би представлявало прекрачване на табу. Победата на крайнодясната формация, макар и в една малка провинция, е в контекста не само на повишаването на подкрепата за партията в цяла Германия. Тя става на фона на възхода на крайнодесните на други места в Европа – примерно, на Марин Льо Пен във Франция. Без да пропускаме немислимата до неотдавна симбиоза между Русия и доскорошния „лидер на демократичния свят“, където по време на втория мандат на Тръмп наблюдаваме антидемократичен завой. И Москва, и Вашингтон изразяват последователна подкрепа за АзГ, а Европа остава все по-самотна в опита си да удържа демократичните ценности.

На всичко отгоре в АзГ съществуват различни фракции, а тази в Саксония-Анхалт е сред най-радикалните от тях. Германските служби квалифицират местната структура на партията като екстремистка организация. А лидера ѝ Улрих Зигмунд вестник Spiegel нарича „най-опасния мъж в Германия“ и пише, че е предводител на дясноекстремистка мрежа, предизвикваща страх дори у ръководството на партията. Същевременно обаче Зигмунд има излъчването на симпатяга, който е близо до хората, и е популярен в социалните мрежи.

А източногерманците, които се чувстват изоставени от системните играчи, като Социалдемократическата партия (СДП) и Християндемократическия съюз (ХДС), имат нужда точно от това – някой да ги чува и да облича неудовлетвореността им в политически послания.

Част от предизборните обещания на АзГ в Саксония-Анхалт са такива, че демократично настроените германци (а и европейци) ги побиват тръпки – например децата с увреждания да не учат с останалите, а да бъдат изпратени в специални училища, нещо като т.нар. училища за бавноразвиващи се, както се наричаха тези учебни заведения по времето на социализма. За децата бежанци също се предвижда да учат отделно от здравите и „нормални“ германчета. За да си знаят, че са в Германия само временно.

Домовете за деца – между институционалното наследство и човешкото лице на грижата

Социалистическото наследство на домовете на деца тегне и днес, когато тези институции са вече уж закрити. Евгения Тонева разказва защо дехуманизиращите нагласи, порядки и стигмата продължават да се възпроизвеждат.

В предизборната програма за ЛГБТИ+ хората се говори като за „отклонения“, които не могат да се възпроизвеждат. Предвижда се изгонване на голяма част от хората с мигрантски произход. А нуждата от работна ръка би се очаквало да се задоволи не с миграция, а с насърчаване на раждаемостта с финансови стимули – мярка с меко казано, спорен ефект.

Дали АзГ ще управлява в Саксония-Анхалт, зависи от това как ще се развият отношенията ѝ с малката партия „Съюз Сара Вагенкнехт“ (ССВ),

която спечели пет места в местния парламент. ССВ е партия, кръстена на председателката ѝ Сара Вагенкнехт и отцепила се от друга партия – „Левицата“. „Левицата“ пък е създадена през 2007 г. от отцепници от СДП и от Партията на социалистическото единство от времето на социализма, тоест БКП-то на ГДР. Въпреки че е против НАТО, критична към ЕС и толерантна към Русия, „Левицата“ застава зад социалнолиберални ценности – човешки права, защита от дискриминация на чужденци, ЛГБТИ+ хора и пр., равенство на половете и т.н.

Партия като „Левицата“ в България нямаме, но Сара Вагенкнехт можем да оприличим на Корнелия Нинова, както и на остатъците от БСП и всичките ѝ производни, включително „Прогресивна България“. „Лявото“ на Вагенкнехт е близо до крайнодясното на АзГ, както Нинова и Костадин Костадинов са си лика-прилика. ССВ е антиимигрантска партия, която недолюбва човешките права и харесва Путин, и затова стана първата партия, готова да подаде ръка на АзГ за съвместно управление. Друг е въпросът дали Улрих Зигмунд ще е склонен на компромиси, каквито изискват коалициите, или ще се опита да предизвика нови избори с цел да вземе цялата власт. Останалите варианти биха били нестабилни опити за правителство на малцинството.

Как е възможно хомосексуална жена да е начело на „Алтернатива за Германия“

В случай че някой се чуди как точно се връзва личността на Алис Вайдел с посоката и позициите на партията ѝ, Светла Енчева дава някои много интересни отговори. И не, „Алтернатива за Германия“ не е германското „Възраждане“, защото радикализацията там върви по съвсем друга линия.

АзГ и Източна Германия

Но да се върнем към историята с Alphaville. Отношението на организаторите на Фестивала на щастието предизвика такова възмущение не на последно място заради културата на паметта, която се възпитава в Западна Германия след Втората световна война. Тя включва съзнание за вината и отговорността за националсоциализма и убеждението, че той не трябва да се допуска никога повече и че колкото по-голяма е опасността, толкова повече и от всяка възможна трибуна трябва да се посочва тя.

ГДР обаче не е минала през подобен процес. Част от идеологията на социалистическите страни е, че те са от „правилната страна на историята“. Осъзнаване и преработване на вина не са нужни – „другите“ са фашисти, „ние“ сме добрите. Ето защо много източногерманци имат чувството, че след Обединението им се натрапва чужда вина. Това е една от причините АзГ, която предлага скъсване с виновното отношение към историята, да вирее по-успешно в източните федерални провинции. Същевременно липсата на имунна система по отношение на националсоциализма е благодатна почва за възраждането му под една или друга форма.

Усещане за онеправданост

Периодът между падането на Берлинската стена и Обединението на Германия е еуфоричен за гражданите на ГДР – те получават свободата да пътуват, да се изразяват, изобщо – да бъдат част от доскоро забранен за тях свят. Освен това Западна Германия е привлекателна и с по-добре развитата си икономика.

Ала когато обединената държава става реалност, еуфорията постепенно отстъпва място на разочарованието. Много хора мигрират в западните провинции, а източните се обезлюдяват. Производството от времето на ГДР се прекратява, а на негово място идват, ако изобщо дойдат, западногермански концерни. Известните от времето на социализма автомобили „Трабант“ и „Вартбург“ например остават в миналото, а в някогашния завод на „Вартбург“ в Айзенах днес се произвежда „Опел“. Към гордостите на ГДР, които вече не се изработват, принадлежат и мотопедите Simson, какъвто демонстративно кара Улрих Зигмунд.

Но не е само индустрията – за източногерманците много от нормите, ценностите и дори немалка част от езика на държавата, от която стават част, са чужди. Уж всички в страната говорят немски, но навлизат нови думи за доскоро несъществували реалности, а езикът, описващ бившата социалистическа реалност, е непонятен за германците на Запад.

Така желаното някога обединение се превръща в усещане за колонизираност – налагане на чужд свят и обезценяване на собственото минало заедно с постиженията и уникалността на всекидневието му.

Между 1995 г. и 2019 г. федералното правителство на Германия и западните провинции отделят общо 238 млрд. евро за източните провинции чрез програма, наречена „Пакт за солидарност“. Първият пакт е до 2004 г. и основната му цел е подобряване на инфраструктурата на територията на бившата ГДР. Вторият е предназначен за компенсиране на трудностите, произтичащи от разделянето на Германия, и за ускоряване на икономическото догонване на западната част от страната.

Всички тези средства обаче не допринасят за премахването на убеждението на голяма част от жителите на източните провинции, че са третирани като втора категория германци. На този фон АзГ обещава да върне достойнството им и предлага визия за миналото и бъдещето на Германия, в която те се припознават. Визия, в която се акцентира върху гордостта и превъзходството, а не върху срама и вината.

Отношение към миграцията

Отрицателното отношение към миграцията, особено от мюсюлмански страни, е основната характеристика на АзГ. Затова на пръв поглед изглежда парадоксално, че партията е толкова популярна точно в източните провинции на Германия. В тях (без да броим столицата Берлин) делът на чужденците и на хората с миграционен произход е значително по-нисък, отколкото в западните. С миграционен произход (това включва чужденци, също хора, получили германско гражданство, както и такива, които може да са родени в Германия, но поне единият от родителите им е от чужбина) е 34,4% от населението в западните провинции, а в източните делът му е почти три пъти по-нисък – 12,1%. Що се отнася конкретно до чужденците (които нямат германско гражданство), в западните провинции делът им е 15,7%, а в източните (без Берлин) – 7,6%, тоест около два пъти по-нисък.

Не е задължително обаче отношението към миграцията да съвпада с реалното ѝ присъствие. В България преди 11 години например, когато „бремето на бежанската вълна“ от Сирия се усещаше като „непосилно“, търсещите закрила бяха малко над 26 000, като по-голямата част от тях не останаха в страната. Но популистки партии насаждаха омраза към бежанците, а хората се страхуваха от тях, понеже не ги познаваха.

Подобна е ситуацията и в източните германски провинции – колкото по-рядко местните хора срещат чужденци, толкова по-чужди са те за тях и толкова по-голяма е вероятността да ги възприемат като опасност и да ги нападат с идеята, че се защитават.

Конкретно в Саксония-Анхалт най-много са чужденците от Украйна (около 36 000) следвани от сирийците (малко над 29 000) и поляците (близо 15 000). Като цяло преобладават чужденци от Източна Европа, включително около 4700 българи и още толкова руснаци. Но АзГ е и против украинските бежанци, а ако плановете на партията за „ремиграция“ се осъществят, е доста вероятно от тях да пострадат и българи.

Случаят Джем (Йоздемир). Защо „Зелените“ победиха в Баден-Вюртемберг

Победата на „Зелените“ изглежда като статистическа грешка – 0,5 пункта пред ХДС. Но е важна, защото зад този резултат стои човек. Джем Йоздемир – син на турски гастарбайтери, „шваба“, както сам се нарича, и попкултурна фигура – се оказа факторът, който преобърна изборите. Как – от Светла Енчева.

Упражнения по демократично безсилие

Докато АзГ в Германия и други антидемократични популисти в Европа набират скорост, европейските институции и партиите, отстояващи принципите на либералната демокрация, реагират със заучена безпомощност.

Първият симптом на тази заучена безпомощност е подчинението пред популистите и убеждението, че ако заприличаме поне малко на тях, ще си запазим демокрацията. Може би сме станали твърде либерални, твърде толерантни, прекалили сме и затова вече не гласуват за нас. Настоящият канцлер на Германия Фридрих Мерц дойде на власт с обещанието за по-твърда ръка, която според него би трябвало да свали подкрепата за АзГ наполовина. Резултатите от изборите в Саксония-Анхалт обаче показват обратното: преполовена е подкрепата за ХДС – партията на Мерц.

Вторият симптом е подценяването и обвиняването на избирателите на популистки партии. Много от тях нямат екстремистки убеждения, но се чувстват изоставени и неразбрани. И когато някой демонстрира, че е на тяхна страна и прехвърля отговорността за неуспехите им върху „другите“, е лесно да му повярват. Когато им се каже „тези са лоши, не гласувайте за тях“, това само ги ожесточава.

Електоралната енигма – защо българите в чужбина гласуват за „Възраждане“

За пореден път след поредните избори се чудим защо българите в чужбина – тази тъй митологизирана група хора – гласуват за „Възраждане“. Марина Лякова прави анализ кои са българите в чужбина, които реално гласуват, и когато го правят, защо вотът им изглежда тъкмо по този начин.

Третият симптом, свързан с предишните два, е, че в настоящия период на тотална несигурност – геополитическа, икономическа, екологична и пр. – демократичният свят като че ли няма вдъхновяващо послание, с което да противостои на популизма. Популистката вълна разполага с конспиративни теории, дезинформационни кампании, инфлуенсъри… Чудовището, родено от симбиозата между ултраконсервативни фундаменталисти и наследството на тоталитарните разузнавателни и репресивни служби, притежава не само умения да влияе, а и много пари да осъществява целите си. И лесно ги постига, защото насреща си има не здрава демократична имунна система, а мрънкане и тюхкане.

Междувременно поколението, което помни Втората световна война, си отива. А с него и демократичният рефлекс „никога повече“. Днес мечтата за бъдещето е мечта за едно идеализирано минало, в което има сигурност и перспективи за всички; в което ние сме си ние – без чужденци, а малцинствата и различните си знаят мястото. Минало, което e forever young.

Водещо изображение: „Синьо небе над Магдебург“, както пише АзГ в страницата си във Facebook броени часове преди излизането на последните изборни резултати. На снимката се вижда и Улрих Зигмунд – лидерът на АзГ в Саксония-Анхалт. Източник: Facebook / AfD

Julia 1.13 released

2026-09-10 jzb

Post Syndicated from jzb original https://lwn.net/Articles/1093567/

Version
1.13 of the Julia programming language
has been released. Highlights include faster precompilation of packages,
improvements to Julia’s REPL, and Juliaup,
a graphical interface for the Julia version manager. A full list of changes can
be found in the release
notes. LWN covered Julia
1.12 in November 2025.

Security updates for Thursday

2026-09-10 jzb

Post Syndicated from jzb original https://lwn.net/Articles/1093566/

Security updates have been issued by AlmaLinux (389-ds-base, ansible-core, buildah, expat, glib2, gpsd, gpsd-minimal, gzip, kernel, kernel-rt, opentelemetry-collector, osbuild-composer, perl-DBI, python-lxml, python3.12-lxml, qt5-qtbase, thunderbird, valkey, vim, and xz), Debian (pyasn1), Fedora (darktable, freeipa, freerdp2, gdk-pixbuf2, GitPython, libsoup3, openssl, perl-Net-DNS, rust-ppmd-rust, samba, and valkey), Mageia (ceph, firefox, nss, perl-DBI, thunderbird, and wget), Oracle (389-ds-base, buildah, expat, git-lfs, glib2, glibc, gpsd, gpsd-minimal, grafana-pcp, kernel, libssh, nginx, perl-GD, python3.14-cryptography, redis:7, skopeo, thunderbird, valkey, xmlrpc-c, and xz), Slackware (xz), SUSE (bzip2, cpio, curl, dracut, fuse-overlayfs, golang-github-vpenso-prometheus_slurm_exporter, helm, java-1_8_0-ibm, kbfs, kernel, kernel-devel, libopenslide-devel, libsoup, libssh2_org, libusb-1_0, libvirt, libzypp, zypper, mcphost, multipath-tools, NetworkManager, opensc, openssl-3, perl-Net-DNS, python-aiohttp, python-Authlib, python-pip, python-sqlparse, python313-dnspython, python313-idna, rpcbind, sssd, strongswan, systemd, tomcat11, ucode-intel, and wget), and Ubuntu (dotnet8, dotnet10, ffmpeg, flatpak, netty, and perl).

Qualcomm Talks Next-Gen Oryon CPU, Adreno GPU, and Hexagon NPU

2026-09-10 Vic A

Post Syndicated from Vic A original https://www.servethehome.com/qualcomm-details-next-gen-oryon-cpu-adreno-gpu-and-hexagon-npu/

Qualcomm disclosed more on its next-gen Oryon CPU, Adreno GPU, and Hexagon NPU that will power next-gen devices

The post Qualcomm Talks Next-Gen Oryon CPU, Adreno GPU, and Hexagon NPU appeared first on ServeTheHome.

1.1.1.1 now supports post-quantum DNSSEC, all 2,420 bytes of it

2026-09-10 Sebastiaan Neuteboom

Post Syndicated from Sebastiaan Neuteboom original https://blog.cloudflare.com/post-quantum-dnssec-1111/

1.1.1.1 now validates DNSSEC signatures made with ML-DSA-44, a post-quantum signature algorithm standardized by the National Institute of Standards and Technology (NIST). This is a first step toward preparing DNSSEC for a future in which today’s signature algorithms are no longer secure.

Cloudflare plans to achieve full post-quantum security by 2029. Much of the work so far has focused on TLS, but public-key cryptography is used in many other systems, including DNSSEC.

While we began experimenting with post-quantum key agreement in TLS in 2019 and enabled support for all customers in 2022, post-quantum signatures have not yet received comparable testing in DNSSEC. There is also some urgency. Widespread client adoption of post-quantum TLS took years, partly because larger messages exposed assumptions and bugs in existing network software. That experience showed why early large-scale testing matters. We cannot wait until quantum computers become an immediate threat.

The problem is that post-quantum signatures are large. Each ML-DSA-44 signature is 2,420 bytes, exceeding common DNS-over-UDP limits before the response includes anything else. At the same time, zones will need to publish conventional signatures for older resolvers for years, creating a potential downgrade path if not validated correctly. The challenge is carrying these much larger responses reliably, without allowing compatibility with older resolvers to weaken protection for newer ones.

With ML-DSA-44 validation enabled, 1.1.1.1 lets us test both challenges at Internet scale: carrying larger DNS responses and preventing fallback to conventional signatures.

Why post-quantum DNSSEC matters

DNS responses are not authenticated by default. An attacker who can forge a response may be able to redirect users to an address of their choosing. DNSSEC prevents this by signing DNS records. A validating resolver such as 1.1.1.1 follows a chain of signed records from the DNS root to the requested domain, checking that the answer is authentic and has not been modified.

DNSSEC supports multiple signature algorithms, but nearly all of those used today are vulnerable to future quantum computers. RSA and ECDSA rely on mathematical problems that are believed to be infeasible for conventional computers to solve at deployed key sizes. We are preparing for the possibility that in 2030 a sufficiently powerful quantum computer could be built that breaks these keys. An attacker could then recover the corresponding private key and create forged signatures that validators would accept. The attack path is shown below.

Quantum computers capable of carrying out these attacks do not exist today. DNSSEC provides authenticity rather than confidentiality, so it is not subject to “harvest now, decrypt later” attacks. The reason to begin now is that changing DNSSEC requires coordination across authoritative servers, registries, registrars, and validating resolvers. The migration must eventually reach the top of the DNS hierarchy, where a compromised key has the greatest impact. An attacker who recovers a root zone signing key using a quantum computer could forge a validation path to any zone below it: “break once, forge everywhere”. ML-DSA-44 gives that migration a standardized starting point, and supporting it in 1.1.1.1 lets us, and the DNS ecosystem at large, gain operational experience.

Why replacing the algorithm is difficult

DNSSEC was designed to support new algorithms. In principle, supporting ML-DSA-44 means publishing its public key and teaching validators to verify its signatures. In practice, two properties make the transition difficult: the signatures are large, and the old algorithm cannot always be removed safely.

A 2,420-byte signature changes the packet

DNSSEC algorithms commonly used today produce relatively small signatures. ECDSA P-256, for example, produces a 64-byte signature. An ML-DSA-44 signature is 2,420 bytes, almost 38 times larger.

That difference matters because many of the systems that send, carry, and receive DNS messages are sensitive to message size. DNS originally restricted messages sent over UDP to 512 bytes. EDNS(0) later allowed a resolver to advertise the largest UDP response it is willing to accept from a nameserver. Many DNS implementations use a conservative UDP payload limit of 1,232 bytes, chosen to fit within IPv6’s minimum MTU (maximum transmission unit) of 1,280 bytes. More recently, RFC 9715 recommended a maximum of 1,400 bytes for DNS over UDP. An ML-DSA-44 signature exceeds that budget on its own, before accounting for the signed RRset, domain names, DNS headers, and other DNSSEC records. Sending such a response as fragmented UDP is unreliable and should be avoided. Instead, the authoritative server should return a truncated response, prompting the resolver to retry using another transport protocol, usually TCP.

The effect is most visible in DNSKEY responses, which contain the keys a resolver needs to validate the zone. An ML-DSA-44 public key is 1,312 bytes, and the DNSKEY RRset also carries a 2,420-byte signature. ML-DSA-44 cannot fully replace conventional signing algorithms until it is widely supported across the DNS ecosystem, a process likely to take years. Until then, DNSKEY responses may contain both conventional and post-quantum keys and signatures to remain compatible with older validators. Key rollovers can add still more keys, making these responses larger again.

Handling DNS over transports other than UDP is not itself unusual. Cloudflare Radar shows that around 85% of queries to 1.1.1.1 arrive over UDP. The platform behind 1.1.1.1, Big Pineapple, also powers other DNS services, including Gateway DNS. Across all services handled by Big Pineapple, around 60% of queries arrive over UDP. The remaining 40% use transports such as TCP, DNS over TLS (DoT), and DNS over HTTPS (DoH).

Those figures describe how queries reach Cloudflare’s resolver services, not how 1.1.1.1 communicates with authoritative servers. Large ML-DSA-44 responses can still cause additional TCP retries on that side, but handling DNS over transports other than UDP is already a normal part of operating 1.1.1.1 at scale.

Supporting two algorithms introduces a downgrade risk

Replacing an existing DNSSEC algorithm cannot happen all at once. If a zone publishes only ML-DSA-44, resolvers that do not support it cannot validate the zone. The practical migration path is therefore to publish conventional and post-quantum keys and signatures together.

That preserves compatibility, but it does not provide post-quantum security by itself. RFC 6840 specifies that “validators SHOULD accept any single valid path.” This rule lets validators use whichever published algorithm they support.

Once a conventional algorithm such as ECDSA is no longer secure, however, the same behavior creates a downgrade path. An attacker could forge an ECDSA-only answer that a resolver accepts despite supporting ML-DSA-44, as illustrated below.

Preventing this downgrade requires an authenticated signal that a zone should be validated with ML-DSA-44. 1.1.1.1 uses DS records published by the parent zone for this purpose. If the authenticated DS RRset contains a record for a supported post-quantum algorithm, the signal is present.

1.1.1.1 then deliberately applies a more restrictive local validation policy. It requires at least one valid post-quantum validation path; a conventional path is no longer sufficient. If no ML-DSA-44 path validates, validation fails. This is not (yet) normal DNSSEC validation behavior, but RFC 4035 allows local resolver policy to determine whether additional signatures must be checked and how conflicting results are handled.

Conventional signatures can remain available for older resolvers without allowing post-quantum-capable resolvers to fall back to them. The downgrade signal is only post-quantum secure if ML-DSA-44 deployment and downgrade protection extend from the trust anchor through every delegation. Rotating the zone key more frequently does not solve the problem: an attacker can target a vulnerable key anywhere higher in the chain and forge every delegation below it.

The road to post-quantum DNSSEC

Adding a post-quantum algorithm to DNSSEC requires more than standardizing the cryptography. It needs implementations in cryptographic libraries, an IANA-assigned DNSSEC algorithm number, support from authoritative servers and validating resolvers, and adoption throughout the DNS delegation chain. ML-DSA-44 now has the initial prerequisites for deployment. NIST has standardized it, and common cryptographic libraries implement it. Its use in DNSSEC is described in the ML-DSA for DNSSEC Internet-Draft, and IANA recently assigned it DNSSEC algorithm number 18.

Adding ML-DSA-44 validation to resolvers is one of the first deployment steps, but it does not create a complete post-quantum chain of trust. Authoritative servers must sign zones with ML-DSA-44, registrars must accept and submit the corresponding DS records, and registries must publish them in parent zones.

This adoption must extend through every parent zone to the DNS root. The root must adopt ML-DSA-44, and its post-quantum key must become a trust anchor for validating resolvers. Any level without post-quantum protection remains a downgrade point.

There is little value in signing a zone with ML-DSA-44 if no resolver validates its signatures. Enabling ML-DSA-44 validation by default on 1.1.1.1 is therefore an important early step. It lets us measure the operational cost of signature verification, additional bandwidth, and increased TCP use between resolvers and authoritative servers.

As with previous migrations, we will also test real-world deployability using background probes on a small fraction of Cloudflare Challenge Pages. These probes will test whether clients can resolve and reach an ML-DSA-44-signed test domain across real networks. We invite other DNS operators and implementers to begin testing ML-DSA-44 at scale. Together, these measurements will show what adjustments are needed as adoption grows.

What this means for you

If you use 1.1.1.1, you do not need to change anything. ML-DSA-44 validation happens automatically when a zone publishes the necessary DNSSEC records, while existing DNSSEC zones continue to validate as before.

This work covers the resolver side of DNS. Our next step is adding ML-DSA-44 signing support to Cloudflare Authoritative DNS and corresponding DS record support to Cloudflare Registrar, which will be available to all customers for free. That will let us test the complete path, from generating signatures and publishing DNSKEY records to transporting and validating them through 1.1.1.1.

Want to see post-quantum DNSSEC in action… all 2,420 bytes of it? Query our dnstest.dev zone using 1.1.1.1:

You can also use Is your DNS resolver post-quantum ready? to test your current resolver. The community is tracking ML-DSA-44 software support on GitHub.

Вътрешни езикови правила за външна употреба (на ваш риск)

2026-09-10

Post Syndicated from original https://www.toest.bg/vutreshni-ezikovi-pravila-za-vunshna-upotreba-na-vash-risk/

Вътрешни езикови правила за външна употреба (на ваш риск)

Редовните читатели на „Тоест“, вярвам, са забелязали, че се стараем да поднасяме не само съдържателни, но и езиково издържани статии. Всеки публикуван материал, след като е бил написан от автора, е минал през ума и очите (а понякога и през сърцето) на поне две инстанции – редактор(ка) и моя милост, коректор(ка). В медията зорко следим текстовете да са съобразени с книжовноезиковите норми. Имаме обаче и някои разминавания, които сме уговорили в нашите Вътрешни правила за стил, граматика, правопис и пунктуация.

Стандарти, кодекси, наръчници

Надали е случайно, че световноизвестни медии и информационни агенции като The Washington Post, The Guardian, Der Spiegel, BBC, Reuters имат редакционни/етични кодекси, в които са заложени основни журналистически стандарти за неутралност, проверка на фактите, етично отношение и т.н. Свой редакционен кодекс има и „Тоест“. С прокламирането на тези стандарти – и с придържането към тях, разбира се – медиите се ангажират да бъдат лоялни към читателите, да им предоставят обективна и достоверна информация, но същевременно улесняват работата в самата редакция, предпазват се от съдебни дела и т.н.

Някои медии си изготвят и собствени езикови правила, скрепени в специални наръчници. Сред най-известните примери са Libro de estilo¹ на испанския El País и Manual of Style and Usage на американския The New York Times, които съдържат стотици страници. С течение на времето тези справочници са станали авторитетни и служат като ръководство за добър стил на много пишещи хора.

Българските медии като че ли са плахи в това отношение и засега освен „Тоест“ ми е известен само още един сайт – „Нула32“, който е заявил публично своята езикова политика.

Тук също може да се запитаме защо е необходимо да има отделни езикови правила, валидни за дадена медия, особено след като разполагаме с подробната кодификация в БЕРОН. Ето го и отговора в синтезиран вид, даден в наръчника на El País:

Наръчникът по стил не е граматика или речник в общоприетия смисъл. Той е вътрешният правилник на редакцията на определено средство за информация, който се опитва да уеднакви изразни системи и форми, така че медията да добие собствен облик и да улесни задачата на читателя.

И така, две са основните причини или по-скоро цели. По-важна ми се струва втората, защото медиите съществуват, за да са посредници² между събитията и читателите. Затова водещо и за авторите, и за редакторите е старанието информацията да се предава максимално точно, ясно и разбираемо за по-широк кръг хора, за да могат те бързо да възприемат прочетеното³. Основното средство за постигане на тази прагматична цел е езикът. А книжовният език с все нормите и правилата си (нека си сложим ръката на сърцето) е малко или много скован. По-точната дума всъщност е ригиден, защото освен „скован“ означава също „неподатлив или трудно податлив на въздействие“ – и ето го, надявам се, нагледно проявен този стремеж да бъдем разбираеми и за онези хора, които не са срещали чуждата дума.

Бързо, скоростно, моментално

Това е повикът на нашето време. Макар да сме декларирали, че журналистиката в „Тоест“ е бавна, сме наясно, че когато са в интернет, повечето читатели включват поне на четвърта скорост, а в много случаи просто сканират текста.

Едно от нашите правила, съдействащи за по-бързото четене, е, че

собствените имена и заглавията, написани с латиница, са в курсив.

Примери има само няколко абзаца по-горе – „Libro de estilo на испанския El País и Manual of Style and Usage на американския The New York Times“. БЕРОН указва в тези случаи да не се използват кавички, защото чуждата азбука разграничава името или заглавието от останалите думи в изречението. В редакцията обаче смятаме, че не е достатъчно, и с курсива сигнализираме, че това е различен код (латински букви) и оригинално име/заглавие.

Въпреки че слятото, полуслятото и разделното писане създават неоправдано много проблеми в реалната езикова практика, ние се придържаме към официалните правила и се разграничаваме от тях само в някои случаи:

1. Пишем като две думи изразите отгоре надолу, отляво надясно, оттогава досега и под. Според БЕРОН те се пишат като три или четири думи: от горе надолу, от ляво надясно, от тогава до сега. След като читателят е свикнал със слято написаните отгоре, надолу, отляво, надясно, оттогава и досега, когато ги среща употребени поотделно, той би се запитал – и съвсем основателно – защо пък в други изрази се налага да ги разделяме. И защо веднъж ги делим на три думи, а друг път – на четири. Филологическото обяснение сигурно е, че в съчетанието от ляво надясно имаме начална точка и посока (в пространството), а в от тогава до сега – начална и крайна точка (във времето), но уважаеми колеги, постановили тези изключения, наистина ли и в момента продължавате да смятате, че си струват?

2. Пишем винаги страна членка и държава членка, дори и когато се посочва част от коя организация е тя, например:

Позорно е за страна членка на ЕС да подлага руски дисиденти на такъв зловещ кафкиански административен ад.

Според официалните правила страна членка (респ. държава членка), се пише разделно, но ако се добави пояснение – на ЕС, вече членка на ЕС се третира като обособена част и следва да се огради с тире и запетая или с две тирета: Позорно е за страна – членка на ЕС, да подлага… Отново смятаме, че това е ненужно издребняване, защото в резултат текстът се натоварва с препинателни знаци, които по-скоро спъват читателя, отколкото го улесняват при възприемането на информацията. За мен лично обособяването тук изобщо не е задължително.

3. Пишем разделно сложните съществителни с първа част арт – арт инсталация, арт галерия, арт карта, макар че арт не се употребява самостоятелно в българския език и тези думи следва да се пишат слято. Със сигурност четенето се затруднява, когато втората част започва с гласна: артинсталация, артидея, артогледало, затова сме приели разделното писане.

В статиите, публикувани в „Тоест“, се разграничаваме и от официално приетия начин за писане на числителните редни имена с арабски цифри и букви: 1-ви, 2-ри, 3-ти, 4-ти, но 5-и, 6-и и т.н. Правилото и всъщност обяснението на кодификатора е, че се пишат тези „букви от края на числителното редно, които не съвпадат със съответното числително бройно“⁴. Добре, за пет – пети, шест – шести и т.н. това важи, но в един – първи, две – втори, три – трети и четири – четвърти сякаш имаме повече несъвпадащи букви. Ако се придържаме строго към това правило, изобщо не можем да напишем първи и втори с арабски цифри, а 3-ети и 4-върти следва да са в този вид.

Затова сме решили да пишем числителните редни имена винаги с последната сричка от думата след дефиса: 1-ви, 2-ри, 3-ти, 4-ти, 5-ти, 6-ти и т.н. Факт е, че в езиковата практика това е преобладаващият начин и много хора се изненадват, когато разберат, че в детската градина и в училище са изработвали поздравителни картички за празника на мама с грешен (спрямо официалните правила) надпис: Честит 8-ми март!

Критика приемаме аргументирана (ние също имаме аргументи)

Когато обявяваш, че в практиката си се разграничаваш от някои официални правила и следваш свои, трябва да си подготвен за критика. Получавали сме и ние – например за това, че употребяваме глаголните форми на -л в състава на учтивата форма в мъжки или в женски род ед.ч., като се съобразяваме с пола на събеседника. В интервютата в „Тоест“ може да срещнете правил сте, бихте искала вместо кодифицираните правили сте, бихте искали.

Официалното правило се нарушава често, особено в устната реч, и е едно от най-разколебаните в българския книжовен език. Важно е, че се нарушава не от необразовани люде, а от хора, които иначе имат сравнително висока езикова култура, включително от журналисти в устни интервюта. Обяснението е във факта, че правилото е извънсистемно – всички други имена и причастия освен миналото свършено деятелно (правили) се употребяват в мъжки или в женски род: любезен/любезна сте, уведомен/уведомена сте. В нашата медия сме решили да бъдем по-близо до съвременната езикова практика.

Срещали сме хапливи забележки и относно предпочитанието ни на каталунски и каталунец пред каталонски и каталонец. В БЕРОН формите са дублетни и дори само това е достатъчно, за да не се налага да влизаме в обяснителен режим. Все пак, избрали сме да пишем Каталуния, а не Каталония, тъй като е по-близо до оригиналното име: Cataluña – на испански; Catalunya – на каталунски. След като сме казали А, е редно да кажем и Б – каталунец и каталунски, за да сме последователни.

Това, че Христо Стоичков и спортните журналисти са популяризирали преди години тези форми, както отбелязват критикуващите, не означава, че трябва да ги отречем тотално. Как да каже човекът каталонски или каталонци, като е живял седем години в столицата на Catalunya и е чувал името на провинцията точно в тази форма? Езикът не е само на високограмотните (за каквито често се имаме) – той е обществено явление, а книжовната му форма следва да зачита реалните езикови употреби все пак.

Медиите по презумпция трябва да са актуални, да са в крак със събитията, проблемите и промените в обществото. На тази актуалност в плана на съдържанието би трябвало да съответства и актуална езикова форма. Книжовният език по природа е консервативен и крета след говоримия – не можем да му се сърдим толкова, но можем да го подсетим, че е време за някакви промени.

В „Тоест“ уважаваме книжовните норми и се придържаме максимално близо до тях. Сами може да прочетете нашите вътрешни правила и да прецените в каква степен ревизираме официалните. Вкопчването в кодификацията обаче и придържането към нея на всяка цена ни прави сковани, по-точно ригидни.

¹ Това е заглавието на наръчника, дадено от El País, но копия от него носят и заглавието Manual de estilo.

² Думата медия води началото си от латинския език, в който едно от значенията на прилагателното medius e „посредничещ“.

³ Естествено, има медии с по-тясна и/или специализирана аудитория.

⁴ Правилото е формулирано в Нов правописен речник на българския език. София: БАН, Хейзъл, 2002, с. 115, т. 83.11. В следващото хартиено издание – Официален правописен речник на българския език. София: БАН, Просвета, 2012, както и в БЕРОН то липсва; дадени са само примерите 1-ви, 5-и, 5-ия, 5-ият за употребата на дефиса. Дали пък кодификаторът не е решил, че правилото не е издържано, и затова го е спестил?

Езикът може да е вкусен и извън блюдото – онзи, българският език, на който говорим от малки и на който около 24 май се кълнем в обич. А той в същността си е средство за общуване и за да ни служи добре, непрекъснато се променя. Да го погледнем в неговата динамика и да се опитаме да разберем какво става и защо, кои са движещите механизми и как те са свързани с обществените процеси. И тъй като задачата не е лека, ще го правим постепенно – на порции.

Признанието като самопризнание

2026-09-10 Дарина Сарелска

Post Syndicated from Дарина Сарелска original https://www.toest.bg/priznanieto-kato-samopriznanie/

Това е признание за всички вас. За българските медии, които предоставят територия за многоликото ни общество и възможности за диалог.

Признанието като самопризнание

С тези думи Кирил Вълчев обясни защо е приел поканата на Илияна Йотова да се кандидатира за неин вицепрезидент. С което стана поредният в списъка с бивши журналисти, пристанали на моментен политически интерес. Бивш, защото през 2021 г. оглави БТА и така излезе от активната журналистика, която допреди това пак съвместяваше с друга професия – освен водещ на „Седмицата“ по Дарик радио беше и юридически консултант на медията (това само по себе си е особен хибрид, но да не придиряме).

Иначе в кампанията ще си е напълно настоящ шеф на БТА, макар и в неплатен отпуск. Да, точно така, журналистите от БТА ще отразяват „безпристрастно“ и „равноотдалечено“ кампанията на своя началник, който съвсем реално може да се завърне в кабинета си на „Цариградско шосе“, ако „Дондуков“ се окаже негостоприемен. Но отразяването, разбира се, ще е професионално. Все пак няма да ни е за първи път! Пак така професионално преди две години Петър Волгин изкара една кампания в неплатен отпуск, преди да прескочи от журналист на общественото БНР до народен представител на „Възраждане“ в Европарламента.

„Няма места.“ Журналистиката след журналистите

Ако се чудите какво стана с медиите в България, този текст ви е напълно достатъчен, за да си дадете отговори на много въпроси. А иначе, вие въпроси може и да си задавате, но в много медии у нас вече няма кой да ги задава – гледайте какво нещо… Защо стана така – от Дарина Сарелска.

От Клара Маринова до Антон Хекимян

От 1990 г. до днес поне една дузина разпознаваеми журналисти от БТА, БНТ, БНР, bTV, TV7, Nova и други национални медии преминават в политиката – като депутати, евродепутати, кметове или кандидати за президент. За същото това време доверието на българските граждани в новините чертае последователен и устойчив спад: според последния доклад на Института „Ройтерс“ то пада до рекордно ниските 21%, което е срив от 5 процентни пункта само за година. С това страната ни се нарежда сред държавите с най-ниско доверие в медиите, два пъти по-ниско спрямо средните стойности в глобален мащаб.

Не твърдя, че има причинно-следствена връзка. Но връзка има. И трябва да сме слепи, за да не видим дебелата не-винаги-червена линия, която разделя обществото на себеобслужващи се елити и пренебрегната и подценена публика. Когато аудиторията види „днес си водещ/директор, утре си кандидат на властта“, „днес интервюираш премиера в поза „уж критичен журналист“, а утре се снимаш с него в предизборен клип“ това размива доверието не само в конкретния човек, а в цялата система.

Защото журналистиката по дефиниция е срещу властта. Не против конкретна партия, а срещу всяка власт.

Добрата журналистика винаги е критична, проверяваща и възпираща нагона на властта. Когато водещи медийни лица преминават от другата страна на тази желязна завеса, това девалвира освен техните авторитетни имена – най-ценното им, пак по думите на Вълчев, но и самата идея за журналистиката като критичен агент и обществен контрол.

И нека не се чудим защо тази година социалните мрежи настигнаха телевизиите като предпочитан източник на новини за българите, а догодина се очаква и да ги задминат. С всички мрачни последствия от това под формата на заливаща ни пропаганда, конспиративни теории, поляризация, радикализация и обикновено Дънинг-Крюгер оглупяване. Последствия, които берем къде с наивна детска изненада, къде с високомерната претенция за интелектуално превъзходство.

Но добре ще е следващия път, като се яви някой Тръмп или местен искащ-да-бъде-важен и получи залпова обществена подкрепа само защото „всички други са маскари“, да спрем за малко, преди да отсъдим. Да не бързаме със заключението „Ето, хората са прости“. И да погледнем дали наистина не се препълни с маскари, и то в редиците на елитите: умните, добре артикулираните, медийно симпатичните емблеми на всякакви кръгове, гилдии и общества, брандирани с признание и статус. Натрупали социален капитал под значката „равноотдалечени от всяка власт“ и скочили скоропостижно в листите на точно всяка власт, която ги пожелае.

Краят на журналистиката

Журналистиката губи не само пари и трафик, а и необходимостта да я има. Алгоритми, нюзинфлуенсъри и политици, които вече говорят директно на публиката, променят правилата на играта. Въпросът вече e не дали медиите са в криза, а дали обществото изобщо още иска журналистика. От Дарина Сарелска.

Първата вълна

Моделът „журналисти, преминали под партиен пагон“ не е нов. И не е само наш. Но у нас се вижда ясно в две вълни: първата – в края на 90-те, когато голямата политическа промяна трябва да се захрани с доверието, инвестирано в разпознаваеми лица и имена от единствените дотогава държавни медии, основно БНТ; втората – след 2005 г., когато частните телевизии стават стартова площадка за политически проекти.

В първата вълна се помнят имената на спортната журналистка Клара Маринова, после депутатка от БСП, както и на Асен Агов и Диляна Грозданова. Грозданова успява да сбъдне мечтата на всеки опортюнист и прави завъртане на 360 градуса. Първо взема завоя от емблематично лице на БНТ към политиката като пиар на Стефан Софиянски, влиза и сред учредителите на партията му „Съюз на свободните демократи“; след това застава и на депутатската банка (2001–2005) в редиците на НДСВ, а после се връща в медиите като изпълнителна директорка и водеща на частната TV7, чийто собственик тогава е съпругът ѝ Любомир Павлов.

Асен Агов пък минава през БТА, БНР и БНТ, където стига до директор на новините, а после и на цялата телевизия след идването на власт на СДС (1992–1993). За да няма никакви съмнения кои са двигателите зад кариерното му развитие, е уволнен веднага след падането на правителството на Филип Димитров и преминава в активна политическа кариера с дълга поредица депутатски мандати от листата на СДС. Години след него този висш пилотаж пробва и Антон Хекимян, но стигна само до поста общински съветник от ГЕРБ в София.

Да, ама не

Единствен Петко Бочаров казва своето прословуто „Да, ама не“ на предложенията за политическа кариера. Дългогодишен журналист в БТА, по-късно зам.-главен редактор, добил популярност през 80-те като лице на предаването „Всяка неделя“, през 1991 г. е предложен за депутат от СДС, но отказва, защото смята депутатстването за несъвместимо с журналистическата си роля. Запомнете това. Защото е

първият документиран публичен дебат в България за конфликта на интереси между медийна и политическа роля след промените. И първият публичен отказ. Следват тихи отлагания или признателни съглашателства.

Между другото, Петко Бочаров е сред малкото публични личности с минало на сътрудници на Държавна сигурност, които сами го изваждат на светло, преди то да бъде огласено официално, и се разкайват за това. Изглежда, интегритетът понякога позволява на човек да демонстрира доблестно поведение дори с подобна биография. Обратното обаче също важи: фактът, че името ти никога не е било в архивите на един мракобесен апарат, сам по себе си не те прави по-достоен.

Да не се посочваме!

Властта в България може да се смени, да се прекръсти на „прогресивна“ и да обещае нов обществен договор, но едно остава непроменено – страхът от журналистически въпроси. А когато медиите са заключени в мазето, „демокрацията“ неизбежно започва да си говори сама със себе си. От Дарина Сарелска.

Частните телевизии като политически стартови площадки

Периодът на ширпотребата идва със зората на частните телевизии. Пионери са Волен Сидеров и неговата проруска партия „Атака“, която се ражда като политически проект от едноименното предаване на Сидеров по телевизия СКАТ, а само няколко години по-късно се изкачва до четвърта политическа сила в 40-тото Народно събрание.

Николай Бареков се опитва да повтори модела през 2014 г. След седем години, в които гради образа си в най-гледания сутрешен блок – на bTV, и то в най-силните години на телевизията, за кратко става началник в TV7 на банкера Цветан Василев, който тогава все още е в съдружие с Делян Пеевски. На 25 януари 2014 г. Бареков учредява партия „България без цензура“, с която достига до мандат в европарламента. Бареков е може би най-яркият пример за употребата на медия като политически трамплин. По-късно сам заявява: „Аз помогнах на Пеевски да разврати медиите“ – самопризнание, което независимо от мотивацията илюстрира дълбочината на проблема.

Интересното е, че по пътя си Бареков успява да приласкае и друг влиятелен за времето си телевизионен журналист – Росен Петров, който обръща палачинката на живо в ефира на bTV. По време на интервюто си с политика Бареков в предаването си „Нека говорят“ на 9 февруари 2014 г. водещият Петров подарява на госта си своята офицерска сабя и му се врича във вярност в ефир, като напуска телевизията с изчитане на нарочна декларация, за да се влее в редиците на Барековата партия. Без цензура. Един от най-срамните моменти, записани от камера в най-новата ни телевизионна история, е все още достъпен в интернет, макар и не на страницата на bTV, насладете се.

Списъкът продължавa с Елена Йончева, Тома Томов, водещите от Nova Калина Крумова (започнала кариерата си от СКАТ) и Цвета Кирилова, Александър Симов и така до Антон Хекимян и Петър Волгин. През 2023 bTV поне от кумова срама изпрати Хекимян на партийна служба, приемайки оставката му с „незабавен ефект“ и демонстрирайки престорена институционална чувствителност към репутационния риск.

За да гаси имиджовия пожар тогава, телевизията обеща външен мониторинг на обективността на новините и актуалните предавания. Не се чу какво е установила тази проверка. Макар другата проверка – публичната, Хекимян системно да не издържаше години преди това, утвърждавайки се като предпочитан интервюиращ на Бойко Борисов, който в най-мрачните времена на брутално политическо превземане на прокуратурата започваше интервютата си с главния прокурор Цацаров с въпроса какво значи името Сотир (значи „спасител“) и с покана да сподели кого последно бил спасил от кабинета си в Съдебната палата. Това само му вдигна цената до шеф на новините и неуспял кандидат-кмет на ГЕРБ.

И така до днес, когато никой не се скандализира при прескачането на шефовете на медии в политиката. Няма нужда да се правят проверки наужким, нито да се мятат оставки. Просто си пускаш неплатен. Така Кирил Вълчев е и възможен, и направо закономерен кандидат за вицепрезидент, а в инициативните комитети се прескачат журналист през журналиста – от Константин Вълков при Андрей Гюров, до Валерия Велева, Кристина Патрашкова и Явор Дачков при Илияна Йотова.

Има за всички. Хубаво е, и е готово

Хубавото е, че става видимо. Журналисти с двойно предназначение се самоосветяват, заемайки видими публични позиции. Тази прозрачност, макар да убива доверието в професията, все пак е и малко полезна. Защото вдига завесата пред тези авторитети под прикритие. Разбира се, не бива да сме наивни – със сигурност се отглеждат приемливи фасади от ново поколение. А и границата отдавна е твърде размита.

Да, между журналистиката и политическата власт трябва да има защитна стена.

За съжаление, все по-често тя прилича на въртяща се врата. Съвсем нормализирана е вече практиката действащи журналисти да предлагат медийни обучения и консултантски PR услуги – и на корпоративни, и на партийни клиенти. Защо пък да не водят направо и собствените си кампании от студиата на новините!

Къде е границата?

Журналистите, разбира се, могат и е нормално да бъдат политически хора – с убеждения, с позиция, с обществен ангажимент. Безпристрастността не означава безразличие. Границата минава другаде –

да не злоупотребяваш с капитала на собственото си доверие. Да не го търгуваш, да не го превръщаш в разменна монета за пост или привилегия.

Журналистите имат право на гражданска активност, дори и на граждански протест – защото това е лична позиция. Не злоупотреба или тайно превалутирано доверие в полза на кампания или на конкретен политически интерес. Стандартът за конфликт на интереси и недопускане на пристрастност е описан ясно и с примери в принципите на Associated Press – световния аналог на БТА, с който родната агенционна журналистика иначе обича да се сравнява:

От служителите в редакцията се очаква стриктно да избягват всякаква политическа дейност, независимо дали редовно отразяват политика, или не. Те не могат да се кандидатират за политически длъжности или да приемат политически назначения, нито да извършват дейност по връзки с обществеността за политици или техни организации. При никакви обстоятелства не бива да даряват средства на политически организации или за политически кампании. Те трябва да преценяват особено внимателно дали да членуват, или да правят дарения в други организации, които могат да заемат политически позиции.

Дори служителите извън редакцията – разбирайте счетоводители, шофьори и юрисконсулти, трябва да се въздържат от политическа дейност и от дарения, освен ако не получат одобрение от прекия си ръководител. Ограничения има даже за членовете на техните семейства, чиито политически каузи и ангажименти също подлежат на публично деклариране.

Разбира се, преквалификацията на журналисти в политици не е само роден феномен – справка: от Мусолини, през Уинстън Чърчил, до Борис Джонсън. Но и тук има нюанс и той е в наличието на т.нар. охлаждащ период.

Има ли кой да ги накаже? За забранителните списъци в bTV и кебапчетата в медиите

Имало едно време един Асен. Вървял, вървял през девет телевизии в десета и попаднал в bTV, където решил, че ще е „яли, пили и се веселили“, докато той каже и само с когото той прецени. Извинявайте, но няма нищо приказно в тази история. От Дарина Сарелска.

Липса на „охлаждащ период“

Според етичните стандарти в много страни, когато журналист става политик, се изисква, или поне силно се препоръчва, пауза между двете роли, за да не се компрометира доверието в медиите. Нали заради това доверие се търсят подобни кандидати все пак?! Не е много умно да се реже клонът, на който седиш.

Така преходът е плавен и бившите журналисти стават политици обикновено след като вече са изградили партийна кариера (говорители, съветници, експерти). Такъв е примерът и на самата Илияна Йотова, която е доста последователна в кариерната си траектория: започва като репортер, налага се като разпознаваемо лице в БНТ, после оставя журналистиката зад гърба си, заема се с комуникациите на БСП, откъдето бавно, полека и отвътре изгражда партийна кариера, за да стигне днес до президент. Това е класическият и по-приемлив път. При Вълчев и при повечето български примери преходът изглежда рязък: от днес за утре. Без междинни стъпки в партийната йерархия или време за хигиенна дистанция. Нещо като да вдигнеш сватба, преди комшиите да са разбрали, че си се развел.

Но да се върнем към (само)признанието на Кирил Вълчев:

Приемам поканата на президента като признание не за мен, а за БТА – институция на съгласие, на общуване с българите по света, на памет, нещо, от което има нужда в Президентството.

А Илияна Йотова допълва:

Дълги години имаме обща кауза с господин Вълчев. Работихме заедно за тази кауза – за опазване на българщината, за развитието на българската идентичност, на българския дух, на българската писменост, на буквите.

Да, от такава БТА има нужда Президентството. От такива медии имат нужда всички овластени. Институции на дългогодишното съгласие. Медии, тихи за злоупотребите на властта и споделящи шумно нейните каузи – днес буквите и българщината на „Прогресивна България“, вчера евроатлантизмът ала ГЕРБ и ДПС – Ново начало, утре – каквото нуждата покаже. Стига да е споделена.

Good night, and good luck, motherf*ckers

Историята на американското предаване „60 минути“ е разказ за механизмите, чрез които се опитомяват медиите. И тези механизми са удивително сходни, независимо от пазара или знамето пред сградата на телевизията. Един текст в стил „Думам ти, дъще, сещай се, журналистическа снахо“ от Дарина Сарелска.

А журналистиката – тя отдавна е пратена в отпуск. Обикновено платен. Така стигаме до реалността, в която нищо не е вярно и всичко е възможно. Времена, в които всеки, който има YouTube канал, се пише журналист. Всеки, който е натрупал два грама доверие като журналист, може да го капитализира за политически пост. Всеки новоизлюпен политик може да привижда в собствените си котерии обществения интерес, обяснявайки ни как „работи за хората“. А пък хората гледат „Ергенът“. Може би там някъде е бъдещият спасител.

AIs Compress Exploit Timeline

2026-09-10 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/09/ais-compress-exploit-timeline.html

Give an AI agent a mere rumor of an exploit, and it’s enough for them to find it.

What’s worse, I found I could use my own agents to find the exploit just by knowing roughly what it was about and so could have been exploiting it well before the public patch was available! Given that just the rumour of a security issue seems enough to give attackers enough info to find new exploits, we’re going to need to change the way we deal with security responses in open source.

Simon Willison comments:

Anil points out that this rate of discovery appears incompatible with existing open source embargo practices for new issues. If an issue can become an exploit this fast, we need to figure out new processes for keeping our communities safe.

A Playbook for Defeating MAGA

2026-09-10 The Atlantic

Post Syndicated from The Atlantic original https://www.youtube.com/watch?v=tI2X0G1vGBA

Президентски избори 2026 – заявление за гласуване

2026-09-10 Боян Юруков

Post Syndicated from Боян Юруков original https://yurukov.net/blog/2026/pres2026-zayavlenie/

Тази информация беше изпратена на над 3000 абонирали се за бюлетина на Glasvam.org заедно с новини и полезни съвети за подготовката и провеждането на гласуването в чужбина.

На 25-ти октомври 2026 ще се проведат избори за президент и вицепрезидент. Вече може да подавате заявление за гласуване зад граница. Ще намерите формуляра на страницата на ЦИК. Там може да проверите и дали правилно е записано заявлението ви. Ето няколко важни неща, които трябва да знаете:

Крайният срок за подаване е 29-ти септември в полунощ българско време
Подаването на заявление за гласуване в секцията най-близо до Вас, ще Ви улесни и ще ускори изборния процес, тъй като ще сте вече вписани в списъците
Заявление се подава за всеки вот поотделно. Т.е. не се пренасят от предходни избори
Дори да подадете заявление, а се окаже, че на 25-ти октомври сте в България, ще може да гласувате в секцията си по постоянен адрес с попълване на декларация
Вече са предварително одобрени 362 места за секции в чужбина, където в последните 5 години е имало поне 100 гласували. Те са в 293 града в 57 държави. Това е значително по-малко от предишни години, тъй като автоматичното одобрение на секции по чл. 14, ал. 2, т. 2 беше ограничено до рамките на Европейския съюз. По-малко е дори от преходните, където бяха силно ограничени секциите в чужбина.
За отваряне на секции извън Европейския съюз остава единствено възможността да се съберат поне 40 заявления и след преценка на дипломатическите представителства. През 2026 г. поне отпаднаха ограниченията за броя секции извън ЕС, което засягаше основно Великобритания, Турция и САЩ. Това означава, че подаването на заявление е още по-важно от предходни години.
Автоматичното одобрение и събраните заявления не означава, че непременно ще има секции на тези места. Това зависи от възможностите на помещенията и дали има комисии и доброволци към тях. Решението е на ЦИК по препоръка на Външно. Могат да се увеличат шансовете като се подават заявления за тези места и повече хора се включат като членове на комисии и доброволци.
На някои места като Германия е нужно да се иска разрешение от местните власти. Това вече би трябвало да се случва предвид предварително одобрените места. Очакваме информация от МВнР
Подаването на заявления освен, че подпомага изборния процес, показва и повишен интерес на съгражданите ни в чужбина към вота

В началото на 2026 г. трябваше да се въведе избирателен район Чужбина с 4 депутата, които щяха да представляват единствено българските граждани зад граница. Тази възможност обаче беше отложена отново до януари 2028-ма в последния момент преди парламентарните избори в началото на годината. Повече по темата описах преди предишния вот.

Събирането на заявленията ще може да следите в реално време на картата, както и на подробната таблица. Възможно е автоматичното зареждане на данните да спре заради промени по сайта на ЦИК, каквито случаи имаше в последните години. Ще допълня, ако има промяна.

[$] LWN.net Weekly Edition for September 10, 2026

2026-09-10 jzb

Post Syndicated from jzb original https://lwn.net/Articles/1092273/

Inside this week’s LWN.net Weekly Edition:

Front: Rust’s never type; Debian and CERN; memory tiering; testing multithreaded Python; fixing TCMalloc; Typst.
Briefs: Rustls; Asahi Linux; Buildroot 2026.08; Grml 2026.09; Audacity 4.0; Jellyfin 12.0; LibreOffice Base; Quotes; …
Announcements: Newsletters, conferences, security updates, patches, and more.

Remembering George W. Bush’s Iconic Moment After 9/11

2026-09-10 The Atlantic

Post Syndicated from The Atlantic original https://www.youtube.com/shorts/G_j5A232b70

Customize Amazon API Gateway destinations for execution logs

2026-09-10 Giedrius Praspaliauskas

Post Syndicated from Giedrius Praspaliauskas original https://aws.amazon.com/blogs/compute/customize-amazon-api-gateway-destinations-for-execution-logs/

Amazon API Gateway execution logs help you trace request processing step by step through your REST API stages. They capture authorization results, integration latency, mapping template output, and error details that are otherwise invisible at the API surface. When a production request fails in a way the access log cannot explain, the execution log is usually where you find the explanation.

Until now, execution logs had two constraints. Every log event was truncated at 1 KB, so a request carrying a moderately sized JSON body would exceed that limit and the remainder was dropped. Logs could only go to the auto-managed log group that API Gateway creates for you (API-Gateway-Execution-Logs_{rest-api-id}/{stage_name}).

With Amazon CloudWatch Logs delivery for REST API execution logs, you can now route execution logs to Amazon CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), or Amazon Data Firehose. Log events can be up to 1 MB per entry, and you benefit from vended logs pricing.

In this post, you learn how CloudWatch Logs delivery works with API Gateway execution logs, how to configure it, and what patterns work best for common observability scenarios.

Understanding API Gateway execution logs

API Gateway produces two categories of logs: access logs and execution logs. Access logs record a summary line per request, similar to an HTTP server access log. You configure the format and destination yourself.

Execution logs are different. They capture the internal processing of each request as it moves through the API Gateway pipeline: authorizer evaluation, request validation, integration dispatch, response mapping, and error handling. These logs exist so you can answer questions such as “why did my authorizer reject this token?” or “what did the mapping template produce before it reached my backend integration?”

API Gateway manages execution log creation automatically. When you set loggingLevel to INFO or ERROR in your stage’s method settings, the service writes execution log events to a CloudWatch Logs log group it manages on your behalf. You do not choose the log group name or configure retention directly on it.

The auto-managed model works for many customers but may create friction for teams with specific observability requirements. Compliance frameworks that require logs in S3 with a particular prefix structure need an extra subscription filter and delivery mechanism. Sending execution logs into a security information and event management (SIEM) tool through a Firehose stream requires a forwarding layer.

Configurable log delivery with CloudWatch Logs

CloudWatch Logs delivery separates log routing from log content. Two concepts control the behavior:

DeliverySource is scoped to your API Gateway stage ARN. It defines where logs go. You create a delivery source, then attach one or more delivery destinations (CloudWatch Logs log group, S3 bucket, or Firehose stream).

MethodSettings controls what gets logged. The loggingLevel setting (INFO, ERROR, or OFF) and dataTraceEnabled flag still determine which log events API Gateway produces. These settings work the same way regardless of whether you use the auto-managed log group or CloudWatch Logs delivery.

When you create a delivery using the CloudWatch Logs APIs, CloudWatch Logs activates your log delivery on your API Gateway stage. When you delete the delivery, CloudWatch Logs disables it accordingly. You do not need to flip any flags on the API Gateway side, and the execution logs automatically resume flowing to the auto-managed log group.

Your existing method settings keep their meaning. The loggingLevel and dataTraceEnabled values continue to control log content. If loggingLevel is already INFO or ERROR, creating a delivery redirects those logs to your chosen destination with no further configuration.

The following diagram shows how the pieces fit together.

Diagram showing one API Gateway stage delivery source fanning out to CloudWatch Logs, Amazon S3, and Firehose destinations.

Figure 1 — A single delivery source scoped to an API Gateway stage feeds one or more deliveries, each of which writes to a delivery destination backed by CloudWatch Logs, Amazon S3, or Amazon Data Firehose

The following table summarizes what changes when log delivery is active.

Aspect	Standard execution logging	Log delivery
Destination	Auto-managed CloudWatch Logs log group	CloudWatch Logs, Amazon S3, or Firehose
Multi-destination	No	Yes
Pricing	Standard CloudWatch Logs ingestion	Vended logs pricing
Log event size	Truncated at 1 KB	Up to 1 MB
Setup	Set `loggingLevel` in MethodSettings	Create delivery through CloudWatch Logs APIs
Teardown	Set `loggingLevel` to `OFF`	Delete delivery

What stays the same

Only execution log routing changes. Access logs continue to flow through accessLogSettings to whatever log group you configure, and unrelated stage features such as AWS X-Ray tracing, detailed CloudWatch metrics, throttling, and caching behave exactly as they did before.

Configuration and integration options

Before you create a delivery, confirm the following requirements:

The API Gateway REST API is deployed to a stage.
loggingLevel is set to INFO or ERROR in MethodSettings.
The account-level CloudWatch Logs IAM role is configured. For setup steps, see Set up CloudWatch logging for REST APIs in API Gateway.
For cross-account delivery, the destination has an appropriate resource policy attached through PutDeliveryDestinationPolicy.

Sending logs to a custom CloudWatch Logs log group

The most common starting point is redirecting execution logs to a log group you own. You get direct control over retention policies, metric filters, and subscription filters. The following steps use the AWS Command Line Interface (AWS CLI) with the fictitious REST API ID abc123, stage prod, Region us-east-1, and account 111122223333.

Create a delivery source referencing your stage ARN. The log type for REST API execution logs is EXECUTION_LOGS:

aws logs put-delivery-source \
    --name my-apigw-execution-logs \
    --resource-arn arn:aws:apigateway:us-east-1:111122223333:/restapis/abc123/stages/prod \
    --log-type EXECUTION_LOGS

Create a delivery destination pointing to your custom (existing) log group, then create the delivery that connects them:

aws logs put-delivery-destination \
    --name my-execution-log-destination \
    --delivery-destination-configuration \
        destinationResourceArn=arn:aws:logs:us-east-1:111122223333:log-group:/my-api/execution-logs

aws logs create-delivery \
    --delivery-source-name my-apigw-execution-logs \
    --delivery-destination-arn arn:aws:logs:us-east-1:111122223333:delivery-destination:my-execution-log-destination

Verify that the delivery is active by listing deliveries for the source:
```
aws logs describe-deliveries
```

The response includes the delivery ID, source, and destination ARN after delivery is established. Execution logs flow to /my-api/execution-logs instead of the auto-managed group.

Note: Log delivery adds structured fields (resource_arn, event_timestamp, api_id, stage, resource_path, http_method, and payload) to each event, so a new delivery emits more than your previous logs. To keep the traditional execution log format with nothing extra, set output format and record fields while creating delivery destination and creating delivery:

aws logs put-delivery-destination \
    --output-format "plain" ...

aws logs create-delivery \
    --record-fields "payload" \
    --field-delimiter "" ...

Routing logs to Amazon S3

S3 works well for long-term retention at lower cost, or for feeding logs into analytics tools such as Amazon Athena. The bucket must be in the same region as your API. Create a delivery destination pointing to your bucket:

aws logs put-delivery-destination \
    --name s3-archive-destination \
    --delivery-destination-configuration \
        destinationResourceArn=arn:aws:s3:::amzn-s3-demo-apigw-logs

Then create a delivery using the same source name. CloudWatch Logs delivers the events to your bucket, where you can query them with Athena or catalog them with AWS Glue.

Streaming to Amazon Data Firehose

For real-time analytics pipelines or third-party SIEM integration, Firehose delivery sends execution log events directly to your stream. The setup is identical: create a delivery destination with your Firehose stream ARN, then create a delivery. With direct Firehose delivery, you no longer need to maintain CloudWatch Logs subscription filters and AWS Lambda forwarders to route execution logs to external analytics systems.

Multi-destination delivery and per-destination shaping

A single delivery source supports multiple destinations. You can route the same execution logs to CloudWatch Logs for real-time alerting, S3 for long-term compliance retention, and Firehose for your SIEM, all from one stage. Create additional deliveries using the same delivery source with different destination ARNs.

Each destination receives identical log events. To shape what reaches each destination, apply a CloudWatch Logs subscription filter on the CloudWatch Logs destination. For example, you can forward only ERROR-level events to a Lambda function that pushes alerts to a SIEM, while the same delivery source writes the full event stream to S3 for compliance.

Management console experience

You can also add a log delivery destination in the management console after you enable logging for the stage.

API Gateway console showing the option to add a log delivery destination after logging is enabled for the stage.

You can specify multiple destinations, both in the current or in a different account:

API Gateway console showing multiple delivery destinations configured, including cross-account options.

Keeping existing monitoring intact

If you have dashboards or alarms on the auto-managed log group, use that same log group as one of your delivery destinations. Your existing monitoring keeps working, and you gain the ability to send logs to additional destinations such as S3 or Firehose in parallel.

Best practices

Update dashboards and alarms before enabling log delivery. When you activate log delivery, the auto-managed log group stops receiving logs. Any CloudWatch alarms, dashboards, or Contributor Insights rules pointing to API-Gateway-Execution-Logs_{rest-api-id}/{stage_name} stop working. Migrate these references to your new log group before creating the delivery.

Keep loggingLevel at INFO or ERROR. Log delivery controls routing, not content. If loggingLevel is OFF, no execution log events are produced regardless of whether a delivery exists. Verify your method settings before troubleshooting missing logs.

Treat the 1 MB log event capacity as a security decision, not only a debugging convenience. With dataTraceEnabled set to true, execution logs include complete request and response payloads up to 1 MB. Those payloads might contain personally identifiable information (PII) or other sensitive data. Confirm your log destinations have appropriate access controls, encryption, and retention policies. Mask or filter sensitive fields in mapping templates upstream of logging and enable data tracing selectively per method or only in non-production stages.

Start with a single destination, then expand. Validate that your log group or bucket receives events correctly before adding Firehose or additional destinations.

Log delivery is best-effort. In rare cases, some log events might not be delivered. For audit-critical workloads, build retention and reconciliation that account for occasional missing events rather than treating execution logs as the system of record.

Cleaning up

To avoid ongoing charges from the resources you created while following this post, delete the delivery and then remove the destinations and any example S3 bucket or Data Firehose delivery stream you no longer need. Deleting the delivery returns the stage to standard auto-managed logging.

aws logs delete-delivery --id <delivery-id>

When the delivery is deleted, CloudWatch Logs disables log delivery on the API Gateway stage automatically. The delivery source and delivery destination remain as independent objects. Delete them with delete-delivery-source and delete-delivery-destination if you do not plan to reuse them.

Conclusion

CloudWatch Logs delivery for API Gateway REST API execution logs helps address the 1 KB event truncation and single managed destination constraints. You can now route full execution logs to CloudWatch Logs, Amazon S3, or Amazon Data Firehose, use multiple destinations from a single stage, and pay vended logs pricing.

The feature works alongside existing method settings. No changes to your current logging configuration are required beyond creating the delivery itself.

To get started, refer to Route execution logs with Amazon CloudWatch Logs delivery in the API Gateway documentation. For more about CloudWatch Logs delivery configuration, see Enable logging from AWS services. For pricing details, review the Amazon CloudWatch pricing page. Try it on a test stage and share your experience in the comments.

Introducing Amazon EBS Volume Clones across AWS accounts

2026-09-10 Channy Yun (윤석찬)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/introducing-amazon-ebs-volume-clones-across-aws-accounts/

Last year, we introduced Volume Clones of Amazon Elastic Block Store (Amazon EBS), a new capability that lets you create instant point-in-time copies of your EBS volumes within the same Availability Zone.

Today, we are extending Volume Clones with cross-account copy, so you can create copies of your EBS volumes into other AWS accounts and optionally re-encrypt them with an AWS Key Management Service (AWS KMS) key in the target account.

With this new feature, you can use your latest application data to develop, test, and experiment in a secondary environment, while protecting and isolating the information in the production environment. For example, you can create copies of a production environment to refresh test and development environments set up in separate accounts with the desired EBS encryption.

Copy EBS volumes across AWS accounts in action
To create a copy of an EBS volume across accounts, the owners of the volume can first grant the target account access to their volume in AWS Resource Access Manager (RAM), which provides a way to share resources across AWS accounts or within an AWS Organization. Then, from the target account, they can locate the volume they have access to create a copy of it.

To get started, choose Share volume for the volume you want to share with the target account in the Amazon EBS console.

Share the volume with other AWS accounts by adding it to existing resource shares, or create a new resource share in the AWS RAM console. For more details, refer to the AWS RAM User Guide.

You can now see confirmation that the volume has been shared in the Volume sharing tab of the volume detail page.

A target account must accept the resource share on the RAM console.

Once they accept the resource share, they can see the volumes in the EBS volume page of the target account. Choose Copy volume for any shared volume.

To share and copy EBS volumes across AWS accounts programmatically, including calling APIs and searching documentation, try the AWS MCP Server and plugins with your preferred AI coding tool. To learn more, visit the Amazon EBS User Guide.

Things to know
Let me share some important technical details that I think you’ll find useful.

Encryption: You can share unencrypted volumes and volumes encrypted with a customer managed key (CMK). Volumes encrypted with the default AWS managed key (AMK) cannot be shared. When copying a shared volume encrypted with a CMK, the CMK must also be shared with the target account. You can specify a different CMK to re-encrypt the copy in the target account.
Monitoring: You can monitor SharedVolumeCopyInitiated through AWS CloudTrail event in your account. You will also receive events in Amazon EventBridge at the start of the copy operation when the state of the copied volume is initializing, and at the end of the operation when the state of the copied volume changes to completed. You can see the shared volume ID, consuming account ID, and event time.
Pricing: Once a copy is initiated, you’ll pay a one-time fee based on your volume size, charged to the account where the copy will reside. There’s no cost for sharing EBS volumes through AWS RAM. The copied volume will incur regular EBS volume charges upon creation.
Availability Zone: The volume copy must be created in the same Availability Zone as the source volume. Use Availability Zone IDs (such as use1-az1) to identify the same physical location across accounts.

Now available
Cross-account volume clones for Amazon EBS are available in all AWS Regions that support Amazon EBS Volume Clones. For Regional availability and a future roadmap, visit the AWS Capabilities by Region.

Give this feature a try in the Amazon EC2 console today and send feedback to AWS re:Post for Amazon EBS or through your usual AWS Support contacts.

— Channy

Testing application resilience with Amazon SQS and AWS Fault Injection Service

2026-09-10 Richard Whitworth

Post Syndicated from Richard Whitworth original https://aws.amazon.com/blogs/architecture/testing-application-resilience-with-amazon-sqs-and-aws-fault-injection-service/

When your application can no longer send or receive messages through an Amazon Simple Queue Service (Amazon SQS) queue, downstream processing can stall. The cause might be a misconfigured Identity and Access Management (IAM) policy, a network partition, a bad deployment, or a transient service event. Your application sees much the same thing regardless: SQS operations start failing. How your services handle those failures (failing fast on what won’t succeed, opening circuit breakers, buffering on the producer side) can be the difference between a brief disruption and a cascading outage.

If you’ve never tested those mechanisms under failure, you’re relying on assumptions. With AWS Fault Injection Service (AWS FIS), you can find out first. The goal isn’t to verify that SQS works, it’s to learn what your application does when SQS operations fail, and whether you’d notice. A resilience experiment tests your recovery mechanisms and your observability at once.

In this post, you’ll learn how to:

Structure a resilience experiment with a clear, measurable hypothesis and success criteria.
Use AWS FIS and AWS Systems Manager (SSM) Automation to simulate progressive access disruption to SQS queues.
Interpret Amazon CloudWatch metrics to determine whether your resilience mechanisms are working, distinguishing producer-side from consumer-side behavior.
Identify and fix gaps in your application’s failure handling.

Solution overview

In this experiment, you block your application’s access to SQS with a scoped deny resource policy, then restore access and observe recovery. The policy you apply rejects the data-plane operations your application depends on (sending, receiving, deleting, and changing message visibility, plus purging) while leaving queue management untouched. Disruption duration increases across four phases to surface different classes of failure. To learn more, see Control planes and data planes.

Important: Don’t deny sqs:*. In IAM policy evaluation, an explicit Deny overrides every Allow, including the automation’s own permission to remove the policy later. A deny covering sqs:SetQueueAttributes, sqs:AddPermission, and sqs:RemovePermission can lock the queue so that even the role that applied it can’t clean it up. Scope the deny to data-plane actions only. See Configuring the experiment for the safe policy shape, SQS troubleshooting: access denied, and this re:Post article on deny-policy lockout.

You can find the fault-injection code (the SSM Automation document, the FIS experiment template, and example IAM policies for both roles) in the FIS template library on GitHub.

Progressive experiment phases

Short disruptions can reveal whether your failure-handling mechanisms activate. Longer ones can expose systemic issues that might only appear under sustained failure.

Note: This experiment tests your application’s resilience patterns, not SQS itself. The scoped deny simulates what your application would experience during a network partition, a permission change, or another access disruption.

You can watch two distinct failure surfaces at once:

Producer side: The component that calls SendMessage. When sends are denied, you’re testing how the producer handles failed enqueues: does it fail fast, open a circuit breaker, buffer locally, or drop messages?

Consumer side: The component that calls ReceiveMessage and DeleteMessage. When receives are denied, you’re testing backlog growth during the outage and, on recovery, redelivery and whether a consumer working through the accumulated backlog keeps up or starts pushing messages toward the DLQ.

To isolate a consumer outage instead, where a bad deployment or scaling issue stops your consumers while producers keep sending, deny only sqs:ReceiveMessage and sqs:DeleteMessage. This can be done by editing the SSM Automation document.

Producer service sends to an SQS queue, a consumer service reads from it, a dead-letter queue attaches to the source queue, and CloudWatch collects metrics

Architecture diagram: producer service → SQS queue → consumer service, with a dead letter queue attached to the source queue and CloudWatch collecting metrics from the producer, the consumer, and both queues.

A note on the example workload. The behaviors here (circuit breakers, local buffering, thread pools) assume long-running producer and consumer services rather than short-lived Lambda invocations.

Define your hypothesis

Start with one question: can you reason about what your system should do when SQS access disappears? That question, not whether this is your first experiment, determines what kind of hypothesis you write.

If you can, state the expectation and the metrics you’ll judge it by: When our application loses access to SQS for [duration], we expect [specific, observable behavior]. Our system will [recovery expectation] within [time] of access being restored, as measured by [metric(s)].

A team with resilience patterns already in place might write: When our order processing application loses access to SQS for 5 minutes, we expect the producer to open its circuit breaker within 30 seconds, fail fast, and buffer messages in local durable storage rather than dropping them. On recovery it will replay the buffer and return to normal processing rates within 2 minutes, as measured by NumberOfMessagesSent returning to baseline and ApproximateNumberOfMessagesVisible draining to near zero within 15 minutes.

If this is your first test or this failure mode has never been exercised, that isn’t a prerequisite. You don’t necessarily need to read the code and settings first, though a basic understanding of the implementation and its normal load helps you set guardrails that bound the test’s impact. Frame the hypothesis as discovery, stating what you’ll observe instead of what you predict:

Our order processing application has never been tested under SQS access loss. We’ll block access for 2 minutes and observe how the producer handles failed sends and whether the consumer recovers unaided, as measured by NumberOfMessagesSent, ApproximateNumberOfMessagesVisible, ApproximateAgeOfOldestMessage, and application error rates.

Either way, write it down before you proceed. The gaps between what you wrote and what happens are where your system needs work.

Prerequisites

The GitHub repo ships working examples. The following bullets note which file to start from. You’ll need:

An instrumented producer and consumer: the application under test. This is the one prerequisite with no example in the repo. The library ships the fault injection, not the workload. The observation tables in this post assume your application emits circuit-breaker state, failed-send and dropped-message counters, fallback-store writes, and duplicate-processing metrics. Without that instrumentation you’ll watch the queue metrics move and learn little about your application.
An IAM role for AWS FIS, trusted by fis.amazonaws.com and able to run the SSM Automation document (ssm:StartAutomationExecution and related, plus iam:PassRole). The repo provides both pieces: sqs-queue-impairment-tag-based-fis-role-iam-policy.json for the permissions and fis-iam-trust-relationship.json for the trust policy. Add the Amazon CloudWatch Logs permissions only if you enable experiment logging. See Logging for AWS FIS.
An IAM role for the SSM Automation document, able to read and modify the target queues’ policies (sqs:GetQueueAttributes, sqs:SetQueueAttributes, sqs:ListQueues, sqs:ListQueueTags). Start from sqs-queue-impairment-tag-based-ssm-automation-role-iam-policy.json and ssm-iam-trust-relationship.json in the repo. The example policy conditions the write on aws:ResourceTag/FIS-Ready, which helps prevent the automation from touching untagged queues. Keep that condition.
SQS queues tagged FIS-Ready: True. This scopes which queues the automation targets. Tag only non-production queues or use planned test windows.
A CloudWatch dashboard and alarms combining those application metrics with the queue metrics across the producer, consumer, and queue (see Monitoring strategy). The repo’s README includes an example put-metric-alarm command for a customer-impact alarm you can adapt as your stop condition.
A documented rollback plan in case the automation can’t remove the deny policy (if a deny ever locks out queue management, see the re:Post article on deny-policy lockout).

Important: Run these experiments in a non-production environment first. In production, confirm you have change management approvals.

Configuring the experiment

AWS Systems Manager Automation applies and removes the deny policy; AWS FIS orchestrates the sequence.

Systems Manager Automation document

The SSM Automation document follows four steps:

getTargetQueues: finds SQS queues tagged with FIS-Ready: True. It calls ListQueues once, which returns at most 1,000 queue URLs, so in an account with more queues than that, add pagination or a QueueNamePrefix filter before you rely on it to find every tagged queue.
applyDenyAllPolicyToQueues: adds a scoped deny statement to each queue’s resource policy. Deny only the data-plane actions your application uses, never the management actions, so the automation can remove its own statement during cleanup. If you adapt the automation, consider adding a validation step that refuses to apply any deny covering management actions. A lockout would then require changing both the policy and the validation.

Tip: You can make the deny self-expiring by adding a DateLessThan condition on aws:CurrentTime to the statement, so the deny stops applying at a set time even if the cleanup step never runs. See IAM condition operators for date and time.

waitForDuration: sleeps for the specified impairment duration (ISO 8601 format, for example PT2M).
removeDenyAllPolicyFromQueues: removes the FISTemporaryDeny statement, restoring normal access. The document routes onFailure and onCancel to this step so that an aborted run attempts to clean up, and the step raises if it can’t restore a policy rather than reporting success.

Choose your blast radius with Principal. "Principal": "*" denies the data plane to every caller: the application under test, but also any admin, canary, or other consumer of that queue. That faithfully simulates a service partition, but on a shared queue it impairs more than your app. To impair only the application, the more common “my app lost access” case, scope the deny to its IAM role:

"Principal": { "AWS": "arn:aws:iam::<account-id>:role/<application-role>" }

A role arn matches all sessions of that role, catching the app’s calls without applying the deny to other callers. On a shared queue, a principal-scoped deny also changes what you measure: queue-level metrics blend impaired and healthy traffic, so lean on your application’s client-side metrics and read recovery as a return to pre-event levels rather than to zero and back. Set the automation’s optional targetPrincipalArn parameter to scope the deny to one principal, or leave it empty to deny all. The rest of this post assumes the full-queue deny ("Principal": "*").

FIS experiment template

The FIS template chains the four impairment phases with recovery periods between each, calling the SSM Automation document with an increasing duration; startAfter fields enforce sequential execution. The escalation is the point. You watch cause and effect at increasing severity:

Phase	Duration	What this duration tends to surface
Impair 1	2 minutes	Fail-fast behavior and circuit-breaker activation
Recover	3 minutes	Buffered messages replay. Metrics return to baseline
Impair 2	5 minutes	Backlog accumulation as the queue fills undrained
Recover	3 minutes	Backlog burndown
Impair 3	7 minutes	Thread-pool and memory pressure from sustained failure
Recover	2 minutes	Recovery under a larger backlog. Whether the consumer keeps up
Impair 4	15 minutes	Systemic limits under prolonged loss of access

The FIS experiment template with four impairment actions of 2, 5, 7, and 15 minutes chained by recovery waits

Figure: The FIS experiment template: four impairment actions (2, 5, 7, and 15 minutes) chained with recovery waits between them.

Stop conditions. A stop condition halts the experiment automatically if a specified CloudWatch alarm fires, an essential control for an escalating experiment. A triggered stop condition also unwinds what it can: FIS cancels the run, and the automation’s onCancel step removes the deny, restoring access. That rollback is a property of this experiment’s design, not of stop conditions in general: an action like EC2 instance termination does not support rollback, so check each action’s rollback behavior before relying on a stop condition to help limit damage. The library template ships with "stopConditions": [{"source":"none"}], because the right alarm depends on health signals the template can’t assume.

Metric choice matters: alarming on a queue metric like ApproximateAgeOfOldestMessage or NumberOfMessagesSent would be incorrect, as those are supposed to move during impairment. So, the alarm would trip in the first 2-minute phase and abort the run before the longer phases surface anything interesting. You’d be alarming on the effect you’re injecting.

Instead, tie the stop condition to a signal that should stay healthy if your resilience mechanisms are working. This would reflect real customer impact. If that signal degrades more than you’ll tolerate (error rates that don’t recover within the 2 minutes your hypothesis allows), your resilience has already failed and continuing risks further customer impact. Some metrics to consider alarming on:

An application error rate or transaction-success metric (a custom CloudWatch metric your app emits, for example failed orders per minute), the most direct measure of customer impact and independent of the SQS metrics you’re perturbing.
Load balancer 5xx count or target response time (for example HTTPCode_Target_5XX_Count on an Application Load Balancer), a good proxy when you don’t yet emit a business metric.
DLQ depth: ApproximateNumberOfMessagesVisible on the dead-letter queue crossing a threshold, which signals messages are failing permanently rather than only backing up recoverably.

See Stop conditions for AWS FIS for more information.

Deriving the threshold from your hypothesis. The preceding hypothesis expects recovery within 2 minutes of access being restored. That number is also your alarm. If failed orders per minute is your customer-impact metric and its baseline is near zero, set the alarm to failed orders per minute > 10 for 2 consecutive 1-minute periods: long enough that a spike while the circuit breaker opens shouldn’t abort the run, short enough that failing to recover inside your hypothesis window stops it. Design the alarm for how the metric behaves during failure rather than for the test: when the circuit breaker opens, a low-volume custom metric might stop emitting data points entirely. Tighten it as you approach production.

AWS FIS experiment in Stopped state after the customer-impact alarm breached and halted the run

Figure: When the customer-impact alarm breached, AWS FIS halted the experiment automatically (State: Stopped)

Running the experiment

To start the experiment with AWS FIS you can use the console or the AWS CLI:

aws fis start-experiment --experiment-template-id <YOUR_TEMPLATE_ID> --region <YOUR_REGION>

What to observe during impairment

During each phase, SQS operations return AccessDenied errors. Note what that does and doesn’t exercise: a 403 is non-retryable, so this experiment validates that your code recognizes it and stops, not your backoff path. To exercise retries and backoff, inject a retryable fault such as throttling or timeouts. The producer and the consumer fail differently, so watch them separately.

SQS queue access policy showing the scoped FISTemporaryDeny statement blocking SendMessage and ReceiveMessage

Figure: During impairment, the queue’s access policy carries the scoped FISTemporaryDeny statement; SendMessage and ReceiveMessage return AccessDenied while management actions still work.

Producer side (the component calling SendMessage):

Stage	Healthy response	Unhealthy response	Signal to watch
First failed `SendMessage`	Recognizes AccessDenied and fails fast	Crashes, hangs, or blocks the calling thread	`NumberOfMessagesSent` drops to ~0. Producer error rate rises
After 3 to 5 consecutive failures	Circuit breaker opens. Sheds or buffers load	Continues retrying indefinitely	Circuit-breaker state metric. Producer CPU / threads / connections
Send gives up (non-retryable error, or retry budget exhausted)	Fails fast and persists the payload to durable fallback storage, or alerts, does not silently drop	Drops the message silently (permanent loss)	Producer “failed send / dropped” counter. Fallback-store writes
Application state	Stays responsive. Degrades gracefully	Returns 500s to callers. Unbounded in-memory queueing	Producer health checks, request latency
Resource usage	Bounded by backoff and circuit breaker	CPU/memory/connections climb (tight retry loops)	Producer CPU, memory, connection-pool usage

Note: a producer that gives up on a send does not route anything to the DLQ.

Consumer side (the component calling ReceiveMessage / DeleteMessage):

Stage	Healthy response	Unhealthy response	Signal to watch
First failed `ReceiveMessage` / `DeleteMessage`	Backs off its poll loop rather than hammering. Any in-flight message returns to the queue after the visibility timeout	Crashes or hangs the consumer loop	`NumberOfMessagesReceived` / `NumberOfMessagesDeleted` drop
Backlog accumulates (consumers can’t drain)	Backlog alarm fires. Scaling responds if keyed to queue depth (for example, backlog per worker)	Backlog grows unbounded. Consumers idle-loop	`ApproximateNumberOfMessagesVisible` stops draining (goes flat or climbs); `ApproximateAgeOfOldestMessage` climbs
Application state	Idempotent processing. Safe to retry	Duplicate side effects on redelivery	Downstream idempotency / duplicate-write metrics
Resource usage	Bounded by visibility timeout and backoff	In-flight messages pile up. Consumer saturation	`ApproximateNumberOfMessagesNotVisible`. Consumer CPU/memory

Don’t expect the DLQ to fill during impairment. Redrive is driven by maxReceiveCount: a message moves to the DLQ only after a consumer has received it that many times without deleting it. With ReceiveMessage denied, nothing is delivered, the receive count doesn’t increment, and nothing redrives. The DLQ depends on the very call that’s blocked, so it’s something to watch for during recovery, not during the outage.

Key CloudWatch metrics, and how to read them:

NumberOfMessagesSent: drops to zero when the deny policy takes effect and producers can no longer enqueue.

NumberOfMessagesSent dropping to zero during each impairment window and spiking on recovery, ending at the stop-condition halt

Figure: NumberOfMessagesSent drops to zero during every impairment window (red) and spikes on recovery (green) as buffered messages replay. The final phase ends at the stop-condition halt (orange).

ApproximateNumberOfMessagesVisible: the current backlog of messages available for retrieval. During impairment this often stops changing, a signal that tells you something is wrong precisely because it goes flat (nothing is being sent or drained).
ApproximateAgeOfOldestMessage: increases as unprocessed messages age, but only if the queue already held a message when the deny took effect. On an empty queue it won’t climb, which is why you read it alongside the visible-message count.

ApproximateAgeOfOldestMessage climbing while the visible backlog stays undrained during the 15-minute phase, then both collapsing on recovery

Figure: Consumer-side impact: ApproximateAgeOfOldestMessage climbs while the visible backlog sits undrained during the 15-minute phase, then both collapse the moment access is restored.

Application error rate: spikes initially, then stabilizes if circuit breakers engage.

Producer circuit breaker opening within seconds of each impairment and closing on recovery as a square wave

Figure: The producer’s circuit breaker opens (1) within seconds of each impairment and closes (0) on recovery, a clean square wave that lags each fault window slightly because it opens only after a few sustained failures.

Count-based metrics (NumberOfMessagesSent/Received/Deleted) reflect system-level activity and can include retries and duplicates, so treat them as trend indicators rather than exact unique-message counts.

What to observe during recovery

When the deny policy is removed, the producer and consumer recover on different timelines.

Producer side:

What to observe	Healthy response	Unhealthy response
Send resumes	`NumberOfMessagesSent` climbs back to baseline. Circuit breaker half-opens, then closes within ~30 seconds	Circuit breaker stays open (stale failure state). Manual restart needed
Buffered / fallback payloads	Replayed from durable fallback storage and re-sent idempotently	Lost permanently (if silently dropped during impairment)

Producer buffering to durable fallback storage during impairment and replaying the buffer on recovery

Figure: The producer buffers to durable fallback storage during impairment (no dropped messages) and replays the buffer on recovery: send success, buffered writes, and replays over the run.

Consumer side:

What to observe	Healthy response	Unhealthy response
Receive / delete resumes	`NumberOfMessagesReceived` / `NumberOfMessagesDeleted` recover	Consumers stay wedged. No auto-recovery
Backlog burndown	`ApproximateNumberOfMessagesVisible` drains steadily; `ApproximateAgeOfOldestMessage` falls	Drain stalls: the rate spikes, then drops to zero and stays there (consumer overwhelmed or stuck)
DLQ contents	Genuinely-poison messages redriven and reprocessed in controlled batches within the DLQ retention period	Reprocessed all at once (overwhelming downstream), or left to age out of the DLQ and be deleted

Recovery is when the DLQ can move. A consumer overwhelmed by the accumulated backlog can re-fail messages and push some to the DLQ. If healthy messages land there, your maxReceiveCount is too low or your consumer isn’t keeping up.

Analyzing results

After the experiment completes, compare what happened against your hypothesis. Focus on these questions:

Did your circuit breakers activate? Measure from the first AccessDenied to when your application stopped attempting SQS operations. Over your target (typically 30 seconds) means your detection threshold is too high.
Did your system preserve messages? Reconcile attempted sends against messages processed after recovery, plus the DLQ and producer-side fallback storage. If the numbers don’t add up you have message loss, and the gap tells you which side lost them.
Are the recovered messages still worth processing? Preservation and relevance are different questions. After a long outage, some buffered sends and backlogged messages represent requests the client has already given up on, and processing them spends recovery capacity acting on stale intent. Compare each message’s timestamp to the current time as you consume it, and drop or sideline anything no longer actionable, deliberately rather than by letting it age out. See REL05-BP04: Fail fast and limit queues.
How did recovery behave? Look at ApproximateNumberOfMessagesVisible after each recovery period. A healthy system drains steadily. If the drain stalls (the rate spikes, then drops to zero and stays there), your consumer is overwhelmed or stuck.
Did longer disruptions reveal new failure modes? Compare the 2-minute phase against the 15-minute one. What tends to surface only under sustained failure:
- Thread pool exhaustion from accumulated retry threads.
- Memory pressure from buffered messages.
- Connection pool starvation.
- DLQ messages aging out: messages that sit in the DLQ longer than its retention period are deleted (see Best practices).

Results that match your hypothesis are evidence your resilience mechanisms work. Results that don’t are your work list.

Best practices

The sections below cover the resilience patterns that turn the gaps this experiment surfaces into fixes: retry logic, circuit breakers, dead-letter queues, and monitoring.

Retry logic with exponential backoff

Don’t retry everything. Retry only errors that might succeed on a repeat, such as throttling, timeouts, and transient 5xxs, and fail fast on non-retryable ones like the AccessDenied (403) this experiment injects. For the errors worth retrying, use exponential backoff with jitter: each failure increases the wait exponentially (1s, 2s, 4s, 8s, and so on) with a random offset that prevents producers who failed together from retrying together and spiking a recovering dependency.

The AWS SDKs have configurable retry behavior built in, so configure it rather than rolling your own. See Timeouts, retries, and backoff with jitter.

Circuit breakers

A circuit breaker stops attempting operations after a threshold of consecutive failures, then lets a single test call through after a recovery timeout. That saves resources on calls that are likely to fail and gives the dependency room to recover. Choose the open-state behavior deliberately: shedding or buffering load is safer than silently switching to an alternate path, because fallback paths are exercised only during failures and tend to fail with them. See Using load shedding to avoid overload and Avoiding fallback in distributed systems.

Dead letter queues

Configure a DLQ for every queue. It’s a consumer-side safety net for poison messages, not a producer overflow buffer. Set maxReceiveCount to the number of processing attempts that make sense for your workload (typically 3 to 5). Because redelivery is what feeds a DLQ, every consumer must tolerate seeing a message twice. See Making retries safe with idempotent APIs. For what a large post-recovery backlog can do, see Avoiding insurmountable queue backlogs.

A DLQ has no depth limit. The constraint is the retention period, after which SQS deletes the message (default 4 days, maximum 14). Set the DLQ’s retention longer than the source queue’s to provide more investigation time before messages are deleted. For standard queues, note that the retention clock runs from the original enqueue time and does not reset on the move to the DLQ, so time in the source queue counts against it. (FIFO queues do reset it.) After the experiment, confirm messages were preserved rather than aged out.

Monitoring strategy

For what to emit and at what granularity, see Instrumenting distributed systems for operational visibility. Build a CloudWatch dashboard combining these across the producer, the consumer, and the queue:

Queue-level metrics: NumberOfMessagesSent, NumberOfMessagesReceived, NumberOfMessagesDeleted, ApproximateNumberOfMessagesVisible, ApproximateNumberOfMessagesNotVisible, and ApproximateAgeOfOldestMessage, read as described in what to observe during impairment.

Application-level metrics:

Error rates by type (distinguish AccessDenied from other failures), tagged by producer vs. consumer.
Circuit breaker state changes (open / closed / half-open transitions).
DLQ message count.
End-to-end message processing latency.

Alarm on ApproximateAgeOfOldestMessage exceeding your SLA threshold as a production alert, but not as an experiment stop condition, since the metric is supposed to rise during impairment. Use a customer-impact signal there instead (see Stop conditions).

Clean up your environment

Verify the deny policy is gone. Check each queue’s access policy on the console or run aws sqs get-queue-attributes --queue-url <URL> --attribute-names Policy. If FISTemporaryDeny is still there, retrieve the policy, delete the statement, and reapply with aws sqs set-queue-attributes.
Process messages that landed in your DLQs during the experiment.
Review CloudWatch metrics to confirm your queues have returned to normal operation.
Document your findings: What matched your hypothesis, what didn’t, and what you’re fixing.

Expand your resilience testing

Once the basics hold, extend the experiment:

Partial failure: Impair only a subset of your queues to test whether your application handles mixed healthy/unhealthy dependencies.

Note: Don’t run two impairment experiments against the same queue concurrently. The automation reads the policy, modifies it, and writes it back. Concurrent runs can overwrite each other and leave a stale deny behind. Target distinct queues, or run them in sequence.

Consumer-side only: Block only ReceiveMessage and DeleteMessage while allowing SendMessage, to simulate a consumer outage while producers keep filling the queue (the most common real-world scenario).

Combine with other failures: Run the SQS experiment alongside EC2 instance termination or network latency injection to test compound failure scenarios.

Explore AWS Resilience Hub: Use AWS Resilience Hub to assess your application’s resilience posture and get recommendations for improvement.

Using FIS scenarios

A scenario is an AWS-authored template bundling the actions, targets, and duration for a recognizable event, so you start from a reviewed definition instead of assembling actions yourself. While AWS provides multiple scenarios in the library, here are two that are a good place to start.

AZ Availability: Power Interruption induces the symptoms of losing power in one Availability Zone: zonal EC2, ECS, and EKS compute stops, new launches in that AZ fail, and subnet connectivity is lost. It’s the sharper test of the queue-based decoupling this post exercises, because producers and consumers lose capacity while the queue itself is not targeted. You learn whether surviving consumers absorb the backlog, whether Auto Scaling replaces capacity in the remaining AZs rather than retrying in the impaired one, and whether the backlog drains inside your hypothesis window. It defaults to 30 minutes of impairment plus 30 of recovery, twice this post’s longest phase.

AZ: Application Slowdown introduces additional latency between resources within a single Availability Zone (AZ). This latency creates many of the symptoms of an application slowdown, a partial disruption, sometimes known as a gray failure. It adds latency to network flows between target resources. Network flows represent the traffic between computing resources: the data packets carrying requests, responses, and other communications between your servers, containers, and services. The scenario can help to validate observability setups, tune alarm thresholds, discover application sensitivity to slowdowns, and practice critical operational decisions like AZ evacuation.

Scenarios carry the same obligations: write the hypothesis first and set the stop condition on a customer-impact metric rather than one the scenario is designed to move. Your derived threshold works unchanged. Copy a scenario into your own template to narrow the targets or change the duration. See Working with the AWS FIS scenario library.

Conclusion

In this post, you learned how to discover what your application does when SQS operations fail, and whether you’d notice. Every gap between your hypothesis and the results is an opportunity to improve your system’s resilience and its observability.

Start with the 2-minute phase in a non-production environment. Fix what breaks. Then run the full sequence and keep running it as the application evolves. Each phase of growth brings failure modes you might only find under load.

For more information, see:

About the authors

Validating multi-Region DR for Terraform Enterprise with AWS FIS

2026-09-10 Frenil Randeria

Post Syndicated from Frenil Randeria original https://aws.amazon.com/blogs/architecture/validating-multi-region-dr-for-terraform-enterprise-with-aws-fis/

In October 2025, Athenahealth, a major North American Electronic Health Record (EHR) provider, discovered a gap. An AWS regional service event in us-east-1 made their single-Region HashiCorp Terraform Enterprise (TFE) deployment inaccessible to their developers. This post shares the architecture, the AWS Fault Injection Service (AWS FIS) validation approach, HashiCorp best practices, and lessons learned from the collaboration between AWS and HashiCorp. Together, these help increase resiliency and verify that the customer’s critical workloads remain active during regional service events.

Currently, Terraform Enterprise (TFE) deployments are only supported within a single AWS Region. This means that, for TFE customers without a well-tested DR plan, a regional service event can block your engineering teams from deploying, modifying, or recovering infrastructure. A multi-Region disaster recovery (DR) strategy addresses this risk. The architecture in this post is a customer-operated DR pattern: HashiCorp supports TFE within a single Region and the HVD module targets single-Region deployments, so the multi-Region failover described here is designed, operated, and tested by the customer rather than provided as a supported product configuration. That strategy only works if you validate your regional failover workflow before you need it. Reacting during an event that impairs your primary Region costs developer productivity and business continuity. AWS FIS exposes hidden dependencies and configuration issues by injecting real failures into your AWS environment.

The following sections walk you through how to design three-phase AWS FIS experiments for TFE, expose hidden dependencies in failover automation, and validate both failover and failback for a multi-Region TFE deployment. You can help prevent extended downtime that impacts your infrastructure deployment capabilities and achieve 12-14 minute recovery times.

Prerequisites

To follow the validation approach in this post, you should have the following:

Familiarity with Terraform Enterprise concepts and administration.
An understanding of Amazon Aurora PostgreSQL-Compatible Edition global databases and Amazon Simple Storage Service (Amazon S3) cross-Region replication.
Basic knowledge of AWS FIS experiment templates, actions, and targets.
AWS Identity and Access Management (IAM) permissions to create and run AWS FIS experiments.
(Optional) A non-production TFE environment to run the sample Terraform code from the GitHub repository.

Starting architecture

If you’re running TFE in a single Region within AWS, this section describes the starting point. Athenahealth’s deployment used HashiCorp’s Terraform Enterprise Validated Design (HVD) module with the following components:

Amazon Elastic Compute Cloud (Amazon EC2) instances running TFE application servers.
Amazon Aurora PostgreSQL-Compatible Edition for application state.
Amazon Simple Storage Service (Amazon S3) for Terraform workspace state files.

Athenahealth hosted these components in the us-east-1 Region. The deployment provided Availability Zone-level resilience but lacked regional failover capabilities. The October 2025 event highlighted what was missing: no cross-Region database replication, no secondary Region compute capacity, no Terraform state file backup outside us-east-1, and DNS pointing exclusively to the primary Region.

Following the October 2025 event, Athenahealth engaged both their AWS and HashiCorp account teams for guidance on protecting not only TFE, but other critical workloads as well. The three organizations worked as a single team to design and implement a multi-Region DR strategy for the TFE environment. By combining the AWS Well-Architected Framework guidance regarding operational excellence and reliability, along with HashiCorp’s best practices regarding DR strategies using Terraform, the team came up with the multi-Region architecture (Figure 1) that would replace the customer’s current single-Region deployment.

Multi-Region DR solution

With an active-passive multi-Region design across us-east-1 (primary) and us-west-2 (DR), Athenahealth achieved a 12-14 minute Recovery Time Objective (RTO) and less than 1 minute Recovery Point Objective (RPO). In the AWS disaster recovery taxonomy, this is a pilot light strategy: data replicates continuously to the DR Region while compute stays at zero. A warm standby variant (DR minimum capacity of 1) trades higher cost for faster RTO. This section covers the architecture components that make this possible and the four-step failover process you can follow during a regional service event.

Multi-Region active-passive DR architecture for Terraform Enterprise with bidirectional S3 replication and a Route 53 health check.

Figure 1: Multi-Region active-passive DR architecture for Terraform Enterprise on AWS. Note the bidirectional S3 replication arrows between Regions and the Amazon Route 53 health check that determines the active Region.

The example Terraform code used to configure and manage the core architecture components, along with the failover process can be found in this sample GitHub repository. You can use this code to test a similar pattern for your TFE workload.

Core architecture components

You can route traffic to the active Region with Amazon Route 53 DNS alias records pointing to an Elastic Load Balancing (ELB) Network Load Balancer. Alias records for ELB targets use a 60-second time-to-live (TTL), which limits how long DNS resolvers cache the record. Clients begin resolving to the DR Region within about a minute of failover rather than waiting for longer cached entries to expire.

In each Region, an Amazon Virtual Private Cloud (Amazon VPC) spans three Availability Zones, and Amazon EC2 Auto Scaling groups manage the TFE instances. To help minimize cost, Athenahealth scaled DR Region compute to zero during normal operations by setting the Auto Scaling group minimum capacity to 0.

For cross-Region database replication, Athenahealth uses Aurora PostgreSQL-Compatible global databases, which provide sub-second replication lag and managed failover. The primary cluster runs one writer and two readers across three Availability Zones. The secondary cluster maintains an inactive writer that’s ready for promotion.

You can replicate Terraform workspace state files bidirectionally between primary and DR Amazon S3 buckets with S3 cross-Region replication. This design supports failback without data resynchronization.

AWS Secrets Manager and AWS Key Management Service (AWS KMS) provide cross-Region credential and encryption key management. Two TFE-specific dependencies deserve attention when you design for multi-Region. First, the TFE encryption password protects the internal Vault unseal key and root token. DR instances configured with a different value cannot start or decrypt existing data, so verify that this secret is replicated to your DR Region and referenced by your DR launch configuration. Second, if you run TFE in Active/Active mode, external Redis holds the job queue and cache. Account for a Redis equivalent in the DR Region and decide what in-flight job loss is acceptable at failover. Amazon CloudWatch alarms in each Region monitor the TFE instances, Auto Scaling groups, and Aurora clusters in that Region. Detection of a primary Region impairment does not depend on the primary Region itself: the Amazon Route 53 health check shown in Figure 1 probes the TFE endpoint from a globally distributed checker fleet, and alerts publish through Amazon Simple Notification Service (Amazon SNS) topics in both Regions.

Failover process

The architecture uses a four-step failover sequence:

1. Activate DR Auto Scaling group (2-5 minutes). Scale from 0 to 1 instance and validate health checks. TFE exposes a health check endpoint (/_health_check) that returns a 200 OK response when the application is running. The Network Load Balancer target group and the Route 53 health check probe this endpoint to determine instance health.

2. Promote Aurora PostgreSQL-Compatible global database (~1 minute). Promote the DR writer. Complete this step before DNS failover shifts traffic to the DR Region, to help prevent both Regions from accepting writes simultaneously (known as a split-brain scenario in distributed databases).

3. Confirm Amazon Route 53 DNS failover (~60 seconds). The pre-configured Route 53 failover routing policy detects the unhealthy primary endpoint and routes traffic to the DR Region’s Network Load Balancer. This happens in the Route 53 data plane, with no record modifications at failover time.

[Important: Your failover process should not depend on control plane API calls during an event. Modifying Route 53 records to perform failover is a documented anti-pattern because the Route 53 control plane operates from a single Region. Athenahealth avoided this dependency with pre-configured health check-based failover routing. For manually initiated Region switches through a highly available data plane, consider Amazon Application Recovery Controller (ARC) Region switch, which Athenahealth plans to evaluate in a future phase.]

4. Scale out for production load (5-10 minutes). Increase Auto Scaling group capacity while monitoring Amazon CloudWatch.

Note: Run failover scripts from outside the primary Region—for example, from the DR Region, a separate management Region, or a CI/CD system that is not dependent on the primary Region. If your failover automation runs in the primary Region, it might be unreachable during the event you are trying to recover from.

The ordering between Aurora promotion and traffic shift is enforced procedurally rather than by an automated control. The DR Auto Scaling group runs at zero capacity during normal operations, so the DR endpoint cannot pass health checks until an operator executes the runbook, and the runbook sequences promotion ahead of scaling for traffic. For an orchestrated Region switch with explicit sequencing controls, consider Amazon Application Recovery Controller Region switch, which Athenahealth plans to evaluate in a future phase.

Total failover execution time: 12-14 minutes, meeting the established RTO.

Validating with AWS Fault Injection Service

With the multi-Region architecture now in place, we still needed to confirm it would work under real failure conditions. This is where AWS FIS was introduced into the DR workflow. AWS FIS injects controlled failures into your AWS environment that can be used to measure actual recovery times and catch configuration issues before an actual disruption. The following three phases show how Athenahealth validated their architecture, and you can apply the same approach to your TFE deployment as well.

Progressive experiment approach

Rather than testing full regional failover immediately, the team validated resilience in three progressive phases starting with individual compute failures, then database failover, and finally simulated S3 connectivity loss. Each phase built confidence in a specific layer of the architecture before combining them, and each surfaced issues that manual review had missed.

Phase A: Amazon EC2 and Auto Scaling group failure injection

Athenahealth hypothesized that if TFE instances were stopped or Amazon EC2 capacity became unavailable, the Auto Scaling group would launch replacement instances within five minutes without manual intervention. To test this, they ran the following AWS FIS actions: aws:ec2:stop-instances, aws:ec2:asg-insufficient-instance-capacity-error, and Auto Scaling group suspend and resume operations. You can use these same actions to validate your own Auto Scaling group recovery behavior.

The results confirmed the hypothesis. The Auto Scaling group detected failed instances and launched replacements within 2-3 minutes. Network Load Balancer health checks removed failed instances from rotation within 30 seconds.

These experiments also revealed an outdated Amazon Machine Image (AMI) reference in the DR Region’s Auto Scaling group launch template. Athenahealth builds custom AMIs and copies them to the DR Region, but the DR launch template still referenced an older version. This configuration drift only surfaced when AWS FIS forced the Auto Scaling group to launch new instances. If you’re running similar experiments, check the launch template AMI references in both Regions as part of your validation.

AWS FIS console experiments list showing one experiment in the Running state.

Figure 2. FIS experiments list showing experiment EXP5vVgVbYvM7G7CFk in Running state, created July 27, 2026 at 12:29:20 IST.

AWS FIS experiment details page showing the Suspend-ASG action completed and Stop-TFE-Instances running.

Figure 3. FIS experiment details showing template TFE-Primary-Region-Full-Outage, CloudWatch log destination /tfe/lab/fis/logs, Suspend-ASG completed, and Stop-TFE-Instances running.

Amazon EC2 console showing the primary instance in us-east-1 in the stopped state.

Figure 4. Primary EC2 instance in us-east-1 stopped after the FIS stop-instances action.

AWS FIS action summary showing Stop-TFE-Instances completed and Wait-For-Failover-Test running.

Figure 5. FIS action summary showing Stop-TFE-Instances completed at 12:35:17 IST and Wait-For-Failover-Test running.

AWS FIS experiment running during the wait window, with the stop action completed and the resume action pending.

Figure 6. FIS experiment still running during the wait window, with stop action completed and resume action pending.

AWS FIS experiment completed with all four actions showing completed, including Resume-ASG-Launch-via-Automation.

Figure 7. FIS experiment completed at 12:51:00 IST. All four actions show completed, including Resume-ASG-Launch-via-Automation.

Phase B: Aurora PostgreSQL-Compatible database cluster failover

Athenahealth hypothesized that if the Aurora PostgreSQL-Compatible global database cluster failed over, TFE would resume writes within one minute without manual intervention. To test this, they ran the aws:rds:failover-db-cluster AWS FIS action. The results confirmed the hypothesis for the database layer. Aurora promoted the secondary cluster’s writer in 58 seconds. During the promotion, TFE experienced approximately 15 seconds of write unavailability.

The Amazon Relational Database Service (Amazon RDS) Global Endpoint automatically redirected connections to the new writer. Athenahealth also discovered that the application layer did not meet the hypothesis. TFE connection pooling settings caused extended reconnection delays.

They reduced the connection pool timeout from 60 seconds to 10 seconds, which improved recovery time significantly. If you’re running TFE with Aurora PostgreSQL-Compatible, review your connection pool settings as part of your AWS FIS validation.

Note that this experiment exercised the coordinated failover path of Aurora, which requires the primary Region to be reachable to synchronize before promotion. During an actual event impairing the primary Region, you would instead use Aurora Global Database managed failover (the failover-global-cluster command with the --allow-data-loss option) or a manual detach-and-promote. These paths do not wait for replication to synchronize, so promotion timing differs and the RPO is bounded by the replication lag at the time of the event rather than zero. Treat the 58-second promotion and sub-second lag measured here as coordinated-path results, and plan unplanned-path expectations using the Aurora Global Database disaster recovery documentation.

Phase C: Amazon S3 connectivity disruption

Athenahealth hypothesized that if TFE lost connectivity to Amazon S3 in the primary Region, the DR Region bucket would hold current replicated state files without data loss. Testing this required a workaround, because AWS FIS doesn’t provide a direct action to disrupt Amazon S3 access. You can use the aws:network:disrupt-connectivity action instead to inject network ACL rules that block S3 traffic at the subnet level.

The aws:network:disrupt-connectivity action targets subnets, not S3 buckets directly. AWS FIS injects network ACL rules on compute private subnets, which blocks egress traffic to S3 service endpoints and simulates Regional S3 disruption for TFE instances in those subnets.

This experiment validates the S3 consumer, meaning TFE losing access to S3. It does not disrupt S3 cross-Region replication, because service-side replication between buckets does not traverse your subnet network ACLs. To test delayed or paused replication between Regions, use the Cross-Region: Connectivity scenario described in Next steps. Note that this approach assumes your TFE instances reach S3 over an in-VPC path, such as a gateway VPC endpoint, so the injected network ACL rules sit on the egress path to S3.

To configure this AWS FIS experiment, use the following template. Replace <your-tfe-compute-private-subnet-prefix> with your actual subnet name prefix, which you can find in the Amazon VPC console under Subnets.

{
    "actions": {
        "DisruptS3Connectivity": {
            "actionId": "aws:network:disrupt-connectivity",
            "parameters": {
                "duration": "PT10M",
                "scope": "all"
            },
            "targets": {
                "Subnets": "TFE-Compute-Private-Subnets"
            }
        }
    },
    "targets": {
        "TFE-Compute-Private-Subnets": {
            "resourceType": "aws:ec2:subnet",
            "resourceTags": {
                "Name": "<your-tfe-compute-private-subnet-prefix>-*"
            },
            "selectionMode": "ALL"
        }
    }
}

The results confirmed the hypothesis for data durability. TFE detected S3 connectivity loss within 5 seconds. The DR Region S3 bucket contained replicated state files with less than 30 seconds of replication lag, and bidirectional replication prevented state file loss. The experiment also exposed a failure mode Athenahealth had not anticipated: the state file dependency issue detailed in the following section.

Combined experiment: primary Region impairment

After validating each layer individually, the team combined the faults into a single AWS FIS experiment template (shown in Figures 2-7). The experiment suspends the primary Region Auto Scaling group, stops the TFE instances, holds the faults in place during a wait window while the team executed the four-step failover runbook, and then resumes the Auto Scaling group. This end-to-end run validated the complete failover process under simultaneous compute impairment. The combined run surfaced no new failure modes beyond those found in the individual phases, which was itself the confirmation the team wanted.

Measuring recovery times

Across the three AWS FIS experiment phases, Amazon CloudWatch measured the following recovery times:

Amazon EC2 failure recovery: 2-3 minutes (automated Auto Scaling group replacement)
Aurora PostgreSQL-Compatible failover: 1-2 minutes (managed promotion)
Failover execution time: 12-14 minutes (operator-triggered four-step process)
Aurora replication lag: less than 1 second (99th percentile)
S3 replication lag: less than 30 seconds (99th percentile)
Data loss during failover: 0 bytes (across each experiment)

[Note: These measurements reflect controlled testing conditions. Aurora Global Database and S3 cross-Region replication are both asynchronous. During an actual event, writes committed within the replication lag window (sub-second for Aurora, up to 30 seconds for S3) may not yet be available in the DR Region. Plan for near-zero rather than zero data loss when setting RPO expectations.]

Failback RTO: approximately 20 minutes (including approximately 5 minutes for Aurora PostgreSQL-Compatible global database re-establishment)

The 12-14 minutes measure failover execution time, from the operator triggering the runbook to full recovery. End-to-end recovery from event onset also includes detection time and the decision to fail over, so plan for a larger overall RTO.

Lesson learned: the state file dependency pitfall

Athenahealth first identified this risk during production failover and failback testing: their automation scripts depended on Terraform S3 state file outputs from both Regions. Subsequent AWS FIS experiments (Phase C) confirmed the severity. When S3 access is lost, those scripts fail entirely.

How the dependency breaks failover

Athenahealth’s failover and failback scripts automate the four-step process described earlier: scaling the Auto Scaling group in the target Region, promoting the Aurora global database writer, and verifying application health. An operator triggers them as part of the manual failover runbook, and they run from outside the primary Region. In their original form, the scripts retrieved infrastructure identifiers such as the Amazon RDS global cluster ID and Auto Scaling group name from state files stored in Amazon S3:

# Failover script excerpt (problematic approach)
# Retrieve RDS Global Cluster ID from primary Region state file
RDS_GLOBAL_CLUSTER_ID=$(terraform output \
    -state=s3://<primary-region-bucket>/terraform.tfstate \
    rds_global_cluster_id)

# Retrieve DR Auto Scaling group name from primary Region state file
DR_ASG_NAME=$(terraform output \
    -state=s3://<primary-region-bucket>/terraform.tfstate \
    dr_asg_name)

# Run Aurora failover
aws rds failover-global-cluster \
    --global-cluster-identifier $RDS_GLOBAL_CLUSTER_ID \
    --region us-west-2

For an unplanned Regional impairment, add the --allow-data-loss option to this command to perform a managed failover instead of a switchover, because a switchover requires the primary Region to be healthy.

During a Regional service impairment, the primary Region’s S3 state file may be unreachable. The same applies in reverse during failback. The failover script tries to read the state file, S3 times out, and the script stops. This creates a circular dependency: you can’t run the failover without access to the infrastructure you’re trying to recover from.

How to remove the dependency

The underlying principle is to remove every recovery dependency on the Region you are recovering from. Failover automation must not read configuration from the control plane or data plane of the impaired Region. Athenahealth implemented this principle by hardcoding infrastructure identifiers directly in their failover scripts. You can obtain these values from your Terraform outputs during normal operations:

# Failover script excerpt (resilient approach)
# Infrastructure identifiers hardcoded, not from dynamic lookups
RDS_GLOBAL_CLUSTER_ID="<your-tfe-global-cluster>"
DR_ASG_NAME="<your-tfe-dr-asg-us-west-2>"
ROUTE53_HOSTED_ZONE_ID="<your-hosted-zone-id>"

# Run Aurora failover with no state file dependency
aws rds failover-global-cluster \
    --global-cluster-identifier $RDS_GLOBAL_CLUSTER_ID \
    --region us-west-2

The trade-off is maintenance: hardcoded values require manual updates when infrastructure changes. Athenahealth addressed this with a CI/CD pipeline that compares hardcoded values against Terraform outputs and alerts on drift.

Hardcoding is one implementation of the principle. A Region-independent configuration source outside the primary Region achieves the same resilience with less drift risk, such as an AWS Systems Manager Parameter Store parameter replicated across Regions, an Amazon DynamoDB global table, or values committed to the repository that holds your failover scripts.

Testing failback

The state file dependency affected both failover and failback. Athenahealth validated failback by running full failover to DR, operating there for 30 minutes, then returning to primary. Bidirectional S3 replication prevented state file loss.

Collaboration model

If you’re planning a multi-Region DR project for TFE, consider a cross-functional approach. Athenahealth’s five-month engagement combined AWS resilience and AWS FIS expertise, HashiCorp TFE architecture knowledge, Terraform DR best practices, and HVD modules. This combination helped the customer reach production-validated DR faster than working independently. You can engage AWS Support or AWS Professional Services for similar guidance.

Conclusion

You can maintain infrastructure deployment capabilities during events that impact a single Region with a validated multi-Region DR architecture for TFE. This architecture achieved a validated RTO of 12-14 minutes and an RPO of less than 1 minute.

Multi-Region DR is not the right choice for every TFE deployment. Athenahealth chose this approach because TFE manages infrastructure for critical healthcare workloads. The October 2025 event showed that losing the ability to deploy during a regional event was a risk the business could not accept. Costs vary based on your configuration, but Athenahealth observed costs approximately 30-40% higher than their single-Region deployment, primarily from Aurora PostgreSQL-Compatible global database replication and S3 cross-Region replication. Weigh this cost against your own RTO and RPO requirements. For less critical workloads, a single-Region deployment with regular backups and a tested restore process may meet your needs.

Key takeaways

Give your infrastructure as code (IaC) tools the same resilience as production workloads. When your TFE deployment becomes unavailable during a Regional service event, you can’t deploy fixes or recover infrastructure.
Validate DR with controlled failure injection. AWS FIS experiments simulating real S3 connectivity loss exposed the state file circular dependency, a failure mode that only surfaces under actual disruption conditions.
Remove recovery dependencies on the Region you are recovering from. Dynamic lookups from state files tie failover to the infrastructure being recovered. Hardcoded identifiers or a Region-independent configuration source both work. Use drift detection to keep values current.
Test failback, not only failover. Without failback validation, you risk getting stuck in the DR Region or causing data loss when returning to primary.
Use subnet-level network disruption to simulate S3 connectivity disruption. The aws:network:disrupt-connectivity action targeting compute subnets simulates Regional S3 connectivity loss, which is the recommended approach because AWS FIS doesn’t offer a direct S3 disruption action.

Next steps

You can implement this solution in your environment with the following steps:

1. Run Phase A AWS FIS experiments on non-production TFE instances to validate Auto Scaling group recovery.

2. Review the HashiCorp’s Terraform Enterprise Validated Design module and DR guidance.

3. Establish your RTO and RPO targets before designing your Aurora replication strategy.

4. Create your first AWS FIS experiment to validate your DR architecture.

After you validate these three phases, extend your testing with additional AWS FIS scenarios. The AZ Availability: Power Interruption scenario validates recovery from the loss of an Availability Zone. The Cross-Region: Connectivity scenario simulates disrupted network connectivity between Regions, including paused S3 replication, which would delay state file replication to your DR Region.

If you need help designing or validating a multi-Region DR strategy, contact AWS Support or AWS Professional Services.

Cleanup

If you deploy this architecture for testing, delete the following resources in both Regions to avoid ongoing charges:

Aurora PostgreSQL-Compatible global database clusters.
Amazon S3 buckets with cross-Region replication.
Amazon EC2 instances in the DR Region Auto Scaling group.

If you used the sample GitHub repo to set up a test multi-Region TFE environment, verify that you also run terraform destroy to avoid any additional charges.

Resources:

AWS Fault Injection Service documentation.
Amazon Aurora PostgreSQL-Compatible global databases.
Amazon S3 cross-Region replication.
AWS Well-Architected Framework, Reliability Pillar.
Disaster recovery strategies with Terraform
HashiCorp Terraform Enterprise AWS HVD module