Beyond DKIM key rotation: understanding email replay attacks

2026-11-02 Michael Araba

Post Syndicated from Michael Araba original https://aws.amazon.com/blogs/security/beyond-dkim-key-rotation-understanding-email-replay-attacks/

As the email administrator for your organization’s domain, you receive a Domain-based Message Authentication, Reporting, and Conformance (DMARC) aggregate report showing email passing DomainKeys Identified Mail (DKIM) verification as though it came from your domain, sent from IP addresses you don’t recognize to recipients you didn’t contact. To clear the issue, you rotate your DKIM key, confirm the new selector is live, and verify that your DMARC policy is already set to p=reject. Days later, you see that the same unfamiliar traffic is still authenticating as you.

In this post, you learn why rotation alone can’t stop a DKIM replay attack, how to confirm that you’re looking at a replay rather than a legitimate relay or ordinary spoofing, and the DNS cleanup step that most rotation guidance leaves out. You also learn how you can use Amazon Simple Email Service (Amazon SES) to detect and respond to this activity faster.

How DKIM replay works

DKIM verifies one thing: that specific parts of a message didn’t change between the signer and the receiver. The signer signs the message with a private key, and receivers fetch the matching public key from your domain’s DNS to check the signature. A valid signature continues to validate wherever the message travels afterward. Mailing lists, .forward rules, and third-party relays all preserve it, which is the protocol working as designed.

A replay attack takes advantage of that durability. The attack works as follows:

Your ecommerce platform sends a customer a legitimate response email, signed with your DKIM key.
An attacker obtains one copy of that message. They might subscribe to your mailing list, compromise a mailbox, retrieve it from a list archive, or capture it through a forwarding rule.
The attacker resends that message, unchanged, from their own servers to thousands of new recipients.
Each receiving server fetches your public key from DNS and checks the signature. The signature passes, because the body wasn’t modified. Sender Policy Framework (SPF) fails, because the sending IP isn’t yours, but DMARC requires only one of SPF or DKIM to pass with alignment. Because DKIM passes, DMARC passes, and the message is delivered as authenticated email from your domain.

Understanding what an attacker can and can’t do matters, because the limits define the problem. An attacker can’t modify your message, add new content, or read your email. The signature is only valid while the message remains byte-for-byte identical.

What an attacker gains is your domain’s reputation, which they use to deliver your genuine message to an audience you didn’t intend. A receipt or invoice becomes a lure, and email that would normally be filtered is delivered because receivers trust your domain. This is abuse of reputation and reach, not content forgery, and the distinction matters when you explain the incident to your security team.

This also explains why an enforcing DMARC policy doesn’t stop an attack. DMARC passes when SPF or DKIM passes with alignment. During a replay, DKIM passes, so DMARC passes, and p=reject does not apply. A p=reject policy is designed to stop spoofing, where a forger has no valid signature and DKIM fails. It does nothing against a message that you signed. Determining which of the two was used is what tells you how to respond.

Is it replay, spoofing, or a legitimate relay?

Email from an unfamiliar IP address that passes DKIM has three common explanations, and each requires a different response. Confirm which you’re dealing with before you make any changes.

DMARC aggregate (rua) reports provide the source IP address, message volume, and SPF and DKIM results. This is enough to identify an anomaly, but not to prove a replay. A report row might read as follows:

source IP: 203.0.113.45 | count: 4,200 | SPF: fail | DKIM: pass (d=yourdomain.com)

You need to evaluate that row against the three possibilities:

The first possibility is a legitimate relay that you haven’t inventoried, such as a mailing list, a forwarder, or an email security gateway such as an inline cloud access security broker (CASB) that you route through. The signature validly survives the relay, and this is normal email flow. In the report, the source IP address usually resolves to a known provider, and the volume tracks your actual sending.
The second possibility is replay abuse, in which an attacker redistributes a message that you legitimately signed. In this case, the DKIM-passing volume comes from an IP address that has no business relationship with you, often well above the usual number of recipients that you send to, and frequently to recipient domains that you haven’t worked with.
The third possibility is ordinary spoofing, which is a forgery with no valid signature. That row shows DKIM: fail. DKIM that passes from an unknown IP address is therefore usually not ordinary spoofing.

Aggregate counts can’t prove a replay on their own. To confirm a replay, you need message-level evidence: a specific message that you sent matched to one that reached recipients you didn’t address. If you can’t make that match, treat the activity as an unconfirmed anomaly, and begin with the enforcement and monitoring steps that follow rather than removing keys.

Set up an enforcing DMARC policy first

Before you take any other action, confirm that you’re enforcing DMARC. Many domains that encounter this problem are still at p=none, which is monitoring only. This situation is common. Every DMARC deployment begins at p=none, and many organizations do not advance beyond it because nothing appears to be broken. At p=none, receivers take no action on failures, so no change you make affects delivery, and you can’t determine what enforcement would have caught.

The first step is therefore not key removal; it’s completing your sender inventory and reaching enforcement:

Inventory every legitimate sender and relay for your domain, including your email service providers, mailing lists, and any security gateways that send on your behalf.
Bring each sender into SPF and DKIM alignment so that legitimate email authenticates.
Advance the policy one stage at a time, from p=none to p=quarantine to p=reject. Review your rua reports at each stage before you advance, typically for a week or two per stage, and longer if you have many senders to work through. The pct tag lets you ramp up gradually, for example by quarantining 25% of failing email before you increase the percentage.

After you’re enforcing DMARC, receivers that honor DMARC drop unauthenticated spoofing on your behalf. Only after this baseline is in place, and only after you have confirmed a replay of genuinely signed email, does the next step apply.

Remove the old key from DNS: routine hygiene after rotation

Most rotation guidance omits this step. Rotating a key and removing the old key are two separate actions.

Rotation means that your provider begins signing new email with a new key and selector. Removal means deleting the old public key from DNS, so that previously signed messages, including the copies that an attacker captured, can no longer be verified. If you rotate but leave the old public key published, every message ever signed with that key remains replayable for as long as the record exists. Removing the old key is what closes that window and is the step that’s commonly overlooked.

Removal affects only messages that are still in transit and were signed with the old key. After email is delivered, the receiver no longer needs your key, so we recommend that you leave the old key published long enough for in-flight email to arrive and then remove it. How much of this process you manage yourself depends on your DKIM method, which the following section describes. During a confirmed active replay, closing this window is essential.

How you manage keys depends on the DKIM method that you use with Amazon SES. With Bring Your Own DKIM (BYODKIM), you generate the key pair and publish the public key as a TXT record that you control. You can rotate and remove keys whenever you want, including removing an old key on demand during an incident. As the BYODKIM documentation states, “you can rotate the DKIM keys for your domains as often as you want.”

With Easy DKIM, a feature where Amazon SES generates and manages the keys for you, you publish CNAME records that point to AWS-hosted keys. Because your DNS points to AWS-hosted keys through those CNAME records, Amazon SES manages the underlying key material for you, so you don’t publish or remove individual keys yourself. Unlike BYODKIM, Easy DKIM doesn’t provide an on-demand option to delete a specific key immediately.

If you’re an Easy DKIM sender during a confirmed replay, your defenses are early detection (described in the following section), the duplicate-message throttling that receivers already apply, and, over the longer term, the emerging DKIM2 effort, an IETF initiative to update the DKIM specification with built-in replay resistance, which is being designed to give receivers better signals to identify and act on replayed messages at the protocol level.

How Amazon SES helps you detect and respond

Your first response is detection, not DNS changes. The goal is to identify anomalous DKIM-passing volume early, while you build the message-level evidence that distinguishes a replay from a legitimate relay.

Begin by enabling logging. Create an Amazon SES configuration set with an event destination (see Monitor email sending using Amazon SES event publishing) that streams per-message events (Send, Delivery, Bounce, and Complaint) to Amazon Data Firehose or Amazon EventBridge (see Add an event destination), tagged with the From domain.

Separately, confirm that you’re receiving rua reports for your domain. You enable these reports by adding a rua= address to your domain’s DMARC TXT record, for example rua=mailto:[email protected]. For the record syntax and a phased p=none to p=quarantine to p=reject rollout, see Complying with DMARC authentication protocol in Amazon SES. Reports arrive as daily XML digests, and most teams process them with a DMARC analyzer rather than reading the raw XML.

With both sources in place, you can correlate them. Compare the email that your account emitted—based on your Amazon SES delivery events—against what DMARC reports show that receivers accepted as your domain. When DMARC shows DKIM-passing volume from IP addresses outside your SPF record that exceeds the delivery events that Amazon SES recorded for that From domain in the same window, that gap is your signal. It’s also how you build the message-level case that distinguishes a replay from a legitimate relay.

We recommend three additional controls. Amazon SES sending authorization restricts which AWS principals can send using your verified identity. It doesn’t stop an external replay, but it addresses the insider and third-party vector that SPF, DKIM, and DMARC don’t cover on their own, so use identity-level policies to limit sending to specific AWS Identity and Access Management (IAM) principals.

Use Amazon Route 53 to set short DNS time-to-live values (TTLs) (for example, 300 seconds) on your DKIM records so that changes propagate quickly; and enable DNS query logging with an Amazon CloudWatch Logs metric filter that watches for queries to a selector that you’ve rotated away from.

A spike in lookups for an old selector is a useful early indicator of a replay attempt using a retired selector. For a BYODKIM domain during a confirmed replay, Route 53 is also where you remove the old selector’s TXT record. You can automate that removal by using an Amazon CloudWatch alarm that starts an AWS Lambda function to call the Route 53 API and retire an old selector’s record.

Scope the function’s IAM role to only the specific Route 53 hosted zone and record that it needs to change, so that it follows least privilege. Because an incorrect DNS change can have a wide effect, gate any automation behind confirmed-replay criteria and test it against a non-production domain first. For most teams, alarm-to-human is safer than alarm-to-automatic-removal.

When your domain is the one being abused

Most guidance assumes that you’re either the provider preventing abuse or the receiver detecting it. In a replay, your domain is the one whose reputation is being consumed, and you aren’t on the recipient list. As a result, you don’t have the message that everyone asks for when you report the issue.

When you file an abuse report with the provider that hosts the source IP address, they ask for full message headers. The email went to people you didn’t contact, so you don’t have the message headers. You then provide DMARC aggregate reports that show the source IP address and volume, and the provider asks for headers again. At this point, you’re unable to proceed.

To break this cycle, go to the recipients. Review your rua reports for the top recipient domains—which are often .gov and .edu domains—contact those recipients directly and request a sample message with full headers. In addition to providing the headers, that sample lets you confirm a replay rather than a legitimate relay. Build your abuse report around the IP address and the DMARC evidence rather than the message content. If the source IP address belongs to Amazon Web Services (AWS), report it through the AWS abuse process and include your DMARC reports and SPF record as evidence.

Incident response: ordered actions

If your reports show suspicious DKIM-passing email from unfamiliar IP addresses, complete the following procedure.

To respond to a suspected DKIM replay

Confirm that it’s a replay. Match a message that you sent to one that reached recipients you didn’t address and rule out legitimate relays and gateways. Aggregate data alone isn’t proof.
Reach an enforcing DMARC policy. Inventory your legitimate senders, align them, and advance from p=none to p=quarantine to p=reject. This stops spoofing.
Harden SPF to -all rather than ~all, but only after your sender inventory is complete, since -all hard-fails any legitimate source you haven’t listed.
Check the DKIM-Signature header of a fresh message to confirm that your provider signs only with the current selector.
Report the source IP address to the hosting provider and include your DMARC evidence.
Investigate the capture point, such as a compromised mailbox, a mailing list, a forwarding rule, or a newsletter subscription.
Remove the old DKIM public key after in-flight email has cleared. For BYODKIM, delete the old selector’s TXT record. For Easy DKIM, rely on the managed Amazon SES rotation together with the monitoring and receiver-side defenses described earlier.

The following table summarizes what each action does and doesn’t do.

Action	What it does	What it doesn’t do
Confirm replay (message-level)	Separates a replay from a legitimate relay or spoofing	Can’t be done from aggregate DMARC data alone
DMARC enforcement (p=none to p=reject)	Drops unauthenticated spoofing	Does nothing against a replay because DKIM still passes
Key rotation	Signs new email with a new key	Doesn’t invalidate already signed messages
Remove old key (routine, post-rotation)	Invalidates old-key signatures, including captured ones	Doesn’t protect in-flight old-key email; on-demand removal available only with BYODKIM
Amazon SES sending authorization	Restricts which principals send as your identity	Doesn’t stop external replay of captured messages

Conclusion

DKIM replay succeeds because a valid signature keeps validating, and an enforcing DMARC policy doesn’t change that. To respond effectively, confirm that the activity is a replay with message-level evidence, reach an enforcing DMARC policy, and then remove the old key from DNS as routine post-rotation hygiene. Keep your primary defense on monitoring and DMARC enforcement rather than on emergency DNS changes. The durable fix for replay is receiver-side: duplicate-message counting today, and stronger protocol-level signals through DKIM2 in the future.

To get started, review your current DMARC policy and confirm that you’re receiving rua reports, enable Amazon SES event publishing, and document how you rotate and remove DKIM keys for each of your domains.

Learn more

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

[$] The second part of the 7.4 merge window

2026-11-02 daroc

Post Syndicated from daroc original https://lwn.net/Articles/1102739/

The 7.4 merge window, which
closed on November 1,
included a record-breaking 18,591 non-merge changesets.
That is almost as many as the entire 7.3 development cycle, which had 18,765, itself a record breaker. Of
those, 11,838 landed after the article covering the
first part of the merge window. Most of those were bug fixes of
various forms; it seems as though bug fixes may be crowding out feature
additions across the open-source ecosystem to
some extent. Still, a good number of new features were included, and a handful
removed, in the tail end of the merge window.

How to run a DR readiness engagement

2026-11-02 Peter Garnepudi

Post Syndicated from Peter Garnepudi original https://aws.amazon.com/blogs/architecture/how-to-run-a-dr-readiness-engagement/

Part 3 of 3

The framework from Part 1 (three business decisions) and Part 2 (progressive validation) works at any pace. You can apply it over months as part of normal operations, or compress it into a structured engagement that moves from undefined targets to a validated recovery commitment in weeks. This post walks through how to run that engagement yourself, with concrete steps, outputs, and a before-and-after example from a real enterprise engagement.

Step 1: Kickoff (define the scope)

Bring three groups into one room: the application owner who knows the technical architecture, the business owner who understands the consequence of downtime, and the platform team that owns the shared services your application depends on. If a dependency is owned by a team not represented in the room, that’s your first action item before proceeding.

Ask these questions in order:

What does the user need to be able to do for this function to count as “recovered”?
Which applications, shared services, and accounts are in the path of that user journey?
Who owns each piece, and have they confirmed their recovery timeline?

The output artifact is a boundary map: a document listing every service and account in the recovery path, with an owner assigned to each and their stated recovery timeline recorded. This single artifact becomes the foundation for every subsequent step.

Step 2: Assessment (surface the gaps)

Run a structured review of each component in the boundary map. For each dependency, ask: does it recover in the target AWS Region, does it recover within the stated recovery time objective (RTO), and does an upstream service’s recovery target cap the RTO you can actually commit to?

A Well-Architected Framework Review against the Reliability pillar gives you a systematic way to surface single points of failure, missing health checks, and unverified quotas.
An Operational Readiness Review checks whether the runbook’s assumptions hold: are the resources pre-provisioned, do upstream dependencies recover within the target, and are there any manual steps that depend on a specific person being available?

The output is a prioritized list of dependency gaps with owners assigned and a target fix date for each. “Done” at this step means every gap has an owner, a planned fix, and a committed date. Gaps without owners don’t get fixed.

Step 3: Validation plan (structure the tests)

Build a test calendar that progresses through the levels from Part 2. A practical sequence for a first engagement:

Week 2: Tabletop exercise on the highest-consequence function. Confirm ownership, escalation path, and expected duration per step.
Week 4-6: Planned failover after gap remediation. Measure actual recovery time against the committed RTO. Verify data consistency against the recovery point objective (RPO).
Week 8-10: Surprise drill (if the planned failover passed). Test coordination without advance notice.

Don’t schedule a game day with fault injection until you’ve passed a planned failover cleanly. Each level builds confidence for the next. Attempting a higher level before the lower one passes cleanly wastes effort and produces noisy findings that obscure the real gaps.

Step 4: Execute and learn (close the loop)

Each test produces findings. Document them in a format that maps each finding back to the decision it informs:

Boundary findings (a shared service discovered mid-test that wasn’t in the recovery scope) → feed back into Decision 1.
Consequence findings (recovery took longer than the business can tolerate) → feed back into Decision 2, where the business owner either accepts the longer timeline or invests in a faster architecture.
Dependency findings (an upstream service whose recovery target capped the application’s own, a resource that wasn’t pre-provisioned) → feed back into Decision 3.

The output is an updated recovery commitment that reflects what you actually measured, not what you assumed. This document replaces the prior version and becomes the baseline for the next cycle.

Step 5: Ongoing cadence (prevent drift)

Recovery posture drifts as applications change. A new microservice adds a dependency nobody mapped. A team deploys a configuration change that shifts traffic routing. A quota that covered peak load 6 months ago no longer does. Schedule a quarterly review that checks: has the boundary changed (new shared services, new accounts)? Has the business owner confirmed they still accept the RTO/RPO targets? Have any dependencies changed their recovery behavior?

A trigger for re-engagement outside the quarterly cycle: any architecture change that adds a cross-account dependency or changes the failover path.

What this looks like in practice

In the enterprise engagement referenced throughout this series, the team applied this pattern across its portfolio of applications and accounts. The first exercise missed a contractual recovery commitment of 5 hours, taking more than twice that long. Root causes mapped to all three decisions: the boundary excluded a shared authentication platform (most downstream applications couldn’t authenticate users), the dependency chain included an upstream service whose recovery target capped what the applications calling it could achieve, disaster recovery (DR) resources weren’t pre-provisioned, and DNS failover depended on a manual step that took over 3 hours under coordination pressure.

Over 4 months, the team redefined the boundary to include shared platforms, documented the full dependency chain per application, reconciled each application’s target against its slowest dependency, pre-provisioned the recovery environment, and replaced manual DNS failover with automated routing built on Amazon Application Recovery Controller (ARC) routing controls. ARC routing controls are on/off switches on a data plane that shifts traffic without depending on the Amazon Route 53 control plane, so a cutover doesn’t rely on modifying DNS records mid-event. This follows the Well-Architected guidance to rely on the data plane, not the control plane, during recovery (REL11-BP04), the same static-stability principle behind pre-provisioning. The second exercise passed. The organization operated a live customer-facing system from the recovery Region for the first time. Senior leadership cited the structured engagement as the factor that connected the two different outcomes.

Accelerating the timeline

You can run every step in this post with your own teams. If you want to compress the timeline, AWS Enterprise Support provides specific mechanisms that map to each step: a Technical Account Manager to facilitate the kickoff and maintain engagement momentum between sessions, Well-Architected and Operational Readiness Reviews for the assessment, specialist Solutions Architects for deep dependency analysis, and AWS Countdown for coordinated operational-readiness and engineering support during critical events, including planned failovers and DR game days. These are acceleration options, not prerequisites. The steps and outputs are the same either way.

The AWS Resilience Lifecycle Framework and AWS Resilience Analysis Framework provide the detailed practices behind each decision, and AWS Resilience Hub helps you assess applications against your defined RTO and RPO targets.

About the authors

Validating your DR commitment: From controlled test to real-world readiness

2026-11-02 Peter Garnepudi

Post Syndicated from Peter Garnepudi original https://aws.amazon.com/blogs/architecture/validating-your-dr-commitment-from-controlled-test-to-real-world-readiness/

Part 2 of 3

In Part 1 of this series, you defined three business decisions that shape your disaster recovery (DR) commitment: the recovery boundary, the business-anchored target, and the dependency-reconciled commitment. Those decisions give you a target on paper. Testing tells you whether it holds under real conditions.

This post covers a progressive approach to DR validation. Each level of testing surfaces a different class of gap, and each gap feeds back into the three decisions from Part 1 so your commitment gets stronger with every cycle.

Why a single test isn’t enough

A scheduled DR exercise with advance notice proves that the failover mechanism works. That’s a necessary starting point. You can confirm that traffic shifts correctly, data remains consistent in the recovery Region, and the application functions as expected after the transition.

A scheduled exercise doesn’t test the assumptions that only surface under pressure. It doesn’t show whether teams who haven’t rehearsed together can coordinate, or whether everyone agrees on who declares the failover and under what conditions. It also can’t confirm that you provisioned and validated your recovery environment before you triggered the failover. These are the conditions where real recoveries stall, and they’re invisible to a test where every team knows it’s coming and watches closely.

In the same engagement from Part 1, several applications had DR instances in a stopped state. The failover automation assumed it could launch instances, create database replicas, and scale container fleets on demand during the exercise. That approach worked only if the team confirmed two things in advance: that the recovery AWS Region had capacity for the requested instance types, and that the account quotas could absorb the full production load. The team didn’t verify these assumptions in advance.

The architecture was correct. The team didn’t verify those assumptions in advance, and it didn’t pre-provision the recovery environment. To close this gap, the team needs to pre-provision the recovery environment and configure auto scaling groups, confirm that its account quotas match production capacity requirements, and remove any runbook step that assumes your team needs to create resources during the event.

Four levels of progressive validation

The AWS Resilience Lifecycle Framework treats testing as a practice you deepen over time. The four levels below are a maturity model you can apply yourself, ordered from lowest-cost awareness to highest-fidelity validation. We’ve drawn it from patterns across the DR exercises we’ve seen. Each surfaces a different class of gap.

Level 1: Tabletop exercise. A facilitated walkthrough of the recovery plan without touching production. The goal is shared awareness and ownership clarity.

Questions to ask during the tabletop:

Who declares the failover, and under what conditions?
For each step in the runbook, who owns it and what’s their backup?
What’s the expected duration of each step, and what’s the total end-to-end timeline?
Which steps depend on a specific person being available?

In the same case study from Part 1, the first tabletop surfaced three boundary gaps: the shared authentication platform wasn’t in the recovery scope, a networking account owned by a different team had no failover runbook, and the DNS update step listed a generic team alias with no named owner.

Level 2: Planned failover. A scheduled failover with advance notice. The goal is confirming the mechanism works under known conditions.

What to measure:

Actual recovery time per application compared to the stated recovery time objective (RTO).
Data consistency validation in the recovery Region to confirm the recovery point objective (RPO).
Any step that requires manual intervention (potential automation candidate).
Any dependency that recovered later than expected.

In the same case study, the planned failover revealed the DNS misconfiguration: the shared platform’s Amazon Route 53 records still resolved to the primary Region 45 minutes into the test. Most downstream applications appeared healthy but couldn’t serve users. Actual recovery took more than twice the 5-hour target.

A successful planned failover produces a measured gap (or confirmation) between the RTO target you defined and your actual recovery time. That number is the starting point for every subsequent conversation with the business owner about whether the target is realistic.

Level 3: Surprise drill. A failover triggered without advance notice to the responding team. The goal is testing execution readiness, not capability.

This level finds the gaps that preparation hides: the runbook that only its original author can interpret, the step that depends on a specific person’s tribal knowledge, and the escalation path that nobody has practiced. In the case study, a later surprise drill revealed that only one engineer understood how to validate database consistency in the recovery Region. With that person unavailable, the team lost 90 minutes locating and interpreting the procedure.

Level 4: Game day with fault injection. An exercise where you inject specific dependency failures using AWS Fault Injection Service, which ships purpose-built scenarios such as “AZ Availability: Power Interruption” and “Cross-Region: Connectivity”. The goal is testing resilience under conditions you didn’t fully script.

You discover the dependency you assumed would be healthy, the monitoring gap that delays detection, and the cascade that your tabletop didn’t model. This is the closest approximation to a real event without waiting for one to happen.

What each level reveals

The tabletop finds the step no team owns. The planned failover finds the misconfiguration. The surprise drill finds the coordination failure. The game day finds the unmodeled dependency. Each level surfaces a different class of gap.

Run only the calm, scheduled test and you validate the mechanism while leaving the conditions that cause real recovery failures untested. The point isn’t to skip levels. You build confidence at each level before increasing realism. But stopping at the planned failover, a pattern we often see once the mechanism works, leaves team, process, and dependency gaps invisible until an event exposes them.

Closing the loop

Each test produces specific findings that map directly to the three decisions from Part 1.

A boundary gap (a shared service discovered mid-test that wasn’t in the recovery scope) feeds back into Decision 1.
A recovery that exceeds the time the business can tolerate feeds back into Decision 2, where the business owner either accepts the longer timeline or invests in a faster architecture.
A dependency that recovered slower than the target assumed feeds back into Decision 3 for a revised commitment that reflects what was actually measured.

In the same engagement, the first exercise surfaced all four patterns described across this series: a boundary gap (shared platform not in scope), a dependency gap (an upstream service whose recovery target capped the application’s own), a stability gap (resources not pre-provisioned), and a coordination gap (manual DNS failover). The team mapped each finding to a specific decision and a specific fix. Four months later, the second exercise passed. The organization operated a live customer-facing system from the recovery Region for the first time.

The value of progressive testing is not passing a single exercise. It’s building a practice where each cycle tightens the recovery commitment, and each finding has a clear path back to the decision it informs.

Where to start

If you haven’t run a tabletop yet, start there. Gather the owners of one critical business function and walk the recovery plan together. Ask: who declares the failover, what’s the first action, who owns each step, and what’s the expected duration of each. If the room disagrees with any answer, this indicates an opportunity to clarify and strengthen your recovery plan before touching production.

If you’ve passed a planned failover, consider whether a surprise drill would surface coordination gaps that preparation masks. In this engagement, the surprise drill cost the team 90 minutes that the planned failover never surfaced. Whatever level you’re at, document what each test reveals and feed it back into the three decisions. A posture that improves every test cycle is worth more than one that passes a single exercise and never evolves. In Part 3, we walk through how to structure a full DR readiness engagement: who to involve, what to produce at each step, and how to compress the timeline from months to weeks.

About the authors

Three business decisions that define your DR success

2026-11-02 Peter Garnepudi

Post Syndicated from Peter Garnepudi original https://aws.amazon.com/blogs/architecture/three-business-decisions-that-define-your-dr-success/

Part 1 of 3

A disaster recovery (DR) strategy is only as strong as the business decisions behind it. The architecture and failover mechanisms matter, but they execute against targets that you define. In our experience facilitating enterprise DR exercises, the difference between teams that meet their recovery targets and teams that don’t often comes down to three decisions made (or deferred) before the first architecture diagram: what you’re actually recovering, what the business needs from that recovery, and what your dependencies allow you to commit to.

This post helps you validate whether your current approach accounts for each one. It draws on a pattern from a real engagement where revisiting these decisions turned a missed recovery target into a successful test four months later.

This is Part 1 of a three-part series. It walks through those three decisions and shows how to make them before you choose an architecture pattern. Part 2 covers progressive validation testing. Part 3 covers how to run a DR readiness engagement. The approach builds on the AWS Resilience Lifecycle Framework and the AWS Resilience Analysis Framework.

Define the recovery boundary

The first decision is drawing the line around what “recovered” means. You cannot meaningfully define a recovery time objective (RTO) until you specify precisely what you need working for you to consider that objective met.

The Resilience Lifecycle Framework defines a business function as the end-to-end capability a user depends on (completing a transaction, filing a report, accessing an account). A single business function often spans multiple applications, accounts, and teams. When you set a recovery target at the application level without accounting for the shared services that the application calls, you might define the boundary too narrowly to deliver the capabilities your users require.

Here’s a pattern we observed in a DR exercise for a large enterprise customer. The customer tested dozens of applications across many AWS accounts. Each application had its own failover automation, and each one failed over correctly. But a shared authentication platform (running in a separate account, managed by a different team) used Amazon Route 53 simple routing pointed at the primary AWS Region. It had no health checks and no failover configuration. Because the team didn’t include it in the recovery scope, the platform remained in the primary Region while everything else moved. The result: Most of the downstream applications that depended on it couldn’t authenticate users. Although the team had successfully failed over the applications themselves, the business function (user login, transaction, confirmation) did not work. This single boundary gap, not the individual application architectures, was the primary driver of the missed recovery target.

The solution involved redefining the recovery scope rather than changing the architecture. The organization redefined recovery boundaries around business functions (what capabilities must you deliver to your users?) rather than individual applications (is this service running?). That single change pulled the shared authentication platform inside the scope and helped the team discover the dependency. A business impact analysis is the structured starting point for this work.

Anchor the target to business consequence

The second decision is settling what recovery means in terms the business owns. Business stakeholders strengthen RTO and recovery point objective (RPO) when they help set them. In organizations where these targets are defined by engineering alone, they can become disconnected from what the business actually tolerates. A 4-hour RTO means something different for a payments platform that clears interbank settlement than it does for an internal reporting dashboard, even when both share the same database engine and replication setup.

In the same engagement, the business carried a contractual recovery commitment of 5 hours. The business treated it as a firm commitment to their regulator. The engineering team treated it as a stretch goal based on the architecture’s theoretical capability. Neither side had validated whether 5 hours was achievable given the dependency chain. That disconnect only became visible when the first exercise took 9–11 hours. The target had never been tested against the actual recovery path, and the business stakeholder who owned the penalty had never been asked whether the engineering team’s assumptions matched theirs.

RPO is a statement about data loss. For a payments platform clearing interbank settlement, an RPO of 5 minutes might still be too wide if those minutes contain transactions that can’t be reconstructed. The question for the business owner is: Which records must survive a Regional impairment, and can you prove they will? The business owner uses that framing to decide whether asynchronous replication (with its window of in-flight data loss) is acceptable. The alternative is synchronous writes, with the latency cost that comes with them.

RTO is a statement about downtime tolerance. A 4-hour recovery time is a technical figure until someone attaches a consequence to each hour. For regulatory filings, the fourth hour might mean a missed deadline with financial penalties. For a storefront, every minute carries lost orders.

The AWS Resilience Lifecycle Framework recommends defining objectives around business outcomes directly (for example, the percentage of successful orders maintained when a component degrades). A service-level agreement (SLA) is a contract with a financial penalty. A service-level objective is a target you manage toward. Raising an SLA doesn’t make an application more resilient.

The output of this decision is a recovery requirement stated in language the business recognizes and engineering can build against: the function, the consequence of losing it, the data that must survive, and the tolerable downtime before the consequence lands.

Account for the dependency chain

The third decision is confirming that your recovery target accounts for everything in the critical path, not just the components you own. Your application recovers only as fast as its slowest dependency. If a service you call takes 4 hours to recover in the recovery Region, your achievable RTO is 4 hours regardless of your own architecture.

In the same engagement, one customer-facing application had committed to a 1-hour recovery target and had the architecture to meet it on its own. But it called a shared identity service, owned by a different team, to validate every user session. That identity service carried a 4-hour recovery target. No matter how fast the customer-facing application recovered, users couldn’t do anything until the identity service came back. The application’s real, achievable RTO was 4 hours, set entirely by a dependency its own team didn’t own and hadn’t accounted for when they committed to 1 hour.

This is why the dependency chain, not the application in isolation, defines the commitment you can make. Identify and document every service your business function calls, note the recovery target each one holds, and set your own target against the slowest link you depend on. Where another team owns a dependency, its recovery target becomes a commitment you need from them, in writing, not an assumption. The AWS Fault Isolation Boundaries whitepaper helps you reason about which dependencies hold during a Regional impairment, and which introduce risk. The distinction that matters is what your runbook asks a service to do. Reading existing data keeps working during an impairment: existing credentials still authenticate, and existing DNS records still resolve. Changing things is what carries risk, because creating or modifying AWS Identity and Access Management (IAM) roles and Route 53 records runs through control planes that live in a single Region. A recovery path that only reads existing state holds up. One that must create or change resources mid-event might not. That’s the case for settling those changes ahead of time.

Where you can, reduce the chain itself. When you pre-provision and run a dependency in the recovery Region before the event, you reduce how many services need to recover before you can restore your application. The shorter the chain of things that must recover in sequence, the closer your achievable target gets to the one you want to commit to.

Where to start

Start your validation by examining one business function. Assess whether: Is the recovery boundary drawn around the full user journey, including shared services in other accounts? Has a business stakeholder confirmed the consequence-per-hour attached to the current RTO and RPO? Does the target account for every dependency in the path, including global endpoints and any runbook step that assumes resource creation during failover?

If you can answer yes to all three with evidence, your targets are well-founded. If any answer is uncertain, you’ve identified where to focus next. In Part 2, we cover how to validate your answers through progressive testing that closes the distance between a controlled exercise and real-world readiness.

About the authors

Automating the full AWS DRS agent lifecycle: Install, heal, and offboard

2026-11-02 Manisha Yadav

Post Syndicated from Manisha Yadav original https://aws.amazon.com/blogs/architecture/automating-the-full-aws-drs-agent-lifecycle-install-heal-and-offboard/

If you protect a large Amazon Elastic Compute Cloud (Amazon EC2) fleet with AWS Elastic Disaster Recovery (AWS DRS), you manage the AWS Replication Agent across its whole lifecycle. You install it, keep it healthy and registered, and remove it cleanly when a server leaves disaster recovery (DR) scope. An enterprise IT customer with more than 300 Linux and Windows instances in DR scope ran into two of these problems while planning and validating a scheduled DR drill. Some agents had stalled after a network change, and some descoped servers were still replicating and adding to the bill. A server with a stalled agent can’t meet your recovery point objective (RPO), the maximum data loss you can accept after a disruption. The customer’s team checked replication status in the AWS DRS console before each drill, so a problem that started between drills could go unnoticed until the next one.

Working with that customer, we automated the install, heal, and offboard phases of the lifecycle. Most of the design choices in this post came from fixing those two problems. Each phase runs as an AWS Systems Manager (SSM) Automation document, started by Amazon EventBridge or on a schedule. A single EC2 tag decides which phase applies to an instance. Because the logic lives in the Automation documents, there are no AWS Lambda functions to maintain. When you take an instance out of scope, the automation records whether replication billing actually stopped.

You can find the SSM Automation documents, AWS Identity and Access Management (IAM) policies, and deployment scripts in the aws-samples/sample-drs-agent-lifecycle-automation GitHub repository.

Solution overview

You apply a DR tag to each EC2 instance, and its value (yes, pause, or no) tells the automation what to do. Three SSM Automation documents divide the work. Onboarding installs and repairs the agent and keeps registration healthy. Healing responds when replication stalls, and offboarding runs teardown when you descope an instance.

AWS Systems Manager State Manager runs onboarding on a 30-minute schedule against tagged instances. Amazon EventBridge triggers healing when AWS DRS reports a stalled server, and offboarding when the DR tag changes. The automation runs primarily in your source AWS Region and calls the DR Region only where AWS DRS requires it.

Prerequisites

To deploy this solution, you need:

An AWS account with administrative access.
Two AWS Regions configured: a source Region (where your EC2 instances run) and a DR Region (where AWS DRS replicates to).
AWS Elastic Disaster Recovery initialized in both the source Region (for failback) and the DR Region (for replication).
Network connectivity between Regions with TCP 443 (AWS DRS API and Amazon Simple Storage Service (Amazon S3)) and TCP 1500 (replication data) open. For private connectivity, use AWS Transit Gateway or VPC peering. Public endpoints are also supported.
(Optional) Amazon Virtual Private Cloud (Amazon VPC) endpoints for AWS DRS and Amazon S3 in the DR Region, if using private connectivity.
IAM permissions to deploy AWS CloudFormation stacks that create named IAM roles (CAPABILITY_NAMED_IAM), Amazon EventBridge rules, and a State Manager association, and to register SSM Automation documents. You also need iam:PassRole on DRS-Automation-Role, because the Amazon EventBridge rules and the State Manager association use that role.
AWS Command Line Interface (AWS CLI) installed and configured.
An instance profile on each source EC2 instance with the AWSElasticDisasterRecoveryAgentInstallationPolicy and AmazonSSMManagedInstanceCore managed policies, so that each instance is managed by Systems Manager. If your instances already have an instance profile, attach both policies to its role.
An existing Amazon Simple Notification Service (Amazon SNS) topic for notifications.
(Optional) An existing Amazon S3 bucket for audit reports.
Git, Bash 4 or later, and yq. The deployment scripts use yq to read config.yaml. macOS includes Bash 3.2, so install a newer version, for example with Homebrew.

You don’t need to create the automation’s IAM roles yourself. The CloudFormation templates create DRS-Automation-Role (which runs every lifecycle action) and DRS-EventBridge-Forward-Role (deployed with the DR Region stack), along with the Amazon EventBridge rules in both Regions and the State Manager association. Amazon CloudWatch metrics are published automatically.

One tag controls the entire lifecycle

After you deploy the solution, the value of the DR tag on each EC2 instance determines what the automation does:

DR value	State	What happens
yes	Protected	Agent is installed, failback tag applied, health maintained every 30 minutes
pause	Quiesced	Agent stays installed, stalled alerts suppressed, offboarding blocked
no or deleted	Descoped	Replication stopped, source server deleted, agent uninstalled
Anything else (for example, `DR=maybe`, `DR=true`)	Misconfigured	Fail-safe: automation skips the instance and logs a `SKIPPED_MISCONFIGURED` status. No install, no heal, no teardown, preventing an unrecognized value from triggering the wrong action

The tag value is normalized (trimmed and lowercased) before evaluation. A value like DR=Yes normalizes to DR=yes and matches correctly.

Three SSM Automation documents divide the lifecycle:

DRS-Agent-Onboard runs every 30 minutes through State Manager against tag:DR=yes instances. It installs or repairs the agent, applies the AWSDRS:AllowLaunchingIntoThisInstance tag, and validates AWS DRS registration. The tag tells AWS DRS to launch recovery onto the original instance during failback and acts as the drift-correction backstop. State Manager re-evaluates tagged instances every 30 minutes. Each run catches instances that lost the Replication Agent, newly tagged instances that aren’t onboarded yet, and earlier runs that failed silently.
DRS-Agent-Heal fires when Amazon EventBridge receives a replication-stalled event (cross-Region forwarded from the DR Region). The rule filters on the STALLED state, so recovery transitions don’t trigger unnecessary runs. It inspects the replication error to distinguish agent-side failures from infrastructure issues. For agent problems, it restarts the service first, then reinstalls in place only if the restart doesn’t fix it. For infrastructure-caused stalls, it escalates to operators through Amazon SNS.
DRS-Agent-Offboard fires when Amazon EventBridge detects a DR tag change or deletion. The offboarding document first verifies safeguards: a tag re-check, a terminated-instance guard, and an optional approval gate. After those pass, it records replication state and stops replication. It then disconnects the source server, deletes it, uninstalls the Replication Agent, and removes the tags.

How the components connect

The solution operates primarily in the source Region. Cross-Region calls happen only where AWS DRS requires them (registration checks and teardown API calls target the DR Region).

The following diagram shows how the automation components connect across the source and DR Regions. State Manager and Amazon EventBridge operate in the source Region; AWS DRS replication and the stall-forwarding rule run in the DR Region.

State Manager and EventBridge in the source Region connecting to AWS DRS replication in the DR Region

Figure 1: AWS DRS agent lifecycle automation architecture across the source and DR Regions

In the source Region, State Manager runs the onboarding document every 30 minutes, and Amazon EventBridge rules start the heal and offboarding documents when their events arrive. AWS Systems Manager Parameter Store holds the DR-event stand-down flag (/drs/control/event-in-progress). The DRS-Automation-Role IAM role runs every automation step. Amazon SNS delivers alerts, Amazon S3 stores per-event audit reports, and Amazon CloudWatch captures outcome metrics.

In the DR Region, AWS DRS manages source server registration and ongoing replication. AWS DRS emits replication events only in the DR Region, where the source servers are registered. An Amazon EventBridge forwarding rule filters replication-stalled events by the STALLED state. It then sends them to the source Region for healing. Staging subnets and Amazon Elastic Block Store (Amazon EBS) volumes, managed by AWS DRS, support the replication infrastructure.

For networking, AWS Transit Gateway (which connects VPCs across Regions) carries TCP 443 (AWS DRS API and S3) and TCP 1500 (replication data) between Regions. Amazon VPC endpoints for AWS DRS and S3 keep traffic on the AWS backbone without internet exposure. An Amazon Route 53 Private Hosted Zone (for DNS resolution within your VPC) resolves endpoint names to private IPs. This removes the need for --endpoint installer flags and protects failback compatibility.

Tag and rule examples

Run these commands after you deploy the solution. Tag an instance for protection:

aws ec2 create-tags --resources i-0abc123def456 --tags Key=DR,Value=yes

Descope an instance (triggers offboarding):

aws ec2 create-tags --resources i-0abc123def456 --tags Key=DR,Value=no

Or remove the tag entirely (also triggers offboarding):

aws ec2 delete-tags --resources i-0abc123def456 --tags Key=DR

The Amazon EventBridge rule that triggers offboarding matches this pattern:

{
  "source": ["aws.tag"],
  "detail-type": ["Tag Change on Resource"],
  "detail": {
    "changed-tag-keys": ["DR"],
    "service": ["ec2"],
    "resource-type": ["instance"]
  }
}

The Amazon EventBridge rule that triggers healing filters on the STALLED state so recovery transitions are ignored:

{
  "source": ["aws.drs"],
  "detail-type": ["DRS Source Server Data Replication Stalled Change"],
  "detail": {
    "state": ["STALLED"]
  }
}

Three layers protecting the RPO

The 15-minute RPO referenced throughout this post is a customer-configured target, not an AWS service guarantee. The architecture helps protect it with three complementary layers, each covering a different failure mode.

Layer 1: OS service auto-restart (seconds). The agent service uses systemd Restart=always on Linux and Windows Service Recovery. When the process stops, the OS brings it back automatically. On Linux, systemd restarts the service almost instantly. On Windows, the first recovery action fires after a configurable delay (30 seconds by default). This layer handles agent process crashes without any automation running.

Layer 2: Amazon EventBridge heal (minutes, not seconds). If the agent disappears or can’t maintain registration, AWS DRS detects the stall and emits a replication-stalled event in the DR Region. A forwarding rule filters on the STALLED state and sends it cross-Region to the source Region’s event bus. A matching receive rule triggers the heal document.

The heal document restarts the agent first (most stalls are transient). If restart fails, it re-runs the installer in place, preserving driver state to avoid a full-disk rescan. The restart path is the faster of the two, and total time depends on your environment and on AWS DRS stall-detection latency. The reinstall path takes longer because the agent must re-establish block-level tracking. Stall-detection latency also counts against the window, because lag must accumulate before the service marks a server as stalled.

Layer 3: State Manager backstop (up to 30 minutes). The onboarding document re-evaluates DR=yes instances every 30 minutes, catching drift and edge cases that faster layers miss. This exceeds the 15-minute RPO target, so it functions as a safety net rather than the primary defense.

Layer 2 requires the Amazon EventBridge data plane in both Regions to be operational, while Layers 1 and 3 run entirely in the source Region.

Onboarding: How ongoing convergence works

State Manager targets tag:DR=yes. Instances that already have the tag and instances you tag later are included automatically, without manual registration.

The onboarding document is idempotent. It skips healthy, registered instances. For instances that need work, it checks the stand-down flag and instance state, validates the operating system (Linux or Windows), and flags unsupported architectures as TAGGED_UNSUPPORTED. If AutoInstallEnabled is false, the run is audit-only.

For actual installs, a concurrency lock (flock on Linux, a named mutex on Windows) prevents races when multiple automation runs overlap on the same instance. This lock is built into the documents and needs no configuration. The State Manager association sets MaxConcurrency to 10 and MaxErrors to 100% in templates/state-manager-association.yaml. MaxConcurrency limits how many instances are processed at once. With MaxErrors at 100%, a failure on one instance doesn’t stop the run for the rest of the fleet. For larger fleets, raise MaxConcurrency gradually and watch for AWS DRS API throttling.

The failback tag (AWSDRS:AllowLaunchingIntoThisInstance) is applied only when the agent is confirmed present. Registration is assessed cross-Region, and if an instance stays in PENDING_REGISTRATION beyond a configurable threshold (default: 3 runs), the document escalates through Amazon SNS. Each execution writes an S3 audit record and emits a CloudWatch metric.

Healing agent issues within RPO

The heal document reads the source server ARN from the event and looks up the matching EC2 instance ID with drs:DescribeSourceServers in the DR Region. It then checks the stand-down flag and the tag state. Values other than yes suppress the heal with no action and no alert.

If restart doesn’t resolve the issue, the automation re-runs the installer in place as a fallback. It doesn’t uninstall first, so the existing replication driver stays intact where possible. Reinstallation can trigger a block-level rescan whose duration scales with total disk size. On large volumes, this might temporarily exceed the RPO window while the rescan completes.

Not every stall is an agent problem. Bandwidth throttling, undersized replication servers, or staging-area disk issues can also cause stalls. The automation escalates these non-agent stalls to your operators through Amazon SNS rather than attempting agent remediation. If the Replication Agent is still not functioning after reinstall, you receive a HEAL_FAILED notification with diagnostic details.

Only the AWS DRS agent service is restarted, not the EC2 instance itself. Your workloads keep running without interruption.

Offboarding with honest billing status

When you set DR=no or delete the tag, Amazon EventBridge fires a Tag Change on Resource event. The offboarding document applies several safeguards before teardown.

Terminated-instance guard: EC2 termination strips tags, generating the same event as a deliberate descope. The document sends an alert and stops. It doesn’t delete the source server, which contains the recovery point.

Tag re-check: If the tag has been restored, the document exits.

Approval gate: If AutoOffboardEnabled is false, the document sends a notification and stops.

State recording: Before modifying resources, it captures replication state, lag duration, and last snapshot time.

Recovery-instance check: Before completing teardown, the offboarding document checks for active recovery instances. If found, it flags them in the audit report and sends an SNS alert. Recovery instances incur separate EC2 billing that persists after source server deletion.

Adaptive retry: A maximum of 10 attempts with exponential backoff handles API throttling during bulk descope.

After teardown, the document checks what actually succeeded:

Outcome	Meaning
`COMPLETED`	Source server deleted, billing halted
`BILLING_STOPPED_ORPHAN`	Replication stopped but delete didn’t succeed. Operator must manually delete.
`FAILED`	`stop_replication` didn’t succeed. Billing might still be active. Operator review recommended.
`NO_SERVER`	Instance was not registered as an AWS DRS source server. Clean no-op.

Because each outcome reflects what actually succeeded, you know whether replication billing has stopped or whether a source server needs manual follow-up.

Security: Least-privilege IAM roles

The solution creates two IAM roles and relies on one that you provide. SSM Automation and Amazon EventBridge assume the DRS-Automation-Role to run all lifecycle actions. Its permissions cover SSM, EC2 tags, cross-Region AWS DRS calls, SNS, S3, Parameter Store, and iam:PassRole on itself. The iam:PassRole permission only allows passing this role to Systems Manager. Run Command is limited to instances tagged DR=yes. In the DR Region, the DRS-EventBridge-Forward-Role can only put events on the source Region’s default event bus.

On each EC2 instance, the instance profile from the prerequisites lets the Replication Agent and SSM Agent reach their services. The automation stores no long-lived credentials. Each role receives temporary credentials when it’s assumed. When you use private connectivity, cross-Region traffic stays on Transit Gateway and VPC endpoints.

Deploy the solution

Start in audit mode, where the onboarding document reports what it would install without changing any instance and offboarding sends a notification instead of running teardown. Review the audit reports, then turn on installation for a small set of instances before you scale to the full fleet.

To configure config.yaml

Clone the repository and change to its directory:

git clone https://github.com/aws-samples/sample-drs-agent-lifecycle-automation.git
cd sample-drs-agent-lifecycle-automation

The deployment scripts read your settings from config.yaml in the root of the repository. Set the values in the following sample before you deploy. The file in the repository also sets document names and stack names. Keep those at their defaults, because the commands in this post use the default names.

# Region configuration
source_region: us-west-1        # Region where your EC2 instances run
dr_region: us-east-2            # Region where AWS DRS replicates to
account_id: "111122223333"      # Your AWS account ID

# Observability
sns_topic_name: drs-alerts      # Existing SNS topic for notifications
amzn-s3-demo-audit_bucket: ""           # Existing S3 bucket for audit reports; empty turns off audit reports
cloudwatch_namespace: DRSAutomation

# Automation toggles (start in audit mode)
auto_install_enabled: "false"   # false = report only; true = install and repair agents
auto_offboard_enabled: "false"  # false = notify only; true = tear down on descope

# Tag configuration
dr_tag_key: DR
dr_protected_value: "yes"
dr_descope_value: "no"

# Stand-down control and schedule
stand_down_parameter: /drs/control/event-in-progress
schedule_rate: "rate(30 minutes)"

The following table describes the values you’re most likely to change.

Parameter	Required	Description
`source_region`	Yes	Region that hosts your EC2 instances and the automation
`dr_region`	Yes	Region where AWS DRS replicates to
`account_id`	Yes	AWS account that hosts the automation
`sns_topic_name`	Yes	Name of an existing SNS topic in the source Region
`amzn-s3-demo-audit_bucket`	No	Existing S3 bucket for JSON audit reports. Leave empty to skip them
`auto_install_enabled`	No	Set to `false` to report only, or `true` to install and repair agents
`auto_offboard_enabled`	No	Set to `false` to notify only, or `true` to run teardown when you descope an instance
`dr_tag_key`	No	EC2 tag key that controls the lifecycle (default `DR`)

To deploy the solution

The following steps deploy each component separately so you can see what each one creates. Run the commands from the root of the cloned repository. Replace the example account ID (111122223333) and Regions (us-west-1 and us-east-2) with your own values.

To deploy everything in one command instead, run ./scripts/deploy.sh after you edit config.yaml, and then go to the confirmation commands after step 6. The script prints the status of every stack when it finishes. Without yq, it falls back to built-in defaults, including a placeholder account ID. Use either the script or the individual steps, not both.

Deploy the IAM role stack. The template creates a role with an explicit name (DRS-Automation-Role), so CAPABILITY_NAMED_IAM is required.

aws cloudformation deploy \
  --template-file templates/iam-roles.yaml \
  --stack-name drs-automation-iam \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-west-1 \
  --parameter-overrides \
    AccountId=111122223333 \
    SourceRegion=us-west-1 \
    DRRegion=us-east-2 \
    SNSTopicName=drs-alerts

Register the three SSM Automation documents. The script reads source_region from config.yaml and creates the documents from the documents/ folder on the first run. On later runs, it publishes a new default version of any document that changed.
```
./scripts/register-documents.sh
```

Deploy the Amazon EventBridge rules in the source Region. This stack creates the tag-change rule that starts offboarding and the rule that starts healing. It uses the existing DRS-Automation-Role, so it doesn’t need CAPABILITY_NAMED_IAM.

aws cloudformation deploy \
  --template-file templates/eventbridge-source.yaml \
  --stack-name drs-eventbridge-source \
  --region us-west-1 \
  --parameter-overrides \
    AutomationRoleArn=arn:aws:iam::111122223333:role/DRS-Automation-Role \
    TargetRegion=us-east-2 \
    SNSTopicArn=arn:aws:sns:us-west-1:111122223333:drs-alerts \
    AutoOffboardEnabled=false

Deploy the forwarding rule in the DR Region. This stack creates DRS-EventBridge-Forward-Role, which lets the rule send stalled events to the source Region’s event bus, so CAPABILITY_NAMED_IAM is required.

aws cloudformation deploy \
  --template-file templates/eventbridge-dr.yaml \
  --stack-name drs-eventbridge-dr \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-2 \
  --parameter-overrides \
    SourceRegion=us-west-1 \
    AccountId=111122223333

Create the stand-down flag in Parameter Store with a value of false. The script reads the parameter name and Region from config.yaml and doesn’t overwrite an existing value.
```
./scripts/init-parameters.sh
```
If you prefer not to use the script, run the following command instead:
```
aws ssm put-parameter --name /drs/control/event-in-progress \
  --value false --type String --region us-west-1
```

Deploy the State Manager association in audit mode. The association runs once as soon as it’s created and then every 30 minutes.

aws cloudformation deploy \
  --template-file templates/state-manager-association.yaml \
  --stack-name drs-state-manager \
  --region us-west-1 \
  --parameter-overrides \
    AutomationRoleArn=arn:aws:iam::111122223333:role/DRS-Automation-Role \
    TargetRegion=us-east-2 \
    AutoInstallEnabled=false \
    SNSTopicArn=arn:aws:sns:us-west-1:111122223333:drs-alerts

To confirm the deployment, run the following commands:

aws cloudformation describe-stacks --region us-west-1 \
  --query "Stacks[?starts_with(StackName, 'drs-')].[StackName,StackStatus]" --output table

aws cloudformation describe-stacks --stack-name drs-eventbridge-dr --region us-east-2 \
  --query "Stacks[0].StackStatus" --output text

aws events list-rules --name-prefix DRS- --region us-west-1 --query "Rules[].Name"

aws events list-rules --name-prefix DRS- --region us-east-2 --query "Rules[].Name"

aws ssm list-documents --filters Key=Owner,Values=Self --region us-west-1 \
  --query "DocumentIdentifiers[?starts_with(Name, 'DRS-Agent-')].Name"

aws ssm get-parameter --name /drs/control/event-in-progress --region us-west-1 \
  --query "Parameter.Value" --output text

A successful deployment shows the following results:

All four stacks show CREATE_COMPLETE, or UPDATE_COMPLETE after a re-run.
The source Region lists the DRS-Tag-Change-Offboard and DRS-Stalled-Heal-Receive rules.
The DR Region lists the DRS-Stalled-Forward rule.
The source Region lists the DRS-Agent-Onboard, DRS-Agent-Heal, and DRS-Agent-Offboard documents.
The stand-down flag returns false.
In the AWS Systems Manager console, State Manager lists DRS-Agent-Onboard-Association.

When you’re ready to turn on installation, set auto_install_enabled: "true" in config.yaml and deploy the State Manager stack again with AutoInstallEnabled=true:

aws cloudformation deploy \
  --template-file templates/state-manager-association.yaml \
  --stack-name drs-state-manager \
  --region us-west-1 \
  --parameter-overrides \
    AutomationRoleArn=arn:aws:iam::111122223333:role/DRS-Automation-Role \
    TargetRegion=us-east-2 \
    AutoInstallEnabled=true \
    SNSTopicArn=arn:aws:sns:us-west-1:111122223333:drs-alerts

To turn on teardown, set auto_offboard_enabled: "true" and deploy the source Region Amazon EventBridge stack again with AutoOffboardEnabled=true:

aws cloudformation deploy \
  --template-file templates/eventbridge-source.yaml \
  --stack-name drs-eventbridge-source \
  --region us-west-1 \
  --parameter-overrides \
    AutomationRoleArn=arn:aws:iam::111122223333:role/DRS-Automation-Role \
    TargetRegion=us-east-2 \
    SNSTopicArn=arn:aws:sns:us-west-1:111122223333:drs-alerts \
    AutoOffboardEnabled=true

Safety guards prevent unintended actions

The following table lists the safety guards across the three automation documents. The stand-down flag and the tag value allow-list apply to every document. The terminated-instance guard and honest billing status protect offboarding, and the remaining guards protect onboarding and healing.

Guard	Prevents
DR-event stand-down	Acting during failover/failback
Terminated-instance guard	Deleting a recovery point on instance termination
Reverse-direction detect	Overwriting an AWS DRS-managed failback agent
Concurrency lock	Race conditions from overlapping runs
AutoInstallEnabled gate	Unintended installs in audit-first deployments
OS pre-check	Running installer on incompatible operating system
Honest billing status	Reporting “billing stopped” when teardown partially completed
Tag value allow-list	A typo triggering the wrong lifecycle action
Agent-presence-gated tagging	Applying failback tag to an agent-less instance

To pause the automation during a declared DR event:

aws ssm put-parameter --name /drs/control/event-in-progress \
  --value true --type String --overwrite --region us-west-1

To resume after the event:

aws ssm put-parameter --name /drs/control/event-in-progress \
  --value false --type String --overwrite --region us-west-1

Monitoring and observability

The three SSM Automation documents produce telemetry across three channels, each tuned for a different audience.

Amazon SNS notifications go to the on-call team. Onboarding stays quiet unless an instance needs intervention. Heal notifies on both success and intervention-needed, because a successful auto-heal still tells you a real issue occurred and was resolved. Offboarding sends a notification at each stage: start, complete, orphan, incomplete, and approval-needed.

Amazon S3 audit reports serve compliance and troubleshooting. One JSON object per lifecycle event at s3://<amzn-s3-demo-audit_bucket>/drs/<account>/<instance>/<event>-<timestamp>.json captures instance ID, host name, platform, source server ID, status, and pre-teardown replication state. To control storage costs, add a lifecycle rule to the bucket (for example, expire objects under drs/ after 90 days).

Amazon CloudWatch metrics power fleet dashboards. An Outcome metric (1=handled, 0=needs attention) dimensioned by event type provides at-a-glance fleet health.

Clean up

Removing the automation doesn’t stop replication. Instances tagged DR=yes keep replicating until you stop replication through the AWS DRS console or API. To remove the automation infrastructure, run the following commands:

Delete the State Manager association and the source Region Amazon EventBridge stacks.

aws cloudformation delete-stack --stack-name drs-state-manager --region us-west-1
aws cloudformation delete-stack --stack-name drs-eventbridge-source --region us-west-1

Delete the Amazon EventBridge forwarding stack in the DR Region.

aws cloudformation delete-stack --stack-name drs-eventbridge-dr --region us-east-2

Delete the SSM Automation documents and the stand-down parameter.

aws ssm delete-document --name DRS-Agent-Onboard --region us-west-1
aws ssm delete-document --name DRS-Agent-Heal --region us-west-1
aws ssm delete-document --name DRS-Agent-Offboard --region us-west-1
aws ssm delete-parameter --name /drs/control/event-in-progress --region us-west-1

After the other stacks are deleted, delete the IAM role stack.

aws cloudformation delete-stack --stack-name drs-automation-iam --region us-west-1

(Optional) If you created the SNS topic and S3 bucket only for this solution, delete them.

Conclusion

With a DR=yes tag in place, onboarding keeps the agent installed and registered, and healing restarts or reinstalls it when replication stalls. When you descope an instance, offboarding stops replication, removes the source server, and writes a JSON audit record to Amazon S3. That record tells you whether billing stopped or whether a partial teardown needs an operator’s follow-up. The whole lifecycle runs from one tag, without function code for your team to maintain.

The June 2023 post Deploying AWS Elastic Disaster Recovery at scale with AWS Systems Manager covered the initial install. This solution builds on it with ongoing health checks and offboarding.

Next steps

Start by cloning the repository and deploying it in audit mode in a test account. Tag a few instances with DR=yes and let onboarding run for a day. The audit reports in Amazon S3 show what the automation would install or repair on each instance, without changing anything. When the reports match what you expect, turn on installation for those instances. Then set DR=no on one of them to check the offboarding notification before you turn on teardown. If you have questions or improvements, open an issue in the GitHub repository.

Additional resources

About the authors

Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

2026-11-02 Saman Irfan

Post Syndicated from Saman Irfan original https://aws.amazon.com/blogs/big-data/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg/

Enterprises increasingly measure their readiness for AI by the readiness of their data, and that often means moving to an open lakehouse where data stays open and portable. The organizations moving fastest are the ones whose data gives them the flexibility to choose the engines and models that fit each workload. A tightly coupled, single-engine cloud data warehouse begins to hold them back. A generation of analytics tools was built inside vertically integrated systems, where the storage format, the catalog, and the query engine were fused into one vendor’s runtime. That design traded openness for convenience, and as generative AI, multi-engine analytics, and cross-team data sharing become the norm, the cost of that trade is rising. Increasingly, enterprises are choosing the opposite: an open, modular architecture where governance is defined once and enforced everywhere. This is the foundation for AI-ready data. It isn’t a new architecture to adopt, but an open lakehouse that keeps your data yours.

In this post, we explain why an open lakehouse architecture on AWS is the natural next step for organizations that are looking to future-proof their data architecture rather than re-architect onto another closed system. We then present the target architecture on AWS for migrating BigQuery workloads. Finally, we provide a step-by-step migration guide covering workload assessment, schema and query conversion, data movement strategies, and post-migration optimization to help you transition from BigQuery to an open lakehouse on AWS with confidence.

Why organizations are moving toward an open lakehouse

Open data, and the freedom to run engine of your choice. When data is held in a vendor’s proprietary format, it becomes difficult to use with other engines, tools, or AI models outside the vendor’s environment. Openness is what breaks it, but it comes in two degrees that are worth distinguishing. At the base is interoperability, which means many engines can reach the same data and it’s fast becoming table stakes now that even proprietary warehouses federate to open formats and query tables where they sit. Beyond that is portability: the freedom to take your data, your choice of engine, and your choice of AI model elsewhere without re-architecting. It’s the more durable property an open lakehouse is built to give you. Apache Iceberg is an open table format that removes these barriers: Iceberg compatible engines can read and write the same tables concurrently at petabyte scale while maintaining ACID compliance. On AWS, Apache Iceberg is a first-class citizen. Amazon Redshift, Amazon Athena, Amazon EMR, and AWS Glue all operate on one governed copy of the data in Amazon Simple Storage Service (Amazon S3). Because storage is decoupled from compute, you match the engine to the workload such as Athena for interactive SQL, Amazon Redshift for high-performance warehousing and business intelligence (BI), Amazon EMR for large-scale transformation, and Amazon SageMaker AI for machine learning (ML) without duplicating data or building extract, transform, and load (ETL) pipelines to shuffle it between systems. Interchangeability becomes a property of the architecture rather than something you engineer around.

Governance defined once, enforced across every engine. Openness doesn’t mean giving up control. On AWS, a single governance layer spans every engine that touches the data. The AWS Glue Data Catalog exposes an Apache Iceberg REST catalog endpoint that implements the Apache Iceberg REST Catalog open specification. AWS Lake Formation applies fine-grained access controls at the database, table, column, row, and cell level, defined once and enforced consistently whether a data engineer runs a Spark job on Amazon EMR, an analyst queries through Athena, or a data scientist trains a model in SageMaker AI. For discovery and sharing, you can use Amazon SageMaker Unified Studio to find, understand, and request access to governed data across the estate. Because the catalog follows an open specification rather than a single vendor’s runtime, governance travels with the data instead of locking it in.

A foundation that is AI-ready and model agnostic. The reason all this matters now is AI. Generative and agentic workloads are only as good as the data they can reach, and they perform best when data stays in open formats you fully control. But openness can’t stop at the data format. The intelligence layer matters too: the AI model, the semantic catalog, the enrichment layers, and the agents. An open foundation keeps both layers open. A single governed copy of open data sits in Amazon S3, and the same tables that serve BI and interactive analytics also feed Amazon SageMaker AI for model training and Amazon Bedrock and Amazon Bedrock AgentCore for generative and agentic work. You choose the foundation model (FM) that best fits your needs, with no extra copies to move or secure.

Target architecture: Open lakehouse on AWS

The following architecture diagram represents the recommended target state for organizations migrating BigQuery workloads to AWS and is especially well suited to customers wanting to build a unified, open data foundation. It’s built on Apache Iceberg as the foundational layer, with purpose-built consumption engines chosen per use case: Amazon Athena for interactive analytics, Amazon Redshift for BI reporting and dashboarding, and Amazon SageMaker AI for ML and agentic workloads, all on one governed copy of the data that stays open, interoperable, and portable.

The architecture follows the flow of data from left to right: source systems are ingested into an open lakehouse on Amazon S3, refined through medallion layers, and served to a diverse set of consumers such as BI tools, AI agents, and ML systems. It rests on four foundational pillars: open storage, purpose-built analytics, unified governance, and an integrated experience.

Figure 1: Open lakehouse target architecture on AWS, from source ingestion through the Amazon S3 Apache Iceberg storage layer to analytics, BI, and ML consumers

Ingestion

This is where it all begins. You must bring data from a variety of sources such as databases, software as a service (SaaS) applications, streaming systems, on-premises systems, IoT devices into your data lake.

Bulk migration: For the initial BigQuery migration and other large batch transfers, use BigQuery’s EXPORT DATA statement to export tables to Google Cloud Storage (GCS) in Parquet format. Then use AWS DataSync to move Parquet files from Google Cloud Storage to Amazon S3. For more information, see Step 4.
Zero-ETL: For SaaS and operational sources such as Google Analytics, Salesforce, SAP, ServiceNow, and HubSpot, Zero-ETL integrations land data directly in the lakehouse without building or maintaining custom pipelines.
Change data capture (CDC): For operational data stores, Debezium connectors running on Amazon MSK Connect stream row-level changes into the lakehouse to keep it continuously current.
Streaming: For real-time sources, Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka (Amazon MSK) provide streaming storage that feeds directly into the raw layer.

Storage layer: Amazon S3 with Apache Iceberg

Amazon S3 serves as the central storage layer, providing virtually unlimited scalability, 99.999999999% (11 nines) of data durability, and multiple storage classes, so you can match cost to the access pattern. Data is stored in Apache Iceberg table format, which brings ACID transactions, schema evolution, time travel, and partition evolution to your data lake. Amazon S3 Tables, a capability of Amazon S3, provides a fully managed Apache Iceberg experience with automatic compaction, snapshot management, and optimization removing the undifferentiated heavy lifting of table maintenance.

Data is progressively refined through a medallion architecture, with each layer stored as Apache Iceberg tables on S3:

Bronze/Raw: Ingested data lands here as-is, in Apache Iceberg format either on fully managed Amazon S3 Tables or as self-managed Apache Iceberg tables on Amazon S3.
Silver/Trusted: Silver layer serves as single source of truth. AWS Glue and Amazon EMR Spark jobs transform, clean, and conform the raw data into trusted, query-ready Apache Iceberg tables. Amazon Athena provides serverless, interactive analytics directly against this layer for interactive exploration and data discovery.
Gold/Serving: Amazon Redshift Serverless serves curated, business-ready data for dashboarding and reporting. Amazon Redshift is a first-class Apache Iceberg engine capable of managing the full Apache Iceberg table lifecycle (see Using Apache Iceberg tables with Amazon Redshift). Data can also be natively stored into Amazon Redshift Managed Storage for workloads that require increased performance for dashboarding and reporting use cases.

Governance: Unified across every engine

A unified foundation depends on governance that is defined once and enforced everywhere. The AWS Glue Data Catalog serves as the central metadata repository, and AWS Lake Formation provides fine-grained access control at the database, table, column, row, and cell level, enforced consistently whether data is accessed through Athena, Amazon Redshift, EMR, or SageMaker AI. AWS Glue Data Quality, a capability of AWS Glue, validates data as it moves through the layers and Amazon SageMaker Unified Studio provides a single integrated development environment for data discovery, SQL analytics, visual ETL, and artificial intelligence and machine learning (AI/ML) development all governed by the same Lake Formation permissions.

Consumers: One lakehouse, many workloads

Because data sits in open Apache Iceberg format under unified governance, a broad set of consumers can access a single source of truth without data movement:

BI tools: Amazon Quick Suite, Looker, Power BI, and Tableau connect for dashboarding and reporting.
AI agents: Agentic tooling such as MCP-based clients, Kiro, and Claude, along with Amazon Bedrock AgentCore, can reason over governed lakehouse data.
ML/SageMaker: Amazon SageMaker AI trains models and serves inference directly against lakehouse data.
Amazon SageMaker Unified Studio provides a single integrated development environment (IDE) for data exploration, SQL analytics, visual ETL, and AI/ML development. For organizations accustomed to a single-pane experience, Unified Studio addresses the concern of managing multiple services by providing one place to discover data, run queries, build dashboards, and develop ML models, all governed by the same Lake Formation permissions.

Data interoperability

The Apache Iceberg REST catalog endpoint of the AWS Glue Data Catalog, part of Amazon SageMaker Lakehouse, enables interoperability with third-party engines. Partners such as Snowflake and Databricks can read and write the same Apache Iceberg tables through that endpoint, so you choose the best tool for each job without duplicating data.

Cross-cutting layers: Orchestration, security, and observability

Two layers span the entire lakehouse rather than sitting at any single stage. An orchestration layer coordinates how data moves through the medallion layers: Amazon Managed Workflows for Apache Airflow (Amazon MWAA), AWS Step Functions, and Amazon EventBridge schedule and sequence ingestion, transformation, and load jobs across Bronze, Silver, and Gold.

A security and observability layer applies across every engine and layer. AWS Identity and Access Management (IAM) and AWS Key Management Service (AWS KMS) provide authentication, authorization, and encryption, while Amazon CloudWatch, AWS CloudTrail, and AWS Cost Explorer provide monitoring, audit logging, and cost visibility across the lakehouse.

Step-by-step migration guide

Migrating from BigQuery to an open lakehouse on AWS follows a structured, phased approach. Each phase builds confidence incrementally, so you can validate value before committing further.

Step 1: Assess and discover

Begin by evaluating your current BigQuery environment:

Inventory workload: Catalog all datasets, tables, views, stored procedures, scheduled queries, and downstream consumers (BI tools, applications, ML pipelines). Name the consumers that most often need a decision: Looker Studio, Connected Sheets, scheduled queries, and any production service reading BigQuery directly.
Classify query complexity: Analyze your SQL workloads to identify standard ANSI SQL (which migrates directly), BigQuery-specific syntax (such as STRUCT, ARRAY, UNNEST, and JavaScript UDFs), and procedural logic in stored procedures.
Map data dependencies: Document which tools and applications read from and write to BigQuery. Identify upstream data sources (streaming, batch, CDC) and downstream consumers (dashboards, reports, ML models).
Assess data volumes: Record active storage, long-term storage, data growth rates, and each dataset’s storage billing model (logical or physical) with its measured physical bytes. The compressed physical size can be a small fraction of the logical size and changes the storage cost estimate materially. This informs your S3 storage class strategy and cost modeling.
Evaluate governance: Document current access controls, row-level security policies, and audit requirements. Map these to Lake Formation capabilities.

The output of this phase is a migration readiness report with per-table effort scoring, query complexity classification, and a recommended migration pattern.

AWS account teams and partners have metadata-driven assessment accelerators. They read only BigQuery metadata (schemas, INFORMATION_SCHEMA job history, view definitions, and routine signatures), not raw data. From that metadata, they produce a cost comparison with the projected savings, per-table migration effort scoring, a complexity classification for every view, routine, and observed query, and a map of every tool reading from or writing to BigQuery with how each one reconnects on AWS.

Reach out to your AWS account team for an initial assessment of moving BigQuery workload to an open lakehouse on AWS.

Step 2: Design target architecture

Based on your assessment findings, design the target-state architecture:

Storage design: Define your S3 bucket structure with a multi-layer layout (raw, cleaned, and aggregated) aligned with data lifecycle best practices. For Apache Iceberg storage, we recommend Amazon S3 Tables for fully managed Apache Iceberg with automatic table maintenance and optimization and general-purpose S3 buckets with self-managed Apache Iceberg for cases that need granular control over table layout and compaction.
Partitioning strategy: Design Iceberg partition schemes aligned to actual query access patterns. Use Iceberg’s native partitioning transforms (bucketing, time slicing) for optimal query pruning.
Engine selection: Choose the engine from the workload you measured in Step 1. Amazon Athena fits estates that scan a moderate volume per day and tolerate a few seconds of latency whereas Amazon Redshift fits estates with sustained high concurrency, sub-second dashboard targets, or a daily scanned volume where warm compute is cheaper than paying per terabyte scanned. For large-scale data transformations, position Amazon EMR (including Amazon EMR Serverless) for complex, custom open source frameworks like Spark and Hive, or AWS Glue for serverless, integration-heavy ETL pipelines.
Governance model: Design your AWS Lake Formation permission model to enforce granular database, table, column, and row-level access policies. Verify the architecture plans for multi-tenant access patterns if applicable.
Ingestion architecture: Design data ingestion pipelines. Options include Zero-ETL integrations for operational data sources, Amazon Data Firehose for streaming data, and AWS Glue for batch ETL.

Step 3: Convert and validate (POC)

Pick a proof-of-concept slice, a handful of representative tables and the dashboards that read them, and convert BigQuery SQL, stored procedures, and data pipelines to their AWS equivalents.

Schema conversion: A BigQuery dataset becomes an AWS Glue database when Athena is the engine, or an Amazon Redshift schema when Amazon Redshift is. The BigQuery project maps to your AWS account with its top-level namespace being the AWS Glue Data Catalog for Athena or to an Amazon Redshift database.
SQL conversion: Convert BigQuery SQL dialect to ANSI SQL or engine-specific syntax. Common conversions include:
- BigQuery STRUCT to Iceberg struct and ARRAY to Iceberg list, and BigQuery JSON maps to a string column queried with JSON function when Athena is the engine and data is stored in Apache Iceberg format. For Amazon Redshift, the recommendation is to build the gold layer which is optimized for reporting use cases where data is flattened out for high-performance queries. If you need to store nested data inside Amazon Redshift, use the SUPER data type.
- On Amazon Redshift, the target depends on where the data lands: for data stored natively in Amazon Redshift, BigQuery JSON maps to the SUPER type (queried with PartiQL), and STRUCT/ARRAY are represented within SUPER.
- BigQuery MERGE statements to Athena or Amazon Redshift MERGE INTO. Choose copy-on-write for tables updated in bulk and merge-on-read for frequently updated ones.
- BigQuery JavaScript UDFs to a SQL expression where possible, otherwise an AWS Lambda-backed UDF on Athena (USING EXTERNAL FUNCTION) or Amazon Redshift.
- BigQuery scheduled queries to Amazon EventBridge Scheduler running Athena queries, Amazon Redshift scheduled queries in Query Editor v2, or dbt for anything that is a model. Chains of schedules become AWS Step Functions.
Pipeline conversion: Rebuild ETL pipelines using AWS Glue, dbt (with dbt-athena or dbt-redshift adapters), or Apache Spark on EMR.
Validation: Run converted workloads against the slice. Compare row counts and checksums per table, query results for the dashboards on the slice, and run times against the agreed target, between BigQuery and AWS, to verify correctness.

BigQuery offers two pricing models for running queries. On-demand pricing is a scan-based model that charges $6.25 per terabyte of data scanned, while capacity-based pricing is a compute-based model that charges for the compute capacity used to run queries. Amazon Redshift is also compute-based, so if you currently run BigQuery on demand, its scan-based pricing is not directly comparable to Amazon Redshift’s compute-based pricing. Because the two models differ, benchmark a representative mix of queries on the target engine rather than a single query type.

A good practice is to run three tiers of queries and compare total cost and performance against your current BigQuery baseline: lightweight queries for simple lookups and aggregations, medium queries for moderate joins and transformations, and heavyweight queries for complex joins, window functions, and large scans. To build an accurate, workload-representative total cost of ownership (TCO), multiply each query tier by how often it runs, add storage costs, and produce a monthly figure you can compare directly to your current monthly BigQuery bill.

Step 4: Migrate data

Execute bulk data migration from BigQuery to Amazon S3 with Apache Iceberg:

Use BigQuery’s EXPORT DATA statement to export tables to Google Cloud Storage (GCS) in Parquet format. For large datasets, partition exports by date or key ranges to parallelize the transfer.
Use AWS DataSync to move Parquet files from Google Cloud Storage to Amazon S3. DataSync reads Google Cloud Storage directly through its S3-compatible XML API and writes the objects to your S3 bucket, with scheduling and incremental transfers built in. For most transfers you deploy a DataSync agent, either as a virtual machine in Google Cloud (closest to the source bucket) or as an Amazon Elastic Compute Cloud (Amazon EC2) instance in your virtual private cloud (VPC). DataSync Enhanced mode transfers between Google Cloud Storage and Amazon S3 run without an agent and offer higher performance and scalability.
Create the Apache Iceberg tables in Athena or Amazon S3 Tables and load them from the exported Parquet with CREATE TABLE AS SELECT or INSERT INTO, or register the existing files with Spark’s add_files procedure on Amazon EMR or AWS Glue.
During the transition period, implement incremental data synchronization to keep AWS data current while BigQuery remains active. Use BigQuery’s change history functions (APPENDS and CHANGES) for scheduled delta exports, change data capture (CDC) patterns, or Amazon AppFlow.
Apply for Google’s data transfer fee waiver before the bulk export starts: Google waives egress fees for customers leaving Google Cloud who request it through support and complete the export within the agreed window.

Step 5: Cutover and go-live

Execute production cutover with minimal disruption:

Parallel run: Run workloads on both BigQuery and AWS simultaneously for a validation period. Compare outputs to confirm data consistency and query correctness.
Redirect consumers: Every consumer takes one of three paths:
- Re-point: In this case, the tool stays the same, only its connection changes from BigQuery to Athena or Amazon Redshift (dbt, Fivetran, Tableau, Power BI, Looker). This is usually a JDBC/ODBC or connector update. Athena and Amazon Redshift both use the AWS Glue Iceberg REST catalog (IRC) endpoint to locate the physical Parquet files on Amazon S3. For more information, see Querying Apache Iceberg tables in Athena and Querying Apache Iceberg tables in Amazon Redshift.
- Rebuild: In this case, the workload has no direct AWS connection and is re-implemented on an AWS counterpart (BigQuery scheduled queries → Amazon Athena Saved Queries orchestrated by AWS Step Functions, or managed through Amazon EventBridge to run an ETL pipeline on a schedule, Dataform → dbt (using the Athena or Amazon Redshift connector), BigQuery ML → SageMaker AI or Amazon Redshift ML), Google Analytics 4 and Connected Sheets → Amazon AppFlow native connectors ingest the data (Google → AWS) into Amazon S3 or Amazon Redshift on a schedule. From S3, an AWS Glue or Athena step lands it in Apache Iceberg.
- Relay: In this case, the tool stays on Google Cloud for now and reads AWS data synced back to it (Connected Sheets, GA4 raw export).

Looker Studio, Connected Sheets, and scheduled queries are the consumers that most often need an explicit decision on which path to take.

Validate end-to-end: Confirm that all dashboards, reports, scheduled jobs, and ML pipelines produce correct results.
Decommission: After successful validation, decommission BigQuery resources and terminate cross-cloud data synchronization.

Step 6: Optimize and expand

Post-migration, optimize your lakehouse for performance and cost:

Query optimization: On general-purpose buckets with self-managed Apache Iceberg, tune file size targets, compaction frequency, and sort orders to the actual query patterns whereas Amazon S3 Tables handles compaction and snapshot expiry automatically. Enable Athena query result reuse and Amazon Redshift materialized views for frequently accessed datasets.
Cost optimization: Use S3 Intelligent-Tiering for infrequently accessed data, Athena workgroups and Amazon Redshift usage limits to control costs, and AWS Cost Explorer to monitor.
Enable new capabilities: With data in open Apache Iceberg format on Amazon S3, you can take on use cases that were previously difficult or impossible:
- Train ML models directly on lakehouse data using SageMaker AI.
- Build generative AI applications with Amazon Bedrock Knowledge Bases.
- Implement real-time analytics with Amazon Data Firehose or Amazon Kinesis Data Streams writing Apache Iceberg tables to Amazon S3 Tables, or with Amazon Redshift streaming ingestion.
- Share Apache Iceberg tables with partner engines such as Snowflake and Databricks through the Apache Iceberg REST catalog endpoint.

Customer success: Yggdrasil Gaming

Yggdrasil Gaming, a global iGaming studio, migrated from BigQuery to an Apache Iceberg-based lakehouse on AWS, working with GOStack, an AWS Partner. The company had been operating dual-cloud infrastructure across AWS and Google Cloud, creating significant operational overhead that limited their ability to implement advanced analytics and AI/ML initiatives.

The migration followed a phased approach: establishing the lakehouse foundation on S3 with Apache Iceberg and AWS Glue Data Catalog, implementing real-time data ingestion using Debezium for change data capture, migrating processing pipelines to AWS Lambda and Amazon EMR, modernizing the transformation layer with dbt and Amazon Athena, and enabling governance through AWS Lake Formation.

The results were significant:

60 percent reduction in data processing costs.
75 percent improvement in analytics latency (from 2 hours to 30 minutes).
Alleviated multi-cloud complexity by consolidating on AWS.
Enabled AI/ML use cases including player behavior modeling and fraud detection.

For the full story, read Building a modern lakehouse architecture: Yggdrasil Gaming’s journey from BigQuery to AWS on the AWS Big Data Blog.

Customer success: Global technology leader performed zero-downtime GCP exit at petabyte scale

A global technology leader in the network security space migrated from Google BigQuery to AWS to reduce costs and enable streaming ingestion for network security analytics. Working with Quantiphi, an AWS Partner, the company migrated 10 core customer-facing applications across 3 regions with zero business downtime for more than 1,000 customers.

Highlights of the migration:

Migrated over 1,000 BigQuery projects to Amazon Athena with Apache Iceberg on Amazon S3 (setting up over 5,000 Apache Iceberg tables).
Key drivers: cost reduction (30 percent savings in cloud operational costs), streaming ingestion for security analytics, and regulatory alignment.
Zero business downtime across a petabyte-scale migration spanning three AWS Regions, with minute-level data refresh meeting a 99.9 percent service level agreement (SLA).
A full GCP exit consolidating entirely on AWS through Quantiphi’s Qatapult modernization and migration platform.

For the full story, read Engineering Excellence: How a Tech Leader Migrated to AWS with Zero Business Downtime (Quantiphi case study).

Conclusion

Migrating from BigQuery to an open lakehouse architecture on AWS with Apache Iceberg is an architectural change and a strategic investment in data openness, multi-engine flexibility, and AI readiness. By storing data in open formats on Amazon S3, governing it centrally through AWS Lake Formation, and accessing it with purpose-built engines, you build a data foundation that adapts to your evolving needs without vendor lock-in.

The phased migration approach (assess, design, convert, migrate, cutover, and optimize) means you can validate value at each step and maintain business continuity throughout the transition. Organizations like Yggdrasil Gaming have demonstrated that this migration can be completed in months, not years, with measurable cost savings and performance improvements.

To get started, explore the following resources:

About the authors

Midterms & Trump #lastweektonight

2026-11-02 LastWeekTonight

Post Syndicated from LastWeekTonight original https://www.youtube.com/shorts/uEpFWeXkZ3s

Саманта Швеблин: Да избягаме от езика, за да докоснем тялото на читателя

2026-11-02 Стефан Иванов

Post Syndicated from Стефан Иванов original https://www.toest.bg/samanta-schweblin-da-izbyagame-ot-ezika-za-da-dokosnem-tyaloto-na-chitatelya/

Саманта Швеблин: Да избягаме от езика, за да докоснем тялото на читателя

Саманта Швеблин е аржентинска писателка от Буенос Айрес, която от 2012 г. живее в Берлин. Смятана е за един от най-силните гласове в съвременния испаноезичен разказ, а книгите ѝ са преведени на повече от двайсет езика. Носителка е на Националната литературна награда на САЩ за преводна литература и на наградата „Шърли Джаксън“; три пъти е номинирана за Международния „Букър“. През април 2026 г. последната ѝ книга „Доброто зло“ – пет разказа за сложните връзки между родители и деца, спечели новоучредената награда „Аена“ за испаноамериканска проза в размер на един милион евро. Швеблин гостува в България по покана на Културния център на Софийския университет, за да представи именно „Доброто зло“ в превод на Калоян Игнатовски (изд. „Лабиринт“). Срещата с нея беше на 30 октомври от 17:00 ч. в аулата на Софийския университет.

Разказите Ви често започват с обикновена домашна сцена, която почти незабележимо се накланя към ужаса. Откъде започва историята за Вас – от образ, страх или изречение?

Ако трябва да избера от трите, бих казала страха, защото той идва пръв. После нещо привлича погледа ти или чуваш изречение, което те сепва, но то е интересно само защото страхът вече е бил там. Всичко започва със силна и много конкретна емоция, а колкото по-конкретна е, толкова повече време отнема да я обясниш. Американската есеистка Ребека Солнит има едно изречение, което непрекъснато повтарям, и то е, че

добрата книга е сърце, което бие в чужди гърди.

Когато пишеш, имаш страха, той те свързва с образи, образите ти дават думи. После идва читателят, взема думите, изгражда образа и получава страха. И този страх трябва да е съвсем точен. Писането е мост между твоите гърди и гърдите на някой друг.

Много от текстовете Ви сякаш измерват разстояния – между родител и дете, между изречение и страх, между непознати, между видяното и почувстваното. Колко близо допускате читателя, преди да задействате капана на страха?

Възможно най-близо. Може би в началото не толкова, защото обичам историите да изглеждат кротки, прости, реалистични. Но щом приближиш нещо съвсем до себе си, виждаш колко е странно. Може би затова обичам разказа, движението в него е бързо. Не ти трябват двеста страници, за да доближиш до очите си нещо, което е изглеждало далечно. Като зуум е. Буквално. Не бях се замисляла досега.

Критиците често описват прозата Ви с думи като „хорър“ или „нечовешко“. Осветляват ли тези етикети това, което правите, или го опитомяват?

Опитомяват го, но разбирам какво имате предвид. Книгата в крайна сметка е предмет, който трябва да бъде сложен на определен рафт, защото живеем в капитализъм. Но литературата, която най-много обичам да чета, е тази, за която самата аз не знам на кой рафт да я сложа – там, където жанровете се смесват и идеите си противоречат. Литературата е обратното на радикализацията, в която живеем. Тя винаги показва как едно нещо може да бъде едновременно добро и лошо, като в заглавието на книгата. Хорърът е по-скоро това, което се случва в главата на читателя. Ако подчертаете думите на ужаса в книгата ми, няма да намерите призраци, Дракула, кръв, никой никого не убива. Когато пишем, слагаме думи на страницата, но пишем и в съзнанието на читателя и не го казвам метафорично. Има думи, които можеш да докоснеш в книгата, и думи, които се случват само в главата на читателя. Мисли като „винаги е била тя“ или „той ще я убие“. Или по-лошо – „това може да се случи на мен и ако се случи, не знам дали ще оцелея“. Това е ужасът, и то най-лошият, защото не се случва на персонажа. Случва се на теб.

Това е буквално реалното.

Точно така.

Кои автори, които не се побират на никой рафт, обичате и Ви вдъхновяват?

Първо се сещам за Шърли Джаксън. Някои романи могат да се нарекат хорър, но разказите ѝ не са хорър, макар да са ужасяващи и великолепни. Изключително важен за мен, когато като дете четях първите си книги за възрастни, беше Рей Бредбъри – уж кротка научна фантастика, а всъщност страшна, много страшна. И в същото време невероятно позитивна. Оттогава не съм чела друг автор, който да пише за толкова мрачни неща и да е такъв оптимист за бъдещето. После открих Балард. Спомням си и разказите на Кафка – толкова мрачни. Никой не говори за това, но Кафка е много смешен.

Кундера винаги го е казвал…

Може би защото забравяме, че четенето е наполовина писател и наполовина читател. Когато слагаме книгите по рафтовете, слагаме там само половината. Читателя го няма, трябва да отвориш книгата и да четеш, за да се случи. Затова една книга може толкова да се промени с времето. Читателите се променят.

Саманта Швеблин: Да избягаме от езика, за да докоснем тялото на читателя — Саманта Швеблин в София © Снимки: Илиян Ружин

Прозата Ви сякаш усеща тревогите на времето, преди да станат заглавия във вестниците за отровени пейзажи, наблюдение или близост, опосредствана от машини. Пишете ли с поглед към определен социален проблем, или той се промъква в историята, която е била за нещо друго?

Истинският отговор е, че не знам. Никога не взeмам съзнателно решение – „това ще е проблем в бъдеще, трябва да пиша за него“. Нещата не се случват от днес за утре. Има шум и по-късно разбираш, че е бил звукът на нещо, което идва. Когато писах „Хиляди очи“, нямаше начин човек като мен да реши да пише за технологии, няма нищо по-скучно за мен. Но ме вълнуваше това, че живеем затрупани от технологии, без да мислим за тях. Едно и също се повтаряше отново и отново. Чета например стихотворение в списание, в което героят два-три пъти праща съобщения по WhatsApp, и някой го нарича „технопоезия“. Или роман, в който присъстват социални мрежи, и изведнъж той е научна фантастика. Това е шумът. Натурализираме технологиите в живота, но щом се появят в книга, решаваме, че не принадлежат на реалността. А литературата е пространството, в което мислим следващите си стъпки и в което се изпитваме и подготвяме за бъдещето. Започнах да си играя с този проблем и така се появи „Хиляди очи“. Темите не са планирани, но усещаш, че вече са в живота ти и са неудобни. И искаш да мислиш за тях.

Днес говорим с чатботове и се доверяваме на алгоритми. Започнала ли е реалността да пише историите, които някога сте си представяли?

В „Хиляди очи“ има устройство, но големият въпрос на романа е как общуваме с другите чрез него. Фокусът ми остава върху хората. С чатботовете говорим с нещо друго. Много е важно да престанем да наричаме тези неща изкуствен интелект, защото те в никакъв случай не са такъв. Хитро е, че ни научиха да ги наричаме така. Това са езикови системи за вероятности и предположения. Пълната противоположност на онова, което хората са правили през цялата история, когато са изобретявали или откривали нещо ново. Ако кажеш „куче“, езикът, основан на предсказване, ще каже „кафяво“, защото това е най-често свързаната дума. А прочутият разказ на Габриел Гарсия Маркес се казва „Очи на синьо куче“. Ето това правим ние. Всеки път, когато намерим нещо извън системата, предприемаме първата крачка към истинско откритие. Когато продължаваме да говорим с тези машини за предположения, се придвижваме към най-очевидния си начин на съществуване. Ставаме глупави, скучни и предвидими, а в това има само нещастие. Убедена съм. Не бих казала дори, че е опасно. Скучно и тъжно е. Като да затваряш една след друга всички врати навън. Точно обратното движение на онова, което прави литературата.

Слушайки Ви, си спомних, че в „Махалото на Фуко“ Умберто Еко измисля компютър на име Абулафия, който комбинира текстове и пише литература. Но там това е шега.

Трябва да препрочетем тази книга.

Какво може да направи писателят днес, във време на разкъсано внимание, моментално възмущение и лесен гняв? Разказът – със своята сгъстеност и отказа му да обяснява – форма на съпротива ли е?

Разказът изисква много от читателя. Добрият роман също, но дори да си страхотен читател, ще ставаш много пъти, ще продължаваш живота си и ще се връщаш. Разказът иска да си напълно съсредоточен двайсет-трийсет-четирийсет минути. Това вече не е лесно, губим вниманието си. Но има нещо особено в несигурността. Обичам това състояние и мисля, че е сърцевината на всяка история. Историята започва, когато изведнъж настъпи несигурност. Имаш персонаж, звънецът звъни и се чудиш кой е. И историята потегля. В живота бягаме от несигурността, не я обичаме. Но във фикцията, когато наистина внимаваш, тя е съвсем отворено пространство. Умира бащата на персонажа, твоят е жив и затова не си толкова разтревожен. Но си задаваш много интересни въпроси. Каква е връзката ми с баща ми? Ами ако го загубя сега? Казах ли му всичко, което исках? Вниманието е напрежение. Когато в книгата има напрежение, ти се отваряш. Споменахте отказа да се обяснява; когато историята стане абстрактна, следиш по-малко сюжета и повече това, което става в главата ти. Затова е толкова силна. Това е танц с писателя. В абстракцията той ти дава време да си играеш с мислите си, без да изпускаш сюжета. Двете трябва да се случват едновременно. Толкова е сложно, а се случва. Като магия е.

Симон Вейл, философката и писателката, казва, че вниманието е най-висшата форма на молитва.

Обичам тази нейна мисъл. Тя казва още, че когато наистина внимаваме, нещо зло в нас умира. Защото, когато внимаваш истински, се свързваш със себе си и с това, което е важно за теб, свързваш се и със света. Злото започва след прекъсването на тази връзка.

Децата във Вашата проза са уязвими, но и странни, непрозрачни, понякога плашещи. Какви възрастни си представяте, че ще станат днешните деца?

Много страшен въпрос. Говорихме за вниманието и тези деца вероятно ще трябва много да се борят за своето. Как ще знаят какво искат, кое е важно? Мрачно е да се мисли за това. Но преди няколко седмици сръбският ми преводач ми каза нещо, за което още мисля. Говореше за протестите в Сърбия. „Тези деца, които току-що промениха света в моята страна – каза ми той, – са децата, на които съм преподавал, без да им вярвам. Мислех, че не внимават, че не мислят за това, за което трябва, че не учат, че не се интересуват от политика. Това беше огромен урок за мен.“ Малко ужасяващо е, защото ние, възрастните, сигурна съм, че мнозина ще се съгласят, виждаме как и колко губим от способностите си. Не мога да се съсредоточавам както преди. Помня как четях Гарсия Маркес по два часа без прекъсване, пълна с радост. Днес половин час четене би бил пълен успех, защото тялото ми вече посяга към телефона. Тази битка сме я загубили. И се плашим, мислим, че може би и младите ще се изгубят. Не знаем.

Пак несигурност.

Пак несигурност.

Страхът често е Вашата суровина. Може ли страхът да бъде форма на внимание? Дори на грижа?

Абсолютно. Затова казвам, че напрежението е внимание. Щом включиш в текста опасност, която наистина може да докосне реалността на читателя, той внимава. Но е малко по-сложно, защото страхът включва и предразсъдъците ни и това е нормално – така работят телата и умовете ни. Ако четеш реалистична проза и някой влезе в стаята с пистолет, предразсъдъците ти казват, че това е лош човек, ще умрем, може да избягаме по стълбите или да се скрием. Помагат ни да реагираме. Във фантастиката и фентъзито също има много правила как да се защитим. Ако се появи Дракула, знаеш, че трябва да дръпнеш завесите, за да влезе светлина, и може би да имаш чесън в джоба. Но в несигурността няма правила. Това ѝ е хубавото. Когато се случи нещо извън правилата и очакванията, предразсъдъците не помагат. Не знаеш какво да правиш, трябва ти информация. И тогава наистина внимаваш.

Как научихте – чрез писане, опит, четене или четене на глас на собствените си разкази – какво да оставите встрани, за да има сгъстено напрежение?

Мисля, че си играя със себе си като читател. Научих се да пиша, разбира се, пишейки. Но още повече се научих, четейки книги с изключително внимание, и то не само към книгата, но и към собственото си тяло и ум. Питам се непрекъснато какво става с мен като тяло, като мозък, като сърце, като стомах. Може би така не се чете най-добре, защото разсейва. Но откритията са невероятни. Откриваш, че един ред не намясто те изважда от книгата. Когато се разсея, спирам и започвам да чета назад, за да разбера къде съм се отклонила. И винаги има такова място. Винаги е конкретна фраза, понякога само две думи. Тогава се опитвам да разбера защо точно те. Когато изучаваме литература и творческо писане, се фокусираме много върху писателя – защо е взел това решение или какво иска да каже. Но трябва да внимаваме и с читателя. Читателят знае много, защото работи с истинския универсален материал на фикцията – тялото. То е нещо изключително органично и мъдро, по-мъдро от нас. Всеки читател е натоварен със собствения си език, култура и живот, но тялото е едно и също в Китай, в Куба и в Германия. Може да звучи странно да се говори за тяло, когато говорим за литература, защото литературата е изградена от език, а езикът е нещо, което сме измислили. Но най-добрата литература се случва, когато фикцията успее да се изплъзне от езика и да докосне тялото. Градим литературата с език, но трябва да избягаме от него, за да докоснем наистина тялото на читателя.

Попитах Ви, защото, когато Ви чета, имам усещането, че пишете като поет, но и като хирург. Много прецизно. Представете си света след тринайсет години, през 2039-та. За какво се страхувате, че ще е изчезнало? Какво се надявате да е оцеляло? И каква история бихте искали да пишете тогава?

Надявам се книгите още да са тук и мисля, че ще бъдат. Не съм песимистка в това отношение, защото книгата е много странно нещо. Технологията промени всички изкуства – киното, театъра, музиката. Книгата не се промени. Може би носителят се е сменил – може да четеш на електронен четец. Но единствената технология, от която книгата има нужда, е нашият мозък. Затова книгата определено ще остане. Надявам се да не сме изгубили вниманието си и да не сме спрели да мислим заедно с другите хора вместо с машините. За първи път в историята започваме да мислим само със себе си, защото, когато говорим с машини, говорим със собствените си най-обикновени и очевидни мисли. А вратите и прозорците се отварят, когато сме достатъчно смели да мислим с някой, който мисли различно от нас и знае други неща. Ако изгубим това, свършено е с нас. Какво бих искала да пиша? Добра книга, която ни позволява да продължим да правим това. Надявам се да пиша и след трийсет години.

Какво Ви дава утеха и оптимизъм?

Знам, че е странно, но утеха ми дават онези много редки моменти във фикцията, когато наистина внимавам. Тогава имам чувството, че мозъкът ми е спокоен. Вече не е изгубен, съсредоточен е, а когато съм съсредоточена, съм по-спокойна. Склонни сме да мислим, че вниманието е нещо стресиращо, което изисква огромно усилие. Така е, когато не си трениран. Когато си добре трениран, вниманието ти помага да си почиваш. Това е утеха за мен.

Значи обичате медитацията?

Да, абсолютно.

AWS Weekly Roundup: GPT-6.1 Terra on Bedrock, Step Functions as agent tools, and more (November 2, 2026)

2026-11-02 Daniel Abib

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-gpt-6-1-terra-on-bedrock-step-functions-as-agent-tools-and-more-november-2-2026/

The agentic launches that stayed with me last week had nothing to do with how capable a model is. They were about reach. AWS Step Functions, which already orchestrates over 220 AWS services, can now be handed to an agent as a tool, so workflows teams already trust became callable without a rewrite. In the same week, the AI blog published how one engagement platform exposed roughly 400 internal API operations the same way. Neither story involves rebuilding anything, only describing what exists and letting the agent find it.

The other thread was cost. OpenAI’s newest Terra model reached Amazon Bedrock priced well below the model beside it, and two post-quantum launches landed for a deadline nobody can put on a calendar yet. Now, let’s get into this week’s AWS news…

Headlines
OpenAI GPT-6.1 Terra is now generally available on Amazon Bedrock, and what interests me is where it sits rather than what it tops. Terra is the everyday-workload line of the GPT-6 family. In OpenAI’s Agents’ Last Exam evaluation it scored 15 percentage points higher than GPT-5.6 Terra, performed on par with GPT-6 Astra, and completed tasks more than three times faster than both. Those are OpenAI’s figures, not an independent test. The price is what I would check first: input and output tokens are at least 50% lower than GPT-6.1 Sol and GPT-5.6 Terra. It takes up to 1 million input tokens with text and image input, and you can reach it from the Bedrock APIs or ChatGPT Work and Codex.

AWS Step Functions workflows are now available as AI agent tools through Amazon Bedrock AgentCore Gateway, which connects agents to tools over the Model Context Protocol (MCP). The division of labor is the idea: the agent decides when to run a workflow and what input to pass; Step Functions runs each step exactly as you defined it. A support agent can trigger a refund, and the eligibility check, the manager approval, and the credit still happen in that order every time, because the ordering was never the model’s to improvise. Your existing Standard and Express workflows qualify as connector targets, and agents can also look up runs. It’s available in all Regions where AgentCore Gateway is, at standard rates.

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

AWS Key Management Service now supports ML-KEM for post-quantum key encapsulation – You can now establish a shared secret with a post-quantum algorithm using AWS KMS keys, with private keys in FIPS 140-3 Level 3 validated HSMs. All three NIST FIPS 203 parameter sets are supported through Encapsulate, which produces a random 32-byte shared secret and its ciphertext, and Decapsulate, which recovers it. The threat is harvest now, decrypt later: an adversary records today’s key exchange and recovers the secret once quantum computers are powerful enough. It’s available everywhere KMS runs except the Middle East (UAE and Bahrain) and China Regions. Amazon Aurora PostgreSQL added post-quantum TLS key exchange the same week, on versions 18 and higher.

AWS IAM Identity Center now supports cross-Region trusted identity propagation – You can now carry a user’s identity with their request to AWS services in another Region. This removes a genuine architectural constraint: trusted identity propagation used to require every service accessing that identity to sit in the same Region, which left two unappealing options, broad IAM roles that cannot distinguish individual users, or consolidating all your services into a single Region. Now the identity travels with the request automatically, so you grant access by user and group membership and audit who accessed what. It’s available at no additional cost everywhere IAM Identity Center runs.

Amazon ECS now supports GPU sharing with NVIDIA Multi-Process Service (MPS) – You can now run multiple Amazon ECS tasks concurrently on a single NVIDIA GPU, the practical answer to work that only ever uses a fraction of one, like small-model inference and multi-model serving. You declare the GPU memory each container needs and can optionally cap its share of GPU compute, and ECS places tasks on GPUs with available capacity as your services scale, with no code changes. It works on ECS Managed Instances and ECS on Amazon EC2, and on Managed Instances it watches the MPS control daemon and replaces the instance if it turns unhealthy. It’s available in all commercial and AWS GovCloud (US) Regions.

AWS Lambda supports Seekable OCI (SOCI) lazy loading for Lambda Managed Instances – You can now have AWS Lambda fetch only the files needed at each stage of startup instead of pulling a whole container image first, which AWS puts at up to 70% off container startup time. You create a SOCI index, push it to Amazon ECR, point the image URI at the indexed tag, and Lambda detects it automatically. The technique also reached Amazon SageMaker AI training and processing jobs, with benchmarks: lazy loading held at 3 to 11 seconds regardless of image size, against a median 3 minutes 42 seconds for a 13.2 GB image, a range they report as 8x to 46x.

Amazon EKS Auto Mode now supports Kubernetes Gateway API – You can now declare how traffic reaches your applications with Gateway and HTTPRoute resources on Amazon EKS without installing a separate controller, and EKS Auto Mode provisions and manages the underlying AWS load balancing infrastructure. You get HTTP and gRPC routing, cross-namespace routing, weighted traffic splitting, header-based routing, and automatic TLS certificate discovery through AWS Certificate Manager. Existing Ingress workloads keep working alongside them, so this is additive rather than a migration. Gateway API support is included with EKS Auto Mode; standard load balancing charges apply.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts and resources that you might find interesting:

How Nubank automated disaster recovery with the AWS DevOps Agent – Nubank serves more than 140 million customers across Brazil, Mexico, and Colombia, and classifies over 300 business flows as critical, each tested on a regulator’s schedule. Nubank reports that handing evidence collection and report writing to the AWS DevOps Agent cut a test cycle from about 4.3 hours to roughly 40 minutes, and quadrupled flow coverage in 2026 to more than 160. One execution terminated more than 900 EKS instances under synthetic traffic with AWS Fault Injection Service, producing the evidence its regulator needed.

How MoEngage turned over 400 APIs into agent tools with AgentCore Gateway – Roughly 400 internal REST API operations, built by 25 teams, sitting in private VPCs across seven Regions, with tenant identity that could never become a parameter the model gets to set. MoEngage estimates the managed path saved about six months of platform engineering. The detail worth stealing: a full tool listing would have crossed 20,000 tokens per connection, so agents search for tools by describing a task instead of receiving the catalog.

Reducing petabyte-scale S3 storage costs with evidence-based lifecycle rules – The hard part isn’t deciding to act, it’s knowing which data has gone cold and where it can safely live. This uses S3 Storage Lens groups and Storage Class Analysis to prove access patterns before moving anything. In the engagement described, 5.8 PB of logs in Amazon S3 Standard at approximately $241,000 per month showed an access ratio below 0.1% past 90 days, and a 90-day transition to Glacier Instant Retrieval cut the bill by roughly 70%, a figure the author ties to that workload.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

AWS re:Invent – re:Invent returns to Las Vegas from November 30 to December 4, with more than 2,200 sessions. The event catalog is live and reserved seating is open, so register and claim your spots before they fill up.

AWS Community Days – Community-led conferences planned and delivered by community leaders. The 2026 season is winding down as re:Invent approaches, so check the hub for what’s still on.

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Daniel Abib

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

По буквите: Линч, Рой

2026-11-02 Зорница Христова

Post Syndicated from Зорница Христова original https://www.toest.bg/po-bukvite-linch-roi/

„Отвъд морето“ от Пол Линч

По буквите: Линч, Рой

превод от английски Иглика Василева, София: „Лист“, 2026

Двама души в рибарска лодка. Не е ясно докога, може би завинаги. Звучи като притча, като „Животът на Пи“, но с хора (всъщност в края на „Животът на Пи“ Ян Мартел подсказва, че също говори за хора…). Звучи като кръстоска между морско приключение и филм на ужасите – защото повечето съвременни сюжети, в които човек е принуден да търпи неопределено време компанията на друг човек – приятел, любима, роднина, – ни водят именно в този жанр. Човекът отдавна не мечтае за себе си в духа на „робинзониадите“ – как ще сътвори цивилизацията наново с голи ръце и с малко помощ от ближния.

Още повече че авторът е Пол Линч – създателят на смразяващата „Пророческа песен“, в която една днешна жена и майка не успява да повярва навреме, че страната ѝ бързотечно се е превърнала в опасна диктатура – и плаща непосилна цена за своята цивилизационна инерция, за желанието да продължи още малко постарому.

„Отвъд морето“ започва без заобикалките на „нормалността“. Още в началото виждаме „стария моряк“ Боливар, усещаме какво ще се случи, когато влезе в морето късно, въпреки предупреждението за буря, защото разни съмнителни хора го търсят. Дори младият му спътник Хектор предусеща нещастието и все пак ето ги – откъснати от света, живи след бурята, но с неработещ мотор, насред нищото,

насред времето – нашето време с плаващите в океана острови от боклуци, но и насред вечното време на всички истории за захвърлени сред вълните човеци.

Двамата герои са противоположности, но не дотолкова, че да представляват някакви архетипи – единият е стар, облъскан от живота, другият е млад, леко смотан… Различават се помежду си дотолкова, доколкото всяко човешко същество се различава от съседното. Друг човек.

Редуват се моменти на сътрудничество и конфликти на фона на съпътстващия ги лош късмет в набавянето на храна и вода и кратките изблици на надежда. Опитният „стар моряк“ се ядосва на липсата на инстинкт за самосъхранение у младия; „младият“ – на готовността на стария да жертва всичко в името на въпросното оцеляване, включително кратките, но опасни възможности за радост, като да поплуваш малко в океана. И разбира се, готовността му да бъде жесток към другите, в случая – към уловените птици. Препратката към Колридж и неговия албатрос е съвсем явна и отчетлива.

Това, което прави романа потресаващ обаче, не е постапокалиптичното съжителство между двамата, а онова, което се случва на финала, след като сюжетът завива в ненадейна посока, но старият моряк не разбира – отказва да разбере и да приеме какво се е случило. И точно този смразяващ финал носи и надеждата на книгата – показва каква физическа, неудържима нужда имаме един от друг.

„Моят подслон и моята буря“ от Арундати Рой

превод от английски Юлия Гешакова, София: изд. „Лабиринт“, 2026

В известен смисъл това също е книга за съжителството и сложните отношения между двама души, този път майка и дъщеря. Те не са сами насред океана, напротив, общуват през море от хора, понякога не се виждат с години по свое желание и все пак остават „в една лодка“ от началото до края. Дори когато се оттласкват една от друга.

Книгата проследява пътя на самата писателка – от детските ѝ години до днес. В романа има куп други неща отвъд отношенията с майката: проблемите на идентичността, любовта, реалностите на това да си жена в Индия, социалната ангажираност, политическите борби… Но отношенията с Мери Рой – амалгама от обич, възхита и негодувание, нужда от оттласкване и потребност от валидация, гняв и жалост – са като че ли основата, на която се развива всичко останало.

Мери Рой е достатъчно ярък персонаж сама по себе си – основателка на легендарно училище в Керала, истинска героиня за местните жени, успяла да промени закона, така че дъщерите да не бъдат оставени без наследство… феминистка, която успява да даде ясен урок на момчетата, опитващи се да тормозят съученичките си. И властна, безмилостна, нетърпима жена, която крещи по помощниците и най-вече по собствените си деца, която е способна многократно да те изкара от колата и да те остави сам на магистралата, независимо на колко си години, да те нарече какъв ли не публично, да те замеря с разни неща…

Отношенията на Мери и Арундати са магнетични за читателя не само защото осигуряват нужните за сюжета спойка и напрежение, а и защото човек неизбежно открива нещо от себе си там – било в едната роля, било в другата, не непременно в отношенията родител–дете. Независимо дали сте били някога в Индия, дали знаете какво точно е направила Индира Ганди по време на Извънредното положение, какво прави днес Моди, как изглежда национализмът в тази страна, в какъв свят са принудени да живеят жителите на изумително красивия Кашмир, какво е да си в Индия и да си мюсюлманин, да си жена или пък независим творец и т.н. Може да не знаете или да не ви интересува колко села са залети при строежа на язовира, срещу който протестира носителката на наградата „Букър“, може да не сте много наясно защо е готова да живее в гората с партизани, чиито методи не споделя напълно, но

основата на книгата – трудни отношения с близък човек – е универсална.

Възрастният, който иска да защити детето, което е бил. Смесицата от обич и страх. Болката при разрив с близък човек. Напрегнатата драма на семейния скандал. Но и облекчението, когато усетиш близост и обич, и симпатия след разрива, когато се научиш от уж прокудения вуйчо да обикнеш провала си, когато впиянченият баща е смешно, безобидно и някак симпатично присъствие, когато майката, от която си бягал, се окаже твоя вярна подкрепа точно когато трябва.

Универсалността на този разказ те засмуква и изведнъж ти се иска да узнаеш и онова, което уж е било удобно далече извън полезрението ти. И в това според мен е истинската сила на книгата – че те поставя в центъра на каузите си, че те пробужда за твоите собствени. Семейната драма става социална драма, а ти си част от нея.

В емблематичната си колонка „Ходене по буквите“, започната още през 2008 г. във в-к „Култура“, Марин Бодаков ни представяше нови литературни заглавия и питаше с какво точно тези книги ни променят. В началото на 2020 г. той я пренесе в „Тоест“. Вярваме, че е важно тази рубрика да продължи. От човек до човек, с нова книга в ръка. От края на 2021 г. по буквите тръгна Зорница Христова.

Активните дарители на „Тоест“ получават 20% отстъпка от коричната цена на всички книги на над 15 български издателства. Кои са те – вижте в условията на Читателски клуб „Тоест“.

Midterm Elections #lastweektonight

2026-11-02 LastWeekTonight

Post Syndicated from LastWeekTonight original https://www.youtube.com/shorts/iJIPA0cYK_8

Shutting the Door on the XCPE Botnet (Threat Intelligence)

2026-11-02 corbet

Post Syndicated from corbet original https://lwn.net/Articles/1104847/

The Google Threat Intelligence blog describes,
in great detail, the XCPE botnet, which has just been taken down.

Last week, Google and industry partners dismantled the XCPE
backdoor which enrolled more than 1.3 million set-top boxes into a
botnet. We assess with high confidence that the backdoor and botnet
were intentionally developed, deployed, and operated for financial
gain by Videostrong Technology Co., an Android Original Equipment
Manufacturer (OEM) and Original Device Manufacturer (ODM).

Videostrong monetized the backdoor by selling access to criminal
affiliates and loading those affiliates’ malware modules onto
infected devices. These secondary modules were conducting ad-fraud;
silently enrolling infected devices into residential proxy
networks; and loading additional malware onto XCPE-compromised
devices.

[$] Managing the kernel CVE flood

2026-11-02 corbet

Post Syndicated from corbet original https://lwn.net/Articles/1100894/

We recently looked at how the
kernel community is dealing with the firehose of vulnerability reports, but
said little about how users of the kernel are responding. At the 2026 Linux Plumbers Conference, Amazon employees
Dylan Johnson and Justinien Bouron discussed the problem from the point of
view of engineers trying to keep AWS running and secure in the face of a
storm of CVE disclosures. The kernel community’s longstanding advice is to
run current kernels, and that turns out to work — sort of.

Tor Browser 16.0 released

2026-11-02 jzb

Post Syndicated from jzb original https://lwn.net/Articles/1104846/

Version
16.0 of the Tor Browser
has been released. This is the first stable version based on Firefox ESR
153. Notable changes in this release include support for Linux on 64-bit Arm,
redesigned settings, and a “generic window titles” feature to help prevent other
applications from snooping on one’s browser activity. Tor Browser for
Android has had a number of interface updates to Google’s Material
Design 3 design language.

This release also follows Firefox in ending support for 32-bit x86 Linux
systems. See the full
changelog for all of the updates and bug fixes.

Security updates for Monday

2026-11-02 jzb

Post Syndicated from jzb original https://lwn.net/Articles/1104844/

Security updates have been issued by AlmaLinux (firefox, java-17-openjdk, kernel, kernel-rt, mariadb:10.5, podman, python-jwt, python27:2.7, python38:3.8 and python38-devel:3.8, python39:3.9 and python39-devel:3.9, ruby:2.5, ruby:2.7, thunderbird, util-linux, and virt:rhel and virt-devel:rhel), Debian (chromium, eog, imagemagick, python-aiohttp, thunderbird, and vim), Fedora (asciinema, atuin, btrfs-backup-ng, clevis, gcc, janus, kernel, libheif, libreswan, mingw-expat, mupdf, nautilus, nicotine+, ntpd-rs, openscap, perl-DBI, python-streamlink, python3.12, retsnoop, ruby, rust-cargo-deny, rust-cargo-rpmstatus, rust-ctutils, rust-gst-plugin-reqwest, rust-rbspy, rust-resctl-bench, rust-rpki, rust-rustls-webpki, rust-sequoia-chameleon-gnupg, rust-sequoia-git, rust-sequoia-octopus-librnp, rust-webpki-root-certs, rust-webpki-roots, tpm2-tss, and vhs), Oracle (.NET 8.0, .NET 9.0, aspell, bind9.18, curl, httpd:2.4, java-1.8.0-openjdk, kernel, libXi, mod_auth_openidc:2.3, nodejs:24, python-pillow, and tomcat), Red Hat (libxml2), Slackware (libarchive), and SUSE (buildah, curl, docker, gnutls, nodejs22, and suseconnect-ng).

Qualcomm Validates HBC Stacked Memory Ahead of 2027 Dragonfly AI250 Sampling

2026-11-02 Ryan Smith

Post Syndicated from Ryan Smith original https://www.servethehome.com/qualcomm-validates-hbc-stacked-memory-ahead-of-2027-dragonfly-ai250-sampling/

Qualcomm today is announcing that they have completed initial validation on their High Bandwidth Compute technology. The DRAM-stacked-upon-compute architecture moves on to commercialization, with Dragonfly AI250 set to sample in the first half of 2027

The post Qualcomm Validates HBC Stacked Memory Ahead of 2027 Dragonfly AI250 Sampling appeared first on ServeTheHome.

How we halved DNS build delay and cut memory use by 85%

2026-11-02 Alex Fattouche

Post Syndicated from Alex Fattouche original https://blog.cloudflare.com/dns-targeted-zone-builds/

When I last wrote about our DNS propagation pipeline in 2022, the Zone Builder received about 250 DNS record changes per second. Today it receives about 700, nearly three times as many, plus about 100 other events, such as zone setting changes. However, the number of events was never the real problem. The problem was the kind of work some of those events needed, and how that work grows with the size of a zone. Some zones now hold more than 20 million records, and we expect them to keep growing toward hundreds of millions.

Each event starts a clock that measures how long it takes us to turn that change into data our authoritative DNS servers can use. We call that work a build. A build reads the current database state, converts it into the format used at the edge, and writes any changes to Quicksilver.

In 2022, per-record builds were 150 times faster than full builds on average. On a test zone with one million records, per-record builds were 4,250 times faster. We ended that post with the obvious next step: do the same thing for zone settings.

Why doesn't the Zone Builder just rebuild the setting that changed?

It should, and that is where we ended up. However, proving when it was safe took most of the work. A record event identifies what changed. A zone event can come from many Cloudflare products, may not say why the zone changed, and can alter data shared by several zones with the same name. Choosing too much work wastes capacity. Choosing too little can leave stale or incorrect DNS data at the edge.

We built a targeted build planner that chooses the smallest build we can prove is safe, and a new scheduler that combines work for the same zone. Zone setting changes no longer rebuild zones with tens of millions of records. Together, they cut peak memory by 85%, and the extra headroom let us safely shorten waits the pipeline has always needed for resilience, halving the time from a DNS change to its data being written to Quicksilver.

How DNS changes reach the edge

A DNS zone is the collection of records and settings for a domain. Cloudflare uses a multi-stage pipeline to turn a change to that zone into data available across our global network.

A customer changes a DNS record through the DNS Records API or dashboard, or another Cloudflare service changes a zone setting.
The change is stored in a database.
The database change or another system produces a Kafka event.
The DNS Kafka Aggregator combines events from several Kafka topics and sends them to the Zone Builder.
The Zone Builder reads the required zone data and converts it into keys understood by our authoritative DNS servers.
The Zone Builder compares those keys with Quicksilver, our distributed key-value store. It writes only what changed, and Quicksilver propagates those changes across Cloudflare's network.

This is a simplified view. The Zone Builder does not copy database rows directly to the edge. It combines records with zone settings, applies DNS serving rules, and organizes the result so our authoritative DNS servers can read it quickly.

Why full builds stopped scaling

The original Zone Builder had one useful property: it did not need to know why a zone changed. It rebuilt the desired state from the current database state.

Read the current zone and its settings.
Read every DNS record in the zone.
Convert those records into Quicksilver keys.
Compare the new keys with those already stored.
Remove stale keys, write changed keys, and update zone metadata.

The final write contains only changed keys. However, a full build still reads, encodes, and compares every record, and holds the whole zone in memory while it does so. That is cheap for a small zone. For our largest zones, with tens of millions of records, the zone does not fit in memory at all. I explain how we handle that below.

Zone settings made targeting harder. Universal SSL is one example. Enabling or disabling it updates metadata and can change Certification Authority Authorization (CAA) records at the zone apex, the bare domain, such as example.com. CAA records tell certificate authorities whether they may issue a certificate for the domain.

A full build only needed to know that the zone changed. A targeted build must answer three questions:

Which DNS names can change?
Can shared metadata or the list of published zones change?
What previous state must be removed from Quicksilver?

Answering those questions safely meant solving five problems, which the rest of this post covers in order:

Events had to say what changed.
Each change had to map to the smallest build that is still correct.
Builds had to account for data shared by every zone with the same name.
The database could change between the event and the build.
Combining work could not lose a Kafka message.

Making events say what changed

We changed the event producer to include changed zone fields and settings, with previous and new values when available. We then traced each setting through the code that generates DNS output and classified what it could affect.

Not every event was that specific. Some database functions only update a timestamp to say, in effect, "something about this zone changed." Sometimes that is harmless bookkeeping. Sometimes it means data changed elsewhere and must be read again.

Those cases look identical in the event. Until we can identify the caller and its purpose, a timestamp-only update still causes a full build. The same rule applies to missing details, unknown events, or any change that may affect records anywhere in the zone. The planner never turns uncertainty into less work.

Choosing the smallest safe build

Quicksilver output has two parts that matter here. Record keys contain the DNS answers for one zone and name. A smaller shared metadata key tells our authoritative DNS servers how to use that zone, including where to find special records such as nameservers and wildcards. Some events change only record keys; others must update both.

With that distinction, the planner can choose one of five build types:

Resources only: Rebuild the records at specific DNS names, which we call resources, without changing zone metadata. Ordinary A, AAAA, TXT, and CNAME changes usually take this path.
Resources and metadata: Rebuild specific names and shared metadata. Nameserver and wildcard changes use this path.
Apex and metadata: Rebuild the zone apex and metadata. Settings such as Universal SSL can use this path.
Purge: Remove a zone that should no longer be published.
Full zone: Rebuild every record when smaller work cannot be proven safe.

The rule is simple: every key a targeted build writes must be byte-for-byte identical to what a full build would write. If we cannot prove that, we do a full build.

Removing old state works as it did for per-record builds in 2022. Record events include deleted records and the previous version of updated ones. The Zone Builder then scans Quicksilver for the existing keys under each name it rebuilds, removing any that the new build did not write.

Full builds remain necessary when a change can affect records anywhere in the zone or when a zone may need to be published for the first time.

There is one deliberate exception. A full build holds every record from the database and every existing Quicksilver key for the zone in memory at once, and our largest zones do not fit. Above a fixed record-count threshold, the Zone Builder never escalates to a full build. It rebuilds the apex, the metadata, and every name identified by the queued events instead. We accept this tradeoff because a full build of those zones cannot finish, and, while it retries, it holds back every other change on its Kafka partition. Many of the cases that would otherwise need a full build involve several zones sharing one name, and zone holds reduce that risk by stopping the same name from being added to another account. For the rare cases that remain, such as enabling a zone on another network, we can manually rebuild individual names in the zone.

When one name means more than one zone

Quicksilver stores each zone's record keys separately. It also stores one metadata key shared by every Cloudflare zone with that domain name. This shared key acts like a directory: it tells our authoritative DNS servers which internal zone IDs they may use when answering a query.

Why can one name have several zone IDs? During customer onboarding, offboarding, and other lifecycle changes, multiple Cloudflare zone objects can briefly exist for the same public domain. We use a priority policy to choose a small, bounded set to publish.

Consider four internal zones for example.com: Zone 101, Zone 202, Zone 303, and Zone 404. The metadata key currently lists Zones 101, 202, and 303. Zone 404 exists, but it is not selected. The numbers identify separate Cloudflare zone objects; customers still see one domain name.

A TXT record change at www.example.com in Zone 202 affects only Zone 202's record keys, so the metadata key does not change. Nameserver and wildcard changes are different because the metadata key records information about them. Those builds must update both the zone's records and the metadata key.

At execution time, the Zone Builder reads the current list again. If it writes the metadata key, it rebuilds the whole key from that fresh view rather than patching an older copy. It writes the triggering zone's records before the metadata that refers to them.

Planning from the event, verifying before writing

An event describes what happened in the past. A build may wait while more events arrive and the database changes, so planning happens twice. The event first creates a conservative work list. Immediately before execution, the Zone Builder reads a fresh publication snapshot: the current zone settings, whether this Zone Builder should publish the zone to its network, and which same-name zone IDs belong in the metadata key. Here, a network is a separate, isolated copy of Quicksilver for one environment. Each Zone Builder deployment writes to one network, and a zone can be enabled on some networks but not others. For example, a zone can be served from Cloudflare's global network and also from the China Network.

Fresh state can change the result in four ways:

A targeted build can become a full build if the zone now has a feature that requires complete-zone work.
A build can become a purge if the zone should no longer be published to this network.
A build can be skipped if the triggering zone no longer belongs in the metadata key.
A metadata build can reconstruct the shared key from the current candidate set.

The fresh read comes from read-only copies of the database, called replicas, so builds do not compete with customer writes. Kafka delivery and database replication move independently: an event can arrive before its database change is visible on a replica. The DNS Kafka Aggregator delays events long enough to cover normal lag, and a final database check rejects the read if that replica is still behind.

Combining changes in one scheduler

Record work and full-zone work previously used different schedulers. Each could combine its own work, but not work from the other scheduler. We replaced them with one unified scheduler. The planner decides what must be built, and the scheduler decides when it can run.

When several events for one zone arrive before a build starts, the scheduler combines their work:

An A record changes at www.example.com, requiring that name.
A wildcard changes at *.example.com, requiring that name and shared metadata.
The scheduler combines both names and the metadata update into one build.

Merging only adds work. If any event requires the complete zone, the combined build becomes a full build. A purge overrides every rebuild because its desired result is that the zone is no longer published.

The scheduler tracks two things separately: the DNS work that must be performed and the Kafka messages waiting to be acknowledged. A retry preserves both. Even when a full build or purge no longer needs the individual record names, the messages stay attached until the result is safely written.

Keeping Kafka moving

One slow build can delay unrelated customers even when workers are available. Kafka divides messages into ordered partitions. For each partition, it stores one committed offset: the point before which every message is known to be complete. Moving that offset past unfinished work could lose a customer change after a restart.

We use soft committing, which I previously described in my Secondary DNS deep dive. A build can finish out of order and be marked complete in memory, but Kafka's committed position advances only after every earlier message has also finished.

In the diagram, message 2 is still running, so messages 3 through n cannot be committed. If the Zone Builder restarts, those in-memory completions disappear and Kafka replays messages 2 through n. We deliberately choose repeated work over losing a change.

Targeted plans reduce how long the oldest message stays unfinished. The scheduler also prevents one busy zone from taking every worker. After a zone builds, its next build waits for a per-zone rebuild interval, which differs by Cloudflare plan. Changes that arrive in the meantime are merged into that next build. The scheduler normally enforces this interval, but lets old queued work run before it blocks a Kafka partition for too long. Zone locks and retry delays still apply, so a failing build cannot run in a tight loop.

Releasing safely

A smaller build is useful only if it produces the same DNS data. We added equivalence tests for nameserver, wildcard, and apex paths. Each test runs a full build and a targeted build from the same database state, pins time-dependent values, and compares every Quicksilver key owned by the targeted plan byte for byte. Planner and scheduler tests cover classification, same-name zones, merging, purges, and escalation.

During rollout, we also ran a full build next to every targeted build and compared the output. It was expensive by design, reading the complete zone every time, so we used it only until the mirror comparison below gave us the same confidence.

Tests cannot reproduce every customer combination, so every release also runs through a mirror Zone Builder. It consumes the production Kafka stream and reads production database state, but writes to a separate Quicksilver environment that serves no customer traffic.

A comparer checks a 20% sample of zones for missing keys, extra keys, and different values. The two environments do not always build a zone at the same instant, so a temporary mismatch is not automatically a regression. We look for new types of differences that remain after both pipelines catch up. That gives us confidence that targeted builds produce the same output as production.

We built the DNS Mirror Dashboard to group those differences and keep examples for investigation. Naturally, it runs on the Cloudflare Developer Platform: a Worker receives results, Durable Objects maintain state, and Cloudflare Access protects it. The critical groups in the screenshot below were each checked and confirmed safe before release. The diff rate is the share of compared zones where production and mirror disagreed on at least one key at the moment of comparison. Most of those are timing differences that disappear once both pipelines catch up, so we watch two things: that the rate stays low, and that no new kind of difference appears.

After the mirror results remained healthy, we rolled the scheduler out by a hash of the zone name. Hashing the name rather than the zone ID kept every zone sharing a name on the same scheduler, so two schedulers never wrote the same metadata key. We increased the percentage for free zones, then repeated the process for paid zones.

Results

Over 24 hours, we measured 86.81% of builds as resources only, 2.54% as resources and metadata, 2.30% as apex and metadata, and 1.30% as purges. Together, 92.95% of builds avoided rebuilding the complete zone. Full builds were the remaining 7.04%.

The two metadata build types were full builds before, even for large zones. Together, they account for 4.84% of builds. Combined with the 7.04% that are still full builds, that is 11.88%, so full builds fell by about 40%. Their share of the count also understates the work avoided, because they include the largest zones. The full builds that remain average 0.35 seconds, because the events that still trigger them, such as a zone being published for the first time, mostly affect new zones with few records.

In 2022, we also planned a batching system. The unified scheduler makes it unnecessary: it does not wait to form batches, and it merges events only when work is already queued for the same zone. Before, long full builds held back the committed offset for their partition. With fewer of them, we now consume Kafka at the rate events arrive, instead of falling behind and catching up in bursts of up to 1,550 messages per second.

We also measured build completion delay: the time from a DNS record or zone setting change until the Zone Builder finished writing to Quicksilver. It does not include Quicksilver's distribution across our network, which varies by location and is unaffected by this work. Production ran the old schedulers, and the mirror ran the unified scheduler on the same events over the same 24 hours. The two environments are identical apart from the code: they consume the same Kafka events, read from the same database, and write equivalent data to Quicksilver. That makes this a controlled, side-by-side comparison. The figures combine every build type and customer plan, so they are not a latency guarantee for an individual zone. A large share of this time is the rebuild interval described earlier, not build work. At each measurement point, p99 is the time within which 99% of events finished. Typical delays barely changed here, because most builds were already small. The difference is in the tail: the largest p99 spike fell from 14.6 minutes to 1.21 minutes, a 12-fold reduction. The chart uses a logarithmic scale.

Targeted builds also load complete large zones far less often. Aggregate working-set memory, the memory actively used by the deployment, fell from an average of 49 GiB to 7 GiB, an 86% reduction. Peaks fell from 79 GiB to 12 GiB, an 85% reduction.

Those were the immediate results of the new scheduler. They also let us revisit two safeguards. The first is the rebuild interval, which stops one busy zone from rebuilding constantly and taking capacity from everyone else. With builds now cheaper, and far fewer of them reading complete zones, we could safely shorten that interval on most plans. The second is how long the DNS Kafka Aggregator holds each event, to make sure database replicas have caught up before we read from them. Once our databases began rejecting reads from replicas that had fallen behind, that buffer could safely be smaller. Together, these halved the median build completion delay, from about 7.7 seconds to about 3.3 seconds. In the chart below, the first step down is the DNS Kafka Aggregator change. The red band marks the Zone Builder release with shorter rebuild intervals. That release rolled out gradually across our servers, so the delay falls in two further steps rather than all at once.

The p90 and p99 lines in that chart are less precise than they look. The histogram had buckets at 1, 5, 10, and 30 seconds, and a percentile that falls between two edges is estimated as if delays were spread evenly across that range. Most delays sit near the bottom of their range, so those estimates come out high: p99 reads close to 30 seconds, and p90 above 7 seconds. The exact average, which does not depend on buckets, confirms the halving. We have since switched to a native histogram, which chooses fine-grained buckets automatically. It puts p90 at about 5.5 seconds and p99 at about 12 seconds.

Why full builds still exist

In 2022, we said we planned to eliminate full builds entirely. We now know that is neither realistic nor useful. A zone being published for the first time needs every record written. A change that can affect records anywhere in the zone needs every record checked. An event that does not say what changed still gets a full build, because the planner never turns uncertainty into less work. For a small zone, a full build is also cheap.

As we learn what more events change, we will keep moving them to targeted builds. However, the biggest gain has already landed: the events that used to rebuild our largest zones no longer do, and the full builds that remain are mostly triggered by events, such as first publication, that affect new, small zones.

Want to know what happens after the Zone Builder writes a change? Read how Quicksilver distributes configuration to every Cloudflare data center in seconds.

The Great Paris Flood of 1910

2026-11-02 The History Guy: History Deserves to Be Remembered

Post Syndicated from The History Guy: History Deserves to Be Remembered original https://www.youtube.com/watch?v=djEqjQpr7eI

AI Vulnerability Finding

2026-11-02 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/11/ai-vulnerability-finding-2.html

A comment:

3% of ‘critical’ software security bugs, in all of recorded history, have been found in the last month. This is on pace to find another ~100,000 bugs this year alone, adding ~26% more vulnerabilities to the all time total Do you realize how fucking insane that is?

Yes. It is a new world out there.