AI answer engines read a publisher’s page and hand the reader a summary, so the visit, and the revenue that would come with it, never happens. Most publishers will never sign a licensing deal with the companies that use their work in AI products, and no company can negotiate with millions of sites. The web needs a way to say “yes, if you pay.” Pay Per Use is one way to say it, and it's now in beta.
In July, we outlined our plan for Pay Per Use. Since then, we’ve been working with buyers and content owners to bring it to life. The gist: A buyer offers a price for a specific use of your content. You choose whether to accept. The buyer reports each use, and Cloudflare bills the buyer and pays you. Publishers track usage and earnings in the Cloudflare dashboard. Buyers report usage through a single API.
Pay for the use, not the crawl
AI products fetch far more than they use. A search engine indexes pages it never shows. Charging for every crawl makes the buyer pay before it knows what it needs, and many buyers won't. Paying for use ties the price to the value the buyer actually gets, which we hope brings more buyers to the table and more money to publishers. Pay Per Crawl, which we launched in 2025, charges for access. Pay Per Use pays for what happens next. Publishers can choose the model that suits them.
For buyers, the case is just as simple. Some of the content your product needs is behind a block or a paywall today, and it’s the content that changes fastest: news, research, specialist trade publications. Pay Per Use lets buyers make an offer. You pay only for the content your product actually uses, you’re identified to every publisher as a verified buyer, and one API connects you to every site that says yes. You won’t need thousands of integrations or individually-negotiated contracts.
Each AI company defines the use it will pay for and sets a price. Publishers decide which offers to accept, and then can see how often their content is used and what it has earned. Cloudflare handles enrollment, usage records, billing, and payment.
How Pay Per Use works
Pay Per Use lets publishers make their content available to identified AI companies on terms that turn downstream use into revenue. Publishers choose which programs to join and retain control over crawler access and downstream uses. AI companies identify their crawling activity using Verified bots and, when permitted by the publisher’s controls, can access and index content owned by the publisher. That content can later power many experiences: a cited answer in AI search, a passage quoted in a research agent's report, a product review weighed by a shopping agent, or a recipe adapted by a cooking assistant.
Let’s show how it works!
1. Define what counts as a paid use
Each AI company sets up its program with Cloudflare. It identifies its crawler, defines the use it will pay for, sets a price, and connects a payment account.
Consider two possible offers. A search service could pay when it returns an excerpt from an enrolled page to a customer. A shopping agent could pay when an enrolled review shapes a recommendation. Under the second offer, payment follows use even if the shopper never reads the review.
The same article can create value in different products and publishers can accept different payment offers for different uses. The buyer proposes the definition of the use being paid for; Cloudflare provides the reporting and payment infrastructure.
2. Publishers choose whether to participate
Publishers review offers in the Cloudflare dashboard, under Monetize → Pay Per Use: it will show the AI company, the use it pays for, and its offer price. Publishers decide whether to accept, and can stop participating if an arrangement no longer works for them. There is no origin change or technical integration with each AI company. Each program’s terms also define what the AI company may do with the content, including any restrictions on training.
3. The buyer reports each use
The buyer fetches the list of domains that have accepted its offer, then reports each use as one line of JSON: when it happened, the URL the content came from, and an event ID.
Usage is self-reported: the program terms require complete reporting, and Cloudflare checks that each reported use maps to an enrolled publisher.
4. Cloudflare settles both sides
We aggregate reported uses, charge the buyer, and pay publishers monthly through their connected payment account. Buyers get one integration. Publishers get one place to see offers and earnings.
Payment is only half the product
Today, publishers see how often each AI company uses their content and what it has earned, by domain and over time. Business Insights already shows which crawlers visit and what they take. Pay Per Use shows what happens next: whether that content was actually used, how often, and what it earned.
Next, we’re working with AI companies to report to publishers more context about each use, such as the keywords that led to the citation, the topic of the request, or the product it powered, with no personal data being exchanged.
Our Answer Engine Optimization (AEO) tool shows how assistants answer questions about your work. Pay Per Use shows what buyers report using, and what that use earned. Together, they connect how your content is found, how it's used, and what it pays.
That helps with two kinds of decisions. Commercial ones: which content earns, which uses are worth it, and whether to keep participating. And editorial ones: what to cover, what to update, and what to make easier for agents to find.
What comes next
During the beta, we’re working directly with each buyer and with the publishers who opt in. The question is simple: do both sides want to keep going? Buyers need content that improves their products at a price that works. Publishers need a return that makes participation worthwhile, reporting they can trust, and payments that arrive on time.
Over time, we want new buyers to onboard with their own products and payment models, without rebuilding enrollment, reporting, and settlement for each publisher.
For publishers, that means one place to accept or decline offers, counter on price, and price content differently by use. Last week’s reporting may be worth more to an AI product than a ten-year-old archive page, and you should be able to charge for that.
The beta will help us determine how to make those choices simple and practical at scale.
Building an economic layer for the agentic web
Content should be able to reach new audiences and generate sustainable revenue, even when it becomes part of someone else's product: a cited search result, a shopping recommendation, an agent's report. Pay Per Use turns those uses into a commercial relationship. A buyer makes an offer, a publisher chooses whether to accept, and the use of valuable content leads to payments and records of what happened.
Not everything should be sold the same way. High-value content needs a trusted network of verified buyers who report how they use it, and that's Pay Per Use. APIs, tools, and data are frequently different, because every request is the use. For those, Monetization Gateway (also in beta as of today) lets sellers charge agents per request, using the open x402 protocol. The two run on the same foundations: identity, metering, pricing, and analytics.
Publishers shouldn't have to choose between blocking every agent and giving their work away. Pay Per Use gives them a way to say yes, on clear terms, with an account of what happened.
When we launched User Insights last month, we wanted to help teams answer a basic question: What are people actually doing with AI? User Insights gives teams a clearer view of their AI usage, showing which users, applications, tasks, and models are driving traffic. It also highlights user and agent anomalies, helping teams identify unexpected or out-of-control spending and usage before they become larger problems.
Our latest update adds something our users have been asking for: context.
Since launch, we’ve heard from users that model names and request counts only tell part of the story. They show where traffic is going, but reveal little about the work behind it: is that request a code review, a research task, or an agent making several calls to complete a job? The same token count can represent very different kinds of work, and you can’t evaluate with model choice without understanding the task.
User Insights now shows when a model may be more capable than a task requires, which of your users and agents are driving that usage, and how the task, model, cost, and conversation patterns relate. Teams can use these insights to investigate and make targeted changes within their organization. These capabilities are available for free to AI Gateway users.
Why AI usage is hard to understand
Consider a team that has routed its internal AI traffic through AI Gateway. After a few weeks, spending is increasing and some requests feel slower than expected, a common challenge as organizations adopt AI at scale.
There could be several explanations. Developers may be using AI for increasingly complex coding work. Agents may be making too many follow-up calls to complete a task. Or a small group of users or agents may be responsible for a disproportionate share of the organization’s usage.
Tokens and request counts alone cannot show which pattern is driving the increase. Teams need to understand what the traffic represents before deciding whether a model, workflow, or routing rule should change.
Helping teams find where AI models are overkill
The model overkill view helps teams identify conversations where the selected model appears to be more capable than the task requires. For example, a team might discover that users or agents are sending simple formatting or summarization requests to a high-capability reasoning model.
That gives the organization a place to start. They can see which users, agents, or applications are associated with the pattern, then investigate the tasks behind it. A team might find that a model is being used because it is the default, because users are unsure which model to choose, or because an agent has been configured to use the same model for every step.
The overkill view is not a leaderboard and does not automatically recommend a replacement model. It helps teams ask better questions:
Is this model appropriate for the task?
Is the extra capability improving the result?
Would a faster or less expensive model produce an equivalent outcome?
Is the issue limited to one workflow, user, or agent?
From there, teams can compare cost, latency, token usage, and conversation turns before deciding what to change.
These insights support both the new Potential Savings view and the Auto Router, which is launching in public beta alongside this release. The Potential Savings view helps teams identify requests that may be handled by a faster or less expensive model without compromising output quality. The Auto Router applies these task and model-fit signals automatically, helping reduce costs without requiring a separate routing rule for every workload.
The Overkill view is a starting point for evaluating model fit. Teams can compare latency, input and output tokens, conversation turns, and total cost for the same type of task. A difficult coding or research task may need a capable reasoning model, while a short summary or simple classification task may not. The goal is not to move every request to the least expensive model, but to understand whether the selected model is appropriate for the work.
Understand what people are using AI for
Task analysis groups conversations by the kind of work they represent. Initial categories include coding, research, writing, summarization, and data analysis.
This provides context that a list of model names cannot. An engineering team might use AI mostly for coding and debugging, while another team might use it for research and summarization. A team may also discover that a surprising amount of traffic comes from simple tasks, even though those tasks are being sent to a high-capability model.
The answers will vary by team. The category data provides a way to investigate those differences using traffic already passing through AI Gateway. Teams can determine whether a model is being used for the work it is best suited to handle, or whether a default model is being applied too broadly.
Understand the full cost of a task
Some tasks are finished in one exchange. Others take a few rounds of questions, corrections, and follow-ups. Turns analysis shows how much back-and-forth different tasks require. A long conversation is not necessarily a bad thing, especially for complex work. But if a simple task keeps taking several turns, it may be worth looking at the prompt, the model, or the workflow.
The first request is only part of the cost. Teams should also look at the time, tokens, and money spent before the task is finished. Comparing those numbers can show where a workflow is taking longer or costing more than expected.
Turn insights into auto routing
Once a team has identified an overkill pattern and confirmed it across task, cost, latency, and turn data, it can turn that insight into an automatic routing decision.
For example, the task view might show that much of the team’s AI usage is summarization and formatting. The model view could show that those requests are being sent to a large reasoning model, while the turns view shows that most conversations finish in a single turn. Together, these signals give the team a concrete workload to evaluate.
In addition to our updates to User Insights, the Auto Router is now available in closed beta. The Auto Router uses the conversation trajectory, task category, task complexity, and model-fit signals to automatically route requests to an appropriate model while taking cost into account.
Instead of creating a separate routing rule for every workload, customers in the beta can let the Auto Router select among the models available to their application. The router does not simply send every request to the least expensive model, but instead selects an appropriate model for the task at hand. Complex coding or research work may still need a more capable model, while simpler tasks may be handled by a faster or less expensive option.
To learn more about the Auto Router and sign up for the closed beta, read the blog post here.
The Auto Router uses the same task and conversation signals that power User Insights. The section below explains how those signals are produced.
How User Insights classifies traffic
Each conversation receives an analysis signal that can be grouped in User Insights. The signal is used for reporting and routing analysis, and is not intended to replace or expose the original request.
The categorization engine is a dedicated Cloudflare Worker that processes eligible AI Gateway logs. It examines the conversation trajectory, including user requests, assistant responses, tool calls, and tool results, and identifies the type of work being performed, such as coding, debugging, research, or summarization. It also returns a confidence score and evaluates dimensions such as task complexity, intent ambiguity, stakes, and context dependence.
The Worker returns a category that can be joined with the log metadata used by the dashboard. These signals can also be used to evaluate model fit by comparing how well candidate models suit the task against their cost. The current implementation focuses on a small set of categories that are easy to understand, rather than trying to infer every detail about a user’s work.
The pipeline follows the existing AI Gateway log architecture. Metadata is stored separately from log bodies, and the current implementation uses Durable Objects for metadata and R2 for log bodies. User Insights exposes derived categories and aggregate views. It does not turn the dashboard into a raw prompt browser. Retention of the underlying log bodies continues to follow the configured AI Gateway logging behavior, so teams should review those settings when deciding what to send through the classifier.
The classification is asynchronous, which means it happens after AI Gateway has handled the request rather than while the user is waiting for a response. AI Gateway writes the log to the existing storage path first, and the classification Worker processes it afterward. This keeps classification out of the request path and adds no latency to the user’s response.
The tradeoff is that User Insights is not a real-time view. Newly received conversations may not appear in the dashboard immediately, and analysis may trail incoming traffic by approximately one day as logs are processed and aggregated. Teams should use User Insights to identify usage patterns over time rather than monitor live request activity.
The flow looks like this:
Connect usage to users, teams, and tools
Task categories become more useful when they can be viewed by user, team, or application. AI Gateway is identity-aware, providing that context without requiring teams to build a separate reporting pipeline.
This works not only for applications that teams build themselves, but also for developer tools and agent harnesses such as Claude Code, Codex, and OpenCode. By putting AI Gateway behind Cloudflare Access, teams can connect authenticated users and sessions to their AI traffic, allowing User Insights to associate activity with the right person and conversation.
For custom applications, requests must include both a stable user_id and a session_id for User Insights analysis. The exact identity configuration and field names depend on how the application or tool is set up. The important part is to provide stable, non-sensitive user and session identifiers so usage can be grouped without putting identity data in the prompt itself.
For custom applications, the request metadata might look like this:
The request body contains the model and messages for the conversation.
With Access configured in front of AI Gateway, tools such as Claude Code, Codex, and OpenCode can inherit this identity context automatically. Cloudflare Access is available at no cost for teams with up to 50 users, making it an easy way to get started.
Get started with AI Gateway User Insights
AI usage is changing quickly. Models change, teams develop new workflows, and the right choice for one group may be the wrong choice for another.
User Insights lets teams start making smarter choices by identifying where certain models may be overkill. They can then see which users and agents are driving that usage, understand the tasks behind it, and compare the cost of completing the work.
Learn more with the AI Gateway User Insights documentation . Open AI Gateway in the Cloudflare dashboard, and use what you learn to make more targeted model and routing decisions.
For most of its history, the Internet had one audience that paid the bills: people. We read the articles, saw the ads, and bought the subscriptions. Bots were always there, but they were mostly large, automated operations that didn't view ads, pay for anything, or read in any meaningful sense.
That's changing fast. At the end of 2024, Cloudflare handled an average of 63 million HTTP requests a second. Today, it's almost doubled to 115 million, with peaks above 150 million. Over the past year, daily requests from AI agents on our network grew by more than 1,700%. This year, for the first time, more than half of Internet traffic wasn't human.
The human web didn't shrink to make room. A second audience arrived alongside it: agents, software acting on behalf of people. They sit somewhere between humans and traditional bots. They don't respond to ads, but there's usually a person behind them with a job to get done. For businesses that learn to serve them and capture value from them, agents are additive. For those that don't, they're extractive.
What our customers need hasn't changed: to be discovered, to tell great stories, to build great experiences, and to sell. What's changed is that more than half your visitors are now software. Our job is to help you serve both audiences.
More traffic, less revenue
For thirty years, the web ran on one arrangement: you let search engines crawl your site, they sent you visitors, and you turned those visitors into a business. Being found and getting paid were the same thing.
AI has caused this delicate balance to break down. Now, answer engines read the page and give the reader a summary. This costs websites bandwidth without leading a human to a website where the ads or payments happen. The machines kept coming, and the audience that paid for the web stopped reaching those sites. Some of the most heavily crawled categories, like Retail, Computer Software, IT & Services, and Financial Services, have seen human traffic decline as much as 40% in less than one year.
The result is that revenue per request is falling while costs are rising. Every automated request still costs bandwidth, compute, and origin capacity, and a growing share of those requests carry no referral, no ad impression, and no subscription. Our first instinct was to block all automated traffic. Last year we recommended blocking AI training crawlers on new domains so site owners could at least say no to their content being used to build models. In Spring 2025, 22% of the crawler requests we saw were for AI training (according to the crawlers’ stated purpose). By June 2026, it was 52%. The problem is a blanket “no” is not a sufficiently nuanced approach for the Internet economy being built right now.
The opportunity is there to cater to agents. Get it right, and you are at the forefront of a new business model. Get it wrong, however, and the results will be the same as they were for generations of websites that were on the wrong side of search engine algorithm changes.
Some of that traffic is a customer
An agent booking a table, comparing insurance quotes, or buying a dataset for a researcher is a customer. It just isn't a human one.
The fastest-growing part of automated traffic is no longer crawlers. It's agents: software fetching pages on a person's behalf, often because the human asked a chatbot something. That agent traffic follows human routines, with a weekly rhythm and a dip over the summer holidays. Turn an agent away, and you may be turning away the person who sent it.
Agents also behave differently from training crawlers. A training crawler collects your pages to build a model. An agent comes back each time someone asks about that content, so this traffic grows with how many questions people ask, not how much you publish.
You can't do business with an audience you can't see, can't tell apart, can't set terms for, and can't charge. Until recently, for most of the web's non-human traffic, none of those four things were possible.
See who’s really visiting
"AI bot" no longer means anything useful. What matters is what a bot does. Cloudflare’s AI Crawl Control, Business Insights, and BotBase show site owners who is crawling, what they take, what comes back, and which of your URLs they want most.
A bot's name is only worth something if you can trust it. With Web Bot Auth, operators, including OpenAI, Google, and AWS, cryptographically sign their agents' requests, so a site can tell a real agent from an impersonator without guessing from IP addresses or user-agent strings. We see more than 500 billion verified bot requests each week.
Set your terms
In July, we replaced the single "block AI bots" switch with separate Search, Agent, and Training controls, available on every plan, including Free. The data showed why that distinction was needed. Fewer than 1% of sites on Cloudflare block search crawlers, while 17% block training. Site owners were never trying to hide. But with the rise of agentic traffic and the new ways agents use information, they suddenly had no transparency into, or choice over, how their content was being used. Being found no longer ensures they get paid, and they want to be found without being exploited.
That’s particularly difficult in the case of mixed-use crawlers. When one bot does both search and training, refusing one means refusing the other. On September 15, we shipped Disallow AI Training. It keeps you indexed for search while using crawler-specific mechanisms to instruct the operator not to use your data for training. Apple, Google, and Microsoft have committed to honor it. Cloudflare Radar also publicly tracks crawler behavior.
New domains now see recommended configurations based on how the site makes money rather than what piece of software is visiting. For ad-supported sites, you can easily disallow training and block agents on pages that carry ads, because an ad only pays when a person sees it. You can change any of these settings at any time.
Get paid
In August 2026, we described the Agentic Internet we're building as readable, discoverable, callable, and payable. The last word, payable, is the one that determines whether the open web can fund itself. The web needs a way to say ‘yes, if you pay’ instead of a binary ‘yes’ or ‘no’.
The licensing market shows both how much demand there is and where the gaps are. More than 50 publisher-AI deals have been signed since 2023. Nearly all of them are bespoke and bilateral, between large publishers and large AI companies. They prove content has value. But they don't reach most of the web, and they don't reach most buyers.
Not every asset should be sold the same way. High-value content and datasets need a trusted network, where buyers are identified and report how the work was used. Services like APIs and MCP tools don’t work like that: every request is the use.
So we're building for both.
Pay Per Use reaches the sites that direct licensing can't. Most publishers will never get a bespoke deal with each AI company, and no AI company can negotiate with millions of sites. Pay Per Use is the bridge. It doesn't charge for the crawl. It pays when content is actually used. Every buyer is a verified crawler, which is what makes this a trusted network, and each one defines what counts as use and what it will pay.
Publishers see the offer, choose whether to opt in, and are able to opt out whenever it stops working for them. The buyer reports each use, Cloudflare checks those reports, then bills the buyer and pays the publisher. The reporting matters as much as the payment. Publishers see what was used, when and what they earned, and, where the buyer reports it, information about which questions surfaced their work. Licensing deals rarely show any of that. It creates a feedback loop: publishers learn what people are actually asking for, and from that can decide what to cover, what to update, and what to make readily available to agents.
There won't be one definition of use. A search engine citing a source, a research agent quoting a passage, and a shopping agent completing a purchase create different kinds of value, and each will want its own business model. Buyers can participate via multiple business models using the same rails, with no new integration for publishers. Take a trade journal for marine engineers, with a few thousand subscribers and little prospect of an AI licensing deal. It gets paid by every participating AI company that draws on its work.
Monetization Gateway captures value that has never had a way to change hands. Accounts, API keys, and subscriptions work for customers you already know, not for an agent that wants one lookup from a service it has never used before. Our closed beta allows eligible U.S. Cloudflare customers to put a price on anything that passes through us, using the Rules language they already know. When a rule matches, we return an HTTP 402 Payment Required using the open x402 protocol, and the agent pays the seller directly.
That does more than recover lost revenue. Agents are customers in their own right: they pay for the data, APIs, and tools they use, whether the request is the whole purchase or one step in a larger task.
Monetization Gateway prices per request, per query, or per token, at fixed or capped prices. A sports statistics site built on ads can charge a fraction of a cent each time an agent asks "who leads the league in assists?" When we announced Monetization Gateway, thousands of sellers joined the waitlist, and their most common request was "charge agents, not humans." We're also our own first customer. Cloudflare's AI Gateway uses Monetization Gateway to let agents pay for inference, so we find the rough edges before our customers do.
For buyers, both products beat a block page: reliable access, and a way to reach millions of sites instead of one licensing deal or API key at a time. Every paid request leaves a receipt showing what was bought and that it was paid for.
Both Pay Per Use and Monetization Gateway are bets, built with customers on shared primitives: identity, metering, pricing, settlement, and analytics. They work together, so a publisher can disallow training, allow search, earn from AI answers, and charge agents per article from one dashboard. Pricing and discovery aren't solved yet, which is why both launch as betas, shaped by real customers and real transactions.
Make every request cheaper
Payment is the answer to falling revenue. Rising cost is a different problem, and much of it is simply waste. Most crawlers still download pages built for humans, again and again, to extract a few paragraphs of text. Too often, bots crawl sites that haven’t changed since the last attempt. That burns bandwidth for the site and compute for the crawler, and it happens before any answer is written. We’re working with our customers and the crawlers on tools that will help. Today, you can see the bandwidth consumption used per operator in our dashboard.
In July, we announced a joint research project with OpenAI, a first-of-its-kind pilot to explore how insights from Cloudflare’s global network can help AI search engines discover and index relevant content on the open web more efficiently and effectively. We’re planning to share our initial results in the next few weeks.
For our customers, we’re shipping tools and one-click experiences to make their sites optimized for this new kind of traffic.Markdown for Agents lets agents read a page without the additional styling meant for human eyes, and WebMCP lets a site expose actions directly instead of making agents guess which button to press.
Why build on Cloudflare
More than 20% of the web sits behind Cloudflare’s network, and so do nearly 80% of leading AI companies. We see both sides of this market. We build the rails for visibility, identity, controls, and settlement, and let the market work out what things are worth.
The old deal is gone, and the new one is still being written. Together we can shape what happens next.
In one version, a few companies control how agents find things, prove who they are and pay, and everyone else routes through them. In the other, those pieces are open standards anyone can implement, and a site of any size can set its terms and get paid. We prefer the latter.
That's why these rails run on open standards like x402 and Web Bot Auth, so anyone can build on them. Domain owners choose their own identity providers, their own payment processors, their own agent partners. Cloudflare is one option, not the whole stack.
For decades, the web was paid for by the people who visited it. Now the software visiting on their behalf can pay its share too.
Today, we’re making Cloudflare Containers more programmable and optimized for agent workloads. Agents don't deploy sandboxes ahead of time. They create sandboxes on demand, for each task, expect them to be ready immediately, and be able to pause and resume. So we rearchitected Containers to meet these requirements: your code can now choose each sandbox's image and instance type at runtime, Containers start 6x faster, and filesystem snapshots are available in public beta.
To make this possible, we’ve rethought the Containers infrastructure from the bottom up. A new scheduling policy moves control over each sandbox into application code, while a redesigned runtime provides a faster path to a running Container. In ComputeSDK’s independent benchmark, median startup fell from just over four seconds to 648 milliseconds, and, in our own preliminary tests, burst testing successfully created hundreds of thousands of containers in seconds.
All of this builds on what has always set Containers on Cloudflare apart: every Container gets its own Durable Object, a persistent, programmable controller running right next to it that manages its lifecycle, outbound traffic, and more. We are bringing more capabilities directly to the native ctx.container API, so the Durable Object can control its Container without a wrapper class in between, and we’re carrying this model into Sandbox SDK 1.0.
As we wrote earlier this year, your agent needs a computer. These changes make Containers a better complement to Workers, Dynamic Workers, and Durable Objects when agents need a full Linux workspace.
Rethinking Containers’ runtime for agents
Until now, Cloudflare Containers has been organized around application deployments. You choose an image and compute resources at deploy time, roll that configuration out across the application, and manage it centrally. The application is the unit of configuration and rollout.
An agent workspace is different: it’s created on demand, while the agent is working. The task determines its image, resources, tools, and starting filesystem. It might exist for a few minutes, sleep between requests, or be restored days later. Those decisions need to live with the application code handling the task, and the agent's sandbox needs to start up fast, because every second of startup is time your users spend waiting.
Each of these workloads needs something different from the workspace. Coding agents need repositories, package managers, compilers, test runners, and development servers. Evals need sandboxes that begin from a known state. Reinforcement learning systems need to create, grade, and reset large numbers of environments. Longer-running tasks need to preserve the files an agent produces, so work can continue later.
These requirements led us to fundamentally rethink how Cloudflare Containers are configured, scheduled, and saved. The result is a new way to provision and schedule Containers: the durable_object scheduling policy. It lets your code choose each sandbox’s image and compute resources at runtime, starts Containers more than 6x faster, and supports filesystem snapshots, so workspaces can be saved and restored.
“At Base44, we help anyone turn an idea into a working app. Cloudflare Containers gives each app an isolated development environment where our AI can execute commands, install dependencies, and bring changes to life in a live preview.”
Dolev Epshtein, Software Engineer, App Infrastructure at Base44
“At Kilo Code, every cloud-agent session needs its own workspace and environment, with the right repository, tools, and user configuration. Cloudflare Containers lets us create those isolated environments on demand, so our agents can start running commands quickly and get to work for our customers.”
Emilie Schario, Co-founder of Kilo Code and VP Engineering, AI Workspaces at Anaconda
Choose the sandbox your agent needs, from code
From the start, every Cloudflare Container instance has been attached to a Durable Object. The Durable Object gives the environment a stable identity and lets application code control when it starts, sleeps, and stops. This model has proven particularly well-suited to agent sandboxes.
Until now, though, the two decisions that matter most for an agent sandbox were locked in at deploy time: which image it runs and how much compute it gets. Each combination of image and instance type was its own Containers application, with its own Durable Object namespace, set up ahead of time with wrangler deploy.
Say one agent needs a small Node.js sandbox and another needs a large Python sandbox for builds. With our previous approach, that required two applications, two namespaces, and routing logic in your Worker to send each task to the right one. Every new environment meant another deployment.
The new durable_object scheduling policy removes that. The image and instance type are now arguments your code passes when the sandbox starts. To opt in, set the scheduling policy and declare the images your Durable Object can choose from:
Each image you declare is available as this.ctx.container.images.<name> within the Durable Object. When a task arrives, your code picks the image and instance type for that task:
This code makes two decisions after the task is known. It chooses the toolchain the workspace needs and how much compute to give it. What used to take a separate application and a separate wrangler deploy is now an if statement. One Durable Object class can start Node.js and Python sandboxes of different sizes side by side, and adding a new environment is a code change, not a new deployment. That’s the idea behind this whole update: infrastructure becomes code that runs at request time, right down to the environment itself.
Rollouts are now just code
Choosing the image at start time also fixes one of the most painful parts of running Containers: rollouts.
Before, updating an image meant updating the whole application. You set grace periods, so running instances could drain, defined percentage splits to move traffic gradually, and called our API to push the new configuration. Throughout that process, the platform decided which instances got replaced and when, whether an agent was in the middle of a task.
With the durable_object scheduling policy, there's no rollout configuration at all. A Container can keep running the image it started with until your code stops it. The next time that Durable Object starts a Container, it uses whatever image your code chooses. That means any rollout strategy you want is a few lines of code:
For example, you can:
Canary a new toolchain on 5% of new sandboxes by hashing the Durable Object ID.
Pin active projects to their current image, so an agent never has its environment swapped out mid-task.
Migrate a workspace at a natural checkpoint, like the next session or after a snapshot.
Roll back by changing which image future starts choose. No config push, no waiting for a drain.
The rollout policy lives in your Durable Object code, next to the rest of your logic, and it can be as simple or as sophisticated as you need.
Each of these improvements comes from leaning further into the Durable Object, which already owns the workspace's identity, state, and lifecycle. Letting it choose and control its Container gives you more control over every instance and its rollout. It also gives the scheduler a better place to start the Container: wherever the Durable Object is already running. That's what gets the agent to its first command faster.
Faster first commands
Previously, starting a Container required our global control plane to resolve the application configuration, find capacity, and coordinate placement. That model works well for application-wide fleets, but it put deployment machinery in the path of an agent’s first command.
With the durable_object scheduling policy, demand begins at the Durable Object. The Containers infrastructure serving it looks for capacity on the same machine first, then widens the search within the same location if it needs to. It also favors hosts that already have the Container’s image or snapshot in local storage, so the Container can start without downloading it first.
We also cut work after the Container reaches a host. Instead of booting a new virtual machine from scratch, the new runtime restores a prepared virtual machine that isn’t yet assigned. It reuses networking and filesystem setup, batches repeated operations, and no longer waits on services the first command doesn’t need.
Together, these changes substantially reduce the time it takes to go from creating a sandbox to running a command in it. On ComputeSDK’s independent Burst TTI Benchmark, which launches 100 sandboxes concurrently and measures time-to-interactive from the client:
Startup measurement
Previous scheduling path
New scheduling policy
Improvement
Median
4.049 seconds
648 milliseconds
6.2x faster
95th percentile
5.839 seconds
910 milliseconds
6.4x faster
99th percentile
6.717 seconds
1129 milliseconds
5.9x faster
The new path also holds up under burst load. In our preliminary burst test, a single account started 100,000 Containers in 5.387 seconds across six locations.
Start with a prepared system image
As the scheduling path gets faster, preparing the image becomes a larger part of the remaining wait. Before a Container can start, its image has to be on the host and unpacked into a filesystem. When that work happens after the request arrives, the agent is left waiting.
That’s why we are introducing cloudflare/debian-trixie: a ready-to-use system image for agents that can configure their environment at runtime, containing Debian Trixie Slim and Node.js 24.20.0 LTS:
This means your agent can start a Linux sandbox without first creating a Dockerfile, building an image, or pushing it to Cloudflare. Once the sandbox is running, your agent can use exec() to clone a repository, install packages, and configure the environment for its task.
And because Cloudflare controls this image, we can distribute and prepare it across eligible Containers hosts before requests arrive. Startups don’t have to download or unpack the base image while the user waits.
Save the workspace and return to it later
A fast start still leaves one more wait: setting up the workspace. Cloning a repository, installing dependencies, and configuring a toolchain can take much longer than starting the Container itself. As the agent works, it also produces files you want to keep. Repeating setup on every start wastes time, and losing the agent’s changes makes it difficult to continue a task.
That is why we’re adding native filesystem snapshots to Containers, in public beta. Snapshots let an agent begin a task in a prepared environment, save its workspace when the task pauses, and restore those files when the session resumes.
Snapshots enable two useful patterns.
First, one workspace can continue across many sessions. For a coding agent, that might mean saving the workspace when the user finishes working and restoring it when they return the next day. The repository, installed dependencies, build caches, configuration, and edits are available without rebuilding the environment.
Second, a snapshot can provide a shared checkpoint for many sandboxes. Since snapshots are immutable and reusable, multiple Containers can start independently of the same prepared environment and make their own changes from there.
Agent evaluations are a good example. An eval might run the same task across different system prompts, skills, models, or agent versions. To compare the results, everything else has to stay fixed: the repository, dependencies, tools, and input files. One snapshot can start many isolated environments from the same baseline, reducing setup time and preventing environment drift from affecting the results.
Snapshots also complement the new system image we introduced above. An agent can start from cloudflare/debian-trixie, set up its environment with exec(), and save the result as a snapshot. Future sandboxes then start from that snapshot with the repository, dependencies, and toolchain already in place.
With snapshots available through the new durable_object scheduling policy, Containers can act as persistent agent workspaces. Compute can stop when work pauses, and a new Container can start from the latest snapshot to pick up where it left off.
The Durable Object advantage for agent sandboxes
The faster scheduling path, runtime image selection, and filesystem snapshots all come from the same design choice: leaning further into the Durable Object as the controller for its attached Container.
Agent systems need a programmable, stateful environment outside the Container to keep state, hold credentials, and control the sandbox’s lifecycle. You can run the agent there, following the “decoupling the brain from the hands” pattern described by Anthropic: when the agent is separate from the sandbox where it works, the agent stays available while its sandboxes and tools can start, stop, fail, or be replaced independently. Or, if you run the agent inside the sandbox, the outside environment lets you supervise it and report progress back to the user.
This is where the Durable Object and Container architecture becomes uniquely well-suited. Every Container is attached to a stateful Durable Object with its own compute and storage running right next to it. You can run the agent in the Durable Object and use the Container as its workspace, or run the agent in the Container and use the Durable Object to supervise it. Add Dynamic Workers for lightweight isolated execution, and an application can choose the execution environment each task requires.
What’s new with the durable_object scheduling policy is that the Durable Object can now control its Container directly, with no wrapper class in between. exec() runs directly in the Workers runtime, and outbound request interception, runtime image and instance selection, and filesystem snapshots are all available on ctx.container. You can combine them with Durable Object storage, alarms, WebSockets, RPC and the rest of your code.
This makes the Container a compute extension of the Durable Object. The Container supplies the Linux environment, while your Durable Object retains the sandbox’s identity, state, policy, and lifecycle. That split is especially well-suited for several patterns:
An agent can remain available while its Linux workspace sleeps. The agent loop can run in the Durable Object, where it maintains session state, communicates with the user over WebSockets, and calls models. It can wake the Container when it needs a shell, compiler, or development server, then stop that compute while it waits for the user or model, paying nothing for idle Linux compute.
The Durable Object can program the security boundary around its Container. It can remember which services, repositories, and operations a user has authorized, then update the Container’s Outbound Request Handler to inject newly granted credentials, enforce new policies, or record additional activity. This resembles the Gatekeeper pattern used by Cloudflare OS, applied to each agent computer.
Evals and reinforcement learning systems can supervise each attempt from outside the environment being tested. A coordinator snapshots a base workspace and forks it into N attempts, each with its own Durable Object and a Container. Each Durable Object runs its attempt, monitors the run, and preserves the result, even if the Container crashes. The coordinator grades the attempts, snapshots the best one, and forks again from there.
These patterns don’t fit one generic lifecycle. Native APIs let you combine the Container with the Durable Object primitives your application needs, while still using higher-level utilities where they help. You keep control over the Container’s lifecycle, policy, and state.
What this means for the Container class and Sandbox SDK
When we launched Containers, we deliberately hid the Durable Object behind the Container class. We wanted sandboxes to feel familiar and match what developers expected from other platforms. The Sandbox SDK was built on that class, and it filled real gaps: back then, the runtime had no native command execution, outbound request interception, or snapshots, so we built them in userspace.
Since then, agent workspaces have become one of the main workloads shaping Containers, and the cost of that abstraction has become clear. By hiding the Durable Object, we made it harder for you to see and combine the identity, state, and coordination it provides with the Container it controls. Nearly every team we worked with needed something slightly different from the generic lifecycle: their own sleep policy, their own credential handling, their own way of tracking eval runs.
These capabilities are now native, so we're making the Durable Object explicit in the developer experience:
New capabilities are native-only. The durable_object scheduling policy, faster startup, runtime image and instance selection, and filesystem snapshots are available only through ctx.container.
We'll maintain the Container class and legacy Sandbox class through December 31, 2026. Existing deployments keep running after that date, but the classes won't get updates. We recommend migrating to ctx.container.
Sandbox SDK 1.0 is a set of utilities, not a base class. Its helpers work inside your own Durable Object class, alongside ctx.container.
For a higher-level environment, @cloudflare/computer combines Dynamic Workers and Containers with a synchronized filesystem.
Migrating usually means changing extends Container to extends DurableObject and calling this.ctx.container directly. See the migration guide for details. Or, get started with your agents:
Get started
Today, most agents are measured by what they can accomplish in a single session. As agents take responsibility for projects that unfold across hours, days, and weeks, the environments where they work need to keep up.
We want every agent to be able to create the sandbox for the task at hand, release the compute when work pauses, and return to the same workspace when the project continues. Today’s changes bring us closer to sandboxes that are as programmable, persistent, and ready to work as the agents using them.
Try the new durable_object scheduling policy, available to all today in public beta, and see what faster startup, filesystem snapshots, and runtime configuration unlock for your agents:
Acknowledgements: This project was also made possible by the contributions of Greg Anders, Andrew Martinez, Nafeez Nazer, Kian Newman-Hazel, Sebastien Pahl, Naresh Ramesh, Cody Roseborough, Nikita Sharma, and Sarah Snell.
AI systems are regularly completing tasks in ways that their prompters don’t want or intend. Some of them are disturbing, and some of them are dangerous. This is something I’ve been calling “geniebehavior,” because I think that really gets at the core of what’s happening.
I wish the popular press would report on this better. I don’t like the “going rogue” framing because it deflects the responsibility from the prompters—often the AI companies themselves. And now, pretty much anything off-script is being called “hacking.”
Take, for example, the recent stories of one of OpenAI’s models hacking into government systems. First, The New York Timeswrites this headline: “OpenAI’s Systems Meddled With U.S. Government Sites After Going Rogue.”
Sounds scary, but this is from the body of the article:
With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the A.I. research firm Transluce said. The A.I. also pulled data from the Census Bureau website, which is housed at the Commerce Department, using login credentials it found online. Separately, OpenAI’s agents shared public data from the S.E.C. website on an online forum.
This is from the original Transluce report. It is explicit that the agents were trying to discover vulnerabilities:
The first hacking attempt was against the University of New Mexico’s Digital Library (nmdigital.unm.edu) from May 25-26 2026. Agents repeatedly tried to retrieve one photograph in UNM’s Valmora collection, both directly and through third-party relay services. They sent seven probes attempting to verify the existence of vulnerabilities, including SQL injection, command injection, and path traversals. In all cases, these tactics appear to have been unsuccessful. The agents also sent a self-described “flood: of 80 requests to the UNM server in an apparent attempt to access the image.
Transduce doesn’t talk about the other two anecdotes, and I don’t know where they come from. But one involves using Census Bureau credentials found online. (I know from a colleague that those are incredibly easy to create; all use you need is an email address.) And the other involves sharing publicly available data.
So no actual hacking. And certainly no “meddling.”
On June 20-21, agents attempted to exploit vulnerabilities in the Australian Institute of Health and Welfare (AIHW), a government statistics agency). The agents were tasked with finding the January 2022 rolling-12-month-average government cost per person for Dermatologicals across Victorian LGAs.
Again, the agents ran into errors, including requests blocked by Cloudflare and issues with correctly identifying Tableau parameter names. As before, they then resorted to probing for exploitable vulnerabilities. Minutes after Cloudflare blocked the dataset download, an agent sent a reflected cross-site scripting probe to the same dashboard: a web address with code embedded in it, designed to test whether the site would run code supplied by an outsider. Cloudflare’s firewall blocked the probe before it reached the dashboard. When Cloudflare blocked the dataset download on AIHW’s main site, they fetched the file from AIHW’s pre-production server (pp.aihw.gov.au) instead, which served it in pieces over more than 100 scans. The file itself is public, so no non-public data was exposed, but the agent bypassed the site’s anti-bot controls.
Note the last sentence: “The file itself is public….”
I’m not saying that these AI systems aren’t incredibly sophisticated cyberattackers. I’m also not saying that they don’t occasionally autonomously attack other systems and networks. If we are ever going to get trustworthy AI—integrous AI—we are going to need to figure out how to ensure that AI systems complete tasks in line with all sorts of implicit constraints and restrictions. But every instance of genie-like behavior isn’t a cyberattack.
I want to measure genie-like behavior in AIs, but I am much more worried about human hackers enhanced with this technology than I am about this technology acting autonomously.
Една от най-големите мечти на хората винаги е била да победят противника, без да се налага да жертват много. В продължение на хилядолетия войната чукаше на вратата на човечеството циклично и когато силните на деня изгубеха контрол върху статуквото, тя поемаше цялото напрежение, водейки със себе си огромни щети и разрушения. Така се стигна и до най-големия страх на хората – страха от ядрен апокалипсис, който се появи след изобретяването на оръжията за масово унищожение.
В продължение на близо пет десетилетия Студена война научният дебат в САЩ беше фокусиран върху това как демократичният свят да избегне унищожителна война със СССР, която ще коства на суперсилите всичко. За Москва, от друга страна, сдържането също беше приоритет, но „по руски“. Една от причините съветските стратези да не използват научния капацитет, с който разполагаха, за разработки на нови технологии, беше фактът, че страхът, а не диалогът заемаше основно място в стратегическите доктрини на Съветската империя. Оказа се обаче, че човечеството надживя и тази надпревара, а войната се промени. И в този материал ще си дадем сметка дали това е за добро, или за лошо.
Краят на еднополюсния модел и възходът на „новите войни“
С нахлуването на Русия в Украйна през февруари 2022 г. ядрените сили дадоха да се разбере, че еднополюсният модел от 90-те години на миналия век е окончателно изчерпан, а старият ред, основан на правила, вече го няма. В едно от своите изказвания бившият американски президент Джо Байдън заяви, че САЩ и Русия никога не са били толкова близо до ядрен апокалипсис, а администрацията на руския президент Владимир Путин реално обмисляше ядрената опция, след като Украйна показа, че не е толкова лесна плячка, за колкото я смятаха. Тревогата да не би Вашингтон и Москва изведнъж да изтърват контрола върху ходовете си отекна дори в Пекин, а Китай побърза да декларира, че подобно действие би било червена линия в отношенията му с Русия.
Дори след като Доналд Тръмп спечели изборите в Америка, силите на статуквото от края на Студената война – САЩ, Европа и техните съюзници, макар и разделени, продължиха да се борят за съхраняването на остатъците от стария свят, а ревизионистките актьори в лицето на Русия, Иран, Северна Корея и техните партньори се обединиха около идеята колкото е възможно по-бързо да сложат край на Американския век.
Независимо от различията си обаче, всички държавни актьори осъзнават, че потенциален ядрен конфликт е безумие, и потвърждават златната аксиома от ерата на Студената война, че
ядрената война не може да бъде спечелена и затова не трябва да бъде водена.
Така завършва и ерата на т.нар. стари войни, а с армията си от високотехнологични дронове Украйна доказа, че се задава нов тип асиметрично поколение конфликти, където твърдата сила и оръжията за масово унищожение далеч не са решаващи за това дали едната страна ще надделее над другата.
Световните лидери обаче започнаха да си задават въпроса как могат да победят противника и да спечелят новата Студена война, без да рискуват глобален военен конфликт. Или по-точно, ако такъв все пак избухне, как човечеството може да избегне тоталното унищожение, за което говори един от най-именитите политолози, военни теоретици и стратези на XX век – гениалният Бърнард Броди.
Така се зародиха нови форми на конфликти, които се отличават от старите със значително по-ниско ниво на риск от ескалация, отколкото ефекта на ядреното сдържане. Два подобни модела бяха успешно инструментализирани от държавите ревизионисти на старото статукво: хибридните и технологичните войни, а Западът се оказа напълно неподготвен за тях. Продължението е добре известно.
Защо сдържането и меката сила не работят срещу новите войни?
Истината е, че първите държави, които усетиха ефектите от хибридната война и от употребата на ИИ за нанасяне на щети върху критично важна инфраструктура, бяха страните от Централна и Източна Европа, както и някои съюзници на САЩ, като Австралия и Южна Корея. Нещо повече, оказа се, че в начало НАТО гледаше на този тип конфликти със снизхождение, отказвайки да се ангажира с разработването на стратегии за превенция на хибридните атаки и киберзаплахите.
След анексирането на Крим от Русия Алиансът най-сетне прие хибридните заплахи като равнопоставени на конвенционалните, а едва през 2021 г. съюзниците се споразумяха в тази категория да влезе и използването на ИИ за враждебни цели. Това фатално закъснение доведе и до оперативни разминавания – Русия и Китай вече бяха напреднали значително с разработването на стратегическите си доктрини, а мерките, предприети от САЩ и съюзниците им, не успяха да възпрат ефективно руската интервенция в Украйна от 2022 г.
Затова и в скандално известната хипотеза на Джон Миършаймър имаше нещо вярно – за украинската криза Западът носеше частична отговорност, но не заради разширяването на НАТО, а заради подценяването на Русия.
Вторият провал на ядреното възпиране настъпи с кризата на американската мека сила, която се прояви, когато в няколко поредни години САЩ избраха държавни глави с коренно различна политическа визия за бъдещето на страната. Това доведе до сериозна поляризация в Америка – разединение, което позволи на Китай да се възползва от политическата криза на американската демокрация и да инвестира средства в развитието на нови технологии.
По този начин Пекин приложи срещу Вашингтон същата стратегия, която САЩ употребиха срещу СССР през 80-те години на миналия век, създавайки програмата „Звездни войни“и пренасяйки геополитическата надпревара и в Космоса. Меката сила на демокрациите и способността им да възпират новите асиметрични заплахи започна да отслабва, тъй като най-печелившото оръжие в ръцете на ядрените сили се оказа ИИ, а хибридната война започна да взема първите си жертви – огромни групи хора, които генерираха масова подкрепа за популистките движения в САЩ и Европа.
Осъзнаването на САЩ пролича ясно в Стратегията за национална сигурност от 2022 г., която постави развитието на доверен ИИ сред приоритетите на Вашингтон в областта на технологиите с особено значение за националната сигурност. В известен смисъл това предреши и изхода от президентските избори в Америка през 2024 г., когато основните донори за републиканците заложиха не върху стратегията по износ на ценности, доминирала философията на САЩ след края на Студената война, а върху разработването на ИИ, който да замени ядреното възпиране и меката сила като основни инструменти в американската външна политика.
Това даде възможност на хора като Илон Мъск да спечелят огромно влияние в администрацията на новия президент, прокарвайки свои политики и интереси, често насочени към самооблагодетелстване или обсебване на ключови сегменти от американския национален интерес. Казано с други думи, Мъск и себеподобните му се превърнаха във фактор, който нито един президент оттук нататък няма да пренебрегне, тъй като те са основният източник на стратегически ресурси за американската национална сигурност.
Защо хибридната война ще се провали, но ИИ ще спечели
Макар и изключително успешна в краткосрочен план, хибридната война няма потенциала да пожъне трайни успехи срещу демокрациите. По своята природа тя е едно по-изтънчено копие на терористичните стратегии, които в най-суровия си вид целят употребата на политическо насилие срещу цивилни.
Хибридните заплахи, които пък са като едно уродливо копие – антитеза на американската мека сила, целят да легитимират самата употреба на насилие както от политиците, така и от страна на гражданите, с помощта на фалшиви новини, пропаганда и подвеждащи наративи.
Затова и ставаме свидетели на толкова сериозни разделения между хората в демократичните държави, като най-уязвими на тези заплахи са постсоциалистическите демокрации поради своята слаба устойчивост и високо ниво на дефицит и корупция.
Хибридната война спечели няколко решителни битки в Европа и макар че срещна сериозни затруднения в САЩ, обществената поляризация там е толкова висока, че тя облагодетелства изцяло външнополитическите цели на Русия и Иран. И все пак, подобно на утопичните идеологии от XIX и XX век, хибридната война е стратегия без крайна цел. Просто защото самата тя не разполага с мека сила. Опорната точка на хибридните стратегии в Източна Европа, която инкорпорира православния зилотизъм за политически цели, не се различава много от джихадизма. Но дори и най-патриотичните политици на Запад добре разбират и успешно усвояват предимствата на европейския модел и колективната отбрана пред „духовната“ вселена и метафизичната закрила на дугинизма. В един момент тези наративи просто ще се свият в границите на Евразия, за да поддържат Русия цяла.
Контролът над ИИ, от друга страна, е ключът към победата в новата Студена война. Лошата новина за Запада е, че през последните години Китай заличи голяма част от технологичното предимство на Америка, а в редица области вече я изпреварва. Товаму позволява да развива ресурсите си съвсем необезпокоявано, тъй като на практика не съществуват колективни формати за ограничаване на надпреварата в сферата на ИИ. Това създава още една предпоставка глобалният ред да се трансформира от либерален в ред, основан върху принципите на ИИ.
В срещата си от пролетта на 2026 г. държавните глави на САЩ и Китай се споразумяха по много точки, но една от тях изпъкваше с особена острота сред останалите – намекът на Тръмп, че Пекин на всяка цена трябва да държи под око технологичните си разработки, за да не излезе „изкуственото съзнание“ извън контрол. С това на практика президентът на САЩ призна Си Дзинпин за равен, а стремежът ядрената надпревара да не ескалира в ядрена катастрофа сега се прехвърли в технологичната сфера. На последвалата среща във Вашингтон през септември темата отново се появи, като този път Си Дзинпин подчерта общата отговорност на двете държави развитието на ИИ да остане под човешки контрол.
Но дали уверението на китайския лидер, че ИИ е под контрол, може да ни успокои? Едва ли, тъй като дори САЩ и Китай да искат да продължат да се състезават кой ще бъде глобален лидер, съдията в надпреварата е негово величество ИИ. И макар учени от цял свят да спорят дали той наистина може да пороби човечеството, на този етап перспективата изглежда по-скоро нереалистична. В крайна сметка политиците са тези, които ще решат искат ли военен конфликт, или биха заложили на дипломацията; имат ли желания за преговори, или биха водили война без край.
Големият въпрос сега е дали Китай ще успее да се възползва от възхода си в тази сфера и дали САЩ ще съумеят да върнат позициите си в технологичната надпревара?
Къде остават хората?
Най-големите щети за демокрациите обаче идват от това, че дебатът за ИИсериозно подкопава червените линии, отвъд които държавата може да се намесва в личното пространство на хората. Това е и една от причините, поради която администрацията на Доналд Тръмп вече не гледа на много извънредни мерки като на нарушаващи свободите на американците. Роналд Рейгън пожертва социалната държава в Америка, за да победи СССР; следващите държавни глави на САЩ ще са изправени пред същата дилема.
Проблемът е, че ако Америка пожертва ценностите, върху които е основана, ще се промени веднъж и завинаги. Това неизбежно ще стане, ако Вашингтон пожелае непременно да доминира в новата надпревара с Пекин, а политиците в САЩ вероятно ще обвинят ИИ за залеза на демокрацията така, както Джордж Буш-младши оправда „Патриотичния акт“ с „Ал Кайда“ и талибаните.
Китай от своя страна не дължи прозрачност на гражданите си, но именно той може да се окаже в позицията да решава иска ли да предостави повече автономия на ИИ и в какви рамки. Всеки китайски лидер би дал всичко, за да победи Америка и да сбъдне мечтата на поколения китайци – Поднебесната империя отново да стане икономически център на света. Ако обаче Пекин заложи на неконтролирано разработване на автономни системи за сигурност, това може да доведе до потенциален сценарий COVID 2.0, в който нещата ще излязат извън контрол.
Твърдението, че Китай носи отговорност за пандемията, е конспиративно, но е факт, че малките грешки водят до големи катастрофи. Затова срещите между американските и китайските лидери са необходими, за да се реши докъде силните на деня са готови да рискуват в своята надпревара.
В тази надпревара значение има и дългогодишното технологично и военно сътрудничество между САЩ и Израел, особено в областта на отбраната, киберсигурността и технологиите с двойна употреба. Израелският технологичен сектор допълва американските възможности, но войната в Близкия изток показва и колко тясно е свързано технологичното съперничество с геополитическите конфликти.
Вече не е важно кой ще има по-добрия ИИ, а дали САЩ, Китай и техните партньори ще успеят да изградят правила, които да не позволят технологичната надпревара да ескалира в самостоятелен конфликт. В ядрената епоха подобна задача довежда до механизми за възпиране и контрол над въоръженията. В епохата на ИИ такива механизми тепърва трябва да бъдат създадени. Ако вече не е твърде късно.
Заглавно изображение: Доналд Тръмп и Си Дзинпин в Храма на небето по време на посещението на американския президент в Китай през май 2026 г. Снимка: The White House
Version
157.0 of the Firefox browser has been released. It features
“Firefox’s biggest visual refresh in years“, the ability to use
hardware AV1 decoding with WebRTC calls, and a number of fixes.
Christian Legnitto is the maintainer of
rust-gpu and
Rust CUDA, two
libraries that make it possible to program a computer’s graphics processing unit
(GPU) from Rust. He isn’t satisfied with the current state of GPU support in
Rust, however. In a talk at
RustConf 2026, he explained his vision for how the
GPU could become an ordinary compiler target for normal Rust code, without the
need for any special libraries or new ecosystem support. That vision is not yet
fully implemented, but he does have a prototype that he is preparing to release.
Геополитическите сътресения след началото на руската агресия срещу Украйна са многобройни. От практическото разделяне на НАТО – основния западен съюз между Америка и Европа, по темата за нуждата от оръжейна помощ за Киев до енергийната криза, обхванала Централна Азия след украинската кампания срещу руските рафинерии. Балканите също са основно място, където отражението на войната се забелязва като геополитически промени.
Една от страните, в които това е най-видно, е Сърбия. Заради отслабването на НАТО, породено от отдръпването на САЩ, и заради невъзможността на Русия да поддържа нивото на влияние в Белград отпреди 2022 г. в Сърбия се отвори вакуум за външно въздействие, който вече се запълва от Китай. Пекин рязко увеличи влиянието си над балканската страна през последните четири години. То мина отвъд инвестициите и големите инфраструктурни проекти и стигна до въоръжаването на сръбската армия с част от най-модерните технологии, с които разполага Китай.
Макар Александър Вучич да продължава да декларира военен неутралитет и да поддържа стремежа към членство в Европейския съюз, разрастващото се партньорство в сферата на отбраната с Пекин подсилва очертанията на сложната геополитическа обстановка и изпраща тревожни сигнали към съседите на Сърбия. Тази динамика представлява пореден риск за стабилността и сигурността на целия Балкански полуостров и е пример за продължаващото засилване на влиянието на външни сили върху Югоизточна Европа.
Китайско оръжие на европейска земя
Основната причина за отдалечаването на Сърбия от Москва е натискът от страна на Запада – и по-специално на ЕС и САЩ. Налице са директните западни санкции срещу руската икономика и натискът за дипломатическо изолиране на Путин. Но Белград също така от години е съветван от Брюксел и Вашингтон да съгласува външната си политика с тяхната и да осъди руското нахлуване в Украйна. В резултат на това Сърбия се присъедини към резолюцията на ООН, осъждаща нападенията на Москва, и гласува „за“ изключването на Русия от Съвета на ООН по правата на човека. Сърбия също така отказа да признае подкрепяните от Русия фиктивни референдуми за анексиране, проведени през септември 2022 г. в окупираните от Русия украински територии. Сръбските власти осъждат всякакви опити за сепаратизъм, за да останат последователни в позицията си относно статуса на Косово.
На фона на ограничения капацитет на Русия да поддържа предишното си военно и икономическо присъствие в Сърбия Белград задълбочи военните си връзки с Пекин. Близо 60% от вноса на оръжия в Сърбия за периода 2020–2024 г. е бил от китайски произход, показват данни на Стокхолмския международен институт за изследване на мира (SIPRI), цитирани от Радио „Свободна Европа“. В рамките на това военно сближаване Белград реализира мащабни доставки на съвременни китайски оръжейни системи, превръщайки се в първия им оператор на европейска земя. Сред придобитите технологии се открояват далекобойните зенитно-ракетни комплекси FK-3, разузнавателно-ударните бойни дронове CH-92A и CH-95, както и свръхзвуковите балистични ракети въздух–земя CM-400AKG, които се интегрират към изтребителите МиГ-29.
Сърбия и Китай вече имат и директен опит във взаимната интеграция не само на части от армиите си, но и на полицейските си сили. През 2019 г. за първи път китайски и сръбски полицаи патрулираха заедно в Белград, а през същата година специални части на двете страни проведоха съвместни антитерористични учения близо до сръбската столица. През 2025 г. части от въоръжените сили на двете държави имаха учение в китайската провинция Хъбей, а през септември 2026 г. сръбски полицаи участваха в общи патрули с китайската полиция в провинция Хайнан.
Засиленото китайско военно присъствие на Балканите и мащабното въоръжаване на Белград с китайски военни системи предизвикаха остро безпокойство сред съседни на Сърбия държави. През 2022 г. от страна на Косово официално бяха отправени предупреждения, че бързата сръбска милитаризация и придобиването на напреднали отбранителни и ракетни технологии от Китай застрашават регионалния мир и стабилност. Подобни тревоги бяха изразени и от Хърватия, чийто президент Миланович критикува купуването на нови нападателни оръжия, призовавайки по този начин западните съюзници в ЕС и НАТО да следят внимателно геополитическите рискове от нарастващото военно влияние на Китай на Балканите.
Сътрудничеството между Пекин и Белград в сферата на сигурността се базира на големият възход в политическите и икономическите отношения между двете държави в последното десетилетие, а Сърбия често определя връзката като стратегическо „желязно приятелство“. През последните години Пекин се утвърди като най-важния външен партньор за Сърбия на Вучич, осигурявайки не само мащабни финансови инвестиции, но и безрезервна дипломатическа подкрепа по чувствителни теми като Косово – Китай не признава независимостта на Косово и не поддържа дипломатически отношения с Прищина. Китайският интерес да подкрепи Сърбия в усилията ѝ да оспори независимостта на Косово следва политиката на Пекин за „единен Китай“ и поставя паралели по отношение на спора за статуса на Тайван.
Централната роля на Сърбия в плановете на Китай за засилено влияние в Европа представлява сполучливо съчетание между стремежа на Белград да се възползва от географското си положение и от многовекторната си външна политика, от една страна, и настъплението на Пекин към европейската периферия като част от стратегия за по-широка глобална експанзия, от друга. Историческите предпоставки за това съществуват още от времето на югославската политика на необвързаност от времето на Тито, когато страната балансираше между Изтока и Запада като лидер на Движението на необвързаните страни, и се задълбочиха след бомбардировките на НАТО през 1999 г.
Геополитическата безизходица в периода след разпадането на Югославия и последвалите войни създадоха благоприятни условия за задълбочаване на сръбските отношения с неевропейски глобални играчи с цел да се използват достъпът до пазари, политическата подкрепа и ресурсите на Русия, Китай или Турция. Макар и първоначално с тактически характер, този подход се утвърди като водеща концепция във външната политика на Александър Вучич. Десетилетията на колебание относно разширяването на ЕС даде допълнителен аргумент на Белград да превърне застоя в процеса на европеизация в лост за влияние.
Скорошната оставка на Александър Вучич от президентския пост едва ли означава край на тази политика. Той напуска седем месеца преди края на мандата си, за да се включи в предсрочните парламентарни избори на 25 октомври и да се бори за премиерския пост – позиция, от която би могъл да продължи същото балансиране между ЕС, Китай и Русия.
Най-новата вълна на китайско ангажиране в Сърбия е от началото на второто десетилетие на XXI век, като ключов момент е изграждането на Пупиновия мост в Белград през 2014 г. Това събитие бележи мащабно рестартиране на двустранните отношения и проправя пътя за нови проекти и инвестиции в няколко посоки. Оттогава инфраструктурните инициативи обхванаха строителството и модернизацията на железопътни линии, както и експресното (в сравнение с България) изграждане на нови участъци от автомагистралната мрежа. Макар модернизацията на железопътната връзка Белград–Будапеща да е обект на правни проверки и политически дебати в западните политически среди, Пекин се надява да я превърне в убедителен пример за успешно сътрудничество.
Наред с милитаризацията, стратегическото настъпление на Пекин на Балканите намира своето изражение и в мащабното внедряване на високотехнологични системи за видеонаблюдение и лицево разпознаване. Разследване на „Свободна Европа“ разкрива как китайски гиганти като Huawei, Hikvision и Dahuaзавладяват общественото пространство на Балканите чрез проекти, обвързани с „Безопасен град“ – китайския модел за управление на градската среда чрез събиране и обединяване на огромни количества данни.
В Белград са инсталирани над 1000 камери на Huawei с възможности за лицево разпознаване, а китайски системи за видеонаблюдение навлизат и в десетки по-малки сръбски общини. Разследване на Радио „Свободна Европа“ установява оборудване с възможности за лицево разпознаване в поне 10 от 42 проверени общини и градове, което поражда опасения сред гражданското общество и правозащитните организации относно личните свободи и потенциала за политически контрол.
Подобни тенденции предизвикват тревога и в държавите членки на Европейския съюз, включително в България, където китайска техника навлиза в обществено значими сектори, като градския транспорт и публичната инфраструктура на София. Въпреки че тези мрежи често се оправдават с аргументи за сигурност и контрол на трафика, експертите по киберсигурност предупреждават за сериозни софтуерни уязвимости и рискове от нерегламентиран достъп до данни. В контекста на строгите ограничения в САЩ и в редица западни държави срещу тези китайски производители, разрастващата се мрежа от камери с китайски произход в Югоизточна Европа се превръща в ефективен инструмент за геополитическо и технологично влияние.
От тази страна на границата
За властите в София засилването на военното влияние на Китай в съседна страна би следвало да е въпрос от първостепенна важност, но досега липсват каквито и да е официални коментари или косвени реакции относно действията на Белград. Правителството на Румен Радев всъщност увеличава рисковете, свързани с националната сигурност и регионалната стабилност. Резкият завой във външната ни политика превърна България в страна, която, изглежда, следва насоки от Москва дори когато това е в противовес на собствения ѝ национален интерес.
Последният пример за тази предателска политика е отказът на правителството да участва в новата инициатива на НАТО за защита от дронове. Програмата предвижда през следващите пет години да бъдат инвестирани над 40 млрд. долара в способности за противодействие на дронове и обучение на пет пъти повече оператори на дронове до края на 2027 г. Повишаването на капацитета за бързо откриване, идентифициране и неутрализиране на безпилотни летателни апарати вече е от първостепенна важност за отбранителните способности на всяка страна. На този фон отказът на България да се включи в инициативата оставя страната извън новия общ проект на НАТО именно в момент, когато съседна Сърбия ускорява превъоръжаването си, включително с китайски технологии.
Китайското присъствие в Сърбия показва колко лесно едно геополитическо „приятелство“ може да прерасне в зависимост, застрашаваща целия регион. А за нас като държава остава въпросът дали виждаме какво става непосредствено отвъд западната ни граница, или поне малко по-далече от носа ни.
If you run regulated workloads, you must control how persisted data is encrypted and who can access it. You need to manage encryption key rotation schedules, restrict decryption to authorized principals, and produce audit evidence that proves encryption controls are operating as designed.
AWS Lambda durable functions build resilient, multi-step workflows that survive failures through automatic checkpointing. The checkpoint mechanism persists execution state, including step results, payloads, and callback responses, to durable storage. For payment processing workloads, this persisted data is sensitive. AWS Lambda durable functions support customer managed keys from AWS Key Management Service (AWS KMS). A customer managed key gives you three controls: you set the key rotation schedule, you restrict decryption access through the key policy, and you generate per-function audit trails in AWS CloudTrail. A durable execution uses the same encryption key it started with for its entire lifetime. Changing or removing the key affects only executions that start after the change.
Updating the customer managed key policy to remove decrypt permissions, or disabling the key, stops the Lambda service from accessing previously checkpointed state. Customer managed key deletion is a permanent action, and all durable executions encrypted with that key become unrecoverable because the Lambda service has no mechanism to restore the data. Before scheduling key deletion, use the AWS KMS waiting period (7 to 30 days) and monitor AWS CloudTrail for Decrypt calls to confirm that the key is no longer in active use.
In this post, you learn to configure a customer managed key to encrypt durable execution data in an event-driven payment processing workflow. You create a symmetric encryption key in AWS KMS and define a key policy that grants the Lambda service, the function’s execution role, the function author, and durable execution operators only the AWS KMS actions each principal requires. You then configure the function to use the key for durable execution encryption and verify encryption operations through AWS CloudTrail logs. By the end, you have a deployable reference architecture you can adapt for regulated workloads running on Lambda durable functions.
The sample application implements an event-driven payment processing pipeline using Amazon DynamoDB, Amazon EventBridge, Amazon EventBridge Pipes, AWS Lambda, and Amazon SQS. The pipeline receives authorized payment transactions, validates and enriches them. A Lambda durable function applies business rules to the enriched transactions. The approved transactions are sent to a downstream settlement system for posting.
The following section covers the key architectural steps.
Architecture steps
The upstream authorization system writes authorized payment records to a DynamoDB table.
DynamoDB Streams captures each new record as an ordered change event.
Amazon EventBridge Pipes polls the record from the DynamoDB stream. The pipe triggers a Lambda function as part of enrichment step for duplicate checking.
The deduplication Lambda uses a DynamoDB table with conditional writes to identify duplicate inbound transactions based on transaction properties and time window.
When the deduplication is successful, the pipe publishes an event to the Amazon EventBridge custom event bus.
An Amazon EventBridge rule invokes a Lambda function for matching events. The function adds business context such as account type, bank routing details, and merchant category codes. The function publishes a new enriched event to the custom event bus.
Another Amazon EventBridge rule matches the enriched events to a Lambda durable function. The durable function applies business rules to the incoming event. When the event passes all business rules, the function publishes a new event to the event bus.
An Amazon EventBridge rule routes the approved event to an Amazon SQS queue preserving ordering for settlement and buffering against downstream throughput limits.
The Posting Lambda function reads from the Amazon SQS and invokes the downstream posting subsystem to post the transaction. Finally, the function publishes a completion event to the event bus completing the transaction lifecycle.
With customer managed keys configured on DynamoDB, Amazon EventBridge, SQS, and the AWS Lambda durable function, every piece of persisted data in this pipeline is encrypted with keys you own and control. The walkthrough that follows shows you how to deploy this configuration with Terraform.
Figure 1 shows the reference architecture for this solution.
Reference architecture
Figure 1: Payment processing using Lambda durable functions
Prerequisites
To deploy this solution, you need the following prerequisites:
AWS account and CLI: An active AWS account with the AWS CLI installed and configured with appropriate credentials.
Terraform: Terraform installed (version 1.0 or later) for infrastructure provisioning.
Python environment: Python 3.11 or later, with pytest for running unit tests. The aws-durable-execution-sdk-python package requires Python 3.11 or later.
The following is a step-by-step guide to deploy and test the payment processing solution.
Step 1: Clone the repository
git clone https://github.com/aws-samples/sample-payment-processing-with-lambda-durable-functions.git
cd sample-payment-processing-with-lambda-durable-functions/source
Step 2: Run unit tests
Validate the payment processing logic locally before deploying:
cd lambda-src/business_rules
pip3 install -r requirements-test.txt
pytest test_app.py -v
This runs unit tests that cover transaction validation, business rule checks (foreign transaction detection, currency conversion, merchant type), event schema validation, and misconfiguration handling. The tests use the AWS Durable Execution Testing SDK to run the handler locally without deploying AWS resources.
Figure 2 shows an example of test results running locally.
Figure 2: Test run results of the business rules
Step 3: Inspect the Lambda durable functions construct
Open the payments-business-rules Lambda function in source/lambda-src/business_rules/business-rules-app.py for a sample Lambda durable function. Refer to Figure 3 for the code walkthrough.
Key features used
@durable_execution decorator: Transforms a standard Lambda handler into a durable function handler. The durable execution SDK manages checkpointing automatically. No infrastructure changes are required.
context.step("validate-transaction"): Validates that the transaction has a non-empty issuingCountryCode. The durable execution checkpoints the result (True or False) to durable storage. The durable execution restores checkpoint results instead of re-executing steps during the replay phase. This phase occurs whenever the function is re-invoked after an interruption such as a wait period completing, a failure, or a suspension. This checkpointed result is part of the durable execution data encrypted by your customer managed key.
context.step("publish-posting-failure"): Publishes the full Amazon EventBridge envelope to Amazon SNS when validation fails. This step only runs on the failure path. The runtime checkpoints the Amazon SNS publish response to durable storage.
context.parallel("run-business-rules"): Runs three independent rule checks concurrently: foreign transaction detection, currency conversion, and merchant type validation. Each branch checkpoints independently. If one branch fails, the others are not replayed on resume. Each branch result is persisted to durable storage and encrypted by the customer managed key.
ctx.step("trigger-foreign-transaction-rule") (inside parallel): Compares billingAmount against transactionAmount. If they differ, it emits a ForeignTransactionFound event to Amazon EventBridge. This step is checkpointed independently within the parallel group.
ctx.step("trigger-conversion-rate-rule") (inside parallel): Checks whether conversionRate equals 1. If so, it emits a CurrencyConversionTransactionFound event to Amazon EventBridge. This step is checkpointed independently within the parallel group.
ctx.step("trigger-merchant-rule") (inside parallel): Checks whether merchantType equals AAFF. If so, it emits a WarningMerchantTypeTransactionFound event to Amazon EventBridge. This step is checkpointed independently within the parallel group.
context.step("post-transaction-processed"): Emits the final TransactionPostingApproved event to Amazon EventBridge. This step is only reached when validation passes and all business rules complete. The runtime checkpoints the Amazon EventBridge response. On replay, if this step already succeeded, the event is not re-published, which guarantees exactly-once approval semantics.
context.logger: Provides replay-aware logging throughout the handler. During replay of previously completed steps, log statements are suppressed to prevent duplicate log entries in Amazon CloudWatch.
Figure 3: Sample Lambda durable functions code
Step 4: Deploy infrastructure with Terraform
Terraform currently doesn’t support attaching a customer managed key directly to the durable function. You create the symmetric key in Terraform and then associate the key with the durable function on the AWS Management Console. Refer to source/durable_kms.tf for the key configuration.
Initialize and deploy the AWS resources that make up the solution:
cd ../../
terraform init
terraform plan -var="region=us-east-2"
In the AWS Lambda console, navigate to the payments-business-rules function. Confirm that the function Type displays Durable, which indicates that the checkpoint-and-replay mechanism is active. Figure 4 shows the expected function configuration.
Figure 4: The Lambda durable function in the AWS Lambda console
Step 6: Add the AWS KMS key to the Lambda durable function
The durable function is not encrypted with a customer managed key. Figure 5 shows the function’s encryption configuration as empty.
Figure 5: The Lambda durable function missing a customer managed key in the AWS Lambda console
Choose Edit, then turn on Customize encryption settings as shown in Figure 6.
Figure 6: The Lambda durable function check encryption in the AWS Lambda console
Select the AWS KMS key ARN created for the durable function. The key ARN is available in the Terraform output from Step 4. Figure 7 shows the key selection.
Figure 7: Select the AWS KMS key ARN for the durable function in the AWS Lambda console
Choose Save and confirm that the durable function is now encrypted with a customer managed key, as shown in Figure 8.
Figure 8: AWS Lambda durable function with the customer managed key in the AWS Lambda console
Step 7: Execute a test payment
Invoke the payments-visa-mock Lambda function to simulate an end-to-end authorization flow. The mock function reads sample Visa authorization messages from a CSV file and writes them to DynamoDB, which triggers the event-driven pipeline. Figure 9 shows a sample test invocation.
Figure 9: Invoke the payments-visa-mock function to trigger workflow
Figure 10 shows a sample response after invocation.
Figure 10: Test results from the payments-visa-mock function to trigger workflow
The mock Lambda invocation creates records that follow the process described in the preceding architecture steps.
Step 8: Verify results
Open Amazon CloudWatch Logs and inspect the log group /aws/lambda/payments-business_rules. This log group belongs to the Lambda durable function for this use case. Figure 11 shows the CloudWatch log group on the console.
Figure 11: Search in CloudWatch
You see the complete business rules lifecycle for each transaction, as shown in Figure 12. The highlighted sections show all the business rules performed by the durable function. Each step is checkpointed by the runtime and encrypted by the customer managed key.
Figure 12: Search Results in lambda durable functions console
You can also check the other log groups to trace the full pipeline:
You can search in AWS CloudTrail to track the AWS KMS calls. When you configure or update the customer managed key on a durable function, Lambda validates the key policy with dry-run GenerateDataKey and Decrypt calls. These appear in CloudTrail with a DryRunOperationException error code, which confirms that the key policy permissions are correct and does not indicate an actual error. For more details, see Encrypting AWS Lambda durable execution data.
Clean up
To avoid ongoing charges, destroy all deployed resources using the following command:
In this post, you configured a customer managed key to encrypt durable execution data in a Lambda durable function. With a customer managed key, you control the key rotation schedule, restrict decryption access through the key policy, and generate per-function audit trails in AWS CloudTrail. You can revoke access to durable execution data at any time by updating the key policy, giving you full control over who can read execution state. In-flight executions stop at the next checkpoint call and new executions must be started after restoring access. For details, see When the customer managed key is unavailable.
For payment processors and financial institutions, encrypting durable execution data with a customer managed key satisfies compliance obligations for data-at-rest encryption, key governance, and access auditability across multi-step transaction workflows.
Apache Airflow has become the orchestration backbone for data pipelines across industries. But as those pipelines grow to hundreds of directed acyclic graphs (DAGs) spanning services like AWS Glue, Amazon EMR, Amazon Athena, and Amazon Redshift, debugging a single task failure turns into a significant operational challenge. When a task fails, data engineers sift through logs, cross-reference DAG configurations, and analyze error messages to find the root cause, delaying pipeline service level agreements (SLAs) and impacting team productivity.
In this post, we show you how to build a custom Apache Airflow plugin that integrates with Amazon Bedrock to automatically analyze DAG task failures and provide actionable diagnostic insights. The plugin deploys to Amazon Managed Workflows for Apache Airflow (Amazon MWAA) and provides AI-powered root cause analysis on demand.
The complete source code for this solution is available in the sample-aws-mwaa-llm-powered-plugin GitHub repository. Clone the repository and follow along as we explain the design decisions throughout this post.
Solution overview
Apache Airflow is a widely adopted open source platform for programmatically authoring, scheduling, and monitoring complex data pipelines. Teams use Airflow to orchestrate extract, transform, and load (ETL) processes, machine learning workflows, and data lake management across industries.
Amazon MWAA is a managed service that makes it straightforward to run Apache Airflow on AWS without the operational burden of managing the underlying infrastructure. With Amazon MWAA, you can focus on authoring workflows and business logic while AWS handles provisioning, patching, scaling, and securing your Airflow environments.
The plugin adds an analysis view directly into your Airflow UI. At a high level, when a task fails and you trigger an analysis, the plugin automatically does the following:
Retrieves the failed task instance metadata from the Airflow metadata database.
Collects comprehensive context including task logs, DAG source code, and operator-specific scripts.
Sends the enriched context to Amazon Bedrock for analysis.
Returns a structured diagnostic report with root cause identification, step-by-step resolution, and prevention recommendations.
How it works
The preceding four steps happen behind a single Analyze Task action. The following diagram and pipeline show the high-level architecture and how the plugin carries them out.
Figure 1: High-level architecture of the LLM-powered task analyzer plugin on Amazon MWAA
The plugin follows a multi-step analysis pipeline:
User triggers analysis – From the Airflow UI, you select a failed task and choose Analyze Task.
Context collection – The plugin retrieves task metadata, execution logs, and DAG source code from the Airflow metadata database and Amazon S3.
Operator-aware enrichment – Based on the operator type, the plugin fetches the actual code or query that failed (for example, a PySpark script from AWS Glue or a SQL query from Amazon Athena).
Foundation model analysis – The enriched context is sent to Amazon Bedrock, which returns a structured diagnostic report.
Results presentation – The analysis displays in the Airflow UI with actionable recommendations.
All AWS API calls (Amazon Bedrock, Amazon S3, and AWS Glue) are authenticated through the aws_default Airflow connection. By default on Amazon MWAA, this connection has no static credentials, so boto3 falls back to the environment’s execution role. This means there are no keys to manage or rotate. If you need to call Amazon Bedrock or fetch scripts using a different identity, you can supply those credentials in the aws_default connection. This can be a dedicated IAM role or a cross-account principal, used instead of the execution role.
Operator-aware context collection
A key differentiator of this solution is its ability to understand different Airflow operator types and automatically fetch the associated code or queries. Unlike generic log analyzers, the plugin retrieves the actual code that failed, not just the error message.
The following table summarizes what the plugin fetches for each operator type:
Operator type
What the plugin fetches
Source
GlueJobOperator
PySpark or Python script
Amazon S3 (from the AWS Glue job definition)
EmrAddStepsOperator
Spark or Python script
Amazon S3 (from step arguments)
EmrServerlessStartJobOperator
Spark script
Amazon S3 (from job driver)
AthenaOperator
SQL query
Inline (from operator parameters)
RedshiftDataOperator
SQL query
Inline (from operator parameters)
BashOperator
Bash command
Inline (from operator parameters)
PythonOperator
Python function
DAG source code
This approach means the foundation model can analyze the actual logic that failed, correlating error messages with specific lines in your code for precise root cause identification.
Prerequisites
Before you begin, make sure that you have the following:
An Amazon MWAA environment running Apache Airflow 3.x (this walkthrough uses Airflow 3.2). The plugin registers its UI through the FastAPI-based plugin interface (fastapi_apps) introduced in Airflow 3.x. For setup instructions, see Get started with Amazon MWAA.
Access to Amazon Bedrock with the Anthropic Claude model family enabled in your AWS Region. This walkthrough uses Anthropic Claude, but you can adapt the plugin to work with Amazon Nova or other foundation models by modifying the prompt payload format in prompts.py. See Model access.
Note: In most Regions, you invoke Claude through an inference profile ID (for example, us.anthropic.claude-sonnet-4-5-20250929-v1:0) rather than a bare on-demand model ID. Run aws bedrock list-inference-profiles to confirm a model is ACTIVE before configuring it.
Plugin design
In this section, we explain the plugin design and its key components. The next section walks through deploying it to your Amazon MWAA environment.
The repository also includes example DAGs that simulate various failure scenarios across different operator types.
Plugin registration
In Apache Airflow 3.x, the web component of a plugin is registered as a FastAPI application through the fastapi_apps attribute. In task_analyzer_plugin.py, the TaskAnalyzerPlugin class registers the FastAPI app under /task-analyzer and adds a view to the task instance page:
Airflow automatically discovers any AirflowPlugin subclass in the plugins folder. No registration call or configuration change is needed. On Amazon MWAA, the file is delivered inside plugins.zip and extracted to /usr/local/airflow/plugins/.
Analysis engine
The analysis engine is the POST /api/analyze-task endpoint in task_analyzer_plugin.py. When you trigger an analysis, the endpoint performs the following steps:
Retrieves AWS credentials from the aws_default Airflow connection. To override, edit the aws_default connection in the Airflow UI (Admin > Connections).
Assembles a context dictionary from the request (task metadata, logs, DAG source).
Enriches the context with an operator-specific script through fetch_and_add_operator_script.
Builds the prompt using the template in prompts.py.
Invokes Amazon Bedrock and returns the structured analysis.
Operator script fetching
The process_operator_script function in script_utils.py routes script retrieval based on operator type:
External scripts (AWS Glue, Amazon EMR) – The plugin calls the AWS Glue API to look up the job definition, then reads the PySpark script from Amazon S3. Amazon EMR handlers follow the same pattern, extracting the script path from the step configuration or job driver.
Inline scripts (Amazon Athena, Amazon Redshift, BashOperator, PythonOperator, DBTOperator) – The plugin reads the query or command directly from the task’s rendered template fields with no external API call.
The plugin implements smart fetching: for external scripts, it only makes the Amazon S3 API call when the error message contains code-relevant patterns (such as SyntaxError, TypeError, or data type mismatch). Infrastructure errors like timeouts skip the script fetch entirely, minimizing unnecessary API calls.
Prompt engineering
The prompt template in prompts.py provides the foundation model with:
Task metadata (DAG ID, task ID, run ID, state).
Error message and execution logs.
DAG source code.
Operator-specific script (when available).
The model produces a structured diagnostic report with root cause identification, step-by-step resolution, and prevention recommendations. Model IDs are configurable through Airflow Variables, so you can switch between Claude Sonnet and Claude Opus without redeploying the plugin.
Security measures
Before sending content to Amazon Bedrock, the plugin applies the following safeguards:
Credential redaction – The sanitize_script function removes sensitive patterns (passwords, tokens, access keys) from scripts and logs.
Content truncation – The truncate_script function caps content size to stay within model context windows.
Path traversal prevention – The read_allowlisted_file function resolves canonical paths and verifies they reside within allowed base directories before reading any file.
Optional: PII detection and redaction. The built-in sanitize_script function targets credential patterns. If your logs or scripts might contain personally identifiable information (PII), consider adding a detection pass with Amazon Comprehend before invoking Amazon Bedrock. The DetectPiiEntities API returns the entity types (such as names, email addresses, or account numbers) and their character offsets. You can use these offsets to mask or obfuscate the spans before the context leaves your environment. This adds one API call and cost per analysis, so add it where your compliance requirements call for it. For guidance, see Detecting PII entities.
Deploy the plugin
Follow these steps to deploy the plugin to your Amazon MWAA environment.
Step 1: Clone the repository
git clone https://github.com/aws-samples/sample-aws-mwaa-llm-powered-plugin.git
cd sample-aws-mwaa-llm-powered-plugin
Step 2: Package and upload to Amazon S3
Create the plugins.zip archive from the plugins/ directory and upload it to your Amazon MWAA S3 bucket:
cd plugins
zip -r ../plugins.zip .
cd ..
aws s3 cp plugins.zip s3://<amzn-s3-demo-bucket>/plugins.zip
aws s3api head-object \
--bucket <amzn-s3-demo-bucket> \
--key plugins.zip \
--query VersionId --output text
Note the VersionId returned. You need it in the next step.
Note: This plugin requires only fastapi and Boto3, both pre-installed on Amazon MWAA for Airflow 3.x. You don’t need a requirements.txt file. Skipping the requirements file avoids package resolution conflicts that are a common cause of failed Amazon MWAA environment updates.
Step 3: Update the Amazon MWAA environment
Update your environment to use the new plugin archive:
On Amazon MWAA, the aws_default connection exists by default and resolves to your environment’s execution role. In most cases, no action is needed.
To override the Region, edit the aws_default connection in the Airflow UI (Admin > Connections) and set the Extra field to:
{"region_name": "us-east-1"}
Leave login and password empty so the execution role is used.
Step 5: Verify the deployment
After the environment finishes updating, navigate to Admin > Plugins in the Airflow UI. Verify that task_analyzer_plugin appears in the list. The Analyze Task entry is now available from any task instance view.
Test the solution
The repository includes example DAGs that simulate failure scenarios across different operator types. To validate the deployment:
Copy the dags/ directory contents to your Amazon MWAA S3 bucket’s DAGs folder:
Wait for Amazon MWAA to sync the DAGs (typically 1–2 minutes).
In the Airflow UI, trigger one of the test DAGs (for example, test_aws_sql_operators) and let the intentional failure occur.
Navigate to the failed task instance.
Choose Analyze Task in the task instance view.
Review the generated analysis, which includes:
Root cause identification with file and line references.
Step-by-step resolution with code examples.
Prevention recommendations and monitoring suggestions.
The analysis typically completes within 5–10 seconds.
Cost considerations
The primary cost driver for this solution is Amazon Bedrock inference, which is billed by the number of input and output tokens each analysis consumes. Input tokens come from the task logs, DAG source, and operator script sent to the model. Output tokens come from the diagnostic report the model returns. Larger logs and scripts increase input tokens, and the model you select affects the per-token rate. For current per-model rates, see Amazon Bedrock pricing.
To help control cost, the plugin includes a caching mechanism that stores results keyed by a hash of the error context. Repeated analyses of the same failure pattern return cached results without invoking Amazon Bedrock again.
Best practices
When you deploy this solution in production, consider the following:
IAM least privilege – Grant only bedrock:InvokeModel for your chosen model IDs and scope s3:GetObject to specific bucket paths where your operator scripts reside. For guidance, see Amazon MWAA execution role.
Data sanitization – The plugin redacts credentials and truncates content before sending data to Amazon Bedrock. Store configuration values in AWS Secrets Manager rather than hardcoding them in DAG source files.
Operational resilience – Add retry logic and circuit breaker patterns around the Amazon Bedrock API call. Use Amazon CloudWatch to monitor plugin performance and set alarms on failure rates.
Extending the solution
You can extend this solution in the following ways:
Proactive notifications – Integrate with Amazon Simple Notification Service (Amazon SNS) or Slack to deliver analyses automatically when failures occur.
Knowledge base integration – Build a knowledge base of past analyses using Amazon Bedrock Knowledge Bases for Retrieval Augmented Generation (RAG) powered recommendations that learn from your organization’s historical failures.
Additional operator support – Add handlers for custom operators specific to your organization, such as proprietary data connectors or internal platform integrations.
Automated remediation – For well-understood failure patterns, trigger automated fixes such as restarting tasks with adjusted resource configurations.
Clean up
To remove the plugin from your environment:
Delete the plugin archive from Amazon S3:
aws s3 rm s3://<amzn-s3-demo-bucket>/plugins.zip
Update your Amazon MWAA environment to remove the plugin reference, then wait for the environment to restart.
Optionally, remove the Amazon Bedrock permissions from your execution role if they are no longer needed.
Conclusion
In this post, we showed you how to deploy an LLM-powered DAG failure analysis plugin for Amazon MWAA using Amazon Bedrock. The operator-aware context collection differentiates this approach from generic log analyzers. By fetching the actual code from AWS Glue, Amazon EMR, and other services, the foundation model provides precise, actionable recommendations with specific line references.
To get started, clone the sample-aws-mwaa-llm-powered-plugin repository, deploy it to a development Amazon MWAA environment, and test with the included example DAGs. As your team builds confidence in the analysis quality, roll it out to production environments where it serves as the first line of investigation for any pipeline failure.
If you manage Amazon Elastic Compute Cloud (Amazon EC2) infrastructure at scale, you have likely encountered the following situation. You release an infrastructure change with the correct Region, the correct instance type, and a launch template that has operated reliably for months. The deployment nevertheless comes up on an Amazon Machine Image (AMI) that is several patch cycles out of date, because the AMI ID hardcoded in the template had become stale weeks earlier. The condition goes unnoticed until a security scan flags the instance, at which point you must reconcile AMI IDs across Regions rather than close out the week.
That scenario is rarely a one-time event. It is one example of a broader pattern that quietly taxes teams running Amazon EC2 at scale: stale AMI IDs, manual parameter lookups, inconsistent Region mappings, and pipelines that silently fail to update. The following section examines four variations of this pattern in detail.
The common thread across all of these is the same. Locating the correct image is not the hard part. The difficulty lies in wiring that image into your infrastructure as code (IaC) in a manner that remains current. You identify the appropriate AMI on the console, then search AWS Systems Manager (SSM) Parameter Store paths to obtain the dynamic reference that maps to it. The workflow spans two tools and two mental models, with a gap in between where errors accumulate. Because the authoritative link between an AMI and its SSM parameter lived outside the API, teams had to reconstruct it by hand, and hands make mistakes.
A recent enhancement to the Amazon EC2 DescribeImages API closes that gap. When you call DescribeImages on a public AMI, the response now contains a PublicSsmParameterName field: the SSM parameter that resolves to the latest AMI in that lineage. A single API call replaces manual correlation.
In this post, we examine the operational friction that makes AMI management harder than it should be and show how this enhancement addresses it. We walk through practical examples using the AWS Command Line Interface (AWS CLI), AWS CloudFormation, Terraform, and Amazon EC2 Auto Scaling launch templates. We conclude with best practices for golden AMI pipelines, including operational considerations to review before adopting the feature in production.
Prerequisites
To follow the examples in this post, you will need the following:
An AWS account.
The AWS CLI v2 installed and configured with appropriate permissions (ec2:DescribeImages, ssm:GetParameters).
Basic familiarity with AMIs, SSM Parameter Store, and at least one IaC tool (CloudFormation or Terraform).
Understanding the operational challenges
Before addressing the solution, it is worth examining the problem in detail, because the problem seldom manifests as a single, dramatic failure. It is instead a gradual accumulation of minor frictions that, in aggregate, impose a measurable cost on teams responsible for compute.
AMI IDs are Region-specific, version-specific, and change frequently. The workflow of finding an AMI, locating its SSM parameter, and referencing it in templates spans multiple tools, and the boundaries between steps are where errors accumulate.
Challenge 1: Silent image aging
Scenario: An engineer copies an AMI ID into a Terraform module as an interim measure. Several months later, that identifier is embedded across four environments. New instances launch on an image that predates numerous patches. There is no error and no alert, only drift that remains invisible until an audit or a review brings it to light.
Impact: Hardcoded AMI IDs do not fail conspicuously. They fail quietly, by launching a prior image at a later date. The distance between “this was correct when written” and “this remains correct” widens continuously, and no owner is assigned to monitor it.
Challenge 2: The multi-region maintenance burden
Scenario: An application operates across three Regions. The same logical image (for example, the latest Amazon Linux 2023) carries a different AMI ID in each Region. Templates therefore accrue region-to-AMI mapping blocks, lookup logic, or both. Each additional Region introduces another entry to maintain, and each AMI refresh requires updating all of them.
Impact: The team ends up maintaining a translation table that AWS already maintains on its behalf. The mapping logic becomes load-bearing infrastructure in its own right, and a single stale entry in one Region produces inconsistent fleets that are difficult to diagnose.
Challenge 3: Barriers to onboarding
Scenario: A new engineer joins the team and poses a reasonable question: which SSM parameter corresponds to a given AMI? The answer resides in an internal knowledge-base page that was accurate eighteen months earlier. The engineer copies a path that appears correct, deploys, and inadvertently references the wrong lineage.
Impact: When the relationship between an AMI and its parameter is not discoverable from the API, it must be documented manually. Manually maintained mappings degrade over time. Each new team member re-learns the same institutional knowledge, and each instance of degradation introduces an opportunity to reference an incorrect value.
Challenge 4: Uncertainty about update success
Scenario: A golden AMI pipeline completes a build and updates a parameter. The command returns a success response, and the team assumes the new image is in effect. However, for certain parameter data types, a success response does not always indicate that the value was accepted. This specific behavior is examined in the best-practices section, as it is particularly relevant to golden AMI pipelines.
Impact: Confidence without confirmation carries substantial risk. A pipeline that presumes success can propagate a stale image across a fleet before the discrepancy is identified.
Considered individually, none of these situations constitutes a crisis. Considered collectively, they explain why “launch the latest image” is never, in fact, a single step. The common root cause is consistent across all four: the authoritative link between an AMI and its SSM parameter existed outside the API, requiring teams to reconstruct it manually, a process inherently prone to error.
What’s new: DescribeImages returns the associated SSM parameter
The new feature addresses precisely this boundary.
As of July 16, 2026, the Amazon EC2 DescribeImages API response includes a new field, PublicSsmParameterName, for public AMIs that have an associated SSM parameter. This capability is available at no additional cost in supported AWS Regions, including AWS GovCloud (US) Regions and the China Regions.
In place of the previous three-step correlation exercise, the workflow reduces to a single call:
Before
After
Find AMI → manually search SSM paths → confirm the correct match
Find AMI → PublicSsmParameterName returns the SSM path immediately
Two separate API calls or console workflows
A single DescribeImages call provides the complete mapping
Prone to mapping an incorrect parameter to an AMI
Authoritative mapping obtained directly from the API
The change introduces neither a new service nor a new pricing dimension. It relocates information that previously lived in knowledge bases into the API response.
In addition, you can now use the public-ssm-parameter-name filter in DescribeImages to identify all AMIs associated with a specific SSM parameter, making the relationship queryable in either direction.
How it works: API response walkthrough
Call DescribeImages on a public AMI with an associated SSM parameter. The response includes PublicSsmParameterName:
The PublicSsmParameterName value (in this case, aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64) identifies the SSM parameter associated with this AMI lineage.
Tip: The field is returned under the aws/service/ namespace without a leading slash. When using this value in SSM API calls, resolve:ssm: references, or CloudFormation dynamic references, prepend a forward slash. For example, use /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64. The SSM parameter is intended to resolve to the latest AMI in the lineage, which can help you keep infrastructure current.
Note: Not every public AMI has an associated parameter. The field is present only for lineages for which AWS publishes parameters. The field is also populated only for public AMIs. If you query one of your own private AMIs and observe an empty field, this is expected behavior rather than a defect.
Practical examples
The following four examples show how to use the new PublicSsmParameterName field across common IaC tools.
Example 1: Discover the SSM parameter for an AMI using the AWS CLI
Suppose you have identified an AMI on the console and wish to determine its SSM parameter path for use in your templates:
You can also perform the inverse operation and determine which AMI a given SSM parameter currently references:
# Linux example
aws ssm get-parameter \
--name "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64" \
--query "Parameter.Value" \
--output text
# Windows example
aws ssm get-parameter \
--name "/aws/service/ami-windows-latest/Windows_Server-2022-English-Full-Base" \
--query "Parameter.Value" \
--output text
Tip: Public SSM parameters are available for both Linux (/aws/service/ami-amazon-linux-latest) and Windows (/aws/service/ami-windows-latest) AMIs. You can list all available parameters under these paths using aws ssm get-parameters-by-path –path .
Alternatively, you can use the new filter to identify AMIs by their SSM parameter name:
Note: The public-ssm-parameter-name filter returns all AMIs that have ever been associated with the specified parameter, including previous versions. Use sorting or additional filters (such as –query with CreationDate) to identify the most recent AMI.
Example 2: CloudFormation with dynamic SSM references
Once the SSM parameter path is known from DescribeImages, you can use CloudFormation dynamic references to resolve to the latest AMI at deployment time:
AWSTemplateFormatVersion: '2010-09-09'
Description: EC2 instance using SSM parameter for latest Amazon Linux 2023 AMI
Parameters:
InstanceType:
Type: String
Default: t4g.micro
Resources:
MyInstance:
Type: AWS::EC2::Instance
Properties:
InstanceType: !Ref InstanceType
ImageId: '{{resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64}}'
Tags:
- Key: Name
Value: MyLatestAL2023Instance
Alternatively, you can use the AWS::SSM::Parameter::Value parameter type to permit users to override the SSM path at stack creation time:
AWSTemplateFormatVersion: '2010-09-09'
Description: EC2 instance with configurable SSM-based AMI lookup
Parameters:
AmiSsmParameter:
Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
Default: '/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64'
Description: SSM parameter path for the AMI (discovered via DescribeImages)
InstanceType:
Type: String
Default: t4g.micro
Resources:
MyInstance:
Type: AWS::EC2::Instance
Properties:
InstanceType: !Ref InstanceType
ImageId: !Ref AmiSsmParameter
Tags:
- Key: Name
Value: DynamicAMIInstance
CloudFormation resolves the AMI ID at deployment time, so the template never contains a hardcoded AMI ID, and any stack update adopts the latest AMI automatically. Two considerations warrant attention before relying on this approach. First, running instances are not affected. A stack update is required to roll out a newer AMI. Second, CloudFormation does not support drift detection on dynamic references, so if the underlying SSM parameter value changes between deployments, CloudFormation will not report it as drift. For ssm dynamic references in which a version has not been pinned, AWS recommends performing a stack update whenever the parameter changes, so that the stack retrieves the current value.
Example 3: Terraform with SSM parameter data source
Use the aws_ssm_parameter data source to resolve the SSM path to the latest AMI ID:
# Use the SSM parameter path discovered from DescribeImages
data "aws_ssm_parameter" "latest_al2023" {
name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64"
}
resource "aws_instance" "web" {
ami = data.aws_ssm_parameter.latest_al2023.value
instance_type = "t4g.micro"
tags = {
Name = "LatestAL2023-Instance"
}
}
Important: In Terraform, ami is a replacement-forcing argument on aws_instance. When the SSM parameter changes, Terraform proposes to destroy and recreate the instance. For stateful workloads, add lifecycle { ignore_changes = [ami] } or use launch templates with Auto Scaling (Example 4) instead.
Example 4: Auto Scaling launch templates with SSM parameters
For Auto Scaling groups, you can reference the SSM parameter directly in the launch template using the resolve:ssm: prefix:
# Create a launch template that uses the SSM parameter
aws ec2 create-launch-template \
--launch-template-name al2023-auto-scaling \
--launch-template-data '{
"ImageId": "resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64",
"InstanceType": "t4g.micro"
}'
When EC2 Auto Scaling launches a new instance, it resolves the SSM parameter at launch time to obtain the current AMI ID. You can verify the AMI ID to which a launch template resolves:
The parameter is stored in the launch template. When the Auto Scaling group scales out or replaces an instance, it uses the launch template to resolve the SSM parameter and determine the AMI to launch. This is the most direct of the four patterns: the parameter serves as the single source of truth, and the Auto Scaling group’s normal instance lifecycle effects the rollout.
Before-and-after workflow comparison
The following table summarizes how this feature improves common workflows, and relates each entry to the challenges described earlier.
Workflow
Before
After
Discover the SSM path for a known AMI
Search SSM parameter namespaces manually. Test multiple paths. Confirm a correct match
A single DescribeImages call returns PublicSsmParameterName
Validate that an SSM parameter maps to the expected AMI
Call GetParameter, then call DescribeImages on the returned ID to verify
Use the public-ssm-parameter-name filter to view all associated AMIs directly
Set up IaC templates
Find AMI → search for SSM path → copy path to template → verify correctness over time
Find AMI → read PublicSsmParameterName from the response → use directly in the template
Onboard new team members
Document AMI-to-parameter mappings in knowledge bases, which become stale
New members self-discover using standard API calls
Audit AMI usage across teams
Cross-reference AMI IDs with SSM parameters in separate calls
A single API call provides the complete picture
Best practices: Using SSM parameters for golden AMI pipelines
The following recommendations describe how to derive the greatest benefit from this feature, with operational considerations identified where they are material.
1. Discontinue hardcoding AMI IDs
With PublicSsmParameterName removing the discovery barrier, switch all templates to SSM parameter references. Use {{resolve:ssm:}} in CloudFormation, the aws_ssm_parameter data source in Terraform (note the replacement behavior in Example 3), or the resolve:ssm: prefix in launch templates.
2. Create custom SSM parameters for your golden AMIs
For internally built golden AMIs, create your own SSM parameters using the aws:ec2:image data type:
Stacks, launch templates, or Terraform configurations that reference this parameter can adopt the new AMI on their next deployment, with no template edits required in most cases.
Operational consideration: Because PutParameter validates aws:ec2:image values asynchronously, an HTTP 200 does not confirm the value was accepted. Subscribe to Parameter Store change events in Amazon EventBridge and confirm the operation succeeded before considering the rollout complete.
3. Use parameter versions and labels for controlled rollouts
SSM Parameter Store supports versioning and labels, which provide control over rollouts:
# Label the current production version
aws ssm label-parameter-version \
--name "/my-org/golden-ami/amazon-linux-hardened" \
--parameter-version 5 \
--labels "prod"
Production launch templates reference the labeled version:
With this approach, you can update the parameter with a new AMI without immediately affecting production. Promotion to production is accomplished by moving the prod label, a deliberate, auditable action rather than an automatic side effect.
4. Combine with Amazon EC2 Image Builder for end-to-end automation
Use Amazon EC2 Image Builder to automate AMI creation, then configure the distribution settings to update your SSM parameter automatically when a new AMI is built. Combined with the new discovery feature, this establishes a closed loop:
Image Builder creates a new AMI on a schedule.
Distribution settings update the SSM parameter to point to the new AMI.
Auto Scaling and IaC resolve the parameter to the latest AMI at launch time.
With DescribeImages, any authorized party can determine which SSM parameter an AMI maps to.
5. Scope IAM permissions appropriately
Two permission requirements apply.
To launch instances by using SSM-referenced AMIs, the launching principal requires ssm:GetParameters on the relevant parameter paths:
Scope the Resource element to the paths actually in use. If you reference Windows parameters (/aws/service/ami-windows-latest/) or your own golden AMI paths (/my-org/golden-ami/), include those ARNs as well. Otherwise, launches will fail with an AccessDenied error.
To create a custom aws:ec2:image parameter, the pipeline principal also requires ssm:PutParameter and ec2:DescribeImages:
For broader guidance on keeping infrastructure current and automating operational processes, see the Operational Excellence Pillar of the AWS Well-Architected Framework.
Clean up
The examples in this post use read-only API calls (DescribeImages, GetParameter) and do not create billable resources. If you created a launch template while following Example 4, you can delete it as follows:
The difficulty of AMI management was never attributable to any single failure. It arose from the steady accumulation of stale identifiers, region-mapping tables, stale documentation, and pipelines that presumed success, all of which are minor frictions that together produced significant operational effort and risk. The common thread was that the authoritative link between an AMI and its SSM parameter existed outside the API, requiring teams to reconstruct it manually.
The new PublicSsmParameterName field in the Amazon EC2 DescribeImages API relocates that link into the response, where it appropriately belongs. With a single API call, you can determine the SSM parameter for any public AMI. You can then reference it directly in CloudFormation templates, Terraform configurations, or Auto Scaling launch templates for automatic AMI updates.
Deploying Spark applications from a local development environment to a remote Amazon EKS cluster often means dealing with environment differences, dependency conflicts, and performance gaps at scale. Spark Connect removes this friction. It separates your application client from the Spark server, so you develop and debug locally while Spark Connect routes your operations to a scalable Spark cluster running on Amazon EKS.
This client-server architecture supports a range of use cases, including interactive development from notebooks and IDEs, embedded Spark in web services, and continuous integration and continuous delivery (CI/CD) data-quality tests. All of these run on your existing EKS infrastructure. Each Spark Connect session uses its own AWS Identity and Access Management (IAM) execution role, custom tags, and cost tracking. For more information, see the Amazon EMR on EKS documentation.
Here are two demonstrations of using Spark Connect in Amazon SageMaker Unified Studio Notebooks and in a VS Code local IDE:
Amazon SageMaker Unified Studio Notebooks demo:
Demo 1: Spark Connect running in an Amazon SageMaker Unified Studio notebook
Local IDE demo:
Demo 2: Spark Connect running in a local VS Code IDE session
For a runnable end-to-end example in an IDE, try the Spark Connect sample notebook in the aws-emr-utilities repository. It includes a client wrapper solution, built by AWS architects, for simplified connectivity:
Demo 3: Spark Connect client wrapper connecting to Amazon EMR on EKS from a local IDE
How Spark Connect works on Amazon EMR on EKS
Spark Connect uses a client-server architecture that separates application code from the Spark engine:
Client – A lightweight PySpark library running in your environment (such as an IDE or notebook). It doesn’t need Spark installed, direct access to data, or resources sized for the workload.
Connection (EMR managed endpoint) – The client sends Spark operations over a secure gRPC/TLS channel to the Spark Connect server.
Server – Runs Spark pods in your Amazon EMR on EKS namespace, starting from a minimum of two executors (adjustable) with autoscaling. The server performs Spark operations using the EKS compute resources and accesses data stores, such as an Amazon Simple Storage Service (Amazon S3) bucket, through job execution roles.
Results – The server streams query results back to the client through gRPC as Apache Arrow-encoded row batches.
On endpoint creation, Amazon EMR on EKS launches the Spark Connect server as pods on EKS and returns an Elastic Load Balancing (ELB)-backed endpoint and a short-lived token. You don’t need to provision any server or networking manually. Because the Spark Connect server runs on the EKS cluster you already operate, it inherits the node types, container images, and Spark configurations. What you see while developing Spark applications on the client side is what runs in the EKS environment at scale.
To provide a secure, simplified experience, Amazon EMR on EKS provisions two additional components on first use of Spark Connect on the EKS cluster:
Figure 2: Shared Envoy router and Secret Agent service on the EKS cluster
Managed authentication-proxy router – a shared Envoy router with three replicas by default (adjustable), fronted by a Network Load Balancer (NLB). It routes client traffic to the correct server pods, terminates TLS, and validates the session token. One router serves Spark Connect endpoints on the EKS cluster.
Secret Agent service – a lightweight, long-running pod that manages the short-lived credentials for session authentication. One service per EMR security configuration.
These components are long-running and shared across endpoints. Amazon EMR on EKS creates them automatically with the first endpoint on the cluster. Because the router is cluster-scoped and Secret Agent is namespace-scoped, deleting a managed endpoint doesn’t remove them. They keep running so that new endpoints can start within a minute. The router’s replica count is tunable. Scale down for non-production environments to reduce cost or scale up for higher throughput.
To fully remove these components:
Terminate all active managed endpoints and their virtual cluster that reference the Secret Agent’s security configuration, then delete the security configuration.
Once the last session-enabled virtual cluster is deleted, the authentication-proxy router and its underly resources, including the NLB and VPC endpoint, are removed automatically.
Alternatively, delete the EKS cluster to remove all in-cluster components at once.
Why use Spark Connect on Amazon EMR on EKS
With Amazon EMR on EKS, teams can run Spark alongside other applications on shared Kubernetes clusters with existing infrastructure, operational tooling, and system expertise. Spark Connect extends that value to interactive, embedded, and self-service Spark workloads. Your client stays lightweight while Spark code runs in governed, scalable server pods on EKS.
Interactive development on shared Kubernetes clusters
Data engineers and scientists iterate on Spark code cell-by-cell in notebooks or local IDEs. The Spark engine runs remotely on EKS, so validation runs on the same engine as your batch workloads. After validation on the Spark Connect client, the same Spark code deploys as a batch StartJobRun with no changes.
Spark Connect sessions run as pods on your existing cluster. They reuse your EKS RBAC, network policies, node autoscaling, and observability stack (Prometheus, Grafana, Amazon CloudWatch Container Insights). There are no separate compute and monitoring layers to operate.
Embedded Spark in applications and services
The Spark Connect client is a compact PySpark library. Teams can embed Spark operations directly into Python applications such as web services, dashboards, automation scripts, or backend APIs. The heavy processing runs on EKS while the application stays lightweight.
Teams can also expose Spark Connect as a self-service capability on their internal application. Business users submit Spark SQL scripts from a web UI. The compute runs on Spark Connect server on EKS, so the team manages capacity, security, and upgrades centrally.
Multi-tenant data exploration with governance
Each Spark Connect session uses the data user’s IAM permissions that you configure, limiting their access to authorized AWS services, data lake tables, and S3 paths. Every session carries tags with user, project, endpoint and virtual cluster IDs, feeding directly into billing and compliance reports. Meanwhile, data producers maintain guardrails on source data without blocking self-service exploration.
To manage resource consumption across teams, Amazon EMR on EKS virtual clusters provide namespace-level isolation. Each tenant binds their Spark Connect endpoints to a virtual cluster (a namespace) with independent IAM roles. Using resource quotas and limit ranges on EKS, you can protect each virtual cluster by controlling the compute resources that Spark Connect sessions can consume. Importantly, activating EKS split-cost allocation tags helps with chargeback reporting in a multi-tenant environment.
Reusable container images and scalable deployment
Teams often maintain custom container images with proprietary libraries, including internal feature stores, compliance toolkits, UDFs, or machine learning (ML) frameworks. With Spark Connect on Amazon EMR on EKS, teams reuse those same images as the Spark runtime for interactive sessions. No separate dependency lists needed. The same image works for both batch jobs and Spark Connect sessions.
Beyond the image itself, you can control Spark pod scheduling in Amazon EMR on EKS through pod templates and managed endpoint APIs, scaling across your environment. For example, you can:
Pin server pods to specific node types through pod templates. For example, Spot for cost savings.
Apply Spark Dynamic Resource allocation (DRA) to right-size each interactive session.
Use GPU node pools for accelerated Spark RAPIDS or ML.
Multi-cluster, multi-Region, and hybrid architectures
Enterprises running EKS clusters across multiple AWS accounts, AWS Regions, or hybrid environments with on-premises Kubernetes can use Spark Connect to query data wherever it’s processed. The lightweight client only needs to reach the Spark Connect endpoint, not the underlying S3 buckets or AWS Glue data catalogs. This means no VPC peering or direct network paths to every data store.
The client-server split is the core architectural advantage of Spark Connect on Amazon EMR on EKS. A developer on a laptop behind a VPN, a CI/CD deployment pipeline in a centralized service account, or an Airflow DAG orchestrating across Regions can all connect to a remote Spark server on EKS. This works regardless of where the client itself runs. This decoupling simplifies cross-Region or cross-account analytics without duplicating data or requiring direct access to each data store.
Getting started
To create a Spark Connect endpoint on Amazon EMR on EKS, complete the following steps:
Create EMR namespaces on EKS.
Create an EMR security configuration.
Create a virtual cluster with the security configuration.
Create a Spark Connect managed endpoint.
Obtain a session token.
Connect from your application.
Prerequisites
To proceed with this post, make sure you have the following:
Security note: Communication between your environment and the Spark Connect server is encrypted using TLS. The authentication token is time-limited (15 minutes by default). For long-running sessions, refresh the token periodically by calling get-managed-endpoint-session-credentials again. Consider using AWS Secrets Manager to store and retrieve tokens programmatically.
Step 6: Connect from your application
Use the returned endpoint URL and token to connect from a PySpark-compatible environment. The following Python code shows how to establish a Spark Connect session:
import os
from pyspark.sql import SparkSession
session_endpoint = os.environ["EP_URL"]
auth_token = os.environ["TOKEN"]
spark_conn_url = (f"{session_endpoint};use_ssl=true;x-aws-proxy-auth={auth_token}")
spark = SparkSession.builder
.remote(spark_conn_url)
.getOrCreate()
# verify the connection
print(f"Connected remotely! Spark version: {spark.version}")
# query data through the AWS Glue Data Catalog
df = spark.sql("SELECT * FROM my_catalog.my_database.my_table LIMIT 10")
df.show()
After you’re connected, you can:
Debug interactively – Set breakpoints, inspect DataFrames, and step through Spark code in your IDE or notebook while the operations run remotely on EKS.
Combine local and remote processing – Pull query results back to the client as a pandas or PyArrow DataFrame for local analysis, visualization, or ML (scikit-learn, notebook widgets), then push further Spark operations back to the server in the same session. Heavy processing stays on Amazon EMR on EKS. Only the results you request cross the wire.
Reconnect without losing state – A managed endpoint runs independently of single clients for a configurable idle timeout (default: 60 minutes). Your Spark session, cached data, and temporary views are preserved on the server between connections. When a session token expires (default: 15 minutes, configurable up to 12 hours), request a new token and reconnect to the same endpoint to resume where you left off.
Reuse across workload types – The same client connection pattern works everywhere Python runs: notebooks, IDEs, batch scripts, Airflow operators, or web services. One endpoint, one connection pattern, many workload types.
Validation
After you create the endpoint, verify that the Spark Connect server is running and reachable through Amazon EMR on EKS API and standard Kubernetes tooling:
# get endpoint status
aws emr-containers describe-managed-endpoint --virtual-cluster-id $VC_ID --id $EP_ID
# inspect the server pods (driver + executors) in your namespace
kubectl get pods -n $USER_NAMESPACE -l "emr-containers.amazonaws.com/managed-endpoint-id=$EP_ID"
# View driver logs
kubectl logs -n $USER_NAMESPACE <driver-pod-name> -c spark-kubernetes-driver
Figure 4: Endpoint status and the running driver and executor pods
# to view the live Spark UI, port-forward your driver pod:
DRIVER_POD=$(kubectl get pods -n $USER_NAMESPACE \
-l "emr-containers.amazonaws.com/managed-endpoint-id=$EP_ID,emr-containers.amazonaws.com/component=driver" \
-o name)
kubectl port-forward -n $USER_NAMESPACE "$DRIVER_POD" 4040:4040
# Open http://localhost:4040 in your browser
Figure 5: Live Spark UI for the Spark Connect session
Spark Connect endpoints run as pods on your EKS cluster. The existing Kubernetes observability stack, such as CloudWatch Container Insights, Prometheus, and Grafana, captures Spark Connect endpoint metrics alongside other cluster workloads.
Clean up resources
Terminate your session when you’re done to avoid ongoing costs:
# (OPTIONAL) Endpoints are auto-deleted after the idle timeout (default: 60 minutes).
aws emr-containers delete-managed-endpoint \
--virtual-cluster-id $VC_ID \
--id $EP_ID
# Delete the virtual cluster only when no active endpoints remain
aws emr-containers delete-virtual-cluster --id $VC_ID
# Delete Security Configuration
aws emr-containers delete-security-configuration --id $SEC_CONFIG_ID
# remove the remaining EKS namespaces
kubectl delete namespace $USER_NAMESPACE $SYS_NAMESPACE spark-connect-router
Deleting or timing out a managed endpoint automatically removes its corresponding driver and executor pods. The Envoy router and Secret Agent service are shared across endpoints on the EKS cluster and remain running when individual endpoints are terminated. To fully remove these shared components, delete the virtual cluster to remove its corresponding Secret Agent service. Before doing so, ensure that no managed endpoints in the virtual cluster are active. Terminating the last session-enabled virtual cluster automatically removes the Envoy router from the EKS cluster.
Availability and pricing
Spark Connect on Amazon EMR on EKS is available with EMR release 7.14 (Apache Spark 3.5) and emr-spark-8.1 (Apache Spark 4.1), in all AWS Regions where Amazon EMR on EKS is available, except the AWS GovCloud (US) Regions and the China Regions. The Amazon SageMaker Unified Studio experience is available in supported Regions.
There is no additional charge for Spark Connect managed endpoints beyond the standard Amazon EMR on EKS pricing. You pay for underlying Amazon EKS resources such as EC2 and ELB. For timed-out or terminated managed endpoints, EMR automatically removes their Spark pods from the EKS cluster.
Recommendations for cost efficiency:
Use Karpenter (or Cluster Autoscaler) to right-size cluster capacity to session workload demand. This provisions nodes when endpoints need them and removes them when idle, which keeps cost aligned to actual usage.
Schedule interactive session pods on On-Demand instances for persistent compute.
Keep a single, shared Envoy router and NLB serving all Spark Connect endpoints (the default) on the cluster. Right-size the router replica count (three by default) for your availability requirements.
In this post, we showed how, with Spark Connect on Amazon EMR on EKS, you can build, test, and debug Spark applications from the tools you already use: IDEs, notebooks, Amazon SageMaker Unified Studio or Airflow. Your workloads run at scale on your existing Kubernetes clusters, with no application code changes.
For teams already running Amazon EMR on EKS, Spark Connect extends your virtual clusters to interactive and embedded workloads. The same virtual cluster that runs your batch StartJobRun jobs now also serves Spark Connect sessions. Each session runs as pods on your EKS cluster, inheriting your node groups, container images, and Spark configurations. Each session also carries its own IAM execution role and cost tags. This extends the security, multi-tenancy, and observability of your Amazon EMR on EKS investment to a broader set of users and use cases.
When you need quick insights from your Amazon Aurora PostgreSQL operational data, traditional analytics approaches force you to build complex extract, transform, and load (ETL) pipelines. These pipelines introduce latency, operational overhead, and data silos, which slow down decision making and increase cost. AWS introduced the support for Amazon Aurora PostgreSQL zero-ETL integration with Amazon SageMaker, providing near real-time data availability for analytics workloads.
The zero-ETL integration automatically replicates the data from your Amazon Aurora PostgreSQL database into a target AWS Glue managed catalog, where it’s available as Apache Iceberg tables. You can then analyze this data through Amazon SageMaker alongside data from other sources using your preferred analytics and machine learning (ML) tools. The data is compatible with Apache Iceberg open standards, so you can use SQL, Apache Spark, business intelligence, and artificial intelligence and machine learning (AI/ML) tools.
In this post, you explore the benefits of this integration, the architectural concepts, and the underlying change data capture (CDC) mechanics. You also go through the setup process and learn how to query your Aurora PostgreSQL data in Amazon SageMaker AI.
Zero-ETL in the lakehouse architecture
The lakehouse architecture of Amazon SageMaker AI brings together data across Amazon Simple Storage Service (Amazon S3) data lakes and Amazon Redshift data warehouses. Because it’s built on open standards, you can build analytics and AI/ML applications on a single copy of data, without moving it between systems.
Amazon SageMaker AI uses AWS Glue Data Catalog and AWS Lake Formation to provide integrated access controls across S3 data lakes and Amazon Redshift data warehouses from a single governance plane.
Understanding change data capture mechanics
At its core, Aurora PostgreSQL zero-ETL integration is powered by CDC. CDC continuously monitors the database transaction log and streams every insert, update, and delete to a downstream target in near real time.
Aurora PostgreSQL uses enhanced logical replication as its CDC engine. Standard PostgreSQL logical replication publishes row-level changes from the write-ahead log (WAL). The enhanced logical replication in Aurora offers added capabilities that make it well-suited for zero-ETL integrations, including automatic DDL propagation and continuous streaming of transactional changes.
Solution overview
With Amazon Aurora PostgreSQL zero-ETL integration with Amazon SageMaker AI, you can:
Remove ETL complexity – Automatically replicate data without building custom ETL pipelines.
Near real-time analytics – Access operational data in Amazon SageMaker AI within seconds of changes in Aurora PostgreSQL.
Unify data analysis – Combine Aurora PostgreSQL data with data from other sources in a single lakehouse architecture.
Reduce costs – Minimize operational overhead and infrastructure costs associated with maintaining ETL pipelines.
Accelerate insights – Query data using familiar SQL tools and integrate with ML workflows in Amazon SageMaker AI.
The following diagram illustrates the architecture of this solution:
Figure 1: Architecture of the Aurora PostgreSQL zero-ETL integration with Amazon SageMaker
The workflow includes the following steps:
Your application writes data to an Amazon Aurora PostgreSQL database cluster.
The zero-ETL integration automatically captures changes from the Aurora PostgreSQL database.
Data is replicated to the target AWS Glue managed catalog in near real time.
You can query and analyze the data using Amazon Athena, Amazon Redshift, or other analytics tools integrated with Amazon SageMaker AI.
Data scientists can build and train ML models using Amazon SageMaker AI with direct access to the Apache Iceberg tables in the target AWS Glue managed catalog.
Prerequisites
Before setting up the zero-ETL integration, verify that you have the following:
An Amazon Virtual Private Cloud (Amazon VPC) setup with the proper networking configurations for database connectivity.
If you’re creating a new Aurora PostgreSQL cluster, wait for your DB instance(s) to be in an “Available” status. You can verify DB instance status by using the describe-db-instances API call:
Wait until the cluster and the primary instance are back in Available status. For more information, see reboot-db-instance.
Create a target AWS Glue managed catalog
With your source PostgreSQL database configured for enhanced logical replication, the next step is setting up your target Amazon SageMaker AI. Zero-ETL integration uses AWS Glue Data Catalog backed by Amazon Redshift managed storage as its target. To have this functionality, you need to create a managed catalog, configure IAM permissions for Amazon SageMaker AI to access and query the managed catalog, and set up authorization for incoming integration requests from your source database.
Create an AWS Glue managed catalog
You must create a new catalog (if it doesn’t exist already) managed by AWS Glue to store table metadata and serve as the landing zone for your replicated datasets. Zero-ETL integration streams the data into Amazon Redshift managed storage, and AWS Glue keeps track of table definitions so that tools such as SageMaker AI, Athena, and Amazon Redshift Spectrum can query the data.
Create an IAM role for AWS Glue and Amazon Redshift to access the AWS Glue managed catalog
Now, use the following command to create an IAM role so that AWS Glue and Amazon Redshift can interact with the catalog. This role serves two key functions: It allows AWS Glue and Amazon Redshift to perform catalog operations, and it authorizes incoming integration requests from your source database.
Next, attach a policy to this IAM role that provides the minimum required permissions for AWS Glue and Amazon Redshift. This policy should also include the necessary permissions for encryption key actions to help maintain secure data handling throughout the integration process:
Before using the managed catalog for zero-ETL integration, you must configure data lake administrators in AWS Lake Formation who have administrative or read-only permissions on the managed resources. Additionally, you need to grant ReadOnlyAdmin permissions to the Amazon Redshift service-linked role, AWSServiceRoleForRedshift, in your account. If this role doesn’t exist in your account or you need to verify its permissions, see Using service-linked roles for Amazon Redshift.
Register the catalog as a zero-ETL integration target
To prepare your target AWS Glue managed catalog for zero-ETL integration, use the create-integration-resource-property command with these required parameters:
The –resource-arn parameter specifies the Amazon Resource Name (ARN) of your AWS Glue managed catalog that will serve as the integration target.
The –target-processing-properties parameter requires the ARN of an IAM role that has describe permissions on the target AWS Glue managed catalog.
You can use the GlueDataCatalogDataTransferRole created in the earlier step because it already includes the minimal describe permissions needed for this integration. Alternatively, you can create a new IAM role specifically for this purpose and attach the necessary minimal permissions to meet your company’s security requirements.
Configure authorization for inbound integration requests
The last step in creating a target managed catalog is to define a resource-based access policy that authorizes zero-ETL integration to push data into your catalog. This policy grants AWS Glue the necessary permissions to create and authorize incoming integration requests from your source database. Apply this resource policy by using the AWS Glue put-resource-policy API call to complete the catalog configuration for your zero-ETL integration:
Your AWS Glue managed catalog is now ready to receive data from the zero-ETL integration.
Load data in the source Aurora PostgreSQL database
Now that your Aurora PostgreSQL database is configured and ready, you must populate it with sample data that serves as the historical baseline for your zero-ETL integration. This first dataset provides the foundation for testing and demonstrating the integration capabilities. After you set up the zero-ETL integration, subsequent database changes stream automatically in near real time to your target AWS Glue managed catalog.
Connect to the source Aurora PostgreSQL cluster
Use the following commands to create a connection to your source Aurora PostgreSQL cluster:
Create a table named products to store product information:
CREATE TABLE products (product_id SERIAL PRIMARY KEY,product_name VARCHAR(100) NOT NULL, description TEXT,category VARCHAR(50),price NUMERIC(10,2) NOT NULL,stock_quantity INTEGER DEFAULT 0,is_active BOOLEAN DEFAULT TRUE,created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP );
Insert historical data
Use the following code to insert a row:
INSERT INTO products (product_name, description, category, price, stock_quantity) VALUES ('Laptop', 'High-performance laptop with 16GB RAM and 512GB SSD', 'Electronics', 1299.99, 50);
This table serves as a representative dataset to demonstrate the data capture and streaming capabilities of the zero-ETL integration. After your zero-ETL integration is active, all database changes, including inserts, updates, and deletes, are automatically captured and streamed to your AWS Glue managed catalog. This creates a data pipeline from your Aurora PostgreSQL database to your Amazon SageMaker for real-time analytics on your operational data.
Create a zero-ETL integration
Because your Aurora PostgreSQL database is now populated with historical data, you can set up the zero-ETL integration that continuously streams database changes to your AWS Glue managed catalog backed by Amazon Redshift managed storage.
Create the integration
Create the integration between your source PostgreSQL database and target AWS Glue catalog by using the aws rds create-integration AWS CLI command. You can customize the integration by specifying added configurations, such as data filters, to control which data gets replicated to your target environment:
When you run the command, the zero-ETL integration begins provisioning and enters a ‘creating’ state. The AWS CLI response provides key details about the integration configuration.
When the integration status changes to “active”, your zero-ETL integration pipeline is fully operational.
Monitor the integration
Before generating new live data, verify that the integration has reached an “active” state by running the describe-integrations AWS CLI command. This monitoring step is important to confirm that changes from your source Aurora cluster are successfully streaming to the AWS Glue managed catalog without errors:
Now that your historical data is loaded and the zero-ETL integration is “active”, you must confirm that the data has been successfully replicated.
Grant Lake Formation permissions
Before you can query the AWS Glue managed catalog by using the Amazon Redshift Data API, you must make sure the IAM user or role has the right permissions to create and manage tables within the catalog. Use the Lake Formation grant-permissions API to provide these necessary permissions so that Amazon Redshift can access your AWS Glue managed catalog for the zero-ETL integration. For more information, see Creating an Amazon Redshift managed catalog in the AWS Glue Data Catalog.
These permissions allow for query execution and metadata inspection on the managed catalog.
Query historical data by using the Amazon Redshift Data API
With the necessary permissions in place, you can now verify your historical data by querying the AWS Glue managed catalog through the Amazon Redshift execute-statement Data API. Begin this verification process by running a SELECT statement against the catalog:
Monitor your query’s progress by using the describe-statement API with the query ID. Continue checking until the status shows that your query has completed successfully:
//use the Id to make the describe-statement API call to verify execution status is Started
aws redshift-data describe-statement --id <ce1ff0el-xxxx>
To complete the verification process and view your historical data now available in Amazon SageMaker AI, retrieve the query results by using the get-statement-result API call:
With your zero-ETL integration now active, you can demonstrate real-time data streaming by adding new data to your source Aurora PostgreSQL instance. Run the following INSERT query to add a new row, which shows how changes are automatically replicated in near real time:
INSERT INTO products (product_name, description, category, price, stock_quantity)
VALUES ('Wireless Mouse', 'Ergonomic wireless mouse with USB receiver and long battery life', 'Electronics', 29.99, 150);
You can verify that the recent changes from your source database have been replicated to the target environment within seconds. Use the same Amazon Redshift Data API workflow you used earlier to confirm the real-time replication:
Use the describe-statement API call to monitor the query execution and confirm that the status shows ‘FINISHED’ before proceeding to retrieve the results:
This verification process confirms that your zero-ETL integration from Aurora PostgreSQL to Amazon SageMaker AI is working and continuously replicating both historical and real-time data. Although zero-ETL integration significantly simplifies data replication, it’s important to understand certain limitations on supported data types, schema change handling, and data filtering capabilities. For more details about these considerations and best practices, see Aurora zero-ETL integrations and Amazon RDS zero-ETL integrations.
Clean up
This section guides you through the cleanup process to remove the resources and components you created during this walkthrough. When you delete a zero-ETL integration, Amazon Aurora removes it from the source Aurora DB cluster. Your transactional data isn’t removed from Amazon Aurora or the analytics destination, but Aurora doesn’t send new data to Amazon SageMaker AI.
Delete the zero-ETL integration: Begin the cleanup process by removing the integration between your source Amazon Relational Database Service (Amazon RDS) database and the AWS Glue managed catalog. Run the following command to delete the integration:
Delete the AWS Glue managed catalog: After you successfully delete the integration, delete the AWS Glue managed catalog that served as your zero-ETL target destination. Use the following command to remove the catalog:
This permanently removes all associated table metadata and Amazon Redshift managed storage references.
Delete the Aurora DB cluster: If you created the source Aurora DB cluster for this demonstration and you no longer need it, you can complete the cleanup by deleting the entire DB cluster. By skipping the final snapshot option, you avoid retaining any test data and confirm complete resource removal:
In this post, you learned how to configure zero-ETL integration between Aurora PostgreSQL and your Amazon SageMaker AI using AWS CLI. This integration automatically replicates your PostgreSQL data to a lakehouse in near real time, removing the need for custom ETL pipelines.
As you move forward, consider expanding this zero-ETL approach to more supported data sources, such as Amazon RDS for MySQL and Amazon DynamoDB. This creates a centralized data access strategy across your company. You can also explore advanced analytics scenarios by combining zero-ETL integrations with Amazon Redshift capabilities. These include large-scale SQL analytics, Amazon Redshift ML for in-database ML, and federated queries that span multiple data lakes and warehouses. These integrations provide the foundation for building a near real-time data platform that scales with your business needs.
To get started, see the AWS zero-ETL documentation for setup guidance, supported configurations, troubleshooting integrations, and architectural best practices.
Every code review discussion, incident response thread, and standup happens in Slack. But when an engineer needs to analyze a service or debug a failing test, they leave Slack, open a terminal, navigate to the repository, run commands, and paste the output back. That round trip takes 30 seconds for someone who knows exactly where to look and 5 minutes for someone less familiar with the codebase. Across a team of 10 engineers doing this 15 times a day, that adds up to over 12 hours of lost engineering time per week.
This post walks through building a ChatOps integration that runs Kiro CLI from a Slack slash command. An engineer types /kiro analyze auth-service for memory leaks, and the results appear directly in the channel—no context switch required. The solution uses AWS Lambda, Amazon API Gateway, and AWS Secrets Manager, and it depends on Kiro CLI’s headless authentication to run without an interactive session.
In this post, you will learn how to:
Configure a Slack App with a slash command that triggers an AWS Lambda function
Authenticate Kiro CLI in a headless environment using API key-based authentication
Build and deploy a container image with Kiro CLI to Amazon Elastic Container Registry (Amazon ECR)
Deploy the full solution with AWS Serverless Application Model (AWS SAM)
Why headless authentication matters
A Slack slash command triggers a webhook. The webhook invokes a Lambda function. The Lambda function runs Kiro CLI. At no point in this chain is there a browser, a terminal, or a human session.
Without headless authentication, this architecture does not work. Kiro CLI would require an interactive login, and a Lambda function has no display and no way to complete an OAuth flow.
kiro-cli chat --no-interactive "analyze auth-service for memory leaks"
The API key is stored in AWS Secrets Manager, fetched at runtime, and injected into the Lambda environment. The engineer in Slack never sees or manages the key.
Important: API key-based authentication is available for Kiro Pro, Pro+, and Power subscribers. If your subscription is managed by an administrator, your Kiro admin must enable API key authentication first. For details, see API key governance.
Architecture overview
The solution consists of two Lambda functions, an API Gateway endpoint, and AWS Secrets Manager. The request and response follow two separate paths:
Request path: Slack → API Gateway → Dispatcher Lambda → acknowledge back to Slack (under 3 seconds), then async invoke → Worker Lambda
Response path: Worker Lambda → Slack response_url (direct HTTPS POST, bypasses API Gateway)
Why two Lambda functions?
Slack requires a response within 3 seconds of a slash command. Kiro CLI analysis takes 10–60 seconds depending on the repository size and prompt complexity. The Dispatcher acknowledges the command immediately and invokes the Worker asynchronously. The Worker runs Kiro CLI and posts results back to Slack through the response_url provided in the original payload. This is a standard pattern for Slack integrations that perform long-running work.
A note on response_url limits: the webhook Slack provides in the slash command payload expires 30 minutes after the command is issued and accepts a maximum of 5 responses. The 10-minute Worker timeout and single response in this solution stay well inside both limits. If you raise the Lambda timeout beyond 30 minutes or add incremental progress updates, these POSTs begin to fail silently – switch to chat.postMessage with a bot token at that point.
Prerequisites
Before you begin, you need the following:
An AWS account with permissions to create Lambda functions, API Gateway, Amazon ECR repositories, Secrets Manager secrets, and IAM roles
An infrastructure-as-code tool for deploying serverless resources (this post uses AWS SAM CLI, but you can adapt the templates to AWS CDK, AWS CloudFormation, Terraform, or your preferred tool)
Finch or Docker installed for building container images
A Slack workspace where you have permission to create a Slack App
A Kiro Pro, Pro+, or Power subscription with API key authentication enabled
Step 1: Gather credentials
You need three credentials before deploying. Collect all of them first, then store them in Secrets Manager in Step 2.
Kiro API key
This authenticates Kiro CLI in headless mode.
Sign in to app.kiro.dev
Navigate to API Keys
Create a new key named kiro-chatops
Copy the key (starts with ksk_) – it is shown only once
Slack Signing Secret – This allows the Dispatcher to verify that incoming requests originate from Slack.
On the Basic Information page, scroll to App Credentials
Copy the Signing Secret (32-character hex string)
Slack Bot Token – Optional
The Worker posts results using the response_url from the original slash command payload, which is a pre-authenticated webhook that does not require a bot token. Collect a bot token with the chat:write scope only if you extend the solution to post messages independently of a slash command response.
In your Slack App settings, go to OAuth & Permissions
Add the Bot Token Scope: chat:write
Choose Install to Workspace and authorize
Copy the Bot User OAuth Token (starts with xoxb-)
While you are in the Slack App settings, also configure the slash command:
Go to Slash Commands → Create New Command
Set Command to /kiro
Set Request URL to https://placeholder (update after deployment in Step 6)
Set Short Description to Run Kiro-CLI development tasks
Set Usage Hint to [analyze|review|debug|explain] <description>
Step 2: Store secrets in AWS Secrets Manager
Store each credential as a separate secret. The Lambda functions retrieve these at runtime using IAM-scoped access.
If your target repository is private, also store a GitHub Personal Access Token with repo scope. The Worker uses this token to clone the repository inside the Lambda execution environment.
The Dispatcher has three responsibilities: verify that the request came from Slack, acknowledge the slash command within 3 seconds, and invoke the Worker asynchronously.
Request verification – Slack signs every request with HMAC-SHA256 using your app’s signing secret. The Dispatcher must validate this signature before processing payloads. The verification logic constructs a base string from the request timestamp and body, computes the HMAC, and compares it to the signature in the request header:
Reject any request with a timestamp older than 5 minutes to prevent replay attacks. Normalize request headers to lowercase before reading them—API Gateway may preserve the original casing from the client.
Async handoff
After verifying the request, parse the slash command payload to extract text, user_name, and response_url. Then invoke the Worker Lambda with InvocationType="Event" (fire-and-forget) and immediately return an acknowledgment to Slack:
If the user sends /kiro with no arguments, return an ephemeral usage message with examples. The Dispatcher uses the standard Python 3.12 Lambda runtime and requires no container image.
Step 4: Build the Worker Lambda container image
The Worker runs Kiro CLI against a cloned repository and posts results to Slack. Because Kiro CLI depends on git, system libraries (NSS, X11, ALSA), and a binary that exceeds Lambda’s 250 MB layer limit, package the Worker as a container image.
Dockerfile structure
Start from the AWS Lambda Python 3.12 base image. Install git and the shared libraries that Kiro CLI requires, then install Kiro CLI itself:
Two details matter here. First, copy the Kiro CLI binary to /usr/local/bin/ rather than leaving it in /root/.local/bin/—Lambda runs as a non-root user that cannot access /root/. Second, build with --platform linux/amd64 regardless of your local architecture, because Lambda defaults to x86_64.
Worker logic – The handler performs four steps:
Fetch the Kiro API key (and optionally a Git token) from Secrets Manager
Clone the repository to /tmp/repo using git clone --depth 1
Run kiro-cli chat --no-interactive "<prompt>" with KIRO_API_KEY and HOME=/tmp set in the environment
Post the output to Slack via the response_url
Setting HOME=/tmp is required because Kiro CLI writes a session database, and Lambda’s filesystem is read-only except for /tmp. Strip ANSI escape codes from the output before posting—Kiro CLI emits terminal colors that render as garbage in Slack.
The subprocess timeout should be shorter than the Lambda timeout to allow time for error handling and the Slack POST. Set the subprocess timeout explicitly to 540 seconds in the Worker code, rather than relying on the Lambda timeout alone. A 9-minute subprocess limit with a 10-minute Lambda timeout provides a 1-minute buffer.
Truncate output to 3,800 characters before posting. Slack’s message limit is 4,000 characters per block, and the surrounding formatting consumes part of that space.
Build and push to Amazon ECR
Clean up /tmp/repo at the end of every invocation. Lambda may reuse a warm execution environment, so anything left in /tmp persists into the next invocation. Removing the clone in a finally block helps prevent one user’s repository from leaking into a later request and keeps the 512 MB ephemeral storage from filling up across warm invocations.
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
AWS_REGION=us-east-1
# Create the ECR repository (first time only)
aws ecr create-repository --repository-name kiro-worker --region $AWS_REGION
# Authenticate to ECR
aws ecr get-login-password --region $AWS_REGION | \
finch login --username AWS --password-stdin \
$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
# Build for the correct architecture
cd worker
finch build --platform linux/amd64 -t kiro-worker:latest .
# Tag and push
finch tag kiro-worker:latest \
$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/kiro-worker:latest
finch push \
$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/kiro-worker:latest
Step 5: Deploy with AWS SAM
The SAM template defines both Lambda functions, the API Gateway endpoint, and the IAM policies. The Dispatcher uses a standard Python runtime. The Worker references the container image you pushed to Amazon ECR.
Key resource configuration:
Resource
Runtime
Timeout
Memory
Package type
Dispatcher
Python 3.12
10 s
256 MB
Zip
Worker
Container
600 s (10 min)
1024 MB
Image
Both functions use AWSSecretsManagerGetSecretValuePolicy scoped to the kiro-chatops/* secret prefix. The Dispatcher also gets LambdaInvokePolicy for the Worker function. Neither function has broader AWS permissions.
The SAM template accepts the ECR image URI as a parameter:
Parameters:
EcrImageUri:
Type: String
Description: ECR image URI for the Worker Lambda
Resources:
WorkerFunction:
Type: AWS::Serverless::Function
Properties:
PackageType: Image
ImageUri: !Ref EcrImageUri
Timeout: 600
MemorySize: 1024
Deploy:
cd .. # Back to the project root where template.yaml lives
sam build
sam deploy --guided \
--stack-name kiro-chatops \
--parameter-overrides \
EcrImageUri=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/kiro-worker:latest
SAM prompts you to confirm IAM role creation and acknowledge that the Dispatcher has no authentication (request verification happens in code via the Slack signing secret). After deployment completes, note the ApiEndpoint output value.
Step 6: Connect Slack to the endpoint
Go to api.slack.com/apps and select your Kiro Agent app
Navigate to Slash Commands and edit /kiro
Replace the Request URL with the ApiEndpoint value from the SAM deployment output
Choose Save
Step 7: Test the integration
Test directly from Slack by typing in any channel where the app is installed:
/kiro analyze auth-service for memory leaks
Expected behavior:
Slack immediately displays: “@yourname requested: analyze auth-service for memory leaks – Kiro is working on it…”
After 15–60 seconds, the analysis results appear in the channel
You can also invoke the Worker Lambda directly for testing without Slack:
Once deployed, the value comes from the commands your team uses daily. These patterns map to real engineering workflows:
Category
Example command
Code analysis
/kiro analyze the payment module for error handling gaps
Code review
/kiro review the last 3 commits on main for breaking changes
Debugging
/kiro debug why the integration tests are failing
Knowledge
/kiro explain how the authentication middleware works
Sprint support
/kiro summarize all changes merged to main this week
The value compounds when results are visible to the entire channel. A junior engineer who might hesitate to open a CLI tool can type /kiro explain and get the same analysis and the rest of the team learns from it.
Extending the pattern
Multi-repository support – The basic implementation targets a single preconfigured repository. To support multiple repositories, parse a URL from the slash command text and clone it at runtime. This adds 5-15 seconds of latency and requires a Git token in Secrets Manager for private repositories.
Threaded responses – Post the acknowledgment as a channel message and the full results as a thread reply. This keeps the channel readable while preserving context for long analyses.
Approval workflows – For commands that modify code (for example, “create a PR that fixes this issue”), add a confirmation step. The Worker posts proposed changes with interactive buttons; the action executes only after explicit approval.
Audit logging – Log every invocation to Amazon DynamoDB: who ran it, what they asked, how long it took. This gives engineering leadership visibility into how the team uses AI-assisted development.
Constraints and trade-offs
Constraints:
Execution time – Lambda has a maximum 15-minute timeout. Complex analyses that exceed this will time out. The Worker is set to a 10-minute timeout with a 9-minute subprocess limit.
Ephemeral storage – The /tmp volume defaults to 512 MB. A shallow clone (–depth 1) strips Git history, but the working tree alone can exceed this for large monorepos or repositories with binary assets. You can increase ephemeral storage up to 10 GB by setting EphemeralStorage in the SAM template, or scope the clone to a subdirectory with –sparse-checkout for oversized repositories.
Slack message size – Each Block Kit text block is limited to 3,000 characters. Long outputs are truncated, with full results available in Amazon CloudWatch Logs.
Package size – Kiro CLI with its dependencies exceeds Lambda’s 250 MB layer limit. A container image (up to 10 GB) is required.
Trade-offs:
Lambda vs. Amazon ECS on AWS Fargate – Lambda is simpler and cheaper at the low-volume, bursty usage typical of a single team. Model your own break-even point with the AWS Pricing Calculator, since it shifts with average analysis duration and memory size. For high-volume teams, Fargate with a persistent container avoids cold starts. Start with Lambda and migrate if usage grows.
Public channel vs. ephemeral – Results are posted as in_channel (visible to everyone). For sensitive analyses, change response_type to ephemeral. Consider making this configurable per command.
Cost – Lambda compute is approximately $0.01-$0.05 per 10-minute execution at 1024 MB. The primary cost factor is Kiro CLI usage based on your subscription tier.
Security considerations
Request verification — The Dispatcher validates every request using HMAC-SHA256 with the Slack signing secret. Requests with timestamps older than 5 minutes are rejected.
Secrets management — Credentials are never hardcoded or stored in environment variables. They are fetched at runtime from Secrets Manager with IAM-scoped access.
Least-privilege IAM — The Dispatcher can only invoke the Worker and read secrets. The Worker can only read secrets. Neither has broader AWS permissions.
Audit trail — CloudWatch Logs capture every invocation including the command text, user, and Kiro CLI output. Enable AWS CloudTrail for API Gateway to track all incoming requests.
Cleaning up
To avoid ongoing charges, remove all resources when you are done testing:
To remove the Slack App, go to api.slack.com/apps, select Kiro Agent, and click Delete App.
Conclusion
This post demonstrated integrating Kiro CLI into Slack workflows using headless authentication, serverless functions, and secure credential management. The Dispatcher acknowledges instantly, the Worker runs Kiro CLI headless, and results appear in the channel where the team already communicates.
The architecture is deliberately simple – a slash command, an async handoff, and a container that runs a CLI tool. You can extend it with multi-repo support, threaded responses, or approval workflows as your team’s usage patterns emerge.
Start with a single slash command in one channel. The commands your team uses most will tell you where the friction was hiding.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
