Transforming search at Delivery Hero: A migration journey to OpenSearch Service with radial search

Post Syndicated from Sayan Das original https://aws.amazon.com/blogs/big-data/transforming-search-at-delivery-hero-a-migration-journey-to-opensearch-service-with-radial-search/

Have you ever searched for something like “low fat yogurt” at any online grocery store and noticed how the results seem to understand what you mean? Instead of only showing items with an exact match, the top-ranked products are often semantically related. You might see items like “Greek yogurt” or “yogurt with 0.5% fat,” even when only one word matches lexically. This is the power of semantic search, and when combined with traditional lexical search, it creates a hybrid search experience that delivers both precision and recall.

Semantic search returning products semantically related to a low fat yogurt query

At Delivery Hero, one of the world’s leading online food delivery platforms, the search team has been using semantic search for grocery verticals since 2024. What started as a proof-of-concept has evolved into a production-grade hybrid search system powered by Amazon OpenSearch Service. This system combines radial vector search with lexical retrieval to deliver highly relevant product results at scale.

In this post, we walk through how Delivery Hero migrated their semantic search infrastructure to Amazon OpenSearch Service, why they chose radial search over traditional k-nearest neighbor (k-NN) search, and the optimizations that made the system fast, cost-effective, and flexible for experimentation.

Legacy system overview

The original semantic search system was built as a standalone service using SpringBoot and Apache Lucene 9.9, deployed on Kubernetes. The retrieval flow worked as follows:

  1. A user starts a search on the application.
  2. The semantic search system retrieves the top 50 nearest-neighbor candidates from a static in-memory Lucene index.
  3. These candidates passed through a filtering layer to remove out-of-stock items.
  4. The filtered semantic results were merged with a parallel set of lexical search results.
  5. A final ranking step combined both candidate sets to produce the response.

The team iterated on this system over seven versions and conducted multiple A/B tests to refine the approach. The initial system performed well, however as the business scaled, several pain points emerged:

  • Scalability limitations: Running vector indices as static, in-memory structures inside Kubernetes pods meant that scaling required provisioning larger pods or adding replicas. Both options were expensive and operationally complex.
  • Multi-model experimentation was difficult: Running A/B/C tests with three different product embedding model variants required fitting all models within a Kubernetes stateless workload. This created memory pressure and complicated deployment pipelines.
  • Operational overhead: Managing index builds, deployments, and version rollouts for a custom Lucene-based service required significant engineering effort compared to a managed service.

Architecture modernization with OpenSearch Service

By the end of 2025, Delivery Hero had migrated their entire search infrastructure from self-managed Elasticsearch 7.x on Google Kubernetes Engine (GKE) to the fully managed Amazon OpenSearch Service 3.x. This migration created a natural opportunity to consolidate the legacy semantic search service into OpenSearch as well.

The new architecture separates concerns into two distinct pipelines: an ingestion pipeline for indexing product embeddings, and an inference pipeline for real-time hybrid retrieval.

Ingestion pipeline

For the ingestion pipeline, Delivery Hero chose Amazon OpenSearch Ingestion (OSIS) to sync product embedding data from Amazon Simple Storage Service (Amazon S3) to the OpenSearch domain.

Ingestion pipeline syncing product embeddings from Amazon S3 to Amazon OpenSearch Service through OpenSearch Ingestion

The flow works as follows:

  1. ML model
  2. Airflow job: An existing Apache Airflow job periodically generates product embeddings using an external machine learning (ML) model and periodically dumps the results (product parent ID + embedding vector) to an S3 bucket.
  3. OpenSearch Ingestion pipeline: An OpenSearch Ingestion pipeline is configured with a scheduled S3 scan that performs a nightly scan from S3 and updates the new k-NN index in OpenSearch Service.
version: '2'
embedding-pipeline:
  source:
    s3:
      acknowledgments: true
      scan:
        buckets:
          - bucket:
              name: my-bucket-name
              filter:
                include_prefix:
                  - vector-search/json-index/latest
        range: PT24H
        scheduling:
          interval: PT24H
      aws:
        region: eu-central-1
        sts_role_arn: arn:aws:iam::<aws-account-id>:role/osis-pipeline-role
      codec:
        ndjson: {}
      compression: none
  workers: '1'
  sink:
    - opensearch:
        hosts:
          - "https://<search-domain>.<aws-region>.es.amazonaws.com"
        aws:
          serverless: false
          region: eu-central-1
          sts_role_arn: arn:aws:iam::<aws-account-id>:role/search-xxx
        index_type: custom
        index: emb_products_v1
        template_content: ...
        template_type: index-template
        routing: '${global_entity_id}'
        document_id: '${global_entity_id}:${master_code}'
        max_retries: '3'

Because the index stores product parent IDs and embeddings are regenerated in batch, there is no need for real-time updates. This allows the team to refresh and force-merge the index once per day, resulting in highly optimized segment structures and fast retrieval speeds (p99 < 35 ms during peak hours).

Setting up the OSIS pipeline required only a few lines of Terraform, making it straightforward to provision and maintain as infrastructure-as-code.

Inference pipeline

On the retrieval side, the system runs a hybrid search strategy that combines radial vector search with lexical search in parallel:

Hybrid inference pipeline running radial vector search and lexical search in parallel before merging and re-ranking results

  1. Query embedding: A user’s search query first reaches the Query Understanding (QU) service, where it is encoded into an embedding using the same live ML model employed for product embeddings. To optimize performance, embeddings for top queries are cached.
  2. Parallel lexical and semantic retrieval:
    • A radial k-NN search runs against the product embeddings index using min_score to retrieve all semantically similar products above a similarity threshold.
    • A lexical BM25 search runs against the product catalog index.

      Chart comparing p95 OpenSearch take-time for lexical and semantic search

      Comparing p95 OpenSearch time for both lexical and semantic search.

  1. ID resolution and inventory filter: Because the k-NN index stores product parent IDs, a resolution step maps these to individual product IDs via a secondary index that maintains near real-time inventory updates. This approach satisfies two key business requirements within a single retrieval call: product-id resolution and real-time availability filtering.
  2. Merge and re-rank: A custom post-processing step combines results from both lexical and radial search, applies re-ranking logic, and returns the final result set.

Traditional k-NN search in OpenSearch uses a top-k approach: you ask for the k nearest neighbors, and you get exactly k results regardless of how similar they actually are. This works well for many use cases, but it has a fundamental limitation for product search. It always returns a fixed number of results, even when some of those results are not semantically relevant.

Radial search solves this by flipping the paradigm. Instead of asking “give me the 50 closest items,” you ask “give me all items that are at least this similar.” This is done using the min_score parameter in the k-NN query:

GET product-embeddings/_search
{
  "query": {
    "knn": {
      "embedding": {
        "vector": [0.12, 0.45, 0.78, ...],
        "min_score": 0.72
      }
    }
  }
}

When using radial search with cosine similarity as the space type, OpenSearch normalizes scores using the related formula (score = (1 + cosine_similarity) / 2), as documented in the OpenSearch knn-spaces reference.

This means a min_score of 0.72 in the query example, does not directly correspond to cosine similarity. Instead, 0.72 is the normalized OpenSearch score which translates to 44% cosine similarity (that is, cosine_similarity = 2 × 0.72 – 1 = 0.44).

If you need results with at least 90% cosine similarity, apply the formula:

min_score = (1 + 0.90) / 2 = 0.95. So, you would set “min_score”: 0.95 in your query.

This approach offers several advantages for product search:

  • Quality over quantity: Low-relevance results are excluded at the retrieval stage rather than relying on downstream re-ranking to filter them out.
  • Variable result set size: The system naturally adapts to query specificity. Niche queries return fewer, more precise results. Broad queries return more candidates for the re-ranker to work with. For example, a highly specific query like “Oatly oat milk barista edition” might return 5 results, while a broader query like “milk” might return 200.
  • Better recall-precision trade-off: By tuning the min_score threshold, the team can directly control the balance between returning too many irrelevant results and missing relevant ones.

Choosing the right min_score threshold is important. Set it too high and you miss relevant products. Set it too low and you flood the re-ranker with noise.

Delivery Hero approaches threshold selection through systematic experimentation. To achieve optimal precision across diverse markets, a tailored min_score threshold is assigned to each country and query type. These thresholds are meticulously determined through rigorous offline evaluations, which use historical user interaction and manually labeled data to establish a rough estimate. This initial estimate is then further refined and validated through a series of live A/B experiments.

Evaluation of the new search system

One of the key advantages of the new architecture is how naturally it supports experimentation. At Delivery Hero, we store three variants of product embeddings within a single document:

PUT product-embeddings/_doc/1?routing=FP_DE
{
  "master_product_code": "abc123",
  "embedding_variant_1": [0.12, 0.45, 0.78, ...],
  "embedding_variant_2": [0.21, 0.4, 0.98, ...],
  "embedding_variant_3": [0.13, 0.65, 0.58, ...],
  "global_entity_id": "FP_DE"
}

In this example, embedding_variant_1, embedding_variant_2, and embedding_variant_3 are generated from three different models for A/B/C testing. After each test, the winning variant is designated as the control, while the other two are replaced with new models for further experimentation. With this approach, the team can iterate continuously while maintaining constant space complexity.

Optimizations of large scale production system

Engine upgrade: OpenSearch 2.17 to 3.3

Production k-NN query latency metrics from one of the busiest countries after the OpenSearch 3.3 upgrade

Production metrics from one of the busiest countries.

OpenSearch 3.x introduced significant performance improvements for vector search workloads. Post-upgrade to OpenSearch 3.3, we observed a ~18% reduction in p95 latency for k-NN queries.

For Delivery Hero’s use case, the k-NN search latency was already very low on OpenSearch 2.17 (p99 of 20–30 ms), which meant the upgrade to 3.3 was not strictly necessary for all clusters. The cluster serving the control group in A/B tests still runs on OpenSearch 2.17.

Shard routing

To minimize cross-shard overhead during k-NN queries, Delivery Hero implemented custom shard routing based on geographic market. Because each market (for example, Germany, Sweden, and Finland) has its own product catalog, routing queries to market-specific shards avoids unnecessary fan-out across the entire index.

This is an example of how to configure routing at index time and search time using the _routing field:

PUT product-embeddings/_doc/1?routing=FP_DE
{
  "master_product_code": "abc123",
  "embedding_variant_1": [0.12, 0.45, 0.78, ...],
  "embedding_variant_2": [0.21, 0.4, 0.98, ...],
  "embedding_variant_3": [0.13, 0.65, 0.58, ...],
  "global_entity_id": "FP_DE"
}

And at query time:

GET product-embeddings/_search?routing=FP_DE
{
  "query": {
    "knn": {
      "embedding_variant_2": {
        "vector": [0.12, 0.45, 0.78, ...],
        "min_score": 0.72
      }
    }
  }
}

This ensures that a query for the German market only hits shards containing German products, reducing latency and compute overhead.

Refresh interval

Because the product embedding index is updated only once per day via the OSIS batch pipeline, there is no need for the default 1-second refresh interval. Delivery Hero configured the index with a longer refresh interval during ingestion and triggers a manual refresh + force merge after the nightly batch completes.

Impact on the business

The migration from self-managed Lucene on Kubernetes to Amazon OpenSearch Service achieved a ~50% reduction in p95 latency, dropping response times from a variable 200ms+ to a stable 100ms baseline. This transition significantly improved system consistency by eliminating the high variance and rhythmic latency spikes seen in the previous architecture.

End-to-end service latency dropping to a stable 100 ms baseline after rolling out semantic search on OpenSearch for foodpanda and yemeksepeti

End service latency after rolling out semantic search with OpenSearch for foodpanda and yemeksepeti.

Beyond raw latency, the operational benefits were significant:

  • Reduced infrastructure complexity: Eliminating the standalone Lucene service removed an entire deployment pipeline, monitoring stack, and on-call rotation.
  • Faster experimentation: New embedding models can be tested by creating a new index and adjusting query routing, without requiring code deployments.
  • Cost efficiency: Using OpenSearch’s managed infrastructure and the batch ingestion pattern (refresh once per day) reduced compute costs compared to running always-on Kubernetes pods with in-memory indices.

Conclusion

By combining radial search with lexical retrieval, Delivery Hero’s team built a system that adapts dynamically to query intent. It returns precise results for specific queries and broader candidate sets for general ones.

The migration to Amazon OpenSearch Service demonstrates how a managed search platform can simplify the operational complexity of vector search while improving performance.

To get started with vector search on Amazon OpenSearch Service, see the AI search documentation and the OpenSearch radial search guide.

About the authors

Sayan Das

Sayan Das

Sayan is Staff Software Engineer at Delivery Hero specializing in high-performance search infrastructure and large-scale distributed systems. With a deep background in Big Data engineering and core search internals (Solr, Lucene, OpenSearch)

Hajer Bouafif

Hajer Bouafif

Hajer is a senior solutions architect in Data Analytics and ML search with a background in Big Data engineering. Hajer provides organizations with best practices and well-architected reviews to build large-scale Machine Learning search solutions

Computer Backup vs. Cloud Storage: Which Do You Need?

Post Syndicated from Kari Wilson original https://www.backblaze.com/blog/computer-backup-vs-cloud-storage-which-do-you-need/

An illustration of a bar chart, stacked blocks and computer screens with the Backblaze flame logo.

Organizations rarely struggle with a lack of storage options. More often, they struggle with determining which solution best fits the way their data is created, accessed, and protected: backup versus cloud storage.

That’s especially true when evaluating backup and cloud storage solutions.

The terms are often used interchangeably, but backup and cloud storage are designed to solve different problems. Understanding those differences can help you build a more effective data protection strategy—whether you’re protecting a personal laptop, a growing media archive, employee endpoints, or critical business data.

At Backblaze, Computer Backup and B2 Cloud Storage serve distinct purposes. For some customers, one solution is the clear choice. For others, the strongest approach combines both.

Before comparing features, it’s helpful to start with a few foundational questions.

Three questions to ask before choosing a solution

When evaluating Computer Backup and B2 Cloud Storage, consider:

  1. Where does your data live today?
  2. Who—or what—needs access to it?
  3. What event are you trying to recover from?

The answers often reveal whether you’re primarily trying to protect a computer, store data in the cloud, or address both needs at the same time.

When the goal is protecting a computer

For many individuals and businesses, the most important data still lives on laptops, desktops, and attached external drives.

A photographer may keep active projects on a workstation. A consultant may store client files locally. A small business may rely on employee laptops as the primary location where work is created and managed.

In these situations, the primary concern isn’t cloud infrastructure. It’s protecting the device where the work happens.

That’s where Backblaze Computer Backup fits.

Computer Backup is designed to automatically protect data stored on a Mac or Windows computer, including connected external hard drives (but not NAS devices). Once installed, it runs continuously in the background, backing up files without requiring users to manually manage folders, storage allocations, or backup schedules. For organizations looking to protect NAS data, B2 Cloud Storage can serve as a backup destination through a variety of supported third-party backup and sync tools. 

The value becomes clear when something goes wrong:

  • A laptop is stolen.
  • A hard drive fails.
  • Files are accidentally deleted.
  • A ransomware attack impacts local data.
  • A computer needs to be restored after a hardware issue.

In each case, the goal is recovery.

Computer Backup is often a good fit when:

  • Your most important data lives on a computer.
  • You want automatic, continuous protection.
  • You need to recover from device loss, hardware failure, or accidental deletion.
  • You want a solution that requires minimal administration.
  • Your primary concern is protecting endpoints.

For many professionals, families, and small businesses, those requirements align closely with their day-to-day reality.

When the goal is storing and managing data in the cloud

As organizations grow, data often becomes less tied to individual devices.

Files are shared across teams. Backup software protects servers and NAS devices. Applications generate and consume data continuously. Data needs to remain accessible and manageable independent of the original device, whether that’s for long-term retention, team access, application workflows, or infrastructure backups. 

At that point, the challenge shifts from protecting a computer to managing data itself.

That’s where Backblaze B2 Cloud Storage comes in.

Unlike endpoint backup, cloud object storage is designed to store data independently of any single device. Data can be uploaded, accessed, managed, shared, and integrated into workflows across users, systems, and applications.

Organizations use B2 Cloud Storage for a wide range of use cases, including:

In these environments, accessibility, scalability, and integration often matter just as much as protection.

B2 Cloud Storage is often a good fit when:

  • Data needs to exist independently of a specific computer.
  • Multiple users or systems require access.
  • You need API-based access and automation.
  • You use third-party backup software that requires cloud object storage.
  • You need centralized storage for growing datasets.
  • You are building applications or data-driven workflows.

The focus isn’t on protecting a device. It’s on providing a durable, accessible home for data.

Understanding the data lifecycle

One reason organizations often use both backup and cloud storage is that data requirements change over time.

Consider a video production team.

While a project is actively being edited, the files may live on a workstation and several external drives. During that phase, protecting the editing environment is critical.

Once the project is complete, however, the priorities often change. The team may need to retain the content for future revisions, client requests, or compliance purposes. The files are no longer active, but they still need to remain available.

The same pattern appears across industries.

Architectural firms retain project files after construction is complete. Marketing teams archive campaign assets. Businesses preserve records for operational or regulatory reasons.

Not all data serves the same purpose throughout its lifecycle.

Active data often benefits from continuous endpoint protection, particularly when it lives on laptops, workstations, or attached drives. As that data ages, becomes shared across teams, or moves into long-term retention, cloud storage often becomes a more appropriate solution.

This is one reason many organizations use both Computer Backup and B2 Cloud Storage. The two solutions address different stages of the data lifecycle rather than competing for the same role.

When your storage requirements change

A common misconception is that organizations eventually “graduate” from backup to cloud storage. In reality, most environments become more complex over time, adding new requirements rather than replacing existing ones. As data volumes grow, teams collaborate across more systems, and retention needs increase, organizations often find themselves adding cloud storage to support those evolving demands. The shift isn’t typically about moving away from backup—it’s about addressing new use cases that emerge as data becomes more distributed, accessible, and valuable to the business. Common signs that additional cloud storage may make sense include: 

Your data is no longer centered around one device

When multiple people need access to the same information, storing everything on a single workstation becomes limiting.

You’re building long-term archives

Completed projects, historical records, and large media libraries often benefit from dedicated cloud storage.

You’re adding automation and integrations

Applications, backup platforms, and workflows frequently require API-accessible storage.

You’re managing more than endpoints

As NAS devices, servers, and infrastructure become part of the environment, storage requirements often extend beyond individual computers.

In these scenarios, cloud storage isn’t replacing endpoint backup. It’s addressing new requirements.

The blind spot many cloud storage users discover

The reverse scenario is also common. An organization adopts cloud storage and establishes a centralized repository for important data, only to discover that important risks still exist at the endpoint level. An employee may accidentally delete a local project folder, lose a laptop, or experience a workstation failure before files have been synchronized elsewhere. Cloud storage protects the data stored in cloud storage, but it does not automatically protect every device where work is created. This is one reason endpoint backup remains an important part of many modern data protection strategies. The risks are different, and each solution is designed to address a different recovery scenario. 

Why many organizations use both computer backup and cloud storage

One of the most persistent myths in data protection is that a single tool should solve every challenge. In practice, resilient environments are typically built in layers, with different solutions addressing different risks and recovery scenarios. Employee laptops may be protected with Computer Backup, while a NAS backs up to B2 Cloud Storage. Completed projects may be archived in the cloud while active work remains protected on local devices. Together, these layers create a more comprehensive approach to protecting data throughout its lifecycle. 

Example: Creative teams

For creative teams, active projects often live on editing workstations and attached storage where they are constantly being updated. Computer Backup helps protect that work in progress, while completed projects can be moved to B2 Cloud Storage for long-term retention, future revisions, or client requests. This approach allows teams to safeguard current work without keeping every finished project on production systems. 

Example: Growing businesses

As businesses grow, their data often becomes distributed across employee devices, shared storage, and business applications. Computer Backup can help protect employee endpoints where work is created, while B2 Cloud Storage provides a centralized location for shared assets, backups, and archives. Together, they support both day-to-day operations and longer-term data retention needs. 

Example: IT and infrastructure teams

IT teams frequently manage a mix of endpoints, servers, NAS devices, and other business systems. In these environments, B2 Cloud Storage often serves as a destination for infrastructure backups, while Computer Backup protects employee devices that may not be covered by server or storage backup workflows. Rather than competing with one another, the two solutions often work together as part of a broader data protection strategy. 

A quick comparison

Question Computer Backup B2 Cloud Storage
Is the primary goal protecting a computer? Yes No
Is it designed to protect endpoint data automatically? Yes No
Is the data primarily tied to a specific device? Yes Not necessarily
Is it designed for shared access across users, systems, or applications? No Yes
Is API access a core feature? No Yes
Can it serve as a destination for third-party backup tools? No Yes
Is the primary goal storing and managing cloud-resident data? No Yes

Choosing the right solution

The decision ultimately comes down to what you’re trying to protect and how your data is used.

If your primary concern is recovering files from a lost, stolen, damaged, or compromised computer, Computer Backup is likely the right starting point.

If you need scalable cloud storage for archives, applications, infrastructure backups, or shared datasets, B2 Cloud Storage is likely the better fit.

And if your environment includes both endpoints and cloud-resident data—as many organizations do—you may benefit from using both.

The most effective data protection strategies rarely rely on a single layer. They account for where data is created, where it lives, and how it needs to be recovered.

Understanding those requirements is often the first step toward choosing the right solution.

The post Computer Backup vs. Cloud Storage: Which Do You Need? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Another npm worm

Post Syndicated from daroc original https://lwn.net/Articles/1087108/


StepSecurity is

reporting the emergence of a new worm affecting npm packages.
The design of the worm is nothing new, but the rapidity with which it is
exploiting captured npm
packager credentials is noteworthy.

TL;DR: A self-propagating worm, which we are calling ChainDrop, is spreading rapidly through the npm ecosystem. So far 435 packages and more than 1,550 compromised versions have been flagged, starting with [email protected]. If you are using any of the packages listed below, assume your environment is compromised. We are still investigating the full scope; check back on this post for updates.

Accelerate CloudFormation development with the IaC MCP Server

Post Syndicated from Shuto Yukawa original https://aws.amazon.com/blogs/devops/accelerate-cloudformation-development-with-the-iac-mcp-server/

Organizations adopt Infrastructure as Code (IaC) to manage cloud environments reliably, repeatably, and at scale. As teams grow and infrastructure complexity increases, IaC becomes the backbone of consistent deployments, compliance enforcement, and operational agility. The developer’s experience around IaC, however, remains fragmented — engineers routinely context-switch between documentation portals, linting tools, deployment consoles, and logging systems just to complete a single deploy cycle. This friction compounds across teams: slower iteration means delayed feature releases, longer incident recovery times, and increased operational risk. When a deployment fails, diagnosing the root cause across disconnected interfaces can take longer than writing the template itself — turning a feedback loop that could take hours of manual investigation into a more streamlined process.

The AWS Infrastructure as Code (IaC) MCP Server brings AWS CloudFormation documentation search, template validation, and deployment troubleshooting into your AI assistant, so you can move through a full AWS CloudFormation development cycle without leaving the chat interface. Developing AWS CloudFormation templates often means switching between documentation pages, linters, the deployment console, and AWS CloudTrail Logs. Each context switch adds friction to the inner development loop — the tight cycle of writing, validating, deploying, and fixing infrastructure code. This fragmented workflow increases time-to-deployment, delays feedback, and reduces developer productivity, particularly for teams managing complex, multi-resource stacks at scale.

The AWS Infrastructure as Code (IaC) Model Context Protocol (MCP) Server unifies these capabilities in one place. This post demonstrates how the IaC MCP Server tools work together in a real workflow — from authoring and validation through deployment and runtime troubleshooting — all within a single AI assistant conversation.

In this post, you can move through a complete CloudFormation development cycle using your AI assistant. You generate a template for an Amazon Simple Storage Service (Amazon S3) bucket, an AWS Lambda function, an AWS Identity and Access Management (IAM) execution role, and an Amazon CloudWatch Logs log group. You then validate, deploy, diagnose a deployment failure, and redeploy, all in a single interface.

Solution overview

The walkthrough follows four steps that map to IaC MCP Server tools:

  1. Author: Search CloudFormation documentation and generate a template
  2. Validate: Check syntax with cfn-lint and compliance with cfn-guard
  3. Deploy: Deploy the stack using a CloudFormation service role
  4. Troubleshoot: Diagnose a deployment failure using CloudTrail correlation

Figure 1 shows the four-step workflow. Steps 1, 2, and 4 run inside the IaC MCP Server, while Step 3 uses the AWS CLI directly.

Architecture diagram showing the end-to-end CloudFormation workflow. You send a prompt to your AI assistant. Inside the AI assistant, the IaC MCP Server handles Step 1 (Author using search_cloudformation_documentation), Step 2 (Validate using cfn-lint and cfn-guard), and Step 4 (Troubleshoot using stack events and CloudTrail). Step 3 (Deploy) runs outside the IaC MCP Server using the AWS CLI with a CloudFormation service role.

Figure 1. End-to-end CloudFormation workflow with the IaC MCP Server

In the prerequisites, you deploy a CloudFormation service role stack that deliberately omits the iam:PassRole permission. During the walkthrough, you use the AI assistant to generate and deploy an application stack. When CloudFormation tries to assign the Lambda execution role, the deployment fails with AccessDenied. The troubleshoot tool then correlates stack events with CloudTrail to pinpoint the root cause.

For an introduction to each IaC MCP Server tool, see Introducing the AWS Infrastructure as Code MCP Server.

Prerequisites

Before you start the walkthrough, set up your AWS account and AI assistant and deploy the service role stack that the walkthrough depends on.

To follow along, you need:

This walkthrough uses the us-east-1 Region. You can use a different Region, but make sure to use the same Region consistently across each step.

Clone the companion repository and deploy the service role stack:

git clone https://github.com/aws-samples/sample-accelerate-cloudformation-with-iac-mcp-server.git

cd sample-accelerate-cloudformation-with-iac-mcp-server

aws cloudformation deploy \
  --template-file iac-mcp-blog-role-stack.yaml \
  --stack-name iac-mcp-blog-role-stack \
  --capabilities CAPABILITY_NAMED_IAM

This role grants CloudFormation permission to create S3 buckets, Lambda functions, and CloudWatch Logs log groups, but deliberately omits iam:PassRole — you’ll diagnose this gap in Step 4.

You use the --capabilities CAPABILITY_NAMED_IAM flag to acknowledge that the stack creates IAM resources with custom names.

We provide this role template for demonstration purposes only and do not intend it for production use.

Note the role ARN from the stack outputs. You must use this ARN in Step 3:

aws cloudformation describe-stacks \
  --stack-name iac-mcp-blog-role-stack \
  --query "Stacks[0].Outputs[?OutputKey=='ServiceRoleArn'].OutputValue" \
  --output text

Walkthrough

The four steps that follow map to IaC MCP Server tools: authoring with documentation search, validating with cfn-lint and cfn-guard, deploying with a CloudFormation service role, and troubleshooting with CloudTrail correlation.

Step 1: Generate a CloudFormation template

Start by asking your AI assistant to search CloudFormation documentation and generate a template. The IaC MCP Server calls the search_cloudformation_documentation tool behind the scenes to retrieve up-to-date resource property references.

Prompt:

Create a CloudFormation template with an S3 bucket, a Lambda function (Python 3.13 runtime, inline hello-world code), an IAM execution role for the function, and a CloudWatch Logs log group. Include common security configurations. Save it as iac-mcp-blog-app-stack.yaml in the current directory.

The AI assistant calls the search_cloudformation_documentation tool to look up resource properties for AWS::S3::Bucket, AWS::Lambda::Function, AWS::IAM::Role, and AWS::Logs::LogGroup. You can see the tool invocations in Kiro’s chat interface. The search results include up-to-date property references and example configurations, which the AI assistant uses to generate a template.

The generated template should include resources similar to the following (your output may vary):

  • An S3 bucket with versioning, encryption, and public access block
  • A Lambda function with inline Python code
  • An IAM role with a least-privilege policy for CloudWatch Logs
  • A log group with a retention policy

The following snippet shows the key resources. Your AI assistant’s output may differ in naming or structure, but the core configuration should be similar:

Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: Enabled

  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.13
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          def handler(event, context):
              return {"statusCode": 200, "body": "Hello from Lambda!"}

Step 2: Validate the template

Before deploying, ask the AI assistant to validate the template. The IaC MCP Server provides two validation tools that wrap open source checkers: cfn-lint for syntax validation and cfn-guard for policy-as-code compliance checks.

Prompt:

Validate iac-mcp-blog-app-stack.yaml for syntax errors and compliance violations.

The AI assistant runs two checks:

  1. Syntax validation (validate_cloudformation_template): Uses cfn-lint to catch structural errors, invalid property names, and schema violations.
  2. Compliance check (check_cloudformation_template_compliance): Uses cfn-guard to evaluate the template against security rules such as S3 bucket encryption, public access block settings, and log group retention.

If either check reports issues, ask the AI assistant to fix them. Continue iterating until both checks pass.

Note that the compliance check might flag violations related to S3 object lock, access logging, replication, and inline IAM policies. For a production workload, you would address each of these issues. In this walkthrough, the AI assistant resolves them to demonstrate the iterative validate-and-fix workflow. Your results might vary depending on the template the AI assistant generated in Step 1.

After the AI assistant resolves the violations, the S3 bucket resource gains access logging and object lock properties. The following snippet shows the typical shape of these additions (see iac-mcp-blog-app-stack-fixed.yaml in the companion repository for the complete hardened template):

  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      # ... existing properties ...
      LoggingConfiguration:
        DestinationBucketName: !Ref S3LoggingBucket
        LogFilePrefix: access-logs/
      ObjectLockEnabled: true
      ObjectLockConfiguration:
        ObjectLockEnabled: Enabled
        Rule:
          DefaultRetention:
            Mode: GOVERNANCE
            Days: 30

Your template now passes both cfn-lint and cfn-guard checks. These security improvements improve your template’s security posture but are unrelated to the deployment failure you’ll encounter next. The failure in Step 3 is caused by missing permission on the service role, not by anything in the template itself.

Step 3: Deploy the stack

With validation complete, deploy the template. This deployment will fail — not because of a template error, but because the CloudFormation service role deployed in the prerequisites is missing iam:PassRole. This is the scenario you’ll diagnose in Step 4.

Now deploy the validated template using the service role you created in the prerequisites:

Prompt:

Deploy iac-mcp-blog-app-stack.yaml as a stack named “iac-mcp-blog-app-stack” in us-east-1 using the service role ARN from iac-mcp-blog-role-stack.

The AI assistant runs the AWS CLI deployment command for you. If your AI assistant doesn’t support running shell commands directly, you can deploy manually with the AWS CLI:

Manual CLI deployment

ROLE_ARN=$(aws cloudformation describe-stacks \
  --stack-name iac-mcp-blog-role-stack \
  --query "Stacks[0].Outputs[?OutputKey=='ServiceRoleArn'].OutputValue" \
  --output text)

aws cloudformation deploy \
  --template-file iac-mcp-blog-app-stack.yaml \
  --stack-name iac-mcp-blog-app-stack \
  --role-arn $ROLE_ARN \
  --capabilities CAPABILITY_NAMED_IAM

The deployment fails. The stack event shows an AccessDenied error on the IAM role resource, but doesn’t identify which permission on the CloudFormation service role is missing or why. At this point, we move from static analysis to runtime troubleshooting.

Step 4: Troubleshoot the failure

Ask the AI assistant to diagnose the failure:

⚠ Note: CloudTrail events typically take 5–15 minutes to appear. Wait at least 5 minutes after the deployment failure before running the troubleshoot tool for the most complete analysis.

Prompt:

Troubleshoot the failed deployment of iac-mcp-blog-app-stack in us-east-1.

The AI assistant calls troubleshoot_cloudformation_deployment, which:

  1. Retrieves the stack events and identifies the failed resources
  2. Correlates the failure timestamps with CloudTrail API calls
  3. Identifies AccessDenied errors and the missing permissions that caused them

The troubleshoot tool identifies that the CloudFormation service role is missing iam:PassRole — the permission required to assign the Lambda execution role to the function. If your template includes the cfn-guard hardening from Step 2 (access logging, object lock), the tool may also surface additional missing S3 permissions such as s3:PutBucketObjectLockConfiguration for the logging bucket.

Prompt:

Fix iac-mcp-blog-role-stack.yaml to add the missing permissions identified by the troubleshoot tool. Save it as iac-mcp-blog-role-stack-fixed.yaml.

The AI assistant adds the missing permissions to the service role template. Now ask the AI assistant to deploy the fix, delete the failed stack, and redeploy:

Prompt:

Deploy iac-mcp-blog-role-stack-fixed.yaml to update iac-mcp-blog-role-stack, then delete the failed iac-mcp-blog-app-stack and redeploy it with the same service role.

The AI assistant runs the necessary CLI commands: updating the role stack, deleting the failed application stack, and redeploying the application stack. The failed stack is in ROLLBACK_COMPLETE state, a terminal state that CloudFormation cannot update in place, so you must delete it before redeploying.

The stack deployment succeeded.

Cost considerations

For information about costs associated with the resources in this walkthrough, including S3 storage, Lambda invocations, CloudWatch Logs, and CloudFormation operations, see AWS Pricing. Confirm that your account usage falls within any applicable free tier limits. If you enabled S3 access logging or object lock through the validation-and-fix workflow in Step 2, the logging bucket stores a small amount of access log data that falls under S3 standard pricing. See AWS Pricing for current rates and confirm that your account is within the Free Tier limits before you deploy.

Cleaning up

To avoid ongoing charges, delete both stacks.

Option A: Clean up with your AI assistant

Ask your AI assistant to run the cleanup for you. The IaC MCP Server lets the AI assistant inspect stack outputs, empty buckets, and delete both stacks in the correct order:

Clean up the iac-mcp-blog-app-stack and iac-mcp-blog-role-stack stacks in us-east-1. Empty any S3 buckets they created (including access log buckets) before deleting the application stack, then delete the role stack.

Option B: Clean up manually

Delete the application stack first because it was deployed with the service role:

⚠ Warning: If your template included access logging, the logging bucket may contain objects. CloudFormation cannot delete a non-empty bucket. Empty it first:

aws s3 rm s3://<logging-bucket-name> --recursive

Then proceed with stack deletion.

aws cloudformation delete-stack --stack-name iac-mcp-blog-app-stack
aws cloudformation wait stack-delete-complete --stack-name iac-mcp-blog-app-stack

aws cloudformation delete-stack --stack-name iac-mcp-blog-role-stack
aws cloudformation wait stack-delete-complete --stack-name iac-mcp-blog-role-stack

If any S3 bucket was created with DeletionPolicy: Retain or still contains objects (for example, server access logs), CloudFormation leaves it in place. Empty and delete those buckets from the S3 console or with aws s3 rb s3://<bucket-name> --force.

Next steps

If you manage CloudFormation infrastructure and find yourself losing time to context-switching between docs, linters, consoles, and logs, here’s how to streamline your workflow starting today:

  1. Set up the IaC MCP Server — Install and configure the IaC MCP Server with an MCP-compatible AI assistant such as Kiro to bring documentation search, validation, and troubleshooting into a single conversational interface.
  2. Run the walkthrough end-to-end — Clone the companion repository and follow this post step by step to experience the full author-validate-deploy-troubleshoot loop in your own AWS account.
  3. Integrate into your team’s workflow — Replace manual context-switching by embedding the IaC MCP Server’s tools into your day-to-day CloudFormation development process, reducing iteration time from hours to minutes.
  4. Extend to AWS CDK — Apply the same conversational workflow to CDK-based infrastructure using the IaC MCP Server’s CDK capabilities described in the introductory blog post.
  5. Contribute and share feedback — Report issues or suggest enhancements on the AWS MCP GitHub repository to help shape future capabilities.

Conclusion

In this walkthrough, you used the IaC MCP Server to move through a complete CloudFormation development cycle without leaving your AI assistant. The documentation search tool retrieved up-to-date resource property references that the AI assistant used to generate a template. The validation tools caught syntax errors and compliance gaps before deployment. When the deployment failed due to missing permissions on the service role (an issue that static analysis cannot detect), you used the troubleshoot tool to correlate stack events with CloudTrail and pinpoint the root cause in seconds.

By combining static validation with runtime diagnostics, you shorten your develop-validate-fix cycle for CloudFormation. Instead of switching between browser tabs, CLI sessions, and the CloudTrail console, you stay in one interface — turning a multi-step troubleshooting session that previously meant switching between consoles, CLI sessions, and CloudTrail into a few prompts in a single conversation.

To get started, explore the companion GitHub repository for the complete sample code. Learn more about the IaC MCP Server in the introductory blog post and the AWS CloudFormation documentation. To set up Kiro, visit kiro.dev.

About the authors

Shuto Yukawa is an Associate Delivery Consultant at AWS Professional Services. He helps customers modernize their applications and adopt cloud-native practices on AWS.

G SS Harsha Vardhan is an Associate Delivery Consultant at AWS Professional Services. He guides customers to migrate and transform their workloads to AWS, driving modernization across people, process, and technology.

[$] The beginning of a process-builder API

Post Syndicated from corbet original https://lwn.net/Articles/1086330/

The recent discussion on “spawn templates”
raised questions about whether it was time to provide an alternative to the
classic Unix fork()/exec() pattern for process creation.
One idea that was raised there was to shift the template pattern into an
interface that could be used to efficiently assemble new processes from
bare cloth, without duplicating the parent process. Preferably, that
interface would be able to implement posix_spawn().
Li Chen, the author of the spawn-template work, has now responded with a patch series
(written with significant LLM assistance) showing what a process-builder
API for Linux might look like.

Security updates for Tuesday

Post Syndicated from jzb original https://lwn.net/Articles/1087068/

Security updates have been issued by AlmaLinux (frr, ldns, mingw-glib2, and perl-Archive-Tar), Debian (ruby2.7), Fedora (borgbackup, nebula, python-nh3, rust-ammonia, and seamonkey), Mageia (librabbitmq, libvncserver, packages, perl, perl-GD, perl-Unicode-LineBreak, squid, and unbound), Oracle (compat-libtiff3, frr, gstreamer1-plugins-good, javapackages-tools:201801, libreswan, nodejs:22, nodejs:24, p11-kit, perl-Archive-Tar, perl-DBI, php, pki-deps:10.6, and python-tornado), and SUSE (aws-iam-authenticator, bind, containerd, gawk, google-cloud-sap-agent, ignition, ImageMagick, java-11-openjdk, libpng16, libssh, mcphost, nginx, openssh, openssl-1_1, perl-DBI, perl-HTTP-Date, perl-Net-DNS, python-urwid, python3-dulwich, python312, python313, python3, python313-pydantic, python313-sentry-sdk, rrdtool, s390-tools, samba, spice-vdagent, vim, and xen).

The Agent Development Lifecycle has arrived on Cloudflare

Post Syndicated from Brendan Irvine-Broque original https://blog.cloudflare.com/agent-development-lifecycle/

Engineering managers spent the past few decades figuring out ways for many programmers to work together on a shared codebase. This work dates all the way back to the “Systems Development Lifecycle” (RAND, 1975) – today commonly referred to as the “Software Development Lifecycle” (SDLC), which defines the following phases:

  • Plan
  • Design
  • Implement
  • Test
  • Deploy
  • Maintain
  • Retire

AI has made the step that was previously the slowest and most expensive — implementation — the fastest and cheapest. That, in turn, has had an impact downstream: overwhelming the people responsible for all the other steps in the SDLC. This ranges from open-source maintainers bombarded with thousands of pull requests and issues, to production engineers trying to save production from falling over as the rate of software delivery increases orders of magnitude.

We are all trying to save our systems, our customers, and ourselves from slop.

The answer — paradoxically — is to empower agents to do more. It’s only fair! You’d never let an engineer on your team write code, expect someone else to validate it, merge it, deploy it, hold the pager in production, and triage incoming bugs. But that’s what most companies are doing right now with agents. Models have improved remarkably, and agents are running over longer time horizons, able to take on much larger tasks. But they are not yet used evenly across the SDLC.

Cloudflare treats agents as our customers. They can buy domains, create temporary accounts and use the entire Cloudflare API. We know that agents need APIs and tools to be able to manage the full SDLC on behalf of our customers — not just the start of it.

And so today we’re introducing the start of a new set of tools that let agents step beyond just generating code and take on more of the SDLC. We’re sharing what we’ve built and learned trying to solve this for ourselves:

There’s something bigger here though. When we look at the SDLC, even with the best automation, its assumptions do not scale for the volume of code agents can write and the pace at which software teams must move to compete. We think it’s time to replace the SDLC with the ADLC — the Agent Development Lifecycle.

The SDLC is for software teams. The ADLC is for software factories.

Right now, everyone is talking about building “software factories” — agent-driven systems that take input and autonomously build, improve, deploy and manage software. Take an input, whether it’s a production error, a bug report from a customer, or an idea for a new feature, and delegate it entirely to an agent.

Even with agents, most software projects are constrained by human-in-the-loop steps. Humans prompting agents, telling them to keep going, instructing agents to apply feedback from a code review, constantly babysitting many agents and giving them instruction. On most software teams, the human still manages each step in the SDLC model — the only change is that they delegate tasks within each step to an agent.

And so the dream behind software factories is: what if you reimagined this approach and built a factory for the entire process of building software? How can we shift more human time towards the things that truly require human inspiration, taste, and judgement? It would leave us more time to design, to talk to customers, and to dream bigger.

A software factory has to manage the same steps in the SDLC, but it demands much more from the platform it is built on. Because when you hand over the keys and let the agent drive, every manual step that previously relied on a human must be adapted to be:

  • Programmatic — ”ClickOps” was bad practice for humans, but it’s a non-starter for agents. Every last operation needs APIs that agents can call, debug, and rely on.
  • Horizontally scalable — preview deployments were a nice-to-have when humans stared at the screen while building or manually took over a staging server to catch issues before production. For agents to drive, every agent must have its own preview that matches production.
  • Reproducible — what happens if there’s a bug that you can only reproduce when simulating 4G on an iPhone 15? Or from an IP in a certain country? Typical unit testing and integration testing tools aren’t going to help here.
  • Real-time, push based — relying on humans to look at the right dashboard has always been a bad way to know if things are working, but it completely breaks down with agents. You need an event that triggers an agent to do work.
  • Atomic — every change needs to be independently testable, releasable, observable, and reversible without affecting unrelated behavior.
  • Permissioned — you know you probably shouldn’t, but today you give a few trusted engineers the keys to SSH into prod in case things really go haywire. There’s no way you let an agent do that — but without the ability to escalate and get more permissions, how can it do its job?
  • Self-improving — people learn from experience. The first week ship or the first on-call rotation, humans are slow and need to shadow someone else, but then get better and faster. Agents, too, need ways to learn from experience.

We need something new if we are going to make software factories safe to use for real production software. Software factories face the same challenge that other autonomous systems like self-driving cars do — the challenge of going from working successfully 80% of the time, to some number of nines past 99%.

To give agents the keys to drive the SDLC, you can’t give them a car designed for humans

An autonomous vehicle is loaded with sensors and technology that a regular car doesn’t have. Lidar sensors, cameras, powerful compute to run inference, and connectivity to a central command system that can take over remotely if needed.

For an autonomous vehicle to be 80% as good as a human at driving, we probably don’t need all of this. Self-driving got to around 80% as good as humans 10 years ago. But that’s not the bar to clear — the bar is to be much better and safer than a human driver. That’s what we expect when we hand over the keys to a machine, in order to feel safe taking a nap driving down the 101 at 60 mph. And that’s why autonomous vehicles have technology that is purpose-built for self-driving — it’s what builds trust and handles the edge cases that cannot be designed for upfront.

The same is true of self-driving software. Ask yourself — why haven’t you yet just let your agent auto-approve and merge its own PRs to your production services? The higher the stakes of what you build, the longer your list of reasons almost surely is.

When you start to unpack not only all the things that can go catastrophically wrong in this process, but also that are necessary to building the right thing for customers, it is remarkably complex. It doesn’t fit into a linear set of steps in a GitHub Actions YAML file, and it goes way beyond running traditional automated tests. Even a small change to a dashboard can span roles, specializations and org structures, and subjective changes are the hardest to test and to delegate. Most of these things are probably not part of your CI/CD pipeline at all today. But they will need to be, if you want them to still happen, while giving full control to the agents running the software factory.

To let agents drive the whole process, we need a better way to orchestrate these dynamic series of steps. We think that is a Workflow, with the capability to spawn containers, agents and browsers. A Workflow that can set feature flags and enable them for a test user, investigate logs and traces, observe production metrics as a change gradually rolls out, and do everything else that is needed in order to ship safely.

A CI/CD pipeline is just a Workflow. But a Workflow can be so much more than a CI/CD pipeline.

Cloudflare Workflows let you chain together multiple steps, automatically retry failed tasks, and persist state for minutes, hours, or even weeks. They are designed to encode complex and dynamic business processes in a logical and well-understood program. This blog post breaks down why Workflows, in tandem with Artifacts, make defining and triggering CI/CD pipelines fundamentally simpler. For example:

Workflows go beyond a series of linear steps though. They can be defined dynamically, and they can spawn agents or other Workflows. This example shows a Workflow that reviews new data from the past day. The Workflow has full control over when and how the agent is prompted, and can pass along context between steps: 

Once you see this pattern, and are “Workflow-pilled” as Cloudflare is, you start to ask: what else could I have a Workflow handle for me? What other human-bottlenecked steps could I delegate to this combination of Workflow + Flue agents?

The full ADLC, on the Cloudflare stack

With Workflows able to orchestrate complex steps, and Artifacts as the storage layer for code, when you look at the SDLC stages, everything an agent needs to own the whole process of building, shipping, and maintaining software is on Cloudflare:

Primitives to build your software factory

Right now, the people on the bleeding edge are building the software factories of the future. Eventually software factories will become, just like agents and AI, the normal way people build software. But for most people and most organizations, we’re not there yet.

We want to change that.

In order to do so, the questions we’ve asked ourselves are: how can we make things simple and accessible so that everyone on the Internet can benefit from a paradigm shift like this? And what are the base layer primitives that we can open up to everyone, from the smallest startup to the largest platforms in the world?

In this case, we think the primitives are here. There’s more to do to connect them, to keep building our own software factory and learn from it, but right now, today, we’re ready for you to build your machine that builds the machine, on Cloudflare. Get started with @cloudflare/ci, build an agent, and see how much of the SDLC you can make autonomous.

Run CI/CD for millions of repos — on your platform, on Cloudflare

Post Syndicated from André Venceslau original https://blog.cloudflare.com/ci-workflows/

We are moving toward a world in which you can store, build, test, and deploy your code fully on Cloudflare. We built the first piece with Artifacts, versioned code storage that scales to millions of repos. 

We have stitched the store, build, and deploy steps together with the CI SDK, built on Cloudflare Workflows, so that you can run your continuous integration (CI) pipeline on Cloudflare. You can send artifact push events directly to your Workflow, triggering an instance of its execution — a CI job, essentially — through a new events field in your wrangler configuration file. 

Then, directly from the Workflow with @cloudflare/ci installed, you can:

  • Automate builds: compile code from your Artifacts repo in a safe, isolated environment 
  • Run linters and typechecks: enforce code style, catch type errors, and flag any potential issues
  • Cache dependencies: run your install once and cache dependencies across steps in the CI job
  • Execute unit tests: verify that each piece of your code works as expected
  • Self-heal: integrate an AI review agent to catch broken steps in your build and push commits to fix 
  • Deploy conditionally: automatically deploy your code, only if your build step is successful

Today, everyone is building a platform, whether it’s an internal vibe coding platform or an extension of your customer-facing product via customization through code. Platforms are now using millions of repos on Artifacts to store their code, and their customers’ code, and version control across the two. But every team has their own needs for a continuous integration and deployment pipeline. For platforms, they might want to define a CI job for their own code differently from that of their customers. 

Many of the end customers building on these platforms don’t want the extra headache of managing their continuous integration and continuous deployment (CI/CD) pipeline. Instead, the platform can manage the build process on their customers’ behalf: write the CI/CD pipeline once and share it across all the applications that their customers are building. Some of the platform’s customers might want to define their own CI; if so, they can write their own Workflow and run custom CI jobs on just their repo, facilitated by dynamic workflows. The beauty is, you don’t have to pick and choose: both platform-managed and custom CI can run at the same time, in the same namespace.

A CI/CD pipeline is just a Workflow

Before today, we had all the pieces to allow platforms to wire their CI/CD pipeline together on Cloudflare. Now, we’re bringing a better developer experience to make it simple. 

A CI/CD pipeline — commonly orchestrated with GitHub Actions — is a series of steps that run in a specific order where, if any step fails, you stop running the pipeline and report the error. In essence, a CI/CD pipeline is just a Workflow. CI/CD, when defined by a YAML file, can get complicated quickly, given the constraints that so often lead to YAML fatigue. But each step in a CI/CD pipeline can translate simply to a Workflow step.do(). Instead of YAML, you can define your CI/CD pipeline in Typescript for greater customization and configurability. 

We are launching new tools in the CI SDK that allow you to run each step in your CI pipeline (e.g. build, lint, and typecheck) in a safe, isolated environment, built directly on Cloudflare’s developer platform via Workflows and the Sandbox SDK. Plus, you can now kick off a CI job directly on push instead of configuring an event subscription, a queue, and a queue consumer. 

Previously, you’d have to call the Sandbox API directly and manage state yourself across different steps in the CI pipeline. The SDK allows you to run each sandboxed command in its own Workflow step, providing the retries and timeouts built into Cloudflare Workflows. 

You can also speed up your CI pipeline by caching step results — for example, your install step — so that you don’t need to reinstall for all subsequent operations. Dependency caching reduces the latency of your CI/CD pipeline since every CI step won’t need to rerun the install.

To define your CI job, all you need to do is:

  1. Define your install step for any dependencies (external packages or tools that your CI job needs), such as bundlers (e.g. esbuild), linters (e.g. eslint), or test runners (e.g. vitest).
  2. Specify the command for each step in the CI job (e.g. bun run build, bun run test, bun run lint). With your dependencies cached, each CI step can execute in parallel, reducing the latency of the overall run. 
  3. Pass wrangler deploy in a deploy step. Your Worker will automatically deploy when the CI pipeline passes.

Writing your own CI pipeline in a Workflow allows you to customize as much as you want. For example, you could call an agent from your CI Workflow to give your CI jobs self-healing functionality: if a step in your build errors, the agent can fix it automatically, and push a commit for your approval.

Try an example of self-healing CI Workflows with Project Think: https://github.com/cloudflare/ci/blob/main/examples/self-healing

Write your own CI Workflow

To write your own CI Workflow, get started with import { CIWorkflow } from@cloudflare/ci.
Start with an install step:

  • Download your dependencies, including any external tools or libraries that your CI steps will need (e.g. vite, react).
  • Specify your lockfile, which tracks whether your dependencies have changed.
  • Cache your dependencies via a sandbox snapshot so that all subsequent steps have access. The snapshot will be stored in an R2 bucket on your account.

Then define steps for the build and checks, each executed in its own safe, isolated sandbox environment.

By default, each step in a Workflow starts independently, meaning the steps will execute concurrently unless otherwise specified. Running each step in parallel reduces the latency of your CI run. To ensure that all checks complete before the CI pipeline continues (for example, finish build, lint, test, and typecheck before the deploy step starts), wrap in a Promise.all(): 

Now, to actually trigger your CI Workflow, add an events field to your Worker’s wrangler configuration, alongside your Workflow and Artifact bindings. The events field is a new field supported within your triggers field. 

You could already subscribe to Artifacts through Cloudflare Queues via event subscriptions and kick off a build pipeline every time there’s a push event. But that requires setting up the event subscription, Queue, consumer, and queue handler. Now, you can target a Workflow with that event — every time that event fires, it will trigger an instance of the Workflow. 

Specify the CI Workflow as your artifact push trigger’s target to automatically trigger a Workflow instance on every cf.artifacts.repo.pushed event. Each CI run surfaces as a Workflow instance so you can view its step-by-step execution and observability directly in the Workflows dashboard. This is an Artifacts-first integration; coming soon, the types will support events from sources across your Cloudflare account to allow for programmatic consumption across the product suite.

If you want to run the CI Workflow on every repo in your namespace — for example, if you are a platform running CI on all of your customers’ repositories — omit repoName and only specify the namespace in filter.

To fully configure your CI Workflow, add bindings to each piece of the infrastructure which powers the pipeline: artifacts, workflows, containers and durable_objects (+ exports config) bindings (to access your sandboxes), plus an r2 binding if you are using cache. The R2 binding is required as the snapshot of your install step sandbox is stored in a bucket.

Self-healing CI runs

To allow your CI job to self-heal, you’ll need two pieces: the LLM and its agent harness. In the example above, we included a Think agent using Workers AI to catch errors in your pipeline and run the fixes on your behalf. Your CI job can be run and re-run remotely — no need to watch with your laptop open or check back every few minutes. Instead, Cloudflare handles it in the cloud, running your healer agent alongside the CI steps in a container. Instead of babysitting the CI job, making a manual fix, and re-running the pipeline, you’ll just need to merge the commit after your agent has made the fix. 

To set up an agent that self-heals your CI pipeline, add a Durable Object binding for your Think agent: 

Create your Think agent — Healer — by extending the HealingAgent class, which includes a heal method for you to call on failure. Pass whichever model you’d like to use: 

Then, wrap your steps in a try/catch block where a failure triggers the healing agent:

This example demonstrates a self-healing CI pipeline, but really, the Bring Your Own Workflow model allows you to customize the CI job however you want. This can be a place to add security rules, filters, or conditional CI steps. Using the BYO-W model, platforms can configure their CI/CD pipelines across different teams, customers, or applications according to each individual use case. 

The benefits of using a Workflow

By running your CI pipeline on a Cloudflare Workflow, you automatically inherit:

  1. Resilient retries (durable execution): if any step in your CI job fails, it will automatically retry with state persisted, meaning that no progress is lost. Every step supports custom retry and timeout behavior, so you can define different failure logic for each one. Plus, you can restart from a specific step, so if just lint fails, for example, you don’t have to rerun the entire CI pipeline. 
  2. Workflows observability: inspect your CI job step-by-step in the Workflows dashboard, where each instance surfaces the steps with their inputs, outputs, and wall and CPU time. You can visualize your CI job through Workflows diagrams in the dashboard, allowing you to easily see which steps run concurrently versus sequentially. You can also inspect Workflows logs through Workers Observability and GraphQL to understand more about runs of your CI job. 

  1. The power of code: by running CI in a Workflow, you can write a step for anything you want. For example, you might want to run an AI code reviewer as part of your CI/CD pipeline. You can make a call to your code review agent — or handle any custom logic you can put into code — with Workflows step.do(). Other examples might include writing build artifacts to R2 and sending an email when CI fails, completes, or merges to main.

What’s next

A CI/CD pipeline is just a Workflow — and with the CI SDK, you can define your CI across your code, and that of your customers, in simple Typescript rather than inflexible YAML. Building off the Cloudflare Workflows primitives, you can define whatever logic you’d like, whether that’s a healing agent, like our Think example, or writing build artifacts to R2. Running CI on Workflows helps bridge the gap between storage (via Artifacts), builds, and deployments. As a platform, this allows you to easily manage each step on your own code and on behalf of your customers.

Request to join the Artifacts private beta and get started with our Workflows CI guide. If you have any feature requests or notice any bugs, share your feedback directly with the Cloudflare team by joining the Cloudflare Developers community on Discord. 

What’s coming next:

  1. Direct integrations for Workers & Workers for Platforms: build.preview() and build.deploy() primitives to automatically deploy on push to main and create previews on push to non-default branches
  2. Gradual deployments: manage percentage-based rollouts via Workflows to customize your deployment progression and rollback logic
  3. Monorepos: simplified management for multi-Worker deployments using one CI pipeline
  4. Triggers: send push events from different sources to run CI jobs on a repo from any version control system, not just Artifacts

How Cloudflare enforces engineering standards using AI

Post Syndicated from Timo Reimann original https://blog.cloudflare.com/engineering-standards-enforcement/

Over the past four months, our AI code reviewer has flagged nearly a quarter of a million deviations from Cloudflare engineering standards (what we’ll call “violations” in this post) and blocked 16,000 merges. Our spec reviewer agent has evaluated close to 600 technical designs against the same standards before implementation began. Both systems draw from the Cloudflare Codex, a shared source of engineering guidance built for people and agents. This post explains why we built the Codex, how it supports the engineering lifecycle, and what we plan to do next.

Before the Codex (which we briefly introduced in a previous post about our AI engineering stack), developer guidance at Cloudflare lived in many places: formal documentation, repository files, chat threads, and the accumulated knowledge of individual engineers. Engineers often spent too much time searching for guidance instead of working on the problem they were trying to solve. Even after finding an answer, they could not always tell whether it was current, authoritative, or applicable to their situation.

As Cloudflare grew, that model became increasingly difficult to sustain. No engineer could read every standard, and reviewers could not reliably check every requirement. Institutional knowledge became harder to recover when people moved between teams, and guidance that was not consistently surfaced or enforced led to drift between projects.

We rebuilt this body of knowledge as the Cloudflare Codex: a governed set of engineering standards that agents can retrieve and apply at the point of work. The same guidance can now inform code review, technical design review, incident report review, and many other use cases, while engineers focus their time and judgment on the resulting findings.

Codex organization and workflow

A dedicated Codex governance model divides the Codex into distinct domains covering the engineering areas we care about. These include architectural matters (for example, frontend and control plane), cross-cutting concerns (security and reliability), specific languages (TypeScript and Rust), and several other areas. Each domain is led by an owner who is responsible for the content, consistency, and overall quality of the documents they oversee.

Codex standards use a Request for Comments (RFC) format. Requirements use the SHOULD and MUST keywords defined by RFC 2119. We also expect a front matter header to hold metadata such as the domain and RFC status. Any Cloudflare employee with a key interest and domain competency can propose an RFC through a merge request that follows the prescribed structure. The proposal then passes through several rounds of feedback from an increasingly broad group of reviewers. Once the domain owner gives final approval, the RFC becomes part of the Codex and is published to an Astro-powered internal site.

Approved RFCs can be consumed by Codex clients and agents, which may then start to flag Codex violations in code, configuration, or documentation immediately. However, they block based on Codex statements only after an RFC moves from the approved to the enforced lifecycle state. This separate promotion step gives teams time to absorb new requirements and accommodates cases where enforcement needs additional work.

The following diagram illustrates the steps in the Codex workflow:

A naive process could stop here and feed the entire Codex to a large language model (LLM) as is. Given the increasing number of RFCs we have already (60+ and counting), however, the corpus volume would put a lot of stress on the context window and impact LLM results negatively. To help guide models to the most relevant RFCs, we invoke a purpose-built agent to automatically extract and compact the SHOULD and MUST statements into a dedicated JSON structure and enrich it with metadata that supports lazy discovery and progressive disclosure. The following abridged excerpt shows the result for our control plane services RFC:

Each statement receives a stable slug identifier that remains unchanged during the extraction process even when its RFC is updated. The identifier lets us track the same statement across different systems over time, which is essential for monitoring, analysis, and exception handling.

Initially, we extracted the statements into another, more concise Markdown file rather than JSON. Over time, we moved to a richer structured format so that agents could filter the content they needed more accurately. We plan to include additional metadata for even tighter scoping, such as indicators for the software development life cycle (SDLC) stage a statement applies to (e.g., design, implementation, runtime).

Codex consumers

Several systems already use the Codex in day-to-day engineering work. Three agents show how the Codex works in practice: our AI code reviewer, spec reviewer, and incident report reviewer.

AI code reviewer

Our AI code reviewer agent, covered in a separate blog post, evaluates merge requests across several dimensions, including Codex compliance.

For each review, the agent retrieves the RFCs and parses the Codex statements. It loads full RFC bodies only when the model or coordinator needs additional context. In most cases, the statements provide enough information to explain a reported violation.

The distinction between SHOULD and MUST, together with an RFC’s status, determines how the reviewer responds. Findings from approved RFCs are non-blocking recommendations. Once an RFC is enforced, an unsatisfied MUST requirement causes the reviewer to withhold approval or block a merge request, depending on the severity. 

Since the Codex’s inception earlier this year, the AI code reviewer has flagged close to 230,000 violations. Among these, almost 16,000 caused approval to be withheld (i.e., they referred to MUST statements on enforced RFCs).

Code review alternatives

A single AI code reviewer run usually takes a couple of minutes to complete due to the coordinator framework and sub-agent execution. Although the wait is very often worth the money (or tokens), engineers were calling out the delay and extra round trip involved in remediating the findings. We looked into how we could improve the experience and came up with two additional options:

  1. For language-specific Codex requirements that can be verified mechanically, we provide custom linter configuration packages. These are aligned with our Codex specification and make it possible to surface problems in milliseconds. TypeScript was the first language to receive Codex linter support while also standardizing on oxlint (maintained by the VoidZero team who joined Cloudflare recently) for performant linter execution. A linter for Rust projects is currently under development, and Go will eventually follow to complete coverage of Cloudflare’s most commonly used languages.
  2. To cut out the continuous integration (CI) leg from the review cycle, we made it possible to run the AI code reviewer locally through a command-line interface (CLI). It matches the coordinator functionality from CI and runs the same (OpenCode-based) agents against an automatically determined diff set, with results presented in the terminal.

We believe the linters would be useful to almost every developer and codebase, while the CLI remains an optional alternative for engineers who prefer it.

Spec reviewer

Engineers at Cloudflare regularly write design documents and technical specifications (or specs in short) before implementation. A significant subset of the Codex pertains to design, architecture, and other themes relevant to technical reviews. To catch architectural mistakes before implementation begins, we built the spec reviewer, an agent that discovers specs and evaluates them against relevant Codex requirements.

The spec reviewer operates on the Developer Platform: it runs as a Cloudflare Worker, stores its results and state in D1, routes model requests through AI Gateway, and kicks off scanning for new specs via a Cron Trigger. It starts by filtering the Codex by domains and sections relevant to specs (for example, language features and implementation-focused RFCs are disregarded). Several guiding prompts instruct the model on how to run the assessment and frame the results. The findings get rated based on severity (influenced by SHOULD and MUST keywords) and include general quality and architectural advice. On completion of a review run, a note is left on the spec document linking to a custom dashboard where review details can be inspected.

Since the beginning of May 2026, almost 600 unique open specs have been reviewed. Including reruns triggered on demand or by spec changes, we tracked over 3,200 review invocations to this date. The vast majority of findings had a “major” (65%) or “minor” (29%) severity, with “critical” findings being the minority (6%).

The following image gives an impression of what the spec reviewer UI looks like:

We plan to integrate the spec reviewer more tightly by posting comments directly on the spec documents, embedding human-agent conversations that can influence the review assessment, and flagging high-impact proposals for additional human review.

Incident report reviewer

The incident report reviewer applies the same approach to incident reports (also known as postmortems). In addition to checking that each report is complete, it evaluates whether the report clearly explains what happened, identifies contributing factors, documents the resolution, and proposes meaningful follow-up actions. These expectations are defined in a dedicated Codex RFC.

The incident report reviewer uses the same Developer Platform building blocks as the spec reviewer. This shared architecture is becoming a common pattern for our Codex agents.

Since May 2026, the reviewer has assessed more than 200 incident reports and identified gaps such as missing follow-up action items, incomplete timelines, and omitted detection signals. Among those reports, 93% covered incidents that were low-impact, internal-only, or declared preemptively. For high-severity incidents, we’ve made the reviewer mandatory as part of our comprehensive central review process, and reports are not considered complete until all findings have been addressed.

Future work

The Codex already supports agents that review code, technical designs, and incident reports. We plan to extend that model throughout the SDLC, allowing agents to surface issues consistently across design, implementation, and operations. The longer-term goal is for agents to identify issues as well as propose fixes with increasing autonomy, while engineers remain responsible for reviewing and approving those changes.

We are also expanding the Codex beyond engineering. Product, security, compliance, and trust and safety teams are beginning to add their own standards, allowing agents to evaluate work against considerations that extend beyond design and implementation alone.

Across a number of engineering workflows, Codex-backed agents have helped us surface issues sooner and apply standards more consistently. We have found AI most useful when it brings the right guidance to engineers at the point of work, and plan to keep extending the approach across Cloudflare.

If you’re interested in building systems like these, our engineering teams are hiring.

Introducing: Cloudflare Agents

Post Syndicated from Nevi Shah original https://blog.cloudflare.com/agents-on-cloudflare/

We're bringing together everything you need to deploy and manage hosted agents on Cloudflare, starting with observability.

We've spent the last nine years building a developer platform, and agents are the perfect use case. They're really just another type of application, but what you need to build them — model access, durable runtime, orchestration, sandboxed execution, persistent storage — happens to be exactly what we've already built.

Now, we’re making it even easier to deploy and manage your agents on Cloudflare. Cloudflare Agents brings all of your deployed agent sessions into a single experience, surfacing key information and insights into how your agents perform at scale.

First stop: agent tracing

We are launching agent tracing for more direct visibility and insight into agent behavior. With agent-aware traces, you can now understand exactly what your agent is doing and what it costs: every model call, tool execution, and token is measured and presented here. Agent tracing launches today with support for OpenTelemetry-compatible agent harnesses including Think, Flue, and AI SDK, and more.

Agent traces are just the beginning. Once you have observability into your agent’s thought process and real-world behavior, you can start to analyze this data and make real improvements. Plug this data into your agent development lifecycle, and you suddenly have autonomous, self-improving agents. This is the vision for Cloudflare Agents: one place to deploy, observe, and continuously improve every agent you run.

Making agents observable

An agent can return HTTP 200 and still fail. It may choose the wrong tool, pass stale context to a subagent, or spend tokens in a retry loop. Traditional application telemetry might show the API request or database query, but not the agent behavior that caused it.

Agent-level telemetry should answer questions such as:

  • Where did the time go: the model, the tool, or the infrastructure?
  • Did the turn pause for approval?
  • Which model did the agent call, and how many tokens did the turn use?
  • Did the agent choose the right tool?
  • When the tool called an external API, did it receive a successful response or time out?
  • Which subagent performed the work, and how did that work affect the final response?

Workers tracing already covers the infrastructure layer, including fetch calls, KV reads, and D1 queries, but until now, traces for agents running on Workers contained those infrastructure spans without the agent operations surrounding them. Agent tracing closes that gap, adding spans for agent invocations, model calls, tool execution, approval events, and supported subagent calls alongside the Workers data already captured. You also get context such as the model and token usage attached as metadata. 

Starting today, agents built with Think, Flue, and AI SDK will send agent traces to Cloudflare, letting you visualize them in the dashboard or export them to a supported OpenTelemetry-compatible destination.

All your agents in one place 

The Cloudflare dashboard now has a dedicated Agents view that lists observed agents and their traces alongside runs, sessions, instances, and reported token usage. 

When you open an agent, you can visualize, understand, and debug what it’s doing in two ways:

  • Replay a session to review captured context across all turns 
  • View a trace to inspect the execution of each turn

Replay a session

The Messages tab assembles the full conversation for a given turn: system instructions, user messages, the model's thinking, tool calls with their arguments and results, and the final response. It's a replay of recorded data, not a re-execution of the agent. This lets you catch a malformed tool argument, see the context available when a tool was selected, understand handoff to subagents, or identify how an earlier turn influenced a later result.

In this example, a user asks to plan a two-day trip to Lisbon. You can see the model's reasoning, watch it call destination_researcher twice (it retried), read the tool results, and follow its thinking as it moves on to building the itinerary. If the agent made a bad decision, this is where you find it.

Exactly what gets recorded depends on your harness or framework. For Think, Flue, and the AI SDK, storeMessages and storeTools control whether message and tool payloads are captured. You can turn payload recording off when that data may contain personal information, secrets, or other sensitive data.

Check the trace

The Traces tab shows the execution waterfall, where you can determine how time was spent and connect agent operations to Workers infrastructure. 

In this trace, a Travel_Planner agent delegates to an itinerary_builder subagent, which calls a model, runs a tool, hits D1, and writes to KV — all visible in a single waterfall:

  • invoke_agent TravelPlanner: The parent agent invocation, 2.72 minutes total. Identifiers for the agent class, conversation, and Durable Object are attached so you can correlate across traces.
  • invoke_agent itinerary_builder: The subagent, nested under the parent, taking 1.83 minutes of that time.
  • chat @cf/zai-org/glm-4.7-flash: Model calls at each level, with duration and provider-reported token usage attached. The first call (17.59s) was the parent's routing decision; the subagent made its own calls underneath.
  • execute_tool record_itinerary_builder_execution: The tool execution, 104ms.
  • cloudflare-d1 run d1_run: A D1 query triggered by the tool, also 104ms.
  • execute_tool record_respond_ready: The tool execution, 232ms.
  • cloudflare-kv put kv_put: A KV write from a later tool, 232ms.

Workers tracing already instruments bindings such as KV, D1, Durable Object, service-binding, and fetch calls, so the Cloudflare infrastructure used by a tool appears under the agent operation that triggered it. Supported subagent calls nest under the parent when child work runs within the active traced context. That lets you follow a turn from the parent agent, through delegated work, to the Cloudflare resources each agent used.

How to enable agent tracing

First, enable tracing in wrangler.jsonc, the Worker's project configuration:

Setup after that depends on the stack

Soon any OpenTelemetry-compliant toolkit will just work

We’re working to support the OpenTelemetry API directly inside Workers. This means frameworks that already emit OpenTelemetry Generative AI semantic conventions spans will be able to visualize them in the Agents view without waiting for a Cloudflare-specific adapter. When those spans include standard agent and conversation identifiers, the Agents view can group them into agents and sessions just like our built-in integrations. Cloudflare can already export OpenTelemetry data; this adds the other direction by accepting standard telemetry generated inside Workers.

Export traces with OpenTelemetry

Your agent telemetry isn’t locked into Cloudflare. You can export traces to any OTLP-compatible provider by configuring a destination in your Worker’s Wrangler configuration file. Because every trace is structured, the same data that helps you debug agents can also power evaluations, analytics, and token-usage reporting. This means traces aren’t just something you inspect when things break, but also a feedback loop for improving your agent’s quality, performance, and cost.

Pricing

Agent traces are built on Workers tracing, so pricing is straightforward. The Agents view shows your agent's operations, but the full Worker trace may include additional spans from SDK internals and other Worker-level operations. To see the full trace, click “View in Observability”.

Every span counts as an observability event, not just the ones visible in the Agents view. All tracing is currently free while in beta. Starting October 1, 2026, tracing pricing will be included as part of existing Workers Observability pricing:

Get started

Tracing is the first piece as we keep building out Cloudflare Agents into the place where you easily deploy, observe, and continuously improve every agent you run. 

Ready to see what your agents are doing? Check out our documentation to enable observability on your agent and head over to the Agents dashboard to inspect your first trace or replay a session.

How we built a software factory to drive Astro’s GitHub issue count to zero

Post Syndicated from Matthew Phillips original https://blog.cloudflare.com/astro-issue-triage/

Everyone is talking about software factories: the idea that AI agents can be assembled into a pipeline that produces working software on their own, the way a factory turns raw materials into finished goods. There’s endless debate over whether that’s actually possible, how far the automation can really go, and whether the “loops” people are demoing count for anything. Some have already written them off as a failure.

Running alongside that is a quieter, more worried conversation: open source maintainers are burning out. The AI boom has made it nearly free to generate issues, pull requests, and security reports, and enormously expensive for a maintainer to read through them all. The old ways of keeping a project healthy are buckling under the volume.

Everyone has a hot take on both topics. We think we have something rarer to offer: real results. For the past several months we’ve run an automated triage pipeline on the Astro repository. It reads incoming bug reports, reproduces them in sandboxes, diagnoses the root cause, and ships preview releases for the reporter to verify. The engine underneath it grew into Flue, an open framework for building this kind of agent automation, and it’s the same tool you could use to build your own.

It wasn’t an instant success. But through a lot of iteration, we’ve used it to bring our open issues down from over 200 to about 30, and we expect to hit zero sometime in the next month. That would be the first time this repository has seen zero open issues in its 5+ year history. 

We didn’t get there by declaring "issue bankruptcy," auto-closing cold tickets, or ignoring reports. We did it by automating issue triage with a team of isolated AI subagents running right inside GitHub Actions. Here’s the story of how we got there, and what you might take back to your own projects.

Starting with an agent skill

At the start of the year, we focused on automating one specific area of development: issue triage. As an open source project, manual issue triage can be one of the more time-consuming, least-rewarding parts of the job. A single issue can sometimes take hours just to reproduce, let alone fix. It was a natural (yet often overlooked) place for us to start our automation journey.

We began by developing an agent skill. This allowed us to develop and test the automation locally as maintainers, running a coding harness on our own machines. We could then run that same harness in a GitHub Action on our repo, and get total reuse of that exact same triage workflow skill.

The triage skill mirrors the exact steps we take during manual issue resolution:

  1. Reproduce: Clone the provided reproduction repository to verify the reported issue.
  2. Diagnose: Instrument the codebase and introduce logging to pinpoint the root cause of the bug.
  3. Verify: Review relevant test suites, code comments, and documentation to determine if the behavior is genuinely a bug or intended functionality.
  4. Fix: Convert the reproduction into failing unit tests, identify the appropriate solution via the architecture guide, and deploy the fix.

To prevent the frequent LLM bias toward forcing a solution when a bug might not actually exist, each phase is executed by an isolated subagent. These subagents pass information forward sequentially by compiling their discoveries into a report.md file.

Turning the skill into an automation

Following initial internal testing of the triage skill, our focus shifted toward building a fully automated pipeline. We specifically wanted to integrate this logic directly into a GitHub workflow, ensuring complete transparency so that anyone could easily audit the agent's sequential reasoning and operational steps.

As we wired it up, we realized the whole pipeline was really just a state machine driven by issue labels. Every new submission starts with the label triage needed, and once a user confirms a fix it moves to fix verified. Beyond those label transitions the pipeline holds no state of its own; it simply reads back through the issue’s existing comments to work out where a given issue is and what should happen next.

From there the flow runs on its own. When the agents land on a fix, the pipeline spins up a preview release with pkg.pr.new and posts everything back to the issue: a summary of what it found, the full logs, and instructions for installing the preview. The original reporter can then try the patch against their own project, and if they confirm it works, the automation opens a pull request linked to the issue.

From triage to a framework

As we built this out, we kept noticing that nothing about it was really specific to GitHub. Reacting to an event, running a sequence of isolated subagents, and separating their reasoning from the actions they’re allowed to take — it’s all just a workflow. One that could run just as well from a Slack message, a cron job, or a webhook as from a GitHub issue. Generalizing that realization into a runtime that works the same way regardless of where it’s deployed, or which model it’s driving, is what became Flue: an open, platform-agnostic framework for building durable agents and workflows.

Benefits of agent automation

When we first launched this automated system, we had shared concerns about its efficacy and the potential negative impacts it might have on our developer community. There was a valid fear that relying on automated bot responses might feel impersonal and create just one more disconnect between us as maintainers and our user base.

That did not happen. If anything, we talk to users more now, just in more useful places:

  • Engaging directly with our community members within Discord.
  • Actively participating in RFC discussions and addressing new feature requests.
  • Collaborating closely with contributors to help integrate their ideas into the framework.

Regarding the quality of automated patches, our core philosophy is that our AI agents should successfully resolve the vast majority of incoming issues. When an agent fails to identify a correct solution, we interpret that failure as an indicator of an underlying architectural or documentation issue within the codebase, pointing to one of three areas:

  • Opaque Abstractions: If an agent cannot interpret the boundaries between components, human developers likely struggle with the code structure as well.
  • Missing Documentation: Critical code segments lack explicit comments explaining the rationale behind their implementation.
  • Insufficient Testing: The repository suffers from a lack of comprehensive test coverage, particularly unit tests.

A clear example occurred with a series of related Hot Module Replacement (HMR) bugs. The triage bot repeatedly attempted to modify a specific if condition to resolve the issue. While this change fixed the targeted bug, it introduced regressions elsewhere due to a lack of test coverage for that specific condition. Once we added a descriptive comment explaining the exact logic governing that statement, the bot adapted and stopped attempting incorrect modifications in that area.

Every time we chase down one of these failures and add the missing comment, test, or clearer boundary, the bot gets noticeably better at that part of the codebase, and so does the next human who works on it.

Turning the workflow into a GitHub Action

Initially, our triage logic lived directly within the Astro monorepo. This coupling made iteration difficult; upgrading Flue or modifying the workflow felt like performing surgery on live infrastructure without a safety net. To solve this, we decoupled the logic into a standalone, testable repository: triagebot-action. This isolation allowed us to introduce automated testing and ensure stability before ever touching our primary codebase.

Today, this action powers issue management in Astro, and it has spread from there. Several other teams have picked it up, some using it directly, and others forking it to build their own automated "factories" tailored to their projects. That second path is really the point: triagebot-action is young and still actively evolving, so we’re sharing it less as a finished product and more as a working reference you can read, learn from, and adapt. 

The wiring for the action itself looks like this:

Or point your own agent at the repository and have it read through the setup, including adding the labels the state machine relies on.

Whichever route you take, the underlying idea matters more than our specific implementation: a sustainable feedback loop that frees maintainers to focus on the framework itself instead of administering a backlog. The code is open. Fork it, strip it down, or just borrow the parts that fit your project.

Want to build something like this? Dig into the code of the triagebot-action to see how it works, or fork it as a starting point for your own repository’s automation. And if you’re building agent-based infrastructure more seriously, that’s exactly what Flue is for: dive into the Flue framework to build your own. We’d love to see what you build. Come share your "factory" stories in the Astro Discord.

Announcing Cloudflare Wallets: the programmable wallet for the agentic Internet

Post Syndicated from Will Papper original https://blog.cloudflare.com/wallets/

Today, it is difficult for AI agents to try out new APIs. They often have to navigate through a login page designed for humans and not agents, contact a human to add a payment method, generate an API key, and then figure out how to call the API.

This flow is very difficult for agents for two reasons: Agents do not have a stable identifier to sign up for an API, and they do not have a native way to pay for APIs. Because they lack these things, they often struggle to onboard onto software, which limits the growth of agentic commerce. AI agents often give up on these tasks entirely, kicking registration, payment methods, and API key generation back to humans. This makes it very difficult for agents to try out and compare many APIs.

To solve this, we’ve created Cloudflare Wallets. Starting today, you can claim a Cloudflare Wallet handle for your account, which will provide a unique username to help you better connect with merchants. Soon, you will be able to set up and use your Cloudflare Wallet to pay for APIs and content.

Earlier this month, we announced the Monetization Gateway to help Cloudflare customers get paid for their websites and applications. Monetization Gateway will support micropayments using the x402 protocol, which allows for payments to be attached to HTTP requests. These micropayments will be able to pay for uses ranging from AI inference to data to content. If you want to pay or get paid for services behind Monetization Gateway and other x402-compatible endpoints, you’ll need a wallet. 

Cloudflare Wallets will allow you to store stablecoins, purchase services, and receive funds across the web. Each account with a wallet will also be able to create Virtual Wallets for its agents to enable them to buy APIs, MCP Tools, content, and more. You will be able to define guardrails for your Virtual Wallets (such as an allowance, an allow list, and a maximum transaction size) to help your agent spend money safely from your account. This will allow your agent to try out many APIs with low friction and managed risk. Wallet users will have the option to share their Cloudflare Wallet handles, which will give them a stable identity when interacting with merchants.

Building the two-sided agentic market

Cloudflare’s Monetization Gateway will allow eligible Cloudflare customers to sell their resources (such as content or APIs) headlessly to agentic buyers. But for that market to truly develop, agents need more tools to buy from merchants in a machine-native way. Wallets will add another tool to Cloudflare’s Agents SDK, enabling AI agents to easily purchase necessary APIs and content using micropayments.

There will be two types of Cloudflare Wallets: Account Wallets and Virtual Wallets.

Account Wallets are designed for humans who are owners and users of Cloudflare accounts. They will be able to add funds, delegate spend to virtual wallets managed by agents, and remove funds as needed. 

Virtual Wallets, by contrast, are designed for agents and operate via API keys. Within a Virtual Wallet, an agent will be able to spend funds according to its permissions. Its maximum spend will be capped by the limit set by the owner of the Account Wallet. This framework gives agents freedom to act on behalf of users without constant manual approval while limiting an agent’s ability to overspend.

The freedom to explore

Virtual Wallets are exciting because they will allow agents to do what they’re best at: explore dozens or hundreds of services and find the best one for a particular use case. Stablecoin micropayments via x402 will make it simple to try an API without an account, allowing agents to test new options with little friction. The spending caps on Virtual Wallets are designed so that humans can let agents explore autonomously within safe spending limits. These limits may seem like constraints, but counterintuitively they give agents more freedom. If an agent is responsible for $10, you can worry less about its spending than if it is responsible for $1,000. If an API only costs a few cents to try, then $10 is more than sufficient to pursue and evaluate many options.

Once you or your agent has picked an API to use, policies set by you in your Account Wallet will act as cost controls for Virtual Wallets. Want to give every employee a $100 per week budget for AI inference? Simply provision an Account Wallet with the right balance and create Virtual Wallets for each employee with that rule. Anyone who exceeds the limits on their Virtual Wallet will be able to request a manual override from a human who is authorized to make changes to the Account Wallet.

We want to make it easy for Account Wallets to set flexible yet firm spending policies that do not require daily, active monitoring. When something anomalous happens, such as unexpectedly fast spending, a human will be able to review and confirm whether everything is operating as intended. If the spend was intentional, then the administrator of the Account Wallet will be able to raise the limit or approve a one-time injection of funds. If the spend was unintentional, then the spending policies for adding funds to virtual wallets did their job by imposing caps.

We are working to make it as easy as possible to fund and use these wallets. We will start with simple ways to onramp and offramp funds within supported geographies, with self-funding via stablecoins available as an alternative for eligible users. The Internet will not shift completely overnight, but with a majority of traffic on the web now being driven by bots, we are excited to give agents and merchants first-class tools for agentic commerce.

Beyond payments alone

Allowing humans to delegate authority to agents to easily buy and sell services is a helpful starting point. But this delegation is not always obvious to the merchants as they interact with agents. Today, if an agent comes to your website, you may know little about them as a user, despite the fact that the agent is acting on behalf of an individual or an organization. This lack of attribution challenges many traditional web business models. It’s easy to give a one-week free trial or sign-up credits to a human or an organization. It’s hard to give these same perks to an agent that lacks a stable identity and when one human can spin up dozens of agents under their control.

We solve this problem by linking wallets to a Cloudflare account via cloudflare.pay. cloudflare.pay will allow agents to optionally identify themselves, since their identity is a delegate of the account. A research agent could live at research.example.cloudflare.pay, allowing merchants to know that it is an agent from a particular organization. This approach will permit agents to maintain consistent and persistent identities, making the experience better for all parties. It will be completely optional for agents to choose to declare their identity or not, and it will be up to businesses to decide whether they want to prioritize transacting with known agents.

Agent identifiers should be human-readable

We believe that the approach to dealing with agents will look like the approach to dealing with VPNs: If someone is unidentified, they are not inherently untrustworthy, but they need to prove themselves more. This is why we have Turnstile and other initiatives to detect bots within Bot Management. Our identity primitive will build on top of this prior work. For example, Web Bot Auth already allows agents to register their identity via a keypair. IDs attached to Cloudflare Wallets allow this keypair to become human-readable.

We know that agentic identity standards are changing quickly, which is why we wanted to keep our approach simple. We are proposing a human-readable identifier for a not-very-readable keypair, similar to the URL and IP-address pairings used in DNS. We are not trying to define a particular schema or other verification system. We only want to make identity simple to remember and easy to declare. As schemas to enrich agentic identity develop through the x402 Foundation’s initiatives, we will seek to adopt them and intend to encourage others to do the same.

The future of agentic commerce

At Cloudflare, we want to offer all the building blocks for agentic commerce to succeed. Monetization Gateway will offer a way for sellers to get paid without setting up traditional payment infrastructure. Wallets will offer a way for buyers to pay headlessly via agents. Identity will allow merchants to communicate with buyers who identify themselves or enforce identification requirements.

All of these building blocks will create a headless marketplace for the Internet. If you are excited about this and want to participate, you can claim your handle now. We’re excited to see what you build and monetize.

Your agent can now debug Workers with local tracing

Post Syndicated from Zin Khant original https://blog.cloudflare.com/local-tracing/

Starting today, wrangler dev and vite dev automatically capture OpenTelemetry traces for local Worker invocations. When Cloudflare's tooling detects an agent session, it points the agent to the Local Explorer API, a local debugging API where it can query those traces. You do not need to install an SDK, enable tracing, configure your agent, or even mention observability in the prompt.

A prompt can be as simple as:

This builds on years of investment in local development, from introducing Miniflare to making local mode the default in Wrangler 3. Local traces give coding agents structured feedback from that development environment before code is deployed.

Agents discover the Local Explorer API automatically

As part of its normal workflow, an agent starts wrangler dev or vite dev to run and test the Worker. When the development server recognizes a supported coding-agent session, it automatically prints a hint that looks like this:

The Local Explorer is a browser-based interface and REST API for viewing and editing local resource data and querying observability data during development. The API root serves an OpenAPI schema, so agents can discover available endpoints at runtime without hardcoded instructions.

The automatically captured traces are available through a read-only observability endpoint in that API, together with their correlated console logs. The agent can query this telemetry, then use the API's other operations to inspect local Workers and bindings or examine state in D1, KV, R2, Durable Objects, and Workflows. 

Find the failure and verify the fix

Consider POST /api/orders, which retrieves an active cart from KV, saves the checkout details into D1, and sends a message to a Queue for order processing. After a schema change, the endpoint suddenly starts returning a 500 status.

Without local traces 

The 500 does not identify which operation failed. The agent adds logs around KV, D1, and the Queue, reruns the request, inspects the output, and repeats. Each cycle takes time and burns tokens while the agent reconstructs the request from text.

With local traces 

The agent reproduces the error and queries the read-only observability endpoint. The trace shows that the KV read succeeded, the D1 insert failed with no such column: delivery_window, and the Queue was never called. Your agent uses the Local Explorer API to access the same trace data you would see here: 

The agent uses the API to inspect the D1 schema. It finds that the migration adding delivery_window exists in the repository but has not been applied locally, applies it, sends the request again, and queries the new trace. Issue resolved. 

In one local loop, the agent identifies the failed operation, fixes the local environment, and verifies the result without deploying or adding temporary logs.

Explore traces and logs in Local Explorer

Agents query local telemetry through the API, but you as a human can visualize the same data in the Local Explorer, the browser-based interface built into the local development server. Alongside browsing local binding state, you can select a request to inspect its spans, timing, attributes, errors, and correlated console logs. 

Local Explorer runs on the same localhost origin as your Worker, not in the Cloudflare dashboard. Press e in Wrangler or visit /cdn-cgi/explorer on the local server to open it.

How it works 

When we launched Workers Tracing, we built instrumentation directly into workerd, the open-source runtime that powers Workers. Without requiring an SDK or any code changes, the runtime captures spans for:

  • Fetch calls: All outbound HTTP requests, including timing, status codes, and request metadata.
  • Binding calls: Every interaction with KV, R2, D1, Durable Objects, Queues, and other bindings.
  • Handler calls: The full lifecycle of each invocation, from fetch to scheduled to queue handlers.

Any custom spans emitted by your application will also appear alongside these automatic spans. 

Wrangler and the Cloudflare Vite plugin use Miniflare to run your Worker locally in the same runtime, making this instrumentation available during local development.

Miniflare collects runtime events and console output, assembles them into OpenTelemetry traces and correlated logs, then writes the telemetry to an internal SQLite-backed Durable Object that serves as the local trace store. The Local Explorer API exposes that data through the local development server where agents can easily query traces and logs and inspect local state. 

Get started

Update Wrangler or the Cloudflare Vite plugin, whichever your project uses:

Then ask your agent to debug locally as you normally would. Your agent can already write and run your Worker locally — now it can see what happened, fix what failed, and verify the result before you deploy. Check out the docs to learn more!

OFAC действа, прокуратурата спи – симптомите на завладяната съдебна система

Post Syndicated from Bozho original https://blog.bozho.net/blog/4613

С документи на американските власти, публикувани от BIRD, отново се поставя темата за подкупа, даден от Божков на Борисов и Горанов, за да си спести милиони в данъци за хазартната му дейност.

Затова днес от ДБ пратихме писма до OFAC, прокуратурата и ДАНС. За да се изясни докрай с какво разполага САЩ, с какво разполагат българските органи и не следва ли наказателното производство да бъде възобновено.

И както винаги въпросът е не само за това какво установяват в САЩ, а защо българските правоохранителни органи се интересуват по-малко от корупционни престъпления в България, отколкото се интересува друга държава.

Въпросът е риторичен, разбира се – защото Борисов и Пеевски се ползват от чадъри в прокуратурата, която е задкулисно завладяна вече дълги години.

Но конкретната механика на завладяването е важна. И конкретните лица – също. Защото докато десетки независими прокурори учредяват втора прокурорска асоциация, сред останалите са най-верните слуги на модела, и то позицинирани на ключови постове.

Говорил съм и преди за прокурора, който не се интересува от показанията на Божков и други свидетели за дадения подкуп – прокурор Марина Ненкова от СГП, която винаги се „пада“ по особено интересните дела. Дела за ББР, за контрабандата в Митниците, за подкупа към Борисов, както и делото срещу мен.

За това, което OFAC установява за Божков и Борисов, прокурор Ненкова казва „дума срещу дума“ и прекратява досъдебното производство. Същата обаче кредитира показания на един единствен свидетел, че съм му дал флашка (каквато флашка не съществува) и ми повдига обвинение.

Но как Марина Ненкова се оказва прокурор по толкова много дела от висок обществен интерес? В СГП има около 120 прокурори, как така малка група прокурори винаги се падат по важните дела?

Отговорът е в специализацията и в административния ръководител, който може да определя групи прокурори, които гледат отделни групи дела. И така, скрити зад валидния аргумент за специализацията, с помощта на едни изтекли инструкции за донагласяне на случайното разпределение, административните ръководители в Софийска градска прокуратура си правят шпиц-команди за политически поръчки.

Само че за да имаш десетина прокурори за политически поръчки и съдии, които да потвърждават безобразията им, те трябва да са зависими – хванати в някое прегрешение и държани с него. Иначе едва ли доброволно биха газили закона – по-скоро биха си правили отводи.

Затова в СГП съществува „6-ти отдел“ – прокурори, разследващи само престъпления, извършени от съдии, прокурори и следователи. В момента 6-ти отдел е най-малък, само с верни хора на джуджето Емилия Русинова. Тези прокурори образуват досъдебни производства срещу “неизвестен” техен колега, искат СРС-та срещу магистрати, и всичко това после се превръща не в обвинителни актове, а потъва в компроматните банки. И така, малко по малко, системата се овладява.

Главният прокурор може да контролира всичко това. Да иззема дела, да отменя постановления, да налага организационни ограничения, за да няма шпиц-команди. Но рядко го прави, защото „е в играта“. Когато го направи, е защото някой е решил да “играем сам” или “да пропее”. А подчинените му прокурори, временно установени в прокурорската колегия, чинно избират административни ръководители, особено на СГП, които да гарантират, че смазаната машина ще продължи да функционира.

Разбира се, в картинката винаги участват и службите за сигурност – ДАНС, ДАТО, КПКОНПИ/КПК, чиито ръководители виждат и знаят какво става, разписват разпореждания за подслушване, притискат свидетели, но не носят отговорност – и изгряват назначение на някой нов пост – дали пак като шефове на ДАНС, дали като прокурори.

И така ОПГ-то си работи – част от него, облечено в прокурорски тоги, а друга част – в изпълнителна и законодателна власт.

Това е “моделът”, който трябва да бъде демонтиран. Това става с политическа воля, със законодателни и с кадрови действия – смяна на ВСС, така че да не се назначават такива административни ръководители, смяна на законодателството, така че злоупотребите със СРС-та, държането на трупчета, съзнателното наказателно-процесуално бездействие и др. способи да бъдат много по-трудни за реализиране, смяна на структурата на вземане на решения и управление на съдебната власт.

Без да бъде демонтиран този модел, все ще гледаме някъде навън да ни решават корупционните проблеми, които ограничават благосъстоянието и развитието. И това все няма да се случва, защото корупционните проблеми трябва да си ги решим ние като суверенна държава.

Материалът OFAC действа, прокуратурата спи – симптомите на завладяната съдебна система е публикуван за пръв път на БЛОГодаря.

CVE-2026-18577: N-able N-central Authentication Bypass Exploited in the Wild

Post Syndicated from Rapid7 original https://www.rapid7.com/blog/post/etr-cve-2026-18577-n-able-n-central-authentication-bypass-exploited-in-the-wild

Overview

On August 2, 2026, N-able published a security advisory for CVE-2026-18577, an authentication bypass vulnerability affecting N-central that was discovered being exploited in-the-wild after an incomplete fix for an earlier authentication bypass issue, CVE-2026-18556 was disclosed. CVE-2026-18577 allows a remote unauthenticated attacker to bypass authentication and obtain administrative control of vulnerable N-central servers in affected deployments.

N-able N-central is a widely deployed Remote Monitoring and Management (RMM) platform used by managed service providers (MSPs) and enterprise IT teams to centrally administer servers, workstations, network devices, and other managed assets. Because the platform operates with extensive administrative privileges across customer environments, successful compromise of an N-central server can provide attackers with an efficient path to compromise downstream managed systems.

According to N-able, exploitation of CVE-2026-18577 has been observed in the wild since August 1, 2026. Following successful exploitation, attackers leveraged the platform’s Take Control functionality to remotely access managed endpoints, and deployed Cloudflare Tunnel (cloudflared) to establish persistent remote access. On August 3, 2026, CVE-2026-18577 was added to CISA’s Known Exploited Vulnerability (KEV) catalog. 

Mitigation guidance

Organizations operating vulnerable N-central deployments should prioritize remediation on an urgent basis, outside of normal patching schedules. Hosted N-central environments are upgraded automatically by the vendor, while on-premise deployments require manual remediation.

Affected versions:

  • All versions of N-able N-central up to and including version 2026.3.1, prior to Hotfix 1.

Fixed version:

  • N-able N-central 2026.3.1 Hotfix 1 (2026.3.1.7).

The vendor also recommends:

  • Upgrading N-central agents after applying the server hotfix.

  • Reviewing systems for indicators of compromise.

  • Contacting N-able Support immediately if evidence of compromise is discovered.

  • Engaging internal incident response teams if malicious activity is identified.

For further information, see the vendor advisory.

IOCs

N-able has published several artifacts that administrators should investigate during incident response.

Endpoint Artifacts:

  • Presence of a Cloudflared service.

  • A suspicious svchost.exe located within the user’s Documents folder.

Network Indicators:

  • Administrators should review historical network logs for inbound or outbound communication involving the malicious IP addresses identified by the vendor:

    • 173[.]249[.]252[.]200

    • 87[.]249[.]138[.]34

    • 37[.]19[.]210[.]32

    • 37[.]153[.]90[.]88

    • 92[.]118[.]112[.]181

    • 68[.]235[.]46[.]214 

Organizations should also review:

  • Authentication logs

  • Administrative account creation or modification

  • Take Control session activity

  • Remote management logs

  • Windows service installation events

To assist affected organizations running N-central, the vendor has provided a detection template for CVE-2026-18577, which organizations can use to help identify potential compromise.

Rapid7 customers

Exposure Command, InsightVM, and Nexpose

Exposure Command, InsightVM, and Nexpose customers can assess exposure to CVE-2026-18577 with a vulnerability check expected to be available in the August 4 content release. Note that potential check type must be enabled in the scan template before scanning.

Updates

  • August 4, 2026: Initial publication.

Some Claude Chats Are Searchable on Google

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/08/some-claude-chats-are-searchable-on-google.html

And it’s personal information (alternate link):

The exposed data includes an AI-powered therapy app that someone appears to have vibe-coded, notes on meetings, and a dashboard someone made apparently to analyze medical billing data. Exposed chats reportedly include private cryptocurrency wallet keys and personal information like peoples’ addresses.

What seems to be the issue is a user setting about data sharing. Anthropic’s position is that it’s not their problem:

“We give people control over sharing their Claude conversations publicly, and in keeping with our privacy principles, we do not share chat directories or sitemaps with search engines like Google,” the company said in a statement. “These shareable links are not guessable or discoverable unless people choose to share them themselves. When someone shares a conversation, they are making that content publicly accessible, and like other public web content, it may be archived by third-party services.”

Here’s how to fix it.

Supporting AI education for 150,000 learners in Aotearoa New Zealand and Australia

Post Syndicated from Anna Burton original https://www.raspberrypi.org/blog/supporting-ai-education-for-150000-learners-in-aotearoa-new-zealand-and-australia/

We’re pleased to share that we are expanding our Experience AI programme to Australia and Aotearoa New Zealand to train 5000 educators who can reach 150,000 students by 2028, thanks to generous funding of $1.2 million from Google.org.

CSER Team delivering a workshop at Adelaide University

Working with local education organisations, we will support young people to develop a foundational understanding of AI technologies, their social and ethical implications, and the role that AI can play in their lives.

AI literacy across the world through Experience AI

AI systems are a common part of everyday life and influence how we access information, and how we work and solve problems. We believe that young people need more than the ability to use AI tools: they need the knowledge, skills, and confidence to understand how AI technology works, to think critically about its impact, and to create AI-based solutions of their own. 

Experience AI is our free educational programme co-developed with Google DeepMind that helps teachers and students learn about artificial intelligence. Through the Experience AI training, lessons, classroom resources, and hands-on activities, teachers introduce young people to how AI systems work, how they can be used, and what their impacts may be.

CSER Team example of teacher workshop 2026

We bring AI literacy to young people across the world with Experience AI by building trusted partnerships with local organisations that lead sustainable delivery of the programme in ways that suit their contexts. Through this global network of Experience AI partners, we have trained over 50,000 educators who can reach an estimated 4.8m young people. Today, Experience AI resources are used in over 195 countries and available in 22 languages. In recognition of its global impact, Experience AI was named a laureate of the 2025 UNESCO King Hamad Bin Isa Al-Khalifa Prize for the Use of ICT in Education.

Experience AI partnerships in Australia and Aotearoa New Zealand

In Australia, the first partner we are working with is the Computer Science Education Research Group (CSER), based at Adelaide University. Professor Katrina Falkner, the university’s Pro-Vice-Chancellor, Learning and Teaching, says about the partnership:

“We are thrilled to partner with the Raspberry Pi Foundation to bring the Experience AI programme to Australian schools. It is so important that teachers are provided opportunities to understand AI so they can help students develop the self-regulated learning skills needed to thrive in a world where AI is increasingly part of everyday learning and work. Educators can play a critical role in ensuring that a human lens of critical thinking, ethical judgement, creativity, and meaningful human connection remain at the heart of AI education and adoption, preparing students for careers where effective collaboration with AI tools will be essential.”

In Aotearoa New Zealand, we are working with Tōnui Collab Charitable Trust, a Maōri-led organisation dedicated to creating innovative STEM learning opportunities.

Collaboration with Tonui

Shanon O’Connor, Director of Tōnui Collab, says about the partnership (1):

“We are partnering with the Raspberry Pi Foundation to provide this kaupapa to educators in Aotearoa, adapting and contextualising their global Experience AI program to make it meaningful and relevant in Aotearoa.

This kaupapa isn’t about learning to code; it’s not a kaupapa designed solely for the ‘tech enthusiasts’, it’s about fostering digital equity, ensuring rangatahi have the tools and knowledge to thrive in a world increasingly shaped by technology. It’s our attempt to ensure the digital divide doesn’t become a digital chasm. 

We’re also facilitating robust conversations about data bias and the impact this has on the ways we as Māori engage with AI-powered technologies, creating space for kōrero about tech tikanga and our collective responsibilities when using or engaging with AI-powered technologies.”

Looking ahead

All young people need opportunities to develop the skills, knowledge, and confidence to navigate and shape a world where AI technologies are widely used. With support from Google.org and education partners across Aotearoa New Zealand, Australia, we will continue to expand access to high-quality AI education.

Find out more about Experience AI at experience-ai.org

(1) Shanon uses some Māori words that are common in Aotearoa New Zealand for both speakers and non-speakers of reo Māori:

  • kaupapa: the guiding purpose, philosophy, or approach underpinning the work
  • rangatahi: younger generation, youth
  • “kōrero about tech tikanga”: having discussions about the correct protocols, ethics, and practices for engaging with technology

The post Supporting AI education for 150,000 learners in Aotearoa New Zealand and Australia appeared first on Raspberry Pi Foundation.

The collective thoughts of the interwebz