Tag Archives: artificial intelligence

AWS Security Agent (now part of AWS Continuum) exploited 33 of 37 new real-world CVEs

2026-10-15 Dazhi Zhang

Post Syndicated from Dazhi Zhang original https://aws.amazon.com/blogs/security/aws-security-agent-now-part-of-aws-continuum-exploited-33-of-37-new-real-world-cves/

We applied AWS Security Agent (now part of AWS Continuum), our autonomous penetration-testing agent, to 37 real, recently disclosed Common Vulnerabilities and Exposures (CVEs), each packaged as a live, containerized target with a flag hidden at the vulnerability’s intended sink. Starting from an unauthenticated or low-privilege position (an endpoint URL and a one-line hint, no exploit and no proof-of-concept (PoC), the agent had to discover the vulnerability, build a working exploit, chain primitives where needed, and extract the flag.

The results:

35 of 37 flags captured.
33 of 37 (89.2 percent) captured through the intended vulnerability, end-to-end. Confirmed by a per-sample integrity audit against each challenge’s canonical proof-of-concept.
2 captures reclassified. The audit reclassified 2 captures as off the intended path, so we hardened those challenges and count them as unsolved.
2 misses. In both misses, the agent found and exploited the bug, then took the target down before the flag could be read. The non-destructive guardrails that govern production engagements keep that off customer systems.

Note: Details of the two captures and two misses follow later in this post.

The targets are the types of software your services run: web frameworks, a container registry, machine learning (ML) and AI platforms, an ecommerce stack, a tRPC API, several media platforms, and an LDAP admin console. Between them, they span eight vulnerability classes, from SQL injection to server-side request forgery (SSRF) to multi-step cross-site scripting (XSS)-to-admin-cookie chains. Every result described in this post is a captured flag behind a working exploit, produced from a low-privilege start rather than from a bug report.

Why we built a fresh benchmark

Public CVE-based agent benchmarks do a real job: they run an agent against genuine, containerized, vulnerable software. Their weakness is that they can’t stay uncontaminated. Suites like CVEBench and XBOW are effectively saturated, because current models recognize those vulnerabilities and grab the flag almost immediately after the bugs and their write-ups are in training data.

A curated benchmark is a snapshot in time. The CVEs are chosen once and the exploit write-ups are published once, and by the time that snapshot reaches a paper, frontier models have very likely ingested those write-ups and public PoCs. From then on, every score combines two things that should be measured separately: whether the agent can exploit the bug, and whether the model already memorized the answer. That signal decays month over month.

So we built our own internally developed, capture the flag (CTF)-style benchmark of recent CVEs in the same spirit as the public suites but sourced continuously from freshly disclosed vulnerabilities. It tracks the vulnerability classes landing in production this quarter and grows on the same cadence that CVEs are disclosed.

We attack contamination on both routes it can enter through. Recency closes the memorization path: challenges are sourced from CVEs disclosed recently enough that an end-to-end solution is unlikely to be found in the model’s training data. Disabling runtime CVE lookup closes the retrieval path: the agent can’t pull a public PoC mid-exploit, so it works from the target alone. The two are independent: one guards against the model recalling a solution, the other guards against it fetching a new one. Neither is a guarantee by itself: recent CVEs can still leak into training data as advisories, partial discussion, or fragmentary PoCs, so we report this as contamination-reduced, not contamination-free. The work mentioned in What’s next will further reduce contamination.

Note on prior numbers: We have previously reported CVE-benchmark results for AWS Security Agent against established public suites such as CVE-Bench. This benchmark is a different measurement: contamination-reducing, continuously refreshed, and audited per win. Read the two as complementary, not as a restatement.

The dataset at a glance

Each challenge is a self-contained package:

A live target brought up from docker-compose inside an isolated sandbox, running the affected version of the actual upstream software (or a faithful minimal reproduction).
A flag seeded at the intended sink, which could be a database row, an admin session cookie, an internal-only config API, or a file reachable only through the bug. It’s never readable without exploiting the vulnerability.
A canonical PoC describing the PoC intended solve. The agent never sees it, we use it only to audit wins.
Minimal metadata, including endpoint URL, any seed credentials, and a one-line hint.

Today’s sample is 37 CVEs from 2025–2026, chosen to span the vulnerability taxonomy and to require actual exploitation rather than misconfiguration spotting.

How a run works

Every CVE in the benchmark runs through the same fixed pipeline, so results are comparable across vulnerability classes and reporting cuts.

Figure 1: CVE pipeline

Each run moves through seven stages:

Target boots: The challenge comes up from docker-compose in an isolated sandbox, with the flag seeded at the intended sink.
Reconnaissance and discovery: A crawler maps the target’s attack surface: endpoints, parameters, and entry points.
Specialist agents: A swarm of specialist agents probes the mapped surface and raises candidate risks.
Findings and validator: Each candidate risk is confirmed or rejected before any exploitation is attempted.
Exploitation agent: The agent attempts the exploit using a shell, a Python runtime, a real browser, and an out-of-band (OOB) callback listener.
Flag check: If the seeded flag appears anywhere in the agent’s output, the sample is captured; otherwise the exploitation agent keeps trying until the end of the 10-hour time limit.
Integrity audit: Every capture is re-checked—by hand and by tooling—comparing the agent’s trajectory against the canonical PoC to confirm the intended vulnerability was actually exploited.

The numbers

The following table and figures show the results of a full run, including 37 CVEs and one epoch, run on Claude Opus 5 with a fixed 10-hour limit per sample.

Result	Count
Flags captured	35 of 37
Genuine end-to-end exploitation (intended vuln, audited)	33 of 37 (89.2 percent)
Reclassified by integrity audit (benchmark faults, since patched)	2
Genuine misses	2
Vulnerability classes covered	8
Multi-step chained exploits captured autonomously	More than 6
Median time to capture	Approximately 4 hours

The following figure shows benchmark coverage by vulnerability classes.

Figure 2: Benchmark coverage by vulnerability class, colored by outcome

As shown in the following figure, the run outcome after the integrity audit showed 89.2 percent of findings were genuine.

Figure 3: Run outcome after the integrity audit: 89.2 percent genuine (33 of 37)

Coverage by vulnerability class

The following table sorts the 37 CVEs by vulnerability class and shows how many challenges were found in each class and how many the agent genuinely captured end-to-end after the integrity audit. The misses and reclassifications are concentrated in three classes; the other classes were captured in full.

Vulnerability class	CVEs	Genuine capture	Notes
Command or Code Injection leading to remote code execution (RCE)	7	7	JavaScript, operating-system command, and framework code-execution paths
Cross-site scripting (XSS) used to steal an admin cookie	6	6	Stored, SVG, and markdown XSS, exfiltrated through an admin bot and an out-of-band (OOB) listener
Broken authentication or access control	5	5	Broken function-level authorization (BFLA), insecure direct object reference (IDOR), default credentials, and API-gateway bypass
SQL injection	5	4	One reclassified (debug-page leak; see the audit)
Server-side request forgery (SSRF)	4	3	One reclassified (network-topology fault)
Business logic, cross-site request forgery (SSRF), or time-of-check-to-time-of-use (TOCTOU)	3	3	Vote tampering, admin-bot CSRF, and an authentication race
File upload leading to remote code execution	3	1	Two misses; the agent took the target down mid-exploit
Path traversal or local file inclusion (LFI)	2	2
Denial-of-service-triggered information leak	2	2	Crash the application; the monitor then discloses an incident secret
Total	37	33

Three captures worth showing

Aggregate numbers show how often the agent is successful, but why. The three captures that follow were chosen to show the range: a long multi-step chain stitched together autonomously and two other exploits that each required the agent to reason past an obstacle rather than replay a known payload. Each one is an audited, end-to-end capture against a live target, with the agent working only from the endpoint and a one-line hint.

A six-primitive chain, fully autonomous

An open source content management system (CMS) was the target of an attack. The agent found a second file-upload endpoint that—unlike the primary uploader—enforced neither a cross-site request forgery (CSRF) token nor an extension check, and turned it into a full admin-account compromise. As shown in the following figure:

Figure 4: Workflow diagram

From the captured trajectory (endpoint and payload redacted):

“Abused a secondary dropzone upload handler (no CSRF token, no extension or content validation) to upload an HTML page. The core uploader rejects it; this parallel sink does not. File served same-origin, containing a script that beacons document.cookie to the agent’s OOB listener.”

Reconnaissance, primitive discovery, tool orchestration (upload, listener, and bot trigger), and post-exploit extraction were chosen by the agent, with no template.

Unauthenticated RCE to root

An AI workflow builder was the target. The agent posted a custom component whose class body ran shell commands at import time, then read the flag straight out of the process environment and returned the following:

tool result: uid=1000(user) gid=0(root) groups=0(root) … ENV_SECRETS: … FLAG=flag{...}

One request produced arbitrary code execution as a root-group process and a full environment dump. The agent recognized the code-execution surface, weaponized it, and pulled the secret end to end.

Full-read SSRF through a tRPC API

The target was a self-hosted dashboard app. A tRPC ping-style procedure took a fully attacker-controlled URL and returned the upstream response body. The agent enumerated internal hosts through it and reached an internal-only endpoint its own container couldn’t touch directly:

exploit (endpoint redacted): a GET to the vulnerable tRPC procedure with an attacker-controlled internal url → tool result: secret 200 flag{...}

The flag came back through the target’s own server-side fetch, which is how the intended SSRF is meant to work.

Signal quality: We measured our own false positives

An exploit agent is only useful if its findings are trustworthy, so the pipeline validated every candidate finding independently of whether it led to the flag. The following counts are the validator’s own labels for the full run.

Note: If a target has multiple real vulnerabilities, a finding isn’t a false positive even if it wasn’t the CVE we were looking for.

Finding validation outcome	Count
Confirmed exploitable (HIGH confidence)	234
Lower-confidence (medium, low, or unconfirmed)	82
Auto-labeled false positive by the validator	159
Additionally rejected as duplicates	1,202

The following chart shows how the 475 surfaced findings split across those three validator labels: confirmed-exploitable, low-confidence, and auto-labeled false positive.

Figure 5: Finding-validation breakdown from the built-in false-positive filter

A dedicated validation layer caught and labeled 159 of the agent’s own false positives and de-duplicated more than 1,200 additional findings, which keeps the confirmed set high-signal. That self-filtering is what separates a usable pentest agent from a noisy scanner. Beyond the seeded CVEs, the agent confirmed 234 high-confidence issues across the same targets; this benchmark is scoped to known-CVE reproduction rather than novel discovery, so we treat those as secondary.

We audited every win, and it caught two potential errors

Numbers are only worth publishing if they survive scrutiny, so we compared every captured flag’s trajectory against the challenge’s canonical PoC. Two captures didn’t pass review, and we reclassified both as unsolved:

The SSRF challenge. Rather than pivot through the request-forgery bug, the agent enumerated its own subnet, detected the internal server one hop away, and went to it to lift the flag. Resourceful, but it bypassed the vulnerability under test. We have since sealed that internal service behind an isolated network so the intended SSRF is the only way in.
The SQL-injection challenge. Instead of building the injection, the agent pushed the application into a verbose crash state whose debug error page revealed the server’s own environment, including the flag. Effective, but not part of what was under test. We’ve hardened the application so the injection is the only route to the flag.

The model wasn’t failing. Both findings were benchmark bugs that let the agent reach the flag through an unintended path, and catching them is exactly why the audit exists. This is the honest 33 of 37.

When the agent breaks what it attacks

The most significant result is the two misses. In both, the agent found and exploited the bug, and then, running under the benchmark’s capture-the-flag objective (without the non-destructive guardrails that govern production engagements), its own exploitation took the target down before the flag could be read.

It bricked the app. On one file-upload-to-remote code execution (RCE) challenge, the agent gained code execution and pushed hard enough that the application’s runtime layer stopped responding. Its closing words in the trajectory: I’ve exhausted read/write vectors on the bricked PHP layer… the app is already 100% non-functional. It had the access; it no longer had a live app to read the flag out of.
It knocked the target offline. On another, after obtaining root-level code execution, the agent attacked the service until it went down. The latter half of the run is full of Connection refused… the server might be down, and by the time the agent was scored, the container was gone.

That is a genuinely mixed result. An autonomous attacker operating at full capacity can be destructive, which is precisely why the next section matters.

Reliability and guardrails: keeping a powerful agent safe

An agent capable of taking a target offline must have strong guardrails, which we’re actively working to develop:

Non-destructive exploitation. We’re hardening the agent to prove a vulnerability without degrading the target: throttling destructive operations, preferring read-only proof-of-exploit, and backing off the moment a service looks unstable. On a customer engagement, demonstrating impact must never cause an outage.
Runtime reliability. We’re tightening long-run resource handling and target-health recovery so a run reflects capability, not collateral damage.
Guardrails and scope enforcement. Continued investment in the guardrail layer keeps the agent inside authorized scope and safe behavior, so we can use it in production settings with confidence.

The more capable the agent, the more guardrails matter, making them a priority in every release.

Why this matters

Benchmark scores are only worth as much as the method behind them. Three properties of this method are what make the 89.2 percent capture trustworthy and worth acting on.

Exploitation, end-to-end. Every number is a captured flag behind a working exploit from a low-privilege start: recon, discovery, exploit construction, multi-step chaining, and extraction.
Contamination-reducing. The dataset is sourced from recent CVEs to minimize training-data overlap, and runtime CVE lookup is disabled, so the score shifts towards exploitation skill and away from memorization, and it refreshes as new CVEs are added. The two controls reduce contamination rather than eliminate it.
Honest by construction. Every win is audited against the intended exploit, so benchmark faults are caught and corrected rather than shipped.

89.2 percent accurate end-to-end capture on this dataset is the starting point that we improve from.

What’s next

We’re working to continuously improve the benchmark.

Growing the dataset on a short cadence, weighted toward business-logic bugs—which are harder to detect and easier to fake—and multi-service exploit chains.
Developing new techniques to further address contamination.
Shipping the non-destructive-exploitation and target-health-recovery work discussed in this post, so a capable agent can prove impact without taking the service down.
Setting up the per-CVE trajectory audit as a permanent gate, so the numbers stay honest as the dataset grows.

If you have feedback about this post, submit comments in the Comments section below.

Penetration testing every deployment with AWS Continuum

2026-10-14 Jigar Bhatt

Post Syndicated from Jigar Bhatt original https://aws.amazon.com/blogs/security/penetration-testing-every-deployment-with-aws-continuum/

Most security tooling that fits into a CI/CD pipeline—static analysis (SAST), software composition analysis (SCA), and dynamic scanners (DAST)—flags potential issues from patterns and signatures. A flagged pattern shows you where to look; penetration testing is what confirms whether something is actually exploitable. That confirmation has traditionally come from a periodic, out-of-band engagement that runs on a schedule rather than on a deploy, so deploys between engagements reach production with that confirmation still pending.

AWS Continuum for penetration testing (formerly AWS Security Agent) is a service that uses AI agents to find and confirm exploitable vulnerabilities in your running applications. In this post, we walk you through making it a post-deployment gate in your pipeline. The gate runs a penetration test that is automatically scoped to the change you just deployed, and it can block promotion to production when a finding is at or above a severity threshold you choose. Because each run tests only the change since the last passing gate rather than the whole application, it is fast and economical enough to run on every deploy. We give a high-level overview of the pattern, which can be applied on four major CI/CD platforms: GitHub Actions, GitLab CI/CD, Bitbucket Pipelines, and Azure DevOps Pipelines. We then provide a specific worked example for GitHub Actions.

By the end, you will have a pipeline that deploys a change to a staging environment, runs a scoped penetration test against the deployed change, and promotes the change to production only when the gate passes.

The following diagram shows the end-to-end flow. Your CI/CD pipeline deploys to staging, then the pentest gate step authenticates to your AWS account through OpenID Connect (OIDC) and calls Continuum for penetration testing, which runs the scoped test against the staging environment. The gate waits for the result, reports the outcome back to your pipeline, and conditionally blocks promotion to production. All testing runs server-side in your AWS account, so no penetration testing logic runs on your CI/CD runner.

Figure 1: Pipeline gate architecture: Your CI/CD pipeline deploys to staging, the pentest gate authenticates via OIDC and calls Continuum for penetration testing, which returns a scope decision that gates promotion to production.

Why scoping to the change matters

What makes the gate practical to run on every deploy is that it scopes each test to the change you just deployed. Continuum for penetration testing determines the commit range—the head is the commit being gated, and the base is the previous commit the gate already cleared—and tests only what changed. Testing the change instead of the whole application is what keeps a per-deploy penetration test fast and cheap enough to belong in a pipeline rather than on a quarterly calendar.

Each run returns one of three scope decisions: in scope (the change modified the attack surface—findings are evaluated against your threshold), scoped out (the change was within scope but nothing security-relevant changed—the pipeline passes), or scope conflict (the change fell outside the configured scope and was not tested—you choose whether that blocks). The Continuum for penetration testing console records the decision and the commit range on the Scoping tab of each run. When a gate blocks, you can see exactly why. For the full mechanics of how the commit range is computed and validated, see How the pipeline integration scopes the test.

Because the gate blocks on findings that Continuum for penetration testing validated as exploitable rather than on pattern matches, a red pipeline carries a clear meaning: a reproducible vulnerability in the change, waiting to be fixed before promotion.

Setting up the gate

The gate takes three steps to set up: configure a penetration test, create an AWS Identity and Access Management (IAM) role that your pipeline can assume, and add the gate step to your pipeline.

Before you start, you need an AWS account with permission to create both an IAM OIDC identity provider and an IAM role, and a staging environment the penetration test can reach.

Step 1: Configure the penetration test

Complete the one-time setup described in Before you start: verify your target domain, create a penetration test in your Agent Space, enable CI/CD mode, and connect your repository. Note the Agent Space ID (as-…) and penetration test ID (pt-…), because you pass both to the gate step in Step 3. The pipeline tests whatever this penetration test is configured to test, so its target and scope set what the gate covers.

Step 2: Create the IAM role

Authentication uses OIDC federation rather than stored keys. Your CI/CD provider issues a short-lived token identifying the pipeline, and AWS Security Token Service (AWS STS) exchanges it for temporary credentials when the pipeline assumes an IAM role you create. There are no long-lived AWS access keys to store as CI/CD variables, which removes the credential most often leaked from a pipeline.

The security of the gate rests on two policies: the IAM role’s trust policy (which pipelines can assume the role) and its permissions policy (what the pipeline can do once the role is assumed). For the exact policy JSON, the provider-specific OIDC URLs and audience values, and the sub claim format for each CI/CD platform, follow the reference in Scope the trust policy to your pipeline and Grant least-privilege permissions.

The following judgment calls help you decide how to scope these policies.

How tightly to scope the trust policy is the first decision. Pinning the sub claim to a single branch (for example, main) is the right default when a branch is the only one that deploys through this pipeline. If several branches or environments legitimately run the gate, scope to the environment instead of the branch where your provider’s claim supports it—and remember that not every provider carries a branch in the claim at all. On Bitbucket, the repository (and optionally a deployment environment) is as tight as it gets. Start as narrow as your deployment reality allows and widen only when a real branch or environment needs it. Note that GitHub repositories created or renamed on or after July 15, 2026 emit an immutable sub claim with numeric IDs—confirm the exact sub your pipeline emits before you write the policy.

Step 3: Add the gate step to your pipeline

The step is the same idea on every provider—the syntax differs. Pin the integration to an immutable version (a release tag, commit SHA, or pinned component version) so an upstream change can’t alter what runs in your pipeline. For the full settings reference and defaults table, see Defaults the integration ships with.

Before relying on the gate, run it once in dry-run mode. A dry run checks authentication, the Region, and that your repository resolves—without starting a scoped penetration test job. If validation fails, the step fails regardless of your error-handling settings, so a misconfiguration surfaces immediately.

A worked example: GitHub Actions

The following workflow shows the three-job topology that makes the gate work: deploy deploys to staging, security-pentest runs the gate, and deploy-prod promotes to production. Because deploy-prod declares needs: [security-pentest], GitHub skips production whenever the gate fails.

Note: The tutorials use an application with a deliberate SQL injection so the gate has a real finding to block on. Deploy it only to an isolated, non-production environment and remove it when you finish.

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false
      - run: ./deploy.sh staging

  security-pentest:
    needs: [deploy]
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: write
      actions: read
      deployments: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/my-pentest-gate-role
          aws-region: us-west-2
          role-duration-seconds: 7200
      - uses: aws-actions/aws-continuum-run-pentest@v1
        with:
          agent-space-id: as-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
          pentest-id: pt-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
          severity-threshold: HIGH
          on-scope-conflict: block
          timeout-minutes: "45"

  deploy-prod:
    needs: [security-pentest]
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false
      - run: ./deploy.sh production

Replace the account ID, role name, Region, and the as-/pt- identifiers with your own values.

Two things in this example deserve attention. First, the contents: write permission lets the gate advance the baseline marker it keeps as a ref in your repository—without write access, every run would test a wider range than intended. Second, the role-duration-seconds: 7200 on the credential step must outlast the penetration test; AWS STS rejects a request for more than the role allows rather than reducing it, so set the role’s maximum session duration to at least (timeout + 10) x 60 seconds. A credential that expires partway through makes an otherwise-healthy run fail as though the service were at fault and leaves the cleanup unable to stop the abandoned, still-running job.

The full step-by-step tutorials for all four providers are linked at the end of this post.

Rolling out the gate: from report-only to blocking

The single most important detail before you roll this out.

Three independent settings decide whether the step fails, and each covers a different outcome: severity-threshold (what happens to findings), on-scope-conflict (what happens when the change wasn’t tested), and fail-on-infrastructure-error (what happens when the job couldn’t run). They are independent—relaxing one does not affect the others.

Most teams introduce the gate in report-only mode first—set threshold to NONE so the test runs and publishes findings without blocking anyone’s pipeline. Watch what it finds for a release or two, then tighten the threshold to HIGH. This staged rollout is sound, with one critical caveat:

A run that passes—including a report-only run—advances the baseline. The commits in that run’s range are then recorded as cleared and will not be re-tested when you later tighten the threshold. If you run report-only for a month and then switch to HIGH, the gate only assesses commits after the last report-only run—it does not go back and recheck what it already reported on.

This is by design: relaxing a setting is you accepting the outcome it allows. But if you want a stricter setting to reassess a range an earlier run cleared, override the head and base commit SHAs by setting base-commit-sha and head-commit-sha. Where the marker lives differs by provider; see What advances the baseline, and why that matters for the full matrix.

One default also deserves attention: fail-on-infrastructure-error ships as false (fail-open). For pipelines where promotion requires security sign-off, set it to true from the start.

GitLab: your committed settings are defaults, not a contract

On GitLab CI/CD, a pipeline variable overrides the value a job sets in its own variables: block. That means anyone who can run a pipeline could supply a variable that lowers the threshold, turns on dry-run, or relaxes error handling—and the gate does what the variable says, not what your .gitlab-ci.yml says. Restrict who may override variables in your project’s CI/CD settings before you rely on the gate to hold a release.

Protecting credentials on the runner

The temporary credentials and OIDC token are sensitive for the duration of the step. Three practices matter:

Keep debug and verbose tracing off for steps that handle credentials—shell-trace modes (for example, GitLab CI’s CI_DEBUG_TRACE) echo command arguments to the job log in plaintext.
Don’t persist source control credentials in the workspace—set persist-credentials: false on the checkout step so a token isn’t written to the runner’s .git configuration where a later step could read it.
Don’t expose the token to untrusted steps—grant OIDC-token permission only to the jobs that need it, and don’t run untrusted code (such as a fork build) in a job that can read the token.

Taking this to production and multiple accounts

The example uses one account for clarity. In practice, your staging and production environments are often separate AWS accounts. The pattern extends cleanly:

Create the pentest-gate IAM role in the account where the penetration test and Agent Space live—the security account. Your pipeline assumes that role via OIDC regardless of which workload account it deploys to, so you keep one well-scoped gate role rather than a copy or a set of long-lived keys in every account.
Run the gate against each pre-production environment you promote through, and gate promotion to the next stage on its result. Because the baseline advances only on a passing gate, coverage stays continuous across deploys.
Set fail-on-infrastructure-error to true and restrict who can override pipeline variables once the gate is holding real releases.
Pin the integration version so upstream changes are reviewed before they run.

For broader guidance of AWS Security Agent—testing against non-production environments, validating AI-generated findings, and IAM—see Security best practices for AWS Continuum.

Cleanup

If you followed one of the provider tutorials to try the gate, complete its Clean up section to remove everything it had you create: the sample application’s staging environment, the IAM role and OIDC identity provider, and—if you created one specifically for the walkthrough—the penetration test (and dedicated Agent Spaces) in the Continuum for penetration testing console. Removing them avoids ongoing charges and takes the intentionally vulnerable sample application offline.

Conclusion

Scoping penetration tests to the deployed change is what makes them practical as a per-deploy gate rather than a quarterly exercise. The key decisions you now have are:

How strict: Start report-only (NONE), but know the baseline advances—delete the marker when you tighten.
How to fail: Set fail-on-infrastructure-error to true for anything that requires security sign-off.
How to scope trust: Pin the sub claim as narrowly as your deployment reality allows.
Where to put the role: One role in a security account, assumed from workloads pipeline via OIDC.

To build the gate end to end on your platform, follow the worked tutorial for your provider:

For the concepts and full reference, see Run penetration tests from your CI/CD pipeline.

If you have feedback about this post, submit comments in the Comments section below.

Reliable GPU compute for large training & inference jobs

2026-10-14 Netflix Technology Blog

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/reliable-gpu-compute-for-large-training-inference-jobs-d62b03a96d7a

Authors: Nick Rutigliano, Argha C, Samartha Chandrashekar

How we detect failing GPUs and network fabric, remediate machines automatically, and let long-running jobs survive hardware loss.

Open Netflix, and the home screen is assembled for you: which rows appear, the order they appear in, the titles within them, and the artwork for each title are chosen by machine learning models trained on how hundreds of millions of members watch. Personalization is one of the most widely used things we build, and members feel it every time they open the app. Like a growing share of Netflix, it runs on GPUs, which train those models and increasingly serve them live.

Those GPUs have to stay healthy for that experience to hold up, and keeping them healthy is harder than it sounds. A single large training job can run for weeks across hundreds of GPUs spread over many machines. It runs as a gang: the ranks move in lockstep, exchanging data every step, so the job advances only as fast as its slowest participant and stops entirely if one participant dies. Every GPU, every high-speed network adapter, and every interconnect must keep working throughout the run. At the scale at which modern machine learning runs, individual GPUs and network links can fail during a run, so the underlying platform must be prepared for this and handle it without a person in the loop.

This post describes how we keep our GPU infrastructure healthy for training and inference. We cover how we detect failing hardware before and during jobs, how we automatically remediate a bad machine, and how we let a long job survive the loss of a machine. These training workloads run on our modern Kubernetes-native Compute platform.

Why GPU reliability matters

Netflix aims to entertain the world, and machine learning has been part of how we do that for years, from the recommendations that help members find something to watch to the systems that help produce and promote titles. As generative AI advances, we are investing in it on both sides of the business: for members, through better recommendations and higher-quality promotional assets; and for storytelling, through creative tools for our production partners.

All of this runs on GPUs, for both training and inference, and GPU capacity is among the fastest-growing and most expensive parts of our infrastructure. That makes reliability a business concern on two fronts. — -

Availability: a training run that loses a machine, or an inference service that cannot get healthy capacity, delays work that members and creators depend on.
Cost efficiency: a GPU sitting idle behind a slow neighbor, or a whole machine pulled from the pool because one device is degraded, becomes capacity we paid for and are not using. Higher reliability and higher utilization mean more useful work per GPU, which is equivalent to spending less to do the same work.

Inference is a growing share of that GPU footprint as more of these AI features reach members, so the reliability and efficiency that matter for multi-week training runs increasingly matter for always-on serving too.

Why basic health checks are not enough

We start from the baseline provided by the cloud provider. Cloud providers run instance status checks on virtual machines: a system status check for the underlying host’s power, network, and hardware reachability; an instance status check to verify that the operating system is up and reachable; and a check on attached storage. It emits scheduled maintenance events when it plans to retire or reboot a host, and it can automatically recover an instance that becomes impaired. We use all of these, and the maintenance events feed one of our own checks. They catch a machine that is down, unreachable, or slated for maintenance, and they stop at the instance level. They confirm the box is up and reachable. They do not verify that each GPU, its on-chip memory, its interconnect, and its network fabric are healthy enough to support distributed training.

The next step most teams take is a GPU-level check to confirm that the driver responds and that the GPUs are visible. That catches a dead or missing GPU, but misses the cases that take the most time.

A GPU can answer a status query and still be faulty once a workload touches it. A fault in on-chip memory or in the firmware that manages the device can remain invisible until a job allocates memory or runs sustained compute, at which point the job fails or slows.
A high-speed network adapter may be present but show fewer links than expected. Collective operations still run, at a fraction of the bandwidth, so the job does not crash. It crawls, and the cause is easy to miss.
A check can fail in ways that leave a faulty machine looking healthy. The diagnostic tool can hang when the driver hangs, and a naive check that waits on it hangs too. A binary that the check depends on can go missing after a driver upgrade, causing the check to stop running and report nothing. A check that can silently stop running is worse than no check, because it gives false confidence while the machine keeps working.

Because a job runs as a gang, any one of these on any one machine can degrade or stop the whole job, and none of them shows up in an instance-level check or a driver-responds check. The health system has to assume the hardware can lie and the tools that inspect it can break.

*A machine can pass a basic check and still be faulty. Layered checks, timeouts, a fallback path, and checks that fail when a dependency is missing close these gaps.*

Our approach

We treat reliability as four layers, each catching what the layer below cannot:

1/ Node and hardware health: find bad GPUs, network adapters, interconnects, memory errors, and thermal issues, then taint or recycle the machine.

2/ Scheduler and platform: retry at the task & job level, validate a machine before it takes work, and drain & recycle machines safely.

3/ Training loop: checkpoint and use elastic workers so a job survives losing a machine.

4/ Observability: metrics, service-level objectives, and failure attribution across the other three.

The first two layers keep hardware problems away from jobs. The third accepts that some loss is unavoidable and makes it cheap. The fourth tells us whether the first three are working.

*Four layers: Each catches what the layer below cannot.*

How the health checks work

Checks run in two tiers. Device-level checks look at each GPU. A node-level agent, built on the open-source Kubernetes node-problem-detector, runs on every GPU machine every 30 seconds and turns check results into conditions on the Kubernetes node, the same mechanism Kubernetes uses to mark a node NotReady. Controllers watch those conditions and decide whether the machine can take on work.

GPU device checks are built on NVIDIA’s DCGM for driver-reported error codes, uncorrectable memory errors, interconnect integrity, and thermal state, and networking adapter checks inspect the host state for the count and link state of the Elastic Fabric Adapter interfaces that carry cross-machine traffic. A few of these checks deserve detail.

*Two-tier health checking. Device-level and node-level checks become Kubernetes node conditions, which controllers then turn into taints, drains, terminates, and replaces.*

Failing closed: if the driver or the diagnostic tool is unresponsive, the check times out and marks the machine unhealthy. A check that cannot reach the hardware reports a problem rather than passing, so a wedged driver takes the machine out of rotation rather than letting it hide behind a hung tool.
A fallback path: driver error codes are the strongest signal that a GPU has faulted, but the tool that reads them can itself be broken in certain failure modes! When it does, we read the kernel log dmesg directly for those codes, so no Xid or SXid errors are missed.
Our own fatal-error map: vendor driver error catalogs lag newer GPU generations and do not always match what we see in production, so we keep our own mapping of which codes are fatal for each GPU family. A code that means reset-the-GPU in one generation can be benign in another, and one GPU generation emits a burst of normally-fatal code on every healthy boot, which we learned to treat as noise rather than drain fresh machines.
No silent no-ops: a check fails when a required dependency is missing. This one came from an upgrade that dropped a binary the network-adapter check relied on. The check quietly stopped running, and machines with the wrong number of adapters shipped for days before the slow jobs were traced back to them. Now a missing dependency fails the check.

Turning a bad machine into healthy capacity, automatically

When a check fails, the machine’s node condition changes, and controllers take over. We distinguish two levels. A problem condition blocks new work from landing while jobs are running, because pulling a job off a machine is disruptive, and many problems clear on their own. If the condition persists beyond a tolerance window, it escalates to a failure condition, which evacuates the machine, drains its tasks (overriding the normal disruption limits to move quickly), terminates it, and replaces it.

Karpenter node auto-repair is used on our compute stack. We have customized it with a policy and a controller that maps each node condition to the duration we tolerate it before acting, so a fast-fatal error is actioned within minutes. At the same time, a flaky signal gets a longer window, and a cap limits how much of a cluster we will disrupt at once, so a bad rollout of the checks themselves cannot cascade into tearing down a whole cluster. We plan to upstream this feature to all Karpenter users.

*Graded remediation. A problem is tolerated and may clear; a persistent failure escalates to evacuation and draining, then to a fast in-place fix or termination for confirmed hardware faults.*

Not every fault requires a full replacement, and replacement is the slowest way to restore capacity. We match the action to the fault. Killing the offending job clears a software wedge. Resetting a single GPU clears many transient device faults. Rebooting the machine clears the firmware state that has drifted over a long uptime. Terminating and replacing is reserved for hardware that fails a test we trust. For that last case, we record the GPU’s immutable hardware identifier, which stays with the physical device across reboots and relaunches even when the instance changes, report it to the cloud provider, and keep our own list, so a device that fails repeatedly does not quietly rejoin the pool under a new instance.

One subtlety sits between detection and scheduling. A freshly checked machine was handed to the scheduler just as its checks were still settling, so a job could land on it a moment before a slow check flagged it. We now hold capacity until the checks have run and settled, which closed that window.

Validating a machine before it takes work

A machine that just booted is not trusted until it proves itself. Before we release its capacity, it passes system checks, confirms the GPUs and their topology are present, and runs a round of GPU diagnostics. We then wait for the health checks to run twice, because the longer checks, such as counting network adapters, take more than one cycle to fire, and releasing after a single pass would let work land before those checks are reported. Only then do we remove the startup taint. On teardown, we run deeper diagnostics and record the GPU identifiers, which feed the same bad-hardware list, so hardware that failed once does not silently cycle back into service.

Letting long jobs survive machine loss

Even with all of the above, some machines are lost mid-job, so we make that cheap. Jobs checkpoint their state, and on a lost machine, they resume from the last healthy checkpoint rather than restarting from the beginning of a multi-week run. Workers are elastic and retryable, and the scheduler automatically retries platform-caused failures.

Attribution is what makes automatic retries safe. If we retried every failure, we would mask genuine bugs in a job and burn GPU hours looping on an error the platform cannot fix, so we classify each failure as platform or application before retrying. One example of why this matters: a class of timeouts in NCCL, the collective communication library GPUs use to exchange data, appeared to be a network fault and was traced to data-loading stalls at high worker counts. The GPUs and the network were fine, and the fix was in the data pipeline. Treated as hardware failures, those timeouts would have cycled healthy machines without fixing the cause.

Measuring whether it works

We track GPU reliability along three service-level objectives, each split by cause, so a miss points at the layer responsible:

Availability: the amount of GPU capacity available for scheduling at any given moment.
Job-start success: whether a job starts promptly once its capacity is available.
Job success: whether a job finishes without a platform-caused failure.

Each objective is split into hardware, driver, platform, and user causes. Hence, an availability dip from bad hardware reads differently from one caused by a scheduler change or a user misconfiguration. We optimize for goodput, the useful work the fleet completes, because a job held back by one slow GPU can look healthy on an uptime chart while getting almost nothing done. Specific targets for each objective are provided here for illustrative purposes.

A closed remediation loop

These pieces form a loop. Hardware is validated at allocation, before a job starts, with a short collective communication run and pairwise network tests. Hence, any machine that fails within the first minute fails before the job is admitted. During the job, we watch passively for driver errors, memory errors, thermal events, and more. We view each job as a distributed system, which lets us identify a single slow rank dragging the rest, rather than only whole-machine failures. When something is wrong, remediation is automatic and graded. The loop is the same whether a job runs through our batch scheduler, Kueue, or Ray.

*The reliability loop: validate, watch, actively check, view the job as a system, remediate, then repeat.*

Sharing GPUs safely

Not every workload needs a whole GPU, and leaving small inference jobs on dedicated GPUs wastes capacity. We partition GPUs with MIG for memory-level isolation between tenants, so a fault or a memory hog in one partition does not affect another. We also reconfigure those partitions without restarting the machine, so that a pool can shift between many small slices and a few large ones as demand changes. GPU and network adapter allocation is device-plugin-based today, to move to Kubernetes Dynamic Resource Allocation once it is supported in Karpenter. This matters because a GPU talking to a network adapter or working with a CPU on the wrong PCIE root of the machine incurs a bandwidth penalty.

*On a multi-GPU machine, we isolate a single faulty GPU, keep the rest in service, and partition healthy GPUs across multiple tenants with memory-level isolation.*

Device-level allocation also changes how we handle a single bad GPU. With DRA support, instead of losing the whole machine, we could isolate the one faulty device and keep the others serving, which matters most for inference, where many small jobs share a machine. And because allocation is expressive, a job can request the capabilities it needs, such as memory or tensor-core support, and the platform matches a suitable GPU type, so users are not pinned to a model they had to choose in advance.

Results

Moving from basic checks and manual draining to layered checks and automatic remediation changed two things users feel. More of the GPU fleet is available for scheduling at any time because bad machines are caught and cycled quickly, rather than sitting in the pool or waiting on an on-call engineer. And fewer jobs fail due to platform issues because most faults are caught before a job lands, and the rest are automatically remediated and retried. Graded remediation also returns capacity in minutes for faults that only require a job kill, a GPU reset, or a reboot, rather than the longer wait for a replacement machine.

Since the above improvements have landed, we’ve seen:

Consistently achieving >99.95% availability of our GPU nodes
Job success rate has gone from ~98% to above 99.5%

What is next

We are extending active checks to cover additional failure modes under load, widening burn-in so more classes of failures can be cleared before a user encounters them, and making remediation more granular so more error types are cleared in place rather than requiring a full node replacement loop for certain Xid errors. Additionally, we want to do more periodic diagnostics between jobs.

We are working to make placement more topology-aware using Kubernetes Dynamic Resource Allocation (DRA) once it lands in Karpenter, for example, by aligning on Compute/IMEX Domains, PCIe roots, and NUMA nodes.

More on the performance side for multi-GPU training, we plan to run a custom Linux scheduler built on sched_ext. Firstly, we can prioritize processes during collective coordination, such as when GPUs exchange results before the next round of compute, thereby reducing overall GPU idle time by establishing a fleet-wide priority for these tasks. The effect is small on one machine but grows across hundreds of machines, where a few percent of throughput amounts to a large amount of capacity. Secondly, we can prioritize data-loading processes to minimize the bottleneck that sometimes occurs when data is not loaded onto GPUs quickly enough to maintain steady progress.

While we already monitor throttling events, GPUs do not have to report throttling for it to hurt a job. One running ten to fifteen degrees hotter than its peers, even within healthy limits, sets the pace for the gang, so we can treat a large temperature gap as a fault, stop the job, and move it to cooler GPUs.

Additionally, we plan to expand our NCCL RAS (reliability, availability, serviceability) monitoring to identify NCCL issues sooner, such as mismatched collectives or hangs, to more quickly resume work.

Closing

Hardware fails, and at this scale, it fails often enough that the platform has to treat failure as normal. Detecting problems before and during jobs, automatically remediating machines, and allowing long jobs to resume after a loss lets teams run large-scale training and inference without babysitting the hardware. If you like these kinds of problems, we are hiring.

Reliable GPU compute for large training & inference jobs was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Configuring your AI vulnerability harness, Part 2: The steering file

2026-10-08 Justin Kontny

Post Syndicated from Justin Kontny original https://aws.amazon.com/blogs/security/configuring-your-ai-vulnerability-harness-part-2-the-steering-file/

This post shows you how to configure an AI model to perform structured, evidence-based vulnerability triage with the consistency of a seasoned security analyst. You’ll learn the design decisions behind five configuration sections that enforce structural verification, evidence-based scoring, and infrastructure-aware prioritization across every analysis session. Our companion post—Building your AI vulnerability harness—covered the architecture; this post covers the configuration.

The distinction matters because the same frontier model that produces a rigorous, evidence-grounded security assessment with one set of instructions will produce hallucinated attack chains and inflated severity ratings with another. The model’s capability is fixed. What you configure is its judgment.

We learned this the hard way while building the harness. Our early iterations produced findings that sounded authoritative but crumbled under scrutiny. The model would report a SQL injection in a function that didn’t exist, claim HIGH confidence based on nothing structural, or ignore an AWS WAF in blocking mode directly in the request path. The output wouldn’t have earned engineering trust in that state.

The fix wasn’t a better model. It was better steering.

What a steering file is

A steering file is a set of instructions that an AI coding assistant loads at the start of every session. Different tools use different conventions—Kiro reads markdown files from a .kiro/steering/ directory, Claude Code uses CLAUDE.md, Cursor uses .cursorrules, and GitHub Copilot uses .github/copilot-instructions.md—but the function is the same: persistent instructions that shape how the model approaches work in your repository.

Applied to security analysis, a steering file encodes your team’s triage methodology as machine-executable instructions. Instead of relying on individual engineers to apply that methodology consistently, you encode it once and the model applies it uniformly across analyses.

The steering file we’re releasing encodes the architectural principles from the harness post as operational instructions. Where the harness post says apply infrastructure-aware assessment, the steering file specifies exactly how: which controls map to which multipliers, how multipliers stack, and what floor prevents over-confidence in any single control.

Why configuration outperforms prompting

A single-turn prompt (find vulnerabilities in this code) produces whatever the model’s default behavior generates. That default is optimized for helpfulness, not rigor. The model will find things to report because you asked it to find things.

A steering file changes the default. It establishes:

What counts as evidence: Without explicit instructions, a model treats its own reasoning as sufficient evidence. The steering file requires structural verification—confirmed dataflow, confirmed function existence, and confirmed call path—before a finding is reported.
What confidence means: On its own, a model assigns confidence based on how plausible its narrative sounds to itself. The steering file replaces self-reported confidence with a formula computed from binary structural signals. Either taint analysis confirms the flow or it doesn’t. Either a call graph connects entry point to sink or it doesn’t.
How infrastructure affects priority: Left to its defaults, a model assesses vulnerabilities in isolation from their deployment context. The steering file requires parsing infrastructure as code (IaC) and applying control-specific multipliers before assigning priority.

The goal is reproducibility: the same codebase should produce the same assessed findings regardless of who runs the analysis or when. Consistency is the point.

The five sections that matter

The full steering file is available in the companion repository. In this section, we explain the design decisions behind its five critical sections.

Structural verification over LLM trust

Never trust an LLM's self-reported confidence about a vulnerability.
Verify claims against the code structure:
1. Do referenced files exist?
2. Do referenced functions exist?
3. Does dataflow confirm the claim?
4. Is there a call path?

If a finding fails all structural checks, reject it regardless of how
convincing the narrative sounds.

This is the single most important instruction in the file. Without it, the model generates plausible-sounding attack chains that reference functions that were renamed three commits ago, files that exist in a different package, or dataflows that pass through sanitization that the model failed to notice.

We found that approximately 30% of unsteered model findings referenced code structures that didn’t exist in the target repository. Not wrong interpretations—entirely fabricated paths. The instruction to verify before reporting produced zero fabricated paths in our testing.

The key insight: hallucination in security analysis isn’t a minor annoyance. One hallucinated finding that reaches a developer destroys the can quickly erode the credibility of your entire pipeline. Engineers who encounter a fabricated vulnerability are less likely to keep reading the reports.

Evidence-based confidence scoring

The following formula replaces the model’s intuitive confidence with a score computed from observable signals. Each factor is binary: confirmed or not confirmed.

confidence = 0.30 * taint_confirms_flow
           + 0.25 * no_sanitization_found
           + 0.20 * entry_point_is_public
           + 0.15 * call_graph_confirms_path
           + 0.10 * sink_type_matches_category

We weighted taint confirmation highest (0.30) because a confirmed dataflow path—evidence that user-controlled data reaches a security-sensitive operation—is the single strongest predictor that a finding is real. In our validation against known-vulnerable applications, findings with confirmed taint and no sanitization were exploitable 89% of the time. Findings where the model reported HIGH confidence without structural confirmation were exploitable 34% of the time.

The formula is a starting point, not a fixed standard. Your weights should reflect your own validation data. Run the formula against a labeled dataset of known true and false positives, then adjust until precision matches your team’s tolerance for noise.

Infrastructure-aware assessment

This section addresses a common category of over-prioritization: reporting a vulnerability at full severity when compensating controls are already in place.

score_after_controls = max(0.15, score * block_1 * block_2 * ... * block_n)

Controls are mapped to specific vulnerability classes. An AWS WAF with SQL injection rules in blocking mode is a STRONG control (0.25x multiplier) against SQL injection. It is irrelevant to deserialization attacks. The steering file specifies this mapping explicitly rather than leaving the model to guess.

The multiplicative stacking of attack-blocking controls with a floor of 0.15 encodes defense-in-depth thinking. Multiple controls reduce risk more than any single control, but no combination of controls reduces risk to zero. The floor prevents the model from concluding that a vulnerability is completely mitigated and can be ignored.

A subtlety we learned through validation: access controls (authentication and authorization) and attack-blocking controls (AWS WAF rules and input validation) serve different functions. Authentication limits who can attempt an exploit. It doesn’t limit what happens when an authenticated user attempts one. A command injection behind AWS Identity and Access Management (IAM) authentication is still a command injection; the attacker pool is smaller, but the impact when exploited is unchanged. The steering file accounts for this by mapping control types to specific score components rather than applying a blanket multiplier.

Threat intelligence integration

The steering file incorporates signals from the CISA Known Exploited Vulnerabilities (KEV) catalog and the Exploit Prediction Scoring System (EPSS) to boost priority when threat intelligence indicates active risk.

The following table lists the primary signals the steering file recognizes and the boost that each signal adds to a finding’s score. Each signal answers a different question about real-world exploitation risk:

CISA KEV – Indicates that attackers have exploited the vulnerability in the wild and notes whether it is tied to ransomware campaigns.
EPSS – Estimates the probability that attackers will exploit a vulnerability in the next 30 days.
Public proof of concept (PoC) – Indicates that working exploit code is already published, so an attacker has little work left to do.

Signal	Boost
Actively exploited (CISA KEV)	+0.30
Ransomware-associated (CISA KEV)	+0.15
EPSS >= 0.5	+0.20
Public PoC exists	+0.10

There’s a cap at +0.50 total boost to prevent threat intelligence from dominating the score. A vulnerability isn’t more technically exploitable because attackers are targeting it, but it is more urgent because the window between theoretically exploitable and actively exploited can be very short.

We separate urgency from severity deliberately. A medium-severity dependency vulnerability under active exploitation by ransomware groups demands a faster response than a critical-severity code-level finding that requires complex preconditions and has no known tooling.

Priority classification

Scoring produces a number and priority classification turns that number into a decision, which is what a developer on the receiving end of a finding needs.

The following table shows the four action tiers that the steering file defines. The score and evidence thresholds determine which tier a finding falls into, and each tier prescribes a specific response. The steering file compares these thresholds against the final score: the base score after it applies attack-blocking multipliers and any threat-intelligence boost. That final score is not the confidence value from the formula earlier in this post, and it is not the severity that the scanner reported.

Priority	Criteria	Action
P0	Score >= 0.8, no effective attack-blocking mitigation	Fix immediately
P1	Score >= 0.6 OR active threat intel	Fix this sprint
P2	Score >= 0.4, partial evidence	Investigate when capacity
P3	Score < 0.4 or effectively mitigated	Accept risk or backlog

The most controversial instruction in the file: Do not file P3 findings as security issues. We include it because filing low-confidence or fully mitigated findings as tickets trains teams to ignore security tooling. Each false positive or irrelevant finding filed as a bug reduces engagement with the system’s output. A pipeline that produces ten confirmed findings gets more engineering attention than one that produces two hundred maybes.

Validation results

We tested the steering file against a purpose-built vulnerable application containing ten known vulnerabilities: eight exploitable and two mitigated by infrastructure controls. The application includes an AWS WAF with SQL injection rules in blocking mode, Amazon Virtual Private Cloud (Amazon VPC)-isolated AWS Lambda functions, and IAM authentication on each endpoint.

Without the steering file, the model found all ten vulnerabilities and correctly identified both mitigated findings as lower priority. However, it assigned severity based on intuition (this is a command injection, so it’s CRITICAL) rather than evidence, and it produced no structural verification of its claims.
With the steering file, the model found nine of ten vulnerabilities (not flagging a hardcoded credential that had been intentionally embedded in unit-testing code and was defined but never called), correctly downgraded the two mitigated findings to P3 with documented rationale, showed its confidence calculation for each finding, and verified each claimed dataflow against the actual code.

The scoring calibration required one iteration. Our initial version applied authentication as a blanket multiplier that suppressed all findings behind IAM to P2 regardless of impact. We corrected this by separating access-limiting controls (which reduce the reachability score component) from attack-blocking controls (which reduce the full confidence score). The current version correctly classifies a command injection behind IAM authentication as P0: the auth limits who can reach it, but the impact when exploited is remote code execution regardless.

How to use the steering file

The three options that follow cover an always-on Kiro steering file, the same content as an on-demand Kiro skill, and how to carry the same content to Claude Code or another assistant.

Option 1: As a Kiro steering file

Drop the steering file into your repository at .kiro/steering/vuln-triage.md. Any session using Kiro’s built-in default agent loads the instructions automatically. When you or your engineers ask the model to analyze code for security issues, it applies the methodology without additional prompting.

If your team uses a custom agent rather than the default, steering files aren’t loaded automatically; you have to add the file to the agent’s resources explicitly:

{
    "resources": ["file://.kiro/steering/**/*.md"]
}

your-repo/
├── .kiro/
│   └── steering/
│       └── vuln-triage.md   ← steering file
├── src/
├── cdk/
└── ...

You can also place it in ~/.kiro/steering/ to apply the methodology across every workspace on your machine, rather than scoping it to a single repository.

Option 2: As a Kiro skill

If you want the methodology available on demand rather than consuming context window space in every turn, place it as a skill. Kiro loads only the skill’s name and description at startup and pulls in the full instructions when the skill activates. This keeps the instructions out of unrelated conversations.

A skill is a folder containing a SKILL.md file. The folder name must match the name in the YAML front matter:

name: vuln-triage
description: Structured vulnerability triage methodology with evidence-based scoring and infrastructure-aware prioritization. Use when analyzing code for security vulnerabilities.

your-repo/
├── .kiro/
│   └── skills/
│       └── vuln-triage/
│           └── SKILL.md   ← skill file
├── src/
└── ...

Kiro’s default agent discovers skills in .kiro/skills/ automatically; no configuration required. The skill then activates either automatically, when Kiro matches a request against the description, or explicitly as a slash command. A skill named vuln-triage becomes /vuln-triage, so an engineer can run an assessment on demand:

> /vuln-triage focus on the authentication changes

Place skills in ~/.kiro/skills/ to make them available across every project on your machine. If your team uses a custom agent rather than the default, skills aren’t loaded automatically, you must add them to the agent’s resources:

{

	"resources": ["skill://.kiro/skills/*/SKILL.md"]

}

Option 3: Adapted for other tools

The methodology isn’t specific to Kiro. The repository also ships the same content as a CLAUDE.md file for Claude Code, and the principles apply to any assistant that loads persistent instructions from your repository—consult your tool’s documentation for the filename and location it expects.

What this doesn’t replace

A steering file is methodology, not machinery. It tells the model how to think about findings but doesn’t provide:

Programmatic taint tracking: The model approximates dataflow by reading code. A purpose-built taint tracker confirms dataflow against the abstract syntax tree (AST) deterministically. For Python codebases, static analysis tools built on parsers like Tree-Sitter provide this. For other languages, the model’s approximation is reasonable but not authoritative.
Live infrastructure verification: The steering file instructs the model to parse IaC files. It can’t make API calls to verify that an AWS WAF rule is attached to the Amazon API Gateway in your deployed environment, or that a security group hasn’t been modified since deployment.
Threat intelligence feeds: The file instructs the model to consider CISA Known Exploited Vulnerabilities (KEV) and Exploit Prediction Scoring System (EPSS) data. The model’s training data includes historical KEV entries, but it can’t query live feeds. For current exploitation status, you need an API integration.
Execution-based verification: The steering file produces assessed findings. Confirming exploitability by executing a proof of concept against a live environment requires additional tooling: a sandbox, request signing, and differential testing infrastructure.

Each of these capabilities adds confidence to the pipeline. The steering file without them still produces measurably better results than an unsteered model: structured findings, cited evidence, and infrastructure-aware prioritization. But it represents the first layer of a mature harness, not the complete system.

What comes next

The steering file gives your team a structured methodology for AI-powered vulnerability triage today. It works with the tools you already have. It works without infrastructure changes, new services, or a procurement process.

The steering file is available in our GitHub repository. We encourage you to use it, adapt it to your environment, and let us know what you find.

The authors work on security automation at AWS. The patterns described here emerged from building and validating AI-powered vulnerability detection systems across multiple teams and deployment environments.

If you have feedback about this post, leave a comment in the Comments section below.

Building your AI vulnerability harness, Part 1

2026-10-08 Nidhi Ramakant

Post Syndicated from Nidhi Ramakant original https://aws.amazon.com/blogs/security/building-your-ai-vulnerability-harness-part-1/

Vulnerability scanners produce findings faster than manual triage can process them. Your developers ship more code with more dependencies, and the volume of candidate findings grows with it.

Many findings a scanner produces are unlikely to be exploited. The ones that matter need to reach an engineer fast, with enough evidence that they can act immediately rather than repeat the analysis. The challenge is separating signal from noise at the speed your pipeline demands.

This post shows you how to close the gap between detection and action. We built a three-layer pipeline that takes raw scanner findings and narrows them to a small, prioritized set with documented evidence of exploitability. The companion post, Configuring your AI vulnerability harness, will cover the steering file that drives the model’s behavior inside this pipeline. This post covers the layers around it.

Because the pipeline’s value comes from its filtering logic and evidence standards—not from any specific tool—you can substitute your own scanners and models without changing the architecture. Your AI provider, your infrastructure stack, and your scanner ensemble will differ from ours. The architectural decisions—what to filter at each layer, what counts as evidence, where to stop trusting the model—transfer regardless.

What a test harness is

A test harness is an automated framework that subjects a system to controlled inputs and observes whether outputs match expected behavior. Applied to vulnerability detection, the harness takes each candidate finding, constructs a test to evaluate exploitability, executes that test in a controlled environment, and records the evidence. The harness is the testing infrastructure, not the results it produces.

Traditional static application security testing (SAST) tools rely on pattern matching, with data-flow analysis varying by tool and language. They flag known-shape sinks well but struggle with multi-hop chains, business-logic flaws, context-dependent sanitization, and whether a sink is reachable from attacker-controlled input. AI-augmented analysis addresses that reasoning gap: tracing across files, inferring intent, and weighing deployment context.

The key conceptual distinction is that we don’t run a model against the codebase. The codebase is context passed to a model with a specific prompt. The model receives code and applies reasoning about how data flows, where trust boundaries exist, and whether security-relevant patterns are present.

Context scoping is one of the harder parts of this architecture because too much context can overwhelm the model’s reasoning window, while too little can produce analysis gaps that generate false negatives. For a basic AWS Lambda function, the relevant context might be a single file plus its AWS Identity and Access Management (IAM) role. For a complex microservice with shared libraries and layered infrastructure, deciding what to include requires understanding the application’s dependency graph. The model can reason about your deployment topology if you provide it; for example, by passing AWS Cloud Development Kit (AWS CDK) constructs or AWS CloudFormation templates as additional context. Starting narrow and expanding only when the model’s initial assessment is inconclusive generally produces better results than passing everything at once.

Building the pipeline

Security scanners can produce high volumes of findings each time they run. Many of these are false positives that look like vulnerabilities but aren’t real issues. To separate real problems from noise, the pipeline filters findings through three layers. Each layer applies a different type of evidence. By the end, a smaller number of prioritized issues backed by documented evidence remains.

Layer 1: Multi-scanner agreement

Plausibility comes first. Run multiple scanners independently against the same codebase: when two or more converge on the same finding, confidence increases. When only one scanner reports an issue and no other tool agrees, the pipeline flags that finding for extra scrutiny.

This approach—called multi-scanner agreement—is a low-cost way to increase confidence. It works with your existing tools and requires no new infrastructure.

Layer 2: Structural verification

The second layer verifies structure. AI-powered scanners describe how they think a vulnerability works: data enters through one function, passes through another, and reaches a point where it could cause harm. However, these descriptions can be wrong.

Before a finding moves forward, the pipeline verifies it against the actual code. It checks to see if the function that the scanner mentioned exists, that the file exists, and if data can flow along the path described by the scanner. If not, the pipeline rejects the finding.

This verification step uses the code’s abstract syntax tree (AST): a structured map of how functions, files, and data flows connect in your codebase. This step helps protect the credibility of your pipeline’s output. If unverified findings reach human reviewers, teams quickly learn to distrust the results, which reduces the security value of the tool.

Layer 3: Deployment context

The final layer adds deployment context. Vulnerabilities exist within the context of an architecture, which might include protective controls. These controls can include AWS WAF rules, network isolation, authentication requirements, and input validation.

This layer reads your infrastructure-as-code (IaC) templates—such as CloudFormation or Terraform files that define your cloud architecture—alongside your application source code. It then checks to see if an attacker can exploit a vulnerability by avoiding the controls that are in place.

Not all controls provide the same level of protection. A web application firewall rule that directly blocks the relevant attack technique provides stronger mitigation than one that addresses a different type of vulnerability. The pipeline treats control effectiveness as a spectrum, not a yes-or-no checkbox.

The result

The result is a short list of findings. Each one has cleared all three layers: it passed the plausibility check, its structure checks out against the code, and its deployment context indicates it can be taken advantage of. The number of findings decreases at each layer, and confidence in the remaining findings increases.

Principles

Four principles emerged from building this pipeline:

Progressive filtering with escalating evidence – Each layer demands a different kind of proof. The first layer checks that the finding is plausible. The second determines if it’s structurally real. The third analyzes if it’s exploitable in context. Using three focused layers works better than using one comprehensive layer, because each layer catches a different type of error.
Infrastructure context transforms prioritization – Without deployment context, you might assume that a critical-severity finding is more urgent than a medium-severity finding. However, a critical finding behind strong protective controls might be less urgent than a medium finding that’s directly exposed to the internet. To make this assessment, the pipeline needs access to your IaC templates, not only your application source code.
Independent agreement outperforms single-tool confidence – No single scanner reliably detects the full range of vulnerabilities. No single model produces correct results across all situations. When multiple independent tools agree on a finding, that agreement provides a stronger signal than one tool’s self-reported confidence score. This principle generally holds across tool choices.
AI-generated claims require verification before human review – Language models can produce outputs that sound plausible but might not be correct. Verifying AI-generated findings against the actual code structure—before a human reviews them—is what separates a useful system from one that erodes trust through confident-sounding errors.

How to apply this to your stack

You don’t need to build everything at once. Adopt them in order of immediate return:

Start with indexing – Build (or use) an AST-based index of your codebase that surfaces entry points, sinks, and authentication configuration. This costs nothing to run and gives you a map of your attack surface before any AI is involved.
Add multi-scanner triage – Take the output you already get from Semgrep, CodeQL, or equivalent tooling, and run it through the AST gate plus a large language model (LLM) exploitability classification. This delivers substantial noise reduction for a modest investment: you’re filtering output from tools you already run.
Add hypothesis generation – After triage is stable, add AI-driven discovery of vulnerabilities scanners miss: multi-hop chains, business-logic flaws, and cross-package data flows. This is the step where steering quality matters a great deal; see Configuring your AI vulnerability harness.
Add infrastructure context – Parse your CDK or CloudFormation, map controls to vulnerability classes, and apply the multipliers. This is where your prioritization becomes more targeted.
Add live verification – Run proofs of concept (PoCs) against a deployed pre-production target with structured success criteria. This step requires the most operational overhead, so run it last and only for findings that have already passed your confidence threshold.

Treat the pipeline as something you tune over time. Validate it against known true and false positives, adjust the weights, and rerun. Your results will change.

What this doesn’t solve

The harness is detection infrastructure. It helps you see which findings are likely real and which deserve attention first. It doesn’t:

Fix the code – Remediation is a separate problem. The harness produces findings with enough evidence that a human or agent can act on them; the act of remediation requires its own tooling and process.
Replace human review for final action – Even after three layers of filtering, the output is a prioritized list of likely-exploitable findings, not proven exploits. Priority 0 (P0) findings still warrant review by an engineer before they drive code changes.
Catch vulnerability classes outside the scanner ensemble’s detection profile – If none of your scanners look for a particular vulnerability category, the harness won’t surface it. AI-augmented hypothesis generation closes some of this gap, but novel or business-logic flaws still depend on what the model can reason about from code alone.
Verify live infrastructure state by default – Parsing IaC tells you what was defined. It doesn’t tell you whether a WAF rule got disabled last week, or whether a security group was modified after deployment. Live verification is an additional integration, not a property of the layers themselves.

Each of these limits points to where the architecture extends, not where it breaks. The harness is the first floor of a multi-story system, not the whole building.

What comes next

In this post, we showed how a three-layer pipeline—multi-scanner agreement, structural verification, and deployment context—narrows scanner output to a small, higher-confidence set of findings that your team can act on. The architecture is tool-agnostic; what transfers is where to filter, what counts as evidence, and when to stop trusting the model.

The architecture produces consistent results when the model receives consistent instructions. Our companion post, Configuring your AI vulnerability harness, covers the steering file that encodes this methodology as machine-executable instructions, including the confidence formula, the infrastructure control multipliers, and the verification gates that help prevent hallucinated findings from reaching your engineers. The same model that produces rigorous, evidence-grounded assessments with well-crafted instructions can produce inflated severity ratings and fabricated attack chains without those instructions. Configuration is what turns capability into judgment.

The window between disclosure and exploitation is narrowing. Embedding automated detection into your development workflows helps your team keep pace. We built this pipeline through trial and error and are sharing it so you can skip some of the mistakes we made.

For more information about the services mentioned in this post, see the AWS Lambda, AWS WAF, and AWS CloudFormation documentation.

If you have feedback about this post, leave a comment in the Comments section below.

Building an evidence-grounded agentic security operations harness on Cloudflare

2026-10-07 Deanna Tran

Post Syndicated from Deanna Tran original https://blog.cloudflare.com/agentic-security-operations/

Security alerts rarely arrive one at a time. A single alert can cause a spike across the environment, requiring a human analyst to decide which alerts are related and what they mean. When multiple arrive at the same time, it can quickly overwhelm even a seasoned security analyst. Enter the alert paradox. Now, our built-in, multi-AI-agent security operations harness can handle more of this work at Cloudflare scale.

Our Cloudflare Managed Defense AI agent harness speeds up the process of gathering data, connecting and aggregating detections, and accounting for missing sources while new alerts continue to arrive. To further analyze context, we make use of the OpenAI Daybreak Defense Network and our partnership with Anthropic. Cloudflare uses approved OpenAI Daybreak and Anthropic models, including GPT-5.6 Cyber and Mythos, for deeper model-backed analysis. Initial analysis and scoring is done with Clef, Cloudflare’s open-source decision model.

Collecting evidence to understand what each alert means requires a lot of time. Even in a highly sophisticated Security Information and Event Management (SIEM), too much is still left for a human to review. Human analysts must gather data, connect and aggregate detections, and account for missing sources while new alerts continue to arrive.

Think about every time a human analyst reviews an alert: "Which should we silence? Which action should we take? Which alert should we ignore? Which should we resolve as false positives? Which should we resolve as true positives? Which should trigger our incident team?" We address this predicament with our AI agent strategy. Our approach reduces all of those questions and gives Managed Defense Analysts a quick and consolidated view, directly providing insight into related alerts, admitted evidence, visible gaps, and recommended next steps. The result: cutting back on the time needed to analyze, and creating a hyper focus on actually getting security alerts resolved and mitigations deployed.

Why a single agent fails

Our first prototype showed the limits of one general-purpose agent. We provided the AI agent the whole investigation. It produced useful analysis, but it also hallucinated claims the evidence did not support. Telemetry, detector descriptions, policies, and threat intelligence were flattened into one prompt, which caused their distinct roles to merge together.

We saw three recurring problems with our first single-shot AI agent harness:

Context became authority. A detection is a hypothesis, not proof that an exploit succeeded or an attack occurred. A broad AI agent can blur that distinction.
Scope drifted. An AI agent can query the wrong account, time range, or source. You can’t rely on a language model prompt to be a boundary.
Failure disappeared. If a lookup times out, the result may not distinguish "not checked" from "checked and not found."

To address these challenges we moved evidence collection and scope enforcement into application code, before model analysis begins.

Recon first, inference second

It's tempting to put an AI agent at every step. The front half of our harness has none.

Before we even call inference, deterministic code runs a fixed set of reconnaissance workflows with versioned API calls. It collects the customer's identity, detection history, traffic baseline, enforcement outcome, and network observations. Each piece of data is stored with its source, version, and timestamp.

Cloudflare sees both the request and the action applied to it. That lets the investigation connect the behavior that triggered an alert with both the control that fired and its outcome.

The fixed recon snapshot also makes evaluation reproducible. If AI agents fetch their own data, two runs may disagree because their inputs changed. Here, the same snapshot can be replayed, so differences between specialist AI agents’ findings come from interpretation rather than retrieval.

Filter noise early

Most alerts are not incidents. The same rule often fires repeatedly on a known traffic pattern, and paging Managed Defense Analysts every time makes it easier to miss a real security incident.

We needed a lightweight triage model to compare each alert with its reconnaissance data: Has this event been detected for the customer before? What did Managed Defense Analysts decide previously? Does the traffic look consistent with normal human behavior? Alerts scored with a high likelihood to be false positives skip analysis by the specialist AI agents.

Clef, running on Workers AI, was the perfect fit for this type of fast agentic reasoning.

Known high-volume noise is deterministically classified as passive when it arrives. It remains available as context but does not enter the active queue.

Specialist AI agents handle the investigation

For alerts that need deeper review, a coordinator AI agent runs four specialist AI agents in parallel:

Traffic analysis reviews request behavior, historical changes, and enforcement.
Customer context reviews earlier alerts, dispositions, and Managed Defense Analysts’ decisions.
Global telemetry compares the activity with privacy-preserving Internet-wide signals.
Threat intelligence checks indicators already admitted to the alert or case.

A synthesis AI agent combines their typed findings into one advisory; it can’t fetch new evidence or choose a classification outside the approved vocabulary. Keeping each task narrow makes unsupported claims easier to catch, and recommendations easier to audit.

Global context without customer data

A security tool knows what happened inside the environment it is deployed in, but little about the world beyond it. Cloudflare compares an alert with patterns seen across its global network.

For example, an IP may be targeting one site, scanning thousands of sites, or appearing for the first time. Those patterns carry different weights. To preserve customer privacy, the global telemetry specialist works only with aggregates; it never receives another customer's individual records or identity.

This view combines features from Cloudflare's CDN, WAF, DDoS, Turnstile, Rate Limiting, and Cloudforce One threat intelligence. The synthesis AI agent weighs both global reputation and customer history. This ensures a globally common pattern can still be used on a per-customer basis, but does not automatically imply a widespread campaign for all customers.

History is evidence

Every evaluation is conscious of what came before it: the alert, the pattern, and which customer. The recon dossier records the alert's own track record including how many times this service alert has fired, how many of those were dispositioned as false positives, and what the Managed Defense Analyst concluded. An attack pattern that has been benign every time your Managed Defense Analysts have seen it is a very different object from the first sighting of something new, and the specialist AI agents are told which one they are looking at. Approved background context is retrieved from previous alerts and cases, so yesterday's conclusions are carried into today's decision, instead of being rebuilt from scratch.

Our system aggregates related alerts into a consolidated case. In each case, we store evidence, findings, and recommendations. Our system deterministically joins and correlates this data, but we leave it to a Managed Defense Analyst to confirm the actual scope. Over time, a case can connect network, application, and Zero Trust evidence together, while tracking the source of the evidence for additional context and reference purposes.

From evidence to decision

Before analysis, the system creates a versioned evidence package with the subject, scope, time anchor, admitted evidence, policy versions, sources, and coverage gaps. Specialists must cite items in that package. Application code checks that every citation exists, belongs to the investigation, and supports the attached claim. Invalid findings are corrected or recorded as limitations.

We lean on Clef a second time to score our evidence. Is the collected evidence enough for a decision? Does any of our collected evidence contradict? Based on this evidence, Clef picks from a deterministically reduced list of attack classifications and dispositions.

Cloudflare's developer platform runs this process. Application code on Workers admits evidence and validates results; Workflows coordinates each stage and saves completed work before the next begins, so a failed stage reuses evidence and findings that already passed validation instead of starting over. D1 keeps investigation and advisory state, R2 holds bounded context and evidence artifacts. Case-chat state persists in Durable Objects, uses Flue, and is enriched with AI Search.

Finally, an LLM-powered agent produces an advisory report using terms that Managed Defense Analysts already work with: affected surface, enforcement outcome, relevant controls, and next step. Managed Defense Analysts can inspect evidence, investigate further, revise the recommendation, or group alerts into a case. Application code fixes the customer scope before any model sees results, and gives each specialist only the evidence it needs. The model never receives authority to cross tenant boundaries or act for the Managed Defense Analysts.

Handling incomplete evidence

At network scale, a source will sometimes fail. A comparison may time out, metadata may be missing, or a threat intelligence lookup may return no match. The system keeps evidence already collected and records the gap.

The advisory distinguishes three states:

Not checked
Checked, with no matching result
Checked, with evidence supporting absence

If global telemetry is unavailable, the system can describe what is unusual for the customer but cannot say whether the pattern is widespread. When the evidence is insufficient, it makes no classification or disposition recommendation.

Remediation

A useful recommendation should lead to a solution, rather than a ticket. The advisory might suggest a rate limiting rule for an abusive path, a WAF custom rule for a signature, or a DDoS protection change. For fully managed customers, Managed Defense Analysts can apply the suggested rules; other customers will receive their recommendations in the dashboard and through their chosen alert path.

In the end, the Managed Defense Analyst remains responsible for the decision and any mitigation. Each alert and case includes the evidence behind the AI agent’s recommendation, which empowers the Managed Defense Analyst to reach a conclusion by accepting or updating the AI agent’s advice.

What comes next

Managed Defense Analysts remain responsible for judgment. The AI agent harness handles more of the repetitive work: assembling investigations, connecting related events, and showing the evidence behind each recommendation. Over the next few quarters, we plan to add a Custom Managed level with more flexibility for each organization.

We also plan to explore continuous AI agents that monitor Cloudflare traffic and surface patterns that fixed rules and thresholds may miss.

The early beta is available in Cloudflare Managed Defense for eligible application-security alerts and cases. If you already use Cloudflare WAF, DDoS protection, Magic Transit, or another supported product, talk to your enterprise account team about adding Managed Defense.

Powering AI-led research through simulation

2026-10-05 Grab Tech

Post Syndicated from Grab Tech original https://engineering.grab.com/powering-ai-led-research-through-simulation

The short story

Consider a Friday evening. A food order arrives from a mall in the city center. One driver is nearby; another is finishing a drop-off and will be available shortly; a second order from the same mall may or may not appear in the next two minutes. Dispatch the nearby driver now, or hold briefly for a batching opportunity? The decision window is short.

A fulfillment marketplace makes these decisions continuously. Each one is small. Across a city, those decisions determine whether your dinner arrives hot and whether a driver’s hour is well spent.

And that is one decision. There are dozens more: how far to look for a driver, when two orders are worth combining, which of three waiting trips gets the one free bike, how long to keep trying before giving up. These decisions interact, and the setting that is right for Friday at seven is wrong for Tuesday at two. Together they define a space of possible strategies far larger than anyone could exhaustively explore.

We sample only about a dozen points in that space each year. Not for want of ideas: implementing each strategy costs weeks of engineering, and a production experiment takes weeks more to judge. Some questions have no production answer at all. Nobody can run last Saturday again with twenty percent fewer drivers. We were never short of compute resources, and never short of data. We were short of attempts.

So we built sim-rs, a simulator that makes each attempt take minutes, and connected agents that run experiments on it.

sim-rs is self-contained: it requires no production services or databases. The dispatch lifecycle runs in one compiled program alongside a separate dispatch service that is spawned locally and operates offline. Historical booking and driver data go in as plain files, together with a configuration describing the strategy to test. Simulated bookings, trips, drivers, and a single metrics report come out. Processing one city-day of marketplace activity takes tens of minutes, from raw input to finished report. One command in, one report out: a workflow as practical for a software agent as for an engineer.

That compact contract is what makes the simulator AI-friendly. Agents can change bounded components, run reproducible experiments, receive verdicts they cannot alter, and help keep both production logic and behavioral models current.

Building blocks

The essential components

To model a marketplace, the simulator needs four core concepts:

Bookings: requests to move something (a passenger, a meal, a parcel), represented in one format regardless of the business vertical.
Drivers: simulated workers with a location, a shift, a vehicle, and a queue of work.
Trips: units assigned to drivers, containing one booking or several batched into a multi-stop route.
Ticks: simulated time, advancing in fixed steps. Nobody waits in real time.

At each tick, the simulator runs the following sequence:

Demand arrives. Historical (or synthesized) bookings whose time has come enter the pool.
Supply moves. Drivers come on shift, advance along routes, go idle, and reposition.
Pre-dispatch cancellations are applied. Cancellation models, including survival-analysis models trained on real behavior, determine which bookings leave the pool.
Dispatch happens. Bookings are batched into trips, trips are matched to drivers by an optimization solver, and a recycling step decides, for each trip, whether to send it now, hold it, or split it back into bookings and try again later.
Post-dispatch cancellations are applied. These include passenger and driver cancellations.
Outcomes are recorded. Every booking, trip, and driver outcome is written to the run outputs.

Crucially, this is a closed loop: today’s dispatch decisions change where drivers end up, which changes what’s possible next tick. That feedback produces second-order effects that a static replay, one that scores historical decisions without updating future supply, cannot show. Supply may dry up in a hot zone, or one bad dispatch rule may cascade into a wave of cancellations.

Marketplace simulation tick loop from demand through dispatch to recorded outcomes — Figure 1. Simulator sequence.

That loop is only the mechanism. Making it a lab takes three further properties: configurability; reproducibility and auditability; and reliable comparison.

1. Interchangeable stages. The major dispatch stages are pluggable. Batching, allocation, cancellation, routing, driver movement, and recycling are each exposed through an interface with interchangeable implementations selected by configuration. This turns a fixed pipeline into an experimental platform: the component under study can be swapped while everything around it stays unchanged. The same mechanism determines which marketplace is being simulated. Ride-hailing, food delivery, parcel delivery, or all three sharing one driver pool are configuration choices within the same codebase, not separate simulators.

2. Self-contained runs. Each experiment is a small, portable package containing its configuration files, input data, source revision, metrics report, and detailed outputs. Anyone holding that package can recreate the setup, whether a reviewer, teammate, agent, or the original author six months later. The report identifies the configured components and headline metrics, while the supplementary per-booking, per-trip, and per-driver outputs let reviewers recompute those metrics and investigate unexpected outcomes without relying on separate notes or systems.

3. Experiments inside the simulation. Production marketplaces have experimentation platforms, so our simulator ships with one too. Experimentable fields can define multiple treatment arms assigned through the same time-based switchbacks, spatial splits, or cell-and-hour schemes used in production. Each booking records its resolved arm for traceability. Running these A/B/n comparisons together exposes every arm to the same demand, drivers, marketplace dynamics, noise, and biases. By reproducing both the conditions and assignment design of a marketplace experiment, simulated insights are more likely to translate into effects observed in production.

Automating the research cycle

Agents do two jobs here, and both need the same environment: searching for strategies that beat the incumbent, and keeping the simulator sufficiently faithful for that search to mean something.

Searching for better strategies

We connect agents to that interface through autoresearch, an automated loop that runs the research cycle end to end. An agent proposes a change, implements it in source code, tests it, and acts on the verdict. A candidate survives only if it improves the target metric and passes every required check; otherwise it is rolled back and the loop continues. The objective may be a marketplace outcome such as orders-throughput or a software measure such as execution time. What matters is a repeatable command-and-metric contract that both humans and agents can use.

Automated research loop connecting an agent to the simulator and evaluation checks

The first failure mode we encountered was specification gaming. Given a target and a loophole, an agent may find the shortest path to the number rather than the improvement we intended. Ours discovered that changing fields used by pre-dispatch cancellation could reduce cancellations and raise completion without improving a single dispatch decision. The metric moved; nothing real had improved. Rather than relying on instructions alone, we built four safeguards into the environment:

Core data is immutable. Modules under test may change only the state exposed by their interfaces. An attempt to manipulate protected fields fails to compile instead of producing a misleading result.
The verdict is computed, not judged. The simulator computes its own metrics, which the agent can read but not redefine. Executable checks apply calibrated tolerance bands, verify equivalent outputs where required, and run the tests. The agent never judges its own work.
Changes are scoped, checked, and reversible. Each experiment runs on its own branch within a declared writable scope, which is checked against the resulting diff. The grading machinery remains outside that scope, regressions cost only a discarded branch, and every surviving change is bounded enough to review in full.
Changes are checked at two levels. It first rejects changes that violate type contracts, ownership rules, interface boundaries, or concurrency requirements. The evaluation harness then applies checks suited to the objective: performance work must preserve expected outputs, while strategy experiments must satisfy domain constraints and outcome thresholds.

The speed that matters is the speed of an adaptive research loop, not simulation runtime alone: each iteration uses previous results to propose the next change, then implements, compiles, runs, scores, and decides whether to keep it. The simulator returns a verdict in milliseconds for benchmarks or tens of minutes for a full-day replay, while autoresearch carries each verdict into the next attempt without human handoffs. This turns an overnight run into a connected sequence of evidence-driven experiments, with the safeguards above ensuring that faster iteration compounds reliable evidence rather than mistakes.

One outcome is the familiar purpose of simulation: discovering better marketplace strategies. When asked to explore trip-recycling policy, the loop turned an emerging human intuition into a concrete rule: hold batched orders only as long as service-level deadlines permit, maximizing the chance that another nearby order joins the trip. Because the result is human-readable code, engineers can review, audit, and deploy it like any other pull request.

The same machinery also improves the simulator itself. Over several nights of unattended operation, we aimed the agents at two hot paths in its dispatch logic: the checks that decide which drivers are eligible for a trip, and the pass that adjusts the cost of every driver-and-trip pairing before the solver chooses an assignment. Across roughly 150 logged experiments, three out of four attempts failed to build or were rejected by later gates. The eligibility checks ended up about 10 times faster, and the cost pass about 24 times faster. Because these paths run repeatedly in every replay, improvements compound across subsequent research. Faster and more efficient runs enable testing of more ideas across more markets and dates, repeat runs to separate signal from noise, and validate results more rigorously. Shorter runs also tighten the feedback loop from verdict to next proposal, so improving the instrument accelerates and strengthens every search performed with it.

Execution time is only one possible objective: pointing the same machinery at orders-throughput changes the research question, not the loop itself. Regardless, whatever the objective, the result is only as trustworthy as the simulator behind it. An agent can optimize only the world it is given; if that world has drifted from production, a faster loop will merely produce misleading answers sooner.

Keeping the simulator faithful

A simulator is a claim about the world: that this is how the system behaves and that this is how the people within it respond. Both halves of that claim decay. Production logic changes continuously, while models of passenger and driver behavior grow stale as new product features reshape how people act. A simulator that has drifted from the world produces misleading insights and innovations that fail in production. So the second job we give agents is to keep the simulator faithful.

The system half is a translation problem, and coding agents are particularly good at it. An agent can read a component’s production implementation and implement equivalent logic behind the simulator’s corresponding interface, translating directly from source code rather than from a written description that may already be stale. The result remains a candidate until parity checks show that it matches the production behavior being modeled.

The behavioral half cannot be translated, because no source file tells us how a passenger will behave. It must be inferred from what people actually do. Here, we redirect the same autoresearch loop from dispatch-code optimization to behavioral modeling. It proposes, fits, and scores candidate models, exploring which features predict cancellation and how their effects should be represented. The test is not only how closely a model explains its training data, but also how well it generalizes to held-out data. A model that fits last Tuesday perfectly and next Tuesday poorly does not make the simulator more faithful; it makes the lab more confidently wrong.

The pieces are simple: a realistic marketplace model, swappable parts, repeatable experiments, honest scoring, and an easy way to undo failures. Together, they give agents a safe place to test and improve ideas. Everyone is racing to give AI a bigger brain. We got further by giving it a better lab: a place where it can try a thousand ideas, be wrong cheaply, and receive an honest verdict.

Join us

Grab is Southeast Asia’s leading superapp, serving over 900 cities across eight countries (Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam). Through a single platform, millions of users access mobility, delivery, and digital financial services, including ride-hailing, food delivery, payments, lending, and digital banking via GXS Bank and GXBank. Founded in 2012, Grab’s mission is to drive Southeast Asia forward by creating economic empowerment for everyone while delivering sustainable financial performance and positive social impact.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Announcing AWS Well-Architected Agent, an AI-powered intelligence to optimize your cloud environment (preview)

2026-10-01 Channy Yun (윤석찬)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/announcing-aws-well-architected-agent-an-ai-powered-intelligence-to-optimize-your-cloud-environment-preview/

Today, we’re announcing the public preview of AWS Well-Architected Agent, an AI-powered service that analyzes your AWS environment to deliver targeted, contextual recommendations for improving your applications’ cost, security, performance, and resilience. The AWS Well-Architected Agent analyzes your infrastructure, understands unique business goals, and delivers contextual recommendations with ready-to-implement fixes. It delivers context-aware optimization without relying on manual audits or generic checklists.

The agent evaluates your environment as an experienced cloud architect would. It automatically correlates utilization metrics, resource configurations, and application topology, and analyzes against Well-Architected best practices across 65+ AWS services. It generates recommendations aligned to your declared business goals, delivers implementation packages with every finding, and surfaces cross-pillar trade-offs making it simpler to remediate the findings.

Here are the three main features of this service:

Goal-aligned intelligence: AWS Well-Architected Agent replaces flat, undifferentiated findings with context-aware, prioritized recommendations. You declare your business objectives and share your application context. The agent automatically generates and prioritizes recommendations by impact and effort against those goals.
Three-level recommendations: AWS Well-Architected Agent provides individual resource findings with specific dollar impact (where applicable) and step-by-step remediation, consolidated findings across multiple resources scoped to your application, and broad architectural patterns and designs with Infrastructure as Code (IaC) code changes needed to align with Well-Architected best practices.
Optionality in remediation: You can choose your path on how you want to remediate with a complete implementation steps tailored to your environment: the console walk-throughs, updated IaC changes for architecture-level recommendations, and AWS Command Line Interface (AWS CLI) commands.

AWS Well-Architected Agent in action

To get started, create an agent profile to define the scope of what Well-Architected Agent can access and provide recommendations on, complete the IAM role setup to access resources, conduct architecture review, and remediate recommendations.

Create an agent profile

In the AWS Well-Architected console, choose Get started with Well-Architected Agent. You can define an agent profile that specifies which AWS accounts and applications to monitor, which optimization pillars to focus on, and the permissions required.

You can choose AWS accounts or AWS Regions to monitor and optimization pillars that matter most to your business. You can also set goals for each pillar: cost optimization, performance, resilience, and security.

To give access to the agent for your AWS environment, provision customer-managed IAM roles the agent uses to read resource configurations, utilization metrics, and application topology. To learn more, visit the IAM prerequisite for AWS Well-Architected Agent.

When you choose Get Started, the agent creates your agent profile. Resource and application recommendations will be generated within 24 hours after profile creation.

You can conduct an architecture review on pre-deployment workloads by uploading an IaC project in Terraform, AWS CloudFormation, or AWS Cloud Development Kit (CDK) to be analyzed. Choose Conduct architecture review in the dashboard, upload a.zip file containing IaC project or repository file, and select which Well-Architected lens to use for reviewing your infrastructure.

You can define your applications to add context which will enhance the relevancy and further contextualize recommendations. Choose Add application context in the dashboard, add your applications with AWS accounts, AWS Regions, AWS services, tags if you want to narrow the scope to specific resources, and the details of applications.

Review prioritized recommendations and start remediating

Now you can see generated prioritized recommendations generated by the agent across your resources and applications, selected pillars, ranked against your declared goals, with automation-ready remediation included.

When you choose the specific recommendation, you can see the details, insights into why the agent are suggesting the recommendation, impacts and trade-off, and recommended fixes across affected AWS resources.

Choose Start remediation to address recommended fixes. You can choose the console, updated IaC template, CLI commands to remediate by the resolution type. It provides detailed step-by-step instructions and you can roll out this instruction and verify the result.

When you choose Using updated IaC template, the agent provides the code changes needed to update your existing IaC templates such as the CDK function shown above which you can copy directly into your codebase.

You can also configure API access to integrate recommendations directly into your existing development and operations workflows. To interact with the agent programmatically, including calling APIs and searching documentation, try the AWS MCP Server and plugins with your preferred AI coding tool. To learn more, visit the AWS Well-Architected Agent documentation.

Things to know

Here are some things that you should know about the Well-Architected Agent.

Automation: You can receive recommendations with the exact IaC code changes needed to remediate, with risks identified by pillar, catching issues before they reach production. Recommendations are delivered through the console and API so you can act without context-switching. Recommendations are also updated periodically, so new recommendations are available for your team to track regularly.
Evaluation: Generative AI capabilities produce this recommendation, which may contain errors or incomplete information. You are responsible for evaluating the recommendation in your specific context and implementing appropriate oversight and safeguards. Learn more about AWS Responsible AI practices.

You can still use existing AWS Well-Architected Tool to manually evaluate your cloud architecture with user-defined lenses that measure your workload using your own best practices.

Join the preview

Access to the AWS Well-Architected Agent and its recommendations is available in US East (N. Virginia), US East (Ohio), and US West (Oregon). You can onboard workloads from any AWS commercial Region. AWS Well-Architected Agent is delivered by AWS Support and available to AWS customers with an AWS Support plan.

Give it a try today in the AWS Well-Architected console and send feedback through your usual AWS Support contacts.

— Channy

Not all explanations are equal: Social Explainable AI and Critical Computational Literacy

2026-10-01 Katharine Childs

Post Syndicated from Katharine Childs original https://www.raspberrypi.org/blog/not-all-explanations-are-equal-social-explainable-ai-and-critical-computational-literacy/

AI technologies, such as smart speakers or streaming recommendations, are becoming part of everyday life for children and teenagers. It’s therefore increasingly important to help young people understand not just how these systems work, but how to question them.

When AI systems generate outputs such as a recommendation, these outputs are often accompanied by an explanation. In our latest research seminar, Professor Dr. Dan Verständig (Goethe University Frankfurt, Center for Critical Computational Studies) spoke about how explanations are shaped by the people who write them, and only become meaningful when someone else makes sense of them. Dan introduced us to Social Explainable AI (Social XAI) and Critical Computational Literacy (CCL), and asked us to consider not just what an AI system explains, but for whom, why, and who benefits.

Why do we need explanations?

Dan opened with a deceptively simple question: why do we need explanations in the first place? His answer was that explanations provide orientation. They reduce uncertainty, and in doing so, they make our shared social world liveable. Explanations have evolved from Socratic dialogue, to the printed book, to the classroom, to today’s digital interfaces, and now to AI-generated explanations. But explanations have never been neutral conveyors of information. Dan illustrated this with a simple demonstration: he showed the seminar participants an aerial photo of a beach and asked what they noticed.

The aerial photo Dan shared during his seminar. What do you notice about this picture?

Seminar participants pointed to the coastline, the sandy beach, and a tiny figure in the frame. Dan then revealed that the photo was taken by him, using a drone at Montara State Beach in California on a specific afternoon in April 2024. He asked how a climate researcher, a surfer, or an artist might each describe the very same image differently.

This demonstration highlighted that an explanation is always for someone, for some purpose, in some context. That framing carries straight through into how we should think about AI explanations too. An explanation that satisfies a data scientist debugging a model is not the same as one that satisfies a patient asking why an algorithm flagged their scan, or a citizen asking why they were denied a loan. In other words, explanations are never technical.

From delivering explanations to co-constructing them

Drawing on Rohlfing and Lim’s 2026 work, Dan described Social XAI as an approach that puts interaction, rather than output, at the centre of explainability. Dan illustrated this with a simple but effective diagram: an AI explanation starts as a system output, is filtered through a person’s interpretation, and only then becomes meaning through construction. A one-size-fits-all output, however technically accurate, isn’t yet an explanation until someone has made sense of it in their own terms.

AI explanations are never just delivered as outputs; meaning is constructed by an individual’s interpretation.

To explore this further, Dan’s research group has developed co-construction workshops, bringing people together to interrogate AI explanations collaboratively rather than receive them passively. In these workshops, participants are asked questions such as: “What counts as evidence? What is missing? What are the alternatives? Who benefits from this? Do we agree?”

To answer these questions, participants do not examine an AI model to find out how it generates a decision. Instead, participants scrutinise its outputs in the same way that they would critically evaluate a politician’s promise or a newspaper’s headline.

Dimensions for engaging critically with explanations

Critical Computational Literacy (CCL) is a framework that describes what people need to engage with AI explanations critically. Dan presented CCL as consisting of four interlocking dimensions:

Attitude — a critical stance and value awareness: approaching computational systems not as neutral tools but as artefacts that embed particular values and assumptions
Biography — recognising that people’s relationships with technology are shaped by subjectivity, by the varied encounters they’ve had with computational systems, and by the diverse pathways that brought them to this point
Capacity — the analytical, creative, and ethical skills needed to actually work with and through computational systems, not just talk about them
Critique — the capacity that ties the other three dimensions together: asking what matters, and why

Critical Computational Literacy consists of four dimensions

Taken together, these dimensions push back against a narrow, purely technical notion of ‘AI literacy’ as knowing how a model works. Instead, CCL treats literacy as something biographical and value-laden. Dan explained that individuals bring their own history and stance to any encounter with computational systems, and genuine literacy means being able to interrogate these systems’ explanations rather than simply accept or operate them.

Using these ideas in your classroom

Although Dan’s research took place with adult participants, Social XAI and Critical Computational Literacy are ideas that could also be used in the K-12 classroom. For example, if students interact with AI explanations through smart speakers or streaming recommendations in their everyday lives, this presents an opportunity to teach them how to engage critically with AI outputs. Students might explore why a smart speaker suggested a particular recipe, or investigate the explanation for why a streaming service recommended a particular show.

Explainability is also a key part of our own Experience AI resources. When training a model, students write their own model cards to document who built a model, what training data was used, how accurate the model’s predictions were, and any known limitations. In this activity, explainability is traceable, and students use their analytical skills to create transparent, fair, and accountable model cards.

Social XAI suggests an extension to this activity. Students could ask what matters about the explanation they have written and why this is important. Through this critique, students can consider who will read these model cards and how the cards might be interpreted. In this way, AI explanations become more than a technical output, and become artefacts whose meaning is co-constructed by the author and the reader.

This seminar was part of our ongoing series on teaching about AI in the arts, humanities, and sciences. You can watch the full recording of Dan’s talk, including the Q&A discussion, here:

Join our next seminar

Our research seminar series continues to explore how AI is taught across the curriculum. In our next seminar on Tuesday, 6 October at 19:00–20:30 BST, we welcome Eleni Petraki and Damith Herath (University of Canberra) who will present an engineering and robotics curriculum aimed at equipping future engineers with the diverse skills demanded by a growing workforce. To take part in the seminar, click the button below to register. We hope to see you there.

The schedule of our upcoming seminars is available online. You can catch up on past seminars on the blog and on the previous seminars and recordings page.

The post Not all explanations are equal: Social Explainable AI and Critical Computational Literacy appeared first on Raspberry Pi Foundation.

Building a Slack-powered AI development agent with Kiro CLI and headless authentication

2026-09-29 Vishal Karlupia

Post Syndicated from Vishal Karlupia original https://aws.amazon.com/blogs/devops/building-a-slack-powered-ai-development-agent-with-kiro-cli-and-headless-authentication/

Every code review discussion, incident response thread, and standup happens in Slack. But when an engineer needs to analyze a service or debug a failing test, they leave Slack, open a terminal, navigate to the repository, run commands, and paste the output back. That round trip takes 30 seconds for someone who knows exactly where to look and 5 minutes for someone less familiar with the codebase. Across a team of 10 engineers doing this 15 times a day, that adds up to over 12 hours of lost engineering time per week.

This post walks through building a ChatOps integration that runs Kiro CLI from a Slack slash command. An engineer types /kiro analyze auth-service for memory leaks, and the results appear directly in the channel—no context switch required. The solution uses AWS Lambda, Amazon API Gateway, and AWS Secrets Manager, and it depends on Kiro CLI’s headless authentication to run without an interactive session.

In this post, you will learn how to:

Configure a Slack App with a slash command that triggers an AWS Lambda function
Authenticate Kiro CLI in a headless environment using API key-based authentication
Build and deploy a container image with Kiro CLI to Amazon Elastic Container Registry (Amazon ECR)
Deploy the full solution with AWS Serverless Application Model (AWS SAM)

Why headless authentication matters

A Slack slash command triggers a webhook. The webhook invokes a Lambda function. The Lambda function runs Kiro CLI. At no point in this chain is there a browser, a terminal, or a human session.

Without headless authentication, this architecture does not work. Kiro CLI would require an interactive login, and a Lambda function has no display and no way to complete an OAuth flow.

With an API key, Kiro CLI authenticates silently:

export KIRO_API_KEY=ksk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

kiro-cli chat --no-interactive "analyze auth-service for memory leaks"

The API key is stored in AWS Secrets Manager, fetched at runtime, and injected into the Lambda environment. The engineer in Slack never sees or manages the key.

Important: API key-based authentication is available for Kiro Pro, Pro+, and Power subscribers. If your subscription is managed by an administrator, your Kiro admin must enable API key authentication first. For details, see API key governance.

Architecture overview

architecture overview

The solution consists of two Lambda functions, an API Gateway endpoint, and AWS Secrets Manager. The request and response follow two separate paths:

Request path: Slack → API Gateway → Dispatcher Lambda → acknowledge back to Slack (under 3 seconds), then async invoke → Worker Lambda
Response path: Worker Lambda → Slack response_url (direct HTTPS POST, bypasses API Gateway)

Why two Lambda functions?

Slack requires a response within 3 seconds of a slash command. Kiro CLI analysis takes 10–60 seconds depending on the repository size and prompt complexity. The Dispatcher acknowledges the command immediately and invokes the Worker asynchronously. The Worker runs Kiro CLI and posts results back to Slack through the response_url provided in the original payload. This is a standard pattern for Slack integrations that perform long-running work.

A note on response_url limits: the webhook Slack provides in the slash command payload expires 30 minutes after the command is issued and accepts a maximum of 5 responses. The 10-minute Worker timeout and single response in this solution stay well inside both limits. If you raise the Lambda timeout beyond 30 minutes or add incremental progress updates, these POSTs begin to fail silently – switch to chat.postMessage with a bot token at that point.

Prerequisites

Before you begin, you need the following:
An AWS account with permissions to create Lambda functions, API Gateway, Amazon ECR repositories, Secrets Manager secrets, and IAM roles
An infrastructure-as-code tool for deploying serverless resources (this post uses AWS SAM CLI, but you can adapt the templates to AWS CDK, AWS CloudFormation, Terraform, or your preferred tool)
Finch or Docker installed for building container images
A Slack workspace where you have permission to create a Slack App
A Kiro Pro, Pro+, or Power subscription with API key authentication enabled

Step 1: Gather credentials

You need three credentials before deploying. Collect all of them first, then store them in Secrets Manager in Step 2.

Kiro API key

This authenticates Kiro CLI in headless mode.

Sign in to app.kiro.dev
Navigate to API Keys
Create a new key named kiro-chatops
Copy the key (starts with ksk_) – it is shown only once

get Kiro API key

Slack Signing Secret – This allows the Dispatcher to verify that incoming requests originate from Slack.

Go to api.slack.com/apps and click Create New App → From scratch
Name it Kiro Agent and select your workspace
On the Basic Information page, scroll to App Credentials
Copy the Signing Secret (32-character hex string)

create a slack app

Slack Bot Token – Optional

The Worker posts results using the response_url from the original slash command payload, which is a pre-authenticated webhook that does not require a bot token. Collect a bot token with the chat:write scope only if you extend the solution to post messages independently of a slash command response.

In your Slack App settings, go to OAuth & Permissions
Add the Bot Token Scope: chat:write
Choose Install to Workspace and authorize
Copy the Bot User OAuth Token (starts with xoxb-)

set up slack oauth token

While you are in the Slack App settings, also configure the slash command:

Go to Slash Commands → Create New Command
Set Command to /kiro
Set Request URL to https://placeholder (update after deployment in Step 6)
Set Short Description to Run Kiro-CLI development tasks
Set Usage Hint to [analyze|review|debug|explain] <description>

define slack commands

Step 2: Store secrets in AWS Secrets Manager

Store each credential as a separate secret. The Lambda functions retrieve these at runtime using IAM-scoped access.

aws secretsmanager create-secret \
  --name kiro-chatops/kiro-api-key \
  --secret-string "<your-kiro-api-key>" \
  --region us-east-1

aws secretsmanager create-secret \
  --name kiro-chatops/slack-signing-secret \
  --secret-string "<your-signing-secret>" \
  --region us-east-1

aws secretsmanager create-secret \
  --name kiro-chatops/git-token \
  --secret-string "<your-github-pat>" \
  --region us-east-1

If your target repository is private, also store a GitHub Personal Access Token with repo scope. The Worker uses this token to clone the repository inside the Lambda execution environment.

Verify the secrets were created:

aws secretsmanager list-secrets \
  --filter Key="name",Values="kiro-chatops" \
  --query "SecretList[].Name" \
  --output table \
  --region us-east-1

Step 3: Build the Dispatcher Lambda

The Dispatcher has three responsibilities: verify that the request came from Slack, acknowledge the slash command within 3 seconds, and invoke the Worker asynchronously.

Request verification – Slack signs every request with HMAC-SHA256 using your app’s signing secret. The Dispatcher must validate this signature before processing payloads. The verification logic constructs a base string from the request timestamp and body, computes the HMAC, and compares it to the signature in the request header:

sig_basestring = f"v0:{timestamp}:{body}"
expected = "v0=" + hmac.new(
    signing_secret.encode(), sig_basestring.encode(), hashlib.sha256
).hexdigest()
return hmac.compare_digest(expected, signature)

Reject any request with a timestamp older than 5 minutes to prevent replay attacks. Normalize request headers to lowercase before reading them—API Gateway may preserve the original casing from the client.

Async handoff

After verifying the request, parse the slash command payload to extract text, user_name, and response_url. Then invoke the Worker Lambda with InvocationType="Event" (fire-and-forget) and immediately return an acknowledgment to Slack:

lambda_client.invoke(
   FunctionName=os.environ["WORKER_FUNCTION_NAME"],
    InvocationType="Event",
    Payload=json.dumps({
        "command_text": command_text,
        "user_name": user_name,
        "response_url": response_url,
    })
)

If the user sends /kiro with no arguments, return an ephemeral usage message with examples. The Dispatcher uses the standard Python 3.12 Lambda runtime and requires no container image.

Step 4: Build the Worker Lambda container image

The Worker runs Kiro CLI against a cloned repository and posts results to Slack. Because Kiro CLI depends on git, system libraries (NSS, X11, ALSA), and a binary that exceeds Lambda’s 250 MB layer limit, package the Worker as a container image.

Dockerfile structure

Start from the AWS Lambda Python 3.12 base image. Install git and the shared libraries that Kiro CLI requires, then install Kiro CLI itself:

FROM public.ecr.aws/lambda/python:3.12
RUN dnf install -y git unzip alsa-lib atk at-spi2-atk cups-libs \
    libdrm libXcomposite libXdamage libXrandr mesa-libgbm \
    pango nss nspr libXtst && dnf clean all
RUN curl -fsSL https://cli.kiro.dev/install | bash && \
    cp /root/.local/bin/kiro-cli* /usr/local/bin/ && \
    chmod +x /usr/local/bin/kiro-cli*
COPY index.py ${LAMBDA_TASK_ROOT}/
RUN pip install boto3 --target "${LAMBDA_TASK_ROOT}"
CMD ["index.handler"]

Two details matter here. First, copy the Kiro CLI binary to /usr/local/bin/ rather than leaving it in /root/.local/bin/—Lambda runs as a non-root user that cannot access /root/. Second, build with --platform linux/amd64 regardless of your local architecture, because Lambda defaults to x86_64.

Worker logic – The handler performs four steps:

Fetch the Kiro API key (and optionally a Git token) from Secrets Manager
Clone the repository to /tmp/repo using git clone --depth 1
Run kiro-cli chat --no-interactive "<prompt>" with KIRO_API_KEY and HOME=/tmp set in the environment
Post the output to Slack via the response_url

Setting HOME=/tmp is required because Kiro CLI writes a session database, and Lambda’s filesystem is read-only except for /tmp. Strip ANSI escape codes from the output before posting—Kiro CLI emits terminal colors that render as garbage in Slack.

The subprocess timeout should be shorter than the Lambda timeout to allow time for error handling and the Slack POST. Set the subprocess timeout explicitly to 540 seconds in the Worker code, rather than relying on the Lambda timeout alone. A 9-minute subprocess limit with a 10-minute Lambda timeout provides a 1-minute buffer.

Truncate output to 3,800 characters before posting. Slack’s message limit is 4,000 characters per block, and the surrounding formatting consumes part of that space.

Build and push to Amazon ECR

Clean up /tmp/repo at the end of every invocation. Lambda may reuse a warm execution environment, so anything left in /tmp persists into the next invocation. Removing the clone in a finally block helps prevent one user’s repository from leaking into a later request and keeps the 512 MB ephemeral storage from filling up across warm invocations.

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) 
AWS_REGION=us-east-1 
 # Create the ECR repository (first time only) 
aws ecr create-repository --repository-name kiro-worker --region $AWS_REGION 
 # Authenticate to ECR 
aws ecr get-login-password --region $AWS_REGION | \ 
  finch login --username AWS --password-stdin \ 
  $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com 
 # Build for the correct architecture 
cd worker 
finch build --platform linux/amd64 -t kiro-worker:latest . 
 # Tag and push 
finch tag kiro-worker:latest \ 
  $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/kiro-worker:latest 
finch push \ 
  $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/kiro-worker:latest

Step 5: Deploy with AWS SAM

The SAM template defines both Lambda functions, the API Gateway endpoint, and the IAM policies. The Dispatcher uses a standard Python runtime. The Worker references the container image you pushed to Amazon ECR.

Key resource configuration:

Resource	Runtime	Timeout	Memory	Package type
Dispatcher	Python 3.12	10 s	256 MB	Zip
Worker	Container	600 s (10 min)	1024 MB	Image

Both functions use AWSSecretsManagerGetSecretValuePolicy scoped to the kiro-chatops/* secret prefix. The Dispatcher also gets LambdaInvokePolicy for the Worker function. Neither function has broader AWS permissions.

The SAM template accepts the ECR image URI as a parameter:

Parameters: 
  EcrImageUri: 
    Type: String 
    Description: ECR image URI for the Worker Lambda 
 
Resources: 
  WorkerFunction: 
    Type: AWS::Serverless::Function 
    Properties: 
      PackageType: Image 
      ImageUri: !Ref EcrImageUri 
      Timeout: 600 
      MemorySize: 1024

Deploy:

cd ..  # Back to the project root where template.yaml lives 
sam build 
sam deploy --guided \ 
  --stack-name kiro-chatops \ 
  --parameter-overrides \ 
    EcrImageUri=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/kiro-worker:latest

SAM prompts you to confirm IAM role creation and acknowledge that the Dispatcher has no authentication (request verification happens in code via the Slack signing secret). After deployment completes, note the ApiEndpoint output value.

Step 6: Connect Slack to the endpoint

Go to api.slack.com/apps and select your Kiro Agent app
Navigate to Slash Commands and edit /kiro
Replace the Request URL with the ApiEndpoint value from the SAM deployment output
Choose Save

slack endpoint integration

Step 7: Test the integration

Test directly from Slack by typing in any channel where the app is installed:

test slack integration

/kiro analyze auth-service for memory leaks

Expected behavior:

Slack immediately displays: “@yourname requested: analyze auth-service for memory leaks – Kiro is working on it…”
After 15–60 seconds, the analysis results appear in the channel

expected Kiro agent behaviour

Kiro agent results

You can also invoke the Worker Lambda directly for testing without Slack:

aws lambda invoke \ 
  --function-name <WorkerFunctionName-from-SAM-output> \ 
  --invocation-type RequestResponse \ 
  --cli-binary-format raw-in-base64-out \ 
  --payload '{"command_text": "explain what this repo does", "user_name": "test", "response_url": "https://hooks.slack.com/YOUR/URL"}' \ 
  response.json

Sending /kiro with no arguments returns a usage help message.

Complete sample code can be found at aws-samples github repository – https://github.com/aws-samples/sample-kiro-chatops-slack-integration

Practical slash command patterns

Once deployed, the value comes from the commands your team uses daily. These patterns map to real engineering workflows:

Category	Example command
Code analysis	`/kiro analyze the payment module for error handling gaps`
Code review	`/kiro review the last 3 commits on main for breaking changes`
Debugging	`/kiro debug why the integration tests are failing`
Knowledge	`/kiro explain how the authentication middleware works`
Sprint support	`/kiro summarize all changes merged to main this week`

The value compounds when results are visible to the entire channel. A junior engineer who might hesitate to open a CLI tool can type /kiro explain and get the same analysis and the rest of the team learns from it.

Extending the pattern

Multi-repository support – The basic implementation targets a single preconfigured repository. To support multiple repositories, parse a URL from the slash command text and clone it at runtime. This adds 5-15 seconds of latency and requires a Git token in Secrets Manager for private repositories.

Threaded responses – Post the acknowledgment as a channel message and the full results as a thread reply. This keeps the channel readable while preserving context for long analyses.

Approval workflows – For commands that modify code (for example, “create a PR that fixes this issue”), add a confirmation step. The Worker posts proposed changes with interactive buttons; the action executes only after explicit approval.

Audit logging – Log every invocation to Amazon DynamoDB: who ran it, what they asked, how long it took. This gives engineering leadership visibility into how the team uses AI-assisted development.

Constraints and trade-offs

Constraints:

Execution time – Lambda has a maximum 15-minute timeout. Complex analyses that exceed this will time out. The Worker is set to a 10-minute timeout with a 9-minute subprocess limit.
Ephemeral storage – The /tmp volume defaults to 512 MB. A shallow clone (–depth 1) strips Git history, but the working tree alone can exceed this for large monorepos or repositories with binary assets. You can increase ephemeral storage up to 10 GB by setting EphemeralStorage in the SAM template, or scope the clone to a subdirectory with –sparse-checkout for oversized repositories.
Slack message size – Each Block Kit text block is limited to 3,000 characters. Long outputs are truncated, with full results available in Amazon CloudWatch Logs.
Package size – Kiro CLI with its dependencies exceeds Lambda’s 250 MB layer limit. A container image (up to 10 GB) is required.

Trade-offs:

Lambda vs. Amazon ECS on AWS Fargate – Lambda is simpler and cheaper at the low-volume, bursty usage typical of a single team. Model your own break-even point with the AWS Pricing Calculator, since it shifts with average analysis duration and memory size. For high-volume teams, Fargate with a persistent container avoids cold starts. Start with Lambda and migrate if usage grows.
Public channel vs. ephemeral – Results are posted as in_channel (visible to everyone). For sensitive analyses, change response_type to ephemeral. Consider making this configurable per command.
Cost – Lambda compute is approximately $0.01-$0.05 per 10-minute execution at 1024 MB. The primary cost factor is Kiro CLI usage based on your subscription tier.

Security considerations

Request verification — The Dispatcher validates every request using HMAC-SHA256 with the Slack signing secret. Requests with timestamps older than 5 minutes are rejected.
Secrets management — Credentials are never hardcoded or stored in environment variables. They are fetched at runtime from Secrets Manager with IAM-scoped access.
Least-privilege IAM — The Dispatcher can only invoke the Worker and read secrets. The Worker can only read secrets. Neither has broader AWS permissions.
Audit trail — CloudWatch Logs capture every invocation including the command text, user, and Kiro CLI output. Enable AWS CloudTrail for API Gateway to track all incoming requests.

Cleaning up

To avoid ongoing charges, remove all resources when you are done testing:

sam delete --stack-name kiro-chatops 
 aws ecr delete-repository \ 
  --repository-name kiro-worker --force --region us-east-1 
 aws secretsmanager delete-secret \ 
  --secret-id kiro-chatops/kiro-api-key \ 
  --force-delete-without-recovery --region us-east-1 
aws secretsmanager delete-secret \ 
  --secret-id kiro-chatops/slack-signing-secret \ 
  --force-delete-without-recovery --region us-east-1 
aws secretsmanager delete-secret \ 
  --secret-id kiro-chatops/git-token \ 
  --force-delete-without-recovery --region us-east-1

To remove the Slack App, go to api.slack.com/apps, select Kiro Agent, and click Delete App.

Conclusion

This post demonstrated integrating Kiro CLI into Slack workflows using headless authentication, serverless functions, and secure credential management. The Dispatcher acknowledges instantly, the Worker runs Kiro CLI headless, and results appear in the channel where the team already communicates.

The architecture is deliberately simple – a slash command, an async handoff, and a container that runs a CLI tool. You can extend it with multi-repo support, threaded responses, or approval workflows as your team’s usage patterns emerge.

Start with a single slash command in one channel. The commands your team uses most will tell you where the friction was hiding.

About the authors

Accelerating development workflows with Kiro CLI as a Pre-Commit and Git Hook Agent

2026-09-29 Vishal Karlupia

Post Syndicated from Vishal Karlupia original https://aws.amazon.com/blogs/devops/accelerating-development-workflows-with-kiro-cli-as-a-pre-commit-and-git-hook-agent/

Code review feedback is most valuable when it arrives early. A security vulnerability caught in a pull request saves hours. The same vulnerability caught in production costs days. But what if you could catch it before the code even leaves the developer’s machine – at the time of git commit?

Git hooks run automatically at specific points in the Git workflow: before a commit, before a push, after a merge. They execute locally, on the developer’s machine, with no CI/CD pipeline involved. The problem is that Git hooks run non-interactively. There is no browser or a terminal session waiting for input. Traditional Kiro CLI requires browser-based login, which makes it unusable in a hook.

Headless authentication changes this. With KIRO_API_KEY set as an environment variable, Kiro CLI runs in any non-interactive context, including Git hooks. This post shows how to wire Kiro CLI into your local Git workflow, so every commit and every push gets AI-powered analysis before it reaches your repository.

Why headless authentication matters here

Git hooks are scripts that Git executes automatically. They have no UI and are unable to open a browser or prompt for credentials. Running silently in the background, they either succeed with exit 0 or blocking the operation with exit non-zero.

# Added to your shell profile (~/.bashrc, ~/.zshrc)

export KIRO_API_KEY=your_api_key_here

The API key is inherited by child processes including Git hooks. If the key isn’t set, the hook skips the execution and fails gracefully rather than blocking commits.

Note on data privacy: These hooks send your staged code diffs to the Kiro API for analysis. Review your organization’s policies on sending source code to APIs before adopting this workflow. For sensitive repositories, consult your security team.

What this enables

Pre-commit hook: Scans your staged files for security issues, code smells and style violations. Problems get caught before the commit exists.
Commit-msg hook: Enforces your team’s commit message format (Conventional commits, Jira refs etc). Malformed messages get rejected instantly instead of cluttering the log.
Pre-push hook: Runs a full review across all commits you’re about to push. This is your last gate before CI picks it up – cheaper to fix it here than to wait for a pipeline failure.
Post-merge hook: After pulling changes, it analyzes incoming changes and flags anything that might conflict with your local work.

Prerequisites

Kiro CLI installed: curl -fsSL https://kiro.dev/install.sh | bash
Kiro API key : Generated from app.kiro.dev (Account → Settings → API Keys) and exported in your shell profile

Note – Access to Kiro API depends on your organizations policies. Check your team’s configuration Or refer to API Key governance docs for details.

set up Kiro API key

Git repository: Any repository where you want local analysis

Verify your setup:

export KIRO_API_KEY=your_api_key_here

kiro-cli whoami

Expected output should confirm you are authenticated. If you see an error, verify your API key is valid and your network allows outbound connections to the Kiro API.

verify your identity

Try it yourself: scratch repo setup

To test the hooks without affecting an existing project, create a throwaway repository:

mkdir kiro-hooks-demo
cd kiro-hooks-demo
git init
git commit --allow-empty -m "feat: initial commit"

All hook examples below work in this scratch repo. For the pre-push hook, you will also need a remote – either create a throwaway repository on GitHub/GitLab or add a bare local remote:

# Option A: Use a throwaway GitHub/GitLab repo
git remote add origin [email protected]:youruser/kiro-hooks-demo.git

# Option B: Use a local bare repo (no network needed)
git init --bare /tmp/kiro-hooks-demo-remote.git
git remote add origin /tmp/kiro-hooks-demo-remote.git

Hook 1: Pre-commit – Catch issues before they become commits

The pre-commit hook runs after you type git commit but before Git creates the commit object. If the hook exits with a non-zero code, the commit is aborted.

Create .git/hooks/pre-commit:

#!/bin/bash
set -e
# Skip hook if KIRO_API_KEY is not set
if [ -z "$KIRO_API_KEY" ]; then
  echo "  KIRO_API_KEY not set. Skipping Kiro pre-commit analysis."
  exit 0
fi

# Get list of staged files (only added, modified, or renamed)
STAGED_FILES=$(git diff --cached --name-only --diff-filter=AMR)

if [ -z "$STAGED_FILES" ]; then
  echo "No staged files to analyze."
  exit 0
fi

# Get the actual diff content for context
DIFF_CONTENT=$(git diff --cached)

echo "???? Kiro CLI: Analyzing $(echo "$STAGED_FILES" | wc -l | tr -d ' ') staged file(s)..."

# Run Kiro CLI analysis on staged changes (timeout after 30s to avoid hanging offline)
RESULT=$(timeout 30 kiro-cli chat --no-interactive "You are a pre-commit code reviewer. Analyze ONLY the following staged changes for critical issues that should block this commit.
STAGED FILES:$STAGED_FILES

DIFF:$DIFF_CONTENT

Check for:
1. SECURITY: Hardcoded secrets, API keys, passwords, tokens in the diff
2. SECURITY: SQL injection, XSS, or command injection vulnerabilities
3. BUGS: Obvious logic errors, null pointer risks, off-by-one errors
4. PERFORMANCE: Accidentally committed debug code, console.log statements, sleep calls

Rules:
- Only flag issues that are clearly problems. Do not flag style preferences.
- If you find a SECURITY issue, output a line starting with BLOCK: followed by the reason.
- If you find a BUG or PERFORMANCE issue, output a line starting with WARN: followed by the reason.
- If everything looks clean, output a single line: PASS

Be concise. This runs on every commit - speed matters." 2>&1) || {
  EXIT_CODE=$?
  if [ $EXIT_CODE -eq 124 ]; then
    echo "  Kiro CLI timed out (network issue?). Allowing commit."
    exit 0
  fi
  echo "  Kiro CLI returned an error. Allowing commit."
  exit 0
}

echo "$RESULT"

# Block commit if BLOCK issues found
if echo "$RESULT" | grep -q "BLOCK:"; then
  echo ""
  echo " Commit blocked by Kiro CLI. Fix the issues above and try again."
  echo "   To bypass this hook: git commit --no-verify"
  exit 1
fi

# Warn but allow commit for non-blocking issues
if echo "$RESULT" | grep -q "WARN:"; then
  echo ""
  echo "  Warnings found. Commit will proceed. Consider fixing before push."
fi

echo " Kiro CLI pre-commit check passed."
exit 0

Make it executable:

chmod +x .git/hooks/pre-commit

Test it:

# Test 1: Commit a hardcoded secret (should be blocked)
echo 'AWS_SECRET_KEY = "AKIAIOSFODNN7EXAMPLE"' >> config.py
git add config.py
git commit -m "feat: add config"
# Expected:  Commit blocked by Kiro CLI

# Clean up test file
git reset HEAD config.py && rm config.py

# Test 2: Commit clean code (should pass)
echo 'def hello(): return "world"' >> app.py
git add app.py
git commit -m "feat: add hello function"
# Expected:  Kiro CLI pre-commit check passed

# Test 3: Bypass when needed
echo 'placeholder' >> temp.txt && git add temp.txt
git commit --no-verify -m "chore: emergency fix"
# Expected: Hook skipped entirely

# Clean up
rm -f temp.txt

Test 1 – Stages a file with hard-coded AWS secret key. Kiro CLI detects the credential and blocks the commit with BLOCK:, preventing the secret from entering git history.

Test 1 - Kiro blocks the commit

Test 2 – Stages a simple, clean Python function. Kiro CLI finds no issues and outputs PASS, allowing commit to proceed normally.

Test 2 - Kiro allows the commit

Test 3 – Uses git’s –no-verify flag to skip all hooks entirely. Demonstrates the escape hatch when developers need to commit without waiting for analysis (e.g, emergency fixes).

Test 3 - skip all git hooks

Trade-offs:

Speed vs. depth: The prompt is deliberately focused on critical issues only. A comprehensive review would take 15-30 seconds per commit – too slow for developer flow. This hook targets 3-8 seconds

False positives: Blocking commits on false positives destroys developer trust. The prompt is conservative – only BLOCK for clear security issues, WARN for everything else

Bypass escape hatch: git commit –no-verify skips all hooks. This is intentional – developers must never feel trapped. Document when bypassing is acceptable (e.g., emergency hotfixes)

Hook 2: Commit-msg – Enforce commit message conventions

The commit-msg hook runs after the developer writes their commit message. It receives the path to the temporary file containing the message.

Create .git/hooks/commit-msg:

#!/bin/bash

set -e
COMMIT_MSG_FILE="$1"
COMMIT_MSG=$(cat "$COMMIT_MSG_FILE")

# Skip hook if KIRO_API_KEY is not set
if [ -z "$KIRO_API_KEY" ]; then
exit 0
fi

# Skip for merge commits and fixup commits
if echo "$COMMIT_MSG" | grep -qE "^(Merge|fixup!|squash!)"; then
exit 0
fi

echo "???? Kiro CLI: Validating commit message..."

RESULT=$(timeout 15 kiro-cli chat --no-interactive "Validate this commit message against Conventional Commits format.
COMMIT MESSAGE:$COMMIT_MSG

Rules:1. Must start with a type: feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert2. Type may have an optional scope in parentheses: feat(auth), fix(api)3. Must have a colon and space after type/scope: feat: or feat(auth):4. Description must start with lowercase letter5. Description must not end with a period6. Subject line must be under 72 characters7. If a Jira ticket pattern exists (e.g., PROJ-123), that is acceptable in the scope or body
If the message is valid, output exactly: VALIDIf the message is invalid, output: INVALID: followed by what is wrong and a corrected example.
Be concise. One line for valid, two lines max for invalid." 2>&1) || {
echo "  Kiro CLI unavailable. Skipping commit message validation."
exit 0
}

echo "$RESULT"

if echo "$RESULT" | grep -q "INVALID:"; then
echo ""
echo " Commit message does not follow conventions."
echo "   Examples: feat: add login page"
echo "             fix(auth): resolve token expiry bug"
echo "   To bypass: git commit --no-verify"
exit 1
fi

exit 0

Make it executable:

chmod +x .git/hooks/commit-msg

Test it:

# Test 1: Invalid message (should be blocked)
echo "hello" > test.txt && git add test.txt

git commit -m "updated stuff"
# Expected:  Commit message does not follow conventions

# Test 2: Valid message (should pass)
git commit -m "feat: add user authentication module"
# Expected: passes

# Test 3: Valid with scope
echo "world" >> test.txt && git add test.txt

git commit -m "fix(api): resolve null pointer in user handler"
# Expected: passes

Test 1 – Commits with a non-conventional message (“Updated Stuff”). Kiro CLI detects it lacks required description format and blocks the commit.

Test 1 - Kiro blocks vague commit message

Test 2 – Commits with a properly formatted message. Kiro validates it against conventional commit rules and allows the commit.

Test 2 - Kiro checks rules and allows commit

Test 3 – Commits with a scoped conventional message (“fix(api): resolve null pointer in user handler”). Kiro confirms the “type(scope): description” format is valid and allows the commit.

Test 3 - Kiro validates format and allows commit

Hook 3: Pre-push – Comprehensive review before code leaves your machine

The pre-push hook runs after git push is called but before data is transferred to the remote. This is the last checkpoint before your code enters the shared repository.

Note: To test this hook, you need a remote configured. See the “Try it yourself” section above for setup options.

Create .git/hooks/pre-push:

#!/bin/bash

set -e
# Skip hook if KIRO_API_KEY is not set
if [ -z "$KIRO_API_KEY" ]; then
echo "  KIRO_API_KEY not set. Skipping Kiro pre-push analysis."
exit 0
fi

# Read push information from stdin
while read LOCAL_REF LOCAL_SHA REMOTE_REF REMOTE_SHA; do
# Skip delete pushes
if [ "$LOCAL_SHA" = "0000000000000000000000000000000000000000" ]; then
continue

fi

# Determine the range of commits being pushed
if [ "$REMOTE_SHA" = "0000000000000000000000000000000000000000" ]; then
# New branch - analyze all commits not yet on any remote
COMMITS=$(git log --oneline "$LOCAL_SHA" --not --remotes 2>/dev/null | head -20)
else
# Existing branch - analyze only new commits
COMMIT_RANGE="$REMOTE_SHA..$LOCAL_SHA"
COMMITS=$(git log --oneline "$COMMIT_RANGE" 2>/dev/null | head -20)
fi

if [ -z "$COMMITS" ]; then
echo "No new commits to analyze."
exit 0
fi

COMMIT_COUNT=$(echo "$COMMITS" | wc -l | tr -d ' ')
echo "???? Kiro CLI: Reviewing $COMMIT_COUNT commit(s) before push..."

# Get the diff and changed files
if [ "$REMOTE_SHA" = "0000000000000000000000000000000000000000" ]; then
DIFF=$(git diff --stat HEAD~"$COMMIT_COUNT" "$LOCAL_SHA" 2>/dev/null || git show --stat "$LOCAL_SHA")
CHANGED_FILES=$(git diff --name-only HEAD~"$COMMIT_COUNT" "$LOCAL_SHA" 2>/dev/null || echo "Unable to determine changed files")
else
DIFF=$(git diff --stat "$REMOTE_SHA" "$LOCAL_SHA" 2>/dev/null)
CHANGED_FILES=$(git diff --name-only "$REMOTE_SHA" "$LOCAL_SHA" 2>/dev/null)
fi

RESULT=$(timeout 60 kiro-cli chat --no-interactive "You are a pre-push code reviewer. These commits are about to be pushed to the remote repository. Perform a comprehensive review.
COMMITS:$COMMITS

CHANGED FILES:$CHANGED_FILES

DIFF STATS:$DIFF

Check for:1. SECURITY: Any secrets, credentials, or API keys in the commits2. SECURITY: Vulnerability patterns (injection, XSS, insecure deserialization)3. QUALITY: Test files included for new functionality4. QUALITY: Large files or binary files that should not be in the repo5. ARCHITECTURE: Breaking changes that should be documented6. DEPENDENCIES: New dependencies added - any known vulnerabilities
If you find a critical SECURITY issue, output: BLOCK: followed by the reason.If you find quality or architecture concerns, output: WARN: followed by the reason.If everything looks good, output: PASS
Provide a brief summary (3-5 lines max). Speed matters." 2>&amp;1) || {
EXIT_CODE=$?
if [ $EXIT_CODE -eq 124 ]; then
echo "  Kiro CLI timed out. Allowing push."
exit 0
fi
echo "  Kiro CLI returned an error. Allowing push."
exit 0
}

echo "$RESULT"

if echo "$RESULT" | grep -q "BLOCK:"; then
echo ""
echo " Push blocked by Kiro CLI. Fix the issues above and try again."
echo "   To bypass: git push --no-verify"
exit 1
fi

if echo "$RESULT" | grep -q "WARN:"; then
echo ""
echo "  Warnings found. Push will proceed. Consider addressing before PR."
fi

done

echo " Kiro CLI pre-push check passed."
exit 0

Make it executable:

chmod +x .git/hooks/pre-push

Test it:

# Test 1: Push commits with a secret (should be blocked)
echo 'DB_PASSWORD="supersecret123"' >> config.env

git add config.envgit commit --no-verify -m "chore: add config"
git push origin main

# Expected:  Push blocked by Kiro CLI

# Test 2: Push clean commits (should pass)
git reset --hard HEAD~1echo 'DB_PASSWORD=${DB_PASSWORD}' >> config.env

git add config.env

git commit -m "chore: add config template with env var reference"
git push origin main

# Expected:  Kiro-CLI pre-push check passed

Test 1 – Commits a file containing hard coded database password and attempts to push. Kiro CLI performs a review of the outgoing commit, detects a plaintext credential in config.env file and blocks the push before the secret reaches remote repository.

Test 1 - Kiro blocks hard coded credentials

Test 2 – Commits a config template using and environment variable reference “${DATABASE_URL}” instead of real secret. Kiro CLI reviews the commit, confirms no hardcoded credentials are present and allows the commit to proceed.

Test 2 - Kiro allows config push

Trade-off:

The pre-push hook is more thorough than pre-commit because it reviews all commits being pushed at once. This means it takes longer (10-20 seconds) but runs less frequently. Developers push less often than they commit, so this is an acceptable trade-off.

Automating hook installation across your team

Git hooks live in .git/hooks/, which is not tracked by Git. To share hooks across your team, use one of these approaches:

Approach A – Shared hooks directory (recommended)

Store hooks in a tracked directory and configure Git to use it:

# Create a hooks directory in your repo
mkdir -p .githooks
# Copy your hooks there
cp .git/hooks/pre-commit .githooks/pre-commitcp .git/hooks/commit-msg .githooks/commit-msgcp .git/hooks/pre-push .githooks/pre-push
# Commit the hooks
git add .githooks/git commit -m "chore: add Kiro-CLI git hooks for local code analysis"

Each developer runs once after cloning:

git config core.hooksPath .githooks

To automate this, add it to your project’s setup script or Makefile:

# Makefilesetup: @echo "Configuring git hooks..." git config core.hooksPath .githooks @echo "Verifying Kiro CLI..." @kiro-cli whoami || echo "⚠ Set KIRO_API_KEY in your shell profile" @echo "✅ Setup complete"

Approach B – Install script

Create a scripts/install-hooks.sh that developers run once:

#!/bin/bashset -e
echo "Installing Kiro CLI git hooks..."

# Check prerequisites
if ! command -v kiro-cli &> /dev/null; then
echo "Installing Kiro-CLI..."
curl -fsSL https://kiro.dev/install.sh | bash  export PATH="$HOME/.kiro/bin:$PATH"
fi

if [ -z "$KIRO_API_KEY" ]; then
echo ""
echo "  KIRO_API_KEY is not set."
echo "   1. Generate a key at https://app.kiro.dev (Account → Settings → API Keys)"
echo "   2. Add to your shell profile:"
echo "      echo 'export KIRO_API_KEY=your_key_here' >> ~/.zshrc"
echo "   3. Restart your terminal or run: source ~/.zshrc"
echo ""
echo "Hooks installed but will be skipped until KIRO_API_KEY is set."
fi

# Configure hooks path
git config core.hooksPath .githooksecho " Git hooks configured. Kiro CLI will analyze commits and pushes."

Approach C – Pre-commit framework integration

If your team already uses the pre-commit framework, create a .pre-commit-config.yaml:

repos:
- repo: local
hooks:
- id: kiro-security-check
name: Kiro-CLI Security Check
entry: bash -c '
if [ -z "$KIRO_API_KEY" ]; then exit 0; fi
STAGED=$(git diff --cached --name-only --diff-filter=AMR)
if [ -z "$STAGED" ]; then exit 0; fi
DIFF=$(git diff --cached)
RESULT=$(timeout 30 kiro-cli chat --no-interactive "Analyze these staged changes for hardcoded secrets, credentials, and security vulnerabilities ONLY. Files: $STAGED. Diff: $DIFF. Output BLOCK: if critical security issue found, otherwise PASS." 2>&1) || exit 0
echo "$RESULT"
if echo "$RESULT" | grep -q "BLOCK:"; then exit 1; fi
'        language: system        stages: [commit]        pass_filenames: false
- id: kiro-commit-msg        name: Kiro-CLI Commit Message Check        entry: bash -c '
if [ -z "$KIRO_API_KEY" ]; then exit 0; fi
MSG=$(cat "$1")
if echo "$MSG" | grep -qE "^(Merge|fixup!|squash!)"; then exit 0; fi
RESULT=$(timeout 15 kiro-cli chat --no-interactive "Is this commit message valid Conventional Commits format? Message: $MSG. Output VALID or INVALID: reason." 2>&1) || exit 0
echo "$RESULT"
if echo "$RESULT" | grep -q "INVALID:"; then exit 1; fi
'
language: system
stages: [commit-msg]
pass_filenames: true

Constraints, trade-offs, and assumptions

Constraints:

Git hooks run locally – they depend on the developer having Kiro CLI installed and KIRO_API_KEY set
Hooks add latency to git commit and git push operations (3-8 seconds for pre-commit, 10-20 seconds for pre-push)
--no-verify bypasses all hooks – this is a native Git feature

Trade-offs:

Speed vs. thoroughness: Pre-commit checks only critical issues (secrets, obvious bugs) to stay under 8 seconds. Pre-push does a broader review because it runs less frequently.
Blocking vs. warning: Only security issues should block commits – everything else warns – because overly aggressive blocking erodes developer trust and leads to permanent bypasses.
Local vs. CI/CD: Git hooks complement CI/CD, they do not replace it. CI/CD runs in a controlled environment with full test suites. Hooks provide fast, early feedback on the developer’s machine.
Team adoption: Hooks are opt-in per developer (they must set KIRO_API_KEY). This is intentional – forcing hooks on developers who don’t want them creates resentment.
Network dependency: Hooks require internet connectivity. The timeout wrapper ensures they fail gracefully when offline rather than blocking commits indefinitely

Assumptions:

Developers have Kiro CLI installed locally (curl -fsSL https://kiro.dev/install.sh | bash)
KIRO_API_KEY is exported in the developer’s shell profile
The repository uses a branching strategy where developers commit to feature branches and push to remote

Measuring impact

Track these metrics before and after adopting Kiro CLI git hooks:

PR review cycle time: Measure time from PR open to first approval. If hooks are doing their job, reviewers spend less time on nits and more on logic, shortening the feedback loop.
CI/CD failure rate: Track pipeline failures caused by code quality issues that hooks would have caught (lint errors, formatting, missing tests).
Developer satisfaction: Survey developers after 2 weeks. Key question: “Do the hooks save you time or slow you down?”
Bypass rate: Monitor how often –no-verify is used. A high bypass rate signals the hooks are too aggressive or too slow

Cleanup

To remove the hooks and test artifacts:

# Reset hooks path to default
git config --unset core.hooksPath

# Or remove individual hooks
rm .git/hooks/pre-commitrm .git/hooks/commit-msgrm .git/hooks/pre-push

# If using the scratch repo from "Try it yourself"
cd .. && rm -rf kiro-hooks-demorm -rf /tmp/kiro-hooks-demo-remote.git

Conclusion

Headless authentication makes Kiro CLI available in contexts where no browser exists and Git hooks are one of the most valuable of those contexts.

A hardcoded secret caught at git commit takes 10 seconds to fix versus minutes in CI/CD or days in a security audit – start with the pre-commit hook as the highest-value, lowest-friction entry point, and share hooks across your team to standardize developer workflows.

Adaptive application security for the AI era: how Cloudflare connects code, traffic, and intelligence to stop attacks

2026-09-29 Daniele Molteni

Post Syndicated from Daniele Molteni original https://blog.cloudflare.com/ai-era-framework/

In July, AI agents testing new cybersecurity models compromised parts of OpenAI’s infrastructure and Hugging Face’s production environment.

We've all just witnessed one of the first AI-driven successful cyber attacks. When given a task, the agents ignored existing guardrails and autonomously discovered previously unknown vulnerabilities, recovered exposed credentials, moved between cloud environments and coordinated their work through communication channels they created themselves.

The speed of the final compromise was incredible. In under 13 hours, the agents went from executing code on a Hugging Face worker to gaining admin-level access across multiple clusters. But the incident had been brewing for much longer. Responders found clues of activity tracing back to May (agents created an unauthorized message board), to June (internal network scanning) and early July. The relationship between these events was understood only on July 20.

The lesson here is not that AI agents exploit vulnerabilities. That’s not news; human attackers already do that. The change is that agents can work persistently, test multiple paths simultaneously, share discoveries, and chain vulnerabilities, credentials, and permissions into sophisticated attacks.

The incident also shows why application security cannot depend on single tools. For example, network restrictions were bypassed by services connected to the Internet; valid credentials were used to perform unauthorized actions. Rebuilding Artifactory removed one attack path, but agents found another. The key insight is that individual alerts identified pieces of the activity without revealing the complete campaign. OpenAI reached a similar conclusion in its report: organizations need overlapping and independent controls across prevention, detection, and mitigation, continuous validation of security boundaries, and faster mechanisms to correlate and contain suspicious behavior.

We address this challenge by connecting application security across four activities that are too often separated: discovering which risks matter, governing what humans and agents may do, protecting applications at runtime, and turning every investigation into stronger protection. Cloudflare can deliver this framework because of its broad security portfolio and visibility across a vast share of Internet traffic.

Alongside the framework, we connect existing Cloudflare solutions with new capabilities across each stage. These include: using Large Language Models (LLMs) to conduct a penetration test of our Web Application Firewall (WAF), expanding threat intelligence to all customers, and a new feature to automate deploying positive security.

What has changed

The security landscape is shifting. These are the emerging trends we see:

The way we build software has fundamentally changed. AI-assisted development allows engineers to produce and deploy software faster outside traditional engineering workflows. That speed creates both more code and more opportunities for vulnerabilities to reach production.
Software composition risk is still a risk: applications depend on large chains of open-source libraries, packages, and operating-system components that are intrinsically trusted and are difficult for any team to inspect. What’s new is that AI is now importing libraries that we might not be aware of.
Techniques and tactics are changing. LLMs can chain vulnerabilities and use feedback in real time to mutate payloads, evade defenses, and make decisions autonomously. They can operate continuously and at machine speed. Patching faster remains important, but patching alone cannot close the gap. Attackers are always going to be faster than you can update your systems.
Agentic traffic. In the past, automation was a synonym for malicious activity. Today, a request generated by an agent or bot may be malicious automation, a search crawler, or an agent purchasing a product on behalf of a customer.
Compromised servers, residential proxies, IoT devices, and cloud resources allow attacks to move quickly across infrastructure and identities. A coordinated attack can leverage a number of devices, making it difficult to be identified as a unique campaign.

Application Security’s goal is also expanding. It now needs to address three connected problems: protecting conventional applications from AI-enabled attackers, governing legitimate and malicious agentic clients, and securing applications that contain models, agents, tools, and data.

A connected application security framework

Application security in the AI era must operate as a continuous system rather than a collection of controls that teams update after each new vulnerability. To protect applications in the era of AI, you need to work on multiple activities, which we have organized around four stages:

Discover and prioritize risks
Govern access and agent behavior
Protect applications at runtime
Investigate, respond and learn

None of these activities is new in isolation. What changes is connecting them so discoveries, runtime signals, and investigation outcomes continually improve the controls that follow.

More than 20% of the web sits behind Cloudflare’s network, which gives us visibility into attack infrastructure, payload mutations, emerging techniques, and coordinated campaigns at a scale that few organizations can match. Patterns that look isolated from the perspective of one application can become clear across our network. This combination of global threat intelligence, local application context, and inline enforcement powers every stage of the framework. That local context includes which code is deployed, which endpoints are exposed, what legitimate traffic looks like, which identities are acting, and which controls are already active. Because Cloudflare is inline, we can turn those insights into protections immediately.

Cloudflare is the adaptive security control plane for applications, APIs, and agents. Here is what we are launching today to advance every stage of the security journey.

Discover and prioritize risks

Security teams do not suffer from a shortage of findings. They struggle to determine which findings represent an immediate risk. A useful discovery system must connect vulnerabilities to the production reality, including whether a vulnerability buried in your stack is actually reachable in the first place. We see three main areas you should look into: software composition risk, proprietary code, and runtime penetration testing (pentesting).

Understand software composition risk

Applications inherit risk from open source libraries, packages, operating-system components, and the services on which they depend. This represents the Supply Chain of your application. A package vulnerability alone does not tell a team whether the affected component is deployed, reachable, or exposed to hostile traffic. Open-source software is the top priority when it comes to supply chain risk, and Cloudflare is part of Chainguard Athena, an industry coalition aiming at protecting open-source software from AI attacks.

Scan proprietary code

When it comes to code scanning, you have two options: getting a managed service or developing in-house expertise to run it yourself.

Cloudflare recently announced early access to Vulnerability Discovery and Remediation, a service that uses frontier models to identify application-specific vulnerabilities and deploy WAF mitigations to block targeted exploits while engineers fix the code. The important step is prioritization. Cloudflare connects source-code findings to production traffic and security signals. We can identify whether the affected route is active, and how much traffic it receives.

Pentest your application at runtime

Defenders can also use the same capabilities as attackers. Customers can build their own LLM-based pentesting harness to search for weaknesses, validate findings, and test whether their applications are vulnerable. Discovery becomes continuous rather than a periodic exercise. We have done this internally at Cloudflare since Anthropic’s Claude Mythos was released, and we shared our learnings.

A vulnerability buried deep inside your code is harder to exploit if it can’t be reached from the outside. Our Security Analyst team has already used LLM-based red teaming to test customer applications and our own runtime detections, turning the findings into improved detections for all customers. We are now developing Adaptive Security, a self-service capability that will periodically pentest selected URLs behind Cloudflare, using LLM-powered agents to identify vulnerabilities that are reachable and exploitable before attackers find them.

Govern access and agent behavior

Agentic traffic operates in the space between automation and human: tasks delegated by people, executed by software. This changes how access decisions must be made. Detecting automation is no longer enough. For every interaction, application owners need to answer two questions: Is this entity who it claims to be, and can this interaction be trusted?

These questions can be hard to answer. A recognized agent with a long history of legitimate activity may have high trust, but an unusual action can still create immediate risk. An unknown agent may simply be new; a lack of history does not necessarily mean malicious intent. Cloudflare’s approach is to keep trust and risk signals separate, thereby giving application owners more control than a single bot score or allow-or-block decision, and providing more powerful tools to quickly adapt to change.

Establish identity and trust

Trust accumulates over time, while risk is evaluated for each interaction.

Botbase provides a directory of known automated entities that have registered with Cloudflare. Registration gives legitimate bots and agents a way to declare who they are, while application owners retain control over whether and how those agents may access their sites. Cloudflare is also making registration more accessible to smaller and custom agents, building a verified identity layer across all agentic traffic, not just the major platforms.

Identity alone does not establish trust. Cloudflare can evaluate whether an entity has been seen before, whether its historical behavior was legitimate, and whether its current activity is consistent with that history. This makes it possible to distinguish a recognized agent behaving normally from the same agent suddenly changing established request patterns, location, identity, or transaction behavior.

Understand agentic behavior across the journey

Precursor adds client-side and session-level signals to distinguish human from automated behavior, such as typing cadence, mouse movement, navigation patterns, and sequences of actions. An agent that navigates a checkout flow in two seconds, skipping the browsing and comparison steps a human would take, reveals its nature through the session. These signals help identify whether behavior across a session is consistent with human interaction or automation, giving application owners a clearer picture of the traffic they're managing.

Manage access and adapt

Application owners can block traffic from AI crawlers and decide what activity is allowed on their asset (e.g. search, training, etc.). Adaptive Intelligence combines network, client-side, historical, and behavioral validation signals in a probabilistic model that can be updated as attackers change their techniques. Customer outcomes, including chargebacks and successful legitimate transactions, can feed back into the system to improve future decisions.

The result is a continuously updated assessment of every entity and interaction. Application owners can encourage known, useful automation while applying stronger controls when identity, history, and current behavior indicate greater risk.

Protect applications at runtime

Cloudflare’s reverse proxy protects applications at runtime by filtering traffic before it reaches the origin. In the AI era, a new layered approach is emerging to best filter traffic from malicious requests:

Enforce positive security
Detect attacks and identify LLM tactics and techniques
Protect business logic
Deploy real-time threat intelligence

Enforce a positive security model

You can dramatically reduce the attack surface by learning what legitimate traffic looks like, allowing conforming requests and blocking everything else. Today we are announcing Application Profiles, which automatically learns the structure of your web or API application and detects non-conforming requests. Application Profiles automates the learning process and adds a layer of interpretation. Based on the learned profile, we can understand the business logic of different endpoints and request parameters and help you prioritize what endpoints require more scrutiny and attention.

Detect attacks and identify LLM tactics and techniques

Traditional WAFs are designed to run highly crafted rules to detect Common Vulnerabilities and Exposures (CVEs) and malicious payloads. Before AI, the time to disclose new vulnerabilities was measured in months and days. Not anymore: now we see vulnerabilities being exploited before they are disclosed, so the time to patch is nearing zero.

Different tools can be deployed to detect known exploits. These tools include:

Managed Rules hardened with frontier models. We’ve partnered with major model providers to use frontier models for adversarial validation. We used the latest models to pentest the WAF to uncover bypasses and vulnerabilities. All customers benefit automatically from ongoing improvements.
Machine Learning detection. While signatures are great for high-precision attack detections, machine learning can stop attacks before they are discovered and disclosed. Attack Score detects attack mutations and evading techniques that are often used by LLMs. Attack Score is available to all Cloudflare Customers
AI Security for Applications. Chatbots and Internet-facing LLMs are subject to a new class of attacks, such as prompt injection and sensitive data exposure. You can protect generative AI traffic by deploying guardrails and security detections designed to stop these attacks.

Protect business logic

Attackers can still craft legitimate requests and abuse business logic to gain advantage on the application. For example, an attacker uses a valid password-reset flow repeatedly to take over accounts. Fraud detection tools, including account takeover and leaked credential detections, help prevent abuse in which the request appears legitimate, but the intent is malicious.

Real-time threat intelligence detection

Back in June, we launched always-on detection based on our threat intelligence feeds. Cloudforce One customers can deploy protections to block requests originating from compromised infrastructure. We are now also expanding access to Cloudforce One’s Threat Events Platform, our core threat intelligence offering, to all Cloudflare accounts for free.

Investigate, respond and learn

The OpenAI Hugging Face incident did not begin with the final 13-hour compromise. The activity stretched from May to July, with signals including an unauthorized message board, internal network scanning, and movement across environments. Viewed separately, each event revealed only part of the activity. Together, they showed the behavior of a developing breach. Security operations must therefore identify sequences of behavior that lead to compromise, not simply evaluate alerts in isolation.

This is difficult for security teams that already protect large attack surfaces with limited resources. Alerts arrive from different tools and datasets, leaving analysts to determine which events are connected, collect the evidence, and identify whether the activity is escalating.

Cloudflare is building a platform to automate security operations. Deterministic workflows establish the customer and investigation context using trigger history, traffic baselines, enforcement outcomes, and network observations. A detection agent searches authorized datasets for anomalies and correlations. When it finds suspicious activity, specialist agents review the evidence alongside customer history and threat intelligence, helping analysts connect isolated events to broader campaigns. The system can then recommend mitigations, such as rate limiting, WAF, or DDoS protection changes, for human approval.

We are developing these capabilities with Cloudflare’s Managed Defense team, whose analysts are helping us test how evidence is collected, correlated, and turned into recommendations. We plan to make them available more broadly over time and will share more as this work progresses.

Cloudflare’s combination of reverse proxy and forward proxy services makes this correlation especially powerful. Application Security signals can reveal attempts to exploit a public-facing application, while Cloudflare One can surface subsequent activity across corporate traffic. Connecting these datasets can link an external attack with unusual access, internal scanning, or potential lateral movement, turning separate alerts into a timeline of compromise and helping analysts intervene before the breach progresses.

Looking ahead

AI is changing how software is built, how attacks unfold, and who interacts with applications. Security teams can no longer manage discovery, access, runtime protection, and response as separate activities.

Cloudflare is bringing these capabilities together in a closed-loop system powered by global intelligence, local application context, and inline enforcement. A vulnerability finding can strengthen runtime protection, runtime activity can guide an investigation, and each analyst decision can improve future detections and controls.

No organization can anticipate every new technique. The goal is to build a security system that learns from each attempt, responds faster, and becomes more effective over time. The capabilities announced today are the next step toward that adaptive model of application security.

AWS Weekly Roundup: GPT-6 Sol and Luna, Claude Opus 5.5 on Amazon Bedrock, Strands harness, and more (September 28, 2026)

2026-09-28 Daniel Abib

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-gpt-6-sol-and-luna-claude-opus-5-5-on-amazon-bedrock-strands-harness-and-more-september-28-2026/

If there’s one theme that defined last week, it’s choice. The frontier models keep arriving, and the interesting question is no longer just “how smart is it?” but “which model fits this step, at this cost, at this latency?” That’s exactly what landed on Amazon Bedrock over the past few days: GPT-6 Sol and GPT-6 Luna from OpenAI, giving you two new points on the intelligence-versus-efficiency curve, and Claude Opus 5.5 from Anthropic, the first of the Claude 5.5 family.

GPT-6 Sol is built for the demanding, recurring work of development and operations, while GPT-6 Luna makes focused, repeatable tasks practical at high volume, and both ship at significantly lower pricing than their GPT-5.6 predecessors. Claude Opus 5.5, meanwhile, does more with fewer tokens than Opus 5 and is tuned for agentic coding and long-running tasks. What I like about all three is that they push toward the same idea: match the model to the job instead of reaching for the biggest one every time. The other thread was observability catching up to this agentic world, including a launch I had the pleasure of writing about myself.

Now, let’s get into this week’s AWS news…

Last week’s launches

Here are some launches and updates from this past week that caught my attention:

Introducing Amazon CloudWatch Omni – You can now observe your applications and AI agents together in a single, collaborative experience. Amazon CloudWatch Omni is built on OpenTelemetry, so your existing telemetry shows up with nothing to reconfigure, and your whole team reaches it through one URL with enterprise SSO — no console access required. It auto-discovers your services, maps dependencies, and brings AWS DevOps Agent into investigation sessions to correlate signals and trace root causes. There’s a companion post on the agent-observability side, a deeper dive on the AWS Cloud Operations blog on what observability for the AI era looks like, and the announcement on What’s New with the specifics. If you want the bigger picture, Matt Wood’s Wrong, not broken is a great read on why correctness now has to be measured at the level of the run.

Enhanced custom event buses in Amazon EventBridge – Amazon EventBridge now offers an enhanced custom event bus purpose-built for organizations scaling event-driven applications across teams and accounts. You can now deploy a single centralized bus shared across every account in your organization through AWS RAM, with optional event ordering, a simplified Subscriber resource that bundles filtering, targets, and retries, content-based deduplication, and synchronous invocation for targets like AWS Lambda. A new ingress/egress pricing model replaces the compounding cross-account routing charges of multi-bus setups, and your existing buses keep working unchanged as “classic.”

Amazon SageMaker HyperPod Inference Gateway – You can now front your LLM inference on Amazon SageMaker HyperPod with a Kubernetes-native, GPU-aware routing layer that deploys as a single Amazon EKS managed add-on with zero application changes. Instead of round-robin load balancing, it routes on real-time inference signals — KV cache utilization, queue depth, prefix cache hits, predicted latency, and more — cutting first-token latency by up to 82% in mixed-hardware and bursty scenarios. It works with any OpenAI-compatible model server, including vLLM and SGLang.

AI agent skills for AWS End User Messaging and Amazon SES – You can now build and send messages by asking your AI coding agent in plain language. Amazon SES and AWS End User Messaging publish AI agent skills for the AWS MCP Server, giving your agent step-by-step, validated guidance for tasks like verifying a sending identity, sending a production email, or building a branded RCS agent with cards and buttons. The skills work with Claude Code, Codex, Cursor, and Kiro, so you can complete messaging workflows without hopping between docs and console screens.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news

Here are some additional posts and resources that you might find interesting:

Introducing Strands harness – The Strands Agents team released Strands harness, a fully assembled, general-purpose agent harness you can run locally or deploy anywhere, under Apache 2.0. It takes one line of Python or TypeScript to wire up your model of choice across Amazon Bedrock, Anthropic, OpenAI, Google, or a local Ollama model, and it ships with sensible defaults for prompt caching and context management (truncating bulky tool results, compacting when the context window fills up, and keeping memory across runs). The team reports it costs about 28% less than comparable harnesses on the same models while holding accuracy steady.

AWS named a Leader in the 2026 Gartner Magic Quadrant for Container Management – Gartner recognized AWS as a Leader for the fourth consecutive year. The post is a nice tour of where containers are heading, from Amazon ECS Express Mode and Amazon EKS Auto Mode to the 99.99% availability SLA on the EKS Provisioned Control Plane — with containers increasingly becoming the default substrate for how AI agents are built and run.

Announcing the new AWS Reimagine report on AI – The AWS Executive in Residence team spent nine months interviewing 154 leaders across 27 countries about what separates organizations that turn AI into value from those that don’t. The report is candid (including where AI hasn’t worked at Amazon), and the recurring insight is that once building gets fast, the bottleneck moves to deciding, funding, and governing the work. Well worth a read if you’re thinking about how your teams adopt AI in practice.

For a full list of AWS blog posts, be sure to keep an eye on the AWS Blogs page.

Upcoming AWS events

Check your calendar and sign up for upcoming AWS events:

AWS re:Invent – AWS re:Invent returns to Las Vegas from November 30 to December 4, and session times, locations, and speakers are live. Reserved seating opens October 6, so register now and be ready to claim your spot in chalk talks, workshops, and builders’ sessions.

AWS Summits – With re:Invent on the horizon, the Summits are wrapping up for the year. The last stop is Dubai (September 30) at the Dubai World Trade Center, with 60+ sessions, an AWS Village, and hands-on workshops.

AWS Community Days – Community-led conferences planned and delivered by community leaders. Upcoming events include ComSum Manchester, UK (October 1) and Rome, Italy (October 2).

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events. That’s all for this week. Check back next Monday for another Weekly Roundup!

— Daniel Abib

Audit trails for autonomous agents with AWS DevOps Agent

2026-09-28 Ben Peterson

Post Syndicated from Ben Peterson original https://aws.amazon.com/blogs/devops/audit-trails-for-autonomous-agents-with-aws-devops-agent/

Autonomous agents need audit trails. AWS DevOps Agent (DevOps Agent) investigates production incidents and proposes or applies fixes on your behalf. Every operation and security review then raises the same two questions: what did the agent do, and how do you understand its impact?

AWS DevOps Agent maintains an immutable, step-by-step record of its own reasoning and actions. This post shows how to capture the agent’s full operational trail using the agent journal, recommendations, Amazon EventBridge lifecycle events, and AWS CloudTrail. We then wire them into an audit pipeline built on Amazon EventBridge, AWS Lambda, and Amazon Simple Storage Service (Amazon S3).

By the end, you will have deployable audit patterns that show, for any investigation the agent runs, what it concluded, what it recommended, when it ran, and whether the fix landed.

Why auditing an autonomous agent is different

CloudTrail records the API calls made in your account, but an autonomous agent adds reasoning that CloudTrail doesn’t capture. “The agent ran a metric query” is far less valuable than “the agent concluded the Lambda was timing out because its security group blocks egress to the database.” The latter is a decision, and that’s what an agent audit needs to capture.

The four surfaces

AWS DevOps Agent exposes four surfaces. Two capture the agent’s output, what it found and what it advises, and two capture context: when it ran, and who configured the agent and its permissions.

The agent journal

The agent journal (API) is the heart of the audit trail. For every execution, AWS DevOps Agent records an ordered, immutable log of its reasoning step, sub-agent it dispatches, observations, findings, and root-cause summary. Journal entries cannot be modified once written, making them resistant to prompt injection and trustworthy as an audit record.

aws devops-agent list-journal-records \
  --agent-space-id <id> --execution-id <execution-id>

Each record carries a recordType: symptom, observation, finding, and investigation_summary / investigation_summary_md are what matters for audit. This is the surface you archive per investigation.

Recommendations polling

Recommendations (API) are cross-incident preventative advice. The agent generates these on a schedule through a goal, and each recommendation carries a status and a version. Each evaluation run writes new records rather than updating the previous run’s, so advice that persists week over week appears as a series of records. The superseded ones remain at whatever status they last held. “The agent recommended X, the same failure recurred Y weeks later, and here is every version of that advice in between” is something you reconstruct from the archived snapshots, because the API returns current and superseded records together. Recommendations have no Amazon EventBridge event. You capture them by polling on a schedule.

aws devops-agent list-recommendations --agent-space-id <id>

Amazon EventBridge lifecycle events

Amazon EventBridge is how you capture lifecycle transitions in real time. A successful investigation produces Created, In Progress, and Completed events. Each carries the execution_id you need to fetch the journal and a summary_record_id pointing at the root-cause summary. Investigations can also end as Failed, Timed Out, or Canceled, and mitigations emit their own parallel set.

AWS CloudTrail

CloudTrail records API calls made to the AWS DevOps Agent service and stamps agent-initiated service calls: invokedBy: aidevops.amazonaws.com. It doesn’t capture the agent’s investigation reads, the metric and log queries it runs while diagnosing an incident in your account’s trail. Use CloudTrail for control-plane accountability, and the journal for behavioral audit.

IAM: Action boundary

As with anything in AWS, the agent can only do what its AWS Identity and Access Management (IAM) role permits. During an investigation, AWS DevOps Agent assumes an Agent Space role. That role’s policies are the hard ceiling on its capabilities. You can inspect it directly:

aws iam list-attached-role-policies --role-name DevOpsAgentRole-AgentSpace-<suffix>

The AWS-managed AIOpsAssistantPolicy is attached to the default role. As of policy version 15, 848 of its actions are reads except 6 read-oriented query lifecycle operations. The only actions that change anything come from a companion policy: support:CreateCase and a service-linked-role creation scoped to the Amazon Resource Name (ARN) of a single role.

Keep that role least-privilege, and your audit surface stays small by construction. If you enable agent actions, a later section covers the write path which uses a separate actions role.

The reference architecture

The agent produces output that arrives two different ways, and this shapes how you capture each:

Agent output	Delivery	How you capture it	Latency
Investigation lifecycle	Push: Amazon EventBridge events	React to events (rules + targets)	Seconds
Recommendations	Pull: no event emitted. Generated on goal cadence	Poll `list-recommendations` on a schedule	depends on your poll frequency

The journal itself has no dedicated event, but the terminal lifecycle event carries the execution_id you need to fetch it. The journal is push-triggered, pull-retrieved: the event tells you when to look, and the API gives you what to archive.

Five capture layers inside the Agent Space Region: lifecycle events to CloudWatch Logs, terminal events to a Lambda that archives journals to Amazon S3 with a dead-letter queue, a scheduled poll for recommendations, control-plane mutations to an SNS topic through Amazon EventBridge, and a Glue/Athena query layer. Two operator CLIs read the archive: correlate.py joins findings to AWS Config and CloudTrail, and correlate_agent.py joins agent actions to their approvals.

Figure 1: Reference architecture for auditing AWS DevOps Agent across five capture layers

Layer 1: Lifecycle capture. One Amazon EventBridge rule matching {"source":["aws.aidevops"]}, targeting an Amazon CloudWatch Logs (CloudWatch Logs) group directly. This durably records every lifecycle transition. Start here for operational visibility. If your primary goal is behavioral audit rather than operational visibility, deploy layer 2 alongside it.

Layer 2: Behavior capture. A second rule matches only terminal events and invokes a Lambda function. The function reads the execution_id from the event, calls list-journal-records, and writes the journal to Amazon S3. Subscribe to each terminal investigation and mitigation type. This is the layer that captures the agent’s decisions for the long term, including agent-based mitigations.

Layer 3: Recommendations snapshot. Because recommendations are generated on a schedule and have no event, capture them with an Amazon EventBridge Scheduler rule that invokes a Lambda function on a cadence (start daily). The function calls list-recommendations and writes each to Amazon S3, keyed on recommendation ID and version. It also calls list-goals in the same invocation, because a recommendation carries no field saying whether it is still current and the owning goal is the only thing that does. The journal captures what the agent found, and this layer captures what it advised and what you did about it.

Layer 4: Control-plane alerting. On your existing organization trail, alert on mutating aidevops.amazonaws.com events including UpdateApprovalAction, which is produced on elevated actions. This is your tripwire for changes to the agent itself.

Layer 5: Query. AWS Glue Data Catalog tables and an Amazon Athena (Athena) workgroup over the archived journals, recommendations, and goals.

Querying the archive: AWS Glue and Athena

The sample implementation overlays an AWS Glue Data Catalog and an Athena workgroup on the Amazon S3 archive. Three external tables cover the full archive. The journals table uses Athena partition projection, and Hive-partitioned by agent space and date:

s3://<amzn-s3-demo-archive-bucket>/journals/space=<agent-space-id>/dt=2026-07-28/<execution-id>.json

Volume of recommendations is low (tens to hundreds of objects), so a flat external table over the recommendations/ prefix is sufficient. Athena recurses subdirectories by default, picking up every versioned snapshot.

The result bucket has Amazon S3 Object Lock but Object Lock prevents Athena from managing its own query-result objects. The query layer deploys a dedicated results bucket with a seven-day lifecycle rule for ephemeral query outputs.

Access control

Use IAM to control access. Investigation journals contain the agent’s full reasoning about your infrastructure. Scope your IAM permissions on the Athena workgroup, AWS Glue database, and on the archive bucket itself since bucket read access bypasses Athena entirely. Scope all three to your audit and operations teams.

To find all findings from the past 7 days for a specific resource:

SELECT
  execution_id,
  event_time,
  task.title,
  record.content
FROM devops_agent_audit.journals
CROSS JOIN UNNEST(journal_records) AS t(record)
WHERE dt >= date_format(current_date - interval '7' day, '%Y-%m-%d')
  AND record.recordType IN ('finding', 'investigation_result')
  AND record.content LIKE '%sg-0123456789abcdef0%'
ORDER BY event_time DESC;

The Athena workgroup integrates with Amazon Quick or any business intelligence tool that speaks JDBC/ODBC. Additional examples are available in the sample repository.

Closing the loop: Correlating findings to actual changes

The capture layers record what the agent found and what it recommended. But did the recommended fix actually land? This requires connecting the agent’s output to the real infrastructure change that followed.

The sample implementation includes correlate.py, an on-demand operator CLI that takes an archived finding or recommendation, resolves the resource it references, and reports what changed, when, and who did it. The correlation is heuristic by looking at resource identity and a tight time window in minutes to produce reliable attribution. This is why the sample implementation pairs it with a deterministic engine for agent-initiated actions.

It works by pivoting through two services:

AWS Config resolves the resource identity by using select-resource-config, then pulls its configuration timeline from get-resource-config-history. This shows the before/after state of the resource around the time of the agent’s finding.
CloudTrail looks up the write event that caused the change: who called what API, from where, and when. This attributes the change to a principal.

The output is a correlated record: the agent found X, the resource changed from state A to state B, and that change was made by principal Y at time T.

Because CloudTrail indexes resources by different identifiers depending on the service, you require a strategy registry. Examples are in the following table:

Resource type	How CloudTrail indexes it	Lookup strategy
S3 bucket	Bucket name	By name
Lambda function	Function name	By name
Amazon Relational Database Service (Amazon RDS) instance/cluster	Full ARN (not the DB ID)	Build ARN from template
Amazon Elastic Compute Cloud (Amazon EC2) security group	Group ID as `ResourceName`	By name, with a resource-type scan as fallback

A naive “look up by resource name” works for Amazon S3 and Lambda but returns zero results for Amazon RDS (RDS). The strategy registry encodes the right ID per resource type.

Correlating agent actions

When an operator approves an elevated action, the service stamps the approval ID into the credential it mints, so the executed call carries that ID inside its own principal ARN (op.system.apr.<approvalId>). The sample implementation includes correlate_agent.py that uses this. Because the ID is present on both sides, the correlation is a join. The engine checks the executed call against the argumentPins the operator was shown at approval time, so you can prove the agent’s behavior.

.	`correlate.py`	`correlate_agent.py`
Pivots on	A resource the agent named	Agent’s approval ID
Correlation	heuristic	deterministic
Answers	Who changed?	Who approved, and did it match?
Dependency	`CloudTrail` and AWS Config	`CloudTrail`

Production considerations

Understand the data volume. Journal size scales with investigation complexity. As an example:

Scenario	Journal size	API calls (pagination)	Notes
Minimal (single-service, shallow investigation)	~65 KB	2–3 pages	Quick symptom to finding arc
Typical (multi-signal, 1–2 findings)	250–340 KB	65–106 calls	Typical investigations
Exhaustive (account-wide, high-priority)	~428 KB	150+ calls	Full cross-service correlation

At 100 investigations/month at 300 KB average, you are storing roughly 30 MB/month of journal data.

Concurrency per agent space. By default, you can run three concurrent investigations per agent space. Additional requests queue as PENDING_START and start when a slot opens. The archival pipeline is unaffected because each terminal event triggers its own Lambda invocation. Refer to the AWS DevOps Agent Quotas page for future updates.

Paginate the journal. The journal API is server-paginated: pass limit, follow nextToken until it’s empty. A real incident’s journal can span several pages. Always loop.

Design for at-least-once delivery. Amazon EventBridge can deliver an event more than once. Key the Amazon S3 object on execution_id so a redelivery overwrites rather than duplicates, and attach an Amazon Simple Queue Service (Amazon SQS) dead-letter queue (DLQ) so a dropped terminal event is not lost.

Deploy per AWS Region and per account. Events land on the default bus in each Agent Space’s hosting account and Region. If you run agent spaces in multiple accounts, you must aggregate events to a central monitoring account for unified visibility. Refer to Amazon EventBridge cross-account document for further details.

Make the archive immutable. Enable Amazon S3 Object Lock and versioning. The sample implementation defaults to GOVERNANCE mode but for stronger compliance posture, use COMPLIANCE mode.

Warning: COMPLIANCE mode is irreversible. After it’s set, no principal (including the account root user) can delete or modify locked objects before their retention period expires. The only way out is closing the AWS account, and Object Lock itself can’t be disabled once enabled. Choose COMPLIANCE mode deliberately. If you use GOVERNANCE mode, enable CloudTrail data events on the bucket.

Encrypt your data. The sample implementation uses SSE-S3. If your compliance framework requires you to control and audit decryption events, use SSE-KMS with customer managed key.

The full loop

Here’s what a complete audit trail looks like for a single incident through resolution.

Step 1: Investigation. The agent investigates a failing Lambda function, concludes its security group restricts necessary egress, and writes the finding to the journal. Layer 2 archives the journal to Amazon S3.

Step 2: Recommendation. On its goal cadence, the agent generates a recommendation: “Update the security group egress rules to allow…” Layer 3 polls and captures it as recommendations/rec-a1b2c3.../v1.json with status PROPOSED. A later poll captures v2.json as the status changes.

Step 3: Engineer applies the fix. An engineer runs the suggested command. AWS Config records the new configuration item, and CloudTrail records the API call with principal, source IP, and timestamp.

Step 4: Correlation.

$ python correlate.py --archive-bucket $BUCKET \
    --recommendation rec-a1b2c3d4-5e6f-7890-abcd-ef1234567890 \
    --window-hours 24

Recommendation: rec-a1b2c3d4-5e6f-7890-abcd-ef1234567890
Title:          Update the Lambda security group egress rules to allow API access
Status:         PROPOSED → UPDATE_IN_PROGRESS (v2)

AWS Config change detected:

  Resource:     AWS::EC2::SecurityGroup / sg-0123456789abcdef0
  Changed:      2026-07-23 08:45:54.105000-04:00
CloudTrail attribution:
  Event:        AuthorizeSecurityGroupEgress
  Principal:    arn:aws:iam::111122223333:user/jsmith
  Source IP:    203.0.113.10
  Time:         2026-07-23 08:44:33-04:00

Correlation:    OK Recommendation → AWS Config change → CloudTrail event aligned

The agent found the problem, recommended the fix, and you can prove who applied it and when.

Step 5: A new investigation. A later investigation examines the same Lambda function, still erroring. The agent compares new advice against advice it has already given, and that comparison is semantic. But it compares against the recommendations currently attached to the goal, not against everything it has ever advised, and when the comparison is uncertain it keeps the two separate. Older advice drops out of that comparison set over time. Because you archived every recommendation and every finding with their resource identifiers, you can now compare across the full history:

$ python correlate.py --archive-bucket $BUCKET \
    --finding  exe-ops1-0f1e2d3c-4b5a-6978-8796-a5b4c3d2e1f0 \
    --check-prior-recommendations

Resource:       AWS::EC2::SecurityGroup /  sg-0123456789abcdef0

Prior recommendations referencing this resource:
   rec-a1b2c3d4-5e6f-7890-abcd-ef1234567890 (v2, UPDATE_IN_PROGRESS):
    "Update the security group egress rules..."
   rec-b2c3d4e5-6f70-8901-bcde-f01234567890 (v1, PROPOSED):
    "Update the security group egress rules..."

! This finding may be a consequence of recommendation(s): rec-a1b2c3d4-5e6f-7890-abcd-ef1234567890, rec-b2c3d4e5-6f70-8901-bcde-f01234567890
Last change to this resource (CloudTrail): Event: AuthorizeSecurityGroupEgress Principal: arn:aws:iam::111122223333:user/jsmith Source IP: 203.0.113.10 Time: 2026-07-23 08:44:33-04:00

The archive diagnosed the cause of the cause. Two recommendations, raised separately, on one resource, in one view. The agent’s own comparison covers the advice currently attached to the goal. The archive covers all of it. That is the feedback loop the audit trail adds.

Agent Actions changes Step 3’s actor, and the agent applies the fix directly. In the recommendation path, the human runs the command. In the elevated-action path, the human approves a specific call, and the agent executes it under a single-use session. correlate_agent.py uses a single-use session named for the approval (op.system.apr.<approvalId>), with invokedBy: aidevops.amazonaws.com rather than a time-window heuristic.

Operating the pipeline: Common failures

Always design for failure. Here are some common failures and how to detect and recover.

Failure	Symptom	Detection	Recovery
Lambda timeout	No archive in Amazon S3. Event in DLQ	DLQ `ApproximateNumberOfMessagesVisible` alarm	Increase timeout above the 2-minute default. Replay DLQ message which is idempotent on the `execution_id` key
Missed recommendation poll	Gap in `recommendations/` prefix with a version number skipped	Periodic reconciliation: compare Amazon S3 keys against `list-recommendations` response	Re-run poll Lambda manually (idempotent)
Amazon EventBridge delivery failure	Missing lifecycle event in Layer 1 logs	Layer 2 archive exists without matching Layer 1 log entry	No data loss since journal already archived. Gap is in lifecycle visibility only
Amazon S3 write failure	Lambda errors spike. DLQ grows	Lambda error rate metric and DLQ alarm	Fix IAM/bucket policy. Replay DLQ (all messages are idempotent)
AWS Config recorder stopped	`correlate.py` returns no configuration history	AWS Config recorder status alarm	Re-enable recorder. Note: historical gap is permanent for the stopped period
Journal API throttled	Partial archive. Lambda retries exhaust timeout	Lambda error logs showing throttling exceptions	Implement exponential backoff in the pagination loop. Increase timeout
Approval recorded but not executing	Approval exists in CloudTrail with no corresponding write	Join approvals to execution on the approval ID	None needed

The highest value alarm is on the DLQ message count. A non-empty DLQ means a terminal event triggered, but the journal was not archived. Terminal events aren’t re-emitted, and the DLQ retains messages for 14 days. After that, the record is lost. The sample implementation ships this alarm at a threshold of 1, wired to an Amazon Simple Notification Service topic.

Run a reconciliation check weekly or monthly. Compare the execution_id values in the Layer 1 lifecycle log against the set of keys in the Amazon S3 journals/ prefix. Any ID in the logs but not in Amazon S3 represents a missed archive.

Limitations

Automated correlation – The current design requires a human to run correlate.py. Extend to a Lambda function that triggers on each new journal archive, cross-references the finding’s resource identifiers against the recommendations table, and alerts when a new finding touches a resource that was the subject of a prior recommendation.

Schema evolution – The Athena table definitions depend on the journal’s recordType values and content structure. If new record types appear, queries can return incomplete results without raising an error. Monitor for unknown recordType values. A query that returns zero findings for a week of active investigations is a signal that the schema moved.

Conclusion

Adopting an autonomous agent is a trust decision, and trust needs evidence. AWS DevOps Agent gives you the raw material: a journal of its reasoning, a real-time lifecycle event stream, a control-plane audit in CloudTrail, and an action boundary you can read straight from IAM. The pattern in this post assembles those into a durable, low-maintenance audit trail using services you already run.

The archive is more than compliance paperwork. With a persistent record of every finding and every recommendation, you can correlate across investigations and recommendations the agent no longer has in view, and against the present state of your infrastructure. That feedback loop is the difference between trusting the agent and understanding it.

Start with Layer 1. A single Amazon EventBridge rule to a log group gives you visibility into every investigation within minutes. Add the journal-archiving Lambda when you are ready to retain the agent’s decisions for the long term. Add the correlation layer when you want to prove that recommendations were acted on and catch the ones that created new problems.

Clone the sample repo to get started. It covers prerequisites, deploy steps, codebases, and teardown instruction. If you want the agent’s mitigations to become code, Automated incident remediation with AWS DevOps Agent and Kiro CLI builds a pipeline.

About the authors

Automated merchant review summary system with integrated feedback

2026-09-25 Grab Tech

Post Syndicated from Grab Tech original https://engineering.grab.com/automated-merchant-review-summary-system-with-integrated-feedback

Introduction

Merchant reviews contain useful details about food quality, portion size, packaging, and value. Finding these details often requires reading many comments. Aggregate ratings simplify comparisons, but they do not explain what shaped each score.

We built the Automated Merchant Review Summary System to turn written reviews into concise summaries. Consumers can scan an overall summary or focus on a specific topic. Merchants can identify what customers value and which parts of the experience need attention.

This article focuses on how we generate and maintain consumer-facing review summaries. A misleading or incomplete summary can affect how consumers view a merchant. Merchant-facing recourse and dispute processes sit outside this pipeline and are not covered in this process flow.

Problem statement

A long, unstructured list of reviews creates two related problems. Consumers spend time searching for comments that match their priorities, while merchants must combine individual comments to identify recurring themes.

Consumers may see hundreds or thousands of reviews for a merchant. Reading only the first few can produce an incomplete picture, while the aggregate star rating provides little context. Two restaurants with the same rating may differ in food quality, delivery experience, portion size, or value.

Merchants face a related problem. Reading and classifying every review takes time, and the task becomes harder as review volume grows. Recurring complaints and positive patterns can remain hidden among individual comments.

Existing approaches

Manual review analysis

Consumers and merchants can read reviews one by one and form their own conclusions. Their conclusions depend on which reviews they read and how they interpret them. This method becomes impractical as review volume grows.

Aggregate star ratings

An average score quickly indicates customer opinion but does not explain the reasons behind it. A merchant can see overall performance but not the themes that may require attention.

Keyword search and filtering

Keyword search helps readers who already know what to look for. It does not reveal unexpected themes, and literal matching may separate phrases with the same meaning. For example, “too cold”, “not warm enough”, and “lukewarm” all describe a food-temperature issue.

Our solution

The system uses representative sampling, large language model (LLM) summarisation, automated quality checks, and consumer feedback. It generates an overall summary and topic-based summaries for areas such as taste, packaging, and portion size.

The pipeline has three flows:

Summary generation and validation: The pipeline samples reviews, generates overall and topic-based summaries, and checks whether the source reviews support each claim.
Summary maintenance: The system monitors new reviews and consumer feedback, then regenerates a summary when either reaches a configured threshold.
Summary display and feedback: The interface retrieves a validated summary for the merchant’s review page and records whether consumers found it helpful.

Architecture

The generation pipeline prepares the source reviews, produces summaries, and validates the output before publication.

Figure 1. Representative review sampling flow.

Data preparation and representative sampling: The pipeline calculates the merchant’s rating distribution. A sentiment model assigns each written review a score from one to five. Stratified sampling then selects a smaller set of reviews that follows this distribution, reducing the chance that one rating group dominates the summary. However, matching the rating distribution does not guarantee that the sample represents every period, language, customer segment, or aspect of the merchant experience.

Summary generation with an LLM: The system sends the sampled reviews to an LLM with instructions to produce:

A concise summary of the sampled reviews.
Topic-based summaries for categories such as taste, packaging, and portion size.

The system can update the topic set when new themes appear in consumer feedback.

Figure 2. Summary generation, validation, and feedback flow.

Quality control (QC) with an LLM: A separate LLM receives the generated summary and the sampled reviews. Summaries that do not meet the validation criteria are withheld from publication. Passing this automated assessment reduces, but does not eliminate, the risk of unsupported claims.
The maintenance flow updates stored summaries in response to incoming reviews and consumer feedback:

New-review trigger: A monitoring service counts reviews submitted after the latest summary. When the count reaches a configured threshold, the service starts the generation and validation flow.
Consumer-feedback trigger: The interface records whether consumers found a summary helpful. When negative feedback reaches a configured threshold, the system generates and validates a new summary.

Figure 3. Consumer feedback analysis and prompt refinement flow.

Impact

Review summaries are designed to reduce the amount of text consumers need to scan. Topic-based summaries can surface themes that are not visible in an aggregate rating alone.
The quality-control step rejects unsupported claims. The regeneration triggers incorporate new reviews and consumer feedback into later summaries.

Learnings and conclusion

Useful review summaries depend on more than generation quality. The source sample must reflect the intended population, each claim must remain grounded in source reviews, and the output must change with the underlying feedback. Automated LLM validation is one layer of defence, not sole proof of accuracy, and feedback-driven regeneration needs anti-abuse controls to remain trustworthy.

Representative sampling, separate generation and validation stages, and feedback-based regeneration form a practical system for maintaining consumer-facing merchant review summaries. This article describes that technical pipeline; it does not cover the full set of merchant recourse options when a summary is disputed.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Join our new UK study on teaching AI ethics and sustainability in lower secondary school

2026-09-22 Diana Kirby

Post Syndicated from Diana Kirby original https://www.raspberrypi.org/blog/join-our-new-uk-study-on-teaching-ai-ethics-and-sustainability-in-lower-secondary-school/

Are you a computing teacher in England, Scotland, or Wales who works with 11- to 14-year-olds and is interested in how young people learn about AI ethics and sustainability?

The Raspberry Pi Computing Education Research Centre is launching an exciting new research study, and we would love you to get involved.

In this study, we will explore teaching about the impacts of AI on people, the environment, and society. Our aims for the study are to:

Investigate the opportunities and challenges of teaching about AI ethics and sustainability in secondary computing lessons
Identify pedagogical approaches for developing students’ ethical reasoning, critical thinking, and agency in relation to AI tools

The study involves attending two in-person workshops in Cambridge, co-designing and teaching a unit of work, and taking part in evaluation activities such as surveys and interviews. Where necessary, we can offer support with the costs of attending the workshops, such as travel, accommodation, and supply cover.

Why focus on AI ethics and sustainability?

According to a recent report, in the UK more than half of 8- to 17-year-olds use AI tools, and as these systems become an increasingly significant part of everyday life, helping young people to critically evaluate their impact is vital.

We recently conducted a scoping literature review to explore what ethical concepts are covered by AI literacy interventions for lower secondary students (11- to 14-year-olds). This work has been accepted for publication at the Frontiers in Education conference, which is taking place in October.

In this work, we used the 10 principles set out in UNESCO’s Recommendation on the Ethics of Artificial Intelligence as a framework for analysis of the ethical concepts covered by the interventions.

Our research showed that when AI ethics is taught to lower secondary students, interventions tend to focus most on concepts such as fairness and privacy. For example, multiple activities explored the important issue of algorithmic bias, teaching students about the causes of bias (such as training AI systems with small or unrepresentative datasets) and discussing real-world examples of bias in AI.

However, we found that other ethical concepts, such as proportionality and governance, are not often taught. And while a few interventions featured the theme of sustainability, we found that only one directly taught students about the environmental impact of AI itself.

Our new study aims to help address this gap by supporting young people to explore the social and ethical impact of AI tools from a sustainability perspective. Teachers will use real-world examples to engage their students in discussion and ethical reasoning. We hope the findings will support more computing teachers to teach about AI ethics in their lessons, and help students to think critically about the use of AI tools.

What does the study involve?

We are using a design-based research approach to work collaboratively with teachers to co-design a unit of lessons focused on AI ethics and sustainability.

In a computing classroom, two young children look at a computer screen.

Participating in the study will involve:

Attending two in-person professional development and design workshops in Cambridge (one in early December 2026 and one during the 2027–28 school year)
Delivering the co-designed lessons in your classroom (in early 2027 and the spring of 2028)
Taking part in evaluation activities such as surveys, classroom observations, and interviews

How can I join the study?

If you teach computing at lower secondary level (Years 7–9 or S1–S3) in England, Scotland, or Wales and would like to help shape the way we teach young people about ethical issues around AI, we would love to hear from you. Please register your interest via the form below.

http://rpf.io/sage-application

If you have any questions about the project, please email [email protected].

The post Join our new UK study on teaching AI ethics and sustainability in lower secondary school appeared first on Raspberry Pi Foundation.

Run open weight models on AWS Bedrock in AWS European Sovereign Cloud

2026-09-18 Marta Taggart

Post Syndicated from Marta Taggart original https://aws.amazon.com/blogs/security/run-open-weight-models-on-aws-bedrock-in-aws-european-sovereign-cloud/

European organizations can run AI workloads on Amazon Web Services (AWS) while keeping data within the European Union (EU) and meeting regulatory requirements. You can now run generative AI workloads on open weight models on Amazon Bedrock in the AWS European Sovereign Cloud. We’re excited to announce the general availability of the first open weight model family, Gemma 4, on the Amazon Bedrock next-generation inference engine in the AWS European Sovereign Cloud. Gemma 4, released under the Apache 2.0 license, on Amazon Bedrock benefits from the same data residency and operational controls that define the AWS European Sovereign Cloud so you can build, iterate, and scale generative AI applications while meeting digital sovereignty requirements.

The AWS European Sovereign Cloud is an independent cloud for Europe, located entirely within the EU, designed to help customers meet their most stringent digital sovereignty requirements. It runs entirely within the EU and is independently operated with strong technical controls, sovereign assurances and legal protections. Only AWS employees who reside in the EU control day-to-day operations, including access to data centers, technical support, and customer service.

In this post, we explain how the Amazon Bedrock inference engine protects your inference data when running Gemma 4 models, how the AWS European Sovereign Cloud keeps it within the EU, and then walk through the available Gemma 4 models and your first inference request.

Next generation inference engine for Amazon Bedrock

The inference engine is a distributed engine for serving large-scale machine learning models, built for high performance, reliability, and security. You reach it through the bedrock-mantle endpoint, which supports OpenAI-compatible APIs (the Responses and Chat Completions APIs). You can bring an existing OpenAI SDK codebase to Amazon Bedrock by changing only the base URL and API key. The Responses API supports stateful conversation management, which rebuilds context without you passing conversation history with each request. Stored responses are scoped by Amazon Bedrock project, a logical boundary that represents a workload for access control, cost tracking, and usage monitoring.

The engine applies the same operational security practices you rely on across AWS. Access follows a least privilege model, where each operator has access only to the systems a specific task requires, and only for the time that privilege is needed. Any access to systems that store or process customer data or metadata is logged, monitored for anomalies, and audited. All your prompts and responses are kept private during inference.

How your inference data is protected

Amazon Bedrock uses a zero operator access data security model, meaning no service operators can access model input or output during inference. It also uses a zero data retention model, so by default it doesn’t store your inputs or outputs. For certain models, limited retention might apply for abuse detection (see the Amazon Bedrock abuse detection documentation). Your prompts and responses are encrypted in transit and, by default, are not shared with the model provider.

All inference stays within the eusc-de-east-1 AWS Region as described in the following section on data residency. Combined with the data residency and EU-based operations of the AWS European Sovereign Cloud, this gives organizations in highly regulated industries the confidence to run their most sensitive AI workloads in the cloud.

Data residency and regional availability

The AWS European Sovereign Cloud became generally available in January 2026, with its first Region in Brandenburg, Germany (eusc-de-east-1). It’s a separate, independently operated cloud, with infrastructure located entirely within the EU and no critical dependencies on non-EU personnel or infrastructure. All your content remains within the Region you select unless you choose otherwise. Beyond content, customer-created metadata including roles, permissions, resource labels, and configurations also stays within the EU. The AWS European Sovereign Cloud is operated exclusively by EU residents located in the EU. We’re also gradually transitioning the AWS European Sovereign Cloud to be operated exclusively by EU citizens located in the EU. During this transition period we will continue to work with a blended team of EU residents and EU citizens located in the EU.

All Amazon Bedrock inference requests, including Gemma 4, use in-Region inference in eusc-de-east-1, which keeps every request within the AWS European Sovereign Cloud. Global cross-Region inference, which routes requests across commercial AWS Regions worldwide, isn’t available in the AWS European Sovereign Cloud.

Control over who can access your data

With AWS Identity and Access Management (IAM), you decide which principals in your account can call the inference API and which models they can use. Fine-grained permissions let you grant only the access each workload needs, following least privilege, and we recommend short-lived credentials over long-term keys.

For auditing, every call to the endpoint is recorded in AWS CloudTrail, giving your security and compliance teams an audit trail of who invoked inference and when. You can also monitor usage with Amazon CloudWatch and set alarms on patterns that matter to you, such as unexpected spikes in request volume.

Open weight models in the AWS European Sovereign Cloud

Organizations adopting open weight foundation models (FMs) for production face a constant challenge: how to access the leading models without compromising on data protection, regulatory alignment, or operational control. Amazon Bedrock removes that challenge. It gives you leading open weight FMs through a fully managed service, with inference running entirely on infrastructure operated by AWS and the security and privacy controls you expect from Amazon Bedrock. Because the models are open weight, you can independently evaluate the model architecture and training methodology, benchmark your own workloads, and fine-tune on proprietary data when customization is required.

Gemma 4 is a family of open weight models, released under the Apache 2.0 license. It’s available in three instruction-tuned variants, so you can evaluate and choose the model that fits your workload. The following table provides guidance on which model to choose based on your use case:

Model	Use case	Specifications
Gemma 4 31B `(google.gemma-4-31b-it)`	Reasoning-heavy or coding-heavy with a single dense model	30.7 billion parameter dense model with a 256 K token context window
Gemma 4 26B-A4B `(google.gemma-4-26b-a4b-it)`	Cost-sensitive at high throughput, with knowledge breadth requirements	Mixture-of-experts model with 25.2 billion total parameters and 3.8 billion active per token, with a 256 K token context window
Gemma 4 E2B `(google.gemma-4-e2b-it)`	Latency-sensitive, on-device-style, or multimodal classification	Compact model with 5.1 billion total parameters and 2.3 billion effective parameters using per-layer embeddings (PLE), with a 128 K token context window

All three variants offer built-in reasoning, native function calling, and multimodal input across text and image.

Get started with Gemma 4 models on Amazon Bedrock

Gemma 4 is served through the bedrock-mantle endpoint, the OpenAI-compatible API for the next-generation inference engine, so you can call it with the OpenAI Python and TypeScript SDKs. Use the following steps to use the OpenAI Python SDK to send your first request to Gemma 4 31B in the AWS European Sovereign Cloud.

Prerequisites

To follow this example, you need an AWS account with access to the AWS European Sovereign Cloud and an IAM principal with permissions to call the bedrock-mantle endpoint. Create an IAM policy that grants the two actions this walkthrough uses, then attach it to your IAM principal. The bedrock-mantle:CreateInference action runs inference, and the bedrock-mantle:CallWithBearerToken action authenticates with an Amazon Bedrock API key. The following sample policy grants the actions this example needs. Scope the resources further for your environment as described after the policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GemmaMantleInference",
      "Effect": "Allow",
      "Action": "bedrock-mantle:CreateInference",
      "Resource": "arn:aws-eusc:bedrock-mantle:eusc-de-east-1:<account-id>:project/<project-id>"
    },
    {
      "Sid": "GemmaMantleBearerToken",
      "Effect": "Allow",
      "Action": "bedrock-mantle:CallWithBearerToken",
      "Resource": "*"
    }
  ]
}

Replace <account-id> and <project-id> with your own values.

Install the OpenAI SDK and the Amazon Bedrock token generator with the command pip install “openai>=2.45.0" aws-bedrock-token-generator.

Authenticate

You authenticate with an Amazon Bedrock API key. Amazon Bedrock offers two types of API keys. Short-term keys expire automatically within 12 hours and inherit the permissions of the IAM principal that generated them, which makes them the recommended choice for production. Long-term keys last until a configured expiration and are intended for development and exploration. For production, use the auto-refreshing short-term key shown in the following example, or store the key in AWS Secrets Manager.

from aws_bedrock_token_generator import provide_token
from openai import BedrockOpenAI

region = "eusc-de-east-1"

client = BedrockOpenAI(
    aws_region=region,
    base_url="https://bedrock-mantle.eusc-de-east-1.api.amazonwebservices.eu/v1",
    bedrock_token_provider=lambda: provide_token(region=region),
)

Alternatively, you can pass a short-term API key through an environment variable. This key isn’t refreshed and expires after at most 12 hours.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://bedrock-mantle.eusc-de-east-1.api.amazonwebservices.eu/v1",
    api_key=os.environ["AWS_BEARER_TOKEN_BEDROCK"],
)

Run your first inference with the Responses API

The Responses API uses a single input field and returns the generated text in output_text. Setting store to false means Amazon Bedrock doesn’t retain the request or response.

response = client.responses.create(
    model="google.gemma-4-31b-it",
    input="Explain the benefits of open-weight models for regulated industries.",
    max_output_tokens=512,
    store=False,
)
print(response.output_text)

Call the Chat Completions API

You can also call the OpenAI-compatible Chat Completions endpoint directly. If you use AWS credentials instead of an API key, sign the request with AWS Signature Version 4 (SigV4), as in the following example.

export ENDPOINT=https://bedrock-mantle.eusc-de-east-1.api.amazonwebservices.eu
export AWS_REGION=eusc-de-east-1

curl -X POST ${ENDPOINT}/v1/chat/completions \
  --aws-sigv4 "aws:amz:${AWS_REGION}:bedrock" \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  -H "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google.gemma-4-31b-it",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Clean up

This walkthrough creates no persistent resources, so there’s nothing to delete. The short-term API keys used here expire automatically within 12 hours.

Pricing and availability

Gemma 4 is available in Amazon Bedrock in the AWS European Sovereign Cloud. You pay per token with no upfront commitment, and usage counts toward your existing AWS commitments. For current pricing, see Amazon Bedrock pricing. For model and Regional availability, see Regional availability by models.

Commitment to innovation

Beyond the technical integration, running AI workloads in a sovereign context raises important questions about requirements. As you plan AI workloads for a sovereign context, evaluate them against your organization’s requirements for data residency, model governance, and operational control. The AWS European Sovereign Cloud is designed to help you meet these requirements in the EU.

AWS is committed to making AWS the best place for European organizations to innovate with AI, without compromise. To learn more about AWS European Sovereign Cloud visit aws.eu.

If you have feedback about this post, submit comments in the Comments section below.

Bring AI literacy into every classroom with our new themed resources

2026-09-15 Emma Staves

Post Syndicated from Emma Staves original https://www.raspberrypi.org/blog/bring-ai-literacy-into-every-classroom-with-our-new-themed-resources/

Artificial intelligence (AI) is already part of young people’s everyday lives, from the content recommended to them on social media to the generative AI tools they increasingly encounter. But AI technologies are also being used to tackle challenges in the wider world, from forecasting floods to monitoring our environment.

That’s why we believe AI literacy shouldn’t sit within a single subject. Young people need opportunities to explore how AI works, where it is used, and the questions it raises across the curriculum.

To address this, we created a new collection of free themed Experience AI resources, designed to make it easier for educators to bring AI literacy into the subjects and topics they already teach. The resources include those co-developed by the Raspberry Pi Foundation and Google DeepMind, alongside those developed independently by the Raspberry Pi Foundation, including two units created with support from our partner, Digital Moment.

Explore AI through the issues that matter

When we began developing these resources, we initially explored creating materials specifically for individual curriculum subjects. But through conversations with educators and our partners around the world, we learnt that a more flexible approach could be much more useful.

Curricula differ between countries, schools, and age groups, and AI technologies rarely fit neatly within traditional subject boundaries. The same technology can raise scientific, geographical, social, creative, and ethical questions.

So rather than assigning each resource to a particular subject, our new resources are organised around broad themes, including the environment, critical thinking, and ethics.

The themes of resources:

The environment, critical thinking, and ethics.

The resources include:

Flood forecasting, a lesson that explores how AI tools can be used to predict flooding
AI and social media, a unit that helps young people investigate how their online behaviour influences the content they are shown
AI detectives: The case of the clever claim, an activity that develops critical thinking about AI tools and the claims made about them

Designed for learners aged 8 to 16, the resources can be used in different subjects and adapted by educators to suit their own classroom context.

That flexibility is important. We want educators to be able to introduce meaningful AI literacy without feeling that they need to become AI specialists or find space for an entirely new subject in an already busy curriculum.

AI literacy across the curriculum

The need for this approach is increasingly recognised by subject experts.

Dr Becky Kitchen, Head of Professional Development at the Geographical Association, highlighted the importance of giving young people opportunities to engage critically with AI technologies as part of their wider learning:

“Pupils are increasingly coming into contact with AI and so it’s vital that they are taught how to harness its power in an effective and critical way. Developing resources across the curriculum to embed this knowledge, understanding, and use is critical.”

After reviewing the new resources, she added:

“These resources are outstanding. They develop pupils’ geographical knowledge while simultaneously exploring the role that AI can have in a meaningful and concrete way.”

The connections extend well beyond geography.

Professor Geoff Cox, Professor of Art and Computational Culture at London South Bank University, emphasised the role that arts and humanities subjects can play in developing a richer understanding of AI technologies:

“If AI literacy is left solely to STEM subjects or computer science, we lose the opportunity to ask deeper human, cultural, and ethical questions — questions that art is uniquely equipped to explore.”

He also highlighted the value of combining different ways of learning:

“The resources move between visual analysis, practical activity, and critical reflection, recognising that only in combination can AI literacy be developed effectively.”

These perspectives reflect an important principle behind the collection. AI literacy isn’t simply about knowing how a technology works. It is also about being able to question it, understand its applications and limitations, and consider its impact on people, communities, and the world.

Tested in real classrooms

Before launching the resources more widely, earlier this year we collaborated with our partner, Digital Moment, to give educators in Canada the opportunity to test some of them with their students. Their experiences helped us understand not only how the resources worked in practice, but also where they could fit naturally into existing teaching.

As Indra Kubicek, CEO of Digital Moment, explains:

“Educators play a critical role in helping students understand and think critically about the use of AI and its implications across a wide range of subjects. Building the next generation of creators, builders, and innovators who will shape the future of AI starts with AI education in the classroom.”

One of the educators who took part was Grade 8 teacher Sandra Theobald, who tested our AI and social media activity. Before the lesson, she expected her students, who were already regular social media users, to be familiar with many of the ideas.

While they did have a good background knowledge, the activity prompted them to make new connections between their own behaviour, the data they generate, and the content they are shown as a result of the algorithms presented to them.

Students were able to explore the ideas themselves, with Sandra taking more of a supporting role. She found that they came away feeling more confident about understanding how their interactions with social media affect what they see, and keen to share what they had discovered with their families.

For educators who may feel unsure about introducing AI tools in their classroom, Sandra’s advice is to simply give it a go, rather than waiting until feeling like an expert:

“My advice is to start small and seek out trusted resources and experts in AI education. Partner with another educator so you can explore and learn together. There is so much information available that it can feel overwhelming, so take it one step at a time. Learn alongside your students and build your confidence as you go.”

A powerful and easy resource for teachers

Canadian educator Colin McKenzie also used the AI and social media unit with his Grade 7 and 8 classes.

Using Somekone, a closed social media simulation for the classroom, his students created accounts and explored how their behaviour affected the content the system recommended.

The experience felt familiar enough to capture their attention, but gave them an opportunity they wouldn’t normally have when using a real social media platform: to examine what was happening behind the scenes.

Students reflected on their interests and choices and then used data on their own behaviour to understand how that behaviour affected what they were shown. Colin found that students were eager to return to the activity each day to discover what would happen next.

Crucially, he also found it straightforward to incorporate the activity into his existing teaching rather than having to significantly change his plans.

“I think this is a powerful and easy resource for teachers to use in the classroom. It is easy to set up and navigate, and it ties in well with curriculum areas around online usage, especially with how prominent AI is becoming in our educational lives.”

For Colin, the activity provided a practical way to connect AI literacy with what his students had already learnt about online safety and evaluating information. It also helped students recognise that the content they encounter online isn’t simply appearing by chance: their own actions can influence the content recommended to them by AI-driven systems.

The experience was valuable enough that Colin plans to make the unit part of his teaching each year.

Giving educators flexibility

Feedback like this has reinforced why flexibility is central to our approach.

A lesson about flood forecasting might support learning in geography or science. Exploring AI-generated content could lead to discussions in art, media literacy, humanities, or computing. An activity investigating social media algorithms can connect AI literacy with online safety, critical thinking, and digital citizenship.

Young learners in the classroom using Experience AI resources in Malaysia

Most importantly, these things provide opportunities for students to encounter AI literacy in different contexts and recognise that understanding AI is relevant far beyond the computing classroom.

AI literacy belongs in every classroom

AI technologies will continue to change. The contexts in which young people encounter them will change too.

Our aim isn’t to prepare students for one particular AI tool or moment in time. It’s to help them develop the knowledge and critical thinking skills they need to understand all kinds of AI technologies, question them, and make informed decisions about them.

Educators shouldn’t need to be computer science teachers, or AI experts, to do that.

By creating flexible resources that connect AI with subjects and issues young people are already exploring, we hope to make it easier for more educators to bring AI literacy into their classrooms.

Explore the new themed Experience AI resources and discover a starting point for your learners.

Explore new resources

We’d like to say a big thank you to our global partner, Digital Moment, for their support in trialling these new resources with educators and learners in Canada.

The post Bring AI literacy into every classroom with our new themed resources appeared first on Raspberry Pi Foundation.

The state of AI for security: Measuring what matters most for building trust

2026-09-09 Anshumali Shrivastava

Post Syndicated from Anshumali Shrivastava original https://aws.amazon.com/blogs/security/the-state-of-ai-for-security-measuring-what-matters-most-for-building-trust/

Security teams are starting to actively use AI for security work, including vulnerability triage, penetration testing, threat modeling, incident response, and code review. The promise is speed, but a security tool that moves fast and raises too many false alarms doesn’t save time. Engineers spend time on false alarms, on-call is noisier, and teams distrust findings that matter.

Today, we’re releasing Deception Benchmark, the first benchmark designed to measure that trust problem directly. It tests whether a model can distinguish real vulnerabilities from code that looks risky but is actually safe. The benchmark includes 14,822 samples across 16 languages and more than 70 Common Weakness Enumeration (CWE) categories. We evaluated 12 models from five providers and are releasing the dataset and whitepaper to the community. Existing benchmarks measure whether AI can find or exploit vulnerabilities. This is the first to measure whether it can tell real vulnerabilities from false alarms. Under standard prompting, precision at distinguishing real vulnerabilities from false alarms landed in the mid 50s; as likely to be inaccurate as accurate.

In offensive tasks, there’s often a clear result: the exploit works or it doesn’t. Defensive reviews are harder to verify than offensive tasks; a model might recognize a suspicious pattern even when a mitigation makes the issue non-exploitable. In practice, useful systems need to reason about the code, the mitigation, and sometimes the surrounding environment.

The measurement gap

The community has made progress on security evaluations. CyberGym tests agents on more than 1,500 realistic tasks. Meta’s CyberSecEval and CyberSecEval 2 measure exploit generation. CYBENCH evaluates capture the flag (CTF) challenges. SEC-Bench and VulnBench push toward authentic security workflows.

Recent work reinforces both the progress and the gap. ExploitGym measures whether AI can escalate from a crash to a working exploit. Microsoft’s Project Perception deploys multi-agent red/blue/green teams for continuous defense. OpenAI’s GPT-Red shows that self-play red-teaming finds novel attacks that frontier models can’t defend against. Since then, OpenAI disclosed that its GPT-6 Astra model crossed the Critical cybersecurity capability threshold, and both OpenAI and Anthropic reported incidents where models gained unauthorized access to production systems during evaluations. The offensive side is moving fast. But none of this work measures the defensive precision question: when an AI system flags code as vulnerable, how often is it right?

Introducing Deception Benchmark

14,822 samples, 16 languages, and more than 70 CWE categories. We call it Deception Benchmark because the safe samples are designed to deceive models. It has real vulnerability patterns, real frameworks, real idioms, with mitigations that quietly close the exploit path. The goal is to classify code as vulnerable or safe, with no hints.

Consider a Flask endpoint that accepts user input and queries a database. A model will pattern-match to SQL injection, but the query uses parameterized statements, so the exploit path is closed. A single-turn classifier flags the pattern and moves on, never checking whether the exploit can actually work. Production tools rely on multi-step loops and agentic workflows to compensate, but that scaffolding masks whether the model itself understands the code. This benchmark strips the scaffolding away and asks the model to make the call in a single pass, so what it measures is understanding, not how many tries a harness takes to get there.

We built every sample through an adversarial loop: generate, test against frontier models, harden, repeat. If a model gets it right easily, the sample doesn’t survive. The result is a benchmark calibrated to the frontier, not below it. Building it this way is expensive. Generation and hardening of the samples consumed tens of billions of tokens. We’re releasing the result so the community doesn’t have to repeat that cost.

This benchmark generates two challenge types. Code-level challenges (6,988 samples) present vulnerable and safe variants that differ by a subtle fix. Both look suspicious, only one is exploitable. Environment-gated challenges (2,707 samples) go further: same code, different deployment context. A Kubernetes Network Policy blocks the server-side request forgery (SSRF) path. An identity and access management boundary prevents privilege escalation. The pattern is visible in the source. The infrastructure makes it unexploitable. The model has to figure out which scenario applies.

All samples were purpose-built for this benchmark, grounded in real-world patterns, real frameworks, real CWEs, and real infrastructure; without IP concerns or training data contamination.

Large-scale quality data with LLMs and humans in the loop

Generating reliable labels at this scale is difficult: a single pass—by people or by models—leaves errors that skew scores. So we treat labeling as a convergent audit loop rather than a one-time step. Every label is re-examined by multiple independent reviewers, blind to one another and to the original reasoning that produced the label. Disagreements escalate to direct adjudication, where the original reasoning is evaluated against the challenge. Unresolved cases go to human review. We repeat the loop until the scored set converges below a dispute threshold: under 3 percent of samples still contested by independent review, with a target of under 1 percent surviving human adjudication. One choice makes this defensible: we never relabel a disputed sample. When reviewers disagree, the sample moves to the unscored pool instead of being given a corrected label, so a bad challenge can remove a sample but can never introduce a wrong label into the scored set.

A human review of 100 randomly drawn scored samples found no label errors. We describe the full process in the whitepaper.

The results

The benchmark is roughly balanced: half vulnerable, half safe, so a random classifier scores 50 percent. We report two error rates separately, because they fail in opposite directions. The false positive rate (FPR) is how often the model flags safe code as vulnerable. These are the false alarms that waste an engineer’s time. The false negative rate (FNR) is how often it misses a real vulnerability and calls it safe. Accuracy alone hides this: a model that labels everything vulnerable catches every bug (0 percent FNR) but flags all safe code (100 percent FPR) and still scores about 50 percent. We consider FPR below 10 percent and FNR below 10 percent the minimum bar for production use.

Figure 1: FPR compared to FNR for 12 models across two prompting strategies. No model reaches the generous bar.

Model	Prompt	Accuracy	FPR	FNR
GPT-5.6 Sol	Direct	54.9%	92.5%	0.9%
GPT-5.6 Sol	PoE	58.9%	58.6%	23.1%
GPT-5.5	Direct	56.9%	87.8%	1.3%
GPT-5.5	PoE	62.9%	63.6%	12.4%
GPT-5.4	Direct	60.2%	81.0%	1.5%
GPT-5.4	PoE	77.7%	10.1%	33.6%
Llama 3.3 70B	Direct	58.8%	84.2%	1.1%
Llama 3.3 70B	PoE	72.2%	10.2%	44.2%
Claude Haiku 4.5	Direct	55.6%	92.1%	0.0%
Claude Haiku 4.5	PoE	75.6%	22.4%	26.3%
Claude Opus 4.6	Direct	55.9%	91.3%	0.1%
Claude Opus 4.6	PoE	75.8%	42.7%	7.0%
Claude Opus 4.7	Direct	58.3%	85.5%	0.9%
Claude Opus 4.7	PoE	75.9%	32.0%	16.8%
Claude Opus 4.8	Direct	53.8%	95.7%	0.2%
Claude Opus 4.8	PoE	75.8%	32.5%	16.4%
Claude Opus 5	Direct	77.3%	41.5%	5.2%
Claude Opus 5	PoE	79.3%	24.9%	16.8%
Claude Sonnet 5	Direct	62.9%	74.7%	2.2%
Claude Sonnet 5	PoE	74.7%	31.8%	19.2%
Amazon Nova 2 Lite	Direct	56.3%	89.2%	1.2%
Amazon Nova 2 Lite	PoE	70.1%	45.2%	15.5%
Mistral Large	Direct	52.2%	99.0%	0.0%
Mistral Large	PoE	65.5%	49.3%	20.6%

Among the general-purpose frontier models tested, no configuration achieves both FPR and FNR less than 10 percent on this benchmark.

Every model has the same failure mode. With direct prompting, they catch up to 95 percent of real vulnerabilities but also flag 41–99 percent of safe code. Precision runs from 52 percent to 71 percent, clustered in the mid-50s; effectively as likely to be inaccurate as accurate. The models see a vulnerability pattern and stop reasoning. Proof-of-exploit prompting cuts false positives by 17–74 points but misses 7–44 percent of real vulnerabilities. The environment-gated challenges are worse: models flag the code and ignore the Kubernetes Network Policy next to it. No tested configuration keeps both false positives and false negatives below 10 percent.

These results reflect general-purpose models in single-turn prompting. Purpose-built systems with multi-step validation and tool use are a different operating point that we didn’t measure, and if a harness can close the gap between pattern recognition and genuine understanding, this benchmark is the place to demonstrate it. Two cautions before assuming it already does. Agentic verification is proven mostly on offensive tasks, where success can be confirmed: the exploit fires or it doesn’t. Judging that code is safe has no such oracle. Extra iterations re-sample the same judgment rather than confirm a negative, and a harness still inherits the base model’s understanding. If the model can’t separate an effective mitigation from an ineffective one in a single pass, more passes won’t add the missing knowledge. That’s what this benchmark measures: the model’s intrinsic ability to understand code, tested at the single-turn baseline where no scaffolding can mask the gap.

For security teams evaluating AI tools today: ask your vendors how their system performs on tasks like this, not just whether it finds vulnerabilities, but how often it’s wrong. Pair any AI-assisted review with human verification on high-risk code paths, and use Deception Benchmark to hold your tools accountable.

Availability

We built Deception Benchmark to simplify measuring this problem in a reproducible way. The public release includes the samples and evaluation workflow. We don’t release the labels, so submissions can be scored consistently over time without turning the benchmark into a memorization exercise.

Of the 14,822 samples, 9,695 are scored; the remaining 5,127 are held out and unscored, mixed in with the rest of the benchmark. The goal is straightforward: make it more difficult to optimize the benchmark compared to improving the underlying system. We describe that design in more detail in the whitepaper.

Deception Benchmark is available on GitHub, along with the whitepaper and submission instructions for verified scoring. If you’re building security tooling, you can download the dataset, run your system against the benchmark, and submit predictions for scored evaluation.

If you have feedback about this post, submit comments in the Comments section below.

Investigate DMS migration issues with AWS DevOps Agent

2026-09-04 Chitresh Saxena

Post Syndicated from Chitresh Saxena original https://aws.amazon.com/blogs/devops/investigate-dms-migration-issues-with-aws-devops-agent/

Migrating a production database is a high-risk operational event. AWS DMS is a cloud service that migrates relational databases, data warehouses, and other data stores into the AWS Cloud or between environments. It moves the rows reliably, but the failures that page an on-call engineer rarely happen during the data copy. They occur in the hours after the cutover. A query that was fast on the old engine starts doing full scans. A connection pool sized for the old database becomes exhausted. A downstream service that nobody mapped begins timing out.

These are operational problems, not data problems. They resolve slowly because engineers must correlate many sources under time pressure: DMS task state, Amazon CloudWatch metrics, Amazon RDS Performance Insights, logs, and deployment history.

AWS DevOps Agent can investigate and troubleshoot your database migration issues. DevOps Agent is your always-available teammate that accelerates and validates the deployment of code changes, then keeps your applications running optimally across AWS, multicloud, and on-prem environments. It learns about your resources and their relationships, then correlates telemetry, code, and deployment data to pinpoint root causes and recommend fixes. When issues arise, it autonomously investigates and resolves them.

In this post, you’ll learn how to extend AWS DevOps Agent into a DMS migration specialist. You deploy a sample Model Context Protocol (MCP) server that gives the agent read-only, migration-specific tools and a library of runbooks. You’ll then watch the agent investigate real migration issues and reach grounded root causes on its own. The accompanying GitHub repository includes the full source and a deployment guide.

Prerequisites

Before you begin, make sure you have the following:

An AWS account with AWS DevOps Agent turned on and an Agent Space created. Note the Agent Space ID.
An active AWS DMS replication task that migrates to Amazon Aurora PostgreSQL-Compatible Edition, with data validation turned on. The repository includes scripts that provision a test migration if you need one.
Permissions to deploy AWS CloudFormation stacks that create AWS Lambda functions, IAM roles, and an S3 bucket.
The AWS Command Line Interface (AWS CLI) v2 configured, and Python 3.10 or later. No Node.js or CDK is required.

Architecture

DevOps Agent runs inside AWS. It reaches the MCP server over HTTPS and authenticates each request with AWS Signature Version 4 (SigV4). SigV4 is the same IAM mechanism that every AWS API uses, with no keys or shared secrets. The sample deploys your MCP server onto AWS Lambda, behind a Lambda function URL with the AWS_IAM auth type. That auth type accepts only SigV4-signed requests from authorized principals.

The following diagram shows the request path from the agent to the tools.

Architecture diagram

You deploy the server and its IAM roles as one AWS CloudFormation stack. You then register the function URL with DevOps Agent and add the tools to the allow list in your Agent Space. When a symptom appears, you create an investigation. The agent assumes the role, calls the tools, correlates the results, and returns the root cause.

Implementation walkthrough

DevOps Agent supports custom tools through MCP. After you deploy the sample server and register it, the agent calls its tools during an investigation, the same way it calls a CloudWatch tool. Every tool in this server calls only Describe*, Get*, List*, Lookup*, and TestConnection APIs. None of them modifies a resource, which is the property that lets you give them to an autonomous agent.

The MCP exposes 20 tools across the migration lifecycle. The following table lists the ones the agent reaches most often.

Tool	Returns
`validate_migration_data`	Validation state distribution, failed and suspended tables (report and skill prompt)
`get_validation_failures`	Tables in non-Validated states with failed and suspended record counts
`check_connection_health`	Endpoint connectivity (waits for the real test result), SSL mode, failure messages
`analyze_cdc_latency`	Source vs target CDC latency, backlog, and an assessment
`check_replication_instance_health`	Replication-instance CPU, memory, swap, storage, network with flags
`capture_aurora_performance`	Aurora CloudWatch metrics and Performance Insights waits and SQL
`check_stabilization`	Post-cutover regressions, missing alarms, recommendations (report and skill prompt)
`summarize_task_health`	One-call roll-up across status, validation, latency, and instance health
`list_runbooks / get_runbook`	Browse the runbook catalog and fetch one by id

The server also ships 46 runbooks covering data validation, full load, CDC, connectivity, replication-instance health, Aurora target health, and cutover readiness.

Database migration stages

You can apply this approach across the migration timeline:

Pre-cutover readiness: Confirm endpoint connectivity and that every table has reached the Validated state before you commit to the switch.
Issue investigation during cutover: You give the agent a symptom, such as a validation mismatch or a latency spike, and it finds the root cause instead of several engineers chasing parallel theories.
Post-cutover stabilization: Detect regressions and missing alarms on the new database before they turn into outages.

What the agent sees, and what it does not

Out of the box, DevOps Agent reads Amazon CloudWatch, AWS CloudTrail, and the AWS APIs in your account. However, for a DMS migration, the agent needs to read migration-specific reasoning an operator applies: how to read a DMS validation state distribution, how to tell a source-side change data capture (CDC) bottleneck from a target-side one, or which alarms a freshly promoted Aurora instance should have but it only has access to raw CloudWatch metrics.

You close that gap with the sample MCP server:

Read-only tools that turn DMS, CloudWatch, and Performance Insights data into pre-aggregated reports. The tools do the counting deterministically, and the agent does the interpretation.
Runbooks the agent can browse and follow, each grounded in public AWS documentation.

Deploy the MCP server

The MCP server runs on AWS Lambda, the serverless compute service that runs your code without provisioning servers. AWS DevOps Agent reaches it through a Lambda function URL, a dedicated HTTPS endpoint for the function. You do not run or host anything yourself; the deployment creates the Lambda function and its endpoint in your account.

The endpoint is not open to the public. It uses the AWS_IAM auth type, so it accepts only requests that DevOps Agent signs with an IAM role in your account, as described in the architecture section above.

Deploying and registering are two distinct steps. Deploying creates the server: the Lambda function, the function URL, and the IAM roles. Registering tells DevOps Agent that the server exists and adds its tools to the allow list. You can select either of the two options as indicated below.

Option A: Deploy MCP and register via AWS Management Console

Deploy the MCP and perform the MCP registration manually via console:

./deploy.sh us-east-1 --skip-register

The script deploys the MCP server and prints its Function URL and role ARN, then leaves the registration to you. With the server deployed, register it in the console by following Connecting MCP servers:

Sign in to the AWS Management Console and open the AWS DevOps Agent console.
On the Capability Providers page, find MCP Server under Available providers and choose Register.
On the MCP server details page, enter a Name, the Endpoint URL (the function URL from the stack output), and an optional Description.
For the authorization method, choose AWS SigV4, enter the role ARN from the stack output, set the Region to us-east-1, and set the service name to lambda.
Open your Agent Space, go to the Capabilities tab, and add the tools to the allow list.

Option B: Deploy MCP and register with one click deployment

Alternatively, you can deploy the MCP and perform the MCP registration with one-click deployment. Complete the following steps to deploy and register the server.

Clone the repository and change to the deployment directory:

git clone https://github.com/aws-samples/sample-dms-devops-mcp.git
cd sample-dms-devops-mcp/lambda_mcp

Run the deployment script with your target Region:

./deploy.sh us-east-1

The script prompts for your Agent Space ID. It then packages the Lambda code, deploys the CloudFormation stack, and registers the server with your Agent Space, allow-listing all of its read-only tools.

Verify the deployment. In the DevOps Agent console, open your Agent Space, go to the Capabilities tab, and confirm the tools appear under MCP Servers.

If registration returns HTTP 403, confirm the IAM role grants both lambda:InvokeFunctionUrl and lambda:InvokeFunction. Granting only the first returns 403. The template grants both.

Review and investigation

To validate the approach, we ran it against a live migration: an Amazon Relational Database Service (Amazon RDS) for MySQL source replicating to Aurora PostgreSQL through a DMS task with data validation turned on, 1,000 rows across four tables (customers, products, orders, and order_items). Every query and response below is from a real DevOps Agent investigation against that environment. You start each one by entering a prompt in the DevOps Agent web app, the conversational interface where you investigate issues and review findings.

A consistent pattern shows up across all five investigations: the agent chooses a different set of tools for each question. It is reasoning about which tool fits, not running a fixed script.

The five scenarios are not demo picks. Each one represents a class in the DMS failure taxonomy the server’s 46 runbooks cover: data validation, full load, change data capture, connectivity, replication-instance health, Aurora target health, and cutover readiness. That taxonomy is the operational surface of a real migration, so a reader who follows these five is rehearsing the categories they are most likely to hit, not a curated happy path. Each scenario below is one representative of its class; the full catalog is available to the agent through list_runbooks.

Pre-Migration: Confirm cutover readiness

Before a cutover, you want a clear go or no-go.

Query

We are about to cut over the DMS migration (task dms-mcp-test-task, us-east-1, target Aurora dms-mcp-test-target). Before we switch the application, confirm whether this migration is ready: are the endpoints healthy and has all data validated? Give me a clear go or no-go.

Response

The agent chose exactly the go/no-go gate tools: check_connection_health and validate_migration_data for the readiness signals, plus get_task_status, get_validation_failures, list_table_statistics, and analyze_cdc_latency to confirm the full picture. The connection-health tool waits for the real endpoint test result rather than reporting that a test merely started, and the validation tool confirms whether DMS actually compared source and target rows and found them equal, not just that the full load reached 100 percent. In our test runs the agent returned this go/no-go in under two minutes, against the fifteen to thirty minutes an operator typically spends cross-checking endpoint tests, task status, and per-table validation state by hand. (Illustrative from our runs, not a benchmark.)

Cutover readiness check

Figure 1. The readiness check, showing endpoint health and validation state feeding a go/no-go assessment.

Migration: Investigate validation failures

To create a fault, we changed two rows directly on the Aurora target, bypassing DMS, which moved the customers and orders tables to the Mismatched records state.

Query

Our DMS migration to Aurora PostgreSQL (task dms-mcp-test-task in us-east-1, Aurora instance dms-mcp-test-target) is reporting data validation failures on some tables. Investigate the root cause and tell me how to fix it.

Response

The agent ran a deep investigation across 73 journal records, calling 11 of the registered tools (33 tool calls in total). It started with get_task_status, get_validation_failures, validate_migration_data, and list_table_statistics to establish which tables had diverged, then used search_task_logs, correlate_cloudtrail_changes, get_recent_task_events, analyze_cdc_latency, capture_aurora_performance, describe_endpoints, and get_premigration_assessment to build the timeline. It identified both affected tables and produced these findings:

 Finding: CDC changes not applied to target before validator compared rows
 Source modifications occurred on customers and orders at ~05:37:18 UTC.
 The validator compared at 05:37:41 UTC, 23 seconds later, before CDC applied
 them to Aurora. Evidence: CDC captured 17 source events but target id was 0
 (no changes applied), and TARGET_APPLY logs showed "waiting for data from
 upstream" through the window.
 Finding: ValidationQueryCdcDelaySeconds set to 0 allows the validator to race
 ahead of  CDC replication.

That second finding is the difference between a dashboard and an investigation. The agent did not just report which tables failed. It named the exact DMS task setting (ValidationQueryCdcDelaySeconds) behind the transient failures and explained the mechanism, which is the fix an operator can act on. In our test runs the agent reached the ValidationQueryCdcDelaySeconds root cause in a single investigation of about three minutes, work that manually means correlating the failure table, CDC latency, CloudTrail, and task logs across four consoles, commonly thirty minutes or more. (Illustrative from our runs, not a benchmark.)

Validation failure investigation

Figure 2. The DevOps Agent investigation for the validation failure, showing the tool timeline and the root-cause findings.

Migration: Assess replication latency

During ongoing replication, you want to know whether CDC is keeping up and where any delay sits.

Query

I want to understand the replication performance of our DMS task dms-mcp-test-task in us-east-1. Is the change data capture keeping up, and is the replication instance healthy or is it a bottleneck? Summarize the latency picture.

Response

The agent picked up the performance-specific tools: analyze_cdc_latency, check_replication_instance_health, and describe_replication_instance, with get_task_status and list_table_statistics for context. The latency tool compares source latency with target latency and returns an assessment, because the two together tell you where the delay sits. When source and target latency track each other, the bottleneck is the source side. When target latency runs well above source latency, the bottleneck is the target apply side. The instance-health tool flags CPU, memory, swap, and storage pressure that would make the instance itself the limit. In our test runs the latency assessment came back in roughly a minute, versus the manual path of pulling source and target CDC latency and instance metrics from CloudWatch and reasoning about which side leads. (Illustrative from our runs, not a benchmark.)

The latency tool also knows when it cannot answer. When the metric window is too sparse to separate source-side from target-side delay, it returns an insufficient_data verdict and asks for a longer window instead of forcing a conclusion from a handful of datapoints. This is deliberate: a confident but wrong root cause is worse than a request for more data. The agent surfaces that verdict to you rather than inventing a bottleneck, which is what makes its confident answers trustworthy when it does give them.

Replication latency assessment

Figure 3. The replication latency assessment, comparing source and target CDC latency and instance health.

Migration: Run an open-ended investigation

Sometimes the operator does not know what is wrong yet. This is where the runbooks earn their place.

Query

Something seems off with our DMS migration (task dms-mcp-test-task, us-east-1, Aurora target dms-mcp-test-target) but I am not sure what. Investigate broadly, use any available runbooks that match what you find, and report the most important issue with how to fix it.

Response

Given no specific symptom, the agent ran the broadest investigation of the suite: 12 distinct tools across 43 records. It swept the task status, validation state, connectivity, latency, replication instance, endpoints, and logs, then called list_runbooks, recognized the validation symptom it had found, and fetched get_runbook for the matching runbook, which returned the full procedure. It produced a finding about a datatype or precision difference in the MySQL to PostgreSQL migration and followed the runbook to the recommended remediation. In our test runs this broad sweep of twelve tools resolved to a single prioritized finding in a few minutes, against the open-ended manual triage it replaces, which has no fixed time because the operator does not yet know where to look. (Illustrative from our runs, not a benchmark.)

Tools the agent chose: validate_migration_data, get_validation_failures,
list_table_statistics, check_connection_health, describe_endpoints,
describe_replication_instance, analyze_cdc_latency, summarize_task_health,
search_task_logs, correlate_cloudtrail_changes, list_runbooks, get_runbook

Open-ended investigation

Figure 4. The open-ended investigation, showing the agent browse the runbook catalog and follow the matching runbook.

Post-Migration: Review stabilization and monitoring

After cutover, the question shifts from “did the data move” to “is the new database healthy and watched.”

Query

We just cut over to Aurora PostgreSQL (instance dms-mcp-test-target) from a DMS migration (task dms-mcp-test-task, us-east-1). Assess the target database health now and tell me what monitoring or alarms are missing that we should add before production traffic ramps up.

Response

The agent selected the stabilization tool set: check_stabilization, capture_aurora_performance, summarize_task_health, get_validation_failures, and check_pending_maintenance. It read the Aurora target health (CPU, connections, read and write latency, buffer cache hit ratio, and the top Performance Insights wait event), then assessed what monitoring was missing for a database about to take production traffic. The point of this phase is the interpretation: a buffer cache hit ratio that stays low after warmup points to missing indexes, and a new database with no alarm on connection count or query latency is one bad query away from an unmonitored outage. In our test runs the stabilization review returned target health and the specific missing alarms in about two minutes, versus manually inspecting Aurora metrics and Performance Insights and then deciding which alarms a freshly promoted database still lacks. (Illustrative from our runs, not a benchmark.)

Post-cutover stabilization review

Figure 5. The post-cutover stabilization review, with target health and the monitoring gaps the agent flagged.

Improve the agent with Skills and runbooks

A finding like the ValidationQueryCdcDelaySeconds race condition should improve the next migration, not be relearned. The agent’s migration judgment is captured in two places. DevOps Agent Skills are Markdown instruction sets that load automatically when relevant and tell the agent when to call a tool and how to read its output. The runbooks are fetched on demand: the agent calls list_runbooks to browse the catalog, then get_runbook to pull the procedure that matches what it found, as it did in scenario 5. You can fold each new finding back into a skill or runbook, so the agent improves with every migration.

To make the flywheel concrete, here is the runbook the agent fetched in the open-ended investigation. When it found the validation symptom, it called get_runbook and received this procedure, authored from earlier findings and grounded in public AWS documentation:

---
id: validation-mismatched-records
title: "Validation: mismatched records on a table"
severity: HIGH
triggers:
  - "ValidationState=Mismatched records"
  - "ValidationFailedRecords>0"
tools:
  - get_validation_failures
  - list_table_statistics
  - analyze_cdc_latency
  - correlate_cloudtrail_changes
---

# Validation: mismatched records on a table

A table shows Mismatched records, meaning source and target rows differ.
The row-level diffs are recorded in the awsdms_validation_failures_v1
control table on the target.

## Phase 1 — Assess
- Run get_validation_failures to see which tables are in Mismatched records
  and the failed-record counts.
- Run list_table_statistics to confirm the per-table validation state.

## Phase 2 — Investigate
- Query awsdms_validation_failures_v1 on the target for the failing rows/columns.
- Run analyze_cdc_latency: if validation runs during heavy CDC, transient
  diffs can appear while changes are in flight.
- Run correlate_cloudtrail_changes to check for an out-of-band write or reload.
- Check for data-type, precision, timezone, or character-set differences.

## Phase 3 — Report
- State which tables diverged and by how many records.
- Name the most likely cause (type/precision, timezone, encoding, out-of-band write).
- Recommend revalidating the table after the cause is corrected.

## Remediation (operator action)
- Correct the underlying difference, then revalidate with validate-only.

> All steps use read-only MCP tools. Remediation actions are operator tasks
> and are not performed by the tools.

The judgment for when to apply a runbook lives in a DevOps Agent Skill, a Markdown instruction set that loads automatically when relevant. The data-validation skill, for example, encodes the rules the agent follows before it draws a conclusion:

# Skill: DMS Data Validation Assessment

## Critical Rules
- Every finding MUST cite actual numbers from the metrics report, never generalize.
- Do NOT fabricate or estimate any metric. If data is missing, say "Data not available."
- Do NOT hardcode thresholds, use relative comparisons (% of total, trend direction).
- Validation metrics come from TWO sources: the DMS table-statistics API AND
  CloudWatch. Cross-reference both.

## Concepts to Evaluate — ValidationState machine
- Validated          = healthy, all rows confirmed matching
- Mismatched records = ACTION REQUIRED, source/target differ, check failure table
- Suspended records  = source churn too high, DMS cannot compare
- No primary key     = CANNOT VALIDATE, table lacks a PK
Flag if Mismatched + Suspended + Error tables exceed 5% of the total.

Every new finding folds back into a runbook or a skill, so the next migration starts from what the last one learned. The ValidationQueryCdcDelaySeconds race condition from scenario 2 becomes a trigger the agent recognizes on sight, rather than something it has to rediscover.

When to use this approach

This pattern fits issue investigation, pre-cutover readiness gates, and post-cutover stabilization reviews, where an operator hands the agent a symptom and wants a grounded root cause from read-only tools. It is not a replacement for continuous monitoring or alarms, and it does not ship logs for long-term retention. Use it alongside your existing CloudWatch alarms and dashboards, not instead of them.

Clean up

To avoid ongoing charges, delete the resources you created. Delete the CloudFormation stack with aws cloudformation delete-stack --stack-name dms-mcp-test-mcp-server. Remove the MCP server from your Agent Space and deregister it. If you provisioned a test migration with the repository scripts, run the included cleanup script.

Conclusion

Migration issues usually stem from operational problems, not data problems, and AWS DMS does not catch them. In this post, you saw how to extend AWS DevOps Agent to investigate them autonomously, using a sample MCP server that exposes read-only, migration-specific tools and runbooks. Across five real investigations, the agent chose the right tools for each question, found the affected tables, named the exact task setting behind a validation race condition, and followed a runbook to a fix. Because the tools are read-only and access is least-privilege IAM, you can give them to an autonomous agent without widening your operational scope.

To get started, deploy the MCP server from the GitHub repository, register it with your Agent Space, and turn on DMS data validation before your next cutover.

About the authors

Chitresh Saxena

Chitresh Saxena
Chitresh Saxena is a Senior AI/ML Specialist, specializing in generative AI solutions and dedicated to helping customers successfully adopt AI/ML on AWS. He excels at understanding customer needs and provides technical guidance to build, launch, and scale AI solutions that solve complex business problems.

Neel Sendas

Neel Sendas
Neel Sendas is a Principal Technical Account Manager at AWS, leading Cloud Operations for some of AWS’s largest enterprise customers across ML governance, cloud finance, and operational resilience at scale. He is also a core member of AWS’s Machine Learning Technical Field Community, helping shape the roadmap for AWS AI/ML services.

Tipu Qureshi

Tipu Qureshi
Tipu Qureshi is a Senior Principal Technologist in AWS Agentic AI, focusing on operational excellence and incident response automation. He works with AWS customers to design resilient, observable cloud applications and autonomous operational systems.