Nothing last week added a new kind of compute. What it did instead was shorten the distance between the code you already have and that code being live. AWS Lambda learned to run a plain HTTP server with no adapter in front of it, and Amazon S3 became something you can provision from inside the tool you were already using, without opening the AWS Management Console. Different services, same direction.
The counterweight arrived on Thursday. As it gets easier to put things into production, it matters more that you can say what those things are allowed to do, and this week’s most interesting read is about exactly that. Now, let’s get into this week’s AWS news…
Headlines
AWS Lambda Web Functions is a new form factor that runs a standard HTTP web application on Lambda without rewriting it into an event handler. You take a port-listening server built with Express, Hono, Fastify, or Next.js, run aws lambda-web deploy, and get back a live HTTPS URL in under a minute. No adapters, no infrastructure to set up first, no framework lock-in. Lambda now has three form factors on the same Firecracker isolation, which compose inside one application: Event Functions (the model you already know) for async work, Web Functions for the user-facing layer, and MicroVMs for stateful sandboxes running untrusted code.
Two details make this more than repackaging. Each execution environment handles up to 128 concurrent requests, so a burst of traffic shares warm environments instead of mapping one request to one environment, and billing meters the CPU time your code actually uses rather than wall-clock time. For server-side rendering, API backends, and inference endpoints streaming tokens, where most of the elapsed time is spent waiting, that changes the arithmetic. Adding --regions to the same deploy command replicates your code and configures latency-based routing behind a single global URL, and you can split traffic across revisions by weight for a canary rollout. It’s available in five Regions, running Node.js 24 on Arm64 with VPC connectivity. The AWS Compute Blog has a longer walkthrough, and Amazon CloudFront can now use a Web Function endpoint as an origin with origin access control.
The week’s other headline isn’t a launch. Trust in Depth: giving agents more autonomy without losing control makes an argument I keep running into in practice: putting a person in the loop for every agent action works at small scale and then quietly stops working, because people asked to check everything stop catching what matters. The alternative is four questions answered at four moments, which AWS calls scope, know, bound, and observe. The part worth sitting with is that an agent’s limits belong outside its own reasoning, in Amazon Bedrock AgentCore policies enforced at the gateway, because an agent cannot turn off a control it does not run. AWS has run automated reasoning in production for more than a decade, including AWS Identity and Access Management at more than 2.1 billion requests per second, and the case here is for pointing it at the rules you write for agents.
Last week’s launches
Here are some launches and updates from this past week that caught my attention:
Amazon S3 is now available on the Vercel Marketplace and v0 by Vercel – You can now provision S3 general purpose buckets, S3 Tables, and S3 Vectors from the Vercel dashboard, or by describing what you want to build in v0. It joins existing integrations for Amazon Aurora, Amazon DynamoDB, and Amazon OpenSearch Serverless, so the whole stack sits in one AWS account with one bill. The authentication detail is the good part: a short-lived OIDC token is exchanged at runtime with AWS Security Token Service for credentials scoped to that resource, so no static keys are stored anywhere.
Amazon EC2 T8i instances are now available in South America (São Paulo) – The newest burstable generation reached the São Paulo Region, which matters for the unglamorous half of a workload: login gateways, small databases, CI/CD runners, low-traffic sites. Amazon EC2 T8i delivers up to 30 percent better price performance than T3, with up to 70 percent higher compute performance and 2.4 times the Amazon EBS bandwidth. It keeps the same CPU credit system as T3, so moving over is a size change rather than a redesign, and t8i.micro and t8i.small are available under the AWS Free Tier.
Reflection Beam, an open-weight model for agentic coding, is now on AWS – You can now deploy Beam, Reflection AI’s open-weight model for agentic coding and long-horizon tasks, from Amazon SageMaker JumpStart. It’s a sparse mixture-of-experts model with 501 billion total and 23 billion active parameters, released on Hugging Face under Apache 2.0, and Reflection reports an effective context length of 1 million tokens. Start with a JumpStart real-time endpoint, or use SageMaker HyperPod Inference with SGLang or vLLM when you need Kubernetes-native control. It was also ported and optimized for AWS Trainium3, and launches in N. Virginia and Ohio.
AWS Lambda Durable Execution SDK for Go is now generally available – You can now write multi-step Go workflows that checkpoint their own progress and pause for up to a year while waiting on an external event, without building progress tracking yourself. That covers deployment rollouts with approval gates and pipelines that call a model and then stop for human review. It behaves like ordinary Go tooling: go get to add it, deploy a single static binary on the provided.al2023 runtime, and test a full workflow locally with the durabletest package, no network access or credentials required.
Amazon CloudWatch announces log routing and chained log parsers – You can now define routes in an Amazon CloudWatch pipeline that send each event to a different log group based on conditions evaluated against processed fields, so severity, source, or event type can each land somewhere with its own retention and access controls. Chained parsers handle the other half: parsers run in sequence, with later ones consuming fields that earlier ones produced, so you can parse a JSON envelope first and then run a Grok parser over a field it extracted.
For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.
Other AWS news
Here are some additional posts and resources that you might find interesting:
AWS Security Agent exploited 33 of 37 new real-world CVEs – The team pointed their autonomous penetration-testing agent, now part of AWS Continuum, at 37 recently disclosed CVEs packaged as live containerized targets, starting from nothing but an endpoint URL and a one-line hint. It captured 35 flags, and after auditing every win against the challenge’s canonical proof of concept, 33 of 37 (89.2 percent) held up as genuine end-to-end exploitation. The two misses are what make the post worth your time: in both, the agent found and exploited the bug, then took the target down before it could read the flag. A capable autonomous attacker is also a destructive one.
Check your calendar and sign up for upcoming AWS events:
AWS re:Invent – re:Invent returns to Las Vegas from November 30 to December 4, with more than 2,200 sessions. The event catalog is live and reserved seating is open, so register and claim your spots before they fill up.
AWS Community Days – Community-led conferences planned and delivered by community leaders. Here in Brazil, the 2026 season closes with AWS Community Day Brasil Norte in Belém on October 31, the final stop in a year that covered every region of the country for the first time.
Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.
That’s all for this week. Check back next Monday for another Weekly Roundup!
— Daniel Abib
This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!
Developers building web and AI applications on Vercel can now add Amazon S3 storage resources directly from the Vercel Marketplace and v0, combining the durability, scalability, and performance of S3 with the convenience of Vercel’s provisioning workflow. Starting today, Amazon Simple Storage Service (Amazon S3), including Amazon S3 Tables and Amazon S3 Vectors, is natively available on the Vercel Marketplace and v0 by Vercel, covering three distinct workloads: everyday storage, analytics, and AI.
Why this matters
Millions of developers build, deploy, and scale web and AI applications on Vercel. Storage is essential to performant applications, but provisioning it previously required navigating to the AWS Management Console to configure policies and credentials. Now, S3 storage integrates directly into the Vercel developer experience, with resource provisioning and connection configuration handled automatically. Together with existing integrations for Amazon Aurora, Amazon DynamoDB, and Amazon OpenSearch Serverless, you get a complete application stack inside Vercel (from your database layer through to object, vector, and open table storage), all on one AWS account with one bill.
Storage for every application
S3 general purpose buckets: Store virtually unlimited objects, including files, images, videos, and documents, for data lakes, cloud-native applications, and mobile apps. S3 delivers the most durable storage in the cloud, designed to provide 99.999999999% (11 nines) data durability and 99.99% availability by default.
S3 Tables: Fully managed Apache Iceberg storage optimized for analytics with automated table maintenance and compaction. S3 Tables are directly queryable from any Iceberg REST catalog API-compatible engine, including DuckDB.
S3 Vectors: Purpose-built vector storage and query for AI applications such as retrieval-augmented generation (RAG) and semantic search, reducing the cost of uploading, storing, and querying vectors by up to 90%.
Figure 1 – The AWS page on the Vercel Marketplace.
Getting started
From the Vercel dashboard, go to the Storage tab and choose Create. Select AWS, pick your S3 bucket type, choose a Region, and create the bucket. Provisioning takes a few seconds.
Figure 2 – Creating an Amazon S3 general purpose bucket in Vercel by choosing a Region and a name.
Connect the bucket to a project, then pull the environment variables with the Vercel CLI:
vercel link
vercel env pull .env.local
The integration injects AWS_REGION, AWS_ROLE_ARN, AWS_RESOURCE_ARN, and a bucket name such as S3_BUCKET_NAME, S3_TABLE_BUCKET_NAME, or S3_VECTOR_BUCKET_NAME. At runtime, the awsCredentialsProvider from @vercel/functions/oidc exchanges a short-lived OIDC token with AWS Security Token Service (AWS STS) for temporary credentials scoped to your resource. No static keys are stored anywhere.
Object storage with S3
This Next.js route uploads a file to a general purpose bucket:
Use ListObjectsV2Command and GetObjectCommand with the same client to list and read objects.
Analytics with S3 Tables
S3 Tables offer S3 storage purpose-built for analytics and are designed to automatically optimize query performance as data volumes grow. They provide fully managed Apache Iceberg tables with automated table maintenance and compaction, and they work with any Apache Iceberg REST Catalog-compliant engine. You can use DuckDB inside a Vercel Function for lightweight queries. This example attaches the table bucket by its ARN and runs an aggregate query:
import { DuckDBInstance } from "@duckdb/node-api";
import { awsCredentialsProvider } from "@vercel/functions/oidc";
const creds = await awsCredentialsProvider({
roleArn: process.env.AWS_ROLE_ARN,
})();
const db = await DuckDBInstance.create();
const conn = await db.connect();
await conn.run("INSTALL aws; LOAD aws; INSTALL httpfs; LOAD httpfs; INSTALL iceberg; LOAD iceberg;");
await conn.run(`
CREATE SECRET s3cred (
TYPE s3,
KEY_ID '${creds.accessKeyId}',
SECRET '${creds.secretAccessKey}',
SESSION_TOKEN '${creds.sessionToken}',
REGION '${process.env.AWS_REGION}'
);
`);
await conn.run(`
ATTACH '${process.env.AWS_RESOURCE_ARN}' AS s3tables (
TYPE iceberg,
ENDPOINT_TYPE s3_tables
);
`);
const rows = await (await conn.run(`
SELECT town_city, AVG(median_price) AS avg_price
FROM s3tables."${process.env.S3_TABLES_NAMESPACE}".town_monthly
GROUP BY town_city
ORDER BY avg_price DESC
LIMIT 10
`)).getRowObjects();
If you run DuckDB inside a Next.js route, add @duckdb/node-api to serverExternalPackages in next.config.mjs so the native bindings are not bundled.
Vector search with S3 Vectors
S3 Vectors deliver purpose-built, cost-optimized vector storage for AI agents, AI inference, and semantic search of your content stored in Amazon S3. S3 Vectors provides native support for storing and querying vectors without a separate vector database, but it does not generate them. Create the embeddings with your model of choice, then query the index for the closest matches:
From v0, describe what you want to build and it provisions the right resource. Ask for “a photo gallery where users upload images using AWS” to get a general purpose bucket, “a support bot that answers questions from my PDFs” to get S3 Vectors, or “an analytics dashboard for my clickstream data” to get S3 Tables. v0 then configures the connection and generates working code.
Figure 3 – v0 generating a PDF support bot from a natural language prompt, with Amazon S3 Vectors connected and the application in the Preview panel.
Combining S3 with AWS databases
Every AWS service on Vercel shares the same account and authentication model, so you can combine them with no extra setup. For example, store file metadata in Aurora with the files in S3, or store document text in DynamoDB with embeddings in S3 Vectors for RAG retrieval.
Using Vercel with your AWS account
You can create a new AWS account through the integration, which supports S3 features for building and testing. You can also link an existing AWS account to use your full set of S3 and AWS capabilities and keep your current billing. If you link an existing AWS account, Vercel works only with new buckets that you create through the platform. Because every S3 resource lives in your own AWS account, you keep AWS-native tooling and a single place to manage storage.
Pricing and availability
There is no charge for the integration, and you pay standard Amazon S3 pricing for the resources you use. New AWS accounts created through Vercel receive $100 in credits valid for six months. Amazon S3, including S3 Tables and S3 Vectors, is available from v0 and the Vercel Marketplace in all commercial AWS Regions supported by Vercel. To get started, visit v0 or the AWS page on the Vercel Marketplace.
Join the conversation
If you want to call APIs, search documentation, find regional availability, and check troubleshooting about this new feature, try using the AWS MCP Server and plugins with your preferred AI tool. Share your feedback on AWS re:Post for Amazon S3.
Some numbers are hard to picture until you sit with them for a moment. Last week we published the Prime Day 2026 metrics, and the one that stuck with me is 2.3 quadrillion: the daily requests Amazon ElastiCache peaked at during the event. Prime Day ran back in June, so this post always lands as a retrospective, and that gap is part of what makes it worth reading. By the time the numbers reach you, the only evidence that any of it happened is that nothing went wrong.
The rest of the week pulled in a different direction: smaller, cheaper, and more constrained. Claude Haiku 5.5 arrived on Amazon Bedrock at roughly 75 percent less than Haiku 4.5, Amazon Nova 2.5 Sonic made voice agents quicker on their feet, and Strands Box showed up to put real boundaries around what an agent is allowed to do. Now, let’s get into this week’s AWS news…
Headlines
All the numbers: Amazon Prime Day 2026 powered by AWS is the latest entry in a tradition that goes back to 2016, and the scale is still difficult to hold in your head. Over the four days of Prime Day, Amazon DynamoDB processed more than 59 trillion requests while peaking at 192 million requests per second, and kept single-digit millisecond responses throughout. AWS Lambda handled over 2.3 trillion invocations per day. Amazon EBS peaked above 24.8 trillion I/O operations, moving more than an exabyte of data daily. Amazon ECS launched an average of 158.3 million tasks per day on AWS Fargate, a 47.7 percent jump over last year, and AWS Graviton powered up to 49 percent of the Amazon EC2 compute that Amazon.com used.
The number I keep coming back to, though, isn’t a throughput record. The team ran more than 44,000 AWS Fault Injection Service experiments ahead of the event, more than six times what they ran in 2025. Those are deliberate failures injected on purpose, months in advance, so that the trillions of requests above turn into an ordinary shopping afternoon. If you’re preparing for your own peak moment, AWS Countdown Premium is built for exactly that kind of planning.
Last week’s launches
Here are some launches and updates from this past week that caught my attention:
Claude Haiku 5.5 is now available on AWS – You can now use the fastest model in the Claude 5.5 family on Amazon Bedrock and Claude Platform on AWS, at around 75 percent less than Claude Haiku 4.5 for most tasks. It’s the first Haiku with effort controls, so you can tune cost against intelligence per task instead of fixing one setting for a whole workload. It pairs naturally with Claude Opus 5.5: Opus plans and makes the judgment calls, and because Haiku is fast and cheap, you can run many Haiku subagents in parallel for routing, classification, and small code changes across many files.
Amazon Nova 2.5 Sonic for voice agents – You can now build real-time voice agents on a speech-to-speech Amazon Nova model with better reasoning, instruction following, and tool-calling accuracy, plus lower latency. That combination matters for multi-step work: a support agent can look up an order, check return eligibility, and start the return inside one conversation. It keeps expressive voices in seven languages, controllable turn-taking, and a 256K context window, and pairs with Strands Bidi Agents, now generally available, to get a production voice agent going in a few lines of code.
Amazon GuardDuty RDS Protection now detects data exfiltration and destruction – Amazon GuardDuty now looks past login anomalies to the queries themselves on Amazon Aurora PostgreSQL and Amazon RDS for PostgreSQL. RDS Protection for data activity builds behavioral baselines and uses ML-based anomaly detection plus threat intelligence to flag unusual data access, with no agents, extra infrastructure, or database configuration changes. This closes a real gap: once an attacker has valid credentials from phishing or a misconfiguration, their queries can look indistinguishable from your application’s.
AWS Lambda supports OAuth for self-managed Apache Kafka event sources – You can now authenticate Lambda Kafka consumers through your enterprise identity provider, such as Amazon Cognito or Okta. Event source mappings previously supported only SASL/PLAIN, SASL/SCRAM, and mTLS, which left teams in regulated industries unable to use Kafka ESM at all. Now you get the automatic scaling, batching, error handling, and event filtering of Kafka ESM with the same identity governance you apply everywhere else. It works with self-hosted clusters and managed offerings including Confluent Cloud, Aiven, and Redpanda.
For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.
Other AWS news
Here are some additional posts and resources that you might find interesting:
Introducing Strands Box: AI agent sandboxes powered by Dogwood – As more of us let agents run shell commands and call APIs without approving every step, isolation alone stops being enough. Strands Box is an open source sandbox (Apache 2.0, developer preview) that combines OS-level containment with fine-grained policies written in Dogwood, so permissions can depend on what the agent has already done. You can say “let the agent post to the incident channel, but no more than three times every 10 minutes” and have that enforced from outside the agent’s process. It also keeps real credentials out of the agent’s environment by attaching them at an egress gateway.
AWS Continuum sets a new standard in autonomous code security – Related to that theme, AWS Continuum for Penetration Testing now integrates into your CI/CD pipeline in public preview, turning penetration testing into a deploy-time event instead of something that happens off to the side. Developers get findings with severity, affected endpoints, and remediation guidance right in the pipeline output, and the pipeline re-tests to verify fixes.
Check your calendar and sign up for upcoming AWS events:
AWS re:Invent – re:Invent returns to Las Vegas from November 30 to December 4, and reserved seating is now open. The session catalog is live, so register and claim your spots in chalk talks, workshops, and builders’ sessions before they fill up.
AWS Community Days – Community-led conferences planned and delivered by community leaders, and this is a busy stretch. Upcoming stops include San Francisco (October 13), Miami (October 14), Brisbane (October 16), Cape Town (October 16), Malta (October 16), and Mumbai (October 17). Here in Brazil, the community calendar has more editions coming before the end of the year.
Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.
That’s all for this week. Check back next Monday for another Weekly Roundup!
– Daniel Abib
This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!
You want to set a cryptographic algorithm policy and stop non-compliant choices from being made in your Amazon Web Services (AWS) environment. This post shows you how. As you adopt post-quantum cryptography, one of the first decisions you face is which algorithms and parameter sets fit your business needs, balancing security against signature size, key size, and performance. Your cloud environment’s job is to let you express that decision and enforce it consistently, so an approved choice stays fully usable while an excluded one cannot be introduced. This post works through one concrete example on AWS: taking an organization’s cryptographic algorithm policy and turning it into preventive and detective controls. The same patterns apply to whatever your approved set is today and whatever it becomes as standards evolve.
This post is Part 2 of a two-part series on quantum-resistant signing on AWS. Part 1, Build a software supply chain with ML-DSA, is for builders and constructs an end-to-end ML-DSA signing pipeline for binaries, container images, and software bills of materials. This part is for the security, governance, and compliance teams who decide which algorithms that pipeline, and everything else in the organization, is permitted to use, and who need that decision enforced.
AWS Key Management Service (AWS KMS) supports the ML-DSA digital signature scheme (FIPS 204) through three parameter sets, ML-DSA-44, ML-DSA-65, and ML-DSA-87, targeting NIST security categories 2, 3, and 5. All three are strong, standardized choices. Each one trades signature size, key size, and speed against security margin, the amount of cryptographic strength a parameter set holds in reserve above its target security category. ML-DSA-44 is the most compact and fastest. ML-DSA-65 and ML-DSA-87 carry a larger security margin, which is what you want for signatures that must stay trustworthy over a long lifetime.
Throughout this post, the examples use the same example policy: an organization that has standardized on ML-DSA-65 and ML-DSA-87 and wants ML-DSA-44 kept out of its approved set. That is a standardization decision, not a judgment that ML-DSA-44 is insecure (see the following note). Wherever you see ML-DSA-44 in the controls that follow, read it as the variant your policy doesn’t include; the mechanics are identical for any allow/deny split you choose. The approved set here is also the one the Part 1 architecture deploys: ML-DSA-87 for the root certificate authority (CA) and ML-DSA-65 for the subordinate CA and the code-signing key. Part 1 makes the choice; this post makes it stick.
Note: ML-DSA-44 is a valid choice. ML-DSA-44 is a fully NIST-standardized parameter set (FIPS 204) at NIST security category 2, which targets roughly 128-bit-class classical security (category 2 is benchmarked against the collision resistance of SHA-256). It produces the smallest signatures and public keys of the three variants (a 1312-byte public key and 2420-byte signature) and is the fastest to sign and verify, which makes it a strong fit for constrained devices, high-volume signing, and bandwidth- or storage-sensitive workloads. You would choose ML-DSA-65 or ML-DSA-87 instead to gain extra security margin for long-lived signatures or to align with an internal standard, not because ML-DSA-44 is weak.
What success looks like
The approved variants (ML-DSA-65 and ML-DSA-87 in the example) remain fully usable throughout. What you want to rule out is the introduction of a variant outside the policy (here, ML-DSA-44). Concretely, within the AWS service surfaces in scope, principals shouldn’t be able to:
Create ML-DSA-44 key material
Use existing ML-DSA-44 keys for signing or verification
Stand up private certificate authorities (CAs) that would introduce ML-DSA-44.
And any drift from those settings should be detected.
Certificate material brought in through acm:ImportCertificate or iam:UploadServerCertificate is also not algorithm-aware (see the Private CA section). Other TLS-terminating or signing services (for example Amazon CloudFront and Amazon API Gateway) and AWS IAM Roles Anywhere aren’t covered in this post and should be assessed separately. (Part 1 of this series uses IAM Roles Anywhere with ML-DSA certificates; controls for that service are out of scope here.)
These controls are intended to be applied as defense in depth. You can apply all of them or only a subset, depending on how prescriptive you want to be:
A blanket SCP at the organization root prevents member accounts from creating or using ML-DSA-44 keys. (SCPs don’t apply to the management account, so workloads there aren’t restricted.)
An SCP at a production organizational unit (OU) restricts keys only in production accounts.
If you need flexibility inside an account, use IAM permission boundaries instead of account-wide settings enforced through an SCP.
Choose the combination that matches your operating model. The policy examples here are written to apply across all AWS Regions. If you need to scope a control to specific Regions, add an aws:RequestedRegion condition.
The policies in this post are illustrative starting points. Authoring, validating, and maintaining the specific SCPs, IAM policies, and permission boundaries for your environment is your responsibility. Test all policies in a non-production environment before deployment and adjust example values (such as policy names and key Amazon Resource Names (ARNs)) to match your organization’s approved standards.
A note on cryptographic agility
Your algorithm policy isn’t fixed forever, and these controls shouldn’t be either. The point of expressing the policy as AWS controls is that you can revise it as your needs and the relevant standards change, tightening, loosening, or swapping the approved set without re-architecting. Treat the examples here as a template you adjust to whatever your current policy is.
A deny-based control that names specific algorithms is, in effect, an allow-list of everything it does not name. The risk is default-allow: any parameter set you have not explicitly denied is permitted, so a newly introduced variant, including one you have not yet evaluated, becomes usable until you notice it and add it to the deny list. Deprecation of an already-approved variant is easy to handle in this model, because you simply add the deprecated variant to the deny list. What the model does not catch on its own is the arrival of something new. Review these controls periodically and pair the deny-list with detection so a variant outside your policy surfaces quickly rather than waiting to be discovered.
The alternative is to write the condition against the approved set rather than the excluded one. A Deny that uses StringNotEquals with a list of approved key specs fires when the request’s key spec matches none of them, so a new or not-yet-evaluated variant is blocked by default until you add it. That closes the default-allow gap, at the cost of a policy update before anyone can use a newly approved variant. The family-level SCP in Part 1 of this series takes this shape: it denies creation of any signing key whose key spec isn’t ML_DSA_*. The examples in this post take the other shape and name the excluded variant, so that approved variants stay usable without a policy change. Both are valid. Choose based on whether you would rather block something new first and approve it later, or notice it first and decide then.
The StringLike clause scopes the statement to ML-DSA key specs so it doesn’t affect other key types; the StringNotEquals clause denies any ML-DSA variant outside the approved list.
Just as algorithms and key sizes will be introduced over time, or move from approved to unapproved, so will AWS service operations. The set of operations through which an ML-DSA-44 key can be used today (AWS KMS Sign, Verify, and GetPublicKey) reflects current service capabilities and might expand as AWS releases new features. If your policy is to block use in addition to creation, you should periodically review the list of operations you restrict, because services evolve over time.
Background: Key spec compared to signing algorithm
AWS KMS support for ML-DSA introduces three key specs—ML_DSA_44, ML_DSA_65, and ML_DSA_87—that all use a single post-quantum signing algorithm: ML_DSA_SHAKE_256. This distinction drives which controls are effective.
kms:KeySpec: Reflects the KeySpec property of the key. It’s evaluated both on CreateKey (the requested key spec) and on AWS KMS key resource operations that act on a specific key, including Sign, Verify, and GetPublicKey. This makes kms:KeySpec a reliable lever for both preventing creation of ML-DSA-44 keys and restricting use of existing ones, because it can single out ML_DSA_44 from ML_DSA_65 and ML_DSA_87.
kms:SigningAlgorithm: Reflects the signing algorithm supplied on Sign and Verify, which is ML_DSA_SHAKE_256 for all three variants. It therefore can’t distinguish ML-DSA-44 from the approved variants and shouldn’t be used for this purpose.
The practical consequence: for keys held in AWS KMS, preventing ML-DSA-44 key material from being created is the most robust single control. If no ML-DSA-44 KMS keys exist, none can be used for signing through AWS KMS and none can become the subject key of a certificate. CA keys and TLS server keys are separate material: AWS Private CA generates and holds a CA’s key pair in its own HSMs, and ACM-issued, imported, or uploaded TLS certificates carry their own keys, so those surfaces are governed by the AWS Private CA and certificate controls later in this post. Where ML-DSA-44 KMS keys already exist, their use can additionally be restricted with kms:KeySpec.
Organization and account-level blocking using SCPs
SCPs can be applied at the organization root or to a specific OU to restrict ML-DSA-44. Placement determines the effective scope: it applies organization-wide when attached at the root; attached to an OU, it applies to the accounts within that OU. An SCP is the broadest-scope preventive control available.
Note that SCPs apply to member accounts and OUs but not to the organization’s management account, so avoid running ML-DSA-44-capable workloads in the management account, where an SCP will not restrict them.
The following example SCP denies both the creation of ML-DSA-44 keys (kms:CreateKey) and their use (kms:Sign, kms:Verify, kms:GetPublicKey), matching on the ML_DSA_44 key spec in each case. The creation deny stops new keys from being created; the usage deny covers keys that already exist, whether created before the SCP was attached or living in an account that later joins the OU, because kms:KeySpec is also evaluated at request time on those usage operations.SCPs are guardrails: a Deny in an SCP can’t be overridden by any IAM policy within member accounts.
Attach the following SCP at the organization root (or relevant OU):
Scope the usage deny to the three key-spec-aware actions. Limit the use of Deny to kms:Sign, kms:Verify, and kms:GetPublicKey (testing confirmed kms:KeySpec is evaluated on all three). Avoid a blanket kms:*: a wildcard that would also catch management operations such as DescribeKey, DisableKey, and ScheduleKeyDeletion, which you need to inspect or clean up those same keys, effectively locking you out of managing the keys you’re trying to remove. If you use multi-Region keys, also add kms:ReplicateKey to the creation deny: replicating an ML-DSA-44 primary key creates a new ML-DSA-44 key in another Region, and kms:KeySpec is evaluated on ReplicateKey as well.
Combining with a family-level guardrail. This SCP governs which ML-DSA parameter set is allowed; it says nothing about RSA or ECDSA. Part 1 of this series includes a complementary SCP that denies creation of any signing key whose key spec isn’t ML-DSA. Attached together, they require every new signing key to be ML-DSA and to be an approved variant.
Operational considerations. A blanket Deny at the organization root is the strongest control but has trade-offs: it removes any ability to create ML-DSA-44 keys anywhere in scope, with no exception path, which can block legitimate testing and might cause infrastructure as code (IaC) or pipeline runs that attempt ML-DSA-44 creation to fail. To preserve flexibility, consider scoping the Deny to a production OU and using IAM permission boundaries (see the following section) in build and non-production accounts.
What the caller sees. When an SCP denies a request, the caller receives an AccessDenied error indicating an explicit deny in a service control policy. For same-account and same-organization requests, the error includes the type of the denying policy and, as the enhanced error messages roll out across services, its ARN. The error doesn’t expose the SCP policy contents, and the user being blocked often can’t see the SCPs applied to their environment. Document the SCP ARN and its purpose internally so support teams can map an AccessDenied quickly.
Prevent ML-DSA-44 key creation in AWS KMS (principal-level)
The following IAM policy blocks ML-DSA-44 at the principal level. It is the identity-based counterpart to the SCP shown earlier, useful when you want per-account or per-role control rather than an organization-wide guardrail. Attach it to the IAM role or user you want to constrain.
The condition value is ML_DSA_44 (underscores, not hyphens).
This policy blocks CreateKey only. To also block use of existing ML-DSA-44 keys at the principal level, add the Sign, Verify, and GetPublicKey deny shown in the Restricting use of existing ML-DSA-44 keys section.
For comprehensive, non-overridable coverage, deploy as an SCP rather than relying solely on principal-level policies.
IAM permission boundaries
IAM permission boundaries set the maximum permissions an identity-based policy can grant to an IAM role or user and can be used as an alternative or complement to using SCPs. The effective permissions are the intersection of the identity-based policy and the boundary, and a Deny in a boundary can’t be overridden by an attached policy. Boundaries are useful where you want per-account or per-role flexibility (for example, in build or non-production accounts) rather than a blanket, multi-account deny.
Operational consideration. A permission boundary only takes effect on principals that have it attached. To rely on boundaries as a control, you need a mechanism to enforce the boundary on all IAM principals programmatically. One example enforcement mechanism would be an SCP that denies iam:CreateRole, iam:CreateUser, iam:PutRolePermissionsBoundary, and iam:PutUserPermissionsBoundary unless the approved boundary is set (using the iam:PermissionsBoundary condition key), and denies iam:DeleteRolePermissionsBoundary and iam:DeleteUserPermissionsBoundary outright so the boundary can’t be removed after the fact. You would also want to probe for non-conforming principals periodically and respond, perhaps by automatically attaching the PermissionsBoundary. IAM users shouldn’t be used, but the enforcement should account for the possibility that they exist.
Because a boundary only sets a ceiling and does not itself grant anything, it opens with Allow on all actions, then denies ML-DSA-44 key creation specifically. The following permission boundary denies creation of ML-DSA-44 keys:
To also block use of existing ML-DSA-44 keys for the principals carrying this boundary, add the Sign, Verify, and GetPublicKey deny statement shown in the following section (which also uses kms:KeySpec).
Restricting use of existing ML-DSA-44 keys
As described in the Background section, kms:KeySpec is also evaluated on Sign, Verify, and GetPublicKey, reflecting the KeySpec of the key being used. You can therefore deny those operations when kms:KeySpec equals ML_DSA_44 while continuing to permit ML_DSA_65 and ML_DSA_87. The IAM policy below, attached to an IAM principal, does exactly that.
This deny can also live in an AWS KMS key policy. The same kms:KeySpec deny can be placed directly in the key policy of an AWS KMS key. A key policy is a resource policy, so the statement must include a Principal element. Identity policies and SCPs do not name principals to which they apply. A deny in the key policy is a resource-side backstop that protects a specific key, while an SCP gives non-overridable organization-wide coverage. To turn the previous example into an AWS KMS key policy, add a Principal element to the statement (for example "Principal": {"AWS": "*"}, or a more specific principal), and keep the rest of it the same.
Additional approaches
Depending on your goals you might also want to remove the key material itself rather than only deny its use. Remember that it is difficult to know what data has been signed by a specific key, unless you track that somehow. Deleting a key that was used will make signature verification impossible. Take great care to make sure keys are not used before deleting them.
Disable or schedule deletion of the keys. Disabling stops key use immediately and is reversible. Key deletion is definitive but irreversible after the waiting period, so confirm before scheduling.
Deny Sign, Verify, and GetPublicKey by specific key ARN. You can target named keys rather than the whole key spec. To do this, you would list the exact key ARNs in the Resource element of the IAM policy (no condition key required). This is effective for targeting a single key specifically, but it is difficult to maintain lists of key ARNs. Use this strategy sparingly. If you’re considering using an IAM policy naming a key ARN, remember that a Deny in the key policy of the target key accomplishes the same goal and is attached to the key itself.
Finding existing keys (only needed if you disable or delete). A Deny in an IAM policy or SCP, based on kms:KeySpec, doesn’t require you to know which keys exist – it applies by key spec at request time. If you want to disable or delete existing ML-DSA-44 keys, you need to locate them. AWS KMS has no server-side filter for key spec, so you must enumerate keys with ListKeys and call DescribeKey and inspect each KeySpec. This client-side filtering pattern is common across AWS APIs and is well supported by pagination in the AWS SDKs. You can also use AWS Config to record and query KMS key resources.
Approved ML-DSA variants and control of AWS Private CA
AWS Private CA sets a certificate authority’s (CA’s) signing algorithm at creation time, in the CertificateAuthorityConfiguration parameter of CreateCertificateAuthority (valid values include ML_DSA_44, ML_DSA_65, and ML_DSA_87). There is currently no condition key for the algorithm, so an SCP cannot enforce signing algorithm. The approach in this section controls who can create CAs and pins the algorithm in that trusted path, backed by AWS CloudTrail-based detection.
Recommended approach
Restrict who can create CAs. Deny acm-pca:CreateCertificateAuthority broadly via SCP, and allow it only for a small set of trusted roles or an approved provisioning pipeline. This removes the ability for arbitrary principals to stand up a CA with an unapproved algorithm.
Pin the algorithm in the trusted path. Have that approved pipeline or infrastructure-as-code template hard-code SigningAlgorithm to ML_DSA_65 or ML_DSA_87 in CertificateAuthorityConfiguration. Because AWS Private CA exposes no algorithm condition key, this provisioning path is the enforcement point. Note that the CA’s algorithm pins the signature on the certificates it issues; the subject key in each certificate comes from the CSR, which AWS Private CA copies without validating its algorithm, so pin the CSR key algorithm in the same issuance pipeline.
Control the trust chain. On the PKI side, only add approved CAs to your trust stores and issuance hierarchies, and do not trust or subordinate CAs that could issue ML-DSA-44. This prevents an unapproved CA from being introduced into a trusted path.
Detect drift. Use CloudTrail to capture CreateCertificateAuthority calls and AWS Config to flag any CA whose configuration uses ML_DSA_44, so an unapproved CA is caught quickly.
Part 1 of this series shows what this looks like in practice: it creates an ML-DSA-87 root CA and an ML-DSA-65 subordinate CA with CreateCertificateAuthority, activates them, and distributes the resulting chain to verifiers. In the model described here, those are the calls that only your approved provisioning pipeline is allowed to make.
Setting the ML-DSA value in CertificateAuthorityConfiguration. The same variant goes in both the KeyAlgorithm and SigningAlgorithm fields (for example, KeyAlgorithm=ML_DSA_65 with SigningAlgorithm=ML_DSA_65). Note a difference in signing algorithms between AWS KMS and AWS Private CA. With AWS Private CA, the SigningAlgorithm takes the variant value directly (ML_DSA_44, ML_DSA_65, and ML_DSA_87), and doesn’t accept ML_DSA_SHAKE_256. In contrast, AWS KMS uses ML_DSA_SHAKE_256 as its signing algorithm. Be aware that some older AWS Command Line Interface (AWS CLI) builds omitted descriptions of the ML-DSA variants from the built-in CLI help, so don’t rely on the help in the AWS CLI to determine valid values. For more information, see CertificateAuthorityConfiguration API Reference.
Known limitation. Some surfaces have no algorithm condition key, so you enforce them through provisioning and detection rather than policy. acm-pca:CreateCertificateAuthority has no condition key for the algorithm, and acm:ImportCertificate and iam:UploadServerCertificate are not algorithm-aware, so a disallowed signature setting on a created, imported, or uploaded certificate cannot be blocked by an IAM policy or SCP condition. Govern these through your provisioning process, restrict who can call them, and deny the import and upload actions entirely where they are not required. The controls above remain the available mitigation.
In particular, iam:UploadServerCertificate doesn’t support any method to prevent use of particular algorithms, and AWS doesn’t recommend using this API to generate and upload certificates. This API predates ACM and exists mainly to cover AWS Regions or partitions where ACM is not yet available. While a new Region or partition is being built out, workloads can still need a TLS server certificate before ACM support has shipped there, and uploading a certificate to IAM gives you a place to store and reference it in the meantime. Once ACM is available in that Region, you should use ACM instead. Outside that narrow case, deny iam:UploadServerCertificate unless it is genuinely required.
ACM-issued leaf certificates. A CA’s own key and signing algorithm are separate from the leaf certificates that ACM issues from it. If you use ACM to request leaf certificates, you can constrain the leaf’s key algorithm with the acm:KeyAlgorithm condition key on acm:RequestCertificate. This lets you allow-list approved algorithms for leaf certificates using SCPs or IAM policies, similar to the AWS KMS examples earlier in this post. Be aware that, at the time of writing, ACM doesn’t offer ML-DSA leaf certificates. So a policy like this is a forward-looking defense-in-depth control rather than an immediate ML-DSA-44 risk. The IAM example below is illustrative (using example approved algorithms). Since this is an evolving part of the service, be sure to validate this in your environment before relying on it.
This example uses StringNotEqualsIfExists so it denies only requests that specify a disallowed algorithm. With plain StringNotEquals, a request that omits acm:KeyAlgorithm would also be denied, because the missing key makes the condition evaluate true. Choose the operator that matches whether you want to block requests that leave the algorithm unspecified.
Prevent ML-DSA-44 usage in TLS for NLB and ALB
Elastic Load Balancing, for both network load balancers (NLB) and application load balancers (ALB), terminates TLS on behalf of your applications. The certificates used by these listeners are requested through or imported into ACM, or uploaded to IAM. A listener’s configured security policy and its certificate determine what’s offered during negotiation. At the time of writing, ACM requests and imports accept RSA and ECDSA keys only, and NLB and ALB listeners accept RSA and ECDSA certificates only, so no ML-DSA certificate can be bound to a listener today. As with the ACM leaf-certificate policy earlier, the controls in this section are forward-looking defense in depth rather than a mitigation of an immediate ML-DSA-44 risk.
Certificate controls (the effective mechanism for ML-DSA-44)
Because the listener API calls reference a certificate only by ARN, you can’t write an SCP on CreateListener, ModifyListener, or AddListenerCertificates that inspects the certificate’s algorithm. IAM conditions can’t reference attributes of a separately-referenced resource. Keeping ML-DSA-44 out of TLS therefore relies on there being no ML-DSA-44 certificate material to associate:
Configure your private CA to issue only ML_DSA_65 or ML_DSA_87 certificates, and control who can create CAs.
Prevent ML-DSA-44 key creation in AWS KMS, so that no KMS-held ML-DSA-44 key can become the subject key of a certificate. TLS server keys for ACM-issued, imported, or uploaded certificates are not KMS keys; those are governed by items 1 and 3.
Control imported and uploaded certificates through your provisioning process or deny the import and upload actions where not required, as described in the AWS Private CA section.
Detective controls
CloudTrail records the certificate and listener API calls in this section: certificate creation and import (acm-pca:CreateCertificateAuthority, acm:ImportCertificate, iam:UploadServerCertificate) and listener configuration (CreateListener, ModifyListener, AddListenerCertificates). Whether the algorithm shows up in the event depends on what the call itself carries. Creation calls that take the algorithm as an input record it: kms:CreateKey logs the KeySpec, and acm-pca:CreateCertificateAuthority logs the signing algorithm. That makes CloudTrail effective for catching the creation of ML-DSA-44 key material or a CA. A listener binding references its certificate only by ARN, so the certificate’s algorithm does not appear in that event. To evaluate the algorithm of a bound or imported certificate, use AWS Config resource evaluation as the detective backstop.
Summary for TLS
The primary preventive control is the certificate material itself: with no ML-DSA-44 CA, no ML-DSA-44 KMS key, and certificate import and upload controlled, no ML-DSA-44 certificate can be issued or bound. Where you also operate an allow-list of permitted TLS security policies, the elasticloadbalancing:SecurityPolicy condition key (on CreateListener and ModifyListener) can additionally constrain which TLS policies listeners can use.
Summary of controls
The following table maps the example policy (allow ML-DSA-65 or ML-DSA-87, exclude ML-DSA-44) to the available controls. Modify the controls to limit use to the approved set of algorithms your policies require.
Area
Control
Status
Block ML-DSA-44 key creation (Organization-wide or for all accounts in an organization OU)
SCP on kms:CreateKey using kms:KeySpec
Fully supported via condition key
Block ML-DSA-44 key creation (per account or per role)
IAM policy or permission boundary using kms:KeySpec
Supported; requires programmatic enforcement of boundaries
Block use of existing ML-DSA-44 keys
Deny Sign, Verify, and GetPublicKey using kms:KeySpec (key policy, IAM, or SCP); optionally disable and delete keys
Supported using kms:KeySpec (not kms:SigningAlgorithm)
Only allow approved variants in Private CA
Restrict who can create CAs; pin ML_DSA_65 or ML_DSA_87 in provisioning path
No algorithm condition key; enforce using process and detection
Limit variants in imported or uploaded TLS certificates
Process controls, or deny import or upload APIs (prefer ACM over iam:UploadServerCertificate)
APIs aren’t algorithm-aware
Block use of ML-DSA-44 in TLS with network load balancers (NLB) and application load balancers (ALB)
Restrict TLS policy using elasticloadbalancing:SecurityPolicy; control cert material
TLS-policy restriction supported; no per-cert algorithm condition
Detect use of unapproved variants
CloudTrail (API-level) and AWS Config custom rules
Supported using custom AWS Config rules; not managed rules
Conclusion
In this post, you learned how to use ML-DSA as an example of turning a cryptographic algorithm policy into enforceable AWS controls. The key lever is the kms:KeySpec condition key: it keeps your approved parameter sets (ML-DSA-65 and ML-DSA-87) fully usable while preventing the creation and use of a parameter set you’ve excluded (ML-DSA-44) on the CreateKey, in addition to the Sign, Verify, and GetPublicKey operations, where kms:SigningAlgorithm can’t tell the variants apart. Applied as defense in depth across SCPs, IAM policies, permission boundaries, and KMS key policies, and extended to the surfaces that have no algorithm condition key (AWS Private CA, certificate import, and Elastic Load Balancing TLS) through provisioning controls plus AWS CloudTrail and AWS Config drift detection, these controls give you a consistent way to keep an unapproved variant out of your accounts.
The pattern isn’t specific to ML-DSA-44. Because your algorithm policy will change as standards and your own requirements evolve, treat these controls as a reusable template and re-point the same condition keys and guardrails at whatever your approved set becomes. This adaptability is the practical meaning of cryptographic agility: changing what you allow without re-engineering how you enforce it.
To see the workloads these controls govern, read Part 1 of this series, which builds a complete ML-DSA signing pipeline, from key creation and PKI through artifact signing and verification, using the approved set enforced here.
Software supply chain attacks have grown in both frequency and sophistication. Recent high-profile compromises of build systems and open source dependencies demonstrate that attackers increasingly target the trust mechanisms that verify software integrity. Those trust mechanisms rely on digital signature algorithms, and today many use traditional algorithms such as Rivest-Shamir-Adleman (RSA) and the Elliptic Curve Digital Signature Algorithm (ECDSA). If a sufficiently powerful quantum computer could forge these signatures, an attacker could potentially inject unauthorized code into signed artifacts, fabricate build attestations, or impersonate legitimate publishers. Using quantum-resistant signatures today defends against threats that might emerge in the future.
Since 2025, the Commercial National Security Algorithm Suite 2.0 (CNSA 2.0), published by the US National Security Agency (NSA), has required US National Security Systems (NSS) to “support and prefer” quantum-resistant signatures for software and firmware, with exclusive use mandated by 2030. The EU Cyber Resilience Act (CRA) takes full effect in December 2027. It requires manufacturers to maintain verifiable software integrity throughout a product’s lifecycle. Organizations that ship software to US federal agencies or sell products in the European market face converging deadlines that demand action now.
This post is Part 1 of a two-part series on adopting ML-DSA signatures, standardized in NIST FIPS 204 (ML-DSA) signatures, on AWS. As of Oct 2026, this series is the first end-to-end guide from any cloud provider for post-quantum signing across the software supply chain, from establishing an ML-DSA root of trust to enforcing algorithm policy across an organization. This part is for builders and walks through constructing and operating the signing pipeline. Part 2, Enforce your ML-DSA algorithm policy with SCPs and IAM, is for security and governance teams and shows how to turn the algorithm choices made here into preventive and detective controls across an organization.
Understanding the harvest now, forge later threat
After a cryptographically relevant quantum computer (CRQC) exists, the mathematical problems protecting today’s public-key cryptography can be solved efficiently. Attacks that are impractical today because of the computational complexity suddenly become practical using a CRQC. Most discussions of post-quantum cryptography (PQC) focus on the harvest now, decrypt later threat to data confidentiality. The asymmetric keys protecting data in transit can be efficiently derived using a CRQC, exposing the data that was meant to be protected. Digital signatures rely on the same asymmetric mathematical problems that a CRQC would break. This means that software supply chains face a parallel threat that targets integrity rather than confidentiality. An attacker who records signed artifacts today can forge those signatures in the future on a totally different artifact, using a CRQC.
Consider firmware signed with ECDSA P-256 and deployed to a fleet of embedded devices in 2026. These devices verify the firmware signature on every boot for their 15-year operational lifetime and can’t be recalled to re-sign after deployment. If a CRQC becomes operational in 2035, an attacker could produce a valid-looking signature over tampered firmware, and every device in the fleet would accept it as authentic on next restart. For organizations that rely on classical digital signatures as the primary mechanism for verifying software integrity, this weakens a critical link in the verification chain. This is why CNSA 2.0 specifies aggressive timelines for digital signatures: the artifacts you sign today must remain trustworthy for their full operational lifetime.
ML-DSA is designed to resist attacks from both classical and quantum computers. AWS KMS, AWS Private CA, and AWS IAM Roles Anywhere all support ML-DSA, making it possible to address the forge-later threat across your supply chain today without additional tooling.
Architecture overview: Five pillars of a quantum-resistant supply chain
The architecture comprises five trust pillars that together protect your software delivery pipeline:
ML-DSA key management in AWS KMS – All signing keys are generated and protected in FIPS 140-3 Security Level 3 hardware security modules (HSMs). Private keys never leave the HSM boundary.
ML-DSA public key infrastructure (PKI) throughAWS Private CA – A certificate hierarchy (root CA, subordinate CA, end-entity certificates) binds ML-DSA public keys to identities, establishing who signed what. CA private keys are generated and stored in hardware-protected HSMs managed by the service and never leave the HSM boundary.
Quantum-resistant workload identity (for external build agents) – For continuous integration and delivery (CI/CD) agents running outside AWS that authenticate through X.509 certificates, IAM Roles Anywhere with ML-DSA certificates ties the signer’s identity back to the same quantum-resistant root of trust.
ML-DSA SBOM integrity attestation – The SBOM for each release is wrapped in an in-toto Statement that names the artifact it describes by digest, and the Statement is signed with the same ML-DSA code-signing key. This provides quantum-resistant proof that the dependency manifest hasn’t been tampered with and that it belongs to the artifact it accompanies.
These pillars create a chain of trust: AWS Private CA protects the root CA key in its own HSMs, issues certificates that bind identities to AWS KMS code-signing keys, IAM Roles Anywhere uses those certificates to authenticate build agents, and authenticated agents sign every artifact and SBOM they produce.
Figure 1: End-to-end quantum-resistant software supply chain on AWS
Figure 1 illustrates this architecture in four layers, numbered in the order you build them:
Root of trust (Pillars 1 and 2) – Establish your AWS Private CA hierarchy and create your code-signing key in AWS KMS.
Workload identity (Pillar 3) – Register the CA as an IAM Roles Anywhere trust anchor and issue ML-DSA certificates to build agents, ensuring the signer’s identity chains back to the same quantum-resistant root of trust.
Artifact signing (Pillars 4 and 5) – Sign binaries, container images, and SBOMs using the authenticated build agent.
Verification and enforcement – Validate signatures at deployment and enforce ML-DSA-only policies.
Prerequisites
To follow along, you need:
An AWS account with permissions to create resources in AWS KMS, AWS Private CA, IAM Roles Anywhere, and Amazon ECR.
IAM Roles Anywhere credential helper (aws_signing_helper) v1.8.0 or later, the first release that supports ML-DSA keys and certificates, for workload authentication. Download from the IAM Roles Anywhere documentation.
OpenSSL 3.5 or later (required for ML-DSA signature verification and mu computation).
ORAS CLI v1.2 or later (for attaching Open Container Initiative (OCI) referrer artifacts to container images). Alternatively, you can use the Notation CLI with a custom plugin or any OCI-compliant client that supports the OCI Referrers API.
jq, for constructing and parsing the in-toto Statements used for SBOM attestation and for parsing ORAS output.
Amazon Inspector SBOM Generator (Sbomgen), or another tool that produces a CycloneDX JSON SBOM, for the SBOM example. Sbomgen runs only on Linux (x86-64 and ARM64), so run that step on a Linux build agent or container. It uses the registry credentials that oras login stores in the Docker config file, and it also accepts them through --username or the INSPECTOR_SBOMGEN_USERNAME and INSPECTOR_SBOMGEN_PASSWORD environment variables.
Java 11 or later and Apache Maven, to run the diy-code-signing-kms-private-ca sample that produces the code-signing CSR in Step 4 of Building the ML-DSA PKI foundation.
A container image pushed to Amazon ECR (for the container signing example).
Building the ML-DSA PKI foundation
Start by creating the certificate hierarchy that will anchor your supply chain trust. AWS Private CA generates and stores its own CA signing keys internally in hardware-protected HSMs when you create a certificate authority. Use ML-DSA-87 (NIST security category 5, defined as at least as hard to break as exhaustive key search on AES-256) for the root CA because it protects the longest-lived trust anchor. Use ML-DSA-65 (NIST security category 3) for the subordinate CA to balance security with smaller signature sizes for frequently issued certificates.
The parameter sets used here, ML-DSA-87 and ML-DSA-65, are also the approved set in the example algorithm policy that Part 2 of this series turns into enforceable controls. If your organization has standardized on a different set, substitute it in the commands that follow; the build steps don’t change.
Step 1: Create the root CA in AWS Private CA
When you specify KeyAlgorithm: ML_DSA_87, Private CA generates an ML-DSA-87 key pair internally. You don’t need to create a separate AWS KMS key for the CA.
After creation, the root CA is in PENDING_CERTIFICATE state. Activate it by self-signing its certificate. This is where you set the root CA’s validity period (10 years is typical for a root):
After creation, the subordinate CA enters PENDING_CERTIFICATE state with no link to the root. Activate it by signing its certificate signing request (CSR) with the root CA:
# Retrieve the subordinate CA's certificate signing request
aws acm-pca get-certificate-authority-csr \
--certificate-authority-arn <sub-ca-arn> \
--output text > sub-ca.csr
# Sign the CSR using the root CA (this establishes the trust hierarchy)
SUB_CERT_ARN=$(aws acm-pca issue-certificate \
--certificate-authority-arn <root-ca-arn> \
--csr fileb://sub-ca.csr \
--signing-algorithm ML_DSA_87 \
--validity Value=1825,Type=DAYS \
--template-arn arn:aws:acm-pca:::template/SubordinateCACertificate_PathLen0/V1 \
--query CertificateArn --output text)
# Retrieve the signed certificate and import it to activate the subordinate CA
aws acm-pca get-certificate \
--certificate-authority-arn <root-ca-arn> \
--certificate-arn $SUB_CERT_ARN \
--query Certificate --output text > sub-ca-cert.pem
aws acm-pca import-certificate-authority-certificate \
--certificate-authority-arn <sub-ca-arn> \
--certificate fileb://sub-ca-cert.pem \
--certificate-chain fileb://root-ca-cert.pem
Step 3: Create the code-signing key in AWS KMS
Your build agents use this key to sign artifacts, so their IAM roles need the kms:Sign permission on it. The end-entity certificate issued by your subordinate CA binds an identity to this key’s public key.
Step 4: Issue the end-entity code signing certificate
To issue a certificate, you need a CSR. Because the ML-DSA private key never leaves the AWS KMS HSM, you can’t generate the CSR locally with OpenSSL. Instead, you construct the CSR’s TBS (to-be-signed) structure containing your subject name and public key, sign it remotely by calling aws kms sign, and assemble the final CSR from those components.
For a working implementation that assembles an ML-DSA CSR around a KMS-managed private key and signs it with AWS KMS, see Post-quantum (ML-DSA) code signing with AWS Private CA and AWS KMS and its diy-code-signing-kms-private-ca sample on GitHub (Java, built with Maven). There is no AWS CLI-only path for this step: the CSR must be constructed and signed in code, and the sample is the reference implementation.
After you have the code-signing.csr file, issue the certificate from your subordinate CA:
# Issue a code-signing certificate valid for one year, signed by the subordinate CA using ML-DSA-65.
# Replace <sub-ca-arn> with the ARN that create-certificate-authority returned for the subordinate CA in Step 2.
CODE_SIGNING_CERT_ARN=$(aws acm-pca issue-certificate \
--certificate-authority-arn <sub-ca-arn> \
--csr fileb://code-signing.csr \
--signing-algorithm ML_DSA_65 \
--validity Value=365,Type=DAYS \
--template-arn arn:aws:acm-pca:::template/CodeSigningCertificate/V1 \
--query CertificateArn --output text)
# Retrieve the issued certificate
aws acm-pca get-certificate \
--certificate-authority-arn <sub-ca-arn> \
--certificate-arn $CODE_SIGNING_CERT_ARN \
--query Certificate --output text > code-signing-cert.pem
This certificate binds the signer identity to the AWS KMS code-signing key material. Note that the certificate validity period (365 days in this example) determines how long verifiers can validate signatures by checking the certificate alone. AWS Private CA lets you choose the validity period of an issued certificate, as long as it ends no later than the issuing CA’s own certificate; in this example the subordinate CA certificate from Step 2 is valid for 1,825 days, so the three-year option below fits inside it. For artifacts that must remain verifiable beyond the certificate’s expiry, consider these approaches:
Longer certificate validity – For private PKI, code-signing certificates are commonly issued with multi-year validity (for example, Value=1095,Type=DAYS for 3 years) because you control the issuance policy.
Trusted timestamping (RFC 3161) – Countersign each signature with an RFC 3161 timestamp from a Time Stamping Authority (TSA) at signing time. The timestamp proves the signature was created while the certificate was valid, allowing verification indefinitely. AWS doesn’t provide a managed TSA; use a third-party TSA or operate your own.
Re-signing – Before a certificate expires, re-sign active artifacts with a newly issued certificate.
Distribute the certificate chain (root CA certificate, subordinate CA certificate, and end-entity certificate) to systems that need to verify your signatures.
Signing artifacts with ML-DSA in your CI/CD pipeline
With the PKI hierarchy in place, integrate ML-DSA signing into your build process. The approach depends on artifact type. ML-DSA-65 signatures are 3,309 bytes (compared to 64 bytes for ECDSA P-256), and the latency of the AWS KMS Sign operation for ML-DSA is comparable to existing asymmetric operations. The larger signature size has negligible impact when stored alongside multi-megabyte container images or packages, but is worth accounting for in size-constrained environments like embedded device images transmitted over low-bandwidth links.
Binary and package signing
Most build outputs, including compiled binaries and packages, are far larger than the 4,096-byte limit that AWS KMS places on the message passed to aws kms sign. Make the EXTERNAL_MU method your pipeline’s default: compute the 64-byte message representative (mu) externally with OpenSSL 3.5, then pass only mu to AWS KMS. This handles an artifact of any size and never needs to change as your artifacts grow. AWS KMS also accepts content directly with --message-type RAW, up to 4,096 bytes. Use RAW only for inputs whose size is fixed by construction, such as the 71-character manifest digest in the container image example that follows. An artifact that fits today is likely to outgrow the limit later, and a pipeline that depends on RAW fails when it does. Both methods produce cryptographically identical signatures. The verifier doesn’t need to know which method was used; standard ML-DSA verification applies in either case.
# Retrieve the public key
aws kms get-public-key --key-id alias/pqc-code-signing \
--query PublicKey --output text | base64 --decode > pk.der
# Extract raw public key bytes from the DER-encoded SubjectPublicKeyInfo.
# The BIT STRING starts at offset 17 for all ML-DSA key sizes.
# Expected output: 1,312 bytes (ML-DSA-44), 1,952 bytes (ML-DSA-65), or 2,592 bytes (ML-DSA-87).
openssl asn1parse -inform DER -in pk.der -strparse 17 -noout -out pk_raw.bin
# Compute mu per FIPS 204 Section 6.2
printf '\x00\x00' | cat - artifact.bin > m_prime.bin
openssl dgst -shake256 -xoflen 64 -binary pk_raw.bin > pk_hash.bin
cat pk_hash.bin m_prime.bin | openssl dgst -shake256 -xoflen 64 -binary > mu.bin
# Sign the mu value
aws kms sign \
--key-id alias/pqc-code-signing \
--message fileb://mu.bin \
--message-type EXTERNAL_MU \
--signing-algorithm ML_DSA_SHAKE_256 \
--query Signature --output text | base64 --decode > artifact.bin.sig
Store the .sig file alongside the artifact in Amazon S3 or your package repository. Protect the integrity of both the artifact and its signature by enabling S3 Object Lock (WORM protection) or at minimum S3 Versioning with restricted write access. If an attacker can modify or delete either file, verification can’t be trusted. For comprehensive guidance, see Security best practices for Amazon S3.
Container image signing
For container image signing, use the OCI referrer pattern, defined in the OCI Image and Distribution Specifications v1.1. A referrer is an artifact, such as a signature or an SBOM, whose manifest names another image in its subject field. Registries that support the pattern, including Amazon ECR, return these artifacts when a client asks for an image’s referrers. This approach uses AWS KMS ML-DSA keys to generate the signature and stores it as a referrer of the image:
# Repository URI used by ORAS here and in the SBOM steps; replace the account ID and Region
ECR_URI=111122223333.dkr.ecr.us-east-1.amazonaws.com/myapp
# Authenticate ORAS to Amazon ECR
aws ecr get-login-password --region us-east-1 | \
oras login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com
# Get the image manifest digest
IMAGE_DIGEST=$(aws ecr batch-get-image \
--repository-name myapp \
--image-ids imageTag=v1.2.0 \
--query 'images[0].imageId.imageDigest' --output text)
# Sign the digest. It is a fixed 71 characters, so RAW is safe here (see Binary and package signing).
# AWS CLI v2 expects blob parameters to be base64, so write the digest to a file and pass it with fileb://
echo -n "$IMAGE_DIGEST" > digest.txt
aws kms sign \
--key-id alias/pqc-code-signing \
--message fileb://digest.txt \
--message-type RAW \
--signing-algorithm ML_DSA_SHAKE_256 \
--query Signature --output text | base64 --decode > image.mldsa.sig
# Attach signature as OCI referrer using ORAS
oras attach $ECR_URI@$IMAGE_DIGEST \
--artifact-type "application/vnd.example.mldsa-signature.v1" \
image.mldsa.sig:application/octet-stream
At deployment time, your admission controller retrieves the referrer artifact and verifies the ML-DSA signature before allowing the image to run. See Verifying a container image signature before deployment later in this post for the full verification workflow and admission control implementation.
SBOM signing
A software bill of materials (SBOM) is a structured inventory of every component, library, and dependency included in a software release. It lists package names, versions, suppliers, and dependency relationships. Organizations use SBOMs to track vulnerabilities, verify license compliance, and respond quickly when a compromised component is disclosed.
Signing the SBOM file on its own proves that the inventory hasn’t been tampered with since it was signed, but not which artifact it describes. If the image signature and the SBOM signature are independent of each other, an attacker who holds a legitimately signed SBOM for a clean image can present it alongside a different signed image and claim that the two correspond. Attaching the SBOM to the image as an OCI referrer doesn’t close this gap by itself, because the subject field of a referrer manifest is set by whoever pushes the referrer; it’s a discovery mechanism, not a cryptographic binding. The binding must be inside the bytes that the signature covers.
To create that binding, wrap the SBOM in an in-toto Statement before signing it. A Statement names the artifacts it describes in its subject field, by digest, and carries the SBOM as its predicate. Because the image manifest digest is part of the signed content, a verifier can confirm that the SBOM in hand was issued for exactly the image it is about to deploy, and a swapped SBOM or a swapped digest fails verification. Embedding the SBOM in the image filesystem is another way to bind the two, because the image digest then covers the SBOM bytes. This post uses the Statement approach because it also works for SBOMs generated by scanning a finished image, for artifacts that have no filesystem such as binaries and firmware, and for images that are already in your registry, and because the SBOM stays readable from the registry without pulling the image.
Generate the SBOM with your preferred tool (such as Amazon Inspector SBOM Generator (Sbomgen) for container images), wrap it in a Statement, and sign the Statement with the same EXTERNAL_MU step you use for binaries:
# Scan the pushed image by digest so the inventory describes exactly the manifest you signed.
# Sbomgen reuses the registry login from the oras login step in the previous section.
inspector-sbomgen container --image $ECR_URI@$IMAGE_DIGEST -o sbom.json
# Wrap the SBOM in an in-toto Statement whose subject is the image manifest digest.
# This puts the digest inside the signed bytes, which binds this SBOM to this image.
jq -n \
--arg name "$ECR_URI" \
--arg digest "${IMAGE_DIGEST#sha256:}" \
--slurpfile sbom sbom.json \
'{
"_type": "https://in-toto.io/Statement/v1",
"subject": [{"name": $name, "digest": {"sha256": $digest}}],
"predicateType": "https://cyclonedx.org/bom",
"predicate": $sbom[0]
}' > sbom.statement.json
# Sign using EXTERNAL_MU (same mu computation as binary signing above,
# with sbom.statement.json as the input)
# ... compute mu from sbom.statement.json ...
aws kms sign \
--key-id alias/pqc-code-signing \
--message fileb://mu.bin \
--message-type EXTERNAL_MU \
--signing-algorithm ML_DSA_SHAKE_256 \
--query Signature --output text | base64 --decode > sbom.statement.json.sig
# Attach the Statement and its signature to the image as an OCI referrer so
# verifiers discover them alongside the image signature
oras attach $ECR_URI@$IMAGE_DIGEST \
--artifact-type "application/vnd.example.mldsa-sbom-attestation.v1" \
sbom.statement.json:application/vnd.in-toto+json \
sbom.statement.json.sig:application/octet-stream
Ship the Statement and its .sig file together; the SBOM itself is the Statement’s predicate field, and verifiers extract it after verification. The same pattern binds an SBOM to a binary or package: set the subject digest to the SHA-256 of the file instead of an image manifest digest. The Verifying a binary or SBOM signature section later in this post walks through the full verification workflow, including the subject check that completes the binding.
Quantum-resistant workload identity for build agents
Signing artifacts with ML-DSA keys addresses the integrity of the artifacts themselves. Separately, build agents running outside AWS (on-premises Jenkins, self-hosted GitHub Actions runners, or third-party CI/CD systems) authenticate using X.509 certificates or OIDC tokens typically signed with RSA or ECDSA. These authentication credentials represent a classical cryptographic dependency in the pipeline. IAM Roles Anywhere with ML-DSA certificates replaces that dependency: the credential that the external build agent presents, and the CA that vouches for it, no longer rely on algorithms that are vulnerable to quantum attack. (The TLS connection that carries the request still authenticates the AWS endpoint with a traditional certificate, as public TLS does today; that is a transport concern separate from the signing identity this post is about.) Build agents running on AWS with an attached IAM role don’t have this dependency: their API requests are authenticated with AWS Signature Version 4 (SigV4), which uses a symmetric hash-based message authentication code (HMAC-SHA256). Symmetric schemes like this are already considered quantum-resistant, so no additional action is needed.
Step 1: Register a workload identity CA as a trust anchor
For this walkthrough, reuse the subordinate CA that you created earlier; that avoids the monthly charge for a second CA, and the commands that follow assume it. In production, use a dedicated subordinate CA for workload identity certificates, separate from your code-signing CA. This limits the blast radius if either CA is compromised and has to be revoked, and it lets you apply different issuance policies to the two certificate types.
# Use the ARN that create-certificate-authority returned when you created the
# subordinate CA earlier. If you no longer have it, list your CAs and copy the
# ARN of the SUBORDINATE one.
aws acm-pca list-certificate-authorities \
--query "CertificateAuthorities[?Type=='SUBORDINATE'].[Arn,Status,CertificateAuthorityConfiguration.Subject.CommonName]" \
--output table
SUB_CA_ARN=arn:aws:acm-pca:us-east-1:111122223333:certificate-authority/abc-123 # replace with yours
# Register the subordinate CA as an IAM Roles Anywhere trust anchor
aws rolesanywhere create-trust-anchor \
--name "PQC-CI-CD-Trust-Anchor" \
--source "sourceType=AWS_ACM_PCA,sourceData={acmPcaArn=$SUB_CA_ARN}" \
--enabled
Step 2: Create the build agent role and a profile that references it
An IAM Roles Anywhere profile specifies which IAM roles a workload can assume after it authenticates with a certificate from your trust anchor. Create the role first. Its trust policy lets IAM Roles Anywhere assume it only through your trust anchor and only for a certificate whose subject common name identifies the build agent; without those conditions, any certificate that the trust anchor CA issues could assume the role. Its permissions policy grants only what signing requires, on only the code-signing key. Save the trust policy as build-agent-trust.json, replacing the trust anchor ARN with the one that create-trust-anchor returned in Step 1:
Save the permissions policy as build-agent-permissions.json, replacing the resource with your key’s ARN (aws kms describe-key --key-id alias/pqc-code-signing --query KeyMetadata.Arn --output text):
Step 3: Issue the build agent certificate and configure the credential helper
Generate the build agent’s ML-DSA-65 key pair and CSR locally with OpenSSL, have the subordinate CA issue a client authentication certificate for it, and then configure the IAM Roles Anywhere credential helper to present that certificate:
# Generate the build agent's key pair and CSR. The CN must match the
# aws:PrincipalTag/x509Subject/CN condition in the role's trust policy.
openssl genpkey -algorithm ML-DSA-65 -out build-agent-mldsa-key.pem
chmod 400 build-agent-mldsa-key.pem
openssl req -new -key build-agent-mldsa-key.pem \
-subj "/C=US/O=Example Corp/CN=pqc-build-agent-01" \
-out build-agent.csr
# Issue a 30-day client authentication certificate from the subordinate CA
# registered as the trust anchor in Step 1.
BUILD_AGENT_CERT_ARN=$(aws acm-pca issue-certificate \
--certificate-authority-arn $SUB_CA_ARN \
--csr fileb://build-agent.csr \
--signing-algorithm ML_DSA_65 \
--validity Value=30,Type=DAYS \
--template-arn arn:aws:acm-pca:::template/EndEntityClientAuthCertificate/V1 \
--query CertificateArn --output text)
aws acm-pca get-certificate \
--certificate-authority-arn $SUB_CA_ARN \
--certificate-arn $BUILD_AGENT_CERT_ARN \
--query Certificate --output text > build-agent-mldsa.pem
# In production, store the private key in a TPM or HSM rather than as a plaintext file.
aws_signing_helper credential-process \
--certificate build-agent-mldsa.pem \
--private-key build-agent-mldsa-key.pem \
--trust-anchor-arn arn:aws:rolesanywhere:us-east-1:111122223333:trust-anchor/ta-id \
--profile-arn arn:aws:rolesanywhere:us-east-1:111122223333:profile/profile-id \
--role-arn arn:aws:iam::111122223333:role/PQCBuildAgentRole
The helper produces short-lived AWS Security Token Service (AWS STS) credentials scoped to the build role, replacing long-lived AWS access keys. In production, protect the ML-DSA private key with hardware-backed storage such as a Trusted Platform Module (TPM), an HSM, or an encrypted credential store with restricted file permissions. The plaintext file path shown here is for demonstration purposes. The private key authorizes access to your code-signing infrastructure and should be treated with the same protection level as any signing key. On Linux build agents, at minimum restrict the file to chmod 400 owned by the service account and consider using a PKCS#11 provider to avoid storing the raw key on disk entirely.
Verification and policy enforcement
A signing architecture is only as strong as its verification. Producing signatures is half the story. The other half is giving the systems and people that deploy your artifacts, such as deployment pipelines, Kubernetes admission controllers, and engineers installing a package, a concrete way to verify those signatures and make trust decisions before an artifact is deployed or run.
Distributing your trust chain
Before any verifier can check a signature, it needs to trust the CA that issued the code-signing certificate. This trust is established through a trust store: a local collection of root CA certificates that a system treats as authoritative. Trust stores exist in operating systems (such as /etc/pki/ca-trust/ on Linux, Keychain on macOS), language run-times (Java cacerts), container base images, and custom application configurations.
For private PKI hierarchies like the one in this post, your root CA certificate isn’t in any default public trust store. You must explicitly add it to the trust stores of every system that will verify your signatures. Common approaches include:
Baking the root CA certificate into your container base images during the build process
Publishing it to an internal S3 bucket with versioning and access logging
Storing it as an OCI artifact in Amazon ECR alongside the images it protects
Treat trust store management with the same rigor as credential management. If an attacker can inject a rogue root certificate into your trust stores, they can make forged signatures appear valid. Restrict write access to trust stores and audit changes through AWS CloudTrail or your system-level audit logs.
Exporting your certificate chain requires exporting the root CA, and then any subordinate CAs that issue certificates that will be involved in verifying signatures:
# Export the root CA certificate
aws acm-pca get-certificate-authority-certificate \
--certificate-authority-arn <root-ca-arn> \
--query Certificate --output text > root-ca.pem
# Export the subordinate CA certificate. Select only the Certificate field: for a
# subordinate CA this call also returns CertificateChain, and --output text would
# print both on one tab-separated line, producing a file that OpenSSL can't parse.
aws acm-pca get-certificate-authority-certificate \
--certificate-authority-arn <sub-ca-arn> \
--query Certificate --output text > sub-ca.pem
# Bundle the full chain for verifiers
cat root-ca.pem sub-ca.pem > trust-chain.pem
Verifiers use this chain to confirm that the code-signing public key was issued by a CA they trust.
The CAs in this post are created without a revocation configuration, and the verification commands that follow don’t check revocation status, which keeps the walkthrough short. In production, configure a certificate revocation list (CRL) or an Online Certificate Status Protocol (OCSP) responder on the subordinate CA (see Plan your AWS Private CA certificate revocation method in the AWS Private CA User Guide; both work with ML-DSA CAs). If you use a CRL, distribute it alongside the trust chain and add -crl_check -CRLfile <crl-file> to the openssl verify command below, where <crl-file> is the CRL that AWS Private CA publishes to your S3 bucket, so that a revoked code-signing certificate fails verification.
Verifying a binary or SBOM signature
Verify the code-signing certificate against the trust chain, confirm that it is a code-signing certificate issued to the signer you expect, extract its public key, and then verify the signature with OpenSSL 3.5. The second check matters: a chain check alone accepts every certificate that the CA hierarchy has issued, including the build agent client certificates from the previous section, so pin the expected subject in your verifier configuration:
# Step 1: Verify that the code-signing certificate chains to the trusted root CA and carries the
# Code Signing extended key usage. -purpose codesign (OpenSSL 3.2 and later) rejects a certificate
# that chains correctly but was issued for another purpose, such as a build agent's client
# authentication certificate from the same subordinate CA.
# trust-chain.pem contains the root CA and subordinate CA certificates distributed in the previous section.
openssl verify -CAfile trust-chain.pem -purpose codesign code-signing-cert.pem
# Confirm that the certificate was issued to the signer you expect. Replace <signer-cn> with the
# common name in your code-signing certificate's subject.
openssl x509 -in code-signing-cert.pem -noout -subject -nameopt RFC2253 | grep -q "CN=<signer-cn>" || { echo "Unexpected signer"; exit 1; }
# Step 2: Extract the public key from the verified certificate.
openssl x509 -in code-signing-cert.pem -pubkey -noout > signing-key.pem
# Step 3: Verify the artifact signature using the trusted public key.
openssl pkeyutl -verify \
-pubin -inkey signing-key.pem \
-in artifact.bin \
-sigfile artifact.bin.sig \
-rawin
A successful verification prints Signature Verified Successfully and returns an exit status of 0. A failed verification prints Signature Verification Failure and returns a nonzero exit status. In shell scripts, check the exit status of each command, or run the block with set -e, so that a failed certificate chain check stops the script before signature verification and a failed signature check blocks downstream actions.
For an SBOM, run the same three steps with sbom.statement.json as the -in file and sbom.statement.json.sig as the -sigfile, then add a fourth: confirm that the subject digest inside the verified Statement is the digest of the artifact you are about to deploy. Signature verification proves the Statement is authentic; the subject check proves it describes this artifact and not another one. The digest you compare against must come from the artifact itself, resolved from the registry for an image or computed from the file for a binary, never from the Statement. For an image, this is the same digest you verify the image signature against, so the two checks together establish that this SBOM describes this signed image.
# Step 4 (SBOM only): the signed Statement must name the image being deployed.
# Resolve the digest from Amazon ECR, not from the Statement.
IMAGE_DIGEST=$(aws ecr batch-get-image \
--repository-name myapp \
--image-ids imageTag=v1.2.0 \
--query 'images[0].imageId.imageDigest' --output text)
SUBJECT_DIGEST=$(jq -r '.subject[0].digest.sha256' sbom.statement.json)
[ "sha256:$SUBJECT_DIGEST" = "$IMAGE_DIGEST" ] || { echo "SBOM does not describe this image"; exit 1; }
# Only now extract the SBOM for downstream tooling
jq '.predicate' sbom.statement.json > sbom.json
For a binary, compare the subject digest to the output of sha256sum artifact.bin instead. Reject the SBOM if either the signature or the subject check fails.
Verifying a container image signature before deployment
Pull the ML-DSA signature from the OCI referrer artifact in Amazon ECR, then verify it against the image manifest digest:
# Pull the signature referrer. ORAS 1.2 lists referrers under "manifests";
# ORAS 1.3 and later list them under "referrers". The jq filter accepts either.
oras discover $ECR_URI@$IMAGE_DIGEST \
--artifact-type "application/vnd.example.mldsa-signature.v1" --format json | \
jq -r '(.referrers // .manifests)[0].digest' | \
xargs -I{} oras pull $ECR_URI@{} -o /tmp/sig/
# Re-derive the image digest and verify
echo -n "$IMAGE_DIGEST" > /tmp/digest.txt
openssl pkeyutl -verify \
-pubin -inkey signing-key.pem \
-in /tmp/digest.txt \
-sigfile /tmp/sig/image.mldsa.sig \
-rawin
# Exit code 0 = image is authentic; non-zero = reject deployment
The same oras discover command with --artifact-type "application/vnd.example.mldsa-sbom-attestation.v1" retrieves the SBOM attestation and its signature; verify them as described in the previous section, including the subject check. The jq filter shown takes the first referrer of the requested type. Don’t rely on the order in which a registry lists referrers, and expect more than one after an image has been re-signed: a production verifier should check every referrer of that type and accept the image if at least one signature verifies against a certificate that chains to your trust store. Taking only the first can reject a correctly signed image, but it can never accept an unsigned one.
To enforce these checks at deployment time, run them in a Kubernetes admission controller, a component that the API server calls before it accepts a pod and that can reject the request. You can write one as a validating admission webhook (see Dynamic Admission Control in the Kubernetes documentation), or use Open Policy Agent (OPA) Gatekeeper, an admission webhook that enforces policies evaluated by the OPA policy engine, and package the verification steps as a Gatekeeper external data provider that a constraint calls. For a worked example of gating Amazon EKS deployments on image signatures, see Announcing Container Image Signing with AWS Signer and Amazon EKS, which verifies AWS Signer signatures at admission with Kyverno and discusses the Gatekeeper and custom webhook alternatives. The ML-DSA signature referrer in this post is a custom artifact type that those tools don’t verify out of the box, so the verification steps in this section run in code you supply. In either design, the controller runs before any pod is scheduled, rejecting images that lack a valid ML-DSA signature referrer in Amazon ECR or whose SBOM attestation doesn’t name the image being deployed.
Preventive and detective controls
Signature verification stops the use of an untrusted artifact at deployment. Organization-level controls apply earlier and more broadly: preventive controls stop teams from creating new traditional signing keys, and detective controls record every signing operation so you can audit which principal signed each artifact and when.
Preventive guardrails. Use an AWS OrganizationsService control policy (SCP) to block creation of traditional signing keys in security-critical accounts, forcing all new signing workloads to use ML-DSA:
This SCP prevents teams from creating new RSA or ECDSA signing keys, enforcing quantum-resistant signatures as the only option for new workloads. Note that this SCP constrains the algorithm family, not the parameter set: it would still permit an ML-DSA-44 key, which this architecture doesn’t use. Part 2 of this series picks up at exactly this point. It shows how to express a complete algorithm policy (in its example, approve ML-DSA-65 and ML-DSA-87 and exclude ML-DSA-44) as layered controls using the kms:KeySpec condition key in SCPs, IAM permission boundaries, and AWS KMS key policies; how to restrict the use of keys that already exist; and how to govern surfaces such as AWS Private CA that have no algorithm condition key. This SCP and the parameter-set SCP in Part 2 work together: attach both, and every new signing key must use an approved parameter set.
Audit trail.AWS CloudTrail logs every kms:Sign and kms:Verify API call, providing an immutable record of when each artifact was signed and by which principal. Use an organization trail and query its logs with Amazon Athena to analyze signing activity across accounts and correlate it with your deployment events.
Regulatory timeline and recommended actions
The following table maps regulatory milestones to specific actions you can take today using the architecture described in this post. This information is provided for general awareness only and does not constitute legal or compliance advice. Consult your legal and compliance teams to determine your organization’s specific obligations. For a broader overview of global PQC regulatory timelines, see Migrating to Post-Quantum Cryptography.
Regulation
Deadline
Regulatory language
Action on AWS
CNSA 2.0: Support and prefer
2025 (in effect)
CNSA 2.0 timeline (Sep 2022, FAQ v2.1 Dec 2024): Software and firmware signing must “support and prefer” CNSA 2.0 algorithms by 2025
Create ML-DSA keys in AWS KMS, begin signing artifacts with ML-DSA alongside existing traditional signatures
CNSA 2.0: New system acquisitions
January 2027
CNSA 2.0 FAQ v2.1 (Dec 2024): “All new NSS acquisitions must be CNSA 2.0 compliant at deployment, effective 1 January 2027”
Full ML-DSA PKI operational, IAM Roles Anywhere trust anchors using ML-DSA, all new builds carry ML-DSA signatures
EU Cyber Resilience Act
11 December 2027
Regulation (EU) 2024/2847, Annex I, Part I(2)(f): Products must “protect the integrity of stored, transmitted or otherwise processed data, personal or other, commands, programs and configuration against any manipulation or modification not authorised by the user, and report on corruptions”; Part II(7): Manufacturers must “provide for mechanisms to securely distribute updates”
ML-DSA code signing for all software updates shipped to EU markets, signed SBOMs accompany every release
CNSA 2.0: Exclusive use
2030
CNSA 2.0 timeline: Software and firmware signing must “exclusively use” CNSA 2.0 algorithms by 2030
Remove traditional signing keys from supply chain, SCP enforcement of ML-DSA-only key creation
PCI DSS 4.0 Req 12.3.3
Mandatory since March 2025
PCI DSS v4.0.1, Requirement 12.3.3: “Cryptographic cipher suites and protocols in use are documented and reviewed at least once every 12 months”, including an up-to-date inventory of the cipher suites and protocols in use
Document cryptographic inventory in AWS KMS, tag ML-DSA versus traditional keys, maintain migration progress tracking
Note for NSS customers and other CNSA 2.0 adopters: CNSA 2.0 is the quantum-resistant algorithm suite that the NSA requires for NSS, the federal systems that carry classified information or support intelligence, cryptologic, military command and control, and weapons functions. When ML-DSA is used for any signature purpose, including software and firmware signing, CNSA 2.0 permits only the ML-DSA-87 parameter set. (For software and firmware signing it also approves the stateful hash-based schemes LMS and XMSS, which this post doesn’t cover.) The ML-DSA-65 parameter set used in this post’s code-signing examples is appropriate for commercial workloads but isn’t permitted on NSS. CNSA 2.0 is binding only for NSS, but it’s followed voluntarily well beyond that boundary: by suppliers to the US defense and intelligence market, in the United States and allied countries, whose products must meet NSS requirements, and by commercial organizations in and outside the United States that adopt it as a conservative baseline because it specifies the highest NIST security category and removes the need to revisit parameter choices later. If your systems fall under CNSA 2.0 scope, or you’ve chosen to align with it, substitute ML-DSA-87 for all signing keys and certificates. Part 2 of this series shows how to make that substitution enforceable: the same kms:KeySpec controls that exclude ML-DSA-44 in its example can be pointed at ML_DSA_87 as the only approved key spec.
For organizations serving both US government and EU commercial markets, the January 2027 and December 2027 deadlines converge into a single migration window. Starting now provides margin for testing, key rotation planning, and certificate chain validation.
Clean up
You created a number of cryptographic resources in the prior steps and examples. AWS Private CA charges a monthly fee per CA and AWS KMS charges per key per month. To avoid ongoing charges, delete them if you aren’t actively using them. Start by listing what exists, so that you catch every resource you created, including duplicates from repeated runs. These commands list one Region; if you worked in more than one, repeat them with --region for each.
# List every private CA with its type, status, key algorithm, and subject.
# A CA you already deleted shows DELETED until its restoration window ends.
aws acm-pca list-certificate-authorities \
--query 'CertificateAuthorities[].[Arn,Type,Status,CertificateAuthorityConfiguration.KeyAlgorithm,CertificateAuthorityConfiguration.Subject.CommonName]' \
--output table
# List every ML-DSA key in AWS KMS with its state and creation date.
# A key already scheduled for deletion shows PendingDeletion.
for KEY_ID in $(aws kms list-keys --query 'Keys[].KeyId' --output text); do
aws kms describe-key --key-id "$KEY_ID" \
--query 'KeyMetadata.[KeyId,KeySpec,KeyState,CreationDate]' --output text
done | grep ML_DSA
# Show which alias points at which key. A key that no alias points at
# is usually a leftover from an earlier run.
aws kms list-aliases --query 'Aliases[].[AliasName,TargetKeyId]' --output text | grep -v '^alias/aws/'
AWS Private CA has no command that lists the certificates a CA has issued. If you need serial numbers for the revocation step that follows, generate an audit report with aws acm-pca create-certificate-authority-audit-report, which writes a list of every certificate the CA has issued or revoked to an S3 bucket that grants the service write access.
Delete each resource that you don’t need, including any extra copies that the listing revealed, then run the listing again to confirm that nothing is left:
# Revoke any issued end-entity certificates while the subordinate CA is still active
aws acm-pca revoke-certificate \
--certificate-authority-arn <sub-ca-arn> \
--certificate-serial <serial-number> \
--revocation-reason CESSATION_OF_OPERATION
# Delete the IAM Roles Anywhere resources that reference the CA
aws rolesanywhere delete-profile --profile-id <profile-id>
aws rolesanywhere delete-trust-anchor --trust-anchor-id <trust-anchor-id>
# Delete the build agent role that the profile referenced, and the local copy of its private key
aws iam delete-role-policy --role-name PQCBuildAgentRole --policy-name pqc-code-signing
aws iam delete-role --role-name PQCBuildAgentRole
rm -f build-agent-mldsa-key.pem
# Delete the AWS KMS code-signing key (schedule 7-day waiting period).
# ScheduleKeyDeletion doesn't accept an alias, so resolve the key ID first.
KEY_ID=$(aws kms describe-key --key-id alias/pqc-code-signing \
--query KeyMetadata.KeyId --output text)
aws kms schedule-key-deletion --key-id $KEY_ID --pending-window-in-days 7
# Remove the alias. It survives key deletion and would otherwise show up in the
# listing above on your next run.
aws kms delete-alias --alias-name alias/pqc-code-signing
# Disable and delete the subordinate CA
aws acm-pca update-certificate-authority \
--certificate-authority-arn <sub-ca-arn> --status DISABLED
aws acm-pca delete-certificate-authority \
--certificate-authority-arn <sub-ca-arn> --permanent-deletion-time-in-days 7
# Disable and delete the root CA
aws acm-pca update-certificate-authority \
--certificate-authority-arn <root-ca-arn> --status DISABLED
aws acm-pca delete-certificate-authority \
--certificate-authority-arn <root-ca-arn> --permanent-deletion-time-in-days 7
If you are adapting this for long-term use, plan for log retention. CloudTrail Event History keeps only the last 90 days of management events and is not a substitute for a trail. For the audit trail described earlier, create a trail that delivers to an S3 bucket with the lifecycle policy and access controls your retention requirements call for.
Conclusion
In this post, I showed you how to build a quantum-resistant software supply chain on AWS by combining ML-DSA signing in AWS KMS, ML-DSA certificate hierarchies in AWS Private CA, and ML-DSA workload authentication through IAM Roles Anywhere. The architecture covers the full pipeline; from authenticated build agents through signed binaries, container images, and SBOMs, to verification and policy enforcement at deployment.
Every ML-DSA capability used in this post is generally available. AWS Private CA and IAM Roles Anywhere support ML-DSA in all commercial AWS Regions, the AWS GovCloud (US) Regions, and the China Regions, and AWS KMS supports ML-DSA in commercial AWS Regions. You don’t need to wait for a standards body or third-party tool to finalize PQC support. The primitives are ready.
To get started, create your first ML-DSA key in AWS KMS and sign a test artifact. From there, extend to your PKI hierarchy, integrate IAM Roles Anywhere for your build agents, and add ML-DSA signatures to your container and SBOM workflows.
Building the pipeline is half the work. The other half is making sure the algorithm choices you made stay in force as accounts, teams, and standards change. That’s the subject of Part 2 of this series, Enforce your ML-DSA algorithm policy with SCPs and IAM, written for the security and governance teams who own that problem.
When you build multi-Region architectures on AWS, one question comes up early: “What services are available in each AWS Region?” The answer shapes architecture decisions, from which AWS Regions to expand into, to how you design for resilience. For teams navigating data residency requirements and compliance reporting, getting the answer wrong has real consequences: deployment failures, compliance gaps, and delayed launches.
AWS publishes Regional availability data covering services, features, APIs, and AWS CloudFormation resource types across all AWS Regions. You can explore this data on the AWS Capabilities by Region page, access it through Amazon Simple Storage Service (Amazon S3) for pipeline integration, or query it using the AWS Knowledge MCP server. These options work well for exploration and automation. However, teams told us they need this data deployed as infrastructure they own, refreshing on their schedule, inside their network, filtered to their workload. That means running availability data the same way you run the rest of your stack: in your Amazon Virtual Private Cloud (Amazon VPC), under your governance.
In this post, we introduce two open source solutions that deliver that ownership. Capability Insights for AWS deploys a Regional availability dashboard into your VPC that auto-refreshes every 24 hours. The dashboard, its API, and availability data all run in your own account, and the scheduled refresh makes the only call that leaves your account when it reads the dataset that AWS publishes. Workload Analysis scans your AWS CloudTrail logs and CloudFormation stacks, then narrows 200+ services to the 20–30 your account actually runs, significantly reducing the scope of a Regional gap analysis.
Together with the Capabilities by Region S3 Access Point, you now have three levels of control: consume from Amazon S3, deploy into your VPC, or filter to your workload. We walk through how to deploy each solution, run a workload analysis, and view personalized results during your Regional expansion planning. Whether you’re building multi-Region recovery strategies, standardizing compliance reporting, or accelerating expansion timelines, these solutions put the data and the decisions it drives inside your perimeter.
Figure 1 shows how Capability Insights for AWS integrates with the VPC and subnets in your existing AWS account. A client in the public subnet accesses the dashboard through an S3 gateway endpoint and calls the private API through an API Gateway VPC endpoint.
Figure 1: Capability Insights for AWS architecture, including private dashboard access and scheduled Regional availability data refreshes.
An Amazon EventBridge schedule invokes the data-fetch Lambda function every 24 hours. The data-fetch function runs outside your VPC, so it reads the S3 access point published by AWS over the AWS network. The function pulls Regional availability data from the Capabilities by Region S3 bucket published by AWS and writes it to the website bucket within your account. The dashboard application accesses the data within the website through the S3 gateway endpoint from your VPC. The API Lambda can also invoke the data-fetch function on demand through the Lambda VPC endpoint. The deployment assets bucket supplies the Lambda code during deployment only.
Prerequisites
To follow along, you need:
An AWS account with permissions to deploy the CloudFormation stacks, including permission to create named AWS Identity and Access Management (IAM) roles. At runtime, the roles created by the stacks use scoped permissions for Amazon S3 object read and write access, AWS CloudFormation read operations, Amazon Athena queries, AWS Glue and AWS Lake Formation catalog access, AWS Lambda invocation, and AWS Step Functions execution. For starting-point deployment policies, see the documentation folder in the repository.
A VPC with DNS resolution and DNS hostnames enabled, one public subnet (routed to an internet gateway) for dashboard and API access, and one private subnet (no internet route) for the in-VPC Lambda function. Both subnets need a route to Amazon S3 through a gateway VPC endpoint.
Node.js v24.18.0 (includes npm and npx) for automated deployment.
An S3 bucket for deployment assets.
An active CloudTrail configuration where you store logs in an S3 bucket (required for Part 2).
Part 1: Deploy a self-hosted dashboard (Capability Insights for AWS)
Capability Insights for AWS deploys a searchable Regional availability dashboard into your own AWS account. The solution pulls data from the AWS Capabilities by Region S3 bucket, stores it inside your VPC, and serves it through a static website backed by Amazon API Gateway, AWS Lambda, and Amazon EventBridge.
If your organization has access to additional data sources beyond the public dataset, the solution incorporates those as well, giving you a unified view across all AWS partitions you have access to. To learn more about accessing additional data, work with your AWS representative.
The dashboard covers:
Services and features: availability status, expected launch dates, and expansion plans per Region.
API operations: individual API action availability per Region for each AWS service.
CloudFormation resource types: which resource types each Region supports.
You provide your own VPC, subnets, and S3 bucket so the solution integrates with your existing infrastructure and security controls.
Clone and install
git clone https://github.com/aws/capability-insights-for-aws.git
cd capability-insights-for-aws
npm install
Create a deployment assets bucket
Create an S3 bucket with public access blocked to store the Lambda code package during deployment. We recommend naming it capability-insights-assets-<ACCOUNT_ID>-<REGION>.
Deploy the stack
Automated deployment:
npm run deploy
The script builds all assets, prompts for parameters, deploys the CloudFormation stack, uploads the website, and triggers an initial data sync. You will be prompted for SourceFolders, a comma-separated list of data sources to pull from. The default is public.
To skip interactive prompts, pass all parameters as flags:
VPC ID with DNS resolution and DNS hostnames enabled
–backend-subnet-id
Private subnet with no internet route. Needs a route to Amazon S3 through a gateway VPC endpoint.
–api-access-subnet-id
Public subnet routed to an internet gateway (user access, API Gateway VPC endpoint)
–deployment-assets-bucket-name
S3 bucket for deployment assets
–source-access-point-arn
Public Capabilities by Region S3 access point ARN published by AWS. Account ID 686591367145 identifies the account managed by AWS that hosts the public dataset. Use the ARN as shown.
EC2 instance with SOCKS proxy: SSH into an instance in the VPC and proxy browser traffic through it.
Explore the dashboard
Once connected, the dashboard provides:
Search and filter: find services by name across all Regions.
Expandable service details: select any service to see individual feature availability per Region.
Status indicators: Available, Planning, Not Expanding, and projected dates (for example, “2026 Q3”)
Export: download the current view as JSON or CSV for sharing or further analysis.
Settings: view last sync time and trigger manual data refreshes.
Figure 2 shows the Capability Insights for AWS dashboard and its controls for comparing service availability across Regions.
Figure 2: Dashboard overview showing service availability across Regions with status indicators and export options.
Summary counts show coverage for services and features, API operations, CloudFormation resources, and Regions. Tabs switch between catalog views, while filtering, Region columns, status values, and the expand and export controls help you inspect and download availability data.
At this point you have a working dashboard inside your VPC, refreshing daily, with no external dependencies during normal operation.
Part 2 adds personalization by filtering this catalog to the specific services your account runs. You can view the workload analysis results through the dashboard or access them through dedicated API endpoints.
Part 2: Personalize the catalog with Workload Analysis
The catalog covers 200+ services across 35+ Regions. When you plan expansion to a new Region, you don’t need to evaluate all of them. You need to evaluate the 20–30 your account runs. Without that filter, a gap analysis is a multi-week project. With it, it’s a 30-minute review.
Workload Analysis deploys as an additive CloudFormation stack alongside the Capability Insights stack. The pipeline runs as an AWS Step Functions state machine with three stages:
Parallel analyzers (two branches run in parallel):
CloudTrail Analyzer: Creates an AWS Glue Data Catalog table pointing at your CloudTrail log bucket, then runs an Amazon Athena query that extracts distinct service/API/Region/account combinations from the last N days (configurable, default 30). This identifies services your account has called.
CloudFormation Analyzer: Calls ListStacks and GetTemplate for every active stack in the account. Extracts AWS:: resource types and scalar property values (strings, numbers, booleans), maps them to service names, and records which stack contributed each resource. This identifies what your account deploys, not only what it calls.
Usage Decorator runs after both analyzers complete. It reads the primary capability catalogs (products.json, apis.json, cfn_resources.json) from the website bucket, intersects them with the analyzer outputs, and writes personalized files back to the bucket for the dashboard to consume.
Scheduled execution through an Amazon EventBridge rule triggers the state machine daily (configurable schedule expression), keeping personalized data fresh without manual intervention.
Figure 3 shows how the Workload Analysis components coordinate scheduled and on-demand analysis.
An Amazon EventBridge schedule or an on-demand API request starts the AWS Step Functions state machine. The workflow runs two branches in parallel: the CloudTrail analyzer uses Amazon Athena with AWS Glue Data Catalog and AWS Lake Formation catalog access to query activity, while the CloudFormation analyzer inspects active stacks. After both branches complete, the Usage Decorator combines their results with the primary capability catalogs and writes personalized data to the Capability Insights website bucket for the dashboard and API to consume.
Deploy Workload Analysis
Verify that CloudTrail is active and your logs land in an S3 bucket. Then deploy with the usage analysis flag:
This deploys both the core Insights stack and the Usage Analysis stack, then wires them together so the API Lambda can trigger analysis and serve personalized results.
Additional flag
Description
--enable-usage-analysis
Enables the Workload Analysis pipeline
--cloudtrail-bucket
S3 bucket containing your CloudTrail logs
A typical first run completes in 2–5 minutes depending on account size and the number of active CloudFormation stacks.
Retrieve API endpoint
Use the following command to retrieve the API base URL from the API configuration file on the dashboard bucket:
Use apiBaseUrl as the base URL for the dedicated API endpoints. You must run API requests from within the VPC because the API Gateway endpoint is private.
Run an analysis
Start an analysis by calling POST /analysis through the API Gateway endpoint:
The Amazon EventBridge rule triggers subsequent runs daily. You can also trigger ad-hoc runs through the dashboard’s Settings page.
View personalized results
After an analysis completes, the dashboard toggles between the full catalog and your personalized My Stuff view, showing only services and resources your account uses.
Through the API:
GET /capabilities?usageFilter=combined&scope=account
The response contains filtered products, APIs, and CloudFormation resources with usage attribution, including which stacks deploy each resource type and which property configurations you use:
Consider an account running a typical web application. Without Workload Analysis, the dashboard shows all 200+ services across 35 Regions. With the combined filter applied:
Before: 200+ services, thousands of features, hundreds of CloudFormation resource types.
After: 28 services your account uses, with per-stack attribution showing which CloudFormation stacks deploy each resource type.
The CloudFormation resource view goes deeper. For each resource type your stacks deploy, you can see which property configurations are in use and which stacks contributed them. For example, your AWS::EC2::Instance resources might show InstanceType: t3.medium from your application stack and InstanceType: m5.xlarge from your data processing stack, each traced back to its source.
This targeted view means your Regional expansion gap analysis focuses on the 28 services you care about rather than the full catalog. For example, if you plan to deploy into the Europe (Zurich) Region (eu-central-2), the dashboard shows which 28 services are available there, which have planned launch dates, and which have no roadmap entry yet. These results highlight service-availability gaps to consider during Regional expansion. From here, you can evaluate your options: wait for planned launches, architect around unavailable features, or evaluate a different target Region. You’re not evaluating 200+ services against a Region matrix. You’re scanning a personalized list where every row is something your stacks deploy.
Clean up
To remove the resources created in this walkthrough:
Workload Analysis (if deployed): Delete the Usage Analysis stack first:
Optionally delete the deployment assets bucket you created during setup. This solution uses standard AWS service pricing for Lambda, S3, API Gateway, Athena, and Step Functions. There is no additional charge for the solution itself.
Automated cleanup: Run npm run teardown to remove both stacks and empty the website bucket in one step.
Conclusion
Knowing which services are available in your target Region is the first step in designing for resilience. You can’t build a multi-Region recovery strategy for services that aren’t there yet. With Capability Insights for AWS and Workload Analysis, you own that data inside your VPC, filtered to what your account runs, refreshing on your schedule.
Amazon S3 Tables now support all data types in the Apache Iceberg V3 specification. You can create V3 tables or upgrade existing V2 tables to take advantage of V3 features like deletion vectors, row lineage, and new data types such as variant, nanosecond timestamps, unknown, geometry, and geography.
Apache Iceberg has become the open standard for managing large analytics datasets. It lets you manage petabyte-scale tables with features like schema evolution, hidden partitioning, and time travel queries, while keeping your data in open Parquet files in data lakes on object storage like Amazon S3. Amazon S3 Tables offer storage purpose-built to keep Iceberg tables performant and cost-effective as they grow, with fully managed features like automatic compaction, maintenance, replication, and Intelligent-Tiering.
Teams running analytics on Apache Iceberg V2 tables often hit the same limits as their data grows. A compliance request to delete 50,000 user records from a 2-billion-row table leaves behind positional delete files that slow queries until compaction runs. Semi-structured events land as JSON strings that every query has to parse. Geospatial coordinates and nanosecond-precision timestamps get encoded as strings or integers. Each workaround adds storage cost, query latency, and pipeline code. With V3, Iceberg solves these challenges by offering native support for semi-structured and geospatial data, faster row-level operations, and built-in row lineage for data governance.
Starting today, Amazon S3 Tables support all V3 data types, including variant, nanosecond timestamps, geometry, geography, and unknown, along with deletion vectors and row lineage. You can create new V3 tables or upgrade existing V2 tables in place, and S3 Tables continue to run compaction and maintenance for you.
Apache Iceberg V3
V3 is the latest version of the Iceberg specification. Among its many improvements, V3 introduces capabilities that address the most common pain points in V2. This includes:
Deletion vectors replace V2’s positional delete files with a compact binary format. That 50,000-row compliance delete now writes a single deletion vector file instead of thousands of small deletes, significantly reducing compaction time and delete file overhead.
Row lineage adds _row_id and _last_updated_sequence_number to each record automatically. Your downstream pipelines can query these fields to find changed rows without scanning the full table.
New data types let you store semi-structured, geospatial, and nanosecond-precision data natively instead of encoding it as strings or integers:
Nanosecond timestamp(tz) for nanosecond-precision timestamps
Geometry and geography for geospatial data
Unknown for columns with no known type
Variant data type stores semi-structured data in columnar format. During writes, the engine shreds variant data into hidden columns and collects statistics. At query time, those statistics enable file pruning that significantly reduces I/O compared to parsing JSON strings.
The following sections walk through how to use these V3 capabilities in practice, with examples that show how to create tables, work with the new data types, and manage data at scale.
Getting started
A retail analytics team tracks user behavior across web and mobile apps. Each event has a different structure: page views include URLs and duration, purchases include items and amounts, and searches include query terms and result counts. With V3’s variant type, you store all event shapes in one table without predefined schemas:
Now query the variant column directly, without PARSE_JSON at read time. With Amazon EMR Spark, use variant_get:
SELECT
event_id,
user_id,
variant_get(payload, '$.action', 'string') AS action,
variant_get(payload, '$.amount', 'double') AS amount
FROM my_catalog.namespace.clickstream
WHERE variant_get(payload, '$.action', 'string') = 'purchase'
AND variant_get(payload, '$.amount', 'double') > 50.00
To enable deletion vectors for write operations, configure merge-on-read mode:
ALTER TABLE my_catalog.namespace.clickstream
SET TBLPROPERTIES (
'write.delete.mode' = 'merge-on-read',
'write.update.mode' = 'merge-on-read',
'write.merge.mode' = 'merge-on-read'
)
Now when you run a compliance delete, V3 writes a small deletion vector instead of rewriting data files:
DELETE FROM my_catalog.namespace.clickstream
WHERE user_id = 'user-42'
S3 Tables compaction handles these deletion vector files automatically on the next maintenance cycle.
Upgrading from V2
AWS provides backwards compatibility for both versions to minimize disruption during migration to V3. Existing V2 readers continue to work on upgraded tables until you’re ready to fully adopt V3 features. For more details, see the S3 Tables Iceberg V3 documentation.
Upgrade an existing table atomically without rewriting data:
ALTER TABLE my_catalog.namespace.existing_table
SET TBLPROPERTIES ('format-version' = '3')
On the next compaction cycle, S3 Tables remove old V2 delete files. New modifications use deletion vectors automatically. Row lineage fields initialize on the first data modification after the upgrade.
This is a one-way operation. The Apache Iceberg specification does not support downgrading from V3 to V2. Verify that all engines accessing the table support V3 before upgrading.
Using row lineage for incremental pipelines
After your table has V3 data, use row lineage to build efficient incremental pipelines:
SELECT *, _row_id, _last_updated_sequence_number
FROM my_catalog.namespace.clickstream
WHERE _last_updated_sequence_number > 42
This returns only rows modified after sequence number 42. Your downstream jobs can checkpoint this value and process only new changes on each run, instead of scanning the full table.
Compatibility across AWS analytics services
AWS offers the broadest native Apache Iceberg support of any major cloud provider, with Iceberg-compatible services at every layer of the data stack: ingestion, storage, catalog, and analytics. You can store and automatically optimize V3 tables in Amazon S3 Tables, write data with Amazon EMR Spark, integrate and manage data with AWS Glue, and run analytics with Amazon Redshift. To learn more about AWS analytics support for V3, see the Apache Iceberg on AWS prescriptive guidance.
Both S3 Tables and AWS Glue Data Catalog support the Iceberg REST Catalog (IRC) API, enabling interoperability across engines regardless of the catalog endpoint.
The new V3 data types (variant, nanosecond timestamps, geometry, geography, and unknown) require an engine built on Apache Spark 4.0 or later, such as AWS Glue 6.0 or later, or Amazon EMR release 8.1 or later.
You can create V3 tables from the Amazon S3 console, AWS CLI, or any engine that supports the Iceberg REST Catalog API.
The new V3 data types are supported only for tables that use the Parquet file format (not ORC or Avro).
Columns of type variant, geometry, geography, or nanosecond timestamp can’t be included in a table’s sort order for compaction. Tables containing these columns still compact under the sort and Z-order strategies when the sort order uses columns of other types.
Now available
Amazon S3 Tables support for all Apache Iceberg V3 data types is now available in all AWS Regions where S3 Tables are supported. Apache Iceberg V3 support is available at no additional charge; standard S3 Tables pricing applies.
To get started, visit the Amazon S3 Tables documentation or create a table bucket from the Amazon S3 console. If you want to call APIs, search documentation, find regional availability, and check troubleshooting about this feature, try using the AWS MCP Server and plugins with your preferred AI tool. Send feedback to AWS re:Post or through your usual AWS Support contacts.
Today, we’re announcing metadata pre-filtering for Amazon S3 Vectors, which delivers higher recall on filtered queries by evaluating your metadata filter before the similarity search. You can filter on attributes such as tenant, category, status, or time, and pre-filtering adds prefix matching with $startsWith for paths, URLs, and hierarchical keys. Each vector carries up to 2 KB of filterable metadata, and a single query supports up to 100 filter constraints. There is no additional cost, no re-ingestion, and no change to your queries.
Most applications never search a whole index. They search the part of it that belongs to a particular user, account, or category, and they express that scope as a metadata filter. Semantic search, retrieval-augmented generation (RAG), and agentic applications all need the same thing from a filtered query: a similarity search that covers the vectors matching the filter, and returns the closest of them. With pre-filtering, a filtered query returns more of the relevant matches your index contains, giving you higher recall on filtered searches.
Common use cases
Pre-filtering applies wherever results have to be both relevant and correctly scoped:
Legal and professional services: A law firm or e-discovery platform searches documents scoped to a single client, and with $startsWith narrows further by matter number, folder path, or document ID prefix. A single client is a small share of a firm-wide archive, and filters this narrow are where pre-filtering improves recall most.
Financial services: An investment research platform searches analyst notes, filings, and call transcripts scoped by issuer, document type, and publication date.
Media and entertainment: A streaming service filters by content rating and regional licensing before the semantic search, finding similar titles restricted to G and PG content licensed in one territory.
Agentic applications: An agent working within a user’s session filters on fields such as owner, document set, and timestamp so its searches cover the material relevant to the task at hand. Higher recall means more of that material reaches the agent, which improves task reliability
How pre-filtering works
Each vector in an S3 Vectors index can carry application-defined metadata, and a query can filter on those fields.
Every vector index has an index mode. On an index whose index mode is ENHANCED, S3 Vectors resolves your filter first, then searches only the vectors that match. On an index whose index mode is CLASSIC, S3 Vectors performs the vector search and filter evaluation in tandem, validating each candidate vector against your filter as it searches. Existing indexes use CLASSIC until you update them.
Consider a support knowledge base of 8 million tickets, where an agent searches one customer’s history for a recurring error. If that customer accounts for 400 of those tickets, resolving customer_id first means the similarity search runs across all 400 of them, so the agent sees that customer’s prior occurrences. Before the index was updated, the same query drew its candidates from the full 8 million, and the result set contained fewer of that customer’s matching tickets.
On highly selective filters, pre-filtering returns up to 5x more of the matching vectors than the same query returned before on CLASSIC indexes.
Getting started
Before you start, make sure your IAM policy grants permissions for the new actions.
You can get started in three steps. The walkthrough below builds a small product-catalog index and runs a selective filter against it, the same pattern you would use for a multi-tenant RAG store or a document search scoped to one client.
The dimension must match the output size of your embedding model, and distance-metric should match how that model was trained (cosine is common for text embeddings). Second, write vectors with the PutVectors API, attaching up to 2 KB of filterable metadata to each vector:
Each vector carries the attributes your application filters on. In this example, tenant_id scopes results to a single customer, category narrows by document type, created_date records when the document was created, and active is a boolean flag. By default every metadata field is filterable, so you can query on any of them without declaring a schema up front.
Third, run a filtered similarity query with the QueryVectors API. The filter uses a compact JSON syntax where a bare key-value pair is an equality match, and operators such as $and, $or, and $gt combine or refine conditions. Pass --return-metadata so the query returns each vector’s metadata:
S3 Vectors first narrows the search space to vectors matching all three filter conditions, then returns the 50 most similar vectors from that subset. Because the filter is applied before the search, those results are drawn from across all the vectors that match it.
Prefix matching with $startsWith
Pre-filtering adds a prefix match operator for filtering on paths, URLs, and hierarchical keys. A document store that encodes case and folder structure into a document ID can scope a search to a subtree in one condition:
Pre-filtering takes effect in place. Your existing vectors are not re-ingested, your queries do not change, and the new filter operators are available immediately.
Here is the difference on the same index and the same query. Before the update, a query scoped to one tenant returns two of the ten results requested:
Once you have validated pre-filtering on an index, set the default index mode on the vector bucket so that new indexes use ENHANCED without a follow-up call:
To bring the rest of your existing indexes across, list them and check the index mode on each one, then call UpdateIndexMode on the ones still using CLASSIC:
Indexes created in vector buckets created on or after September 30, 2026 use index mode ENHANCED. Indexes in buckets that existed before that date use CLASSIC until you set the bucket default, including indexes created in those buckets afterward.
A single query supports up to 100 filter constraints, counted per value the filter evaluates. If a query exceeds that, you can usually consolidate the filter, replacing a 300-value $in over legal cases with a single caseId field, for example, or split it into smaller queries, run them in parallel, and merge the results by distance.
Get started today
Metadata pre-filtering is available at no additional cost in all commercial AWS Regions where Amazon S3 Vectors is available, and in the AWS China Regions. You pay standard S3 Vectors pricing for storage, PUT requests, and queries. For full pricing details, visit the Amazon S3 pricing page. For regional availability, visit Amazon S3 Vectors Regions and quotas.
Whether you’re scoping a RAG application to one tenant, scoping an agent’s searches to one user’s documents, or narrowing a catalog search to a licensing window, pre-filtering lets you apply those filters without trading away recall. To learn more and get started, visit the Amazon S3 Vectors documentation. Send feedback to AWS re:Post for S3 or through your usual AWS Support contacts.
Today, we’re announcing a new capability for Amazon Aurora PostgreSQL that you can use to directly query operational data together with data stored in your data lake in Apache Iceberg and Apache Parquet formats, using your existing PostgreSQL applications and tools. By eliminating the need to extract, transform, and load (ETL) structured data from data lakes into your operational database, you can reduce operational complexity and simplify application development. You can also use Aurora PostgreSQL to query data from data lakes managed in Iceberg REST Catalog (IRC)-compatible catalogs, giving you access to data across a breadth of analytics systems without moving or duplicating it. Whether you’re powering real-time dashboards, enriching transactions with historical context, or building AI agents that reason over both live and archived data, you can now do it all through a single, familiar interface.
Previously, if your application needed to combine recent transactional data in Aurora with historical records stored in Amazon S3, a common approach was to build reverse ETL pipelines that duplicated data, increased infrastructure costs, and required ongoing engineering effort to keep everything synchronized. This challenge only grows as you increasingly embed AI agents into your applications, where it is impractical to predict and pre-replicate every dataset an agent might need.
DuckLabs, the team that maintains the DuckDB project, recently joined Amazon, and this capability is an example of how the efficiency of DuckDB is being integrated into our services. DuckDB is now embedded directly within Aurora PostgreSQL, so you can query live operational data (including uncommitted writes) alongside your data lake in a single query. Query processing stays within Aurora, with no additional network hops and no ETL pipelines that duplicate data. You can query Apache Iceberg tables managed through the AWS Glue Data Catalog, as well as Parquet and Iceberg data stored in Amazon S3 and S3 Tables. You do all of this using familiar PostgreSQL syntax and your existing applications and tools.
We’re excited to bring the speed and simplicity of DuckDB directly into Aurora PostgreSQL, so you and your agents can query and combine operational and Iceberg data using the familiar PostgreSQL applications, tools, and endpoints already in use. By building this capability around DuckDB, future improvements to the open source engine can continue to bring performance and functionality gains to Aurora and other AWS services.
What is new
This capability is supported on two Aurora PostgreSQL major versions: 17 (starting with 17.11) and 18 (starting with 18.6). To use it, you create an Aurora PostgreSQL cluster, attach an IAM role with the AuroraAnalytics feature, and enable the aurora_analytics extension. The IAM role is what gives Aurora access to your data in Amazon S3 and the AWS Glue Data Catalog. You then create foreign tables that point to your Iceberg or Parquet data in the data lake, and query them using familiar PostgreSQL syntax. You can complete this setup through the Amazon RDS console, or with any PostgreSQL client such as psql. The process is well documented in the Aurora PostgreSQL documentation.
You can query data across external IRC-compatible catalogs through AWS Glue Data Catalog federation. You register the external catalog once with Glue, and then create foreign tables for the tables you want to query, the same way you would for any Glue-native table. A single query can then join data stored in Aurora with Iceberg tables registered across multiple catalogs, so applications get a unified view without moving data or replacing your existing catalog investments.
Aurora also applies optimizations such as predicate pushdown and column pruning so that only the relevant data is read. This keeps queries efficient even as the underlying data grows. Frequently accessed data is also cached in your Aurora instance, so subsequent queries against the same data return faster. You can inspect this behavior per query using aurora_analytics_stat_statements(), which reports metrics such as rows scanned, bytes read from Amazon S3, and cache hits.
To see how direct querying works, I connected to my Aurora PostgreSQL database using psql and created the extension:
CREATE EXTENSION aurora_analytics;
For my walkthrough, I set up a simple financial scenario. I have a recent_transactions table in Aurora with the last 7 days of customer transactions, and a Parquet file in Amazon S3 containing 5 years of historical transaction data. To make Aurora aware of the historical data, I created a foreign table pointing at the Parquet file in S3:
CREATE FOREIGN TABLE transaction_history ()
SERVER aurora_analytics_server
OPTIONS (
location 's3://<my-bucket>/finance/transaction_history.parquet',
format 'parquet'
);
Notice the empty parentheses in the CREATE FOREIGN TABLE statement. Aurora automatically reads the schema from the Parquet file metadata, so you do not need to define columns manually. For workloads with many tables, you can skip creating them one at a time: a single IMPORT FOREIGN SCHEMA statement bulk-creates foreign tables for every Iceberg or Parquet table in an AWS Glue Data Catalog database, inferring schemas automatically.
With both tables in place, I ran a single query that combines the recent operational data in Aurora with the historical data in S3:
SELECT merchant, category, amount, transaction_date, 'recent' AS source
FROM recent_transactions
WHERE customer_id = 'C-1001'
UNION ALL
SELECT merchant, category, amount, transaction_date, 'historical' AS source
FROM transaction_history
WHERE customer_id = 'C-1001'
AND transaction_date >= CURRENT_DATE - INTERVAL '5 years'
ORDER BY transaction_date DESC
LIMIT 15;
The result shows both recent and historical transactions in a single result set. The 7 most recent rows come from Aurora, and the rest come directly from the Parquet file in S3. DuckDB handles the analytical scan of the Parquet data under the hood, while Aurora handles the operational data. That single query would have previously required a pipeline to move the historical data into the database first.
If a query pattern needs single-digit-millisecond latency, you can materialize data from the data lake into a native Aurora PostgreSQL table using familiar commands such as CREATE TABLE AS SELECT, INSERT INTO ... SELECT, or MERGE INTO. The materialized table lives in Aurora and is queried like any other PostgreSQL table, giving you a low-latency path for hot data without operating a separate ingestion pipeline. The read queries can run on any Aurora PostgreSQL instance in your cluster, whether the writer or a read replica, so you can offload analytical scans from your operational workload. The materialization commands write data into Aurora, so they run on the writer instance.
Get started today
Direct querying of Apache Iceberg and Parquet data from Amazon Aurora PostgreSQL is available today in all commercial AWS Regions and AWS GovCloud (US) Regions, at no additional charge. You pay only for the incremental Aurora compute the queries consume and Amazon S3 request costs for reading data lake files.
This is a guest post by Denis Bauer, Yatish Jain, Anuradha Wickramarachchi, Brendan Hosking, and Nick Edwards of CSIRO, in collaboration with the ASP Prototyping and Scaling Team at AWS.
In this post, we describe how researchers at CSIRO, Australia’s national science agency, built Serverless Beacon (sBeacon), a scalable serverless solution for securely querying genomic variant data on AWS, underpinning production-scale clinical and research applications.
The Beacon protocol is the widely adopted standard for exchanging genomic and phenotypic data developed by the Global Alliance for Genomics and Health (GA4GH). It uses an API to define how data is shared, with the goal of enabling efficient and secure data discovery across international research and clinical networks.
sBeacon is a production-ready implementation of this standard, built using AWS services: Amazon Simple Storage Service (Amazon S3), AWS Lambda, Amazon DynamoDB, and Amazon Athena. By using these foundational AWS serverless services, sBeacon is able to provide the following benefits to researchers and clinicians needing to perform genomic variant querying:
Highly scalable for large cohorts: sBeacon can scale to support hundreds of millions of individuals (and billions of genomic locations), which makes it suitable even for mega-biobank-scale datasets.
Low cost to run: Because it uses a serverless, cloud-native architecture, sBeacon can operate for approximately USD 0.40 per month for a 1000 Genomes-scale dataset. The following case study breaks down ingestion, query, and storage costs in detail.
High performance and fast query response: Real-world queries return in seconds (about 5 seconds) because of the serverless compute and efficient architecture, for near real-time data lookups.
No heavy data ingestion or transformation needed: sBeacon can directly consume standard VCF files (a common format for genomic variant data), which reduces the need to load data into databases or transform it to different data structures.
Rapid onboarding of new data: Genomic data generation is accelerating because it underpins clinical diagnosis and treatment, and because its complexity demands ever-larger cohorts to study complex traits. As a result, both clinical services and research cohorts must continuously onboard new data, and Beacon supports real-time generation-to-use life cycles (about 18 seconds).
Improved privacy, data ownership, and decentralization: Because sBeacon doesn’t require central databases and supports federated networks, data stays under the control of original holders, which can help data custodians address privacy and ethical considerations in sensitive genomic and medical data sharing.
Lower barrier to entry for broader participation: Its affordability, simplicity, and small operational footprint can help make it more accessible for smaller or resource-limited institutions and countries, which can increase participation from underrepresented populations and improve data diversity.
Zero trust model: sBeacon enforces explicit authentication, least-privilege data access, ephemeral compute isolation, and strict cloud-native boundary controls that help confirm no component, user, or request is implicitly trusted.
Prerequisites
sBeacon is deployed as a container that sets up the necessary development environment, with Terraform defining the resources for the deployment. To get started, clone the terraform-aws-serverless-beacon repository on the GitHub website.
Make sure that your development environment contains Docker and has the necessary permissions for you to use it without super user access. Press Ctrl+Shift+P (Cmd+Shift+P on macOS) to open the command palette in VS Code, and then choose Reopen in Container. This opens the workspace in the container environment that we have defined.
Now, run the following command to initialize the necessary libraries and Lambda layers.
bash init.sh
Next, run the following command to initialize the Terraform environment.
terraform init
Optionally, you can define a backend by following the instructions in the repository. After the preceding command runs successfully, you can run the deployment command.
terraform apply
Enter yes when prompted to proceed with the deployment. After the deployment is complete, you receive information such as the API URL and the command to sign in as the admin or guest user. To shut down the entire service, run terraform destroy. Any created datasets are lost (but not the VCFs on which they are based).
Solution walkthrough
CSIRO developed sBeacon for sharing and querying genomic and medical data. sBeacon uses AWS serverless technology for the elastic scaling of compute resources.
The architecture of sBeacon performs two broad processes:
Data onboarding: the ingestion and indexing of genomic metadata into sBeacon.
Data querying: the querying of the genomic metadata by end users.
Data onboarding
During the onboarding process, you define where the genomic data and the metadata (such as disease status, age, and location) is located. Note that genomic data is not copied out of its original location but rather is referenced when needed. In contrast, metadata is loaded to sBeacon’s storage mechanisms because it is necessary to perform indexing that allows efficient querying. The user will need to ensure no sensitive or privacy-revealing data is disclosed. The example details the approach using CSIRO’s Ontoserver. However, sBeacon supports the API schema of the Ensembl OLS V4 specification.
Figure 1. Data onboarding.
The data onboarding process is summarized by the following steps:
The onboarding starts with the user submitting the location of the genomic data as request payloads to an API Gateway endpoint.
The request payloads are forwarded to an AWS Lambda function that handles the data indexing.
The metadata is written to an Amazon S3 bucket in the ORC format, to allow future querying and processing by Athena.
An AWS Lambda function is called to orchestrate the indexing process.
The CSIRO Ontoserver is called to build the ontology index for advanced metadata queries.
The resulting index files are written to Amazon S3.
CREATE TABLE AS SELECT (CTAS) queries are run on Amazon Athena to build the metadata tables.
Athena loads the metadata from Amazon S3 into the metadata tables.
The metadata tables are written back to Amazon S3 in ORC format.
Data querying
Querying in sBeacon is flexible, catering to a wide range of applications from human genetic disease to pathogen queries. We achieved this by designing the query architecture modularly. This approach let us separate the querying logic into several Lambda functions based on their querying scope, while maintaining a similar architecture.
The following architecture diagram describes the workflow for metadata querying, which uses the Variant Querying Module described later in this section.
Figure 2. Data querying.
The user submits their query to the API Gateway endpoint.
API Gateway calls the Microservice Lambda function.
The Microservice Lambda function looks up the relevant query ontology terms in an Amazon DynamoDB table.
The matching ontology descendent terms (and their codes) are returned to the Microservice Lambda function. The descendent terms are those that match a hierarchical descendent of each term, or each term itself, from the query.
Using the ontology codes from step 3, the metadata tables on Athena are queried.
The metadata associated with the query is returned from Athena.
If required by the query, the Microservice Lambda function queries the Variant Querying Module.
The variant data associated with the genomic conditions in the query is returned to the Microservice Lambda function.
The result is formatted according to the Beacon protocol and is returned to the user through Amazon API Gateway.
The response is received by the user.
Figure 3. Variant Querying Module.
Genomic variant queries are performed using the Variant Querying Module. The workflow of this module is as follows:
The Microservice Lambda function calls an Initiator Lambda function.
The Initiator Lambda function fans out the splitQuery Lambda function across the VCF files.
The performQuery Lambda function is then fanned out across the VCF regions in each of the files involved in the query.
The performQuery Lambda function fetches the VCF files from Amazon S3.
The query results are synchronously returned to the parent Initiator Lambda function.
If requested by the user, metadata can optionally be queried, where the Initiator Lambda function queries the metadata from Athena.
Athena queries the metadata from Amazon S3 (through an external table).
The metadata results are returned to Athena.
The Initiator Lambda function receives the metadata from Athena.
All the query results, including any optional metadata, are returned to the calling Microservice Lambda function.
Case study: 1000 Genomes dataset
We demonstrate sBeacon on chromosome 1 of the 1000 Genomes Project to report how it handles large-scale variant queries. We measure ingestion efficiency, query scalability, and cost for typical population-scale analyses, such as identifying SNP variants across defined genomic regions. The case study uses chromosome 1 (chr1, 8% of the genome) from the 1000 Genomes Project, which contains 2504 samples. This multi-sample VCF is approximately 1.1 GB compressed, with data stored in Amazon S3. Note that sBeacon can also process cohorts of single-sample VCF files. All costs in this section are for the Asia Pacific (Sydney) Region (ap-southeast-2), exclude applicable taxes, and reflect pricing at the time of writing.
sBeacon can ingest chromosome 1 from the 2504 individuals in 18 seconds, for less than 1 cent (USD 0.00052). This is because sBeacon does not copy the large genomic information but instead creates index files that enable random access. Cost is therefore driven predominantly by storing the copied metadata. After ingestion, sBeacon can be maintained for USD 0.000025 per month (1 MB of compressed metadata stored for 2504 samples in ORC format, plus genomic index files). If you store the genomic data as well, this would be USD 0.032 for chr1 (at USD 0.025 per GB in ap-southeast-2) or about USD 0.425 for the whole genome.
Query time is similarly near real time. For example, querying across a region of 10,000 base pairs to determine the genotypes in this region takes 1.52 seconds across the 2504 individuals. This would serve a query such as “Fetch all individuals with a specific BRCA1 mutation who have stage 3 cancer.” The cost for such a query is USD 0.00013. Note how the query time stays constant even with an increasing number of variants returned (for example, from 4 to 400).
Table 1. Query example costing and times (whole chromosome 1).
Query region size (bases)
Number of variants found
Average Time
Compute Cost (per query in USD)
10
4
1.51 s (+- 0.26)
0.00013
100
18
1.52 s (+- 0.25)
0.00013
1,000
84
1.62 s (+- 0.24)
0.00014
5,000
229
1.65 s (+- 0.29)
0.00014
10,000
400
1.52 s (+- 0.11)
0.00013
Table 2. Cost for ingestion, querying, and idling (whole chromosome 1 for 2504 genomes with less than 10 MB of metadata).
Scenario
Metric
Cost (USD) per month
Ingestion Cost
per 1000 ingestions
0.53 (32.82 GB seconds of Lambda)
Query compute cost
per 1000 queries
0.28 (9.8 GB seconds of Lambda)
Query Athena Cost
per 1000 queries
0.05
Idle Cost (Storage Cost)
1.1 GB
0.03
Query DynamoDB Cost
Per 1000 queries
0.0005
Security features
Security and compliance is a shared responsibility between AWS and the customer. AWS is responsible for protecting the infrastructure that runs the AWS services described in this post, and you are responsible for your use of those services, including how you configure them, which identities you grant access to, and which data you choose to onboard. Consider the services you choose carefully, because your responsibilities vary depending on the services used, how you integrate those services into your IT environment, and applicable laws and regulations. For more information, see the AWS Shared Responsibility Model.
Zero trust model
Explicit authentication and authorization – Every API request must carry a valid JWT issued by the Amazon Cognito user pool (aws_api_gateway_authorizer.BeaconUserPool-authorizer, type COGNITO_USER_POOLS). The authorizer runs at API Gateway before any Lambda function is invoked, so requests do not reach a handler without Cognito validation. Token validation includes signature, expiry, and audience (Cognito app client ID). You can disable authentication during the first deployment with BEACON_ENABLE_AUTH = false for intentionally public or open beacons. This is an explicit operator decision, not a default.
Authorization (what a valid user can do) is enforced inside the Lambda layer, not in Amazon API Gateway:
Group membership (sbeacon-record-access-user-group, and so on) controls the maximum granularity returned.
Admin-only operations (dataset submission, deletion) check for sbeacon-admin-group membership before proceeding.
Least-privilege data access – sBeacon implements role-based access control (RBAC) through Cognito groups that map directly to disclosure tiers. You assign each user one or more of the following:
Cognito group
Maximum disclosure
sbeacon-boolean-access-user-group
exists: true/false only
sbeacon-count-access-user-group
aggregate counts
sbeacon-record-access-user-group
full variant details and sample names
sbeacon-admin-group
preceding tiers plus dataset management
The JWT carries the user’s group memberships as claims. The query Lambda function reads these claims to determine requested_granularity and include_details, then passes both flags to performQuery. performQuery computes only what was requested. A boolean-tier user’s request does not cause sample-level data to be computed or returned, even if it exists in the VCF.
Ephemeral compute isolation – Lambda execution environments are stateless by design. Each cold start is a fresh container, /tmp (1,024 MB for performQuery) is cleared between cold starts, and concurrent invocations run in separate sandboxes with no shared memory. The bcftools subprocess inside performQuery runs and exits within the Lambda function lifetime (10 second timeout). No state persists after invocation.
Cloud-native boundary controls – API Gateway is the public entry point in this architecture. Amazon S3 buckets, DynamoDB tables, Athena, and Amazon SNS topics have no public resource policies. Amazon S3 buckets are created with private ACLs and BucketOwnerPreferred ownership controls. Lambda functions run on AWS-managed VPCs with no inbound network access. Amazon SNS topics are account-private (no external principal grants).
Privacy and data ownership
Each institution deploys the entire Terraform stack into its own AWS account, so there is no shared infrastructure, no central data lake, and no cross-account trust. VCF files live in the deploying institution’s Amazon S3 bucket and do not leave it. performQuery passes the Amazon S3 URL directly to bcftools as a subprocess argument, which uses htslib HTTP byte-range requests to read only the tabix-indexed region of interest (about 1 KB per query). The raw genomic sequence bytes do not pass through Lambda memory as returnable data. What the query returns upstream (exists as a boolean, call_count as an integer, and variant representations) is aggregate result data, not source sequence.
Decentralization in sBeacon is achieved at the storage layer, not the compute layer. The _vcfLocations registered for a dataset are Amazon S3 URIs, and these can point to buckets owned by entirely different organizations. When a query runs, performQuery passes each URI directly to bcftools, and htslib issues HTTP byte-range requests (Range: bytes=X-Y) against the Amazon S3 REST API of whichever organization owns that bucket. The raw VCF bytes do not leave the source organization’s Amazon S3 bucket. Only the query result (exists, count, or variant record) is returned.
Data onboarding privacy
The submitDataset endpoint sits behind the same API Gateway Cognito authorizer as all other endpoints. An unauthenticated request receives a 401 response before reaching any Lambda function. Beyond authentication, the handler also checks that the caller is a member of sbeacon-admin-group. A valid token from a user in only record-access or count-access is rejected. This means the beacon operator explicitly controls the set of people who can introduce data into the system, so onboarding is not a self-service capability.
Further considerations
We chose AWS Lambda over AWS Step Functions in this architecture because it can process much larger payloads. Given the size and complexity of genomic data and the fan-in and fan-out architecture for parallel handling, AWS Lambda emerged as the lower-cost and more flexible approach for this workload.
As demonstrated in the sBeacon publication, the architecture can cater to population-scale datasets. However, if you accidentally attempt to run a range query of the entire genome, the architecture times out at the Amazon API Gateway level. Applying functional operations over the whole genome requires further architectural considerations.
Because a single fan-out query spawns many parallel Lambda invocations, you need to monitor concurrency consumption to confirm that burst queries do not exhaust the account’s concurrency pool and starve other functions. Tracking the ConcurrentExecutions metric at both the account and function level provides early visibility into capacity pressure.
Similarly, because synchronous Lambda invoke does not automatically retry on throttle, a 429 response from a performQuery invocation means the result is silently lost unless the application handles it explicitly. Setting Amazon CloudWatch alarms on the Throttles metric for performQuery allows you to take corrective action, such as requesting a concurrency limit increase, before throttles affect query accuracy. Alternatively, we have produced a separate architecture that sends alert email with diagnostic information when Lambda functions fail, available in the error-catcher repository on the GitHub website. You can implement this in the repository or set it up as a standalone service to catch Lambda errors thrown by sBeacon.
After idle periods, simultaneous performQuery invocations might encounter cold starts that add latency to query responses. Enabling provisioned concurrency on the query-path Lambda functions helps reduce this cold-start latency during burst fan-out scenarios at the price of increasing the idle cost.
Conclusion
In this post, we described how CSIRO built sBeacon, a fast, scalable, and low-cost way to run genomics workloads on AWS. sBeacon implements the GA4GH Beacon standard with a fully serverless and modular architecture. This publicly available solution supports near real-time querying of standard VCF data, scales to mega-biobank cohorts, minimizes ingestion effort, and supports privacy and zero-trust security. If you are considering genomics on AWS, you can deploy sBeacon on existing Amazon S3-hosted VCF data, integrate it with clinical or research workflows through the Beacon API, and progressively federate with other Beacons for secure, cross-institutional genomic data discovery. Set up sBeacon to query your genomic data and explore the possibilities of securely sharing insights with your collaborators. You can read more about sBeacon in our publication: Scalable genomic data exchange and analytics with sBeacon. The source code for sBeacon can be downloaded from our GitHub repository.
Misconfigured Amazon Simple Storage Service (Amazon S3) buckets can expose your data to unauthorized access. Without proactive review, S3 bucket policies or Access Control Lists (ACLs) configured with broad access may go unnoticed in your environment. In this post, you learn how to identify and fix over-permissioned S3 buckets across your AWS environment, along with best practice recommendations and automation opportunities to help you prevent security gaps. This post provides a workflow framework and methodology recommendations for your security team to adapt. The focus of this post is on the what and why rather than a prescriptive implementation. You will need to customize the approach based on your organization’s requirements and existing security tooling.
This solution is intended for security engineers, cloud architects, and DevOps teams managing single- or multiple-account AWS environments with Amazon S3 workloads that require access management.
Prerequisites
Before you begin, make sure you have the following in place:
For multi-account environments, AWS Organizations should already be configured.
Basic familiarity with IAM policies and Python will help you customize the solution to your needs.
Solution overview
This solution uses a five-phase workflow diagram to detect, remediate, and continuously monitor over-permissioned S3 buckets across your AWS accounts. The following workflow diagram illustrates the high-level end-to-end process for identifying and remediating over-permissioned S3 buckets across your Amazon Web Services (AWS) environment.
Setup and prerequisites – Configure AWS Organizations or multi-account access, designate a central security account, deploy AWS Config across all accounts, and enable AWS Security Hub with a central administrator.
Detection and identification – Deploy AWS Config rules (such as s3-bucket-public-read-prohibited and s3-bucket-public-write-prohibited) and run an audit Lambda function that scans each S3 bucket. The function checks three areas: Public Access Block configuration, bucket policy status, and bucket ACL grants. Buckets with issues are added to a risky buckets list. The function then generates a report in CSV and JSON format, uploads it to an output S3 bucket, and sends an SNS alert.
Remediation – Address findings using one or more approaches – Apply restrictive bucket policies to deny public read/write access and restrict access to specific IAM principals; deploy a remediation Lambda function to automatically update bucket policies and disable public access settings; or use CloudFormation StackSets to deploy standardized policies across multiple accounts.
Continuous monitoring – Schedule the audit Lambda function for recurring scans (daily or weekly) using Amazon EventBridge. Use EventBridge to detect policy changes, configure automated notifications for new violations, enable IAM Access Analyzer for S3 to identify external access, and run regular compliance scans.
Resource cleanup – Review and delete resources created during the audit that are no longer needed, including Lambda functions and IAM roles, EventBridge rules, SNS topics and subscriptions, audit output S3 buckets, AWS Config rules, and Security Hub (if enabled only for this audit).
Cost considerations
This section covers the AWS services used in this solution and their associated costs so you can estimate spend before deployment. The primary cost drivers are AWS Config and Security Hub, which scale with the number of accounts and resources you monitor. Lambda, Amazon EventBridge, Amazon SNS, and Amazon S3 typically add minimal costs for most environments. Start with a pilot in one or two accounts to validate costs before scaling.
AWS Config – Charges per configuration item recorded and per rule evaluation. Costs scale with the number of accounts and resources tracked.
Security Hub – Charges per account per AWS Region for security checks and finding ingestion.
Lambda – Charges per request and per GB-second of compute time.
EventBridge – Scheduled rules are free. Custom event bus usage might incur charges.
Amazon SNS – Charges per notification delivered.
Amazon S3 – Storage costs for audit report output files. Minimal for most environments.
Check the service pricing pages for current rates. Use the AWS Pricing Calculator to estimate costs for your specific environment before enabling services across all accounts. Consider starting with a pilot in one or two accounts to validate costs before scaling.
Detect and report over-permissioned buckets
This section walks you through setting up the audit environment, deploying the Lambda-based scanner, and generating reports of over-permissioned S3 buckets across your accounts. Follow these steps to identify over-permissioned S3 buckets in your multi-account environment, starting with preparing your environment for an Amazon S3 audit.
To set up the multi-account audit environment:
Set up AWS Organizations or multi-account access. Set up centralized management of your AWS accounts using AWS Organizations or configure cross-account IAM roles.
Choose a central security account. Choose one account as your security/audit account. This account will run the audit Lambda function and collect results from member accounts.
Create an Amazon SNS topic for alerts. Subscribe your security team to receive notifications when over-permissioned buckets are detected. Note the topic Amazon Resource Name (ARN) from the output—you will need it when creating the Lambda execution role (step 6) and the Lambda function (step 9). Confirm the email subscription before testing; Amazon SNS doesn’t deliver alerts until the subscription is confirmed. Learn more in the Amazon SNS Developer Guide.
(Optional): Create an S3 bucket for audit reports. If you plan to use Script v2 for historical reporting and trend analysis, create a dedicated bucket now. Skip this step if you only need real-time alerts using Script v1.
Plan cross-account IAM roles. The central security account needs permission to scan member accounts. Design cross-account roles that:
Grant minimum Amazon S3 read permissions (list buckets, read policies, ACLs, public access configurations).
Include an external ID condition to mitigate the confused deputy problem.
Note: The specific trust policy and permissions policy for your cross-account roles will depend on organizational requirements. Work with your IAM administrators to grant minimum necessary access for the audit function.
Create the Lambda execution role. Create an IAM role for your Lambda function with the permissions it needs to scan buckets, publish alerts, and write logs. Apply the principle of least privilege—grant only the minimum Amazon S3 read permissions required for the audit (such as, listing buckets, reading bucket policies, ACLs, and public access block configurations), Amazon SNS publish permission for the alert topic created in step 3, Amazon S3 write permission for the output bucket created in step 4 (Script v2), and Amazon CloudWatch Logs permissions. For multi-account scanning, also include sts:AssumeRolepermission for the cross-account role ARNs created in step 5. The AWS Lambda execution role documentation has instructions on creating and configuring execution roles.
To deploy the S3 audit solution Deploy the audit components
Enable AWS Config in member accounts. AWS Config provides compliance monitoring and can detect when S3 buckets are created or modified with public access settings. This will enable the Lambda-based audit to receive real-time detection between scheduled scans. The AWS Config Developer Guide has setup instructions. Deploy pre-defined AWS Config rules to identify overly permissive settings. These managed rules provide automated compliance checking. When AWS Config detects violations, it sends findings to Security Hub (configured in step 8) for centralized visibility alongside the Lambda audit results.
s3-bucket-public-read-prohibited
s3-bucket-public-write-prohibited
Create AWS Config rules for specific permission patterns. For the full list of available rules, see the AWS Config managed rules reference
Enable Security Hub for centralized visibility. Enable AWS Security Hub in member accounts and configure the central security account as the administrator. Security Hub aggregates findings from AWS Config rules (step 7), IAM Access Analyzer (enabled later), and can receive custom findings from your Lambda audit function, providing a single dashboard for Amazon S3 security issues across your organization. See the Security Hub User Guide for setup details.
Deploy the audit Lambda function. Deploy a Python Lambda function using the Boto3 library to list S3 buckets, check their policies, ACLs, and IAM permissions, and identify over-permissioned buckets. See the example scripts that follow.
Important: These code examples aren’t production ready. Adapt them to meet your organization’s requirements and test them in a non-production environment before deployment.
Choose your approach:
Script v1 – Best for immediate SNS alerts when issues are detected.
Script v2 – Best for historical reports, trend analysis using BI tools.
Both scripts – Best for different schedules and ongoing needs.
Audit Lambda function – Example script v1 (Scan and alert)
The following is an example of a Lambda function script for reference purposes. Review, adapt, and test before use in your environment, it scans all S3 buckets in the current account and checks for:
Public Access block configuration gaps
Bucket policies that allow public access
ACL grants to AllUsers
Note: Replace placeholder values with actual values before deployment:
<REGION>– Your AWS Region (for example, us-east-1)
<ACCOUNT_ID>– Your 12-digit AWS account ID
<TOPIC_NAME>– The name of your SNS topic created in step 3
import boto3
import json
def lambda_handler(event, context):
s3 = boto3.client('s3')
sns = boto3.client('sns')
risky_buckets = []
errors = []
try:
buckets = s3.list_buckets()['Buckets']
except Exception as e:
return {'statusCode': 500, 'body': f'Failed to list buckets: {str(e)}'}
for bucket in buckets:
bucket_name = bucket['Name']
issues = []
try:
# Check Public Access Block — all four settings should be enabled
try:
pab = s3.get_public_access_block(Bucket=bucket_name)
config = pab['PublicAccessBlockConfiguration']
if not all([
config.get('BlockPublicAcls'), # Block new public ACLs
config.get('BlockPublicPolicy'), # Block new public bucket policies
config.get('IgnorePublicAcls'), # Ignore existing public ACLs
config.get('RestrictPublicBuckets') # Restrict access to public buckets
]):
issues.append('Public Access Block not fully enabled')
except s3.exceptions.NoSuchPublicAccessBlockConfiguration:
issues.append('No Public Access Block configured')
# Check bucket policy — flag if policy status is public
try:
policy_status = s3.get_bucket_policy_status(Bucket=bucket_name)
if policy_status['PolicyStatus']['IsPublic']:
issues.append('Bucket policy allows public access')
except s3.exceptions.NoSuchBucketPolicy:
pass # No bucket policy is acceptable
# Check bucket ACL
acl = s3.get_bucket_acl(Bucket=bucket_name)
for grant in acl.get('Grants', []):
grantee = grant.get('Grantee', {})
uri = grantee.get('URI', '')
# 'AllUsers' = anonymous public access
# 'AuthenticatedUsers' = any AWS account (still overly permissive)
if grantee.get('Type') == 'Group' and ('AllUsers' in uri or 'AuthenticatedUsers' in uri):
issues.append('Bucket ACL grants public access')
break
if issues:
risky_buckets.append({'bucket': bucket_name, 'issues': issues})
except Exception as e:
errors.append(f'{bucket_name}: {str(e)}')
# Send alert if risky buckets found
if risky_buckets:
message = f'Found {len(risky_buckets)} buckets with public access:\n\n'
for item in risky_buckets:
message += f" {item['bucket']}: {', '.join(item['issues'])}\n"
sns.publish(
TopicArn='arn:aws:sns:<REGION>:<ACCOUNT_ID>:<TOPIC_NAME>',
Subject='S3 Public Access Alert',
Message=message
)
return {
'statusCode': 200,
'body': json.dumps({
'risky_buckets': risky_buckets,
'errors': errors,
'total_checked': len(buckets)
})
}
Multi-account scanning: This script scans the current account only. To scan across member accounts, see the Multi-account extension section later in this post.
Audit Lambda function – Example script v2 (CSV and JSON report)
The following is an example Lambda function script for reference purposes. Before deploying any script, review error handling, logging, output structure, and permissions. This script generates CSV and JSON output files and uploads them to an S3 bucket for reporting and business intelligence (BI) dashboard integration.
You can deploy both functions with different EventBridge schedules, for example, Script v1 daily for alerts and Script v2 weekly for reports.
Note: Before you deploy this script, replace <OUTPUT_BUCKET_NAME> with the S3 bucket you created for audit reports in step 4.
import boto3
import csv
import json
import os
def lambda_handler(event, context):
s3 = boto3.client('s3')
buckets = s3.list_buckets()['Buckets']
full_access_buckets = []
for bucket in buckets:
bucket_name = bucket['Name']
try:
bucket_policy = s3.get_bucket_policy(Bucket=bucket_name)['Policy']
policy = json.loads(bucket_policy)
for statement in policy['Statement']:
if (statement['Effect'] == 'Allow'
and statement['Principal'] == '*'
and 'Action' in statement
and 's3:*' in statement['Action']):
full_access_buckets.append({'BucketName': bucket_name})
break
except s3.exceptions.ClientError as e:
if e.response['Error']['Code'] != 'NoSuchBucketPolicy':
print(f'Error checking bucket policy for {bucket_name}: {e}')
# Output CSV
csv_output = os.path.join('/tmp', 'full_access_buckets.csv')
with open(csv_output, 'w', newline='') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=['BucketName'])
writer.writeheader()
writer.writerows(full_access_buckets)
# Output JSON
json_output = os.path.join('/tmp', 'full_access_buckets.json')
with open(json_output, 'w') as jsonfile:
json.dump(full_access_buckets, jsonfile, indent=2)
# Upload to Amazon S3
output_bucket = '<OUTPUT_BUCKET_NAME>'
s3.upload_file(csv_output, output_bucket, 'full_access_buckets.csv')
s3.upload_file(json_output, output_bucket, 'full_access_buckets.json')
return {
'statusCode': 200,
'body': json.dumps(f'CSV and JSON files uploaded to {output_bucket}')
}
Important: If this function runs on a schedule, consider implementing a file naming strategy with timestamps to prevent overwriting previous reports or establish a lifecycle policy to manage retention. Include the output bucket in your cleanup procedures when the auditing process is no longer needed.
What if no over-permissioned buckets are found?
If the audit scan returns zero risky buckets, document the clean baseline for future comparison and move to the verification and monitoring phase to so new buckets or policy changes don’t introduce risk over time.
Multi-account extension
The preceding example scripts scan buckets in the current account only. To scan across member accounts in your organization, add the following AssumeRole logic. This function assumes the cross-account IAM role you created during setup, then returns an Amazon S3 client with temporary credentials for each member account.
Note: Before you deploy, configure the following Lambda environment variables:
<MEMBER_ACCOUNTS> – Comma-separated list of 12-digit account IDs to scan (for example, 111111111111,222222222222)
<CROSS_ACCOUNT_ROLE_NAME> – The IAM role name created in each member account (for example, S3AuditRole)
<EXTERNAL_ID> – The external ID configured in the trust policy (for example, s3-audit-external-id)
import boto3
import os
def get_member_s3_clients():
"""
Assumes the cross-account audit role in each member account
and returns a list of (account_id, s3_client) tuples.
"""
sts = boto3.client('sts')
member_accounts = os.environ.get('<MEMBER_ACCOUNTS>', '').split(',')
cross_account_role_name = os.environ.get('<CROSS_ACCOUNT_ROLE_NAME>')
external_id = os.environ.get('<EXTERNAL_ID>')
clients = []
for account_id in member_accounts:
account_id = account_id.strip()
if not account_id:
continue
try:
assumed_role = sts.assume_role(
RoleArn=f'arn:aws:iam::{account_id}:role/{cross_account_role_name}',
RoleSessionName='S3AuditSession',
ExternalId=external_id
)
# Create S3 client with assumed credentials
s3_client = boto3.client(
's3',
aws_access_key_id=assumed_role['Credentials']['AccessKeyId'],
aws_secret_access_key=assumed_role['Credentials']['SecretAccessKey'],
aws_session_token=assumed_role['Credentials']['SessionToken']
)
clients.append((account_id, s3_client))
except Exception as e:
print(f'Failed to assume role in account {account_id}: {e}')
return clients
To scan each member account, replace the single-account s3.list_buckets() call with a loop over member accounts:
def lambda_handler(event, context):
all_risky_buckets = []
all_errors = []
# Scan each member account
for account_id, s3_client in get_member_s3_clients():
try:
buckets = s3_client.list_buckets()['Buckets']
for bucket in buckets:
# ... same scanning logic as the single-account scripts ...
# Use s3_client instead of s3 for each API call
pass
except Exception as e:
all_errors.append(f'Account {account_id}: {e}')
# ... same alerting/reporting logic ...
The Lambda execution role in the central security account needs sts:AssumeRole permission for the cross-account role ARNs. Add this to the execution role policy you created in step 5.
Remediate elevated access
This section describes how to fix over-permissioned buckets using account-level controls, bucket policies, and optional automation. Any elevated access that you find needs to be remediated.
Enable Amazon S3 Block Public Access (account level)
Before applying individual bucket policies, enable Amazon S3 Block Public Access at the account level. This prevents buckets in the account from being made public, regardless of individual bucket policies or ACLs. See theS3 Block Public Access documentation for configuration details. See the following example AWS CLI command; replace <ACCOUNT_ID> with the ID of the account you’re using to manage resource access:
For multi-account environments, deploy this setting across member accounts using AWS CloudFormation StackSets or AWS Organizations service control policies (SCPs).
Important: Before enabling account-level S3 Block Public Access, check whether any workloads need public bucket access (for example, static website hosting, public dataset sharing). Coordinate with your application teams to identify any exceptions.
Remediate using bucket policies
Implement bucket policies that restrict access to specific IAM users, roles, or accounts. When crafting policies, apply the principle of least privilege and include only the actions and principals required for your use case.
Example S3 bucket policy: deny public read/write access. Modify the resource ARN, actions, and conditions to match your requirements:
Automate remediation with Lambda or CloudFormation StackSets (optional):
You can also remediate using Lambda or CloudFormation Stacksets:
Create Lambda functions to automatically update bucket policies or disable public access settings for flagged buckets
Use CloudFormation StackSets to deploy standardized bucket policies and S3 Block Public Access settings across multiple accounts
Verify your remediation
This section explains how to confirm that your fixes are effective before moving to ongoing monitoring. After applying remediation, verify the fix is effective before setting up ongoing monitoring:
Re-run the audit Lambda function – Confirm the previously flagged buckets no longer appear in the risky buckets list.
Check Security Hub compliance – Verify the compliance status has changed from FAILED to PASSED for Amazon S3-related controls.
Validate with IAM Access Analyzer – Review findings for the remediated S3 buckets. Active findings should resolve automatically after public access is removed.
Test application functionality – Confirm that legitimate workloads continue to function correctly.
Document the verification results for your auditing needs. If any S3 buckets still show issues, investigate whether the policy was applied correctly or if there are conflicting permissions.
Automation opportunities
This section covers optional strategies to automate ongoing detection and maintain your security posture without manual intervention.
(Optional) Schedule recurring scans with Amazon EventBridge
Regular security scans help identify new issues arising from configuration changes or newly created S3 buckets. When new security risks are detected, Amazon SNS sends an alert and automatically initiates the remediation phase (Workflow 2 in Figure 1). To avoid repeated alerts, you can configure the audit Lambda function to run on a schedule and compare current results with the previous baseline to generate notifications when new findings are discovered.
For ongoing monitoring, you can schedule the audit Lambda function to run on a recurring basis using EventBridge. Create a scheduled rule with a cron expression (for example, daily at 6:00 AM UTC or weekly on Mondays), add the Lambda function as the target, and grant EventBridge permission to invoke it. See Amazon EventBridge scheduling documentation for instructions on creating scheduled rules and configuring targets.
Enable IAM Access Analyzer for Amazon S3
IAM Access Analyzer monitors bucket policies, ACLs, and access points to identify buckets accessible from outside your account or organization. Create an analyzer scoped to your organization or individual account, then review findings to identify unintended external access. Findings automatically flow into Security Hub when both services are enabled, giving you a dashboard view for Amazon S3 security findings. See the IAM Access Analyzer documentation for setup and usage instructions.
Automate notifications for policy drift
Recurring scans might surface new findings from policy drift or newly created buckets. When new risks are detected, Amazon SNS alert triggers and the remediation cycle repeat (as shown in Workflow 2 in Figure 1) sends email notifications. Configure the audit Lambda function to compare current scan results against the previous baseline and alert on new findings for ongoing reviews.
Clean up
This section lists the resources created during this walkthrough that you should review and remove when they are no longer needed. If the following services were not previously active in your account, leaving them enabled might result in additional ongoing charges. See the Cost considerations section for details. Review and remove unused resources to optimize costs.
Delete or disable the following script-generated resources if they’re not required after outputs are generated. Focus first on Lambda functions and EventBridge rules if you’re not running recurring scans. If you enabled AWS Config or Security Hub specifically for this audit, evaluate whether you need them for other compliance requirements before disabling.
Lambda – Functions, IAM roles, and policies created for auditing
Amazon EventBridge – Scheduled rules created for recurring audit triggers
Amazon SNS – Topics and subscriptions created for notifications
AWS Config – Rules and recorders if no longer needed for compliance
Security Hub – Disable if enabled solely for this audit
IAM Access Analyzer – Delete the analyzer if no longer needed for ongoing monitoring
Note: Be careful when deleting data and consider temporarily disabling services first to check for dependencies. Only delete resources generated as part of your audit outputs. Verify you have retained any necessary results before proceeding. Verify resources are not used by other workloads before deletion.
Best practices
This section provides recommendations to maintain secure Amazon S3 configurations long-term. To learn more about maintaining secure Amazon S3 configurations, review the AWS documentation links provided in the conclusion. The following recommendations aren’t exhaustive. Adapt and extend them based on your organization’s evolving security requirements and AWS best practices guidance. After you’ve fixed existing issues, these practices help you maintain secure Amazon S3 configurations.
Start with account-level controls – Enable S3 Block Public Access at the account level. This prevents buckets from becoming public even if someone misconfigures an individual bucket policy. For multi-account environments, enforce this through AWS Organizations SCPs.
Automate detection – Use IAM Access Analyzer to detect external access. Schedule your audit Lambda function with EventBridge to catch new issues weekly or daily, depending on your change frequency. Compare scan results against previous baselines to identify drift.
Standardize across accounts – Use CloudFormation StackSets to deploy the same secure configuration to all accounts in your organization, reducing the chance of configuration drift. Use StackSets for IAM roles, AWS Config rules, and S3 Block Public Access settings.
Additional security measures
Regularly review and rotate cross-account IAM role credentials and external IDs
Implement Amazon S3 server-side encryption (SSE-S3 or SSE-KMS) for data at rest
Enable S3 access logging and AWS CloudTrail data events for audit trails
Conclusion
This section summarizes what you accomplished and suggests next steps to maintain your S3 security posture. By implementing the detection, remediation, and monitoring workflow outlined in this post, you can proactively identify and secure over-permissioned S3 buckets across your AWS environment. To maintain your ongoing security posture, enable IAM Access Analyzer for continuous monitoring and schedule recurring audits with EventBridge. To learn more about Amazon S3 security best practices, see Security best practices for Amazon S3
For legal and compliance teams, auditing a repository of thousands of contracts for a single regulatory obligation shouldn’t take weeks. But with keyword search, it often does. A search for “inadvertent access notification” returns exact matches while missing functionally equivalent clauses such as “security incident disclosure” or “unauthorized access reporting.” This creates two problems:
Discovery gap: Critical risk exposure goes undetected because keyword search cannot match semantically equivalent terms across different contracts.
Review latency: After finding relevant contracts, legal counsel must manually scan lengthy documents to locate the specific clauses that matter. This process can stretch from minutes to hours per document.
Amazon OpenSearch Service is a fully managed search and analytics service that configures, manages, and scales OpenSearch clusters in the AWS Cloud. It supports use cases from log analytics and application monitoring to full-text search and real-time security analytics. It also supports AI-powered semantic search.
Amazon OpenSearch Service addresses both problems through two capabilities:
Semantic search retrieves contracts based on meaning rather than exact keyword matches, closing the discovery gap.
Semantic highlighting pinpoints the exact clauses within retrieved contracts that answer the query, reducing review time from hours of manual scanning to seconds of targeted reading.
In this post, you build a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. You deploy the solution using two AWS CloudFormation stacks, test it with synthetic contract documents, and see how a single query surfaces both the right contracts and the right clauses within them.
Solution overview
The solution uses a two-stage retrieval and extraction pipeline. First, semantic search identifies relevant contracts across the repository. Then, semantic highlighting marks the specific clauses within those contracts that match the query intent.
The following diagram illustrates the solution architecture:
Upload contracts toAmazon Simple Storage Service (Amazon S3) – Contract documents (JSON format) are uploaded to an Amazon S3 bucket, which serves as the centralized document repository.
Amazon OpenSearch Ingestion (OSI)reads from S3 – A serverless OSI pipeline detects new documents in the S3 bucket and reads them for processing.
OpenSearch ingest pipeline generates embeddings through Amazon Bedrock – As documents arrive, the ingest pipeline’s text_embedding processor invokes Amazon Titan Text Embeddings V2 through an ML Commons Bedrock connector. This converts contract text into 1024-dimension vector representations, stored in a k-NN index that uses the faiss engine.
User submits a search query – A user queries the system with natural language (for example, “data protection regulations”) through a test AWS Lambda function that forwards the request to OpenSearch using the neural query type.
OpenSearch generates the query embedding – OpenSearch converts the user’s natural language query into a vector embedding using the same machine learning (ML) Commons Amazon Bedrock connector and Amazon Titan V2 model.
Amazon OpenSearch Service performs semantic search – OpenSearch uses k-NN vector similarity to retrieve contracts that are semantically relevant to the query, even when exact terminology differs.
Amazon SageMaker AI performs semantic highlighting – The opensearch-semantic-highlighter-v1 model, hosted on an Amazon SageMaker AI GPU endpoint, scores sentence relevance using cross-encoder inference and wraps the matching clauses in <em> tags for targeted reading.
How semantic search and semantic highlighting work together
The system processes queries in two steps:
Step 1 – Semantic search (document discovery): You query the contract corpus using natural language. The system retrieves contracts with semantically similar concepts, even when exact terminology differs. For example, searching for “force majeure” returns contracts discussing “natural disasters” or “unforeseeable circumstances” because the system understands these concepts are related.
Step 2 – Semantic highlighting (clause identification): After relevant contracts are retrieved, semantic highlighting automatically marks the clauses that semantically match your search intent. Instead of scanning pages of legal text, you immediately see the specific paragraphs that answer your question.
Keyword highlighting wraps individual matching words: <em>termination</em> and <em>rights</em>.
Semantic highlighting wraps entire relevant clauses: <em>Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days.</em>.
This reduces false positives, cuts review time, and provides explainability for why each document was retrieved.
Semantic highlighting model deployment
Before the system can highlight clauses based on meaning, the opensearch-semantic-highlighter-v1 model must be deployed to an Amazon SageMaker AI GPU endpoint and registered with the OpenSearch ML Commons plugin through a remote connector.
Stack 2 of the CloudFormation deployment automates this process. It performs the following steps:
Downloads the model artifact from an AWS-managed source and deploys it to an Amazon SageMaker AI endpoint (ml.g5.xlarge).
Creates a remote ML Commons connector in OpenSearch that points to the SageMaker endpoint.
Registers the model with the QUESTION_ANSWERING function so that OpenSearch can use the model’s cross-encoder capabilities to score sentence relevance at query time.
The equivalent manual registration call (handled automatically by the stack) is:
You don’t need to run this manually. The deployment script and CloudFormation stack handle model registration end-to-end. The resulting model ID is automatically passed to the query Lambda function for use in semantic highlighting requests.
Index configuration
The index uses a k-NN vector field with 1024 dimensions (matching the Amazon Titan V2 output) and the faiss engine with HNSW method. The mapping includes both a knn_vector field for semantic retrieval and a standard text field for keyword matching and highlighting. When you search for “liability limits,” OpenSearch first retrieves documents through vector similarity, then uses the Amazon SageMaker AI model to identify and wrap the specific relevant sentences in <em> tags.
This section walks you through deploying the solution using two AWS CloudFormation stacks and two shell scripts. You first set up the core infrastructure (OpenSearch, ingestion pipeline, and ML Commons Bedrock connector), then deploy the semantic highlighting model on Amazon SageMaker AI.
Prerequisites
To deploy this solution, you need:
An active AWS account with permissions to create Amazon S3 buckets, AWS Lambda functions, Amazon SageMaker AI endpoints, Amazon Bedrock model access, Amazon OpenSearch Ingestion pipelines, Amazon OpenSearch Service domains, and AWS Identity and Access Management (IAM) roles (including iam:PassRole and sts:AssumeRole). For the exact least-privilege policy, see iam-deployer-policy.json in the repository. Both CloudFormation stacks require the CAPABILITY_NAMED_IAM acknowledgement.
Amazon Bedrock model access enabled for Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0).
Estimated deployment time: approximately 35 minutes.
Estimated cost: approximately USD $ 2.00–3.00 for a quick demo. Delete the stacks promptly after testing.
This post uses US East (N. Virginia) as the deployment AWS Region. Verify service availability in your preferred Region before deploying.
Deploy the solution
The solution deploys using two AWS CloudFormation stacks and two shell scripts. The demo includes synthetic contract documents covering common contract types including software licenses, data processing agreements, managed services, and software as a service (SaaS) subscriptions.
Clone the repository and run the deployment script:
git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./deploy.sh
The deployment script creates the following resources across two stacks:
Stack 1:
An Amazon OpenSearch Service domain with fine-grained access control.
An Amazon OpenSearch Ingestion (OSI) pipeline that reads contracts from S3 and sends them to OpenSearch for indexing.
An ML Commons Bedrock connector and ingest pipeline that automatically generates 1024-dimension vector embeddings through Amazon Titan Text Embeddings V2 during document indexing.
A test Lambda function for querying the OpenSearch index using keyword, neural, or hybrid search with semantic highlighting support.
An S3 bucket for storing contract documents.
IAM roles for Lambda functions, the OSI pipeline, and OpenSearch access.
Stack 2:
An Amazon SageMaker AI endpoint hosting the semantic highlighting model.
A Lambda function that creates an ML Commons remote connector in OpenSearch and registers the highlighting model.
After both stacks deploy, the script automatically configures OpenSearch (role mappings, Amazon Bedrock connector, embedding model, k-NN index), ingests the sample contract data, and registers the semantic highlighting model.
The total deployment takes approximately 35 minutes to complete.
(Optional) Automated deployment with Claude Code CLI
If you have Claude Code CLI installed, you can deploy the solution using an AI-assisted workflow that creates a least-privilege IAM role scoped to this demo before deploying:
git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./scripts/create-deployer-role.sh
export OS_DEMO_DEPLOYER_ROLE=arn:aws:iam::<ACCOUNT_ID>:role/os-demo-deployer-role
export AWS_DEFAULT_REGION=us-east-1
claude "Deploy the OpenSearch semantic search demo following README.md"
Claude Code reads the repository instructions, assumes the deployer role, deploys both CloudFormation stacks in order, runs the setup scripts, and verifies the deployment end-to-end. The deployer role restricts actions to resources prefixed with os-demo-*, following the principle of least privilege.
Test the solution
After the deployment succeeds, follow these steps to test the solution.
On the Lambda console, choose Functions in the navigation pane.
Choose the function that has os-demo-query in its name.
On the Test tab, in the Event JSON paste this keyword search query {"query": "data protection regulations?", "type": "keyword", "k": 3}
Choose Test to run the Lambda function.
The following screenshot shows the Lambda function test configuration on the AWS Management Console with the keyword search query.
The function processes the query in two ways depending on the search type:
For keyword search (enter: keyword): The function sends a standard match query to OpenSearch, which returns documents containing the exact query terms. The highlight fragments wrap individual matching words like <em>termination</em> and <em>rights</em>.
For neural search (enter: neural): The function sends a hybrid query to OpenSearch combining k-NN (semantic similarity) with keyword matching. OpenSearch automatically generates the query embedding through the ML Commons Amazon Bedrock connector using the same Amazon Titan V2 model. This returns semantically related documents even if they don’t contain the exact query terms. The SageMaker endpoint powers the semantic highlighting, identifying the most relevant clauses within each retrieved document. It wraps entire passages like <em>Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days.</em>.
Download the highlight viewer HTML file and open it in the browser. This file helps you view the highlighted text.
Copy the entire execution output of the Lambda execution, paste it into the placeholder in the HTML file, and then choose Load Results.
The following screenshot shows that only the matching keywords are highlighted.
Next, paste the neural search query as input to the Lambda function to see how semantic highlighting works: {"query": "data protection regulations", "type": "neural", "k": 1}
Choose Test to run, and then paste the entire output into the HTML viewer.
The viewer now displays entire sentences highlighted instead of individual keywords.
Optimizing for scale: batch semantic highlighting
In a standard search, a query might return dozens of relevant contracts. Using the default single inference mode, OpenSearch makes a separate ML call for every document in the result set. For a compliance officer reviewing 50 contracts, this sequential processing introduces noticeable latency.
OpenSearch 3.3 introduced batch inference mode to address this. Batch inference collects the matching documents and processes them in a single ML inference call. In the contract compliance use case, this shifts the performance characteristic from multiple sequential roundtrips to a single parallel execution on the Amazon SageMaker AI GPU.
To enable batch inference, first configure the cluster setting:
PUT _cluster/settings
{
"persistent": {
"search.pipeline.enabled_system_generated_factories": ["semantic-highlighter"]
}
}
Then add batch_inference: true to your highlight options. The following query searches for data privacy clauses across the contracts and highlights the top 10 results using a single batch call:
Follow these recommendations to optimize performance, security, and cost-efficiency when deploying the contract compliance search system in production.
Experiment with overlapping chunk sizes (for example, 500 characters with a 10 percent overlap) in your OSI pipeline to verify that context is preserved for long indemnification or liability clauses.
Verify that your Amazon S3 buckets and OpenSearch domains are encrypted using AWS Key Management Service (AWS KMS). For production workloads containing sensitive data, make sure that all traffic stays within your virtual private cloud (VPC) through interface endpoints.
This demo uses simplified configurations for learning purposes. For production deployments, implement VPC isolation, AWS KMS encryption with customer-managed keys, and multi-AZ OpenSearch clusters.
Clean up resources
To avoid ongoing charges, delete the AWS CloudFormation stacks and associated resources:
On the AWS CloudFormation console, choose Stacks in the navigation pane.
Select the os-demo-highlighting stack (Stack 2) and choose Delete. Wait for deletion to complete.
Select the os-demo-search stack (Stack 1) and choose Delete. Stack deletion takes approximately 10–15 minutes to complete.
The stack deletion will automatically remove:
OpenSearch domain.
SageMaker model and endpoint.
Lambda functions.
IAM roles and policies.
After both stacks are deleted, manually delete the S3 bucket (opensearch-cfn-semantic-highlighting-us-east-1-<ACCOUNT_ID>) created for model artifacts. This bucket is provisioned at deploy time and is not managed by CloudFormation. Replace <ACCOUNT_ID> with your AWS account ID in the bucket name.
Conclusion
In this post, you built a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. The system helps close the discovery gap by retrieving contracts based on meaning rather than exact keywords, and it reduces review latency by highlighting the specific clauses that answer your query.
While we focused on legal agreements, the architecture described here is a blueprint for domains requiring high-stakes document discovery, including:
Regulatory filings: Identifying specific compliance mandates in financial reports.
Technical documentation: Pinpointing troubleshooting steps across massive product manuals.
Research and academia: Isolating specific methodologies within thousands of scientific papers.
Internal knowledge bases: Empowering employees to find exact policy language instantly.
AWS GlueData Catalog view is a multi-dialect view that supports querying from multiple SQL query engines, such as Amazon Athena, Amazon Redshift Spectrum, Apache Spark in Amazon EMR and AWS Glue. You can create a Data Catalog view in one account, using an AWS Identity and Access Management (IAM) definer role in the same or different account and use AWS Lake Formation to share the view across multiple accounts. The definer role has the required full SELECT on the base tables to create the view and share it with other users for querying. The Data Catalog assumes the definer role and manages access of the base tables when the view is queried, thus allowing to share a subset of data without sharing the underlying base tables.
AWS Glue now adds AWS SDK support for creating and updating the ATHENA dialect of Glue views. With this addition, you can now create ATHENA and SPARK dialects of Glue views simultaneously, using a cross account IAM definer role. This feature enhances the automation to create and update Glue views, like that of Data Catalog tables. In our earlier blog Create AWS Glue Data Catalog views using cross-account definer roles, we had introduced IAM definer roles in a cross-account use case to create Data Catalog views with SPARK dialects using the APIs – CreateTable() and UpdateTable() – while creating and adding ATHENA dialects using Athena query editor. As a continuation to it, this post shows you how to use the Catalog objects API CreateTable() to programmatically create ATHENA and SPARK dialects using cross-account IAM definer roles, and how to add the ATHENA dialect programmatically for the views that were created earlier with only SPARK dialect.
Cross account definer roles enable enterprise data mesh architectures where multiple accounts are interconnected in a central governance and multiple producers and consumers. The central governance account hosts the database, tables and permissions, while the producer accounts maintain CI/CD pipelines to create and manage those data assets. Having the definer role in producer accounts allows those CI/CD pipelines to be fully managed by IAM roles in the individual accounts.
Key points on creating multi-dialect views using cross-account definer roles
ATHENA dialects are validated and asynchronously created. Hence, a cross-account Glue connection is required for validation for every producer account-central governance account pair. This is a one-time setup.
SPARK dialects are not validated. Hence SPARK dialect’s create syntax requires SubObjects list of the base tables and StorageDescriptor fields for the columns of the view.
Though queries on cross account views can be run using database resource link names, the view definition SQL query for creating the view requires the original database and base table names from the central governance account.
If a view has SPARK and ATHENA dialects available, we recommend updating both the dialects of the view simultaneously using update_table() API/SDK, for any changes in the SQL definition of the view or the base table. This will keep both the dialects queryable.
Creating and updating both SPARK and ATHENA dialects using cross account definer role is supported using AWS CloudFormation.
The Data Catalog view that can be created using cross account IAM definer roles are available in SPARK and ATHENA dialects and currently not supported for Redshift Spectrum dialect.
Prerequisites
We use the same setup used in Create AWS Glue Data Catalog views using cross-account definer roles for the sample database, tables, definer role, resource link, IAM and Lake Formation permissions on those resources and principals between the two AWS accounts. Summarizing the requirements as below.
The setup includes a central governance account with Data Catalog database bankdata_icebergdb and two tables transaction_table1 and transaction_table2, a producer account with a Data-Analyst role used as view definer role.
Lake Formation permissions on the central account’s database and tables are granted to the producer account Data-Analyst role as per the earlier blog. The definer role in producer account should have database DESCRIBE and CREATE_TABLE permissions, table SELECT and DESCRIBE permission on all columns and rows of the base tables. The IAM permissions required on the definer role are detailed in Prerequisites for creating views. Similarly, follow the earlier blog to create resource link for the shared database and grant Lake Formation permissions on the resource link to the Data-Analyst
An Athena data source named centraladmin in the producer account, pointing to the Data Catalog of the central governance account.
Creating ATHENA and SPARK dialects at the same time
Creating both ATHENA and SPARK dialects of a Glue catalog view simultaneously is now supported by the AWS SDK. In the producer account, create a new Glue connection, required for the Athena dialect validation. This is a prerequisite for creating the ATHENA dialect of the Glue catalog view using cross account definer role. Then we create a Glue view with both dialects.
Sign in to the producer account as the Lake Formation admin role, or any role with permission to create AWS Glue connections.
Note: If you are using Athena for the first time in your account or using Primary workgroup, setup the query results location bucket using Specify a query result location.
Sign out as the Lake Formation admin and sign back in to the producer account as the definer IAM role, Data-Analyst.
Create an AWS Glue view using the create-table CLI command and JSON file, or using the AWS SDK for Python (Boto3) script.
The content of create_multipledialects.json is as follows.
{
"DatabaseName": "rl_bank_iceberg",
"TableInput": {
"Name": "view_2dialects_2basetables_fromcli",
"StorageDescriptor": {
"Columns": [
{
"Name": "transaction_id",
"Type": "string"
},
{
"Name": "transaction_type",
"Type": "string"
},
{
"Name": "transaction_amount",
"Type": "double"
},
{
"Name": "transaction_location",
"Type": "string"
},
{
"Name": "transaction_date",
"Type": "date"
}
},
"ViewDefinition": {
"SubObjects": [
"arn:aws:glue:us-west-2:<central-account-id>:table/bankdata_icebergdb/transaction_table1",
"arn:aws:glue:us-west-2:<central-account-id>:table/bankdata_icebergdb/transaction_table2"
],
"IsProtected": true,
"Representations": [
{
"Dialect": "SPARK",
"DialectVersion": "1.0",
"ViewOriginalText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id",
"ViewExpandedText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id"
},
{
"Dialect": "ATHENA",
"DialectVersion": "3",
"ViewOriginalText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id",
"ValidationConnection": "glue-view-validation-connection"
}
]
}
}
}
Notes about fields in the above CLI input JSON (applies to all SDK):
The definer is by default the API caller, but a Definer field can be set to explicitly specify a different IAM role.
In the ViewDefinition, database qualifiers are required for SPARK dialect. That is, the SQL definition provided for ViewOriginalText and ViewExpandedText should be in <source_database_name>.<source_table_name> format.
After the view is created, you can inspect the details on the Lake Formation console. The SQL definitions show both ATHENA and SPARK as shown in the following screenshot.
If your view creation fails for any of the dialects, you can use the AWS Glue get-table CLI command with --include-status-details to see what the error is and rectify it.
The PySpark script for creating a view with ATHENA and SPARK dialects are provided below. Download and edit the Pyspark script with your bucket name, producer and central account ids, region and relevant Glue resource names: bdb_5773_createview_bothdialects.py
Provide the following settings to run the script in your Glue Studio. For details on running a Spark job in Glue, refer Working with Spark jobs in AWS Glue.
Choose Data-Analyst as the job execution IAM role.
Choose Glue 5.1 for Glue version.
For the Requested number of workers, provide >=4. This is an FGAC Spark driver requirement, which is needed for Glue catalog views. Below screenshot shows these settings.
Add the following 2 properties as additional job parameters. A screenshot is shown for reference.
Adding ATHENA dialect using SDK to an existing AWS Glue view
You can update an existing AWS Glue view that was created with the SPARK dialect and add the ATHENA dialect using the SDK. The following example uses the update-table CLI command.
The content of add-athena-dialect.json is as follows.
{
"DatabaseName": "rl_bank_iceberg",
"ViewUpdateAction": "ADD",
"TableInput": {
"Name": "view_sparkfirst_athenanext",
"ViewDefinition": {
"Representations": [
{
"Dialect": "ATHENA",
"DialectVersion": "3",
"ViewOriginalText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id",
"ValidationConnection": "glue-view-validation-connection"
}
]
}
}
}
Verify the added dialect on the view by reviewing the SQL definitions of the view in Lake Formation console or using GetTable(). If you want to edit the SQL definition or change the base tables of an existing view that has both SPARK and ATHENA dialects, you can do so using the update_table API (using SDK or CLI), with "ViewUpdateAction": “REPLACE” and provide both the dialect definition under ViewDefinition.
You can run queries on the view from the producer account as Data-Analyst. The view can be shared using Lake Formation Tags or named method, just like sharing tables, to additional consumer accounts from the central governance account. The consumer accounts will create a resource link and query the views.
Cleanup
To avoid incurring ongoing costs, clean up the resources you used for this post:
Revoke the Lake Formation permissions granted to the Data-Analyst role and the producer account from the central governance account.
Drop the Data Catalog tables, views, and the database.
Delete the Athena query results from your Amazon Simple Storage Service (Amazon S3) bucket.
Delete the Data-Analyst role from IAM.
Delete the AWS Glue connection and the Athena data source.
Delete the AWS Glue job, if you tried the Python script as an AWS Glue job.
Conclusion
In this post, I demonstrated how to use cross-account IAM definer roles with AWS Glue Data Catalog views, how to create and update ATHENA and SPARK dialects using the Data Catalog CreateTable() and UpdateTable() APIs. The multi-dialect Data Catalog views allow sharing a subset of data from different tables using Lake Formation permissions, including LF-Tags based access control. The cross-account definer roles support multi-account data mesh architectures so that the producer IAM roles can run the CI/CD pipelines in its account. We encourage you to try the feature and share your feedback in the comments.
Acknowledgements: I would like to thank all the team members who worked to add AWS SDK support for creating ATHENA and SPARK dialects together for AWS Glue views – Daniil Arushanov, Wyatt Hawes, Yuxi Wu, Santhosh Padmanabhan and Karthik Devaraj.
In this 33rd quarterly recap post, discover the most impactful AWS serverless launches, features, and resources from Q2 2026 that you might have missed. Stay current with the latest serverless innovations that can improve your applications.
In case you missed our last ICYMI, read about what happened in Q1 2026.
AWS Lambda MicroVMs
AWS Lambda MicroVMs is a new serverless compute primitive for running user or AI-generated code in isolated, stateful execution environments. Built on the same Firecracker virtualization that powers over 15 trillion monthly Lambda invocations, MicroVMs give you VM-level isolation with near-instant launch and resume. Each MicroVM runs in its own Linux environment with no shared kernel or resources between sessions. This isolation makes it a useful solution for AI coding assistant sandboxes, interactive code or multi-tenant development environments, CI/CD build environments, data analytics platforms, vulnerability scanners, and game servers that run user-supplied scripts.
Standard Lambda functions are best for event-driven, request-response workloads which have a 15-minute timeout. MicroVMs are purpose-built for single end user or session workloads and can preserve state for up to 8 hours. You get full lifecycle controls including launch, suspend, resume, and terminate. You can suspend them during the 8 hours if you don’t need them active. MicroVMs retain memory and disk state for the length of the session, even while suspended. They can auto resume when you need to use them again.
Amazon S3 Files makes your S3 buckets accessible as high-performance file systems. S3 files is a fully featured, POSIX-compatible file system to access to your data with approximately 1ms latency.
For serverless workloads, the Lambda integration with S3 Files lets your functions mount an S3 bucket as a local file system. Your function reads and writes files at a local mount path like /mnt/data, and the file system handles synchronization with S3 automatically. You can avoid downloading objects to /tmp from S3 within your function and work directly with files. Applications that assume a file system can now run on Lambda without rewriting their I/O layer. Use cases include sharing data between functions, ML model loading, document processing, media transcoding, or any pipeline that treats data as files rather than objects.
AWS Lambda durable functions
The Lambda durable functions SDK for Java is now generally available, joining Python and TypeScript. This allows Java developers to build multi-step workflows with automatic checkpointing and recovery without adding external orchestration. Durable functions is also now available in 16 additional AWS Regions. Learn how to build fault-tolerant multi-agent AI workflows to coordinate multiple AI agents that call tools, make decisions, and hand off work. There is automatic recovery if any agent fails mid-task. Voice analytics with Amazon Bedrock shows building a pipeline that processes call recordings through transcription, sentiment analysis, and summarization with durable checkpoints between each stage. For best practices, AI patterns, and futures, view the live stream.
AWS Lambda Managed Instances
Lambda Managed Instances now allows you to build memory-intensive apps with up to 32 GB (3x more than standard Lambda). This allows use cases like in-memory caching, large dataset analytics, and ML inference that previously required considering other services.
Scheduled scaling lets you pre-warm capacity for predictable traffic patterns with Amazon EventBridge Scheduler. This helps reduce cold start latency during known demand spikes. Tag propagation automatically applies your function tags to the underlying Amazon EC2 instances, Amazon Elastic Block Store volumes, and network interfaces. This helps finance teams with cost allocation visibility without manual tag management.
Other Lambda updates
Response streaming is now available in all commercial AWS Regions, bringing full regional parity for progressively streaming data back to clients. This is useful for LLM-powered applications where users expect to see tokens as they generate rather than waiting for a complete response.
If you have a fleet of functions on older runtimes, you can now upgrade runtimes at scale using AWS Transform custom. This uses AI to analyze your function code, identify breaking changes for the target runtime version, and generate the code modifications needed. This can help teams save manual migration effort across many functions. The Serverless Office Hours live stream has more information.
AWS Serverless Application Model (AWS SAM) CLI now supports BuildKit for building container images from Dockerfiles. This allows faster multi-stage builds with better caching, cross-architecture image builds, and Docker secrets to keep credentials out of final image layers.
Containers with Mama J
Serverless with Mama J
Mama J is back in the second video of a series where Eric Johnson explains what he does all day at work to his mother. Previously, they talked serverless and Lambda. This time it’s containers, what they are, why they exist, and how AWS manages them at scale. Eric goes through the “it works on my machine” problem, how Docker builds images, container orchestration and how containers differ from Lambda.
AWS Step Functions has an Amazon Bedrock AgentCore-powered agentic reasoning step. You can embed AI agent reasoning directly inside a workflow as a native step type. This bridges structured orchestration with autonomous agent behavior. Your workflow handles the deterministic parts such as branching, retries, timeouts, parallel execution, while the agentic step handles the parts that require flexible reasoning.
The Amazon BedrockAgentCore harness reached general availability. Two API calls give you a running agent in seconds which runs in its own isolated environment with a filesystem and shell. It can read files, run commands, and write code safely.
AgentCore Payments (preview) allows agents to autonomously access and pay for APIs and MCP servers, opening up agent-to-agent commerce. AgentCore Memory has metadata for long-term memory so agents retain and recall context across sessions. Web Search on AgentCore grounds agents in current, cited web knowledge. The Runtime now supports bring-your-own file systems from S3 Files and Amazon Elastic File System, and Node.js for direct code deployment.
Strands Agents SDK
The open source Strands Agents SDK shipped three capabilities. Context management that cuts token costs in half by intelligently pruning what goes into the model context window, Strands Shell for sandboxed agent code execution, and Strands Evals 1.0 with chaos testing and adversarial red teaming. This can reduce costs to help make production agent workloads cheaper without sacrificing quality. A Serverless Office Hours live stream covered the new features in depth.
The Agent Toolkit for AWS became generally available with three plugins (aws-core, aws-agents, aws-data-analytics), over 30 curated skills, and the AWS MCP Server. View this video for an introduction. This gives AI coding agents such as Kiro, Claude Code, and Cursor expert AWS knowledge which helps to reduce errors and lower token costs. For more information on the serverless tools available when using AI, see this Serverless Land resources page.
Serverless Office Hours ran a live stream series finding out how experts use AI to build serverless applications. Hear from:
Kiro launched Kiro Pro Max and an iOS mobile app for approving and monitoring agentic coding sessions from your phone. Amazon Q Developer IDE plugins are transitioning to Kiro. The Kiro power for AWS DevOps Agent connects your IDE directly to production intelligence. You can investigate incidents and generate fixes without context switching.
Join our live stream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. View episodes on-demand at serverlessland.com/office-hours.
The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.
You can also follow the Developer Advocacy team to get the latest news, follow conversations, and interact with the team.
Last week, my team visited Seoul to meet AWS Korea User Group (AWSKRUG) leaders. AWSKRUG is the largest cloud developer community in Korea, with 20 meetup groups organized by topic and area that collectively host over 100 events each year, primarily in Seoul.
My team regularly visits countries across the Asia-Pacific region, listens to feedback from user group leaders, and works to support their communities. At this meeting, leaders honestly shared what they did well in the first half of the year, what needs improvement, and what they asked of AWS Developer Experience team. We also enjoyed a pleasant conversation during our Chimaek time together.
Now, let’s take a closer look at key launches of last week.
A one-click Lambda setup prompt for coding agents caught my eye most last week. This prompt configures your agent with AWS Serverless skills and the Serverless Model Context Protocol (MCP) server, embedding serverless best practices from the start. This prompt references the Lambda agent setup guide, which includes installation commands for Claude Code, Kiro, Cursor, GitHub Copilot, Codex, Devin Desktop, and OpenCode.
To get started, choose the Copy agent prompt button on the Lambda console screen or copy fetch https://docs.aws.amazon.com/lambda/latest/dg/samples/aws-lambda-agent-setup.md directly, and paste this URL in your preferred AI agent.
You can also use Agent Toolkit for AWS to give your coding agent current AWS knowledge and safe resource access. Use fetch https://raw.githubusercontent.com/aws/agent-toolkit-for-aws/refs/heads/main/setup-instructions/setup.md for installing AWS MCP Server.
Last week’s launches Here are last week’s launches that caught my attention:
OpenAI GPT-5.6 Sol, Terra, and Luna on Amazon Bedrock: You can use the smartest family of models from OpenAI yet on Bedrock’s next-generation inference engine built for high performance, security, and reliability. The three models span capability tiers from flagship reasoning (Sol) to balanced performance (Terra) to fast, cost-efficient inference (Luna), all accessible through the Responses API on Amazon Bedrock.
Same-day transitions to Amazon S3 Standard-IA and S3 One Zone-IA: You can now transition objects to S3 Standard-Infrequent Access (S3 Standard-IA) and S3 One Zone-Infrequent Access (S3 One Zone-IA) as soon as the day they are created, without the previous 30-day minimum retention period in S3 Standard. These storage classes offer up to 40% lower storage costs than S3 Standard while still providing millisecond access when needed, making them ideal for backups, log analytics, and compliance workloads where data becomes cold within hours or days.
Self-managed code storage on AWS Lambda: With self-managed Amazon S3 buckets for code storage, you can reference source code directly from your own S3 buckets without Lambda creating intermediate copies. This eliminates code storage limits and reduces function activation time after function creates and updates by removing the copy step.
Importing users with password hashes on Amazon Cognito: You can now import users with password hashes in CSV user imports. Previously, imported users had to reset their passwords on first sign-in. Now, you can include password hashes in the CSV import, enabling users to sign in immediately with their existing credentials. When creating a CSV import, you specify the password hashing algorithm used by your source system.
For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.
Additional updates Here are some additional news items that you might find interesting:
Open Protocols with the Strands Agents SDK: Learn how open AI protocols such as MCP, A2A, UTCP, AG-UI, and x402 work together using Strands Agents SDK for building AI agents as an example implementation, though the patterns apply to any agent framework.
Open source Bulk Executor for Amazon DynamoDB: Performing bulk operations against all items in a DynamoDB table has historically required custom coding. The Bulk Executor for DynamoDB simplifies bulk tasks like these. You can use this feature to invoke commands like count, find, delete, or update. No coding is required, even when running at large scale.
Transform AWS Support Case Workflows with Kiro CLI: Explore how Kiro CLI’s MCP integration accelerates support case workflows by combining investigation, documentation lookup, and case creation into a single conversational interface across three real-world scenarios: AWS Glue job failures, AWS Lambda cold start investigation, and AWS WAF false positive analysis.
For a full list of AWS blog posts, be sure to keep an eye on the AWS Blogs page.
Finally, some customers experienced an issue with Cost Explorer displaying inaccurate estimated billing data in last weekend. They may have received erroneous budget and cost anomaly detection alerts, and observed inflated estimated cost and usage data. The issue has been resolved, and all AWS services are operating normally. We apologize for the concern this incident caused our customers and are conducting a thorough retrospective to prevent events like this from reoccurring, as well as improving our response when billing incidents occur. For more information, visit the AWS Health Dashboard.
That’s all for this week. Check back next Monday for another Weekly Roundup!
If you manage Lambda functions at scale, you’ve likely hit the 75 GB code storage limit or explained to your security team why deployment artifacts live in an S3 bucket you don’t control. Today, we’re announcing self-managed Amazon S3 buckets for AWS Lambda deployment packages. Lambda reads your code directly from your bucket, eliminating quota pressure and giving you full security control.
Previously, the default AWS-managed code storage created three challenges at scale. First, all copies count toward your 75 GB code storage quota. Second, you cannot apply your own encryption, access controls, or compliance tags to the internal bucket. Third, the copy cannot be incorporated into your disaster recovery strategies.
With self-managed S3 buckets, Lambda reads your function code directly from your bucket. No copy, no duplication. Your S3 object becomes the single source of truth for your functions. Deployment packages no longer count against your account’s code storage limit. You manage the bucket’s security and compliance posture: choosing the encryption, defining the access policies, managing lifecycle transitions, and maintaining the audit trail. And because you own the bucket, you can use S3 Cross-Region Replication to maintain fallback copies of your code in a secondary Region, so that your functions remain deployable even if your primary Region experiences an issue. Using self-managed S3 buckets also results in a faster time to first invoke for new functions and after function updates, because Lambda no longer needs to copy your zip package to a Lambda-managed S3 bucket.
You can use this feature today in all AWS standard regions where Lambda is available, at no additional charge beyond your standard Amazon S3 storage and request costs. Let’s look at some use cases, how it works, and how to use it at scale.
Use cases
Here are a few patterns where owning your deployment bucket makes a real difference.
CI/CD pipelines and artifact management
With self-managed storage, your CI/CD pipeline uploads once, and Lambda references the same object. One set of lifecycle rules and access controls covers all artifacts, and rollbacks mean pointing the function to a previous S3 object version.
Multi-account and multi-team architectures
Organizations using AWS Organizations often separate workloads into multiple accounts: a development account, a staging account, and a production account. They centralize shared resources in a tooling or shared-services account.
Self-managed buckets integrate naturally with this pattern:
Store all deployment artifacts in a central “artifact account” bucket.
Grant cross-account s3:GetObject access to Lambda execution roles in each workload account through bucket policies.
Maintain a single inventory of what code is deployed where, managed by your platform or DevOps team.
Enforce consistent encryption, versioning, and retention policies from one place.
Disaster recovery and business continuity
Because your deployment artifacts live in a bucket you own, you can use the built-in replication features of S3, Cross-Region Replication (CRR) or Same-Region Replication (SRR), to maintain copies of your code artifacts in backup locations. Combined with S3 Versioning and Object Lock, this gives you a durable, tamper-proof code archive that supports rapid recovery if a deployment is accidentally corrupted or deleted.
How it worked before
When you deploy a Lambda function using a .zip deployment package stored in Amazon S3, the process has traditionally worked like this:
You upload your .zip deployment package to your S3 bucket.
You call CreateFunction or UpdateFunctionCode, specifying the S3 bucket and S3 key.
Lambda copies the .zip artifact from your bucket into an internal, service-managed bucket.
Lambda uses this copy to create the optimized version of your function that runs at invocation time.
The copied artifact counts toward your account’s 75 GB code storage quota.
Figure 1 — Standard Lambda deployment
This model is straightforward and works well for most workloads. However, it creates three friction points at scale:
Storage quota pressure: Every deployment package copy counts toward your account’s 75 GB total code storage limit. Organizations with hundreds of functions and multiple published versions can exhaust this quota.
No control over stored artifacts: You cannot configure encryption (beyond the service default), access logging, lifecycle policies, Object Lock, or compliance tags on the internal bucket.
Redundant storage: Your original artifact remains in your bucket while a copy lives in the Lambda bucket used for provisioning new instances of your Lambda function.
What’s new: REFERENCE mode
This launch introduced a new function configuration setting, S3ObjectStorageMode. The default value is COPY, which provides the existing behavior described in the preceding section. To enable self-managed S3 buckets, set S3ObjectStorageMode to REFERENCE when creating or updating a function. In this mode, Lambda no longer copies your deployment package. Instead, it stores a reference to your S3 object and reads the code directly from your bucket when needed. If you do not specify S3ObjectStorageMode, Lambda still takes a copy by default.
Figure 2 — Lambda deployment with self-managed storage
This gives you:
No quota consumption. Deployment packages in your bucket don’t count against the 75 GB Function and layer storage account limit.
Improved performance. Lambda no longer copies the code to an internal bucket, so function creation and updates are faster.
Full security and compliance control. Apply your own bucket policies, encryption, Object Lock, versioning, access logging, and compliance tags.
Single source of truth. Your S3 object is the canonical artifact with no additional copies and no drift.
Disaster recovery options. Use S3 Cross-Region Replication to maintain fallback copies in a secondary Region.
How it works
To use this feature, specify the S3ObjectStorageMode parameter when creating or updating your function.
We recommend including the aws:SourceArn condition key scoped to your specific function ARN to allow for least-privileged access. Note the Resource is scoped to the exact S3 key rather than a wildcard prefix. This follows least-privilege and matches how the aws:SourceArn condition locks down which function can access which object.
Bucket requirements
Your S3 bucket must meet the following requirements:
Versioning (required). You must enable S3 versioning to make sure that Lambda references a specific, immutable artifact and to protect against accidental overwrites.
Encryption. The following encryption types are supported: SSE-S3, SSE-KMS (including customer-managed KMS keys), and DSSE-KMS. If you use SSE-KMS, the Lambda principal must have kms:Decrypt permission on the key.
Object Lock. Supported. You can apply Object Lock in Compliance or Governance mode to prevent accidental deletion of deployment artifacts.
Access logging. You can enable S3 server access logging or AWS CloudTrail data events to audit every time Lambda reads your code.
What happens when the object is unavailable
Lambda periodically accesses the source object from your S3 bucket to reoptimize your function code. You must maintain access to the source object for your function to remain active.
If Lambda loses access to the source object for a function, the function transitions to the Inactive state. To restore the function, restore access to the source object and then update the function.
Performance considerations
Lambda functions with self-managed code storage behave the same as standard Lambda functions with one difference during function creation and update. Lambda does not copy your deployment package to a Lambda-managed S3 bucket. In our testing with a 200MB Python 3.13 function, functions using self-managed storage showed function creation times approximately 5s less than the default COPY mode. Reading directly from your S3 bucket without an intermediate copy step can provide a modest advantage, particularly for larger deployment packages.
Getting started with infrastructure as code
Self-managed code storage can be implemented using infrastructure-as-code tooling with either the AWS CLI or AWS CloudFormation today.
Once you adopt self-managed S3 buckets for your Lambda deployment packages, your artifact bucket grows over time as you deploy new versions of your functions. This section covers strategies for managing that growth efficiently, keeping your versions organized, and planning for cross-Region deployments.
Managing artifact lifecycle with S3 lifecycle policies
Every time you update a function’s code, S3 creates a new object version in your bucket. The previous objects don’t disappear. They accumulate. Without a cleanup strategy, your storage grows indefinitely and old artifacts clutter your bucket.
S3 Lifecycle policies let you automate this entirely. You define rules that transition or delete objects based on age, and S3 executes them on your behalf: no scripts, no cron jobs, no manual intervention.
Strategy 1: Archive old versions to Glacier
If compliance or audit requirements mandate that you retain all historical deployment packages, but you rarely need to access them, transition old object versions to a lower-cost storage class:
This rule transitions any non-current object version to S3 Glacier Flexible Retrieval after 30 days. Your active deployment packages remain in S3 Standard for fast access, while historical versions move to archival storage at a fraction of the cost.
For artifacts you need to retain for years but will rarely access again, consider a tiered approach: moving to Glacier Flexible Retrieval first, then to Deep Archive:
This rule keeps the 2 most recent non-current versions of each object (giving you a rollback path) and deletes anything older than 14 days beyond that. This aligns well with a deployment strategy where you want the ability to quickly roll back to your previous one or two releases, but don’t need to retain anything older.
Tracking object and function versions
With REFERENCE mode, there is a direct relationship between your S3 object version and your Lambda function version. We recommend the following practices:
Tag your objects with metadata from your CI/CD pipeline (commit SHA, build ID, pipeline run ID) so you can trace any deployed function back to the exact source that produced it.
Document the mapping between Lambda function versions (or aliases) and S3 object version IDs. This makes rollbacks straightforward: update the function to reference the previous object version.
Cross-account considerations
How you organize your artifact buckets across AWS accounts depends on your operational model:
Centralized artifact account: A single bucket in a shared-services or tooling account, with bucket policies granting cross-account s3:GetObject access to Lambda execution roles in workload accounts. This gives your platform team a single inventory of all deployment artifacts with consistent lifecycle, encryption, and access policies.
Per-account buckets: Each workload account owns its own artifact bucket. Requires less effort to set up, but harder to enforce consistent governance across many accounts.
Either pattern works with self-managed storage. Choose based on how your organization balances centralized control against team autonomy.
Cross-Region considerations
With REFERENCE mode, your S3 object is the authoritative copy for your function. Self-managed code storage supports cross-Region function creation within a partition for all default Regions (non-opt-in Regions). You can store your code packages in one Region and deploy your functions in another. This makes cross-Region planning critical. There are four factors to balance:
Disaster recovery
This is the most critical consideration. Because your S3 object is the single source of truth in REFERENCE mode, you do not want all your deployment artifacts in a single Region with no fallback. A recommended pattern:
Primary Region: Your main artifact bucket where CI/CD pipelines deposit new deployment packages.
Fallback Region: A secondary bucket populated via S3 Cross-Region Replication (CRR). If your primary Region becomes unavailable, you can update your Lambda functions to reference the replicated objects in the fallback Region.
Weigh replication + storage costs against per-deploy cross-Region data transfer. If you deploy frequently, storing replicated copies in each target Region is usually cheaper. For infrequent deployments, a one-time transfer may suffice.
Governance and data residency
Some organizations, particularly in regulated industries, have strict requirements about where code artifacts can reside. Before configuring cross-Region replication, confirm that your data is permitted to leave its current Region. Certain regulatory frameworks (for example, data sovereignty laws, FedRAMP boundaries) may restrict replication to specific Region pairs.
Performance
If your workload requires fast function creation and activation times, for example, in a CI/CD pipeline where deployment speed is critical, keep your S3 objects in the same Region where you are creating your Lambda functions. Cross-Region reads add latency to the initial code download, which directly impacts how quickly a new function version becomes active after deployment.
For workloads where creation speed is less critical (for example, batch processing functions that are updated infrequently), the latency of a cross-Region read might be acceptable and can simplify your architecture.
Things to know
Before adopting self-managed S3 buckets for your Lambda functions, keep the following in mind:
Availability: You can use this feature today in all AWS standard regions where Lambda is supported.
Pricing: There is no additional Lambda charge. You pay standard S3 costs for storage, and any cross-Region data transfer.
Maximum deployment package size: The existing limits apply: 250 MB unzipped.
Supported runtimes: All Lambda runtimes that support .zip deployment packages are compatible. Container image deployments are not affected by this feature.
Migration: You can switch an existing function from service-managed to self-managed storage by calling UpdateFunctionCode with the --s3-object-storage-mode REFERENCE parameter. Lambda recreates the function by referencing the object in your S3 bucket and deletes the saved copy.
Reverting: You can switch back to service-managed storage at any time by updating the function with --s3-object-storage-mode COPY. Lambda resumes copying the artifact to its internal bucket.
Object availability is your responsibility: In REFERENCE mode, Lambda depends on your S3 object being accessible. If the object is deleted, the bucket policy changes, or the KMS key is disabled, new invocations requiring a cold start will fail.
Conclusion
In this post, we showed how self-managed S3 buckets for Lambda give you more capacity, more control, and simpler compliance, all without changing how you write or invoke your functions. Your deployment packages no longer count against account quotas, your security team can apply the same policies to code artifacts that they apply everywhere else, and your disaster recovery story is as strong as the replication capabilities of S3.
Companies increasingly need to query and analyze data across platforms without the cost and complexity of moving it. Salesforce and AWS have collaborated to make this possible by providing Zero Copy access to Apache Iceberg tables stored in Amazon Simple Storage Service (Amazon S3) directly from Salesforce Data 360, using the Iceberg REST endpoint from AWS Glue Data Catalog with data access managed by AWS Lake Formation. This integration helps customers federate their Amazon S3 data lakes with Data 360, preserving data governance, freshness, and business semantics without replication.
Zero Copy file federation plays an important role in activating applications and experiences. By removing the need to physically move or copy data, and connecting to data at the storage level, it addresses key challenges including:
Cost efficiency – Reduce storage duplication costs and minimize the compute resources required for data pipelines.
High scale – Access data with near-native performance at scale through in-Region access.
Enhanced agility – Access and analyze data in real time, accelerating time-to-insight and supporting faster response to evolving business needs.
Streamlined operations – Remove the complexity of building and maintaining intricate data pipelines, clearing up valuable data engineering resources.
In this post, we demonstrate how AWS and Salesforce customers can access their enterprise data lakes on AWS from Data 360 using Zero Copy file federation.
What is Data 360?
Data 360 is the real-time data engine that activates trusted context across the entire Salesforce platform. It connects all your enterprise data — data warehouses, data lakes, third-party signals, and more — to the business context, logic, and governance that already live in Salesforce, without moving or copying it. With Zero Copy federation, your teams and AI agents always operate from a complete, current, and trusted picture of your business in the moment it’s needed. It serves as the essential system of context for Agentforce, enabling agents to reliably get real work done.
What is Apache Iceberg?
Apache Iceberg is a high-performance, open table format for huge analytic datasets that brings the reliability and simplicity of SQL tables to big data. It’s a thriving open source project under the Apache Software Foundation. Data engineers use Apache Iceberg because it’s fast, efficient, and reliable at any scale and keeps records of how datasets change over time. Apache Iceberg offers integrations with popular data processing frameworks such as Apache Spark, Apache Flink, Apache Hive, Presto, and more.
Why Amazon S3 for Apache Iceberg data lakes?
Amazon S3 is regarded as the best place to build data lakes because of its durability, availability, scalability, security, compliance, and audit capabilities, and its ability to integrate with a broad portfolio of AWS and third-party tools for data ingestion and processing. Apache Iceberg was designed and built to interact with Amazon S3, and provides support for many Amazon S3 features as listed in the Iceberg documentation.
What is Zero Copy file federation?
File federation, also termed catalog federation, uses the Data Catalog to communicate with remote catalog systems to discover catalog objects and to authorize access to their data in Amazon S3. When you query a remote Iceberg table, the Data Catalog discovers the latest table information in the remote catalog at query runtime, getting the table’s Amazon S3 location, current schema, and partition information. Your analytics engine then uses this information to access Iceberg data files directly from Amazon S3, and Lake Formation manages access to the table and data by vending scoped credentials to the table data stored in Amazon S3. This approach avoids metadata and data duplication while providing real-time access to remote Iceberg tables through your preferred AWS analytics engines.
Solution overview
Apache Iceberg file federation lets Data 360 directly query data stored in Amazon S3 without copying or moving the data. This Zero Copy approach provides several benefits:
Real-time access to Amazon S3 data from Salesforce.
Reduced data movement and storage costs.
Simplified data architecture.
Improved data freshness.
The following diagram illustrates the architecture of the integration between Data 360 and Amazon S3 using Apache Iceberg file federation.
Key components:
Amazon S3 stores the source data in Apache Iceberg format.
AWS Glue Data Catalog maintains the metadata for Iceberg tables.
AWS Lake Formation manages metadata and underlying data access for Amazon S3-based data lakes.
Data 360 processes and analyzes the data.
Apache Iceberg connector provides direct access to query Amazon S3 data from Salesforce.
Walkthrough
The following walkthrough shows you how to set up Zero Copy file federation.
Prerequisites
Before you begin, you need the following:
An AWS account.
A data-lake-admin role following the steps in Lake Formation personas and IAM permissions reference, with access to AWS Identity and Access Management (IAM), AWS Glue Data Catalog, AWS Lake Formation, Amazon S3, and Amazon Athena.
Sign in as the data lake admin and complete the following steps:
Open the Amazon S3 console.
Choose Create bucket to create a bucket.
For Bucket type, choose General purpose, provide a Bucket name, and choose Create bucket.
In the bucket, create two prefixes by choosing Create folder.
Name the prefixes athena_iceberg and athena_results.
Inside the athena_iceberg prefix, create another prefix named customer_iceberg.
Create an Iceberg table using Athena
Open the Amazon Athena console.
Choose Query your data in Athena console, then choose Launch query editor.
In Athena, choose Edit settings.
Set s3://<your-bucket-name>/athena_results/ as the Location of query result, then choose Save. Replace <your-bucket-name> with your bucket name.
Choose Editor to return to the query editor page.
To create the database, copy the following query into the query editor and choose Run. You need to be in the Athena Query Editor to run the following commands.
create database iceberg_db;
To create the Iceberg table, copy the following query into the query editor, replace <s3 bucket location> with your Amazon S3 bucket location hosting the Iceberg table, and choose Run.
In Data Cloud, choose Setup, then choose Data Cloud Setup.
Under External Integrations, choose Other Connectors.
Choose New.
On the Source tab, choose AWS Glue Data Catalog, then choose Next.
Complete the following information shown in the following screen:
In the Authentication Details section, enter the AWS access key ID and AWS secret access key for the IAM user. Make sure that the IAM user has a policy that grants the user read-only access to AWS Glue Data Catalog. Use Lake Formation to configure storage credential vending. This approach is for AWS Glue Data Catalog to vend temporary credentials at run time so that Data 360 can access the underlying storage bucket.
For Catalog ID, enter the 12-digit AWS account ID linked to AWS Glue Data Catalog.
For Signing Region, enter the host AWS Region where AWS Glue Data Catalog is located.
For Signing Service, enter glue. Data 360 requires the Signing Service, in addition to the AWS access key ID, secret access key, and Signing Region, to sign requests to AWS Glue Data Catalog by using AWS Signature Version 4.
Test the connection and check for the success message.
Save the connection details.
After the configuration is complete and saved, the new AWS Glue Data Catalog connection shows up with “Active” status in the Connectors screen.
Create and configure the data stream
In Data Cloud, on the Data Streams tab, choose New.
Under Other Sources, choose the AWS Glue Data Catalog source, then choose Next.
From the menus, choose the connection that you just set up, choose a database in your AWS Glue catalog where you have an Iceberg table, choose the table that you want to stream, and choose Next.
Choose the category to specify the type of data to ingest. For more information, see Category.
Choose a primary key to uniquely identify the incoming records. For more information, see Primary Key.
Choose the source fields you want to ingest, then choose Next. Fields with convertible data types are listed under Supported Fields.
Choose the relevant data space. Choose “Default” if you don’t have any other data space provisioned in your org. For more information, see Data Spaces.
Choose Deploy.
After the setup is complete, the new data stream appears in your Data Cloud environment.
The data stream is ready. You can now go to the Data Explorer in your Data Cloud environment and start viewing the Iceberg tables that reside in your external AWS account.
Best practices and considerations
Use IAM roles with least-privilege access. Grant only the specific permissions each service or user needs.
Implement appropriate Amazon S3 bucket policies. Define bucket-level policies that restrict access by AWS account, VPC endpoint, or IP range.
Monitor access patterns. Enable Amazon S3 server access logging or AWS CloudTrail data events to track who reads from and writes to your table buckets.
Optimize Iceberg table partitioning. Choose partition keys that align with your most common query filters.
Consider data access patterns. Design your table layout around how data is actually queried.
Implement lifecycle policies for Amazon S3 objects. Configure Amazon S3 lifecycle rules to transition older data files to other storage classes.
Use appropriate Iceberg file compaction strategies. Run compaction regularly to merge small files produced by streaming or frequent batch appends.
Monitor data transfer costs. Track cross-Region and internet egress charges using AWS Cost Explorer as applicable.
Clean up
After you finish testing, clean up all the resources in your AWS account that you created (including the Amazon S3 bucket, Athena tables, and other AWS services) to avoid recurring costs.
Conclusion
By implementing Apache Iceberg file federation between Data 360 and Amazon S3, you can create a more efficient and streamlined data architecture. This solution gives you real-time access to Amazon S3 data while using the analytics capabilities of Data 360. As businesses continue to prioritize data-driven decision-making, Zero Copy data sharing plays an important role in unlocking the full potential of customer data across platforms.
Amazon Redshift continuously innovates to deliver improved performance and advanced features. In some releases, Amazon Redshift patches might introduce behavior changes. Testing patches in a non-production environment confirms that production workloads continue to function and you can maintain your applications’ service level agreements. As a best practice, keep Dev/QA clusters on the Current patch track and Production on the Trailing track. Test on Dev/QA when a patch lands, allowing 1–6 weeks of review before the scheduled production deployment.
In this post, we demonstrate an automated test suite that validates your Amazon Redshift cluster automatically after any patch, reboot, or modification. It uses standard drivers against real workload patterns to provide a verified gate between a patch landing and that patch reaching production.
Architecture
The solution uses native AWS services to create an automated validation pipeline.
Test Execution: A Docker container runs a comprehensive test suite in four phases:
JDBC Driver Tests – Validates the official Amazon Redshift JDBC driver, testing DatabaseMetaData API calls, connection handling, and queries that tools like SQL Workbench/J depend on.
ODBC Driver Tests – Validates the PostgreSQL ODBC driver with SQLTables, SQLColumns, and other ODBC API calls that RStudio and similar tools use.
Catalog SQL Queries – Runs approximately 35 queries against pg_catalog, information_schema, and svv_* views, organized by client (SQL Workbench, DBeaver, RStudio, JDBC metadata API).
Performance Benchmarks – Executes your custom workload queries and compares execution time against known baselines, flagging regressions. For convenience, the solution includes sample queries to be replaced with performance validation queries from your workloads.
Reporting: Detailed JSON results land in Amazon Simple Storage Service (Amazon S3) for historical analysis. An Amazon Simple Notification Service (Amazon SNS) notification sends your team an email immediately with a pass/fail summary. Full JSON results are written to Amazon S3 with timing data for every individual query, row counts, error details, and the Amazon EventBridge event that triggered the run. If tests fail, you have specific, actionable evidence (which queries broke, which drivers failed, which benchmarks regressed) to open a support case requesting a rollback and defer maintenance until the case is resolved. When tests succeed, you can move forward with confidence to production.
For real-time feedback while the tests are running, a quick command tells you the current state:
The test suite covers two critical areas: client tool compatibility and query performance.
Client compatibility queries
The test suite replicates the connection behavior of popular SQL clients by issuing the same metadata API calls and queries they perform when connecting to your cluster.
ODBC-specific catalog queries, column type mapping
JDBC Metadata API
getTables(), getColumns(), getPrimaryKeys(), and other DatabaseMetaData method equivalents
The package contains the exact queries these clients execute upon connection.
Performance regression detection
The benchmark phase of the suite automatically detects whether it has been run before. On the first execution, it captures baseline query execution times as the “known good” state for your pre-patch environment. On every subsequent run, it compares current query timings against the stored baseline and flags any regressions. If a query that previously completed in 2 seconds now takes 15, the report calls it out immediately. This phase is designed to test your most performance-sensitive queries.
Prerequisites
Before deploying, make sure your environment meets the following requirements:
Amazon Redshift cluster reachable from the private subnets.
An AWS Secrets Manager secret with your Amazon Redshift credentials (created automatically with managed clusters).
Docker installed. Consider building the image with AWS CloudShell, which comes with Docker pre-installed. You can do this either by uploading the customized repo to Amazon S3 and then downloading it to AWS CloudShell, or by cloning and customizing the repo directly within AWS CloudShell.
Getting started
The full solution is available on GitHub. It includes the AWS CloudFormation template, Docker build scripts, test suite, and documentation.
Clone the GitHub repo, customize it for your workload, deploy it against a Dev/QA cluster.
Detailed instructions are included in the package README.md. Reference those for deployment.
Step 2: Customize the scripts for your environment
The test suite ships with comprehensive default queries. After cloning and before deployment, edit the scripts as described in the following sections for each phase.
Add your performance-critical queries
Edit bundle/run_tests.py and replace the example queries with queries where performance is critical:
BENCHMARK_QUERIES = {
"daily_patient_summary": """
SELECT department, COUNT(DISTINCT patient_id), AVG(los_days)
FROM clinical.encounters
WHERE admit_date >= CURRENT_DATE - 30
GROUP BY 1
""",
"revenue_rollup": """
SELECT payer_type, SUM(total_charges)
FROM billing.claims
WHERE service_date >= DATE_TRUNC('month', CURRENT_DATE)
GROUP BY 1
""",
}
Add client-specific catalog queries
If your team uses custom views or schemas, add them to bundle/client_catalog_queries.py:
"custom_view_check": {
"description": "Verify our reporting view works after patching",
"sql": "SELECT * FROM analytics.monthly_kpis LIMIT 10",
},
Step 3: Build the Docker image
Execute build-image.sh, which creates an Amazon ECR repository, builds the Docker image (with JDBC and ODBC drivers bundled), and pushes it, outputting the image URI for the next step.
# Upload project to S3, then build in CloudShell
./build-image.sh --stack-name my-redshift-tests
Step 4: Deploy the stack
Use the AWS Command Line Interface (AWS CLI) to deploy the AWS CloudFormation stack with your environment-specific parameters. The stack creates the required components: Amazon Elastic Container Service (Amazon ECS) cluster, AWS Fargate task definition, security groups, VPC endpoints (to keep AWS Secrets Manager and Amazon SNS traffic off the NAT gateway), Amazon S3 bucket, Amazon SNS topic, AWS Lambda trigger, and Amazon EventBridge rules.
Here are the core principles that make automated patch testing effective:
Dev/QA on Current track, Production on Trailing: This separation creates the buffer window between when a patch is available and when it reaches production. Without it, there’s no opportunity to catch regressions before they affect users.
Automate the validation: The track split is most effective if the test suite runs after every patch. Event-driven automation helps confirm no patch goes untested during the buffer window.
Test with real drivers: Simulated queries aren’t sufficient. The test suite exercises the Amazon Redshift JDBC and PostgreSQL ODBC drivers that your SQL clients depend on. This validates the same code paths your tools use in production.
Event-driven, not scheduled: Tests run the moment a patch is applied. They don’t run on a fixed cron schedule. Patch applied, then test executed, then results delivered in minutes.
Low operational overhead, minimal cost: The entire solution is serverless (AWS Lambda and AWS Fargate). There are no instances to manage and no agents to install. The Fargate task spins up only when a patch event fires, runs the test suite, and shuts down. You pay only for the compute each test run consumes.
Clean up
When you no longer need the automated test suite, delete the associated resources so you don’t incur ongoing costs.
Delete any created prerequisites, if not needed.
Amazon Redshift cluster (removes the managed secret).
Automated patch testing ensures consistent and predictable performance of your production workloads. By deploying Dev/QA clusters on the Current track with event-driven validation, you gain weeks of advance notice before patches reach production. The solution presented here provides comprehensive testing of JDBC drivers, ODBC drivers, catalog queries, and performance benchmarks. It requires zero manual intervention. Deploy it once, customize it for your workload, and gain confidence that the next Amazon Redshift patch will be validated before it matters.
To learn more about Amazon Redshift, explore the following resources:
Enterprise video surveillance is operating at an unprecedented scale as organizations across retail, banking, quick-service restaurants (QSR), convenience stores, and transportation networks generate petabytes of video data across thousands of distributed locations. As retention requirements grow and organizations seek to extract more operational insights from video, traditional on-premise storage models are becoming increasingly difficult and expensive to scale.
March Networks is a global provider of intelligent video surveillance and business intelligence solutions serving enterprises across banking, retail, quick-service restaurants, transportation, and other multi-site environments. With more than 25 years of experience in video technology, the company helps organizations transform video data into operational insights through cloud-based platforms, AI-powered analytics, and enterprise-scale video management.
Unlocking the power of video data
In this post, we show how March Networks built a scalable cloud architecture on Amazon Web Services (AWS) to support large-scale enterprise video storage and analytics. The solution uses Amazon Simple Storage Service (Amazon S3) and Amazon S3 Glacier to manage long-term video retention, while integrating with additional AWS services to support ingestion, lifecycle management, monitoring, and secure access. We also explore how this architecture enables advanced video analytics using technologies such as Amazon S3 Vectors and Amazon Bedrock, helping organizations store petabyte-scale video data more cost-effectively while accelerating investigations and operational insights.
The challenge: Managing enterprise video at scale
Historically, enterprise video has been stored on local network video recorders (NVRs) and on-premise servers deployed at each site. Although this model provides localized control, it creates fragmented storage environments that require frequent hardware expansion, ongoing maintenance, and inconsistent retention policies across locations. This also limits organizations’ ability to centrally access, analyze, and govern video data across their enterprise.
As organizations increase video retention periods for compliance, liability protection, and operational intelligence, infrastructure requirements grow rapidly. Adding local storage hardware across hundreds or thousands of sites increases operational complexity and introduces lifecycle management challenges.
The economic impact of cloud video storage
Cloud storage introduces a more flexible model by consolidating distributed video data into centralized, elastic storage infrastructure. Even partial migration (such as moving long-term retention or compliance archives to the cloud), can significantly reduce infrastructure overhead while enabling centralized data management and analytics.
The financial impact of this shift can be substantial. For example, one retail organization evaluated the benefit of moving to a hybrid cloud storage model to extend video retention for a period of up to 5 years — a common retention window driven by compliance standards and laws — without adding new on-premise hardware. This customer operated more than 580 cameras, generating approximately 5,600 TB of archived video. The total storage required depends on factors such as video bitrate and quality, camera count, and backup duration. Their estimated cloud storage cost using a third-party cloud provider was approximately $347,000 per year, compared to roughly $1.7 million annually to store the same volume of video on-premise. For long-term cloud storage, data is not expired or deleted; customers are notified as their storage quota approaches capacity and can purchase additional storage as needed. By retaining recent footage locally while archiving older video to a third-party cloud provider, the organization significantly reduced storage costs while maintaining access to archived footage when needed.
Solution overview: March Networks cloud storage on AWS
March Networks Cloud Storage is a cloud-based video storage solution built on AWS. It is designed for distributed enterprise environments such as retail chains, financial institutions, convenience stores, and transportation systems that operate thousands of cameras across geographically dispersed locations.
The solution leverages Amazon S3 and Amazon S3 Glacier to provide scalable and durable storage for large volumes of video data while integrating AWS services that support secure ingestion, lifecycle management, monitoring, and access control. By combining AWS cloud infrastructure with March Networks’ video surveillance expertise, organizations can modernize video retention strategies while maintaining operational flexibility.
The platform supports multiple deployment models that allow organizations to adopt cloud storage at their own pace. Hybrid architectures allow recent footage to remain on-site for immediate access while older video is archived to the cloud. In other deployments, organizations can move a majority of video storage into AWS to reduce on-premise infrastructure and simplify long-term retention management.
Because the platform is built on AWS, storage capacity scales automatically as organizations add cameras, extend retention periods, or onboard new sites. This allows customers to grow video storage environments without hardware planning, or infrastructure expansion.
Architecture deep dive
The Cloud Storage architecture integrates on-premise video infrastructure with AWS services that manage ingestion, storage, monitoring, and secure access to video data.
At a high level, the architecture connects local video systems, including NVRs, cameras, and client applications, to AWS cloud services through secure network connections. March Networks securely ingests video data into AWS storage infrastructure, where customers can retain, monitor, and retrieve it based on their defined policies.
Figure 1: March Networks Architecture on AWS.
Video ingestion and storage
March Networks securely uploads video recorded on local NVRs to Amazon S3 buckets using encrypted transmission protocols. Amazon S3 provides highly durable object storage designed to store large volumes of data while enabling efficient retrieval and lifecycle management.
Once stored, organizations can retain video data for active investigations or operational review. Organizations configure lifecycle management policies that automatically move older footage to lower-cost storage tiers based on their access patterns.
Tiered storage with Amazon S3 and Amazon S3 Glacier
Video storage requirements vary depending on how frequently footage must be accessed. The platform uses multiple Amazon S3 tiers to align performance and cost with real-world video access patterns.
Amazon S3 Standard and Amazon S3 Standard-Infrequent Access (S3 Standard-IA) support video that must remain readily accessible for investigations, operational review, or analytics. For long-term retention, the platform uses Amazon S3 Glacier storage tiers to provide ultra-low-cost archival storage for footage that must be preserved but is rarely accessed.
Lifecycle policies automatically transition videos between tiers according to customer-defined retention policies. This allows organizations to store high-value recent video on high-performance storage while archiving older footage economically.
Supporting AWS services
Several AWS services support the reliability, scalability, and operational visibility of the platform:
Amazon Simple Queue Service (Amazon SQS) manages asynchronous messaging between system components, enabling reliable communication between ingestion, processing, and storage services.
Amazon Simple Email Service (Amazon SES) provides notification capabilities for operational alerts and system events.
Amazon CloudWatch monitors system performance, logs activity, and provides operational visibility into cloud infrastructure.
AWS Security Token Service (AWS STS) enables secure authentication and temporary credentials for system components accessing cloud resources.
For metadata management and caching, the platform uses PostgreSQL and Amazon ElastiCache for Redis to maintain high-performance access to video metadata and system state.
Together, these services enable March Networks to deliver a secure, scalable cloud architecture capable of supporting petabyte-scale video workloads across distributed environments.
Outcomes and benefits
By building its video storage architecture on AWS, March Networks enables organizations to modernize video infrastructure while reducing operational complexity and long-term storage costs. This includes:
Reduced storage costs
Tiered storage using Amazon S3 and Amazon S3 Glacier allows organizations to align storage costs with actual video access patterns. Frequently accessed footage remains readily available, while older video can be archived at significantly lower cost.
Elastic scalability
AWS infrastructure enables organizations to scale video storage across hundreds or thousands of locations without adding on-premise hardware. As organizations add cameras or extend retention periods, storage capacity expands automatically.
Centralized investigations and governance
Cloud-based video storage enables security and operations teams to investigate incidents across multiple sites using a centralized platform. Organizations can apply consistent retention policies, maintain audit trails, and enforce standardized governance across all locations.
AI-driven analytics and search
Centralized video storage also enables advanced analytics capabilities. March Networks integrates AI-powered tools such as AI Smart Search, which allows users to locate relevant footage using natural-language queries across large video archives.
These capabilities leverage technologies, including Amazon S3 Vectors and Amazon Bedrock to support semantic search and AI-driven video intelligence across enterprise-scale datasets.
Conclusion
As organizations generate increasing volumes of video data, scalable cloud infrastructure becomes essential for managing long-term storage and enabling advanced analytics. By building its Cloud Storage platform on AWS, March Networks provides organizations with a durable, secure, and cost-efficient foundation for enterprise video retention.
Services such as Amazon S3, Amazon S3 Glacier, Amazon SQS, Amazon CloudWatch, and AWS Security Token Service support a scalable architecture capable of storing and managing petabytes of video data across distributed environments. This cloud-native approach allows organizations to modernize video infrastructure today while preparing for future AI-driven analytics and operational intelligence.
Specification-driven composition addresses a common scalability bottleneck in data pipelines. Data pipelines often start as simple scripts, but as they grow, you duplicate transformation logic and small changes cascade across multiple workflows. Copying and modifying data transformation logic across scripts leads to workflows that become difficult to manage at scale. Tracking what each pipeline does becomes harder because workflow intent is embedded in code. This lack of visibility complicates governance, especially in regulated environments such as healthcare, finance, and life sciences.
Many implementations combine orchestration, transformation logic, and validation rules in the same scripts. Supporting new datasets requires modifying and redeploying code, while validation often happens only during processing. As a result, issues surface late in the lifecycle which increases the operational risk. Specification-driven composition separates workflow intent from implementation so you can build flexible data workflows.
You can separate workflow intent from processing logic with specification-driven composition. This approach reduces duplication, shortens the time required to onboard new datasets, and improves consistency across workflows. Instead of embedding logic in scripts, the system describes workflow intent in a structured specification, validates the specification before processing, and dynamically assembles a processing pipeline.
This approach moves pipeline configuration outside application code and composes pipelines from reusable processing components. To separate concerns, it organizes a workflow into three layers, as shown in Figure 1. The intent layer defines workflow behavior using specifications. The composition layer validates specifications and assembles pipelines. The processing layer runs the pipeline of transformation steps.
Specification-driven composition provides several practical benefits when you manage multiple pipelines. First, it improves governance because specifications provide a clear, traceable description of workflow behavior that you can review and validate before invocation. In regulated industries, this traceability shortens audit preparation time and reduces the review burden for new dataset submissions.
Second, the pattern supports reusable transformations. You implement transformation logic once and reuse it across multiple workflows. This reduces duplication and improves consistency. In practice, teams adopting this pattern report being able to onboard new datasets in days rather than weeks because most required capabilities already exist in the registry.
Third, specification-driven composition enables flexible pipeline design. You define new specifications to create pipelines supporting new datasets and use cases without modifying application code and registering new system release.
Fourth, the pattern separates business intent from execution artifacts. The specification expresses what the workflow should do in domain terms, while the generated state machine remains a system artifact. This separation matters in regulated environments (for example, GxP) where business users author intent but are not allowed to author or modify execution code directly.
Finally, the declarative representation of workflows lets AI tools assist with capability discovery, specification authoring, and pipeline analysis, while runtime behavior stays predictable as it relies on validated capabilities.
Core components
You work with four key components to define, validate, and run workflows.
Specification
A specification is a structured document, typically JSON or YAML, that describes datasets, mappings, and transformations. It defines what the workflow should do without including processing logic. Because specifications are explicit and versioned, they provide a clear record of workflow intent.
Composer
The composer converts specifications into runnable steps, so workflows run consistently without embedding transformation logic in application code. It checks that referenced capabilities exist, retrieves metadata, and builds a workflow that can run on the processing layer. The composer does not perform transformations. It only assembles the workflow. In practice, the composer compiles business intent expressed in the specification into a code artifact such as an Amazon States Language (ASL) definition. This abstraction lets domain users author specifications without producing runnable code, which is important in environments with strict separation of duties.
Capability registry
The registry stores metadata about reusable transformation functions. This includes identifiers, input/output formats, invocation details, and permission boundaries. The composer uses the registry to validate specifications and locate capabilities. Treat this registry as a governed artifact rather than a manually edited lookup table. Capability definitions live in version control, and your CI/CD pipeline validates metadata and runs tests before you publish new versions. Specification authors include explicit capability version references in the specifications to support reproducible workflow runs. In regulated systems, you must validate new capabilities and obtain approval through a separate workflow before you register them.
Capability pipeline
Once assembled, the pipeline runs a sequence of transformation steps. Each step performs a specific operation such as formatting, validation, or enrichment. Because these steps are reusable, you can apply them across many workflows.
Technical implementation
Let’s walk through a technical implementation of this pattern using serverless AWS services. In this example, workflow specifications are uploaded to an S3 bucket. An AWS Lambda composer retrieves and validates the specification, looking up capability metadata in Amazon OpenSearch Service. The composer then assembles the workflow in AWS Step Functions, which orchestrates the workflow and invokes AWS Lambda capability processors. Each processor emits traces to Amazon CloudWatch Logs.
Figure 2 shows the architecture.
Figure 2. AWS implementation of Specification-Driven Composition
Interpreting the workflow specification
The workflow specification defines datasets and transformation logic using a structured format (in this example, JSON). A specification is a declarative document that describes what the pipeline should produce rather than how to produce it. Your composer reads the specification, validates it against a schema, and uses it to construct the pipeline.
The following example maps source fields to target fields using reusable capabilities (Figure 3). It takes data from a source dataset (‘raw_orders’), maps specific fields (‘order_date’, ‘amount’) and applies reusable transformation capabilities such as ‘format_date’ and ‘normalize_currency’. Each mapping explicitly links a source field to a target field and references a capability that performs the transformation. Source dataset(s) are listed under the ‘source’ section of the JSON document, target datasets under the ‘target’ section, and mappings ‘mappings’. You can define your own specification structure and build custom validation logic in your composer to make sure specifications are valid.
You can model preprocessing steps such as column standardization, numeric casting, or unit conversion as first-class capabilities and reference them earlier in the specification (for example, in a dedicated ‘preprocessing’ section) before downstream mappings run. This way, preparation logic uses the same metadata, versioning, validation, and observability model as the rest of the workflow, which simplifies lineage and review.
In this example, an S3 event notification invokes the composer Lambda function when you upload a specification. In practice, the same composer can be invoked through several mechanisms depending on the use case: S3 events for new specifications, an Amazon EventBridge schedule for recurring runs, an API or UI action for on-demand invocation, a direct Step Functions StartExecution call, or an upstream pipeline. This flexibility lets you re-run an approved specification against refreshed data without re-authoring or re-approving the workflow.
The composer parses the specification and validates the referenced capabilities by querying Amazon OpenSearch Service to retrieve capability metadata such as Amazon Resource Names (ARNs). OpenSearch Service is used here because the composer does more than direct-key lookups. It supports capability discovery through full-text and semantic search over capability metadata such as capability descriptions, input/output schemas, and tags. This lets data authors and AI tools find reusable capabilities by intent rather than by exact identifier. After validation, the composer assembles and starts an AWS Step Functions state machine which invokes each capability in sequence. Each capability runs independently, making the pipeline modular and reusable.
Securing sensitive data flows
This pattern suits regulated workloads, so handle security as part of the design. Use SSE-KMS with a customer managed key on the specification and data S3 buckets and enable encryption at rest on the Amazon OpenSearch Service domain. Enforce HTTPS (TLS) access with an S3 bucket policy (aws:SecureTransport) and enable node-to-node encryption on Amazon OpenSearch Service. AWS Step Functions and Lambda calls use TLS by default. Each capability processor receives only the fields its mapping references, and you can apply IAM policy at the source bucket to restrict access. Processors emit traces to Amazon CloudWatch Logs.
For data classification, you can tag sensitive fields in the specification (example: "sensitivity": "PHI") and declare the sensitivity of each capability in the registry: a direct-move capability preserves the source classification, a date-of-birth-to-age capability clears it, and an enrichment that introduces sensitive data sets it. The composer combines the source tag with the capability’s behavior to derive the target field’s sensitivity, validating that the combination resolves clearly before assembling the pipeline. It then generates the masking artifact for the output (for example, AWS Lake Formation column grants) so consumers get correct masking without a separate masking specification or manual effort.
Recognizing the pattern
This approach works well when you describe workflows using structured specifications, reuse transformation logic across pipelines, require validation before invocation, and require workflows to be deterministic and auditable. You are likely to recognize the pattern in regulated data pipelines for the submission of tabular datasets to oversight agencies.
Consider clinical trial reporting as a concrete example. Data analysts collect raw data from clinical sites and must transform it into a standard submission format such as the Study Data Tabulation Model (SDTM) before submission to agencies such as the US Food and Drug Administration. With specification-driven composition, data analysts define specifications that map collected data about adverse events, demographics, and vital signs to the standard target variables, and the validated output feeds downstream systems such as patient safety and medical monitoring.
That said, this pattern might add unnecessary complexity for simple, one-time data transformations or pipelines with fewer than three to five workflows. Evaluate this pattern’s impact by tracking the reduction in duplicated transformation logic across pipelines, the time required to onboard new datasets, and the number of workflows you can create without modifying application code.
Conclusion
Script-based data pipelines accumulate hidden costs as they grow, including duplicated logic across files and late-breaking validation failures. Specification-driven composition separates workflow intent from processing so you can manage data workflows more consistently at scale. The result is faster dataset onboarding, stronger governance, and pipelines that are transparent enough to trust in regulated environments.
This pattern is especially valuable for regulated reporting pipelines, multi-source data integration, and reusable ETL frameworks where traceability and flexibility matter. By investing a small set of reusable capabilities and a disciplined specification format, you can reduce the engineering effort for new pipelines by treating them more as configuration tasks which may be delegated to system users.
For a practical first use case, try applying the pattern to a reporting pipeline you maintain today that has three or more variants such as monthly finance reports generated from different source systems. Replace the duplicated scripts with a single composer and a shared capability library, and measure the onboarding time for the next variant. As you expand your capability library, you can apply the same pattern across additional workflows and standardize how transformations are defined and run.
About the author
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.