Tag Archives: Intermediate (200)

Extend Amazon Inspector SBOM Generator with Plugins

2026-07-30 Michael Long

Post Syndicated from Michael Long original https://aws.amazon.com/blogs/security/extend-amazon-inspector-sbom-generator-with-plugins/

Amazon Inspector is an automated vulnerability management service that continually scans Amazon Web Services (AWS) workloads for software vulnerabilities. The vulnerability management capabilities of Amazon Inspector are powered by an asset inventory engine known as the Amazon Inspector SBOM Generator (inspector-sbomgen), a standalone command-line tool that produces a software bill of materials (SBOM) from container images, directories, archives, local systems, compiled binaries, and more. Over the past two years, we’ve expanded inspector-sbomgen’s coverage across dozens of programming language ecosystems, operating systems, and widely deployed applications.

We’re pleased to announce a new capability for builders using inspector-sbomgen: a plugin system for writing your own custom package collectors that you can use right away, without requiring source code compilation nor waiting for an official release.

You can download the latest version of inspector-sbomgen from the Amazon Inspector User Guide.

In this post, we walk you through what the inspector-sbomgen plugin system does, why we built it, and how you can write your first plugin in a few minutes. Along the way, we also cover how plugin-generated package components integrate with Amazon Inspector for vulnerability scanning, and we explore the plugin safety model, which helps ensure security-hardened and predictable plugin behavior.

Why we built a plugin system

Software ecosystems are dynamic. New language package managers, lockfile formats, and end user applications ship constantly, and many are adopted quickly, in some cases with little security scrutiny. That leaves security teams with a visibility gap: production workloads running software that their SBOM tooling doesn’t yet recognize. Customers have asked us to inventory many of these ecosystems directly, and until recently, the only path to support was to open a feature request and wait for the inspector-sbomgen team to onboard the ecosystem and deploy a new release.

The inspector-sbomgen plugin system changes that. With plugins, you can:

Onboard ecosystems that inspector-sbomgen doesn’t support out of the box. New open source ecosystems, niche or fast-moving package formats, and internal or proprietary tooling can all be inventoried without modifying inspector-sbomgen.
Prototype detection for an ecosystem quickly. We designed a plugin system that is friendly to developers and AI coding assistants alike. Plugins are written in Lua, loaded at runtime, and require no Go toolchain nor compilation. You can use the built in test harness to iterate on a plugin and see results immediately.
Build on a stable foundation. The plugin API abstracts away artifact-type differences, so you write your detection logic once and it works seamlessly across container images, archives, local systems, and more. And because plugins stay decoupled from the internals of sbomgen, the core tool’s regression surface stays small.

Internally, we’ve used the plugin system to ship new ecosystem coverage faster than before. In our 1.13 release, more than 20 ecosystems that were previously implemented in Go, including Apache Tomcat, NGINX, MySQL, Redis, WordPress, and the OpenSSH toolchain, are now embedded as plugins inside the sbomgen binary. The same release also added more than ten brand-new ecosystems as plugins, including Apache Cassandra, Apache Struts, Conda, Swift packages, and AI-agent collectors (Amazon Q Developer, Kiro CLI, Claude Code, GitHub Copilot, and Ollama).

How inspector-sbomgen plugins work

Sbomgen plugins follow a two-step pipeline:

Discovery – Scan the artifact’s file system to identify files that contain installed package metadata.
Collection – Open each discovered file, parse file contents, and publish findings into the SBOM.

Under the hood, an event bus connects discovery and collection plugins. Discovery plugins publish events listing discovered files, and one or more collection plugins subscribe to these events, triggering package collection. Developers might recognize this behavior as the observer pattern.

This decoupling lets a single discovery plugin feed multiple collectors, for example, one extracting package metadata, another scanning for secrets, and another checking policy. Each collection plugin works from the same file list without re-walking the artifact filesystem, a computationally expensive operation.

Write your first plugin in 5 minutes

Inspector-sbomgen makes it straightforward to bootstrap a plugin environment. The plugin new command tells sbomgen to create a new plugin workspace, and the —-with-example flag populates the workspace with a discovery-collection plugin pair, that you can run immediately.

inspector-sbomgen plugin new --with-example

After invoking the preceding command, you will be prompted to provide a plugin name and a directory that will contain your plugin workspace. You can provide custom values or use the default values:

Plugin name (identifies the software ecosystem your plugin will inventory, e.g. debian-dpkg, rhel-rpm, python-pip, cmake) [my-custom-ecosystem]: <enter>
Project directory [my-sbomgen-plugins]: <enter>

Created plugin "my-custom-ecosystem" in my-sbomgen-plugins/

Note that you can skip interactive prompts by specifying the plugin name and directory using the corresponding command line interface (CLI) arguments:

inspector-sbomgen plugin new \
    --with-example \
    --name my-custom-ecosystem \
    --path my-sbomgen-plugins

After creating your plugin workspace, inspector-sbomgen will display a next steps screen, which guides developers and AI code assistants to the source files they need to change and to supporting documentation:

Next steps:

  Get started:
    1. Open plugin folder in a code editor (VS Code recommended)
    2. Add test files that your plugin will discover and parse
       (e.g., config files, lockfiles, binaries, etc.):
       my-sbomgen-plugins/discovery/cross-platform/extra-ecosystems/my-custom-ecosystem/_testdata/

  Develop:
    3. Edit discovery:    my-sbomgen-plugins/discovery/cross-platform/extra-ecosystems/my-custom-ecosystem/init.lua
    4. Edit collection:   my-sbomgen-plugins/collection/cross-platform/extra-ecosystems/my-custom-ecosystem/init.lua

  Test:
    5. Write unit tests:  my-sbomgen-plugins/discovery/cross-platform/extra-ecosystems/my-custom-ecosystem/init_test.lua
    6. Run unit tests:    inspector-sbomgen plugin test --path my-sbomgen-plugins

  Deploy:
    7. Distribute your plugin directory wherever you run inspector-sbomgen:
       inspector-sbomgen <arguments> --plugin-dir /path/to/my-sbomgen-plugins

       Example:
       inspector-sbomgen container --image alpine:latest -o /tmp/sbom.json --plugin-dir /path/to/my-sbomgen-plugins

For code completion, install the VS Code Lua language server extension:
  https://luals.github.io/#vscode-install

For more information:
  - Plugin guide:    my-sbomgen-plugins/docs/sbomgen-plugin-developer-guide.md
  - Testing guide:   my-sbomgen-plugins/docs/sbomgen-plugin-testing-guide.md
  - API reference:   my-sbomgen-plugins/docs/sbomgen-plugin-api-reference.md
  - Documentation:   https://docs.aws.amazon.com/inspector/latest/user/sbom-generator.html

Now that you have a plugin workspace, let’s explore its contents in greater detail:

tree my-sbomgen-plugins

├── AGENTS.md
├── collection
│   └── cross-platform
│       └── extra-ecosystems
│           └── my-custom-ecosystem
│               └── init.lua
├── discovery
│   └── cross-platform
│       └── extra-ecosystems
│           └── my-custom-ecosystem
│               ├── _testdata
│               │   ├── empty
│               │   └── example.lock
│               ├── init_test.lua
│               └── init.lua
├── docs
│   ├── sbomgen-plugin-api-reference.md
│   ├── sbomgen-plugin-developer-guide.md
│   └── sbomgen-plugin-testing-guide.md
├── library
│   └── sbomgen.lua
└── README.md

The scaffolded project includes a working discovery and collection plugin pair, passing unit tests with test fixtures under _testdata/, a .vscode/settings.json for integrated development environment (IDE) integration, and a local copy of the developer documentation.

The scaffolding is deliberately succinct and complete, so it reads well for both humans and AI coding assistants. Every file has clear comments that explain what each function does and what the plugin author needs to fill in.

To test a plugin, you first need something to scan, such as a package lock file or a compiled binary. The example plugin inventories a fictional example.lock with the following contents:

my-package-alpha==1.0.0 
my-package-beta==2.3.1 
my-package-gamma==0.9.5

The provided discovery plugin knows how to look for instances of example.lock within the artifact file system:

-- my-custom-ecosystem discovery plugin
-- Discovers example.lock files in the artifact file list.

function discover()
    return sbomgen.find_files_by_name({"example.lock"})
end

And the provided collection plugin knows how to parse the contents of example.lock and publish package findings to the output SBOM.

-- my-custom-ecosystem collection plugin
-- Parses example.lock files and extracts package name and version.

function collect(file_path)
    local content = sbomgen.read_file(file_path)
    if content == nil then
        return
    end

    for line in content:gmatch("[^\n]+") do
        local name, ver = line:match("^(.+)==(.+)$")
        if name and ver then
            sbomgen.push_package({
                name = name,
                version = ver,
                purl_type = "generic",
                namespace = "my-custom-ecosystem",
                component_type = sbomgen.component_types.APPLICATION,
            })
        end
    end
end

Run the tests

Plugins ship with a built-in test framework so you can validate your logic before scanning a real artifact. Tests are written in Lua, live next to the plugin in init_test.lua, and reference fixture data in _testdata/:

function test_discovers_packages() 
    local result = testing.scan_directory("_testdata") 
    testing.assert_equals(3, #result.findings) 
    testing.assert_equals("my-package-alpha", result.findings[1].name) 
    testing.assert_equals("1.0.0", result.findings[1].version) 
end 
 
function test_no_findings_for_empty_directory() 
    local result = testing.scan_directory("_testdata/empty") 
    testing.assert_equals(0, #result.findings) 
end

Run the tests with the following command:

inspector-sbomgen plugin test --path my-sbomgen-plugins -v

=== RUN   my-custom-ecosystem/discovery/init_test/test_discovers_packages 
--- PASS: my-custom-ecosystem/discovery/init_test/test_discovers_packages (0.04s) 
=== RUN   my-custom-ecosystem/discovery/init_test/test_no_findings_for_empty_directory 
--- PASS: my-custom-ecosystem/discovery/init_test/test_no_findings_for_empty_directory (0.04s) 
ok    2 tests passed

This is the tightest development loop we could design: no Go toolchain, no rebuild, no container spin-up. Write a test, run it, iterate.

Scan a real artifact

For plugins to produce findings, inspector-sbomgen needs an artifact that contains the files your plugin looks for. For the example plugin, any directory with an example.lock file works. The fixture we generated earlier is a good stand-in:

inspector-sbomgen directory \ 
    --plugin-dir ./my-sbomgen-plugins \ 
    --path ./my-sbomgen-plugins/discovery/cross-platform/extra-ecosystems/my-custom-ecosystem/_testdata \ 
    -o sbom.json

The --plugin-dir flag tells inspector-sbomgen where to load your Lua plugins from. The resulting SBOM contains a CycloneDX component for each of the three packages in example.lock, for example:

{
  "bom-ref": "comp-2",
  "type": "application",
  "name": "my-package-alpha",
  "version": "1.0.0",
  "scope": "optional",
  "purl": "pkg:generic/my-sbomgen-plugin/[email protected]",
  "properties": [
    {
      "name": "amazon:inspector:sbom_generator:source_path",
      "value": "./my-sbomgen-plugins/example.lock"
    }
  ]
}

Every plugin-generated component carries an amazon:inspector:sbom_generator:source_path property that records the file the component was collected from, so you can always trace a component back to the artifact that produced it.

Vulnerability scanning with Amazon Inspector

Plugin-generated findings are first-class SBOM components. They work with every downstream consumer that reads CycloneDX SBOMs, including Amazon Inspector. To send an SBOM to Amazon Inspector for vulnerability analysis, add the --scan-sbom flag (this requires an active AWS account):

inspector-sbomgen directory \ 
    --path ./my-sbomgen-plugins/discovery/cross-platform/extra-ecosystems/my-custom-ecosystem/_testdata \ 
    --plugin-dir ./my-sbomgen-plugins \ 
    --scan-sbom \ 
    --aws-profile your_profile \ 
    --aws-region your_region \ 
    -o /tmp/sbom.json

An important caveat when you onboard a brand-new ecosystem: Plugin authors can inventory arbitrary ecosystems, but Amazon Inspector can only report vulnerabilities for components it has advisories for. When you point Amazon Inspector at a component whose ecosystem isn’t in its advisory feeds yet, Inspector will return the component with a property, Component skipped: no supported rules found. For example:

{ 
  "bom-ref": "comp-1", 
  "name": "my-package-alpha", 
  "properties": [ 
    { 
      "name": "amazon:inspector:sbom_scanner:path", 
      "value": "my-sbomgen-plugins/discovery/cross-platform/extra-ecosystems/my-custom-ecosystem/_testdata/example.lock" 
    }, 
    { 
      "name": "amazon:inspector:sbom_scanner:info", 
      "value": "Component skipped: no supported rules found." 
    } 
  ], 
  "purl": "pkg:generic/my-custom-ecosystem/[email protected]", 
  "type": "application", 
  "version": "1.0.0" 
}

This is expected behavior, not an error. The SBOM is still generated correctly, the component is still tracked, and the source_path tells you exactly which file produced it. If and when Amazon Inspector adds advisory coverage for the ecosystem, the same SBOM will start producing vulnerability findings without any change to your plugin. For ecosystems Inspector already supports, plugin-generated components are indistinguishable from components produced by built-in scanners.

First class IDE support

We care about productivity and efficiency when writing plugins. Writing Lua without modern conveniences such as autocomplete isn’t fun, so every plugin project scaffolded with the plugin new command ships with a library/sbomgen.lua definition file and a .vscode/settings.json that automatically wires it up to the Lua Language Server extension for VS Code.

For code completion and IDE support, first install the sumneko.lua extension, open your plugin project in VS Code, and every sbomgen.* function will get:

Parameter hints with types.
Hover documentation.
Autocomplete for constants (sbomgen.component_types.*, sbomgen.groups.*, sbomgen.platform.*).
Type checking on function calls.
Inline warnings when required fields are missing from push_package().

The same definition file makes plugin development work well with AI coding assistants. The types and documentation are embedded in a form that tools can read, so assistants can generate correct plugin code with far less monitoring than writing against a raw language would require.

A safe foundation

Plugins run real code inside the same process as inspector-sbomgen, so we designed the execution environment to keep that code stable and security-hardened. Every Lua plugin runs in an isolated sandbox. Every Lua virtual machine (VM) has access to a restricted subset of the Lua standard library to ensure only safe operations are permitted:

No direct filesystem access. The Lua io library isn’t loaded. All file operations go through sbomgen.* functions, which route through sbomgen’s internals so your plugin behaves identically whether it’s scanning a directory on disk, a container image, a compressed archive, or a mounted volume.
No subprocess execution or environment mutation. The Lua os library is blocked, so plugins can’t spawn processes, modify environment variables, or touch files outside the artifact.
No VM introspection. The Lua debug library is blocked.
No unbounded code loading. dofile, loadfile, and loadstring are removed. require() is available but restricted to the plugin’s own directory tree, so plugins can share helper modules with themselves but cannot load code from other plugins or system paths.

If a plugin raises an unhandled Lua error, inspector-sbomgen logs a warning and continues with the next file or plugin; one faulty plugin does not prevent other plugins from running. Plugins never override inspector-sbomgen’s built-in package collectors. Every plugin must declare a unique name. If a custom plugin uses a name that’s already claimed by an official built-in plugin, the custom plugin is skipped with a warning. Built-in plugins always take precedence, so a custom plugin can never silently replace or shadow the tool’s own detection behavior.

Next steps

To start building your own plugins today:

Install the latest inspector-sbomgen from the Amazon Inspector user guide.
Run inspector-sbomgen plugin new --with-example and follow the prompts.
Run inspector-sbomgen plugin test --path ./my-sbomgen-plugins -v to see the example tests pass.
Replace the example logic with detection for your own ecosystem.

The full reference documentation covers every function, constant, and command in depth:

Lua plugin developer guide: plugin concepts, directory structure, and lifecycle.
Lua plugin testing guide: test framework reference and fixture conventions.
Lua plugin API reference: complete catalog of the sbomgen.* API surface.

Conclusion

Whether you’re adding support for an internal lockfile format, prototyping detection for a new open source ecosystem, or replacing a home-grown scanner with something your whole organization can run at scale, the plugin system is designed to make the path from idea to working SBOM as short as possible. We can’t wait to see what you build with it.
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Accelerate Spark on EMR Serverless with larger workers and shuffle-optimized disks

2026-07-29 Karthik Prabhakar

Post Syndicated from Karthik Prabhakar original https://aws.amazon.com/blogs/big-data/accelerate-spark-on-emr-serverless-with-larger-workers-and-shuffle-optimized-disks/

With Amazon EMR Serverless, you can run open source big data frameworks such as Apache Spark and Apache Hive without managing clusters or infrastructure. Customers are increasingly choosing EMR Serverless for their analytics workloads because of the simplicity of a fully managed, serverless experience. As adoption grows, teams want to bring their most demanding jobs to Serverless too. These jobs include large-scale joins, shuffle-heavy ETL, and memory-intensive analytics that previously required carefully sized clusters. Customers migrating these heavyweight workloads from their Spark clusters often need the same compute shapes on EMR Serverless to achieve the same price-performance and make migration easier.

Today, we’re excited to announce a new 32 vCPU / 244 GB worker configuration on Amazon EMR Serverless, giving you the headroom to run your most intensive workloads without leaving the serverless experience.

Overview of larger workers

An EMR Serverless application uses workers to run your Spark tasks, and you can choose a worker size that matches your workload. The new 32 vCPU worker offers a large compute and memory footprint (32 vCPUs and 244 GB of memory) that supports attaching up to 2,000 GB of shuffle-optimized disk. This combination benefits three common workload patterns:

Shuffle-intensive workloads – Wide transformations such as join, groupBy, sortBy, and repartition redistribute large amounts of data across the cluster. Larger workers keep more shuffle data local to each executor and read and write shuffle blocks on higher-throughput disk, reducing remote fetches and shuffle wait time.
I/O-heavy workloads – Queries that scan large datasets or spill intermediate data to disk are limited by disk throughput and IOPS. The shuffle-optimized disk raises the ceiling on both sides. Large workers also improve network bandwidth.
Memory-intensive workloads – Higher per-executor memory (244 GB versus 30 GB) lets more data be cached and processed in memory without spilling, which helps with data skew and caching.

Benchmark setup

We compared the recommended large-worker configuration against common small-worker defaults at identical total compute (192 vCPUs).

Large workers – 6 executors × (32 vCPU / 244 GB / 2,000 GB shuffle-optimized disk).
Small workers – 48 executors × (4 vCPU / 30 GB / 200 GB standard disk).

The two configurations differ in both worker shape and disk class. We compare them as paired configurations, since shuffle-optimized disk is the intended disk type for the 32 vCPU worker. The reported gains reflect this combined effect.

Both configurations set spark.dynamicAllocation.enabled=false and spark.scheduler.minRegisteredResourcesRatio=1 to reduce variance from worker launch times. Requiring full registration ensures each query starts only after the cluster is ready. We ran every query with 3 iterations and reported the median. Both benchmarks ran on EMR release emr-7.13.0 in a virtual private cloud (VPC) with an Amazon Simple Storage Service (Amazon S3) gateway endpoint attached to the private subnets.

Benchmark results

This post presents benchmark results comparing the new 32 vCPU workers against the existing 4 vCPU standard workers using the industry-standard TPC-DS and TPC-H benchmarks. Across 126 queries (104 TPC-DS and 22 TPC-H), large workers delivered an average of 29% faster query execution and 29% lower query-attributed cost, with zero regressions and improvements peaking at 45–55% on shuffle-heavy, multi-table join queries. Both the larger executor shape and the shuffle-optimized disk contribute to these gains. The disk advantage is most pronounced on the shuffle and I/O-heavy queries where the largest improvements appear.

The following table summarizes the results across both benchmarks. Large workers won every query on both performance and cost.

Note: The benchmark results in this post are derived from the TPC-DS and TPC-H benchmark specifications. TPC-DS and TPC-H are trademarks of the Transaction Processing Performance Council.

Benchmark	Queries tested	Avg performance improvement	Avg cost improvement
TPC-DS (3 TB)	104	26.7%	27.4%
TPC-H (1 TB)	22	38.5%	37.2%
Combined	126	28.8%	29.1%

TPC-DS 3 TB benchmark

TPC-DS is an industry-standard decision support benchmark that models complex analytical workloads with multi-table joins, subqueries, and aggregations. We ran 104 queries from the TPC-DS v2.4 suite against a 3 TB partitioned Parquet dataset, with 3 iterations per query for statistical confidence.

The test environment was as follows:

Dataset: 3 TB partitioned Parquet (24 TPC-DS tables).
Queries: 104 (full suite minus 4 incompatible with the dataset schema).
Iterations: 3 per query (one query per start-job-run, a fresh Spark application each time).
Networking: VPC with an Amazon S3 gateway endpoint.
EMR release: emr-7.13.0.

The following chart shows the top 10 and bottom 5 queries by performance improvement. All 104 queries show a positive improvement, with the largest gains on shuffle-heavy queries such as q58 (45%), q21 (43%), and q12 (42%).

Bar chart of per-query performance improvement for the top 10 and bottom 5 TPC-DS queries

Performance improvement for the top 10 and bottom 5 TPC-DS queries, 32 vCPU compared to 4 vCPU standard workers

Why large workers are faster for TPC-DS

TPC-DS queries are characterized by complex multi-table joins that generate large shuffle operations. With 6 large executors instead of 48 small ones, the shuffle-optimized disk provides significantly higher random I/O throughput for reading and writing shuffle blocks. In addition, fewer executors mean less network coordination during shuffle. Each executor fetches shuffle data from only 5 remote sources instead of 47. This increases the share of shuffle data read locally, which improves performance.

TPC-H 1 TB benchmark

TPC-H is a decision support benchmark that focuses on ad hoc analytical queries. We ran all 22 TPC-H queries against a 1 TB dataset, with 3 iterations per query. Each query was submitted as a separate start-job-run (a fresh Spark application) to simulate the realistic pattern of independent ad hoc queries arriving without session warmup.

The test environment was as follows:

Dataset: 1 TB partitioned Parquet.
Queries: 22.
Iterations: 3 per query (one query per start-job-run, a fresh Spark application each time).
Networking: VPC with an Amazon S3 gateway endpoint.
EMR release: emr-7.13.0.

Bar chart of per-query performance improvement across the 22 TPC-H queries

Performance improvement for TPC-H queries, 32 vCPU compared to 4 vCPU standard workers

Why large workers are faster for TPC-H

The shuffle-optimized disk accelerates this initial table-scan phase. The subsequent query execution benefits from higher per-executor memory (244 GB versus 30 GB), which lets more data be processed in memory without spilling to disk.

Cost calculation and improvement results

The following table compares the cost of running TPC-DS and TPC-H benchmarks on larger workers (32 vCPU / 244 GB / 2,000 GB shuffle-optimized disk) versus smaller workers (4 vCPU / 30 GB / 200 GB standard disk) on EMR Serverless. Both configurations use identical total compute (192 vCPUs).

TPC-DS 3 TB

Metric	Larger Workers (6 × 32 vCPU)	Smaller Workers (48 × 4 vCPU)
Runtime	2,780.8s	3,670.2s
Resource Billed	vCPU = 148.31 \| Memory = 1,130.87 \| Disk = 9,269.47	vCPU = 195.74\| Memory = 1,468.07 \| Disk = 9787.15
Cost	$15.37	$19.87
Total vCPU	192	192
Total Disk	12 TB (shuffle-optimized)	9.6 TB (standard)
Improvement	27.4% lower cost	Baseline

TPC-H 1 TB

Metric	Larger Workers (6 × 32 vCPU)	Smaller Workers (48 × 4 vCPU)
Runtime	940.7s	1,529.5s
Resource Billed	vCPU = 50.17 \| Memory = 382.54 \| Disk = 3,135.60	vCPU = 81.57 \| Memory = 611.81 \| Disk = 4,078.72
Cost	$5.20	$8.28
Total vCPU	192	192
Total Disk	12 TB (shuffle-optimized)	9.6 TB (standard)
Improvement	37.2% lower cost	Baseline

Notes:

Runtime represents the sum of median query execution times across all queries in the benchmark (3 iterations per query, median reported).
Calculated cost is computed using EMR Serverless on-demand pricing: vCPU-hr ($0.052624), Memory GB-hr ($0.0057785), Storage GB-hr ($0.000111).
Illustrative TPC-DS run cost calculation:
- Large worker: (148.31 * $0.052624) + (1,130.87 * $0.0057785) + (9,269.47 * $0.000111) = $15.37.
- Smaller worker: (195.74 * $0.052624) + (1,468.07 * $0.0057785) + (9787.15 * $0.000111) = $19.87.
Cost is proportionally attributed to query execution time, excluding Spark initialization and shutdown overhead.
Both configurations use identical total vCPU (192) with Dynamic Resource Allocation disabled.
The improvement percentage represents the cost reduction achieved by larger workers relative to smaller workers.
Disk Sizing: The larger workers provisioned 25% more disk (12 TB versus 9.6 TB), yet the total cost came out 22.6–37.2% lower. Disk is by far the lowest-priced billing dimension on EMR Serverless ($0.000111 per GB-hour, versus 52x that for memory and 474x for vCPU) and made up only $1.03 of the $15.37 TPC-DS total. Many customers under-provision disk to trim this smallest line item, and it backfires. Shuffles slow down, jobs run longer, and every extra second is billed on the costlier vCPU and memory dimensions. On large workers (8+ vCPUs), shuffle-optimized disks scale IOPS and throughput with capacity, which helps move shuffle data faster. Size disk as a performance lever, not a cost lever.

When to use large workers

To determine whether the 32 vCPU worker with shuffle-optimized disk will benefit your Spark applications, consider the following:

Check the Stages tab of the Spark History Server for your EMR Serverless application and review the Shuffle Read and Shuffle Write columns. The larger the shuffle volume relative to the number of executors, the more a job benefits from keeping shuffle data local on high-throughput disk. Jobs that shuffle tens of gigabytes or more per executor are strong candidates.
Check the Stages tab for the Spill (memory) and Spill (disk) columns and the Executors tab for peak JVM memory. If data is spilling to disk or peak memory is close to the configured executor memory, the higher memory of the large worker can remove the spill and improve performance.

When not to use large workers

Large workers are not the best fit for every workload:

I/O-bound jobs – For workloads whose runtime is dominated by reading and writing data (rather than shuffle or memory pressure), a larger number of smaller workers (for example, 8 or 16 vCPU) with the same disk sizes can deliver better aggregate throughput. Spreading the work across more executors increases read/write parallelism, although a few large workers can leave disk and network bandwidth underused.

Conclusion

In this post, we demonstrated that EMR Serverless 32 vCPU workers deliver performance and cost benefits for shuffle-intensive Spark workloads. Across 126 TPC-DS and TPC-H queries, larger workers achieved an average 29% faster execution and 29% lower cost.

We recommend evaluating the 32 vCPU worker with shuffle-optimized disk for your shuffle-intensive and I/O-heavy EMR Serverless Spark workloads. To get started, use the following configuration:

--conf spark.executor.cores=32
--conf spark.executor.instances=6
--conf spark.executor.memory=220g
--conf spark.emr-serverless.executor.disk=2000G
--conf spark.emr-serverless.executor.disk.type=SHUFFLE_OPTIMIZED

For more information about worker configurations, refer to Worker configurations in the Amazon EMR Serverless User Guide. We also recommend staying up to date with the latest EMR releases to take advantage of ongoing performance improvements.

About the authors

Secure your npm and pip package updates in Amazon Linux

2026-07-29 Norbert Manthey

Post Syndicated from Norbert Manthey original https://aws.amazon.com/blogs/security/secure-your-npm-and-pip-package-updates-in-amazon-linux/

If you use and install packages from npm or PyPI, the first hours after a package is published are the riskiest because scanners can’t analyze packages before publication. Recent supply chain events affecting NodeJS and Python packages have been detected and removed within hours. However, while those packages were available to the general public, it’s possible that they were installed by users, creating the potential for a security incident. As you will see from the data that follows, if users had waited 1 day before accessing those packages, none of the recent supply chain security events would have had an impact.

In this post, I show you a one-line configuration that you can use to eliminate this exposure in your environment: a dependency cooldown for npm and pip. This change tells your package manager to skip versions published in the last 24 hours, giving the security community time to detect and remove unexpected packages before they reach your systems. These settings secure the default setup. There’s another use case of package updates: receiving security fixes to address security risks. This process involves updating packages to a more recent version. I also show you how to override the cooldown configuration so you can install the latest security patches while newly installed package updates are delayed. We recommend that you assess the severity of code defects and apply security fixes if there’s known risk. Handling security fixes based on their severity—and how to specify SLAs for these fixes based on severity—is beyond the scope of this blog post.

Background: Two risks pull in opposite directions

Software delivered by Amazon Linux packages go through review by Amazon package maintainers and pass guardrails before release. Open source software is developed and maintained with similar processes and guardrails. The npm and PyPI registries have open publishing access and don’t enforce reviews. Unexpected packages are potentially added to the registries because of risks like impersonation or stolen credentials. You’re caught between two risks: older software accumulates unpatched vulnerabilities, while new packages potentially contain unexpected vulnerabilities that haven’t been detected yet. The best approach is to stay current without adopting the newest releases immediately, while applying recommended security fixes. The following diagram illustrates the relation between the two types of risks in an abstract way, where the supply chain risk is highest immediately after a package is published, because unexpected updates can potentially bypass guardrails. After a package is published, auditing can review it and identify potential defects over time. If no security fixes are applied, the risk of all the code defects adds up.

Figure 1: Software risk over lifetime. Unpatched vulnerabilities risk increases over time. Very recent software also carries more supply chain risk.

The problem: The first day presents the highest risk

Supply chain events follow a consistent pattern. An unexpected author publishes an unexpected package or package version and waits for automated systems and users to pull it in. Security researchers and automated scanners typically detect and remove these packages within hours, but by then, systems have been exposed to the risk.

Datadog’s 2026 State of DevSecOps report found that 54% of JavaScript applications install at least one dependency within a day of its release. That’s the time window that presents the highest supply chain risk. Recent events show how fast detection happens:

Event	Exposure window
Nx s1ngularity (Aug 2025)	4–5 hours
axios (Mar 2026)	2–3 hours
Bitwarden CLI (Apr 2026)	93 minutes
TanStack (May 2026)	30 minutes
node-ipc (May 2026)	less than 24 hours

The solution: Skip packages published today

A dependency cooldown tells your package manager to skip recently published versions. If a version hasn’t existed on the registry for the configured timespan, for example, 1 day, it won’t be installed, giving the security community time to detect and remove unexpected versions.

A 1-day cooldown blocks each event listed in the preceding table. Notably, several of these events produced valid provenance attestations and passed build verification. These provenance checks alone didn’t stop them. A cooldown works independently of authorization mechanisms, because it blocks by age rather than by trust.

Both npm (v11.10.0+) and pip (v26.1+) support cooldowns . Amazon Linux 2023 ships these packages in NodeJS 24 and Python 3.14 since release 2023.11.20260608.

If you use lockfile-based installations through npm ci or pip install -r requirements.txt with pinned versions, you won’t pull latest package updates. The cooldown doesn’t apply to those installations. The cooldown only affects resolution of new or updated packages. See the Lockfile-based installs and the cooldown section for details.

Prerequisites

To implement the following solution, you first need to have the following prerequisites in place:

Node.js 24 with npm 11.10.0 or later (in nodejs24-24.14.1-1.amzn2023.0.1 or later).
Python 3.14 with pip 26.1 (in python3.14-pip-26.1.1-1.amzn2023.0.1 or later)
pip-audit (tool to scan python packages required for defect-based override scripts). Use python3.14 -m pip install pip-audit to install.

Future versions of Node.js and Python will bring new commands. The following tool commands work for Amazon Linux 2023 with Node.js 24 and Python 3.14. The provided commands target specific package versions. Adjust the commands if you use later releases.

To set up the npm cooldown

Create the global configuration directory, depending on your NodeJS version.
sudo mkdir -p /usr/lib/nodejs24/etc
Add the npm configuration file with the cooldown setting.
sudo npm-24 config set min-release-age 1 --location=global
Check that the cooldown is active by running the next command.
npm-24 config list

You will see before = "<timestamp from 24 hours ago>" in the output, confirming npm converted the 1-day cooldown into a date filter.
For more information, see the npm min-release-age documentation.

To set up the pip cooldown

Create the system-wide pip configuration file with the cooldown setting.
sudo python3.14 -m pip config set --global global.uploaded-prior-to P1D
Verify the configuration (for Python 3.14 and pip 26.1+).
python3.14 -m pip config list

You will see global.uploaded-prior-to='P1D’ in the output.

This configuration is safe to deploy immediately, because older pip versions (25.x) silently ignore the setting.

To install a package’s latest version without cooldown

What if you want to install the latest version of a package, for example to receive security fixes? The following sections describe how to override the flag using the tool command line. To identify which packages need urgent updates, run the appropriate audit command for your package manager.

npm auditor python3.14 -m pip_audit

For npm packages

Install the package with the cooldown override.
npm-24 install <package-name> --min-release-age=0

For pip packages

Install the package with the cooldown override.
python3.14 -m pip install <package-name> --uploaded-prior-to="P0D”

Update packages that need urgent updates

We recommend that you apply security fixes for packages that have known security risks. You don’t need to turn off the cooldown entirely to apply security fixes. Use the audit tools to identify packages with known issues, then override the cooldown for only these packages.

Prerequisites: Ensure you have Python 3 and pip-audit installed (python3.14 -m pip install pip-audit).

Important: These scripts demonstrate the concept. For production use, add error handling, logging, and testing. Review packages before updating them in automated pipelines.

For npm packages

The following script demonstrates the required steps to identify npm packages with a known security fix. The npm audit command prints these packages as JSON. Next, packages in this list are updated with an npm install command, where their cooldown is overridden so that the latest version is picked up.

npm audit --json | python3 -c "
import json, sys, subprocess
data = json.load(sys.stdin)
for pkg in data.get('vulnerabilities', {}):
subprocess.run(['npm-24', 'install', f'{pkg}@latest', '--min-release-age=0'])
"

For pip packages

The following script demonstrates the required steps to identify pip packages with a known security fix. The pip_audit command prints these packages as JSON. Next, all packages in this list are updated with an pip install command that overrides the cooldown so that the latest version can be picked up.

python3.14 -m pip_audit --format=json | python3 -c "
import json, sys, subprocess
data = json.load(sys.stdin)
for vuln in data:
    pkg = vuln['name']
    fix = vuln.get('fix_versions', ['latest'])[0]
    subprocess.run(['python3.14', '-m', 'pip', 'install', f'{pkg}=={fix}', '--uploaded-prior-to="P0D"'])
"

Lockfile-based installs and the cooldown

If you use npm ci or pip install -r requirements.txt with pinned versions, the cooldown doesn’t apply. These commands install what the lockfile specifies, regardless of package age. The cooldown only affects resolution of new or updated packages.

Industry adoption: Cooldowns are now used across PyPI and NodeJS

Major package managers and enterprises have started to adopt dependency cooldowns. As of May 2026, several popular package management tools now include cooldown features: pnpm (a fast Node.js package manager), Renovate (an automated dependency update tool), and StepSecurity (a supply chain security platform).

pnpm 11 ships with minimumReleaseAge enabled by default. It’s one of the first major package manager to make cooldowns opt-out rather than opt-in.
Renovate’s config best-practices preset has included a 3-day npm cooldown since 2025 and is widely adopted across enterprises.
StepSecurity Secure Registry uses a configurable cooldown period for enterprise customers. StepSecurity recommends a 10 day delay as default.

How AWS is helping protect the open source supply chain

AWS scans upstream package registries to catch unexpected packages before they reach customers.

Amazon Inspector, a security management service that continuously scans workloads for software vulnerabilities and network exposure, uses AI-assisted detection rules to scan upstream package registries. In 2025, Amazon Inspector researchers identified over 150,000 unexpected npm packages linked to a token farming campaign.
AWS Security published detailed response guidance for the Shai-Hulud worm and the axios event.
AWS contributes to the Open Source Security Foundation $12.5M grant for open source security, funding proactive scanning infrastructure that benefits the entire ecosystem.

Unexpected packages are typically caught within hours of publication. A 1-day cooldown ensures you don’t install them during that detection window.

Recommendations

To secure your Amazon Linux 2023 configuration:

Set a 1-day cooldown for npm and pip as shown in the preceding sections. External registries don’t have human review, so give the defenders time to catch problems.
Override when needed for urgent security patches using the per-command flags.
Run npm audit or pip_audit regularly to identify packages that need immediate attention.

Set up the cooldown with one line of configuration, and the protection is immediate.

Conclusion

By implementing the solutions presented in the post, you secure your npm and PyPI environment from most instances of unexpected code. The update delay of 1 day protects your environment, while still allowing to apply the latest security fixes. To learn about how to protect your environment further, see the following resources:

If you have feedback about this post, submit comments in the Comments section below.

Getting started with AWS End User Messaging Notify

2026-07-28 Brett Ezell

Post Syndicated from Brett Ezell original https://aws.amazon.com/blogs/messaging-and-targeting/getting-started-with-aws-end-user-messaging-notify/

One-time passwords (OTPs) are the backbone of modern user verification, from account creation to wallet additions, to password resets. But for businesses operating globally, delivering OTP messages reliably across dozens of countries is complex. It has traditionally meant navigating a maze of country-specific registrations, originator provisioning, and carrier compliance requirements. AWS End User Messaging Notify changes that equation entirely by removing the complexity of global OTP delivery.

Before we dive into the technical implementation, let’s explore what Notify is, why it matters for businesses sending verification codes at scale, and how you can go from zero to sending OTPs in minutes, not weeks.

The problem with traditional OTP sending

Sending OTP messages through traditional SMS channels requires significant upfront investment. For each country you want to reach, you need to research which origination identity types are supported: short codes, long codes, toll-free numbers, sender IDs, or 10DLC. Each has different registration timelines, throughput limits, and costs. Some countries require pre-registered message templates. Others mandate sender ID registration or risk having your messages displayed as “LIKELY-SCAM” to recipients.

For a business targeting multiple countries, that complexity multiplies fast. Each market carries its own registration process, approval timeline, and compliance requirements. A US short code can take 8–12 weeks to provision. India requires Distributed Ledger Technology (DLT) registration with the Telecom Regulatory Authority of India (TRAI), including entity verification, template approval, and header registration. Even straightforward markets like the UK now require sender ID registration to prevent carrier filtering.

The result? Businesses spend weeks or months on registration paperwork before sending their first verification code. And that assumes every registration is approved on the first attempt.

What is Notify?

Notify is a fully managed OTP and verification code sending service within AWS End User Messaging. Instead of provisioning your own phone numbers, managing carrier registrations, and building routing logic, you create a single Notify configuration and AWS handles the rest: the origination identities, carrier compliance, message routing, and even fraud protection.

The core benefit behind Notify is simplicity. You provide your brand name, select your target countries, choose a pre-approved message template, and start sending. AWS automatically validates your account, assigns the appropriate origination identities for each destination country, configures SMS Protect for fraud prevention, and routes your messages through the optimal delivery path. What previously required weeks of registration work now takes minutes.

The business case for Notify

The value of Notify becomes clear when you look at the traditional OTP implementation journey. A business targeting the United States, United Kingdom, Japan, and India would face four completely different registration processes:

United States: Choose between toll-free (15 business days, best case), 10DLC (moderate timeline), or short code (8-12 weeks or more). Sender IDs are not supported.
United Kingdom: Sender ID registration is required. Unregistered sender IDs risk being displayed as “LIKELY-SCAM.” Short codes are available through an AWS support case.
Japan: Sender IDs are still supported without pre-registration, but number display isn’t guaranteed across all carriers (particularly KDDI). Dedicated short codes are available through an AWS support case.
India: Requires DLT registration including entity verification, content template approval, and header (sender ID) registration. Traditionally one of the longest lead-time markets to onboard.

With Notify, your OTP use case in all four of these countries can be operational in minutes. For markets that previously required weeks, you can send immediately while working through dedicated registrations in parallel.

To illustrate the business impact across industries:

Hospitality and entertainment: Theme parks and resorts with global audiences can verify guest accounts across 50+ countries without managing individual country registrations, so international visitors can add passes to digital wallets, create accounts, and complete elevated security actions.
Ecommerce: Global marketplaces can onboard customers from any supported country with verified OTPs, eliminating the registration bottleneck that delays international expansion.
Financial services: Banks and fintech companies can deploy multi-factor authentication globally, with built-in fraud protection that automatically filters artificially inflated traffic and SMS pumping.
Healthcare: Patient portals can verify identities across international locations, with pre-approved templates that help maintain compliance without custom message body management.

Where to start?

When evaluating Notify for your OTP program, we recommend starting with your highest-volume verification use case. OTP and code verification messages follow predictable patterns, making them ideal candidates for Notify pre-approved templates. More importantly, they provide the most immediate return on investment (ROI) by eliminating the registration overhead for your most critical customer touchpoint.

Consider a phased approach:

Phase 1 – Quick win with Notify: Set up Notify for your primary markets. Get OTP sending operational in minutes. Use this to validate your integration and monitor deliverability.
Phase 2 – Evaluate and scale: As volumes grow, evaluate whether specific high-volume countries benefit from dedicated origination identities for higher throughput or branding purposes. Notify supports a hybrid approach in which you can use your own originators alongside Notify managed identities.
Phase 3 – Complex markets: For countries with strict registration requirements, begin the dedicated registration process in parallel. After they’re approved, integrate those dedicated identities into your sending architecture while Notify continues to handle the rest.

Now that you have decided where to start, the next sections walk you through setting up your first Notify configuration, from prerequisites to sending your first OTP.

Prerequisites

Before you begin, make sure that you have the following prerequisites in place:

An active AWS account with billing configured.
Access to the AWS End User Messaging SMS console.
AWS Identity and Access Management (IAM) permissions for AWS End User Messaging SMS operations.
A brand display name for your Notify configuration.
A planned list of target countries for your OTP messages.

Note: The display name must contain only letters, numbers, spaces, hyphens, or underscores, and can be up to 15 characters.

Understanding Notify tiers

Notify offers two tiers, each designed for different stages of your OTP implementation:

Feature	Basic tier	Advanced tier
Transactions per second (TPS)	1	25
Daily message limit	200 messages/day	Unlimited
Country availability	30 pre-approved low-risk countries	Full country list
Short code access	No	Yes
Fraud protection (SMS Protect)	Mandatory – AWS managed	Mandatory – AWS managed
Compliance verification	Trust-based with audit	AWS verifies opt-in compliance

The Basic tier is designed for getting started, testing, and low-volume use cases. It provides immediate access with conservative limits, which is well suited to validating your integration before scaling to production.

The Advanced tier unlocks higher throughput, unlimited daily sending, and access to the full list of supported countries. To upgrade, you complete a streamlined verification process where AWS confirms your opt-in compliance. This is significantly faster than traditional carrier registration because the verification stays within the AWS boundary. No downstream carrier approval is required.

The Basic tier is live almost immediately. The Advanced tier upgrade requires a brand verification registration that demonstrates a compliant opt-in flow, and most requests are processed within 3-5 business days, which is still much faster than traditional carrier registrations.

Spend limits: Notify has a separate spend limit from standard SMS sending. AWS auto-approves more generous limits for Notify, but plan accordingly if you anticipate high volumes at launch.

Testing tip: While testing, keep in mind there is a per-recipient cap of 10 messages per day per Notify configuration (and 10 per day per account) for any single destination phone number. If repeated test sends to your own phone suddenly stop arriving, this limit, not a configuration error, is the likely cause.

A closer look at Advanced tier country coverage

The Advanced tier documentation states it supports all countries available on AWS End User Messaging SMS. While technically accurate, there is an important distinction not immediately obvious from the console: not all countries are fully managed with Notify.

At the time of writing, 68 of the 247 countries available on the Advanced tier (28%) require you to provide your own origination identity. For those destinations, you still need to go through number provisioning and carrier registration. AWS does not automatically assign managed identities for these countries.

To identify which countries require customer-owned identities, use the AWS End User Messaging SMS v2 API:

aws pinpoint-sms-voice-v2 list-notify-countries --tier ADVANCED

For table view, use:

aws pinpoint-sms-voice-v2 list-notify-countries \
  --tier ADVANCED \
  --output table \
  --query 'sort_by(NotifyCountries, &CountryName)[].{
    "Country Name": CountryName,
    "ISO Code": IsoCountryCode,
    "Supported Channels": join(`, `, SupportedChannels),
    "Supported Use Cases": join(`, `, SupportedUseCases),
    "Supported Tiers": join(`, `, SupportedTiers),
    "Customer Owned Identity Required": CustomerOwnedIdentityRequired
  }'

The output is a formatted table as seen here:

Terminal table of Notify countries showing ISO code, channels, use cases, tiers, and the Customer Owned Identity Required column

Look for the CustomerOwnedIdentityRequired field in the response.

Coverage type	Count	Percentage
Fully managed by AWS	179	72%
Customer-owned identity required	68	28%
Total Advanced tier countries	247	100%

Note: These figures reflect coverage at the time of publication. As AWS continues expanding Notify fully managed coverage, these numbers may change – always use the API for the most current breakdown.

Generating the countries list as a CSV

Because Notify’s country coverage evolves over time as AWS expands fully managed support, we recommend always generating the latest list with the AWS CLI, in a spreadsheet-friendly format, whenever you need it.

aws pinpoint-sms-voice-v2 list-notify-countries \
--tier ADVANCED \
--output json \
--query 'sort_by(NotifyCountries, &CountryName)[].{CountryName: CountryName, IsoCountryCode: IsoCountryCode, SupportedChannels: join(`, `, SupportedChannels), SupportedUseCases: join(`, `, SupportedUseCases), SupportedTiers: join(`, `, SupportedTiers), CustomerOwnedIdentityRequired: CustomerOwnedIdentityRequired}' \
| jq -r '["Country Name","ISO Code","Supported Channels","Supported Use Cases","Supported Tiers","Customer Owned Identity Required"], (.[] | [.CountryName, .IsoCountryCode, .SupportedChannels, .SupportedUseCases, .SupportedTiers, (.CustomerOwnedIdentityRequired | tostring)]) | @csv' > notify-countries.csv

What this means for your planning

For 179 countries, coverage is turnkey. Create your configuration and start sending immediately.
For 68 countries, you need a hybrid setup. Provision origination identities and associate them through a phone pool. Notify still handles routing, templates, and fraud protection.
This reinforces the phased approach: start with the 179 fully managed countries, then register for the remaining 68 in parallel.
Eight countries support SMS only, with no voice: Austria, China, France, Gabon, Germany, Italy, Pakistan, and Slovenia.

Pro tip: Before committing to your target country list, run the ListNotifyCountries API call and look for CustomerOwnedIdentityRequired: true to identify which markets need additional lead time.

How Notify works

Notify simplifies the OTP sending workflow into three steps:

Create a Notify configuration – Provide your brand display name and select your use case (currently code verification). Optionally configure target countries, preferred templates, and channel settings.
Enable countries and channels – Select which countries you want to send to and enable SMS, voice, or both. AWS automatically configures the appropriate origination identities and fraud protection for each country.
Send messages – Use the SendNotifyTextMessage API to deliver OTPs. Pass your Notify configuration ID, the destination phone number, and your template variables (such as the OTP code). AWS handles identity selection, template resolution, and message delivery.

Registration steps

To create your first Notify configuration, open the AWS End User Messaging SMS console, choose Notify configurations, and choose Create configuration.

Enter your brand display name, which cannot be changed after creation.
Select the Code verification use case. Optionally, under Advanced, set your target countries, channels, language code, a default template, and a phone pool (for a hybrid setup).
Choose Create configuration.

For full step-by-step details, see the Getting started with Notify tutorial.

Notify configuration creation form in the AWS End User Messaging SMS console with display name and use case fields

Your configuration activates within moments.

Sending your first OTP

After your configuration is active, you can send a test message directly from the console:

Navigate to your Notify configuration and choose the Test tab.
Enter a destination phone number.
Select a message template and language.
(Optional) Configure the OTP code value and expiration.
Choose Send test message.

The recipient receives a message similar to the following:

“[YourBrandDisplayName], your one-time password verification code is [PASSWORD]. Please do not share this message. Sent by Notify.”

Note: This post focuses on delivering the OTP message itself. Generating a secure OTP code and verifying the code the customer submits back are outside the scope of this post. For guidance on that side of the architecture, including code generation, storage, and verification, see Build a Secure One-Time Password Architecture with AWS.

For programmatic sending, use the AWS End User Messaging SendNotifyTextMessage API (this example uses the AWS End User Messaging SMS phone number simulator):

In your terminal (or AWS CloudShell), create a new Python file:

cat > send_notify_message.py << 'EOF'
import boto3

client = boto3.client('pinpoint-sms-voice-v2')

response = client.send_notify_text_message(
    NotifyConfigurationId='notify-config-1234567890',
    DestinationPhoneNumber='+15555550123',
    TemplateId='your-template-id',
    TemplateVariables={
        'otp': '123456'
    },
    MessageFeedbackEnabled=True
)

print(f"Message ID: {response['MessageId']}")
EOF

Before running, replace the placeholder values with your own:

notify-config-1234567890 – Your Notify configuration ID.
+15555550123 – The destination phone number in E.164 format.
your-template-id – The template ID you created earlier.
123456 – Your OTP or variable value.

Run the script:

python3 send_notify_message.py

You receive output similar to the following:

Message ID: msg-1234567890abcdef0

Note: Notify uses the same pinpoint-sms-voice-v2 API namespace as standard AWS End User Messaging SMS. If you have an existing integration, the migration path is straightforward. The primary difference is the API endpoint and parameters.

Notify and standard AWS End User Messaging SMS: Better together

Notify works alongside your existing AWS End User Messaging SMS infrastructure. This hybrid approach combines the strengths of both:

Notify for rapid global coverage: Use Notify for countries where you don’t have dedicated origination identities. Eliminate registration delays and start sending immediately.
Dedicated identities for high-volume markets: For countries where you need maximum throughput, custom branding, or two-way messaging, continue using your own short codes, toll-free numbers, or registered sender IDs.
Pool-based prioritization: Associate your dedicated numbers in a phone pool with your Notify configuration. Notify prioritizes your numbers first and falls back to AWS managed identities only when needed.

This approach works well for businesses expanding internationally. You can launch OTP verification in new markets on day one with Notify, then transition to dedicated identities as volume and requirements justify the investment.

How Notify compares to standard AWS End User Messaging SMS

The following table compares standard SMS sending with Notify across the capabilities that matter most when planning your OTP program.

Feature	Standard SMS	Notify
Number provisioning	Customer managed	AWS managed
Carrier registration	Customer managed	AWS managed
Time to first message	Days to weeks	Minutes
Message templates	Customer created	Pre-approved by AWS
Fraud protection	Optional (SMS Protect)	Mandatory (SMS Protect)
Country rules	Customer configured	AWS managed with customer controls
Throughput (Basic)	Varies by originator	1 TPS, 200 msgs/day
Throughput (Advanced)	Varies by originator	25 TPS, unlimited daily
Hybrid with own numbers	N/A	Supported via phone pools
API namespace	`pinpoint-sms-voice-v2`	`pinpoint-sms-voice-v2`

Protecting your OTP traffic

Every Notify configuration includes mandatory SMS Protect integration. This means:

Country rules: Control which countries can receive messages. We recommend disabling all countries by default and enabling only your target markets. You can adjust these settings at any time through the console, API, or CLI.
Fraud filtering: AWS automatically filters artificially inflated traffic and SMS pumping. On the Basic tier, filter settings are AWS managed. On the Advanced tier, you retain the same protections with additional controls.
Spend controls: Notify includes a dedicated spend limit separate from your standard SMS spend limit. Monitor your usage through Amazon CloudWatch metrics and set up billing alerts to track spending.

You can view and manage your country rules directly within the Notify configuration. To allow or block specific countries:

Navigate to your Notify configuration.
Choose the Countries tab.
Toggle countries between Allow and Block status.

Note: SMS Protect is part of all AWS End User Messaging, not exclusive to Notify. If you decide to transition from Notify to dedicated origination identities for specific countries, your Protect configurations carry over without additional configuration.

Important considerations

Display name is permanent: Your display name appears in every message and cannot be changed after creation. If you need to experiment, create a test configuration first, then create a production configuration with your finalized brand name.

Templates are pre-approved: You cannot create custom OTP message bodies with Notify. This is by design. Pre-approved templates help maintain carrier compliance and reduce the risk of message filtering. If you need custom message content, use standard SMS sending with your own origination identities.

Architecture compatibility: Notify uses the same V2 API (pinpoint-sms-voice-v2) as standard AWS End User Messaging SMS. If you have an existing architecture using Amazon API Gateway, AWS Lambda, and Amazon Simple Queue Service (Amazon SQS) for message routing, integrating Notify requires only parameter changes, not an architectural redesign.

Message feedback: Notify supports the Message Feedback API for tracking OTP conversion rates. Pass the MessageFeedbackEnabled parameter when sending to track whether recipients successfully verify their codes. This provides first-party data for measuring authentication success rates across countries and carriers.

Managing costs and usage

Notify pricing includes a per-message service fee on top of standard SMS transport rates. The service fee covers origination identity management, fraud protection, and routing optimization.

Monitor your Notify message volume through Amazon CloudWatch metrics and the analytics dashboard within your Notify configuration. Set up billing alerts to track spending against your budget. For the latest rates, see AWS End User Messaging pricing.

Conclusion

In this post, we showed you how AWS End User Messaging Notify eliminates the registration complexity of global OTP sending. You get a fully managed verification code service with built-in fraud protection, pre-approved templates, and coverage across 200+ countries, operational in minutes instead of weeks.

Evaluate your current OTP sending workflow and identify the countries where registration overhead is delaying your go-to-market. Consider starting with Notify for those markets to establish immediate coverage, then layer in dedicated origination identities for high-volume countries as your program scales.

Get started today

Ready to implement Notify? Here are your next steps:

If you’re ready to start: Open the AWS End User Messaging SMS console and create your first Notify configuration. You can be sending test OTPs within 5 minutes.
If you want to learn more: Review the AWS End User Messaging Notify documentation and the Getting started with Notify tutorial.
If you’re evaluating for a global rollout: Check the Notify supported countries list against your target markets and review the Notify tiers and quotas to plan your throughput requirements.
If you need to build a complete OTP architecture: Read our related post, Build a Secure One-Time Password Architecture with AWS, for a comprehensive guide to OTP generation, storage, delivery, and verification.

Resources

About the authors

How Fanatics Commerce built a scalable email platform on Amazon SES

2026-07-28 Paul DeLaria

Post Syndicated from Paul DeLaria original https://aws.amazon.com/blogs/messaging-and-targeting/how-fanatics-commerce-built-a-scalable-email-platform-on-amazon-ses/

Fanatics Commerce is a leading designer, manufacturer, and retailer of licensed consumer products, including fan gear, jerseys, lifestyle and streetwear products, headwear, and hardgoods. Whether it’s a championship jersey or a last-minute gift, fans trust Fanatics to deliver and that trust extends to every digital touchpoint along the way.

Every order confirmation, shipping notification, and account update represents a moment of connection with a fan. Fans check their inbox after buying a jersey, track a package before game day, and verify their account when they sign up. These emails are the backbone of the fan experience.

The Fanatics Commerce engineering team built a modern, scalable email platform on Amazon Simple Email Service (Amazon SES), designed from the start for high deliverability, operational efficiency, and seasonal scale that comes with serving more than 100 million fans. When events like Super Bowl, NBA Finals, or World Series drive a surge in orders, the platform has to keep up without missing a beat.

This post walks through what drove the decision, the platform architecture, migration, the key engineering decisions, and what comes next for a large transactional email platform running on Amazon SES.

The case for change

As Fanatics Commerce grew, the engineering team saw an opportunity to elevate their email infrastructure by using Amazon SES capabilities purpose-built for operating at scale.

Dedicated IP addresses for full reputation control. With dedicated IPs in Amazon SES, Fanatics Commerce could own their sending reputation entirely removing dependency on shared infrastructure and gaining direct control over deliverability outcomes.
Granular traffic segmentation. Amazon SES offered the ability to treat transactional and marketing email as distinct, independently managed streams each with its own configuration sets, sending identities, and performance tuning rather than routing everything through a single pipeline.
Real-time deliverability visibility. At the scale of millions of fans, the team needed domain-level insight into open rates, bounce rates, and complaint rates in real time. The built-in analytics and Virtual Deliverability Manager in Amazon SES gave them the detail to diagnose shifts quickly and act decisively.
Domain-level isolation and authentication. SES enabled the team to assign dedicated subdomains and authentication policies (DKIM, SPF, DMARC) per email type ensuring high-priority transactional messages maintain protected, independent reputations.
Operational automation at scale. IP warming, reputation monitoring, and sending pattern adjustments could be managed programmatically through SES rather than requiring manual intervention keeping pace with Fanatics Commerce’s volume growth.

The team recognized the opportunity to move beyond incremental fixes. Rather than continuing to adapt an existing system, they set out to build a purpose-built transactional email platform on AWS that addressed all of these needs from the ground up.

Why Fanatics Commerce chose Amazon SES

After evaluating their requirements against several email service providers, the Fanatics Commerce team chose Amazon SES for its combination of reputation control, native observability, and tight integration with their existing AWS infrastructure. Several capabilities stood out during their evaluation.

The priority was reputation control. SES supports dedicated IP pools with separate pools for high-priority transactional, account, and lower-priority traffic, ensuring noisy streams cannot contaminate critical flows.

Visibility was equally important. As Rajat Banerjee, Fanatics’ engineering leader, explains:

“SES emits detailed JSON events for every send, delivery, bounce, complaint, open, and click into S3, and we model that data directly in our warehouse. Tagging each event with order, site, and mailbox provider, plus the user agent SES captures on opens and clicks, lets us slice deliverability at the level we need to run at Fanatics Commerce scale. That granularity is what let us refine our NPS survey email, power order attribution reporting, and debug real production issues over the last few months.”

The team also valued owning the full delivery path, from provider through messaging queue, internal processing, and status store, with rendered email HTML stored in-house. This end-to-end visibility strengthens support and debugging workflows.

The migration scope is strictly transactional, service, and survey email with high but predictable baseline volume and large event-driven spikes. SES is purpose-built for this pattern, with configurable IP warm-up strategies and the flexibility to choose between standard and managed dedicated IPs.

Finally, SES integrates natively with AWS metrics, notifications, queues, and storage, allowing monitoring, alerting, and failure handling to follow the same patterns used elsewhere in the Fanatics stack.

“If SES works for Amazon at scale, I figured it would work for us. We had also seen SES handle our load before, during a failover from our primary provider on a shared IP setup. That gave us the confidence to commit early and design around it.”

Platform architecture

The Fanatics Commerce team designed their email platform with the same engineering rigor they apply to their commerce systems. The architecture reflects a technology first approach to email operations.

Figure 1 — Fanatics Commerce transactional email platform on Amazon SES

Application layer

The Fanatics application connects to Amazon SES through IAM role-based authentication, with no stored credentials anywhere in the pipeline. This approach simplified security management and eliminated credential rotation as an operational concern.

Managed dedicated IPs

Fanatics started with dedicated IPs and pivoted to managed dedicated to let SES handle IP warming and management. Managed IPs let Amazon SES handle reputation optimization automatically, adjusting sending patterns, warming new IPs, and responding to reputation signals without manual intervention. This was a deliberate engineering decision: the team wanted to invest their time building great fan experiences, not managing IP reputation.

“We started with standard dedicated IPs and managed warming ourselves. Reputation management at our scale became more challenging than we wanted to own, so on AWS architects’ recommendation we moved to managed dedicated IPs. We would rather have our engineers enhancing the fan experience than tuning IP reputation.”

Domain and subdomain strategy

The domain architecture reinforces sender reputation through isolation. Transactional email sends from a dedicated subdomain with its own DKIM signing, SPF records, and DMARC policy. This ensures mailbox providers evaluate transactional email reputation independently, protecting the deliverability of order confirmations and shipping notifications regardless of what other email streams do.

Multi-tenant email design

The team designed a multi-tenant architecture that separates email streams into distinct tenants with independent configuration sets, dedicated IPs, and domain strategies. Each tenant maintains its own reputation, its own IP warming schedule, and its own deliverability metrics. If one tenant has reputation challenges, that specific tenant will be paused without disrupting other tenants.

This isolation is a core design principle. Transactional email, the email fans depend on, runs in its own tenant with dedicated infrastructure. Commercial email operates in a separate tenant. The architecture ensures each stream scales independently and maintains its own deliverability profile.

Real-time observability

Amazon SES Virtual Deliverability Manager (VDM) gives the Fanatics Commerce team a real-time, centralized view of key deliverability metrics including open rates, bounce rates, and complaint rates at the tenant or configuration set level. With VDM, the team is able to spot deliverability shifts early, diagnose issues with confidence, and take action before fans ever notice a problem in their inbox.

Scaling with the seasons

Sports merchandise is inherently seasonal. The platform needed to handle volume swings, from baseline traffic to peak holiday and playoff demand, without degrading deliverability.

During the 2025 holiday season, the platform scaled sending volume by 48x in five months, from initial rollout to full peak capacity across Black Friday, Cyber Monday, and the holiday gifting season. The architecture handled this surge while preserving deliverability, demonstrating that the multi-tenant design and managed dedicated IPs absorb seasonal spikes while maintaining consistent inbox placement rates.

The team phased their rollout by email type and volume, monitored deliverability metrics at each stage, and adjusted sending patterns based on real time feedback from mailbox providers. As the volume scales rapidly, this methodical approach ensured deliverability remained high.

The partnership model

This platform succeeded because of the partnership between Fanatics Commerce and AWS. The engagement brought together an account team TAM, a Solutions Architect, and a Worldwide Specialist SA, each contributing a different perspective.

The TAM coordinated the engagement by connecting Fanatics Commerce engineering with AWS specialists and driving architecture reviews from initial planning through peak holiday season scale.

Rajat Banerjee, Senior Manager of Engineering at Fanatics Commerce, led this initiative end to end from platform design all the way through production rollout. He and his team designed the domain and subdomain strategy that protects sender reputation across brands and ran a phased migration that scaled sending volume to full peak capacity without any disruption to delivery. SES built real-time analytics and reporting pipelines that give team visibility into delivery rates, bounces, and engagement. That visibility transformed incident response and helped the team optimize sending behavior at scale.

This model, customer engineering plus a cross-functional AWS team, accelerated decisions and shortened the feedback loop between architecture questions and production answers. The team had direct access to SES product expertise whenever they needed it, which enabled them to make timely informed decisions.

What’s next

Tenant-level isolation within SES – Handling each tenant’s sending, reputation, and operational signals independently end to end.

Deep linking from transactional emails into the Fanatics mobile app so fans can tap a link in an order or shipping email and land directly on the right screen in the app instead of the web.

Conclusion

Fanatics Commerce set out to build an email platform that matches the speed and reliability fans expect from the brand. By choosing Amazon SES and investing in purpose-built architecture, multi-tenant isolation, managed dedicated IPs, domain-level reputation control, and real-time observability, the team eliminated the operational trade-offs that come with scaling large email systems.

The results speak for themselves: the platform scaled sending volume 48x in five months, maintained high inbox placement rates through peak holiday and playoff demand, and gave the engineering team the visibility to diagnose and resolve deliverability issues in minutes rather than days.

More importantly, this platform frees the Fanatics Commerce team to focus on what matters most, building great fan experiences rather than managing IP reputation and chasing deliverability problems. Every order confirmation that lands in a fan’s inbox on time is a moment of trust earned.

Whether you’re sending millions of emails or only beginning to outgrow your current setup, the patterns in this post apply at any scale. Start by identifying where your current email infrastructure makes you choose between deliverability and growth. Amazon SES is built so you don’t have to.

To learn more about Amazon SES, visit the Amazon SES product page. To explore the Fanatics Commerce AWS journey, read Migration at Scale: The Fanatics Commerce AWS Journey.

About the authors

Build a contract compliance search system with Amazon OpenSearch

2026-07-28 Durga Prasad

Post Syndicated from Durga Prasad original https://aws.amazon.com/blogs/big-data/build-a-contract-compliance-search-system-with-amazon-opensearch/

For legal and compliance teams, auditing a repository of thousands of contracts for a single regulatory obligation shouldn’t take weeks. But with keyword search, it often does. A search for “inadvertent access notification” returns exact matches while missing functionally equivalent clauses such as “security incident disclosure” or “unauthorized access reporting.” This creates two problems:

Discovery gap: Critical risk exposure goes undetected because keyword search cannot match semantically equivalent terms across different contracts.

Review latency: After finding relevant contracts, legal counsel must manually scan lengthy documents to locate the specific clauses that matter. This process can stretch from minutes to hours per document.

Amazon OpenSearch Service is a fully managed search and analytics service that configures, manages, and scales OpenSearch clusters in the AWS Cloud. It supports use cases from log analytics and application monitoring to full-text search and real-time security analytics. It also supports AI-powered semantic search.

Amazon OpenSearch Service addresses both problems through two capabilities:

Semantic search retrieves contracts based on meaning rather than exact keyword matches, closing the discovery gap.
Semantic highlighting pinpoints the exact clauses within retrieved contracts that answer the query, reducing review time from hours of manual scanning to seconds of targeted reading.

In this post, you build a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. You deploy the solution using two AWS CloudFormation stacks, test it with synthetic contract documents, and see how a single query surfaces both the right contracts and the right clauses within them.

Solution overview

The solution uses a two-stage retrieval and extraction pipeline. First, semantic search identifies relevant contracts across the repository. Then, semantic highlighting marks the specific clauses within those contracts that match the query intent.

The following diagram illustrates the solution architecture:

Solution architecture showing contracts flowing from Amazon S3 through OpenSearch Ingestion and Amazon Bedrock embeddings to semantic search and Amazon SageMaker AI highlighting

Upload contracts to Amazon Simple Storage Service (Amazon S3) – Contract documents (JSON format) are uploaded to an Amazon S3 bucket, which serves as the centralized document repository.
Amazon OpenSearch Ingestion (OSI) reads from S3 – A serverless OSI pipeline detects new documents in the S3 bucket and reads them for processing.
OpenSearch ingest pipeline generates embeddings through Amazon Bedrock – As documents arrive, the ingest pipeline’s text_embedding processor invokes Amazon Titan Text Embeddings V2 through an ML Commons Bedrock connector. This converts contract text into 1024-dimension vector representations, stored in a k-NN index that uses the faiss engine.
User submits a search query – A user queries the system with natural language (for example, “data protection regulations”) through a test AWS Lambda function that forwards the request to OpenSearch using the neural query type.
OpenSearch generates the query embedding – OpenSearch converts the user’s natural language query into a vector embedding using the same machine learning (ML) Commons Amazon Bedrock connector and Amazon Titan V2 model.
Amazon OpenSearch Service performs semantic search – OpenSearch uses k-NN vector similarity to retrieve contracts that are semantically relevant to the query, even when exact terminology differs.
Amazon SageMaker AI performs semantic highlighting – The opensearch-semantic-highlighter-v1 model, hosted on an Amazon SageMaker AI GPU endpoint, scores sentence relevance using cross-encoder inference and wraps the matching clauses in  tags for targeted reading.

How semantic search and semantic highlighting work together

The system processes queries in two steps:

Step 1 – Semantic search (document discovery): You query the contract corpus using natural language. The system retrieves contracts with semantically similar concepts, even when exact terminology differs. For example, searching for “force majeure” returns contracts discussing “natural disasters” or “unforeseeable circumstances” because the system understands these concepts are related.

Step 2 – Semantic highlighting (clause identification): After relevant contracts are retrieved, semantic highlighting automatically marks the clauses that semantically match your search intent. Instead of scanning pages of legal text, you immediately see the specific paragraphs that answer your question.

The difference between standard keyword highlighting and semantic highlighting is significant:

Keyword highlighting wraps individual matching words: termination and rights.
Semantic highlighting wraps entire relevant clauses: Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days..

This reduces false positives, cuts review time, and provides explainability for why each document was retrieved.

Semantic highlighting model deployment

Before the system can highlight clauses based on meaning, the opensearch-semantic-highlighter-v1 model must be deployed to an Amazon SageMaker AI GPU endpoint and registered with the OpenSearch ML Commons plugin through a remote connector.

Stack 2 of the CloudFormation deployment automates this process. It performs the following steps:

Downloads the model artifact from an AWS-managed source and deploys it to an Amazon SageMaker AI endpoint (ml.g5.xlarge).
Creates a remote ML Commons connector in OpenSearch that points to the SageMaker endpoint.
Registers the model with the QUESTION_ANSWERING function so that OpenSearch can use the model’s cross-encoder capabilities to score sentence relevance at query time.

The equivalent manual registration call (handled automatically by the stack) is:

POST /_plugins/_ml/models/_register?deploy=true
{
  "name": "amazon/sentence-highlighting/opensearch-semantic-highlighter-v1",
  "version": "1.0.0",
  "model_format": "TORCH_SCRIPT",
  "function_name": "QUESTION_ANSWERING"
}

You don’t need to run this manually. The deployment script and CloudFormation stack handle model registration end-to-end. The resulting model ID is automatically passed to the query Lambda function for use in semantic highlighting requests.

Index configuration

The index uses a k-NN vector field with 1024 dimensions (matching the Amazon Titan V2 output) and the faiss engine with HNSW method. The mapping includes both a knn_vector field for semantic retrieval and a standard text field for keyword matching and highlighting. When you search for “liability limits,” OpenSearch first retrieves documents through vector similarity, then uses the Amazon SageMaker AI model to identify and wrap the specific relevant sentences in  tags.

PUT /legal-contracts-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "clause_text": { "type": "text" },
      "clause_embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "space_type": "l2"
        }
      }
    }
  }
}

Implementation steps

This section walks you through deploying the solution using two AWS CloudFormation stacks and two shell scripts. You first set up the core infrastructure (OpenSearch, ingestion pipeline, and ML Commons Bedrock connector), then deploy the semantic highlighting model on Amazon SageMaker AI.

Prerequisites

To deploy this solution, you need:

An active AWS account with permissions to create Amazon S3 buckets, AWS Lambda functions, Amazon SageMaker AI endpoints, Amazon Bedrock model access, Amazon OpenSearch Ingestion pipelines, Amazon OpenSearch Service domains, and AWS Identity and Access Management (IAM) roles (including iam:PassRole and sts:AssumeRole). For the exact least-privilege policy, see iam-deployer-policy.json in the repository. Both CloudFormation stacks require the CAPABILITY_NAMED_IAM acknowledgement.
Amazon Bedrock model access enabled for Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0).
Familiarity with AWS CloudFormation.
Estimated deployment time: approximately 35 minutes.
Estimated cost: approximately USD $ 2.00–3.00 for a quick demo. Delete the stacks promptly after testing.
This post uses US East (N. Virginia) as the deployment AWS Region. Verify service availability in your preferred Region before deploying.

Deploy the solution

The solution deploys using two AWS CloudFormation stacks and two shell scripts. The demo includes synthetic contract documents covering common contract types including software licenses, data processing agreements, managed services, and software as a service (SaaS) subscriptions.

Clone the repository and run the deployment script:

git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./deploy.sh

The deployment script creates the following resources across two stacks:

Stack 1:

An Amazon OpenSearch Service domain with fine-grained access control.
An Amazon OpenSearch Ingestion (OSI) pipeline that reads contracts from S3 and sends them to OpenSearch for indexing.
An ML Commons Bedrock connector and ingest pipeline that automatically generates 1024-dimension vector embeddings through Amazon Titan Text Embeddings V2 during document indexing.
A test Lambda function for querying the OpenSearch index using keyword, neural, or hybrid search with semantic highlighting support.
An S3 bucket for storing contract documents.
IAM roles for Lambda functions, the OSI pipeline, and OpenSearch access.

Stack 2:

An Amazon SageMaker AI endpoint hosting the semantic highlighting model.
A Lambda function that creates an ML Commons remote connector in OpenSearch and registers the highlighting model.

After both stacks deploy, the script automatically configures OpenSearch (role mappings, Amazon Bedrock connector, embedding model, k-NN index), ingests the sample contract data, and registers the semantic highlighting model.

The total deployment takes approximately 35 minutes to complete.

(Optional) Automated deployment with Claude Code CLI

If you have Claude Code CLI installed, you can deploy the solution using an AI-assisted workflow that creates a least-privilege IAM role scoped to this demo before deploying:

git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./scripts/create-deployer-role.sh
export OS_DEMO_DEPLOYER_ROLE=arn:aws:iam::<ACCOUNT_ID>:role/os-demo-deployer-role
export AWS_DEFAULT_REGION=us-east-1
claude "Deploy the OpenSearch semantic search demo following README.md"

Claude Code reads the repository instructions, assumes the deployer role, deploys both CloudFormation stacks in order, runs the setup scripts, and verifies the deployment end-to-end. The deployer role restricts actions to resources prefixed with os-demo-*, following the principle of least privilege.

Test the solution

After the deployment succeeds, follow these steps to test the solution.

On the Lambda console, choose Functions in the navigation pane.
Choose the function that has os-demo-query in its name.
On the Test tab, in the Event JSON paste this keyword search query {"query": "data protection regulations?", "type": "keyword", "k": 3}
Choose Test to run the Lambda function.

The following screenshot shows the Lambda function test configuration on the AWS Management Console with the keyword search query.

Lambda console Test tab with the keyword search query entered in the Event JSON field

The function processes the query in two ways depending on the search type:

For keyword search (enter: keyword): The function sends a standard match query to OpenSearch, which returns documents containing the exact query terms. The highlight fragments wrap individual matching words like termination and rights.

For neural search (enter: neural): The function sends a hybrid query to OpenSearch combining k-NN (semantic similarity) with keyword matching. OpenSearch automatically generates the query embedding through the ML Commons Amazon Bedrock connector using the same Amazon Titan V2 model. This returns semantically related documents even if they don’t contain the exact query terms. The SageMaker endpoint powers the semantic highlighting, identifying the most relevant clauses within each retrieved document. It wraps entire passages like Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days..

Download the highlight viewer HTML file and open it in the browser. This file helps you view the highlighted text.
Copy the entire execution output of the Lambda execution, paste it into the placeholder in the HTML file, and then choose Load Results.
The following screenshot shows that only the matching keywords are highlighted.

Highlight viewer showing only individual keywords highlighted in the keyword search results

Next, paste the neural search query as input to the Lambda function to see how semantic highlighting works: {"query": "data protection regulations", "type": "neural", "k": 1}
Choose Test to run, and then paste the entire output into the HTML viewer.

The viewer now displays entire sentences highlighted instead of individual keywords.

Highlight viewer showing entire relevant clauses highlighted in the neural search results

Optimizing for scale: batch semantic highlighting

In a standard search, a query might return dozens of relevant contracts. Using the default single inference mode, OpenSearch makes a separate ML call for every document in the result set. For a compliance officer reviewing 50 contracts, this sequential processing introduces noticeable latency.

OpenSearch 3.3 introduced batch inference mode to address this. Batch inference collects the matching documents and processes them in a single ML inference call. In the contract compliance use case, this shifts the performance characteristic from multiple sequential roundtrips to a single parallel execution on the Amazon SageMaker AI GPU.

To enable batch inference, first configure the cluster setting:

PUT _cluster/settings
{
  "persistent": {
    "search.pipeline.enabled_system_generated_factories": ["semantic-highlighter"]
  }
}

Then add batch_inference: true to your highlight options. The following query searches for data privacy clauses across the contracts and highlights the top 10 results using a single batch call:

POST /legal-contracts-index/_search
{
  "query": {
    "neural": {
      "clause_embedding": {
        "query_text": "standard for inadvertent access notification",
        "model_id": "<TEXT_EMBEDDING_MODEL_ID>",
        "k": 10
      }
    }
  },
  "highlight": {
    "fields": {
      "clause_text": { "type": "semantic" }
    },
    "options": {
      "model_id": "<REMOTE_HIGHLIGHTER_MODEL_ID>",
      "batch_inference": true,
      "max_inference_batch_size": 50
    }
  }
}

Best practices

Follow these recommendations to optimize performance, security, and cost-efficiency when deploying the contract compliance search system in production.

Experiment with overlapping chunk sizes (for example, 500 characters with a 10 percent overlap) in your OSI pipeline to verify that context is preserved for long indemnification or liability clauses.
Verify that your Amazon S3 buckets and OpenSearch domains are encrypted using AWS Key Management Service (AWS KMS). For production workloads containing sensitive data, make sure that all traffic stays within your virtual private cloud (VPC) through interface endpoints.

This demo uses simplified configurations for learning purposes. For production deployments, implement VPC isolation, AWS KMS encryption with customer-managed keys, and multi-AZ OpenSearch clusters.

Clean up resources

To avoid ongoing charges, delete the AWS CloudFormation stacks and associated resources:

On the AWS CloudFormation console, choose Stacks in the navigation pane.
Select the os-demo-highlighting stack (Stack 2) and choose Delete. Wait for deletion to complete.
Select the os-demo-search stack (Stack 1) and choose Delete. Stack deletion takes approximately 10–15 minutes to complete.

The stack deletion will automatically remove:

OpenSearch domain.
SageMaker model and endpoint.
Lambda functions.
IAM roles and policies.

After both stacks are deleted, manually delete the S3 bucket (opensearch-cfn-semantic-highlighting-us-east-1-<ACCOUNT_ID>) created for model artifacts. This bucket is provisioned at deploy time and is not managed by CloudFormation. Replace <ACCOUNT_ID> with your AWS account ID in the bucket name.

Conclusion

In this post, you built a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. The system helps close the discovery gap by retrieving contracts based on meaning rather than exact keywords, and it reduces review latency by highlighting the specific clauses that answer your query.

While we focused on legal agreements, the architecture described here is a blueprint for domains requiring high-stakes document discovery, including:

Regulatory filings: Identifying specific compliance mandates in financial reports.
Technical documentation: Pinpointing troubleshooting steps across massive product manuals.
Research and academia: Isolating specific methodologies within thousands of scientific papers.
Internal knowledge bases: Empowering employees to find exact policy language instantly.

To get started, deploy the solution from the sample repository on GitHub and try semantic search in the Amazon OpenSearch Service console. For more information about semantic search, see Semantic search in the Amazon OpenSearch Service Developer Guide.

About the authors

AWS Shield Advanced is embracing the AWS WAF Anti-DDoS managed rule group: What changes and how to prepare

2026-07-27 Eitav Arditti

Post Syndicated from Eitav Arditti original https://aws.amazon.com/blogs/security/aws-shield-advanced-is-embracing-the-aws-waf-anti-ddos-managed-rule-group-what-changes-and-how-to-prepare/

Application-layer distributed denial of service (DDoS) attacks are difficult to detect because they closely resemble legitimate traffic. HTTP request floods are now among the most common vectors targeting web applications, using valid-looking requests that blend in with normal user activity.

In June 2025, AWS launched the AWS WAF Anti-DDoS managed rule group, built specifically for application-layer (L7) DDoS protection. AWS Shield Advanced is adopting it as the default application-layer protection, and in time as the only one. On July 27, AWS Shield Advanced begins adding the Anti-DDoS managed rule group to eligible web access control lists (ACLs) in Count mode. It will not cause any interruption to your traffic alongside your existing L7 automatic mitigation and WAF rules. In this blog post, we provide details regarding the Anti-DDoS managed rule group and when the change is expected to reach your web ACLs. You will understand the phases and steps that you need to take before the finish date, including how the monitoring and metrics will change.

Anti-DDoS managed rule group features

The Anti-DDoS managed rule group builds on what Shield Advanced automatic mitigation already provides. It profiles your traffic, learns what normal traffic looks like for your application, and establishes a baseline in minutes rather than hours. When an attack starts, it reacts within seconds and there are no health checks to configure. The rule group adds a Challenge action to the Block and Count actions you already use. Challenge decisions are driven by the AMR labels that mark the suspicion level of each inspected request. One option is a silent browser challenge, which has a background verification that runs in the visitor’s browser with no interstitial page, so legitimate users are never interrupted while automated traffic is filtered out. You can also exclude workload paths that don’t support Challenge, which fall back to Block mitigations instead. Sensitivity is configurable to Low, Medium, or High, and you set it separately for Block and Challenge. Block and Challenge are tuned independently; meaning you can run Challenge at high sensitivity to catch more suspicious traffic while keeping Block low to avoid dropping legitimate requests or reverse it for a stricter posture.

The rest is about cost and visibility:

It uses less capacity than before. The rule group needs 50 web ACL capacity units (WCUs), down from the 150 the previous protection required, providing you with capacity for the rest of your rules.
The dashboard ships in the AWS Management Console for AWS WAF. It’s there now, showing live DDoS events, match metrics, and the top URIs, geographies, and IP addresses driving traffic.
It labels everything it inspects. Requests get labels for event-detected, graduated suspicion levels, and specific rules. Match on those labels in your own AWS WAF rules when you need logic the rule group doesn’t cover.
You don’t pay for the attack traffic. During active mitigation, blocked DDoS requests drop out of your monthly count. That exclusion covers AWS WAF request fees, Anti-DDoS managed rule group request fees, and Shield Advanced request charges.

AWS Shield Advanced isn’t required to use any of these features. Shield Advanced subscribers get the rule group included with AWS WAF and any customer can turn it on independently. See AWS WAF pricing for more information on costs.

Implementation details

Shield Advanced upgrades application-layer DDoS protection in five phases. The following dates are when AWS will act automatically, not the earliest date when you can act. After the rule group is deployed in Count mode on July 27, 2026, you can begin migrating right away rather than waiting for the October auto-upgrade. There’s no window where protection lapses. Your current automatic mitigation stays active through every phase until the Anti-DDoS managed rule group takes over. That handoff happens in a single operation, with no cutover window and no gap for your traffic flows.

Phase 1: Anti-DDoS managed rule group deployed in Count mode (rolling out July 27–August 7, 2026)

AWS adds the Anti-DDoS managed rule group in Count mode to every web ACL eligible for this rollout. Eligible means any Shield Advanced web ACL with at least one resource using application-layer automatic mitigation that isn’t already running the Anti-DDoS rule group. This is a broader set than the web ACLs eligible for the October auto-upgrade (Phase 3), which applies a stricter test. The deployment rolls out gradually, starting July 27 and expected to finish by August 7, 2026, so different web ACLs might be updated on different days. There’s no impact to your traffic because the rule group watches and labels requests without acting on them while your existing automatic mitigation keeps running. Throughout the evaluation period, you receive DDoS events, metrics, and AWS WAF labels at no additional charge.

Phase 2: Free evaluation period (July 27–September 30, 2026)

The existing automatic mitigation and the Anti-DDoS managed rule group run side by side each detecting independently. Automatic mitigation continues to protect your resources while the rule group operates in Count mode. To compare their detection results, use the DDoSAttackRequests metric, AWS WAF labels, and the Anti-DDoS dashboard. All Anti-DDoS managed rule group charges are waived during this period, including the subscription fee, per-request fees, and WCU consumption costs for the eligible web ACLs from phase 1.

Phase 3: Auto-upgrade begins (October 1, 2026)

For eligible web ACLs, the auto-upgrade mirrors your existing automatic mitigation configuration. The rule group inherits your current setting, so a Block configuration comes up in Block mode and a Count configuration comes up in Count mode in a single, atomic operation. The rule group takes over in the same step that disables automatic mitigation, so protection never drops for an instant. This is a handoff rather than a cutover with no window where your resources are unprotected. If you’d rather not upgrade you can opt out by contacting AWS Support before the auto-upgrade date.

Phase 4: Guided migration (available July 27–December 31, 2026)

You don’t have to wait for the October auto-upgrade to migrate. As soon as the rule group is deployed in Count mode between July 27 and August 7, 2026, you can move to it on your own schedule. This is the path to use for web ACLs that aren’t eligible for the Phase 3 auto-upgrade, meaning mixed-mode web ACLs or ones with resources that don’t have automatic mitigation enabled. Work with your AWS account team and AWS Support at any point in this window to plan and complete the migration. Eligible web ACLs are also upgraded automatically starting October 1 (Phase 3), so guided migration is mainly for the web ACLs the auto-upgrade can’t cover.

Phase 5: Shield Advanced application-layer automatic mitigation sunset (January 1, 2027)

As of January 1, 2027, the Shield Advanced application-layer automatic mitigation feature will no longer be available. Resources that haven’t migrated to the Anti-DDoS managed rule group will lose automatic application-layer DDoS mitigation.

Capability	Shield Advanced application layer automatic mitigation	Anti-DDoS managed rule group (AWSManagedRulesAntiDDoSRuleSet)
Feature type	Shield Advanced automatic mitigation	AWS WAF managed rule group
Detection and mitigation speed	Requires a baseline period; mitigation varies per event	Enhanced detection and faster mitigation
Configuration scope	Per resource (Shield API)	Per web ACL (AWS WAF API)
Mitigation actions	Count, Block	Count, Block, and Challenge
Sensitivity controls	None	Low, Medium, and High for both Block and Challenge
Non-HTML path handling	N/A	URI regex exemptions for Challenge
WCU consumption	150 WCUs	50 WCUs
Health checks	Required (Amazon Route 53 health-based detection)	Not required, provides automatic traffic profiling
Availability	Shield Advanced only	AWS WAF and Shield Advanced (see pricing)

Observability

The existing automatic mitigation and the Anti-DDoS managed rule group use separate Amazon CloudWatch namespaces and metric structures. The rule group gives you three tiers of observability: tier 1 tells you an attack is happening, tier 2 shows which requests it flagged and why, and tier 3 shows what it did about them. You don’t need all three on day 1 because most customer teams start at tier 1 to confirm detection is working, then add the others as they tune.

Tier 1: Event detection alarms

You can detect DDoS events using two CloudWatch metrics, each with its own namespace.

	DDoSDetected (Shield)	DDoSAttackRequests (Anti-DDoS managed rule group)
Namespace	AWS/DDoSProtection	AWS/WAFV2
Requires Shield Advanced	Yes	No
Scope	L3, L4, and L7 events	L7 events only
Value during event	Binary (0 or 1)	Count of requests observed
Value outside event	Reported once daily (keeps metric alive)	Absent (no data points)
Dimensions	ResourceArn	Resource, ResourceType

What this means for your existing alarms:

After the application-layer automatic mitigation feature is sunset, DDoSDetected still fires for infrastructure layer 3 and layer 4 events, so your existing network layer and transport layer alarms remain valid. For the full list, see AWS Shield Advanced metrics.
DDoSAttackRequests is the Anti-DDoS managed rule group equivalent for application-layer event detection. Alarm on Sum >= 1 to detect any event, or set a volume threshold (for example, more than 10,000 requests per minute) for severity-based alerting.
During the evaluation period, both metrics fire independently and you can validate detection parity before migrating your application-layer alarms.
Because DDoSAttackRequests is absent when there are no active DDoS events, set treat-missing-data to missing or notBreaching for alarms on this metric.

Tier 2: Detection labels for custom monitoring

Every request the Anti-DDoS managed rule group evaluates gets a label. Where tier 1 tells you an attack started, tier 2 shows which requests looked suspicious and how confident the rule group was. The labels surface as AWS WAF metrics in the AWS/WAFV2 namespace: AllowedRequests, BlockedRequests, and CountRuleMatch. Each carries the LabelName and LabelNamespace dimensions under the awswaf:managed:aws:anti-ddos: namespace.

event-detected – Requests observed during a detected DDoS event
ddos-request – Requests identified as part of the attack
low-suspicion-ddos-request, medium-suspicion-ddos-request, high-suspicion-ddos-request – Graduated suspicion levels
challengeable-request – Requests eligible for browser challenge

Chart suspicion-level trends on a CloudWatch dashboard to see how an attack builds. Match on the labels in your own AWS WAF rules or dig into them in your AWS WAF logs with CloudWatch Logs Insights or Amazon Athena when you need to understand a specific event after the fact.

Tier 3: Mitigation action metrics

Where tier 2 shows what the rule group flagged, tier 3 shows what it did about those requests during an event. You’ll find these metrics as ChallengeRequests, BlockedRequests, and CountRuleMatch, each scoped by the rule label that produced it.

ChallengeAllDuringEvent – Requests challenged during an active event
ChallengeDDoSRequests – Suspected DDoS requests challenged based on suspicion level
DDoSRequests – Requests blocked (or counted in Count mode)

Watch these during a live event to see whether mitigation is keeping up. If you’re challenging far more requests than you’re blocking, your configuration might be too cautious, and you can raise the sensitivity level after you trust the numbers.

Observability summary

Tier	Automatic mitigation	Anti-DDoS managed rule group
Event alarm	DDoSDetected in AWS/DDoSProtection (binary, L3/L4/L7)	DDoSAttackRequests in AWS/WAFV2 (request count, L7)
Detection labels	None	event-detected, ddos-request, suspicion levels, challengeable-request
Mitigation actions	Not visible (Shield-managed rule group metrics not exposed)	ChallengeAllDuringEvent, ChallengeDDoSRequests, DDoSRequests
Dashboard	Shield console event history	Shield console and Anti-DDoS dashboard in the AWS WAF console
Historical analysis	Shield event history only	AWS WAF logs (CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), Amazon Data Firehose)

Billing

Your Shield Advanced subscription includes the Anti-DDoS managed rule group for up to 50 billion requests per month, counted across your whole organization at the payer account level. For most customers that ceiling is well above normal traffic, so you won’t see a line item here unless you’re operating at very high volume. For the exact rates, see AWS WAF pricing and Shield Advanced pricing.

You aren’t charged for DDoS traffic while the Anti-DDoS managed rule group is actively mitigating, which means Block or Challenge mode rather than Count. This applies to AWS WAF request fees, Anti-DDoS managed rule group request fees, and Shield Advanced request charges. Leaving the rule group in Count mode past the evaluation period costs you the protection without the billing relief, so avoid staying in Count mode longer than you need to validate.

During the evaluation period (July 27 to September 30, 2026), the eligible web ACLs AWS auto-enrolled don’t incur per-request fees or WCU consumption, even when configured in Count mode.

The Anti-DDoS managed rule group works at the web ACL level, so every resource you associate with a web ACL shares that coverage. Before assuming a single resource accounts for the whole cost, look at how many resources sit behind each web ACL. A web ACL fronting 20 resources bills differently from one fronting 2, so check that count first and familiarize yourself with the workload protected by each web ACL.

Adding the Anti-DDoS managed rule group to a web ACL yourself isn’t part of the upgrade path, so standard pricing applies from the moment you enable it. The same is true for any resource that was already running the rule group before the rollout. To get the free evaluation, let the automatic rollout reach your web ACLs rather than adding the rule group ahead of it. There’s no penalty for adding it yourself; you just don’t receive the waiver on that web ACL.

Update your infrastructure as code

If you manage web ACLs with AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), Terraform, or other infrastructure as code (IaC), the auto-upgrade changes your infrastructure configuration outside your templates. Your code is still the source of truth, so you need to do two things. First, change where the protection is declared. Today you enable application-layer automatic mitigation through the Shield API (EnableApplicationLayerAutomaticResponse), configured per protected resource. The Anti-DDoS managed rule group is configured through the AWS WAF API instead (CreateWebACL and UpdateWebACL), as a managed rule group statement inside the web ACL, scoped per web ACL rather than per resource. In IaC terms, you remove the Shield automatic-response block (for example, Terraform’s aws_shield_application_layer_automatic_response) and add the WAF managed rule group statement shown in the following section. Second, pull the upgraded web ACL back into your tooling before your next deploy, or your pipeline will try to revert the change.

For the full statement in Terraform, CloudFormation, and the AWS CDK, plus how to sync state after the auto-upgrade (terraform plan, CloudFormation drift detection, cdk diff), see the iac-webacl-examples helper.

Update your AWS Firewall Manager policy

If you run a Shield Advanced policy in AWS Firewall Manager today, that policy is what enabled application-layer automatic mitigation across your accounts. To keep that protection, add the Anti-DDoS managed rule group to an AWS WAF Firewall Manager policy. Your Shield Advanced policy still handles L3 and L4, while the application-layer piece moves to the AWS WAF policy. The migration is straightforward: add or reuse an AWS WAF Firewall Manager policy, put the Anti-DDoS managed rule group in it, and scope it to the same accounts and resources your Shield Advanced policy covers.

You can’t add the rule group from the Shield console or by editing an account-level web ACL directly, because Firewall Manager owns the web ACLs it creates and overwrites local edits. Instead, add the rule group to the AWS WAF policy and Firewall Manager pushes it to every in-scope account.

You can make this change in the console or as code. If you manage your Firewall Manager policies as code, don’t edit them in the console: add a new AWS WAF policy or update an existing one in your templates with the Anti-DDoS managed rule group included, and deploy it using the following Firewall Manager policies using IaC steps. Otherwise, use the console.

In the console, follow Creating an AWS Firewall Manager policy for AWS WAF to create the policy and reach the Edit policy rules page. Add the Anti-DDoS rule group, listed there as AWS AntiDDoS Protection for Layer 7 attacks (AWSManagedRulesAntiDDoSRuleSet), as a new rule group under First rule groups so it evaluates before your other managed groups, but below any Allow custom rules you use to fast-path known-good traffic.

If you protect CloudFront distributions, make this change in your Global policy, and repeat it in each AWS Regional policy for regional resources. Save the policy, and Firewall Manager rolls the change out to in-scope accounts, which can take a few minutes.

After being added, the rule group appears as the first rule group in the policy, as shown in the following screenshot:

Figure 1: AntiDDoS enabled

Firewall Manager policies using IaC

If you manage Firewall Manager policies as code, make the change in your template instead of the console. The Anti-DDoS managed rule group goes into the AWS WAF policy’s ManagedServiceData, a WAFV2 policy definition carried as a JSON string, added to the first rule groups so it evaluates early. For the ManagedServiceData JSON with CloudFormation, Terraform, and AWS CDK examples, see the firewall-manager-examples helper.

Whichever path you take, scope the policy to the same accounts and resources your Shield Advanced policy already covers, so no resource loses application-layer protection during the move.

Getting started

Between July 27 and August 7, 2026, AWS will add the Anti-DDoS managed rule group in Count mode to Shield Advanced web ACLs that have resources using application layer automatic mitigation but not yet the Anti-DDoS rule group. After it reaches your web ACL, you can evaluate it, and migrate whenever you’re ready, without waiting for the October auto-upgrade.

Review the Anti-DDoS dashboard in the AWS WAF console. The dashboard shows real-time DDoS events, match metrics, and top traffic sources.
Compare event detection side by side. During Count mode, both systems detect independently. Check the DDoSDetected metric in AWS/DDoSProtection alongside DDoSAttackRequests in AWS/WAFV2 to validate detection parity for your resources. You can deploy the CloudWatch comparison dashboard from the AWS Samples repository to view both systems on a single dashboard.
Explore AWS WAF labels. Enable AWS WAF logging and query for labels in the awswaf:managed:aws:anti-ddos: namespace. Look at suspicion levels (low-suspicion-ddos-request, medium-suspicion-ddos-request, high-suspicion-ddos-request), event-detected, and challengeable-request to see per-request visibility into detected events.
Start with Low sensitivity for Block actions during evaluation to minimize false positive risk. Tune up as you gain confidence from the Anti-DDoS dashboard and AWS WAF label data.
Plan your configuration. Review sensitivity levels, URI exemptions for non-HTML paths, and web ACL priority placement. The Anti-DDoS managed rule group should run at the highest priority in your web ACL, or right below any custom rules with the Allow action.
Sync your IaC templates. After the auto-upgrade adds the Anti-DDoS managed rule group to your web ACL, fetch the current state into your IaC tooling (Terraform refresh, CloudFormation drift detection, AWS CDK import) before your next deployment.

Conclusion

The Anti-DDoS managed rule group profiles your traffic within minutes and mitigates within seconds, where the automatic mitigation it builds on established its baseline over hours, and it gives you granular visibility into what it’s doing. The evaluation period exists so you can watch both systems run on your own traffic before anything changes. Spend the first few weeks in Count mode confirming the new detection matches what you see today, then move your alarms over and pick a sensitivity level you’re comfortable with. If you run a web ACL across several resources, or you manage rules through AWS Firewall Manager, contact AWS Support before you start so you don’t have to unwind anything later. The Shield Advanced application-layer automatic mitigation feature retires on January 1, 2027, and anything still relying on it needs to be migrated by then.

Resources

If you have feedback about this post, submit comments in the
Comments section below.

Introducing the Amazon GuardDuty investigation agent: on-demand AI-powered threat assessment

2026-07-21 Allan Holmes

Post Syndicated from Allan Holmes original https://aws.amazon.com/blogs/security/introducing-the-amazon-guardduty-investigation-agent-on-demand-ai-powered-threat-assessment/

The new Amazon GuardDuty investigation agent (now in public preview) investigates security findings across your Amazon Web Services (AWS) environment, reducing investigation time from hours to minutes.

GuardDuty is our managed threat detection service that continuously monitors your AWS accounts and workloads for suspicious, potentially malicious activity, and unauthorized behavior, delivering detailed security findings for visibility and remediation.

Whether you’re investigating a single suspicious finding or assessing security posture across your entire organization, the investigation agent provides structured assessments providing risk levels, confidence scores, and actionable recommendations.

Security teams can spend hours investigating security findings and correlating data across multiple tools. The GuardDuty investigation agent automates this correlation, providing actionable intelligence, built directly into GuardDuty and accessible on demand through the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS APIs, or AWS SDKs.

This post shows you how to:

Enable the investigation agent in your GuardDuty console.
Create your first investigation through the console or AWS CLI.
Use the investigation agent with the AWS MCP server for AI-assisted security operations

Key features of the GuardDuty investigation agent

The GuardDuty investigation agent provides APIs using the same patterns you already know from GuardDuty. Each completed investigation returns a risk level, confidence assessment, MITRE ATT&CK® technique mapping, resource mapping, and prioritized recommendations.

You can scope investigations from the console for a specific finding, an account, or all accounts across your organization. Alternatively, the AWS CLI and API accept a free-form trigger prompt of up to 2,048 characters, so you can describe what to investigate in natural language and guide the analysis of the agent by specifying areas of concern, suspected root causes, or priorities for the investigation.

The investigation agent APIs are also available through the official AWS MCP server, part of the Agent Toolkit for AWS, enabling integration into your existing security toolchains and AI-powered workflows. You don’t need to manage or interact with the agent directly. Call API endpoints, and the agent investigates findings, correlates evidence, and delivers an assessment without the overhead of managing complex configurations.

How the investigation agent analyzes findings

When you create an investigation, the agent uses cross-Region inference to process your findings based on scope and produces a structured output.

Cross-Region inference – GuardDuty investigation uses the Cross-Region Inference Service (CRIS), which selects the optimal AWS Region within your geography to process the investigation assessment. Your data remains stored only in the Region where the investigation request originates. However, investigation data and summary results might be processed outside that Region. Data is transmitted encrypted across the secure network provided by Amazon.

For more information about which inference Regions your request might be routed to see the Cross-Region inference routing table located in the investigation section of the Amazon GuardDuty User Guide.

Investigation output – Each completed investigation produces the following insights: Risk level (Info, Low, Medium, High, or Critical), Confidence (Unknown, Low, Medium, or High), Summary (description of findings and key observations), Investigation Details (additional context), and Recommended Actions (detailed actions including AWS CLI commands).

Account scoping – Account specification is required only when investigating a specific member account. For broader scopes such as your entire organization, no account ID is needed. The agent will only investigate findings within accounts you’re authorized to access per the authorization model that follows.

Prerequisites

Before you get started, make sure you have the following prerequisites in place:

Amazon GuardDuty enabled in your account
AWS account in a supported Region (see Availability section)

Required IAM permissions

You will need three new permissions: guardduty:CreateInvestigation to start new investigations, guardduty:GetInvestigation to retrieve results, and guardduty:ListInvestigations to view investigations for a given detector.

Example IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "guardduty:CreateInvestigation",
        "guardduty:GetInvestigation",
        "guardduty:ListInvestigations"
      ],
      "Resource": "*"
    }
  ]
}

Authorization model

Administrator accounts can create investigations, retrieve results, and view investigation lists for themselves and their member accounts. Member accounts can only retrieve results and view investigation lists for their own account. Member accounts can’t create investigations and can’t access investigations belonging to other accounts or the administrator account. Account specification is required only when investigating a specific member account. For your own account or accounts across your organization, no account ID is needed.

To enable and create your first investigation

Before you begin, verify you have the required IAM permissions as described in the prerequisites .

Open the AWS Management Console in the desired supported Region and navigate to Amazon GuardDuty.
In the navigation pane, choose Investigations.

Figure 1: GuardDuty investigation dashboard

If investigations aren’t enabled choose Go to Settings and then enable investigations by choosing Enable.

Figure 2: GuardDuty investigations enablement screen

After investigations are enabled, navigate back to the investigations page.
In the navigation pane, choose Initiate Investigation.

Figure 3: GuardDuty initiate investigation

Select a scope for your investigation:
- Enter a GuardDuty Finding ID: Use when you want to investigate a specific GuardDuty finding in depth
- Enter an AWS Account ID: Use when you want to assess the overall security posture of a specific AWS account
- All accounts: Use for organization-wide security assessment or when investigating potential lateral movement
- Choose Initiate investigation.

Figure 4: GuardDuty investigation setup

Wait for the investigation to complete (typically 2–5 minutes for account level and 10–12 minutes for specific finding investigations during preview). The status updates automatically.
When the investigation completes, select the investigation title to view the full assessment.

Figure 5: GuardDuty investigation completed menu

The investigation assessment contains detailed information about the investigation including general information, a summary of the investigation, mapping, assessment of the threat, and recommended actions.

The General Information section displays the investigation ID, status, triggered-by account, and creation timestamp.

Figure 6: General information section of the assessment

The summary section provides a narrative of key observations and findings.

Figure 7: Summary section of the assessment

The mapping section shows attack techniques and affected AWS resources.

Figure 8: MITRE ATT&CK mapping section of the assessment

The Threat Assessment section displays the risk level, confidence score, and detailed threat analysis.

Figure 9: Threat assessment section

The Recommended Actions section lists prioritized remediation steps.

Figure 10: Recommended actions section of the assessment

Investigations can also be conducted with the AWS CLI or SDK using the following API endpoints:

CreateInvestigation – Initiates a GuardDuty investigation that automatically analyzes security findings, correlates related activity, performs account-level analysis, and produces a structured investigation summary with recommended next steps.
GetInvestigation – Retrieve the status and results of a specific investigation, including the assessment from the agent, correlated evidence, and recommended actions when completed.
ListInvestigations – View investigations across your environment with filtering and pagination.

To run investigations using the AWS CLI

Investigations are asynchronous because the agent queries multiple data sources, correlates findings across services, and performs AI-based analysis. After creating an investigation, you’ll need to check its status periodically until it completes.

Step 1: Find your detector ID

Each GuardDuty deployment has a unique detector ID per-account and per-Region that identifies your specific GuardDuty configuration. You will need this for all AWS CLI operations, especially if you have GuardDuty enabled in multiple Regions. You can find your detector ID in the GuardDuty console under Settings, or by running the following command and specifying the Region. For example, if the GuardDuty detector of interest were in the us-east-1 (N. Virginia) Region

aws guardduty list-detectors –-region=us-east-1

Expected response:

{
  "DetectorIds": [
    "12abc34d567e8fa901bc2d34eexample"
  ]
}

Note: the DetectorIDvalue from the response, you will use it in all subsequent commands.

Or if working only in the same Region, the session can be set as an environment variable to avoid repetition, for example on Linux:

export AWS_DEFAULT_REGION=us-east-1

See the AWS CLI documentation for guidance on configuring this for additional operating systems.

Step 2: Create an investigation

The following is an example of code to investigate a specific finding:

aws guardduty create-investigation us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--trigger-prompt "Investigate this finding ID 1ab2c3d4e5f6a7b8c9d0e1f2a3b4c5d6"

The --trigger-prompt parameter is useful when you have context that isn’t captured in GuardDuty metadata or consumable through the API.

Expected response:

{
  "InvestigationId":"a1b2c3d4-5678-90ab-cdef-ef1234567890"
}

To investigate findings across an entire AWS account, use the following example:

aws guardduty create-investigation –-region=us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--trigger-prompt “Investigate findings in Account 123456789012”

To investigate findings across an entire organization:

aws guardduty create-investigation –-region=us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--trigger-prompt “Investigate findings across my AWS Organization”

Step 3: Check investigation status

Check the status of the investigation shown here using the AWS CLI query command to filter and list only the Status section of the output for simplicity:

aws guardduty get-investigation –-region=us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--investigation-id a1b2c3d4-5678-90ab-cdef-ef1234567890 --query 'Investigation.Status'

Repeat this command until the Status field shows COMPLETED.

Example completed response output:

{
  "Investigation": {
    "InvestigationId": "a1b2c3d4-5678-90ab-cdef-ef1234567890",
    "Status": "COMPLETED",
    "TriggerPrompt": "Investigate finding 1ab2c3d4e5f6a7b8c9d0e1f2a3b4c5d6 in account 123456789012",
    "TriggeredBy": "123456789012",
    "RiskLevel": "Critical",
    "Risk": "Active multi-stage runtime compromise on EKS worker node with root-privileged reverse shell, Docker socket access, malicious file execution, and 500 multi-tactic runtime signals — behavioral evidence is consistent with a genuine intrusion.",
    "Confidence": "High",
    "Summary": "{\"keyObservations\":{\"title\":\"...\",\"narrative\":\"...\",\"observations\":[...]},\"countermeasures\":[...],\"threatAssessment\":{...}}",
    "Cloud": {
      "Provider": "AWS",
      "Region": "us-east-1",
      "Account": "123456789012"
    },
    "Metadata": {
      "Product": {
        "Name": "AmazonGuardDuty AI Analyst",
        "Feature": "Investigation"
      },
      "Version": "1.0.0"
    },
    "StartTime": 1705319400.0,
    "EndTime": 1705319700.0
  }
}

Status values – RUNNING, COMPLETED, FAILED
Timing – Investigation times can very. Checking status every 30 seconds should be sufficient to yield results.
If status shows FAILED – Review the error message in the response and verify your permissions match the authorization model requirements.

To list all investigations for a given detector run the following, the max-results command is optional but useful to filter the number of returned results.

aws guardduty list-investigations –-region=us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--max-results=10

Beyond running investigations manually, the API-first design addresses a common customer pattern: sending GuardDuty findings to third-party tools. You can now add automated investigation to those existing pipelines, so your team receives enriched, prioritized intelligence rather than raw alerts.

Consider a customer that routes GuardDuty findings through Amazon EventBridge to their Security Information and Event Management (SIEM) platform, where analysts manually investigate each alert. With the investigation agent, an AWS Lambda function can be placed into the pipeline that calls CreateInvestigation with the finding ID, waits for completion, and forwards the enriched results (risk level, confidence score, MITRE ATT&CK mapping, and recommended actions) to their SIEM alongside the original finding. Critical findings route directly to the customer incident response queue for further analysis or automation. Low-risk findings with high confidence get auto-closed or batched for weekly review. The analyst’s time shifts from repetitive log correlation to validating assessments and acting on confirmed threats.

This pattern works with SIEMs, ticketing systems, or automation platforms that can be customized to use the API or EventBridge messaging. The investigation agent fits into the pipeline as a processing step, not a destination.

The agent is fine-tuned on investigating GuardDuty findings. It’s distinct from other AWS frontier agents such as the AWS Security Agent and AWS DevOps Agent. The scope of the investigation agent is focused to deliver specialized analysis of GuardDuty findings.

Integration with the AWS MCP server

The Model Context Protocol (MCP) is an open standard that allows AI assistants to securely connect to external data sources and tools. Because the AWS MCP server implements this standard for AWS services, you can use it to add GuardDuty investigations into AI-powered workflows using tools like Kiro, Anthropic’s Claude, or other MCP-compatible clients.

To configure the AWS MCP server

Configure your MCP client to connect to the AWS MCP server.
Use natural language to invoke investigations (for example,“Investigate the recent Unauthorized Access finding for account 123456789012″).
Review the investigation results returned through your MCP client. These results can vary depending on the model or agent being used, configuration, and the non-deterministic nature of AI.

Integrate the results into your existing agent automation or take manual action based on the findings.

Additional usage examples

“Investigate the latest high-severity finding in my production account”
“Create an investigation for finding ID abc123 in account 987654321098 and summarize what happened”
“List investigations from the last 24 hours and flag those that need human review”

How the investigation agent relates to AWS Security Incident Response

At re:Invent 2024, AWS launched AWS Security Incident Response (AWS SIR), a managed service that you can use to quickly prepare for, respond to, and recover from security incidents. AWS SIR and the GuardDuty investigation agent address different stages of your security workflow. The GuardDuty investigation agent provides an on-demand assessment capability. When your team needs deeper context on a specific finding, an account security posture, or the overall security posture of your organization. You create an investigation and receive a structured assessment with risk levels, confidence scores, MITRE ATT&CK® technique mappings, and actionable recommendations. Security analysts can use this to quickly understand the scope and severity of what GuardDuty has detected.

When you create an AWS-supported case through AWS SIR, a SIR investigation agent activates, working in parallel with AWS Security Incident Response engineers to gather evidence and deliver an investigation summary within minutes. AWS SIR is purpose-built for active security events where you need both AI-powered automation and human expertise to coordinate containment and recovery.

Security teams can use these capabilities to assess and prioritize findings on demand using the GuardDuty investigation agent, escalate confirmed issues to stakeholders with supporting evidence, and create or update an AWS-supported case to accelerate involvement from the AWS SIR team when additional support is needed.

Availability and pricing

Public preview of the GuardDuty investigation agent is available in 10 AWS Regions including US East (N. Virginia), US East (Ohio), US West (Oregon), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), Europe (Stockholm), and Asia Pacific (Tokyo).

During public preview, the investigation agent is available at no charge. Usage is limited to 10 investigations per account per day, with a cumulative limit of 100 investigations per account during the preview period. Failed investigations do not count toward these quotas.

Start investigating findings today

The Amazon GuardDuty investigation agent reduces investigation time from hours to minutes, letting your security team focus on confirmed security events rather than manual correlation.

Get started by:

Enabling the investigation agent in your GuardDuty console
Creating your first investigation using a recent GuardDuty finding
Reviewing the structured assessment, including risk level and recommended next steps

For organizations using the AWS MCP server, you can also invoke investigations through natural language in your AI assistant of choice.

Learn more

Amazon GuardDuty Investigation Agent documentation – Detailed configuration and usage guidance
IAM permissions reference for GuardDuty – Required permissions for investigation operations
Amazon EventBridge User Guide – Learn how to route investigation events to downstream systems
MITRE ATT&CK®Framework – Getting started resources
AWS Security Incident Response – AWS Security Blog article
AWS Activation Days – Information about upcoming complimentary events that provide you prescriptive guidance to achieve outcomes when using AWS

If you have feedback about this post, submit comments in the Comments section below.

High-performance Remote Shuffle Service on Amazon EMR with Apache Celeborn

2026-07-15 Suvojit Dasgupta

Post Syndicated from Suvojit Dasgupta original https://aws.amazon.com/blogs/big-data/high-performance-remote-shuffle-service-on-amazon-emr-with-apache-celeborn/

Organizations running large-scale Apache Spark workloads often face a trade-off between achieving lower cost and job reliability. These tradeoffs are more prominent when using Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances or when their jobs process highly skewed datasets. Three shuffle-related challenges drive the pain:

Spot interruptions trigger costly recomputation: Spot instances can reduce compute spend by up to 90 percent compared to On-Demand instances, but they can be reclaimed with only two minutes of notice. When a Spark executor on a Spot Instance is interrupted, its local shuffle data is lost, and Spark must recompute entire upstream stages to regenerate that data. For shuffle-heavy jobs processing terabytes of data, frequent interruptions cause cascading recomputation and runtime delays, quickly eroding the savings that made Spot attractive in the first place.
Local shuffle storage causes cluster-wide over-provisioning: In YARN-based Hadoop architectures, including Amazon EMR on EC2, the External Shuffle Service (ESS) stores shuffle data locally on each Node Manager’s node alongside the Spark executor that produced it. Every node must carry large memory and disk allocations to accommodate shuffle output, yet only a few EC2 nodes perform most of the shuffle work. The rest sit oversized and underused. This is a classic coupled storage-compute problem. By decoupling shuffle storage to a dedicated, storage-optimized tier, you can right-size your compute for actual demands.
Shuffle data protection leaves compute idle: To guard against local shuffle data loss, Spark’s scaling logic prevents nodes that still hold shuffle data from scaling down. EC2 nodes sit idle long after Spark tasks complete. Data skew amplifies this effect: tail-end tasks run far longer than typical ones, delaying shuffle reads and postponing scale-down across the cluster.

Together, these challenges force a difficult choice: cheaper infrastructure or predictable jobs. In this post, we show how Apache Celeborn resolves this trade-off for Amazon EMR on EKS and Amazon EMR on EC2, improving job reliability while unlocking additional cost savings.

What is Apache Celeborn?

Apache Celeborn is an open-source Remote Shuffle Service (RSS) that solves the preceding problems by decoupling shuffle data from the executor lifecycle entirely. It uses a Leader-Worker-Client architecture: Leader nodes manage metadata, Workers read and write shuffle blocks, and Clients integrate with compute engines. Instead of writing shuffle output to local disks, Spark executors push data to a shared, storage-optimized Celeborn cluster that persists shuffle data independently of executor location. This means EMR executor nodes can run on 100% Spot Instances. Spot reclamations no longer cause shuffle data loss. Executors scale in and out freely without triggering upstream recomputation.

Celeborn also provides Raft-based high availability, per-job data replication, and a pluggable shuffle manager that replaces Spark’s default mechanism with minimal configuration changes. With its push-based model, Spark executors send shuffle data directly to Celeborn workers, which cache and consolidate partitions. This reduces the N×M network connections during the read phase, improving both performance and stability at scale.

Push-based remote shuffle service where Spark executors on Amazon EMR push shuffle data to a shared Celeborn cluster

Image 1: Push-based Remote Shuffle Service for Spark on EMR

Overview of solution

In this post, we show you how to deploy a Celeborn cluster alongside EMR on EKS and EMR on EC2. The solution also includes an observability stack to provide operational visibility into the Celeborn cluster. Metrics are collected by the AWS Distro for OpenTelemetry (ADOT) collector and routed to two monitoring paths. The AWS managed option uses Amazon Managed Service for Prometheus and Amazon Managed Grafana. The open source option uses self-managed Prometheus with a built-in Grafana.

Apache Celeborn can be deployed in several ways depending on your operational requirements and scale. These two deployment patterns are the main ones:

Co-located on the same cluster: Celeborn runs on the same compute environment as Spark. This is the most straightforward operational model, with no cross-cluster networking. The main constraint is shared cluster lifecycle: any upgrade or termination affects Celeborn and running Spark jobs simultaneously.
Separate Celeborn cluster: Celeborn runs on its own EC2 or EKS cluster, fully isolated from Spark compute. This is the most operationally flexible model and is the focus of this post.

In this solution, Celeborn runs on a dedicated Amazon Elastic Kubernetes Service (Amazon EKS) cluster, separate from the EMR Spark environments. The two workloads have different resource profiles. Celeborn is storage and network I/O intensive, while Spark is CPU and memory intensive. By separating them, each cluster can use instance types optimized for its workload. It also improves independent lifecycle management, so you can upgrade or scale Celeborn clusters without disrupting Spark jobs, and the other way around.

Solution architecture with EMR on EKS and EMR on EC2 connecting to a shared Celeborn cluster through an internal Network Load Balancer

Image 2: Solution Architecture

As the architecture diagram shows, two types of EMR deployment models, EMR on EKS and EMR on EC2, connect to a shared Celeborn cluster through an internal Network Load Balancer (NLB). Behind the scenes, Spark executors register their shuffle partitions to Celeborn workers through this connection, while reducers read consolidated data back during the fetch phase.

The following are the key design considerations for this solution:

Streamlined operation by a shared RSS model: Celeborn runs on a dedicated EKS cluster. This provides lifecycle independence, allows each cluster to use workload-optimized instance types, and allows a single Celeborn cluster to serve multiple EMR clusters as a shared service.
Cross-cluster connectivity: Clusters reside in the same Amazon Virtual Private Cloud (Amazon VPC) and share private subnets. The AWS Load Balancer Controller on the Celeborn cluster provisions an internal NLB exposing its active primary pods on ports 9097 (RPC) and 9098 (dashboard). The NLB DNS name is VPC-resolvable, so any EMR cluster in the same VPC can reach Celeborn by setting spark.celeborn.master.endpoints to the NLB address.
Restricted and secured networking: The Celeborn cluster only allows inbound traffic from the EMR on EC2 and EMR on EKS clusters, sending shuffle data and metrics over the network to the Celeborn cluster.
State persistence: Primary nodes maintain Celeborn’s coordination state through Raft consensus, which requires storage that survives pod restarts. Deploying them as StatefulSets with EBS-backed persistent volume claims (PVC) lets a restarted primary pod recover its Raft log and identity from durable storage rather than starting from scratch. Workers keep shuffle data on local NVMe instance store for performance, but this is ephemeral. To protect data loss on workers, each shuffle partition is replicated to two other workers by setting spark.celeborn.client.push.replicate.enabled=true.
Observability: ADOT collector is deployed on the Celeborn EKS cluster. It scrapes Prometheus metrics from Celeborn’s pods, and simultaneously remote writes them to two monitoring backends. Option 1 uses Amazon Managed Service for Prometheus as the metrics store, with Amazon Managed Grafana surfacing pre-built dashboards for Celeborn cluster health and Java Virtual Machine (JVM) metrics. This option requires AWS IAM Identity Center. Option 2 deploys a self-managed Prometheus stack with a built-in Grafana on the Celeborn EKS cluster, with Prometheus configured as a remote-write receiver for the same ADOT collector. This option suits environments without IAM Identity Center or those preferring a single open-source tooling.

Critical configurations

The following two tables list the key configurations that make up the solution.

Spark configuration (RSS client): Tells the Spark client to use Celeborn as the shuffle manager in place of Spark’s built-in implementation.
Celeborn configuration (RSS server): Controls how the primary and worker Celeborn pods operate on Kubernetes.

Note: The values in the following tables are reference defaults used in this walkthrough. Adjust them based on your workload requirements and cluster sizing.

1. Spark configuration (RSS client)

The following table highlights some key Spark configurations required to use Celeborn as the shuffle manager. You can refer to these configurations applied for each Spark submission method in the scripts below:

EMR on EC2 cluster: emr-on-ec2/submit-job.sh.
EMR on EKS virtual cluster: emr-on-eks/submit-emr-api.sh.
EMR on EKS with Spark operator: emr-on-eks/submit-spark-operator.sh.

Parameter	Value	Purpose
spark.shuffle.service.enabled	false	Disables Spark’s built-in External Shuffle Service. Must be off before Celeborn can take its place
spark.shuffle.manager	org.apache.spark.shuffle.celeborn.SparkShuffleManager	Replaces Spark’s default SortShuffleManager with Celeborn’s shuffle manager
spark.celeborn.master.endpoints	<NLB_DNS>:9097	Points Spark to the Celeborn primary RPC endpoint through the internal NLB. You can add multiple NLB addresses here, separated by a comma.
spark.shuffle.sort.io.plugin.class	org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO	Registers Celeborn’s data I/O plugin alongside the shuffle manager
spark.celeborn.client.push.replicate.enabled	true	Default: false OPTIONAL: if shuffle performance has a higher priority than job stability, turn off the data replication at a job level. This setting replicates shuffle data across multiple Celeborn workers for fault tolerance.
spark.celeborn.client.spark.push.unsafeRow.fastWrite.enabled	false	Default: true COMPULSORY: Disables Celeborn’s optimization for UnsafeRow, making it compatible with the optimized Spark runtime in EMR
spark.dynamicAllocation.shuffleTracking.enabled	false	Default: true COMPULSORY: Disables shuffle tracking.
spark.sql.adaptive.localShuffleReader.enabled	false	Default: true. COMPULSORY: makes sure Spark does not use local shuffle readers to read the shuffle data.
spark.celeborn.client.spark.shuffle.fallback.policy	NEVER	Default: AUTO. COMPULSORY: to make sure we don’t see intermittent writes to local and remote shuffles.

2. Celeborn configuration (RSS server)

The following table provides the server-side settings that control how the Celeborn primary and worker pods operate on Kubernetes. These values are set in the Helm values.yaml file.

Parameter	Value	Purpose
master.replicas	3	Number of Celeborn primary node replicas for HA. A minimum 3 required for Raft quorum
worker.replicas	3	Number of Celeborn workers to store shuffle data, should be less than EC2 node number.
master.volumeClaimTemplates	gp3, 5 GiB	Persistent storage for Raft consensus state
worker.volumes	4 × hostPath (/mnt/nvme/disk1-4)	Shuffle data stored on local NVMe instance store is ephemeral but significantly faster than Amazon Elastic Block Store (EBS) volumes for shuffle I/O
master/worker tolerations	celeborn-dedicated	Schedules Celeborn pods only on dedicated tainted nodes
master/worker podAntiAffinity	preferred, w=100	Spreads replicas across different nodes to limit failure blast radius
image.tag	0.6.2	Pinned Apache Celeborn version

Deploy the solution

This solution contains six layers, each of which is dependent on the previous deployments. See the details in the following deployment steps:

Shared Infrastructure (Step 2).
Celeborn Remote Shuffle Service installation (Step 3).
Prepares sample data and creates EMR compute (Step 4, 5).
Observability Layer (Step 6-7 and Step 9).
Submit job (Step 8).
Cleanup (Step 10).

Note: This walkthrough creates billable AWS resources, including Amazon EKS clusters, EC2 instances, Amazon Managed Grafana, Amazon Managed Service for Prometheus, and a Network Load Balancer. To avoid ongoing charges, follow the cleanup instructions at the end of this post.

Prerequisites

Before you deploy this solution, make sure the following prerequisites are in place:

Access to a valid AWS account with AWS Organizations and IAM Identity Center enabled.
The AWS Command Line Interface (AWS CLI) installed on your local machine.
Git, kubectl, Helm 3 and curl, and Python 3 utilities installed on your local machine.
Docker installed and running.
Permission to create AWS resources, such as Amazon EKS clusters, AWS CloudFormation stacks, AWS Identity and Access Management (IAM) roles, VPC resources, and Amazon EMR clusters.
Familiarity with Kubernetes, Apache Spark, Apache Celeborn, Amazon EKS, Amazon EMR on Amazon EKS, and Amazon EMR on Amazon EC2.

Deployment steps

Step 1: Clone from the source repository

Clone the repository to your local machine and set the AWS_REGION:

git clone https://github.com/aws-samples/sample-emr-celeborn-shuffle-service.git
cd sample-emr-celeborn-shuffle-service
export AWS_REGION=<AWS_REGION>

Step 2: Deploy the shared infrastructure

This step creates the core AWS resources, including the VPC, AWS Key Management Service (AWS KMS) key, security groups, and Amazon Simple Storage Service (Amazon S3) bucket.

./shared-infra/deploy.sh

Step 3: Deploy the Celeborn cluster

This step provisions a dedicated EKS cluster for Celeborn and exposes it through an internal NLB. It follows security best practices by keeping the EKS endpoint private, allowing public access only from a deployment workstation IP, and enabling encryption for secrets plus logging for all cluster components.

./celeborn/deploy.sh

Step 4: Prepare the sample data

Execute the following script to generate sample data:

./spark-jobs/setup-data.sh

Step 5. Deploy a compute engine (choose one or both)

Create at least one of the following EMR deployment models to run Spark jobs.

Option A: EMR on EKS

./emr-on-eks/deploy.sh

Option B: EMR on EC2

./emr-on-ec2/deploy.sh

Step 6. Deploy observability

To monitor the Celeborn cluster, deploy one of the observability options. After deployment, a Grafana URL and login details are available in the .environment-info file located at the repository’s root directory. Follow the instructions in Step 9 to sign in to your Grafana dashboard. You will see two pre-built dashboards on the Grafana web UI:

Celeborn Cluster Overview: Active shuffles, worker status, disk usage, and memory utilization.
Celeborn JVM Metrics: Heap usage, garbage collection, and thread activity.

Option A: AWS managed services

This option deploys an Amazon Managed Service for Prometheus workspace for metrics storage and an Amazon Managed Grafana workspace to host dashboards. Amazon Managed Grafana requires IAM Identity Center, so first enable it at the organization level and create an SSO user:

export SSO_USER_EMAIL=<your-email>
./observability/amp-amg/setup-sso.sh

Then deploy the stack:

./observability/amp-amg/deploy.sh

Option B: Open-source tool

This option deploys a Prometheus stack, including a built-in Grafana service, onto the Celeborn EKS cluster in the monitoring namespace.

./observability/prometheus-grafana/deploy.sh

Step 7: Deploy ADOT Collector

The ADOT collector scrapes Prometheus metrics from Celeborn pods and remote-writes them to all active monitoring backends: Amazon Managed Service for Prometheus, open-source Prometheus, or both.

./observability/adot/deploy.sh

Step 8: Submit a Spark job

The sample job is a PySpark word-count application that creates shuffle through groupBy and orderBy operations. Both EMR deployment options below are configured with Celeborn as the shuffle manager.

Option A: Using EMR on EKS

Submit the job using the StartJobRun API:

./emr-on-eks/submit-emr-api.sh

This code snippet shows the core part of the script (see the full version):

aws emr-containers start-job-run
  --virtual-cluster-id "${VIRTUAL_CLUSTER_ID}"
  --name "${job_name}"
  --execution-role-arn "${JOB_EXECUTION_ROLE_ARN}"
  --release-label "${EMR_RELEASE_LABEL}"
  --job-driver "{
    "sparkSubmitJobDriver": {
      "entryPoint": "${input_path}",
      "entryPointArguments": ["${data_input}", "${data_output}"],
      "sparkSubmitParameters": "
        --conf spark.shuffle.manager= \
        org.apache.spark.shuffle.celeborn.SparkShuffleManager
        ...
      "
    }
  }"

Alternatively, submit the job using a Spark Operator:

./emr-on-eks/submit-spark-operator.sh

The script automatically generates a SparkApplication manifest then applies it to EMR on EKS. For example, kubectl apply -f your-job-manifest-name.yaml

Option B: Using EMR on EC2

Submit the job as an EMR Step through the Steps API:

./emr-on-ec2/submit-job.sh

The following snippet shows the core API call used by the script:

aws emr add-steps
  --cluster-id "${CLUSTER_ID}"
  --region "${AWS_REGION}"
  --steps "Type=Spark,
  Name=${JOB_NAME},
  ActionOnFailure=CONTINUE,
  Args=[
    --deploy-mode,client,
    --conf,spark.shuffle.manager= \
    org.apache.spark.shuffle.celeborn.SparkShuffleManager,
    ...
  ]"

Step 9: Review the Grafana dashboard for remote shuffle metrics

The Grafana endpoint is dynamically generated at deploy time and is unique to your deployment. To access it, open the .environment-info file at the repository root. This file contains the Grafana URL along with login instructions. Sign in using the credentials listed there, then navigate to the Celeborn Cluster Overview dashboard to observe remote shuffle metrics in real time. A sample Grafana dashboard screenshot is shown below:

Grafana dashboard showing Celeborn remote shuffle metrics, including active shuffles and worker status

Image 3: Grafana dashboard for Celeborn Metrics

Step 10. Cleaning up

To avoid incurring future charges, run the cleanup script from the root directory:

./teardown.sh

This script detects which components are deployed and tears them down in reverse dependency order, automatically skipping components that are not present. Shared infrastructure is always deleted last, since other components depend on it.

WARNING: This will permanently remove all resources created previously, including any data stored in S3 buckets and configurations. The action cannot be undone. Make sure you have backed up any data you wish to retain before proceeding.

Considerations for production implementation

This post demonstrates a working end-to-end architecture for integrating Celeborn with Amazon EMR. Before taking this pattern to production, consider the following areas.

1. Security

The following considerations help you secure a Celeborn deployment across data isolation, encryption, and access control.

1.1. Shuffle data isolation between teams

Celeborn partitions shuffle data by application ID, which is designed to prevent jobs from accidentally reading each other’s data. This is sufficient when all jobs share the same trust boundary. For multi-team deployments where data privacy is required, a dedicated Celeborn cluster per team is the most effective isolation boundary: each team gets its own NLB and security group, and shuffle data never co-mingles at the infrastructure level.

1.2. Data in transit

In our implementation, shuffle data travels over plain TCP between Spark executors and Celeborn workers. Access is restricted to nodes using security groups. The internal NLB isn’t reachable outside the VPC, and security group rules block all other intra-VPC traffic.

For workloads requiring encryption in transit, Celeborn supports TLS on both RPC and data channels through the celeborn.ssl.* configuration. This is a cluster-wide setting that applies to all jobs on the cluster and must be enabled server-side in celeborn-defaults.conf.

1.3. Data at rest

EBS volumes (used for Celeborn leader pod Raft state) are encrypted with the shared AWS KMS key. NVMe instance store volumes on Celeborn worker nodes are ephemeral and not encrypted by default. Shuffle data written to NVMe is not protected by AWS KMS. For compliance requirements mandating encryption at rest, consider using EBS-backed worker storage instead of instance store, where worker pods mount PersistentVolumeClaim (through volumeClaimTemplates) that reference the encrypted gp3 StorageClass. This comes at the cost of lower I/O throughput compared to local NVMe.

1.4. EKS secrets encryption

EKS clusters encrypt Kubernetes secrets at rest using AWS KMS (EncryptionConfig in the cluster CloudFormation templates). This covers Kubernetes API objects but not application-level shuffle data.

1.5. Application-level authorization

Our deployment enforces access control in the network layer (that is, VPC and security group rules). Celeborn also provides an application-level authorization framework, which is disabled by default. Turning it on adds a second level of security control where only applications presenting valid credentials can register with the cluster. This is recommended for production deployments where multiple workloads or teams share the same VPC subnets, ensuring that network proximity alone does not grant shuffle service access.

Configuration for the Celeborn server:

celeborn.auth.enabled true # Enable SASL application authentication
celeborn.internal.port.enabled true # Required when enabling SASL authentication

Configuration to enable authentication in every Spark app:

--conf spark.celeborn.auth.enabled=true

2. Autoscaling

You can scale Celeborn workers or primary pods using kubectl. For example:

kubectl scale statefulsets celeborn-worker -n celeborn --replicas=6
kubectl scale statefulsets celeborn-master -n celeborn --replicas=2

Celeborn workers register with a primary node when they start, so scaling out is safe even while the cluster is running. Scaling in, however, requires more care. Removing a worker pod during an active job may cause shuffle data loss unless celeborn.client.push.replicate.enabled=true is enabled. To reduce the risk of accidental disruption during node scale-in or upgrades, add a Pod Disruption Budget to prevent multiple workers from being evicted at the same time.
Spark executors: Dynamic Resource Allocation (DRA) is enabled in sample Spark jobs (spark.dynamicAllocation.enabled=true). This means the number of executors can scale automatically within the configured minExecutors and maxExecutors range.

3. Resiliency

Primary Node HA: 3-replica Raft quorum with EBS-backed durable state is designed to tolerate one leader node failure without job interruption.
Worker replication: the setting celeborn.client.push.replicate.enabled=true copies each shuffle partition to two workers, designed to tolerate a single worker failure mid-job. Without replication, a worker failure causes a fetch failure and job retry.
Pod Disruption Budgets (PDB): not configured in this demo. Add PDBs for Celeborn’s StatefulSets to prevent simultaneous eviction during node upgrades or scale-in.
Multi-AZ placement: podAntiAffinity (preferred, weight 100) spreads pods across nodes and Availability Zones. For strict Availability Zone isolation, switch to requiredDuringSchedulingIgnoredDuringExecution.

4. Monitoring and alerting

Consider extending the Grafana dashboards with the alerting rules on:

Active shuffle partition count: indicator of job load.
Worker disk utilization: prevent shuffle storage exhaustion.
Push failure rate: early signal of connectivity or capacity issues.
Celeborn primary node failover: leadership changes indicate Raft instability.

Conclusion

In this post, we showed how to deploy Apache Celeborn as a Remote Shuffle Service on Amazon EKS and integrate it with Amazon EMR on EKS and Amazon EMR on EC2. By decoupling shuffle storage from Spark’s compute, this architecture delivers resilience to node failures, eliminates disk contention, and enables independent scaling of storage and compute tiers.

Running Celeborn on a dedicated cluster gives you lifecycle independence from Spark, lets multiple EMR clusters share a single shuffle service, and provides fault tolerance through push-based shuffling, Raft-based high availability, and per-job data replication. It also provides automatic fallback to Spark’s built-in shuffle during maintenance windows. Stop choosing between cost and reliability. With Celeborn on Amazon EMR, you get both.

For more information, see the Amazon EMR on EKS documentation and the Apache Celeborn documentation. To explore the full implementation, visit the aws-samples GitHub repository. If you have questions or feedback, leave us a comment.

About the authors

Authenticate legitimate AI agent traffic with AWS WAF Bot Control

2026-07-14 Harith Gaddamanugu

Post Syndicated from Harith Gaddamanugu original https://aws.amazon.com/blogs/security/authenticate-legitimate-ai-agent-traffic-with-aws-waf-bot-control/

As AI agents and automated tools increasingly access web applications, distinguishing legitimate bot traffic from malicious attempts has become a critical security challenge. Traditional approaches such as IP-based filtering and reverse DNS lookups fail in multi-tenant systems (such as Amazon Bedrock AgentCore) where thousands of distinct workloads share the same IP space. Attackers can easily spoof user agents, and manual allowlists don’t scale with growing demand.

Web Bot Authentication (WBA), available in AWS WAF Bot Control since November 2025, solves this challenge by implementing cryptographic signatures that provide tamper-proof verification of bot identities. WBA uses asymmetric cryptography to verify that a request comes from an authorized automated agent, relying on two active Internet Engineering Task Force (IETF) drafts: a directory draft for sharing public keys, and a protocol draft defining how keys attach crawler identity to HTTP requests.

With WBA, you can confidently identify trusted automated access while maintaining granular control through WAF labels, creating a more secure and manageable ecosystem for both bot operators and website owners. AWS WAF Bot Control respects WBA verification status by default, automatically allowing verified AI agent traffic.

This post provides a deeper technical guide to implementing WBA with AWS WAF. You learn how WBA works, explore the new labels and capabilities it introduces, and walk through a step-by-step implementation—including signing code—to authenticate bot traffic using cryptographic signatures.

How Web Bot Authentication works with AWS WAF

WBA uses asymmetric cryptography to verify bot identities through HTTP message signatures. The process works as follows:

Bot registration – Bot operators publish their public keys in a signature directory. AWS WAF regularly polls these directories and maintains a valid key registry.
Request signing – Each bot operator’s request is signed using their private key following the IETF standard HTTP Message Signatures (RFC 9421).
Verification – AWS WAF verifies signatures against known public keys associated with the bot operator and appends labels related to verification status.

A typical WBA-signed request includes headers like the following:

Signature-Agent: https://signature-agent.test
Signature-Input: sig2=("@authority" "signature-agent")
;created=1735689600
;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"
;alg="ed25519"
;expires=1735693200
;nonce="e8N7S2MFd/qrd6T2R3tdfA..."
;tag="web-bot-auth"
Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQ..

The following sequence diagram shows how AWS WAF verifies bot signatures and applies labels for allow or block decisions.

Figure 1 – AWS WAF Web Bot Authentication verification flow

The workflow shown in figure 1 includes the following steps:

A bot sends a signed request to Amazon CloudFront and is inspected by AWS WAF Bot Control
AWS WAF Bot Control retrieves the bot operator’s public key from the signature directory
AWS WAF Bot Control verifies the ed25519 signature
AWS WAF Bot Control appends a verification label (verified, invalid, expired, or unknown_bot)

AWS WAF Bot Control evaluates rules using the label to allow or block the request.

New capabilities added to AWS WAF

With the addition of WBA, the following capabilities were added to AWS WAF.

Cryptographic bot verification

When a bot sends a request, it includes HTTP message signatures that AWS WAF validates at the edge using the AWS WAF Bot Control rule group (version 4.0 and later). This validation process adds minimal latency to requests while providing cryptographic certainty about the bot’s identity. HTTP Message Signatures is an open IETF standard (RFC 9421) that defines a mechanism for signing and verifying HTTP messages using asymmetric keys—in practice, this means a bot cryptographically signs specific headers and metadata of each request, and the receiver can verify the signature using the bot’s published public key.

New labels within AWS WAF for granular control

AWS WAF automatically validates signatures, and successfully validated traffic is immediately marked as verified. This verification status can be used in WAF rules and bot management policies, giving you the ability to write your own rules based on the new functionality.

The following table describes the new labels.

Label	Meaning	Suggested action
awswaf:managed:aws:bot-control:bot:web_bot_auth:verified	Successful cryptographic verification	Allow
awswaf:managed:aws:bot-control:bot:web_bot_auth:invalid	Failed verification attempt	Block or rate-limit
awswaf:managed:aws:bot-control:bot:web_bot_auth:expired	Expired key used	Block and alert
awswaf:managed:aws:bot-control:bot:web_bot_auth:unknown_bot	Unrecognized key	Monitor or block
awswaf:managed:aws:bot-control:bot:vendor:<vendor_name>	Bot vendor or operator	Use for vendor-specific rules
awswaf:managed:aws:bot-control:bot:name:<rfc_name>	Bot name (RFC token from WBA)	Use for bot-specific rules
awswaf:managed:aws:bot-control:bot:account:<hash>	AWS account identifier (Amazon Bedrock AgentCore agents only)	Use for account-level controls

AWS WAF now automatically allows verified AI agent traffic

AWS WAF Bot Control now respects WBA verification status by default, automatically allowing verified AI agent traffic. This includes two specific behavior changes:

Category:AI rule update – Previously, the Category:AI rule under common Bot Control blocked unverified bots. Bot Control now checks WBA verification status before applying this rule.
TGT_TokenAbsent rule update – The TGT_TokenAbsent rule, which detects requests without a WAF token, no longer matches requests that carry the web_bot_auth:verified label.

Key benefits for AWS WAF customers

WBA with AWS WAF delivers several advantages for organizations managing automated traffic at scale.

Enhanced bot visibility – Clear identification of distinct bots operating from multi-tenant platforms like Amazon Bedrock AgentCore, providing transparency into automated traffic sources. The AWS WAF console includes a new AI activity dashboard that provides a centralized view of AI bot and agent traffic across your protected resources.
Enhanced security – Cryptographic verification of bot identities using industry-standard signing mechanisms.
Reduced false positives – Accurate distinction between legitimate and malicious automated traffic, particularly in shared IP environments.
Industry alignment – Alignment with industry standards and major content delivery network (CDN) providers for consistent bot authentication across platforms.

Customer use cases for WBA with AWS WAF

Across industries, organizations use WBA to grant automated agents secure, controlled access to their web applications. The following scenarios highlight where this capability delivers real-world value:

Verified customer support agents – Authenticate AI-powered chat and support bots so websites can recognize them as approved, registered agents. This enables seamless customer service automation while maintaining security controls and audit trails.
Automated crawling and indexing – Allow search engine crawlers and content indexers to fetch pages with clear identity and scoped permissions. This reduces false-positive blocks, improves crawl efficiency, and helps legitimate bots access your content without triggering security controls.
Partner integrations – Third-party agents can access customer portals and APIs with explicit consent and granular, scoped access controls. This facilitates secure business-to-business (B2B) integrations while maintaining visibility into partner bot activity.
Enterprise automations and agents – Internal automation tools—including monitoring systems, QA bots, continuous integration and delivery (CI/CD) pipelines, and robotic process automation (RPA) solutions—get authenticated access to web applications with least-privilege access principles and full auditability.

Availability

WBA was introduced in Bot Control rule group Version_4.0 (November 2025) for Amazon CloudFront distributions, with continued support in later versions. With Version_6.0, WBA is available for resource types supported by AWS WAF across standard commercial AWS Regions.

Getting started: Developers or agents quick start

Whether you’re implementing WBA yourself or working with an AI coding assistant, the following steps walk you through deploying WBA, signing requests, and writing custom rules.

Step 1: Deploy the WBA-enabled Bot Control

Add the AWS WAF Bot Control rule group to your CloudFront-associated web ACL using static Version_4.0 or Version_5.0—both include WBA support for cryptographic bot verification. Version_5.0 (released February 2026) covers more than 650 unique bots and agents spanning categories including AI search engine crawlers, AI data collectors, AI assistants, and large language model (LLM) training crawlers.

Important: You must explicitly select one of these static versions.

The following example CloudFormation YAML snippet shows a bot control rule set configuration:

# Bot Control rule group with WBA support
ManagedRuleGroupStatement:
  VendorName: AWS
  Name: AWSManagedRulesBotControlRuleSet
  # Use Version_4.0 or higher for WBA support
  Version: Version_5.0
  ManagedRuleGroupConfigs:
    - AWSManagedRulesBotControlRuleSet:
        # COMMON level provides WBA verification
        # TARGETED level adds additional bot-specific protections
        InspectionLevel: COMMON

Step 2: Sign requests from your bot

If your agent runs on Amazon Bedrock AgentCore Browser, request signing is handled automatically—no additional configuration is required.

For agents running outside of AgentCore, registration APIs are on the roadmap that you can use to sign requests independently by:

Generating an ed25519 key pair
Hosting your public key in a signature directory
Signing outbound HTTP requests using the Signature-Input and Signature headers with the web-bot-auth tag. For language-specific signing implementations, see the HTTP Message Signatures RFC (RFC 9421) and the AWS WAF Bot Control documentation.

Step 3: Write custom rules using WBA labels

Use the verification labels in custom WAF rules for granular traffic control, for example:

Allow – awswaf:managed:aws:bot-control:bot:web_bot_auth:verified
Rate-limit – awswaf:managed:aws:bot-control:bot:web_bot_auth:invalid
Alert on – awswaf:managed:aws:bot-control:bot:web_bot_auth:expired

Step 4: Monitor WBA traffic

Use AWS WAF metrics and logs to monitor authenticated bot traffic:

Review Amazon CloudWatch metrics for Bot Control rule group matches and set up alarms for anomalous or unexpected spikes in invalid or expired verification attempts.
Analyze AWS WAF logs to identify patterns in bot authentication attempts and filter on web_bot_auth labels.
Use the AI Activity Dashboard in the AWS WAF console for a centralized view of AI bot traffic. Visualize traffic trends, identify top bots and frequently targeted paths, and filter by verification status to decide which bots to allow, rate-limit, or block.

Conclusion

WBA with AWS WAF provides a cryptographically secure, standards-based approach to authenticating legitimate AI agent traffic. By moving from IP-based allowlisting to signature-based verification, you gain accurate bot identification that works across multi-tenant environments.

Looking ahead, our focus is to simplify bot authentication and make it safer by default. Registration APIs that agent owners can use to cryptographically verify bot identity and intent are on the roadmap, helping website owners quickly distinguish trusted automation from unknown traffic.

If you own an agent, adopt WBA and register your agent to receive verified status. In parallel, AWS continues to actively participate in the IETF web-bot-auth working group, advocating for complementary approaches—using both identifying and anonymous verification protocols—and will incorporate these standards into products as they mature to help your deployments stay aligned with the broader ecosystem.

To get started, see the AWS WAF Bot Control documentation and the HTTP Message Signatures RFC (RFC 9421).

If you have feedback about this post, submit comments in the Comments section below.

Build an AI campaign orchestrator with Amazon Bedrock and AWS End User Messaging

2026-07-13 Ruchikka Chaudhary

Post Syndicated from Ruchikka Chaudhary original https://aws.amazon.com/blogs/messaging-and-targeting/build-an-ai-campaign-orchestrator-with-amazon-bedrock-and-aws-end-user-messaging/

Marketing teams running large-scale campaigns often send the same message across SMS, WhatsApp, and email regardless of how each customer engages or how many messages they’ve already received that week. This pattern wastes budget on channels customers ignores and pushes promotional content toward frustrated or message-fatigued customers. A MarketingSherpa study found that 45% of consumers who unsubscribe from email marketing cite messages being too frequent as the reason. Over-messaging therefore erodes the audience a brand has paid to acquire. This post shows how to build a campaign orchestrator on AWS End User Messaging and Amazon Bedrock. The orchestrator predicts the best channel for each customer, adapts content per channel, and holds back messages to fatigued or unhappy customers.

In this post, we describe the following capabilities for enterprise marketing teams:

Channel prediction that selects SMS, WhatsApp, or email for each customer based on engagement history
Content adaptation that takes a single campaign brief and produces channel-appropriate variants: a 160-character SMS, a longer WhatsApp template message, and an HTML email
Sentiment-aware suppression that holds back promotional messages when a customer’s stored sentiment score is negative
Frequency tracking across channels that lowers send rate when a customer shows disengagement signal
Natural-language campaign launch that turns a typed instruction such as “Send the Andaman package to Mumbai customers who haven’t booked in six months” into a segmented, channel-routed send

Prerequisites

You need the following to deploy this solution-

An AWS account
The AWS Serverless Application Model (AWS SAM) CLI installed locally
A WhatsApp Business account linked to AWS End User Messaging Social
Amazon Bedrock model access granted for an Anthropic Claude model in your AWS Region
(Optional) An Amazon SageMaker AI endpoint for channel prediction. The orchestrator calls the endpoint when it’s configured and falls back to the customer’s stored preferred channel otherwise.

Solution overview

A marketer types a plain-language instruction into the campaign launcher. Amazon API Gateway forwards the instruction to an AWS Lambda function, which starts an AWS Step Functions state machine. The state machine walks the campaign through seven stages. Each stage moves the campaign closer to dispatching the right message on the right channel. The stages read and write customer state in Amazon DynamoDB and call Amazon Bedrock for language tasks. The final stage dispatches messages through AWS End User Messaging or Amazon Simple Email Service (Amazon SES).

When you turn on semantic segmentation, the state machine also queries an Amazon OpenSearch Serverless collection. The collection holds customer embeddings.

To deploy the sample in your account, refer to the GitHub repository.

Figure 1 shows the campaign orchestration system.

Message processing

When a marketer submits an instruction, the launcher Lambda function starts a Step Functions execution. The state machine then runs the stages in order. Each stage reads the output of the previous one, applies its own logic, and passes its result forward. The state machine retries transient failures within a stage, so a Bedrock throttle or a DynamoDB timeout doesn’t restart the whole campaign. A choice state redirects the workflow straight to the recording stage when no customers pass the safety check, so empty campaigns skip the content adaptation step. This decoupled design gives operators three things:

If one stage fails, the workflow retries that stage without rerunning earlier work
You can add new stages — for example, a translation step — without changing the others
The system scales with campaign volume

AI conversation engine

Amazon Bedrock does two distinct things in the orchestrator, and they happen at different stages. The parse stage runs first. It takes the marketer’s plain-language instruction and asks the model to return a small JSON object. The JSON has fields such as location, package, age range, and a short semantic query when the instruction implies a lifestyle or affinity. That JSON is what every downstream stage works against, so the parse output sets the shape of the campaign.The parse stage sends the following prompt to Anthropic Claude on Bedrock through the InvokeModel API:

You parse marketing campaign instructions into structured fields.

Instruction:
{instruction}

Return JSON with these fields:
- "sku" (string or null): product SKU or package name
- "location" (string or null): city or region
- "category" (string or null): one of "Electronics", "Travel", "Apparel", "Home"
- "min_age" (integer or null), "max_age" (integer or null)
- "min_purchases" (integer or null)
- "lookback_days" (integer or null)
- "has_cart_items" (bool or null)
- "semantic_query" (string or null): free-text lifestyle/affinity descriptor

Output ONLY the JSON object, no prose.

For the instruction “Send the Andaman package to budget-conscious families in Mumbai”, the model returns:

{
  "sku": "Andaman package",
  "location": "Mumbai",
  "category": "Travel",
  "min_age": null, "max_age": null,
  "min_purchases": null,
  "lookback_days": null,
  "has_cart_items": null,
  "semantic_query": "budget-conscious families"
}

The adapt stage runs later, after segmentation and safety. It takes a single campaign brief and asks Bedrock to produce one variant per channel: a 160-character SMS, a longer WhatsApp template message, and an HTML email. The model never sees customer-level data at this point; the brief and the channel are the only inputs.The orchestrator stores one prompt per channel. The SMS prompt enforces a hard character limit; the WhatsApp prompt allows a longer message; the email prompt asks for structured HTML:

# SMS
Write a single SMS for the campaign brief below. Hard limit: 160 characters.
No emojis, no links unless the brief explicitly includes one. Plain text only.

# WhatsApp
Write a WhatsApp message for the campaign brief below. Up to 1024 characters.
Friendly tone, optional emoji where natural.

# Email
Write an HTML email body for the campaign brief below. Include a single <h1>,
two short paragraphs, and a call-to-action link placeholder {{CTA_URL}}.
No <html> or <body> wrappers.

Each prompt is formatted with the campaign brief and sent to Bedrock; the response becomes that channel’s variant for every approved customer in the segment.

The safety stage runs after the parse stage and before the adapt stage, and it is rule-based rather than model-based. It reads each customer’s stored sentiment score and rolling send count from DynamoDB, and drops customers below the sentiment threshold (default -0.3) or above the fatigue limit. The fatigue limit is a per-customer count of sends over a rolling window, for example five sends in the previous seven days. You set the fatigue window and the sentiment threshold as Step Functions input parameters. You populate the sentiment score upstream. For example, you can run a daily Amazon Comprehend Custom Classification job that scores recent support transcripts and writes the result back to the customer record.

The fatigue check reads the customer’s recent send timestamps from the rate-limits table and counts the entries inside the rolling window:

NEGATIVE_THRESHOLD     = Decimal("-0.3")     # configurable
MAX_MESSAGES_PER_WINDOW = 5
WINDOW_SECONDS         = 7 * 24 * 60 * 60     # 7 days

def _is_fatigued(customer_id):
    item = _rate_limits.get_item(Key={"limiter_key": f"customer:{customer_id}"}).get("Item")
    if not item:
        return False
    cutoff = int(time.time()) - WINDOW_SECONDS
    recent = [t for t in item.get("recent_sends", []) if int(t) >= cutoff]
    return len(recent) >= MAX_MESSAGES_PER_WINDOW

Each successful send writes its timestamp into the customer’s recent_sends list, so the next campaign sees an up-to-date fatigue count without a separate ETL step.

Orchestration

Each stage in the campaign workflow is a small AWS Lambda function. The state machine invokes them in sequence: parse the instruction, segment customers, predict channels, check safety, adapt content, deliver messages, and record results. The predict stage reads each customer’s per-channel engagement history from DynamoDB and picks the channel with the highest historical engagement rate. When you wire an Amazon SageMaker AI endpoint into the stack, the stage calls that endpoint instead and uses its score as the channel ranking signal.The state machine, not the functions, owns the control flow. New stages (for example, a translation step before adapt content) can be inserted without changing the existing handlers. The Step Functions definition lives in statemachine/campaign_orchestrator.asl.json. Refer to it in the GitHub repository for the exact state graph and retry policy.

Semantic search

Consider a marketer who types “Send the Andaman package to budget-conscious families interested in beach vacations.” A keyword filter against the customer table won’t match a profile tagged “economy package, kid-friendly, coastal”, because the words don’t overlap even though the meaning does. To bridge that gap, the seed script embeds each customer profile with Amazon Titan Text Embeddings v2 and writes the vector into an OpenSearch Serverless Vector search collection. The segment stage then embeds the marketer’s phrasing at query time and runs a k-nearest-neighbor search against the collection.

The orchestrator intersects those matches with the structured DynamoDB filter. The final segment respects both the hard constraints (location, age, recency) and the soft ones (lifestyle, affinity). OpenSearch Serverless scales the collection’s compute units to zero when idle, so this capability adds near-zero cost when no campaigns run.

Deployment

To deploy the sample in your AWS account, clone the GitHub repository and run the SAM-based deploy script:

git clone https://github.com/aws-samples/sample-ai-campaign-orchestrator.git
cd sample-ai-campaign-orchestrator
./scripts/deploy.sh --guided

The script prompts you for an AWS Region, a stack name, and the orchestrator parameters (your WhatsApp phone number ID and optional SES sender). It then runs sam build followed by sam deploy, and prints the API endpoint and stack outputs when the deployment finishes.

Test the solution

After the stack finishes deploying, seed the customer profiles table with one sample customer and run a campaign against it:

python scripts/seed_demo_data.py --whatsapp-recipient +1234567890

Then submit a campaign instruction to the API endpoint that the deploy script printed:

curl -X POST $ENDPOINT -H 'content-type: application/json' \
  -d '{"instruction": "Send the Andaman package to Mumbai customers"}'

From here you can:

Watch the campaign execution in the AWS Step Functions console.
Query the delivery tracking table in Amazon DynamoDB to see which customers the safety stage approved or suppressed, and which channel the orchestrator picked for each.
Check the recipient’s phone for the WhatsApp template message that the deliver stage sent.

Sample conversation

The recording in this section shows a marketer using the campaign launcher to send an Andaman travel promotion to a Mumbai segment. It opens with the marketer typing the natural-language instruction and the parse stage extracting structured filters. The segment stage then matches customers in DynamoDB. The safety stage suppresses a customer with a low sentiment score. The predict stage assigns a channel per remaining customer. The recording ends with the adapt stage producing one message variant per channel and the deliver stage dispatching them through AWS End User Messaging.

Clean up

To avoid incurring future charges, delete the resources you created. The sample includes a cleanup script in the GitHub repository. Run ./scripts/cleanup.sh to empty the deployment bucket and delete the stack. The stack deletion removes the AWS Step Functions state machine, AWS Lambda functions, Amazon DynamoDB tables, and (when configured) the Amazon OpenSearch Serverless collection.

Conclusion

You can combine AWS End User Messaging, Amazon Bedrock, and AWS Step Functions to build a campaign orchestrator. The orchestrator routes each message to the channel a customer is most likely to open. It also holds back sends to fatigued or unhappy customers.

The same pattern fits other business-initiated messaging workflows where per-recipient channel and content decisions matter. Examples include transactional banking notifications, appointment reminders, and logistics status updates. To deploy the sample in your account, refer to the GitHub repository. To learn more about AWS End User Messaging, refer to the service documentation.

If you’re applying this pattern, start with the safety check and frequency tracking. Those two stages reduce the risk of damaging customer relationships and produce the engagement data that channel prediction depends on. Once that data is in place, add the prediction and content adaptation stages. Use this implementation as a reference for production messaging on AWS.

About the authors

New compliance guidance available: HITRUST i1 on AWS

2026-07-13 Abdul Javid

Post Syndicated from Abdul Javid original https://aws.amazon.com/blogs/security/new-compliance-guidance-available-hitrust-i1-on-aws/

We are pleased to announce the publication of a new AWS compliance implementation guidance: HITRUST i1 Compliance on AWS: Customer Implementation Guidance with an Illustrative Healthcare Platform.

Healthcare organizations seeking HITRUST i1 certification increasingly rely on Amazon Web Services (AWS) as their cloud foundation. The HITRUST i1 assessment covers 182 curated controls at the Implemented level and is the most widely required HITRUST certification tier in healthcare vendor contracts and Business Associate Agreements required by health plans, hospital systems, and business associates as a condition of working with them.

This guide is designed to close the gap between understanding what HITRUST i1 requires and knowing how to implement it on AWS. It walks cloud architects, security engineers, compliance leads, and assessment preparation teams through the full lifecycle of an i1 engagement from defining the assessment boundary to implementing controls across each technical domain.

What the guide covers

The guide addresses 11 HITRUST i1 technical control domains, with supporting AWS implementation components relative to these domains. The domains include access control, endpoint protection, configuration management, vulnerability management, network protection, transmission protection, incident management, data protection and privacy, audit logging and monitoring, password management, and business continuity and disaster recovery.

The guidance is grounded in a fictional but realistic connected healthcare platform deployed on AWS Landing Zone Accelerator. The scenario is used to make abstract HITRUST concepts concrete, not to suggest that the same architecture or control choices apply universally. HITRUST i1 scoping is inherently organization-specific. The assessment boundary, applicable controls, and evidence requirements are determined by each organization’s system scope and delivered through the HITRUST MyCSF portal. Readers should treat the guidance as a starting point and work with a HITRUST Authorized External Assessor to validate what applies to their specific environment. This guide doesn’t constitute a compliance certification advisory.

Getting started

You can download the guide here: HITRUST i1 Compliance on AWS: A Customer Implementation Guidance with an Illustrative Healthcare Platform.

AWS HITRUST assurance documentation and the Customer Responsibility Matrix are available through AWS Artifact. For assessment readiness support, visit AWS Security Assurance Services.

If you have feedback about this post, submit comments in the Comments section below.

Introducing OAuth Support for AWS MCP Server

2026-07-10 Vaibhav Chowla

Post Syndicated from Vaibhav Chowla original https://aws.amazon.com/blogs/security/introducing-oauth-support-for-aws-mcp-server/

AWS MCP Server using the same credentials and sign-in methods that you already use for connecting to the AWS Management Console or AWS Command Line Interface (AWS CLI) through a familiar browser-based experience powered by industry-standard OAuth. This new sign-in path supports AWS Identity and Access Management (IAM) federation, AWS IAM Identity Center, and root or IAM users.

In addition, AWS is introducing several new security and governance tools, including: new global condition keys for OAuth, token introspection and revocation, dynamic client registration, new AWS CloudTrail elements, and a new API for headless OAuth connectivity. All of this is compatible with your existing IAM configuration including permissions, roles, and federated access.

In this post, you’ll learn how to connect your agents to the AWS MCP Server, understand how AWS Sign-In authorizes agent access, and manage access using new security and governance capabilities.

How to connect an agent to the AWS MCP Server

This walkthrough uses Claude Code, but the same steps apply to any agent that supports Model Context Protocol (MCP) such as Kiro, Codex, and Gemini. See Setting up the AWS MCP Server for how to connect the AWS MCP Server to an agent.

Prerequisite permissions

To connect an agent to the AWS MCP Server, you’ll need the IAM permissions required for OAuth-based sign-in. The following AWS CLI command adds a managed policy with required permissions to your IAM role (remember to replace <MyRole> with your IAM role):

aws iam attach-role-policy \
  --role-name <MyRole> \
  --policy-arn arn:aws:iam::aws:policy/AWSMCPSignInOAuthAccessPolicy

Step 1: Configure the AWS MCP Server on your agent

Run the following command to add the AWS MCP Server endpoint to your agent’s configuration as shown in Figure 1:

claude mcp add --transport http aws-mcp https://aws-mcp.us-east-1.api.aws/mcp

Figure 1: Adding the AWS MCP Server endpoint to Claude Code

Step 2: Review the authorization request

The first time your agent needs to access the AWS MCP Server, it opens a browser and redirects you to an AWS Sign-In page, shown in Figure 2. Authenticate as you would on AWS console or AWS CLI, review the authorization request, and approve access. You should receive an Authorization successful message.

Figure 2: Review authorization request

Note that if you already have an active AWS Sign-In session (e.g., because you previously signed in to the console earlier in the day), you can reuse that session without needing to sign in again.

Step 3: Start using AWS tools

After connecting your agent to the AWS MCP Server, you can begin invoking tools provided by the server. To verify that Claude Code is connected to the AWS MCP Server, start Claude Code and run the following command:

/mcp

The command displays the configured MCP servers and confirms that the AWS MCP Server is connected and ready to use with your AWS credentials.

Figure 3 shows an example of a successful connection to the AWS MCP Server.

Figure 3: Verifying the AWS MCP Server connection in Claude Code

After the connection is established, you can ask Claude Code to invoke tools provided by the AWS MCP Server. For example, enter the following prompt:

Deploy a sample serverless web application into my development AWS account

Claude Code uses the AWS MCP Server to identify the active AWS account, confirm the target account, and describe the deployment it plans to perform before invoking AWS services on your behalf.

Figure 4 shows Claude Code confirming the active AWS account and outlining the resources that will be deployed.

Figure 4: Using Claude Code to deploy a sample serverless application through the AWS MCP Server

Authorization models and how they work

AWS Sign-In supports two authorization models for connecting agents to the AWS MCP Server:

Interactive authorization for developer’s AI agents using browser based authentication
Non-interactive (headless) authorization for applications and AI agents that already have AWS credentials and don’t have access to a browser

Note that authorizing an agent allows it to access the AWS MCP Server on your behalf. It doesn’t grant the agent additional AWS permissions. Every request is still evaluated using your existing IAM policies, SCPs, RCPs, permission boundaries, and other organizational controls.

Interactive access

In the interactive case, the agent first discovers the AWS Sign-In OAuth server and then registers itself as an OAuth client using Dynamic Client Registration (DCR). It then redirects you to an AWS Sign-In page where you authenticate and authorize access (step 2 in the preceding section). After successful authorization, AWS Sign-In then issues short-lived access tokens and refresh tokens that authorize the agent to access the AWS MCP Server on your behalf. AWS Sign-In automatically manages token issuance and token refresh, enabling authorized agents to continue accessing the AWS MCP Server without requiring you to repeatedly sign in.

The interactive authorization model supports three distinct sign-in methods: native AWS IAM credentials for individual developers, managed access through AWS IAM Identity Center for enterprises, and seamless federated access via third-party providers like Okta and Ping Identity for larger organizations

OAuth server metadata and DCR

Before an agent can request authorization, it must discover the AWS Sign-In OAuth endpoints and register itself as an OAuth client. AWS Sign-In supports OAuth metadata discovery and DCR, allowing supported agents to configure themselves automatically without requiring developers to manually provision OAuth client IDs and client secretsWhen an agent connects to the AWS MCP Server for the first time, it retrieves the AWS MCP Server’s protected resource metadata (RFC 9728) and the AWS Sign-In OAuth metadata (RFC 8414). The agent then uses RFC 7591)) to register with AWS Sign-In, obtain a client ID, and initiate the standard OAuth authorization code flow.

AWS Sign-In supports OAuth discovery and DCR for agents running on local workstations and supported hosted environments. For the current list of supported agents and environments, see Supported redirect URIs for the AWS MCP Server.

Non-interactive access to the AWS MCP Server

Non-interactive (headless) authorization is for agents and applications that run without a browser or human in the loop, and thus don’t require interactive sign-in. This allows agents that already have AWS credentials to obtain OAuth access tokens and connect to the AWS MCP Server. The following is an example of how to obtain an access token.

aws signin create-oauth2-token-with-iam \ 
--grant-type client_credentials \ 
--resource aws-mcp.amazonaws.com \  
--region us-east-1 
{ 
"accessToken": "ASOA****************************************...", 
"tokenType": "Bearer", 
"expiresIn": 3600 
}

In the non-interactive case, AWS Sign-In implements the OAuth client credentials grant using AWS security credentials instead of a static client secret. Applications authenticate to the AWS Sign-In token endpoint using SigV4 creds, and AWS Sign-In returns a short-lived OAuth access token that can be used to access the AWS MCP Server.

Please note you may have to update the SDK and AWS CLI, please refer to CLI guide.

Managing OAuth access

AWS Sign-In extends the existing IAM authorization model with capabilities for governing OAuth access to the AWS MCP Server. Administrators can use familiar IAM policies together with new OAuth-specific controls.

Granting OAuth permissions

OAuth access is governed using IAM policies and requires the following IAM actions:

signin:AuthorizeOAuth2Access – Allows users to sign in interactively using the OAuth authorization code flow
signin:CreateOAuth2Token – Allows applications to obtain OAuth access tokens by exchanging authorization codes, refresh tokens, or using client credentials

When an application requests access, AWS Sign-In creates an OAuth authorization grant between the agent and the AWS MCP Server. This grant is represented as an IAM resource, which the preceding AWS Sign-In actions are authorized against.

arn:aws:signin:us-east-1:012345678910:service-principal/aws-mcp.amazonaws.com

OAuth authorization grants are represented as IAM resource enabling administrators to use standard IAM policy constructs, including global condition keys, together with OAuth-specific condition keys to control how authorization grants are created and used.

Governing OAuth access

AWS Sign-In introduces OAuth-specific condition keys that allow administrators to govern how agents obtain OAuth authorization. The following examples demonstrate common governance patterns.

To restrict OAuth authorization to localhost:

In addition to accessing the AWS MCP Server with agents on your local workstation, AWS supports signing into the AWS MCP Server on select hosted providers through dynamic client registration. Click here to view the list of supported remote providers. Many organizations want to allow developers to authorize agents running on their local workstations while preventing OAuth tokens from being delivered to untrusted redirect URIs or using unsupported authorization flows. The following policy allows only the OAuth authorization code and refresh token flows for the AWS MCP server and restricts token delivery tolocalhost.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "signin:AuthorizeOAuth2Access",
        "signin:CreateOAuth2Token"
      ],
      "Resource": "arn:aws:signin:*:*:service-principal/aws-mcp.amazonaws.com",
      "Condition": {
        "StringLike": {
          "signin:OAuthRedirectUri": "http://localhost:*"
        },
        "StringEquals": {
          "signin:OAuthGrantType": [
            "authorization_code",
            "refresh_token"
          ]
        }
      }
    }
  ]
}

To deny access for a specific OAuth session

Use the aws:SignInSessionArn global condition key to deny authorization associated with a specific sign-in session. This allows administrators to contain a suspicious or compromised authorization session without affecting other active sessions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "*"
      ],
      "Resource": "*",
      "Condition": {
        "ArnEquals": {
          "aws:SignInSessionArn": "arn:aws:signin:us-east-1:111122223333:session/abc123-example-session-id"
        }
      }
    }
  ]
}

These examples demonstrate common governance patterns. Additional IAM and SCP examples are available in the AWS Sign-In condition keys reference.

Revoking OAuth tokens

AWS Sign-In provides OAuth token introspection and token revocation APIs that allow administrators to build custom tools for token validation and revocation. Access to these APIs is controlled through the signin:IntrospectOAuth2Token and signin:RevokeOAuth2Token permissions. IAM principals with permissions are allowed to introspect and revoke tokens for the same account.

The introspection API can be used to determine whether a token is active and obtain information about the associated authorization. The revocation API allows administrators and security tools to revoke individual refresh tokens without affecting other active sessions. For example, if an organization needs to invalidate access for a specific OAuth authorization, account admins can revoke the associated refresh token without affecting other active sessions.

Monitoring OAuth activity

OAuth-related activities are recorded in AWS CloudTrail, including authorization requests, token issuance, token revocation, and token introspection events. CloudTrail logs also capture details such as the OAuth client, target the AWS MCP Server, redirect URI, authorization flow, and associated sign-in session. In addition, AWS API calls made using OAuth access tokens include the associated aws:SignInSessionArn context, allowing organizations to correlate API activity with the originating OAuth sign-in session.

This allows security teams to monitor OAuth usage, investigate authorization activity, detect anomalous behavior, and integrate OAuth events into existing auditing, compliance, and incident response workflows alongside other AWS activity.

Here’s a CloudTrail sample for an AuthorizeOAuth2Access event:

{
    "eventVersion": "1.11",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROATJHQDX737YZP****:testuser",
        "arn": "arn:aws:sts::111111111111:assumed-role/Admin/testuser",
        "accountId": "111111111111",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROA2IRT4N5U4RDHM2LG4",
                "arn": "arn:aws:iam::111111111111:role/Admin",
                "accountId": "111111111111",
                "userName": "Admin"
            },
            "attributes": {
                "creationDate": "2026-06-09T05:06:39Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2026-06-09T05:09:00Z",
    "eventSource": "signin.amazonaws.com",
    "eventName": "AuthorizeOAuth2Access",
    "awsRegion": "us-west-2",
    "sourceIPAddress": "192.0.0.2",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36",
    "requestParameters": {
        "resource": "https://aws-mcp.us-west-2.api.aws/mcp",
        "redirect_uri": "http://127.0.0.1:60432/oauth/callback",
        "code_challenge_method": "S256",
        "client_id": "arn:aws:signin:us-west-2::external-client/dcr/609544da-aasa-49a4-ab11-c2r457fa999"
    },
    "responseElements": null,
    "additionalEventData": {
        "success": "true"
    },
    "requestID": "4fb4ff7b-6yu7-9090-78i9-9c0088a65134",
    "eventID": "bb05b222-31ec-4237-b8e7-8eb26d4fd48b",
    "readOnly": true,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "111111111111",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "us-west-2.oauth.signin.aws"
    }
}

Here’s a CloudTrail sample for a CreateOAuth2Token event:

{
    "eventVersion": "1.11",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROATJHQDX737YZP7****:testuser",
        "arn": "arn:aws:sts::111111111111:assumed-role/Admin/testuser",
        "accountId": "111111111111",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROA2IRT4N5U4RDHM****",
                "arn": "arn:aws:iam::111111111111:role/Admin",
                "accountId": "111111111111",
                "userName": "Admin"
            },
            "attributes": {
                "creationDate": "2026-06-09T05:06:39Z",
                "mfaAuthenticated": "false"
            },
            "signInSessionArn":""
            
        }
    },
    "eventTime": "2026-06-09T05:10:04Z",
    "eventSource": "signin.amazonaws.com",
    "eventName": "CreateOAuth2Token",
    "awsRegion": "us-west-2",
    "sourceIPAddress": "192.0.0.2",
    "userAgent": "curl/8.7.1",
    "requestParameters": {
        "resource": "https://aws-mcp.us-west-2.api.aws/mcp",
        "client_id": "arn:aws:signin:us-west-2::external-client/dcr/609544da-b3dd-49a4-ab11-c2e98d7fa999"
    },
    "responseElements": null,
    "additionalEventData": {
        "signInSessionArn": "arn:aws:signin:us-west-2:111111111111:session/daff060f-7871-5tg6-67yu-a07bbdabe61a",
        "grant_type": "refresh_token",
        "success": "true"
    },
    "requestID": "44d6d7ce-e4r5-4cbf-0909-bfb8a8295a76",
    "eventID": "f79cc63f-b383-4e3c-a1e5-97c7db1ab833",
    "readOnly": true,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "111111111111",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "us-west-2.oauth.signin.aws"
    }
}

Additional audit events and logging details for calls made using OAuth access tokens to the AWS MCP Server can be found in Logging AWS MCP Server API calls using AWS CloudTrail.

Conclusion

AWS Sign-In support for OAuth enables you to securely connect to the AWS MCP Server using industry-standard authorization. This release simplifies application and agent integration with AWS while supporting your existing IAM setup, governance, and auditing capabilities.

To learn more, see Sign-In with OAuth 2.0 in the AWS Sign-In User Guide and Setting up the AWS MCP Server in the Agent Toolkit for AWS User Guide.

If you have feedback about this post, submit comments in the Comments section below.

The CISO’s guide to post-quantum mandates and migrations

2026-07-08 Rushir Patel

Post Syndicated from Rushir Patel original https://aws.amazon.com/blogs/security/the-cisos-guide-to-post-quantum-mandates-and-migrations/

Over a dozen major economies have now published post-quantum cryptography (PQC) adoption guidance. As a CISO, you’re probably well into your migration plan and know the most difficult part has little to do with changing algorithms. The real leadership challenge is driving coordinated change across a large, complex organization where asymmetric cryptography is embedded in every protocol, every vendor dependency, and every legacy system that quietly handles key exchange or digital signatures. This guide provides the regulatory context and the strategic playbook for CISOs, CTOs, or any senior leader who needs to deliver a program that meets compliance deadlines while modernizing your organization’s security governance.

Overview for busy executives

There are five key takeaways to the information presented in this post:

Start at the top. Secure board-level sponsorship by framing cryptographic modernization as enterprise risk reduction with a defined timeline and measurable milestones. Stand up a centralized program office that owns the mandate, sets prioritization criteria, and coordinates delivery across business units.
Classify dependencies, don’t inventory everything. At the workload level, you need to understand three things: what your providers will upgrade on your behalf, what they won’t upgrade in time and needs replacing, and what you own and must address directly. The fastest path to reduce your migration scope is to shift cryptographic responsibility to the first category (what providers will upgrade for you) wherever possible.
Invest in cryptographic telemetry. Build visibility and monitoring in parallel with your migration work. Although this capability is critical, it shouldn’t come at the cost of momentum. Track algorithm usage, PQC coverage percentage, and migration velocity at the workload level. Telemetry sustains board sponsorship over a multiyear program and gives your centralized team the feedback loop to set priorities.
Build for agility, not one-time compliance. Your goal should extend beyond deploying PQC one time. Build the organizational muscle to rotate protocols, algorithms, and key lengths as standards evolve, because cryptographic migration will be a recurring operational requirement.
Treat this as security and governance modernization. Strong patching discipline, reliable continuous integration and delivery (CI/CD), and automated lifecycle management are capabilities that will outlast your PQC migration. They’re the same capabilities you need to respond to AI-accelerated threats, where vulnerability discovery timelines are compressing from weeks to hours. An organization that can rotate algorithms on demand can also patch against novel AI-driven exploits.

Read on for the full playbook.

Global regulatory landscape

In August 2024, NIST published the first three post-quantum standards covering key encapsulation (ML-KEM), lattice-based digital signatures (ML-DSA), and hash-based signature alternatives (SLH-DSA). These standards now serve as the baseline that most jurisdictions reference when setting migration deadlines. The United States, European Union, United Kingdom, Germany, France, Australia, Canada, Japan, South Korea, India, Singapore, and the UAE have all published formal guidance. Industry groups like FS-ISAC in financial services and GSMA in telecom have their own additional timelines.

These timelines vary by jurisdiction, but all follow the same direction. Most regions require PQC readiness for new procurement by 2027, with full migration deadlines falling between 2030 and 2035 depending on industry and geography. For any organization operating across borders, navigating the specific requirements in each jurisdiction where you do business is critical to both compliance and competitive positioning. Amazon Web Services (AWS) maintains a detailed breakdown of regional mandates and timelines in the FAQ section of the Migration to quantum-resistant cryptography page.

Scoping your migration

Historically, cryptographic migrations have taken far longer than you might expect. The deprecation of SHA-1 took nearly twenty years from the first published vulnerability until major browsers finally rejected it. MD5, 3DES, and RC4 all followed the same pattern of slow organizational response despite clear technical consensus that migration was overdue. Those transitions also happened without the modern cloud infrastructure, automated orchestration, and real-time telemetry that exists today. Organizations that use these capabilities can migrate faster while simultaneously building a future-ready security foundation.

The migration scoping challenge splits cleanly into two families. The first is software systems that negotiate algorithms as part of short-lived authentication or encryption protocols, such as TLS, IPsec, or SSH. For these workloads, cloud-centered lifecycle management, automated patching, and centralized library upgrades make this more straightforward than previous cryptographic migrations. Managed services can handle upgrades transparently and telemetry tooling gives real-time visibility into algorithm usage across endpoints. CI/CD pipelines enable incremental rollout with clean rollback paths. Organizations with modern cloud infrastructure have never been better positioned to execute this side of cryptographic transition at speed.

The second family of things to migrate are long-lived embedded systems, which are devices with burned-in firmware that contain keys and algorithm code that can’t be updated in place. The fastest way to reduce this surface area is to offload their cryptographic workloads to managed services, where your provider absorbs the hardware refresh cycle and every migrated workload is one fewer device you need to plan around. For what remains on dedicated hardware, build quantum readiness into your annual capex review. Because quantum advances don’t arrive on a fixed schedule, evaluate embedded cryptographic assets yearly against developments in quantum hardware. Some devices will stay operationally sound for years, whereas others will need accelerated replacement as threat timelines compress. Annual evaluation means early deprecation becomes a planned business decision rather than an unbudgeted emergency.

The strategic playbook

The following playbook outlines a strategic approach to PQC migration that you can adapt to your organizational context. Each step is designed to build enterprise-wide alignment, replace ambiguity with actionable frameworks, and deliver measurable progress to keep your program funded and on track.

Secure board-level commitment

CISOs need to bring PQC to the board as a business risk conversation anchored to regulatory compliance and competitive exposure rather than a technical briefing on lattice-based algorithms. During this process, it’s important to battle misconceptions. One common misconception at the board level is that PQC migration requires re-encrypting all stored data. It does not. Data encrypted at rest using standard 256-bit symmetric encryption is not vulnerable to a quantum computer. This distinction significantly narrows the actual scope of change and should be communicated early to prevent over-scoping.

Present the regulatory timeline with specificity. For example, explain how CNSA 2.0 mandates PQC for new products by January 2027 and that these timelines will function as procurement gates in regulated industries like financial services, healthcare, government, and defense. You can also quantify the organizational exposure by mapping revenue and workloads that sit in regulated verticals. This could be using existing contracts and pending opportunities with public sector customers as the quantifiable data for business at risk.

Here’s an example of what this could look like in practice. First, identify existing contracts in regulated verticals where PQC compliance language is appearing or will appear at renewal. Calculate the revenue attached and flag renewal dates within 18 months as compliance cliffs. Second, look at your open pipeline. Do you have RFPs, vendor questionnaires, or procurement requirements already referencing post-quantum readiness? That pipeline value is at risk of disqualification if you can’t demonstrate compliance and a competitor can. Third, size the total addressable opportunity in verticals where mandates are taking effect and frame what share becomes inaccessible without readiness. With customers writing PQ readiness requirements into vendor contracts, organizations that can’t demonstrate compliance risk being disqualified from future business.

Finally, request dedicated headcount and vendor budget with board-level sponsorship. This can’t be a side project absorbed into existing security operations. Prioritize executive reviews with quantifiable outcomes tracked quarterly at the leadership level.

Assign single-threaded migration leaders

Stand up a cryptography center of excellence with a cross-functional mandate that spans security, engineering, compliance, and procurement. Appoint a migration lead with direct executive reporting who owns the program end-to-end. Staff the team with representation from networking, identity, application development, vendor management, and compliance because PQC touches all these domains simultaneously.

Give the team authority to set organizational standards for cryptographic policy, library usage, and migration timelines. Align this body with vendor and supplier engagement so there’s one accountable team driving the cloud provider and third-party vendor relationships on PQC readiness.

Fund this team to drive centralized remediation patterns that individual business units adopt rather than reinvent. They own the reference implementations, the approved library versions, the testing frameworks, and the rollout playbooks. When one team solves a migration pattern for a given workload type, the centralized team packages that solution and distributes it across every similar workload in the organization.

Classify dependencies and reduce migration surface area

Beware of guidance that recommends a comprehensive bottom-up cryptographic inventory, except in jurisdictions where it’s explicitly required. That exercise can consume months and delay actual migration. Instead, classify your dependencies into three categories:

Workloads where someone else will upgrade for you. Managed cloud services, software as a service (SaaS) providers, and infrastructure vendors with active PQC roadmaps fall here. Your job is to validate their timelines and hold them accountable.
Workloads where someone else owns the stack but won’t upgrade in time. These are vendor dependencies that you need to replace, potentially before the end of their planned useful life. Flag them now so replacement decisions enter your procurement and capex cycles early.
The third is workloads you own and must upgrade yourself. For these, the decision is whether to upgrade in place or modernize into the cloud where the cryptographic layer becomes managed for you.

The first two categories fall into a vendor risk assessment program. The third category is the workstream that must be managed within your own organization and driven to completion on a workback schedule. Track which dependencies have been validated, which replacements are in flight, and which of your self-managed stacks have active upgrade plans. The three-category model gives your centralized team a clear decision framework instead of going into an unbounded discovery exercise.

Build observability and continuously monitor progress

Visibility into your cryptographic posture is a necessity for planning, execution, and demonstrating compliance to auditors. However, observability shouldn’t be a prerequisite to migrating workloads and should be viewed as a parallel workstream so it doesn’t come at the cost of momentum. After your visibility tooling is in place, it will retroactively show all previous work completed and give a real-time view of progress at the organization level.

Many organizations start with TLS because it’s typically the broadest deployment of cryptography and the primary mechanism protecting sensitive data in transit across web applications, APIs, and microservices. Sponsor TLS metric dashboards that show algorithm usage across all endpoints, differentiating between post-quantum and classical TLS traffic using metadata fields in service logs. The PQC Readiness Scanner serves as an example of how to build and deploy this type of visibility tooling. Over time, extend the same observability to other transport protocols like IPSec, SFTP, and SSH.

Establish a continuous evaluation program with company-wide KPIs, which can feed into executive reviews. Beyond discovery, telemetry provides the executive-level progress metrics that sustain board sponsorship over a multiyear program. Some examples include:

Percentage of TLS connections using TLS 1.3 and ML-KEM key exchange
PQC coverage percentage across your defined categories
Ratio of validated vendor timelines to unconfirmed ones
Time-to-remediation when a new dependency is flagged as noncompliant.

Track PQC coverage percentage at the workload and organization level. These metrics turn PQC migration from a one-time project into an ongoing governance function, the same way you already govern patching cadence, vulnerability SLAs, and compliance posture. The goal is to develop a standing capability that absorbs future cryptographic transitions as routine operational work rather than requiring a new program each time.

Align with vendors, regulators, and industry groups

PQC migration crosses organizational boundaries and requires coordinated movement across your supply chain. Engage your cloud providers on their PQC roadmaps and understand which services already support PQ-TLS, which are on the roadmap, and when support is expected. Engage third-party software vendors and SaaS providers with explicit questions about PQC support timelines and write PQC readiness into procurement requirements and vendor contracts going forward.

Engage regulators and standards bodies in your jurisdictions to understand the specific timelines, compliance mechanisms, and audit expectations that apply to your industry. Participate in industry forums because financial services, telecom, healthcare, and critical infrastructure each have sector-specific PQC working groups where peer organizations are sharing approaches and lessons learned. This collaborative approach can also help you get the investment you need for a migration when you have unwilling stakeholders across the business.

Prioritize and roadmap the workloads you own

Adopt a phased approach rather than attempting to migrate everything all at once. Prioritize workloads based on risk and use case. The AWS post-quantum cryptography migration plan blog post provides an example of this prioritization. As you execute on your roadmap, build reliable release and rollback mechanisms at every stage. PQC algorithms have different performance and size characteristics that might surface unexpected behavior under production load. Identify legacy dependencies before they become migration blockers. Systems running custom TLS libraries or hardcoded cipher suites need to be flagged early in the process.

The fastest path to reducing your PQC surface area is eliminating custom cryptographic stacks entirely. Every workload you migrate to a managed service is one fewer workload that your team must upgrade manually. AWS has already delivered post-quantum key exchange across several service endpoints with imperceptible performance impact, and post-quantum signing through AWS Key Management Service (AWS KMS) and AWS Private Certificate Authority. For bespoke code on cloud compute or on premises, open source cryptographic libraries like AWS-LC provide production-ready, FIPS 140-3 validated PQC implementations that your teams can adopt immediately.

Transition to a crypto agile enterprise

Crypto agility is the operational capability to rotate algorithms, update protocols, and absorb cryptographic change as business as usual rather than a dedicated program. Cryptographic standards will continue to evolve. Algorithms will be deprecated and replaced. The organizations that build the ability to do this now won’t need a new program next time.

Crypto agility demands excellence at four disciplines:

Patching and upgrade discipline: If you can’t maintain consistent patching cadences across your fleet today, PQC migration will surface that gap at enterprise scale. Mature vulnerability management programs adopt PQC as a natural extension of existing operations.
Incremental release with clean rollback: PQ algorithms carry larger signatures, larger keys, and different performance profiles. You need to be able to deploy changes incrementally, validate behavior in production, and rollback cleanly when something doesn’t perform as expected.
Consistent CI/CD pipelines: Every application touching asymmetric cryptography will need to be evaluated and potentially rebuilt and redeployed with updated algorithms or libraries. Fragile or manual deployment processes will impede the entire migration.
Automated security lifecycle management: Certificate lifecycle, key rotation, secrets vaulting, signature operations, and compliance validation must all operate at machine speed. Manual processes that function today will fail as security requirements evolve.

These aren’t necessarily PQC-specific investments. They’re the foundational capabilities of a well-run security organization. With AI accelerating the speed at which vulnerabilities are discovered and exploited, organizations that have built crypto agility into their operational posture are better positioned to respond to AI-accelerated threats. Savvy security leaders can use PQC as a forcing function to build the operational resilience your organization needs as the threat landscape evolves.

Conclusion

PQC migration will define how the next generation of enterprise security programs are built and measured. The technical tooling exists to execute this transition faster than any previous cryptographic migration. The organizations that move now will shape procurement requirements and set the competitive baseline for their industries. Those that defer will inherit compressed timelines, increased costs, and diminished optionality.

AWS is here to help as you navigate the PQC migration process. You can find our latest guidance and publications in Migration to quantum-resistant cryptography.

AWS Security Assurance Services and AWS Professional Services provide expert guidance, and validated implementation approaches to help you upgrade your own applications and workloads. To get started, you can request a complimentary Post-Quantum Readiness Accelerator introductory call.

If you have feedback about this post, submit comments in the Comments section below.

Introducing Apache Spark Connect support in AWS Glue interactive sessions

2026-07-07 Zach Mitchell

Post Syndicated from Zach Mitchell original https://aws.amazon.com/blogs/big-data/introducing-apache-spark-connect-support-in-aws-glue-interactive-sessions/

When we built AWS Glue interactive sessions, our goal was to make AWS Glue as interactive as running local Python from a notebook. We mostly succeeded. With a straightforward Python package and a Jupyter notebook, you could execute remotely against the AWS Glue ephemeral Spark backend. The Livy-based approach was ahead of its time, but it had limitations from its REST-based protocol. Running local PySpark unlocked powerful integrated development environment (IDE) features such as debugging and linting, so your environment could understand the code and help you develop Spark applications more quickly. Customers would often split their development work. They used local Spark (or Docker containers) to develop in an IDE on a small amount of data, then switched to AWS Glue interactive sessions to validate scaling and tuning against the full dataset.

With modern PySpark releases came a new protocol: Apache Spark Connect. Spark Connect bridges the gap between these two worlds: you develop in local Python, but execute on AWS Glue against actual data. Today, AWS Glue interactive sessions support Spark Connect natively. You can connect from any environment that supports the PySpark remote() API, including VS Code, PyCharm, Amazon SageMaker Unified Studio notebooks, and standalone Python applications. You don’t need to install specialized kernels or manage cluster infrastructure.

What Spark Connect changes

Spark Connect, introduced in Spark 3.4, decouples the Spark client from the server through a lightweight gRPC protocol. Instead of running your driver program on the cluster, your IDE communicates with a remote Spark server through a thin client layer. This architecture unlocks the key workflow improvement: you develop locally and execute remotely.

Spark Connect architecture diagram showing a thin client communicating with a remote Apache Spark server

Spark Connect architecture — thin client with the full power of Apache Spark

With Spark Connect support in AWS Glue interactive sessions, you get:

IDE freedom – Use VS Code, PyCharm, JupyterLab, or any Python environment. No kernel installation required.
Programmatic access – Build Spark into your Python applications and automation scripts with a standard SparkSession.builder.remote() call.
Serverless execution – AWS Glue provisions and manages the Spark cluster. You pay only for the data processing units (DPUs) consumed while your session is active.
Spark Connect monitoring – The Spark Live UI now includes a dedicated Connect tab showing active Spark Connect sessions and operations alongside the existing Jobs, Stages, and Executors views.

Getting started with SageMaker Unified Studio

Amazon SageMaker Unified Studio provides the most direct path to Spark Connect on AWS Glue. The notebook environment handles session creation, endpoint retrieval, and token refresh automatically, so no connection boilerplate is required.

Prerequisite: You need an Amazon SageMaker Unified Studio project to use this workflow. If you don’t have one, create a project in your SageMaker Unified Studio domain first.

To connect to an AWS Glue Spark Connect session:

A notebook open in SageMaker Unified Studio

A notebook open in SageMaker Unified Studio

Choose the compute icon in the left toolbar to open the Compute environment panel. Expand the Spark section.

Compute environment panel in SageMaker Unified Studio with the Spark section expanded

The Compute environment panel with the Spark dropdown list

Select a Glue Spark connection. Depending on your SageMaker domain configuration, you will see either default.spark or named connections such as project.spark.compatibility. Select the appropriate Glue (Spark) connection and choose Apply.

Notebook cell showing spark.version returns 3.5.6-amzn-1 after connecting to Glue Spark Connect

Connected to Glue Spark Connect — running spark.version returns ‘3.5.6-amzn-1’

After you make your selection, you’re connected. The spark session object is available natively. No imports or configuration are needed. Start running PySpark immediately:

spark.sql("SHOW DATABASES").show()

The session manages itself in the background, including automatic token refresh.

Using the sagemaker_studio SDK

The sagemaker-studio Python package extends the Spark Connect experience beyond SageMaker Unified Studio notebooks into local IDEs, continuous integration and continuous delivery (CI/CD) pipelines, and any Python environment. The sparkutils module handles session initialization and connection configuration in a single call. You get the same streamlined experience as in the notebook, anywhere you run Python:

from sagemaker_studio import sparkutils

# Initialize a Glue Spark Connect session using your project connection
spark = sparkutils.init(connection_name="default.spark")

# Run queries immediately
spark.sql("SHOW DATABASES").show()

You can also use sparkutils.get_spark_options() to retrieve pre-configured Java Database Connectivity (JDBC) options for reading and writing to data sources through your project connections. Supported sources include Amazon Redshift, Amazon Aurora, and Amazon DocumentDB (with MongoDB compatibility):

# Get connection options for a Redshift connection in your project
options = sparkutils.get_spark_options("my_redshift_connection")

# Read from Redshift via Spark Connect
df = spark.read.format("jdbc").options(**options).option("dbtable", "analytics.orders").load()
df.show()

Within SageMaker Unified Studio, the sagemaker-studio SDK is native to the environment. The spark session and sparkutils are available without installation. For local IDE use, install it with pip install sagemaker-studio and configure credentials through an AWS named profile or boto3 session.

How it works

Spark Connect sessions in AWS Glue use a three-step workflow:

Create a session – Call the CreateSession API with SessionType set to SPARK_CONNECT. The session provisions in approximately 30 seconds.
Retrieve the endpoint – Call GetSessionEndpoint to receive a sc:// gRPC endpoint URL and a time-limited authentication token.
Connect with PySpark – Pass the endpoint and token to SparkSession.builder.remote() and start running Spark operations.

Spark Connect protocol flow from the DataFrame API to a logical plan, sent over gRPC and protobuf, with results streamed back over gRPC and Arrow

Spark Connect protocol flow — DataFrame API translated to logical plan, sent via gRPC/protobuf, results streamed back via gRPC/Arrow

Connecting with the low-level API

Some environments don’t have the sagemaker-studio SDK, such as custom containers, AWS Lambda functions, or non-Python toolchains. In these environments, or if you’re not using SageMaker Unified Studio, you can use the AWS SDK (Boto3) to manage sessions directly. The following example demonstrates the full workflow:

import time, boto3, urllib.parse
from pyspark.sql import SparkSession

glue = boto3.client("glue", region_name="us-east-1")

# 1. Create a Spark Connect session
session_id = "my-spark-connect-session"
glue.create_session(
    Id=session_id,
    Role="arn:aws:iam::123456789012:role/GlueServiceRole",
    Command={"Name": "glueetl"},
    GlueVersion="5.1",
    SessionType="SPARK_CONNECT",
    DefaultArguments={"--enable-spark-live-ui": "true"},
)

# 2. Wait for the session to reach READY
while True:
    status = glue.get_session(Id=session_id)["Session"]["Status"]
    if status == "READY":
        break
    time.sleep(5)

# 3. Get the Spark Connect endpoint
sc = glue.get_session_endpoint(SessionId=session_id)["SparkConnect"]
endpoint_url = sc["Url"]
auth_token = sc["AuthToken"]

# 4. Connect with PySpark
encoded_token = urllib.parse.quote(auth_token, safe="")
connection_string = f"{endpoint_url}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
spark = SparkSession.builder.remote(connection_string).getOrCreate()
spark.sql("SELECT 1 + 1 AS result").show()

Monitoring with Spark Live UI

When you enable the Spark Live UI at session creation, you gain access to a real-time dashboard showing:

Jobs and Stages – Track active, completed, and failed jobs with stage-level metrics.
Executors – Monitor memory usage, shuffle data, and executor health.
SQL – Inspect query plans and execution details.
Connect tab – View active Spark Connect sessions and operations (specific to Spark Connect).

Access the dashboard through the GetDashboardUrl API or directly from the AWS Glue console.

import boto3, webbrowser

glue = boto3.client("glue", region_name="us-east-1")
dashboard = glue.get_dashboard_url(
    ResourceId="my-spark-connect-session",
    ResourceType="SESSION",
)
webbrowser.open(dashboard["Url"])

In SageMaker Unified Studio, no API call is needed. Choose Ready in the notebook status bar to open the kernel info popover. From there, open the Spark UI link for the live dashboard or Spark Driver Logs for real-time log output.

Notebook status bar Ready button that opens the Spark UI and Spark Driver Logs links

Image showing “Ready” in the status bar to access Spark UI and Driver Logs directly from the notebook

Token refresh

Authentication tokens expire after 30 minutes. In SageMaker Unified Studio, this is handled automatically. For programmatic use, you can use a background thread to keep the connection alive. The following helper reconnects transparently before the token expires:

import threading, time, boto3, urllib.parse
from pyspark.sql import SparkSession

class GlueSparkConnect:
    """Maintains a SparkSession with automatic token refresh."""

    def __init__(self, session_id, region="us-east-1", refresh_margin=300):
        self.session_id = session_id
        self.glue = boto3.client("glue", region_name=region)
        self.refresh_margin = refresh_margin  # seconds before expiry to refresh
        self._lock = threading.Lock()
        self.spark = self._connect()
        self._start_refresh_loop()

    def _connect(self):
        sc = self.glue.get_session_endpoint(SessionId=self.session_id)["SparkConnect"]
        encoded_token = urllib.parse.quote(sc["AuthToken"], safe="")
        remote_url = f"{sc['Url']}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
        self._token_expiry = sc["AuthTokenExpirationTime"].timestamp()
        return SparkSession.builder.remote(remote_url).getOrCreate()

    def _start_refresh_loop(self):
        def _loop():
            while True:
                sleep_for = max(self._token_expiry - time.time() - self.refresh_margin, 30)
                time.sleep(sleep_for)
                with self._lock:
                    self.spark = self._connect()
        t = threading.Thread(target=_loop, daemon=True)
        t.start()

# Usage
session = GlueSparkConnect("my-spark-connect-session")
session.spark.sql("SELECT 1 + 1 AS result").show()

The background thread sleeps until 5 minutes before token expiry, then transparently reconnects. Because the daemon thread exits when your script ends, there is no cleanup required.

Getting started

To start using Spark Connect with AWS Glue interactive sessions:

Use AWS Glue version 5.1 (Apache Spark 3.5.6).
Install PySpark 3.5.6 locally: pip install pyspark==3.5.6.
Grant your AWS Identity and Access Management (IAM) identity permissions for glue:CreateSession, glue:GetSession, and glue:GetSessionEndpoint.
Create a session with --session-type SPARK_CONNECT and connect from your preferred environment.

VPC note: If you connect to AWS Glue interactive sessions through a virtual private cloud (VPC) endpoint, add the new Spark Connect endpoint (com.amazonaws.{region}.glue.sessions) to your VPC configuration. Existing AWS Glue VPC endpoints don’t cover Spark Connect traffic.

For detailed instructions, see Connecting to a Spark Connect session in the AWS Glue Developer Guide.

About the authors

Deploy modern data platforms in minutes with MDAA

2026-07-02 Sudeshna Dash

Post Syndicated from Sudeshna Dash original https://aws.amazon.com/blogs/big-data/deploy-modern-data-platforms-in-minutes-with-mdaa/

Modern Data Architecture Accelerator (MDAA) is an open source framework that replaces infrastructure code with concise YAML configuration, so your team can deploy a governed, production-ready data architecture, reducing deployment time from months to weeks (depending on complexity and team experience).

Organizations building modern data architecture on AWS face a critical challenge: deploying production-ready, governed infrastructure traditionally requires 6–12 months of custom development, thousands of lines of infrastructure code, and continuous remediation cycles to maintain security and compliance. Governance is often added incrementally, treated as an afterthought that creates compliance gaps and engineering rework.

MDAA addresses this by replacing infrastructure code with concise YAML configuration, achieving up to 97.6 percent code reduction (from approximately 1,800 lines of AWS CloudFormation to 45 lines of MDAA YAML) while embedding governance from the start. The complete Governed Lakehouse Starter Kit deploys 491 AWS resources across 12 stacks from approximately 450 lines of YAML configuration, representing a 66x verbosity ratio where each line automatically expands into production-ready infrastructure.

In this post, we explore how MDAA transforms data architecture development from months of manual coding to production-ready deployment through configuration-driven infrastructure and embedded governance, examine a real customer transformation, and provide a clear implementation pathway for your own data modernization journey.

Customer use case and challenge

A university system office needed to modernize its analytics architecture across 17 campuses while managing sensitive educational data. Their third-party dependency created bottlenecks that slowed feature implementation from weeks to months, and their IT team lacked the cloud skillsets to build modern infrastructure independently.

With MDAA, they achieved:

95 percent reduction in time-to-value for dashboard and feature implementation (from weeks to hours).
17 campuses integrated into a unified, secure architecture.
7.2TB of data and over 8,000 dashboards migrated successfully.
Significant cost savings by removing third-party dependencies and reducing license costs.
Enhanced security posture for external stakeholders accessing sensitive educational data.

The team used MDAA to implement a modernization strategy with continuous integration and continuous delivery (CI/CD) for automated deployment. The architecture now supports rapid response to stakeholder requests while maintaining strict data governance through AWS Lake Formation.

Their transformation demonstrates what becomes possible when governance is embedded from launch rather than added incrementally, moving from months-long manual development to weeks of production-ready deployment through configuration-driven infrastructure.

Solution: MDAA and its value propositions

MDAA’s capabilities stem from its modular, composable architecture. The accelerator provides over 40 pre-built modules that encapsulate AWS best practices for security, governance, and operational excellence. Organizations describe the outcomes they want in MDAA-specific YAML configuration files (not CloudFormation or Terraform YAML) and the accelerator automatically translates these configurations into AWS Cloud Development Kit (AWS CDK) constructs, which then deploy via CloudFormation with embedded governance.

Configuration over code. The MDAA framework takes a fundamentally different approach: describe the outcomes you want in YAML, and the accelerator deploys production-ready infrastructure with embedded governance. Consider deploying a governed data lake where fraud detection teams need write access to transaction data, while marketing analytics teams require read-only access to customer behavior data. Traditional approaches require over 1,800 lines of CloudFormation across Amazon Simple Storage Service (Amazon S3) buckets, AWS Key Management Service (AWS KMS) keys, AWS Identity and Access Management (IAM) policies, and Lake Formation permissions. With MDAA, the same governed data lake is expressed in 45 lines of configuration, a 97.6 percent reduction, while helping you apply encryption, least-privilege access, and cross-account governance as built-in defaults.

The configuration deploys multi-zone S3 storage with KMS encryption, Lake Formation permissions with tag-based access control (TBAC) enabled, Amazon SageMaker Unified Studio for data product discovery, and encrypted AWS Glue Data Catalog with automated crawlers. All permissions flow through Lake Formation rather than individual IAM policies.

Embedded governance from day one. Governance is declared in YAML and deployed alongside infrastructure from the first run. Fine-grained access controls, encrypted data catalogs, data quality validation, audit trails, and sensitive data classification are all part of the same configuration. MDAA’s Governed Lakehouse starter kit defines an entire governed data architecture in roughly 450 lines of YAML, which produces approximately 29,700 lines of CloudFormation across 12 stacks (a 98.5 percent reduction in infrastructure code).

Modular, composable architecture. Each module is purpose-built to handle a specific capability within the data architecture. Modules communicate through AWS Systems Manager Parameter Store, passing resource identifiers (Amazon Resource Names (ARNs), IDs, and names) between stacks. This approach removes hardcoded dependencies. A KMS key created in one module can be referenced by another through parameter resolution, with all dependencies resolved automatically at deployment time.

The diagram illustrates the deployed architecture and team-level access flow that MDAA generates from the 45-line configuration.

Progressive architecture patterns. MDAA provides four reference architecture patterns that align to progressive stages of data infrastructure maturity:

Basic Data Lake deploys a governed data lake with built-in security controls, data quality checks, centralized metadata management using AWS Lake Formation and AWS Glue.
Data Science Platform extends the data lake with Amazon SageMaker notebooks, feature stores, and machine learning (ML) pipelines so data science teams can experiment and train models on governed data.
SageMaker Unified Studio adds a single interface for analytics and ML collaboration, connecting data engineers, analysts, and data scientists in one workspace.
Generative AI Platform layers Amazon Bedrock and Retrieval Augmented Generation (RAG) capabilities on top of your existing data foundation, so teams can build generative AI applications grounded in enterprise data.

Each pattern builds the one before it. You can start with the Basic Data Lake and adopt additional patterns as your team’s needs grow. MDAA’s modular design means you add capabilities without rearchitecting what you already deployed.

The infrastructure is versioned through GitHub, repeatable across environments, and auditable through comprehensive AWS CloudTrail logging. Data engineers focus on data pipelines and business logic while MDAA manages infrastructure complexity and governance integration. This represents the fundamental shift: from writing infrastructure code to describing the outcomes you want through configuration, with governance embedded from the start.

Use case of MDAA: Governed data architecture

DataOps teams spend significant time on governance tasks, including permissions management, compliance validation, and access control, rather than building pipelines and analytics. These aren’t data problems, they’re governance problems that consume engineering capacity meant for higher-value work. MDAA addresses this at the architectural level. Governance is declared in YAML and deployed alongside infrastructure from the first run.

The following sections walk through how each governance module works in practice.

Amazon SageMaker Unified Studio provides a governed data catalog where data producers publish data products, and consumers discover and subscribe to them. Your deployment with MDAA includes a pre-configured domain, blueprints (managed and custom), projects, and environment profiles, all defined in a single configuration file:

# sagemaker.yaml --- 16 lines that deploy 114 CloudFormation resources
domains:
  domain1:
    dataAdminRole:
      id: ssm:/{{org}}/govern1/generated-role/data-admin/id
    description: SMUS Domain 1
    userAssignment: MANUAL

    tooling:
      vpcId: '{{context:vpc_id}}'
      subnetIds:
        - '{{context:private_subnet_id1}}'
        - '{{context:private_subnet_id2}}'

    groups:
      team1:
        ssoId: '{{context:team1-group-sso-id}}'
      team2:
        ssoId: '{{context:team2-group-sso-id}}'

Behind this configuration, MDAA deploys an Amazon SageMaker Unified Studio domain with dedicated KMS keys, execution and provisioning roles, and single sign-on group profiles for team access. Data producers tag and publish assets with metadata, ownership, and classification. Consumers browse a searchable catalog, see only authorized assets, and request access through a governed workflow. Cross-account and cross-business-unit data sharing flows through a subscription model, ensuring every access grant is tracked, auditable, and revocable.

Use case of MDAA: Restricting access to cardholder data using Lake Formation

AWS Lake Formation provides fine-grained access control at database and table levels, removing manual IAM policy management. MDAA deploys AWS Lake Formation with pre-configured settings that disable IAMAllowedPrincipals, the critical governance setting that ensures all permissions flow through centralized governance:

# lakeformation-settings.yaml --- 6 lines that deploy 25 CloudFormation resources
lakeFormationAdminRoles:
  - id: generated-role-id:data-admin
createCdkLFAdmin: true
createDataZoneAdminRole: true
iamAllowedPrincipalsDefault: false

That last flag is the single most important governance setting in the platform. Without it, an IAM principal with glue:GetTable can read tables in the catalog, bypassing the entire access control model. Most manual setups miss this or defer it.

With the data lake configuration, you declare roles and access policies in YAML where admins get full control, engineers get read access to curated data, extract, transform, and load (ETL) roles get scoped write access, and MDAA compiles them into the correct S3 bucket policies and Lake Formation registrations.

Use case of MDAA: Ensuring data integrity with AWS Glue Data Quality

AWS Glue Data Quality runs automated validation rulesets continuously as part of the pipeline, not as periodic batch checks. MDAA’s data quality module supports over 15 built-in rule types, from completeness and uniqueness checks to statistical thresholds and data freshness validation:

# data-quality.yaml
projectName: example-project

rulesets:
  customer-data-quality:
    description: Validate customer data completeness and uniqueness
    targetTable:
      databaseName: project:databaseName/customer-data
      tableName: customers
    ruleset:
      - ruleType: IsComplete
        column: customer_id
      - ruleType: Uniqueness
        column: email
        comparisonOperator: ">"
        threshold: 0.95
      - ruleType: RowCount
        comparisonOperator: ">"
        value: 100

Quality metrics flow into Amazon CloudWatch for real-time alerting. If anomalies are detected, automated workflows quarantine affected records and alert data engineering teams before issues reach downstream consumers.

Protecting metadata at rest: AWS Glue Data Catalog encryption

Table schemas, column names, and partition structures can reveal sensitive information about an organization’s data architecture, even without access to the underlying data. AWS Glue Catalog Encryption secures metadata at rest using AWS KMS-managed keys. MDAA configures catalog encryption by default, so schema definitions and connection passwords are encrypted from initial deployment without requiring manual key management setup. Access to catalog metadata follows the same Lake Formation governance controls applied to the data itself, so teams see only the schemas that they’re authorized to query.

Auditing every data access event: CloudTrail integration

Every data access event must be logged and attributable to a specific identity. Without a complete audit trail, demonstrating compliance during a regulatory review becomes a manual, error-prone process. AWS CloudTrail captures API-level activity across the data infrastructure, recording who accesses what data, when, and from which service. MDAA configures CloudTrail integration by default, so audit logging is active from initial deployment rather than added retroactively. Log data flows into a centralized, tamper-resistant store, giving compliance teams a single location to query access history across all business units and accounts.

Identifying sensitive data automatically: Macie integration

In large environments, sensitive information spreads across dozens of S3 buckets through pipelines, transforms, and ad hoc data drops, and self-reporting data owners consistently produce gaps. Amazon Macie uses machine learning to automatically discover and classify sensitive data in S3, surfacing findings at the object level without manual tagging. MDAA configures Macie across your S3 buckets during deployment, routing findings to Amazon EventBridge where automated workflows can alert owners or trigger remediation.

Together, these controls form a layered defense: Lake Formation governs access to cataloged data, Glue Data Quality validates integrity on arrival, and Macie identifies sensitive data that lands outside governed pipelines to reduce compliance risk.

Multi-account data mesh

MDAA provides extensive support for multi-account data mesh setups, with decentralized data ownership across business units and centralized governance. The data mesh starter kit supports cross-account data product publishing and consumption, allowing organizations to scale data sharing while maintaining consistent security and compliance controls.

Technical implementation

Ready to deploy your modern data architecture? Here are the resources to get started:

MDAA Implementation Guide provides detailed instructions for deploying all starter packages, including architecture patterns, configuration examples, security best practices, and troubleshooting guidance.

MDAA Hands-on Workshop offers step-by-step guided implementation with AWS experts. The workshop covers configuration management best practices, implementation patterns, hands-on labs with real-world scenarios, and cleanup instructions.

GitHub Repository and Documentation provide source code, module reference, and comprehensive documentation.

Organizations approach MDAA from different starting points. Some modernize existing data architectures, migrating from on-premises infrastructure or legacy cloud architectures. Others build new architectures for artificial intelligence and machine learning (AI/ML) initiatives or generative AI applications. Financial services organizations require PCI-DSS compliance from day one. Healthcare organizations need controls that can help support HIPAA. Each journey benefits from MDAA’s configuration-driven approach and embedded governance.

Conclusion

MDAA transforms data architecture development from months of manual coding to production-ready deployment. Configuration-driven infrastructure reduces development time by 40–60 percent while embedding governance from the start. The university system’s 95 percent reduction in time-to-value demonstrates the outcome: organizations deploy secure, compliant, governed data architectures in weeks rather than months.

Financial services organizations can deploy architectures to help them align with PCI-DSS compliance requirements using Lake Formation access controls, Glue Data Quality validation, SageMaker Unified Studio data discovery, comprehensive CloudTrail audit trails, and automated Macie data classification, all inherited from configuration rather than built manually.

Data architecture journeys need not follow six-month timelines with governance added incrementally. MDAA provides an alternative: describe the outcomes you want through YAML configuration, inherit pre-validated security controls, and deploy production-ready infrastructure with comprehensive governance from initial deployment.

Security and compliance is a shared responsibility between AWS and the customer. For more information, see the AWS Shared Responsibility Model.

Need help or have questions? Contact AWS ProServe for personalized guidance on selecting the right package and deployment strategy for your organization.

About the author

Amazon Redshift RG: Faster and lower cost, Graviton-powered

2026-07-02 Stefan Gromoll

Post Syndicated from Stefan Gromoll original https://aws.amazon.com/blogs/big-data/amazon-redshift-rg-faster-and-lower-cost-graviton-powered/

Amazon Redshift recently announced the general availability of a new Graviton-powered instance called RG. Built on Amazon’s own Graviton processors, RG delivers:

Up to 2.2x faster performance for data warehouse workloads compared to RA3.
Up to 2.4x faster for Iceberg queries and 1.5x faster for Parquet queries through an integrated vectorized data lake engine.
No per-TB scan charges for data lake queries, eliminating the Amazon Redshift Spectrum cost applied on RA3 clusters.
30 percent lower cost per vCPU compared to RA3.

RG is both faster and cheaper. While cloud vendors typically charge more for faster performance or newer generation hardware, Amazon Redshift delivers better performance at lower cost.

In this post, we describe the innovations that make RG instances so much faster. We also share benchmark results showing that RG delivers up to 4.2x better price-performance than other leading data warehouses.

What makes RG so fast

The new RG instances are built from the ground up to take advantage of Graviton processors. The vectorized engine of Amazon Redshift is optimized with Graviton-based single instruction, multiple data (SIMD) kernels to deliver accelerated, parallelized execution for analytics workloads. Operations like predicate evaluations over Parquet encodings use Graviton vector comparison, table lookup, and vector manipulation intrinsics. To support these increased processing speeds, RG instances use custom-built Nitro SSDs. This lets RG use faster local storage as a caching layer for Amazon Redshift Managed Storage (RMS), data lake scans, and intermediate result sets for computations that can’t fit in memory. RG’s JIT (Just-In-Time) Analyze feature also collects statistics from data lake files automatically as queries run, so the optimizer can produce significantly better query plans. Together, these represent innovation across the entire stack: hardware acceleration with Graviton, vectorized execution with SIMD kernels, high-speed storage with Nitro SSDs, and intelligent query planning with JIT Analyze.

These optimizations, coupled with RG’s purpose-built high-performance vectorized data lake engine, combine to make Amazon Redshift’s new RG instances up to 2.2x faster than RA3 for analytics workloads at 30 percent lower cost.

Purpose-built high-performance vectorized data lake engine

With RA3, data lake queries offloaded scans to a separate compute fleet known as Amazon Redshift Spectrum. Because data lake queries ran on this separate compute, additional overhead was introduced to transfer query metadata and results between RA3 clusters and the Spectrum fleet. Amazon Redshift RG instances include a completely new built-in scan layer designed from the ground up for data lakes. This new scan layer includes a purpose-built I/O subsystem that incorporates smart prefetch capabilities to reduce data latency. The new scan layer is also optimized to process Apache Parquet files, the most commonly used file format for Iceberg, through fast vectorized scans that use SIMD kernels optimized for Graviton. The scan layer includes sophisticated data pruning mechanisms that operate at both partition and file levels, which significantly reduces the volume of data that needs to be scanned. This pruning capability works with the smart prefetch system to create a coordinated approach that maximizes efficiency throughout the entire data retrieval process.

The new purpose-built vectorized data lake engine is up to 2.4x faster than RA3 for Iceberg queries and 1.5x faster than RA3 for Parquet queries.

Because this new vectorized data lake engine integrates directly with the core execution engine of Amazon Redshift, new performance optimizations are possible compared to RA3. With this architecture, data lake queries on RG now benefit from fast local data caching, improved bloom filters, vectorized Parquet scans, and advanced filtering and pruning.

RG also solves a common problem customers face when querying data in the lake: open-format files like Iceberg in Amazon Simple Storage Service (Amazon S3) often lack useful metadata and statistics, which makes it difficult to run a SQL query optimally.

Statistics are metadata about your data, such as distinct value counts, min/max values, distribution patterns, and row counts. The query optimizer uses this information to choose the most efficient way to run a query. For example, when joining two tables, the optimizer needs to know how many unique values each side produces to pick the right join strategy. Without statistics, it has to guess, which often leads to slower joins and unnecessary data movement across nodes. This is where Amazon Redshift’s new feature called JIT (Just-In-Time) Analyze comes in. RG instances automatically fetch and store statistics of your Iceberg files as queries run, so Amazon Redshift can choose query execution strategies that are far more optimized than it could without these statistics.

These improvements make scans of Iceberg and Parquet data much faster than RA3. Removing Amazon Redshift Spectrum compute also means RG instances remove the $5/TB cost for data lake queries, which makes data lake queries cheaper and costs predictable. This is a triple win for data lake price-performance: faster performance, lower compute cost, and no per-TB scan cost.

Faster insights from faster data loads

Amazon Redshift RG’s fast I/O and Graviton-optimized engine result in faster data loads compared to RA3. To measure this improved performance, we ran the data ingestion step of 10TB TPC-DS and TPC-H on equivalently sized RA3 and RG clusters. RG ingested the TPC-DS dataset 2x faster and the TPC-H dataset 1.4x faster, as shown in the following figure.

Bar chart comparing data ingestion time on RA3 and RG, showing RG loads TPC-DS 2x faster and TPC-H 1.4x faster

The new Graviton-based RG instances are up to 2.0x faster for data loads compared to RA3 instances. This means workloads can see the latest data sooner, and users and agents can get up-to-date insights faster. This faster ingestion on RG comes at 30 percent lower cost compared to RA3, resulting in up to 2.9x better price-performance for data loads compared to RA3 instances.

What customers are saying

Amazon Redshift customers are already seeing performance and cost benefits of switching to RG. Southwest Airlines and tombola tested their business-critical workloads, and found they could get better performance and save on cost:

Southwest Airlines

“Amazon Redshift RG instances have the potential to deliver meaningful business impact for Southwest Airlines. Based on initial testing in our development environment, our data warehouse workloads run 50–60% faster, and data lake analytics are 45% faster—enabling teams to get insights sooner, respond to operational conditions faster, and make data‑driven decisions with less latency. These early results are encouraging, and we are excited to validate and scale these improvements in production. All of this comes without per‑terabyte Spectrum scanning charges, delivering 30% lower cost than RA3 at a time when fuel prices continue to pressure industry margins!!”

— Sean Lynch, Vice President, Data and Architecture, Southwest Airlines

tombola

“The new Graviton-based Amazon Redshift RG instances delivered 1.8x–2x faster write throughput and up to 2.2x faster read speeds compared to RA3 across a diverse set of batch and analytical jobs — enabling us to process 40% more within the same window. Compressed ETL cycles, accelerated time-to-insight, and decision-making no longer bottlenecked by the pipeline — together, these translated directly into fresher data reaching our analysts and business teams sooner. What made this even more compelling was a concurrent 30% reduction in compute spend alongside the gains — delivering more for less is a rare outcome, and one worth highlighting. In a volume-heavy gaming industry at tombola, where query latency and cost compound at scale, this has been one of the more impactful platform decisions we’ve made this year.”

— Akshay Srinivasan, Data Engineer, tombola

Qoala

“After migrating our Amazon Redshift cluster from RA3 to the new Graviton-based RG instances, we saw 60–70% faster query processing times across our BI and analytics workloads. As a growing insurtech platform handling millions of policy transactions, faster time-to-insight means our data team can deliver dashboards and reports to the business sooner. We moved to a larger node configuration to accommodate future growth, and the performance gains far exceeded the incremental investment – making this one of the most impactful infrastructure decisions we’ve made this year.”

— Umar Abdul Aziz, VP of Data, Qoala

Performance results

To see how RG stacks up, we ran benchmarks derived from the industry-standard TPC-DS and TPC-H benchmarks at 10TB scale on the new Amazon Redshift RG instances and on leading alternative data warehouses. These benchmarks are designed to run queries of various operational requirements and complexities, such as ad hoc, reporting, iterative online analytical processing (OLAP), and data mining. We sized each data warehouse at approximately the same on-demand cost ($32/hr) and ran three power runs of each benchmark out of the box, with no special tuning or manual customization. The results are shown in the following charts.

Bar chart of TPC-DS 10TB price-performance showing Amazon Redshift RG leading alternative data warehouses

Bar chart of TPC-H 10TB price-performance showing Amazon Redshift RG leading alternative data warehouses

The new RG instance leads, and by a large margin. Better price-performance means better performance and lower cost.

Conclusion

Amazon Redshift RG instances are the next generation of analytics engine, delivering high performance for data warehouse and data lake workloads. Because RG supports all the same workloads and features as RA3, getting started is straightforward. See our migration guide for how to upgrade and start getting better performance at lower cost.

Find the best price-performance for your workloads

The benchmarks used in this post are derived from the industry-standard TPC-DS and TPC-H benchmarks, and have the following characteristics:

We use the schema and data unmodified from TPC-DS and TPC-H.
The queries are generated using the official TPC-DS and TPC-H kits with query parameters generated using the default random seed of the kits. TPC-approved query variants are used for a warehouse if the warehouse doesn’t support the SQL dialect of the default queries.
The test includes the 99 TPC-DS SELECT queries and 22 TPC-H SELECT queries. It doesn’t include maintenance and throughput steps.
Three power runs were run, and the best run is taken for each data warehouse.
Price-performance is calculated as the cost per hour (USD) divided by 3,600 seconds/hour times the benchmark geomean in seconds, which is equivalent to the geomean cost per query. The latest published on-demand pricing is used for all data warehouses.

We call this the Cloud Data Warehouse benchmark, and you can reproduce the preceding benchmark results using the scripts, queries, and data available in our GitHub repository. It’s derived from the TPC-DS benchmarks as described in this post, and as such isn’t comparable to published TPC-DS results, because the results of our tests don’t comply with the official specification.

About the authors

Scale analytics with Amazon Redshift multi-warehouse enhancements

2026-06-29 Raza Hafeez

Post Syndicated from Raza Hafeez original https://aws.amazon.com/blogs/big-data/scale-analytics-with-amazon-redshift-multi-warehouse-enhancements/

Onboard analytics workloads at scale with Amazon Redshift’s improved remote table data definition language (DDL), materialized view improvements, and concurrency scaling enhancements for zero-ETL and auto-copy.

As organizations scale their analytics capabilities, they need the ability to add workloads without disrupting production operation or being constrained by the resources of a single data warehouse. In this post, we introduce new capabilities of Amazon Redshift that enhance our multi-warehouse and scaling capabilities: remote materialized view (MV) operations, remote table DDL support, and concurrency scaling enhancements for zero-ETL and S3 event integration. These features help you build more scalable, performant decentralized analytics architectures on Amazon Redshift.

Let us review how these new features enable you to run analytics at scale.

New remote materialized view operations

Amazon Redshift now classifies CREATE MATERIALIZED VIEW as a user workload, enabling concurrency scaling to execute MV logic on additional warehouses during resource contention. This ensures queries consistently benefit from MV performance advantages, even under heavy load.
Amazon Redshift now supports creating MVs on remote data shares, allowing customers who share data across Redshift warehouses to leverage MV performance benefits on both local and shared data.
Consumer warehouses can now refresh MVs created on a producer and create MVs on top of data-shared MVs, enabling full MV parity across producer and consumer warehouses in a data sharing architecture.

New remote table DDL operations

ALTER TABLE ALTER DISTSTYLE operations now work on remote warehouses through concurrency scaling and data sharing. You can dynamically optimize data distribution across distributed environments, improving query performance and resource utilization without requiring data migration. This is especially valuable for data engineers fine-tuning performance across multiple warehouses and administrators adapting to changing query patterns.
ALTER TABLE APPEND operations now extend to remote warehouses through concurrency scaling and data sharing. This consolidates data across distributed environments, so you can efficiently combine tables without complex data movement or extract, transform, and load (ETL) processes. Organizations managing dynamic table operations across multiple environments can maintain data consistency while reducing operational overhead.

Concurrency scaling improvements

Amazon Redshift’s enhanced zero-ETL feature now supports concurrency scaling for automated data ingestion from applications and operational sources.
Amazon Redshift’s enhanced auto-copy feature now supports concurrency scaling for automated data ingestion from S3.
Amazon Redshift now extends concurrency scaling to support COPY queries from Amazon S3. You can now scale data ingestion automatically with concurrency scaling for batch workloads.

With these new concurrency scaling capabilities, you can maintain consistent data freshness without compromising existing warehouse performance. This eliminates the traditional trade-off between analytics and data loading. Apart from turning on concurrency scaling, no additional changes are required to take advantage of these features.

Customer use cases

This section covers two industry use cases: the first for a financial services customer and the second for a gaming industry customer.

Financial services use case

The following is a sample architecture for a large financial services customer with global operations. This customer uses a multi-warehouse architecture built on Amazon Redshift.

Financial services multi-warehouse architecture using STG, DWH, ETL, and USR Amazon Redshift warehouses

The staging (STG) warehouse serves as a raw zone for data from various sources, like the bronze layer of a medallion architecture. This warehouse also cleanses and standardizes the raw data to the silver layer and makes it available for further processing. The STG warehouse uses MVs to process millions of nested JSON messages and extract attributes into scalar columnar Amazon Redshift tables.

CREATE MATERIALIZED VIEW rawdb.fsi.customer_orders_raw
distkey(c_custkey) sortkey(c_custkey) AS (
    SELECT c_custkey,
        o.o_orderstatus,
        o.o_totalprice,
        o_idx
    FROM customer_orders_lineitem c,
        c.c_orders o AT o_idx
);
REFRESH MATERIALIZED VIEW rawdb.fsi.customer_orders_raw;

The DWH warehouse serves as the primary Amazon Redshift instance and gold layer, providing data to consuming applications like Business Objects and Tableau. The zero-ETL concurrency scaling improvements provide consistent data freshness even when zero-ETL ingestion spikes occur alongside heavy DWH workloads. The DWH MVs provide fast access to aggregated data for Tableau extracts and Business Objects live reports. The DWH warehouse takes advantage of concurrency scaling when multiple MVs need to be refreshed on the DWH instance.

CREATE MATERIALIZED VIEW bodb.final.customer_churn_tbl
AS (
    SELECT state,
        account_length,
        area_code,
        total_charge/account_length AS average_daily_spend,
        cust_serv_calls/account_length AS average_daily_cases,
        churn
    FROM custdb.final.customer_activity_all
);
REFRESH MATERIALIZED VIEW bodb.final.customer_churn_tbl;

The ETL01/02 warehouses serve as dedicated compute environments for running project-specific ETL jobs, while the USR01/02 warehouses handle user workloads such as ad-hoc analysis or model building from dbt. When new objects are required by user workloads, they are created and maintained on the remote producer warehouse (DWH).

ALTER TABLE salesdb.final.sales_report_all
ALTER DISTKEY sales_id;

ALTER TABLE APPEND salesdb.final.sales_report_all
FROM stagingdb.sales.sales_2026_02;

Gaming industry use case

A leading gaming company has built their entire analytics infrastructure on AWS, with their analytics team managing data streaming from games, data warehousing, and business intelligence tools. They standardized Amazon Redshift across the organization, migrating off Vertica running on Amazon Elastic Compute Cloud (Amazon EC2). After overcoming early challenges with cluster resize operations, the team became strong advocates for Amazon Redshift and now runs their primary production cluster on 32 ra3.16xlarge nodes.

As their data ingestion pipeline grew, query workloads began competing with data ingestion processes, creating performance bottlenecks. Rather than scaling up their primary cluster, they implemented a workload isolation strategy using Amazon Redshift data sharing. The customer launched a second 16-node ra3.4xlarge cluster as a data share consumer, with the primary cluster serving as the producer. This architecture allowed them to migrate consumption workloads to the consumer cluster while the producer focused on data ingestion, effectively supporting growth without increasing the primary cluster size.

Gaming company architecture with a producer Amazon Redshift cluster sharing data to a consumer cluster

Recognizing the advantages of this distributed architecture, the gaming company expanded their approach by migrating workloads to Amazon Redshift Serverless, further using the data sharing model for workload isolation. Amazon Redshift’s remote materialized view capability allowed the gaming company to create materialized views directly on the data shared by the producer cluster. Each consumer cluster could now build materialized views optimized for its specific workload patterns. This created pre-aggregated datasets, custom join strategies, and workload-specific data distributions, without impacting the producer cluster’s performance or requiring data duplication. The producer warehouse maintains data distribution and sorting strategies designed for generic enterprise needs, providing consistent data quality across all consumers. Meanwhile, consumer warehouses used remote materialized views to fine-tune query performance for their distinct analytical requirements, whether supporting real-time player analytics, business intelligence dashboards, or ad-hoc data science workloads. This distributed approach to data consumption optimization proved essential for the gaming company. It delivered fast query performance across diverse analytical workloads while maintaining a single source of truth in the producer cluster and avoiding the operational overhead of managing redundant data copies.

Best practices

To get the most out of these new capabilities, consider the following best practices:

Enable concurrency scaling on your Amazon Redshift clusters and Serverless workgroups to allow ETLs and user queries to run even faster, providing consistent report and dashboard performance.
Set up usage limits for concurrency scaling on both Amazon Redshift provisioned clusters and Serverless workgroups by configuring an appropriate MaxRPU setting. This helps you avoid unexpected additional costs. For more information, see the Amazon Redshift usage limits documentation.
Use remote MVs to offload resource-intensive MV creation and refresh operations from your primary warehouse to remote data share clusters.

Conclusion

In this post, we walked through the new MV refresh features, remote table DDL capabilities, and expanded concurrency scaling support for zero-ETL and S3 auto-copy. These features help you move beyond the constraints of a single warehouse. They are particularly valuable for organizations managing distributed data architectures that require dynamic table management across multiple environments while maintaining data consistency and adapting quickly to changing workloads. To get started, make sure you are running the latest Amazon Redshift version. Then visit the Amazon Redshift documentation to learn more about concurrency scaling, data sharing, and materialized views.

About the authors

What the June 2026 Threat Technique Catalog update means for your AWS environment

2026-06-29 Shannon Brazil

Post Syndicated from Shannon Brazil original https://aws.amazon.com/blogs/security/what-the-june-2026-threat-technique-catalog-update-means-for-your-aws-environment/

The AWS Customer Incident Response Team (AWS CIRT) encounters patterns that repeat across engagements when helping customers respond to security incidents. We’re passionate about making sure that information is accessible so that everyone can improve their security posture and their organization’s resilience to disruption. The primary method we use to share this information is the Threat Technique Catalog for AWS (TTC). The latest update to the catalog for June 2026 focuses on container security, organization-level trust, and compute hijacking. Each new entry reflects something we’ve encountered in practice, and each provides straightforward mitigation. This post breaks down what changed, why it matters, and what you can do about it today.

What we’re seeing

We’ve added five new entries to the TTC.

EKS workload modification

Amazon Elastic Kubernetes Service (Amazon EKS) gives teams powerful orchestration capabilities. We’re seeing threat actors who have obtained Kubernetes credentials or an AWS Identity and Access Management (IAM) role with EKS permissions modify running workloads—altering container images, injecting sidecar containers, or changing pod specifications to introduce malicious code into a deployment.

Nothing new is created. The workload already exists, it might be running in production, and by modifying it in place the threat actor inherits the network access, service account permissions, and data access the legitimate workload already had. Without admission controllers or image verification, these changes can go unnoticed until the impact shows up downstream. Enforcing image signing through admission controllers, restricting workload changes with Kubernetes role-based access control (RBAC), and enabling Amazon GuardDuty EKS Protection to surface anomalous cluster activity all reduce this risk. For more information, see EKS Modification – Workload Integrity Degradation.

Exploit public-facing application – EKS

Publicly exposed Kubernetes API servers and misconfigured ingress controllers continue to be an entry point we see exploited. This technique captures threat actors targeting the customer-deployed workloads running on Amazon EKS—not EKS itself—and their exposure to the internet.

The pattern starts with an exposed service and an application-level weakness, then pivots from the compromised pod toward broader cluster access. When inside a pod, a threat actor can query the instance metadata service, read mounted service account tokens, or move laterally across the cluster network. Limiting public exposure of the Kubernetes API server, applying network policies to restrict pod-to-pod communication, and running workloads with least-privilege service accounts reduce the risk of this technique succeeding. For more information about this technique, see Exploit Public-Facing Application.

Assume root into organization member account

AWS Organizations centralizes trust across member accounts, and that trust runs in one direction—from the management account downward. We’ve observed threat actors who compromise a management account—or gain sufficient privilege within one—use that position to assume root access into member accounts using sts:AssumeRoot. Because the trust is inherent to the organization structure, this can avoid the access controls a member account administrator has configured.

With root access to a member account, a threat actor can disable security controls, delete resources, change billing configurations, and establish persistence that survives remediation focused on IAM principals. We strongly encourage implementing service control policies (SCPs) that restrict which principals can call sts:AssumeRoot and under what conditions, and monitoring for sts:AssumeRoot calls in AWS CloudTrail. For more information, see Assume Root into Organization Member Account.

Compute hijacking – EKS

Compute hijacking remains one of the most common motivations we see behind unauthorized access, and Amazon EKS clusters are increasingly the target. Threat actors deploy cryptocurrency mining or other compute-intensive workloads inside compromised clusters, consuming customer resources and generating unexpected cost.

What sets EKS-based hijacking apart is scale. In clusters without resource quotas, a single compromised service account can consume all available capacity across nodes. The workloads use legitimate-looking images pulled from public registries, which makes image scanning alone insufficient. Setting resource quotas and limit ranges, restricting which registries workloads can pull from, and enabling Amazon GuardDuty EKS Protection to flag mining behavior provides effective detection. For more information, see Resource Hijacking: Compute Hijacking – EKS.

Invite accounts to unknown organization

A threat actor with access to a standalone account—or one they’ve removed from its legitimate organization—invites it into an organization they control. After the account joins, it falls under the threat actor’s governance. The threat actor’s organization can apply SCPs that restrict the legitimate owner’s actions, gain visibility into the account’s resources through organizational services, and access consolidated billing information. The legitimate owner finds themselves locked out of their own governance controls. Monitoring organizations:InviteAccountToOrganization and organizations:AcceptHandshake, and implementing SCPs that prevent accounts from leaving their legitimate organization are important preventive measures. For more information, see Modify Cloud Resource Hierarchy: Invite Accounts to Unknown Organization.

What’s updated

We’ve refreshed three existing entries. S3 Object Collection now captures additional API calls used for bulk data staging from Amazon Simple Storage Service (Amazon S3), with refined detection guidance and mitigations that use recent Amazon S3 security features. Compute Hijacking – ECS adds methods threat actors use to deploy unauthorized tasks in Amazon Elastic Container Service (Amazon ECS), including abuse of overly permissive task execution roles. Role Assumption and Federated Access has been expanded to cover new cross-account role assumption variations and identity provider manipulation, with sharper guidance for distinguishing legitimate federated access from unauthorized use.

The current trend

This June update reflects a clear trend: threat actors are increasingly targeting container orchestration platforms and using organizational trust relationships to their advantage. The container techniques show that as organizations adopt Kubernetes at scale, the attack surface grows with it. The organization-level techniques show that threat actors understand organizational trust relationships.

The common thread is that every one of these techniques operates within the boundaries of legitimate functionality. Modifying a workload, assuming cross-account trust, and joining an organization are all expected actions in healthy environments.. Detection, then, depends entirely on context: the principal, the timing, and the sequence of events that follows.

The Threat Technique Catalog for AWS is designed to help with this. We encourage teams to review the relevant entries and assess whether their current monitoring would catch these patterns:

Unexpected modifications to EKS workload specifications
Pod deployments that use unsigned container images
sts:AssumeRoot calls into member accounts
Unbounded compute consumption in your EKS clusters that could be prevented by resource quotas
Unexpected organization invitations to your accounts

Each of the threats leaves traces in AWS CloudTrail and Kubernetes audit logs, and the TTC provides specific guidance on what to watch for and how to respond.

Looking ahead

The Threat Technique Catalog for AWS exists because we believe the patterns we observe during security engagements shouldn’t stay behind closed doors. When we see techniques repeating across customers, the most effective thing we can do is document them and make that knowledge available so you can act on it before you’re in the middle of an incident.

This June update adds five new entries and updates three existing ones, and the catalog will continue to evolve. Our team updates it based on what we’re seeing in the real world when helping customers respond to security events. We encourage security teams to review the catalog, incorporate its techniques into threat modeling exercises, and use it as a shared vocabulary for discussing cloud-specific threats.

Explore the full catalog: Threat Technique Catalog for AWS – Full Matrix

Additional resources

If you have feedback about this post, submit comments in the Comments section below.

Dual-token authentication for Nakama game servers with Amazon Cognito on AWS

2026-06-29 Madhusudan Athinarapu

Post Syndicated from Madhusudan Athinarapu original https://aws.amazon.com/blogs/architecture/dual-token-authentication-for-nakama-game-servers-with-amazon-cognito-on-aws/

When your game server needs both a managed identity provider and its own session system, players face a broken experience if authentication forces a redirect or stalls gameplay. Dual-token authentication for Nakama game servers with Amazon Cognito solves this by connecting two independent session systems, each with its own token lifecycle, without interrupting the player. This post shows you how.

Amazon Cognito handles player identity and Nakama manages game sessions. Cognito issues a JWT, a server-side Go hook validates it and exchanges the verified identity for a Nakama session token. Each token is validated independently on every request. The pattern applies to game servers such as Nakama that support runtime authentication hooks.

The infrastructure wraps Nakama in a default-closed routing layer. Amazon CloudFront serves as the single HTTPS entry point, AWS WAF filters traffic at the edge, an Application Load Balancer (ALB) enforces an explicit route allow-list for HTTP, and a Network Load Balancer (NLB) handles WebSocket TCP passthrough. Nakama runs on Amazon Elastic Container Service (Amazon ECS) on AWS Fargate. In this post, we cover the Cognito configuration, the Go hook, the Terraform infrastructure, and the WebSocket lifecycle controls.

In this post, you learn how to:

Configure an Amazon Cognito User Pool for SRP-based game client authentication with no client secret.
Implement a Go runtime hook that validates Cognito JWTs and bridges player identity to Nakama sessions.
Set up a default-closed routing layer using Amazon CloudFront, an ALB, and an NLB.
Manage the WebSocket connection lifecycle under the NLB TCP idle timeout model.

Solution overview

The architecture has four layers for authenticating and routing traffic.

The following diagram shows the architecture. Amazon CloudFront is the single entry point, routing HTTP API traffic through an Application Load Balancer (ALB) to Nakama on Amazon ECS, and WebSocket traffic through a Network Load Balancer (NLB) via TCP passthrough.

Architecture diagram showing dual-token authentication flow from client through Amazon CloudFront, ALB, and NLB to Nakama on Amazon ECS

Figure 1. Dual-token authentication architecture for Nakama on AWS.

Traffic flows through the system in six steps:

Client → Amazon Cognito — The player authenticates using USER_SRP_AUTH. The password never leaves the client. Amazon Cognito returns a JWT access token.
Client → Amazon CloudFront — Requests enter via Amazon CloudFront (HTTPS). AWS WAF inspects traffic at the edge before it reaches the origin.
CloudFront → ALB (port 80) — /* HTTP API traffic. The ALB is security-group locked to the CloudFront managed prefix list only.
CloudFront → NLB (port 7350) — /ws* WebSocket traffic. The NLB performs TCP passthrough with no HTTP inspection.
ALB → Amazon ECS (Nakama) — For auth requests: the BeforeAuthenticateCustom Go hook validates the Cognito JWT and extracts the sub claim as the Nakama user ID. For other API calls: Nakama validates its own session token.
NLB → Amazon ECS (Nakama) — Persistent WebSocket connection. Nakama validates the session token from the token query parameter at connect time.

Why two load balancers

The ALB and NLB serve different purposes and cannot be combined into one.

The ALB operates at the HTTP layer (Layer 7). It reads the path, applies listener rules, and returns 403 for unlisted routes.

The NLB operates at the TCP layer (Layer 4) and passes the raw stream to Nakama unchanged. Nakama receives the WebSocket upgrade directly from the client, validates the session token, and manages the connection lifecycle end-to-end.

Amazon CloudFront routes /ws* to the NLB and everything else to the ALB, so each connection type gets the appropriate handling behind a single HTTPS endpoint.

Prerequisites

Before you deploy this solution, make sure you have:

Terraform >= 1.5.0 (download).
Go >= 1.21 (to build the Nakama plugin locally).
Docker and the AWS Command Line Interface (AWS CLI) configured with appropriate credentials.

The repository includes a browser-based test app (/app) that demonstrates the full sign-up, sign-in, and Nakama token exchange flow.

Authenticate players with Amazon Cognito

Amazon Cognito provides a managed user directory that issues JWTs without requiring you to run your own identity server or store credentials. The game server validates the JWT independently on each request, with no callback to Cognito needed. This decouples identity from game sessions: Cognito owns the player’s identity, Nakama owns the game session, and neither system depends on the other at runtime.

Players self-register by calling the Cognito SignUp API from the game client. The User Pool verifies their email before the account becomes active. After sign-in, Cognito returns a JWT access token containing the player’s sub claim (a UUID), which becomes the Nakama user ID in the next step.

Authentication uses the USER_SRP_AUTH flow. The password never leaves the client device. The User Pool App Client is configured as a public client with no client secret, since your game client runs in the browser or a native app where any embedded secret is extractable. With SRP, no secret is needed; security comes from the protocol itself.

After a successful sign-in, Amazon Cognito returns a JWT access token. This token carries the player’s identity claims and is signed with an RSA key pair unique to your User Pool. The sub claim — a UUID generated by Cognito — uniquely identifies the player and becomes the Nakama user ID in the next step.

The auth Terraform module configures the App Client with generate_secret=false and permits only ALLOW_USER_SRP_AUTH and ALLOW_REFRESH_TOKEN_AUTH flows. The resulting JWT access token is short-lived (1 hour by default) and carries the sub, iss, exp, and client_id claims that the Go hook validates in the next step.

Bridge Cognito identity to Nakama sessions

Nakama’s server-side runtime supports Go plugins exclusively. The hook in this section is written in Go using Nakama’s runtime.Initializer interface. This is a constraint of the Nakama runtime model.

Once the client has a Cognito JWT, it needs a Nakama session token to make game API calls.

Validate the Cognito JWT in the Go hook

The game server cannot trust the identity claim sent by the client directly. Any client can forge a user ID. JWT validation cryptographically proves the identity was issued by Cognito, preventing player impersonation.

The hook performs five checks in order: token format, algorithm (RS256 only), signature against the JWKS, expiry, and issuer/audience matching your specific User Pool.

func validateCognitoJWT(token string, env map[string]string) (string, error) {
    parts := strings.Split(token, ".")
    if len(parts) != 3 {
        return "", runtime.NewError("invalid token format", 3)
    }

    // Parse the header to get the key ID (kid)
    var header struct {
        Kid string `json:"kid"`
        Alg string `json:"alg"`
    }
    headerBytes, _ := base64.RawURLEncoding.DecodeString(parts[0])
    json.Unmarshal(headerBytes, &header)

    if header.Alg != "RS256" {
        return "", runtime.NewError("unsupported algorithm: "+header.Alg, 3)
    }

    // Fetch the public key from the JWKS cache
    pubKey, err := jwksCache.getKey(header.Kid)
    if err != nil {
        return "", runtime.NewError("token validation failed", 16)
    }

    // Verify the RSA signature
    hash := sha256.Sum256([]byte(parts[0] + "." + parts[1]))
    signatureBytes, _ := base64.RawURLEncoding.DecodeString(parts[2])
    if err := rsa.VerifyPKCS1v15(pubKey, crypto.SHA256, hash[:], signatureBytes); err != nil {
        return "", runtime.NewError("invalid token signature", 16)
    }

    // Validate claims: expiry, issuer, audience
    if time.Now().Unix() > claims.Exp { return "", runtime.NewError("token expired", 16) }
    if claims.Iss != expectedIssuer || claims.ClientID != env["COGNITO_CLIENT_ID"] {
        return "", runtime.NewError("invalid issuer or audience", 16)
    }

    return claims.Sub, nil // sub claim becomes the Nakama user ID
}

Security note: The hook never trusts the identity string sent by the client. It discards it and overwrites the Nakama user ID with the sub claim from the validated JWT. A client that sends a forged sub cannot impersonate another player — the hook ignores the body value entirely.

Cache JWKS keys with thundering herd protection

Amazon Cognito rotates its signing keys periodically. The hook caches keys with a 1-hour TTL. A 30-second re-fetch guard prevents multiple goroutines from calling the JWKS endpoint simultaneously when the cache expires.

func (c *JWKSCache) refresh() error {
    c.mu.Lock()
    defer c.mu.Unlock()

    // Thundering herd protection: if another goroutine already
    // refreshed within the last 30s, use the updated cache
    if time.Since(c.fetched) < 30*time.Second {
        return nil
    }

    // ... fetch and parse JWKS from Cognito endpoint
}

Register the hook

The hook registers itself in InitModule, the entry point called by Nakama when the plugin loads:

func InitModule(ctx context.Context, logger runtime.Logger, db *sql.DB,
    nk runtime.NakamaModule, initializer runtime.Initializer) error {

    if err := initializer.RegisterBeforeAuthenticateCustom(beforeAuthenticateCustom); err != nil {
        return fmt.Errorf("failed to register hook: %w", err)
    }
    logger.Info("Cognito JWT validation hook registered")
    return nil
}

When the client calls POST /v2/account/authenticate/custom with the Cognito JWT as the id field, Nakama calls beforeAuthenticateCustom before processing the request. If the JWT is valid, the hook sets in.Account.Id = sub and returns. Nakama creates or links the account and returns a session token to the client.

If your server is not Nakama, for example, Colyseus, Photon, or a custom WebSocket server, implement the same five checks (algorithm, signature, expiry, issuer, audience) in your server’s middleware or plugin language. The JWKS endpoint and JWT structure follow the OIDC standard, so any OIDC-compliant identity provider (not only Amazon Cognito) works with this pattern.

Deploy the infrastructure

The infrastructure is organized into six Terraform modules: network (Amazon Virtual Private Cloud (Amazon VPC), subnets, security groups), compute (Amazon ECS cluster, ALB, NLB, Amazon Elastic Container Registry (Amazon ECR)), auth (Cognito User Pool), cdn (CloudFront distribution), waf-cloudfront (AWS WAF Web ACL), and ops (IAM, AWS Systems Manager access). A bootstrap module creates the S3 state backend and AWS Key Management Service (AWS KMS) key before the main deployment.

Deploy with:

# One-time: provision the Terraform state backend
cd terraform/bootstrap && terraform init && terraform apply

# Deploy everything
cd terraform && terraform init -backend-config=config/backend-dev.hcl
make deploy

make deploy builds and pushes the Nakama container image to Amazon ECR, then runs terraform apply. The image tag auto-increments from the latest tag in ECR.

ALB routing: explicit allow list

The ALB default listener action returns 403. Only the paths in the following table reach Nakama. Requests to unlisted paths are rejected before they reach the game server.

Priority	Path	Target	Purpose
1	/healthcheck	Nakama port 7350	Health monitoring
2	/v2/account/authenticate/*	Nakama port 7350	Session bridge: Go hook validates JWT
10	/v2/*	Nakama port 7350	Nakama REST API v2
11	/v1/*	Nakama port 7350	Nakama RPC (v1)
Default	*	403 Forbidden	Request never reaches Nakama

The default-403 posture means a misconfigured client or a scanner probing arbitrary paths gets a 403 at the ALB, not an error from the game server. This limits the attack surface to the explicitly listed API surface.

Security group chain

The network layer enforces two security group rules:

The ALB security group allows inbound only from the CloudFront managed prefix list. As an additional application-layer check, CloudFront sends a shared secret in the X-CloudFront-Secret header on every request; ALB listener rules reject any request missing the correct value with a 403. The NLB security group applies the same CloudFront managed prefix list restriction at Layer 4.
The NLB security group allows inbound TCP 7350 only from the CloudFront managed prefix list. The ECS task security group allows inbound port 7350 only from the ALB security group (HTTP API) and from the NLB security group (WebSocket).

Together, the routing and security group chain means the only path to Nakama is: Internet → CloudFront → AWS WAF → ALB or NLB → ECS. No hop can be skipped.

Manage the WebSocket connection lifecycle

The NLB TCP passthrough model creates a lifecycle challenge: the NLB drops idle TCP flows after 350 seconds (the AWS default, not configurable). If a player’s connection sits idle, the NLB closes the underlying TCP connection while Nakama still holds an open socket.

The following table describes the four controls that handle this:

Control	Value	Purpose
NLB TCP idle timeout	350s	NLB drops idle TCP flows. Cannot be changed.
Nakama ping interval	10s	Nakama sends a WebSocket ping every 10s, keeping the TCP flow active.
Nakama pong wait	20s	If the client does not respond to a ping within 20s, Nakama closes the connection.
token_expiry_sec	7200	Nakama rejects session tokens older than 2 hours at connect time.
single_socket	true	A new connection from the same user kills the previous one, preventing stale sessions.

The ping/pong keepalive

The 10-second ping interval is the key control. Nakama sends a WebSocket ping frame every 10 seconds on each active connection. The client responds with a pong. This keeps the NLB TCP flow alive well within the 350-second idle timeout. If the client goes silent, Nakama detects the missing pong within 20 seconds and closes the socket cleanly.

Session expiry at connect time

The NLB performs TCP passthrough, so there is no opportunity to inspect HTTP headers or validate the session token at the network layer. Nakama validates the session token from the token query parameter when the WebSocket upgrade request arrives. A token older than token_expiry_sec is rejected and the connection is closed before any game messages are processed.

Single socket enforcement

single_socket: true verifies that when a player opens a second connection (after a network drop and reconnect, for example) the server closes the first connection. Without this, a player’s Nakama state can be split across two concurrent connections if the client does not cleanly close the first one.

The four-layer model (keepalive, timeout, session expiry at connect, one-connection-per-user enforcement) applies to any real-time server behind an NLB TCP passthrough: Colyseus, Photon, custom WebSocket backends, or any game server that manages persistent connections. If your server does not have built-in ping/pong, implement application-level heartbeat messages that serve the same role.

Security note: The session token travels as a query parameter (?token=...) in the WebSocket upgrade URL. Query parameters appear in server access logs, load balancer logs, Amazon CloudFront logs, and browser history. Mitigations: all connections use TLS (token encrypted in transit), session tokens are short-lived (2 hours), and single_socket invalidates old connections on reconnect. For production deployments, consider log redaction policies for the token parameter.

Clean up

To avoid ongoing AWS charges, destroy all resources when you no longer need them.

Destroy the main infrastructure first:

cd terraform && terraform destroy

Then destroy the Terraform state backend:

cd terraform/bootstrap && terraform destroy

Confirm resources are removed by running terraform state list (should return empty) or checking the AWS Management Console.

Conclusion

In this post, you implemented a dual-token authentication architecture for a Nakama game server on AWS. Amazon Cognito handles player identity through JWT validation; a Go runtime hook bridges verified identity into Nakama sessions; and the infrastructure enforces a routing layer where HTTP API traffic passes through an Application Load Balancer with an explicit allow list and WebSocket connections reach Nakama directly through a Network Load Balancer TCP passthrough.

The four-layer WebSocket lifecycle model can be applied to real-time game servers behind an NLB TCP passthrough, not Nakama exclusively.

For production deployments, consider these next steps:

Replace the PostgreSQL sidecar with Amazon Aurora PostgreSQL-Compatible Edition for persistent, managed player data storage.
Add a custom domain with TLS re-encryption between Amazon CloudFront and the ALB.
Add Amazon VPC endpoints for Amazon Cognito and AWS Secrets Manager to eliminate the NAT Gateway dependency.

The full Terraform modules and Go plugin are available in the GitHub repository.

For more on Cognito-based game authentication patterns, refer to Using Amazon Cognito to Authenticate Players for a Game Backend Service and Web application access control patterns using AWS services.

Share your questions and feedback in the comments.