Trade

2026-08-26 xkcd.com

Post Syndicated from xkcd.com original https://xkcd.com/3290/

"You legs may have a comparative advantage at running, but we arms have a competitive advantage at swinging hammers, so unless you accept that we're the dominant limbs and stop hogging the oxygen, that running advantage won't be around for long."

SambaNova’s SN50 RDU for AI at Hot Chips 2026

2026-08-26 Ryan Smith

Post Syndicated from Ryan Smith original https://www.servethehome.com/sambanovas-sn50-rdu-for-ai-at-hot-chips-2026/

This year’s Hot Chips conference includes a presentation from AI accelerator developer SambaNova, who is at the show to discuss their latest-generation RDU, the SN50

The post SambaNova’s SN50 RDU for AI at Hot Chips 2026 appeared first on ServeTheHome.

Microsoft’s Maia 200 AI Accelerator at Hot Chips 2026

2026-08-26 Ryan Smith

Post Syndicated from Ryan Smith original https://www.servethehome.com/microsofts-maia-200-accelerator-at-hot-chips-2026/

At Hot Chips 2026, Microsoft is going into new detail on Maia 200, their second-generation server AI inference processor

The post Microsoft’s Maia 200 AI Accelerator at Hot Chips 2026 appeared first on ServeTheHome.

Cerebras Talks Going Rack-Scale with Their WSEs at Hot Chips 2026

2026-08-26 Ryan Smith

Post Syndicated from Ryan Smith original https://www.servethehome.com/cerebras-talks-going-rack-scale-with-their-wses-at-hot-chips-2026/

At Hot Chips 2026, wafer scale engine developer Cerebras is talking about the next generation of their giant accelerators, as well as how the company is going rack-scale thanks to their Nexus platform architecture

The post Cerebras Talks Going Rack-Scale with Their WSEs at Hot Chips 2026 appeared first on ServeTheHome.

Fast Track ISM-ready cloud environments and IRAP Assessments with Landing Zone Accelerator on AWS

2026-08-26 Kevin Donohue

Post Syndicated from Kevin Donohue original https://aws.amazon.com/blogs/security/fast-track-ism-ready-cloud-environments-and-irap-assessments-with-landing-zone-accelerator-on-aws/

This post announces the availability of a new independent assessment report available on AWS Artifact analyzing how Landing Zone Accelerator on AWS (LZA) can automatically deploy multi-account environments in Amazon Web Services (AWS) with Australian Government Information Security Manual (ISM) security controls coverage at scale. The report includes findings from an independent third-party analysis conducted by AWS Partner gwi.digital. In addition to the report, we talk about ISM compliance applicability to LZA and a new testing mechanism for measuring configuration drift, which together can provide Australian customers with a documented and validated foundation to accelerate IRAP assessment readiness.

Background

Australian organizations in public sector, defense, and critical infrastructure agencies must build cloud environments that meet (ISM requirements. The ISM defines 1,081 security control requirements across 22 guideline chapters. Demonstrating compliance is central to achieving an IRAP assessment outcome, but assessments typically require months of preparation, evidence gathering, and testing.

In October 2025, we introduced the LZA Universal Configuration and LZA Compliance Workbook. LZA provisions a multi-account security architecture that automates the deployment of nearly 200 security controls based on AWS Well-Architected pillars and AWS security best practices. The LZA Compliance Workbook, available on AWS Artifact, documents how the Universal Configuration (UC) maps to technical security requirements from 17 global compliance frameworks, with more being added. LZA is an ideal solution for customers with security and compliance obligations—both existing and anticipated—because the guardrails it deploys are applied automatically to new accounts as environments grow.

What’s in the report

To see how LZA can help customers in Australia we teamed up with AWS Partner gwi.digital to run LZA as a customer would. gwi.digital is a consultancy partner specializing in cybersecurity and governance, risk, and compliance (GRC) and has deep experience in IRAP assessments and the ISM framework. The team conducted an independent analysis and evaluation of LZA UC against 1,081 ISM controls. The assessment was conducted in a greenfield AWS environment hosted in the ap-southeast-2 (Sydney) AWS Region, positioned within Phase 2 of the ASD Cloud Security Assessment and Authorization Framework and focused on cloud consumers building on already-authorized AWS services. It builds on existing AWS IRAP foundations: Underlying AWS services were most recently independently assessed at the PROTECTED level by CyberCX last year. While this report does not constitute an official IRAP authorization, certification, or accreditation, it provides a professional evaluation of evidence based on what LZA delivers out of the box (and what it does not), so organizations can make informed decisions.

What the assessment found

Of the 1,081 ISM controls, 256 are within the addressable scope of LZA and include the technical infrastructure controls that a solution like LZA can meaningfully address. Of those 256, LZA achieves Full or Partial coverage for 234 (91%). The remaining 825 controls are outside the scope of LZA: physical security, personnel, organizational governance, and classification-level exclusions. A key contribution for the assessment was analysis of the shared responsibility model that goes beyond the traditional AWS/customer binary. It considers a three-tier view—AWS (provided), LZA (enabled), and Customer (responsibility)—and further categorizes the 825 out-of-scope controls into subcategories so customers can quickly determine which controls require their attention compared to which are already addressed at the infrastructure level. By combining automated deployment, ISM-specific compliance mappings, independent validation, and continuous evidence generation, Australian customers can reduce IRAP assessment timelines while achieving more reliable compliance outcomes.

Note: Results are based on the configuration as-provisioned based on the LZA version and during the time of this assessment. Results may vary depending on customer implementation and configuration choices.

Continuous validation with CATS

Confirming controls are implemented and operating effectively across your environment is complex and often lacks complete coverage. To simplify testing your LZA deployment, we developed the Controls Acceptance Testing Suite (CATS). CATS is an automated compliance validation engine that runs purpose-built tests against the security configuration baseline deployed by LZA UC. For the ISM assessment, CATS executed over 3,600 individual tests across six AWS accounts, evaluating account structure, identity and access, network configuration, logging, encryption, and backup. For Australian customers, CATS enables:

Automated evidence generation: Machine-readable results may replace weeks of manual evidence collection
ISM-enriched reporting: gwi-digital developed a conversion script that maps CATS output to ISM control references, enabling auditors to interpret results from an ISM perspective
OSCAL export: Results in Open Security Controls Assessment Language (OSCAL) format for interoperability with assessment tools
Continuous drift detection: Repeatable execution identifies when configurations deviate from the validated baseline between assessment cycles

Note: CATS availability is limited and currently accessible only through AWS Professional Services as a private beta solution and may be subject to change. Availability, features, and pricing are subject to change without notice. Contact your AWS account representative or reach out to a member of the LZA team for questions about CATS, or to provide feedback on the LZA ISM report.

ISM-optimized configuration

Through the assessment, gwi-digital identified configuration adjustments that elevate specific control ratings from Partial to Full, with minimal effort; for example, increasing the default password length from 14 to 15 characters or adjusting log retention to align with Australian Federal Disposal Authority standards. These recommendations have been communicated to the LZA UC team for inclusion in a future ISM-specific guidance section in LZA GitHub documentation.

Getting started

In addition to the LZA ISM report, you can also find the LZA Compliance Workbook available on AWS Artifact. It maps related ISM requirement identifiers to security implementation statements, giving you a starting point from which you can customize and enhance your compliance documentation for your unique use cases after deploying LZA.

Sign in to your AWS account and then download the LZA Australia ISM Third-Party Analysis and Evaluation Report and LZA Compliance Workbook from AWS Artifact.

Figure 1: LZA report and workbook in AWS Artifact
Visit the LZA Universal Configuration GitHub repository to review and download the latest configuration baseline.
Use the LZA Implementation Guide to see use cases, review pre-deployment considerations and then follow deployment steps.
Set-up a security compliance chat agent. Consider creating a knowledge base with Amazon Bedrock using the LZA Compliance Workbook, LZA Implementation Guide, web crawlers to the LZA GitHub, and your custom resources to set up your own chat agent.

If you have questions, contact a gwi.digitalexpert, AWS LZA team member, or your AWS account representative.

Conclusion

The combination of ISM mappings in the LZA Compliance Workbook, the gwi-digital analysis report, and CATS evidence gives Australian customers a head start on IRAP assessments.

Weeks saved on scoping: Pre-determined control applicability significantly reduces upfront scoping effort
Documentation ready: Implementation statements and control requirement alignment reduce the documentation burden
Independent assurance: A third-party report that customers and their assessors can reference directly
Continuous evidence: CATS generates repeatable security evidence between assessment cycles, replacing periodic manual audits with ongoing assurance

Acknowledgements

The Landing Zone Accelerator team would like to thank Baden Hughes, Henrik Melkonyan, Iain Lindsay-German, and Ian Roderick from gwi.digital for their professionalism and expertise in performing an incredibly thorough assessment, helping us to make LZA better for customers, and for their contributions to this blog post.

If you have feedback about this post, submit comments in the Comments section below.

NVIDIA’s Groq 3 LPU Accelerators for Heterogeneous AI Compute at Hot Chips 2026

2026-08-26 Ryan Smith

Post Syndicated from Ryan Smith original https://www.servethehome.com/nvidias-groq-3-lpu-accelerators-for-heterogeneous-ai-compute-at-hot-chips-2026/

The newest member of NVIDIA’s AI hardware family, at Hot Chips 2026 NVIDIA is diving into the use of LPUs as part of Vera Rubin clusters. The specialized chips from acquihire Groq are being tapped to offer significantly lower latency in the decode phase of inference

The post NVIDIA’s Groq 3 LPU Accelerators for Heterogeneous AI Compute at Hot Chips 2026 appeared first on ServeTheHome.

Meta’s MTIA Custom AI Silicon at Hot Chips 2026

2026-08-26 Ryan Smith

Post Syndicated from Ryan Smith original https://www.servethehome.com/metas-mtia-custom-ai-silicon-at-hot-chips-2026/

For Hot Chips 2026, Meta is at the show to discuss their AI inference accelerators. The MTIA family, the company has an ambitious roadmap to release 4 accelerators over the next couple of years

The post Meta’s MTIA Custom AI Silicon at Hot Chips 2026 appeared first on ServeTheHome.

mklinux-v7.0-mk2 released

2026-08-25 corbet

Post Syndicated from corbet original https://lwn.net/Articles/1090582/

For people who would like to experiment with the multi-kernel Linux concept, Cong Wang has
announced the
release of mklinux v7.0-mk2.

mklinux lets one machine run several independent Linux kernels at
the same time on bare metal, without a hypervisor. A host kernel
owns a pool of CPUs, memory and PCI devices, carves that pool into
instances, and boots a spawn kernel into each instance through
kexec_file_load(). Every spawn kernel runs natively on its own
CPUs, its own physical memory and its own devices. Nothing is
emulated and nothing is trapped; the only thing shared is what you
choose to share.

Note that this is not the old MkLinux, which was a port to PowerPC
Macintosh systems.

This 5G Adapter Beats UniFi’s 5G Max (Here’s Why)

2026-08-25 Crosstalk Solutions

Post Syndicated from Crosstalk Solutions original https://www.youtube.com/shorts/UFfNtMPb5YM

PythonOperator and BashOperator Now Available on Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless

2026-08-25 Pradeep Kumar Nalluri

Post Syndicated from Pradeep Kumar Nalluri original https://aws.amazon.com/blogs/big-data/pythonoperator-and-bashoperator-now-available-on-amazon-managed-workflows-for-apache-airflow-amazon-mwaa-serverless/

If you run Apache Airflow workflows on Amazon MWAA Serverless, you can now use PythonOperator and BashOperator to run custom code directly in the serverless runtime. Previously, Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless only supported orchestration of AWS services through operators for scheduling tasks, managing dependencies, and handling retries. It did not support running your own Python functions or shell scripts natively. If you needed custom Python logic or shell commands, you had to wrap code in AWS Lambda functions, start Amazon Elastic Container Service (Amazon ECS) tasks, or use other AWS compute services. These alternatives add complexity, cost, and latency to your orchestration pipelines.

With this launch, you can run custom Python functions and shell scripts directly within the serverless task runtime, without requiring additional infrastructure. This means you can now use PythonOperator and BashOperator many data engineering teams rely on for ETL pipelines and data quality checks – without provisioning additional compute.

In this post, we walk through how this feature works and demonstrate a practical example: building a serverless pipeline that converts CSV files to JSON format using a PythonOperator, and verifies the output using a BashOperator. By the end, you will know how to:

Package a Python module with dependencies and upload it to an Amazon Simple Storage Service (Amazon S3) bucket as a code bundle
Define a multi-task workflow using the dag-factory compatible YAML
Create and run a workflow with the AWS Command Line Interface (AWS CLI)
Verify that your pipeline produced the expected output

How it works

With MWAA Serverless, you can package your custom code, upload it to an Amazon S3 bucket, and reference it when creating a workflow. The service snapshots your code at workflow creation time and uses that snapshot for all subsequent runs of the same workflow version.

Code bundles

A code bundle is the package that contains your custom logic. You package your Python modules or shell scripts and upload them to an Amazon S3 bucket. A code bundle can be:

A single .py file or .sh bash script (uploaded to an Amazon S3 bucket)
A ZIP archive containing multiple shell scripts, Python modules and dependencies (up to 250 MB)

Execution model

When you create or update a workflow, MWAA Serverless snapshots your code bundle from an Amazon S3 bucket provided and stores it on the service side. At task execution time, the service uses this snapshot – not the object currently residing in your Amazon S3 bucket – to run your code in an isolated runtime environment.

Python and Bash tasks do not have internet access. They can reach only Amazon S3, Amazon Elastic Container Registry (Amazon ECR), and Amazon CloudWatch, which are the services the runtime requires to operate. To have internet access, configure the workflow with Amazon VPC so that it can go through the provided VPC.

Supported operators

The following table describes the two operators now available in MWAA Serverless.

Operator	Description
PythonOperator	Executes a Python callable (function) from your code bundle
BashOperator	Runs shell commands or scripts

Security

AWS Key Management Service (AWS KMS) encrypts your code bundles at rest. IAM policies control who can create, update, and trigger the workflows. The execution role scopes what AWS resources your code can access at runtime.

Prerequisites

Before getting started, verify that you have the following resources and tools configured in your AWS account:

An AWS account with access to Amazon MWAA Serverless
AWS CLI v2 (latest version) installed and configured. To install or update, see Installing or updating to the latest version of the AWS CLI.
An Amazon S3 bucket for storing DAG definitions and code bundles
An IAM role that MWAA Serverless can assume (see the execution role setup below)

Walkthrough: Building a serverless CSV-to-JSON pipeline

In this walkthrough, we build a pipeline that converts CSV files to JSON format – a common data transformation for downstream APIs and analytics systems that consume JSON. The pipeline uses a PythonOperator for the conversion logic and a BashOperator to verify the output. Here is what the pipeline does:

Reads a CSV file from an Amazon S3 bucket
Converts it to JSON format with column type inference
Writes the JSON file back to an Amazon S3 bucket
Validates record counts match between source and output

Step 1: Create the execution role

Create an IAM role that your workflow assumes at runtime. The trust policy must allow the airflow-serverless.amazonaws.com service to assume the role:

cat > trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "airflow-serverless.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Create the role and attach an inline policy granting least-privilege access to your S3 bucket:

aws iam create-role \
  --role-name MWAAServerlessExecutionRole \
  --assume-role-policy-document file://trust-policy.json

aws iam put-role-policy \
  --role-name MWAAServerlessExecutionRole \
  --policy-name MWAAServerlessAccessPolicy \
  --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::amzn-s3-demo-mwaa-data",
        "arn:aws:s3:::amzn-s3-demo-mwaa-data/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/mwaa-serverless/*"
    }
  ]
}'

Step 2: Write the Python module

Create a file called csv_to_json.py with the conversion logic:

# csv_to_json.py
import csv
import json
import boto3
import io

def convert(**kwargs):
    """Read a CSV from S3 and write it back as JSON lines."""
    bucket = "amzn-s3-demo-mwaa-data"
    source_key = "raw/sales_data.csv"
    output_key = "processed/sales_data.json"

    s3 = boto3.client("s3")

    # Read source file
    response = s3.get_object(Bucket=bucket, Key=source_key)
    content = response["Body"].read().decode("utf-8")

    # Parse CSV
    reader = csv.DictReader(io.StringIO(content))
    rows = list(reader)

    # Type inference - convert numeric fields
    for row in rows:
        for key, value in row.items():
            try:
                row[key] = float(value)
            except (ValueError, TypeError):
                pass

    # Write as JSON lines
    output = "\n".join(json.dumps(row) for row in rows) + "\n"
    s3.put_object(Bucket=bucket, Key=output_key, Body=output.encode("utf-8"))

    print(f"Converted {len(rows)} rows to JSON lines")
    print(f"Output: s3://amzn-s3-demo-mwaa-data/{output_key}")
    return {"rows": len(rows), "output_key": output_key}

This function uses boto3 (which comes pre-installed with the MWAA Serverless execution environment) and Python’s built-in csv and json modules. The conversion reads the CSV, infers numeric types, and writes a JSON lines file back to the S3 bucket.

Step 3: Write the verification script

Create a file called verify_output.sh. This script validates the pipeline output by comparing the record count in the source CSV against the output JSON file. If the counts do not match, the task fails with a non-zero exit code, which causes the workflow run to fail.

#!/bin/bash
echo "=== Data Validation ==="

# Count source records (skip CSV header)
SOURCE_COUNT=$(python3 -m awscli s3 cp s3://amzn-s3-demo-mwaa-data/raw/sales_data.csv - | tail -n +2 | wc -l)
echo "Source CSV records: $SOURCE_COUNT"

# Count output records
OUTPUT_COUNT=$(python3 -m awscli s3 cp s3://amzn-s3-demo-mwaa-data/processed/sales_data.json - | wc -l)
echo "Output JSON records: $OUTPUT_COUNT"

# Validate counts match
if [ "$SOURCE_COUNT" -ne "$OUTPUT_COUNT" ]; then
    echo "FAILED: Record count mismatch (source=$SOURCE_COUNT, output=$OUTPUT_COUNT)"
    exit 1
fi

echo "PASSED: Record counts match ($OUTPUT_COUNT records)"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

This script runs the AWS CLI, which is bundled as a dependency in the code package. The s3 cp streams the file content to stdout without writing to disk, allowing standard shell tools like wc -l and tail to process it. The execution role credentials are automatically available in the execution environment, so the CLI can access S3 without additional configuration.

Step 4: Package and upload the code to Amazon S3

Since the verification script uses the AWS CLI, bundle it as a dependency in the ZIP archive along with your Python module and shell script:

BUCKET="amzn-s3-demo-mwaa-data"
REGION="us-east-1"

# Install awscli into a package directory
pip install awscli \
  --target my_package/ \
  --platform manylinux2014_x86_64 \
  --python-version 3.12 \
  --only-binary=:all:

# Add your module
cp csv_to_json.py my_package/
cp verify_output.sh my_package/

# Create the ZIP archive
cd my_package && zip -r ../code_bundle.zip . && cd ..
# Upload to S3
aws s3 cp code_bundle.zip s3://$BUCKET/code/code_bundle.zip --region $REGION

Upload a sample CSV file for testing:

cat > sales_data.csv << 'EOF'
date,region,product,units,revenue
2026-07-01,us-east,widget-a,150,4500.00
2026-07-01,eu-west,widget-b,89,2670.00
2026-07-02,us-east,widget-a,203,6090.00
2026-07-02,ap-south,widget-c,67,1340.00
2026-07-03,us-east,widget-b,178,5340.00
EOF

aws s3 cp sales_data.csv s3://$BUCKET/raw/sales_data.csv --region $REGION

Step 5: Define the DAG (YAML)

MWAA Serverless uses a declarative YAML format for DAG definitions. Create a file called conversion_dag.yaml:

csv_to_json_pipeline:
  start_date: "2026-01-01"
  schedule: null
  tasks:
    convert_to_json:
      operator: airflow.operators.python.PythonOperator
      python_callable: csv_to_json.convert
    verify_output:
      operator: airflow.operators.bash.BashOperator
      bash_command: "verify_output.sh"
      dependencies:
        - convert_to_json

This DAG defines two tasks:

convert_to_json – Runs the convert function from the Python module to transform CSV to JSON lines.
verify_output – Runs a shell script that validates the pipeline output by comparing source and output record counts, failing the task if they do not match.

Upload the DAG definition to S3. Note: You can also run inline Bash commands directly without a shell script.

aws s3 cp conversion_dag.yaml s3://$BUCKET/dags/conversion_dag.yaml --region $REGION

Step 6: Create the workflow

Create the MWAA Serverless workflow, referencing the DAG definition and the code bundle:

ROLE_ARN="arn:aws:iam::<your-account-id>:role/MWAAServerlessExecutionRole"

aws mwaa-serverless create-workflow \
  --name csv-to-json-workflow \
  --definition-s3-location Bucket="$BUCKET",ObjectKey="dags/conversion_dag.yaml" \
  --code '{"S3Location": {"Bucket":"'"$BUCKET"'","ObjectKey":"code/code_bundle.zip"}}' \
  --role-arn $ROLE_ARN \
  --region $REGION

The response includes a WorkflowArn that you use to trigger runs:

{
  "WorkflowArn": "arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123",
  "CreatedAt": "2026-07-15T10:30:00.000000+00:00",
  "WorkflowVersion": "a1b2c3d4e5f6"
}

Step 7: Run the workflow

Trigger a workflow run:

WORKFLOW_ARN="arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123"

aws mwaa-serverless start-workflow-run \
  --workflow-arn $WORKFLOW_ARN \
  --region $REGION

The response confirms the run has started:

{
  "RunId": "6OZV9ABF9enHKXk",
  "Status": "STARTING"
}

Step 8: Monitor execution

Check the status of your run:

RUN_ID="6OZV9ABF9enHKXk"

aws mwaa-serverless get-workflow-run \
  --workflow-arn $WORKFLOW_ARN \
  --run-id $RUN_ID \
  --region $REGION

A successful run returns:

{
  "RunDetail": {
    "Duration": 45,
    "RunState": "SUCCESS",
    "TaskInstances": ["ex_abc123_convert_to_json_1", "ex_abc123_verify_output_1"]
  },
  "RunId": "6OZV9ABF9enHKXk",
  "RunType": "ON_DEMAND",
  "WorkflowArn": "arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123",
  "WorkflowVersion": "a1b2c3d4e5f6"
}

Step 9: Verify the output

Confirm the JSON file was written to the S3 bucket:

# List the output file
aws s3 ls s3://$BUCKET/processed/sales_data.json --region $REGION

You should see the JSON file:

2026-07-15 10:32:45 1847 sales_data.json

You can also verify task-level output in Amazon CloudWatch Logs. Open the log group for your workflow and find the convert_to_json task log stream:

Converted 5 rows to JSON lines
Output: s3://amzn-s3-demo-mwaa-data/processed/sales_data.json

Considerations and limits

When planning your workloads on MWAA Serverless with these operators, keep the following considerations in mind:

Code bundle size – ZIP archives must be under 250 MB per bundle.
Network access – Python and Bash tasks do not have internet access. They can reach a limited set of AWS services required for the runtime to function (Amazon S3, Amazon ECR, and Amazon CloudWatch) but cannot call other AWS services or external endpoints. If your workflow requires calls to external APIs, preprocess that data and store it in an Amazon S3 bucket before invoking the workflow.
Runtime dependencies – boto3 and the Python standard library are pre-installed. For additional packages (such as pandas or requests), bundle them in your ZIP archive following the Amazon MWAA Serverless packaging guidelines.
Execution timeout – Tasks are subject to the workflow’s configured timeout limits.
Python version – Check the Amazon MWAA Serverless documentation for the currently supported Python runtime version.
DAG format – MWAA Serverless uses YAML-based DAG definitions, not traditional Python DAG files. If you are migrating from MWAA Provisioned, you will need to convert your DAGs to the YAML format.
Operators not supported – Some Airflow community operators and custom plugins are not available in the Serverless runtime. Refer to the documentation for the full compatibility list.

Clean up

To avoid ongoing charges, delete the resources you created in this walkthrough. The following commands remove the workflow, S3 objects, and IAM role:

Note: $WORKFLOW_ARN is defined in Step 7.

# Delete the workflow
aws mwaa-serverless delete-workflow \
  --workflow-arn $WORKFLOW_ARN \
  --region $REGION

Note: $BUCKET is exported in Step 4. If appropriate, delete the bucket as well.

# Remove S3 objects
aws s3 rm s3://$BUCKET/code/code_bundle.zip
aws s3 rm s3://$BUCKET/dags/conversion_dag.yaml
aws s3 rm s3://$BUCKET/raw/sales_data.csv
aws s3 rm s3://$BUCKET/processed/sales_data.json

# Delete the IAM role
aws iam delete-role-policy \
  --role-name MWAAServerlessExecutionRole \
  --policy-name MWAAServerlessAccessPolicy

aws iam delete-role --role-name MWAAServerlessExecutionRole

Conclusion

With native support for PythonOperator and BashOperator, you can now run the custom code execution patterns that many data engineering teams rely on daily directly in MWAA Serverless. Run data transformations, format conversions, validations, and shell scripts in the serverless runtime – without provisioning additional compute or managing containers.

If you are running Airflow workloads on MWAA Provisioned or self-managed infrastructure, your existing PythonOperator and BashOperator logic requires minimal changes. Convert your Python DAG files to the YAML format, package your code as a bundle, and you are ready to run on MWAA Serverless.

To get started, visit the Amazon MWAA Serverless documentation and try the walkthrough earlier in this post with your own data. For pricing details, visit the Amazon MWAA pricing page. We look forward to your feedback.

About the authors

NVIDIA Spectrum-X Ethernet Multiplane Network Architecture at Hot Chips 2026

2026-08-25 Patrick Kennedy

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/nvidia-spectrum-x-ethernet-multiplane-network-architecture-at-hot-chips-2026/

At Hot Chips 2026, NVIDIA showed how Spectrum-X and a multiplane network is its vision for AI Factory networks

The post NVIDIA Spectrum-X Ethernet Multiplane Network Architecture at Hot Chips 2026 appeared first on ServeTheHome.

NVIDIA BlueField-4 DPU at Hot Chips 2026

2026-08-25 Patrick Kennedy

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/nvidia-bluefield-4-processor-at-hot-chips-2026/

At Hot Chips 2026, NVIDIA is detailing the huge upgrades in its BlueField-4 DPU along with the reasons behind it and its scale-in networking

The post NVIDIA BlueField-4 DPU at Hot Chips 2026 appeared first on ServeTheHome.

Broadcom Thor Ultra Ethernet NIC at Hot Chips 2026

2026-08-25 Patrick Kennedy

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/broadcom-thor-ultra-ethernet-nic-at-hot-chips-2026/

At Hot Chips 2026, we got a lot of detail on how the Broadcom Thor Ultra 800GbE NIC works and some of the performance views

The post Broadcom Thor Ultra Ethernet NIC at Hot Chips 2026 appeared first on ServeTheHome.

Setting up an RCS agent with an AI coding assistant and AWS End User Messaging

2026-08-25 Bruno Giorgini

Post Syndicated from Bruno Giorgini original https://aws.amazon.com/blogs/messaging-and-targeting/setting-up-an-rcs-agent-with-an-ai-coding-assistant-and-aws-end-user-messaging/

Clone a repo, open it in your AI coding assistant, type “go,” and walk away with a working RCS agent.

Creating an RCS agent on AWS End User Messaging normally means juggling 23 registration fields, three different CLI parameter types, brand asset requirements, and a multi-step approval process. An AI coding assistant can handle all of that for you. With AWS End User Messaging, you can create RCS agents that send and receive rich messages complete with your brand’s logo, colors, and verified identity.

Setting up an RCS agent involves creating an agent container, uploading brand assets, configuring a 23-field registration, submitting for approval, adding verified testers, and testing both outbound and inbound messaging. Each field has a specific type (TEXT, SELECT, or ATTACHMENT) that requires a different CLI parameter, and getting any of them wrong means starting over.

We built an open-source sample repository that encodes all of this knowledge into an AGENTS.md file. When you open the repo in an AI coding assistant like Kiro, Cursor, or Windsurf, the assistant reads the instructions and walks you through the entire setup interactively. You provide a brand name and your phone number. The AI handles everything else.

How it works

The repository aws-samples/sample-rcs-agent-setup-and-send-messages contains:

AGENTS.md — A structured instruction file that AI coding assistants read automatically. It contains the complete RCS agent setup workflow: credential checks, brand asset generation, registration field configuration, tester management, and message testing.
brand-assets/ — Template SVG files for the agent logo (224×224 px) and banner (1440×448 px), ready to be customized and converted to PNG.
.kiro/steering/rcs-agent-setup.md — A Kiro-specific steering file with the same instructions, using the inclusion: always frontmatter so Kiro loads it automatically.

The AGENTS.md file is the key. It defines six skills that the AI assistant executes in sequence:

Create RCS agent — Creates the agent container, generates brand assets (logo and banner SVGs), converts them to PNG, creates a test registration, sets all 23 fields with the correct parameter types, and submits for approval.
Add verified testers — Registers test phone numbers and guides you through accepting the tester invitation.
Send a test message — Checks for blockers (protect configuration, opt-out lists) and sends your first branded RCS message.
Set up inbound keyword — Configures an automatic response keyword so you can test inbound messaging without writing backend code.
Verify inbound messaging — Walks you through the console deep link flow to confirm two-way messaging works.
Delete an RCS agent — Removes an agent cleanly by disabling deletion protection, deleting the associated registration, then deleting the agent itself.

Prerequisites

Before you start, you need:

An AWS account with access to AWS End User Messaging.
AWS Command Line Interface (AWS CLI) v2.35.12 or later installed and configured with credentials that have pinpoint-sms-voice-v2:* permissions.
An AI coding assistant that reads AGENTS.md files (Kiro, Cursor, Windsurf, or similar).
librsvg for SVG to PNG conversion (brew install librsvg on macOS).
A test phone that supports RCS messaging.

Getting started

Follow these steps to go from zero to a working RCS agent. The entire process takes about five minutes.

Step 1: Clone the repository

git clone https://github.com/aws-samples/sample-rcs-agent-setup-and-send-messages.git
cd sample-rcs-agent-setup-and-send-messages

Step 2: Open in your AI coding assistant

Open the cloned directory in your preferred AI coding assistant. The assistant will automatically detect the AGENTS.md file (or .kiro/steering/rcs-agent-setup.md if you are using Kiro).

Step 3: Type “go”

In the chat panel, type go. The AI assistant will:

Check your AWS credentials — It runs aws sts get-caller-identity and asks how you authenticate if credentials are not configured. It supports named profiles, SSO, IAM user credentials, and environment variables.
Verify EUM access — It confirms your account can use AWS End User Messaging.
Check tooling — It verifies rsvg-convert is installed for brand asset generation.
Ask for your preference — Quick mode (provide a brand name) or interactive mode (you specify every detail).

Step 4: Provide a brand name

In quick mode, you provide a brand name and the AI generates everything else: a description, an accessible accent color, contact information with placeholder values, privacy and terms URLs, and custom SVG brand assets with your brand name and colors.

In interactive mode, the AI asks for each detail one section at a time: brand name, accent color, logo description, banner description, contact information, and policy URLs.

Step 5: Watch it work

The AI assistant executes every AWS CLI command in sequence:

Creates the RCS agent container.
Enables deletion protection.
Creates a test registration and links it to the agent.
Generates and converts brand asset SVGs to PNG.
Uploads the logo and banner as registration attachments.
Sets all 23 registration fields using the correct parameter type for each (TEXT, SELECT, or ATTACHMENT).
Submits the registration and polls for approval.
Reports when the agent is active.

Step 6: Add a tester and send a message

Once the agent is approved, the AI asks for your test phone number, registers it as a verified tester, and waits for you to accept the invitation. After verification, it checks for blockers (protect configuration and opt-out lists), then sends your first branded RCS message.

Step 7: Test inbound messaging

The AI configures an automatic keyword response and walks you through the console deep link flow to verify two-way messaging. When you send RCSINBOUNDTESTING to your agent, you receive an automatic reply confirming inbound messaging works.

What the AI handles for you

The AGENTS.md file encodes several non-obvious behaviors that would otherwise require trial and error:

Challenge	How the repo handles it
`create-rcs-agent` takes no `--display-name` parameter	The brand name comes from the registration, not the agent creation call. The instructions reflect this.
Three different field parameter types	The instructions include a field reference table mapping each of the 23 fields to its correct CLI parameter: `--text-value`, `--select-choices`, or `--registration-attachment-id`.
`--field-values` does not exist	The instructions explicitly warn against this non-existent parameter and use the correct alternatives.
`--attachment-body` and `--attachment-url` conflict	The instructions use `--attachment-body` only.
Accent color contrast requirements	The instructions include pre-validated color choices with 4.5:1 contrast ratio against white.
Field paths differ from what you might expect	The correct paths are `agentDetails.logoImage` and `agentDetails.bannerImage`, not `logoAttachmentId` or `bannerAttachmentId`.
New registration versions do not inherit field values	The troubleshooting section warns that all 23 fields must be re-populated when creating a new version.

Customizing the repo

You can modify the AGENTS.md file to fit your workflow:

Change default values — Update placeholder contact information, privacy URLs, or terms URLs to match your organization.
Add custom brand assets — Replace the template SVGs in brand-assets/ with your own designs. Keep the logo at 224×224 px and the banner at 1440×448 px.
Extend the skills — Add new skills for richer message types (cards, carousels), event destinations for programmatic inbound handling, or integration with other AWS services.

Cleanup

To remove the resources created during testing:

# 1. Disable deletion protection
aws pinpoint-sms-voice-v2 update-rcs-agent \
  --rcs-agent-id <your-agent-id> \
  --no-deletion-protection-enabled \
  --region us-east-1

# 2. Delete the associated registration (required before deleting the agent)
aws pinpoint-sms-voice-v2 delete-registration \
  --registration-id <your-registration-id> \
  --region us-east-1

# 3. Delete the agent
aws pinpoint-sms-voice-v2 delete-rcs-agent \
  --rcs-agent-id <your-agent-id> \
  --region us-east-1

Note: You must delete the registration before the agent. Skipping this step results in a ConflictException: RESOURCE_NOT_EMPTY error.

Conclusion

The aws-samples/sample-rcs-agent-setup-and-send-messages repository turns a multi-step, error-prone CLI workflow into a guided conversation. Clone the repo, open it in your AI coding assistant, type “go,” and you have a working RCS agent that can send and receive branded messages to verified testers.

The AGENTS.md pattern is reusable. Any complex AWS workflow with non-obvious API behavior can be encoded the same way: document the correct commands, parameter types, and pitfalls in a structured file, and let the AI assistant execute it interactively.

For a detailed manual walkthrough of the same process, see Creating and testing an RCS agent with AWS End User Messaging. For an overview of the business case for RCS, see Upgrade business messaging with RCS on AWS. For more information, see the AWS End User Messaging service page and the RCS documentation.

About the author

XCENA MX1 CXL Computational Memory Device at Hot Chips 2026 with Samsung

2026-08-25 Patrick Kennedy

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/xcena-mx1-cxl-computational-memory-device-at-hot-chips-2026/

At Hot Chips 2026, XCENA showed off how the MX1 combines a CXL memory controller with 3072 RISC-V cores and tiering to SSDs, and Samsung showed how it scales

The post XCENA MX1 CXL Computational Memory Device at Hot Chips 2026 with Samsung appeared first on ServeTheHome.

Samsung LPDDR5X-PIM at Hot Chips 2026

2026-08-25 Patrick Kennedy

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/samsung-lpddr5x-pim-at-hot-chips-2026/

Samsung has a new generation of its LPDDR5X-PIM solution at Hot Chips 2026. This processing in memory solution looks more polished this generation

The post Samsung LPDDR5X-PIM at Hot Chips 2026 appeared first on ServeTheHome.

Enable cross-cloud analytics with Amazon S3 Tables and Google BigQuery, Part 1: IAM-based access control

2026-08-25 Lakshmi Nair

Post Syndicated from Lakshmi Nair original https://aws.amazon.com/blogs/big-data/enable-cross-cloud-analytics-with-amazon-s3-tables-and-google-bigquery-part-1-iam-based-access-control/

Organizations running analytics workloads across multiple clouds often hit the same friction: the data lives on one cloud, but the engine querying it lives on another. Copying data across the boundary creates a second dataset that must be kept in sync, adding cost, latency, and reconciliation overhead. In this post, we address a specific instance of that pattern: your Google BigQuery users need to work with data that lives in Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), on AWS. The ideal outcome is a single, governed dataset that serves teams in both clouds without a standing replication pipeline between them.

With Amazon S3 Tables, you get managed Apache Iceberg tables with built-in compaction, snapshot management, and an integration with the AWS Glue Data Catalog. Because S3 Tables stores data in the open Iceberg format, supported external engines can read it directly if the right access path exists.

This two-part blog series demonstrates how you can connect Google BigQuery to Amazon S3 Tables using the cross-cloud lakehouse with AWS Glue. We cover two access control approaches:

AWS Identity and Access Management (IAM): You can define a single policy that uses IAM permissions to set up access to both table metadata and data.
AWS Lake Formation: You can use temporary vended credentials for data access, with metadata access managed by Lake Formation permissions.

This post focuses on the IAM-based approach. Part 2 covers the Lake Formation approach for organizations that need credential-vended access across multiple engines.

By the end, you will have BigQuery querying Iceberg tables stored on S3 Tables without data copy or duplication, providing live access to Iceberg data.

Cross-cloud analytics scenarios

There are several scenarios where organizations benefit from cross-cloud querying capabilities. Here are some of the common patterns this architecture addresses:

Schema evolution across cloud boundaries

When source schemas change frequently, streaming pipelines writing to BigQuery-managed store require coordinated DDL changes on the BigQuery table and downstream views. Teams often work around this challenge by storing payloads as untyped columns and parsing them later.

With Iceberg on S3 Tables, schema evolution is tracked in table metadata. When the writing engine adds a new column, BigQuery’s Lakehouse refresh picks up the updated schema automatically on the next sync cycle.

Multi-cloud analytics without data duplication

A company has its production data environment on AWS (data lakes, warehouses, streaming) but acquired a business unit that runs analytics exclusively on BigQuery. In-place querying from BigQuery keeps your data in Amazon S3 Tables, so you pay for one copy, work from live data, and avoid the operational overhead of a synchronized second store.

Cost optimization for infrequently queried datasets

An organization has hundreds of datasets on AWS, but only a fraction is queried daily from BigQuery. Replicating all of them to Google Cloud Storage drives unnecessary storage and transfer costs. With Lakehouse catalog federation, you keep your data on S3 Tables. BigQuery reads data only when queried, so you pay per query rather than per-copy storage.

Decoupled compute across engines

Data team wants storage on AWS with the flexibility for multiple engines to read the same data: BigQuery and Amazon Redshift for data warehousing use cases, Amazon Athena for interactive ad-hoc querying, Amazon SageMaker AI for machine learning (ML). With Apache Iceberg’s open format, you can use one storage layer, many compute engines, no data copies between them.

Solution overview

You use the AWS Glue Iceberg REST Catalog (IRC) as the bridge between BigQuery and S3 Tables. BigQuery’s cross-cloud Lakehouse creates a federated catalog that syncs metadata from the Glue IRC, then uses the synced metadata to read Iceberg data files directly.

Figure 1: Architecture diagram showing BigQuery connecting to Amazon S3 Tables through the AWS Glue Iceberg REST Catalog

The key components in this architecture:

Amazon S3 Tables: With Amazon S3 Tables, you get a fully managed Apache Iceberg table experience in Amazon S3, optimized for analytics workloads. You can register table metadata in the AWS Glue Data Catalog for discovery and governance.
AWS Glue Data Catalog: With AWS Glue Data Catalog, you can access the federated s3tablescatalog catalog that maps S3 Tables resources (table buckets, namespaces, tables) into a catalog hierarchy from supported analytics engines. The standard Iceberg REST endpoint of Glue Data Catalog serves table metadata to external engines. BigQuery connects through this endpoint.
Google Cross-Cloud Lakehouse: With Google Cross-Cloud Lakehouse, you can connect BigQuery to external Iceberg catalogs. It assumes an AWS IAM role using OpenID Connect (OIDC), calls the Glue Iceberg REST endpoint, and syncs metadata on a configurable refresh interval.

Prerequisites

Before you begin, you need:

An AWS account with Amazon S3 Tables available in your AWS Region.
A Google Cloud project with billing enabled and the BigLake API activated.
AWS Command Line Interface (AWS CLI) and gcloud CLI installed and configured.
An S3 table bucket with at least one namespace and table containing data.

Setting up Amazon S3 Tables

If you already have S3 Tables with data, skip to the next section. Otherwise, create a table bucket, namespace, and populate a table.

Create a table bucket and namespace

Use the AWS CLI to create resources as follows:

# Create a Table bucket
aws s3tables create-table-bucket \
    --name <TABLE_BUCKET_NAME> \
    --region <REGION>

# Create a Namespace (Database)
aws s3tables create-namespace \
    --table-bucket-arn "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET_NAME>" \
    --namespace <NAMESPACE> \
    --region <REGION>

Integrating S3 Tables with the Glue Data Catalog

For BigQuery to access S3 Tables, the tables must be discoverable through the Glue Data Catalog. S3 Tables integrates with Glue through a federated catalog called s3tablescatalog.

Set up S3 Tables integration with the Glue Data Catalog using IAM mode

Open the Amazon S3 console:

In the navigation pane, choose Table buckets.
Choose Enable integration, and then choose Enable integration again to confirm.

This creates the s3tablescatalog federated catalog in Glue, where access is controlled entirely by IAM policies on the calling role. This is a one-time setup per account and Region. After you enable it, the analytics integration applies to all table buckets in your account.

The Enable integration option on the table buckets page of the Amazon S3 console

Figure 2: Enabling the S3 Tables integration in the Amazon S3 console

Alternatively, create the catalog using the AWS CLI:

aws glue create-catalog --region <REGION> --cli-input-json '{
  "Name": "s3tablescatalog",
  "CatalogInput": {
    "FederatedCatalog": {
      "Identifier": "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/*",
      "ConnectionName": "aws:s3tables"
    },
    "CreateDatabaseDefaultPermissions": [
      { "Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"] }
    ],
    "CreateTableDefaultPermissions": [
      { "Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"] }
    ]
  }
}'

Create a table and insert data

Now, to create the table and insert data, open the Amazon Athena console. In the query editor, select s3tablescatalog/<TABLE_BUCKET_NAME> as your data source and <NAMESPACE> as the database. Then run the following SQL statements one by one:

CREATE TABLE `<NAMESPACE>`.orders (
    order_id STRING,
    customer_id STRING,
    amount BIGINT,
    order_date DATE,
    region STRING
)
TBLPROPERTIES ('table_type' = 'iceberg');

INSERT INTO orders
VALUES
    ('ORD-001', 'C100', 4500, DATE '2024-06-01', 'EMEA'),
    ('ORD-002', 'C200', 8900, DATE '2024-06-01', 'EMEA'),
    ('ORD-003', 'C100', 3200, DATE '2024-06-02', 'NAMER'),
    ('ORD-004', 'C300', 12000, DATE '2024-06-02', 'NAMER'),
    ('ORD-005', 'C400', 6700, DATE '2024-06-03', 'APJ'),
    ('ORD-006', 'C200', 4100, DATE '2024-06-03', 'APJ'),
    ('ORD-007', 'C500', 9500, DATE '2024-06-04', 'EMEA'),
    ('ORD-008', 'C100', 2800, DATE '2024-06-04', 'LATAM'),
    ('ORD-009', 'C600', 15000, DATE '2024-06-05', 'NAMER'),
    ('ORD-010', 'C300', 7200, DATE '2024-06-05', 'LATAM');

Configuring cross-cloud access

BigQuery assumes an AWS IAM role via OIDC federation to access the Glue IRC. This section walks through creating the role, OIDC provider, and permissions.

Create the OIDC identity provider

Register Google as an OIDC identity provider in your AWS account. This allows AWS to validate tokens issued by Google’s identity service:

aws iam create-open-id-connect-provider \
    --url https://accounts.google.com \
    --client-id-list accounts.google.com \
    --thumbprint-list 08745487e891c19e3078c1f2a07e452950ef36f6

The –thumbprint-list parameter is optional. When omitted, IAM automatically retrieves the thumbprint from the OIDC provider’s certificate. See AWS documentation for details.

Create the cross-cloud IAM role

Login into AWS Console, and create the role with a placeholder trust policy. You will update it with the actual BigLake service account ID after you create the federated catalog in Google Cloud.

aws iam create-role \
    --role-name bigquery-cross-cloud-role \
    --max-session-duration 43200 \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["PLACEHOLDER"],
            "accounts.google.com:aud": ["PLACEHOLDER"]
          }
        }
      }]
    }'

The --max-session-duration 43200 allows sessions up to 12 hours, which is needed for long-running BigQuery queries.

Attach permissions

The permissions policy differs based on your access control approach. For the IAM-based approach, attach the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueRead",
      "Effect": "Allow",
      "Action": [
        "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases",
        "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions"
      ],
      "Resource": [
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog/<TABLE_BUCKET>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>/*"
      ]
    },
    {
      "Sid": "S3TablesRead",
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTableBucket", "s3tables:ListTableBuckets",
        "s3tables:ListNamespaces", "s3tables:GetNamespace",
        "s3tables:ListTables", "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation", "s3tables:GetTableData"
      ],
      "Resource": [
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>",
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>/*"
      ]
    },
    {
      "Sid": "S3TablesListBuckets",
      "Effect": "Allow",
      "Action": ["s3tables:ListTableBuckets"],
      "Resource": "*"
    }
  ]
}

Connecting BigQuery to S3 Tables

With the AWS side configured, create the federated catalog in Google Cloud that connects BigQuery to the Glue IRC.

Create the federated catalog

Authenticate to Google Cloud using gcloud auth login, or use Cloud Shell, which is pre-authenticated. Verify that the BigLake API is enabled:

gcloud services enable biglake.googleapis.com --project="<GCP_PROJECT_ID>"

For IAM mode:

gcloud alpha biglake iceberg catalogs create <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --catalog-type=federated \
    --federated-catalog-type=glue \
    --glue-aws-region=<AWS_REGION> \
    --glue-aws-role-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role \
    --glue-warehouse=<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET> \
    --primary-location=<GCP_REGION>

The --glue-warehouse parameter uses the format <AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>. This tells the Glue IRC to scope requests to your specific S3 Tables bucket within the federated catalog hierarchy.

The --primary-location refers to the Google Cloud region where the federated catalog metadata is stored. Use the AWS to Google Cloud region mapping to find the corresponding GCP region for your AWS Region. For example, AWS us-east-1 maps to GCP us-east4.

Retrieve the BigLake service account ID

After catalog creation, Google provisions a dedicated service account for your federated catalog. Retrieve its numeric ID:

BIGLAKE_SA_ID=$(gcloud alpha biglake iceberg catalogs describe <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --format="value(biglake-service-account-id)")
echo $BIGLAKE_SA_ID

Update the AWS trust policy

Back on AWS, replace the placeholder in the IAM role’s trust policy with the actual service account ID:

aws iam update-assume-role-policy \
    --role-name bigquery-cross-cloud-role \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["<BIGLAKE_SA_ID>"],
            "accounts.google.com:aud": ["<BIGLAKE_SA_ID>"]
          }
        }
      }]
    }'

Register the service account ID in the OIDC provider’s audience list. Without this step, AWS rejects the token because the aud claim doesn’t match any registered client:

aws iam add-client-id-to-open-id-connect-provider \
    --open-id-connect-provider-arn "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com" \
    --client-id "<BIGLAKE_SA_ID>"

Set up metadata sync

Wait 3–5 minutes for IAM changes to propagate globally, then set up background refresh:

gcloud alpha biglake iceberg catalogs update <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --refresh-interval=300s

The --refresh-interval (300 seconds in this example) determines how often BigQuery syncs metadata from the Glue IRC. New tables and schema changes appear in BigQuery within this interval.

Querying from BigQuery

After the catalog refresh completes, BigQuery automatically creates external datasets corresponding to the synced namespaces. No manual CREATE SCHEMA is required.

Verify the sync:

gcloud alpha biglake iceberg namespaces list \
    --catalog="<FEDERATED_CATALOG_NAME>" \
    --project="<GCP_PROJECT_ID>"

Run a query in BigQuery:

SELECT * FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders` LIMIT 1000

Sample Query Output:

SELECT
    customer_id,
    COUNT(*) as order_count,
    SUM(amount) as total_spend
FROM `<PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders`
GROUP BY customer_id
ORDER BY total_spend DESC

BigQuery query results showing order count and total spend per customer from the Amazon S3 Tables data

Figure 3: BigQuery query results returned directly from the Amazon S3 Tables data

BigQuery reads the Iceberg metadata to identify which Parquet data files contain relevant data. It also applies partition pruning where applicable, and fetches only the necessary files from S3 Tables managed storage.

Schema evolution

When new columns are added to an Iceberg table on the AWS side (through Spark, Athena, or the Glue IRC), the schema change is captured in Iceberg’s metadata. On the next Lakehouse refresh cycle, BigQuery picks up the new columns automatically. No DDL changes are needed in BigQuery.

Metadata freshness

The s3tablescatalog in Glue is a federated catalog that resolves table metadata live from the S3 Tables service on each request. When a streaming job commits new data to an S3 Table, the latest metadata is immediately available through the AWS Glue IRC. BigQuery sees the update on its next refresh cycle (as configured by --refresh-interval).

OIDC identity federation

The trust relationship between Google Cloud and AWS uses OpenID Connect. When BigQuery Lakehouse needs to access your data, it presents a signed JWT token containing:

iss: accounts.google.com (the issuer)
sub: The BigLake service account ID (identifies which catalog is making the request)
aud: The same service account ID (the intended audience)

AWS validates this token against the registered OIDC provider and trust policy conditions before issuing temporary credentials. Each federated catalog receives a unique service account ID, providing per-catalog isolation and auditability through AWS CloudTrail.

Network path

By default, traffic between BigQuery and AWS travels over the public internet. For workloads requiring private connectivity, Google Cloud supports Cross-Cloud Interconnect or Partner Interconnect. This helps routing queries over a dedicated network path. Refer to the Google Cloud documentation for private interconnect configuration.

Clean up

To avoid ongoing charges, remove the resources created in this walkthrough.

On AWS:

# Delete the table (if created for this walkthrough)
aws s3tables delete-table \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace analytics --name orders --region <AWS_REGION>

# Delete namespace and table bucket
aws s3tables delete-namespace \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace <NAMESPACE> --region <AWS_REGION>

aws s3tables delete-table-bucket --name <TABLE_BUCKET> --region <AWS_REGION>

# Delete IAM role and OIDC provider (if no longer needed)
aws iam delete-role --role-name bigquery-cross-cloud-role

On Google Cloud:

gcloud alpha biglake iceberg catalogs delete <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" --location=<GCP_REGION>

Conclusion

This post demonstrated how to query Amazon S3 Tables from Google BigQuery using the open Apache Iceberg format and the AWS Glue Iceberg REST Catalog as the metadata bridge. Using Apache Iceberg’s open format, you can write data once on AWS and read it from supported engines that speak Iceberg, including BigQuery. We used IAM-based access control to govern access to both Glue Data Catalog metadata and the underlying Amazon S3 Tables data. This is the simpler configuration path with fewer components. In Part 2, we walk through configuring AWS Lake Formation to vend temporary, scoped credentials to BigQuery for data access.

To get started with this pattern in your environment:

Review the Amazon S3 Tables documentation for table bucket setup.
Follow the Google Cloud cross-cloud Lakehouse setup guide for the BigQuery federation configuration.
Explore the AWS Glue Iceberg REST Catalog APIs for programmatic access patterns.

About the authors

Enable cross-cloud analytics with Amazon S3 Tables and Google BigQuery, Part 2: access control with Lake Formation

2026-08-25 Lakshmi Nair

Post Syndicated from Lakshmi Nair original https://aws.amazon.com/blogs/big-data/enable-cross-cloud-analytics-with-amazon-s3-tables-and-google-bigquery-part-2-access-control-with-lake-formation/

In Part 1, we showed how to connect Google BigQuery to Amazon Simple Storage Service (Amazon S3) Tables, a capability of Amazon S3, using access control based on AWS Identity and Access Management (IAM). A single IAM policy governs both table metadata and data access. We also walked through common cross-cloud analytics scenarios where this pattern adds value. This post covers the approach using AWS Lake Formation. Instead of relying solely on IAM policies for data access, Lake Formation manages fine-grained permissions and vends temporary, scoped credentials to the requesting engine. This is a better fit when multiple engines need different levels of access to the same tables, or when you want to manage grants centrally without touching IAM policies every time a new consumer comes along.

Solution overview

Figure 1: Architecture diagram showing BigQuery connecting to Amazon S3 Tables through the AWS Glue Iceberg REST Catalog

The key components in this architecture:

Amazon S3 Tables: With Amazon S3 Tables, data is stored in table buckets, specifically designed for storing tables in the Apache Iceberg format. Table metadata is registered on AWS Glue Data Catalog for discovery and governance.
AWS Glue Data Catalog: With AWS Glue Data Catalog, you can access the federated s3tablescatalog catalog that maps S3 Tables resources (table buckets, namespaces, tables) into a catalog hierarchy from supported analytics engines. The standard Iceberg REST endpoint of Glue Data Catalog serves table metadata to external engines. BigQuery connects through this endpoint.
AWS Lake Formation: With AWS Lake Formation, you define access permissions at the catalog, database, and table level. Instead of granting broad IAM permissions for data access, Lake Formation evaluates permissions at query time and issues short-lived credentials limited to the resources the caller is authorized to read.
Google Cross-Cloud Lakehouse: With Google Cross-Cloud Lakehouse, you can connect BigQuery to external Iceberg catalogs. It assumes an IAM role using OpenID Connect (OIDC), calls the AWS Glue Iceberg REST endpoint, and syncs metadata on a configurable refresh interval.

Prerequisites

Before you begin, you need:

An AWS account with Amazon S3 Tables available in your AWS Region.
A Google Cloud project with billing enabled and the BigLake API activated.
AWS Command Line Interface (AWS CLI) and gcloud CLI installed and configured.
An S3 table bucket with at least one namespace and table containing data.

Setting up Amazon S3 Tables

If you already have S3 Tables with data, skip to the next section. Otherwise, create a table bucket, namespace, and populate a table.

Create a table bucket and namespace

Use AWS CLI to create resources as follows:

# Create a Table bucket
aws s3tables create-table-bucket \
    --name <TABLE_BUCKET_NAME> \
    --region <REGION>

# Create a Namespace (Database)
aws s3tables create-namespace \
    --table-bucket-arn "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET_NAME>" \
    --namespace <NAMESPACE> \
    --region <REGION>

Set up S3 Tables integration with the Glue Data Catalog using Lake Formation mode

Lake Formation needs its own service role to interact with S3 Tables on your behalf. This is the role Lake Formation assumes internally when it reads or writes data on behalf of authorized callers.

Create a Lake Formation service IAM role named LakeFormationS3TablesServiceRole with the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LakeFormationPermissionsForS3ListTableBucket",
      "Effect": "Allow",
      "Action": ["s3tables:ListTableBuckets"],
      "Resource": ["*"]
    },
    {
      "Sid": "LakeFormationDataAccessPermissionsForS3TableBucket",
      "Effect": "Allow",
      "Action": [
        "s3tables:CreateTableBucket", "s3tables:GetTableBucket",
        "s3tables:CreateNamespace", "s3tables:GetNamespace",
        "s3tables:ListNamespaces", "s3tables:DeleteNamespace",
        "s3tables:DeleteTableBucket", "s3tables:CreateTable",
        "s3tables:DeleteTable", "s3tables:GetTable",
        "s3tables:ListTables", "s3tables:RenameTable",
        "s3tables:UpdateTableMetadataLocation", "s3tables:GetTableMetadataLocation",
        "s3tables:GetTableData", "s3tables:PutTableData"
      ],
      "Resource": ["arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/*"]
    }
  ]
}

Attach the following trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LakeFormationDataAccessPolicy",
      "Effect": "Allow",
      "Principal": { "Service": "lakeformation.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:SetContext", "sts:SetSourceIdentity"],
      "Condition": { "StringEquals": { "aws:SourceAccount": "<AWS_ACCOUNT_ID>" } }
    }
  ]
}

In the Lake Formation console, in the navigation pane, choose Catalogs, and then choose Enable S3 Table Integration.

The Enable S3 Table Integration option on the Catalogs page of the Lake Formation console

Figure 2: Enabling the S3 Tables integration in the Lake Formation console

Choose the role you created earlier when prompted for an IAM role, and select Allow external engines to access data in Amazon S3 locations with full table access.

S3 Tables integration performs the following:

Registers the S3 Tables data location with Lake Formation.
Creates the s3tablescatalog federated catalog in Glue.

Important: Before enabling the integration, verify your Lake Formation data lake settings have empty default permissions to prevent IAMAllowedPrincipals from being auto-granted on the catalog:

aws lakeformation put-data-lake-settings \
    --data-lake-settings '{"DataLakeAdmins":[{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ADMIN_ROLE>"}],"CreateDatabaseDefaultPermissions":[],"CreateTableDefaultPermissions":[]}' \
    --region <AWS_REGION>

The S3 Tables integration dialog in Lake Formation with full table access selected

Figure 3: Selecting full table access for external engines during S3 Tables integration

When you select this option, you allow external engines to access data in Amazon S3 locations with full table access, and Lake Formation grants full table-level access to external engines. Column-level and row-level filtering are not enforced for external engine connections. Access is granted at the whole-table level.

Verify the integration by confirming the catalog in Lake Formation console.

Create a table and insert data

CREATE TABLE `<NAMESPACE>`.orders (
    order_id STRING,
    customer_id STRING,
    amount BIGINT,
    order_date DATE,
    region STRING
)
TBLPROPERTIES ('table_type' = 'iceberg');

INSERT INTO orders
VALUES
    ('ORD-001', 'C100', 4500, DATE '2024-06-01', 'EMEA'),
    ('ORD-002', 'C200', 8900, DATE '2024-06-01', 'EMEA'),
    ('ORD-003', 'C100', 3200, DATE '2024-06-02', 'NAMER'),
    ('ORD-004', 'C300', 12000, DATE '2024-06-02', 'NAMER'),
    ('ORD-005', 'C400', 6700, DATE '2024-06-03', 'APJ'),
    ('ORD-006', 'C200', 4100, DATE '2024-06-03', 'APJ'),
    ('ORD-007', 'C500', 9500, DATE '2024-06-04', 'EMEA'),
    ('ORD-008', 'C100', 2800, DATE '2024-06-04', 'LATAM'),
    ('ORD-009', 'C600', 15000, DATE '2024-06-05', 'NAMER'),
    ('ORD-010', 'C300', 7200, DATE '2024-06-05', 'LATAM');

Configuring cross-cloud access

BigQuery assumes an AWS IAM role using OIDC federation to access the AWS Glue IRC. This section walks through creating the role, OIDC provider, and permissions.

Create the OIDC identity provider

Register Google as an OIDC identity provider in your AWS account. This allows AWS to validate tokens issued by Google’s identity service:

aws iam create-open-id-connect-provider \
    --url https://accounts.google.com \
    --client-id-list accounts.google.com \
    --thumbprint-list 08745487e891c19e3078c1f2a07e452950ef36f6

The –thumbprint-list parameter is optional. When omitted, IAM automatically retrieves the thumbprint from the OIDC provider’s certificate. See AWS documentation for details.

Create the cross-cloud IAM role on AWS

Sign in to the AWS Management Console. Create the role with a placeholder trust policy. You will update it with the actual BigLake service account ID after you create the federated catalog in Google Cloud.

aws iam create-role \
    --role-name bigquery-cross-cloud-role \
    --max-session-duration 43200 \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["PLACEHOLDER"],
            "accounts.google.com:aud": ["PLACEHOLDER"]
          }
        }
      }]
    }'

The --max-session-duration 43200 allows sessions up to 12 hours, which is needed for long-running BigQuery queries.

Attach permissions

The permissions policy differs based on your access control approach. For the Lake Formation approach, attach the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueRead",
      "Effect": "Allow",
      "Action": [
        "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases",
        "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions"
      ],
      "Resource": [
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog/<TABLE_BUCKET>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>/*"
      ]
    },
    {
      "Sid": "S3TablesRead",
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTableBucket", "s3tables:ListTableBuckets",
        "s3tables:ListNamespaces", "s3tables:GetNamespace",
        "s3tables:ListTables", "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation", "s3tables:GetTableData"
      ],
      "Resource": [
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>",
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>/*"
      ]
    },
    {
      "Sid": "LakeFormationCredentialVending",
      "Effect": "Allow",
      "Action": ["lakeformation:GetDataAccess"],
      "Resource": "*"
    }
  ]
}

Grant Lake Formation permissions

Lake Formation permissions work as a layered grant model: you grant access at each level of the catalog hierarchy, from catalog down to table. The cross-cloud role needs DESCRIBE on the catalog and database so it can discover what exists, and SELECT plus DESCRIBE on the table so it can read the actual data. Without grants at every level, Lake Formation denies access even if the IAM policy allows it.

If using Lake Formation, grant the bigquery-cross-cloud-role access to your tables:

Grant catalog permission: DESCRIBE.
Grant database permission: DESCRIBE.
Grant table permission: SELECT, DESCRIBE.

Grant Lake Formation permissions on the cross-cloud role (one-time).

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Catalog":{"Id":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>"}}'     --permissions '["DESCRIBE"]'     --region <AWS_REGION>

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Database":{"CatalogId":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>","Name":"<NAMESPACE>"}}'     --permissions '["DESCRIBE"]'     --region <AWS_REGION>

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Table":{"CatalogId":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>","DatabaseName":"<NAMESPACE>","Name":"orders"}}'     --permissions '["SELECT","DESCRIBE"]'     --region <AWS_REGION>

Before granting Lake Formation permissions, revoke the default IAMAllowedPrincipals access. By default, Lake Formation grants IAMAllowedPrincipals full access to all databases and tables, so you first need to revoke this to enforce fine grain access. IAMAllowedPrincipals provides backward compatibility when you start using Lake Formation permissions to secure the Data Catalog resources that were earlier protected by IAM policies for AWS Glue.

Set up Lake Formation for external engines

For table metadata to sync from Glue to BigLake/BigQuery, the following Lake Formation settings are required. You might notice that a similar setting also appeared during the S3 Table integration setup. The first one registers the data location and enables external access at the catalog level, while this one enables the Lake Formation credential vending mechanism at the account level for all external engines. For a clean cross-cloud setup, we recommend that you enable both.

In the Lake Formation console, choose Administration, then Application integration settings, and then select Allow external engines to access data in Amazon S3 locations with full table access.

Application integration settings in the Lake Formation console with external-engine access enabled

Figure 4: Enabling external-engine access in Lake Formation application integration settings

Connecting BigQuery to S3 Tables

With the AWS side configured, create the federated catalog in Google Cloud that connects BigQuery to the AWS Glue IRC.

Create the federated catalog

Authenticate to Google Cloud using gcloud auth login, or use Cloud Shell, which is pre-authenticated. Verify the BigLake API is enabled:

gcloud services enable biglake.googleapis.com --project="<GCP_PROJECT_ID>"

For Lake Formation mode (with credential vending):

gcloud alpha biglake iceberg catalogs create <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --catalog-type=federated \
    --federated-catalog-type=glue \
    --glue-aws-region=<AWS_REGION> \
    --glue-aws-role-arn=arn:aws:iam::<ACCOUNT_ID>:role/bigquery-cross-cloud-role \
    --glue-warehouse=<ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET> \
    --primary-location=<GCP_REGION> \
    --credential-mode=vended-credentials

The --glue-warehouse parameter uses the format <AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>. This tells the AWS Glue IRC to scope requests to your specific S3 Tables bucket within the federated catalog hierarchy.

The --credential-mode=vended-credentials flag (Lake Formation mode) instructs BigQuery Lakehouse to request scoped temporary credentials from Lake Formation rather than using the role’s IAM permissions directly for data access.

Retrieve the BigLake service account ID

After catalog creation, Google provisions a dedicated service account for your federated catalog. Retrieve its numeric ID:

BIGLAKE_SA_ID=$(gcloud alpha biglake iceberg catalogs describe <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --format="value(biglake-service-account-id)")
echo $BIGLAKE_SA_ID

Update the AWS trust policy

Back on AWS, replace the placeholder in the IAM role’s trust policy with the actual service account ID:

aws iam update-assume-role-policy \
    --role-name bigquery-cross-cloud-role \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["<BIGLAKE_SA_ID>"],
            "accounts.google.com:aud": ["<BIGLAKE_SA_ID>"]
          }
        }
      }]
    }'

Register the service account ID in the OIDC provider’s audience list. Without this step, AWS rejects the token because the aud claim doesn’t match any registered client:

aws iam add-client-id-to-open-id-connect-provider \
    --open-id-connect-provider-arn "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com" \
    --client-id "<BIGLAKE_SA_ID>"

Set up metadata sync

Wait 3–5 minutes for IAM changes to propagate globally, then set up background refresh:

gcloud alpha biglake iceberg catalogs update <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --refresh-interval=300s

The --refresh-interval (300 seconds in this example) determines how often BigQuery syncs metadata from the AWS Glue IRC. New tables and schema changes appear in BigQuery within this interval.

Querying from BigQuery

After the catalog refresh completes, BigQuery automatically creates external datasets corresponding to the synced namespaces. No manual CREATE SCHEMA is required.

Verify the sync:

gcloud alpha biglake iceberg namespaces list \
    --catalog="<FEDERATED_CATALOG_NAME>" \
    --project="<GCP_PROJECT_ID>"

Run a query in BigQuery:

SELECT * FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders` LIMIT 1000

Sample Query Output:

SELECT
    customer_id,
    COUNT(*) as order_count,
    SUM(amount) as total_spend
FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders`
GROUP BY customer_id
ORDER BY total_spend DESC

Figure 5: BigQuery query results returned through Lake Formation credential vending

Schema evolution

When new columns are added to an Iceberg table on the AWS side (through Spark, Athena, or the AWS Glue IRC), the schema change is captured in Iceberg’s metadata. On the next Lakehouse refresh cycle, BigQuery picks up the new columns automatically. No DDL changes are needed in BigQuery.

Metadata freshness

The s3tablescatalog catalog in AWS Glue is a federated catalog that resolves table metadata live from the S3 Tables service on each request. When a streaming job commits new data to an S3 Table, the latest metadata is immediately available through the AWS Glue IRC. BigQuery sees the update on its next refresh cycle (as configured by --refresh-interval).

OIDC identity federation

The trust relationship between Google Cloud and AWS uses OpenID Connect. When BigQuery Lakehouse needs to access your data, it presents a signed JWT token containing:

iss: accounts.google.com (the issuer)
sub: The BigLake service account ID (identifies which catalog is making the request)
aud: The same service account ID (the intended audience)

Network path

Clean up

To avoid ongoing charges, remove the resources created in this walkthrough.

On AWS:

# Delete the table (if created for this walkthrough)
aws s3tables delete-table \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace analytics --name orders --region <AWS_REGION>

# Delete namespace and table bucket
aws s3tables delete-namespace \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace <NAMESPACE> --region <AWS_REGION>

aws s3tables delete-table-bucket --name <TABLE_BUCKET> --region <AWS_REGION>

# Delete IAM role and OIDC provider (if no longer needed)
aws iam delete-role --role-name bigquery-cross-cloud-role

On Google Cloud:

gcloud alpha biglake iceberg catalogs delete <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" --location=<GCP_REGION>

Conclusion

This post demonstrated how to query Amazon S3 Tables from Google BigQuery using AWS Lake Formation credential vending, where Lake Formation manages the permissions and issues temporary, scoped credentials for data access. With the open Iceberg format, you can write data once on AWS and read it from supported engines that speak Iceberg, including BigQuery.

Together with the IAM approach covered in Part 1, two access control modes provide flexibility: IAM for teams who want a straightforward setup and Lake Formation for organizations with complex governance requirements where multiple engines need centrally managed access to the same data.

To get started with this pattern in your environment:

Review the Amazon S3 Tables documentation for table bucket setup.
Follow the Google Cloud cross-cloud Lakehouse setup guide for the BigQuery federation configuration.
Explore the AWS Glue Iceberg REST Catalog APIs for programmatic access patterns.

About the authors

AMD Versal RF Series at Hot Chips 2026

2026-08-25 Patrick Kennedy

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/amd-versal-rf-series-at-hot-chips-2026/

The AMD Versal RF devices are getting there time on stage at Hot Chips 2026 with hardend IP blocks to flexible FPGA fabric

The post AMD Versal RF Series at Hot Chips 2026 appeared first on ServeTheHome.

GPU-accelerated Apache Spark with Amazon EMR and NVIDIA RTX PRO 4500 on Amazon EC2 G7 instances runs up to 3.7x faster

2026-08-25 McCall Peltier

Post Syndicated from McCall Peltier original https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/

For years, Apache Spark has been the backbone of large-scale data processing. However, as datasets grow and artificial intelligence and machine learning (AI/ML) pipelines become more complex, modern workloads demand more computational power. Feature engineering for machine learning models, large-scale extract, transform, and load (ETL) transformations, and real-time analytics workloads are computationally intensive by nature. GPU-accelerated instances improve performance and transform jobs that once took hours into minutes, so you can iterate on models faster and reduce operational costs. You can process larger datasets in single batches, make decisions in real time, and achieve strong performance without over-provisioning infrastructure.

We’re excited to share the benchmarking results on Amazon EMR with Amazon Elastic Compute Cloud (Amazon EC2) G7 instances, powered by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. For data engineers and data scientists running Apache Spark workloads, this means faster pipelines, shorter iteration cycles, and more time spent on insights.

Amazon EMR on EKS natively supports the NVIDIA cuDF plugin for Apache Spark. This support is the result of joint engineering between AWS and NVIDIA to qualify the cuDF plugin for Amazon EMR, co-optimize Spark execution paths for RTX PRO 4500, and validate performance at scale through shared TPC-DS benchmarking on Amazon EC2 G7 instances. Now, Apache Spark workloads on Amazon EMR on EKS run up to 3.7x faster with Amazon EC2 G7 GPU instances than with comparable CPU instances, and require no changes to existing Spark code.

In the TPC-DS 3 TB benchmark, at the 64 GB memory tier, EC2 G7 instances with RTX PRO finished in 4.7 minutes. If you run large-scale data processing pipelines, you can cut job run times by more than two-thirds while maintaining full compatibility with the applications you already have in production.

The use cases that benefit most are those where speed directly unlocks business value. In AI/ML feature engineering, faster Spark jobs mean data science teams can iterate on features more quickly, reducing the time from raw data to trained model. In complex ETL pipelines, like financial transactions, clickstream aggregation, or supply chain data consolidation, GPU acceleration compresses multi-hour batch windows into near-real-time processing. For real-time analytics, teams running fraud detection, personalization engines, or operational dashboards can process larger volumes of data within tighter latency windows, without redesigning their architecture.

Beyond data analytics, the G7 instances will support a broad range of AI and graphics workloads, including conversational AI, content generation, recommender systems, and video streaming and rendering. Built on the AWS Nitro System, they deliver the security and resource efficiency that production AI, analytics, and graphics workloads demand.

The following sections walk through the cluster configuration, benchmark methodology, and performance results.

Cluster configuration

We benchmarked four instance types to measure the real-world performance of G7 GPU instances against comparable CPU instances for Spark SQL performance. The g7.4xlarge also provides 80 Gbps network bandwidth (compared to 15–17 Gbps on the CPU baselines) and uses RapidsShuffleManager. However, CPU runs showed no evidence of being network- or shuffle-bound at this cluster scale. All tests used Amazon EMR on EKS 7.12.0 with Apache Spark 3.5.6 and cuDF plugin 26.04.2, running the full TPC-DS benchmark at 3 TB scale across 103 queries. Each experiment ran 5 iterations. We report the median. Data was stored as Parquet on Amazon Simple Storage Service (Amazon S3) (same-region gateway endpoint). All instances were launched in a single Availability Zone.

Instance specifications

All four instance types share the same compute footprint of 16 vCPUs and 64 GB system RAM. The g7.4xlarge additionally includes an NVIDIA RTX PRO 4500 Blackwell GPU with 32 GB of dedicated video memory (VRAM), which the cuDF plugin uses to accelerate Spark SQL operations. The baseline for all speedup and cost comparisons is m9gd.4xlarge (Graviton), the lowest-cost CPU instance in the group.

.	g7.4xlarge	m9gd.4xlarge	m8id.4xlarge	m8a.4xlarge
Architecture	x86_64	arm64 (Graviton)	x86_64	x86_64
vCPU	16	16	16	16
RAM	64 GB	64 GB	64 GB	64 GB
GPU	1× RTX PRO 4500 Blackwell (32 GB VRAM)	–	–	–
NVMe	875 GB	950 GB	950 GB	EBS only (GP3 16k IOPS and 2000 MB/s throughput to match NVMe
Network	80 Gbps	Up to 17 Gbps	Up to 15 Gbps	Up to 15 Gbps

The g7.4xlarge uses the RTX PRO 4500 Blackwell Server Edition GPU. The CPU baselines cover all three major architectures: m8id.4xlarge (Intel x86), m8a.4xlarge (AMD x86), and m9gd.4xlarge (Graviton arm64).

Spark configuration

All instances used eight executor nodes with the following configuration:

Configuration	GPU instances	CPU instances
Amazon EMR release	emr-7.12.0-spark-rapids-latest	emr-7.12.0-latest
executor.cores	14	14
executor.instances	8	8
executor.memory	20G	20G
executor.memoryOverhead	30G	30G
spark.plugins	com.nvidia.spark.SQLPlugin	–
rapids.memory.pinnedPool.size	8G	–
rapids.sql.concurrentGpuTasks	3	–
shuffle.manager	RapidsShuffleManager	default (sort)
sql.adaptive.enabled	true	true
io.compression.codec	zstd	zstd

CPU instances use the same 30 GB memoryOverhead as GPU to make sure that the memory comparison is apples-to-apples. This setting reserves off-heap memory for shuffle and caching on both sides.

For GPU instances, the cuDF plugin offloads Spark SQL operations to the GPU automatically. No code changes are required. The executor.memoryOverhead value is set higher on GPU instances to accommodate GPU memory management and the RAPIDS shuffle manager.

The cuDF plugin automatically falls back to CPU execution for unsupported operators and user-defined functions (UDFs). Your job still completes, but those stages run without GPU acceleration. To identify which operations run on GPU compared to CPU, set spark.rapids.sql.explain=NOT_ON_GPU in your Spark configuration. For a pre-migration assessment of your workloads, use the NVIDIA cuDF tool to estimate GPU acceleration potential before moving to G7 instances.

To tune settings like concurrentGpuTasks and pinnedPool.size, use the Spark History Server on Amazon EMR on EKS, which provides per-stage execution details to identify CPU fallback and shuffle bottlenecks.

Getting started

Reference the Using cuDF Accelerator for Apache Spark with Amazon EMR on EKS for detailed setup instructions.

Prerequisites

Before running GPU-accelerated Spark on Amazon EMR on EKS, make sure the following are in place:

Amazon EMR on EKS release version 6.9.0 or later (this post uses emr-7.12.0-spark-rapids-latest).

The -spark-rapids release variant ships the NVIDIA cuDF plugin pre-installed.

Amazon Elastic Kubernetes Service (Amazon EKS) cluster with a GPU-enabled node group using G7 instances.
Node AMI: AL2023_x86_64_NVIDIA (Amazon EKS optimized accelerated AMI).

NVIDIA device plugin installed in the cluster to expose GPUs to Kubernetes pods:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.9.0/nvidia-device-plugin.yml

Amazon EMR on EKS virtual cluster registered to the EKS namespace.

To validate GPU availability on your nodes:

kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

Note: Getting started with GPU-accelerated Spark on Amazon EMR is straightforward. To use the latest cuDF plugin, overlay the latest version (for example, 26.04.2 as of May 2026) onto the Amazon EMR RAPIDS image using an initContainer technique. This replaces the bundled cuDF JAR with a newer version while preserving all other Amazon EMR dependencies. We recommend using the latest Amazon EMR release to get the most up-to-date cuDF plugin for better performance. In our benchmarks, upgrading from cuDF plugin 25.08.0 to 26.04.2 reduced runtime by 36–38 percent. Download the latest cuDF plugin JAR from the NVIDIA repository. AWS Support covers Amazon EMR. For issues specific to a cuDF JAR, file a GitHub issue or contact NVIDIA at [email protected].

Performance benchmarks and cost efficiency

We ran the full TPC-DS benchmark suite (103 queries) at 3 TB scale on 8-node clusters in us-east-1. The following table summarizes the results:

.	GPU instances	CPU instances
Cost per run	$2.06	$2.93–$3.18
Total time (103 queries)	281s (4.7 min)	1,010–1,043s (16.8–17.4 min)
Speedup compared to CPU instances	3.7×	baseline

Cost per run is the total cluster cost for the benchmark’s duration: Cluster $/hr × (median runtime ÷ 3,600). The hourly rate combines the EC2 On-Demand cost for all 8 nodes and the Amazon EMR on EKS charge for the vCPU and memory the Spark pods consume. Both are billed per second (one-minute minimum), so you pay only for what a job uses while it runs. All runs used Amazon EMR on EKS 7.12.0 in us-east-1, with 8 × 4xlarge nodes (128 vCPU) on both the GPU and CPU sides. The g7.4xlarge cluster runs at $26.35/hr (8 × $3.042 EC2 = $24.34, plus $2.01 for Amazon EMR on EKS) and finishes in 281 seconds, at $2.06 per run. The CPU clusters run at a lower hourly rate ($10.43–$10.99) but take 1,010–1,043 seconds, landing at $2.93–$3.18 per run. All prices reflect On-Demand pricing in us-east-1 as of May 2026. G7 instances are also eligible for EC2 Spot and Compute Savings Plans, which can further reduce costs for recurring batch workloads.

Cost-per-run calculations include EC2 and Amazon EMR charges only. They exclude the EKS control-plane fee, EBS volumes, S3 request and storage costs, and the driver pod.

Bar chart of total TPC-DS runtime by instance type, showing g7.4xlarge finishing far faster than the CPU instances

Figure 1: Total runtime by instance type for all 103 TPC-DS queries at 3 TB scale. The g7.4xlarge with GPU acceleration completed the benchmark in 4.7 minutes, 3.7× faster than CPU instances (16.8-17.4 minutes)

Bar chart of total cost per benchmark run by instance type, showing the g7.4xlarge GPU instance costing less than the CPU instances

Figure 2: Total cost per benchmark run, including both Amazon EC2 instance and Amazon EMR on EKS cost across all 8 nodes. Despite a ~2.5× higher hourly rate, the g7.4xlarge GPU instance costs up to 31% less per run than Graviton because it finishes the workload 3.7× faster

Where GPU acceleration excels

GPU acceleration completed the 103-query power run in 281s compared to 1,032s on CPU, an overall 3.7× speedup that saves 750 seconds per run. GPU was faster on 102 of 103 query executions.

GPU acceleration delivers the largest gains on the long-running, compute- and shuffle-heavy queries where kernel throughput outweighs launch overhead. The biggest absolute time savings:

Query	CPU time	GPU time	Speedup	Time saved
q24 (part 1+2)	81.6s	15.9s	~5.1×	65.7s
q23 (part 1+2)	79.4s	16.4s	~4.9×	63.1s
q93	63.7s	5.6s	11.4×	58.1s
q76	30.4s	3.4s	9.0×	27.0s
q64	35.4s	8.5s	4.2×	26.9s
q50	27.6s	3.5s	7.9×	24.0s

Speedup distribution across all 103 executions:

Speedup band	Queries
≥5×	15
4–5×	13
3–4×	21
2–3×	26
1–2×	27
<1× (CPU faster)	1

Median per-query speedup 2.94× (geomean 2.84×). The heaviest wins (q50, q76, q93) are aggregation- and shuffle-join-intensive queries that convert cleanly to GpuHashAggregate and GpuBroadcastHashJoin.

Where CPU wins

With RAPIDS 26.04.2, the following query showcases a workload pattern where CPU was faster:

Query	CPU time	GPU time	Ratio	Root cause
q16	0.96s	1.44s	CPU 1.5x faster	Trivial/near-empty scan. Sub-second runtime where GPU kernel-launch overhead is not amortized

Choosing the right instance

Instance	Best for	Summary
g7.4xlarge (RTX PRO GPU)	Fastest and most cost-effective	Up to 3.7× faster than comparable CPU instances and up to 31% cheaper per run. Completes in 4.7 min compared to 17.2 min. Best choice for both speed and cost efficiency.
CPU instances (m8a / m8id / m9gd)	Flexibility, availability, and always-on workloads	Multiple architecture options deliver similar Spark SQL performance. Choose CPU when GPUs are unavailable, when clusters need to remain running continuously (for example, overnight jobs ready for next-day analysis), or when workloads cannot use GPU acceleration. CPU instances offer broad availability and predictable capacity without startup delays.

G7 instances require a G-instance vCPU service quota in your account (default is often 0 for GPU types). Request a quota increase through the Service Quotas console, or use On-Demand Capacity Reservations (ODCRs) to guarantee availability for recurring batch jobs.

Based on these benchmark results, consider evaluating GPU acceleration for your own Apache Spark workloads. Start by identifying compute-intensive operations in your current pipelines, particularly those involving large-scale aggregations, joins, or machine learning feature engineering that could benefit from the performance improvements demonstrated here.

Conclusion

Amazon EMR on EKS with NVIDIA RTX PRO 4500 together provide a meaningful step forward for teams running data-intensive Spark workloads at scale. Whether you’re building ML pipelines that demand rapid feature iteration, running complex ETL transformations across massive datasets, or powering real-time analytics that can’t afford to wait on slow batch jobs, GPU-accelerated Spark on G7 delivers the performance and speed to do more. As data and AI workloads continue to evolve, GPU-accelerated analytics on Amazon EMR is becoming the foundation for data teams. Get started with GPU-accelerated Spark on Amazon EMR on EKS today by visiting Amazon EMR documentation to launch your first G7-powered cluster and see the performance gains for yourself.