Post Syndicated from Frank Scarfo original https://aws.amazon.com/blogs/compute/adding-custom-domains-to-aws-lambda-microvms-with-application-load-balancer/

AWS Lambda MicroVMs is a serverless compute building block that provides VM-level isolation, near-instant startup performance, and state retention. You can now give each user or job their own execution environment to securely run just-in-time code, whether user or AI-generated. You do this without managing virtualization infrastructure or choosing between isolation, speed, and state retention. Lambda MicroVMs are powered by Firecracker virtualization, the technology underpinning AWS Lambda.

When you run a workload on AWS Lambda MicroVMs, each MicroVM is reachable at a service-generated endpoint that looks like 92cfc7f9-….lambda-microvm-….on.aws. That works, but many teams want to expose their MicroVMs under a domain they own, such as 92cfc7f9-….microvms.example.com. When a browser is the client, they also want to satisfy cross-origin resource sharing (CORS) without changing the application inside the MicroVM.

Both are achievable today, entirely from load-balancing and networking primitives. There is no Amazon CloudFront distribution and no compute in the request path. All you need is an Application Load Balancer (ALB) that terminates TLS with your AWS Certificate Manager (ACM) certificate, rewrites the Host header, and forwards the request over AWS PrivateLink. In this post you’ll deploy that pattern with the AWS Cloud Development Kit (AWS CDK), map a wildcard of custom domains onto your MicroVMs, and let the ALB handle CORS for you.

The complete, deployable example is available as a pattern on Serverless Land. This walkthrough centers on the reusable networking pattern. The sample also includes a small demo application that provisions a MicroVM and mints an access token, which we reference but do not detail here.

What you’ll build

By the end you’ll have:

A wildcard custom domain like *.microvms.example.com, where each <uuid>.microvms.example.com maps transparently to the corresponding MicroVM.
An internet-facing ALB that rewrites the incoming request’s Host header to the real MicroVM endpoint and forwards requests to it privately over PrivateLink.
CORS preflight and response headers handled at the ALB, with no change to the code running in the MicroVM.

Calling https://<uuid>.microvms.example.com/<path> (with the MicroVM access headers described later) reaches the right MicroVM, with your domain intact end to end.

Solution overview

The request flow looks like this:

Request flow from a browser through the Application Load Balancer, which terminates TLS and rewrites the Host header, then forwards over AWS PrivateLink to the Lambda MicroVM service.

The key component is the ALB host header rewrite, introduced in URL and host header rewrite for Application Load Balancers. A listener rule matches the incoming custom host with a regex condition, captures the MicroVM ID from the left-most label, and a host-header-rewrite transform rewrites the Host header to <uuid>.lambda-microvm.<region>.on.aws before forwarding. Because the MicroVM service front-end routes on the Host header, the request lands on the correct MicroVM, while the customer’s domain stays in the browser’s address bar the whole time.

Why not CloudFront? Why not an ALB redirect?

CloudFront can also rewrite Host/SNI toward the origin, but a single distribution has static origins. Mapping a wildcard of MicroVM IDs through one distribution would require a CloudFront Function to compute the origin per request. The ALB transform performs the same rewrite for the entire wildcard with zero code.
An ALB redirect action only issues an HTTP 301 Moved Permanently/302 Found response. The browser would follow it, and the address bar would then show the .on.aws URL, which breaks our design as it is not a real custom domain. The transform (not a redirect) is what makes the custom domain transparent.

Walkthrough

The example is an AWS CDK application. Configuration lives under the microvm-custom-domains key in cdk.json (hosted zone, wildcard base, the endpoint base to rewrite to, the PrivateLink service name, and the CORS origin). Set those values, then deploy. The sections below explain what the stack creates and why.

Prerequisites

An AWS account.
An existing Amazon Route 53 public hosted zone for your domain (for example, example.com). The stack imports it and adds a wildcard record.
Node.js v22 or higher and the AWS CDK v2.
Credentials for the target account with permission to create Lambda, Amazon Virtual Private Cloud (VPC), Elastic Load Balancing, AWS Certificate Manager (ACM), and Route 53 resources through AWS CloudFormation.

1. Create the network and the PrivateLink endpoint

A small VPC (two Availability Zones, which is the minimum for an internet-facing ALB) hosts the ALB and an interface VPC endpoint to the AWS managed MicroVM service. There are no NAT gateways, because nothing here needs egress, which keeps the footprint lean.

// Interface (PrivateLink) endpoint to the AWS managed MicroVM service.
const endpoint = new ec2.InterfaceVpcEndpoint(this, 'MicroVmEndpoint', {
  vpc,
  service: new ec2.InterfaceVpcEndpointService(cfg.microvmVpceServiceName, 443),
  subnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
});

2. Discover the endpoint’s private IP addresses at deploy time

An ALB IP target group needs the private ENI IP addresses of the interface endpoint (one per Availability Zone). CloudFormation does not expose those IPs as a usable attribute, so the stack resolves them during deployment with an AwsCustomResource that reads the endpoint’s own ENIs by ID (DescribeNetworkInterfaces on vpcEndpointNetworkInterfaceIds).

This is the only compute the package deploys, it runs only during cdk deploy, and it is never in the request path.

3. Request a wildcard TLS certificate

ACM issues a DNS-validated wildcard certificate for *.microvms.example.com, validated through the hosted zone you imported. The ALB presents this certificate for every custom domain under the wildcard.

4. Create the ALB and the MicroVM target group

The internet-facing ALB has an HTTPS:443 listener using the wildcard certificate. The target group holds the endpoint ENI IPs as IP targets, reached over HTTPS:443.

Encrypted in transit. A customer-provided AWS Certificate Manager (ACM) certificate is used to securely terminate encryption between the client and the ALB. The ALB re-originates TLS to the MicroVM service so traffic stays encrypted through the network.
IP-based targets. The target group uses IP-based targets with the local IP addresses of the VPC endpoints.
Health check matcher 200,403,404. The load balancer’s health probes are unauthenticated, so the MicroVM endpoint answers them with 403. A 403 here means “endpoint is reachable,” not “auth is broken,” so the matcher treats it as healthy.

const targetGroup = new elbv2.ApplicationTargetGroup(this, 'MicroVmTargets', {
  vpc,
  protocol: elbv2.ApplicationProtocol.HTTPS,
  port: 443,
  targetType: elbv2.TargetType.IP,
  targets: targetIps.map((ip) => new elbv2t.IpTarget(ip, 443)),
  healthCheck: {
    protocol: elbv2.Protocol.HTTPS,
    path: '/',
    healthyHttpCodes: '200,403,404',
  },
});

const listener = alb.addListener('Https', {
  port: 443,
  protocol: elbv2.ApplicationProtocol.HTTPS,
  certificates: [certificate],
  // Default action for anything that doesn't match our host regex.
  defaultAction: elbv2.ListenerAction.fixedResponse(404, {
    contentType: 'text/plain',
    messageBody: 'Unknown custom domain',
  }),
});

5. Add the host-header rewrite rule

A listener rule matches <uuid>.microvms.example.com with a regex condition and rewrites the Host header to <uuid>.lambda-microvm.<region>.on.aws with a host-header-rewrite transform. The regex captures the left-most label (the MicroVM ID) and reuses it in the replacement.

At the time of writing, the CDK L2 constructs don’t yet model regex host conditions or transforms, so the example reaches the underlying CfnListenerRule to set them:

const escapedBase = customDomainBase.replace(/[.]/g, '\\.');
const matchRegex = `^(.+)\\.${escapedBase}$`;     // capture <uuid>
const replaceWith = `$1.${microvmEndpointBase}`;   // <uuid>.lambda-microvm.<region>.on.aws

const cfnRule = forwardingRule.node.defaultChild as elbv2.CfnListenerRule;

cfnRule.conditions = [{ field: 'host-header', regexValues: [matchRegex] }];

cfnRule.addPropertyOverride('Transforms', [
  {
    Type: 'host-header-rewrite',
    HostHeaderRewriteConfig: { Rewrites: [{ Regex: matchRegex, Replace: replaceWith }] },
  },
]);

6. Point Route 53 at the ALB

Wildcard A and AAAA alias records (*.microvms.example.com) target the ALB, so every MicroVM custom subdomain resolves to it.

7. Deploy

Run the following commands to install the dependencies and then deploy the application.

npm install
npx cdk deploy

Handling CORS at the ALB

If your clients are browsers calling the MicroVM from another origin, CORS is handled entirely at the ALB, with no change to the application inside the MicroVM.

The listener uses ALB header-modification attributes to insert the Access-Control-Allow-* headers on every response. A higher-priority rule answers OPTIONS preflight requests at the edge with a fast 204 response. Otherwise, preflight requests would reach the origin and be rejected without an access token.

// Insert CORS headers on every response on this listener.
const cfnListener = listener.node.defaultChild as elbv2.CfnListener;
cfnListener.addPropertyOverride('ListenerAttributes', [
  { Key: 'routing.http.response.access_control_allow_origin.header_value',  Value: cfg.corsAllowOrigin },
  { Key: 'routing.http.response.access_control_allow_methods.header_value', Value: 'GET,POST,PUT,DELETE,OPTIONS,PATCH,HEAD' },
  { Key: 'routing.http.response.access_control_allow_headers.header_value', Value: 'x-aws-proxy-auth,x-aws-proxy-port,content-type,authorization' },
  { Key: 'routing.http.response.access_control_expose_headers.header_value', Value: 'content-type,content-length' },
  { Key: 'routing.http.response.access_control_max_age.header_value',        Value: '86400' },
]);

// Answer OPTIONS preflights at the ALB.
new elbv2.ApplicationListenerRule(this, 'CorsPreflightRule', {
  listener,
  priority: 10,
  conditions: [elbv2.ListenerCondition.httpRequestMethods(['OPTIONS'])],
  action: elbv2.ListenerAction.fixedResponse(204, { contentType: 'text/plain', messageBody: '' }),
});

Because the ALB adds those headers to both the preflight 204 and the forwarded MicroVM response, a browser’s cross-origin call succeeds without any application change. Set corsAllowOrigin to * for quick testing, and pin it to your own site for anything beyond a demo.

Test it end to end

First, launch a Lambda MicroVM and mint an access token (follow Create your first Lambda MicroVM). When it’s running, the service gives you a generated endpoint that looks like:

012345678-9abc-defg.lambda-microvm.us-east-2.on.aws

To get the custom-domain equivalent, replace the endpoint suffix (.lambda-microvm.<region>.on.aws) with your wildcard base: .microvms.example.com. Everything ahead of that suffix is preserved exactly:

012345678-9abc-defg.microvms.example.com

The ALB’s rewrite rule captures whatever precedes the suffix and re-attaches it to the real endpoint base, so the mapping holds for the entire wildcard. You never register anything per-MicroVM.

With your token in hand, call the custom domain you derived:

curl "https://012345678-9abc-defg.microvms.example.com/<path>" \
  -H "X-aws-proxy-auth: <token>" \
  -H "X-aws-proxy-port: 8080"

The request travels to the ALB, which terminates TLS, rewrites the host header, and forwards over PrivateLink to the MicroVM. The response comes back under your domain.

The reference architecture also includes a single-page demo and a POST /api/provision endpoint that runs or reuses a MicroVM and mints a short-lived token. With it, you can try the flow without wiring up token creation yourself. It even performs this suffix swap for you and hands back a ready-to-click custom-domain URL. See the repository for that piece.

Important considerations

Authentication is still the client’s job. This pattern only rewrites Host. The client must still supply a valid, unexpired access token in X-aws-proxy-auth. This is deliberate. MicroVM tokens are per-MicroVM and short-lived, so baking them into infrastructure would be fragile and insecure.
Region pinning. PrivateLink is regional, so the ALB, the endpoint, and the MicroVM service must all be in the same Region.
Production hardening. If you adapt the sample’s provisioning endpoint, put authentication and rate limiting in front of it, pin CORS to your origin, and scope IAM to the minimum. The sample’s provisioning path is intentionally open for demonstration and is not production-safe as written.
Cost. You pay for the ALB and the interface endpoint (hourly plus data processing) in addition to the Lambda MicroVM usage. There is no CloudFront distribution and no per-request compute in the data path.

Clean up

Run the following command in the same directory where you deployed the application from.

npx cdk destroy

This removes the ALB, target groups, endpoint, certificate, VPC, and Route 53 records created by the stack.

Conclusion

You can front AWS Lambda MicroVMs with customer-owned wildcard custom domains using an Application Load Balancer and AWS PrivateLink. The key is the ALB’s host-header rewrite. Because the MicroVM service routes requests based on the Host header, a single rewrite rule can transparently map an entire wildcard of custom domains onto your MicroVMs. CORS is handled at the edge as well. The whole setup relies only on networking primitives, with no CloudFront distribution and no compute in the request path.

To try it yourself, deploy the reference architecture and review the ALB URL and host header rewrite launch post for more on the transform feature.

[$] Compiling the kernel with gccrs

2026-09-22 daroc

Post Syndicated from daroc original https://lwn.net/Articles/1095553/

Pierre-Emmanuel Patry and Arthur Cohen gave a talk at

RustConf 2026 on the
status of the Rust frontend for GCC (gccrs), with a particular eye toward the
goal of

compiling the Linux kernel. Patry gave a follow-up talk for a more
kernel-focused audience at

Kangrejos the next week, which Cohen could not attend. The

gccrs project is
making good progress overall, but it will still be some time until the compiler
is usable.

Best practices for scaling large consumer groups on Amazon MSK

2026-09-22 Pallavi Jha

Post Syndicated from Pallavi Jha original https://aws.amazon.com/blogs/big-data/best-practices-for-scaling-large-consumer-groups-on-amazon-msk/

When scaling large consumer groups on Amazon Managed Streaming for Apache Kafka (Amazon MSK), a common challenge is managing the size of internal metadata records. During rebalances, Kafka persists a metadata record to the internal __consumer_offsets topic. If the consumer group is large enough, this record can exceed the default 1 MB limit. This causes a RecordTooLargeException and a rebalance retry loop.

In this post, we explain how consumer group metadata grows and how to estimate your metadata size. We provide a step-by-step walkthrough for increasing the topic-level size limit (the most common remediation), along with guidance on three complementary strategies: splitting groups, right-sizing partitions, and optimizing naming conventions. We also discuss capacity planning, monitoring, and how KIP-848 in Apache Kafka 4.0 addresses this constraint at the protocol level.

Prerequisites: This post assumes familiarity with Apache Kafka consumer groups, rebalance protocols, and Amazon MSK cluster configuration. You should have access to the kafka-configs.sh CLI tool or the Amazon MSK console. The configuration approaches described apply to Amazon MSK Provisioned clusters with Standard brokers. The KIP-848 section covers a forward-looking protocol change in Apache Kafka 4.0 that applies across deployment types.

How consumer group metadata grows

The following diagram illustrates how consumer group metadata flows through the system during a rebalance:

Flow of a GroupMetadata record from the Group Coordinator to the internal offsets topic and follower brokers during a rebalance

Figure 1: Consumer group metadata flow during a rebalance

During a rebalance, the Group Coordinator serializes a GroupMetadata record containing information about every member in the group and persists it to the __consumer_offsets topic. This record must fit within the topic’s max.message.bytes limit. Follower brokers must also replicate it, constrained by replica.fetch.max.bytes. For each member, the record includes:

Subscription topics – The list of topics the member subscribes to.
Owned partitions – Partitions currently held by the member.
Assignment – The new partition assignment after rebalancing.
Client ID – The configured client.id.

Apache Kafka’s serialization format repeats topic names multiple times per member: once in subscription, once in ownedPartitions, and once in assignment. The client.id adds further per-member overhead. The broker stores these metadata records uncompressed in __consumer_offsets.

Estimating your metadata record size

You can approximate your consumer group’s metadata record size with the following formula:

record_size ≈ member_count × (3 × avg_topic_name_bytes + 2 × client_id_bytes + ~200 bytes overhead)

Example: A group with 1,000 members, a 50-byte topic name, and a 40-byte client ID:

1,000 × (150 + 80 + 200) = ~430 KB

At 1,500 members with the same parameters: ~645 KB. With multiple topic subscriptions or longer naming conventions, the record can exceed 1 MB well before 2,000 members.

Approaches to handle large consumer group metadata

The following sections describe four strategies for managing large consumer group metadata, starting with the most direct remediation.

Increase max.message.bytes on __consumer_offsets

If your consumer group metadata exceeds 1 MB, you can increase the maximum record size on the internal topic. This is the most direct path to help unblock consumer groups that have already scaled beyond the default. Note that max.message.bytes is the topic-level configuration name, while message.max.bytes is the equivalent broker-level default.

1. Update the topic-level configuration:

kafka-configs.sh --bootstrap-server <bootstrap-server> \
    --entity-type topics \
    --entity-name __consumer_offsets \
    --alter \
    --add-config max.message.bytes=2097152 # topic-level config for max record size

2. Update replica.fetch.max.bytes at the cluster level:

This broker-level setting controls the maximum fetch size for inter-broker replication. Set it equal to or greater than max.message.bytes on __consumer_offsets so follower brokers can replicate large metadata records.

replica.fetch.max.bytes=2097152

You can apply this through the Amazon MSK console under Cluster configuration or using the AWS Command Line Interface (AWS CLI) with update-cluster-configuration.

Amazon MSK console cluster configuration editor with the replica.fetch.max.bytes property set

Figure 2: Setting replica.fetch.max.bytes in the Amazon MSK cluster configuration console

Important: Always make sure that replica.fetch.max.bytes ≥ max.message.bytes for __consumer_offsets. Without this, you might observe UnderReplicatedPartitions on the internal topic.

3. Test in a non-production environment first:

Trigger a consumer group rebalance (restart consumers or scale the group).
Verify no RecordTooLargeException in broker logs.
Confirm broker heap usage and replication lag remain healthy.

Split large consumer groups

Breaking a single large consumer group into multiple smaller groups reduces the per-group metadata record size proportionally. To split a group, deploy multiple connector or consumer instances, each with a distinct group.id, subscribing to the same topic but consuming from a subset of partitions. The system preserves offset tracking within each sub-group independently.

When to use: Consumer group membership is growing unboundedly through Auto Scaling, and you want to keep each group’s metadata well within limits without modifying internal topic configuration.

Trade-off: Increases operational complexity. You have multiple groups to monitor and manage instead of one.

Right-size partition count and auto scaling bounds

Over-partitioned topics require more consumers to fully parallelize, which inflates group membership. Unbounded auto scaling policies can grow consumer groups beyond what was originally planned.

Review whether your topic’s partition count matches your actual throughput requirements.
Configure auto scaling policies with an upper bound on consumer replicas (for example, Horizontal Pod Autoscaler on Amazon Elastic Kubernetes Service (Amazon EKS)).
Align partition count with the maximum number of consumers you intend to support.

This is a proactive measure, best applied during topic design and capacity planning to prevent the metadata size issue from occurring in the first place.

Optimize naming conventions

Consumer group names and client IDs contribute to the overall metadata size. Shorter, standardized naming reduces per-member overhead.

Considerations: Changing an active consumer group’s name means the new group starts with no committed offsets and all tracking history is lost. For this reason, naming optimization is most practical for new deployments rather than existing production groups.

Capacity planning for larger metadata records

When you increase max.message.bytes on __consumer_offsets, larger metadata records consume more broker heap during rebalance processing. Proper capacity planning helps you select the right broker instance type and configuration value before hitting production issues.

Planning steps:

Calculate your current record size using: member_count × (3 × topic_name_bytes + 2 × client_id_bytes + ~200).
Project peak membership based on your auto scaling upper bound (maximum consumer replicas × number of tasks per connector, if using Kafka Connect).
Apply a 2× safety margin to account for protocol overhead, multi-topic subscriptions, and burst scaling events.
Select your max.message.bytes value from the following guidance table.
Choose your broker instance type based on heap requirements. Larger metadata records increase heap pressure during rebalances. For groups exceeding 1,000 members with 2+ MB metadata records, use kafka.m5.xlarge or larger to provide sufficient heap headroom.
Validate in non-production by running a consumer group at projected peak membership and monitoring HeapMemoryAfterGC during rebalances.

The following table provides sizing guidance based on consumer group size:

Consumer Group Size	Guidance
< 500 members	Default 1 MB is typically sufficient. `kafka.m5.large` or larger.
500–1,000 members	Monitor metadata size. Consider increasing to 2 MB. `kafka.m5.xlarge` or larger.
1,000–2,000 members	Increase to 2–5 MB. `kafka.m5.2xlarge` or larger for adequate heap headroom.
> 2,000 members	Combine increased limit with group splitting. `kafka.m5.2xlarge` minimum. Consider `kafka.m5.4xlarge` for high rebalance frequency.

Key metrics to monitor

The following Amazon CloudWatch metrics help you track consumer group metadata health:

Metric	What it tells you
`HeapMemoryAfterGC` (Amazon CloudWatch)	Percentage of heap memory in use after garbage collection. Indicates memory pressure from larger metadata records during rebalances.
`UnderReplicatedPartitions` (Amazon CloudWatch)	Replication health. Non-zero may indicate `replica.fetch.max.bytes` is too low.
GC pause duration (broker logs)	Prolonged GC can trigger session timeouts and cascading rebalances.
Consumer group rebalance rate	Stable groups should not rebalance frequently after configuration changes.
Consumer lag	Confirms consumers are making progress after rebalances complete.

We recommend creating two Amazon CloudWatch alarms for HeapMemoryAfterGC. Set a warning alarm at 60% to indicate potential performance degradation. Set a critical alarm at 80 percent, at which point you should scale brokers or reduce consumer group size. For UnderReplicatedPartitions, alarm at any value> 0 sustained for more than 5 minutes after a configuration change.

Looking ahead: KIP-848 and Apache Kafka 4.0

Apache Kafka 4.0 (released March 2025) adopted KIP-848 as the default consumer protocol. The broker now computes partition assignments server-side rather than delegating to a consumer group leader. Because each member no longer carries full subscription and assignment data on the wire, the new protocol reduces per-member metadata size. For details on the protocol changes that achieve this reduction, see the KIP-848 design document. KIP-848 also introduces incremental rebalances.

Newer Apache Kafka versions on Amazon MSK bring smaller metadata records by default. The following steps help you prepare for KIP-848 adoption:

Track Amazon MSK version support for Apache Kafka 4.0+.
Verify your Kafka client libraries support the new consumer protocol.
Test the new protocol in a non-production environment before migrating production consumer groups.
Plan for a phased rollout, starting with non-critical consumer groups.

Conclusion

The following table summarizes when to apply each approach. The max.message.bytes increase (covered step-by-step earlier) is the primary remediation. The other strategies are complementary guidance you can adapt to your environment:

Situation	Recommended approach
Already hitting `RecordTooLargeException` in production	Increase `max.message.bytes` on `__consumer_offsets` + set `replica.fetch.max.bytes` accordingly
Planning for growth	Right-size partitions, set auto scaling bounds, monitor metadata size
Naming overhead is significant	Optimize naming conventions for new deployments
Operating at very large scale (2,000+ members)	Combine increased limits with consumer group splitting
Long-term architecture	Plan migration path to KIP-848 (Apache Kafka 4.0)

Test configuration changes in non-production first, monitor broker metrics during and after rebalances, and scale incrementally. With these practices in place, you can operate consumer groups at the scale your streaming workloads require.

To get started:

Review the Amazon MSK Developer Guide for cluster configuration steps.
Use the sizing formula in this post to estimate your current metadata record size.
Set up Amazon CloudWatch alarms on HeapMemoryAfterGC to monitor broker health proactively.

About the authors

How Moeve standardized dbt runs across data lakes with Amazon Athena

2026-09-22 Rubén Romero Córdoba

Post Syndicated from Rubén Romero Córdoba original https://aws.amazon.com/blogs/big-data/how-moeve-standardized-dbt-runs-across-data-lakes-with-amazon-athena/

As organizations grow, data processing often becomes fragmented across teams, environments, and orchestration tools. This fragmentation leads to inconsistent patterns, duplicated logic, limited cost visibility, and operational overhead.

At Moeve we were no exception. Our analytics teams build their transformations with dbt, an open source tool that defines transformations as SQL models, resolves the references between them, and works out the order in which they run. dbt describes what to transform, but it does not define where or how a project runs. We left that decision to each team, and as the number of projects grew we found fragmented pipelines, inconsistent compute engines, and limited cost visibility slowing every project down. Standardizing our dbt runs on Amazon Athena was how we worked our way out of that.

This post describes the architecture of the centralized, serverless solution we built on Athena, which reduced onboarding for a new dbt project from days to about 15 minutes. The solution centralizes how dbt runs across our data lakes while staying loosely coupled from orchestration. It uses Amazon Athena as the default processing engine, a centralized dbt launcher, and a shared event bus for downstream orchestration.

In the sections that follow we explain how we decoupled dbt runs from orchestration using AWS Step Functions and AWS Fargate, why Athena fits our workloads from a cost and operational perspective, how storing run parameters in Amazon DynamoDB rather than in pipeline code removed the infrastructure deployment step from onboarding, and how publishing results to Amazon EventBridge lets our run and orchestration layers evolve independently.

The challenge of running dbt at scale

Before the dbt launcher, our dbt runs had grown in different directions. Run logic was embedded in project-specific pipelines, orchestration and processing were tightly coupled, teams selected different compute engines for comparable workloads, and we had no consistent governance over run parameters and retries.

As the number of dbt projects increased, this made it difficult to enforce consistent standards and to evolve the solution without touching every pipeline. We needed a way to standardize dbt runs across our data lakes, decouple running a project from deciding what to run, improve cost control and observability, and deliver faster and safer continuous integration and continuous delivery (CI/CD) iterations.

Why Amazon Athena as the default dbt engine

Choosing the processing engine for dbt is a foundational architectural decision, so we made it first.

Serverless processing

Athena is fully serverless. There are no clusters to provision, scale, or maintain. Our teams run their queries with the default Athena pricing, which charges for the data a query scans and gives us elasticity with no capacity planning.

Because each data lake lives in its own account, each team makes its own decision about Athena payment. A team whose workload grows into continuous, high-concurrency usage can move to Athena capacity reservations with no change to the launcher, to their dbt profiles, or to their project configuration. None of our teams have needed to do so yet, and the architecture keeps that choice independent per team.

Our workload is predictable but not continuous. Each dbt project runs for a few minutes when its schedule fires or when its upstream data lands, then stays idle until the next trigger. That shape is what made serverless the right fit for us.

Why Athena fit our solution

For Moeve, the decision to standardize on dbt and Amazon Athena was driven by our goal of creating a common transformation solution that could be adopted across multiple teams and AWS accounts while keeping operations lightweight.

Our data was already stored in Amazon S3 and registered in the AWS Glue Data Catalog, making Athena a natural processing layer. Athena allowed us to run transformations without managing clusters, capacity, or infrastructure, which was particularly important for a small central team supporting multiple domains.

We evaluated alternative processing engines, but for our workload profile and data volumes, Athena provided the best balance between scalability, operational simplicity, and maintainability. Adapter maturity was another important factor. The dbt Athena adapter offered strong integration with testing, CI/CD workflows, and the broader dbt ecosystem, reducing the operational risk of maintaining custom solutions.

Athena also aligned naturally with our architecture. Transformations run in the AWS account that owns the data through cross-account role assumption, while orchestration remains centralized. As a result, we standardized how projects run across teams while keeping compute close to the data.

Finally, the Apache Iceberg support in Athena underpins the idempotent incremental processing model described in this post, so incremental loads and historical reprocessing follow the same path with minimal operational overhead.

Optimized incremental processing

Our largest cost was not reading source data. It was merging into it.

Our fact tables are Apache Iceberg tables in the data lake, and most of our models are incremental. Each run brings in new or corrected records and merges them into a target that can hold several years of history. A merge has to locate the rows it is about to update, and without a predicate on the target the query reads far more of the table than the incoming data can affect. The common approach is a static filter such as the last 30 days, which is wrong in both directions: too wide for an ordinary daily load, and too narrow as soon as a correction arrives for an older partition.

Instead of a fixed window, the platform derives the predicate from the data. Before the merge runs, it reads the distinct values of the partition column present in the incoming dataset and builds the target predicate from them. For a single partition it applies an equality predicate, for a small set an IN list, and for a larger set a bounded range. The merge then reads only the partitions the incoming data can affect.

The same principle applies on the source side. The launcher builds the source filter from the parameters given for that run: an explicit range, an arbitrary SQL condition, or, when neither is supplied, a default window taken from the project configuration. Input is therefore bounded to the subset each run needs.

Two results mattered to us. Because Athena charges for the data a query scans, narrowing both ends of the merge reduces cost without any team hand-tuning individual models. More importantly, a daily load and a full historical reprocess became the same operation with different inputs. Every run is idempotent, so the same input always produces the same result regardless of how many times it runs. That removed the distinction between processing and reprocessing from our runbooks and simplified incident response.

Table design is what makes this pruning possible. Partitioning, columnar formats, and compression all contribute, and the AWS Big Data Blog post Top 10 performance tuning tips for Amazon Athena covers the general techniques. We have deliberately not published a before and after figure here, because the saving depends so heavily on partition design and data distribution that a single number would mislead without extensive context.

Architecture overview

Moeve built a centralized dbt launcher that runs dbt jobs in a uniform way, regardless of the project or the target data lake.

Architecture diagram

Architecture of the centralized dbt launcher running cross-account dbt jobs on Step Functions, Fargate, and Amazon Athena

Figure 1: Centralized dbt launcher and cross-account run flow

At a high level, the architecture consists of:

AWS Step Functions to control the run lifecycle.
AWS Fargate to run dbt in an isolated, ephemeral container.
Amazon Athena as the default dbt processing engine.
Amazon DynamoDB to store dbt project configuration.
Amazon EventBridge to publish run results.

Every dbt run follows the same contract, which gives us consistency and reduces the operational surface we must maintain.

Cross-account processing model

The solution operates in a centralized account while running transformations in domain-specific data lake accounts: corporate, marketing, and manufacturing. The Fargate container assumes a dbt-child IAM role in the target account, so the container processes data where it lives while governance stays centralized.

Each data lake account keeps control of its own IAM permissions and manages its own storage and catalog without affecting the solution. This also puts costs in the right account. We could have attributed Athena spend using Athena workgroups, but Athena is only part of what a query costs. The Amazon Simple Storage Service (Amazon S3) requests it makes and the AWS Key Management Service (AWS KMS) operations it triggers are real costs as well, and running in the owning account attributes all of them to the team that owns the data, per project and per run.

Centralizing dbt runs with the dbt launcher

To stop every team inventing its own way of running dbt, we built a single launcher that all of them go through. Instead of dbt logic living inside multiple pipelines, every run is triggered through one well-defined path.

Run lifecycle

The launcher is an AWS Step Functions state machine. It receives a run request with its parameters, starts an AWS Fargate task from our dbt container image, and the container assumes the IAM role of the target data lake account. dbt then runs its SQL transformations in Athena, reading the source tables and materializing the targets. Alongside the run, Elementary, an open source dbt package, records model-level results and data quality test outcomes. When the run finishes, the launcher publishes a completion event to Amazon EventBridge.

The state machine can be started in different ways depending on the scenario. Some projects run on a schedule, others are triggered when upstream data lands, and teams can request a run on demand. Those decisions are made by our orchestration layer, which submits a standardized run request to the launcher. The launcher therefore stays focused on running dbt projects, regardless of how the run was initiated.

This sequence is identical for all projects and environments. The launcher is responsible only for running the project it was asked to run. It doesn’t decide what should run next, and that responsibility is intentionally delegated to downstream consumers through Amazon EventBridge.

Configuration-driven runs with Amazon DynamoDB

We had to decide where a project’s run parameters would live. In the pipeline definition, changing a timeout would be a code change, a review, a build, and a deployment, for a value we sometimes need to change while an incident is open. We put the parameters in DynamoDB instead, keyed by project, and the launcher reads them at the start of every run. That lets us decouple code deployment from run behavior, update parameters without redeploying services, and enforce consistent defaults across all dbt projects.

The parameters themselves are modest. They cover the target environment and AWS Identity and Access Management (IAM) role, the default processing engine, how long to allow a run to take, how many times to retry on failure, and how many days of data to process by default. These values change for operational reasons rather than logical ones, which is why we did not want them coupled to a release cycle.

The effect on onboarding was larger than we expected. Deploying a new dbt project is now a merge of the dbt models and one configuration entry. There is no Terraform change, no infrastructure review, and nothing to provision, because the compute the project needs already exists and is shared. What used to be a multi-step infrastructure pipeline is now a single CI workflow that validates the project’s SQL and lineage locally, then merges and registers it. We run that local validation with DuckDB, which returns feedback in under two minutes without consuming cloud resources.

Governance did not weaken as a result. Who may change a configuration entry is controlled the same way as any other production change. What changed is that the change no longer has to travel through an infrastructure deployment to take effect.

CI/CD pipeline diagram

A pull request triggers local validation of the project’s SQL and lineage. On merge, the dbt models are deployed and the project’s configuration entry is registered in DynamoDB, after which the launcher can run the project.

Figure 2: CI workflow for onboarding and updating a dbt project

Athena remains the engine of record. Local validation catches Jinja errors, unresolved references, and obvious SQL mistakes, but Athena-specific behavior, cross-account permissions, and AWS Glue Data Catalog interactions are only proven in the target environment. It’s important to be explicit about that boundary with the teams, so that a green CI run is not read as a guarantee.

Publishing run results with Amazon EventBridge

After a dbt run finishes, the launcher publishes a structured event to a central Amazon EventBridge bus recording whether the run succeeded, which project and source it covered, when it started and finished, how long it took, and which datasets it updated. The launcher does not know which consumers are subscribed.

This event-driven approach gives us loose coupling between running a project and orchestrating what comes next, multiple downstream consumers for the same signal, and independent evolution of both layers.

At Moeve the main consumer is our orchestration layer, which models the dependencies between datasets as a graph. Each completion event tells it that a node is now up to date, so it can determine which downstream projects have all their inputs ready and start them. That consumer has no special status. An AWS Lambda function, a monitoring dashboard, or a notification integration can subscribe to the same events without any change to the launcher.

Validation and observability

Because every project follows the same lifecycle, we get validation and observability in one place instead of per pipeline. Step Functions shows the state of any run and the step at which it failed, and error handling and retries are defined once. Fargate logs carry the container runtime detail. dbt and Elementary report model-level results and data quality test outcomes. The completion event on the bus is the auditable record that a project finished and what it produced. Together these layers give us operational visibility without coupling the components to each other.

Cost control and resource cleanup

The platform is serverless end to end, which keeps idle cost close to zero and removes a class of operational mistake. Fargate tasks are created for a run and destroyed when it ends, so no long-running container needs maintenance. Athena has no persistent compute. An idle Step Functions state machine costs nothing. Nothing is left running unintentionally, which matters when the number of projects on the platform keeps growing.

Results

The metrics in the following table are the ones our teams notice day to day, and the reason the solution is maintainable by a small central team.

Metric	Before	After
Onboarding time for a new dbt project	Days, including pipeline and infrastructure setup	About 15 minutes, configuration only
Run consistency	Varied by team	Same lifecycle and contract for every project
CI feedback time	8 to 10 minutes on Jenkins	Under 2 minutes with local validation
Cost visibility	Per-account aggregate	Per-project and per-run attribution
Operational overhead	One pipeline per project	One solution for all projects

Conclusion

Standardizing how we run dbt turned out to depend less on dbt than on defining two boundaries clearly.

The first is the contract of the launcher: parameters in, event out. We defined that interface before building the internals, and it has stayed stable while the implementation changed several times. The second is the separation between running a project and deciding what to run next. Making the launcher publish events without knowing its consumers is why our orchestration layer could be rebuilt while the launcher stayed as it was, and the launcher has never been modified to accommodate a new orchestration requirement.

Two smaller decisions carried more weight than we expected. Keeping run parameters in DynamoDB rather than in pipeline definitions means timeouts, retries, and engine selection can be changed without a deployment, which is valuable during incident response. Making every run idempotent removed the distinction between processing and reprocessing, so a daily load and a full historical reprocess are the same operation with different inputs.

Athena is serverless, so our teams did not need to set up infrastructure of their own. That is what made it practical for everyone to run dbt in a standardized way, and why onboarding a new project went from days to about 15 minutes.

If your organization runs dbt across multiple accounts and teams, pair Amazon Athena with a clear contract for the component that runs your projects: fixed parameters in, a published event out.

Resources

About the authors

Security updates for Tuesday

2026-09-22 jake

Post Syndicated from jake original https://lwn.net/Articles/1096022/

Security updates have been issued by AlmaLinux (apr-util, corosync, curl, freerdp, gstreamer1-plugins-base, libarchive, libtiff, libxml2, openexr, openssh, rsyslog, sudo, tomcat, unbound, webkit2gtk3, yggdrasil, and yggdrasil-worker-package-manager), Debian (chromium), Fedora (alsa-plugins, amarok, aqualung, atomes, attract-mode, audacious-plugins, audacity, baresip, blender, calibre, cantata, cef, chromaprint, chromium, digikam, doctl, dragon, ffmpeg, ffmpegthumbnailer, ffmpegthumbs, ffms2, fooyin, glaxnimate, goldendict-ng, gpac, gstreamer1-plugin-libav, guacamole-server, guvcview, haruna, hedgewars, icecat, janus, k3b, kdenlive, kf5-kfilemetadata, kf6-kfilemetadata, kpipewire, lazygal, lego, libcamera-apps, libheif, libopenshot, libopenshot-audio, libvncserver, localsearch, mat2, minidlna, mivisionx, mixxx, mlt, monado, mpd, mpv, mpv-mpris, neatvnc, notcurses, nv-codec-headers13.0, obs-studio, obs-studio-plugin-droidcam, obs-studio-plugin-pwvideo, obs-studio-plugin-vaapi, obs-studio-plugin-vkcapture, obs-studio-plugin-webkitgtk, olive, openal-soft, OpenBoard, opencv, openmw, opustags, os-autoinst, patool, Pencil2D, perl-HTML-FormHandler, pianobar, prometheus-podman-exporter, python-audioread, python-torchaudio, python-torchvision, qmmp, qmmp-plugin-pack, qmplay2, qt5-qtwebengine, qt6-qtmultimedia, qt6-qtwebengine, qtox, retroarch, rocdecode, rocdecode7.2, rsgain, siril, squeezelite, swayimg, tigervnc, timg, unpaper, vlc, vtk, waypipe, wf-recorder, wivrn, wxsvg, xine-lib, xmms2, xpra, xscreensaver, yle-dl, znc, and znc-clientbuffer), Mageia (nmap, pcre2, and vim), Oracle (curl, openssl-fips-provider, sudo, tomcat, webkit2gtk3, yggdrasil, and yggdrasil-worker-package-manager), Slackware (util-linux), SUSE (cadvisor, chromium, coredns, fake-gcs-server, freeciv, gh, glibc, google-guest-agent, google-osconfig-agent, hugo, kbd, kbfs, keybase-client, libheif, libpcap, mbedtls, pcre2, python-asteval, python-jwcrypto, python311, python313-ansi2html, shadowsocks-rust, sofia-sip, and trivy), and Ubuntu (clamav, expat, ghostscript, glib2.0, gst-plugins-base1.0, gst-plugins-good1.0, libsoup2.4, libsoup3, libssh2, libxml2, linux-azure-6.8, linux-azure-fde, linux-azure-fde, linux-azure-fde-7.0, linux-azure-fde, linux-intel-iotg, linux-kvm, linux-oracle, linux-xilinx-zynqmp, linux-gcp-6.8, linux-ibm, linux-xilinx, linux-ibm, linux-nvidia-bos, linux-raspi, memcached, openjdk-17, openjdk-21, openjdk-25, openjdk-8, openjdk-lts, rsyslog, and strongswan).

NVIDIA Announces DSX Ready Qualification Program for Data Center Power and Cooling Hardware

2026-09-22 Ryan Smith

Post Syndicated from Ryan Smith original https://www.servethehome.com/nvidia-announces-dsx-ready-qualification-program-for-data-center-power-and-cooling-hardware/

NVIDIA this week has launched DSX Ready, a new qualification program for data center cooling and power hardware, for use with constructing AI data centers based on NVIDIA’s DSX AI factory blueprint

The post NVIDIA Announces DSX Ready Qualification Program for Data Center Power and Cooling Hardware appeared first on ServeTheHome.

Красота и катастрофа в „Светлосянка: Експедиция 33“ (втора част)

2026-09-22 Миглена Николчина

Post Syndicated from Миглена Николчина original https://www.toest.bg/krasota-i-katastrofa-v-svetlosyanka-ekspeditsiya-33-vtora-chast/

<< Към първа част

Красота и катастрофа в „Светлосянка: Експедиция 33“ (втора част)

Северина Станкева: Наред с руините, които говорят за отминали епохи и които обсъждахме в края на предишния разговор, има и съвсем буквални отломъци, които се натрупват. Когато героите излизат от града си Люмиер, те се оказват заобиколени от останките на предишните експедиции. Ако в Люмиер изтриването се осъществява много романтично с червените листенца и превръщането в пепел, които спомена, то извън него смъртта има точно обратния вид. Няма изтриване, телата се вкаменяват там, където са загинали. Остават и се трупат едни върху други. Има места, на които играчът трябва да се катери по тях. Един от основните въпроси отново е: какво да правим с това натрупване, с разпада? Как се продължава нататък? Независимо дали разглеждаме разпада като личната история на персонажите (това семейство, чиито връзки са много сложни), или като цивилизационен упадък. Какво да правим с отломките, на които сме свидетели? Да стъпим върху тях, да ги оплакваме, да се бунтуваме? Героите се опитват постоянно да надскочат ситуацията си, да се справят по някакъв нов начин с наследството, което заварват. И в първата половина на играта изглежда, че това наследство е героично преодолимо. В последна сметка се оказва поредното завъртане на все същия омагьосан кръг, повторение на музикалната тема.

Миглена Николчина: Огромна култура е залегнала в правенето на „Светлосянка“. Ще дам още една следа, която ми се струва важна именно в този ракурс. В някои от текстовете на песните, които – както стана ясно – са много красиви, е използван окситански, езикът на Южна Франция, езикът на трубадурите, на куртоазната култура. Това са големите поети на Късното средновековие, от които нишката води към Данте и изобщо към новите европейски литератури. Вкарването на окситански в контекста на играта ми се струва още една податка, че тук ние се занимаваме по някакъв начин с целия обхват на френската култура и цивилизация, с всичко онова, което тя е създала – между окситанската поезия и голямата живопис от първата половина на XX век. В играта подчертано преднамерено са представени всички изкуства. Има например един остров, на който нищо не се случва. Всичко там е бяло и се чува глас, който чете стихотворение. Безплътен, излъчван от пейзажа, превърнат в пейзаж, бял като празен лист…

И всичко това го намираме в руини. Намираме го в красиви, живописни руини, често пъти увиснали в небето, непризнаващи гравитацията, но така или иначе – руини. Последното, което ще добавя към тезата си, е името на града. То не е Париж, въпреки че навсякъде виждаме знакови за Париж сгради и места, включително Айфеловата кула. Името е Люмиер. Люмиер значи „светлина“. Париж е наричан Град на светлината, но също така, в множествено число (което не се чува на френски), това е френската дума за Просвещението. Така че имаме окситанското наследство – началото, Средновековието, трубадурите, които са от Южна Франция, откъдето са и основните създатели на играта. Труповете из „забравени земи“, за които говориш – дали са от Вартоломеевата нощ, от Френската революция, от други сътресения? Забравени са земите. Има го Просвещението в самото име на града. Просвещението е общоевропейски процес, но Париж е люлката му. Имаме този XIX век, чиято опашка хващаме с разпадането на Бел епок, с „гомажа“… Като разбрах, че се нарича гомаж това изчезване, веднага се досетих, че персонажите са нарисувани.

Северина Станкева: Да, това беше първото ти впечатление, което ми сподели – дали героите не са нарисувани.

Красота и катастрофа в „Светлосянка: Експедиция 33“ (втора част) — Кадри от „Светлосянка: Експедиция 33“

Миглена Николчина: Играта доста дълго отлага осъзнаването, че персонажите обитават „света на платното“. Всъщност сюжетът със своята затвореност в семейната драма също би могъл да бъде привлечен към този реквием на френската цивилизация. С такъв сюжет чисто биологически играта изключва продължение. В картината – да! В картината би могло да има продължение, но извън нея като че ли не. Всичко, което по някакъв начин се доближи до това инцестно семейно ядро, е обречено на изчезване.

Северина Станкева: На мен ми се иска да се върнем към темата за разрухата и разпада. Разказът на това семейство за самото себе си се разпада, умножава се в течение на цялата игра. Същинската ѝ част като че ли започва там, където играчът и героите смятат, че трябва да завърши – с победата над художничката. Оказва се, че тя е крепителка, не разрушителка – всъщност бащата Реноар иска да изтрие платното, а обратното броене е нейният начин да отложи колкото е възможно това. След победата експедицията се завръща в Люмиер, макар и изгубила главния си герой Густав, с когото играта започва, но като същински принцип на реалността бащата се появява и изтрива всичко и всички. След като майката (художничката) е прокудена от платното, започва битка между баща и дъщеря. Дъщерята дори написва на заплашителния монолит, на който в началото на играта се помества обратното броене: „Тате, пръждосвай се“ (Papa, va t’en).

От своя страна нарисуваните герои, за които тук едва ли ще остане време да говорим, а също са много интересни, трябва да се справят със статуса си на такива. Включително с обстоятелството, че за тях смъртта не е толкова категорична, колкото са (и сме) смятали. Густав, уж представен като главен герой в началото на играта, е убит още в първо действие – много смел ход сам по себе си; не се сещам друга игра да отстранява протагониста си толкова бързо, отвъд обстоятелството, че смъртта му е първата – на пръв поглед, а всъщност поредната – загуба на брата. Или загуба на поредния брат.

Семейството се множи на страшно много герои. Всеки негов член си създава рисувани образи на останалите – двойници. Двойниците образуват собствени взаимоотношения, копие и оригинал постоянно сменят местата си и това разпръскване е невъзможно да се фиксира около какъвто и да било стабилен център. Остава въпросът откъде произлиза всичко това. Има „реален“ факт, който впоследствие функционира като център – синът е умрял, жертвайки се, за да спаси сестра си в пожар, – но оттам насетне е много трудно на играча да прецени на кого от героите да вярва.

И двата края са изключително противоречиви. Изборът е между гледната точка на рисуван герой, който знае, че всъщност не е „истински“ човек, но застава на страната на реалността, и „истински“ човек на страната на фикцията. Първият (Версо, брат, създаден от майката ) смята, че трябва да се изправим срещу трагедията ребром, което практически значи да изтрием самия него и целия свят на Люмиер. Тоест фикционалният герой иска да изтрие света на фикцията. Реалният човек (Алисия, влязла в платното под алтер егото Маел), за когото този свят е убежище, иска обратното – да изтрие за себе си реалността, да се претопи във фикцията. Всъщност няма обективно верен отговор, той зависи от перспективата, а и двете възможности са ужасно мрачни. Защото всеки от тези герои, макар че застава на страната, на която не би трябвало да застане, с това повтаря точно грешките, които иска да избегне. Истинската героиня се превръща в майка си, става новата художничка повелителка на платното, а изображението на брата действа като самия брат, жертвайки се, за да спаси сестра си, и с това разрушава цял свят.

Миглена Николчина: Няма правилен изход, защото играта показва историческата безизходица, липсата на изход. И това е ужасяващото в нея. И все пак има светлина в края на тунела. Ще го предложа, както го виждам, макар и с доза ирония. Преди всичко не става съвсем ясно защо и от кого е подпален пожарът, в който е загинал братът. Донякъде се разбира, че има гилдия на художниците, към която принадлежат персонажите. Пък имало някаква гилдия на писателите и те взели че подпалили къщата на художниците. В духа на моите изкуствоведски хипотези шегата ми е, че това не е гилдия на писатели в точния смисъл – ако нечий дом е тежко опожарен през последните стотина години, това е домът на поезията – а е гилдия на концептуалистите, които вече не рисуват, а развиват теоретични концепции, към които прикрепят някакви, така да се каже, подпалвачества.

Така или иначе, някой е подпалил дома на Реноар. (Между другото, срещу реалния художник се разгаря някаква доста нелепа кампания в годините преди правенето на играта – движение да бъдат премахнати картините му от музеите.) Но да се върна към завършека на играта. И двата края изглеждат по различен начин мрачни. В единия край братът и сестрата оцеляват, като остават в света на платното. Сюжети за оставане или озоваване във виртуален свят има много, но тук това е свят на живопис, на живописната картина, на живописното платно. Тази разлика много се хареса на колегите ни играчи, които, разбира се, виждат този свят върху монитор. Българската дума живопис тук ми се вижда по-богата на смисъл от френската или английската – тя предполага изобразяването като нещо живо, като изписване на живота…

В другия случай рисуваният брат загива заедно с рисувания свят на Люмиер. Само сестрата се връща при майка си, баща си и другата си сестра в „истинския“ Париж, но Париж е показан като гробище. Братът в реалния свят е мъртъв, сестрата е обезобразена от пожара, не може да говори… Затова, ако бихме могли да гледаме оптимистично на случващото се, аз го намерих в изречение на една от нарисуваните героини, която ще изчезне, ако се избере вариантът с връщане в реалността. Тя казва на Версо, нарисувания брат, че Маел, нарисуваната версия на сестрата, ѝ е обещала – тъй като въпросната героиня никога не е виждала истинския свят – да го нарисува в друго платно в центъра на платното. Тоест има обещание реалността да се появи като картина в картината, в още едно ниво на виртуалност, но това вече да бъде големият реален свят.

За сравнение, когато играх „Принципът на Талос“ (The Talos Principle), направо се скъсах от старание, за да изведа аватара си, който е робот във виртуален свят, към унищожения реален свят. Играта дава опция да останеш в илюзията, наивно вярвайки ѝ, или като един Бодхисатва да останеш съзнателно в нея, но най-трудният финал е да излезеш от илюзорното в грозната действителност. Там обаче това последното го исках. Усещането беше, че има надежда, има работа, която можеш да свършиш. В „реалния“ финал на „Светлосянка“ не виждам такава надежда.

Така че ето какво е моето позитивно заключение. Великото френско изкуство, което правеше Париж център на света, може би вече е в миналото. То не може да се повтори, не може да бъде спасено, както казваш ти, но може да бъде преродено в различни форми, включително във виртуалния свят на игрите, където да продължи съществуването си в тази нова естетическа вселена. Затова за мен игрите са кауза на едно ново изкуство, на нови възможности, които можем да разгръщаме оттук насетне.

Когато с теб проведохме разговор за „Светлосянка“ пред публиката на „През девет земи“, един от слушателите се намеси, за да каже, че е асоциирал името на града с братята Люмиер, които дават начало на кинематографията точно в този период. Един вид, живописта е поставена под въпрос от киното и дилемата е дали сме готови да приемем кинематографията като ново начало, или трябва да браним живописта като такава.

Съгласих се с него. Със сигурност трябва да се има предвид и това. През 1905 г. се случват изненадващо много неща в киното, включително френски филм „Есмералда“, базиран на „Парижката Света Богородица“ от Юго: ето че отсъстващата катедрала се появява при такава връзка! За да превъртим отново времето, Франция е първата страна, в която започва да се пише сериозно за видеоигрите – във френското списание за кино Les Cahiers du Cinéma. Там за първи път (доколкото аз знам) се печата сериозна критика за видеоигри с идеята, че това е ново изкуство. Французите бяха много силни през 90-те години с видеоигрите си, после малко заглъхнаха, а сега, изглежда, се завръщат с пълна сила.

Северина Станкева: Да, този коментар е много уместен, доколкото при видеоигрите се срещат – поне в академичен план, но мисля, че не само – същите стереотипи и нагласи, които навремето са съпътствали появата на киното („как може да говорите за това, това е за панаири, не е изкуство“), а преди това и на фотографията. Това представя една историческа линия на неприемане на новите форми на изкуство, която е ясно проследима и винаги стъпва на едни и същи основания. Иначе в играта има и преминавания от цветно към черно-бяло, които също си струва да се обсъдят, доколкото не следват плътно дихотомията реалност–платно. Има много неща в тази игра – аз само веднъж съм я преполовила, пак ще я играя, вече с оптиката на твоето изкуствоведско тълкуване. И пак ще говорим.

В рубриката „Игромислие“ публикуваме разговори, в които се срещат, съпоставят и противопоставят различни гледни точки към многоизмерния, многожанров феномен на видеоигрите – не толкова като електронен спорт, колкото като нов синтез на изкуствата и като ново поле на общуване и социалност.

We just shipped support for the ugliest part of HTTP: Vary

2026-09-22 Alex Krivit

Post Syndicated from Alex Krivit original https://blog.cloudflare.com/vary-support/

The response header, Vary, has been called “the ugliest part of HTTP that we haven't yet improved.” The same post describes it as a “horrible, kludgy mechanism” with “pretty abysmal interoperability” across intermediaries. That is usually where sensible engineers back away slowly with their hands raised.

That’s not exactly an endorsement of Vary, but ugly doesn’t mean useless.

One URL can have more than one correct response. A server might, for example, deliver different image formats to different browsers. If a cache ignores Vary, it risks serving the wrong bytes to a request. But if it treats every raw header value as distinct, a handful of similar requests can spread into thousands of barely reusable cache entries. Vary tells a cache which request fields may affect the response, but it does not tell the cache which differences actually matter.

Vary support is now available in Cache Rules on every plan. The origin still names the request headers that may affect a response, but you decide how Cloudflare handles each one. You can normalize known negotiation headers, pass exact values through when those small differences matter, or bypass cache when the variation is too unpredictable. The origin declares what may vary, and you decide how much variation is actually meaningful for the cache.

How Vary works

Vary is a standard HTTP response header that tells intermediary caches (like Cloudflare) which request fields may affect the response sent by the origin. Sites use Vary to serve different languages, image formats, compression schemes, or regional content from the same URL.

Take one URL that produces two valid representations. A browser requests a webpage:

The origin returns HTML and identifies Accept as a field that may affect the response:

An API client can request the same URL with a different preference:

This time, the correct response is JSON. The Vary: Accept header tells the cache that the URL alone is not enough to choose between responses. The request’s Accept value must also be considered.

Without Vary, whichever response enters the cache first can be served to both clients. If HTML wins, the API client receives markup and its JSON parser fails. If JSON wins, a browser expecting a web page receives an API response.

Vary prevents the cache from serving the wrong response to the requesting client. But it introduces a harder question: when two requests contain different header values, do they actually need different responses?

When correct caching becomes useless

Vary can tell a cache which request fields may affect a response. It does not tell the cache what the response represents. For example, take an origin that serves content in only English, French, and German. A client might send:

While another client might request:

Both requests prefer English here. The origin’s response may map both requests to exactly the same English response. But a cache comparing the raw values cannot safely assume they are equivalent. They have different orders and language tags (that the origin doesn’t differentiate). So the cache may store them as separate variants, even when their response bodies contain identical bytes.

This is Vary’s central problem. Applications often produce a small, finite set of representations from an enormous set of possible request values. The origin understands that thousands of language preferences collapse into three supported languages, while a cache usually does not.

This problem compounds when a response varies on multiple fields. Ten possible values across one field create ten variants. Ten values across three fields can create 1,000 combinations. Real headers can have far greater cardinality: User-Agent values are numerous, cookies can be unique to individual visitors, and preference headers can differ in ordering, formatting (spaces and tabs matter!), and quality values.

The result is a cache that can be perfectly correct and almost permanently cold (an entry never reused). Identical responses can be scattered across entries that receive too little traffic to remain hot and in cache. They can consume capacity, evict one another, reduce cache hit ratios, and send more requests back to origin servers. Eviction can remove cold entries, but it cannot merge them just because the responses are identical.

An analysis of more than 120 million responses from nearly 50,000 popular sites found almost 3,000 sites varying on four or more fields. Some varied on 10, 23, or even 47 fields. We want to make sure that customers have the tools they need to use Vary when appropriate, but not so much that they create a useless cache.

Some high-cardinality variation is deliberate. CDNs or reverse proxies may inject values, such as a geographic region, to partition content predictably. That works when the possible values are controlled and every component agrees on their meaning. Without those constraints, the cache fragments into variants it may never reuse.

That was the design problem we needed to solve to support Vary. We needed to preserve enough variation to serve the right response, without allowing incidental differences between requests to destroy cache efficiency.

How Cache Rules control Vary

Cloudflare customers already had several ways to handle negotiated content similar to Vary. They could bypass cache and let their origin deal with it, reproduce the origin's negotiation logic in a custom cache key or other rule, use a Worker, or use features like Vary for images.

Those options remain useful, but they either give up caching, duplicate application logic, need to write additional code, or address a narrower use case. Vary in Cache Rules may fill the gap between these existing features by splitting support into two decisions:

The origin uses Vary to identify the request headers that may affect a response.
The Cache Rule determines how Cloudflare handles the value of each header.

A Cache Rule does not force every response to vary. If the origin does not return Vary, Cloudflare caches the response normally, though the rule may still rewrite Accept and Accept-Language before forwarding the request to the origin.

When the origin does return Vary, Cloudflare uses the configured action for each header it names. Headers without an individual setting use the rule’s default action. The three available actions are:

We recommend normalize as the default. For individual headers with personal or unbounded values, use bypass. Use passthrough when the exact value changes the response.

For example, passthrough preserves distinctions in casing, whitespace, ordering, and duplicate values, even when the origin treats them as equivalent. With Vary: X-View and passthrough, these three values produce separate cache keys:

X-View: compact,full

X-View: Compact,full

X-View: compact, full

Enough incidental variation can turn a reusable response into many one-off variants in your cache.

Regardless of the configured actions, Vary: * always bypasses cache. It means any aspect of the request, even information outside the HTTP message (like the client’s IP address), may affect which response the origin selects. Cloudflare therefore cannot reuse the response for a later request without contacting the origin.

How a response moves through cache

Let’s follow one of the /catalog requests from above through Cloudflare.

On the first request, Cloudflare has no stored Vary data for the resource, so the cache lookup misses. The matching Cache Rule can normalize configured fields before Cloudflare contacts the origin.

This can happen before Cloudflare knows whether the eventual response will contain Vary. The Cache Rule defines the permitted normalization; the response later determines whether those fields become part of the cached variant.

That ordering matters. If Cloudflare grouped several raw values under one normalized cache key, but the origin still received those raw values, the origin could produce different responses that the cache would later consider interchangeable. Forwarding the normalized value keeps origin selection aligned with cache matching.

The origin responds with:

Vary: Accept, Accept-Language

Cloudflare records those header names and stores the response as a cached variant. The header values, processed according to the Cache Rule, distinguish this variant from others for the same resource.

When another request for /catalog arrives, Cloudflare starts with the resource’s base cache key: generally the URL plus any other configured key fields. It then reads the stored Vary fields and applies the Cache Rule to those headers in the new request to identify the matching cached variant.

Suppose they normalize to:

Accept: text/html

Accept-Language: en,fr

Cloudflare uses those values to look up the matching cached variant directly. It does not compare the request against every stored variant one by one.

If a matching variant exists and is fresh, the request is a cache hit. If not, Cloudflare sends the request to the origin and may store the resulting response as another variant.

The origin response closes the loop. For each header named in Vary, Cloudflare uses the action configured for that header, or the rule’s default action if the header is not listed individually:

If it does not contain Vary, Cloudflare caches it normally.
If every named header resolves to normalize or passthrough, Cloudflare can store the response as a cached variant.
If any named field uses bypass, Cloudflare does not store the response.
If the response contains Vary: *, Cloudflare does not store it.

This places an important responsibility on the origin. Every cacheable response that can differ based on request fields must return the appropriate Vary header consistently, including errors and fallback responses. If one response omits it, Cloudflare could cache that response without the variance needed to keep it isolated.

The cache keys in the diagram are conceptual. The later request assumes a fresh cached response.

Any purge targeting a cached resource covers all its Vary variants. Existing requirements for purging custom cache keys still apply.

Changing a Vary configuration does not automatically purge existing content. The new policy may produce different cache keys: requests can miss and refill under the new keys, while old entries remain until they expire or are purged.

Normalization keeps equivalent requests together

Remember the requests from above asking for English and French?

Accept-Language: en-US, fr;q=0.8

Accept-Language: fr;q=0.8, en-GB

Both requests prefer English, but passthrough would treat them as different variants. If the Cache Rule allows en, fr, and de, normalize reduces both to en,fr, allowing them to share a cached response.

To do this, Cloudflare lowercases values in Accept, Accept-Language, and Accept-Encoding, then sorts them by quality value, the highest first, with alphabetical ordering to break ties. The client’s ordering therefore does not affect the cache key. After sorting, Cloudflare strips parameters from entries with a nonzero quality value. It can also lose q=0 (“not acceptable”) when shortening language tags or filtering to the configured formats and languages. For example, en-US;q=0 can become en. Use passthrough for Accept or Accept-Language if the origin needs to see those exclusions.

You can also configure the rule to keep only specified media types or languages in Accept and Accept-Language. Regional language tags such as en-US reduce to their base language, en, unless the full tag is configured. This lets you align normalization with the formats and languages your origin actually serves.

To keep origin selection aligned with cache matching, Cloudflare forwards the normalized Accept and Accept-Language values to the origin. It also forwards normalized Accept-Encoding values when Respect Strong ETags is enabled. Other headers are normalized only for cache matching.

Configure Vary in Cache Rules

In the Cloudflare dashboard, go to Caching > Cache Rules, create or edit a rule, make the response eligible for cache, and add the Vary setting. Set the default behavior, then add the headers your origin is expected to name.

The same configuration is available through the Rulesets API in the http_request_cache_settings phase. The default setting chooses a fallback action for headers your origin names in Vary that you have not configured individually.

This example normalizes Accept and Accept-Language to a configured set of formats and languages. The default normalize action also applies to other headers named in Vary:

This is a complete request body for a PUT to the http_request_cache_settings phase entrypoint. A PUT replaces every rule in that entrypoint. If you already have Cache Rules, include them in the rules array or use the appropriate single-rule create or update operation instead.

If the origin serves one representation for each media type and language pair, there are six content combinations. That does not cap the cache at six keys. Preference order, missing headers, and values that normalize to empty can create more. Keep the supported set small and define the rule’s boundaries clearly. After rollout, test the same URL with different header values that should normalize to the same cached variant. Send the test requests from the same client, confirm they return the expected format and language, and inspect CF-Cache-Status. Look for hits once the cache is populated, and investigate persistent miss responses or unexpected bypass responses.

For limitations, additional examples, and how to set this in Terraform, see the Vary documentation.

Why not use a custom cache key?

At this point, an obvious question is, “why not add Accept and Accept-Language to a custom cache key?”

That works when those fields are always part of the resource’s identity. But a custom cache key adds the configured dimensions to every response covered by the rule, whether the origin used them or not.

Vary is response-driven, but cacheable responses under the same base key need a consistent set of Vary fields.

Use a custom cache key when a request property always defines the resource. Use Vary when the origin declares the same set of request fields across cacheable responses. Avoid placing the same header in both unless the duplication is deliberate and tested.

Use Vary in Cache Rules today!

Vary helps solve an obvious problem: one URL can have more than one correct response. But it hands a cache a harder problem, which request differences actually matter? The origin knows which responses it can serve. The cache needs to know which requests can reuse each response.

Vary in Cache Rules connects those two views. The origin identifies the request fields that may affect a response. You decide whether to normalize values, use passthrough for exact differences, or keep the response out of cache.

Vary was never too ugly to be useful. But configuring supported formats and languages manually may not suit every application. We’re evaluating whether ideas from the expired Availability Hints draft could reduce that work by letting origins describe the representations they serve directly.

Vary in Cache Rules is available today on Free, Pro, Business, and Enterprise plans through the Cloudflare dashboard, Rulesets API, and Terraform.

UnitedHealthcare & UnitedHealth Group #lastweektonight

2026-09-22 LastWeekTonight

Post Syndicated from LastWeekTonight original https://www.youtube.com/shorts/dfguT_tvyGs

Introducing Worker Previews: isolated preview environments for every change your agent makes

2026-09-22 Yomna Shousha

Post Syndicated from Yomna Shousha original https://blog.cloudflare.com/worker-previews/

Nothing is worse than testing out a change that works in staging, only to see it behave differently in production. That’s why we wanted to give you an environment that’s as close to production as possible — so you can battle-test your changes and make sure they behave exactly as you expect them to.

Agents are helping us push more lines of code than ever before, and larger changes mean more ground needs to be tested ahead of release. Ideally, that testing is done in a way that doesn’t slow agents down, but gives them the tools to take on more of the development lifecycle.

That’s why today we’re launching Worker Previews. Each Git branch gets a production-like place to run, with its own code, configuration, URL, observability, and state.

So now, for every change in your codebase, you can:

Deploy an isolated Preview with npx wrangler preview, using its own variables, secrets, and bindings, separate from production configuration and traffic.
Share a stable Preview URL for the branch so that every push updates the same running Preview where you can send requests, click through the UI, and test runtime responses.
Isolate Durable Objects and Containers per branch, keeping state changes, sessions, memory, migrations, and concurrent tests scoped to that Preview.
Inspect logs, errors, metrics, and traces for that Preview to confirm the change works, catch failures, push a fix, and verify it before production sees it.
Start from the Preview configuration you set, so each Preview begins with a copy of the variables, secrets, bindings, and settings you define — just like a code branch starts from main. We call this the base configuration.
Override a Preview’s configuration when needed, like pointing it at its own database or test API key for migrations — without changing production, the base, or other Previews’ configuration.
Serve Preview URLs on a custom domain so that auth providers, cookies, cross-origin resource sharing (CORS), and OAuth redirects work the same way they will in production.

The result is a pre-production feedback loop for every branch. Push your change to a branch, test behavior, inspect performance — before you merge to production.

This enables an Agent Development Lifecycle (ADLC) where each change is atomic, independently deployable, observable, and revisable. And it gives agents and humans the evidence they need to self-improve: catch what failed, push a fix, and verify the next deployment before it hits production.

Every Git branch gets its own environment

When you start work on a new feature, the first thing you do is branch off of main. You get your own copy of the code and make your changes without affecting anything in production.

Worker Previews extend that same model beyond code. Each branch gets its own isolated environment and URL. You can run hundreds of Previews at the same time — each operating independently without affecting other Previews or production.

Production and each Preview have their own configuration — served on their own URL.

When you run npx wrangler preview, the branch gets its own copy of your Previews configuration that you have defined, running on its own URL — all under the same Worker.

In the dashboard, this works like switching branches. Click the breadcrumb next to your Worker's name (it defaults to Production) to see all your Previews:

The dashboard brings every environment into one view. Production sits alongside as many Previews as you need, so contributors can work on separate changes without fighting over a shared staging site. Unlike Wrangler environments, where each environment requires deploying and managing a separate Worker, Previews keep that isolation in one dashboard view.

Each Preview runs as a real version of your Worker. Some changes can only be validated at runtime: an API endpoint has to handle a real request and return the right response. More subjective changes, like a UI update, a new onboarding step, or a different error state, need to be experienced in context before they reach production.

Every Preview has its own isolated and persistent state, with Durable Objects and Containers

For isolation to extend across your application, stateful resources need special treatment. The reason for that is that Durable Objects run on a singleton model. One instance is responsible for a given object ID, and that instance owns its storage.

If a Preview shared the same DO namespace as production, you wouldn't just be reading stale data — you could modify the same instance serving live traffic in real time (scary!).

That is why every time you run npx wrangler preview, Cloudflare automatically creates a new Durable Object namespace and Container application for that Preview — so that a failed migration or a bad schema change stays contained to that branch and that branch only.

All you need to do is export the class, add its migration, and access it through ctx.exports:

In production, ctx.exports.Counter resolves to the production namespace, while in a Preview, it resolves to that Preview’s namespace.

You now have an entire playground to experiment with. Take Sandboxes, for example, where milliseconds of improvement to startup time can make or break the experience. If you have been trying to improve cold-start performance, you can run different configurations across branches at the same time, compare their cold and warm performance side by side, and find the best setup faster.

Test, observe, and revise each Preview (or have your agent do it)

Now that each branch runs at its own URL in an isolated environment with its own state, you can enter the feedback loop and start battle-testing every change before it reaches production.

You can send traffic to the Preview URL however you normally would — from your terminal, probe from CI, an agent, or by clicking through it yourself. Once that traffic starts flowing, every Workers Observability tool you’re already used to is available, scoped to each individual Preview.

As each request hits the Preview, Workers Observability traces its full lifecycle in a waterfall, including fetch calls, binding operations, and handler invocations. So when something fails, you can follow exactly what happened without sorting through production traffic or signals from other changes.

Observability for Previews looks just like you're already used to for production Workers. Select your Preview from the breadcrumb and open the Observability tab to see its events, errors, and traces:

To give your agents even more control, you can have them open the Preview URL in a headless browser, click through a login flow step by step, and capture a screenshot or record the entire session as replayable DOM events – with Browser Run.

Below is an example where an agent opens the Preview, captures what was rendered, and connects a failed request to Workers Observability events from the same run.

A reviewer can watch the session in real time with Live View or step in with Human in the Loop when the automation needs judgment.

If something fails, you see it from both angles: what rendered and what happened at runtime.

That gives the agent enough evidence to keep the pre-production loop running autonomously: deploy, open the URL with Playwright MCP, click through, query the traces through the Workers Observability MCP server, patch, redeploy, and verify. Every iteration stays scoped to the branch.

Configure a base configuration for Previews once, then override as needed

Just like you wouldn't reconfigure your code from scratch every time you branch, you shouldn't have to reconfigure your environment either.

You set base configuration for Previews once, in a previews block in your Wrangler configuration file.

In the dashboard under Worker → Settings, you see this inlined as Production and Previews Base. Once the base is set, run npx wrangler preview from any branch to create a Preview. If your Worker is Git-connected through Workers Builds, it happens automatically on push.

You can override any setting for only one Preview — without affecting production, the base, or other Previews.

Preview URLs on your own custom domain, protected with Cloudflare Access

To bring the whole setup even closer to production, your preview URLs can be served from your own custom domain. If your app runs on example.com, a Preview for a login branch could run at feature-login.previews.example.com.

If you want to keep those URLs private, you can protect your Previews with Cloudflare Access and require visitors to sign in first.

Testing the whole system before production

We’ve already been dogfooding Worker Previews inside Cloudflare, most notably to build and test CloudflareOS, our open-source platform for safely connecting agents to company systems.

CloudflareOS lets agents work with services such as Google, GitHub, and Slack through Gatekeepers, which control what those agents can access and change. That makes Gatekeeper changes especially sensitive, because a bug could expose data or permit an action that should never have been allowed.

Some of these bugs only appear when OAuth callbacks, permissions, approval flows, and application state are running together. Because testing each component separately cannot show us how the complete system will behave, we deploy an isolated Preview of CloudflareOS and its Gatekeepers for every change under review. We then run the full workflow, fix what fails, and test it again before merging.

We’re seeing customers use Previews for the same basic reason: some problems only show themselves when the change is actually running.

"Previews gives us the ability to iterate earlier at the edge. For IKEA.com, custom domain support helps us avoid Content Security Policy and cookie issues. We’re especially excited for Service Binding support, which will enable communication between Previews and be a game changer for end-to-end testing across our Worker chain." — Santosh Kumar Dwivedi, Senior Software Engineer, IKEA

"At Supermemory, we use Cloudflare heavily, and Worker Previews are exactly the kind of developer experience improvement we wanted to see. For HTTP flows, we can preview Worker changes before they reach production, including routes backed by Durable Objects, and catch issues earlier without slowing down shipping." — Dhravya Shah, Founder, Supermemory

"Previews is amazing for Inspect [Ramp’s coding agent]. I used it to review and test an Inspect PR on my phone that is making reviewing and testing PRs with Inspect on phones responsive…with Inspect." — Dylan Garcia, Senior Staff Engineer, Ramp

What’s next?

You might be thinking: Didn't Workers already have preview URLs? It’s true, we did. We're now calling those Version URLs because they point to specific uploaded Worker versions. Unlike Worker Previews, they don't create an isolated environment for each branch and could only point to production resources. To learn more and compare the different workflows, check out our docs.

Worker Previews is a big improvement from what we offered before, but there's still more to come. Here's what we're working on next:

Preview multi-Worker applications. Today, a service binding from a Preview still calls the bound Worker's production deployment. We're working toward keeping the entire request path inside matching Previews.
Run Queue consumers and Workflows inside each Preview. Today, Previews can send messages to Queues but cannot consume them, while isolating Workflow executions requires separate configuration. We want the entire asynchronous flow scoped to the branch automatically.
Support long-lived Previews for staging and QA. We've heard from teams in the private beta that not every branch is short-lived — some maintain staging, QA, or per-developer environments that persist across sprints. We want to support these end-to-end, and we want to hear how you use them, so we can get it right.

Worker Previews are available now. Get started with the docs, and if you have a feature request or run into an issue, open an issue on GitHub or join the Cloudflare Developers community on Discord.

Acknowledgements: This project was made possible by the design and implementation efforts of Greg Brimble, Patrick O’Donnell, Matt Price, Korinne Alpers, Max Peterson, Cina Saffary, Josh Wheeler, Thomas Ankcorn, Matt Rothenberg, and Brandon Strittmatter, with leadership from Brendan Irvine-Broque and Dan Carter.

THG Podcast: Counterfactuals – Stainless Steel

2026-09-22 The History Guy: History Deserves to Be Remembered

Post Syndicated from The History Guy: History Deserves to Be Remembered original https://www.youtube.com/watch?v=R9nC1_khCN8

What Can Happen When Meditation Goes Wrong

2026-09-22 The Atlantic

Post Syndicated from The Atlantic original https://www.youtube.com/shorts/NMPcdwhUOIA

GPT-6 Astra Breaks an Old Enigma Message

2026-09-22 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/09/gpt-6-astra-breaks-an-old-enigma-message.html

This is pretty amazing:

However, the most astonishing thing about this break is that the GPT6 Astra did it entirely on its own. Carter Leffer only directed GPT6 Astra to see if it could break any of the unbroken Enigma messages published on the Crypto Cellar Research web page. After analysing the unbroken messages on the website, it decided that the most promising message was Nr. 172, MVUEH and it also quickly suspected that the plaintext of Nr. 173, SIPVX, might be related to the plaintext of the unbroken MVUEH message. After trying many different approaches, GPT6 Astra focused on using the repeated place name ROSENOW ROSENOW as a crib. After developing the necessary Python and C++ software for an Enigma simulator and an Enigma Bombe, GPT6 Astra started a thorough break with the ROSENOW crib, which in the end resulted in the correct key and plaintext for the MVUEH message being found.

We are still analysing the GPT6 Astra logs to see exactly how it executed the break. And we are discovering amazing details.

Join our new UK study on teaching AI ethics and sustainability in lower secondary school

2026-09-22 Diana Kirby

Post Syndicated from Diana Kirby original https://www.raspberrypi.org/blog/join-our-new-uk-study-on-teaching-ai-ethics-and-sustainability-in-lower-secondary-school/

Are you a computing teacher in England, Scotland, or Wales who works with 11- to 14-year-olds and is interested in how young people learn about AI ethics and sustainability?

The Raspberry Pi Computing Education Research Centre is launching an exciting new research study, and we would love you to get involved.

In this study, we will explore teaching about the impacts of AI on people, the environment, and society. Our aims for the study are to:

Investigate the opportunities and challenges of teaching about AI ethics and sustainability in secondary computing lessons
Identify pedagogical approaches for developing students’ ethical reasoning, critical thinking, and agency in relation to AI tools

The study involves attending two in-person workshops in Cambridge, co-designing and teaching a unit of work, and taking part in evaluation activities such as surveys and interviews. Where necessary, we can offer support with the costs of attending the workshops, such as travel, accommodation, and supply cover.

Why focus on AI ethics and sustainability?

According to a recent report, in the UK more than half of 8- to 17-year-olds use AI tools, and as these systems become an increasingly significant part of everyday life, helping young people to critically evaluate their impact is vital.

We recently conducted a scoping literature review to explore what ethical concepts are covered by AI literacy interventions for lower secondary students (11- to 14-year-olds). This work has been accepted for publication at the Frontiers in Education conference, which is taking place in October.

In this work, we used the 10 principles set out in UNESCO’s Recommendation on the Ethics of Artificial Intelligence as a framework for analysis of the ethical concepts covered by the interventions.

Our research showed that when AI ethics is taught to lower secondary students, interventions tend to focus most on concepts such as fairness and privacy. For example, multiple activities explored the important issue of algorithmic bias, teaching students about the causes of bias (such as training AI systems with small or unrepresentative datasets) and discussing real-world examples of bias in AI.

However, we found that other ethical concepts, such as proportionality and governance, are not often taught. And while a few interventions featured the theme of sustainability, we found that only one directly taught students about the environmental impact of AI itself.

Our new study aims to help address this gap by supporting young people to explore the social and ethical impact of AI tools from a sustainability perspective. Teachers will use real-world examples to engage their students in discussion and ethical reasoning. We hope the findings will support more computing teachers to teach about AI ethics in their lessons, and help students to think critically about the use of AI tools.

What does the study involve?

We are using a design-based research approach to work collaboratively with teachers to co-design a unit of lessons focused on AI ethics and sustainability.

In a computing classroom, two young children look at a computer screen.

Participating in the study will involve:

Attending two in-person professional development and design workshops in Cambridge (one in early December 2026 and one during the 2027–28 school year)
Delivering the co-designed lessons in your classroom (in early 2027 and the spring of 2028)
Taking part in evaluation activities such as surveys, classroom observations, and interviews

How can I join the study?

If you teach computing at lower secondary level (Years 7–9 or S1–S3) in England, Scotland, or Wales and would like to help shape the way we teach young people about ethical issues around AI, we would love to hear from you. Please register your interest via the form below.

http://rpf.io/sage-application

If you have any questions about the project, please email [email protected].

The post Join our new UK study on teaching AI ethics and sustainability in lower secondary school appeared first on Raspberry Pi Foundation.