Run Maestro tests on real Android devices using AWS Device Farm

Post Syndicated from Ronald Teijeira Fernandez original https://aws.amazon.com/blogs/devops/run-maestro-tests-on-real-android-devices-using-aws-device-farm/

ManoMano is a European online marketplace for home improvement and gardening products. Their mobile engineering team builds a React Native application that serves millions of customers across Europe. To verify app quality across a wide range of Android devices, ManoMano adopted Maestro for end-to-end (E2E) testing and needed a way to run these tests at scale on real hardware. This post shares how they solved that challenge using AWS Device Farm.

In this post, you learn how to run Maestro mobile UI tests on real Android devices using AWS Device Farm. Maestro is a mobile UI testing framework that uses a declarative YAML syntax to describe test flows. By using the custom test environment in AWS Device Farm, you can install and run Maestro flows on real physical devices in the cloud.

This post is for mobile developers and Quality Assurance (QA) engineers who use Maestro for E2E testing and want to scale their test suites across real Android devices with different brands, screen sizes, and OS versions.

Many teams that adopt React Native prefer Maestro for its YAML-based simplicity and reliable mobile testing capabilities. With the custom test environment feature in AWS Device Farm, you can run any framework that operates over Android Debug Bridge (ADB), including Maestro. This post shows you how to set it up end to end.

Solution overview

AWS Device Farm is a fully managed app testing service that you can use to test your Android and iOS apps on physical devices in the AWS Cloud. It provides officially supported frameworks (Appium, Espresso, and XCTest) with all dependencies and setup pre-configured. It also offers custom test environments that give you full control over the test host. This means you can install and run any framework that operates over ADB, including Maestro. The custom environment provides the following capabilities:

  • An Amazon Linux 2023 host machine with ADB pre-installed, fully cleaned up and not reused between runs.
  • ADB connectivity to the physical device under test.
  • The ability to install additional tooling through shell commands in the test spec from your test package, downloaded from the internet or accessed through private sources within your virtual private cloud (VPC).
  • Live log and video streaming during test execution.
  • Full artifact collection (logs, screenshots, and other test output) from specified directories.

With these capabilities, you can install the Maestro CLI and, if needed, its primary dependency (for example, Java 21) on the test host. You then run your YAML-based test flows against the connected device, the same way you run them locally.

Architecture of the Maestro on AWS Device Farm solution, showing an Amazon Linux 2 test host connecting to a real Android device over ADB to run Maestro flows

Figure 1: Solution architecture

How it works

The solution uses the following workflow:

  1. AWS Device Farm provisions a real Android device and an Amazon Linux 2023 test host with ADB access to the device.
  2. During the install phase, the test spec downloads and installs Amazon Corretto 21 (Java) and the Maestro CLI on the test host.
  3. During the test phase, Maestro connects to the device through ADB and runs your .yaml test flows.
  4. Maestro writes screenshots, video recordings, and test results to the $DEVICEFARM_LOG_DIR directory.
  5. AWS Device Farm collects all artifacts and makes them available for download after the run completes.

Note: This approach currently supports Android devices only. As of 2026, Maestro iOS support is optimized for simulators and does not work with the real physical iOS devices that AWS Device Farm provides (as indicated in the Maestro documentation).

Prerequisites

To complete this walkthrough, you must have the following:

  • An AWS account with access to AWS Device Farm
  • An Android .apk file built for testing (a release build with animations disabled is recommended for stability)
  • Maestro test flows written in YAML (for more information, see Getting started in the Maestro documentation)
  • Familiarity with the AWS Device Farm Console
  • The Maestro driver APKs (dev.mobile.maestro) for uploading as Extra Apps in AWS Device Farm. After running Maestro locally at least once on a connected device, you can extract the installed APKs using adb shell pm path dev.mobile.maestro.

Walkthrough

Complete the following steps to configure and run Maestro tests on AWS Device Farm.

Step 1: Prepare your Android application package

Build your React Native (or native Android) application as an .apk file. Use a release build or a dedicated test build with animations disabled to reduce test flakiness.

Step 2: Package your Maestro test flows

AWS Device Farm expects a test package in .zip format. Use this zip file as a carrier for your Maestro .yaml flows. The following example shows a typical test directory structure:

.maestro/
├── login_flow.yaml
├── checkout_flow.yaml
└── search_flow.yaml

Run the following command to package the directory into a zip file:

zip -r maestro-tests.zip .maestro/

Upload this zip file to AWS Device Farm as your test package. AWS Device Farm extracts it to $DEVICEFARM_TEST_PACKAGE_PATH, which makes your flows accessible during test execution.

Step 3: Configure the test run

In the AWS Device Farm Console, create a test run with the following configuration:

  1. For Choose your application, upload your .apk file.
  2. For Test framework, select Appium Node.js. This selection determines the expected format of the test package you upload. Appium Node.js accepts a generic zip file containing your test scripts and binaries, which makes it suitable for packaging Maestro flows and the Maestro CLI. Other framework options (such as Instrumentation) require framework-specific package formats like compiled jar files.
  3. For Test package, select the zip file that contains your .maestro directory.
  4. For Test spec, replace the default testspec.yml with the custom configuration in Step 4.
  5. For Devices, select a curated device pool or specific Android devices.
  6. For Extra Apps (located under Additional configuration and then Install other apps), upload the Maestro driver APKs (dev.mobile.maestro). AWS Device Farm installs these on the device before the test begins to help you maintain compatibility with restrictive Android skins.
  7. Review your configuration and choose Confirm and start.

Step 4: Configure the test specification

The following test spec instructs AWS Device Farm to download Amazon Corretto 21, install the Maestro CLI, and run your test flows.

In the following code, replace {YOUR_APP_ID} with your application package ID (for example, com.example.myapp). You can find this in your application’s AndroidManifest.xml or by running adb shell pm list packages | grep <your-app-keyword>.

version: 0.1
# Use the Amazon Linux 2023 host environment
android_test_host: amazon_linux_2023

phases:
  install:
    commands:
      # Set locale (important if your app uses special characters)
      - export LANG=en_US.UTF-8
      - export LC_ALL=en_US.UTF-8
      - export LANGUAGE=en_US.UTF-8
      - locale

      # Install Amazon Corretto 21 (required by Maestro)
      - echo "Downloading Amazon Corretto 21..."
      - curl -L -o /tmp/java21.tar.gz https://corretto.aws/downloads/latest/amazon-corretto-21-x64-linux-jdk.tar.gz
      - mkdir -p /tmp/java21
      - tar -xzf /tmp/java21.tar.gz -C /tmp/java21 --strip-components=1
      - export JAVA_HOME=/tmp/java21
      - export PATH=$JAVA_HOME/bin:$PATH
      - java -version

      # Install Maestro CLI
      - curl -Ls "https://get.maestro.mobile.dev" | bash
      - export PATH="$PATH:$HOME/.maestro/bin"

  pre_test:
    commands:
      # Grant background permissions to Maestro driver (for MIUI devices)
      - adb shell appops set dev.mobile.maestro RUN_IN_BACKGROUND allow 2>/dev/null || true
      - adb shell appops set dev.mobile.maestro START_ACTIVITIES_FROM_BACKGROUND allow 2>/dev/null || true
      - maestro --version

  test:
    commands:
      # Run Maestro test flows
      - maestro test -e APP_ID={YOUR_APP_ID} "$DEVICEFARM_TEST_PACKAGE_PATH/.maestro" --test-output-dir=$DEVICEFARM_LOG_DIR/maestro-results

  post_test:
    commands:
      - echo "E2E tests completed"

artifacts:
  # Collect all Maestro output (screenshots, videos, logs)
  - $DEVICEFARM_LOG_DIR

For more information, see Custom test environments in AWS Device Farm.

Key environment variables

The test spec uses the following AWS Device Farm environment variables:

  • $DEVICEFARM_TEST_PACKAGE_PATH: The path where AWS Device Farm extracts your uploaded test package (zip file).
  • $DEVICEFARM_LOG_DIR: The default directory for storing artifacts. AWS Device Farm collects files from this directory after the run.
  • $DEVICEFARM_APP_PATH: The path to your uploaded .apk file on the test host.
  • $DEVICEFARM_DEVICE_NAME: The unique device identifier (UDID) of the device under test.
  • $DEVICEFARM_DEVICE_OS_VERSION: The Android OS version running on the device.

Reviewing test results

After the test run completes, AWS Device Farm displays results organized into three suites:

  • Setup suite: Contains the output from the install and pre_test phases, including Amazon Corretto and Maestro installation.
  • Tests suite: Contains the output from the maestro test command, with pass/fail status for each flow.
  • Teardown suite: Contains the post_test phase output and device cleanup.
AWS Device Farm test run results page organized into Setup, Tests, and Teardown suites

Figure 2: AWS Device Farm test run results showing the Setup, Tests, and Teardown suites

In the Test Specification Log, you can view the live console output. The following example shows typical output during a test run:

[DEVICEFARM] Setting up your device. This usually takes 2-3 minutes.
[DEVICEFARM] ########### Start executing testspec ###########

[DEVICEFARM] ########### Entering phase install ###########

[DeviceFarm] echo "Downloading Amazon Corretto 21..."
Downloading Amazon Corretto 21...
[DeviceFarm] java -version
openjdk version "21.0.11" 2026-04-21 LTS
OpenJDK Runtime Environment Corretto-21.0.11.10.1 (build 21.0.11+10-LTS)
[DeviceFarm] curl -Ls "https://get.maestro.mobile.dev" | bash
Installation was successful!

[DEVICEFARM] ########### Entering phase test ###########

[DeviceFarm] maestro test -e APP_ID=com.manomano "$DEVICEFARM_TEST_PACKAGE_PATH/.maestro" --test-output-dir=$DEVICEFARM_LOG_DIR/maestro-results

Waiting for flows to complete...
[Passed] login_flow (1m 12s)
[Passed] checkout_flow (2m 29s)
[Passed] search_flow (0m 47s)

3/3 Flows Passed

[DEVICEFARM] ########### Finish executing testspec ###########
[DEVICEFARM] Tearing down your device.

To download detailed results, including screenshots and video recordings, open the Files tab and choose Customer Artifacts. For more information, see Downloading artifacts in a custom test environment.

Handling MIUI device installation prompts

Maestro requires a driver app to be installed on the device under test. On devices with restrictive Android skins (such as Xiaomi MIUI and Oppo ColorOS, which are common in the AWS Device Farm device pool), the operating system blocks USB installations. Installation proceeds only after a user accepts a security prompt, as displayed in Figure 3.

MIUI security dialog on an Android device prompting the user to accept a driver app installation

Figure 3: MIUI installation permission dialog

The way to address this scenario is by pre-installing the Maestro driver APKs as Extra Apps in the AWS Device Farm Console. When you upload the Maestro APKs alongside your application, AWS Device Farm installs them on the device before the test begins. You can then grant the necessary background permissions through ADB using appops commands in the pre_test phase:

adb shell appops set dev.mobile.maestro RUN_IN_BACKGROUND allow
adb shell appops set dev.mobile.maestro START_ACTIVITIES_FROM_BACKGROUND allow

These commands tell the OS to trust the Maestro driver app, allowing it to run in the background and start activities without user interaction. This approach removes the need for the UI watcher script and helps make sure that Maestro can start driving your application when the test begins.

Best practices

The following recommendations are based on the AWS Device Farm best practices for custom test environments:

  • Use environment variables instead of absolute paths: AWS Device Farm provides $DEVICEFARM_TEST_PACKAGE_PATH, $DEVICEFARM_LOG_DIR, and other variables that help maintain consistent behavior across test hosts.
  • Direct artifacts to $DEVICEFARM_LOG_DIR: Use the --test-output-dir flag to write Maestro output to this directory. AWS Device Farm collects artifacts stored here automatically.
  • Verify non-zero exit codes on failure: Maestro returns a non-zero exit code when tests fail, which AWS Device Farm uses to mark the run as failed. Verify this behavior with your Maestro version.
  • Do not log sensitive data: Test spec logs are available as downloadable artifacts. Do not echo API keys or secrets in your commands.
  • Disable animations in your test build: Animations are a common source of test flakiness. Disable them at build time or add ADB commands in the pre_test phase.
  • Use Maestro tags for selective test runs: Use --include-tags or --exclude-tags to run subsets of your test suite, such as smoke tests compared with full regression.

Automate with CI/CD

You can automate this workflow by integrating AWS Device Farm into your continuous integration and continuous delivery (CI/CD) pipeline with the AWS Command Line Interface (AWS CLI) or AWS SDKs. The following are the key API operations:

  • CreateUpload: Upload your .apk and test package zip file.
  • ScheduleRun: Start a test run with your test spec, app, and device pool.
  • GetRun: Poll for run completion and retrieve the result status.
  • ListArtifacts: Download test artifacts such as screenshots, videos, and logs.

For GitHub Actions, use the official GitHub Action, which handles uploading your application and test package, scheduling the run, polling for completion, and downloading artifacts. For AWS CodePipeline, see AWS Device Farm test action reference. For other CI/CD tools or services, use the AWS CLI or AWS SDKs with the AWS Device Farm API.

Cost considerations

AWS Device Farm pricing offers two models for automated testing:

  • Pay per device minute: You pay per minute of device usage. A typical Maestro test run takes 5–15 minutes per device, depending on the number of flows.
  • Unlimited testing (flat rate): A monthly flat fee per device slot gives you unlimited testing on one device at a time. This model is cost-effective for teams that run frequent test suites.

The install phase (downloading Java and Maestro) adds approximately 1–2 minutes to each run. To optimize costs, run critical smoke tests on every commit and full regression suites on release branches only. For current pricing details, see the AWS Device Farm pricing page.

Troubleshooting

The following list describes common issues and their solutions.

  • Maestro times out waiting for the app: Verify that your .apk is a release build or that the app starts quickly. Increase the launch timeout by using the appLaunchTimeout parameter in your flow YAML file.
  • Tests fail on MIUI or Xiaomi devices only: Verify that the Maestro driver APKs are uploaded as Extra Apps in AWS Device Farm and that the appops commands in the pre_test phase grant RUN_IN_BACKGROUND and START_ACTIVITIES_FROM_BACKGROUND permissions.
  • Java not found during Maestro run: Verify that the export PATH commands in the install phase are correct. Environment variables that are set in one phase persist to subsequent phases.
  • Artifacts do not appear after the run: Verify that --test-output-dir points to $DEVICEFARM_LOG_DIR or a subdirectory. AWS Device Farm collects only files in paths that are listed under the artifacts section.
  • Test marked as passed despite flow failures: Verify that your Maestro version returns a non-zero exit code on failure. Some older versions might not exit with a non-zero code by default.

Cleaning up

To avoid incurring future charges, delete the following resources that you created during this walkthrough:

  1. In the AWS Device Farm Console, delete the test run and any associated uploads that you no longer need.
  2. If you created a device pool specifically for this walkthrough, delete it from your AWS Device Farm project.
  3. AWS Device Farm does not incur charges when no test runs are active. No additional cleanup is required unless you are using the unlimited testing plan.

Conclusion

By integrating Maestro with the AWS Device Farm custom test environment, ManoMano transformed their mobile testing strategy. They moved from manual validation on a handful of devices to automated end-to-end testing across real physical hardware on every release. This approach delivers:

  • Broader device coverage: Tests run on devices from different manufacturers with varying screen sizes and OS versions, catching device-specific rendering issues and permission dialogs that emulators miss.
  • Faster feedback cycles: Automated test runs complete in minutes rather than the hours required for manual testing across multiple devices.
  • Consistent reliability: Pre-installing the Maestro driver APKs and granting background permissions through ADB eliminates flakiness caused by restrictive Android skins like MIUI.
  • CI/CD integration: Tests run automatically on every pull request using the official AWS Device Farm GitHub Action, preventing regressions from reaching production.

The key enabler is the AWS Device Farm custom test environment. With it, you can run any framework that operates over ADB, without being limited to officially supported test frameworks. This means you can combine the YAML-based test authoring in Maestro with a managed fleet of real devices and extend the pattern to other ADB-compatible tools as your testing needs evolve.

In this post, you learned how to configure the custom test environment, package your Maestro flows, handle manufacturer-specific restrictions, and automate the workflow in your CI/CD pipeline. To get started, create an AWS Device Farm project and try running your existing Maestro flows by following the steps in this post.


About the authors

Ronald Teijeira Fernández

Ronald Teijeira Fernández

Ronald is a Senior Solutions Architect at Amazon Web Services, helping Public Sector customers in Iberia accelerate cloud adoption and maturity. He specializes in development and security, and is an active member of the Next Generation Development Experience and Security Technical Field Communities, where he enables and supports both the community and customers through technical content and guidance.

Jose Maria Elias

Jose Maria Elias

Jose is a Senior Engineering Manager at ManoMano, where he leads the Foundations team responsible for the core architecture of both web and mobile platforms. Recognizing the critical importance of app quality, he focuses on driving mobile testing automation strategies. He specializes in architecting scalable E2E testing pipelines, currently utilizing the Maestro framework and AWS Device Farm to validate mobile applications across diverse device ecosystems.

Pinpointing Location of Videos from Background Airplane Sounds

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/11/pinpointing-location-of-videos-from-background-airplane-sounds.html

This is fascinating:

The video describes a technique that can reveal where an otherwise unidentifiable recording was made by analyzing its audio for the faint sounds of passing aircraft. The technique is both fascinating and frightening.

It works by looking at Doppler curves, which plot the way a plane’s sound changes as it passes overhead because of the Doppler effect. This is the same familiar Doppler effect that causes the frequency of a siren to rise as it approaches and then fall as it passes by and moves away. More generally, it’s the phenomenon whereby the frequency of a wave perceived by an observer changes when the observer and the source of the wave are moving in relation to one another.

[…]

It’s sound that’s relevant here, though, and specifically the sound of aeroplanes. Early in his video, Colsto brings up a map of the flights that were in the air at the time the video he’s tracing was recorded. Much of this information is publicly available. Aircraft equipped with Automatic Dependent Surveillance–Broadcast (ADS-B) transmit data about their position, velocity, and altitude, which flight-tracking services collect and make available to the public. While not every plane is tracked, the system provides an enormous amount of information about aircraft movements.

Security updates for Friday

Post Syndicated from jzb original https://lwn.net/Articles/1105881/

Security updates have been issued by AlmaLinux (cockpit-files, container-tools:rhel8, delve, gcc-toolset-12-binutils, gimp:2.8, glibc, go-toolset:rhel8, kernel, libfastjson, mariadb:10.3, mod_auth_openidc:2.3, mysql:8.0, php:8.0, pki-core:10.6 and pki-deps:10.6, squid:4, tang, tesseract, thunderbird, and virt:rhel and virt-devel:rhel), Debian (chromium, ntfs-3g, puppet-agent, and trapperkeeper-authorization-clojure), Fedora (freerdp, goose, gpsd, hatch, perl-HTTP-Headers-ActionPack, perl-MIME-tools, perl-Plack, perl-Starman, proftpd, python-mistune, ruby-build, rust-crypto-bigint, rust-ctutils, rust-elliptic-curve, rust-num-bigint, rust-num-bigint0.4, rust-num-modular, rust-num-modular0.6, rust-num-prime, rust-zmij, rust-zstd, rust-zstd-safe, rust-zstd-safe7, rust-zstd-sys, rust-zstd0.13, scribus, sofia-sip, tailscale, tor, and wordpress), Oracle (cockpit-files, delve, firefox, gimp:2.8, kernel, and osbuild-composer), Red Hat (delve, git-lfs, go-toolset:rhel8, grafana, ignition, opentelemetry-collector, osbuild-composer, python-pillow, and yggdrasil), Slackware (openssh), SUSE (bind, brotli, cpupower, expat, fuse-overlayfs, go1.26, google-guest-agent, helm, ibus-typing-booster, kernel, kyverno, libmodplug-devel, librsvg, libX11, nvidia-open-driver-G06-signed, owntone, parted, perl-Cpanel-JSON-XS, perl-Text-CSV_XS, php8, python-Brotli, regclient, roundcubemail, rsyslog, and tomcat), and Ubuntu (ca-certificates, frr, haproxy, linux-nvidia-7.0, linux-nvidia-bos, linux-nvidia-tegra, and nginx).

The post-quantum future of web privacy

Post Syndicated from Christopher Patton original https://blog.cloudflare.com/vole-mole/

The highest priorities for the transition to post-quantum cryptography are the basic building blocks of authenticated key-agreement protocols such as TLS. Without these building blocks in place, secure communication on the Internet will not be possible once cryptographically relevant quantum computers come online. However, today we rely on quantum-vulnerable cryptography for much more than TLS.

Privacy Pass is a protocol used by site operators like Cloudflare to help protect applications from abuse. One way we do this is by rate limiting individual clients so that no one client consumes too many of the site's resources. Normally this involves some combination of friction or loss of privacy for users: CAPTCHAs or other challenges to moderate traffic, along with accounts and fingerprinting to identify traffic.Privacy Pass was designed to avoid this tradeoff between friction and privacy. The basic idea is that a client gets issued a number of tokens, which they can redeem in exchange for access to the site. Crucially, each token is unlinkable to the client it was issued to, ensuring the client can use the site as they always had without sacrificing privacy.

With 10 years of deployment experience under our belts — using Privacy Pass for applications such as privacy proxies (like iCloud Private Relay) and relays (like Privacy Gateway) — we, along with others, are collaborating to bring the next generation of this technology to the web. One of our goals is to develop more sophisticated tools for managing agentic traffic that would otherwise consume too many resources and degrade performance for human visitors. Another goal is to foster a more open ecosystem by making it easier for more types of clients to participate. Achieving these goals requires developing new capabilities beyond what Privacy Pass offers today.

Given the trajectory, in this post we look at the prospects for post-quantum. In fact, we're in better shape than one might think. Privacy Pass still relies on classical cryptography, like RSA and elliptic curves. This means that a quantum attacker will be able to forge tokens at will and use them to bypass whatever security mechanism the site operator wants to implement (such as rate limiting). On the other hand, Privacy Pass is unconditionally private, meaning no attacker, quantum or not, can link the redemption of a token to its issuance. What's more, the new capabilities we're developing for this new generation of private authentication technology  can and will be instantiated in a way that upholds the same guarantee: your interaction remains private even after Q-day.

This gives us a little breathing room, but we will need an alternative soon. This is an active area of research that has made significant strides in recent years. Still, just as is the case with other aspects of the transition, the primary challenge is the sheer size of post-quantum alternatives: ML-DSA-44 signatures are a whopping 2,420 bytes, while ECDSA-P256 signatures are just 64 bytes. And there's not much hope for drop-in replacements any time soon. Just as for TLS, we'll have to settle for primitives with higher cost and re-engineer our systems to bear this cost.

But what will those primitives be? In this post, we'll explore one potential future for post-quantum privacy on the web, made possible by a recent breakthrough in the design of blind signatures.

What we need to build: a brief look at MOLE

Let's start with an overview of this new technology, which is presently called MOLE ("Moderation Of unLinkable Endorsements"). We'll have much more to say about MOLE later on, but for now we'll highlight the details that are most relevant to cryptography.

One of the core components of Privacy Pass is a blind signature scheme, which allows a client to obtain a signature under an issuer's secret key without revealing to the issuer the message that it signed. The client can then reveal the message and blind signature later on without linking the signature to its issuance.

One type of Privacy Pass token consists of the message and blindly issued signature. To verify the token, the verifier needs the issuer's public key, and so necessarily learns the identity of the issuer. Thus while these tokens protect the identity of the client, they don't protect the identity of the issuer.

Of course, this is exactly what we want for some use cases. The identity of the issuer tells us something about what type of client is connecting to our edge, a signal we use to make decisions in some of our products and for some of our customers. For instance, a client might use a token issued by one of our customers to prove it is authorized to use the customer's VPN product.

However, for many use cases, this is more information than we actually need to make a decision to grant a client access to a resource. Normally we only care that the token was issued by some entity we trust. In fact, MOLE takes this a step further: the token should hide the identity of the issuer altogether so that we can't treat tokens differently based on who issued them. This guarantees equal treatment regardless of the type of client, making it easier for more clients to participate.

This issuer-hiding property is achieved through a new object called an endorsement. An endorsement is issued by an anchor and redeemed by a moderator. Before redemption, the client and moderator agree on an anchor set, the public keys of the anchors trusted by the moderator. During redemption, the client proves it was issued an endorsement by one of the anchors in the anchor set without revealing which one. Upon acceptance, the moderator issues the client the MOLE credential it will use to moderate the client's interaction with the site.

An endorsement can take many forms, but its key property is that it can be redeemed while hiding the identity of the anchor that issued it. There will also be many types of credentials, but because they're intended to be used by a particular moderator, they needn't be anchor-hiding or even publicly verifiable. This means that everything from blind signatures to anonymous credit tokens or anonymous rate-limited credentials can be used to instantiate MOLE credentials.

Now that we have a sense of what we're building, let's look at our options for post quantum.

A feast of cryptography

When designing cryptographic protocols, adaptability and efficiency are often in tension. Sometimes we know exactly what we want to build and can tailor a construction for that purpose; but the more we tailor the protocol for that use case, the harder it becomes to adapt that construction to a different use case.

Adaptability is essential for a system like MOLE. We're not just building one kind of credential system, but many. And the types of credentials we need are likely to change over time, especially on the web where we must constantly adapt to new traffic patterns and attacker behavior. To keep up, we need a set of building blocks that can be composed together into different protocols to meet different requirements.

Most of the post-quantum cryptographic schemes in wide use today come from larger families of constructions, but have been tuned for their particular application. ML-DSA is a good example here: the cryptographic framework from which this scheme was derived can be used to build a variety of things, but to make ML-DSA as compact as possible, its parameters have been carefully tailored to build an efficient and secure digital signature scheme. However, the same parameters would not necessarily be secure for other use cases.

We have come to view this as the central challenge for post-quantum anonymous credentials: to decouple parameter selection from protocol design. Classical credential systems take this for granted: for the most part, protocol designers don't need to think about the parameters of the elliptic curve they're using, allowing even non-experts in the design of elliptic curve cryptography to build useful things with it. Unfortunately, designing efficient post-quantum credentials still requires an enormous amount of expertise.

This situation seems to have changed, however. Earlier this year, we saw a breakthrough that represents a potential sea change for anonymous credentials.

VOLE-in-the-Head

The roots of this breakthrough go back several years. We'll pick up the story three years ago, with the VOLE-in-the-Head (VOLEitH) zero-knowledge proof system (Baum et al., Crypto 2023). A zero-knowledge proof system allows one to prove knowledge of a hidden witness, satisfying some public statement while keeping the witness private. For example, a zero-knowledge proof can be used to prove knowledge of an AES key K for which Y=AES(K,X), where X,Y  are public. This proof can be transformed into a digital signature whose security reduces to the security of AES, with X,Y as the public key and K as the secret key. This is exactly the FAEST digital signature, a candidate in NIST's ongoing signature competition.

Compared to many contemporary proof systems, VOLEitH is relatively simple to understand and implement securely. VOLE stands for "Vector Oblivious Linear Evaluation" and is an interactive technique that gives two parties correlated random values. These correlations act like authentication tags, allowing the verifier to check that a computation on the hidden witness was performed correctly. In VOLEitH, the prover prepares locally what the two parties would normally generate together — hence “in the head” — then commits to those values and uses the Fiat-Shamir transformation (a widely used tool in cryptography) to select the parts the verifier checks. This step turns the interactive VOLE process into a non-interactive zero-knowledge proof.

Apart from its simplicity, VOLEitH is a fairly unremarkable proof system. It's not succinct in the way many other proof systems are: its proofs are asymptotically linear in the size of the witness and the arithmetic complexity of the statement being proved. However, VOLEitH is concretely very efficient for a particular type of statement, namely one that can be expressed as a set of equations over small binary fields.

One example is the relation Y=AES(K,X) we just discussed. We don't normally think of AES as a set of equations, but many of its computations are designed around arithmetic in GF(28), a field whose elements are single bytes. FAEST proofs for this statement are only about 4KB (NIST Round 3, 128s).

Another example is the KuMQuat signature scheme, in which the signer proves an instance of the Multivariate Quadratic (MQ) problem. Here we're given a system of random, quadratic equations of multiple variables over a small field, like GF(28). Mathematically, this MQ map P looks something like P(s) = (sTA1s + b1s,…,sTAms + bms) where s and each bi are length n vectors and each Ai is an n-by-n matrix. Given P,P(s) for random s, the MQ problem is to find s. We call it quadratic because we multiply the elements of s by itself.

The MQ problem is believed to be hard to solve even for a quantum computer. But because P is made up of quadratic equations, proving knowledge of a solution with VOLEitH is fairly cheap, yielding KuMQuat signatures as small as 2.6KB.

When compared to other post-quantum signatures like ML-DSA or FN-DSA, these numbers don't look especially impressive. But how VOLEitH compares to other schemes in a particular category of constructions is beside the point. For MOLE, we'll need to strike a balance between performance and the ability to build many types of credential systems. To that end, we need one more ingredient.

Multivariate signatures

Another way to build signatures from MQ maps involves embedding some additional mathematical structure into the map. In the Unbalanced Oil and Vinegar (UOV) signature scheme, the public key is an MQ map P, and a signature is a vector s satisfying P(s) = t, where t is obtained by hashing and salting the message to be signed.

The secret key for this signature scheme is a trapdoor for P with which the signer can derive solutions to the MQ equations. The trapdoor reveals to the signer a hidden partitioning of the variables of each quadratic equation into “oil” and “vinegar” variables. Oil variables are never multiplied by one another, so fixing the vinegar values turns the equations into a linear system that the signer can solve for the oil values. In the public key, oil and vinegar variables are mixed together in a way that hides this structure in the map.

UOV is also a candidate in NIST's ongoing signature standardization effort. It has one of the smallest signatures among the candidates, at just 135 bytes. The trade-off is that it has the largest public keys, at 46.6KB. MAYO is another multivariate candidate with much smaller public keys, at the cost of slightly larger signatures.

The key thing to take away here is that the arithmetic complexity of evaluating a UOV (or MAYO) signature is comparable to evaluating an MQ instance. Both are public quadratic maps over small binary fields, making them a good fit for VOLEitH. This brings us to the breakthrough.

Blind signatures from VOLE-in-the-Head and UOV

Earlier this year, a blind signature called PoMFRIT was published (Baum et al., USENIX Security 2026) that achieved what at the time was a record for candidate post-quantum constructions: 6.7KB for the blind signature itself and just a few hundred bytes for issuance. PoMFRIT is based on VOLE-in-the-Head and the MAYO multivariate signature. We can easily drive the size of the signature down to 4.3KB by swapping MAYO out for UOV. (This figure uses NIST round 3 parameters for UOV (uov-Ip) and FAEST-EM-128s.) The trade-off is larger public keys, but this pays off in applications where a public key can be cached and used sufficiently often.

This is competitive with state-of-the-art lattice-based constructions. But the really good news is that building blocks of PoMFRIT can easily be reconfigured into all of the credential systems we've considered for MOLE, including endorsements. More on this in a bit: for now, let's look at how PoMFRIT works in more detail.

A well-known paradigm for designing blind signatures is the following. First, the client places the message in the cryptographic equivalent of a sealed envelope, called a commitment. For our purposes, a commitment is simply the hash of the message together with some randomness generated by the client. To complete issuance, the signer simply signs the commitment directly.

To construct the blind signature, the client produces a zero-knowledge proof of knowledge of the signature and the randomness it used to compute the commitment, called the opening. A verifier would use the signer's public key to verify this proof.

Let's define a bit of notation to make this clearer:

  1. The client generates an opening r and sends a commitment c=H(m,r) to its message m to the issuer.
  2. The issuer produces a signature s=Sign(sk,c) and sends it to the client. It doesn’t learn the message as it’s hidden behind the hash.
  3. Finally, the client produces a proof π of knowledge of s,r for which Verify(pk,H(m,r))=1,

The proof π is the blind signature. It can be verified by any party in possession of the issuer's public key pk and the message m.

Unlinkability comes from the client presenting entirely different objects during the two phases: a commitment is shown to the issuer during issuance, and a zero-knowledge proof is shown to a verifier during presentation. Even if the issuer and verifier collude or are the same entity, they cannot link the sessions.

The size of the proof π is dominated by evaluation of the hash, so we need to choose a hash function that is friendly for our proof system. For most proof systems, such proof-friendly hashes are not well studied, but for VOLEitH it turns out that the Keccak permutation used in SHAKE and TurboSHAKE is a good choice. Keccak’s bitwise operations translate natively into VOLEitH constraints: XOR becomes addition, AND becomes multiplication, and rotations simply rearrange values. (Many popular SNARKs, by contrast, are designed for large prime fields, making byte- and bit-oriented operations less natural to express.)

Nevertheless, Keccak evaluations end up being the largest component of the VOLEitH proof, so we need to make sure we're using only the minimum necessary number of permutations. PoMFRIT requires two: one for the commitment and another for the signature. Concretely, the statement we prove is roughly P(s)=H(H(m,r),rs) where s is the signature, rs is a salt chosen by the issuer, P is the issuer's UOV public key, and H denotes the Keccak permutation with the output truncated to the appropriate length, which also prevents anyone from inverting.

From blind signatures to endorsements and beyond

Recall that for MOLE endorsements, we need something like a blind signature, but one that doesn't reveal the anchor that issued it. That is, we now have a sequence of anchor keys P1,…,PN, and the statement we now want to prove is  Pi(s)=H(H(m,r),rs) for some anchor key Pi whose index i is now part of the hidden witness.

With some care, we can write this as a set of constraints with roughly the same cost: two hashes and one MQ evaluation. We need to prove a bit more about the witness. What exactly? We won't spoil it for you: ZK circuit design is part of the craft of cryptographic engineering that makes it so fun. We'll just say the kind of thinking needed here is transferable to just about any modern proof system you might encounter.

Even more advanced credential systems can be constructed in this framework, including the aforementioned anonymous credit tokens and anonymous rate-limited credentials. In both of these cases, the primary cost is an additional hash or two. Each of these also requires proving that part of the witness (e.g., the credit count) is in some expected range, but this is low cost in VOLEitH.

Indeed, for every construction we have considered so far for MOLE, the primary cost is the size of the Keccak evaluation and how many times we need to evaluate it. This means that finding ways to reduce the amount of hashing yields improvements across the board.

Optimizations

Before we go any further, we need to acknowledge that cryptanalysis of multivariate cryptography is still relatively immature. We have seen advances against several parameter regimes in just the last couple of years, Both UOV and MAYO have weathered these attacks well, and an outright break of UOV and even MAYO seems more unlikely with every passing year. But we should at least expect the parameters to change in a way that adds a bit of cost to our constructions.

On the other hand, it's not uncommon for credential systems used in practice to adopt a less-than-conservative security posture for the sake of performance in so far as it affects forgery resistance. This typically means relying on stronger-than-usual computational assumptions. That is, it's possible that a (pre-quantum) attack might come along that breaks anonymous credit tokens, but not ECDSA, despite the fact that they both use the same elliptic curve.

Using fewer hashes

PoMFRIT achieved its size record by doing precisely the same thing. Recall that the blind signature requires proving two hash evaluations, one by the client and the other by the issuer. PoMFRIT observes that VOLEitH proof generation can be split into two stages. In the first stage, the client commits to a random mask. It then hashes the message normally (outside of the ZK circuit), combines the result with the mask, and sends the masked target to the issuer. The issuer uses the UOV trapdoor to find a preimage of this target. In the second stage, the client completes the proof that the preimage and previously committed mask are consistent. The hash remains part of the protocol, but the issuer — acting as the VOLEitH verifier — can compute it directly rather than requiring the zero-knowledge proof to reproduce it.

This strategy substantially reduces proof size, but it also changes our security posture. The plain UOV assumption is that no (quantum) attacker can find a pre-image for a random target without knowledge of the trapdoor. The corresponding one-more-UOV assumption (analogous to the one-more-MAYO assumption in the PoMFRIT paper) neatly captures the additional information available to the attacker as a result of this kind of optimization. The assumption, then, is that this additional information doesn't lead to an attack.

This assumption may turn out to be overly optimistic. Recently, researchers at the University of California, San Diego, and INRIA managed to forge a 1024-bit RSA signature using a Privacy Pass issuer as a signing oracle. Fortunately, Privacy Pass uses 2048-bit RSA, which, in concert with other defense mechanisms, provides a pretty good security margin. Likewise, for UOV we might consider using NIST Level 3 parameters instead of Level 1.

In any case, a bit more study will be required before we rely too heavily on these kinds of optimizations. It's worth emphasizing that a break here would only impact an attacker's ability to forge tokens, not the attacker's ability to violate a client's privacy.

Making each hash cheaper

Standard SHAKE uses a 1600-bit Keccak state with 24 applications of an underlying round function. However, since the SHA-3 standard was published, many have come to believe that 24 rounds is just too much crypto. TurboSHAKE works the same way, except the permutation uses 12 rounds instead of 24. This substantially reduces the size of VOLEitH proofs. In fact, we're quite comfortable with this security margin, and not just for this application.

Another change to Keccak that reduces the proof size is to reduce the size of the state. 256 bits of the state are reserved for the capacity to get 128 bits security. That leaves 1344 bits that can be hashed in one go, which is more than we need. Reducing the state by half to 800 bits leaves us with a rate of 544 bits, which is plenty. On top of that, it even increases the security at the same number of rounds, as an attacker has less to play with. The smaller state means we’re using half the multiplications in our circuit, which reduces the size of the proof.

Leaning into multivariate

In an ideal world, we could avoid proving conventional hashes altogether by replacing each hash with something that composes more natively with the algebraic structure of our primitives. Indeed, those familiar with Pedersen commitments and their use in elliptic curve cryptography may wonder if commitments can be constructed from multivariate cryptography.

This might look something like this: Fix public, random MQ maps F and G and commit to a message m using c=(m+F(r),G(r)). The advantage of this approach is that the degree of the constraints is the same as the degree of the constraints for the UOV signature. Not only does this make the proof smaller, it makes it far simpler, since we avoid evaluating symmetric crypto in ZK altogether.

We experimented with this approach for our application, and it yielded some impressive results. However, it's not at all clear yet if the parameter regime in which multivariate commitments yield an improvement is also secure. Notably, the risk here is not just to unforgeability, but also to client privacy. A lot more study is needed.

Another direction is to build an MQ-friendly pseudorandom generator (PRG).

For typical VOLEitH parameters, adding an extra witness bit adds ten bits to the proof. There is a neat trick to make this a lot cheaper: only one bit of proof per bit of witness! Instead of using VOLEitH directly, we commit to a (relatively) short key. That key we stretch into a larger bitstream using a PRG with low-degree so that it can be computed cheaply in the circuit. Now, to add a witness bit w, we get one extra bit k from the PRG, and store the bit w XOR k in the proof.

There is a performance trade-off: we need to compute the PRG and the witnesses start at degree 2 instead of 1. That especially increases prover time. Also the benefits on proof size only start to pay off when you need a decent number of witness bits.

And of course you need a PRG! You can create a PRG using a random MQ instance, but that’s very slow and seems overly conservative. At the other end is the Goldreich PRG, which can be seen as an as-sparse-as-possible MQ PRG. We’d want something in between, but such PRGs haven’t received much study.

Is MOLE ready for post-quantum?

We already have post-quantum privacy today, so we have some time before we need to have a solution in place, but now is the time to develop alternatives.

Multivariate cryptography looks to be a promising basis for easily building a variety of anonymous tokens and credentials. We're currently developing two schemes in this framework: VOLE-IHAT, a candidate for MOLE endorsements, and VOLE-ACT, a candidate for MOLE credentials. Our current cost estimates for these are shown in the table below. The numbers here are subject to change as we analyze and develop these constructions. We use UOV (uov-Ip) in each case, which means the public keys are all 46.6KB. VOLEitH uses the same parameters as FAEST-EM-128s:

issuance

presentation

PoMFRIT (with UOV)

<1KB

4.3KB (redemption)

VOLE-IHAT

5.6KB

6.5KB (redemption)

VOLE-ACT

5.4KB

12.0KB (spend)

These numbers are significantly larger than their classical counterparts. This comes as no great surprise, as "PQ is big" is really the main hallmark of the transition across the board. However, replacing RSA blind signatures with PoMFRIT has comparable overhead to replacing ECDSA with ML-DSA. The runtimes of these protocols are also excellent: each operation takes just a few milliseconds with our implementation. (Runtime can be improved by quite a bit, though at the cost of higher communication.)

If these numbers hold, then we've already reached an important milestone: the transition to PQ anonymous credentials for the web can now shift from cryptography design to protocol engineering. In other words, we think we can make these numbers work. For example, whereas classical blind signatures might be cheap enough to redeem on every request, we'll have to figure out how to amortize a single redemption over multiple requests.

We have some work to do before we get there. First, we need to make sure our protocols are sound, in particular by producing security proofs for them that show they are secure as long as UOV is secure. Second, we need to wait a bit for cryptanalysis of UOV to catch up. We encourage cryptographers to spend time with multivariate cryptography. Once the parameters stabilize, we believe these schemes will find a place in a ton of useful applications. We've only just scratched the surface so far.

Йотова ли спечели, или Гюров загуби?

Post Syndicated from Емилия Милчева original https://www.toest.bg/yotova-li-specheli-ili-gyurov-zagubi/

Йотова ли спечели, или Гюров загуби?

На избори обикновено се състезават обещания за по-добро бъдеще, но на балотажа за президент в България победи обещанието да не стане по-лошо. 

Лидерът на ГЕРБ Бойко Борисов години наред печелеше избори с предупреждения за хаоса, който ще настъпи без него. Румен Радев усъвършенства тази политическа технология, като прибави и страха от война към страха от нестабилност. На президентските избори технологията отново проработи, а премиерът (и президент почти два мандата) си приписа заслугите за победата на Илияна Йотова. Същевременно опозицията им не успя да убеди достатъчно избиратели, че промяната не е поредната заплаха.

За джендъра на машините, или за активните мероприятия

Ако нещо изглежда като активно мероприятие, постига цели като активно мероприятие и отшумява лека-полека – точно както правят активните мероприятия, то по всяка вероятност е активно мероприятие. Безпомощни ли сме пред активните мероприятия, или не съвсем? От Светла Енчева.

Колко страх събра „Ганьо отива на война“?

Да си прав за агресията на Русия срещу Украйна, за европейския избор на България и за опасностите от политическото заиграване с Кремъл не означава непременно да печелиш избори. 

Правотата не е политическа стратегия, нито моралното превъзходство е заместител на доверието. 

Властта успя да превърне войната на Русия срещу Украйна в риск, който уж произтича от противниците на управлението на страната. В държава членка на НАТО, на чиято територия са разследвани (руски) саботажи и която е мишена на дезинформационни и хибридни операции, управляващите представят не Русия, а политическите си противници като заплаха, по-голяма от режима на Путин. Така агресорът остана на заден план, а опасността беше приписана на онези, които му се противопоставят.

Балотажът, както е добре известно, не се печели само със собствените избиратели. Между двата тура кандидатите трябваше да търсят подкрепа извън твърдите ядра и симпатизантите си. 

Резервоарът на гласове за Йотова беше гарантиран още на първия тур от кандидатпрезидентските двойки на „Възраждане“, „Величие“ и дори МЕЧ (независимо от призива на лидера Радостин Василев да се гласува за Гюров). Третите на 25 октомври – „Възраждане“ с лидера си Костадин Костадинов и евродепутата Петър Волгин, спечелиха близо 191 000 гласа. Част от тях преминаха на втория тур към Йотова поради липса на българска Алис Вайдел. Но и наративът на Йотова и подкрепящите я е също като на лидерката на „Алтернатива за Германия“.

Forever Young. Да поговорим за възхода на „Алтернатива за Германия“

Крайната десница печели там, където демокрацията не успява да даде нито убедителни отговори, нито визия за бъдещето. Светла Енчева разглежда възхода на АзГ в Източна Германия и проследява как усещането за изоставеност, носталгията и страхът се превръщат в политическа сила.

Неслучайни съвпадения

Противопоставянето „мир или война“ в кампанията за президент в България не е изолиран феномен. В Европа, не без помощта на Кремъл, се оформя нов политически речник, в който капитулацията се нарича мир, съпротивата – ескалация, а солидарността с нападнатия – подстрекателство към война. 

В интервю за The Economist Алис Вайдел говори с добре познати и на българското общество тези, които се налагат от управляващата „Прогресивна България“, от говорители като Явор Дачков и Мартин Карбовски, от някои социолози и анализатори, от Йотова–Вълчев, Костадинов–Волгин, „Величие“, МЕЧ и Радостин Василев. Един твърде шумен и многолик, но единогласен хор.

Украйна вече е загубила войната. Подкрепата за Украйна е опасност от разширяване на войната и Германия може да стане мишена. 

На въпрос за предполагаем руски опит за атака с дронове на летището в Лайпциг Вайдел заявява, че не е видяла доказателства за руска отговорност. Също като българските управляващи, които упорито отказват да посочат Русия като стояща зад атаките с дронове по търговски кораби в Черно море. 

Когато питат Вайдел за отговорността на Путин, тя измества разговора към президента на Украйна Зеленски. Оказва се, че нападнатата страна трябва да доказва, че иска мир. Така лидерката на германските крайнодесни се измъква от неудобни въпроси, като обръща посоката им. По същия начин действа с въпроса за дроновете – насочва разговора към предполагаемата украинска роля във взривовете на газопроводите „Северен поток“. 

„Джудо“ в комуникацията

В бойното изкуство джудо силата на противника се използва срещу самия него. На този принцип се основава и т.нар. вербално джудо – комуникационна техника, популяризирана от американеца Джордж Томпсън, бивш преподавател и полицай. Нейната цел е да овладява конфликти чрез пренасочване на разговора, без агресия и насилие. 

В политиката обаче подобен подход може да се използва и за манипулация. Тази тактика напомня на т.нар. DARVO (от английските Deny, Attack, Reverse Victim and Offender), при която обвиненият отрича, атакува обвиняващия и разменя ролите на жертва и извършител. Вместо да отговориш на неудобния въпрос, обръщаш го срещу онзи, който го задава. Атакуваш, вместо да се защитаваш. И докато противникът се опитва да опровергае обвиненията, никой вече не помни въпроса.

Съвпадения с пропагандния речник на Кремъл се откриват в наративите на различни европейски политици. На 21 октомври The Insider разкри документи и вътрешна кореспонденция за руската операция за влияние „Допелгенгер“. От тях става ясно, че още през април 2022 г. Сергей Кириенко – първи заместник-ръководител на президентската администрация в Русия, който отговаря за вътрешнополитическите процеси, и един от най-влиятелните хора в Кремъл, поставя пред пропагандистите три задачи: да създадат „ядрена психоза“, да водят „война в името на мира“ и да преувеличават всичко. През март 2024 г. при опасения от нарастваща подкрепа за Зеленски във Франция са предложени посланията „чужда война“ и „Европа има нужда от мир“. 

А днес опити за всяване на страх от ядрена катастрофа и обещания за мир звучат и от най-високите трибуни в България.

Месец преди балотажа премиерът Румен Радев извади ядрената карта. На форум на Euronews в Брюксел на 30 септември той предупреди, че Русия може да прибегне до ядрено оръжие, ако бъде отслабена или притисната. Попита защо никой не оценява риска от опитите за конвенционална победа над най-голямата ядрена сила, и настоя да бъдат чути тревогите и на двете страни. Така заплахата, отправяна от Москва, се превърна в аргумент за сдържаност на Запада.

След първия тур Радев вече не говореше за хипотетична ядрена ескалация, а за избор между мир и война. На 26 октомври той обяви, че политическите му противници не биха се поколебали да въвлекат България във войната. За месец опасността смени адреса си: от руския ядрен арсенал към българската опозиция.

В телевизионния дебат Йотова продължи тази линия. Според нея наливането на повече средства и по-модерни оръжия в Украйна води единствено до нови разрушения и човешки жертви. Тя напомни, че загиват хора от двете страни – славяни и православни християни. Общата религия и произход се оказаха по-важни за аргумента ѝ от разликата между агресор и нападнат. А подписаното от Гюров споразумение с Украйна беше използвано, за да бъде внушена опасността от изпращане на български войници, въпреки че това може да реши единствено българският парламент.

Така войната, започната от Путин, беше превърната в предизборно обвинение срещу противниците на Радев. Докато те трябваше да доказват, че няма да изпращат българите на фронта, управляващите си присвоиха правото да говорят от името на мира.

Кога и защо се разпространява манипулативното съдържание?

В последната част на поредицата Йоанна Елми изследва кога и защо се разпространява манипулативното съдържание из онлайн пространството. В текста проследяваме кризите в обществото (политически…

Технологията на внушенията беше използвана и срещу Андрей Гюров. Анализ на компанията Identrics проследява как твърдение за негов спонсор, свързан с производството на дронове в Украйна, преминава през сайтове, социални мрежи и политически говорители, докато се превърне в обвинение, че кандидатът е финансово обвързан с военната индустрия. Според анализа Мартин Карбовски също разпространява твърдението във Facebook, повтаря го на среща с граждани във Велико Търново и Илияна Йотова.

Според публикация в сайта „Гласове“ на Явор Дачков, чиято съпруга Йотова назначи за съветник в Президентството, Данчев е най-големият дарител на „Продължаваме промяната“ през 2025 г., а за президентската кампания на Андрей Гюров е предоставил общо 150 000 евро. 

Калоян Данчев притежава българската „Дженерал Чери интернешънъл“, в чийто съвет на директорите е Кирил Петков. През юни се появяват публикации за намерения двамата да развиват производство на дронове по украински лиценз. 

На 29 октомври „Гласове“ публикува данни от естонски регистри, според които Петков е участвал в срещи с отбранителни институции като представител на делегация на украинския производител на бойни дронове General Cherry. Тези обстоятелства не доказват, че дарението му обвързва Гюров с военния бизнес или с каквато и да било политика на въвличане на България във войната.

Президентът е върховен главнокомандващ на въоръжените сили, но това не означава, че може еднолично да изпраща български войници на война. Той назначава и освобождава висшия команден състав по предложение на Министерския съвет, а решението за изпращане на български войски извън страната принадлежи на Народното събрание.

Градската десница и изкуството да губиш избори

Първият тур на президентските избори показа, че Йотова може да бъде победена: разликата с Гюров беше под 3% и почти 43 000 гласа. Вторият тур разби надеждите. 

За Илияна Йотова гласуваха 57,6% от отишлите до урните, а за Андрей Гюров, подкрепен от „Продължаваме промяната“, „Демократична България“ и други формации от демократичната общност – 39,5%. В гласове разликата надхвърли 300 000.

Утешението е, че Гюров получи близо 750 000 гласа, една част от които са дошли от ГЕРБ. Все пак вотът е сериозен политически капитал за човек, станал широко разпознаваем благодарение на 78-те си дни като служебен премиер и обявил кандидатурата си само два месеца преди изборите. Така беше пропуснато ценно време не само за утвърждаването му като кандидат, но и за достигане до избиратели извън традиционните среди на ПП, ДБ и подкрепящите ги формации. (За сравнение, на парламентарните избори на 19 април коалицията ПП–ДБ получи 408 846 гласа, впоследствие се разпадна на двете си съставни части.)

Кампанията показваше Гюров как цепи дърва, лови риба и меси питка, но твърде малко от това какъв президент възнамерява да бъде. Вместо да разширява представата за политическите му възможности, тази кампания сякаш се опитваше да го направи симпатичен на хора, които бездруго бяха готови да гласуват за него.

Градската десница от години се сблъсква с един и същ проблем: знае срещу кого се бори, но трудно убеждава избирателите за какво се бори. Борбата с корупцията, съдебната реформа, европейската ориентация на България са важни каузи. Но те не са достатъчни, ако остават политически език, разбираем и убедителен предимно за хората, които така или иначе споделят тези убеждения.

Гюров трябваше да бъде кандидат не само на онези, които не искат Радев и Йотова, а и на хората, които нямат доверие нито на управляващите, нито на техните противници. За тях въпросът не е единствено кой е по-европейски настроен, кой е по-последователен в подкрепата за Украйна или кой има правилна позиция за съдебната власт. А защо и как един различен от управляващите президент би помогнал животът им, държавата им и бъдещето на децата им да станат по-сигурни.

Вместо да постави управляващите в положение да обясняват собствените си решения, по време на кампанията си Гюров твърде често трябваше да отговаря на техните обвинения. Така се състезаваше на терена, избран от противника. Докато единият кандидат обещаваше мир, другият трябваше да доказва, че не иска война.

Гюров не беше безмълвен пред властта. След ударите с дронове по кораби в Черно море той обвини Йотова, че е превърнала президентската институция в ехо на правителството. В телевизионния дебат с нея говори за бюджета, инфлацията, свободата на медиите и президентското вето. Но тези позиции не успяха да определят дневния ред на кампанията. Вместо да се превърне в разговор за отговорността на управляващите, предизборният спор отново и отново се връщаше към въпроса дали Гюров няма да въвлече България във война.

Разбира се, президентът не управлява икономиката, не разпределя бюджета, не се занимава с данъците. Но има право на вето, назначава служебно правителство при определените от Конституцията условия, участва във формирането на ключови институции и може да задава политически дневен ред. Кампанията даваше възможност да се покаже как държавният глава може да бъде коректив на властта. Румен Радев го правеше непрекъснато през двата си мандата, както и чрез служебните правителства.

Но има и още нещо: колко пъти демократичната общност може да обяснява изборните си поражения с пропагандата, зависимите медии, купения вот и страховете на избирателите, без да потърси причините и в себе си? Всички тези фактори имат значение. Не обясняват обаче защо политическите ѝ послания толкова трудно прекрачват границите на собствения ѝ електорат.

А какво следва за Андрей Гюров? Близо 750 000 гласа му дават политическа тежест, но не и автоматично политическо бъдеще. Ще остане ли лице на демократичната общност, ще потърси ли собствено място в партийния живот, или ще се оттегли отново от първата линия? 

Разпадането на коалицията ПП–ДБ прави този избор още по-сложен. Гюров получи подкрепа, която надхвърля електората на двете формации, но тепърва трябва да покаже дали може да я превърне в трайно политическо доверие и за каква цел. 

Иначе тези гласове ще останат само статистика от един загубен балотаж.

Ако прави собствена партия, ще откъсне живо месо от двете сили – типичното за общността самоизяждане, един изчерпан вече модел на политическо роене. Трудно е и да бъде издигнат за кандидат за кмет на София предвид намерението, макар и незаявено публично, на Васил Терзиев за втори мандат. В случай че се мисли в тази посока, от ПП и ДБ се очаква да направят оценка на работата на столичния кмет и да обяснят публично защо не биха го подкрепили за втори мандат.

В интервю за „Дневник“ съпредседателят на „Да, България“ Ивайло Мирчев посочва, че демократичната общност се разширява и Гюров, който е допринесъл за това, има място в нея. Чудесно, но как? Създаването на нови политически конструкции, които пренареждат едни и същи избиратели, не привлича нови.

Дълголетието на политика Йотова

През януари 2027 г. Илияна Йотова ще се закълне като седмия президент и първата жена на този пост, след като изкара 9 години като вицепрезидентка и една като действащ държавен глава. Предстои ѝ петгодишен мандат, след който Конституцията дава шанс да се пробва и за втори. 

Илияна Йотова ще се сдобие с политическо дълголетие на висок пост, по-голямо от това на лидера на ГЕРБ Бойко Борисов като трикратен премиер. Тя е единственият политик от БСП, чиято кариера, започнала още през 90-те, неизменно е била във възходящо движение – между пресцентъра на „Позитано“ 20 и позицията ѝ на „Дондуков“ 2 тя премина през столичната организация на БСП, беше депутатка и евродепутатка. Избирателите на БСП съвсем оредяха, но структурите останаха и работиха енергично за кампанията ѝ. 

Така партията се оказа в парадоксалното положение да остане извън парламента, но да има свой политик за президент – разбира се, с подкрепата на управляващата „Прогресивна България“. 

Йотова си осигури още пет години политическо бъдеще, като превърна бъдещето на България в повод за страх. Ще успее ли да предложи нещо повече?

Гласовете на Америка – брой 20

Post Syndicated from Йоанна Елми original https://www.toest.bg/glasovete-na-amerika-broy-20/

Гласовете на Америка – брой 20
* „Горна и Долна Каролина“ е игра на думи от гей жаргонa за top и bottom, кой е отгоре и кой отдолу

Гласовете на Америка – брой 20

Това са само част от коментарите в социалните мрежи след историческата синя вълна, която заля САЩ на междинните избори във вторник, 3 ноември 2026 г. Както често се случва, никой не успя да предскаже голямата история – дори силно положителните прогнози в полза на демократите не се приближиха до реалността. Американските медии говорят не за вълна, а за цунами. Последвалото цунами от анализи, мнения и прогнози е не по-малко мощно.

Затова ето обобщение за междинните избори в САЩ накратко в десет точки. 

Всяка основна демография в САЩ се е преместила към демократите. 

Независимо дали разглеждаме населението по пол, раса, възраст или образование, Америка прави рязък завой към Демократическата партия. При мъжете под 30 години преместването вляво е с 38 пункта, при белите американци под 30-годишна възраст е с 30 пункта, а най-голям обратен завой са направили мъжете от латиноамерикански произход под 30 години – 47 пункта по-голяма подкрепа за демократите. Дори белите гласуващи без висше образование, които традиционно подкрепят Тръмп, са се преместили 14 пункта вляво. 

Гласовете на Америка – брой 20

Това е най-голямата политическа изборна победа след скандала „Уотъргейт“, заради който през 1974 и 1976 г. републиканците понасят тежко поражение. 

Демократите обръщат някогашни бастиони на републиканизма, като Канзас, който не е имал сини представители в Сената и долната камара от почти сто години насам. За последния половин век нито една партия не е вземала повече от две места, спечелени в предишни избори от опозицията с повече от 10 точки преднина. Демократите със сигурност печелят между пет и седем такива места – в Айова, Канзас, Охайо, Южна Каролина, Тексас, като Аляска и Мисури са почти преброени и сигурно също ще се обърнат в синьо. 

Гласовете на Америка – брой 20
Компилация от реакциите на политическия журналист и анализатор Стив Корнацки, превърнал се в интернет звезда. Корнацки е сред най-добрите в отразяването на изборите на живо и бе гледан от рекорден брой зрители – 29 млн. души.

Седем щата решиха изборите и ще бъдат сред ключовите за президентската кампания след две години. 

Канзас избира демократ за последен път през 1932 г. – три години след началото на Голямата депресия, когато Демократическата партия е много различна. Тексас не е бил син от 1988 г. и бе толкова сигурно републикански, че Илон Мъск се премести там, за да избяга от демократите в Калифорния.

Южна Каролина не е променяла цвета си от 2002 г. насам, Айова и Северна Каролина последно гласуват за Обама през 2008 г. Охайо беше традиционно смятан за колебаещ се щат, но оставаше непоколебимо червен през последните шест години. Това обаче може би отново ще се промени. Мейн също е колебаещ се щат с необичайно висока подкрепа за независими кандидати – този път спечелиха демократите. 

Успехът на демократите се дължи колкото на неуспеха на политиките на Тръмп – републиканците държаха президентството, долната и горната камара и имат мнозинство от консервативни съдии във Върховния съд, толкова и на смелата стратегия на Демократическата партия, която най-накрая, изглежда, осъзнава грешките си. 

Гласовете на Америка – брой 19

Америка отива на междинни избори със скъп живот, републиканци с пълни каси и демократи, които още се чудят каква партия искат да имат. В новия брой на бюлетина ни за САЩ Йоанна Елми гледа отвъд изборната аритметика – към залозите за следващите две години.

Така или иначе, без много за губене демократите инвестираха сериозни средства в кампании, насочени към места, които Тръмп бе спечелил с огромна преднина през 2024 г. На това се дължат изненадите в Канзас, Южна Каролина и Тексас. Те вероятно нямаше да бъдат толкова големи, ако войната на Тръмп в Иран не бе вдигнала цените на горивата, а хаотичните мита не бяха причината за допълнително поскъпване на живота и стоките. Блокадата на Ормузкия проток доведе не само до трусове в световната енергетика, но и смути доставките на торове, което директно се отрази на земеделците в тези щати.

Реалността надделя над театъра на сенките на идеологическите войни. 

Сред причините резултатите от тези избори да са толкова неочаквани е политическото втвърдяване, за което сме говорили и друг път – силното поляризиране на избирателите в две идеологически крайности, от които се излиза трудно. Президентът Тръмп стана емблемата на този феномен: независимо колко скандали, корупция и проблеми породи неговата фигура през годините, изглеждаше, че той има имунитет, защото въплъщаваше идеологическата визия и искания на своя електорат и привличаше и независими избиратели към себе си. 

Речта на преизбрания за сенатор на Джорджия Джон Ософ, сочен като един от фаворитите за кандидат-президент след две години. Пълния превод може да прочетете тук. 

Избирателите обаче посочиха високите цени, войната с Иран, липсата на достъп до добро здравеопазване и корупцията по високите етажи на властта като основни проблеми. И въпреки че вместо да признаят грешките си, през цялата кампания републиканците отново заложиха на културни войни, говорейки за трансджендъри, страшни мигранти и комунизъм, реалността просто се оказа по-силна.

Тактиката с всяване на страх се превръща в сатира, защото все повече хора виждат, че не отговаря на реалността. Не помогна и това, че президентът до последно беше съсредоточен да убеждава хората, че е най-успешният лидер на САЩ в историята. Както и че собствената му партия така и не успя да му противостои – отчасти защото втвърденият ѝ електорат не позволи, а когато вече бе възможно, политиците просто не реагираха навреме и с достатъчно натиск. 

В САЩ десният популизъм не доведе до мащабна промяна, а до корупция, задълбочаване на неравенствата и морален упадък. 

След края на Студената война и двете партии претърпяват промени, които отразяват разклащането на обществото: след разочарованието от ерата на Буш-младши и нахлуването в Ирак расте недоверието към институциите и медиите, което кулминира в победата на Обама – първия чернокож президент, през 2008 г.; това пък хвърля в истерия дясното и медийния му балон (добре известна е историята на Роджър Ейлс и Fox News, превърната и в популярен сериал).

По времето на Обама по-екстремните крила на републиканците бавно превземат партията, а Тръмп се появява като последен симптом на болното си време и срутва окончателно умерените фронтове вдясно. След 2000 г. неравенствата в САЩ се задълбочават, финансовата криза и войните в Близкия изток белязват обществото, животът става все по-скъп, средната класа живее все по-трудно, медийният пейзаж се фрагментира, а парите навлизат все по-безочливо в политиката. 

Клипът, с който Джош Турек обявява кандидатурата си за Сената в Айова. В него бившият баскетболист, печелил злато за страната си, разказва за заболяването си – вродена спина бифида, причината за която е излагането на баща му на хербицида „Агент Ориндж“, използван от САЩ по време на войната във Виетнам. 

Импулсът, който докарва Тръмп на власт, е популистки – той се обявява срещу институциите, системата, обещава да „изчисти блатото“ във Вашингтон. Без повече войни в Близкия изток, по-евтин живот, справедливост. Тези обещания резонират с очакванията на избирателите през 2016 г., въпреки че тогава губи популярния вот.

За завръщането му през 2024 г. пък помага тоталната немощ на демократите да изберат обединяващ кандидат, който да бъде ново лице на партията и да внесе качествени промени – вместо това на власт след пандемията бе Джо Байдън, чийто когнитивен упадък бе отричан системно от партията, преди да стане твърде очевиден. От обещанията на Тръмп обаче минаха десет години, властта никога не е била по-явно корумпирана, а всяко едно от въпросните обещания бе нарушено – не на последно място защото е невъзможно например да осъществиш масови депортации, когато икономиката на държавата ти зависи от мигрантски труд.

Може би това са първите избори, в които виждаме новото лице на демократите след тоталното им омаломощаване в годините след Обама. 

Нито Хилари Клинтън, нито Джоузеф Байдън можеха да бъдат лицето на промяната, от която има нужда страната – напротив, в очите на американците те са част от същата политическа класа, неуспяла да се справи с проблемите в зародиш. Част от привилегията на Тръмп бе именно да бъде възприеман като аутсайдер. Днес демократи като Абдул Ел-Сайeд от Мичиган също използват популистки език, само че с лява насоченост, като в една от предизборните си реклами споменава и факта, че е трябвало да се пребори със собствената си партия, за да бъде издигнат. 

Предизборна реклама на Абдул Ел-Сайед, в която се появяват баба му и дядо му. „Откъде е Абдул Ел-Сайед“, започва рекламата, имитирайки апокалиптичната и плашеща естетика, характерна за републиканския ксенофобски наратив. „Оттук, ние го отгледахме, а той ни носи покупки всяка седмица и знае кое колко струва“, разказват те. 

Макар и все още да нямаме данни за средната възраст на избраните за Камарата на представителите и Сената, със сигурност се отдалечаваме от геронтокрацията на последното десетилетие. През по-голямата част от кампанията на междинните избори прожекторите бяха върху нови лица, които често побеждаваха по-опитни членове на партията в предварителните избори, привличаха огромни публики и обещаха да променят приоритетите на демократите, щом стигнат до Вашингтон. 

Превръщат ли се САЩ в геронтокрация?

Превръща ли се лека-полека Чичо Сам в Дядо Сам и как изглежда това в контекста на предстоящите президентски избори в САЩ? Йоанна Елми продължава поредицата си за американския вот с текст за остаряването на политическата класа в САЩ.

Сред най-разпознаваемите примери са: 37-годишният Джеймс Таларико, който спечели сенаторското място в Тексас; 47-годишният Джошуа Турек от Айова; Адам Хамилтън, на 62, който обърна Канзас; 46-годишната сенаторка д-р Ани Андрюс от Южна Каролина, която победи сестрата на починалия сенатор Линдзи Греъм. Ръководството на партията положи неимоверни усилия да открие хора, които резонират с избирателите във всяка една област. А после им даде път напред, вместо да ги саботира, както ставаше през последните години.

Предизборна реклама на д-р Ани Андрюс, в която участва и баща ѝ – републиканец, гласувал за някогашната ѝ опонентка Линдзи Греъм. Д-р Андрюс набляга върху разочарованието от двете страни на политическия спектър, а баща ѝ казва, че дъщеря му със сигурност не е „луда социалистка“. 

Печели умереният, рационален, многообразен вот; въпрос на време е да разберем дали умереният, рационален, многообразен популизъм съществува и е приложим вляво. 

В кампанията на демократите осезаемо липсваха някои от по-крайните проявления на социалната политика, които им костваха избиратели през последните десет години. Д-р Андрюс например се съсредоточи върху критика към здравния министър Робърт Кенеди-младши и обвини администрацията на Тръмп за епидемията от морбили в щата си.

В своята предизборна агитация Адам Хамилтън заложи на това, че Канзас е щат в центъра на страната и се намира далеч от крайностите и поляризацията във Вашингтон. В началото на годината Хамилтън е регистриран републиканец, променя партийната си принадлежност и води кампания, основна на вярата му – Хамилтън е пастор в най-голямото методистко паство в страната. Таларико от Тексас също залага на вярата си, връщайки християнството вляво. 

Както в Камарата на представителите, така и в Сената мнозинството на демократите ще бъде многообразно – от по-левите представители, като Ел-Сайед и Трой Джаксън в Мейн, до новоизбраните демократи от традиционно консервативни щати, които могат да бъдат добър баланс или обратното – да не могат да се разбират взаимно.

Важно е да се отбележи, че мнозинството от избраните демократи са подчертано умерени и центристи. Групата по интереси на демократите Blue Dog Coalition, които поддържат центристки възгледи, обяви, че 14 нейни членове са избрани в Камарата. Други центристки групи, като New Democrats и Problem Solvers, също са увеличили представителството си.

Предизборна реклама на Джеймс Таларико, концентрирана върху достъпни цени и политически диалог за работа в името на хората

Предстоят решаващи години за републиканците. 

За повечето рационални хора беше ясно, че ерата „Тръмп“ ще има печален край, въпросът беше колко и на каква цена – както за САЩ, така и за света. Онези, които все пак понякога допускаха, че обратното е възможно, по-скоро се надяваха, че грешат за доброто на всички. В тази рубрика и преди сме си задавали въпроса какво ще се случи с партията, която избра да се идентифицира с един-единствен човек, след като неизбежните му човешки и политически провали доведат до отлив на избиратели.

В часовете след загубата Джей Ди Ванс показа една възможна посока, като дипломатично се дистанцира от президента, същевременно успя да го цитира и да поеме някаква отговорност за загубата. Много виждат в бившия сенатор от Охайо новото лице – досега на тръмпизма, но оттук нататък може би на републиканизма. 

Ванс обвини войната в Иран за загубата. Но в щаба на президента от месеци е било ясно, че прогнозите не са добри – и нито президентът, нито партията са сменили курса, за да чуят проблемите на избирателя; напротив – само преди седмици президентът Тръмп каза, че няма проблем да се плащат по-високи цени за горива, ако петролните компании са на печалба. Арогантността, безпринципността и отявлената корупция са не по-малко решаващи за загубата на партията, която от десетилетие разчита на пропаганда, черен пиар и негативна кампания, за да прави политика. Вероятно ще отнеме повече от две години на републиканците да се разделят с този бранд. 

Въпросът не е кой печели и кой губи, а дали наболели и сложни проблеми ще бъдат решени или дали поне ще бъдат направени конструктивни опити в тази посока. Рискът при нов провал е тотално обезверяване и още по-радикални режими. 

Икономиката, състоянието на демокрацията, имиграцията, здравеопазването, външната политика и престъпността са сред приоритетите на избирателите, гласували на тези избори. Мнозинството от тях казват, че Тръмп е влошил икономическата обстановка. 38% заявяват, че са гневни относно състоянието на държавата, а 37% са недоволни. На 8 от 10 гласуващи инфлацията е причинила тежки или умерени проблеми. 74% от запитаните казват, че войната в Иран е била сред основните фактори за кого ще гласуват; 63% смятат, че е било грешка САЩ да атакуват. 

Гласовете на Америка – брой 20

Според проучване на „Галъп“ разпределението на благата и концентрацията на капитал също са сериозен проблем за избирателите. Войната на Израел в Палестина също е посочена като фактор, а вече ви разказахме за безогледните строежи на центрове за данни, които са обединяващ фактор на недоволство в идеологическия спектър. Разделенията кой какво смята за важно, са очевидни между демократи и републиканци и не на последно място са резултат от медийните балони; консервативният е доказано наситен с дезинформация, но множество крайнолеви източници също разпространяват дезинформация и/или са податливи на манипулации. 

Гласовете на Америка – брой 17

Америка сменя меката сила с тестостерон, война и показна грубост. Между кризата на мъжествеността, страха от майчинството и възхода на Китай може да се проследи как разпадът на една култура започва да пренарежда света, превръщайки крайностите в „ново нормално“ пред очите ни. От Йоанна Елми.

Как ще се държи един своенравен, нарцистичен президент през идните две години, в които ще бъде принуден да се подчинява на волята на народа отново чрез демократични механизми? Ще успеят ли демократите да действат достатъчно бързо и ефективно, за да има осезаема промяна и да не изгубят инерцията? Ще се радикализират ли последователите на Тръмп, или ще се оттеглят от него? Каква посока ще изберат републиканците? Какъв обрат ще търси външната пропаганда, която цели да задълбочи и изостри вътрешнополитическото напрежение в САЩ, за да ги отслаби на световната сцена? Ще успеят ли демократите да прекратят подкрепата за Израел в Газа? Каква ще е позицията спрямо Украйна? 

Отговорите на някои от тези въпроси ще се опитаме да дадем в следващите бюлетини. За останалото – ще покаже само времето. 

How OrderServ achieved up to 64% cost savings with multi-tenant ECS

Post Syndicated from Chetan Yewale original https://aws.amazon.com/blogs/architecture/how-orderserv-achieved-up-to-64-cost-savings-with-multi-tenant-ecs/

When your food-ordering platform doubles its brand count, dedicated clusters become your biggest cost problem, and every failed request during a traffic surge is a lost sale.

Here is how Cognizant and AWS Professional Services solved it.

Cognizant is a global services company that helps organizations modernize technology, reimagine processes, and transform customer experiences. Its OrderServ service is an omnichannel ordering solution for the food services industry that supports ordering across devices and integrates with point-of-sale, kitchen display, payment, loyalty, and other enterprise systems. Cognizant is an AWS Premier Tier Services Partner with multiple AWS Competencies and a member of the AWS Managed Service Provider (MSP) and AWS Well-Architected Partner Programs.

In 2022, Cognizant published how OrderServ 2.0 reimagined digital food ordering by migrating from a monolith to microservices on AWS Fargate, achieving 30 percent cost savings over traditional hosted architectures.

Since then, the service grew from serving a single quick-service restaurant (QSR) brand to powering multiple brands across markets. Each brand ran as a dedicated Amazon Elastic Container Service (Amazon ECS) cluster with 21 microservices. This provided hard isolation, but cost scaled linearly with every brand onboarded, and scaling speed could not keep up with promotional traffic surges that spike 5–10× within minutes.

In this post, we describe how AWS Professional Services partnered with Cognizant to transform OrderServ into a multi-tenant software as a service (SaaS) offering, achieving 47–64 percent compute cost reduction, scaling time reduced from 5–7 minutes to 30–60 seconds, and authentication costs cut by 80 percent.

The problem: Dedicated clusters don’t scale with brands

OrderServ 2.0 ran 21 microservices per brand on dedicated AWS Fargate clusters. For five brands, this meant:

  • 105 ECS services (21 × 5), each with idle capacity during off-peak hours.
  • 315 sidecar containers for observability, configuration, and log routing, duplicated per brand.
  • Linear cost growth, because onboarding a new brand required provisioning an entire cluster.
  • 5-minute scaling gap, because large container images increased image-pull and task startup times on AWS Fargate, delaying the addition of capacity during promotional traffic surges.

In QSR, every failed request during a surge is a lost sale and users may switch to competitor services immediately. The service needed to serve over 10 brands cost-effectively while handling unpredictable traffic patterns: lunch rushes, promotional events, and festival surges.

Solution overview

We addressed these challenges through four architectural changes:

  • Shared ECS cluster with a tenancy library: Application-level isolation replaces per-brand infrastructure.
  • EC2 warm pools with Bottlerocket: Sub-minute scaling replaces AWS Fargate cold starts.
  • Crash observability pipeline: Zero-loss logging and 90-day forensics at $10/month per service.
  • Passkeys for multi-tenant authentication: Biometric login reduces recurring SMS costs.

Solution 1: Shared cluster with application-level isolation

As OrderServ scaled from one to five brands, the cost of replicating 21 microservices per tenant made the economics unsustainable. We needed to consolidate infrastructure while preserving the data isolation guarantees each brand required.

Why we moved away from dedicated clusters

The dedicated-cluster model provided strong isolation but no economy of scale. With the platform growing from five brands, onboarding a new brand needed to become a configuration change, not an infrastructure provisioning exercise.

Model Pros Cons
Dedicated cluster per brand (status quo) Hard isolation, simple AWS Identity and Access Management (IAM) Linear cost, idle capacity, slow onboarding
Shared cluster with opt-out Aggregate scaling, config-only onboarding Requires application-level isolation, wider IAM scope

We chose the shared cluster as the default, with the ability to place high-volume brands on dedicated clusters when needed.

Figure 1 illustrates this transformation, from five dedicated ECS clusters (105 services, 315 sidecar containers) to a single shared cluster with 21 services and an opt-out path for high-volume brands. The tenancy library handles tenant resolution, resource routing, and context propagation, while per-tenant data stores (Amazon DynamoDB, Amazon Aurora, Valkey) maintain strict data isolation.

Evolution from dedicated clusters per brand to a shared cluster model with opt-out for high-volume brands

Figure 1: Evolution from dedicated clusters per brand to a shared cluster model with opt-out for high-volume brands

The service uses AWS Key Management Service (AWS KMS) keys to encrypt every data store at rest, and the architecture implements Transport Layer Security (TLS) to protect all service-to-service and client-to-service communication. Amazon DynamoDB automatically encrypts tables by default. Customers enable encryption when creating Amazon Aurora Clusters and the Amazon Elastic File System (Amazon EFS) file system, and customers configure both at-rest and in-transit encryption for the Amazon ElastiCache for Valkey cluster.

The tenancy library as primary isolation

Most multi-tenant architectures enforce isolation at the infrastructure level: separate accounts, separate IAM roles, or separate compute. We took a different approach: a shared .NET Core library as the primary enforcement point for tenant isolation. IAM and resource policies are the backstop that contains the blast radius when the library has a defect.

Every request carries an X-Tenant-Id header injected by the Amazon API Gateway authorizer. The tenancy library:

  • Resolves tenant context per request using request-scoped storage (AsyncLocal<T>), never static or singleton state.
  • Routes all data access: Amazon DynamoDB table names, Amazon ElastiCache for Valkey key prefixes, Amazon Aurora schema selection, and S3 paths.
  • Refuses to construct resource IDs without a tenant prefix, so a missing tenant context fails closed instead of reading another tenant’s data.
  • Propagates tenant context on service-to-service calls and asynchronous messages (SQS/SNS attributes).
  • Enriches every log line and distributed trace with the tenant ID.

This design means the same service binary runs identically in a shared cluster or a dedicated cluster. The compute model becomes a deployment decision, not an architectural constraint.

Application-level isolation moves the isolation boundary into code, and code could potentially have defects. Because an Amazon ECS task role is granted per task rather than per request, a task in the shared cluster necessarily holds permissions spanning every brand it serves. To narrow that scope, the tenancy library acquires tenant-scoped credentials at run time: it calls sts:AssumeRole with the tenant ID as a session tag, and the role’s policies reference that tag so the credentials reach only that tenant’s DynamoDB tables, S3 prefixes, and Aurora schema. Amazon ElastiCache for Valkey enforces its own layer, with per-tenant role-based access control (RBAC) users restricted to the tenant’s key pattern, so the cache engine itself refuses a cross-tenant key access rather than trusting the library to have built the key correctly.

Impact

The following table compares key operational metrics (service count, sidecar overhead, onboarding time, and capacity utilization) before and after the migration to a shared cluster model.

Metric Before (dedicated) After (shared + library)
ECS services to deploy 21 × N brands 21 total
Sidecar containers 315 (for 5 brands) 63 total
Onboarding a new brand Provision full cluster (days) Configuration change (minutes)
Idle capacity Per-brand (wasted) Aggregate (efficient)

Solution 2: EC2 warm pools with Bottlerocket for sub-minute scaling

With all brands now sharing a single cluster, traffic spikes from any one brand could impact the others. We needed a scaling strategy that could absorb sudden surges in seconds, not minutes.

Why we moved away from AWS Fargate

AWS Fargate removes infrastructure management, but OrderServ required additional capacity in under a minute during unpredictable demand surges. New tasks took 5–7 minutes to become healthy, primarily because Fargate had to download and extract the workload’s large container image before starting the application. Task provisioning, application initialization and health checks added to the delay, but image retrieval was the largest contributor. During this interval, existing ECS tasks become saturated, latency increased and some orders started failing.

The solution: EC2 Auto Scaling groups with warm pools

We migrated to Amazon EC2 Auto Scaling groups with Bottlerocket (the AWS purpose-built container OS) and warm pools:

  • The system pre-initializes and stops warm pool instances, and you pay only for Amazon Elastic Block Store (Amazon EBS) storage (~$3/month per instance).
  • When ECS needs capacity, the Auto Scaling group (ASG) starts a stopped instance (image pre-cached, ECS agent registered) in 30–60 seconds.
  • We set capacity headroom (target utilization at 80 percent), so that the first wave absorbs instantly.
  • Bottlerocket provides atomic auto-updates, eliminating the Amazon Elastic Compute Cloud (Amazon EC2) patching burden.

Five-layer scaling strategy

Rather than relying on a single scaling mechanism, we implemented five complementary layers, each targeting a specific traffic pattern to remove gaps between detection and response.

Layer Mechanism Traffic pattern solved
Predictive ECS predictive scaling (machine learning, 14-day history). Scheduled scaling for specific known events (New Year, Christmas) was already handled manually via ASG scheduled actions Daily lunch/dinner peaks, scheduled scaling
Reactive Target tracking on ALBRequestCountPerTarget (60s) Unexpected moderate spikes
Headroom EC2 ASG target capacity at 80% First wave of any spike
Downstream ECS Service Connect RequestCount metric Bottleneck shifting
Throttling API Gateway per-tenant rate limits via AWS Lambda authorizer Extreme surges (noisy neighbor)

Figure 2 illustrates how the five scaling layers work together with the EC2 warm pool. Pre-initialized warm pool instances launch in 30–60 seconds versus 5–7 minutes for an AWS Fargate cold start, at a cost of approximately $3/month per stopped instance.

Five-layer scaling strategy. Predictive scaling handles daily patterns, reactive scaling triggers warm pool activation, capacity headroom absorbs initial spikes, and per-tenant throttling protects system integrity

Figure 2: Five-layer scaling strategy

Cost and performance impact

In addition to improving scaling performance, the move from AWS Fargate to Amazon EC2 reduced the projected monthly cost of running 21 services in the Asia Pacific (Mumbai) AWS Region (ap-south-1). Two instruments do different jobs here: On-Demand Capacity Reservations is designed to make sure capacity is available in the Availability Zone when a surge activates the warm pool, and Compute Savings Plan supply the discount. The reservation carries no term commitment and can be resized or released as the fleet changes, so the following one-year and three-year terms apply to the Savings Plan alone.

Approach Monthly cost (21 services) versus AWS Fargate
AWS Fargate On-Demand (before) ~$1,568 Baseline
EC2 ASG with 1-year Compute Savings Plan and warm pool ~$832
  • ~47%
EC2 ASG with 3-year Compute Savings Plan with warm pool ~$568
  • ~64%

The reservation covers the steady-state fleet, while you stop warm pool instances, adding approximately $30 per month in Amazon EBS storage costs. Scaling time falls from 5–7 minutes to 30–60 seconds.

Solution 3: Crash observability pipeline

When containers crash in production, the support team needs three things: what happened, the root cause, and the container’s execution state at the time of failure. In the prior state, OrderServ used the awslogs driver to send container output to CloudWatch Logs. Although this captured normal application output, it did not provide durable task-stop metadata, cross-service request correlation or persistent crash artifacts.

  • Potential loss of final logs records during crash: When a task terminates abruptly, records still in the log driver’s delivery buffer might not reach Amazon CloudWatch Logs.
  • Short lived task metadata: Amazon ECS makes stopped-task information available through the API for a limited period of 1 hour, reducing the investigation window.
  • No cross-service correlation: Plain-text logs lacked trace IDs and consistent task, container, and service identifiers needed to follow a failed request across services.
  • No persistent crash artifacts: The ephemeral task storage of AWS Fargate disappeared when the task stopped, taking local crash dumps and diagnostic files with it.

The solution: FireLens sidecar + Amazon EventBridge persistence

Figure 3 shows the crash observability pipeline, where the application container emits structured JSON logs with X-Ray trace IDs to a FireLens (Fluent Bit) sidecar. The sidecar survives container termination and routes logs to CloudWatch Logs with Infrequent Access class and 30-day retention, as well as to Amazon Simple Storage Service (Amazon S3) with gzip partitioning and 365-day retention. Amazon EventBridge captures task-stop events with full metadata, while the AWS Distro for OpenTelemetry (ADOT) collector provides auto-instrumented traces to X-Ray and EFS persists crash dumps across availability zones, all at approximately $10/month per service.

Crash observability pipeline. FireLens operates at the container runtime level, surviving application container crashes. Amazon EventBridge persists task metadata for 90 days

Figure 3: Crash observability pipeline. FireLens operates at the container runtime level, surviving application container crashes. Amazon EventBridge persists task metadata for 90 days

FireLens (Fluent Bit) as a sidecar is the key architectural choice. ECS redirects the application container’s stdout at the container runtime level. If the app container receives an out-of-memory (OOM) kill, Fluent Bit still has the logs.

For hard crashes where application logs are not sufficient (stack overflow, native code fault), the .NET Core runtime automatically writes a mini dump to an Amazon EFS mount at /tmp/dumps/<container-id>. Because Amazon EFS writes are immediately durable and the mount survives container death, the process state (thread call stacks, heap objects, loaded modules) helps preserve diagnostic information after a SIGKILL. An AWS Lambda function copies dumps to S3 for analysis without VPC access. Cost: ~$0.02/month.

Amazon EventBridge to Amazon CloudWatch Logs persists the complete Amazon ECS task state change event with 90-day retention. No AWS Lambda function is required.

Every log line is structured JSON with the AWS X-Ray trace ID. The support workflow: search for an error, find the trace ID, open X-Ray, and see the full request flow across all 21 services.

We also use CloudWatch Logs Infrequent Access class with FireLens dual output (CloudWatch + S3), reducing observability costs by 62 percent per service.

Cost: approximately $10/month per service for the complete pipeline.

Solution 4: Passkeys for multi-tenant authentication

QSR mobile apps need instant login. Every second of friction is a lost order. The service previously relied on SMS one-time password (OTP) for every login, which at millions of monthly active users meant significant costs and user friction.

Passkeys with per-brand cryptographic isolation

We implemented passkeys (WebAuthn) using the native support in Amazon Cognito:

  • First-time registration: Phone number + OTP verifies identity, then the device registers a passkey bound to the Secure Enclave.
  • Every subsequent login: Face ID or fingerprint, which completes in about 2 seconds with no SMS cost.
  • New device or stale profile: Because carriers reassign phone numbers, possession of a number is treated as a claim rather than proof of identity. Registering a passkey on a new device repeats the OTP verification, and a profile whose last verification predates our revalidation window is re-verified before any registered credential is honoured.
  • Per-brand isolation: Each brand operates its own Cognito User Pool with a distinct Relying Party ID that is cryptographically isolated at the OS level.

A Post-Authentication Lambda (about 60 lines) maintains a customer authentication profile in DynamoDB, keyed by a hash of the brand/tenant id and the customer’s phone number. Because the brand ID is part of the key, the same customer registering with two brands produces two unrelated records, and the system cannot locate any record without the brand context. The profile stores only authentication state, such as which devices were registered and when each was last verified. It holds no order history, loyalty balance, or other brand-specific data. Passkey credentials are isolated by the same principle: because each brand uses a distinct Relying Party ID, the authentication flow prevents presenting a credential registered for one brand to another brand, and the device enforces that boundary. The implementation required about 95 lines of custom code to integrate with Amazon Cognito’s authentication logic.

Metric Before After
Cost per login ~$0.03 (SMS OTP) $0 (passkey)
Login latency 15–30 seconds ~2 seconds
SMS cost reduction — 80%+ at steady state
Phishing resistance None Cryptographic (device-bound)

Overall results

The architectural evolution delivered compounding savings:

Dimension Before After Improvement
Compute cost AWS Fargate On-Demand EC2 + Warm pools + ODCR + Compute Savings Plan 47–64% reduction
Scaling speed ~ 5-7 minutes 30–60 seconds 5–10× faster
Observability cost Standard CW Logs IA + FireLens + S3 62% reduction
Authentication cost SMS OTP every login Passkeys 80%+ reduction
Tenant onboarding Manual, multi-day Configuration-driven Days → minutes
Crash diagnosis “Exit Code: 1” Full trace + stack trace Blind → full visibility

For a service serving over 5 brands, the combined annual savings exceed 60 percent of the previous infrastructure cost, while simultaneously improving scaling speed, operational visibility, and user experience.

Conclusion

In this post, we showed how Cognizant OrderServ evolved from a single tenant microservices service to a multi-tenant SaaS service by making four architectural changes: shared compute with a tenancy library, Amazon EC2 warm pools for sub-minute scaling, a crash observability pipeline, and passkeys for frictionless authentication. The architecture addresses key failure modes through tenant validation, capacity headroom, per-tenant throttling and dedicated-cluster for high-volume brands. These capabilities introduce trade-offs, including greater application-level isolation, EC2 capacity management and compute Savings Plan commitments.

The key insight is that multi-tenancy does not require infrastructure isolation for every tenant. A well-designed tenancy library with strict resource-routing conventions provides application-level isolation while sharing compute, making the isolation model a deployment decision rather than an architectural constraint.

If you operate a multi-brand or multi-tenant service on Amazon ECS and face similar challenges with cost scaling or surge handling, consider these patterns:

  • Evaluate whether your isolation requirements truly need per-tenant infrastructure, or whether application-level isolation with per-tenant data stores is sufficient.
  • For workloads with unpredictable surges, compare the total cost of AWS Fargate over-provisioning against EC2 warm pools with compute Savings Plan.
  • Use FireLens sidecars for log delivery in any environment where container crashes are possible.

To learn more, explore these resources:


About the authors

Supermicro Super AI Station Review: The AI Server That Doubles as a Workstation

Post Syndicated from Ryan Smith original https://www.servethehome.com/supermicro-super-ai-station-review-the-ai-server-that-doubles-as-a-workstation/

Today we are looking at Supermicro’s Super AI Station, the company’s neat take on a DGX Station System with a design that can stand as a tower or be rackmounted as a 5U server

The post Supermicro Super AI Station Review: The AI Server That Doubles as a Workstation appeared first on ServeTheHome.

Inside GitHub’s new Git storage architecture

Post Syndicated from Karthik Raman original https://github.blog/engineering/architecture-optimization/inside-githubs-new-git-storage-architecture/

Last week at GitHub Universe, we announced our new Git storage architecture, capable of handling thousands of writes per second with reads that scale on demand. Through internal testing, we’ve demonstrated the ability to support 5,000 concurrent agents working on a repository, sending 7,200 pushes per second to a repo, all while serving read traffic with consistently low latency. This new architecture is the foundation for GitHub to support agentic development for hundreds of millions of developers, and we’ve been building it underneath GitHub while it keeps running.

Over the past year, agents changed what a busy repository looks like on GitHub. In our first post in this series, we laid out the two principles behind our next-generation Git storage architecture: minimize coordination, and decouple storage from compute. In our last post, we covered the bridge: moving 99.99999% of repositories into Azure, scoping the write quorum to individual references, and standing up virtual replicas backed by Azure Blob Storage.

We also left off with an observation: very little of a write needs to be ordered. Git objects are immutable and named by their contents, meaning two writers can store them at the same time without agreeing on anything. The part of a push that needs agreement is the reference update, and a reference update is tiny in comparison. Virtual replicas were the first step toward building around that, and the means by which we onboard repositories to the new architecture.

The rest of the architecture builds on that observation. Only reference updates need agreement. That lets the parts of Git infrastructure that used to be tightly coupled (storing data, serving requests, and maintaining repositories) scale independently. For developers, Git stays Git: they push, fetch, and branch the same way, with the same protections in place. Below, we walk through the requirements behind the design, follow a request through the system, and dive deep into two of its components.

Building for agent scale

We think about the scale of the GitHub platform along two axes:

  1. Horizontal scale is how many developers and agents can work across many repositories. Many of GitHub’s current approaches on this axis have been battle-tested over the years and are carried forward to the new architecture: partitioning of tenants across infrastructure, isolation and/or fairness enforcement of tenant traffic, and load-aware routing, to name a few.
  2. Vertical scale is how many developers and agents can work on one repository. As agents write more code, we expect individual repositories to get larger and busier, so most of our requirements came from the vertical axis.

Today, the busiest repositories on GitHub.com receive at most dozens of writes in a single second, typically in short bursts. As more of that work shifts to agents running in parallel and around the clock, we expect those bursts to become a sustained rate within the next year. Each push leads to more reads: CI runs, deployment systems pull the new commit, and other agents fetch it for context. This read fan-out results in a large amplification effect on overall traffic that scales with writes.

We’re already seeing this trend play out on repositories at the top of the curve. The chart below shows daily requests to one repository on GitHub from March to May, when its traffic grew about 4x. Its growth has continued on an exponential trajectory since then.

Line chart of daily requests to a single repository from March through May 2026. The March average is 44M requests per day and the May average is 181M requests per day, about a 4x increase.

We have to plan for a world where agent-driven development keeps accelerating at this speed. Instead of planning for the next increment, we set a target of at least two orders of magnitude more scale. For a single repository, that meant the new architecture needed to:

  • Accept thousands of writes per second.
  • Support multiple writer nodes. There are two reasons for this:
    • Experience has taught us that indexing incoming packs and checking object connectivity are the most CPU-expensive parts of a write, and that cost scales with the size of the write, so a single node will become compute-bound quickly.
    • Spreading that work across many nodes improves fault tolerance and shortens recovery when a node fails.
  • Minimize commit latency. An agent waits on each push before it moves to its next step.
  • Record changes in order for each reference with the ability to quickly linearize across references. A virtual replica that falls behind needs to catch up by downloading only the data it needs to serve the incoming request.
  • Keep reads cheap when nothing has changed, which is the common case given the read-to-write ratio.
  • Let each component scale independently.

On top of that, the system had to be able to improve overall reliability and scale globally.

Introducing “Project Rubicon”

In May of 2026, hot on the heels of the initial Azure migration, we started working on a design that could meet those demands. We tried several approaches. In all of them, pack data lived in Azure Blob Storage, and in some we experimented with keeping reference state there too. We’ll cover some of the trade-offs we found below, but we settled on a design we felt maximized throughput while preserving the correctness, durability, availability, and latency guarantees a platform like GitHub requires. We called it “Rubicon,” a nod to the point of no return the industry crossed with agentic development.

Let’s walk through Rubicon from the top down. Many things stay the same: requests from Git clients, the API, and the web UI arrive at Spokes, which still handles routing and load balancing, and serves the internal API that the rest of GitHub uses to read and write repository data. Spokes also maintains its role as a placement and balancing service, maintaining the partitioning, isolation, and tenant fairness required for horizontal scale. The data plane underneath Spokes, in contrast, is new.

Architecture diagram. Git, API, and Web requests pass through Spokes to groups of virtual replicas per repository. Virtual replicas hold only cached state. Ref-CAS stores refs and the pack manifest, and Azure Blob Storage stores pack data. Compaction nodes compact and garbage collect packs in Blob Storage and update the pack manifest in Ref-CAS.

Spokes routes each request to one of a small group of virtual replicas assigned to that repository. Virtual replicas are a single fleet of disposable compute that serves reads, writes, or both. A virtual replica’s local disk is a cache that can be rebuilt at any time.

Spokes routes a repository’s traffic to the same virtual replicas when it can, which keeps their caches warm and latency low, but nothing about correctness depends on a request landing on any particular virtual replica. When a virtual replica goes away, the repository’s other virtual replicas keep serving, and a replacement fills its cache as traffic arrives.

When a repository gets busy, we add more virtual replicas. Separately, a fleet of compaction nodes works in the background, directly against Blob Storage and Ref-CAS, to merge each repository’s packs and remove unreachable objects. That work never runs on a virtual replica, meaning their CPU and memory can be almost entirely dedicated to serving requests.

Durable state lives in two places:

  1. Git objects, in the form of packs and auxiliary read acceleration data, live in Azure Blob Storage.
  2. References and the list of packs that make up the repository (the pack manifest) live in a separate replicated log service we call Ref-CAS (for “Reference Compare-and-Swap”).

Let’s walk the architecture from the perspective of a write and a read:

  • Write: a virtual replica indexes the incoming pack and uploads it to Blob Storage, then commits one transaction to Ref-CAS that lists the pack in the manifest and updates the references. Later, a compaction node merges that pack with others to keep reads fast.
  • Read: a virtual replica checks its reference version with Ref-CAS, pulls the packs it needs from Blob Storage into its cache, and serves the read from there.

There are many distinct approaches we’ve taken within the new architecture, from the software stack on the virtual replicas to how traffic routing and workload scheduling are tuned for different use cases (e.g., many small repos vs. one large one). Let’s dive deep into two components of the architecture and tell their stories centered on the two design principles.

Minimizing coordination: mutating reference state at high throughput

Object storage services like Azure Blob Storage are incredibly good at what they do, and natural fits for many data-heavy applications like source code storage. They are durable and replicated, hold as much as we need them to, have a relatively low cost per byte, and deliver exceptionally high aggregate throughput so long as traffic is spread across many keys. As a result, nearly everything durable in the new Git storage architecture lives in Azure Blob Storage, including:

  • Packs and indexes written by pushes, compaction, and garbage collection.
  • Various read accelerators (like multi-pack indexes, reachability bitmaps, and some custom formats) that our maintenance process builds from those packs.
  • Records that let compaction and garbage collection resume or audit their work, along with the leases that compaction nodes use to claim work.
  • Cached responses for frequently repeated fetches.
  • Checkpoints for the reference state of the repository, though not the authoritative state used to serve requests.
Diagram of one repository's data in Azure Blob Storage. Push, Compaction, Maintenance, GC, and Ref-CAS write into it, and fetches read from it by exact key. It holds packs and indexes, records and leases, read accelerators such as the multi-pack index and reachability bitmap, cached fetch responses, and reference checkpoints.

Most of that data is written once to a distinct key and never changes afterward, which is what makes it highly suitable for Blob Storage.

References are the opposite. A busy repository’s references are a relatively small piece of state that many writes race to update at the same time. That access pattern is harder to serve from Blob Storage, and our first prototypes showed us why. Object storage for this kind of workload is an intuitive choice, and one that works well up to a certain level of scale: conditional writes give a compare-and-swap for free, and the data is already durable and replicated. However, we ran into a difficult set of choices, illustrated below by two approaches we tried.

Our first approach was a write-ahead log for all of a repository’s references, indexed by a single blob. Every update is a conditional write against that blob. It’s correct, and reads are cheap: a virtual replica asks whether the blob has changed, and if it hasn’t, it’s done. Writes are where it falls short. Only one conditional write to a blob can be in flight at a time, so even when we batch every push that arrives during a write into the next one, the repository’s write ceiling is one write per round trip to a single blob. In practice, this netted us write throughput in the mid-hundreds per second.

Our second approach split the references across several segment blobs, with each reference hashing to one of them. Writes to references in different segments no longer wait on each other, which raises the ceiling. But a read now has to check every segment to know whether anything changed, and it’s only as fast as the slowest segment. We’d traded write latency for read latency, and reads outnumber writes by orders of magnitude. We can visualize this by showing multiple writes and a read side by side:

Neither option met every requirement, so we built something else: a replicated log service dedicated to references, or the Ref-CAS service. This is the (small) consensus group we said at the end of the last post was still somewhere in the stack. Here’s how it works.

Each reference update in Ref-CAS is a compare-and-swap committed by a quorum. A transaction locks only the references it touches, checks that each one still has the value the writer expects, and becomes visible once a majority of the log’s replicas have recorded it. Agents mostly push to their own branches, so most updates don’t contend, and they replicate in parallel. When two pushes do update the same reference, only one will append to the log.

Each commit to Ref-CAS gets a log sequence number (LSN), and the repository’s version is the LSN of its latest commit. Reads start with a version check against that version. A virtual replica sends the version of the reference state it already holds. If that version is still current, the reply carries no references at all, and an unchanged repository costs a single comparison. If the virtual replica is behind, it gets back the current references and version, along with pointers to pack data and anything else that needs to be known, such as when the last garbage collection cycle occurred.

Unlike the consensus group in our legacy architecture, Ref-CAS never holds or validates object data. It only stores pointers to it. A virtual replica uploads a push’s pack to Blob Storage before it commits the reference transaction. A reference only ever points at objects that are already durable.

These approaches allow us to transact reference state alone in the tens of thousands of operations per second, and provide a baseline for additional annotations on the replication log that help reads decide what to download for a given request.

But supporting thousands of writes per second on a single repository creates a second-order problem: maintaining the performance of reads and writes. For that, we needed a different approach to compaction.

Decoupling storage from compute: compaction at thousands of writes per second

Every push adds a pack to Blob Storage. Git can serve a repository from many packs, but each additional pack makes lookups slightly more expensive, so the pack count can’t grow without bound. In our legacy architecture, compaction runs on the same replicas that serve traffic, and it has to run on every replica. In the new architecture, it runs entirely in the background on separate nodes, directly against durable storage, using the Ref-CAS service to notify virtual replicas of the evolving state.

Compaction nodes each claim ownership over a subset of the repository’s data via a lease, which allows us to scale compaction horizontally. Here’s how one compaction node works through a single shard of a repository’s packs:

The node first takes a lease on the shard in Blob Storage, so only one node compacts a shard at a time. Meanwhile, virtual replicas keep uploading packs and listing them in the pack manifest. The node reads the manifest to find new packs and collects them into an open segment, which is effectively a holding ground for data about to be compacted. When the segment reaches a defined limit based on pack count, size, or age, it seals the segment so it accepts no new packs, then combines its packs into fewer, larger ones. While compaction happens, the next segment for the shard can begin capturing its input packs. Before anything is published, the node checks that the new packs contain exactly the same objects as the inputs. It then uploads the new packs and adds them to the manifest. A second transaction removes the inputs from the manifest, and cleanup deletes them later.

To scale out, this process runs independently for many shards at once. Each pack’s hash maps to a shard, so adding shards and nodes adds compaction throughput. A compacted pack that’s still under the size limit goes back into a shard by its own hash and merges again, so packs grow in tiers until they reach the limit. The net result is an effectively flat pack count while pushes climb:

Compaction merges packs, but it keeps every object. Garbage collection is the step that removes objects nothing references anymore, like the ones left behind by deleted branches and force-pushes. In our new architecture, garbage collection works very similarly to compaction by using dedicated workers to handle the compute-heavy portion. It also has a few special behaviors for safely removing data, which are worthy of their own discussion.

Using these approaches, we’ve been able to consistently demonstrate a flat latency curve as writes climb, and we have other planned optimizations to make the compaction and garbage collection processes themselves more performant.

What comes next

The two principles we started with let each part of this architecture scale on its own. We scale reads and writes by adding virtual replicas. Ref-CAS orders each reference independently instead of the whole repository. We scale maintenance by adding compaction nodes. Our goal is for every repository, from one developer’s side project to the busiest monorepo, to run on the same architecture with no special handling for the busy ones. Repositories don’t stay small. A side project can pick up a fleet of agents overnight, and it shouldn’t need a migration or special handling from us to keep up. One architecture also means every improvement we make reaches every repository at once.

This foundation lets us do a lot more. We’re focused on three areas next: scaling the architecture beyond what we can predict we’ll need today, making agents more efficient at the work they do on GitHub, and evolving Git itself for agentic development.

Improving the architecture

We have plans to take this architecture further. Each of these improvements lets a repository handle more activity while reducing the work behind each request:

  • Read only what’s needed. Fetch a single object from Blob Storage instead of the whole pack that holds it.
  • Organize packs by type. Group commits, trees, and blobs into their own packs. Many requests then touch less data.
  • Partial replicas. Serve requests without every pack on local disk.
  • Materialized views. Keep repository data in ready-to-use forms, like prepared archives, a near-live shallow copy of the default branch, and agent session context stored alongside each commit.
  • Global replicas. Place replicas in any region, close to the workloads that use them. Reads stay within bounded staleness of the latest write.
  • Scale to zero. Grow and shrink the fleet with the repositories that are actually active.

Efficient agent operations on GitHub

Scaling the infrastructure is one side of supporting agents. The other side is making each agent operation cheaper: fewer bytes to fetch, fewer round trips, and less waiting on changes. Each of these fast-follow items builds on the storage work in this post:

  • Work without a clone. Many agent tasks never need a full working tree. They need a handful of files and enough context to make a change. They can read both straight from the server.
  • Atomic commit API. Land a multi-file change in one operation. The whole change commits, or it’s rejected cleanly on conflict.
  • Event streams. Agents and the systems around them work best with real-time information. A generic event stream lets them subscribe to changes and react as they happen.

Improving Git for agents

We also have work to do in Git itself. GitHub has been a longtime contributor to the Git project, and we’re not stopping now. A few examples of the work we’re excited to continue:

  • Lightweight clones. Partial clone, shallow clone, and sparse checkout each let an agent fetch less. Each one has gaps. They don’t always work well together, either. We want to improve each piece and make them fit.
  • Remove hidden scaling costs. Git still has code paths that slow down much faster than the repository grows. Recent releases fixed several of them, but we expect to find more.
  • Faster history walks. Metadata like the commit-graph can answer merge-base and reachability questions without walking every commit. That matters most when an agent’s API call turns into Git work on the server.
  • Avoid common gotchas. Agents make the same mistakes developers do. Git can catch more of the common ones and warn before they cause damage.

Agents have changed how software gets built. GitHub and Git are changing with it. We’re excited to take this new architecture across the starting line. It’s available in a limited private preview now, with a broader rollout early next year. We can’t wait to see what teams build when the number of agents working on a repository is limited by their ideas. We’ll keep evolving Git and GitHub alongside our customers and the Git community.

The post Inside GitHub’s new Git storage architecture appeared first on The GitHub Blog.

The collective thoughts of the interwebz