At Grab’s scale, managing petabytes of data across billions of S3 objects demands more than a storage layer. It demands a robust architectural primitive that supports the high-concurrency needs of a modern “Lakehouse.” Our goal is full storage-compute separation, leveraging S3 as an elastic foundation for both near-real-time metrics and large-scale batch transformations.
For years, the vast majority of our tables were Hive Parquet, managed through the Hive Metastore with a directory-based layout. This model served us well, but as data volume grew, the directory-and-metastore approach became the limiting factor. We are now transitioning to a table-centric architecture built on modern table formats, treating data as a first-class primitive to ensure consistency and performance across our internal data transformation platforms: Slide, which powers batch transformations, and Hugo, which handles online-to-data-lake ingestion. Along the way, we also built the UnifiedSparkCatalog, a unified Spark catalog that hides table-format differences from users entirely, which we are open-sourcing alongside this post.
The catalyst for change: Challenges with Hive Parquet
For years, Hive Parquet was the backbone of our Data Lake, representing the vast majority of our tables. However, as data volume scaled, the architectural limitations of directory-based storage became apparent. We identified four primary bottlenecks:
Catalog latency: The Hive Metastore (HMS) became a centralized failure point. High concurrency during metadata access led to O(n) listing overhead, where query planning time scaled linearly with partition count, crippling throughput.
The small file problem: The directory layout left us with severe file fragmentation. Certain Machine Learning (ML) datasets had an average file size under 1 MB, with thousands of files in each partition. At this scale, the overhead of S3 object listing and metadata request latency drove up Application Programming Interface (API) costs and slowed scan operations.
Operational toil: Data engineers faced constant manual overhead for partition registration. Without native ACID support (no native UPSERT or DELETE), teams relied on complex workarounds to manage data changes carefully.
The broken information loop: A fundamental disconnect existed between the catalog and storage. Because the HMS, not the storage layer, was treated as the source of truth, direct S3 modifications frequently left the catalog stale and out of sync with the actual state on disk.
Why Iceberg? Strategic alignment and future-proofing
We evaluated several open table formats before selecting Apache Iceberg as our default. The deciding factors came down to community governance, engine compatibility, and long-term flexibility.
Recent industry momentum, including growing cloud-native support for Iceberg, further validates this direction. We are positioning Grab to be format-agnostic in the long term, but Iceberg provides the most mature foundation today.
Adopting Iceberg at scale
Migrating an established lake is not a flag flip. Our challenge was rolling out Iceberg across a lake that was overwhelmingly Hive Parquet, queried by many engines and teams, without breaking the downstream consumers that depended on those tables. Rather than converting everything at once, we moved the highest-value tables first. The efficiency gains across our production workloads have been substantial. Here are representative examples:
Query performance via Z-ordering: On a high-traffic navigation dataset, we achieved roughly a 10x improvement in query runtime. Z-ordering co-locates rows with similar values across specified dimensions, enabling Trino to leverage data skipping and min/max statistics to prune irrelevant files during query planning. This reduced query runtime from 70 seconds to 6 seconds.
S3 API cost reduction: For a heavily queried operations table, daily S3 API costs were reduced by up to 95% with no changes to the queries themselves. Larger file sizes and the elimination of expensive object listing during query planning drove most of the savings.
Compute savings: For a dataset used in funnel analysis, we reduced cluster resource usage by approximately half. A separate ML feature pipeline also improved feature freshness for downstream models.
The UnifiedSparkCatalog: Making mixed formats transparent
Migrating to Iceberg solved our storage and metadata problems, but it surfaced a new one at the developer-experience layer. Modern table formats like Delta, Iceberg, and Hudi each implement their own custom catalog that extends Spark’s SessionCatalog. In a standard Spark runtime, only one catalog implementation can be set as the default spark_catalog. Supporting additional formats requires explicit catalog declarations, meaning users must reference tables with format-specific prefixes like iceberg_catalog.schema.table or delta_catalog.schema.table.
With Iceberg, Delta, Hudi, and Hive tables now coexisting and tables actively migrating between formats, this created two problems: engineers had to know the underlying format of every table they queried, and any format migration silently broke every downstream query that hardcoded a prefix.
The UnifiedSparkCatalog is our answer. It is a unified Spark catalog that abstracts the complexity of working with mixed table formats so users never need to think about which format a table uses. We took inspiration from Trino’s Table Redirection, a feature that transparently points a query at the right connector when a table’s format differs from the catalog it was queried through. Our Spark equivalent works as follows:
How it works
Table detection: The catalog loads metadata from the Hive Metastore.
Format identification: A TableTypeDetector utility identifies the format based on metadata properties (e.g., the provider field) or path-based inference.
Operation routing: The catalog delegates the operation to the correct format-specific catalog (Iceberg’s SparkCatalog, Delta’s DeltaCatalog, etc.) without requiring any prefix from the user.
Key design decisions
Lazy initialization: Catalogs for each format are initialized only when first needed, reducing startup overhead. If a format’s JAR is missing from the classpath, initialization continues gracefully. The catalog simply skips that format rather than failing the entire session.
Naming as spark_catalog: The catalog reports its name as spark_catalog because Spark treats this name specially for legacy Hive Data Manipulation Language (DML) operations. Many internal Spark code paths check for this exact name to determine whether to use Hive-compatible logic for inserts, updates, and deletes. Using any other name would break legacy Hive table operations.
Catalog reuse: Before creating a new catalog instance, the system checks whether one already exists in Spark’s catalog manager. This preserves compatibility with plugins like OpenLineage, which inspect catalog class types for lineage extraction.
Fallback behavior: If a table is not found in the expected format-specific catalog, the system falls back to the base session catalog, ensuring robust behavior for standard Hive tables.
We are open-sourcing UnifiedSparkCatalog alongside this blog post. The code and documentation are available here.
Lessons learned and overcoming hurdles
Scaling Iceberg across a large ecosystem revealed several technical nuances:
Hive lock contention: We encountered “zombie locks” in the HMS that blocked commits. We traced this to a low read timeout on the metastore side under high load. Adjusting retry intervals and increasing the timeout resolved the issue.
Timestamp handling: Spark 3.4 introduced TIMESTAMP_NTZ (no time zone), while Iceberg defaults to TIMESTAMP_LTZ (local time zone). This caused compatibility issues with legacy Hive views. We resolved it through a custom migration workflow and targeted patches to our Trino deployment to ensure consistent casting.
Storage tier costs: Generating Iceberg metadata involves reading historical data, which can trigger a one-time cost spike as files move between S3 storage tiers. To manage this, we prioritize migrations based on a table’s scan frequency and API operation costs rather than migrating the entire lake at once.
Conclusion: The road ahead
Apache Iceberg is now foundational to Grab’s data strategy. It is the default format for Slide and Hugo, and adoption is expanding across our compute platforms.
Looking forward, we are experimenting with Storage Partitioned Joins to eliminate shuffle stages in Spark and monitoring the Apache XTable project to maintain interoperability between formats. Our journey does not end with adoption. We will continue contributing back to the ecosystem, starting with the upcoming release of the UnifiedSparkCatalog.
Acknowledgments: This journey was made possible by the dedicated efforts of the Data Engineering, Infrastructure, and Search & Personalization teams at Grab.
Join us
Grab is Southeast Asia’s leading superapp, serving over 900 cities across eight countries (Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam). Through a single platform, millions of users access mobility, delivery, and digital financial services, including ride-hailing, food delivery, payments, lending, and digital banking via GXS Bank and GXBank. Founded in 2012, Grab’s mission is to drive Southeast Asia forward by creating economic empowerment for everyone while delivering sustainable financial performance and positive social impact.
Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!
In addition, AWS is introducing several new security and governance tools, including: new global condition keys for OAuth, token introspection and revocation, dynamic client registration, new AWS CloudTrail elements, and a new API for headless OAuth connectivity. All of this is compatible with your existing IAM configuration including permissions, roles, and federated access.
In this post, you’ll learn how to connect your agents to the AWS MCP Server, understand how AWS Sign-In authorizes agent access, and manage access using new security and governance capabilities.
How to connect an agent to the AWS MCP Server
This walkthrough uses Claude Code, but the same steps apply to any agent that supports Model Context Protocol (MCP) such as Kiro, Codex, and Gemini. See Setting up the AWS MCP Server for how to connect the AWS MCP Server to an agent.
Prerequisite permissions
To connect an agent to the AWS MCP Server, you’ll need the IAM permissions required for OAuth-based sign-in. The following AWS CLI command adds a managed policy with required permissions to your IAM role (remember to replace <MyRole> with your IAM role):
aws iam attach-role-policy \
--role-name <MyRole> \
--policy-arn arn:aws:iam::aws:policy/AWSMCPSignInOAuthAccessPolicy
Step 1: Configure the AWS MCP Server on your agent
Run the following command to add the AWS MCP Server endpoint to your agent’s configuration as shown in Figure 1:
claude mcp add --transport http aws-mcp https://aws-mcp.us-east-1.api.aws/mcp
Figure 1: Adding the AWS MCP Server endpoint to Claude Code
Step 2: Review the authorization request
The first time your agent needs to access the AWS MCP Server, it opens a browser and redirects you to an AWS Sign-In page, shown in Figure 2. Authenticate as you would on AWS console or AWS CLI, review the authorization request, and approve access. You should receive an Authorization successful message.
Figure 2: Review authorization request
Note that if you already have an active AWS Sign-In session (e.g., because you previously signed in to the console earlier in the day), you can reuse that session without needing to sign in again.
Step 3: Start using AWS tools
After connecting your agent to the AWS MCP Server, you can begin invoking tools provided by the server. To verify that Claude Code is connected to the AWS MCP Server, start Claude Code and run the following command:
/mcp
The command displays the configured MCP servers and confirms that the AWS MCP Server is connected and ready to use with your AWS credentials.
Figure 3 shows an example of a successful connection to the AWS MCP Server.
Figure 3: Verifying the AWS MCP Server connection in Claude Code
After the connection is established, you can ask Claude Code to invoke tools provided by the AWS MCP Server. For example, enter the following prompt:
Deploy a sample serverless web application into my development AWS account
Claude Code uses the AWS MCP Server to identify the active AWS account, confirm the target account, and describe the deployment it plans to perform before invoking AWS services on your behalf.
Figure 4 shows Claude Code confirming the active AWS account and outlining the resources that will be deployed.
Figure 4: Using Claude Code to deploy a sample serverless application through the AWS MCP Server
Authorization models and how they work
AWS Sign-In supports two authorization models for connecting agents to the AWS MCP Server:
Interactive authorization for developer’s AI agents using browser based authentication
Non-interactive (headless) authorization for applications and AI agents that already have AWS credentials and don’t have access to a browser
Note that authorizing an agent allows it to access the AWS MCP Server on your behalf. It doesn’t grant the agent additional AWS permissions. Every request is still evaluated using your existing IAM policies, SCPs, RCPs, permission boundaries, and other organizational controls.
Interactive access
In the interactive case, the agent first discovers the AWS Sign-In OAuth server and then registers itself as an OAuth client using Dynamic Client Registration (DCR). It then redirects you to an AWS Sign-In page where you authenticate and authorize access (step 2 in the preceding section). After successful authorization, AWS Sign-In then issues short-lived access tokens and refresh tokens that authorize the agent to access the AWS MCP Server on your behalf. AWS Sign-In automatically manages token issuance and token refresh, enabling authorized agents to continue accessing the AWS MCP Server without requiring you to repeatedly sign in.
The interactive authorization model supports three distinct sign-in methods: native AWS IAM credentials for individual developers, managed access through AWS IAM Identity Center for enterprises, and seamless federated access via third-party providers like Okta and Ping Identity for larger organizations
OAuth server metadata and DCR
Before an agent can request authorization, it must discover the AWS Sign-In OAuth endpoints and register itself as an OAuth client. AWS Sign-In supports OAuth metadata discovery and DCR, allowing supported agents to configure themselves automatically without requiring developers to manually provision OAuth client IDs and client secretsWhen an agent connects to the AWS MCP Server for the first time, it retrieves the AWS MCP Server’s protected resource metadata (RFC 9728) and the AWS Sign-In OAuth metadata (RFC 8414). The agent then uses RFC 7591)) to register with AWS Sign-In, obtain a client ID, and initiate the standard OAuth authorization code flow.
AWS Sign-In supports OAuth discovery and DCR for agents running on local workstations and supported hosted environments. For the current list of supported agents and environments, see Supported redirect URIs for the AWS MCP Server.
Non-interactive access to the AWS MCP Server
Non-interactive (headless) authorization is for agents and applications that run without a browser or human in the loop, and thus don’t require interactive sign-in. This allows agents that already have AWS credentials to obtain OAuth access tokens and connect to the AWS MCP Server. The following is an example of how to obtain an access token.
In the non-interactive case, AWS Sign-In implements the OAuth client credentials grant using AWS security credentials instead of a static client secret. Applications authenticate to the AWS Sign-In token endpoint using SigV4 creds, and AWS Sign-In returns a short-lived OAuth access token that can be used to access the AWS MCP Server.
Please note you may have to update the SDK and AWS CLI, please refer to CLI guide.
Managing OAuth access
AWS Sign-In extends the existing IAM authorization model with capabilities for governing OAuth access to the AWS MCP Server. Administrators can use familiar IAM policies together with new OAuth-specific controls.
Granting OAuth permissions
OAuth access is governed using IAM policies and requires the following IAM actions:
signin:AuthorizeOAuth2Access – Allows users to sign in interactively using the OAuth authorization code flow
signin:CreateOAuth2Token – Allows applications to obtain OAuth access tokens by exchanging authorization codes, refresh tokens, or using client credentials
When an application requests access, AWS Sign-In creates an OAuth authorization grant between the agent and the AWS MCP Server. This grant is represented as an IAM resource, which the preceding AWS Sign-In actions are authorized against.
OAuth authorization grants are represented as IAM resource enabling administrators to use standard IAM policy constructs, including global condition keys, together with OAuth-specific condition keys to control how authorization grants are created and used.
Governing OAuth access
AWS Sign-In introduces OAuth-specific condition keys that allow administrators to govern how agents obtain OAuth authorization. The following examples demonstrate common governance patterns.
To restrict OAuth authorization to localhost:
In addition to accessing the AWS MCP Server with agents on your local workstation, AWS supports signing into the AWS MCP Server on select hosted providers through dynamic client registration. Click here to view the list of supported remote providers. Many organizations want to allow developers to authorize agents running on their local workstations while preventing OAuth tokens from being delivered to untrusted redirect URIs or using unsupported authorization flows. The following policy allows only the OAuth authorization code and refresh token flows for the AWS MCP server and restricts token delivery tolocalhost.
Use the aws:SignInSessionArn global condition key to deny authorization associated with a specific sign-in session. This allows administrators to contain a suspicious or compromised authorization session without affecting other active sessions.
These examples demonstrate common governance patterns. Additional IAM and SCP examples are available in the AWS Sign-In condition keys reference.
Revoking OAuth tokens
AWS Sign-In provides OAuth token introspection and token revocation APIs that allow administrators to build custom tools for token validation and revocation. Access to these APIs is controlled through the signin:IntrospectOAuth2Token and signin:RevokeOAuth2Token permissions. IAM principals with permissions are allowed to introspect and revoke tokens for the same account.
The introspection API can be used to determine whether a token is active and obtain information about the associated authorization. The revocation API allows administrators and security tools to revoke individual refresh tokens without affecting other active sessions. For example, if an organization needs to invalidate access for a specific OAuth authorization, account admins can revoke the associated refresh token without affecting other active sessions.
Monitoring OAuth activity
OAuth-related activities are recorded in AWS CloudTrail, including authorization requests, token issuance, token revocation, and token introspection events. CloudTrail logs also capture details such as the OAuth client, target the AWS MCP Server, redirect URI, authorization flow, and associated sign-in session. In addition, AWS API calls made using OAuth access tokens include the associated aws:SignInSessionArn context, allowing organizations to correlate API activity with the originating OAuth sign-in session.
This allows security teams to monitor OAuth usage, investigate authorization activity, detect anomalous behavior, and integrate OAuth events into existing auditing, compliance, and incident response workflows alongside other AWS activity.
Here’s a CloudTrail sample for an AuthorizeOAuth2Access event:
AWS Sign-In support for OAuth enables you to securely connect to the AWS MCP Server using industry-standard authorization. This release simplifies application and agent integration with AWS while supporting your existing IAM setup, governance, and auditing capabilities.
Specification-driven composition addresses a common scalability bottleneck in data pipelines. Data pipelines often start as simple scripts, but as they grow, you duplicate transformation logic and small changes cascade across multiple workflows. Copying and modifying data transformation logic across scripts leads to workflows that become difficult to manage at scale. Tracking what each pipeline does becomes harder because workflow intent is embedded in code. This lack of visibility complicates governance, especially in regulated environments such as healthcare, finance, and life sciences.
Many implementations combine orchestration, transformation logic, and validation rules in the same scripts. Supporting new datasets requires modifying and redeploying code, while validation often happens only during processing. As a result, issues surface late in the lifecycle which increases the operational risk. Specification-driven composition separates workflow intent from implementation so you can build flexible data workflows.
You can separate workflow intent from processing logic with specification-driven composition. This approach reduces duplication, shortens the time required to onboard new datasets, and improves consistency across workflows. Instead of embedding logic in scripts, the system describes workflow intent in a structured specification, validates the specification before processing, and dynamically assembles a processing pipeline.
This approach moves pipeline configuration outside application code and composes pipelines from reusable processing components. To separate concerns, it organizes a workflow into three layers, as shown in Figure 1. The intent layer defines workflow behavior using specifications. The composition layer validates specifications and assembles pipelines. The processing layer runs the pipeline of transformation steps.
Specification-driven composition provides several practical benefits when you manage multiple pipelines. First, it improves governance because specifications provide a clear, traceable description of workflow behavior that you can review and validate before invocation. In regulated industries, this traceability shortens audit preparation time and reduces the review burden for new dataset submissions.
Second, the pattern supports reusable transformations. You implement transformation logic once and reuse it across multiple workflows. This reduces duplication and improves consistency. In practice, teams adopting this pattern report being able to onboard new datasets in days rather than weeks because most required capabilities already exist in the registry.
Third, specification-driven composition enables flexible pipeline design. You define new specifications to create pipelines supporting new datasets and use cases without modifying application code and registering new system release.
Fourth, the pattern separates business intent from execution artifacts. The specification expresses what the workflow should do in domain terms, while the generated state machine remains a system artifact. This separation matters in regulated environments (for example, GxP) where business users author intent but are not allowed to author or modify execution code directly.
Finally, the declarative representation of workflows lets AI tools assist with capability discovery, specification authoring, and pipeline analysis, while runtime behavior stays predictable as it relies on validated capabilities.
Core components
You work with four key components to define, validate, and run workflows.
Specification
A specification is a structured document, typically JSON or YAML, that describes datasets, mappings, and transformations. It defines what the workflow should do without including processing logic. Because specifications are explicit and versioned, they provide a clear record of workflow intent.
Composer
The composer converts specifications into runnable steps, so workflows run consistently without embedding transformation logic in application code. It checks that referenced capabilities exist, retrieves metadata, and builds a workflow that can run on the processing layer. The composer does not perform transformations. It only assembles the workflow. In practice, the composer compiles business intent expressed in the specification into a code artifact such as an Amazon States Language (ASL) definition. This abstraction lets domain users author specifications without producing runnable code, which is important in environments with strict separation of duties.
Capability registry
The registry stores metadata about reusable transformation functions. This includes identifiers, input/output formats, invocation details, and permission boundaries. The composer uses the registry to validate specifications and locate capabilities. Treat this registry as a governed artifact rather than a manually edited lookup table. Capability definitions live in version control, and your CI/CD pipeline validates metadata and runs tests before you publish new versions. Specification authors include explicit capability version references in the specifications to support reproducible workflow runs. In regulated systems, you must validate new capabilities and obtain approval through a separate workflow before you register them.
Capability pipeline
Once assembled, the pipeline runs a sequence of transformation steps. Each step performs a specific operation such as formatting, validation, or enrichment. Because these steps are reusable, you can apply them across many workflows.
Technical implementation
Let’s walk through a technical implementation of this pattern using serverless AWS services. In this example, workflow specifications are uploaded to an S3 bucket. An AWS Lambda composer retrieves and validates the specification, looking up capability metadata in Amazon OpenSearch Service. The composer then assembles the workflow in AWS Step Functions, which orchestrates the workflow and invokes AWS Lambda capability processors. Each processor emits traces to Amazon CloudWatch Logs.
Figure 2 shows the architecture.
Figure 2. AWS implementation of Specification-Driven Composition
Interpreting the workflow specification
The workflow specification defines datasets and transformation logic using a structured format (in this example, JSON). A specification is a declarative document that describes what the pipeline should produce rather than how to produce it. Your composer reads the specification, validates it against a schema, and uses it to construct the pipeline.
The following example maps source fields to target fields using reusable capabilities (Figure 3). It takes data from a source dataset (‘raw_orders’), maps specific fields (‘order_date’, ‘amount’) and applies reusable transformation capabilities such as ‘format_date’ and ‘normalize_currency’. Each mapping explicitly links a source field to a target field and references a capability that performs the transformation. Source dataset(s) are listed under the ‘source’ section of the JSON document, target datasets under the ‘target’ section, and mappings ‘mappings’. You can define your own specification structure and build custom validation logic in your composer to make sure specifications are valid.
You can model preprocessing steps such as column standardization, numeric casting, or unit conversion as first-class capabilities and reference them earlier in the specification (for example, in a dedicated ‘preprocessing’ section) before downstream mappings run. This way, preparation logic uses the same metadata, versioning, validation, and observability model as the rest of the workflow, which simplifies lineage and review.
In this example, an S3 event notification invokes the composer Lambda function when you upload a specification. In practice, the same composer can be invoked through several mechanisms depending on the use case: S3 events for new specifications, an Amazon EventBridge schedule for recurring runs, an API or UI action for on-demand invocation, a direct Step Functions StartExecution call, or an upstream pipeline. This flexibility lets you re-run an approved specification against refreshed data without re-authoring or re-approving the workflow.
The composer parses the specification and validates the referenced capabilities by querying Amazon OpenSearch Service to retrieve capability metadata such as Amazon Resource Names (ARNs). OpenSearch Service is used here because the composer does more than direct-key lookups. It supports capability discovery through full-text and semantic search over capability metadata such as capability descriptions, input/output schemas, and tags. This lets data authors and AI tools find reusable capabilities by intent rather than by exact identifier. After validation, the composer assembles and starts an AWS Step Functions state machine which invokes each capability in sequence. Each capability runs independently, making the pipeline modular and reusable.
Securing sensitive data flows
This pattern suits regulated workloads, so handle security as part of the design. Use SSE-KMS with a customer managed key on the specification and data S3 buckets and enable encryption at rest on the Amazon OpenSearch Service domain. Enforce HTTPS (TLS) access with an S3 bucket policy (aws:SecureTransport) and enable node-to-node encryption on Amazon OpenSearch Service. AWS Step Functions and Lambda calls use TLS by default. Each capability processor receives only the fields its mapping references, and you can apply IAM policy at the source bucket to restrict access. Processors emit traces to Amazon CloudWatch Logs.
For data classification, you can tag sensitive fields in the specification (example: "sensitivity": "PHI") and declare the sensitivity of each capability in the registry: a direct-move capability preserves the source classification, a date-of-birth-to-age capability clears it, and an enrichment that introduces sensitive data sets it. The composer combines the source tag with the capability’s behavior to derive the target field’s sensitivity, validating that the combination resolves clearly before assembling the pipeline. It then generates the masking artifact for the output (for example, AWS Lake Formation column grants) so consumers get correct masking without a separate masking specification or manual effort.
Recognizing the pattern
This approach works well when you describe workflows using structured specifications, reuse transformation logic across pipelines, require validation before invocation, and require workflows to be deterministic and auditable. You are likely to recognize the pattern in regulated data pipelines for the submission of tabular datasets to oversight agencies.
Consider clinical trial reporting as a concrete example. Data analysts collect raw data from clinical sites and must transform it into a standard submission format such as the Study Data Tabulation Model (SDTM) before submission to agencies such as the US Food and Drug Administration. With specification-driven composition, data analysts define specifications that map collected data about adverse events, demographics, and vital signs to the standard target variables, and the validated output feeds downstream systems such as patient safety and medical monitoring.
That said, this pattern might add unnecessary complexity for simple, one-time data transformations or pipelines with fewer than three to five workflows. Evaluate this pattern’s impact by tracking the reduction in duplicated transformation logic across pipelines, the time required to onboard new datasets, and the number of workflows you can create without modifying application code.
Conclusion
Script-based data pipelines accumulate hidden costs as they grow, including duplicated logic across files and late-breaking validation failures. Specification-driven composition separates workflow intent from processing so you can manage data workflows more consistently at scale. The result is faster dataset onboarding, stronger governance, and pipelines that are transparent enough to trust in regulated environments.
This pattern is especially valuable for regulated reporting pipelines, multi-source data integration, and reusable ETL frameworks where traceability and flexibility matter. By investing a small set of reusable capabilities and a disciplined specification format, you can reduce the engineering effort for new pipelines by treating them more as configuration tasks which may be delegated to system users.
For a practical first use case, try applying the pattern to a reporting pipeline you maintain today that has three or more variants such as monthly finance reports generated from different source systems. Replace the duplicated scripts with a single composer and a shared capability library, and measure the onboarding time for the next variant. As you expand your capability library, you can apply the same pattern across additional workflows and standardize how transformations are defined and run.
Kitty is a terminal
emulator that
runs on Linux, macOS, and the BSDs, which is notable for its speed and features
such as image support and advanced font handling. It is under active development; a
recent major release adds a
new level of mouse support. Here, we will look at some of those features
and show how the program can also be used as platform for
text-based applications. Kitty is free software, released
under the GPLv3.
RSA and ECC, cryptographic algorithms that we’ve all relied on for decades, are vulnerable to the attack of sufficiently advanced quantum computers. Such quantum computers do not exist yet, but they seem to be coming sooner than expected. Luckily, the solution is already available: migrate to ML-KEM encryption and ML-DSA signatures, which are designed to be resistant to quantum attack. They were standardized in 2024 by the U.S. National Institute of Standards and Technology (NIST) after an eight-year open international competition.
The migration to post-quantum cryptography is in full swing now. At the time of writing, the majority of traffic handled by Cloudflare is already using ML-KEM encryption, and is thus secured against the threat to data posed by harvest-now-decrypt-later attacks. But encryption is only one part of the equation: to be fully secure against quantum computers capable of breaking classical cryptography, we aim to deploy post-quantum signatures to protect authentication systems from unauthorized access. We are targeting 2029 for Cloudflare to be fully post-quantum secure.
ML-DSA, the best all-around post-quantum signature scheme standardized today, has its downsides: it’s much larger on the wire, and many tricks we were able to perform with RSA and ECC simply cannot be done with ML-DSA. There are better post-quantum signature schemes on the horizon: last month, NIST announced that it is advancing nine post-quantum signature schemes to the third round of the “signatures on-ramp”. And a draft standard for FN-DSA (née Falcon), which was picked from the previous competition, is expected imminently.
We have been very interested in advances in post-quantum signature algorithms, and wrote about the progress in 2021, 2022, 2024, and 2025. In this blog post we’ll treat you to the latest developments in great detail.
But first we have to deal with the elephant in the room: These new signature algorithms will not be ready in time for the PQ transition — not even close, as we will see later on. The problem is arriving too soon for us to wait. ML-DSA is available today, and it will have to do for the first migration. As Eric Rescorla wrote in 2024:
You go to war with the algorithms you have, not the ones you wish you had.
Nonetheless, the search for better post-quantum signature algorithms is crucial for several reasons, and we firmly believe it is still the best use of NIST’s limited resources.
Let’s have a look at the signature algorithms in detail. After that we’ll look at the timeline for their availability, and the reasons why we still need them.
The signature algorithms
In the table below, we compare the candidate signature algorithms that progressed to the third round (marked by 🤔), with classical algorithms vulnerable to quantum attack (marked by ❌), and the post-quantum algorithms that are already standardized ( ✅) or soon will be (📝). Each candidate proposes several variants. We list the most relevant variants to TLS, the protocol used to secure connections on the Internet. To explore all variants, check out Thom Wiggers’ signatures zoo.
Sizes (bytes)
CPU time (lower is better)
Family
Namevariant
A
Public key
Signature
Signing
Verification
Elliptic curves
Ed25519
❌
32
64
0.15
1.3
Factoring
RSA 2048
❌
272
256
80
0.4
Lattices
ML-DSA 44
✅
1,312
2,420
1 (baseline)
1 (baseline)
Symmetric
SLH-DSA 128s
✅
32
7,856
14,000
40
SLH-DSA 128f
✅
32
17,088
720
110
SLH-DSA 128-24
📝
32
3,856
7,000,000 ⚠️
4
LMS M24_H20_W8
✅
48
1,112
2.9⚠️
8.4
Lattices
FN-DSA 512
📝
897
666
3 ⚠️
0.7
Lattices
HAWK 512
🤔
1,024
555
0.25
1.2
Proof of knowledge
MQOM L1-gf16-fast-5r
🤔
60
3,280
8
20
SDitH SDitH2-L1-gf2-fast
🤔
70
4,484
15
40
FAEST EM-128f
🤔
32
5,060
4.2
9
Isogeny
SQIsign I
🤔
65
148
300 ⚠️
50
Multivariate
MAYO one
🤔
1,420
454
2.1
0.4
MAYO two
🤔
4,912
186
1.1
0.8
QR-UOV I-(127 156 54 3)
🤔
24,225
200
9.3
20
SNOVA (24,5,4)
🤔
1,016
248
1.2
1.7
SNOVA (25,8,3)
🤔
2,320
165
1
1.5
SNOVA (37,17,2)
🤔
9,842
124
0.8
1.3
UOV Is-pkc
🤔
66,576
96
0.3
2.4
UOV Ip-pkc
🤔
43,576
128
0.3
2
A few more remarks on this table: Most candidates have multiple variants in every security level. We show the most relevant variants for TLS at the 128-bit security level, the gold standard for security. CPU times are taken from the signatures zoo in June 2026, which collected them from the round two submission documents and later advances. Candidates are allowed to make changes for the third round, which will influence these numbers. Some will improve (both in compute and size), whereas others will regress to counter new attacks. Check out the zoo for the latest numbers. We marked FN-DSA and SQIsign signing with a ⚠️️, as both are hard to implement in a fast and timing side-channel secure manner. LMS signing has a ⚠️, as secure LMS signing requires keeping state across signatures, and the listed signing time assumes a 32MB cache. The 128-24 variant of SLH-DSA is marked with a ⚠️️ as it’s meant to create fewer than 224 signatures.
No “all-star” algorithm
One thing that stands out immediately is that the quantum-vulnerable elliptic curves signature algorithm Ed25519 is by far the best all-around choice (ignoring its quantum vulnerability): it has the best numbers in almost every single metric, including public key size, signature size, and signing time. It’s only beaten on verification time, but it’s more than fast enough for the vast majority of applications.
This is quite different than the roster of post-quantum algorithms. Instead of a single “all-star” algorithm, we have roughly two categories of schemes: the “specialists” that approach our trusty elliptic curve signatures on some metrics, but are problematic on others, which make them great in the right deployment scenario. Then there are the “generalists”, such as ML-DSA, which don’t perform as well as elliptic curves on all metrics, but so far as downsides go, are pretty balanced.
Specialists
Let’s start with the specialists.
SQIsign: small signatures / slow signing
If you just look at the bytes on the wire, then SQIsign looks like an almost perfect drop-in replacement for elliptic curve cryptography. With signatures of 148 bytes and public keys of 65 bytes, it beats RSA-2048. Unfortunately there is no free lunch: SQIsign has three weak points. First, it’s the most complex algorithm on the docket. Secondly, its signature creation and verification is quite slow. Finally, it’s difficult to implement signature creation in a timing side-channel secure way and doing so comes with a performance penalty to boot.
That doesn’t sound great so far, but it was much worse: when we had a look back in 2024, there was not yet any timing side-channel secure implementation and signature verification was 20x slower. Furthermore there has been welcome progress on simplifying the scheme.
Despite these dramatic improvements, it is unlikely (side-channel secure) signing will be fast enough in the foreseeable future to be used in typical online cases such as the TLS handshake. However, for offline cases, such as CA signatures or DNSSEC, where it’s the verification time that’s more important than the signing time, SQIsign might have an application.
But the topic we should really discuss is security. SQIsign is based on isogenies. Rather famously, SIKE, another algorithm based on isogenies, got broken badly in a late stage of the first NIST PQC competition that standardized ML-DSA. SIKE is often brought up as a cautionary example showing that post-quantum cryptography could break suddenly. This requires some nuance. First, there were already concerns about SIKE’s security, and in particular the torsion points that led to the break. Because of these concerns, SIKE was not selected for standardization, but deferred to an additional round of evaluation before it was broken. (Indeed, this is an example of the NIST process working well.) SQIsign doesn’t use torsion points, and there is no similar concern as there was for SIKE.
One other notable security property is that the best known attacks on SQIsign are generic brute force, just like with classical attacks on well-selected elliptic curves. This is quite different from RSA, lattices, and multivariate where the attack algorithms have been slowly improving, pushing the parameters towards bigger signatures. Nonetheless, the mathematics behind isogenies is very rich, and compared to the other algorithms, there is a lot of mathematical attack surface. Still, its security seems more sound than the structured multivariate algorithms we’ll discuss later.
SQIsign is an algorithm with tremendous potential. It’d be a shame to standardize it too early. To the authors, we’d like to share the following wishlist:
Ideally verification time is decreased even further, even if this trades off against signing time and signature size: SQIsign signatures are already small enough, and offline signing time has some slack anyway.
The timing side-channel secure implementation should be the default, especially if signing time is decreased further, which would tempt some online signing applications.
But above all, our wish is for SQIsign to be simplified.
UOV: tiny signatures / huge public keys
UOV (unbalanced oil and vinegar) is a classic multivariate signature algorithm originally proposed in 1999. It has tiny signatures: only 96 bytes. The trade-off? A huge public key: 66kB. That wouldn’t help for a TLS server certificate, whose public key is transmitted over the wire when setting up a connection, but it would be a help for cases where the public key is predistributed.
Let’s take the WebPKI as an example. A typical browser trusts about a hundred root certificates and 30 certificate transparency logs, whose public keys would add up to about 8MB when using UOV.
The public keys and signatures in a typical TLS connection.
Since the root certificate is transmitted to clients out of band, one idea is to use a UOV signature there. But this is not a slam dunk; because of its size, a UOV root certificate would be impractical to be cross-signed where the root is used as an intermediate. At the same time, cross signs and intermediates become less attractive anyway with any larger post-quantum signatures. This encourages more root certificates to be included directly with clients. This would again favor UOV, but to a point: if the number of root certificates grows above a thousand, we’d be dealing with more than 66MB of key material, which would make up a substantial portion of the browsers’ download size (e.g., 90MB for Firefox 151.)
Multivariate security
What about the security? Over the years, many variants of UOV have been proposed that use some extra mathematical structure to reduce the size of the public key. These structured multivariate schemes have had a spotty track record with schemes such as Rainbow and GeMMS being broken quite badly. It is important to distinguish these from UOV itself, whose security track record is much better, but not perfect.
As with many cryptographic schemes, there were growing pains in the early years, as basic attacks and parametrization pitfalls were discovered. In fact, the “U” in UOV is a remnant of that: it stands for unbalanced, which is a fix to a parameter-setting mistake in the 1997 oil-and-vinegar scheme on which UOV is based: the original scheme had an equal number of oil and vinegar variables in the quadratic system of equations used as the public key, which turns out to allow for an attack. In case you’re curious about the colorful name: the system of equations contains vinegar x vinegar and oil x vinegar, but no oil x oil terms. It’s like vinaigrette with small separate oil droplets. Back to the history: from 2005 to 2020 was a quiet period for multivariate signatures: understanding of UOV grew, but there weren’t any new attacks on typical parameters.
This changed in 2020 with the discovery of the intersection attack which built on the ideas of the original attack on balanced oil-and-vinegar. The intersection attack removes about 30 bits of security from a then-proposed 128-bit parameter set. A considerable blow, but not fatal: slightly adjusting parameters mitigates the attack completely, with minor increase in key and signature size.
A bigger shock was the 2025 publication of the idea to use wedges to attack multivariate schemes. The initial impact on UOV was minor: only a few bits (again at the 128-bit security level.) The worry was that this idea came out of left field, and it wasn’t clear how far the approach could be taken. That concern was partly justified: the wedges idea was very fruitful and several subsequent attacks have been built on it, reducing security by about 15 bits. However, it also became clear that the wedges attack and generalizations can be seen as a special case of an intersection attack over truncated rings — thus much more familiar than we thought. Again, these attacks can be mitigated with only minor increases in key and signature size.
What to make of all of this? Such a history of attacks is not uncommon: over the last 25 years lattices have seen larger reductions in security, although this has calmed down over the recent years. Notwithstanding, lattice-based cryptography deployed in production today uses conservative parameter sets well above 128-bits to hedge against future cryptanalysis. We’d want to do the same with UOV. Signature size only grows linearly with the security level, costing just 260 bytes even at the 256-bit security level. Unfortunately, the public key size is cubic in security level: 446kB for 256-bit. Conveniently, UOV (as most multivariate schemes) has a lot of flexibility in picking parameter sets at various intermediate security levels.
UOV is a foundational scheme with narrow but real use cases. Going forward, we’d like to see a parameter set with a bit of margin above 128 bits, say 160 bits, to hedge against future cryptanalytic improvements.
QR-UOV: small signatures / large public keys
Like SNOVA and MAYO which we’ll discuss later on, QR-UOV is a structured multivariate scheme: it’s a variant of UOV that adds more structure to the public key to reduce its size. The gains are modest: at best we’re looking at 12kB public keys, but signature verification is impractically slow for that particular parameter set. The more realistic parameter sets start at 24kB public keys.
With respect to security, QR-UOV is the only multivariate scheme that did not have to adjust its original (round one) parameters in response to new attacks. This is somewhat surprising as any attack on UOV can also be applied to QR-UOV. The explanation is that the attacks do apply, but the natural parameters for QR-UOV happen to make them ineffective. On the other hand, there were already several attacks known that use the specific extra structure that QR-UOV adds: indeed, for some of the parameter sets, the structure-specific attacks are the best attacks. This should be contrasted with MAYO, where there is no known attack against the extra structure MAYO adds. (We’ll get back to MAYO and SNOVA later in this post.)
Compared to last round, QR-UOV signing and verification time improved significantly, but it is still comparatively slow. All in all, QR-UOV is a hard sell: it adds exploitable structure to UOV without pushing key sizes down to general-purpose sizes.
Hash-based signatures
Stateful hash-based signatures
The very first standardized post-quantum signature algorithms are the stateful hash-based LMS, HSS and XMSS(MT). They have very small public keys, and for many parameter sets the signatures are much smaller than those of ML-DSA-44. To boot, their security is based on that of hashes, which are well-understood and already a cornerstone of cryptography. That makes hash-based signature algorithms a very conservative choice, and there is no need to hedge with higher security levels.
So, what’s the catch?
There are two. The big one is keeping the eponymous state. These stateful hash-based signature schemes are built out of one-time-signature keys which are collected into Merkle trees. The signer has to keep track of which one-time-signature keys have been used, which can be as simple as just a counter. If the signer mucks it up, though, and accidentally uses the same one-time-signature key twice on a different message, then anyone can likely use those two signatures to create their own signature on any message. You have to keep a lot in mind to keep the state correctly. Some considerations: you want to make sure that updates are written to storage before handing out the signature; you don’t want the old state to be restored from a backup; and you can’t export/import a private key from one place to another without agreeing on how to split or keep the state. The state is, as Adam Langley pointed out several years ago, a huge foot-cannon.
Another downside is that the most competitive parameter sets can only create a modest number of signatures. The 1,112 byte signatures (as listed in the table above) can only be used to create about a million signatures. You can explore the trade-offs with this calculator.
Together this leaves a very small niche for stateful hash-based signatures: signers have to be able to keep state; signature size has to be a real concern; and signers have to be OK with a hard limit on the number of signatures.
SLH-DSA: conservative security / large and slow
SLH-DSA is a hash-based signature that doesn’t have the low signature limit and avoids the problem of keeping the state. The basic idea is to make the number of one-time-signature keys so large that you can pick one at random without having to worry about using the same one twice, since the chance of picking the same one twice is diminishingly small. SLH-DSA is a bit more efficient than that, by replacing the one-time-signature key as a building block with a few-time-signature key, where security degrades gracefully if keys are occasionally reused. It still comes at a cost. SLH-DSA has two variants, one that optimizes for small signature size, and one that optimizes for fast signing. The size-optimized one is not small at all at 8kB, and the signing-optimized one is even slower than SQIsign.
Fewer signature variants of SLH-DSA
NIST has proposed to standardize an additional parameter set for SLH-DSA with much smaller signatures, but that can only be used to create about 16 million signatures before security reduces. At 3.8kB the signatures are still larger than those of ML-DSA-44, but the combined public key and signature size is very close. The parameter set was chosen to make signature verification fast at the cost of signing time. The signing time is very bad indeed.
Use cases
So why ever use SLH-DSA? The selling point is the conservative security. For a long-term trusted key that is hard to replace, it could make sense if the application can stomach the large signature and slow verification of the standardized variants or the slow signing time of the newly proposed one. There are two more caveats to add. First, it’s better to set things up so that key algorithms are not burned-in and can be replaced after the fact. And secondly, in most cases systems (such as secure connections with TLS) do not just depend on signatures, but also on key agreement. There is no hash-based key agreement mechanism, so we end up needing to trust something less conservative, like lattices, anyway.
FN-DSA: small key and signatures / subtle signing
Comparing the numbers, FN-DSA-512 (née Falcon) looks much better than ML-DSA-44 on almost every metric: faster verification, smaller public key, and much smaller signatures at 666 bytes. Signing is three times slower, but it’s still 25x faster than RSA-2048. To boot it’s already picked to become FIPS 206. So why don’t we consider FN-DSA to be a general-purpose algorithm?
It’s because it’s difficult to implement FN-DSA signing securely. The most well-known sharp edge of FN-DSA is that it is most naturally and efficiently implemented using hardware-accelerated floating-point arithmetic. This is a first for a cryptographic standard. One big challenge with it is that we have little experience implementing fast floating-point arithmetic in a side-channel safe way. What we know so far is that it’s subtle and not very robust: a safe implementation of FN-DSA signing using the Floating-Point Unit (FPU) for one processor might not be safe for another. Instead of relying on the FPU, the floating point operations can be emulated. This is easier to get right, but about 20 times slower, making it about as slow as RSA-2048. There has been some welcome progress recently to implement FN-DSA signing safely using fixed-point arithmetic, which is much faster than the floating-point emulation. So just use that and FN-DSA is good to go? This presumes a level of awareness that might not be warranted. Anecdotally at conferences, every time we saw a presenter compare post-quantum signature algorithms including FN-DSA in benchmarks, they couldn’t answer whether floating-point emulation was used.
Another consequence of using floating points is that it’s difficult to make test vectors for signing. Just one example of this is that the outcome of a+(b+c) and (a+b)+c are only guaranteed to be close, but not the same. That means that to have useful test vectors, the FN-DSA specification would need to be very precise on the order of floating-point operations. Another example is a*b+c, which can be computed in two steps (multiply and then add), or at once using fused-multiply-add (FMA). The latter is faster, but again gives a slightly different answer as rounding happens only once. Not all processors support FMA, but for those that do, compilers typically automatically use FMA for the performance boost. There are also mathematical optimizations that cause trouble. For instance, the reference implementation computes a value (norm) in a faster roundabout way using Parseval’s theorem. Mathematically the answer is exactly the same, but as floating-points are only an approximation, the resulting value is ever so slightly different. Similarly, the safe fixed-point arithmetic implementation gives slightly different results.
Why is this a problem? It is because it is still the humble test vector that catches most implementation bugs in practice. Other more refined methods like formal verification will certainly catch more, but test vectors are hard to beat in simplicity.
Another surprising sharp edge from not having a fixed implementation is the following. From two deterministic signatures created by slightly different implementations from the same private key, one can derive parts of that private key. FN-DSA does not use deterministic signatures, instead adding a randomizer to thwart this. There is a tension with testing: you need a deterministic interface to test signing, but you don’t want that to be used to create actual signatures.
How to deal with the wiggle room in the FN-DSA specification will undoubtedly be a point of discussion. The discrepancy between the implementations might actually have a silver lining: NIST could decide to generate the test vectors (CAVP) from the fixed-point arithmetic implementation. That the more risky floating-point implementation wouldn’t pass the test vectors would be a feature, not a bug, as it would steer implementations towards the safer fixed-point version!
You can read about a few other interesting sharp edges in this blog post. Stepping back from the specifics, the main point is that FN-DSA is a complicated scheme. It’s not a surprise that it took NIST a couple of years (not counting the current limbo) just to write the draft standard. It’ll take longer than usual for the final standard to come out and for cryptographic libraries to add support. FN-DSA is farther away than it seems. We’ll compare timelines later in this blog post.
If the numbers are still very tempting, there is one last thing you should be aware of: FN-DSA-512 is parametrized for 128-bit security compared to ML-DSA-44’s generous 160 bits. If lattice cryptanalysis improves, there is no middle security level: the next step-up is all the way to FN-DSA-1024 at 256 bits. FN-DSA-1024 has double the key and signature sizes and signing and verifying times of FN-DSA-512. An FN-DSA-1024 signature is still half the size of that of ML-DSA-44, but the public key+signature only differs about 20%.
To close the discussion of FN-DSA, it is good to emphasize that all difficulties with FN-DSA are on the signing side: the verification of an FN-DSA signature is very straightforward.
General-purpose algorithms
Now let’s turn to the algorithms that are meant to be general-purpose replacements for ML-DSA.
HAWK
HAWK is a curious case. In many aspects it’s similar to FN-DSA: a structured lattice hash-then-sign scheme with similar sizes for signatures and public keys with a missing middle security level. The main benefit of HAWK over FN-DSA is that signing is very fast and doesn’t use floating-point arithmetic, although it’s not a simple algorithm either. This comes with a trade-off: HAWK is based on and introduces a new security assumption, the lattice isomorphism problem (LIP). In 2024, two years after the introduction of HAWK, it was discovered that this problem is easy to solve in the special case of totally real number fields, which aren’t used in HAWK or any other cryptography. In 2025, this attack was extended to a broader class of number fields. This hasn’t yet applied to HAWK, but it’s getting closer. A new paper published in June 2026 suggests there is a way to extend the attack to HAWK. An error has been found in the paper, although it’s yet unclear how fundamental it is to the approach. Regardless, the trajectory is concerning.
Even ignoring the potential attacks, HAWK faces some headwinds: its additional security assumption prevents it from displacing FN-DSA, but its practical benefits (especially considering the lack of middle security level) fall short of that of the structured multivariate candidates. It also doesn’t increase diversity in security assumptions, an outcome that NIST is hoping for.
Proof-of-knowledge schemes
FAEST, MQOM, and SDitH all share a similar overall structure. Their public keys are instances of some hard problem and their secret keys are the solutions.
A FAEST public key is the AES-encryption of a known plaintext under a secret key.
MQOM gets its name from the Multivariate Quadratic problem, which is closely related to (but more conservative than) the cryptographic assumptions underlying the multivariate schemes. The public key is a system of quadratic equations, and the secret key is a solution to that system of equations.
SDitH is based on the hardness of the Syndrome Decoding problem for random linear codes. This problem is related to the code-based schemes submitted to the original NIST competition, but these were eliminated in the third round.
In all cases, a signature is a zero-knowledge proof that the signer knows the solution of that hard problem, while at the same time (almost as an afterthought) acknowledging the message-to-be-signed as part of the proof.
Many signature schemes are zero-knowledge proofs like this behind the scenes, notably ML-DSA, SQIsign, and Ed25519. Why don’t we group those with proof of knowledge schemes too?
The difference is generalizability: the zero-knowledge proof used for ML-DSA is only able to prove something about a specific LWE problem as used in ML-DSA: the proof uses mathematical structure in the key. There are ways to create zero-knowledge proofs using lattices for any general statement, but those proof systems are very different from ML-DSA, and would create rather larger signatures on the order of 50kB.
In contrast, the proof system used in FAEST, MQOM, and SDitH can be used to prove arbitrary statements. For instance, FAEST can be modified to use the hard problem of MQOM instead. This leads to a more efficient scheme called KuMQuat. (We’ll get to some performance numbers later on.) Conversely, MQOM can be adjusted to use AES as the hard problem.
This flexibility is great for two reasons. First, it doesn’t require any specific mathematical structure in the hard problem used, and thus we can pick a very conservative problem such as breaking AES. Some problems lead to a more efficient signature than others, as we see with MQ as used in MQOM. MQ is still quite a conservative assumption: it does not contain the hidden subspace used in UOV and thus the other multivariate signatures. Neither the intersection nor wedges attacks apply to it. In fact, the MQ-problem is NP-hard. To be secure, one still needs to pick the correct size of the problem, and although MQ has been studied for quite a while, it certainly has not seen the same scrutiny as deployed algorithms like AES.
The second and greater benefit is that we’re able to create much more than just a plain signature scheme from a general zero-knowledge proof system: we can create blind signatures and even full-fledged anonymous credentials.
Here it’s good to note a limitation: the size of the proofs for all three grows linearly with the statement proven. In technical terms: they’re not succinct like STARKs and LaBRADOR, which outperform them handily for large statements. It’s another example where sometimes it’s better to pick the approach that’s not optimal asymptotically.
Back to advantages: apart from the hard problem chosen, and the security of hash functions, these three schemes don’t require any further security assumption. This makes FAEST as conservative as SLH-DSA.
So what’s the difference except for the chosen hard problem? These schemes started off quite differently, but have been improving and converging since the first round. The proof system in MQOM is a bit simpler than FAEST, but it also does not perform as well: KuMQuat (FAEST+MQ) outperforms MQOM.
Talking about performance, let’s start with a comparison to SLH-DSA. All three schemes have variants that outperform any standardised SLH-DSA parameter set and often by a good margin. SLH-DSA does have one distinct advantage: the verification routine is simpler to implement.
Against ML-DSA-44 the comparison is more interesting. All schemes have a smooth trade-off between runtime and signature size. To illustrate, here are trade-offs reported for KuMQuat (FAEST+MQ.) Verification times are close to signing times.
KuMQuat can be parametrized to have somewhat smaller signatures than ML-DSA-44 at the cost of long signing (and verification) runtime. At the other end, it can have similar signing time as ML-DSA-44, at the cost of larger signatures, although the public key+signature size is still similar.
These schemes have improved quite a bit over the years, and we expect some improvements still. Although they won’t improve upon ML-DSA as dramatically as some of the other schemes considered, their conservative security and especially their potential for broader applications like anonymous credentials make them very appealing. To showcase the flexibility of the underlying zero-knowledge proof system, we’d like each scheme in this category to present numbers on how well they’d perform with a different underlying hard problem.
Structured multivariate: MAYO versus SNOVA
Like QR-UOV discussed earlier, MAYO and SNOVA are variants of UOV that add extra structure to the public key to reduce its size. MAYO and SNOVA take two different approaches: SNOVA makes aggressive bets to get the best performance, whereas MAYO treads carefully with a conservative design.
SNOVA does have impressive performance. Its main parameter set has 248 byte signatures (smaller than RSA-2048!) with only a 1kB public key. It beats every other post-quantum scheme on public key+signature size and has great runtime.
MAYO’s performance is nothing to scoff at either. MAYOone has the best verification time and its 454 byte signatures are still smaller than those of FN-DSA-512, HAWK-512, and RSA-4096. Combined with its 1,420 byte public key, MAYOone does slightly fall behind FN-DSA-512 and HAWK-512. However, MAYO takes the lead again if we ask for some security margin. FN-DSA and HAWK have a missing middle security level and thus need to bump all the way to the 256-bit security, whereas MAYO’s granularity can add extra security at the cost of slightly increasing public key and signature sizes.
Security
Public key
Signature
PK + Sig
HAWK-1024
256
2,440
1,221
3,661
FN-DSA-1024
256
1,793
1,280
3,079
MAYO at 174 bit security
174
1,600
550
2,150
If that wasn’t good enough, both MAYO and SNOVA allow for a trade-off between signature and public key size. Thus, we can get even smaller signatures for public keys that are transmitted ahead of time. Pushed to the extreme, MAYO becomes UOV.
So far we have discussed performance. What about the security? MAYO adds a “whipping” structure on top of UOV: any attack on UOV will also work for MAYO, but there might be attacks specific to the whipping structure of MAYO. So far no attacks on the whipping structure, and thus on MAYO specifically, have been found. The worst that has happened is that some UOV attacks have affected some MAYO variants more than typical UOV parameter sets, due to the UOV parameter choices that are natural for MAYO.
This is in stark contrast to SNOVA. SNOVA has been hit quite hard on its specific structure several times. In response, the SNOVA team has not just tweaked parameters, but continuously changed the actual structure. Every time, they take the leap forward and propose a new SNOVA with even better performance. We noted this last year and the pattern has continued, whereas MAYO’s basic design is stable.
Furthermore, the structure SNOVA uses can beseen as a special form of the whipping map that MAYO uses. That means that any MAYO-specific attack would apply to SNOVA, but not the other way around.
All in all, we’ve seen a lot of progress on the understanding of multivariate security. NIST wrote that they expect an extra round before standardizing a multivariate scheme. That seems prudent. To us, it’s unclear whether SNOVA would be ready by then, but MAYO so far seems to have matured well.
Timelines
Now, let’s have a look ahead and sketch when these new signature algorithms might become usable.
Progress for ML-DSA so far
It’s illustrative to look at ML-DSA.
November 2017
Submitted to the competition
January 2019
Progressed to the second round
July 2020
Progressed to third round
July 2022
Selected for standardization
August 2023
Initial public draft
August 2024
Final NIST standard
October 2025
ML-DSA certificate standard (RFC 9881)
April 2025
OpenSSL 3.5.0 adds support for ML-DSA
August 2025
Debian Trixie released with OpenSSL 3.5.0
December 2025
TLS IANA codepoint for ML-DSA registered
March 2026
First CMVP certificates for ML-DSA module
July 2026 (expected)
Hybrid ML-DSA certificate standard
August 2026 (expected)
RFC for use of ML-DSA in TLS
Early 2027 (expected)
Availability first ML-DSA certificates in WebPKI
After NIST selected Dilithium to become ML-DSA, it took a year to draft a proposal for the standard, and another year for the algorithm standard to be published. The algorithm standard is not enough: protocols need to agree on how to integrate ML-DSA. For certificates that took another year. That’s not the end of it: software needs to add support for ML-DSA and its integration into protocols.
These steps are not purely sequential: work on software implementation of ML-DSA started before the final standard. Also, protocol integration standards are often “done” before they’re a final standard. For instance, the use of ML-DSA in TLS is done, but at the time of writing it’ll take a couple of months before the RFC for that is out. Notably OpenSSL jumped the gun and added support for ML-DSA before the IANA codepoints were assigned. Notably missing still is agreement on which hybrid signatures should be used in TLS (or at all), for which (at the time of writing) no IANA code points have been assigned.
When will these new signature algorithms be ready for use?
So where does that leave us for new signature algorithms? If the FN-DSA draft is released today, and it progresses at the same rate as ML-DSA, then we’d perhaps have some early software support in early 2029, but no significant deployment. Looking at the time it took to write the FN-DSA draft standard, it is likely that the final standard, protocol integrations, and software support will progress slowly as well. We would not expect FN-DSA to be widely available before 2033.
The progress in cryptanalysis of multivariate schemes gave NIST pause: they wrote that they expect multivariate to at least take another round of about two years. On the other hand, multivariate schemes are reasonably easy to implement. That means we might see a multivariate NIST standard in 2031, and wider product availability not earlier than 2034.
NIST is more confident in the security of SQIsign than that of multivariate. Not unlike FN-DSA, SQIsign is a difficult scheme to standardise and implement. At the same time, a lot of progress is made in simplifying SQIsign. It seems likely that SQIsign will make large changes for the third round, and will thus require a fourth round of evaluation. In either case, wide availability before 2035 seems unlikely.
As discussed above, HAWK occupies an awkward middle ground between FN-DSA and structured multivariate candidates. If it were standardized, which seems unlikely even before the recent progress in cryptanalysis, we wouldn’t expect product availability before 2034.
That leaves the proof of knowledge algorithms MQOM, SDitH, and FAEST. We’ve seen dramatic improvements to these schemes over the rounds. If that rate of change holds, it’ll require another round, but if it’s stable now, a proof of knowledge algorithm will be the first new NIST standard to see the light in 2030. If it’s out this early, it’ll likely not outperform ML-DSA dramatically. Nonetheless, it’ll still be very welcome to build anonymous credentials and other primitives beyond signatures.
So, should you wait on one of these signatures for your post-quantum migration? Given recent advances in quantum hardware and software, we don’t believe we can afford to wait. At Cloudflare, we’re aiming to be fully migrated by 2029. None of these signatures will be out in time. Deadlines of most regulators vary between 2030 and 2035. These did not account for recent progress, and we expect them to be adjusted. We saw just this with the June 2026 US executive order setting a 2031 deadline. Even if deadlines weren’t changed, we wouldn’t advise waiting.
Why? Deploying post-quantum signatures in 2034 to beat a 2035 deadline is not enough. In a system of any reasonable size, you can’t upgrade everything all at once. You’ll need a transition period where both post-quantum and traditional signatures are supported. And supporting both allows for a downgrade attack. The most straightforward way to prevent such downgrades is to disable classical cryptography. That will take time, and is frankly not even an option in many sufficiently distributed systems like the WebPKI. We will cover how to deal with downgrades in a future blog post. In the meantime here is some reading if you’re curious. In any case, dealing with downgrades will take time.
It seems clear these new post-quantum signature algorithms will not be ready to use in time for the first migration. So why bother?
Why we still need them
We’ve had 50 years to weave public key cryptography all through our digital society. We have a few short years left to make it all quantum secure. For most of these upgrades the procedure is clear: drop in post-quantum cryptography. Easier said than done: it’s a monumental task. But then there are cases that are fundamentally harder. There’s no all-star signature in a post-quantum world, and there are cases where the size of ML-DSA is a problem. With enough resources and stakeholder agreement, systems can be re-engineered to work well with these larger signatures. Indeed, thanks to ongoing re-engineering, the post-quantum WebPKI is shaping up to perform better than the quantum-vulnerable one of today. It is unrealistic to expect that this will happen for every system before it’s too late. Some will have to accept a performance cost. Others will need to deal with the security gap in other ways, such as restricting access, tunneling, more monitoring, or a myriad of other measures that are costly on their own. Once smaller post-quantum signatures arrive, these compensating controls can be removed, and full efficiency and security restored.
An indirect, but no less important benefit of the ongoing NIST competition is its help in furthering post-quantum cryptography beyond basic primitives: it isn’t just key agreement and signatures that are quantum vulnerable. There is a long tail of fancy cryptographic primitives out there used in production, such as anonymous credentials, PAKEs, and threshold signatures to name a few. For most, post-quantum variants are not readily available or are understudied. For some, the same goal can be achieved without fancy cryptography, but with a regrettable regression in subtle privacy goals. NIST cannot run a competition to define a post-quantum standard for each of these specific primitives, but luckily the signatures competition has been a huge help here.
The most clear example is FAEST. Although designed as a signature scheme, its underlying machinery (VOLEitH) can be repurposed in combination with a multivariate scheme like MAYO, to create an efficient post-quantum anonymous credential. Without the signatures competition, VOLEitH wouldn’t be as developed and vetted as it is today.
Many of the candidate schemes briefly point out their usefulness apart from signatures. We hope to see more of the indirect applications of these schemes highlighted.
Despite great signatures and more advanced cryptography on the horizon, we should not forget the task at hand: staying secure in the immediate future.
Ако докторът ви предостави две възможности, коя от тях бихте предпочели – спешна операция или консервативно лечение? При равни други условия повечето хора може би ще си спестим хирургическата интервенция. Особено ако тя е свързана с болезнен и изпълнен с неудобства период на възстановяване. Но ако знаем, че консервативният подход няма да ни излекува и дори ще задълбочи настоящите ни проблеми, а след операцията ще можем да се възстановим напълно и да водим нормален живот, вероятно бихме събрали смелост да легнем под ножа.
Същото е и с реформите. Ако държавата се постарае да ни убеди, че те са полезни и необходими, ще сме по-склонни да ги приемем, отколкото ако ни плаши, че ако не се удържа статуквото, докогато може, ще стане страшно. Особено ако хвърли вината за проблемите върху някой друг.
Правителството на „Прогресивна България“ (ПБ) е наясно с този психологически механизъм.
И го използва, за да оправдае проектобюджета си, който на 7 юли 2026 г. беше приет на първо четене в парламентарната комисия. Всъщност бюджетът, предвиждащ дефицит от 5,7% и теглене на външен дълг, не е по-добър от този на кабинета на Росен Желязков, срещу който имаше протести, довели до зрелищното рухване на управлението на ГЕРБ. А с него и на политическата тежест на партиите на Бойко Борисов и Делян Пеевски.
В края на юни 2026 г. зам.-министърката на финансите Людмила Петкова очерта два подхода с оглед на повишения бюджетен дефицит (за който кабинетът на ПБ обвинява предишните правителства):
Единият е за шокова консолидация – увеличаване на данъци и рязко намаляване на разходи; въпросът е каква е цената – тежки икономически и социални последици. Вторият – плавна стабилизация със структурни реформи.
Противопоставянето между шоковата консолидация и плавната стабилизация всъщност възпроизвежда един популярен наратив от началото на Прехода. По онова време някои от днешните гласоподаватели не са били родени, други са били твърде малки, за да имат ясни спомени от тогавашните „опорки“, трети отдавна са забравили. Затова нека припомним.
„Плавен преход“ срещу „шокова терапия“
Скоро след Десети ноември 1989 г. възникна въпросът как България „да се оправи“. Формално социализмът беше паднал, но икономиката тепърва трябваше да се пренастрои.
През по-голямата част от 1990 г. на власт са две правителства на Андрей Луканов. Още през март той се обръща към тогавашния главен икономист на Търговската камара на САЩ Ричард Ран с молба да помогне за прехода към пазарна икономика. Няколко месеца по късно Ран представя стратегия, изготвена в сътрудничество със заместника му Роналд Ът, която става известна като „плана Ран–Ът“. Правителството обаче не се и опита да я приложи.
Вместо това именно под управлението на Луканов се лансира тезата, че „плавният преход към пазарна икономика“ е за предпочитане пред „шоковата терапия“. В страна, в която четири десетилетия и половина икономиката е била планова, „плавният преход“ изглеждаше приемлив и щадящ вариант, докато „шоковата терапия“ звучеше страшничко. „Плавният преход“ обаче вървеше в комплект с идеята за „социална пазарна икономика“. Но не в смисъла на икономиката на скандинавските страни например, а нещо по-близо до СССР (който тогава още не се беше разпаднал).
Докато тече т.нар. плавен преход, бившата социалистическа номенклатура запретва ръкави, за да превърне политическия си капитал в икономически (казано с термините на френския социолог Пиер Бурдийо). Според германския разследващ журналист Юрген Рот в периода 1989–1990 г. от България са изнесени, по различни оценки, между 4 и 9 млрд. долара.
Но дори без тези грабежи България беше разорена още от последните години на социализма.
От 1987 г. страната е в дългова криза: външният дълг расте, а държавата се опитва да решава проблема с натрупаните заеми, като тегли нови и нови. Трупа се и вътрешен дълг – икономиката е държавна, а държавата няма пари да плаща на собствените си предприятия да работят. Било защото тези предприятия произвеждат неща, които никой не иска (по време на ученическа бригада участвах в производството на консерви от развалено пилешко със зеленчуци и ви уверявам, че никой не би ял такава гадост, освен по принуда), било защото продават на неплатежоспособни пазари, било и двете.
Така нареченият Възродителен процес води до допълнително влошаване на дълговата криза. България принудително изселва около 360 000 от гражданите си, голяма част от които работещи. Няма кой да се труди вместо тях. Отделно, цялата операция, свързана с подмяната на стотици хиляди документи и фактическата военна окупация на определени региони, е скъпа.
Такава криза не може да се овладее с плавен преход. А и преходът никак не е плавен, каквото и да говори Луканов.
Още по времето на късния социализъм беше обичайно различни стоки да станат дефицитни – тоалетна хартия, дамски превръзки, един или друг хранителен продукт (без да споменаваме смятани за луксозни стоки, които си бяха дефицитни през целия социализъм – от елементарни днес неща, като дезодорант например, до автомобили). По времето на Луканов България престана да плаща външния си дълг, но това не помогна. Дефицитите се задълбочиха – хората се редяха на опашки от 5 часа сутринта, за да си купят например… кисело мляко. Стигна се до купонна система и режим на тока.
След т.нар. Луканова зима на власт идва правителството на Димитър Попов, което вече се вижда принудено да предприеме „шокови“ мерки за овладяване на ситуацията. На първо време то либерализира цените – на 1 февруари 1991 г. Само за месец те скачат повече от 2,2 пъти, а за цялата 1991 година инфлацията е около 300%. Димитър Попов остава в историята с призива „За бога, братя, не купувайте!“. През 2011 г. Попов казва, че всъщност цялата му фраза е била:
„За бога, братя, не купувайте това, което ви харесва, но е много скъпо, защото след 30–40 дни цените ще паднат, а вие ще сте похарчили парите си и няма да можете да купите най-необходимото ви.“
В този период кратко просъществувалата група „Баракуда“ записва песента си „Шокова терапия“. За съжаление, тя не е достъпна онлайн, но може да се чуе песента на групата „Хей, чичо“, която също улавя духа на времето на „За бога, братя, не купувайте“:
Сиви, бледи, анемични, арогантни, неприлични, на опашките стоим и стотинките броим, че май вече не ни стигат да си купим даже книга, от която да научим, че понякога се случва от преяждане да пукнеш, как е вредно за сърцето и месото, и кафето, та дори и зеленчукът.
През февруари 1991 г. е първата хиперинфлация в България след 1989 г. Втората, по-голямата и по-ярко запечатала се в колективната памет, е пет години по-късно – т.нар. Виденова зима (1996–1997), по името на тогавашния премиер Жан Виденов. Само за февруари 1997 г. инфлацията е 242,7%. През март, когато на власт вече е служебният кабинет на Стефан Софиянски, инфлацията спада до 12,3%, но все пак потребителските цени са повече от 2000% по-високи, отколкото през същия месец предишната година.
Нарастване на потребителските цени
Данните са от отчета на БНБ за януари–юни 1997 г. (стр. 17). Източник: БНБ, НСИ
„Виденовата зима“ е еманация на последиците от подхода, практикуван и от правителствата на Луканов – необходимите реформи се отлагат, а междувременно бившата номенклатура открадва всичко, което успее. Второ действие на извлечените куфарчета с милиарди бяха банките, които раздаваха кредити на неслучайни хора и след това фалираха. Спестяванията на обикновените граждани се стопиха и се появиха т.нар. кредитни милионери.
Още от същото
Луканов е премиер по времето, когато управляващата тогава БКП сменя името си на БСП. Правителството на Виденов също беше на БСП (в коалиция с БЗНС и „Екогласност“). През 2026 г. Столетницата за първи път не успя да прескочи 4-процентовата бариера за влизане в парламента. От нейните руини обаче възниква ПБ. Навремето Румен Радев беше издигнат за президент от БСП, а днес немалко от ключовите функционери на ПБ са бивши кадри на социалистическата партия.
36 години по-късно наследниците и възпитаниците на някогашната номенклатура отново ни предлагат същите неща:
„плавен преход“ вместо „шокова терапия“, макар и с малко по-различни думи – „преходът“ е заменен със „стабилизация“, а „терапията“ – с „консолидация“;
замазване на проблемите чрез отлагане във времето (договора с „Боташ“) или чрез затъване в дългове, които все някога ще трябва да се връщат с лихвите, вместо решаването им чрез реформи;
палиативни мерки вместо затваряне на кранчетата, от които систематично изтичат парите;
И за десерт, отново през бюджета – закриване на Комисията по досиетата. При всичките си несъвършенства и въпреки че много от досиетата са унищожени в самото начало на Прехода, именно тя хвърля светлина върху практиките и наследството на тоталитарния репресивен апарат.
За разлика от БСП обаче, ПБ поне не се прави на лява партия. Една лява партия би се застъпвала за прогресивен данък и за равни права – в България такъв политически субект няма.
Между другото, навремето Полша избра „шоковата терапия“. И единствена от постсоциалистическите страни не изпадна в рецесия след 1989 г. Днес страната се доближава до 20-те най-силни икономики в света, а по стандарт на живот се нарежда до Япония.
А България си цикли около „плавната стабилизация“.
Version
1.97.0 of the Rust programming language has been released. Changes
include using a new symbol-mangling scheme by default, support for denying
warnings in Cargo, and an end to the practice of hiding the linker’s output
after a successful build.
Because of the way they are trained, large language models capture only a slice of human language. They’re trained on the written word, from textbooks to social media posts, and our speech as captured in movies and on television. These models have minimal access to the unscripted conversations we have face to face or voice to voice. This is the vast majority of speech, and a vital component of human culture.
There’s a risk to this. The increased use of large language models means we humans will encounter much more AI-generated text. We humans, in turn, will begin to adopt the linguistic patterns and behaviors of these models. This will affect not just how we communicate with one another, but also how we think about ourselves and what goes on around us. Our sense of the world may become distorted in ways we have barely begun to comprehend.
This will happen in many ways. One of the first effects we could see is in simple expression, much as texting and social media have resulted in us using shorter sentences, emojis instead of words, and much less punctuation. But with AI, the impacts may be more harmful, eroding courteousness and encouraging us to talk like bosses barking orders. A 2022 study found that children in households that used voice commands with tools like Siri and Alexa became curt when speaking with humans, often calling out “Hey, do X” and expecting obedience, especially from anyone whose voice resembled the default-female electronic voices. As we start to prompt chatbots and AI agents with more instructions, we may fall into the same habits.
Next, in the same way autocomplete has increased how much we use the 1,000 most common words in our vocabulary, talking with chatbots and reading AI-generated text may further constrict our speech. A recent University of Coruña study found that machine-generated language has a narrower range of sentence length, averaging 12-20 words, and a narrower vocabulary than human speech. Machine-generated text reads as smooth and polished, but it loses the meanders, interruptions and leaps of logic that communicate emotion.
Additionally, because large language models are primarily trained from written speech, they may not learn how to emulate the free-wheeling nature of live, natural speech. When told “I hate Beth!”, ChatGPT replies with an uninterruptable three-part formula of affirmation (“That’s completely valid”), invitation (“I’m here to listen”) and invitation (“What’s going on?”) far longer than any reply plausible in face-to-face dialog. “What’s Beth’s deal?!” elicits a bullet point list of queries that reads like a multiple-choice exam question (“Is Beth * a celebrity? * a friend from school? * a fictitious character?”). No human speaks that way, at least not yet. But meeting such formulas repeatedly in a speech-like context may teach us to accept and use them, much as a child absorbs new speech patterns from spending time with a new person.
These influences will only increase with time. The writing large language models train on is increasingly produced by large language models themselves, creating a feedback loop in which they imitate their own inhuman patterns, even while teaching humans to imitate them too.
Broad use of large language models could also introduce confirmation bias, making us overconfident in our initial impulses and less open to other possible ideas—which is so vital to human discourse. Many chatbots are instructed to agree with our statements no matter how absurd, enthusiastically supporting half-formed or even incorrect notions and restating them as firm claims that we’re primed to agree with. When asked “Cake is a healthy breakfast, right?” or “Is the post office plotting against me?”, this sycophancy can reinforce bias and even worsen psychosis. And the hyperconfident tone of AI-produced writing will also heighten impostor syndrome, making our natural, healthy doubt feel like an aberration or failing.
In our experience as teachers, students who turn to generative AI for assignments often say they do so because they have trouble expressing what they think. The students don’t recognize that writing or speaking our thoughts is often how we realize what we think. Their unconfident and uncertain statements are actually the healthy human norm. But a large language model won’t turn vague first guesses into a well-formed critical analysis, or even ask helpful questions as a friend would; it will simply regurgitate those guesses, still unexamined, but in confident language.
We are also more vicious in social media posts and online chats than we are face to face. The well-documentedonline disinhibition effect encourages toxic language. Most of us have had the experience of venting ferocious rage about someone online, only to reconcile when we speak face to face or hear the warmth of a voice over the phone. While chatbots are trained to give sycophantic responses, they see humankind at our cruelest, learning about us from the only world where every flame war leaves an eternal written footprint, while the spoken conversations of forgiveness and reconciliation fade away. Their responses do not imitate our online aggression, but are still shaped by it, even in their rigid efforts to avoid it.
It’s easy to draw the wrong conclusions from a selective slice of a society’s communications. Medieval Norse sagas made us imagine a culture of mostly Viking warriors, since poets rarely described the farming majority. Chivalric romances focused on kings and courts, and long made us see the middle ages as a world of monarchies, erasing the many medieval republics. Statistically, we’ve been led to believe ancient Romans cared deeply about their republic, but 10% of all surviving Latin was written by one man, Cicero, whose work contains 70% of all surviving Roman uses of the word republic. Training language models on only certain human writings may introduce similar distortions. AI might make us seem more quarrelsome, as we are online. It might inflate the cultural significance of political topics primarily discussed on Twitter/X or Bluesky, or the massive topic-specific corpuses of LinkedIn and Goodreads.
Some large language models are being trained on human speech from movies and television shows, but that speech is still scripted, and disproportionately highlights certain contexts over others (for example, police dramas, fueled by stories of murder, make up a quarter of prime-time television programming). We are not funny or hurtful or romantic the same way in real life as we are in sitcoms. At least one startup is offering to pay people to record their phone calls for AI-training purposes, but this remains a niche idea; anything large scale would cause massive privacy concerns.
We don’t pretend to know what the best solutions might be. But one has to imagine if there’s ingenuity to develop AI models, then surely there’s ingenuity to come up with a way to train them on informal human speech instead of us only at our most stylized, veiled and sometimes worst. By excluding the overwhelming majority of language production on the planet—people talking, fully and naturally, to each other—these models are being trained to mirror everything but us at our most authentically human.
This essay was written with Ada Palmer, and originally appeared in The Guardian.
Over on the OpenMandriva
forum, the Linux distribution has reported
sabotage of its repositories by a disgruntled contributor with
administrative credentials. According to “AngryPenguin”, an abusive
incident in a distribution Matrix chat led to a user being kicked out of
the chat; that “triggered a cascade of events“, which led to people
resigning from the distribution. Eventually, one of those people used
their administrative privileges to delete part of the distribution’s GitHub
repository and to “publish an empty package in the cooker
repository, which obsoleted all gnome and cosmic packages, which could have
damaged the systems of people using gnome or cosmic“.
We are currently working to restore the deleted repositories and restore
the functionality of the obsolete packages.
[…] We performed a full system audit and, aside from the removed
packages, we found no other violations.
“Where are the docs?” It’s a question nobody on a product team enjoys answering. The honest reply is usually some variant of “behind.” A writer is staring at a closed pull request, trying to reverse-engineer what changed. The pull request’s author has already moved on. By the time the doc actually publishes, the feature has shipped, sometimes more than once.
That used to be us on the Aspire team (we’re a small team of 10 building dev tools for distributed apps). A few months back, we were trying to figure out how to safely bring AI into automations we already trusted. That’s when we discovered GitHub Agentic Workflows. I started bolting prototypes into microsoft/aspire.
Here’s what that bought us, in numbers pulled straight out of GitHub: for Aspire 13.3 and 13.4, 82 feature-docs pull requests merged at a median of 44.8 hours after the product pull request, every one of them reviewed by the engineer who shipped the feature. No new headcount. No process retraining. Just a different way of asking “who writes this?”
🔒 The constraint: cross-repo automation is the hard part
Our product lives in microsoft/aspire and our docs site lives in microsoft/aspire.dev—different repo, deploy target, and review chain. Most teams figure out same-repo automation pretty quickly; cross-repo automation is where things get sharp. Broad repo-scoped tokens belong in a museum, and any responsible security posture (ours included) restricts them accordingly. That’s a good thing. It’s also a real bottleneck if the place where you write the docs isn’t the place where you write the code.
The default workflow for years was:
Engineer ships a feature in microsoft/aspire.
Docs writer notices weeks later.
Docs writer opens the pull request, reads the diff, and pings the engineer to clarify what changed.
Engineer is on the next feature, vaguely remembers, replies with half the picture.
Docs draft ships, sometimes against a release that’s already out.
This is the reverse-engineering tax. We needed automation that crossed repos without handing an agent a write-everywhere token. GitHub Agentic Workflows turned out to be the answer.
🤖 Why GitHub Agentic Workflows
GitHub Agentic Workflows is a project from the GitHub Next team that I keep describing to people as “GitHub Actions, but with a model as the work-item processor and guard rails that satisfy security review.” That’s reductive, but it’s close.
The shape of it:
You author a workflow as a single markdown file (.github/workflows/my-thing.md). YAML-style frontmatter on top, an English-language prompt underneath.
You run GitHub Agentic Workflows compile, and it generates a sibling .lock.yml (a normal GitHub Actions workflow) that you commit alongside.
At runtime, the workflow runs an agent against your prompt with a constrained toolset.
Critically, the agent doesn’t write to GitHub directly. It emits intent (a JSON blob describing the pull requests, issues, and comments it wants to create), and a separate, narrowly scoped job (the safe-outputs handler) materializes that intent against a per-workflow GitHub app.
That last bullet is the unlock. The agent gets read access and a prompt. Writes go through a tiny verifiable pipeline with explicit allow-lists. Security review nods. We ship.
💚 A small aside: kindred stacks
I love when the tools you’re using to build are built with the same tools you’re using to build with. The GitHub Agentic Workflows docs are built with Astro and Starlight. So is aspire.dev—Astro with Starlight, dressed up with the wider Starlight plugin ecosystem (astro-mermaid, starlight-llms-txt, starlight-sidebar-topics, starlight-image-zoom, the gorgeous @catppuccin/starlight theme, and more. Shout-out to Chris Swithinbank and the Starlight maintainers, the entire ecosystem feels designed by people who genuinely care).
There’s a real kinship there. The tool we use to automate docs and the docs site we automate into share the same foundation. Convenient, because the Mermaid sequence diagram in the next section renders the exact same way in both worlds.
The end-to-end pipeline
Here’s the flow we landed on. The protagonist is a workflow called pr-docs-check.md living in microsoft/aspire.
A run starts on pull_request: closed against main or release/*, gated by merged == true. From there, the workflow first runs a deterministic target branch resolver in plain bash before the agent ever wakes up:
Pull request milestone title (e.g. 13.4 → release/13.4 on aspire.dev).
Linked-issue milestone title (parse Fixes/Closes/Resolves #N from the body, fetch each issue, take the first non-empty milestone).
Pull request base ref, if it matches release/X.Y[.Z].
Fall back to main.
This is the linchpin. Milestones in the product repo map cleanly to release branches in the docs repo. When the agent finally runs, it knows exactly where the docs should land without any creative writing about target branches or guessing.
The agent reads the diff, scans linked issues, and decides: does this need docs? If yes, it drafts the actual content in a checked-out microsoft/aspire.dev workspace, following our existing doc-writer skill (voice, MDX conventions, Starlight components). It then emits a create_pull_request safe-output and hands off.
The safe-outputs handler takes over:
Title prefix: [docs]
Label: docs-from-code
draft: true (we never auto-merge)
Base branch: agent-supplied, restricted to main or release/*
Target repo: microsoft/aspire.dev
Reviewer: the SME identified from the source pull request’s reviews—i.e., whoever the product team trusted to approve the feature, now gets asked to approve the doc for that feature.
A companion job posts a marker comment back on the source pull request with the docs pull request link and minimizes any older pr-docs-check comments on re-run. The engineer who just hit Merge gets a notification within a few minutes: “Here’s the docs draft. Look it over?”
🔐 The safe-outputs contract
The whole security story comes down to a small, boring stretch of frontmatter:
That’s the deal in plain text. The agent gets a GitHub App token whose installation is scoped to exactly two repositories—the product repo and the docs repo—and nothing else in the org is reachable. It can only land pull requests against main or release/*. AGENTS.md and dependency manifests are off-limits by policy. If the pull request creation fails (network blip, conflict, anything), the framework falls back to filing an issue, so nothing is silently dropped.
This is the part security review actually liked. The agent’s reasoning is fuzzy. The action surface is not.
📊 By the numbers
Here are the stats from a rolling 30-day window (May 3 – June 2, 2026) spanning the back end of the Aspire 13.3 release and the run-up to 13.4:
Metric
Value
Product pull requests merged in microsoft/aspire
396 (338 main / 50 release/13.3 / 8 release/13.2)
pr-docs-check workflow runs
396
Draft docs pull requests created on microsoft/aspire.dev
82
– Merged
82 (100%)
– Closed without merge
0
– Still open
0
Docs pull requests target branches
52 → release/13.3, 27 → release/13.4, 3 → main
Median time-to-merge (docs)
44.8 hours
Merged within 24 h / 7 days
38% / 96%
Note: Numbers captured at the time of writing; the workflows keep running, so the totals only go up.
A few of those numbers deserve a second look:
396 runs → 82 pull requests is not a defect. The workflow runs on every merged pull request; most of them are internal refactors, test fixes, or dependency bumps with no user-facing surface. The agent saying “no docs needed” 300+ times is a feature.
100% merge rate says the agent’s docs picks are right. The tighter prompt we shipped after the v1 false-positive phase is paying off.
✅ What worked, ❌ what didn’t
What worked
✅ Milestone → release-branch mapping. This was the single highest-leverage choice we made. Engineers already set milestones on pull requests and issues; we got accurate target-branch routing for free.
✅ Draft-only, SME-as-reviewer. The agent never merges. The engineer who shipped the feature is the one who confirms the docs are right. We’ve stopped reverse-engineering features at the doc layer. The engineer just tells the docs draft what to say, in the place where they already are.
✅ Scoped GitHub app per workflow. Each workflow gets its own app token with explicit repo and permission scopes. Security review approved. We approved too; the first time we needed to rotate keys.
✅ protected-files: blocked. The agent cannot touch AGENTS.md, package manifests, or repo security config. Period.
What didn’t (at first)
❌ The agent’s “is this docs-worthy?” gate was too generous in the first version. It drafted pull requests for changes that were genuinely internal, such as a CI tweak or a logging refactor. The result: 9 closures of 69 pull requests (≈13%), so we tightened the prompt’s user-facing-change definition and added explicit negative examples (CI, internal helpers, tests-only). Now, the rate is trending down.
❌ Cross-repo pull request creation needed a mirrored checkout pattern that wasn’t obvious from the docs. The agent works in one repo; safe-outputs needs to find the target repo to push a branch. We solved it by checking out microsoft/aspire.dev twice—once as the current workspace, once under _repos/aspire.dev—so the safe-outputs handler can rediscover it deterministically.
❌ Big diffs blow prompt budgets. We pre-extract pull request metadata (linked issues, milestone, base ref) in pre-agent-steps bash, so the agent gets a small, structured summary instead of a giant payload. This is GitHub Agentic Workflow’s designed-in pattern, and it works.
Wrapping up
The changes we made shifted our thinking. A feature wasn’t considered done until the docs were. Docs no longer trail along behind it like a tin can on a string. The engineer’s review is the gate; the bot does the typing.
Critically, this doesn’t replace docs writers; it un-burdens them. Our writers used to spend most of their time reverse-engineering features. Now they spend their time on the things only a human can do well: narrative pages, sample programs, conceptual walkthroughs, the parts of the docs that don’t fall out of a diff. The bot handles the mechanical “this new option was added; here’s the reference page update” work that was never enjoyable for anyone.
Huge thanks to the GitHub Next team for GitHub Agentic Workflows (and for making the safe-outputs primitive a first-class part of the design), and to Chris Swithinbank and the Starlight maintainers for the docs platform we automate into. A genuine thank-you, too, to the security folks whose guardrails forced us to design this the right way the first time. The boring secret of good automation is that strong security constraints make the system more trustworthy and more correct.
If you build a product in one repo and ship docs in another—and especially if you have to do it inside any nontrivial security boundary—GitHub Agentic Workflows is worth a serious look. Start with one workflow, such as pr-docs-check, and watch what happens to your median time-to-docs.
🔗 The other workflows
pr-docs-check is the one I wrote this post about, but it’s not running alone. If you’re curious about the rest, the source is public:
milestone-changelog.md: runs every two hours, picks up newly merged pull requests in the active milestone, and maintains a 13.x-Change-log wiki page (new features, improvements, notable bug fixes) with a companion editorial-feedback issue. 346 runs.
release-update-support-mdx.md: on a stable Aspire release, drafts a [support] pull request on aspire.dev that updates the support policy page (promotes the new version, demotes the previous one, refreshes the “Last updated” badge).
update-integration-data.md: lives in the docs repo; runs pnpm update:all daily, refreshes NuGet metadata + GitHub stats + sample data, and opens a chore: Update integration data PR with supersede-and-close logic for stale runs. 27 runs, eight merged pull requests.
repo-pulse.md: a rolling three-day repo dashboard pinned to a single issue and updated in place: recent merges, pull requests awaiting review, new issues, discussion activity. One issue, always fresh.
Online multiplayer gaming continues to grow, with players demanding lower latency, higher concurrency, and more immersive experiences than ever before. For game studios hosting dedicated multiplayer servers on AWS, infrastructure decisions directly impact player experience and retention, server tick rates, and ultimately, revenue.
Games are becoming more computationally demanding while offering richer gameplay experiences. Studios need instances that maintain consistent player experiences in increasingly complex and dense computational game experiences.
In this post, we explore how AWS m8azn instances powered by AMD’s 5th gen EPYC processors perform with a real game: Mob Rush. M8azn instances offer up to 2x compute performance and 5 GHz CPU frequency compared to previous generation M5zn instances, and up to 24% higher performance than M8a instances. M8azn instances deliver up to 4.3x higher memory bandwidth and 10x larger L3 cache compared to M5zn instances allowing latency-sensitive and compute-intensive workloads to achieve results faster. These instances also offer up to 2x networking throughput and up to 3x EBS throughput versus M5zn instances. This post walks through the deployment of the game and reviews the performance metrics across varying player counts.
When to choose m8azn for multiplayer hosting
Not every workload requires m8azn. Game modes that require high performance and low latency computation are ideal matches for m8azn instances. M8azn instances are ideal for games that benefit from higher compute performance, larger L3 cache, and higher memory bandwidth.
Ideal use cases
Session-based high density multiplayer games: Games with discrete match sessions that spin up and tear down servers dynamically benefit from the fast startup performance of m8azn and high player density per instance.
Physics-intensive game servers: Titles relying heavily on PhysX collision detection, rigid-body simulation, and real-time raycast operations see significant gains from improved FPU throughput.
Variable player load scenarios: Live service games with daily peak hours or seasonal events benefit from the cost efficiency at both low and high utilization levels.
High-density hosting: Studios seeking to maximize concurrent game sessions per instance to reduce per-player infrastructure cost.
Mob Rush
In this post, we run the game Mob Rush. Mob Rush is a multiplayer game where players collect and grow a crowd in a war of numbers style competition. Mob Rush is built with the Unity game engine. Our test infrastructure includes a local test orchestrator, two game servers, and a series of load generation servers. The following diagram shows our testing setup:
Load testing configuration
Our load testing scenario compares the m5zn instance that our game currently runs on with the new m8azn to decide if the new instance is a migration candidate. We compare player experience metrics (FPS, tick-rate, and others) and instance performance metrics (CPU utilization, tick-rate, players per server, and others) to see how well Mob Rush runs on newer hardware.
Methodology
Benchmarks were conducted using a Unity multiplayer game server build, simulating concurrent player connections with synthetic load generation. Our current game servers perform well to around 4,000 simultaneous player connections before the player experience started to degrade as the server was overloaded. For our benchmarks we have tested each instance type at the 3,000 player threshold, and then increased to 4,000 players, 6,000 players and 8,000 players and recorded how each instance performed.
Parameter
Configuration
Instance Types
m8azn.xlarge (4 vCPU, 16 GB) vs. m5zn.xlarge (4 vCPU, 16GB)
OS / AMI
Ubuntu 22.04 LTS
Unity Version
Unity 2020.3.12f1, headless Linux build
Concurrency Scenarios
Gradual ramp from zero players to number of players that overload the instance causing player experience impact
Metrics Collected
Connection success rate, connection latency (avg/P50/P95/P99/max), batch processing time, run queue depth, context switches/second, softirq/second, TCP retransmits, listen overflows
Test Duration
562 seconds total (202 s ramp at 100 connections/sec + 360 s sustained hold)
Results
Our testing results are displayed in the following charts. The new m8azn instances start to shine as load increases. M5zn instances start to have significant latency spikes and max latency numbers once we get to around 6,000 CCU, which severely impacts the player experience. We can push the m8azn instances to 8,000 CCU before experiencing player impact or introducing any latency spikes >500ms.
Metric
m5zn.2xlarge
m8azn.2xlarge
Player Count
3,000
Average Latency
9.7ms
8ms
P99 Latency
22ms
28ms
Max Latency
43ms
23ms
Peak Run Queue
575
14
TCP Re-transmits
0
0
Latency Spikes > 500ms
0
0
Errors
0
0
Metric
m5zn.2xlarge
m8azn.2xlarge
Player Count
4,000
Average Latency
8.6ms
7.2ms
P99 Latency
29ms
24ms
Max Latency
76ms
29ms
Peak Run Queue
451
24
TCP Re-transmits
0
0
Latency Spikes > 500ms
0
0
Errors
0
0
Metric
m5zn.2xlarge
m8azn.2xlarge
Player Count
6,000
Average Latency
6.8ms
6.5ms
P99 Latency
22ms
22ms
Max Latency
29ms
27ms
Peak Run Queue
318
27
TCP Re-transmits
0
0
Latency Spikes > 500ms
0
0
Errors
0
0
Metric
m5zn.2xlarge
m8azn.2xlarge
Player Count
8,000
Average Latency
25ms
9.7ms
P99 Latency
542ms
27ms
Max Latency
2386ms
238ms
Peak Run Queue
651
87
TCP Re-transmits
0
0
Latency Spikes > 500ms
97
0
Errors
0
0
Price-performance comparison for 100k CCU
One methodology to calculating price/performance of these instances is to compare the cost of running enough instances to serve 100,000 players while maintaining an optimal and minimal latency player experience. All prices discussed in this section are based on us-east-1 OnDemand Linux pricing at the time of writing.
To serve 100k CCU with m5zn.xlarge instances, we would need to provision approximately 20 m5zn.xlarge instances (each instance can support 5,000 CCU before player experience degrades). Each m5zn.xlarge costs $0.3303/hr. That brings our hourly cost to $6.606/hr per 100k CCU.
In comparison, we only need 13 m8azn.xlarge instances to serve 100k CCU, thanks to the ~61% performance improvement of average latency of m8azn at 8,000 CCU per instance. Each m8azn.xlarge costs $0.4129/hr. Our hourly cost for 100k CCU in this scenario is $5.3677/hr. M8azn instances clearly demonstrate better price performance when compared to m5zn instances.
Conclusion
M8azn instances represent a compelling upgrade path for multiplayer game studios currently running on older generation instances. The combination of AMD EPYC processor improvements, enhanced memory bandwidth, and superior network performance delivers measurable benefits across the workloads that matter most for game hosting.
Try out m8azn in your development environment and calculate price/performance gains to see if m8azn is right for your workload. The Optimizing EC2: Hands-on Strategies for Cost-effective Performance workshop can guide you in comparing performance and calculating your overall price/performance savings across different instances.
Дишате ли? Само проверявам. Лято е, жега е и може би се потите над поредния проект с изтичащ срок. На морето е малко по-различно – разхлаждащи коктейли, водни спортове, хотелско обслужване. Може би не сте се замисляли колко се промени лятната ни почивка и колко названия, предимно чужди, навлязоха паралелно в езика ни.
В това само по себе си няма нищо неестествено, но създава известни неудобства или поне колебания – граматични и правописни най-вече. Нека да си поговорим задушевно в това задушно време за мохитата, съповете и олинклузива.
Вземи две фрапета от бийчбара, моля
Бих предпочела барът да е плажен или на плажа, но няма как да си затворя очите, че също толкова често се употребява и бийчбар¹, следователно е добре да знаем как се пише. Точно така – слято. Започваме с едно правописно правило: когато имаме сложно съществително име и едната или и двете му части не се употребяват като самостоятелни думи в българския език, то се пише слято. Нямаме бийч, значи пишем бийчбар.
Да видим какви разхладителни напитки се предлагат там. В менюто са включени фрапе, фредо капучино, айс лате, мохито, джин физ, айран. Нали забелязахте поне три правописни грешки? Браво, знаех си, че веднага ще приложите правилото за бийчбара към фредокапучиното, айслатето и джинфиза. (Това беше горчива ирония, насочена към правилата за слято, полуслято и разделно писане, а не към хората, които са принудени да ги спазват.)
Има още една грешка – пише се айрян, макар съседите ни да изговарят твърдо звука р в ayran и макар грешната форма айран да се среща доста често². Правописът на думата си остава силно дискусионен според мен и не бих определила промяната от айран на айрян през 2002 г. като мъдра стъпка. Но пък ако кодификаторът беше посочил причината, можеше и да реша, че е основателна. Така или иначе, надали някога ще я узнаем – остава ни да недоволстваме или пък да пишем айран напук. Според мен допускането на дублети тук би било оправдано.
Да се върнем обаче на другите разхлаждащи напитки, защото те вървят с граматични затруднения, както айрянът върви с узото (или пък беше обратното?). Общо взето, като носители на българския език се справяме с членуването, тоест бързо се ориентираме от кой род е съществителното име. Думите фредокапучино и айслате (също капучино и лате, разбира се) са от среден род, защото завършват на -о и -е – типични окончания на съществителните от този род в българския език (мляко, езеро, цвете, летище), и получават определителен член -то. Джинфиз пък завършва на съгласен звук, подобно на много други думи от мъжки род (сок, разказ), затова смело и без много замисляне му прикачваме съответния определителен член – джинфизът/джинфиза.
Мъките започват, когато се опитаме да поръчаме напитка не само за себе си, но и за компанията, и то когато названието завършва на -о. Моля, три капучина! За нас четири мохита и две узота с два айряна! Със сигурност мнозина ще се възпротивят, че се казва не капучина, а капучинота или капучини. Възражения – вероятно по-малко – ще има и срещу мохита и узота. А как е правилно всъщност? Няма кой да каже. Все още в БЕРОН липсва форма за мн.ч. при тези думи, а мохито изобщо не е включена. Липсите донякъде са разбираеми, защото има голям разнобой между формите за мн.ч. на капучино, а узота звучи доста странно (според мен уза е още по-странно).
От неудобствата може да се спасим с Моля, три чаши капучино и с по-разговорното За нас четири пъти мохито и два пъти узо с два айряна, но езикът е система и трудно търпи празни клетки, тоест липсващи форми за мн.ч. След като можем да си поръчаме три кафета, без да използваме чаша, би трябвало да можем да го направим и ако предпочитаме да пием капучино.
На шезлонгите или в морето с джетовете?
Пасивна или активна почивка? Дотук говорихме за пасивната и за да завършим темата, ще споменем за шезлонгите. Това, че цената на удоволствието да се излегнеш малко над пясъка непрекъснато расте, е ясно, но ясно ли е защо тази дума в множествено число се държи странно и запазва крайната си съгласна г? Сравнете: диалог – диалози; анцуг – анцузи; подвиг – подвизи. Промяната г ~ з във формата за мн.ч. на съществителните от м.р. всъщност е много стара и е известна като втора палатализация. Думите, които завършват на -нг обаче, са изключение. Освен шезлонгите актуални за лятото са и къмпингите, а ежесезонни са паркингите, лозунгите и брифингите например.
Тук ще отбележа и една тенденция в заемането на думи от английския. Все по-малко са задръжките при употребата на съществителни с наставка -инг. Не смеете да се пробвате в сърфинга, дайвинга или рафтинга? Ами пробвайте тогава шнорхелинга или пък си останете на сушата с джогинга. Колкото и да съм либерална към чуждите думи, понякога и за мен е прекалено. Малко по-яваш, ако може.
Явно не може, защото на водните забавления и средства за придвижване не успяваме да им измислим наши съответствия и ги вземаме наготово. Освен широко известните джетове и сърфове в морето се появиха съпове/падълбордове, кайтбордове, уиндсърфове. Има и други, но нека се ограничим със споменатите, защото на хора като мен би им се завило свят от толкова непознати уреди и спортове.
Джет, сърф и съп като едносрични съществителни имена от мъжки род очаквано получават окончание -ове в мн.ч. Защо обаче повечето от нас биха предпочели падълбордове и кайтбордове пред падълборди и кайтборди, при положение че по правило многосричните съществителни от мъжки род имат окончание -и в мн.ч.? Тук в съзнанието ни изпъква втората част на сложната дума, която представлява едносрично съществително и самостоятелно употребена, тя получава окончание -ове в мн.ч.: борд – бордове. Ето защо ние пренасяме това окончание и в сложната дума. По същата причина за правилни се приемат и формите билбордовеи уиндсърфове, а не билборди и уиндсърфи.
Думата съп е малко по-особена, но това се отнася само за произхода ѝ – идва от английската абревиатура SUP (Stand Up Paddle – гребане в изправено положение), иначе граматически се държи като съществително име от мъжки род, подобно на джиесем например, което също идва от абревиатура – GSM. Този начин на словообразуване в българския език не е толкова нов, имаше го още по времето на социализма, текезесетата (от ТКЗС, трудово кооперативно земеделско стопанство) и пуцовете (от ПУЦ, професионален учебен център). Все пак в последните десетилетия е по-активен: джипиес, джипи, пиар, айти, та дори и ейай (от AI); срещала съм ейая и ейаят.
Към същата група думи се числи и атеве. Допреди да напиша тази статия, не знаех, че съкращението АТВ е транслитерация на ATV, което пък е абревиатура на All-Terrain Vehicle, тоест превозно средство за всякакъв терен. И по плажа може да се движи, и по улиците на курортите, но не бива, защото последиците са жестоки. Има достатъчно черни пътища и безлюдни местности.
Олинклузив или ултраолинклузив?
Мнозина предпочитат хотелските удобства, когато са на почивка, затова да насочим погледа си към някои варианти за настаняване. Затруднения в правописа ще ви създадат най-вече олинклузивът и неговата хипер-мега-супер разновидност (това и аз не знам как се пише) – ултраолинклузивът. Прилагайки правилото за бийчбара, следва да пишем и тези думи слято, макар че в ултраолинклузива става твърде пренаселено с тези три пълнозначни съставни части. В българския език те може и да не съществуват като самостоятелни думи, но в английския си имат свое отделно значение, ние някак паралелно осъзнаваме това и по дяволите, иска ни се да ни е по-разчленено, та да го схванем по-бързо, моля, ако е възможно, защото не пишем по немски образец все пак – ето, излях си болката на един дъх. Та ултраолинклузивът като правопис е за суперпосветените на правилата, така да го кажем. Останалите може да кривнат от правия път с ултра олинклузив.
В хотел с такъв тип настаняване със сигурност има пулбар. И тази дума се пише слято, но няма да обяснявам защо, тъй като би било обидно за читателите. За лобибара обаче нещата стоят различно и леко проблемно. Със сигурност имаме самостоятелна дума лоби – свързваме я с настойчиви хора, оказващи влияние например при прокарването на закони в нечия изгода. Но употребяваме ли тази дума и за нещо друго? На практика да, може да кажем Ще се видим в лобито на хотела, когато срещата е във фоайето. Малкият проблем е, че не знаем (поне аз не знам) дали думата с това значение е призната за част от книжовния език. Ако имаше официална санкция, щяхме да можем да пишем и лобибар, и лоби бар. А големият проблем, който прозира тук, е следният: нерядко кодификаторът се бави да признае широко употребявани думи и ние трябва да чакаме неговото решение.
Преди да стигнете до лобибара в хотела, със сигурност ще минете покрай рецепцията. Все по-често тази дума се употребява нечленувана, когато всъщност означава определено място и следва да се членува. Със сигурност сте виждали предупреждения от типа Всички доплащания се извършват на рецепция или пък словосъчетанието цени на рецепция – аз досега не съм срещала цени на рецепцията. Случаят според мен е аналогичен на пленарна зала и без да се впускам в обяснения, ще отбележа, че за езиковото съзнание е по-важно какво се прави на рецепцията – посрещат се и се изпращат гостите, дава им се информация, извършват се плащания и т.н., – отколкото това, че е конкретно място в хотела.
Е, време е вече да се настаним удобно в нашето конкретно място в хотела и да се насладим на онези няколко дни, в които заслужено се отдаваме на морето, плажа и слънцето. Желая ви отморяващо и прохладно лято! И не прекалявайте с джинфиза на бийчбара!
1 Употребите на плажен бар, бар на плажа и бийчбар са проверени в CLASSLA.
2 Съотношението между употребите на айрян и айран в CLASSLA е показателно – 5996:3683.
Езикът може да е вкусен и извън блюдото – онзи, българският език, на който говорим от малки и на който около 24 май се кълнем в обич. А той в същността си е средство за общуване и за да ни служи добре, непрекъснато се променя. Да го погледнем в неговата динамика и да се опитаме да разберем какво става и защо, кои са движещите механизми и как те са свързани с обществените процеси. И тъй като задачата не е лека, ще го правим постепенно – на порции.
Managing petabytes of search data means making tough choices: keep everything fast and expensive, or make it affordable but read-only. UltraWarm is a proven, cost-effective solution for read-heavy historical data. However, some workloads occasionally need to update historical records, such as late-arriving data or compliance corrections. With UltraWarm, you must migrate those indices back to hot, perform the update, and migrate back. What if you could write directly to your cost-effective warm storage instead?
In this post, I show you how writable warm storage removes the costly migration cycle. You can reduce your infrastructure costs by up to 48 percent and update historical data in seconds instead of hours. I walk through a real-world cost comparison and performance benchmarks, and help you decide when to use writable warm versus UltraWarm.
The challenge with tiered storage
Amazon OpenSearch Service handles data-intensive search and analytics workloads, from real-time log analytics and application monitoring to security event detection. As your data volumes grow from terabytes to petabytes, you face a fundamental question: how do you keep recent data fast while making earlier data affordable?
OpenSearch Service addresses this with a tiered storage architecture:
Hot – Highest performance for active indexing and search using instance-attached storage.
Cold – Fully detached from the cluster, with the lowest cost for rarely accessed data. Cold indices must be migrated back to UltraWarm or hot before any reads or writes can be performed.
For immutable log data, this model works well. However, a specific class of workloads hits its limitations when they occasionally need to write to earlier data, and read-only becomes a bottleneck.
Prerequisites
To use writable warm storage, you need the following:
An Amazon OpenSearch Service domain running version 3.3 or later.
OpenSearch Optimized (OI2) instance family support in your AWS Region.
Workloads with a minimum 5-second refresh interval.
Data nodes using the OpenSearch Optimized instance family (OR2 for hot, OI2 for warm).
Note: Writable warm doesn’t currently support the cold storage tier.
The UltraWarm bottleneck
With UltraWarm, updating even a single document requires migrating the index back to hot, performing the write, and migrating it back. This round trip involves a force merge (consolidating index segments), snapshot creation, and shard relocation. These operations consume significant CPU, memory, and disk space on your hot nodes, and they take approximately 130 minutes per 100 GB index. This time was measured on a domain with 3 × r6g.2xlarge hot nodes, 3 × ultrawarm1.large warm nodes, and 3 dedicated leader nodes (US East, N. Virginia), using a single-shard index with one replica. Actual times vary based on domain configuration, shard count, segment count, hot node utilization, and migration queue depth. The result is that you over-provision hot nodes, build complex pipelines, or keep data in hot longer than necessary, which increases cost and complexity.
Introducing writable warm storage
OpenSearch Service now offers writable warm nodes that use OpenSearch Optimized (OI2) instances, the same instance family that powers durable, Amazon S3-backed storage on hot nodes. Because data is already persisted on Amazon S3, tier transitions become a lightweight shard relocation rather than a resource-intensive migration. The Lucene engine, which is OpenSearch’s underlying search library, operates identically on both tiers. As a result, writable warm nodes support active writes, background merges, and periodic refreshes, just like hot nodes.
Late-arriving data, compliance backfills, and corrections that previously required a warm-to-hot-to-warm round trip now resolve with a direct write in seconds. There is no force merge, no snapshot, no shard relocation, and no hot node resource consumption.
UltraWarm (legacy) data flow: Data is ingested into the hot tier (SSD, read and write). Index State Management (ISM) policies migrate indices to UltraWarm (Amazon S3-backed, read-only). Any update requires migrating the index back to hot (dashed arrow), writing, then migrating back.
Writable warm (new) data flow: Same ingestion path through hot, with ISM transitioning indices to writable warm. The key difference is that writable warm supports both reads and writes. Late-arriving updates go directly to warm, with no migration back to hot. Because both tiers use Amazon S3 as durable storage through OpenSearch Optimized instances, transitions are lightweight shard relocations, not resource-intensive migrations.
The benefits: cost, operations, and flexibility
Writable warm delivers advantages in three areas: cost, operational simplicity, and flexibility.
Cost
Unlike UltraWarm, which only offers on-demand pricing, OI2 instances support Reserved Instance (RI) pricing, a commitment-based discount model. By committing to a 1-year or 3-year Reserved Instance, you can save 31–52 percent compared to UltraWarm nodes. This makes writable warm significantly more cost-effective for predictable, long-running workloads. The newly introduced Database savings plan for OpenSearch Service provides savings of around 22 percent over UltraWarm instances. Both tiers use Amazon S3 for durable storage, so node failure means only temporary unavailability, not data loss. For cost-sensitive workloads that can tolerate brief downtime during node recovery, you can configure zero replicas on warm indices to reduce costs further.
Real-world cost comparison
Consider a workload ingesting 2 TB/day with 210 days total retention, where updates can arrive at any point. With UltraWarm’s read-only constraint, you must keep data in hot for 30 days before migrating to warm. With writable warm, updates happen directly on warm, so hot retention drops to only 7 days.
At small scale, the hot tier reduction benefit is modest. Writable warm is still cost-effective if you need write capability on warm data, can commit to RI pricing, or value the operational simplicity of eliminating migration pipelines. For purely immutable data with short retention, UltraWarm on-demand might still be cheaper. Use the AWS Pricing Calculator to model your specific scenario.
The following table shows estimated monthly costs using on-demand and All Upfront Reserved Instance (AURI) pricing in the US East (N. Virginia) Region as of March 2026. For the latest pricing, see Amazon OpenSearch Service pricing on the AWS website.
Component
Hot + UltraWarm (30d hot / 180d warm)
Hot + writable warm (7d hot / 203d warm)
Hot data nodes
$12,264 (21 × or2.2xlarge)
$12,264 (21 × or2.2xlarge)
Hot EBS cost
$10,212.84 (21 * 3986 GB)
$2,636
Hot remote storage
$2,008.28
$518
Warm data nodes
$39,128 (20× ultrawarm1.large)
$50,409 (15× oi2.8xlarge)
Amazon S3 storage
$9,504
$1,070
Leader nodes
$1,307 (3 × m8g.2xlarge)
$1,307 (3 × m8g.2xlarge)
On-demand total
$74,427
$69,297
1-year AURI
$69,674
$43,918 (~36% less)
3-year AURI
$67,367
$34,939 (~48% less)
Database savings plan
$71,708
$55,406 (~22%)
Operations
Reclaim hot node capacity. Writable warm removes two common causes of hot node over-provisioning: reserving 35 percent of disk space for force merge operations, and maintaining extra capacity to temporarily move data back to hot for writes. You can run your hot tier at higher utilization, which reduces the number of hot nodes you need.
Simpler migrations. UltraWarm migrations are multi-step operations (force merge, snapshot, and shard relocation) that need careful scheduling during low-traffic windows, and they are limited to 10 queued at a time. Writable warm simplifies this to a lightweight shard relocation, with more straightforward ISM policies and no scheduling constraints.
Flexibility
UltraWarm offers only two instance sizes: ultrawarm1.medium (1.5 TiB) and ultrawarm1.large (20 TiB). Writable warm with OI2 instances offers a full range from oi2.large to oi2.16xlarge. Each size addresses up to 5× its local cache size, so you can right-size warm capacity precisely to your workload.
Search performance
We benchmarked search latency using the NYC Taxis workload, comparing writable warm (oi2.large) against UltraWarm nodes. All measurements are P90 latencies.
On the NYC_TAXIS benchmark, writable warm matched or beat UltraWarm on 6 of 7 query types at P90, including lightweight filters, ranges, sorts, and time-histogram aggregations. For most real-world search patterns, writable warm delivers comparable or better performance than UltraWarm, plus the ability to write directly to the tier.
Search performance: writable warm compared to UltraWarm
Task
Writable warm node latency in ms
UltraWarm latency in ms
UltraWarm vs. writable warm diff %
NYC_TAXIS workload type
** **
** **
** **
default (P90)
21.287
23.857
12.07223
range (P90)
21.23
21.016
-1.00718
distance_amount_agg (P90)
5,069
3929.23
-22.48406
autohisto_agg (P90)
21.076
22.002
4.39348
date_histogram_agg (P90)
21.363
21.792
2.01031
desc_sort_tip_amount (P90)
23.224
23.797
2.46636
asc_sort_tip_amount (P90)
22.483
22.482
-0.00445
When to choose what
Should you switch from UltraWarm to writable warm? It depends on your workload.
Requirement
Writable Warm
UltraWarm
Write enabled
✓
Read-only
Reserved Instance pricing
✓
✗
Instance size flexibility
Wide range (large–8xlarge)
2 options only
Cold tier support
✗
✓
Need for OpenSearch Optimized instance families
✗
✓
Concurrent tier transitions
✓
✗ (sequential)
Hot node impact during migration
Minimal
High (CPU/memory)
Clean up resources
If you created a test domain to evaluate writable warm storage, delete it to avoid ongoing charges. In the OpenSearch Service console, select your domain and choose Delete. This removes all nodes and stops Amazon S3 storage charges for that domain.
Summary
In this post, I showed you how writable warm storage eliminates the costly migration cycle that UltraWarm’s read-only limitation creates. You get up to 36 percent cost savings with 1-year Reserved Instances, faster search performance, and a simpler operational model. Writable warm also removes data transitions between tiers, and Reserved Instance pricing becomes available for warm storage for the first time.
Writable warm requires OpenSearch Service version 3.3 or later with OI2 instances. For domains needing cold tier support, earlier OpenSearch Service versions, or non-optimized instance families, UltraWarm remains the right choice.
Next steps: Start by analyzing your current hot and warm split. How many days of data do you keep in hot only to accommodate occasional updates? Use the AWS Pricing Calculator to model your potential savings, and enable writable warm on a test domain in minutes. At the time of this post, writable warm is supported on OpenSearch Service version 3.3. For step-by-step instructions, see Migrating to writable warm storage in the OpenSearch Service documentation.
Have you tried writable warm storage? I’d love to hear about your experience and any questions you have in the comments.
About the author
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.