Tag Archives: Engineering

The bridge to agent-scale Git infrastructure

Post Syndicated from Eleanor White original https://github.blog/engineering/architecture-optimization/the-bridge-to-agent-scale-git-infrastructure/


Over the past year, while Git traffic grew exponentially, 99.99999% of the Git repositories on GitHub began living in two places at once: our own data centers and Azure. The rest are among the busiest on the platform, and they’re held back by just one thing: a 25ms round trip.

In an earlier blog post, we laid out the two principles behind our next-generation storage architecture:

  • Minimize coordination
  • Decouple storage from compute

In this post, we’ll look at the immediate bridge between our old design and our new one. We’ll talk about how a change to the way our quorum serializes writes gets the busiest repositories into Azure, and how that same step opens the door to replicas that keep no persistent state of their own.

How we got here

The timing and shape of the move were driven by the agentic shift: rapidly growing traffic, a hard ceiling on the power capacity of our data centers and the surrounding geographical region, and physical supply chain constraints. At the beginning of our migration, the remaining capacity with all possible expansion gave us enough time to migrate safely, but not enough to do it slowly.

While we had prototyped new architectural approaches for scaling Git on cloud infrastructure, changing the storage architecture and the hosting platform at the same time, under that growth, carried too much risk. Given our scale and how critical the service is to the community, we needed to sequence the work carefully: move the data first, then evolve the architecture once it arrived.

Git operations are a Tier 0 service: they’re a core function of GitHub, and nearly everything in GitHub depends on the repository data. That meant the data, services, and every dependency had to move first. The first copy of a repository landed in Azure’s Central US region in August 2025. Today, most repositories have two of their five replicas in Azure, and Azure serves about half of Git traffic. For most of that time, Azure absorbed our growth while our own data centers held steady.

During the migration, traffic follows the data. Most repositories have five replicas to balance durability, availability, and scale. To minimize read latency, Spokes uses preferential routing to direct traffic to a replica in the same region and availability zone as the request. As replicas move to Azure, the share of the traffic that can be served from Azure grows with each one, until the last replica arrives and the old region stops receiving requests entirely. Every repository goes through this process:

No repository left behind

So, what about the remaining 0.00001% of repositories?

In our current architecture, a repository hosted in more than one region has a lower write throughput ceiling. Spokes’ quorum only needs a majority to commit a write, but it waits for every healthy replica to respond (up to a timeout) before moving on, so slow replicas delay every write rather than falling behind. In practice, that means the slowest healthy replica sets the latency of every write, and because each write holds a repository-wide lock until it commits, it also sets the throughput ceiling. Once a replica lives in the target Azure region, it’s about 1,000 miles from our data centers, making each network round trip around 25ms, and a write requires multiple round trips. A repository already near its throughput ceiling can’t absorb that on every serialized write; it would fall further behind with every push.

This is where our first principle, minimizing coordination, comes into play. Specifically for this case: don’t serialize what doesn’t need to be. Git’s consistency requirements apply to each reference individually. Once the server accepts an update to a reference, no client should see an older value afterward, only that value or a newer one. That holds whether the update is a fast-forward, a force push, or a delete. But in most cases, updates to different references don’t depend on each other and can be applied independently.

So, we’re changing the scope of the quorum from the whole repository to individual references.

Holding a repository-wide lock for the entire reference update, round trips included, turns the quorum into a single, unordered request queue with one write in flight at a time. Each write in that queue takes as long as the slowest healthy replica. As the write rate grows, so does the queue, and each write waits longer than the one before it.

With per-reference serialization, only writes that contend on a shared reference must wait for each other. In practice, most writes to a repository touch different references—for example, agents and developers pushing to their own branches—so they no longer wait on each other. Each write still experiences the cross-region latency but writes to different references can be in flight at the same time. That removes the single bottleneck on the repository. We can visualize how queueing and timeouts build up by comparing the two approaches side by side under similar load:

These changes raise a repository’s write throughput ceiling roughly in proportion to how many references are being written at once, and get the last remaining repositories into Azure. In internal testing with realistic traffic, writes to different references consistently saw 3–4x higher throughput at lower latency.

Scaling reads on their own

Most repositories are heavily read-skewed, with somewhere between 100 and 1,000 reads for every write. Today, reads and writes hit the same replicas, so they compete for the same resources and can’t be scaled independently.

Decoupling storage from compute changes that. Once repository data is durably stored in Azure Blob Storage and updated with each incoming write, we can begin to decouple read traffic from the write quorum, and independently scale the compute needed for each. For example, reads would benefit from far more in-memory cache than we give them today, but that memory is also needed for compaction. Once they’re separated, each gets its own configuration.

The first step is making object storage a member of the quorum, but a special one that we call a virtual replica. The compute that represents it is our first fully ephemeral node. It serves as a bridge to object storage, streaming incoming writes and serving no reads. Because it’s a member of the quorum, a write isn’t acknowledged until it’s durable in object storage.

That adds a little latency to each write, but just like the cross-region round trip, per-reference serialization keeps it from becoming a throughput bottleneck. The virtual write replica also coalesces transactions on disjoint references into a single set of network I/Os to object storage, so a busy repository pays for one round trip across many concurrent writes instead of one per write. For now, the virtual write replica also handles compaction, and read replicas inherit the compacted state.

With the virtual write replica in place, we can scale out as many virtual read replicas as a repository needs, on demand. Each read replica warms its cache from Azure Blob Storage and starts taking read traffic without joining the quorum, using object storage as the source of truth and reconciling its local state against it in the background.

This step unlocks significant read scale. In internal testing, our distributed load testing harness routinely found ceilings in the current architecture. Against virtual read replicas, the harness hit its own limits first. It also comes with a few challenges that are solved in subsequent steps:

  1. Virtual read replicas start out reconciling with object storage in the background, which makes them a good fit for some read workloads but not all of them. Serving every kind of read requires moving that reconciliation in-line, fetching only the reference state and data that changed since the replica last caught up.
  2. While we’ve incrementally increased write throughput by shifting to a per-reference model, the quorum remains the bounding factor for increasing further.

How can we push write throughput further?

What comes next

The GitHub of today was built on the Spokes quorum, and over the years Spokes has become much more than a quorum coordinator. It’s also an intelligent routing and load balancing layer, a unified API that internal clients use to access and transact repository data at well over 100 billion operations per day, and a collection of background systems we call “sweepers” that handle repository placement, repair, and rebalancing between storage nodes.

The vast majority of what lives under the Spokes service umbrella applies just as well to the new architecture as it does to the existing one. But we also recognize that the commit protocol that has taken GitHub to a billion repositories won’t take it to the next billion and beyond.

We’re not done yet. With these steps in place, the next limit is write throughput. Every write still waits on replicas that each hold a full copy of the repository, yet very little of a write actually needs to be ordered. Git objects are immutable and named by their contents, so they can be stored without anyone agreeing on them. What’s left is small, and the hard part is ordering it as fast as possible without moving the bottleneck somewhere else. Designing for that on a whiteboard is one thing. Doing it underneath a billion repositories that developers and agents are writing to around the clock is another. Getting there forced us to rethink the second-order effects too, like where durable reference state lives and how compaction and garbage collection scale.

Agent traffic isn’t slowing down, and each of these steps gives repositories more headroom as it lands, without waiting for the end state.

Coming next week: At GitHub Universe and in the final post in this series, we’ll take a tour through that end state, what it does for write throughput, the second-order problems we had to solve to get there, and why there’s still a (small) consensus group somewhere in the stack.

Thanks to Brian Celenza for their contributions to this post.

The post The bridge to agent-scale Git infrastructure appeared first on The GitHub Blog.

App stability at scale with a layered defense

Post Syndicated from Grab Tech original https://engineering.grab.com/app-stability-at-scale-with-a-layered-defense

Introduction

Artificial intelligence (AI) coding assistants have changed how fast we can ship. They have not changed how mobile users experience a crash. That gap is the pressure every mobile stability program now operates under.

The Grab superapp, which millions of users across Southeast Asia use to book rides, order food, and more, operates at a large scale. On Android alone, it spans millions of lines of code, with many engineers shipping code into every release on a weekly release cycle. Maintaining app stability is not just a technical goal; it is a product commitment to those users. With the rise of AI coding assistance, the volume of change in every release has grown, and so has the number of ways a single change can break something in an already large codebase.

At this scale, crashes leaking into production are an ongoing risk. A high volume of merge requests lands every week across many product teams, many shipping features behind flags that are switched on independently of releases. Any one of those changes can introduce a crash. Most of the time, the stack trace points straight to the responsible change and team. The hard cases are the exceptions: crashes without a usable stack trace, such as native crashes without debug symbols, where finding the cause means cross-referencing release timelines, flag rollouts, and session metadata.

The cost of a crash is also steeper on mobile than on the web. A web fix can go live in minutes; a mobile crash that ships stays on users’ devices. With more than 100 million downloads on Google Play as of October 2026, not every user has auto-update enabled, and until a force update reaches them, some will run a broken version for months. Meanwhile, the crash degrades their experience, drags down the crash-free session rate, and can hurt the app’s ranking on Google Play. If it is severe enough to need a hotfix, the team also has to prepare an out-of-cycle build, pulling time away from planned work.

We keep the Grab superapp stable with a set of layered practices. Data points and tooling examples come from Android. The Grab iOS app follows the same strategy with platform-equivalent tooling (SwiftLint in place of Android Lint), and both apps share one release pipeline for Grab Early Access (GEA), our early access build, as well as staged rollout, production monitoring, and crash triage.

A layered defense for app stability

Our strategy is rooted in a shift-left mindset: the earlier we catch a crash, the cheaper and less disruptive it is to fix. A crash caught by a lint rule costs nothing. A crash caught in GEA takes an engineer fewer hours to fix. A crash that reaches production costs user trust. That is why we invest most heavily in prevention: the goal is to stop crashes before they reach production, not to fix them there.

The system is a series of filters, each anchored to a specific phase of the feature development lifecycle, so the right check runs at the right time, and a crash that slips past one layer has another waiting downstream.

The colors mark the three phases on the stability side: prevention (pre-merge through pre-prod), the release gate (staged rollout and feature flags), and response (post-release).

Prevention: Our primary line of defense

Lint checks: Stopping crashes at the code level

Lint is the cheapest layer we have: static analysis that flags unsafe patterns, application programming interface (API) misuse, and likely crashes before the code ever runs. We use Android Lint on Android and SwiftLint on iOS. Both run in continuous integration (CI) on every merge request, and high-severity findings are merge blockers. Pre-existing issues never block a merge request: on Android, a lint baseline records them, and on iOS, SwiftLint reports only on the lines a merge request changes. Either way, a merge request fails only on the findings it introduces.

Our rules come from three sources:

  • Platform rules. Built-in checks cover well-known dangerous patterns, such as incorrect API usage and threading mistakes, and form the foundation for every build.
  • Community rules. Open-source rule sets cover third-party libraries and patterns the built-in checks miss. We adopt the ones relevant to our stack.
  • Custom rules. Both Android Lint and SwiftLint support custom rules, and we have built our own on both platforms. When a post-mortem or crash investigation traces a production crash to a specific code pattern, we write a rule that blocks that pattern from coming back.

Custom rules matter more as AI-assisted coding grows. An assistant can write code that looks right in general without knowing which patterns have crashed our app before. Custom rules turn those lessons into checks that apply equally to hand-written and AI-generated code.

Lint routinely blocks merge requests carrying Fatal issues: the kind that typically crash the app immediately or stop it from running.

Testing strategy: Multiple layers of confidence

Lint catches unsafe patterns statically, but it cannot run the code. Tests are where we catch crashes and regressions that only appear at runtime. No single test type covers everything, so we apply multiple complementary layers:

Layer Focus
Unit tests Business logic correctness at the function and class level
User interface (UI) component tests Rendering and interaction behavior of individual UI components
End-to-end (E2E) automation tests Integrity of critical user journeys
Quality assurance (QA) manual testing Real-device exploratory testing for new features and significant code changes

Each layer targets a different class of crash risk. Unit tests block logic errors before they merge. UI component tests catch crashes tied to rendering or state mismatches; like unit tests, they run pre-merge on every merge request, so a rendering regression is blocked before it lands. E2E automation covers the flows with the highest business impact and runs on a per-team schedule and on every release train. QA manual testing focuses on new features and significant code changes, plus a full regression pass on every release, catching the edge cases that automation misses on real devices.

Grab Early Access (GEA): A one-week pre-production gate

Even with strong lint rules and test coverage, some crashes only surface under real-world conditions. GEA is an early build app by Grab. GEA builds are actively monitored for crashes, and any new crash surfaced during the GEA window automatically creates a ticket assigned to the responsible team. Issues reported by internal users, including functional defects and feature breakage, are ticketed the same way. Teams are required to resolve all blocking GEA issues before the release proceeds to production. GEA’s user base is large enough to expose real-world edge cases that automated tests miss, but small enough that issues are contained before reaching production users. GEA is a mandatory quality gate on every release cycle.

A crash that only a handful of internal users see in GEA would, at production scale, translate into a substantial spike in the crash rate, extra support load, and potentially a hotfix. Catching these crashes before they ever reach production users makes GEA one of our most important stability gates.

The impact adds up. Across both platforms, GEA catches new crashes every month, and each is fixed before the release reaches production. Each of them could otherwise have reached production users, including the long tail of users who do not auto-update and would have kept hitting them for months.

The release gate: Staged rollout and feature flags

Prevention keeps crashes out of the build. But the moment a release enters production, it meets something neither lint rules nor GEA can fully replicate: the full diversity of the real world, with thousands of device models, operating system (OS) versions, carrier configurations, and usage patterns that no internal test environment can mirror. So we treat the release boundary as its own critical checkpoint, a release gate between a validated build and full user exposure. Two mechanisms operate at this gate: staged rollout monitoring to limit the blast radius of the release itself, and feature flags to decouple feature activation from deployment.

Staged rollout monitoring

We do not ship to all users at once. Every release follows a staged rollout over the course of a week, starting with a small percentage of users and expanding in steps to the full user base. Each increment gives the monitoring system time to accumulate meaningful signal before the next expansion. During this window, we run a dedicated monitor that tracks the crash rate of the new version in isolation, separately from the baseline of all active versions. The separation matters: a regression in the new release can be masked in aggregate metrics if the older, stable versions dominate the user base.

If the new release shows an abnormal crash rate at any point during rollout, an automated system pauses the release before it expands further. The rollout does not resume until the responsible team has reviewed the issue and explicitly approved continuation. The pause gives us a hard stopping point: a regression introduced by the new release is contained to the fraction of users who received it, instead of compounding across the full install base.

In recent months, several Android rollouts were paused for on-call review. Most resumed after investigation: either the signal turned out to be noise, or the crash sat behind a feature flag that could be turned off without a new build. A small number stayed halted at the first stage until a hotfix shipped. We tune the monitor to be sensitive on purpose: a paused rollout takes an engineer a few hours to fix, while a missed regression can cost weeks. In one of the halted cases, only a small number of users hit the new crash before the halt, and the rest of the user base never saw it. Without the automated halt, the rollout would have kept expanding on schedule, taking the crash to the full user base.

Feature flags: Controlled activation with an instant kill switch

Even when a release ships cleanly, the features inside it are gated independently. All significant new features ship behind feature flags, decoupling deployment from activation. Flags let us roll out a feature gradually, monitor crash rates at each increment, and, critically, disable a feature instantly if its crash rate exceeds a defined threshold, without waiting for the next release cycle. For a team shipping weekly, the kill switch is what makes fast iteration safe, and it is also our first lever when something does reach production.

Response: When prevention falls short

No prevention layer is perfect. Crashes can still reach production: a misconfigured remote flag opens an unexpected code path, an OS update from a device manufacturer changes system behavior in ways no test environment can predict, or a bug slips past GEA and the rollout monitor because it only triggers under a specific combination of device state and user action. When that happens, the goal shifts from prevention to containment. For these scenarios, speed of detection, stopping the bleeding, and accuracy of diagnosis become the priority.

Real-time monitoring and on-call response

We maintain real-time crash rate monitoring across all active versions, with two complementary signals that can trigger an on-call alert:

  • Hard threshold breach. The crash rate of an active version crosses an absolute limit we define as unacceptable.
  • Abnormal trend. The crash rate spikes sharply relative to its own recent baseline, even if it has not yet crossed the absolute threshold.

We watch both because they catch different failure modes. A hard threshold catches slow drift: if the crash rate creeps up a little with each release, every step looks normal against its recent baseline and the trend alert never fires, yet over a year the rate can quietly increase. An abnormal trend catches the opposite case: a sudden regression, early, before it grows large enough to breach the threshold.

When either signal fires, an on-call engineer is immediately notified. The engineer begins by reviewing Firebase Crashlytics reports to identify which crash issues are driving the anomaly, then loops in the team responsible for the affected area. From there, the response follows a three-step decision framework, ordered from cheapest to most expensive:

  1. Disable the responsible feature flag. If the crash is tied to a feature behind a flag, turning the flag off stops new users from hitting the issue immediately, with no new build required. Turning off a flag is by far the fastest mitigation we have.
  2. Apply a server-side fix. If no flag covers the affected code path, the team looks for any server-side lever that can contain the impact: updating an API response, changing a config value, throttling a backend dependency. Any change that takes effect without shipping a new app build is preferred over one that does.
  3. Ship a hotfix. If neither of the above is available, the team evaluates whether the crash warrants an out-of-cycle hotfix release, using defined criteria: business criticality of the affected flow, crash severity, user impact, and time to the next scheduled release. After the hotfix is delivered, a post-mortem is required. The goal is not just to document what went wrong, but to identify concrete action items that close the gap in our prevention layer, such as adding a lint rule to block the same pattern, wrapping a similar feature behind a flag, or improving test coverage for the affected code path. Every hotfix should make the next one less likely.

Automated crash triage

Automated ticket creation in GEA and production is not a collection of separate tools; it is one triage pipeline that follows a release from GEA into production. In production, running in parallel with real-time monitoring, it creates a production issue ticket for crashes that cross a minimum threshold. We assess crashes by type, user impact, and whether they’re actionable. Tickets are created for issues that meaningfully affect users; low-volume, device-specific crashes that we can’t address are monitored but don’t receive tickets. This keeps the team focused on the problems users are most likely to feel.

For crashes that do qualify, the ticket is routed to the owning team with a service-level agreement (SLA) tied to severity: Urgent, High, Medium, or Low. The initial priority is set automatically based on crash rate and user impact, but teams can re-evaluate and adjust the priority based on the business criticality of the affected feature.

Feature flag rollout history: Diagnosing crash spikes

Some of the hardest on-call moments start the same way: the app suddenly begins crashing, alerts fire, and there is no obvious reason why. Nothing shipped that day, and in the worst cases the crash arrives without a usable stack trace to point anywhere. With many product teams working on the same app simultaneously, the culprit is often not new code in the latest release; it is a feature flag quietly rolled out to a wider user segment in the background. Finding that flag by asking around does not scale: without a central view, an on-call engineer can spend hours before the responsible rollout turns up.

So we maintain a feature flag rollout history: a dashboard that correlates flag activation events with crash rate changes over time. Instead of polling every team, the on-call engineer gets an immediate view of whether any flag rollout aligns with the onset of the spike, turning a slow manual search into a quick read of a timeline.

Deep observability: Enriched diagnostics for hard-to-debug crashes

Firebase Crashlytics is our primary crash monitoring tool, but some crashes resist easy diagnosis: crashes with unusable stack traces, native crashes from third-party libraries without debug symbols, or issues that only manifest on specific device configurations. Without a clear stack trace, there is no obvious starting point for investigation.

We handle these cases in two ways.

First, we use Crashlytics’ built-in export to BigQuery, so we own a copy of the crash data. The data is pseudonymized and handled under Grab’s data protection policies. That copy gives us two things Crashlytics alone cannot:

  • Query the data when needed: full Structured Query Language (SQL) analysis across device attributes, app versions, and session metadata, surfacing patterns invisible in individual crash reports.
  • Build internal dashboards that serve app stability investigations, broken down by crash category, device tier, or release version.

Second, we enrich every crash session with custom metadata attached directly to Crashlytics. Among the most useful attributes we track:

  • Last visited screen: the last screen the user was on before the crash occurred (for example, the last Activity and Fragment on Android), helping reconstruct the navigation path that led to the failure.
  • Install source: whether the app was installed from Google Play or a third-party store (Android only). Some third-party stores distribute unofficial copies of the app, and more than once we have traced a crash back to a corrupted copy served by one of them. Knowing the install source lets us recognize these failures as environmental, instead of chasing them as code defects.
  • Session history: from the session data, we can reconstruct the sequence of screens and actions leading up to the crash, which is often the key to a crash investigation.

We attach many other custom attributes, but these three have proven the most useful in hard investigations. Together with structured logs and breadcrumbs, they give us a trail of events leading up to the crash, often the only meaningful signal when the stack trace alone is not enough.

Crashes we control, not eliminate

Different crashes call for different responses. Some are fixed in our code, while others, such as rare native crashes in operating system (OS) code, out-of-memory errors on devices near their limits, or failures from unofficial and sideloaded installs, are better handled through monitoring and containment.

We keep these categories at an acceptable baseline and monitor them on dedicated dashboards, separate from actionable issues. If any of them starts trending up instead of sitting flat, that spike triggers the same triage process as any other anomaly.

Conclusion

Sustaining stability across a codebase of millions of lines on Android alone, with many engineers shipping a release every week and AI assistance accelerating the rate of code change, is not the work of any single tool or team. It is the cumulative effect of a layered defense: lint rules and tests blocking bad code at the merge gate, QA and GEA catching what slips through, staged rollout limiting the blast radius of what reaches production, and on-call response combined with automated triage closing the loop on what production reveals.

The principle behind every layer is the same: shift left. A crash caught earlier is cheaper, smaller in impact, and less disruptive to fix. As the volume of code shipped per release continues to grow, the layered defense matters more, not less: the cost of a crash that reaches production is unchanged, but the number of opportunities to introduce one is rising. AI is speeding up how fast code is written; the layered defense is what makes that speed safe. Stability is not about having zero open problems; it is about knowing which problems you own and which ones you are watching.

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility, and digital financial services sectors. Serving over 900 cities in eight Southeast Asian countries: Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam. Grab enables millions of people every day to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. We operate supermarkets in Malaysia under Jaya Grocer and Everrise, which enables us to bring the convenience of on-demand grocery delivery to more consumers in the country. As part of our financial services offerings, we also provide digital banking services through GXS Bank in Singapore and GXBank in Malaysia. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line. We aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Staying Evergreen: How we automated secure access and repo hygiene in a shared org

Post Syndicated from Greg Mohler original https://github.blog/engineering/platform-security/staying-evergreen-how-we-automated-secure-access-and-repo-hygiene-in-a-shared-org/


Working with outside collaborators can add more work for a repository owner. You need to confirm who still needs access, change permissions as projects evolve, and archive repositories once they’ve served their purpose.

To keep up, repository owners need to track collaborators, deadlines, and follow-ups for each repository. As a project grows, that manual housekeeping becomes harder to keep up with and easier to defer.

We faced this problem in a shared organization where GitHub employees (“Hubbers”) collaborate with customers and partners on product trials and professional services engagements. Our answer was Evergreen, a GitHub App built with native GitHub features. The workflows below exhibit a pattern you can adapt for your own organization.

Meet Evergreen

Evergreen is a GitHub App that automates organization hygiene through two workflows:

  1. Outside collaborator access reviews. Every quarter, each repo’s outside collaborators are reviewed. Keep who you need, and everyone else is removed.
  2. Stale repository detection and archival. Repos with no human activity past a threshold get flagged, warned, and (if still untouched) archived.

Two design principles shaped everything:

Fail secure. If nobody responds to an access review, we don’t leave access in place and hope for the best. We revoke it. The safe outcome is the default outcome.

Never destroy anything. Evergreen will never delete a repository or its contents. Archived repos are recoverable, and every access change is written into the repo’s tracking issue, so reverting a mistake is as easy as a copy-paste to Copilot.

How it works: One brain, many hands

Under the hood, Evergreen is a small Go webhook server running in a container. But the interesting part is the choreography. Every moving piece is a stock GitHub feature: a GitHub App, scheduled actions, webhooks, issues, custom properties, and plain files in a repo. Nothing exotic to reinvent.

A controller that keeps time

A dedicated controller repo acts as Evergreen’s clock and its memory. Four scheduled GitHub Actions workflows fire repository_dispatch events on a quarterly cadence, and the app wakes up and does the work:

Day Workflow What happens
1st Stale repo review Flag repos with no recent human activity, open a warning issue
8th Access review Snapshot outside collaborators, open a checklist issue per repo
16th Stale repo cleanup Archive repos still stale after the grace period
23rd Access cleanup Auto‑revoke access for any review left unanswered

Reviews open two weeks before their cleanup runs, and there’s a hard grace period baked in so a stray manual run can never fast‑forward the guillotine. Each workflow also supports workflow_dispatch, so we can trigger any step by hand as needed.

State you can git blame

Evergreen tracks every repo’s status as a JSON file (one per repository) committed to the controller repo under state/repos/<repo_id>.json. The snapshot of who had access, when the review opened, the deadline, who decided, and what happened all lives there. Here’s a completed review where a Hubber kept Alice’s access and let Bob’s lapse:

{
  "repo_id": 123456789,
  "full_name": "the-org/acme-trial",
  "last_seen_at": "2026-04-13T00:05:00Z",
  "access_review": {
    "deadline": "2026-04-27T00:00:00Z",
    "status": "renewed",
    "collaborators_at_open": [
      { "login": "alice", "name": "Alice Example", "permission": "admin" },
      { "login": "bob", "name": "Bob Example", "permission": "write" }
    ],
    "renewed_logins": ["alice"],
    "decided_by": "carol-hubber",
    "decided_at": "2026-04-15T16:42:10Z",
    "tracking_issue": { "repo_id": 123456789, "number": 42 }
  },
  "staleness": {
    "last_activity_at": "2026-04-15T16:42:10Z",
    "warned_at": null,
    "archive_deadline": null,
    "status": "active",
    "tracking_issue": null
  }
}

The best part: because state is just files in git, the commit history is the audit trail. Want to know why someone lost access last spring? git log the state file. There’s no separate database or bespoke audit pipeline to track.

Issue‑ops, everywhere it matters

Evergreen does its talking through issues inside each repository. When an access review kicks off, every repo gets a checklist issue:

Screenshot of an access review issue titled “[ACTION REQUIRED] Access Review – 2026-04-13.” Outside collaborators Alice Example, with admin access, and Bob Example, with write access, have unchecked boxes. Instructions say to select anyone who should keep access, then close the issue. A warning states that if no action is taken by April 27, all listed access will be automatically revoked.

The interaction model is delightfully simple: check the boxes for who stays, then close the issue. That’s it. When the issue closes, Evergreen listens for the issues.closed webhook event, reads the checkboxes, retains the checked collaborators, and revokes the rest, then posts a receipt comment listing exactly what it kept and what it removed.

Quarterly outside-collaborator access review flowchart. A dispatch snapshots outside collaborators and opens a checklist issue in each repository. If a team member closes the issue, checked collaborators retain access and unchecked collaborators have access revoked. If no one responds by the deadline, everyone’s access is automatically revoked. Both paths log the decision in the issue and state.

Only the right humans can pull the lever

Closing an access review issue is a privileged action, so Evergreen guards it. If someone who isn’t a Hubber closes an issue, the app quietly reopens it and posts a note explaining why. Decisions only count when they come from the people authorized to make them.

Informing the right people

Here’s a subtle problem: we know the people who need to respond to a review are Hubbers, but we didn’t always know which Hubber(s) owned a given repository.

Evergreen resolves the right people to notify through a fallback chain:

  1. If a repo sets an evergreen_repo_owner custom property (a @user or @org/team), use it.
  2. Otherwise, look at recent issue and commit authors and cc the ones who are on the access review team, i.e., the Hubbers actually working in that repo.

The result: the right owners get pinged automatically, without anyone maintaining a spreadsheet of “who owns what.”

Fail secure, and fail closed

When a review deadline passes with no response, cleanup revokes every collaborator in the snapshot. We were careful about a subtle edge case: what if some removals succeed while others hit a transient API error? Marking the review “done” while a removal is still outstanding would be a quiet inconsistency, and we didn’t want that.

So Evergreen fails closed. If any removal fails, the review stays pending and the issue stays open, and the next run retries the stragglers. It only records “revoked” once every removal actually succeeded. And whatever gets removed is written into the issue, so restoring access is a matter of re‑adding the handles it listed.

The user experience: automation that feels considerate

Automated housekeeping has a reputation for being the office smoke alarm: loud, easy to ignore, and usually going off while you’re cooking something perfectly fine. We worked hard to make Evergreen feel like the opposite:

  • It comes to you. The work happens as an issue in your repo.
  • It tags the right people. No “who owns this?” archaeology.
  • It’s crystal clear about the ask. Every issue states what to do, the exact deadline, and precisely what happens if you ignore it.
  • It nudges before it acts. A reminder comment lands before the deadline: no surprise revocations.
  • It has an escape hatch. Org owners (Hubbers) can opt out by setting a custom property (evergreen_exempt_from_access_review or evergreen_exempt_from_archive).

Rolling it out carefully

Automation that manages access and archives repositories deserves a careful rollout, especially into a large, active org (ours has 1,400+ Hubbers, and many more customers and partners). We wanted the very first run to be a complete non‑event. A few practices got us there:

Dry‑run first. Evergreen ships with a DRY_RUN mode that logs every write it would make without touching a single collaborator or repo. We ran it against the real org and read the logs like tea leaves until we trusted the decisions it wanted to make.

Overcommunicate. Before flipping the switch, we broadcast what was coming, when, and what people needed to do, so the first issue anyone saw wasn’t a surprise. We found that when you’re telling a large group the same thing for the seventh time, it’s often when many are still hearing it for the first time.

Watch everything with Splunk. Evergreen emits structured logfmt logs to a dedicated Splunk index. That observability is how we caught edge cases in the wild (empty collaborator lists, unusual permissions, repos with issues disabled) and tightened the logic before they mattered.

Respect the platform’s speed limits. When you open issues and post comments across every repo in a big org, you slam into GitHub’s secondary rate limits for content creation: 80 requests/minute and 500/hour. While this is a thoughtful preventative control to avoid system abuse, it requires a bit of intention when automating IssueOps at large scale.

So Evergreen paces itself by construction. It processes repos in batches, spaces out each content‑creating call by a couple of seconds, and sleeps between batches to let the rolling window clear, and it only counts the repos that actually create content, so a quiet quarter where nothing’s eligible still runs fast. We opted for proactive avoidance over reactive back‑off.

Lessons we’re taking with us

  • Make the safe path the default path. Fail secure, then make responding easy enough that failing secure rarely triggers.
  • Use the primitives you already have. Issues, webhooks, custom properties, and git history did the work of a bespoke UI, a job queue, and an audit database.
  • Meet people where they work. An issue in your repo beats a dashboard you’ll never open.
  • Logs are a revert button. Writing every change into the issue turned “oops” into “paste these three handles into a Copilot session.”
  • Respect the limits. At org scale, rate limits aren’t an edge case: they’re the design constraint.

Keep your org evergreen

Routine hygiene is the kind of work that’s easy to defer and perfect to automate. The trick isn’t heroic cleanup sprints: it’s making the boring, safe thing happen on a schedule, with a friendly face and an undo button.

That’s Evergreen. And because it’s built entirely from native GitHub features, you can stand up the same pattern yourself.

Ready to make repository hygiene routine? Start with one recurring access or archival task, then use GitHub Apps, actions, issues, and custom properties to automate the problem away.

The post Staying Evergreen: How we automated secure access and repo hygiene in a shared org appeared first on The GitHub Blog.

From 2,500 requests to 24: Rebuilding star history without the stargazers

Post Syndicated from Taylor Cuilty original https://github.blog/engineering/architecture-optimization/from-2500-requests-to-24-rebuilding-star-history-without-the-stargazers/


Maintainers look at star history to answer a practical question: did a launch, release, or event change a project’s trajectory? To reconstruct that curve from the GitHub API, tools had to request pages of stargazer records (each person who currently stars the repository and when they starred it), keep the timestamps, and throw the identities away.

Over time, that design created problems at both ends. Identities were available for bulk collection, so bad actors started scraping public stargazer lists for spam and phishing campaigns. For highly starred repositories, the endpoint could also make the database process many thousands of entries to return a single page.

The fix seemed clear: return counts instead of people. But returning less data didn’t actually make the query any cheaper.

Now maintainers have a direct way to get the curve: we built an endpoint that returns up to 30 weeks of daily star counts in a single request, with no identities attached. Keeping that cheap to serve meant reducing how much data the database processed, not just how much it returned, and the endpoint does that by filtering stars by time before it counts anything.

What maintainers need from star history

We first addressed the immediate risks to developers and to our database by restricting stargazer and watcher lists to repository administrators and collaborators. After the access change, callers without the required permission could receive an empty response or a 403 Forbidden response.

That change also cut off tools that had used the old response to calculate star history, often without displaying or otherwise using the usernames it returned. So we talked with maintainers and tool builders to understand what they still needed. They used star history as an approximate signal of interest: to compare activity before and after a release, or to see whether an announcement reached new developers. What they cared about was the curve.

The old API response had made identity look like part of that contract because every timestamp arrived attached to a user. In practice, tools such as Star History needed counts and dates, not usernames or other user data. Once identity was no longer part of the requirement, we could design an aggregate endpoint for the actual job: retrieving changes in star counts over time at a useful resolution.

When the API change disrupted Star History, users rallied around the project. GitHub’s Community team listened and connected us with product engineering, who dug into our use case and worked alongside us to design a purpose-built endpoint. A difficult situation became a constructive collaboration between GitHub, Star History, and the broader community—ultimately leading to a better solution.

Tianzhou Chen, star-history.com maintainer and Bytebase co-founder

Why the original endpoint was expensive

Once we knew what maintainers needed, the next step was understanding why the original design couldn’t provide it cheaply. The original GET /repos/{owner}/{repo}/stargazers endpoint was built to answer a simple question: who starred this repository? It returned a paginated list of people, and if the request specified a custom media type, each entry also included when the star was created.

The star model was built for the same job. It records the current set of people who have starred a repository, which keeps showing a star count or a list of stargazers simple. It was not designed as a historical event log: when someone removes a star, that record is removed. When tools began using the list as a time series, reconstructing a growth curve meant retrieving individual identities and processing every current star. Because each page held at most 100 people, the work grew with the repository’s popularity: building the full curve for react/react, with about 251,000 stars, took about 2,511 requests, so tools often ran into rate limits or sampled pages instead of reading them all.

Serving a page required the database to query the repository’s current stars. Due to a bug in the pagination implementation, every request also counted all of those stars to work out how many pages existed, even though the caller received only one page of up to 100 people. On a highly starred repository, that meant counting many thousands of entries on every request. In the week before we restricted access, nearly 37,000 of these count queries ran slowly enough to be logged, averaging nearly 3 seconds each, and database safeguards killed more than 1,000 of them, which contributed to request timeouts.

The chart below shows what all that counting cost. After we restricted access, the database served about 60% fewer stargazer requests, and the total time it spent on the count query fell by about 80%.

Approaches we considered

Removing identities handled the privacy requirement. The harder part was keeping each request from scanning the repository’s entire current star set.

Keeping events without identities

Our first spike kept one record per star and returned the starred_at value without the stargazer’s identity for callers who lacked collaborator access. At first, this looked like the quickest path forward, because it directly served the request we had heard from tool builders. It also closed the identity-harvesting path with a relatively small change to the existing serialization and authorization flow.

This approach addressed identity exposure and would have allowed existing tools to keep working. But API consumers would still have to paginate through and process potentially many thousands of individual records to build an aggregate history. It also tied the new use case to an endpoint and pagination model designed for listing people rather than retrieving trends.

Aggregating, then paginating

Aggregation seemed like the obvious next step. Instead of returning one row per star, the query could group stars into daily or weekly buckets, sort those buckets, and paginate the much smaller result.

The problem was where pagination happened. In a straightforward SQL query, the database first finds all stars for the repository, groups them into buckets, and sorts those buckets. Only then does it apply LIMIT, which controls how many buckets are returned, and OFFSET, which determines where the page begins:

all matching stars -> bucket every row by day or week -> sort every bucket -> OFFSET/LIMIT -> response

For a popular repository, an anonymous request could still scan and aggregate many thousands of index entries to return a handful of buckets. Returning fewer buckets would shrink the JSON response, but it would not reduce the expensive grouping and sorting the database had already performed. For react/react, reading the full history that way would have meant examining roughly 6 million rows, because every one of the 24 requests would have reprocessed all of its stars. To reduce the query cost, we had to narrow the rows before aggregation began.

Filtering by time before grouping

The design we shipped paginates star history by calendar week. For each request, it translates page and per_page into an indexed created_at range, filters the stars to those weeks, and then groups the matching records into daily counts. A response includes up to 30 weeks, and increasing page moves backward through the repository’s history.

indexed date range -> group rows in that range -> daily buckets -> response

With the date range applied first, each request processes the weeks being returned instead of every current star in the repository. For tools, that means one request now covers up to 30 weeks of history, no matter how many stars the repository has. For react/react, that’s about 24 requests instead of about 2,511, and the database examines each star about once (roughly 251,000 rows) across the full history. This fits the primary use case because maintainers usually inspect growth across weeks or around a known release, launch, or event. Week-based ranges also align with other repository activity APIs: the response follows the same shape as the weekly commit activity endpoint, GET /repos/{owner}/{repo}/stats/commit_activity.

The time-range design limits each request to a defined period, but it does not limit the number of stars within that range. A week with an unusually large number of stars can therefore still produce a slow or expensive query. The normal request path no longer queries every current star, but exceptionally dense weeks remain a scaling limitation.

Comparing the performance of the old and new API paths also surfaced other places where GitHub was counting stars on demand. We found the same pattern on the repository stargazers page on GitHub.com and updated it to use the repository’s cached star count. Slow queries from that page, which had numbered in the hundreds each week, dropped to zero. We recently shipped a similar fix to other stargazer API endpoints.

The chart below shows slow and killed stargazer count queries each week, by where they came from.

What we learned

  1. A smaller response does not always mean a cheaper query. LIMIT and OFFSET reduced the number of results returned, but they did not reduce the grouping and sorting the database had already done. We had to look at the order in which the work happened.
  2. Start with what people are actually trying to do. Maintainers needed to compare trends. A list of identities was simply how the old endpoint happened to provide the underlying data.
  3. Return only what the use case needs. Anything more is data someone else can misuse. Once we understood the job, we could remove individual identities without taking away the information maintainers relied on.
  4. Be clear about the tradeoffs that remain. The new endpoint is safer and more predictable, but calendar boundaries, removed stars, and unusually dense weeks still affect what it can return. Those details are part of the API contract too.

Restricting the original endpoint addressed the immediate problem. Listening to maintainers gave us something better than what we started with: star history that’s easier to get, cheaper to serve, and safer for everyone who stars a project.

Get repository star history, then share feedback in the GitHub Community discussion.

The post From 2,500 requests to 24: Rebuilding star history without the stargazers appeared first on The GitHub Blog.

Rewinding the exponential: How we keep hardware usage down

Post Syndicated from Cathy Chi original https://blog.cloudflare.com/rewinding-the-exponential/

More than 20% of the web sits behind Cloudflare’s network, and our team, the Data team, owns the data pipeline responsible for plumbing that data (and more) from our global network to applicable central data centers. The Cloudflare network is a globally distributed system supporting tens of thousands of servers. Data that power Cloudflare Logs, SQL API, observability, internal analytics, and more flow through this network and use our data pipeline to serve customers.

In May of this year, we were providing data delivery at roughly two billion events per second. This includes customer traffic as well as data sent from internal teams. Year after year, we tackle the same problem — exponential traffic growth against linear hardware growth. We have to react to this increase in order to keep resource costs under control. Over the past five years, we’ve made various improvements to compensate for the data volume increase in comparison to hardware allocation, equivalent to deferring two and a half years of hardware expansion. This post details those changes: breaking up our monolithic data schema in 2022, adjusting compression in 2024, rearchitecting pipeline replication in spring 2026, decentralizing sampling in summer 2026, and introducing aggregations into the pipeline in fall 2026.

Background

Our pipeline starts with an internal service called Logfwdr, where we collect from highly distributed data sources (think: Workers, FL). We then feed this data to a set of centralized receivers, which eventually push them onto our buffering system, Buftee. Consumers fetch data from Buftee and distribute data to their final destinations, like ClickHouse, which we rely on heavily.

Unpacking the monolith

In 2022, we dismantled our one and only schema. Previously, there was a single stream packed with every product's fields, forcing Cloudflare-internal consumers to ingest everything and demux or discard records via filters. At the time, most of Cloudflare’s data products were built on HTTP, Security and CDN metadata. In 2021, our message schema looked like this:

Some fields here belonged under the same product, but some were completely irrelevant to each other. The hypothesis was that none of these teams require 100% of fields in the monolithic schema. In other words, we can create small messages for each product’s required fields. Although we were going to duplicate fields, some of them in every message, replacing this firehose with product-specific schemas actually reduced the backbone bandwidth usage by 35-40%, knocking out roughly a year’s worth of compressed traffic growth.

Compression

By 2024, as traffic steadily grew, we were tackling network congestion issues on a daily basis. To react to this, we changed our compression algorithm from LZ4 to zstd. We had already experimented with zstd compression before, but we noticed a higher CPU usage at the time. Therefore, we were careful to track the metrics to ensure the trade-off was worth it.

Initially, we rolled this change out to a few servers. To our relief, we didn’t observe a noticeable difference in CPU usage.

With more confidence, we gradually rolled out to our entire fleet, which also showed little impact to CPUs. In the end, there was nearly a 40% reduction in compressed traffic throughput. The uncompressed bytes remained at the same growth, meaning we could support more than 1.5x the amount of traffic with the same size pipe. Or another way to put it, the estimated growth-equivalent reductions put us at about 10 months prior.

Now, in 2026, we attempted three strategies to reduce uncompressed bytes (and thus, compressed bytes).

Load balancing

To make analytics resilient against regional disasters, our systems require redundant data to be stored in multiple centralized regions (data subject to Customer Metadata Boundaries is stored strictly within its designated region). Historically, the simplest way to achieve disaster resilience was double-writing: Logfwdr transmitted two full copies of every non-regional analytics event — one to each centralized region. Each region then independently processed a complete copy through to ClickHouse. We also apply sampling in each region, an incredibly scalable technique that works for all volumes of data during any traffic conditions. We downsample high-volume traffic while rare events are preserved.

This infrastructure was manageable when only a handful of streams used secondary regions. However, as we expanded disaster resilience across almost all analytics streams, Logfwdr traffic doubled across the board. Sending every event, twice, over wide-area transit was unsustainable against our exponential traffic growth.

We had already tackled a similar challenge for Logpush. For Logpush, Logfwdr treats the centralized data centers as a unified, shared pool of receivers: each message is sent to only one region, failing over to another destination if a machine or facility is unreachable. That model had already proven its worth during large incidents, shifting traffic seamlessly from one location to another without human intervention. This automatic failover was possible because Logpush delivery is stateless, as customer destinations do not care which centralized data center pushes the logs.

Analytics was trickier. Unlike stateless logs, ClickHouse clusters in both regions still need a full copy of non-regional data to allow instant, independent query failover. The breakthrough was decoupling pipeline transit from database redundancy:

1. Shared Ingest Pool: We adopted Logpush’s model to the analytics streams. Logfwdr dispatches each event exactly once across wide-area transit to whichever centralized receiver is available.

2. Late Replication from centralized regions: Instead of replicating traffic at source, replication happens as late as possible in the pipeline. Data lands in our distributed buffering layer, Buftee, and remote inserters stream cross-data-center copies downstream.

3. Resilient Failure Handling: If a centralized region goes down, Logfwdrs immediately routes all non-regional traffic to the surviving region. If the inter-data center connection blips, each cluster continues ingesting its local share without backpressure; remote inserters catch up from Buftee's retention buffer once the backbone recovers.

For the datasets we migrated, the cutover eliminated approximately 43 GB/s of compressed transit traffic while adding only ~7 GB/s of cross-data-center replication on our pipeline. Two architectural factors explain this asymmetry:

1. Replication happens downstream of sampling: In the old model, Logfwdrs transmitted two copies of raw, unsampled events over wide-area transit. In the new model, Logfwdrs send a single copy to centralized receivers, which immediately apply sampling before buffering into Buftee, carrying only this already-compacted stream.

2. Better compression through uniform partitioning: Data at source is fragmented across thousands of individual servers. After centralization, streams are aggregated and partitioned into uniform batches over wider time windows. This structural uniformity allows compression algorithms to achieve dramatically higher compression ratios than Logfwdr transit batches ever could.

Overall, this was a net reduction of 36 GB/s across wide-area networks. We rolled this out initially to two datasets, HTTP analytics and Ready-Analytics, our in-house one-click analytics product.

We’ve been actively migrating more datasets to this load-balanced architecture since the spring.

Sampling from the get-go

Another strategy we employed was further expanding our sampling mechanism so that we pulled critical processing earlier into the pipeline. Sampling was already used in our centralized services, particularly the receivers, easing pressure on downstream systems. We can be more resource efficient if we send less data downstream (without sacrificing data quality).

The solution here was to simply sample in Logfwdr, in addition to maintaining the sampling in centralized receivers. How the sampling algorithm works here, in short, is each dataset has a key that producer teams have chosen to index and sample the data. When sampling, we first shuffle the records for each key to eliminate bias. Then, for each group of records, we select a sample size equal to the square root of the total count of the records. The result is a representative sample, whether sampled once or twice. To produce estimates, we scale calculations against the sample rate. For example, if we sample 10 out of 100 records, the sample rate would be 10/100, and the unbiased estimate of the original count would be 10*10 = 100.

To ensure this approach was working well, we compared the number of rows written to ClickHouse against the estimated sum for total count using a variety of keys across a variety of traffic footprints, so that “least seen” keys to “most seen” keys were captured across the spectrum.

The result is that for keys with more data, the sample rate decreases. However, keys with less data were sampled comparably. Thus, we still kept a representative sample while delivering and storing fewer events. We’re not compromising the quality of data delivered, even when we deliver less in quantity.

If we look at this more closely, using power laws, we can observe that the double-sampled (red) grows at roughly x0.319 v. sampled-once/control (blue) at x0.485. Another way to picture this is, a key that has 100x more data than another key would have stored roughly 10x the number of rows if single-sampled. Now, it will only store roughly 4x the number of rows. Concretely, a total of 11.6 million events is represented by 18k rows in ClickHouse as opposed to 79k. Double sampling attenuates storage of the “most seen” keys, while still retaining the same detail per event for “least seen” keys.

We initially implemented this strategy against three of our top datasets. Every single one at least halved the amount of compressed data we were originally sending on the wire to centralized receivers. That’s 25GB/s altogether, or over 2PB/day.

Aggregating

Cloudflare offers several analytics datasets built on rollups: aggregates of the raw events computed at ingest time. This makes the opposite tradeoff to sampling: rather than answering any question approximately, we answer a fixed set of questions exactly, provided we choose the dimensions and metrics up front.

For example, in the HTTP dataset, we measure the number of unique occurrences of IP addresses during a given minute rather than the individual IPs themselves. Another example is we sum the total number of GET, POST, PUT, etc. requests for a given domain, for that minute, grouped by the HTTP method. None of these computations actually necessitate waiting until the tail-end of the pipeline to start calculating.

Historically, this aggregation occurred towards the end of the pipeline, inside the inserters which read from Buftee and insert into ClickHouse. Decentralizing this aggregation was an opportunity to spread out load and reduce the amount of data we centralize.

Hence, we built an aggregation functionality that is utilized every step of the way, creating repeated opportunities to combine matching partial aggregates as data converges. Much like sampling, records are grouped by a key, which determines which records can be combined. We aggregate on each instance of Logfwdr. We aggregate again after the partially aggregated events reach the centralized receivers, which push these re-aggregated events onto dedicated partitioned buffers. Because these buffers bring together records from multiple receivers, we aggregate another time after they’ve been read in batches by the ClickHouse inserters. Finally, we aggregate in ClickHouse with aggregating table engines.

The first dataset we rolled this out to was for the HTTP traffic. In just this one dataset alone, we dropped from 9+ GB/s to <2 GB/s in compressed bytes traffic. Even though this is currently the highest cardinality dataset, the aggregations on it still gave us an approximate 80% reduction on the wire.

This aggregation feature is powerful for datasets that can collapse easily, especially for use cases like usage metrics where accuracy on the sums and counts is paramount. It also means we can offer producer teams another option to reduce resource usage besides sampling, which we did. Like Ready-Analytics, we created a generic in-house one-click aggregating analytics product, called Ready-Aggregates. As we continue to iterate on this new feature in the pipeline, we expect to see savings continue to mount.

Impact

The starkest proof of this exponential growth is how rapidly traffic growth consumes headroom.

Take Ready-Analytics for example, where two of the three mechanisms, load balancing and sampling techniques, were applied. In mid-May, load balancing dropped traffic by 50%, plunging from ~26 GB/s to ~13 GB/s. Two months later, traffic had already started surging back up to ~20 GB/s. That’s about half of the reduction, erased by organic traffic growth. We then created even more headroom by reducing this dataset’s throughput rate to the single digits, ~8GB/, when sampling rolled out.

Without both interventions, Ready-Analytics alone would have breached 60 GB/s by September. By processing less data in centralized data centers, we also save a significant amount of CPUs and memory. The following graphs show the impact of load-balancing and sampling in Logfwdr on the CPU and memory usage of just the Ready-Analytics receivers: a ~50% reduction, regaining ~750 CPUs and 3.5 TiB of memory.

Across the entire pipeline altogether, from spring to fall, load balancing (~38 GB/s saved), sampling (~25 GB/s saved across top datasets), and aggregating (~7+ GB/s so far), eliminated ~70 GB/s of compressed traffic from our backbone. These cliff drops provide immediate relief.

If we zoom out such that we look at five years from now, the changes made in this year alone have just as much impact as the short-term benefits. We’ve halved the amount of data we’d be processing in another half a decade’s time. The structural and algorithmic optimizations we have and will continue to make are essential to keep hardware demands sustainable.

If you’re interested in helping us solve these challenges at scale, we’re hiring!

GrabMart data for faster disease outbreak detection

Post Syndicated from Grab Tech original https://engineering.grab.com/grabmart-data-for-faster-disease-outbreak-detection

Abstract

Retail purchases can supplement clinical surveillance because changes in demand for some over-the-counter (OTC) products may appear before confirmed case reports. This article describes an early-stage concept for analysing aggregated GrabMart transaction data to identify unusual changes in product demand. The concept is not a deployed system, and it would require validation by public health experts, privacy review, and regulatory approval before any implementation.

Introduction

Public health agencies use sources such as clinical case reports, laboratory results, and hospital records to monitor disease activity. Retail data may offer an additional signal when demand changes across groups of symptom-related products. It cannot establish a diagnosis or replace clinical evidence.

When using indirect signals, we should assess whether they remain relevant to the outcome we want to measure. This includes considering how changes in user behaviour could affect their interpretation. A retail-based signal can experience similar drift, selection bias, and false alarms. Any useful design must therefore combine statistical validation with careful privacy and governance controls.

Background

Syndromic surveillance tracks patterns that may indicate emerging illness before diagnoses are confirmed. Retail surveillance applies this idea to changes in demand for products associated with symptoms, such as fever remedies or cough medicines.

The proposed method groups eligible products into broad symptom classes and measures changes in aggregate demand over time. An anomaly score can indicate that observed demand differs from an expected seasonal pattern. The score is a statistical signal, not an estimate of how many people have a disease.

The design uses the following terms:


Proposed solution

The proposed pipeline would process only transactions that meet documented eligibility rules. It would map eligible SKUs to symptom classes, aggregate counts across approved time periods and geographic areas, and compare each series with an appropriate baseline. Public health experts would review the resulting Early Outbreak Detection Index alongside independent evidence.

The method deliberately excludes identity linkage and individual-level health inference. It also avoids using a single purchase to classify a customer, target an advertisement, adjust an insurance decision, or change access to a product.

Conceptual data flow

Eligible transactions
        |
        v
Reviewed SKU-to-symptom mapping
        |
        v
Aggregate by approved time period and geographic area
        |
        v
Compare with seasonal and historical baselines
        |
        v
Privacy and quality checks
        |
        v
Aggregate signal for authorised public health evaluation

Analysis and validation

Several statistical methods could model expected demand, including seasonal time-series models and spatial scan statistics. Model selection should follow retrospective evaluation against approved public health data. Evaluation should measure sensitivity, false-alert rate, calibration, geographic stability, and performance during seasonal changes.

The system would also need ongoing monitoring. Product catalogues change, purchasing behaviour shifts, promotions affect demand, and access to GrabMart varies across populations and locations. These factors can produce anomalies unrelated to disease activity.

Privacy and governance

Privacy protection starts with data minimisation. The analysis should use only eligible product categories, coarse geographic areas, and aggregate time windows approved for the stated public health purpose. It should not expose raw transactions or personally identifiable information (PII).

The release process could apply minimum-count thresholds and differential privacy (DP), subject to a documented privacy analysis. Privacy specialists would need to set and review any privacy budget. Access controls, retention limits, audit logs, and contractual restrictions would govern any approved data exchange.

An independent review should cover privacy law, health-related inference, fairness, scientific validity, and the risk of harm from false alerts. An authorised public health body, rather than an automated model, should decide whether and how to communicate a possible outbreak.

Potential applications

If research validates the signal and the required approvals are in place, aggregate trends could support:

  • public health research alongside established surveillance sources;
  • operational planning for broad categories of health supplies; and
  • evaluation of whether retail demand contributes useful evidence during an emerging event.

These applications should use aggregate trends only. The signal should not drive personalised medical advice, individual targeting, insurance decisions, or automatic changes to medicine search rankings.

Limitations

Retail demand is an indirect and incomplete measure of illness. Purchases may reflect promotions, weather, media coverage, household stocking, or changes in product availability. GrabMart customers are not representative of the full population, and the available product mix differs by market.

Mapping a product to a symptom class is also uncertain because many products have several uses. Aggregation and privacy controls reduce disclosure risk but can weaken statistical utility. These limitations require local validation and documented thresholds before anyone interprets the index.

Conclusion

Aggregated retail transactions may add a timely signal to public health research, but they cannot replace clinical surveillance. A responsible design must minimise data, prohibit individual health inference, test the model against independent evidence, and place public health experts in control of interpretation and communication.

This concept remains in the early planning phase. No production deployment, measured performance result, or public health integration is claimed.

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility, and digital financial services sectors. Serving over 900 cities in eight Southeast Asian countries: Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam. Grab enables millions of people every day to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. We operate supermarkets in Malaysia under Jaya Grocer and Everrise, which enables us to bring the convenience of on-demand grocery delivery to more consumers in the country. As part of our financial services offerings, we also provide digital banking services through GXS Bank in Singapore and GXBank in Malaysia. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line. We aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Building an evidence-grounded agentic security operations harness on Cloudflare

Post Syndicated from Deanna Tran original https://blog.cloudflare.com/agentic-security-operations/

Security alerts rarely arrive one at a time. A single alert can cause a spike across the environment, requiring a human analyst to decide which alerts are related and what they mean. When multiple arrive at the same time, it can quickly overwhelm even a seasoned security analyst. Enter the alert paradox. Now, our built-in, multi-AI-agent security operations harness can handle more of this work at Cloudflare scale.

Our Cloudflare Managed Defense AI agent harness speeds up the process of gathering data, connecting and aggregating detections, and accounting for missing sources while new alerts continue to arrive. To further analyze context, we make use of the OpenAI Daybreak Defense Network and our partnership with Anthropic. Cloudflare uses approved OpenAI Daybreak and Anthropic models, including GPT-5.6 Cyber and Mythos, for deeper model-backed analysis. Initial analysis and scoring is done with Clef, Cloudflare’s open-source decision model.

Collecting evidence to understand what each alert means requires a lot of time. Even in a highly sophisticated Security Information and Event Management (SIEM), too much is still left for a human to review. Human analysts must gather data, connect and aggregate detections, and account for missing sources while new alerts continue to arrive.

Think about every time a human analyst reviews an alert: "Which should we silence? Which action should we take? Which alert should we ignore? Which should we resolve as false positives? Which should we resolve as true positives? Which should trigger our incident team?" We address this predicament with our AI agent strategy. Our approach reduces all of those questions and gives Managed Defense Analysts a quick and consolidated view, directly providing insight into related alerts, admitted evidence, visible gaps, and recommended next steps. The result: cutting back on the time needed to analyze, and creating a hyper focus on actually getting security alerts resolved and mitigations deployed.

Why a single agent fails

Our first prototype showed the limits of one general-purpose agent. We provided the AI agent the whole investigation. It produced useful analysis, but it also hallucinated claims the evidence did not support. Telemetry, detector descriptions, policies, and threat intelligence were flattened into one prompt, which caused their distinct roles to merge together.

We saw three recurring problems with our first single-shot AI agent harness:

  • Context became authority. A detection is a hypothesis, not proof that an exploit succeeded or an attack occurred. A broad AI agent can blur that distinction.
  • Scope drifted. An AI agent can query the wrong account, time range, or source. You can’t rely on a language model prompt to be a boundary.
  • Failure disappeared. If a lookup times out, the result may not distinguish "not checked" from "checked and not found."

To address these challenges we moved evidence collection and scope enforcement into application code, before model analysis begins.

Recon first, inference second

It's tempting to put an AI agent at every step. The front half of our harness has none.

Before we even call inference, deterministic code runs a fixed set of reconnaissance workflows with versioned API calls. It collects the customer's identity, detection history, traffic baseline, enforcement outcome, and network observations. Each piece of data is stored with its source, version, and timestamp.

Cloudflare sees both the request and the action applied to it. That lets the investigation connect the behavior that triggered an alert with both the control that fired and its outcome.

The fixed recon snapshot also makes evaluation reproducible. If AI agents fetch their own data, two runs may disagree because their inputs changed. Here, the same snapshot can be replayed, so differences between specialist AI agents’ findings come from interpretation rather than retrieval.

Filter noise early

Most alerts are not incidents. The same rule often fires repeatedly on a known traffic pattern, and paging Managed Defense Analysts every time makes it easier to miss a real security incident.

We needed a lightweight triage model to compare each alert with its reconnaissance data: Has this event been detected for the customer before? What did Managed Defense Analysts decide previously? Does the traffic look consistent with normal human behavior? Alerts scored with a high likelihood to be false positives skip analysis by the specialist AI agents.

Clef, running on Workers AI, was the perfect fit for this type of fast agentic reasoning. 

Known high-volume noise is deterministically classified as passive when it arrives. It remains available as context but does not enter the active queue.

Specialist AI agents handle the investigation

For alerts that need deeper review, a coordinator AI agent runs four specialist AI agents in parallel:

  • Traffic analysis reviews request behavior, historical changes, and enforcement.
  • Customer context reviews earlier alerts, dispositions, and Managed Defense Analysts’ decisions.
  • Global telemetry compares the activity with privacy-preserving Internet-wide signals.
  • Threat intelligence checks indicators already admitted to the alert or case.

A synthesis AI agent combines their typed findings into one advisory; it can’t fetch new evidence or choose a classification outside the approved vocabulary. Keeping each task narrow makes unsupported claims easier to catch, and recommendations easier to audit.

Global context without customer data

A security tool knows what happened inside the environment it is deployed in, but little about the world beyond it. Cloudflare compares an alert with patterns seen across its global network.

For example, an IP may be targeting one site, scanning thousands of sites, or appearing for the first time. Those patterns carry different weights. To preserve customer privacy, the global telemetry specialist works only with aggregates; it never receives another customer's individual records or identity.

This view combines features from Cloudflare's CDN, WAF, DDoS, Turnstile, Rate Limiting, and Cloudforce One threat intelligence. The synthesis AI agent weighs both global reputation and customer history. This ensures a globally common pattern can still be used on a per-customer basis, but does not automatically imply a widespread campaign for all customers.

History is evidence

Every evaluation is conscious of what came before it: the alert, the pattern, and which customer. The recon dossier records the alert's own track record including how many times this service alert has fired, how many of those were dispositioned as false positives, and what the Managed Defense Analyst concluded. An attack pattern that has been benign every time your Managed Defense Analysts have seen it is a very different object from the first sighting of something new, and the specialist AI agents are told which one they are looking at. Approved background context is retrieved from previous alerts and cases, so yesterday's conclusions are carried into today's decision, instead of being rebuilt from scratch.

Our system aggregates related alerts into a consolidated case. In each case, we store evidence, findings, and recommendations. Our system deterministically joins and correlates this data, but we leave it to a Managed Defense Analyst to confirm the actual scope. Over time, a case can connect network, application, and Zero Trust evidence together, while tracking the source of the evidence for additional context and reference purposes.

From evidence to decision

Before analysis, the system creates a versioned evidence package with the subject, scope, time anchor, admitted evidence, policy versions, sources, and coverage gaps. Specialists must cite items in that package. Application code checks that every citation exists, belongs to the investigation, and supports the attached claim. Invalid findings are corrected or recorded as limitations.

We lean on Clef a second time to score our evidence. Is the collected evidence enough for a decision? Does any of our collected evidence contradict? Based on this evidence, Clef picks from a deterministically reduced list of attack classifications and dispositions.

Cloudflare's developer platform runs this process. Application code on Workers admits evidence and validates results; Workflows coordinates each stage and saves completed work before the next begins, so a failed stage reuses evidence and findings that already passed validation instead of starting over. D1 keeps investigation and advisory state, R2 holds bounded context and evidence artifacts. Case-chat state persists in Durable Objects, uses Flue, and is enriched with AI Search.

Finally, an LLM-powered agent produces an advisory report using terms that Managed Defense Analysts already work with: affected surface, enforcement outcome, relevant controls, and next step. Managed Defense Analysts can inspect evidence, investigate further, revise the recommendation, or group alerts into a case. Application code fixes the customer scope before any model sees results, and gives each specialist only the evidence it needs. The model never receives authority to cross tenant boundaries or act for the Managed Defense Analysts.

Handling incomplete evidence

At network scale, a source will sometimes fail. A comparison may time out, metadata may be missing, or a threat intelligence lookup may return no match. The system keeps evidence already collected and records the gap.

The advisory distinguishes three states:

  • Not checked
  • Checked, with no matching result
  • Checked, with evidence supporting absence

If global telemetry is unavailable, the system can describe what is unusual for the customer but cannot say whether the pattern is widespread. When the evidence is insufficient, it makes no classification or disposition recommendation.

Remediation

A useful recommendation should lead to a solution, rather than a ticket. The advisory might suggest a rate limiting rule for an abusive path, a WAF custom rule for a signature, or a DDoS protection change. For fully managed customers, Managed Defense Analysts can apply the suggested rules; other customers will receive their recommendations in the dashboard and through their chosen alert path.

In the end, the Managed Defense Analyst remains responsible for the decision and any mitigation. Each alert and case includes the evidence behind the AI agent’s recommendation, which empowers the Managed Defense Analyst to reach a conclusion by accepting or updating the AI agent’s advice.

What comes next

Managed Defense Analysts remain responsible for judgment. The AI agent harness handles more of the repetitive work: assembling investigations, connecting related events, and showing the evidence behind each recommendation. Over the next few quarters, we plan to add a Custom Managed level with more flexibility for each organization.

We also plan to explore continuous AI agents that monitor Cloudflare traffic and surface patterns that fixed rules and thresholds may miss.

The early beta is available in Cloudflare Managed Defense for eligible application-security alerts and cases. If you already use Cloudflare WAF, DDoS protection, Magic Transit, or another supported product, talk to your enterprise account team about adding Managed Defense.

Building Git infrastructure for agent-scale development

Post Syndicated from Brian Celenza original https://github.blog/engineering/architecture-optimization/building-git-infrastructure-for-agent-scale-development/


Every day on GitHub, millions of developers build the products their customers rely on, contribute to open source, and pursue personal projects. GitHub’s architecture has changed steadily over the years to support that work and the growing demands of the developers and organizations who depend on it.

Agentic software development is driving the next architectural shift. With developers and agents working concurrently in repositories that receive millions of commits a day, these workloads demand a different Git architecture. We’re rebuilding GitHub’s Git infrastructure to support them. This post explores the demands shaping that work and the design principles behind it.

Today’s highest-volume workloads show the scale we’re building for. The gap between a typical repository and the busiest ones is wider than most people expect. Here’s a rough picture of the monthly repository activity distribution on GitHub from August 2026:

Repository activity climbs sharply at the far end of the distribution. The busiest repository on GitHub saw roughly a billion requests in August.

Beyond these highest-volume workloads, total Git activity on GitHub is also growing rapidly: between September 2025 and August 2026, it increased to more than 2x its previous level, from 218.2 billion events per month to 473.3 billion.

In September alone, developers and agents made 7.38 billion commits on GitHub, more than five times as many as a year earlier.

The repositories at the top of this curve show what agentic development looks like at its leading edge: large engineering teams running busy CI pipelines alongside growing fleets of agents. Supporting these teams means building Git infrastructure for sustained, concurrent reads and writes at a scale few repositories reach today. We’re investing deeply in Git infrastructure to meet the demands of agentic software development and give teams a foundation built for their most ambitious workloads.

Building for this scale means addressing several architectural challenges:

  • Commit turnaround becomes a bottleneck per agent. An agent in a tight loop commits or checkpoints after nearly every action. Its speed is bounded by how fast a single push completes, so latency that a human would never notice becomes the limiting factor.
  • Write throughput demand is increasing by orders of magnitude. Pushes grew 4.9x year over year, from 0.69 billion to 3.35 billion per month. Thousands of agents working on their own branches in one repository produce a sustained write rate that converges on a single point in our architecture.
  • Merges contend on one reference. Trunk-based development, release trains, and merge queues funnel all that work onto a single ref that has to absorb every merge. Pull request merges on GitHub grew to nearly 4x their volume a year ago.
  • Each push multiplies into thousands of reads. For example, CI and code scanning clone or fetch the same branch tip thousands of times per minute, and that fan-out has to be cheap. GitHub Actions alone ran 3.26 billion times in September, more than 4x as many as a year ago.
  • Operations on a repository must continue to be fast. To keep them fast, we continually compact repository data and clean up objects that are no longer needed. Every new write adds to that work, and the cost compounds as volume climbs.

This is why fast clones only solve part of the problem. Reads are relatively easy to scale: add caches, add replicas, and serve the same bytes to more clients. Scaling reads is essential, but these workloads require more than that. Writes are way harder. Every push has to be stored durably and made visible consistently before the next agent or CI job can build on it.

Where today’s architecture meets new demands

The current architecture has served developers well for years. Every repository is stored by Spokes, which keeps a full copy on the local disks of several fileservers, five by default. Those fast local disks let Git operations read native repository data with low latency, and the extra copies provide redundancy while spreading reads across fileservers. When a push updates a reference, a three-phase commit protocol uses a quorum to ensure that CI, the web UI, and API clients see a consistent repository state. That pairing serves a billion repositories today.

However, the mechanism we use for durability is the same one we use for scale. The copies on disk are the source of truth, so adding read capacity means adding another durable replica. Every replica participates in every write, so a push is only as fast as the slowest replica in its set. The net effect: adding replicas to absorb read load makes writes slower.

For most repositories, this tradeoff works well. At the highest activity levels, it becomes a ceiling: adding read replicas adds overhead to writes, losing a replica reduces read capacity, and losing quorum stops writes entirely. To meet agent-first demands, we need to separate durability from scale without losing what teams rely on today.

Built for the busiest, better for everyone

We’re rebuilding the infrastructure while GitHub keeps running. There’s no maintenance window where the world’s code stops moving, and no version of this work where we ask people to change how they build software while we do it.

We’re building for the most demanding workloads on GitHub: an enterprise shipping under strict regulatory requirements, a team landing a change across a repo that builds an operating system, and an organization running thousands of agents against a single codebase. Engineering for that scale raises the floor for everyone. The maintainer reviewing contributions from volunteers across time zones and the student opening their first pull request get the same faster, more resilient foundation.

The new architecture also must preserve the controls teams already operate on. A maintainer needs branch protections and required reviews so an unreviewed change never reaches the default branch. A security team needs audit logs and repository visibility to investigate a suspicious access event. An on-call engineer needs dependable automation and enough observability to understand why a deployment failed.

For the platform to keep serving everyone here while it scales for the busiest workloads, these are our guiding principles:

  • Build on the workflows developers already trust. Teams rely on workflows like branching, review, merge, and history to build, ship, and govern software at scale. Our new infrastructure is designed to support those same workflows at much higher volumes of activity.
  • Put reliability first. We measure every decision against the reliability that developers and organizations require. Confidence in the platform is what lets an engineering organization build automation, ship on a schedule, meet compliance obligations, and understand the software it produces. This work will meaningfully improve throughput and scale, and those gains extend a foundation of trust that’s already there.
  • Keep people in control of their code. If the system isn’t helping the people and organizations who use it, and isn’t under their control, it isn’t worth building. As agents take on more of the work, the people who own the code can still review, understand, and approve it.

The approach

We are building a new GitHub architecture that can scale much more effectively. Our approach centers around core distributed systems design tenets, applied to the concurrency and scale of agentic software development. Our goal is to continue the forward momentum for open-source communities and enterprises around the world who have built their projects with Git and GitHub, while preserving and adapting the features and controls around it to meet the new needs of the agentic era.

Minimize coordination

A repository that receives many pushes must accept and publish updates quickly. Coordination is valuable when it protects correctness, but too much of it limits write throughput and can turn a busy repository into a bottleneck. Our current architecture is tightly coupled in places it doesn’t need to be, which limits our ability to scale across reads and writes without tough trade-offs. We’re redesigning the system to preserve the coordination that Git semantics require and let everything else proceed independently.

  • Coordinate only what needs agreement. The part of a push that truly needs agreement is the reference update itself. Storing the underlying objects, validating object connectivity, and secret scanning are much more work, but most of it can happen in parallel to other writes. That shrinks the critical path of a push to the small step that needs coordination, so the rest of the work no longer delays the acknowledgment.
  • Move maintenance off the serving path. Compaction and garbage collection are among the heaviest work a repository does, and today they run on the same hosts that answer live Git requests. In the new architecture, separate workers handle maintenance directly against durable storage. A busy repository can be optimized continuously in the background without slowing pushes and fetches.

Decouple storage from compute

Today, complete repository copies on local disks serve both as durable storage and as the layer that answers Git requests. Separating the two lets us scale each one independently.

  • Scale reads without adding durable copies. In the new architecture, read capacity comes from lightweight workers that cache data to serve requests. The authoritative copy of the repository lives in a durable storage layer underneath. That way, the platform can absorb large read spikes from CI fan-out, agent fleets, and large clones without adding work to every push.
  • Let each layer do one job. Authoritative repository data lives in Azure Blob Storage, which already provides durability and replication at Azure scale. The compute layer is optimized for throughput at the lowest latency.
  • Recover faster from failures. When storage and compute are coupled, losing a host reduces both capacity and durability, and recovery means rebuilding a full repository copy. When they’re separate, losing a compute worker is closer to a cache miss: a replacement worker can start serving requests right away and fill its cache from durable storage as traffic arrives.
  • Match capacity to demand. Compute workers can be added or removed as traffic changes instead of provisioning for peak load in advance. A repository going through a burst of activity, like a release or a new agent fleet coming online, can get extra capacity for the burst. Once it passes, that capacity goes away.

Together, these tenets allow us to support higher throughput and more concurrent work without abandoning the reliability and controls that our users need.

What comes next

We’re building an architecture designed to provide the highest throughput and reliability available. Reads and writes scale independently, and the system recovers gracefully from failures. In internal benchmarks, it has delivered up to 35 times higher write throughput, with read capacity that scales on its own to meet demand.

As automated development increases the frequency and concurrency of software change, GitHub will evolve its foundations without trading away the governance and control that teams rely on. We’re already putting that foundation in place. In the next post in this series, we’ll dive deeper into our future architecture and the journey that led us there.

The post Building Git infrastructure for agent-scale development appeared first on The GitHub Blog.

Smarter personalization: How property data helps us understand user price sensitivity

Post Syndicated from Grab Tech original https://engineering.grab.com/price-sensitivity-profiling-using-llm

Introduction

Existing systems estimate user price sensitivity primarily from spending behavior or demographic proxies. They do not systematically account for residential property values, which can indicate a user’s financial circumstances. This omission creates three limitations:

  1. Limited property insights: Existing profiles do not account for property values.
  2. One-dimensional profiles: Users with similar spending patterns but different living standards receive the same classification.
  3. Regional variation: Manual classification does not adapt well to differences between property markets.

This invention adds property data to user profiles to produce affluence segments that account for regional property markets. The invention remains unimplemented and is not deployed in any live market.

Background

Property values vary across buildings, neighborhoods, and regions. A useful classification method must account for these differences while processing property records in several formats. The same classification method could support other industries that use affluence segments.

Solution

A four-step method that combines public property data with clustering techniques to classify user affluence was created:

  1. Data collection and enrichment.
  2. Building-level property price estimation.
  3. Property tier classification.
  4. User mapping to affluence tiers.

The method combines external property data, large language model (LLM) enrichment, and cluster-based segmentation. The resulting profiles can inform personalized offers, targeted marketing, and operational planning.

Data gathering and processing

The system collects public property data from external sources. These records include transaction prices, building types, and related attributes in raw text formats. LLMs standardize the records and extract key attributes such as postcode, building address, transaction price, price per square meter, property size, and floor level. This stage produces a structured property-level dataset for analysis and modeling.

Building-level price estimation

Under the proposed method, transaction records for properties within the same building would be aggregated to calculate a weighted average sale price. The weighting would account for transaction recency and data volume, with the aim of reflecting recent market conditions.

When transaction data is limited, the method could apply rules based on comparable properties in the area. It could also evaluate LLM-assisted extraction of contextual signals from appropriately licensed external sources, such as real estate listings. Any LLM-derived signals would require validation against independent reference data before contributing to a property-value estimate.

The invention has not been implemented, deployed, or tested in a live market. Its model and version, validation methodology, error bounds, and operational safeguards remain subjects for future technical evaluation.

Property tier classification

A hierarchical clustering model groups buildings into affluence tiers based on their estimated resale values. The model accounts for regional differences and local market-value distributions across property types and locations. This stage assigns each building to an affluence tier.

User mapping to property tiers

The system maps would use appropriately aggregated geographic signals to associate users with broad property-market segments and assign a corresponding affluence classification.

Potential future applications

This invention presents a conceptual approach for exploring new ways to support personalization, advertising, and operational planning. It has not been implemented, deployed, or tested in a live market. Subject to technical validation and approval from Privacy, Legal, PR, and the relevant product owners, potential applications could include:

  • Personalization: Tailoring promotions, discounts, subscription plans, and recommendations for optional products and services.
  • Advertising: Supporting more relevant connections between advertising content, merchants, and broad audience segments, alongside tailored marketing campaigns.
  • Operational planning: Informing aggregated analysis used to plan and prioritize delivery and transport capacity.

Learnings and conclusion

This invention presents a conceptual approach that combines publicly available property data, LLM-assisted data structuring, building-level value estimation, hierarchical clustering, and contextual tier assignment. It demonstrates how regional property-market context could complement existing signals, while highlighting dependencies on data quality, model accuracy, geographic coverage, privacy, and fairness.

The approach shows potential applications in personalization, advertising, and operational planning, but these are illustrative, not demonstrated outcomes.

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility, and digital financial services sectors. Serving over 900 cities in eight Southeast Asian countries: Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam. Grab enables millions of people every day to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. We operate supermarkets in Malaysia under Jaya Grocer and Everrise, which enables us to bring the convenience of on-demand grocery delivery to more consumers in the country. As part of our financial services offerings, we also provide digital banking services through GXS Bank in Singapore and GXBank in Malaysia. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line. We aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Powering AI-led research through simulation

Post Syndicated from Grab Tech original https://engineering.grab.com/powering-ai-led-research-through-simulation

The short story

Consider a Friday evening. A food order arrives from a mall in the city center. One driver is nearby; another is finishing a drop-off and will be available shortly; a second order from the same mall may or may not appear in the next two minutes. Dispatch the nearby driver now, or hold briefly for a batching opportunity? The decision window is short.

A fulfillment marketplace makes these decisions continuously. Each one is small. Across a city, those decisions determine whether your dinner arrives hot and whether a driver’s hour is well spent.

And that is one decision. There are dozens more: how far to look for a driver, when two orders are worth combining, which of three waiting trips gets the one free bike, how long to keep trying before giving up. These decisions interact, and the setting that is right for Friday at seven is wrong for Tuesday at two. Together they define a space of possible strategies far larger than anyone could exhaustively explore.

We sample only about a dozen points in that space each year. Not for want of ideas: implementing each strategy costs weeks of engineering, and a production experiment takes weeks more to judge. Some questions have no production answer at all. Nobody can run last Saturday again with twenty percent fewer drivers. We were never short of compute resources, and never short of data. We were short of attempts.

So we built sim-rs, a simulator that makes each attempt take minutes, and connected agents that run experiments on it.

sim-rs is self-contained: it requires no production services or databases. The dispatch lifecycle runs in one compiled program alongside a separate dispatch service that is spawned locally and operates offline. Historical booking and driver data go in as plain files, together with a configuration describing the strategy to test. Simulated bookings, trips, drivers, and a single metrics report come out. Processing one city-day of marketplace activity takes tens of minutes, from raw input to finished report. One command in, one report out: a workflow as practical for a software agent as for an engineer.

That compact contract is what makes the simulator AI-friendly. Agents can change bounded components, run reproducible experiments, receive verdicts they cannot alter, and help keep both production logic and behavioral models current.

Building blocks

The essential components

To model a marketplace, the simulator needs four core concepts:

  • Bookings: requests to move something (a passenger, a meal, a parcel), represented in one format regardless of the business vertical.
  • Drivers: simulated workers with a location, a shift, a vehicle, and a queue of work.
  • Trips: units assigned to drivers, containing one booking or several batched into a multi-stop route.
  • Ticks: simulated time, advancing in fixed steps. Nobody waits in real time.

At each tick, the simulator runs the following sequence:

  1. Demand arrives. Historical (or synthesized) bookings whose time has come enter the pool.
  2. Supply moves. Drivers come on shift, advance along routes, go idle, and reposition.
  3. Pre-dispatch cancellations are applied. Cancellation models, including survival-analysis models trained on real behavior, determine which bookings leave the pool.
  4. Dispatch happens. Bookings are batched into trips, trips are matched to drivers by an optimization solver, and a recycling step decides, for each trip, whether to send it now, hold it, or split it back into bookings and try again later.
  5. Post-dispatch cancellations are applied. These include passenger and driver cancellations.
  6. Outcomes are recorded. Every booking, trip, and driver outcome is written to the run outputs.

Crucially, this is a closed loop: today’s dispatch decisions change where drivers end up, which changes what’s possible next tick. That feedback produces second-order effects that a static replay, one that scores historical decisions without updating future supply, cannot show. Supply may dry up in a hot zone, or one bad dispatch rule may cascade into a wave of cancellations.

Marketplace simulation tick loop from demand through dispatch to recorded outcomes
Figure 1. Simulator sequence.

That loop is only the mechanism. Making it a lab takes three further properties: configurability; reproducibility and auditability; and reliable comparison.

1. Interchangeable stages. The major dispatch stages are pluggable. Batching, allocation, cancellation, routing, driver movement, and recycling are each exposed through an interface with interchangeable implementations selected by configuration. This turns a fixed pipeline into an experimental platform: the component under study can be swapped while everything around it stays unchanged. The same mechanism determines which marketplace is being simulated. Ride-hailing, food delivery, parcel delivery, or all three sharing one driver pool are configuration choices within the same codebase, not separate simulators.

2. Self-contained runs. Each experiment is a small, portable package containing its configuration files, input data, source revision, metrics report, and detailed outputs. Anyone holding that package can recreate the setup, whether a reviewer, teammate, agent, or the original author six months later. The report identifies the configured components and headline metrics, while the supplementary per-booking, per-trip, and per-driver outputs let reviewers recompute those metrics and investigate unexpected outcomes without relying on separate notes or systems.

3. Experiments inside the simulation. Production marketplaces have experimentation platforms, so our simulator ships with one too. Experimentable fields can define multiple treatment arms assigned through the same time-based switchbacks, spatial splits, or cell-and-hour schemes used in production. Each booking records its resolved arm for traceability. Running these A/B/n comparisons together exposes every arm to the same demand, drivers, marketplace dynamics, noise, and biases. By reproducing both the conditions and assignment design of a marketplace experiment, simulated insights are more likely to translate into effects observed in production.

Automating the research cycle

Agents do two jobs here, and both need the same environment: searching for strategies that beat the incumbent, and keeping the simulator sufficiently faithful for that search to mean something.

Searching for better strategies

We connect agents to that interface through autoresearch, an automated loop that runs the research cycle end to end. An agent proposes a change, implements it in source code, tests it, and acts on the verdict. A candidate survives only if it improves the target metric and passes every required check; otherwise it is rolled back and the loop continues. The objective may be a marketplace outcome such as orders-throughput or a software measure such as execution time. What matters is a repeatable command-and-metric contract that both humans and agents can use.

Automated research loop connecting an agent to the simulator and evaluation checks

The first failure mode we encountered was specification gaming. Given a target and a loophole, an agent may find the shortest path to the number rather than the improvement we intended. Ours discovered that changing fields used by pre-dispatch cancellation could reduce cancellations and raise completion without improving a single dispatch decision. The metric moved; nothing real had improved. Rather than relying on instructions alone, we built four safeguards into the environment:

  • Core data is immutable. Modules under test may change only the state exposed by their interfaces. An attempt to manipulate protected fields fails to compile instead of producing a misleading result.
  • The verdict is computed, not judged. The simulator computes its own metrics, which the agent can read but not redefine. Executable checks apply calibrated tolerance bands, verify equivalent outputs where required, and run the tests. The agent never judges its own work.
  • Changes are scoped, checked, and reversible. Each experiment runs on its own branch within a declared writable scope, which is checked against the resulting diff. The grading machinery remains outside that scope, regressions cost only a discarded branch, and every surviving change is bounded enough to review in full.
  • Changes are checked at two levels. It first rejects changes that violate type contracts, ownership rules, interface boundaries, or concurrency requirements. The evaluation harness then applies checks suited to the objective: performance work must preserve expected outputs, while strategy experiments must satisfy domain constraints and outcome thresholds.

The speed that matters is the speed of an adaptive research loop, not simulation runtime alone: each iteration uses previous results to propose the next change, then implements, compiles, runs, scores, and decides whether to keep it. The simulator returns a verdict in milliseconds for benchmarks or tens of minutes for a full-day replay, while autoresearch carries each verdict into the next attempt without human handoffs. This turns an overnight run into a connected sequence of evidence-driven experiments, with the safeguards above ensuring that faster iteration compounds reliable evidence rather than mistakes.

One outcome is the familiar purpose of simulation: discovering better marketplace strategies. When asked to explore trip-recycling policy, the loop turned an emerging human intuition into a concrete rule: hold batched orders only as long as service-level deadlines permit, maximizing the chance that another nearby order joins the trip. Because the result is human-readable code, engineers can review, audit, and deploy it like any other pull request.

The same machinery also improves the simulator itself. Over several nights of unattended operation, we aimed the agents at two hot paths in its dispatch logic: the checks that decide which drivers are eligible for a trip, and the pass that adjusts the cost of every driver-and-trip pairing before the solver chooses an assignment. Across roughly 150 logged experiments, three out of four attempts failed to build or were rejected by later gates. The eligibility checks ended up about 10 times faster, and the cost pass about 24 times faster. Because these paths run repeatedly in every replay, improvements compound across subsequent research. Faster and more efficient runs enable testing of more ideas across more markets and dates, repeat runs to separate signal from noise, and validate results more rigorously. Shorter runs also tighten the feedback loop from verdict to next proposal, so improving the instrument accelerates and strengthens every search performed with it.

Execution time is only one possible objective: pointing the same machinery at orders-throughput changes the research question, not the loop itself. Regardless, whatever the objective, the result is only as trustworthy as the simulator behind it. An agent can optimize only the world it is given; if that world has drifted from production, a faster loop will merely produce misleading answers sooner.

Keeping the simulator faithful

A simulator is a claim about the world: that this is how the system behaves and that this is how the people within it respond. Both halves of that claim decay. Production logic changes continuously, while models of passenger and driver behavior grow stale as new product features reshape how people act. A simulator that has drifted from the world produces misleading insights and innovations that fail in production. So the second job we give agents is to keep the simulator faithful.

The system half is a translation problem, and coding agents are particularly good at it. An agent can read a component’s production implementation and implement equivalent logic behind the simulator’s corresponding interface, translating directly from source code rather than from a written description that may already be stale. The result remains a candidate until parity checks show that it matches the production behavior being modeled.

The behavioral half cannot be translated, because no source file tells us how a passenger will behave. It must be inferred from what people actually do. Here, we redirect the same autoresearch loop from dispatch-code optimization to behavioral modeling. It proposes, fits, and scores candidate models, exploring which features predict cancellation and how their effects should be represented. The test is not only how closely a model explains its training data, but also how well it generalizes to held-out data. A model that fits last Tuesday perfectly and next Tuesday poorly does not make the simulator more faithful; it makes the lab more confidently wrong.

The pieces are simple: a realistic marketplace model, swappable parts, repeatable experiments, honest scoring, and an easy way to undo failures. Together, they give agents a safe place to test and improve ideas. Everyone is racing to give AI a bigger brain. We got further by giving it a better lab: a place where it can try a thousand ideas, be wrong cheaply, and receive an honest verdict.

Join us

Grab is Southeast Asia’s leading superapp, serving over 900 cities across eight countries (Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam). Through a single platform, millions of users access mobility, delivery, and digital financial services, including ride-hailing, food delivery, payments, lending, and digital banking via GXS Bank and GXBank. Founded in 2012, Grab’s mission is to drive Southeast Asia forward by creating economic empowerment for everyone while delivering sustainable financial performance and positive social impact.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Supporting native Rust in Workers with the new Emscripten target for wasm-bindgen

Post Syndicated from Guy Bedford original https://blog.cloudflare.com/rust-workers-emscripten-target/

Today we’re announcing the first public experimental preview of a feature to better support native Rust code and even Tokio-based applications just running natively on Workers: first-class support for the Emscripten wasm32-unknown-emscripten Rust compiler target on the wasm-bindgen open source toolchain and Cloudflare’s Rust Workers.

wasm-bindgen is the open source toolchain powering Rust-based WebAssembly applications on our V8-based Workers Runtime. Enabling the Emscripten target for wasm-bindgen has been a long-term effort, first initiated by Google over a year ago, and then further reviewed and supported by the Cloudflare engineers maintaining wasm-bindgen.

While still in pre-release, we’re excited to share the new workflow possibilities this work enables in running native wasm-bindgen Rust applications with Emscripten on the web, Node.js, and on Cloudflare’s global Workers platform.

In testing we’ve been able to see significantly improved library compatibility for Rust Workers. To illustrate the sort of capabilities supported, we were able to get a Rust-native Minecraft server (Pumpkin) running inside of a Durable Object with TCP ingress, using real TCP sockets via Tokio. See the end of this post for a full description of this port.

Emscripten is an open-source WebAssembly compiler toolchain initially created by Mozilla and currently maintained by Google engineers, which makes it possible to run and bridge native code with the web platform, including supporting and virtualizing platform features such as timers, file system operations, sockets, and other native functionality.

Since Cloudflare Workers supports Web Platform APIs and Node.js compatibility, we are able to support Emscripten on Workers using its Node.js compilation flags, fully virtualizing native platform features such as timers, file system operations, and sockets on top of our existing Node.js APIs. And with our work on Tokio support, Rust code building on top of Tokio’s async runtime ecosystem can now also be fully integrated into JavaScript-based host environments with this Emscripten target.

We’ve made these current experimental patchsets available with example applications to try out today, including:

Supporting the wasm32-unknown-emscripten target in wasm-bindgen

Mitch Foley on Google’s Portable Toolchains team first encountered the need for Rust WebAssembly toolchains to interoperate with C++ when another internal team at Google was interested in using wasm-bindgen to interface with their JavaScript. The goal was for wasm-bindgen to drive the build and produce the companion JavaScript, while Emscripten’s linker (wasm-ld) would be able to link in any C++ dependencies needed.

Internally, Google uses Emscripten in C++ codebases to generate JavaScript along with the rest of their applications, allowing the C++ and JS to interact with one another. While Emscripten was capable of linking in Rust code as a dependency, the thing it couldn’t do was provide a robust binding system between Rust and JavaScript like wasm-bindgen does.

The problem was both tools assume they are in charge of loading and interacting with JavaScript and generating the final JS and Wasm output. Because of this conflict, Google internal users would have to commit to using one or the other toolchain and never both, and adding a second set of tooling would have doubled the support surface for Google’s Portable Toolchains team.

Mitch and his colleague Yifan Yang crafted a plan to have them work cooperatively: Emscripten would continue to drive the build, load the Wasm module, and provide the companion JS, while wasm-bindgen would produce a smaller portable version of its JavaScript bindings in a format that could be directly included in Emscripten’s library system.

This wasn’t just a technical problem — both Emscripten and the wasm-bindgen maintainers had to support this plan and be willing to maintain integration tests that depended on the other. With feedback from Google’s Portable Toolchains team, Google’s Wasm Tools team, and Cloudflare engineers, this finally resulted in the required changes landing in both projects.

As a result of these efforts, the wasm-bindgen and Emscripten toolchains now have seamless interoperability under the new -sWASM_BINDGEN configuration:

  1. C++ Emscripten code driven by Emscripten’s compiler can be built against static wasm-bindgen Rust code, fully supporting the wasm-bindgen bindings layer alongside the Emscripten bindings layer.
  2. Rust applications using wasm-bindgen and driven by Rust’s compiler can now be built for the Emscripten target, fully supporting Emscripten’s bindings layer alongside wasm-bindgen’s bindings layer.

See the wasm-bindgen Emscripten documentation page for more information about using this target.

Rust library support for Emscripten

After landing the wasm32-unknown-emscripten target support in wasm-bindgen, early prototypes by Cloudflare engineers demonstrated clear success for supporting this target on Cloudflare Workers. Many libraries worked out of the box, even including low-level systems libraries, since Emscripten already supports the target_family = unix in Rust.

Some low-level systems libraries that were unaware of Emscripten required patching, for example libc, socket2, and Mio. Even for these low-level libraries, patches primarily involved adding the Emscripten target to the existing platform gates, for example to explicitly allow target_os = “emscripten”, in place of Wasm platform gates.

Overall we posted these target support patches fairly infrequently, and they were mostly trivial when needed, with the Rust library maintainers very receptive to reviewing the support.

That said, one of the critical foundational libraries that needed significant support was Tokio.

Tokio support

Cloudflare Workers are single-threaded and hosted within a JS event loop, while Tokio async is designed around blocking operations being supported through threaded parking semantics. The two models are clearly not compatible with each other — a blocking operation such as a pending socket read or epoll wait cannot block the shared JS event loop.

To get around this, we had two options: WebAssembly JavaScript Promise Integration or to modify Tokio to support event loop runtime integration.

To support Tokio building for Emscripten, we have contributed full Tokio support patchsets that are being reviewed upstream, with the first target support patch for wasm32-unknown-emscripten already landed upstream in Tokio. With these patchsets, we’ve been able to fully support both approaches in Cloudflare Workers. In our Rust Workers Tokio examples, these patches are required directly pending further integration upstream.

Supporting JSPI

WebAssembly JavaScript Promise Integration (JSPI) maps directly onto Tokio’s existing parking semantics. This is because JSPI allows a blocking Wasm call to suspend the WebAssembly stack on a synchronous operation and return control to the JS event loop, exactly as one would expect of a park.

But when a new Wasm call is made while an earlier one is suspended, JSPI doesn’t break: it instead supports having a new WebAssembly stack being entered while arbitrary existing stacks are suspended at the same time.

The problem here, though, is that Rust itself isn’t aware that its stack is being switched out from under it. Tokio’s runtime context is tracked via a thread local, but a JSPI stack switch is not a thread switch, so the suspended and new stack still share the same thread-local runtime context. As a result, the Tokio runtime context still thinks it is in the parked context when a new Wasm call enters, and then panics because the runtime is already entered.

Supporting fully reentrant JSPI therefore requires careful thread local handling to split context between JSPI context switches. To support this, the thread-local context itself must be swapped on each JSPI enter, exit, suspend, and resume. In effect this is cooperative time-multiplexed threading, with each suspended stack carrying its own runtime context.

With our pre-release Tokio patchset, we were able to implement and verify this model. Finalizing the design and upstreaming it is ongoing in collaboration with the Tokio and Emscripten teams.

Adding an event loop runtime to Tokio

The other runtime approach is full event loop integration. Event loops are of course a common paradigm in native UI applications, so the idea of supporting an event loop runtime for Tokio was certainly not something completely unfamiliar to the maintainers in discussions we had around the Emscripten target support.

The question was rather how to design an event loop for Tokio in such a way that it could work across native applications (including Windows and macOS), as well as for WebAssembly embeddings in JavaScript hosts.

If we could design a general LocalEventLoop runtime for Tokio, we could solve this problem more generally.

A Tokio runtime does two things in a loop: (1) it polls tasks until nothing is ready, and then (2) it waits. The wait is what makes it a runtime rather than a library: the thread parks inside the I/O driver until a socket becomes readable, a timer expires, or another thread wakes it.

But when the host already has its own event loop, Tokio's loop cannot run inside it without blocking the host's. The way around this is to split Tokio's loop in two with both parts able to integrate with the host: (1) becomes an explicit drive() operation that runs one batch of ready tasks and returns, and (2) is replaced by a wake, so that instead of parking, the runtime tells the host it has work and the host calls drive() when it is ready to.

Our proposed LocalEventLoop design for Tokio is a LocalRuntime whose wait has been replaced by a wake. It is built with a standard std::task::Waker that the host owns, which instead of being used to signal that a future should be polled soon, is used by the Tokio runtime to signal that the runtime itself should be driven soon.

Consider the example of a socket read under a regular Tokio runtime:

This would correspond to the following call diagram:

Instead, with LocalEventLoop we can write:

Here, spawn_local returns immediately, with the host event loop taking responsibility for driving the spawned task to completion via el.drive() calls from the host. When stream data is unavailable, the runtime simply returns, handing back control to the host. Once the socket becomes readable, Tokio uses the host_waker to signal that the runtime needs driving.

This LocalEventLoop flow corresponds to the following call diagram:

Everything that would have unparked a native runtime's thread wakes the host instead: a spawn, a task woken from another thread, a socket becoming readable, a timer expiring. Because a Waker is Send + Sync and carries no execution semantics, its implementation is just a notification to the host's event loop. This makes the contract safe by construction: a wake arriving from another thread, from a host callback, or even during a drive, queues work rather than re-entering the runtime. The drive that follows runs on the owning thread with nothing else on the stack.

The one thing you cannot do is wait. block_on still exists and runs a future as far as ready work carries it, but where a normal runtime would park, LocalEventLoop::block_on panics instead. This is because nothing could wake that future from inside the call since its wait belongs to the host.

With this design, the host event loop is never blocked, and interleaves its own work with Tokio's, one batch at a time. The same architecture embeds into a GTK main loop, a Win32 message pump, or a Cocoa run loop. In addition, any number of these runtimes can co-exist due to their cooperative execution semantics.

Supporting sockets and epoll on Emscripten

With both Tokio runtime integration approaches fleshed out for the Rust Emscripten target, we were then able to support most of the Tokio test suite, with one major subsystem still unsupported — the net feature. This includes its sockets APIs: TCP, UDP, and Unix sockets. The reason for this was that Emscripten only supported poll() and a WebSocket emulation layer, but not epoll_wait(), on which Tokio’s I/O driver is built (via mio).

For Cloudflare Workers, we wanted to be able to fully integrate with our TCP sockets APIs, including upcoming inbound TCP. To do this, we would need to build a bridge between Emscripten’s virtualization layer and our own sockets API layer.

Instead of having to build this bridge ourselves, we realized we already have one: the node:net API we support in our Node.js compatibility layer. Emscripten had an -sNODERAWFS mode to bridge natively into Node.js FS APIs, so the same approach could give us a sockets bridge without Emscripten or Cloudflare Workers needing to implement any custom APIs on either side.

We contributed this work in over 40 pull requests to Emscripten, which is now the –sNODERAWSOCKETS layer compilation option, enabling support for epoll, TCP, UDP, and Unix sockets for Emscripten applications in Node.js — and, because Workers implements the same node:net API, on Workers itself as well.

Under JSPI, Emscripten's epoll_wait() simply suspends the stack until readiness, so Tokio's I/O driver works as on native. For LocalEventLoop, readiness instead had to reach the Waker from a JS callback. To support this, we drafted an Emscripten proposal for a new emscripten_epoll_add_listener API to associate a callback on an epoll’s ready events. The next drive() then collects these events with a zero-timeout epoll_wait(), so Tokio's existing I/O driver can be used unchanged.

Running a Minecraft server on Workers

To test out this new target, the challenge was raised to see if a Minecraft server could be deployed to Workers. Dan Lapid then implemented a Pumpkin Minecraft server running on a Durable Object over a single weekend.

Pumpkin is a Rust Minecraft server built on Tokio and designed for multi-core machines. World generation runs on a dedicated thread pool, while the game tick loop and chunk scheduler each run on their own OS threads.

Inside a Durable Object there is exactly one thread, so getting Pumpkin to run there meant turning those threads into cooperative tasks on the event loop. Leaning on the Tokio integration, the tick loop and chunk scheduler became async tasks and each Rayon job became a Tokio task. World generation then runs on the event loop itself, taking one turn per chunk and interleaving with network I/O and game ticks rather than blocking them.

For persistence, Pumpkin writes its world through ordinary std::fs. Under Emscripten’s -sNODERAWFS option file system calls get forwarded to the node:fs Node.js compatibility layer for Workers. Since this bridge is just JavaScript, it is easy to swap out. Dan’s worker-fs-mount library enables mounting a node:fs-compatible file system with its durable-object-fs backend, which stores files as rows in the Durable Object's SQLite storage. With this integrated, every file Pumpkin saves becomes a row in the object's database, written synchronously and committed with the Durable Object's transaction. Pumpkin itself has no idea it isn't on disk, and a restarted object boots straight from the same world.

Networking needed no changes in Pumpkin either. Each player's connection arrives through Workers TCP ingress at the Worker's connect() handler, which forwards it into the Durable Object. There, handleAsNodeConnection() from cloudflare:node dispatches the socket to a net.Server listening on that port inside the object – a new TCP counterpart of handleAsNodeRequest(). Emscripten’s -sNODERAWSOCKETS backend implements TcpListener on top of net.Server, so the server accepts players exactly as it would on Linux, and the returned promise tells the object when the last player has left, so it can save and shut down.

Running a fully persistent Minecraft server inside a Durable Object with multiplayer support demonstrates the level of native compatibility that is possible with this new Emscripten target. We’re excited to see what native Rust applications you can bring to the platform.

Try it out

We’re making these full patchsets and workflows available today for experimental use. Improving wasm-bindgen’s support for modern WebAssembly standards for all users is part of our ongoing commitment to the Rust and Wasm ecosystem.

We welcome all contributions and feedback — find us on GitHub and in the #rust-on-workers channel on Cloudflare’s Discord.

Improving site performance by shipping more CSS

Post Syndicated from Josh Black original https://github.blog/engineering/architecture-optimization/improving-site-performance-by-shipping-more-css/


The Primer Design System powers many of the experiences you see on GitHub today. From buttons to banners to breadcrumbs, these foundational components are required to be accessible, flexible, and performant across a wide variety of scenarios.

Back in 2023, the number of components on certain pages began to explode. This led to several performance-related challenges with our existing CSS-in-JS solution:

  • Initial page loads took longer due to styles being initialized on the client
  • Server-side rendering performance declined as style collection shifted from the client
  • Updates to styles grew out of control as component count grew on a page

It became clear that the Primer team needed to address the issue at the source. We needed to find an alternative that would completely avoid the client and server costs that we were seeing with our current solution. Most importantly, any alternative we pick would need to work in a way that would avoid any breakage to GitHub during the migration.

Introducing CSS (Modules)

The Primer team found a solution that met all of our criteria: CSS Modules. This format would allow us to do one of our favorite things: write and use native CSS features, while still allowing some amount of the colocation and encapsulation that we had come to expect from CSS-in-JS.

With CSS Modules, styles would be authored in a CSS file alongside the JavaScript source for the component. It would also allow us to treat all class names as local by default, preventing some of the collisions and challenges that can come from global selectors. This format also removes the need for any client or server runtime behavior. Instead, styles would roll up into CSS stylesheets that were sent as part of the HTML for a page.

However, this solution was radically different from the CSS-in-JS solution we had at the time. This change would require an update to every Primer component and every component at GitHub authored using this technique. Thankfully, design systems are a perfect vehicle to deliver this kind of change at scale.

A gradual march towards CSS Modules

The situation for moving towards CSS Modules was clear. The Primer team would need to deliver updates to each of its components, moving them from CSS-in-JS to CSS Modules. At the same time, updates we made to these components could not break any usage in GitHub. Finally, the underlying technique we used for CSS-in-JS also had to continue working for any components in GitHub that were currently using it.

With all these constraints in place, we decided on an incremental migration strategy that would allow us to safely ship component updates without breaking the world. For each component, our plan was to:

  • Add a new file that translated existing styles to CSS Modules
  • Add the component to a feature flag that would toggle between the new and old styles
  • Use existing visual regression tests to verify snapshots were identical between our CSS-in-JS solution and CSS Modules
  • Gradually roll out the feature flag to our team, then to GitHub staff, and finally to all GitHub users to catch any issues along the way

This process created a strong feedback loop where issues were flagged early in the process as Primer continuously delivered these changes to GitHub. The use of feature flags allowed us to do this migration safely while giving us clear signals on the performance benefits of CSS Modules.

By December 2024, all components in Primer were migrated over to CSS Modules using this process. We saw performance wins across the board, in particular:

  • 55% less time to server-side render a page
  • 25% less time for components on a page to initialize

With clear performance wins from doing this work in Primer, we began to wonder if we could see similar performance wins by doing these conversions in other parts of GitHub. Similarly, how long until we could ultimately drop support for CSS-in-JS across the company?

Moving away from CSS-in-JS at GitHub

One of the trickiest parts about removing our CSS-in-JS solution from Primer was due to the usage of the sx prop. This prop was the way to style and customize components from Primer. Teams could provide an inline object to customize everything about the component. It represented the best and worst parts of CSS-in-JS:

  • Excellent TypeScript support with integration with our Design Tokens
  • Co-located with the component so that everything was in one place
  • High runtime cost due to the dynamic nature of inline objects used for sx
  • Difficulties scaling as the number of components using sx on a page grew

As a result, the first part of our journey to move away from CSS-in-JS was to reduce sx usage across GitHub. This would allow us to immediately improve performance similar to the wins we saw when migrating Primer components. It also set us up perfectly for removing CSS-in-JS entirely from the product.

The Duality of Primer

It’s important to note that, while the design system itself was officially off styled-components, a large part of the GitHub codebase itself wasn’t. With sx props having been the de facto styling standard at GitHub for years, we were looking at thousands of sx props that needed migration before we could even think of getting GitHub onto the sleek new @primer/react version which didn’t rely on styled-components.

So… how did we do this immense amount of work while increasing confidence and reducing risk? The answer: not all at once.

The original CSS migration was a little bit more nuanced than we led on: in addition to migrating the components to CSS modules, feature flagging to test in production and slowly rolling them out, we also created “wrapper” components in a transitive library we called @primer/styled-react. The whole purpose of this package was to allow sx usage into the newly migrated components. This way, the instances of the GitHub UI codebase that were using this prop could continue to consume them by importing the same component through @primer/styled-react, while we realized the performance gains of importing straight from @primer/react for the cases that didn’t.

Styled Box Zero

The next phase of the migration process was as follows:

  • On a package-by-package basis:
    • Translate all sxusage into equivalent CSS modules files. This included cross referencing (See migrating to CSS variables)
    • Replace @primer/styled-react imports with @primer/react imports
    • Test in pre-production
    • Deploy

Curiously enough, while we were getting ready to undertake this massive effort, styled-components maintenance mode was announced, offering further confirmation that we were taking steps in the right direction.

The work kicked off April 2025 with a peak of ~7,760 sxprops to be migrated; we wouldn’t see it realized until May 2026. Initially, one of our great in-house developers, Ian Sanders, created a VS Code plugin that would assist with per-prop migration. A similar codemod was developed internally and utilized to migrate entire files in the GitHub codebase. The work, while requiring a bit of manual oversight and careful validation, was mostly automated. A rotation of 8 engineers migrated 6,419 props over the course of 6 months, observing Server-Side Rendering time performance gains ranging from 1% up to 22% in some pages.

Table titled “Server Side Rendering” with a search field and columns for CATALOG_SERVICE, CONTROLLER, and IMPROVEMENT. It lists GitHub services and controllers with improvement percentages ranging from 3.05% to 21.97%, including a highest value of 21.97% for github/code_view / commit.

In a different side of GitHub, Copilot’s capabilities were increasing exponentially. AI was getting smarter, more capable; we released Copilot coding agent and Copilot code review while this work was still underway.

By the time we picked this work back up, now April of 2026, the panorama was different; we were able to get down from 895 to 0 sxprops in the span of three weeks with a team of two engineers, relentless determination, and a whole lot of Copilot coding agents.

Area chart titled “Total SX props” showing values declining from about 7,300 in May to near zero by June–July of the following year, with several short-lived spikes. Two highlighted points are labeled 6.87k around August and 5.37k around October.

Battle of the themes

It was a big day: we had finally completed the sx migrations that were standing between us and full styled-components removal, years in the making… we can finally clean up these dependencies and move on to different, more exciting work, right? Wrong!

GitHub supports seven different themes, all of which offer a high contrast mode variation. All of it enabled through, you guessed it, styled-components. Before we can even think of removing these dependencies, we need to decouple our theming.

Now, this isn’t as huge a deal as it sounds. Our theming variables have always been defined in CSS through our @primer/css package, and we already planned forward for non-styled Theming when we migrated @primer/react in late 2025. It’s the JavaScript usage and utilities that are enabled by styled-components that we needed to remove. Once again, we got to work.

You know the drill by now: perform the migrations, roll it out slowly, feature flag everything. Two months and a few hiccups along the way later, we were all-systems go for dependency removal; we even feature flagged that. Better safe than sorry.

All’s well that ends well

GitHub has been running on 100% CSS modules as of June 2026. The safeguards we put in place enabled us to roll out significant architectural changes safely, stress test in production, catch errors, pivot and repair efficiently, ultimately allowing us to succeed in our goals and realize great performance gains along the way.

What looked at first like a CSS migration turned out to be a gradual re-platforming of how GitHub styles, themes, and ships UI at scale. By the end, we had not only removed sx, styled-components, and styled-system from dotcom, but completed it without breaking GitHub along the way. Enhancing the performance, user experience and delight of our products continues to be top of mind for all of us here at GitHub.

The post Improving site performance by shipping more CSS appeared first on The GitHub Blog.

Automated merchant review summary system with integrated feedback

Post Syndicated from Grab Tech original https://engineering.grab.com/automated-merchant-review-summary-system-with-integrated-feedback

Introduction

Merchant reviews contain useful details about food quality, portion size, packaging, and value. Finding these details often requires reading many comments. Aggregate ratings simplify comparisons, but they do not explain what shaped each score.

We built the Automated Merchant Review Summary System to turn written reviews into concise summaries. Consumers can scan an overall summary or focus on a specific topic. Merchants can identify what customers value and which parts of the experience need attention.

This article focuses on how we generate and maintain consumer-facing review summaries. A misleading or incomplete summary can affect how consumers view a merchant. Merchant-facing recourse and dispute processes sit outside this pipeline and are not covered in this process flow.

Problem statement

A long, unstructured list of reviews creates two related problems. Consumers spend time searching for comments that match their priorities, while merchants must combine individual comments to identify recurring themes.

Consumers may see hundreds or thousands of reviews for a merchant. Reading only the first few can produce an incomplete picture, while the aggregate star rating provides little context. Two restaurants with the same rating may differ in food quality, delivery experience, portion size, or value.

Merchants face a related problem. Reading and classifying every review takes time, and the task becomes harder as review volume grows. Recurring complaints and positive patterns can remain hidden among individual comments.

Existing approaches

Manual review analysis

Consumers and merchants can read reviews one by one and form their own conclusions. Their conclusions depend on which reviews they read and how they interpret them. This method becomes impractical as review volume grows.

Aggregate star ratings

An average score quickly indicates customer opinion but does not explain the reasons behind it. A merchant can see overall performance but not the themes that may require attention.

Keyword search and filtering

Keyword search helps readers who already know what to look for. It does not reveal unexpected themes, and literal matching may separate phrases with the same meaning. For example, “too cold”, “not warm enough”, and “lukewarm” all describe a food-temperature issue.

Our solution

The system uses representative sampling, large language model (LLM) summarisation, automated quality checks, and consumer feedback. It generates an overall summary and topic-based summaries for areas such as taste, packaging, and portion size.

The pipeline has three flows:

  • Summary generation and validation: The pipeline samples reviews, generates overall and topic-based summaries, and checks whether the source reviews support each claim.
  • Summary maintenance: The system monitors new reviews and consumer feedback, then regenerates a summary when either reaches a configured threshold.
  • Summary display and feedback: The interface retrieves a validated summary for the merchant’s review page and records whether consumers found it helpful.

Architecture

The generation pipeline prepares the source reviews, produces summaries, and validates the output before publication.

Figure 1. Representative review sampling flow.

Data preparation and representative sampling: The pipeline calculates the merchant’s rating distribution. A sentiment model assigns each written review a score from one to five. Stratified sampling then selects a smaller set of reviews that follows this distribution, reducing the chance that one rating group dominates the summary. However, matching the rating distribution does not guarantee that the sample represents every period, language, customer segment, or aspect of the merchant experience.

Summary generation with an LLM: The system sends the sampled reviews to an LLM with instructions to produce:

  • A concise summary of the sampled reviews.
  • Topic-based summaries for categories such as taste, packaging, and portion size.

The system can update the topic set when new themes appear in consumer feedback.

Figure 2. Summary generation, validation, and feedback flow.

Quality control (QC) with an LLM: A separate LLM receives the generated summary and the sampled reviews. Summaries that do not meet the validation criteria are withheld from publication. Passing this automated assessment reduces, but does not eliminate, the risk of unsupported claims.
The maintenance flow updates stored summaries in response to incoming reviews and consumer feedback:

  • New-review trigger: A monitoring service counts reviews submitted after the latest summary. When the count reaches a configured threshold, the service starts the generation and validation flow.
  • Consumer-feedback trigger: The interface records whether consumers found a summary helpful. When negative feedback reaches a configured threshold, the system generates and validates a new summary.
Figure 3. Consumer feedback analysis and prompt refinement flow.

Impact

Review summaries are designed to reduce the amount of text consumers need to scan. Topic-based summaries can surface themes that are not visible in an aggregate rating alone.
The quality-control step rejects unsupported claims. The regeneration triggers incorporate new reviews and consumer feedback into later summaries.

Learnings and conclusion

Useful review summaries depend on more than generation quality. The source sample must reflect the intended population, each claim must remain grounded in source reviews, and the output must change with the underlying feedback. Automated LLM validation is one layer of defence, not sole proof of accuracy, and feedback-driven regeneration needs anti-abuse controls to remain trustworthy.

Representative sampling, separate generation and validation stages, and feedback-based regeneration form a practical system for maintaining consumer-facing merchant review summaries. This article describes that technical pipeline; it does not cover the full set of merchant recourse options when a summary is disputed.

Join us

Grab is Southeast Asia’s leading superapp, serving over 900 cities across eight countries (Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam). Through a single platform, millions of users access mobility, delivery, and digital financial services, including ride-hailing, food delivery, payments, lending, and digital banking via GXS Bank and GXBank. Founded in 2012, Grab’s mission is to drive Southeast Asia forward by creating economic empowerment for everyone while delivering sustainable financial performance and positive social impact.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Rendering huge pull requests in the GitHub Copilot app

Post Syndicated from Alberto Gimeno original https://github.blog/engineering/user-experience/rendering-huge-pull-requests-in-the-github-copilot-app/


Broad refactors and migrations often have to land as one change.

Stacked pull requests are a great way to split work into smaller changes, which makes reviews easier and helps teams ship with less risk. But some changes, like this one, can’t be split cleanly. That leaves you with a single pull request that can get very large, and the review conversation causes it to grow.

The review experience needs to remain fast and smooth even when the diff and its conversation are enormous. In the GitHub Copilot app, we rebuilt the pull request view with that requirement in mind.

To see how far that goes, we opened the biggest pull request we could find: an open source one with 2,200 files, over a million changed lines, and more than 400 inline review comments. Here’s how we made even this extreme pull request performant.

The scope of the problem

Rendering a large diff at speed is well-understood: virtualize the rows, keep the mounted DOM small, and lean on the fact that every row is a line of code at a known height.

Comments are the hard part. A comment’s height depends on how its markdown wraps, the expandable sections, whether there’s a reply box in it, and whether its images have loaded yet. You find all of that out at render time. This forces a different architecture.

Three problems:

  1. Measurement. You can’t know how tall a comment is until you render it. This breaks the design that lets big diffs stay responsive as you scroll.
  2. The data pipeline. A fast diff surface is worthless if the data pipeline feeding it stalls, or if it throws away work it already did.
  3. How we actually found the bugs. These problems surface under load, on a specific engine, at a specific scroll position. So we defined what healthy meant, instrumented the surface to answer it, and ran the whole change → measure → improve loop unattended.

Part 1: Virtualization, and why comments break it

The first step is to understand the geometry that makes a code-only diff fast. Once comments enter the picture, that geometry is no longer enough.

What makes big diffs fast

You cannot put a million DOM nodes on a page. The standard answer is virtualization: mount only the rows that are on screen, plus a small margin, and recycle those same DOM elements as the user scrolls. The list behaves as if all million rows exist. The scrollbar is the right size, scroll-to-row works. But only about 100 rows are ever real at once.

For this illusion to hold, something has to supply the geometry. The scrollbar height is the sum of all row heights. The position of row N is the sum of the heights of the rows above it. Jumping to a row, drawing the scrollbar, deciding what’s on screen, it’s all arithmetic over a table of heights. You can build that table from estimates and correct it as rows get measured, and general-purpose variable-height virtualizers do exactly that.

But if every row is a line of code at a known font size, you don’t have to. You can compute the whole table up front and it never changes, so there’s nothing to correct later.

Call this the “all heights known before paint” contract. Our diff surface is built around it:

  • An imperative, recycled code-row renderer (no React component per row)
  • Typed-array geometry for the offset math
  • Backend-owned diff documents streamed structure-first
  • An imperative scroll API with exact “scroll to row N“

None of it scales badly, because no per-frame work grows with the total row count. On pure code this design is the right one, and we kept all of it.

How comments change the contract

Now put a review thread in the middle of the diff. How tall is it?

You don’t know, and you can’t know without rendering it. Its height depends on things that only exist at render time, and they can keep changing after first paint:

  • Markdown that wraps differently at different widths
  • <details> blocks the user can expand or collapse in place
  • A reply composer that opens inside the existing thread and grows as you type
  • Suggested-change diffs, reactions, edit mode, resolution banners
  • Images and async assets that change height when they finish loading

The obvious answer is to reserve a fixed-height slot for each comment, sized by an estimator. It falls apart on a big pull request. An estimator that’s right on average is still wrong at the extremes. It over-reserves most comments, leaving gaps of whitespace, and under-reserves the expensive ones, which clip or sprout a nested scrollbar. If you measure the real height after paint and write it back into the shared offset table, everything below moves, while the user is already scrolling. That’s a scroll jump, and on a big pull request it’s a large one.

So comments need a different contract. “All heights known before paint” is unachievable for this content. What we could promise instead: heights are bounded, measured lazily, and corrections are small and anchored to whatever the user is looking at.

Two geometries instead of one

The idea that made this tractable was to stop forcing one geometry to serve both kinds of content. We split the document’s height into two independent domains:

total height = deterministic code height          (exact, known up front)
             + Σ dynamic block effective heights   (estimated, then measured)
             + scroll padding

Code geometry keeps the original world. It’s deterministic, prefix-summed, exact, never rebuilt when a comment resizes.

Dynamic block geometry covers everything whose height we can’t predict, such as review threads, drafts, and reply composers. Each one is a block identified by what it is rather than where it currently sits. It has a stable key that survives its content loading, and it’s anchored to a file, line and side rather than to a pixel coordinate, so a reflow can’t lose track of it. We also keep a fingerprint of everything that could change the block’s height: its content, whether a <details> is open, whether a composer is active. And we record the width it was last measured at, rounded into buckets, so an ordinary window resize doesn’t invalidate every measurement in the document.

A block’s effective height is then simple: the measured height if we have a valid one, a cached height if the fingerprint and width still match, and the estimate otherwise. Those heights live in their own index, separate from the code rows, so a resizing comment never forces the code geometry to be rebuilt. And the number of blocks is bounded by comments, not by rows. A few thousand blocks is fine, as long as first paint never mounts or measures all of them at once.

The measurement scheduler, and the mistake we made first

This part took the longest to get right, because our first design was wrong in an instructive way.

The obvious way to measure dynamic content is one ResizeObserver per block, which watches the element and writes its measured height back into the layout whenever it changes. This is what we designed and then rejected during performance hardening. It is the feedback loop that big virtualized surfaces have to avoid. An observer that writes a height back into the layout of the element it’s watching can retrigger itself, and the cost grows with every mounted block.

What shipped instead is a single idle- and scroll-gated measurement pass, held to the same discipline as the deterministic side:

  • Off the hot path. It runs when the visible range settles, never once per scroll frame, and waits entirely while a scroll is in flight. A reflow mid-scroll is exactly the jank we’re avoiding. It runs again once scrolling stops.
  • Scoped to the viewport. Only blocks within roughly 2400px of the viewport are candidates, so the work is O(viewport). Distant blocks keep riding their estimate and get corrected as they approach.
  • On-screen reads win. A mounted block is on screen, so its rendered height is ground truth. The pass reads every mounted candidate in one batch, a single reflow with no writes in between, and records what it finds. A mounted block is never skipped in favor of a stale estimate. That one rule fixed the nastiest bug we hit: comments that rendered with a strip of blank space underneath, because a mounted block had been filtered out of measurement and left sitting on a too-tall estimate.
  • Off-screen measurement is a bounded fallback. For a nearby block that hasn’t mounted yet, the pass does at most one off-screen render, to correct its reservation before it scrolls into view. Blocks taller than the viewport skip even that. Their over-reservation hides below the fold, so the render isn’t worth paying for.
  • An observer catches the rest. Some height changes don’t move the fingerprint and don’t coincide with a scroll: typing in a reply composer, an image finishing loading, toggling a <details>. Each mounted block keeps a ResizeObserver, but by default all it does is flag the block so the idle pass re-reads it. It never writes a height itself, which is what would close the feedback loop we rejected. It disconnects on unmount, and an inactive pull request tab observes nothing.
  • With one deliberate exception. Waiting was visibly wrong for resizes you caused yourself: expanding a <details>, opening a reply composer, an image landing. The block grew immediately, but the code below it only moved on the next idle pass. For one frame the comment was taller while everything under it sat at its old position, and you could see the two steps. So when a block is mounted and on screen, the observer now measures it and applies the correction in the same frame, before paint. The block grows, the code repositions, and everything below shifts together. Two safeguards keep this from becoming the loop we were avoiding: at most one synchronous commit per frame, so a burst of resizes collapses into one, and never during an active scroll, where it falls back to the batched pass.

Scroll anchoring: Correcting without fighting the user

When a measured height differs from its estimate, the scrollbar arithmetic changes, and the naive result is that the viewport jumps. The fix is to correct by identity rather than by pixel:

  1. Before applying height updates, capture what the user is anchored to (a row or a block, by identity), plus the offset within it.
  2. Apply the height deltas.
  3. Resolve that same anchor to its new pixel position.
  4. Scroll so the anchor stays put in the viewport.

Plus a few rules that keep it from feeling wrong:

  • A block above the viewport changing height → adjust by the delta (keeps your place).
  • Content hydrating below the viewport → don’t adjust (you can’t see it).
  • If you toggled a <details> or opened a reply in a visible block → suppress above-block correction for that block, so the interaction feels direct, and let the content below flow down naturally.
  • Never fight active pointer or wheel momentum; batch the correction after the frame.

That last rule has a sharp edge, and it bit us. “Don’t correct while the user is scrolling” was implemented as a guard on the last observed scroll, and programmatic scrolls refreshed that timestamp too. Toggling the file-tree sidebar changes the width of the diff pane. With line wrapping on, every wrapped line above you reflows to a different number of visual lines, the whole coordinate space shifts, and the surface emits a small scroll of its own as it settles. The guard read that as “the user just scrolled” and skipped the very correction that was supposed to keep your place, so the file you were reading drifted off screen. The fix was to tell user scrolls apart from ones the surface caused itself. Any “is the user interacting?” check has to be one your own side effects can’t satisfy.

So corrections stay small, they reuse measurements we already have, and they follow whatever you’re looking at.

Part 2: The pipeline behind the surface

A diff surface can only be as fast as the data feeding it, and three habits from that side of the work shaped what the UI could do. The first is stream structure before content. The diff is requested incrementally, so the file tree and metadata paint while the document is still loading, and the full set of review threads is resolved up front rather than trickling in. The second is defer per-item work until something needs it. Syntax highlighting runs off the main thread, so rows appear as plain text immediately and get colored when the results arrive. Highlighting improves the surface instead of blocking the scroll. Large markdown bodies and suggested-change context work the same way: nothing is built until it approaches the viewport.

The third habit is about which costs are worth keeping. Releasing a diff document when you navigate away is the right default. These documents are large, and holding on to every one you’ve visited is how a long session ends up eating memory. But pull request metadata persists, so the shell around the diff, the header and the file tree, repaints instantly when you go back, and then sits there for several seconds waiting for a diff it had complete moments ago. An instantly-drawn shell around an empty diff looks broken, even though you’re waiting less time overall. So the policy stayed and we added a cache: keep the last few diffs resident, evict anything beyond that, and let the background refresh notice when one has gone stale.

Part 3: The measurement loop, or how we actually found the bugs

Almost every bug in this project was invisible until it wasn’t, and reproducing one by hand is miserable. A typical report reads: “a strip of whitespace appears below some comments, but only sometimes, only on big pull requests, and it heals if you scroll past and back.” You can’t debug that by staring at the screen, so we built tooling to debug it mechanically.

Instrument with the app’s real signals, not throwaway logs

The naive workflow is to sprinkle console.log calls, exercise the flow by hand, copy the output, paste it to someone (or something) that can analyze it, delete the logs, and repeat. It’s slow, it needs a human in the loop, and worst of all you end up measuring your own hand-rolled instrumentation rather than the app’s real behavior.

So the surface carries permanent, structured probes for its own invariants. They’re plain questions it answers about itself on every render:

  • Is the surface actually viewport-bound? How many rows and comment blocks are mounted right now?
  • Is measurement coalescing to a single commit per frame, and how long does that frame take?
  • How large are the scroll corrections we’re making?
  • Did any comment block get inserted after scrolling started? (Must be zero once the backend topology has landed.)
  • Do the per-block observers actually tear down on unmount, or are we leaking one per block?

These are the objective pass/fail signals, and they’re asserted as budgets in an end-to-end test against a synthetic many-comment huge-pull-request fixture. CI can now tell us whether the surface is healthy.

Put the loop on autopilot

The centerpiece was an autonomous change → measure → improve loop. Two lanes:

A headless probe lane ran a declarative flow (open a pull request, scroll to a fraction, toggle a details block, resize the window) against a mock server, reading the app’s own production instrumentation: React render counts, the performance timeline, and a requestAnimationFrame sampler for jank. It did the whole instrument, drive, collect, analyze, rank cycle by itself and printed the bottlenecks in order. Because the flow is just JSON handed to the probe at runtime, an agent could profile any flow by describing it in plain English, without editing a line of source.

An autopilot drove the actual desktop app through the huge-pull-request flow, unattended, on a loop: first cold, with comments still skeletons, then warm, with comments loaded, toggling <details> blocks, opening and cancelling reply composers, collapsing and expanding files, toggling the sidebar tree, sweeping deep into the file list, resizing the window. Every measurement was mirrored to the app’s on-disk log, so an agent could read runtime behavior with nobody at the keyboard. Each sample carried a health signal, and that was the objective check. A warm sample counted as healthy only if there were no unfilled gaps between comments, no comment blocks left blank, and real thread content actually mounted, across the entire scroll range, deep-file sweep included.

The loop we ran was:

  1. Reproduce unattended, on the real engine. Arm the autopilot, let it loop, read the on-disk log.
  2. Detect with a health signal, not with your eyes. Trust the sample fields.
  3. Probe the suspect seam. When a signal goes bad, add one narrow structured probe there, re-arm, re-read. (Editing the surface hot-reloads the live window and re-arms the autopilot, so a fresh capture is about one cycle away.)
  4. Remove the scaffolding. Once you understand the invariant, pin it in a test and the design doc, and keep only the detector-grade signals.

Where this leaves us

Reviewing a pull request this large used to mean one of two things: waiting, or giving up and reading it somewhere else. A review isn’t a document with known dimensions. It’s a conversation that changes shape while you’re reading it, and the surface underneath has to be built for that from the start rather than patched into it afterwards.

The result is a pull request view where a million-line diff with hundreds of threaded comments opens, scrolls, and behaves like a normally sized pull request. Comments render in full instead of clipping into a scrollable box. Expanding a collapsed section moves the code below it and nothing else. Coming back to a pull request you just left puts you where you were.

If you review code for a living, it’s worth feeling the difference on a pull request you already know is painful. Open the worst one you’ve got.

Try the GitHub Copilot app >

The post Rendering huge pull requests in the GitHub Copilot app appeared first on The GitHub Blog.

Saving another 100TB of RAM with math (and Rust)

Post Syndicated from Kevin Guthrie original https://blog.cloudflare.com/saving-100-tb-of-ram-with-math/

Cloudflare operates at a scale so big that even after working here for years, it doesn’t seem real. We have thousands of servers all over the world with petabytes of RAM and millions of CPU cores, and all of it is pushed to the max. As vast as those resources feel, they are still finite, and when you need every service to run on every node, it doesn’t leave room for wasted space.

At this scale, small improvements are greatly magnified, so even 1%-at-a-time improvements are worth celebrating. And some tweaks add up to a lot more: in this post, we’ll look at how small changes to a single algorithm reduced the memory footprint of one of our Pingora-based services significantly. That allowed us to reclaim more than 100TB of RAM globally, on top of the 100TB of memory the DNS team was able to shed last month.

Waste not

Maintaining equitable resource sharing between teams is not easy, especially in large organizations. One of the ways Cloudflare ensures the balance is kept is through the tireless efforts of the wonderful Performance team. 

This story starts with a ticket filed by Ivan who found: Excessive memory usage from pingora-ketama in Pingora Backend Router. The finding was that our internal load-balancing service, Pingora Backend Router (yes, PBR), was using significantly more memory than expected — specifically in structures associated with pingora-ketama, which is our open-source library for handling consistent hashing.

In order to talk about how we addressed this seeming overuse of memory, we need to talk about what consistent hashing even is, why we are using it in PBR, and how it became so memory hungry. Along the way, we’ll learn some Rust and even a little math.

Consistent hashing

Consistent hashing is a widely used method for distributing tasks across multiple servers in a way that does not require large changes when servers are added or removed. Internally we use it to route cacheable requests to servers by URL. This allows us to keep only one copy of a file stored per data center and gives a stable way to find the location of each file. We have mentioned this system before, but let’s take the time to walk through how and why this algorithm is used and how it works.

The key concept of consistent hashing is that while hash functions can accept any kind of input, their output is limited to a single unsigned integer (32, 64, or 128-bit integers depending on which hash function). This allows us to relate tasks and servers to each other in a consistent way. Most discussions of consistent hashing have you think of that output space as a continuous, circular ring that wraps around from its max value to zero. This depiction makes for some nice visualizations, but it can also make the simple concept of integer ranges seem more complicated than it needs to be. For our discussion, we’ll represent the 32-bit output of our hash function as a number line.

Now, let’s say we have a set of servers, A, B, & C, and a set of tasks t-z. We can map each onto the number line based on the hash of their representative values, so something like IP addresses for servers and cache keys for tasks.

Assigning tasks to servers is now just a matter of finding the first server to the left of each task. We can represent this visually by coloring in the region of hashes that will be associated with each server. Notice that the range covered by server C wraps around to the beginning, hence the idea that hashes exist in a ring.

And that’s it. At a base level, consistent hashing is this simple — but it doesn’t take long to see that there is room for improvement. Notice that the range covered by server A in our example is significantly larger than that of either B or C. This is a problem because the fraction of the requests a server handles is going to be proportional to the size of its range on the number line. Ideally we would like to guarantee each server will have an equal size, but because hashes are essentially random numbers, we have to talk about the size of the regions in terms of statistics. 😨

Math and consequences

First: don’t panic. I promise I'm not about to lie to you and that we will stay safely within the bounds of a day-one probability lesson. When we talk about statistical distributions, there are two big factors that help us quantify uncertainty in helpful ways: expected value and standard deviation. In (over-)simplified terms, expected value gives us a point where measurements based on a distribution will be centered, and standard deviation tells how close to that central point most measurements are likely to be.

For consistent hashing, we can calculate these factors for the fractional size of the range associated with one of N servers. (Details on where this formula comes from later).

In terms of concrete numbers, let’s say we have 100 servers. The formulas above give:

That tells us that we can expect that the range each server handles will be centered around 0.99% of the total and most of the lengths to fall within 1% of what's expected. This sounds good until we realize that that’s 0.99% of the total length. We need to scale the standard deviation by the expected value to see how big the error is as a fraction of the target size. This value is called the coefficient of variation.

What if we add hashes?

The simplicity of consistent hashing is a double-edged sword. It’s easy to understand and implement because everything is turned into easily-relatable hashes on the same numberline, but any improvements to the system will also need to be relatable to that numberline. That means the solution to any consistent hashing problem can only be more hashes. It’s less like a golden hammer (a tool with which all problems look like nails) and more like a golden nail in that it turns all tools into hammers.

To solve the problem of imbalanced workloads, we can add multiple hashes to represent each server instead of just one. We’ll get to the math behind this momentarily, but it should make some intuitive sense that while each individual range has a large standard deviation, adding a bunch together should make their total size even out. If we take our three-server example from the above diagrams and add two more hashes at random for each server, we see that it helps even out each server’s workload. 

This is an admittedly contrived example. The random nature of the system means there’s no guarantee how much improvement you will get from adding 2 additional hashes per server, but it should make some intuitive sense that combining more of these hash segments together produces a more even distribution. Each segment in the sum has a chance of balancing another. Maybe one is too short; maybe one is too long. This is essentially what the law of large numbers tells us should happen… The obvious problem is it only works for large numbers. In NGINX, the baseline number of hashes per server is hardcoded to 160, and Pingora uses the same value as the default. I’ll spare you the math for now, but if we go back to our 100-server example, if we use 160 points per server instead of just one, the coefficient of variation (which we can think of like an error margin) drops from about 99% to about 8%, a significant improvement.

What if we add more hashes?

We saw above that increasing the number of hashes per server by a constant amount allows us to improve how evenly workloads are distributed per server, but what if we don’t want to distribute the work evenly? In Cloudflare’s case, we have some servers that have more storage space than others, so it would be better to have the number of requests allotted to a server be proportional to its disk space. One way to accomplish this is with the ketama algorithm. The naming is a little funny because the algorithm is named after the library where it was first implemented, and the library was named … well you can google it 😶‍🌫️.

For us, since we want workload to be scaled based on storage, we can use the disk space as the weight, which is exactly what the Pingora team has been doing for years. Elsewhere in the company where workloads are more compute-intensive, weights might be based on CPU or GPU count.

What if we add even more hashes???

The last problem we need to address is that so far we are working under the assumption that any server can handle any request, but in practice that is not the case. Things like compliance requirements or enabled caching features mean only a subset of servers can handle any particular request. Unfortunately, unlike before, we can’t solve this problem by adding more hashes to the same ring. We have to add completely new rings, and not only that — every combination of features potentially needs its own specific ring!

Storage improvements

One big improvement came from Zaidoon, who had an insight about our struct for storing hashes in PBR. That struct looks like this:

Unfortunately, Rust doesn’t make it that easy. Changing the size of the index as we did above does nothing to reduce the memory footprint. This is because Rust has alignment rules that require the size of a structure in memory to be a multiple of its largest (or “most aligned”) field. In this case, the hash is the largest with four bytes, so when stored in memory, a Point is required to have size $mN \times 4m$, so the minimum size is eight bytes.

Luckily there are well-known ways around this. You (meaning me) might be tempted to use #[repr(packed)], but that is controversial for good reasons. A safer but less readable solution is to store the hash and index as raw byte array and access them with getters. Both methods compile to the same thing.

This simple (if wordy) change reduces the amount of memory used for consistent hashing by a whopping 25%! In order to do better than that, we’ll need to jump back into the math, so everybody hang on to something; this is the home stretch.

What if we tried fewer hashes?

You may have noticed that we gave the formula for the standard deviation for the case where there is only one hash per server. Deriving the formula for the case where there are $m k m$ hashes per server is not easy, and most sources only give you an approximation or an asymptotic limit, but not us. I might not be a statistician, but I grew up with a calculus teacher (Hi, Mom!), and I wanted to know the actual value. The full derivation is in a supplemental post, but here is the payoff.

To see how increasing the hash count improves the accuracy, we need to look again at the coefficient of variation.

The predictions from my beautiful math only work if we think about hashes in a continuous ring, but in practice we use 32-bit numbers for the hashes that have the potential for collisions, and the probability of collisions goes up surprisingly quickly as the number of hashes increases (see the birthday paradox). Collisions matter because in the ideal case, every hash contributes to the volume and distribution of requests handled by the associated server, but a collision means some contributions are randomly dropped, introducing unpredictable error. If we compare some simulated results with 32-bit hashes with the predicted error rate, we can see that for data centers with 2048 servers, the error rate increases: between 10,000 and 100,000 hashes per server.

Ultimately, even though this realization feels kind of bad, it’s great news for our plan to reclaim some RAM! Now that we have some math to back it up, we determined that we could decrease the number of hashes we were generating for each server by 90% without incurring any appreciable error, so that is what we set out to do.

Migrating without melting origins

There was one more problem: changing the hash ring changes where some cacheable requests go. Even if the new ring is better, switching the whole network at once would effectively invalidate almost all cached content. It would turn a memory optimization into an apocalyptic increase in origin traffic.

So we did not make this a single global flip. For a while, PBR carried both versions of the cacheable load balancer in memory: the old ketama ring and the new smaller one. Each request used our normal migration framework to decide which ring should select the backend. That meant the rollout decision was stable per request hash, and it also gave us a clean rollback path. If anything looked wrong, we could send new requests back through the old ring without redeploying PBR.

We then rolled the migration out in layers. We started with small validation locations, moved through progressively larger groups of data centers, and only then continued toward the rest of the world. 

The important part was that we controlled two dimensions independently: how much traffic used the new ring, and where that traffic was allowed to move. A plain global percentage rollout would have spread cache churn everywhere at once. Data-center-scoped rollout kept the blast radius small and made it much easier to tell whether a change was actually safe.

During the migration, we watched backend-selection traces, ring-version counters, PBR connection errors, process memory, startup time, cache behavior, and origin traffic. Once the migration reached 100%, we removed the temporary old-ring path, and voila!

The chart above shows the comparison of the memory used by PBR the week of the change compared with data from a few weeks before, as well as the result of subtracting one from the other. The sharp drop is the day where the version of PBR with the large (now unused) hash rings was decommissioned forever. Looking at the difference, we get the satisfying result that our changes dropped the used memory by 100TB!

Try it yourself

All the changes we talked about in this post are available now in the pingora-ketama crate in the form of a (for now) unadvertised cargo feature. The v2 ring has the compacted storage format, a faster sorting method, and the ability to scale the base number of hashes per node. Our focus in making these changes had to be on stability and control, so the v1 ring is identical to what pingora ketama has always used, and the library makes it possible to run both simultaneously and decide on a request-by-request basis which to use and when. 

Beyond trying our literal consistent hashing changes, I would like you to take away from this some inspiration to dig into your own systems to see what “simple” or “obvious” decisions are hiding potential wins, if you’re willing to get into the numbers. You might not be able to solve all your problems with Rust, but math is universal.

How we make AI coding more cost efficient without sacrificing task quality

Post Syndicated from Erik Kristensen original https://github.blog/ai-and-ml/github-copilot/how-we-make-ai-coding-more-cost-efficient-without-sacrificing-task-quality/


Output quality is important when working with AI coding agents, but true efficiency comes from getting work done quickly, efficiently, and with the right context.

That’s why token count of individual interactions alone isn’t a meaningful measure of efficiency. The goal shouldn’t be to use fewer tokens, but to tap into the right amount of context to move a task forward. A concise tool response can sometimes require additional calls or work if it leaves out information the agent needs, ultimately making the task slower and more expensive.

That’s why we want to optimize for the outcome rather than the tool call. This post examines four changes in GitHub Copilot that put that principle into practice:

  • Preserve useful context while reducing repetitive output.
  • Remove formatting that adds no value to the task.
  • Shorten instructions without changing useful behavior.
  • Deliver completed background work without an extra retrieval step.

Possible changes were evaluated offline using agentic coding benchmarks. The most promising changes were then validated through controlled online experiments before shipping. The examples in this post come from GitHub Copilot CLI. Multiple other Copilot products, such as the GitHub Copilot app and Copilot code review, use the same underlying harness and also become more efficient through these improvements.

Chart showing 3.1% 'Remove view previxes', 5.5% 'Selective output compaction', 2.9% 'Compact task-tool prompt', and 2.3% 'Reduce notification roundtrips'.
Figure 1: Four independent A/B experiments using the same AI-credit metric. The segments are shown together for comparison; their effects are not necessarily strictly additive. 

The local metric trap

It’s common to shorten the output from each tool call as a way to reduce agent costs. RTK (Rust Token Killer) is a utility that shortens shell output before an agent reads it. We evaluated its effect on GitHub Copilot using our agentic coding benchmarks.

In our harness and benchmark configuration, RTK shortened some responses, but when the omitted text mattered, the model sometimes reopened the original output or reran the command to recover what it needed.

Those recovery steps added turns and carried more context forward. The individual tool response was shorter, but on average, the task used more tokens and took longer. We saved tokens locally and spent more globally.

Flow chart showing: RTK, compresses shell output > Local win, tool output gets shorter > Useful detail is missing > Recovery, reread or rerun > More turns and context carried forward. Then the option of finishing at 'End-to-end result, Tokens and cost up, Task duration up, Task completion: steady,' or 'Recovery repeats' going back to 'useful detail is missing'.
Figure 2: A shorter tool response can make the completed task more expensive when missing details force the agent to reread output, rerun commands, and carry more context forward. 

This result applies to the integration and workloads we tested, not to every RTK configuration or to output compression in general. This meant that tokens per tool call is the wrong objective. An efficiency change has to be evaluated across the complete task, from the user’s request through the final result.

More useful was to look at what can we remove without making the model repeat work.

Compress noise, preserve useful information

The goal was to shorten repetitive output while preserving the context an agent needs to complete its task without retracing steps.

Analysis of benchmark runs showed that install, build, test, and lint output often contains repetitive noise, while source-like output and arbitrary command results are more likely to contain the information an agent needs. That analysis informed a selective output compressor, informed in part by RTK and similar approaches.

The prototype was evaluated on agentic coding benchmarks and a range of open source repositories, exercising their build, test, and lint systems.

Early versions were too aggressive. They made the model repeat work or read the full saved output, increasing end-to-end cost and reducing task success. For example, we initially compressed git diff but removed that filter after benchmark tasks showed agents reopening the original output to recover missing information.

Those early failures led to a three-part policy:

  1. Preserve source-like and arbitrary output. Commands such as cat, git diff, git show, and arbitrary scripts are returned unchanged.
  2. Reorganize search results without dropping content. Matches and file lists from tools such as grep can be grouped more efficiently while retaining every result.
  3. Compress repetitive noise selectively. Install, build, test, and progress output is compressed only when the savings are substantial.

The shipped version emerged through repeated evaluation and refinement. It is conservative not because the goal was to build a conservative compressor, but because that is what the evaluations supported.

When output is compressed, the agent can still retrieve the complete original through a direct recovery path.

Flowchart showing how GitHub Copilot handles shell-command output. Copilot calls a shell command, classifies the output, then chooses one of three paths: keep arbitrary/source output unchanged, reorganize search results without losing any matches, or selectively compress repetitive noise (like install/build/test logs) while preserving full output and providing a recovery path. The processed result is returned to Copilot.
Figure 3: The shipped compressor preserves source-like output, reorganizes search results without loss, and compresses only predictable repetitive noise while retaining the full original.

That recovery path is both a safety mechanism and an evaluation signal. We tracked whether the agent opened the saved original, reran commands, repeated exploration, narrowed its searches, or took additional turns. Frequent recovery would indicate that the compressor had removed something valuable.

On offline tasks where output compression triggered, no statistically significant task-success regression was detected, and agents extremely rarely opened the saved originals. In the online experiment, average cost decreased slightly with no material regression detected in the tracked quality metrics.

Remove formatting before removing information

One clean token optimization came from the view tool, which agents use to read file contents into context.

Previously, view prefixed every line with a number before showing the contents to the model. Earlier file-editing tools used those numbers to target changes, but current tools instead match surrounding code and do not use line numbers. The line-number prefixes remained even though the normal workflow no longer used them.

Each prefix was small. Repeated across every line and every file read, however, that unused formatting accumulated throughout a session. So, we removed it.

Before-and-after image of code snippets. The line-number prefixes re removed from the 'After' image.
Figure 4: Removing line-number prefixes preserves the source exactly while eliminating formatting that was repeated across every file read.

Line numbers remain useful in diffs and short snippets. They were wasteful here because they were attached to every file read without serving the current editing workflow.

Removing them caused model-inference cost to fall by roughly 5% in offline agentic coding benchmarks. Success rates stayed within the expected run-to-run variance, and edit failures did not increase.

We then tested the change with Copilot CLI users. The online experiment reduced average daily model-inference cost per user by about 3%, with no material regression detected in the quality or satisfaction metrics we tracked.

For developers, that means more of the context window is available for the work itself rather than formatting the agent does not use.

This was the ideal change: no new instructions for the model, no source of information to recover, and no additional decision to make. The file contents reached the model unchanged.

Compress prompts without compressing intent

Prompts carry instructions that shape how an agent works, and they are sent to the model on every turn. Shortening them only improves efficiency if the agent keeps the behaviors developers depend on.

In GitHub Copilot, the task tool launches specialized agents for parallel work. Its guidance had accumulated across tool descriptions, schemas, agent definitions, system instructions, and companion tools.

A meta-prompting loop, in which Copilot iteratively wrote its own prompt, reduced that prompt by roughly half. Copilot produced and refined smaller candidates, and targeted behavioral tests checked the requirements we wanted to preserve.

The first online experiment found a regression that the initial offline evaluations had missed. The meta-prompting loop had rewritten cautious parallelism guidance into a hard scheduling policy, causing independent custom agents to run sequentially.

We stopped the experiment. Before changing the prompt again, we wrote a regression evaluation for the behavior users had exposed. The eventual fix replaced an explicit allowlist and denylist with one sentence:

Independent agents can run in parallel; consider side effects.

That sentence was shorter and less restrictive; it deferred the choice of whether to run sub-agents in parallel to the model instead of the previous explicit guidance. With it, our new behavior test passed without causing any existing behavioral tests to fail.

Prompt behavior needs tests. If a behavior is not tested, a shorter prompt can remove it without anyone noticing. 

Three-stage diagram labeled Compression → Regression + fix → Completed. Left panel shows an original prompt compressed by about 50%. Middle panel highlights a regression where agents became serialized, then a fix by editing one sentence to restore parallelism. Right panel shows final shipped prompt with restored behavior and cumulative savings of about 1,300 fewer tokens per turn across steps.
Figure 5 Prompt compression became safe only after a regression test exposed serialized agents and a one-sentence fix restored parallelism; the resulting token savings recur on every model turn.

The shipped prompt removes about 1,300 task-tool prompt tokens per turn, corresponding to approximately 1.8% fewer total prompt tokens per session and 2.9% lower normalized cost per active hour, with no quality regression detected in the measured evaluations.

Deliver completed background work without an extra retrieval turn

Agents often run independent work in the background, such as a long-running shell command alongside a sub-agent investigation. Notifications let the agent continue until that work is ready without spending a tool call waiting.

If the agent does not explicitly wait for either task, the harness wakes the model and notifies it when the shell command or sub-agent finishes.

Previously, that notification did not include the completed result, so the agent had to spend another turn retrieving output Copilot had already received. When several tasks finished close together, that detour could repeat. Copilot now batches eligible completion notifications and delivers completed results directly in the existing tool-result format. The agent can continue with the information it needs, without spending an extra turn asking for it again. Explicit reads for work that is still running behave as before.

Before-and-after sequence diagram comparing orchestration behavior. Before: model waits on separate shell and sub-agent completions, causing retrieval detours and four LLM calls to process two results. After: a harness batches related completions and emits synthetic tool events so background work continues while waiting; both results are processed together in a single LLM call. The visual emphasizes reduced latency and fewer model round trips.
Figure 6 Before, each background completion could wake a retrieval-only model turn. After, the harness batches eligible completions and delivers completed results in the existing tool-result format.

Before this change, each completed task required one model call to request its result and another to process it. For the shell command and sub-agent shown above, that meant four model calls before work could continue.

Now, the harness batches both completions and supplies their results together, so a single model call can process both. Removing those retrieval detours also avoids carrying the full session context through unnecessary calls.

By delivering completed results directly, without compressing, summarizing, or withholding anything, the harness reduced average token-related usage, as measured in AI Credits, by about 2.3%.

Measure changes in context

A change that saves tokens in one Copilot workflow can increase costs in another.

For example, a tighter set of file-tool instructions was inspired by positive results in Copilot code review. In a Copilot CLI online experiment, it increased cost, so we did not ship it.

By contrast, removing line-number prefixes and selectively compressing output each reduced average prompt tokens per review by roughly 5% in independent evaluations across a large set of Copilot code review tasks using the production model. We detected no material change in the tracked review-quality metrics.

These findings are separate from the earlier migration of Copilot code review to the shared file tools, which, together with review-instruction tuning, reduced code review cost by about 20%.

Each change needs to be measured in the workflow where it runs.

Five lessons for building efficient AI coding agents

  1. Optimize the completed task, not the tool call. Shorter output is not cheaper if the agent spends more turns recovering what was removed.
  2. Optimize orchestration, not just model output. Eliminate model turns that perform work the harness can complete deterministically.
  3. Compress by what the output represents. Preserve exact content, prefer lossless transformations, and measure how often agents use the recovery path.
  4. Prompt rewrites sometimes have unintended consequences. Validate that intended behavior is preserved.
  5. Evidence is local to the workload. Re-evaluate changes in offline benchmarks, online experiments, and every product surface where they ship.

None of these changes made the model smarter. They removed work the model never needed to do.

The changes described in this post are shipping across GitHub Copilot experiences that use the same underlying harness.

Bring agentic workflows to your terminal
with GitHub Copilot CLI >

The post How we make AI coding more cost efficient without sacrificing task quality appeared first on The GitHub Blog.

Data Mesh at Grab (Part III): Operationalizing data reliability with automated DPIs

Post Syndicated from Grab Tech original https://engineering.grab.com/data-mesh-at-grab-part-three

Introduction

In the first two parts of this series, we described how Grab approaches data mesh through the Signals Marketplace: a way for teams to publish, discover, and reuse trusted data products across domains. Part II introduced the foundational tools behind certification: Hubble for metadata and ownership, Genchi for data quality observability, and the Data Contract Registry for explicit producer-consumer guarantees.

Certification is the starting point for a trusted data marketplace. It gives downstream consumers confidence in an asset’s ownership, documentation, lineage, and quality controls. Certification does not eliminate runtime failure. A certified table can still arrive late. A certified metric can still be affected by a broken dependency. A certified Kafka stream can still violate a freshness expectation.

Keeping certified data products reliable in production requires more than defining standards upfront. Teams need a consistent way to detect failures, diagnose the root cause, fix the issue, and verify recovery. That is where Data Production Issues (DPIs) come in. At Grab, DPIs turn data quality signals into an operational workflow.

The DPI lifecycle

A good DPI should be clear enough to act on, and it should close automatically when the underlying condition recovers. From the beginning, we designed the DPI lifecycle to be automated, with minimal human-in-the-loop.

The lifecycle starts when Kinabalu, Grab’s incident orchestrator, observes that a data asset may no longer satisfy its contract. The contract captures the reliability expectations that matter for the asset, along with the health checks, exposed through Test Health application programming interfaces (APIs), that evaluate those expectations.

The orchestrator stays decoupled from platform internals. It does not need to know how each platform computes freshness, completeness, or other quality dimensions. It only needs to ask whether the relevant contract tests are healthy. If one or more contract tests are unhealthy, the contract is considered breached, and the DPI lifecycle begins.

Diagram of the automated Data Production Issue workflow from contract-test evaluation through triage, diagnosis, resolution, and close.
Figure 1. Automated DPI workflow.

Triaging DPIs: From alerts to confirmed contract breaches

Data platforms emit many alerts. An Airflow schedule may be delayed, a data quality test may fail, or a pipeline job may exit unexpectedly. These alerts are useful, but they are not automatically DPIs. Triage decides whether an alert represents a real contract breach for a data asset.

As introduced in Part II, a data contract is an explicit, versioned agreement between a data producer and its consumers. It outlines the data’s schema, freshness, completeness, and other semantic guarantees. These guarantees are codified and enforced through data quality tests in Genchi.

When the incident orchestrator evaluates contract tests, it distinguishes an individual test run result from the overall health of a test. A test run can pass or fail at a point in time, but the test itself may only be considered healthy after the underlying issue has been fully resolved. For example, consider a completeness test that checks whether the T-1 daily partition is complete. If the test failed two days ago but passed yesterday and today, the test may still be considered unhealthy until the partition from two days ago has been backfilled and verified as complete.

The orchestrator also deduplicates around the active unhealthy condition. If an asset already has an open DPI for the same breach, new signals update the existing DPI with additional context rather than creating parallel issues. DPIs that share the same underlying root cause can also be grouped. This keeps responders focused on solving the underlying issue rather than chasing a stream of repetitive alerts.

During triage, the workflow also gathers context for the DPI: affected asset, breached contract, unhealthy tests, data interval, and upstream and downstream dependencies. Not every alert becomes a DPI. Triage protects the operational workflow from noise by promoting only meaningful contract breaches into production issues.

Diagnosing DPIs: Assigning owners with root cause analysis (RCA)

Once a DPI is created, the system must answer why the data is unhealthy, who should fix it, and how.

Not every data issue should be assigned to the data asset owner. A data product may be unhealthy because of a platform incident, a failed producing job, or a delayed upstream dependency. Assigning every issue to the asset owner creates unnecessary handoffs and slows down resolution.

This is where the Data Health API matters. It answers the question: “What kind of failure made this asset unhealthy?” The Data Health API keeps the error taxonomy small:

  • UPSTREAM_ERROR: the asset is unhealthy because an upstream dependency is late, failed, or unavailable.
  • PLATFORM_ERROR: the asset is unhealthy because the underlying platform or infrastructure is impaired.
  • JOB_ERROR: the asset is unhealthy because the producing job or pipeline failed.
  • DATA_ERROR: the asset is unhealthy because the produced data violates quality expectations.

The taxonomy is not meant to replace platform-specific diagnostics. The high-level Data Health API gives the orchestrator just enough structure to assign DPIs and manage their lifecycle consistently. An ingestion platform, streaming platform, metrics platform, or machine learning (ML) platform can still maintain detailed internal error catalogs, logs, retry states, and debugging tools. Platforms remain free to evolve their internals, while the incident orchestrator consumes a stable API contract, so the DPI workflow can interoperate across heterogeneous systems.

A simplified Data Health API response might look like this:

Disclaimer: The fields in this API response are mock data generated for demonstration purposes and do not represent real operational metrics.

{
  "assetId": "urn:li:dataset:(urn:li:dataPlatform:hive,schema.table_A,PROD)",
  "healthStatus": "UNHEALTHY",
  "errorCategory": "UPSTREAM_ERROR",
  "context": {
    "upstreamAsset": "urn:li:dataset:(urn:li:dataPlatform:hive,schema.table_B,PROD)",
    "reason": "upstream data has not arrived for the expected data interval."
  },
  "lastCheckedAt": "2026-06-15T08:30:00Z"
}

From this response, the orchestrator can see that table_A is unhealthy because of an upstream dependency rather than a problem in the asset itself. It then traces the active DPI for the upstream asset and links the table_A DPI to that upstream issue. The downstream DPI can inherit the same owner as the upstream DPI, keeping related failures grouped under the team best positioned to resolve the root cause.

The DPI process works only when the issues it raises can be assigned and fixed. If DPIs are frequently noisy, duplicated, or difficult to act on, users will eventually learn to ignore them. Diagnostic accuracy matters because it keeps DPIs useful for the people who receive them. It also creates a forcing function for each data-producing platform to improve its diagnostics. To produce accurate RCA, platforms need to incorporate signals from their dependencies and surrounding systems, not just their own local failure state.

Grab operationalizes DPI diagnosis across its internal data platforms. Our ingestion platform, Hugo, is a primary example of this approach, as outlined in a previous tech blog. Hugo’s intelligent diagnosis architecture uses a three-layered system to automatically detect, analyze, and troubleshoot data pipeline failures within its domain, as shown in Figure 2.

Diagram of Hugo's three-stage diagnosis architecture: signal collection, alert diagnosis, and diagnosis result.
Figure 2. Hugo diagnosis architecture.

Modern data platforms generate alerts from many independent systems. Individually, these signals show only a partial view of a dataset. Hugo consolidates platform-specific signals into a unified diagnostic workflow to pinpoint root causes and recommend pipeline remediations. The diagnosis architecture consists of three stages:

  1. Signal collection collects events from multiple signal sources to build a full view of the dataset and pipeline health.
  2. Alert diagnosis creates a structured alert context, classifies the alert, routes it to the appropriate diagnoser, and identifies the root cause using specialized diagnosis logic.
  3. Diagnosis result persists the structured diagnosis output, including the identified root cause, affected dataset, and recommended fix or action.

For example, when a dataset fails, the workflow orchestrator notifies Hugo with a job failure event. Hugo then routes the alert to its internal diagnostic layer to check for conditions such as upstream database replica lag, storing both the diagnosis and recommended fix alongside the affected dataset.

Decoupling signal ingestion, diagnosis, and result management makes it straightforward to add new signal sources and specialized diagnosers. Immediate RCA removes the need for manual log inspection, which shortens remediation and feeds directly into automated resolution workflows.

Resolving DPIs: Auto-healing first, human judgment when needed

After triage and RCA, the final stage of the DPI lifecycle is resolution. The lifetime of a DPI is a proxy for data downtime: it begins when a contract breach is detected and ends when the affected dataset becomes healthy again. Reducing that window requires more than identifying the correct issue. It also depends on recovering safely and consistently from recurring failure modes.

Many incidents are routine and recoverable, such as transient compute interruptions, database connection timeouts, S3 throttling, or upstream pipelines that are delayed rather than permanently broken. Instead of relying on manual intervention for every incident, Hugo automates recovery for these well-understood failure patterns. Once the diagnosis workflow identifies the root cause, it produces a structured diagnosis result containing the affected dataset, the root cause, and the recommended resolution strategy. The auto-resolution workflow then consumes this result to execute the appropriate remediation automatically. Figure 3 shows Hugo’s auto-resolution architecture in two stages.

Diagram of Hugo's auto-resolution architecture, covering resolution execution plus notification and audit.
Figure 3. Hugo auto-resolution architecture.

  1. Resolution execution applies the recommended resolution strategy, such as retrying a failed job, waiting for an upstream dependency, or executing a custom resolver. After the action completes, the system verifies both pipeline health and data correctness to confirm the issue has been fully resolved. If a failure cannot be resolved safely through automation, such as in cases of data corruption, invalid records, or application code defects, the workflow escalates the incident for human intervention.

  2. Notification and audit records every resolution attempt and its outcome, while notifying the appropriate engineering teams. That record supports operational analysis, auditing, and later improvements to resolution policies.

For example, a dataset may miss its freshness Service Level Agreement (SLA) because the workflow orchestrator becomes temporarily unresponsive and fails to submit the scheduled ingestion job. The diagnosis workflow identifies the incident as a pipeline execution failure and recommends a retry strategy. Hugo automatically retries the job, verifies that the pipeline completes and data health is restored, then logs the recovery and notifies the responsible team. This end-to-end process, from incident detection to resolution, runs automatically without manual intervention.

Hugo closes the loop between detection, diagnosis, and recovery. Rather than stopping at identification, the platform turns diagnosis results into targeted remediation, so routine operational issues can be resolved automatically while preserving human oversight for complex or high-risk incidents. Separating diagnosis from execution also lets new diagnosis capabilities and resolution strategies evolve independently without changing the overall architecture.

The impact is already evident in production. 86.9% of DPI incidents were automatically resolved, significantly reducing manual operational effort. By automating routine recoveries, engineers spend less time performing repetitive operational tasks and more time building new platform capabilities, while overall data downtime is significantly reduced.

Conclusion

Certified data products still need to prove their reliability in production. Freshness delays, upstream failures, platform incidents, and data quality violations can all break consumer trust, even when an asset has already met certification standards.

Automated DPIs are the operating model for managing these failures. By turning contract breaches into structured production issues, the DPI lifecycle makes data reliability operational: triage separates real breaches from alert noise, diagnosis identifies the likely failure domain, ownership routing reduces handoffs, and resolution closes the loop through auto-healing or human intervention when needed.

The most important outcome is not simply that issues are detected faster. It is that data downtime becomes visible, measurable, and reducible. With every DPI tracked from detection to recovery, teams can understand where time is spent, which failure modes repeat, and where automation can safely reduce operational toil. To date, more than 95% of DPIs are raised automatically rather than by humans, with a mean time to resolve (MTTR) that is 6 times faster for automated DPIs than for manually raised ones.

For Grab, this shifts data reliability from reactive firefighting to a managed production workflow. Automated DPIs help keep trusted data products trustworthy after certification, so downstream teams can depend on them with greater confidence.

What’s next

Across the three-blog series, the story is how Grab turns data mesh from an operating principle into an artificial intelligence (AI)-ready foundation for the company.

  • Part I: Building trust through certification. Grab needed the Signals Marketplace because the business had scaled across mobility, deliveries, financial services, and many data-producing domains. The old model of relying on a central Data Engineering team could no longer keep up. Certification became the mechanism for making high-quality data products visible, reusable, and accountable. With clear ownership, data contracts, and measurable adoption, Grab moved more consumption toward trusted assets, reduced duplication, and created stronger incentives for teams to curate the data they publish.

  • Part II: The foundational tools behind certification. Trust becomes operational through platforms. Hubble covers discovery, lineage, ownership, and the certification engine. Genchi runs continuous data quality observability across freshness, completeness, schema, and business-rule checks. The Data Contract Registry formalizes producer-consumer expectations as versioned, enforceable contracts. Combined, these systems keep data certification an actively maintained standard rather than a static label.

  • Part III: Operationalizing data reliability with automated DPIs. Certification tells consumers which data products should be trusted; DPIs keep that trust true in production. Kinabalu evaluates contract breaches, deduplicates noisy alerts, assigns ownership, and tracks recovery. Data Health APIs make RCA portable across platforms, while Hugo’s diagnosis and auto-resolution patterns show how common failures can be remediated faster and with less operational toil. The result is a measurable reduction in time to resolve and a stronger feedback loop back into certification.

The bigger takeaway is that Grab’s data moat is not just the volume of data we have. It is the system that makes our data trustworthy, discoverable, reusable, and continuously reliable. This foundation is what lets us embrace the agentic world: AI agents can search certified assets, reason over contracts and lineage, trust quality signals, detect production issues, draft RCA, and eventually suggest or execute safe remediation. In that world, data reliability becomes a compounding advantage. The better our foundations are, the more confidently Grab can build agentic experiences on top of them.

We would like to thank all the data practitioners across Grab, including engineers and analysts to data scientists and product teams, who have invested in certification, contracts, and data quality to build a solid foundation for AI agents and AI-powered experiences. We are equally grateful for the unwavering sponsorship, strategic guidance, and hands-on support from our leadership (Mohan Krishnan and Nikhil Dwarakanath), without which this long-term data foundation initiative would not have been possible.

Join us

Grab is Southeast Asia’s leading superapp, serving over 900 cities across eight countries (Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam). Through a single platform, millions of users access mobility, delivery, and digital financial services, including ride-hailing, food delivery, payments, lending, and digital banking via GXS Bank and GXBank. Founded in 2012, Grab’s mission is to drive Southeast Asia forward by creating economic empowerment for everyone while delivering sustainable financial performance and positive social impact.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

How we saved 100 terabytes of memory by optimizing 1.1.1.1’s DNS cache

Post Syndicated from Sebastiaan Neuteboom original https://blog.cloudflare.com/dns-cache-memory-optimization-1111/

Big Pineapple, the platform behind 1.1.1.1, Gateway DNS, DNS Firewall, AS112, and several other Cloudflare DNS services, stores over 250 billion DNS cache entries at any given time. At that scale, wasting a single byte per entry costs more than 250 gigabytes of memory across our fleet.

Five successive changes to how cache entries are stored in memory cut the per-entry footprint by over 50%. Across our fleet, these changes freed up roughly 100 terabytes of memory, equivalent to the amount of RAM in 130 of our Gen 13 servers. The cache also got faster. Insert throughput rose 43% and lookup latency dropped 19%, as fewer allocations and better memory locality meant we did not trade speed for space.

What we cache

On cold start, Big Pineapple starts out with an empty cache. As DNS queries arrive, the cache fills until it hits its maximum entry count, at which point we evict older or less popular items to make room.

The exact cache size varies by data center. When EDNS Client Subnet (ECS) is in use, authoritative servers return different answers depending on the client's network, so we cache multiple versions of the same query. This increases both the number of entries and the memory each one consumes, making the optimizations in this post especially impactful for ECS-heavy locations.

Each item in the cache is a key-value pair. The key identifies what was queried:

The value stores the DNS response itself: the answer, authority, and additional record sections, along with metadata like the creation time, a hit counter, and the Time-to-Live (TTL).

Both structs have room for improvement. Several fields use types that carry overhead we don't need once the entry is stored.

Benchmarking memory usage

To measure the impact of each change, we benchmark by filling the cache with randomly generated entries that roughly match the traffic distribution we see in production: 56% A records, 25% AAAA, and 19% TXT. Each entry contains between one and four records.

TXT records serve as a stand-in for all non-A/AAAA record types in the benchmark. Their size is randomized between 64 and 224 bytes, close to the average response size we see for variable-length record types.

We track memory usage using a custom allocator that wraps Rust’s System allocator and records the number and size of allocations per cache entry. Alongside memory, we measure insert throughput and lookup latency across the full cache flow to make sure memory savings don’t come at the cost of performance.

These inputs approximate production rather than reproduce it exactly. Process memory also depends on traffic mix, cache occupancy, allocator state, and memory used outside the cache. We therefore measured resident memory across production instances during the rollout.

The cost of capacity

Vec<T> stores three fields: a pointer to heap-allocated data, the current length, and the total capacity. When you push an item, Vec checks whether the length exceeds the capacity and reallocates if needed. If there’s room, it just appends the item and increments the length.

Once we store a DNS response in the cache, however, we never modify it again. The capacity field serves no purpose, but still costs 8 bytes per Vec. The over-allocated heap space is wasted as well, as a Vec with capacity for eight items but only five stored leaves three slots unused on the heap.

Using Box<[T]> solves both problems. It can’t grow after creation, so it doesn’t need a capacity field or reserve space for future elements. The same applies to String, which also carries a capacity field. Box<str> drops it.

Each cache entry stores 8 Vec and String fields. Replacing them with Box<[T]> and Box<str> saves 8 bytes per field, 64 bytes per entry. It also eliminates the excess heap memory that Vec reserves for future growth. The combined savings add up to over 15 terabytes with over 250 billion cache entries.

Fewer lists, fewer pointers

Rather than storing the answer, authority, and additional sections in separate lists, we can store a single list with offsets to the start of each section. Since DNS record counts per section fit in a u16, we can use a u16 (2 bytes) for each offset, compared to the 8-byte pointer and 8-byte length that each separate Box<[T]> requires.

This removes two lists, each with an 8-byte pointer and 8-byte length, and replaces them with two 2-byte offsets, saving 28 bytes per entry.

These savings do not always map directly to the number of bytes removed from individual fields. Rust inserts padding to satisfy alignment requirements and rounds a struct’s size up to a multiple of its alignment. Removing a small field can therefore eliminate additional padding. For example, we also packed several boolean fields into a single bitflag. This reduced the surrounding padding, causing the struct to shrink by more than the size of the individual booleans.

Dropping the owner

Each DNS record has an owner, the domain the record belongs to. In many cases, this owner is identical to the domain being queried. For example, a query for example.com A returns two records with the same owner:

But when a CNAME is involved, for example, the record owner can differ from the queried domain:

The DNS wire format handles repeated owners using name compression, as defined in RFC 1035. Rather than encoding the same domain twice, subsequent occurrences store a 2-byte pointer to the first occurrence. A domain like www.example.com can encode just www followed by a pointer to where example.com already appeared in the message.

This works well on the wire, but in our cache we store the full owner name alongside each record. Following compression pointers during cache lookups is expensive on the hot path, so we trade memory for speed.

Most records, however, have an owner identical to the queried domain. For those, we can drop the owner entirely and infer it at read time. When the owner differs, such as the A records behind a CNAME, we store the full name.

When owner is None, response construction restores the queried domain from the cache key, avoiding a heap allocation. This means the record is no longer self-contained, but the cache key is already available during every lookup. When the owner differs, Some stores a pointer to the full name on the heap.

In practice, most cached records have an owner identical to the queried domain, so the majority require no heap allocation for the owner field.

Enum sizing

Rust enums are sum types: each variant can carry different data, but the enum is always the size of its largest variant.

Option is either Some and holds a value, or None and holds nothing. Both variants take the same amount of memory. The enum stores a tag indicating the active variant, followed by space large enough for the largest variant’s data. When the variant is None, that space is unused.

For record data, it seems natural to store each DNS record type as an enum variant:

But the enum is always as large as its largest variant. In our case, that’s NAPTR at 136 bytes. It stores three variable-length text fields, a domain name, and two integers. As a result, the full enum, including the variant tag and padding, becomes 144 bytes.

An A record only needs 4 bytes, and an AAAA record needs 16 bytes. A and AAAA make up over 80% of our traffic, so most records waste over 120 bytes on padding. Since a single cache entry can store many records this quickly adds up.

Boxing the variants

To solve this problem, we can box the larger variants of the enum, moving them to a separate heap allocation. The enum then stores an 8-byte pointer to the heap, where the data takes up only the size it actually requires.

For A and AAAA records, this saves 120 bytes per record. Smaller variant types like TXT and CNAME also benefit. They still occupy the 24-byte enum, but their heap allocation is sized to their actual data rather than padded to 144 bytes. NAPTR, the largest variant, actually pays slightly more. It now adds the cost of a heap pointer and allocation overhead. But NAPTR records are rare in practice, so the tradeoff is worth it.

But boxing the larger record variants introduces costs of its own.

The costs of boxing

Boxing has two costs. The first is allocator overhead. Each boxed variant becomes a separate heap allocation, and allocators round up to the nearest size class. Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads. jemalloc groups allocations of similar sizes into fixed-size bins. A TXT record requests 32 bytes and fits exactly into a 32-byte bin, wasting nothing, but an MX record requests 40 bytes and rounds up to 48, wasting 8 bytes.

The second cost is poor memory locality. Without boxing, the record enum values for a cache entry sit in a single contiguous allocation. With boxing, data for each boxed variant lives in a separate heap region. Reading it requires following a pointer, and when that pointer lands far from the rest of the entry, the CPU has to fetch a new cache line. With millions of cache entries, boxed data ends up scattered across the heap rather than packed together.

Neither cost is catastrophic on its own, but eliminating both, as the next section shows, yields a measurable improvement in both memory usage and lookup latency.

Storing records in wire format

An obvious next step would be to store the full DNS response in wire format, patching only per-client fields like the message ID on each lookup. But this has drawbacks. DNSSEC records are only included when the client sets the DO (DNSSEC OK) flag. Storing a complete wire format message means either caching two variants, one with DNSSEC and one without, or filtering them out of an already-built message. There is also a cost to parsing the full message on every lookup, which the enum approach we just described avoids by storing already-parsed records.

As a middle ground, we store just the record data as raw bytes, while keeping the rest of the cache entry as structured fields. Instead of a list of parsed enum variants, we store the records as a single Box<[u8]> containing each record encoded as a 2-byte length prefix followed by its raw bytes.

This eliminates the per-variant enum overhead and the boxed heap allocations from the previous optimization. The data also becomes packed contiguously, which improves CPU cache locality. The tradeoff is that records can no longer be randomly indexed. We have to iterate through the buffer sequentially. This adds some complexity for features like round-robin rotation of A/AAAA records, but since record counts per entry are small, the cost is negligible.

When building a DNS response from cached records, most record types can be copied directly from the buffer into the outgoing message. Previously, each parsed record had to be serialized field by field back into DNS wire format. The new layout skips that work for A, AAAA, TXT, and all DNSSEC record types by copying their encoded bytes directly. Only records containing domain names, such as CNAME, NS, MX, and SOA, still require parsing so we can apply DNS name compression. Since records that support direct copying make up the vast majority of our traffic, this change reduces work on the lookup path. Combined with improved memory locality, this reduced cache lookup latency by 5% in our benchmarks.

To build the record data buffer, we write into a reusable scratchspace buffer that persists across cache insertions. Since previous writes have already grown it, the buffer rarely needs to be reallocated. Records vary in size, so we do not know the exact buffer size until they have been serialized. Once the records are in the scratchspace buffer, we allocate a Box<[u8]> and memcpy the data into it. This replaces the separate allocation for each boxed record with one allocation for all record data. It also avoids the waste from shrinking a Vec<u8>, where the allocator may not be able to reclaim the unused tail of the original allocation. In our benchmark, this change alone increased cache insert throughput by 13%.

The results

The production measurements show how the benchmarked per-entry savings translated to whole-process resident memory. The graph below shows p90, p98, and p99 memory usage across Big Pineapple instances. The first dashed line marks the start of the rollout on May 18, 2026, and the second marks its completion across all services on July 6, 2026. Each release introduced one or more of the optimizations described above, so memory usage dropped in steps rather than all at once.

As each release rolled out, restarted instances began with empty caches and consumed more memory as those caches filled. The stable plateaus therefore represent steady-state memory usage better than the initial dips.

Per-instance memory usage dropped across all percentiles. At p99, memory dropped from 9.3 GB to 5.3 GB, a 43% reduction in resident memory. At p90, memory dropped from 6.5 GB to 3.8 GB, a 42% reduction. Instances with fuller caches saw the largest absolute savings.

In our benchmarks, these five optimizations reduced the per-entry memory footprint from 953 bytes to 420 bytes, a 56% reduction. Per-entry allocations dropped from 1.1 KB to 461 bytes. The reductions measured in production are smaller because resident memory includes the cache alongside all other process data. After the rollouts settled, aggregate working-set memory across the fleet was roughly 100 terabytes lower.

Performance also improved. Cache insert throughput increased by 43%, while lookup latency dropped by 19%.

We plan to reinvest the freed memory into increasing cache capacity without increasing our memory usage, which improves cache hit rates and reduces upstream query volume. We're also exploring further optimizations to the cache itself.

To learn more about Big Pineapple, see How Rust and Wasm power Cloudflare's 1.1.1.1. If you work on DNS or other large systems, share the optimizations that have worked for you in the Cloudflare Community or on the Cloudflare Developers Discord.

Your alt text passes automated checks. That doesn’t mean it’s any good.

Post Syndicated from Taarik Ashenafi original https://github.blog/engineering/user-experience/your-alt-text-passes-automated-checks-that-doesnt-mean-its-any-good/


More than one in four images on the web’s most popular home pages have alt text that’s missing, vague, or copied from adjacent images.

That’s from WebAIM’s 2026 WebAIM Million report, which found that alt text,an HTML attribute containing text describing the content of an image, was missing on 16.2% of images across the top million home pages. Among the images that did have alt text, another 10.8% provided an undescriptive attribute, such as alt="image", a raw filename, or a description duplicated from a neighbor.

While automated tooling reliably flags missing alt text, it isn’t as good at fixing poorly written alt text. Most alt text checkers test whether an accessible name for an image exists, not whether the provided alt text says anything useful about the associated image, and that’s a deliberate design choice: a quality-oriented rule with false positives is a rule teams switch off. So alt="IMG_2847.png" passes. So does the same alt="3/5 stars" on five different star-shaped icons.

We built an alt text plugin for the GitHub Accessibility Scanner to help improve your alt text. This post covers where we drew the line between what a checker can prove and what it can only suspect, why our worst bug turned out to be a layout problem rather than a parsing one, and what changed once we let a model into the loop.

If you’re building automated checks of your own, for accessibility or otherwise, the tradeoffs should transfer.

Proving a string is wrong without seeing the picture

Presence of alt text is an objective fact; the attribute is there or it isn’t. Quality is often a judgment call. A machine can’t prove whether a sentence adequately describes a picture in context from markup.

However, not all quality is subjective. There’s several checks you can perform based on the alt text alone, with no need to consult the image content:

  • The attribute is absent (not empty) or whitespace-only.
  • The alt is a filename, such as hero.png, IMG_2847.jpg.
  • The alt is a placeholder somebody meant to replace, such as TODO, tbd.
  • The alt is one generic word naming the medium instead of the content, such as image, logo, chart.
  • The same alt repeats across adjacent images.

Every one of those is a claim about a string, and that became our dividing line. Five deterministic rules run by default which need no credentials for running AI models or network calls. One opt-in rule calls a model with provided image content and surrounding context, for judgments an alt text string can’t support on its own.

First, we had to determine which images to judge on a scanned webpage. We use Playwright’s role-based locator rather than querySelectorAll('img'), so anything not included in the browser’s accessibility tree drops out, including anything carrying alt="". That last exclusion matters most. An empty alt is the author explicitly saying the image is decorative, and flagging it would punish exactly the behavior you want to encourage.

So, how strict should it be? A quality checker lives or dies on false positives, so we chose closed sets over clever heuristics. The vague-alt rule normalizes a string, then checks it against a curated list of words that carry no information on their own. It fires only on an exact match:

  • alt="image" gets flagged.
  • alt="image of the login screen with the SSO button highlighted" doesn’t.

Rules this literal miss plenty of bad alt text. We took the miss over the false positive, because a reliable checker that developers enable beats one that gets switched off.

Repetition is a layout problem, not a DOM problem

Repeated alt text presented an interesting problem. Picture a row of five star-shaped icons that each say "3/5 stars". A screen reader user hears the same thing five times and learns nothing new from four of them.

Our first version walked the images in document order and flagged any run sharing the same normalized alt. It caught things it shouldn’t have. For example, a footer “GitHub” logo and a header “GitHub” logo might sit next to each other in the extracted list but nowhere near each other on screen, so nobody experiences them as a group.

What matters is where images land on screen, not where they sit in the markup. So the rule now checks page layout, and only extends a run when the gap between two bounding boxes is small compared to the boxes themselves:

const gap = Math.max(horizontalGap, verticalGap) 
const largerDim = Math.max(a.boundingBox.width, a.boundingBox.height, 
                           b.boundingBox.width, b.boundingBox.height) 
return gap > GAP_MULTIPLIER * largerDim

Two details worth noting:

  • The multiplier is a judgment call, not a number we derived from anything. It’s the kind of value you tune against real pages instead of trusting from a spec.
  • When either image has no measurable box, the check fails open and the run continues. A missing finding is invisible; a wrong one isn’t.

Getting a model to act like a reviewer, not a critic

Deterministic rules only need the alt string. Anything smarter needs to know what the page is about, and none of that is tracked by the image element. Whether alt="a smiling person" is fine depends entirely on what surrounds it: on a generic mood shot, it’s probably works. But under a heading where a specific person is named, it doesn’t provide enough detail.

In our optional alt-text-qualitycheck, we extract page context alongside each image: the nearest heading, the page title, any <figcaption>, whether the image sits inside a link or button, and up to 600 characters of nearby prose.

The link signal matters most, because when an image is a link’s only content, its alt becomes the link’s accessible name. The right alt then names the destination instead of describing the picture.

One caution: The plugin only records that an image sits inside a link. We don’t check whether it’s the link’s only content, which is the part that actually turns alt into a link name. So right now both cases look identical to the model.

That context, the alt, and the image go to a vision model through GitHub Models. Our failure modes were rarely the model misreading a picture. They were the model having opinions. Given perfectly good alt text, our first version of the checker would suggest different alt text, because “could this be better?” is a question a language model always answers yes to. Every image becomes a finding, so the signal disappears.

Three changes fixed it:

  • A decision procedure instead of an instruction. The prompt walks four ordered steps, stops at the first that matches, and emits that step’s verdict: decorative, redundant with a caption, functional, or informative.
  • Explicit anti-nitpick rules. Trust the author’s framing. Separate redundant prefixes (“Image of…”) from semantic ones (“Photograph of…”). Treat a short alt as correct when the surrounding prose already analyzes the image.
  • Structured output with a forced field order, so reasoning is generated before verdict and the model has to build an argument before it picks a label.

None of that makes the model unfailingly correct. It makes it consistent enough to iterate against. The repository carries an offline grading harness built from published teaching material: WebAIM, the W3C images tutorial, and POET. The rule and the harness share one prompt, so what you tune offline is what runs in CI. That harness only tests the model’s judgment, though, not the whole pipeline. A case can score perfectly there and never reach the model in a real scan.

Sending images to a model is a privacy and cost decision

The moment a check calls an external model with webpage data, it stops being just a lint rule and requires careful data flow design. A few things follow from that:

  • The rule is off by default. It won’t run unless you deliberately enable it in your plugin configuration, and it needs a token with access to GitHub Models.
  • URLs get redacted. Image URLs and link hrefs often carry signed CDN tokens or session identifiers, so query and fragment are stripped from anything entering the model context or the rule’s error logs. For the same reason, src and srcset are replaced with (omitted) in the markup we send.
  • Everything in that context window is untrusted input. Titles, headings, and prose all come from the page being scanned, and a page can contain text written to steer a model. Structured output constrains the shape of a response, not the reasoning behind it.

One caution, because that list is easy to over-read: findings still carry the real page URL and original HTML into the scanner’s normal reporting pipeline. That’s on purpose, since you can’t fix an image you can’t locate. Redaction narrows what reaches the model and the logs, not what lands in your own issues. And if you set up Azure AI Vision credentials, an optional OCR pre-pass sends image bytes to a second place. Nothing requires Azure, but a data-flow review needs to cover both paths.

Cost follows the same shape. In the common case this is one model call per image per scan, which on an image-heavy site dominates the cost of the whole run. That’s reason enough to put it on a schedule rather than on every commit.

What this still can’t do

  • The deterministic rules are literal. They catch alt text that’s obviously unwritten, not alt text that’s fluent and wrong. They also read the alt attribute rather than the computed accessible name, so an aria-label that fixes the problem won’t stop the finding.
  • The model-backed rule produces false positives. Every finding is a prompt for human attention, not a verdict.
  • Silence isn’t coverage. That rule re-fetches images outside the browser session, so anything behind authentication can fail to load. Fetch and model errors are logged and skipped, which means a page can come back clean because nothing got checked.
  • Suggested alt text is a draft. A model that sees the image and a few nearby words can’t account for your audience, your house style, or the job that image is doing on the whole page.
  • Some findings double up with the scanner’s built-in checks, since our missing-alt rule covers the same ground.
  • We only check HTML <img> tags. SVG, role="img" containers, CSS backgrounds, and canvas aren’t covered yet.
  • This is new code with limited real-world feedback. Rules like these improve when they meet the variety of markup and content found across real sites. This plugin hasn’t had that yet, so treat early findings accordingly.
  • Passing isn’t conformance. Automated checks are a floor. Testing with people who use assistive tech is the goal.

What we’d tell you if you’re building something similar

Separate what you can prove from what you can only suspect, and give them different defaults. Checks that prove something should be cheap, predictable, and on by default. Checks that only suspect something should be opt-in, and should read as a suggestion rather than a verdict. Then, ask what the user experiences rather than what the DOM says. Every gap still open in this plugin has that second shape. We record that an image is inside a link, not that it is the link. We read an attribute, not a computed name.

That distance is the real boundary, and a better model doesn’t close it. Deciding what the functionality of an image is for a user who can’t see it still requires human judgment. What automation buys you is making sure that human is giving the right images a second examination.

Try the alt-text plugin in your accessibility scanning workflow. If it tells you the wrong thing, please report it. Open an issue with the finding and, if public, a link to the affected page.

The post Your alt text passes automated checks. That doesn’t mean it’s any good. appeared first on The GitHub Blog.