This academic year, I am taking a sabbatical from the Kennedy School and Harvard University. (It’s not a real sabbatical—I’m just an adjunct—but it’s the same idea.) I will be spending the Fall 2025 and Spring 2026 semesters at the Munk School at the University of Toronto.
I will be organizing a reading group on AI security in the fall. I will be teaching my cybersecurity policy class in the Spring. I will be working with Citizen Lab, the Law School, and the Schwartz Reisman Institute. And I will be enjoying all the multicultural offerings of Toronto.
Integrating a new storage provider is too complicated. Migrating data, retraining teams, and reconfiguring tools will take too long and create too much risk.
It’s understandable. Data migrations have a reputation for being messy and disruptive. And let’s be honest, nobody wants to babysit infrastructure when there are products to build. For many teams, just the thought of switching cloud providers feels like a detour they don’t have time for.
But in reality, that fear is often bigger than the actual lift. If your workflows already use standard tooling like S3-compatible APIs, switching to a specialized provider is more like a well-marked exit than a hard left turn.
This is the third post in our series debunking persistent myths about cloud storage—check out the first and second posts—and why a best-of-breed, interoperable approach is actually less disruptive than sticking with legacy hyperscaler models.
New Cloud Native Times Call for New Cloud Storage Approaches
Learn more about how the open cloud supports faster development, improved workflows, and reduced cost complexity in our free ebook, “New Cloud Native Times Call for New Cloud Storage Approaches.”
Migration anxiety vs. reality
“Storage migration” can sound like it requires weeks of planning and an army of engineers. But if your apps are already using S3-compatible workflows, most of the heavy lifting is already done.
If you know S3, you’re already ready
Many specialized storage providers now support S3-compatible APIs, allowing teams to keep the tools, scripts, and services they already know, such as Terraform, Kubernetes, ArgoCD, boto3, and MinIO.
And because your teams are already familiar with the S3 API and related tools, retraining isn’t a hurdle. The same skills, scripts, and automation frameworks carry forward, keeping onboarding time minimal. In fact, most teams are surprised by how little they need to change to get started.
That means:
No need to learn a new SDK or storage interface
No retraining your DevOps team
No rewriting automation pipelines or batch jobs
In most cases, all it takes is updating your endpoint URL and refreshing credentials. The mental model stays the same, the tools stay the same, and your workflows continue as-is.
You don’t need to rip and replace
Downtime concerns are one of the biggest sources of hesitation when switching providers. But in practice, migrations to S3-compatible cloud storage providers rarely require full cutovers or risky, all-or-nothing switchovers. With a bit of planning, most teams handle migrations incrementally:
Start by migrating lower-risk datasets, such as backups or archives.
Validate configurations and permissions as data lands in the new system.
Slowly expand to production datasets as confidence grows.
Better yet, you don’t have to move everything at once. Many teams adopt phased transitions, running some buckets side-by-side or writing to both systems during the handoff to minimize risk. With a bit of planning and the right migration tools and guidance, you can keep operations stable while gradually shifting workloads at a comfortable pace.
Metadata isn’t a blocker
Migrating files without metadata continuity can break downstream systems, especially if your applications rely on timestamps or version tracking.
Fear not. S3-compatible cloud storage providers can preserve metadata during migration, including timestamps. That means your historical data stays intact and compliant with internal policies or regulatory needs, and you won’t need to reset or alter your data management policies.
Moving isn’t the risk. Staying locked in is.
Let’s flip the narrative. The real risk isn’t switching; it’s staying stuck.
Major cloud provider ecosystems are designed for lock-in. The deeper you go, the harder it becomes to leave. Features that look like conveniences, such as integrated IAM policies, tiered storage, and custom APIs, often become entanglements over time.
Each of these layers is built to reinforce reliance:
IAM rules tie access tightly to the provider’s own tooling.
Tiered storage creates dependencies on lifecycle rules and retrieval thresholds.
Custom APIs mean even basic storage functions can require provider-specific logic.
And as you expand your usage—adding compute, networking, and security services—everything becomes interdependent. What starts as convenience evolves into constraint. Even small changes to your stack can trigger cascading reviews, system audits, or full reconfigurations.
The result? Innovation slows. Costs creep up. Flexibility disappears.
With a specialized provider, you break that cycle.
Specialized Doesn’t Mean Complicated
Specialized storage doesn’t complicate onboarding. It streamlines it. Solutions like Backblaze B2 are purpose-built to make this shift smooth and sustainable, without the trade-offs or surprises you might expect from switching providers.
S3 compatibility allows for seamless integration with the tools and workflows your team already uses.
Granular control means you can choose the tools and providers that work best for your architecture, not the ones bundled into a vendor’s ecosystem.
Metadata continuity is supported through features like custom upload timestamps, preserving file context during migration.
Transparent pricing ensures there are no hidden egress fees, transaction charges, or retention penalties to catch you off guard.
Hands-on support helps you plan, validate, and scale your migration with confidence and minimal disruption.
Breaking out of a single-vendor ecosystem may feel intimidating, but it’s often the fastest way to simplify operations, improve performance, and regain control over your cloud strategy.
The best part? Once you’ve made the move, you’re free to experiment. Multi-cloud strategies become more accessible. Your architecture becomes more modular. And your team can focus on building, not babysitting infrastructure.
Next Up: In the final post in this series, we’ll tackle Myth #4: Managing multiple clouds is complicated. (Spoiler: It doesn’t have to be.)
The Microdot
web framework is quite small, as its name would imply; it supports both
standard CPython and MicroPython,
so it can be used on systems ranging from internet-of-things (IoT) devices
all the way up to large, cloudy servers. It was developed by Miguel
Grinberg, who gave a presentation about it at EuroPython 2025. His name
may sound familiar from his well-known Flask
Mega-Tutorial, which has introduced many to the Flask lightweight Python-based
web framework. It should come as no surprise, then, that Microdot is
inspired by its rather larger cousin, so Flask enthusiasts will find much
to like in Microdot—and will come up to speed quickly should their needs turn
toward smaller systems.
Преди две години – на 26 юли 2023-та в 14 часа на магистрала Тракия се обръща камион пълен със скрап. По чудо няма пострадали други коли. Бързо се разбира, че камионът е на Господин Вълев Вълев по-известен като Динко. В първите 24 часа след случая медиите основно тикаха микрофони пред лицето му и чакаха някоя институция да каже нещо. Тъй като никой не се зае, аз изрових всичко налично от публични източници и обобщих нарушенията във Facebook. Отне ми 20 мин. на телефона. След това ги описах подробно в статия, която беше прекопирана и цитирана от доста медии. Оказа се, че не само катастрофиралия, но и много от камионите му нямат технически преглед, гражданска и/или са пътували без винетка, което обикновено е сигнал за по-сериозни нарушения.
17 часа след поста ми във Facebook и два часа след статията в блога Министерството на транспорта и съобщенията излезе с новина, че започват проверка на фирмите на Динко и тази, към която е регистриран камиона. Последваха реакции и информация, че ще му отнемат лиценза заради множество нарушения.
В следващите две години не се чу нищо повече за случая. Динко излезе няколко пъти в медиите за други неща, но не стана ясно нито какво се случва с проверката, нито дали има наистина последствия. С оглед на постоянно изникващите проблеми по пътя, данните за жертви, популистки недомислени изказвания при поредния инцидент и призивите за реални действия реших да проверя къде е лицето Вълев и какво се случва с бизнеса му. Търся отговор на въпроса какво са правили институциите, взели ли са мерки, тези мерки имали ли са ефект и дали поведението на лицето се е променило.
Мисля, че знаете отговора, но нека започнем поред.
Камионите на Динко
В статии от 2023-та се говори, че Вълев има 12 камиона към фирмата си „Бозав“ ЕООД. Обърналият се камион с номер A1256HH е на „Логтранс БГ“ ЕООД. към днешна дата виждаме 10 камиона с данни, че на фирмата на Вълев и още 6, които най-малкото използва, а вероятно също са негова собственост. Така стават 16 камиона с различни конфигурации.
9 от тях виждаме в разрешителни за извършване на пътна помощ актуално към настоящия момент още два се намират в разрешителното от 2022. От първите 9 само 3 са с актуален технически преглед. Един от другите 6 му изтече техническия преглед вчера, а други три не са с ГТП от поне три месеца. Въпреки това, Вълев продължава да им плаща гражданска отговорност и присъстват в регистъра за извършване на пътна помощ.
Два от камионите извършващи пътна дейност заедно с още 7 се намират в регистрациите на „Бозав“ ЕООД за превоз на отпадъци от РИОСВ Стара Загора за цялата страна. Беше получил такова три месеца преди инцидента, след което около седем месеца нямаше разрешение. Издали са му обаче ново през февруари 2024-та. В него се споменават шест камиона, включително катастрофиралия през 2023-та – A1256HH. Към онзи момент той няма нито гражданска отговорност, нито ГТП. Това не пречи да бъде включен във всички разрешения след това – през май и октомври 2024-та и последно януари 2025-та. Други камиони се сменят в тези разрешения, но последното от началото на тази година е интересно, защото от 7 камиона споменати в него, три са без валиден технически преглед и продължават да са независимо, че им се плаща гражданска.
Фирмата на Вълев има и разрешения за третиране на отпадъци. Обновяват му го месец преди инцидента през 2023-та. Въпреки заявките на институциите, че ще му отнемат лицензите за нарушения, получава ново само два месеца след катастрофата – на 20-ти септември 2023-та. Следващото е издадено на 15-ти януари 2024-та. Тук интересното е, че на 25-ти април излиза решение на РИОСВ Стара Загора за отнемане на издаденото през януари и от тогава няма издадено следващо.
Може да ги пренася обаче да пренася отпадъци из цялата страна, макар към днешна дата да е законно да го прави само с два от седемте в разрешението му. Останалите, аналогично на катастрофиралия през 2023-та камион, нямат сведения да са технически изправни.
Пътните такси на Динко
Доколкото задълженията към хазната да не влияят пряко на пътните инциденти, сведения са за отношението на човек и бизнес към закона и ефектът на някакви наложени санкции. В статията ми преди две години писах, че откривам над 11 хиляди лева натрупани глоби към тол агенцията от 2021-ва насам. Към днешна дата тази сума надхвърля 24 хиляди лева. Повечето от натрупаните от тогава са по магистралата за Бургас при Ветрен от камион Y4443AH през септември 2024-та.
Горе виждате обновена карта къде са глобявани 13 камиона използвани от Вълев. В червено са 17-те глоби от 2023-та насам. В тъмно червено е мястото с най-много глоби – 13. В зелено е обновен списък от тези преди инцидента преди две години с други негови камиони, които открих.
Ще се фокусирам върху конкретен случай. Споменатият по-горе камион Y4443AH има наложена санкция за каране без маршрутна карта или валидна тол декларация на 24-ти октомври 2024-та. Глобата е 540 лв., които според сайта на тол агенцията не са платени. Притеснителното обаче е, че техническият преглед на този камион е изтекъл пет дни по-рано – на 19-ти октомври. Глобата е сведение, че се е движил по магистралата незаконно подобно на катасофиралия камион през 2023-та. Нещо повече – вероятно е превозвал пак по същия начин отпадъци, тъй като е включен в разрешителното за това три седмици по-рано – на 3-ти октомври.
Делата на Динко
Открих 34 дела, в които фирмата на Вълев е страна. В 24 от тях той обжалва решения на институции или е страна. 16 от делата са заведени след инцидента през 2023-та.
6 от тях се отнасят до негово заведение в с. Прилеп, Сингуляре. Четири от тях са нарушения в трудовото законодателство – работа без договор или неизплащане на двойна заплата през празниците. Две са за изхвърляне на отпадни води от заведението, включително незаконно изграждане на канал за целта към близката река. В едно документите към едно от делата в Районен съд Карнобат се споменава и незаконно строителство на помещение за хазарт към заведението. Не може да знаем дали е имало слот машини или друг хазарт, но фирмата на Вълев, която е собственик на земята и заведението, не е в никой от регистрите за хазартна дейност.
Останалите 10 дела са заведени от Вълев като обжалва актове. Някои от края на 2023-та са за безопасност на труда и трудовото законодателство. Има няколко за нарушения на ЗУО – липсваща ограда, обезопасяване, незаконни площадки и горене на кабели. Всъщност от последните са и потвърдени най-големите глоби – общо 24 хиляди лв.
5 от делата падат в съда – например за незаконна пътна връзка. Според съда е бил глобен вече през 2019-та за това и в новия акт от края на 2023-та не е упоменато дали пътят е бил направен отново или просто не са го накарали да махне стария. Други актове падат, защото според съда в тях се говори за множество различни нарушения, за които трябвало да се съставят отделни актове. При някои обжалвания намаляват глобите.
Общо след делата заведени от Вълев между октомври 2023 и януари 2025-та са потвърдени глоби за 30000 лв. Осъден е да плати адвокатски разходи за 1270 лв., но и различни институции са осъдени да му платят 6780 лв. по хонорари. Така нето за над година трябва да плати на държавата около 24490 лв. Толкова дължи и в глоби за липса на тол такси и винетки. Не знаем дали е имал и други актове или с тези дела се изчерпва всичко около него.
Институциите и Динко
Докато горе ставаше дума за съдебните дела, има обществена информация и за държанието на Вълев. Преди само три седмици излезе запис за опасно неправилно изпреварване с джипа му, при което за малко да смаже моторист. Цената на пикапа е сериозна и дава да се разбере колко смехотворни са глобите, които му се налагат за постоянни нарушения на трудовото и екологичното законодателство, както и безопасността на труда и движението по пътищата. За същия номер, впрочем има глоби за липса на винетка през 2021 и 2022-ра.
През 2024-та пък излезе информация, че го разследват за участие в престъпна група за изнудвания. Не стана ясно дали е имало резултат от това обвинение или и това разследване е потънало или оставено на трупчета за удобен момент, например при следващ инцидент подобен на онзи през 2023-та. Скоро след инцидента през 2023-та, впрочем, го бяха арестували за агресия спрямо просещ човек. Не стана ясно дали е бил съден или дори глобен – само, че излиза срещу гаранция.
Макар да нямаме информация за всички проверки и актове към бизнеса и личността на Вълев, от заведените дела и описаните факти в протоколи и решения е видно, че институциите са се активизирали сериозно след инцидента. Част от тези актове падат, тъй като не са оформени правилно, санкциите са били прекалени или нарушенията са били отколешни. Забелязва се обаче един вид кампанийно стоварване на регулаторния чук върху Динко.
Какво от това?
От останалата публична информация обаче виждаме, че това не му е повлияло по никакъв начин. Надали има изненадани, но в крайна сметка това е идеята на глобите и проверките. Не помага фактът, че изглежда проверки преди инцидента, както и година след него явно няма. Нарушения като незаконно строителство, горене на отпадъци и проблеми с безопасността и трудовите правоотношения са масови и отколешни и това изглежда не е изключение за бизнеса на Вълев. Достатъчно е да се сниши за известно време и да продължава по старо му. Поне до следващия обърнат тир и навярно размазана жертва по пътя.
Направих си труда да извлека цялата тази информация не за да говоря за една банално патологична личност, а за изводите, които може да си направим за институциите ни и как реагираме на тази патология, нарушения и откровени престъпления. След като прахта улегне, катастрофата се разчисти, дъжда измие кръвта, следват няколко проверки – бланкетни по думите на един съдия, няколко акта, няколко дела, няколко шумни заглавия, няколко нескопосани обвинения връщани на прокурорите години наред за фрапантни грешки и забравяме до следващия път.
Това стана с Динко и оная катастрофа. Това продължава да прави той до ден днешен като лакмус за институциите ни. Това се случва с парашутите, катастрофите, болниците ни, корупцията, обществените поръчки и прочие. Това с тира му беше просто по-очевидно. Всичко това е неизбежен ефект от скритото кадруване в регулаторите, прокуратурата и съдебната система, овладяването им до превръщане в бухалка, а не нещо, което помага на хората. Затова Пеевски и Борисов не искаха конкурси за шефове на регулаторите, спирането на командироването на прокурори и промени във ВСС. Затова беше блокиран одита и създаване на публична, проследима и истински случайно разпределение на делата. Затова Сарафов е все още главен прокурор, а Кошлуков е в БНТ.
Защото не е важно дали регулаторът пише неграмотен акт, прокурорът – нескопосано и често противозаконно обвинение и съдията – решение противоречащо на европейски закони, конституция и десетилетия практика. Важното им е тези да се пишат при команда когато някой трябва да се удари. Това се случва масово в местната власт из страната. Случва се отдавна в бизнеси особено в малките населени места. Затова хора като Динко просъществува и се държи така – защото той е наш проблем и утре той или поредния като него ще убие нашите деца. Овладените регулатори и прокуратура са заети са обслужват други.
For over two decades, we’ve built real-time communication on the Internet using a patchwork of specialized tools. RTMP gave us ingest. HLS and DASH gave us scale. WebRTC gave us interactivity. Each solved a specific problem for its time, and together they power the global streaming ecosystem we rely on today.
But using them together in 2025 feels like building a modern application with tools from different eras. The seams are starting to show—in complexity, in latency, and in the flexibility needed for the next generation of applications, from sub-second live auctions to massive interactive events. We’re often forced to make painful trade-offs between latency, scale, and operational complexity.
Today Cloudflare is launching the first Media over QUIC (MoQ) relay network, running on every Cloudflare server in datacenters in 330+ cities. MoQ is an open protocol being developed at the IETF by engineers from across the industry—not a proprietary Cloudflare technology. MoQ combines the low-latency interactivity of WebRTC, the scalability of HLS/DASH, and the simplicity of a single architecture, all built on a modern transport layer. We’re joining Meta, Google, Cisco, and others in building implementations that work seamlessly together, creating a shared foundation for the next generation of real-time applications on the Internet.
An evolutionary ladder of compromise
To understand the promise of MoQ, we first have to appreciate the history that led us here—a journey defined by a series of architectural compromises where solving one problem inevitably created another.
The RTMP era: Conquering latency, compromising on scale
In the early 2000s, RTMP (Real-Time Messaging Protocol) was a breakthrough. It solved the frustrating “download and wait” experience of early video playback on the web by creating a persistent, stateful TCP connection between a Flash client and a server. This enabled low-latency streaming (2-5 seconds), powering the first wave of live platforms like Justin.tv (which later became Twitch).
But its strength was its weakness. That stateful connection, which had to be maintained for every viewer, was architecturally hostile to scale. It required expensive, specialized media servers and couldn’t use the commodity HTTP-based Content Delivery Networks (CDNs) that were beginning to power the rest of the web. Its reliance on TCP also meant that a single lost packet could freeze the entire stream—a phenomenon known as head-of-line blocking—creating jarring latency spikes. The industry retained RTMP for the “first mile” from the camera to servers (ingest), but a new solution was needed for the “last mile” from servers to your screen (delivery).
The HLS & DASH era: Solving for scale, compromising on latency
The catalyst for the next era was the iPhone’s rejection of Flash. In response, Apple created HLS (HTTP Live Streaming). HLS, and its open-standard counterpart MPEG-DASH abandoned stateful connections and treated video as a sequence of small, static files delivered over standard HTTP.
This enabled much greater scalability. By moving to the interoperable open standard of HTTP for the underlying transport, video could now be distributed by any web server and cached by global CDNs, allowing platforms to reach millions of viewers reliably and relatively inexpensively. The compromise? A significant trade-off in latency. To ensure smooth playback, players needed to buffer at least three video segments before starting. With segment durations of 6-10 seconds, this baked 15-30 seconds of latency directly into the architecture.
While extensions like Low-Latency HLS (LL-HLS) have more recently emerged to achieve latencies in the 3-second range, they remain complex patches fighting against the protocol’s fundamental design. These extensions introduce a layer of stateful, real-time communication—using clever workarounds like holding playlist requests open—that ultimately strain the stateless request-response model central to HTTP’s scalability and composability.
The WebRTC Era: Conquering conversational latency, compromising on architecture
In parallel, WebRTC (Web Real-Time Communication) emerged to solve a different problem: plugin-free, two-way conversational video with sub-500ms latency within a browser. It worked by creating direct peer-to-peer (P2P) media paths, removing central servers from the equation.
But this P2P model is fundamentally at odds with broadcast scale. In a mesh network, the number of connections grows quadratically with each new participant (the “N-squared problem”). For more than a handful of users, the model collapses under the weight of its own complexity. To work around this, the industry developed server-based topologies like the Selective Forwarding Unit (SFU) and Multipoint Control Unit (MCU). These are effective but require building what is essentially a private, stateful, real-time CDN—a complex and expensive undertaking that is not standardized across infrastructure providers.
This journey has left us with a fragmented landscape of specialized, non-interoperable silos, forcing developers to stitch together multiple protocols and accept a painful three-way tension between latency, scale, and complexity.
Introducing MoQ
This is the context into which Media over QUIC (MoQ) emerges. It’s not just another protocol; it’s a new design philosophy built from the ground up to resolve this historical trilemma. Born out of an open, community-driven effort at the IETF, MoQ aims to be a foundational Internet technology, not a proprietary product.
Its promise is to unify the disparate worlds of streaming by delivering:
Sub-second latency at broadcast scale: Combining the latency of WebRTC with the scale of HLS/DASH and the simplicity of RTMP.
Architectural simplicity: Creating a single, flexible protocol for ingest, distribution, and interactive use cases, eliminating the need to transcode between different technologies.
Transport efficiency: Building on QUIC, a UDP based protocol to eliminate bottlenecks like TCP head-of-line blocking.
The initial focus was “Media” over QUIC, but the core concepts—named tracks of timed, ordered, but independent data—are so flexible that the working group is now simply calling the protocol “MoQ.” The name reflects the power of the abstraction: it’s a generic transport for any real-time data that needs to be delivered efficiently and at scale.
MoQ is now generic enough that it’s a data fanout or pub/sub system, for everything from audio/video (high bandwidth data) to sports score updates (low bandwidth data).
A deep dive into the MoQ protocol stack
MoQ’s elegance comes from solving the right problem at the right layer. Let’s build up from the foundation to see how it achieves sub-second latency at scale.
The choice of QUIC as MoQ’s foundation isn’t arbitrary—it addresses issues that have plagued streaming protocols for decades.
By building on QUIC (the transport protocol that also powers HTTP/3), MoQ solves some key streaming problems:
No head-of-line blocking: Unlike TCP where one lost packet blocks everything behind it, QUIC streams are independent. A lost packet on one stream (e.g., an audio track) doesn’t block another (e.g., the main video track). This alone eliminates the stuttering that plagued RTMP.
Connection migration: When your device switches from Wi-Fi to cellular mid-stream, the connection seamlessly migrates without interruption—no rebuffering, no reconnection.
Fast connection establishment: QUIC’s 0-RTT resumption means returning viewers can start playing instantly.
Baked-in, mandatory encryption: All QUIC connections are encrypted by default with TLS 1.3.
The core innovation: Publish/subscribe for media
With QUIC solving transport issues, MoQ introduces its key innovation: treating media as subscribable tracks in a publish/subscribe system. But unlike traditional pub/sub, this is designed specifically for real-time media at CDN scale.
Instead of complex session management (WebRTC) or file-based chunking (HLS), MoQ lets publishers announce named tracks of media that subscribers can request. A relay network handles the distribution without needing to understand the media itself.
How MoQ organizes media: The data model
Before we see how media flows through the network, let’s understand how MoQ structures it. MoQ organizes data in a hierarchy:
Tracks: Named streams of media, like “video-1080p” or “audio-english”. Subscribers request specific tracks by name.
Groups: Independently decodable chunks of a track. For video, this typically means a GOP (Group of Pictures) starting with a keyframe. New subscribers can join at any Group boundary.
Objects: The actual packets sent on the wire. Each Object belongs to a Track and has a position within a Group.
This simple hierarchy enables two capabilities:
Subscribers can start playback at Group boundaries without waiting for the next keyframe
Relays can forward Objects without parsing or understanding the media format
The network architecture: From publisher to subscriber
MoQ’s network components are also simple:
Publishers: Announce track namespaces and send Objects
Subscribers: Request specific tracks by name
Relays: Connect publishers to subscribers by forwarding immutable Objects without parsing or transcoding the media
A Relay acts as a subscriber to receive tracks from upstream (like the original publisher) and simultaneously acts as a publisher to forward those same tracks downstream. This model is the key to MoQ’s scalability: one upstream subscription can fan out to serve thousands of downstream viewers.
The MoQ Stack
MoQ’s architecture can be understood as three distinct layers, each with a clear job:
The Transport Foundation (QUIC or WebTransport): This is the modern foundation upon which everything is built. MoQT can run directly over raw QUIC, which is ideal for native applications, or over WebTransport, which is required for use in a web browser. Crucially, theWebTransport protocol and its correspondingW3C browser API make QUIC’s multiplexed reliable streams and unreliable datagrams directly accessible to browser applications. This is a game-changer. Protocols like SRT may be efficient, but their lack of native browser support relegates them to ingest-only roles. WebTransport gives MoQ first-class citizenship on the web, making it suitable for both ingest and massive-scale distribution directly to clients.
The MoQT Layer: Sitting on top of QUIC (or WebTransport), the MoQT layer provides the signaling and structure for a publish-subscribe system. This is the primary focus of the IETF working group. It defines the core control messages—like ANNOUNCE, and SUBSCRIBE—and the basic data model we just covered. MoQT itself is intentionally spartan; it doesn’t know or care if the data it’s moving is H.264 video, Opus audio, or game state updates.
The Streaming Format Layer: This is where media-specific logic lives. A streaming format defines things like manifests, codec metadata, and packaging rules. WARP is one such format being developed alongside MoQT at the IETF, but it isn’t the only one. Another standards body, like DASH-IF, could define a CMAF-based streaming format over MoQT. A company that controls both original publisher and end subscriber can develop its own proprietary streaming format to experiment with new codecs or delivery mechanisms without being constrained by the transport protocol.
This separation of layers is why different organizations can build interoperable implementations while still innovating at the streaming format layer.
End-to-End Data Flow
Now that we understand the architecture and the data model, let’s walk through how these pieces come together to deliver a stream. The protocol is flexible, but a typical broadcast flow relies on the ANNOUNCE and SUBSCRIBE messages to establish a data path from a publisher to a subscriber through the relay network.
Here is a step-by-step breakdown of what happens in this flow:
Initiating Connections: The process begins when the endpoints, acting as clients, connect to the relay network. The Original Publisher initiates a connection with its nearest relay (we’ll call it Relay A). Separately, an End Subscriber initiates a connection with its own local relay (Relay B). These endpoints perform a SETUP handshake with their respective relays to establish a MoQ session and declare supported parameters.
Announcing a Namespace: To make its content discoverable, the Publisher sends an ANNOUNCE message to Relay A. This message declares that the publisher is the authoritative source for a given track namespace. Relay A receives this and registers in a shared control plane (a conceptual database) that it is now a source for this namespace within the network.
Subscribing to a Track: When the End Subscriber wants to receive media, it sends a SUBSCRIBE message to its relay, Relay B. This message is a request for a specific track name within a specific track namespace.
Connecting the Relays: Relay B receives the SUBSCRIBE request and queries the control plane. It looks up the requested namespace and discovers that Relay A is the source. Relay B then initiates a session with Relay A (if it doesn’t already have one) and forwards the SUBSCRIBE request upstream.
Completing the Path and Forwarding Objects: Relay A, having received the subscription request from Relay B, forwards it to the Original Publisher. With the full path now established, the Publisher begins sending the Objects for the requested track. The Objects flow from the Publisher to Relay A, which forwards them to Relay B, which in turn forwards them to the End Subscriber. If another subscriber connects to Relay B and requests the same track, Relay B can immediately start sending them the Objects without needing to create a new upstream subscription.
An Alternative Flow: The PUBLISH Model
More recent drafts of the MoQ specification have introduced an alternative, push-based model using a PUBLISH message. In this flow, a publisher can effectively ask for permission to send a track’s objects to a relay without waiting for a SUBSCRIBE request. The publisher sends a PUBLISH message, and the relay’s PUBLISH_OK response indicates whether it will accept the objects. This is particularly useful for ingest scenarios, where a publisher wants to send its stream to an entry point in the network immediately, ensuring the media is available the instant the first subscriber connects.
Advanced capabilities: Prioritization and congestion control
MoQ’s benefits really shine when networks get congested. MoQ includes mechanisms for handling the reality of network traffic. One such mechanism is Subgroups.
Subgroups are subdivisions within a Group that effectively map directly to the underlying QUIC streams. All Objects within the same Subgroup are generally sent on the same QUIC stream, guaranteeing their delivery order. Subgroup numbering also presents an opportunity to encode prioritization: within a Group, lower-numbered Subgroups are considered higher priority.
This enables intelligent quality degradation, especially with layered codecs (e.g. SVC):
Subgroup 0: Base video layer (360p) – must deliver
Subgroup 1: Enhancement to 720p – deliver if bandwidth allows
Subgroup 2: Enhancement to 1080p – first to drop under congestion
When a relay detects congestion, it can drop Objects from higher-numbered Subgroups, preserving the base layer. Viewers see reduced quality instead of buffering.
The MoQ specification defines a scheduling algorithm that determines the order for all objects that are “ready to send.” When a relay has multiple objects ready, it prioritizes them first by group order (ascending or descending) and then, within a group, by subgroup id. Our implementation supports the group order preference, which can be useful for low-latency broadcasts. If a viewer falls behind and its subscription uses descending group order, the relay prioritizes sending Objects from the newest “live” Group, potentially canceling unsent Objects from older Groups. This can help viewers catch up to the live edge quickly, a highly desirable feature for many interactive streaming use cases. The optimal strategies for using these features to improve QoE for specific use cases are still an open research question. We invite developers and researchers to use our network to experiment and help find the answers.
Implementation: building the Cloudflare MoQ relay
Theory is one thing; implementation is another. To validate the protocol and understand its real-world challenges, we’ve been building one of the first global MoQ relay networks. Cloudflare’s network, which places compute and logic at the edge, is very well suited for this.
Our architecture connects the abstract concepts of MoQ to the Cloudflare stack. In our deep dive, we mentioned that when a publisher ANNOUNCEs a namespace, relays need to register this availability in a “shared control plane” so that SUBSCRIBE requests can be routed correctly. For this critical piece of state management, we use Durable Objects.
When a publisher announces a new namespace to a relay in, say, London, that relay uses a Durable Object—our strongly consistent, single-threaded storage solution—to record that this namespace is now available at that specific location. When a subscriber in Paris wants a track from that namespace, the network can query this distributed state to find the nearest source and route the SUBSCRIBE request accordingly. This architecture builds upon the technology we developed for Cloudflare’s real-time services and provides a solution to the challenge of state management at a global scale.
An Evolving Specification
Building on a new protocol in the open means implementing against a moving target. To get MoQ into the hands of the community, we made a deliberate trade-off: our current relay implementation is based on a subset of the features defined in draft-ietf-moq-transport-07. This version became a de facto target for interoperability among several open-source projects and pausing there allowed us to put effort towards other aspects of deploying our relay network.
This draft of the protocol makes a distinction between accessing “past” and “future” content. SUBSCRIBE is used to receive future objects for a track as they arrive—like tuning into a live broadcast to get everything from that moment forward. In contrast, FETCH provides a mechanism for accessing past content that a relay may already have in its cache—like asking for a recording of a song that just played.
Both are part of the same specification, but for the most pressing low-latency use cases, a performant implementation of SUBSCRIBE is what matters most. For that reason, we have focused our initial efforts there and have not yet implemented FETCH.
This is where our roadmap is flexible and where the community can have a direct impact. Do you need FETCH to build on-demand or catch-up functionality? Or is more complete support for the prioritization features within SUBSCRIBE more critical for your use case? The feedback we receive from early developers will help us decide what to build next.
As always, we will announce our updates and changes to our implementation as we continue with development on our developer docs pages.
Kick the tires on the future
We believe in building in the open and interoperability in the community. MoQ is not a Cloudflare technology but a foundational Internet technology. To that end, the first demo client we’re presenting is an open source, community example.
Even though this is a preview release, we are running MoQ relays at Cloudflare’s full scale, like we do every production service. This means every server that is part of the Cloudflare network in more than 330 cities is now a MoQ relay.
We invite you to experience the “wow” moment of near-instant, sub-second streaming latency that MoQ enables. How would you use a protocol that offers the speed of a video call with the scale of a global broadcast?
Interoperability
We’ve been working with others in the IETF WG community and beyond on interoperability of publishers, players and other parts of the MoQ ecosystem. So far, we’ve tested with:
The Internet’s media stack is being refactored. For two decades, we’ve been forced to choose between latency, scale, and complexity. The compromises we made solved some problems, but also led to a fragmented ecosystem.
MoQ represents a promising new foundation—a chance to unify the silos and build the next generation of real-time applications on a scalable protocol. We’re committed to helping build this foundation in the open, and we’re just getting started.
MoQ is a realistic way forward, built on QUIC for future proofing, easier to understand than WebRTC, compatible with browsers unlike RTMP.
The protocol is evolving, the implementations are maturing, and the community is growing. Whether you’re building the next generation of live streaming, exploring real-time collaboration, or pushing the boundaries of interactive media, consider whether MoQ may provide the foundation you need.
Availability and pricing
We want developers to start building with MoQ today. To make that possible MoQ at Cloudflare is in tech preview – this means it’s available free of charge for testing (at any scale). Visit our developer homepage for updates and potential breaking changes.
Indie developers and large enterprises alike ask about pricing early in their adoption of new technologies. We will be transparent and clear about MoQ pricing. In general availability, self-serve customers should expect to pay 5 cents/GB outbound with no cost for traffic sent towards Cloudflare.
Enterprise customers can expect usual pricing in line with regular media delivery pricing, competitive with incumbent protocols. This means if you’re already using Cloudflare for media delivery, you should not be wary of adopting new technologies because of cost. We will support you.
If you’re interested in partnering with Cloudflare in adopting the protocol early or contributing to its development, please reach out to us at [email protected]! Engineers excited about the future of the Internet are standing by.
Think of the Web as a digital territory with its own social contract. In 2014, Tim Berners-Lee called for a “Magna Carta for the Web” to restore the balance of power between individuals and institutions. This mirrors the original charter’s purpose: ensuring that those who occupy a territory have a meaningful stake in its governance.
Web 3.0—the distributed, decentralized Web of tomorrow—is finally poised to change the Internet’s dynamic by returning ownership to data creators. This will change many things about what’s often described as the “CIA triad” of digital security: confidentiality, integrity, and availability. Of those three features, data integrity will become of paramount importance.
When we have agency in digital spaces, we naturally maintain their integrity—protecting them from deterioration and shaping them with intention. But in territories controlled by distant platforms, where we’re merely temporary visitors, that connection frays. A disconnect emerges between those who benefit from data and those who bear the consequences of compromised integrity. Like homeowners who care deeply about maintaining the property they own, users in the Web 3.0 paradigm will become stewards of their personal digital spaces.
This will be critical in a world where AI agents don’t just answer our questions but act on our behalf. These agents may execute financial transactions, coordinate complex workflows, and autonomously operate critical infrastructure, making decisions that ripple through entire industries. As digital agents become more autonomous and interconnected, the question is no longer whether we will trust AI but what that trust is built upon. In the new age we’re entering, the foundation isn’t intelligence or efficiency—it’s integrity.
What Is Data Integrity?
In information systems, integrity is the guarantee that data will not be modified without authorization, and that all transformations are verifiable throughout the data’s life cycle. While availability ensures that systems are running and confidentiality prevents unauthorized access, integrity focuses on whether information is accurate, unaltered, and consistent across systems and over time.
It’s a new idea. The undo button, which prevents accidental data loss, is an integrity feature. So is the reboot process, which returns a computer to a known good state. Checksums are an integrity feature; so are verifications of network transmission. Without integrity, security measures can backfire. Encrypting corrupted data just locks in errors. Systems that score high marks for availability but spread misinformation just become amplifiers of risk.
All IT systems require some form of data integrity, but the need for it is especially pronounced in two areas today. First: Internet of Things devices interact directly with the physical world, so corrupted input or output can result in real-world harm. Second: AI systems are only as good as the integrity of the data they’re trained on, and the integrity of their decision-making processes. If that foundation is shaky, the results will be too.
Integrity manifests in four key areas. The first, input integrity, concerns the quality and authenticity of data entering a system. When this fails, consequences can be severe. In 2021, Facebook’s global outage was triggered by a single mistaken command—an input error missed by automated systems. Protecting input integrity requires robust authentication of data sources, cryptographic signing of sensor data, and diversity in input channels for cross-validation.
The second issue is processing integrity, which ensures that systems transform inputs into outputs correctly. In 2003, the U.S.-Canada blackout affected 55 million people when a control-room process failed to refresh properly, resulting in damages exceeding US $6 billion. Safeguarding processing integrity means formally verifying algorithms, cryptographically protecting models, and monitoring systems for anomalous behavior.
Storage integrity covers the correctness of information as it’s stored and communicated. In 2023, the Federal Aviation Administration was forced to halt all U.S. departing flights because of a corrupted database file. Addressing this risk requires cryptographic approaches that make any modification computationally infeasible without detection, distributed storage systems to prevent single points of failure, and rigorous backup procedures.
Finally, contextual integrity addresses the appropriate flow of information according to the norms of its larger context. It’s not enough for data to be accurate; it must also be used in ways that respect expectations and boundaries. For example, if a smart speaker listens in on casual family conversations and uses the data to build advertising profiles, that action would violate the expected boundaries of data collection. Preserving contextual integrity requires clear data-governance policies, principles that limit the use of data to its intended purposes, and mechanisms for enforcing information-flow constraints.
As AI systems increasingly make critical decisions with reduced human oversight, all these dimensions of integrity become critical.
The Need for Integrity in Web 3.0
As the digital landscape has shifted from Web 1.0 to Web 2.0 and now evolves toward Web 3.0, we’ve seen each era bring a different emphasis in the CIA triad of confidentiality, integrity, and availability.
Returning to our home metaphor: When simply having shelter is what matters most, availability takes priority—the house must exist and be functional. Once that foundation is secure, confidentiality becomes important—you need locks on your doors to keep others out. Only after these basics are established do you begin to consider integrity, to ensure that what’s inside the house remains trustworthy, unaltered, and consistent over time.
Web 1.0 of the 1990s prioritized making information available. Organizations digitized their content, putting it out there for anyone to access. In Web 2.0, the Web of today, platforms for e-commerce, social media, and cloud computing prioritize confidentiality, as personal data has become the Internet’s currency.
Somehow, integrity was largely lost along the way. In our current Web architecture, where control is centralized and removed from individual users, the concern for integrity has diminished. The massive social media platforms have created environments where no one feels responsible for the truthfulness or quality of what circulates.
Web 3.0 is poised to change this dynamic by returning ownership to the data owners. This is not speculative; it’s already emerging. For example, ActivityPub, the protocol behind decentralized social networks like Mastodon, combines content sharing with built-in attribution. Tim Berners-Lee’s Solid protocol restructures the Web around personal data pods with granular access controls.
These technologies prioritize integrity through cryptographic verification that proves authorship, decentralized architectures that eliminate vulnerable central authorities, machine-readable semantics that make meaning explicit—structured data formats that allow computers to understand participants and actions, such as “Alice performed surgery on Bob”—and transparent governance where rules are visible to all. As AI systems become more autonomous, communicating directly with one another via standardized protocols, these integrity controls will be essential for maintaining trust.
Why Data Integrity Matters in AI
For AI systems, integrity is crucial in four domains. The first is decision quality. With AI increasingly contributing to decision-making in health care, justice, and finance, the integrity of both data and models’ actions directly impact human welfare. Accountability is the second domain. Understanding the causes of failures requires reliable logging, audit trails, and system records.
The third domain is the security relationships between components. Many authentication systems rely on the integrity of identity information and cryptographic keys. If these elements are compromised, malicious agents could impersonate trusted systems, potentially creating cascading failures as AI agents interact and make decisions based on corrupted credentials.
Finally, integrity matters in our public definitions of safety. Governments worldwide are introducing rules for AI that focus on data accuracy, transparent algorithms, and verifiable claims about system behavior. Integrity provides the basis for meeting these legal obligations.
The importance of integrity only grows as AI systems are entrusted with more critical applications and operate with less human oversight. While people can sometimes detect integrity lapses, autonomous systems may not only miss warning signs—they may exponentially increase the severity of breaches. Without assurances of integrity, organizations will not trust AI systems for important tasks, and we won’t realize the full potential of AI.
How to Build AI Systems With Integrity
Imagine an AI system as a home we’re building together. The integrity of this home doesn’t rest on a single security feature but on the thoughtful integration of many elements: solid foundations, well-constructed walls, clear pathways between rooms, and shared agreements about how spaces will be used.
We begin by laying the cornerstone: cryptographic verification. Digital signatures ensure that data lineage is traceable, much like a title deed proves ownership. Decentralized identifiers act as digital passports, allowing components to prove identity independently. When the front door of our AI home recognizes visitors through their own keys rather than through a vulnerable central doorman, we create resilience in the architecture of trust.
Formal verification methods enable us to mathematically prove the structural integrity of critical components, ensuring that systems can withstand pressures placed upon them—especially in high-stakes domains where lives may depend on an AI’s decision.
Just as a well-designed home creates separate spaces, trustworthy AI systems are built with thoughtful compartmentalization. We don’t rely on a single barrier but rather layer them to limit how problems in one area might affect others. Just as a kitchen fire is contained by fire doors and independent smoke alarms, training data is separated from the AI’s inferences and output to limit the impact of any single failure or breach.
Throughout this AI home, we build transparency into the design: The equivalent of large windows that allow light into every corner is clear pathways from input to output. We install monitoring systems that continuously check for weaknesses, alerting us before small issues become catastrophic failures.
But a home isn’t just a physical structure, it’s also the agreements we make about how to live within it. Our governance frameworks act as these shared understandings. Before welcoming new residents, we provide them with certification standards. Just as landlords conduct credit checks, we conduct integrity assessments to evaluate newcomers. And we strive to be good neighbors, aligning our community agreements with broader societal expectations. Perhaps most important, we recognize that our AI home will shelter diverse individuals with varying needs. Our governance structures must reflect this diversity, bringing many stakeholders to the table. A truly trustworthy system cannot be designed only for its builders but must serve anyone authorized to eventually call it home.
That’s how we’ll create AI systems worthy of trust: not by blindly believing in their perfection but because we’ve intentionally designed them with integrity controls at every level.
A Challenge of Language
Unlike other properties of security, like “available” or “private,” we don’t have a common adjective form for “integrity.” This makes it hard to talk about it. It turns out that there is a word in English: “integrous.” The Oxford English Dictionary recorded the word used in the mid-1600s but now declares it obsolete.
We believe that the word needs to be revived. We need the ability to describe a system with integrity. We must be able to talk about integrous systems design.
The Road Ahead
Ensuring integrity in AI presents formidable challenges. As models grow larger and more complex, maintaining integrity without sacrificing performance becomes difficult. Integrity controls often require computational resources that can slow systems down—particularly challenging for real-time applications. Another concern is that emerging technologies like quantum computingthreaten current cryptographic protections. Additionally, the distributed nature of modern AI—which relies on vast ecosystems of libraries, frameworks, and services—presents a large attack surface.
Beyond technology, integrity depends heavily on social factors. Companies often prioritize speed to market over robust integrity controls. Development teams may lack specialized knowledge for implementing these controls, and may find it particularly difficult to integrate them into legacy systems. And while some governments have begun establishing regulations for aspects of AI, we need worldwide alignment on governance for AI integrity.
Addressing these challenges requires sustained research into verifying and enforcing integrity, as well as recovering from breaches. Priority areas include fault-tolerantalgorithms for distributed learning, verifiable computation on encrypted data, techniques that maintain integrity despite adversarial attacks, and standardized metrics for certification. We also need interfaces that clearly communicate integrity status to human overseers.
As AI systems become more powerful and pervasive, the stakes for integrity have never been higher. We are entering an era where machine-to-machine interactions and autonomous agents will operate with reduced human oversight and make decisions with profound impacts.
The good news is that the tools for building systems with integrity already exist. What’s needed is a shift in mind-set: from treating integrity as an afterthought to accepting that it’s the core organizing principle of AI security.
The next era of technology will be defined not by what AI can do, but by whether we can trust it to know or especially to do what’s right. Integrity—in all its dimensions—will determine the answer.
Sidebar: Examples of Integrity Failures
Ariane 5 Rocket (1996) Processing integrity failure
A 64-bit velocity calculation was converted to a 16-bit output, causing an error called overflow. The corrupted data triggered catastrophic course corrections that forced the US $370 million rocket to self-destruct.
NASA Mars Climate Orbiter (1999) Processing integrity failure
Lockheed Martin’s software calculated thrust in pound-seconds, while NASA’s navigation software expected newton-seconds. The failure caused the $328 million spacecraft to burn up in the Mars atmosphere.
Microsoft’s Tay Chatbot (2016) Processing integrity failure
Released on Twitter, Microsoft‘s AI chatbot was vulnerable to a “repeat after me” command, which meant it would echo any offensive content fed to it.
Boeing 737 MAX (2018) Input integrity failure
Faulty sensor data caused an automated flight-control system to repeatedly push the airplane’s nose down, leading to a fatal crash.
SolarWinds Supply-Chain Attack (2020) Storage integrity failure
Russian hackers compromised the process that SolarWinds used to package its software, injecting malicious code that was distributed to 18,000 customers, including nine federal agencies. The hack remained undetected for 14 months.
ChatGPT Data Leak (2023) Storage integrity failure
A bug in OpenAI’s ChatGPT mixed different users’ conversation histories. Users suddenly had other people’s chats appear in their interfaces with no way to prove the conversations weren’t theirs.
Midjourney Bias (2023) Contextual integrity failure
Users discovered that the AI image generator often produced biased images of people, such as showing white men as CEOs regardless of the prompt. The AI tool didn’t accurately reflect the context requested by the users.
Prompt Injection Attacks (2023–) Input integrity failure
Attackers embedded hidden prompts in emails, documents, and websites that hijacked AI assistants, causing them to treat malicious instructions as legitimate commands.
CrowdStrike Outage (2024) Processing integrity failure
A faulty software update from CrowdStrike caused 8.5 million Windows computers worldwide to crash—grounding flights, shutting down hospitals, and disrupting banks. The update, which contained a software logic error, hadn’t gone through full testing protocols.
Voice-Clone Scams (2024) Input and processing integrity failure
Scammers used AI-powered voice-cloning tools to mimic the voices of victims’ family members, tricking people into sending money. These scams succeeded because neither phone systems nor victims identified the AI-generated voice as fake.
This essay was written with Davi Ottenheimer, and originally appeared in IEEE Spectrum.
The Arch Linux Project is currently experiencing an ongoing denial
of service attack that primarily impacts our main webpage, the Arch
User Repository (AUR), and the Forums.
We are aware of the problems that this creates for our end users
and will continue to actively work with our hosting provider to
mitigate the attack. We are also evaluating DDoS protection providers
while carefully considering factors including cost, security, and
ethical standards.
The post contains information on workarounds to use during the
service disruption, and notes that Arch is not sharing technical
details about the attack or mitigation while the attack is still
ongoing.
On August 21, 2025, an influx of traffic directed toward clients hosted in the Amazon Web Services (AWS) us-east-1 facility caused severe congestion on links between Cloudflare and AWS us-east-1. This impacted many users who were connecting to or receiving connections from Cloudflare via servers in AWS us-east-1 in the form of high latency, packet loss, and failures to origins.
Customers with origins in AWS us-east-1 began experiencing impact at 16:27 UTC. The impact was substantially reduced by 19:38 UTC, with intermittent latency increases continuing until 20:18 UTC.
This was a regional problem between Cloudflare and AWS us-east-1, and global Cloudflare services were not affected. The degradation in performance was limited to traffic between Cloudflare and AWS us-east-1. The incident was a result of a surge of traffic from a single customer that overloaded Cloudflare’s links with AWS us-east-1. It was a network congestion event, not an attack or a BGP hijack.
We’re very sorry for this incident. In this post, we explain what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again.
Background
Cloudflare helps anyone to build, connect, protect, and accelerate their websites on the Internet. Most customers host their websites on origin servers that Cloudflare does not operate. To make their sites fast and secure, they put Cloudflare in front as a reverse proxy.
When a visitor requests a page, Cloudflare will first inspect the request. If the content is already cached on Cloudflare’s global network, or if the customer has configured Cloudflare to serve the content directly, Cloudflare will respond immediately, delivering the content without contacting the origin. If the content cannot be served from cache, we fetch it from the origin, serve it to the visitor, and cache it along the way (if it is eligible). The next time someone requests that same content, we can serve it directly from cache instead of making another round trip to the origin server.
When Cloudflare responds to a request with the cached content, it will send the response traffic over internal Data Center Interconnect (DCI) links through a series of network equipment and eventually reach the routers that represent our network edge (our “edge routers”) as shown below:
Our internal network capacity is designed to be larger than the available traffic demand in a location to account for failures of redundant links, failover from other locations, traffic engineering within or between networks, or even traffic surges from users. The majority of Cloudflare’s network links were operating normally, but some edge router links to an AWS peering switch had insufficient capacity to handle this particular surge.
What happened
At approximately 16:27 UTC on August, 21, 2025, a customer started sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare’s cache. These requests generated a volume of response traffic that saturated all available direct peering connections between Cloudflare and AWS. This initial saturation became worse when AWS, in an effort to alleviate the congestion, withdrew some BGP advertisements to Cloudflare over some of the congested links. This action rerouted traffic to an additional set of peering links connected to Cloudflare via an offsite network interconnection switch, which subsequently also became saturated, leading to significant performance degradation. The impact became worse for two reasons: One of the direct peering links was operating at half-capacity due to a pre-existing failure, and the Data Center Interconnect (DCI) that connected Cloudflare’s edge routers to the offsite switch was due for a capacity upgrade. The diagram below illustrates this using approximate capacity estimates:
In response, our incident team immediately engaged with our partners at AWS to address the issue. Through close collaboration, we successfully alleviated the congestion and fully restored services for all affected customers.
Timeline
Time
Description
2025-08-21 16:27 UTC
Traffic surge for single customer begins, doubling total traffic from Cloudflare to AWS
IMPACT START
2025-08-21 16:37 UTC
AWS begins withdrawing prefixes from Cloudflare on congested PNI (Private Network Interconnect) BGP sessions
2025-08-21 16:44 UTC
Network team is alerted to internal congestion in Ashburn (IAD)
2025-08-21 16:45 UTC
Network team is evaluating options for response, but AWS prefixes are unavailable on paths that are not congested due to their withdrawals
2025-08-21 17:22 UTC
AWS BGP prefixes withdrawals result in a higher amount of dropped traffic
IMPACT INCREASE
2025-08-21 17:45 UTC
Incident is raised for customer impact in Ashburn (IAD)
2025-08-21 19:05 UTC
Rate limiting of single customer causing traffic surge decreases congestion
2025-08-21 19:27 UTC
Network team additional traffic engineering actions fully resolve congestion
IMPACT DECREASE
2025-08-21 19:45 UTC
AWS begins reverting BGP withdrawals as requested by Cloudflare
2025-08-21 20:07 UTC
AWS finishes normalizing BGP prefix announcements to Cloudflare over IAD PNIs
2025-08-21 20:18 UTC
IMPACT END
When impact started, we saw a significant amount of traffic related to one customer, resulting in congestion:
This was handled by manual traffic actions both from Cloudflare and AWS. You can see some of the attempts by AWS to alleviate the congestion by looking at the number of IP prefixes AWS is advertising to Cloudflare during the duration of the outage. The lines in different colors correspond to the number of prefixes advertised per BGP session with us. The dips indicate AWS attempting to mitigate by withdrawing prefixes from the BGP sessions in an attempt to steer traffic elsewhere:
The congestion in the network caused network queues on the routers to grow significantly and begin dropping packets. Our edge routers were dropping high priority packets consistently during the outage, as seen in the chart below, which shows the queue drops for our Ashburn routers during the impact period:
The primary impact to customers as a result of this congestion would have been latency, loss (timeouts), or low throughput. We have a set of latency Service Level Objectives defined which imitate customer requests back to their origins measuring availability and latency. We can see that during the impact period, the percentage of requests whose latency fails to meet the target SLO threshold dips below an acceptable level in lock step with the packet drops during the outage:
After the congestion was alleviated, there was a brief period where both AWS and Cloudflare were attempting to normalize the prefix advertisements that had been adjusted to attempt to mitigate the congestion. That caused a long tail of latency that may have impacted some customers, which is why you see the packet drops resolve before the customer latencies are restored.
Remediations and follow-up steps
This event has underscored the need for enhanced safeguards to ensure that one customer’s usage patterns cannot negatively affect the broader ecosystem. Our key takeaways are the necessity of architecting for better customer isolation to prevent any single entity from monopolizing shared resources and impacting the stability of the platform for others, and augmenting our network infrastructure to have sufficient capacity to meet demand.
To prevent a recurrence of this issue, we are implementing a multi-phased mitigation strategy. In the short and medium term:
We are developing a mechanism to selectively deprioritize a customer’s traffic if it begins to congest the network to a degree that impacts others.
We are expediting the Data Center Interconnect (DCI) upgrades which will provide network capacity significantly above what it is today.
We are working with AWS to make sure their and our BGP traffic engineering actions do not conflict with one another in the future.
Looking further ahead, our long-term solution involves building a new, enhanced traffic management system. This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer’s traffic from degrading the service for anyone else on the platform. This system will also allow us to automate many of the manual actions that were taken to attempt to remediate the congestion seen during this incident.
Conclusion
Customers accessing AWS us-east-1 through Cloudflare experienced an outage due to insufficient network congestion management during an unusual high-traffic event.
We are sorry for the disruption this incident caused for our customers. We are actively making these improvements to ensure improved stability moving forward and to prevent this problem from happening again.
Organizations face diverse challenges when it comes to managing encryption keys. While some scenarios demand strict separation, there are compelling use cases where a centralized approach can streamline operations and reduce complexity. In this post, our focus is on a software-as-a-service (SaaS) provider scenario, but the principles we discuss can be adopted by large organization facing similar key management challenges.
Managing encryption across a multi-tenant, multi-service architecture presents a significant challenge. Many organizations find themselves struggling with the complexity and costs associated with provisioning separate AWS Key Management Service (AWS KMS) customer managed keys for each tenant and service. This approach, while secure, often leads to growing operational overhead and increased AWS KMS usage costs over time.
But what if there was a more efficient way?
In this post, we unveil a strategy that uses a single customer managed key (symmetric) per tenant across services. By the end of this post, you’ll learn:
How to implement a scalable, secure, and cost-effective encryption model
Techniques for using one customer managed key per tenant across multiple services and environments
Methods for encrypting tenant data in Amazon DynamoDB and other storage types while maintaining tenant isolation
Multi-tenant encryption requirements for SaaS providers
Data isolation is fundamental to multi-tenant SaaS architectures, serving both compliance requirements and customer confidence. Many SaaS providers need to encrypt sensitive information—from API keys and credentials to personal data—across storage solutions such as DynamoDB and Amazon Simple Storage Service (Amazon S3).
While these storage services provide default encryption at rest, they typically use a single shared key across data items. Consider DynamoDB in a shared pool model, where one table contains data from multiple tenants. In this setup, the tenant data is encrypted using the same AWS KMS Key, regardless of ownership.
KMS key represents a container for top-level key material and is uniquely defined within the KMS, for more information on the different keys involved when encrypting or decrypting data using KMS, see AWS KMS key hierarchy.
This shared-key approach often proves insufficient for SaaS providers operating under strict security and compliance frameworks. Some customers require:
Bring your own key (BYOK) capabilities
Logical isolation of their data through dedicated encryption keys
To meet these requirements, providers can implement customer-specific AWS KMS managed keys, helping to ensure that each customer’s sensitive data remains isolated and inaccessible to other tenants.
Alternatively, providers might consider a silo model with separate tables for each customer. However, this approach introduces its own challenges—as the tenant base grows, managing numerous individual tables becomes increasingly complex and service quota limits might become a constraint.
Managing growth: KMS key management at scale
When scaling a SaaS platform, empowering teams to develop services independently is crucial. A quick way to scale is to have each team develop independently using a dedicated account. This often leads to a decentralized approach where each service manages its own KMS keys per customer. However, this autonomy comes with hidden costs as your customer base and service portfolio expand.
The challenge of key proliferation
As the company grows, the number of keys multiplies with each new customer and service addition. This proliferation creates several organizational challenges:
Cost impact: A single AWS KMS key costs $1 monthly, increasing to a maximum of $3 per month with two or more key rotations.
Operational complexity: Managing many KMS keys across environments and accounts is error-prone and hard to scale.
Organizational waste: Duplicate efforts across teams because each develops and maintains their own code for managing customer key lifecycles.
Governance overhead: It becomes difficult to enforce consistent policies or track KMS key usage across multiple AWS accounts.
A streamlined approach
The solution lies in implementing a centralized key management strategy. One KMS key per tenant, maintained in a central AWS account. This approach effectively addresses the cost, operational, and governance challenges while maintaining security.
In the following sections, we explore how to implement this centralized approach and securely share KMS keys across various services and AWS accounts.
At the heart of our solution lies a centralized tenant key management service (shown as Service A in the following figure). This service handles every aspect of customer KMS key lifecycle—from creation during tenant onboarding to managing aliases, access policies and deletion.
The service achieves secure, scalable key usage across the organization through cross-account AWS Identity and Access Management (IAM) access. It grants other services (for example, the customer-facing service in Account B in the following figure) a permission to perform specific encryption operations using tenant-specific KMS keys through role delegation. This implementation follows AWS best practices for cross-account access, utilizing IAM and AWS Security Token Service (AWS STS) role assumption as described in the AWS documentation and this blog post.
Centralized key management in practice: Encrypting customer data
Let’s examine how this works in practice with a common scenario:
Service A: Our centralized tenant key management service in Account A
Service B: A customer-facing workload running in Account B
When a customer interacts with Service B, it needs to store sensitive information securely, whether that’s secrets, API keys, or license information in a DynamoDB table. Instead of relying on shared KMS keys or default encryption, Service B encrypts data using the customer’s dedicated KMS key managed by Service A. The process works through AWS Identity and Access Management (IAM) role delegation. Service B temporarily assumes a role (ServiceARole) in Account A, receiving fine-grained, scoped down permissions for the specific tenant’s KMS key. With these temporary credentials, Service B can perform client-side encryption operations on sensitive information using the AWS SDK or the AWS Encryption SDK.
In this blog post, we used Boto3. For more advanced use-cases requiring data key caching or keyrings, use the AWS Encryption SDK.
Solution walkthrough
Let’s expand the technical aspects of the solution depicted above. Assumptions and definitions:
Incoming requests include an authentication header with a JSON Web Token (JWT) that includes data identifying the current tenant’s ID. These tokens are signed by an identity provider, making sure that the JWT cannot be modified, and the tenant identity can be trusted.
Account A: Centralized key management service.
Account B: Business service that serves customer requests.
alias/customer-<tenant-id> is the format of the aliases in account A. Each alias points to the KMS key of the corresponding customer identified by value of <tenant-id>. Service A creates these aliases during tenant onboarding and deletes them during tenant offboarding.
ServiceARole: A role in Account A that can encrypt and decrypt a KMS key that has an alias prefixed with alias/customer-*. The permissions are scoped down further using session policies when ServiceBRole assumes ServiceARole.
ServiceBRole: A role in Account B that can assume ServiceARole in Account A to gain access to the customer’s KMS key. This will be the AWS Lambda function’s execution role.
Note that Service B’s compute layer in this case is a Lambda function, but the solution works for other compute architectures. Let’s go over the flow in more detail:
Use service with JWT
A customer who belongs to a tenant signs in to the SaaS solution and is given a JWT that identifies its tenants with a tenant ID (<tenant-id>). The customer makes an action in ServiceB and sends sensitive information.
ServiceB handles the request (in a Lambda function), verifies the JWT token and wants to:
Encrypt the customer’s sensitive data
Save the encrypted data along with other data in the DynamoDB table
To encrypt tenant secrets securely and at scale, we grant application roles cross-account access to KMS keys—but only through their alias, which maps to a tenant identifier present in their JWT authentication token, enforcing strong isolation.
Depending on your environment, you can add additional conditions to this trust policy to further reduce the scope of who can assume this role. For more information, see IAM and AWS STS condition context keys.
Then, each KMS customer managed key will have the following policy. For example, a KMS key for a customer with <tenant-id>: 123 will have a policy that restricts access to the key using the specific customer alias and only through ServiceRoleA.
The following is a Python code example demonstrating how Service B dynamically assumes a role in Account A to encrypt data for a specific tenant using a session-scoped IAM policy that allows access only to that tenant’s KMS key alias.
This pattern follows the same principles outlined in Isolating SaaS Tenants with Dynamically Generated IAM Policies. The idea is to generate and attach a tenant-specific IAM policy at runtime, granting the minimum required permissions to operate on tenant-owned resources—in this case, a KMS key alias. The credentials will allow the Lambda function to use only the KMS key that belongs to a customer (identified by tenant_id).
We will call the assume_role_for_tenant for every tenant.
The condition of "StringEquals" - "kms:RequestAlias": alias is the magical AWS STS sauce, it restricts ServiceB to use the current tenant’s alias in its encryption SDK calls and relies on alias authorization
import boto3
def assume_role_for_tenant(tenant_id: str):
alias = f"alias/customer-{tenant_id}"
# Session policy scoped to only the specific alias
session_policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:GenerateDataKey*"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:RequestAlias": alias
}
}
}
]
}
# Assume ServiceARole in Account A with inline session policy
sts = boto3.client("sts")
assumed = sts.assume_role(
RoleArn="arn:aws:iam::<ACCOUNT_A_ID>:role/ServiceARole",
RoleSessionName=f"Tenant{tenant_id}Session",
Policy=json.dumps(session_policy)
)
return assumed["Credentials"]
Encrypt data and save in DynamoDB
Now, what remains to do is use the assumed role credentials and use AWS SDK to encrypt the sensitive customer data and store it in the DynamoDB table.
# Use temporary credentials to create a KMS client
creds = assume_role_for_tenant(tenant_id, plaintext)
kms = boto3.client(
"kms",
region_name="us-east-1",
aws_access_key_id=creds["AccessKeyId"],
aws_secret_access_key=creds["SecretAccessKey"],
aws_session_token=creds["SessionToken"]
)
# Encrypt using the alias
response = kms.encrypt(
KeyId= f"alias/customer-{tenant_id}"
Plaintext=plaintext
)
# store response["CiphertextBlob"] in DynamoDB table
This post doesn’t address isolation between different services, only between tenants. If such service isolation is required, you can use encryption context, an optional set of non-secret key/value pairs that can contain additional contextual information about the data, for example the service identifier. This helps ensure that services can only encrypt or decrypt data using the relevant service encryption context.
Benefits of centralized key management
Let’s examine how this solution addresses our earlier challenges.
Tenant isolation by design
Despite reducing the total number of KMS keys, we maintain strict tenant isolation. Each customer’s sensitive data remains encrypted with their dedicated key, identified by a unique alias (alias/customer-<tenant-id>). Access control to the tenant key is tightly managed through IAM role delegation, following least privilege principles:
Service A exclusively controls the management of the tenants’ KMS keys.
Service B can only assume a role that grants restricted encrypt, decrypt, and GenerateDataKey access for the customer managed key designated by the alias: alias/customer-<tenant-id>.
Optimized cost management
Our approach significantly reduces costs by moving from multiple service-specific KMS keys per tenant to a single KMS key per tenant that is shared securely across services and environments. This behavior introduces a new centralized account (Account A) that provides access to encryption keys under the right circumstances. It is important to understand AWS STS limits, specifically for AssumeRole calls and consider temporary IAM credentials caching mechanisms if those limits become a bottleneck. Additionally, if KMS limits are a bottleneck, consider using data key caching by using the AWS Encryption SDK.
Streamlined operations and governance
By centralizing key management in Service A, you can achieve:
Consistent KMS key lifecycle management across the organization
Improved audit capabilities using AWS CloudTrail to better understand key access patterns by service
Reduced operational overhead
Simplified compliance monitoring
The only additional complexity is the initial cross-account role delegation setup between Service A and other services. After being established, this framework can be scaled to accommodate new tenants and services.
It’s best to encapsulate the assume-role logic, policy generation, and AWS SDK client initialization within a shared organization-wide SDK. This abstraction reduces cognitive load for developers and minimizes the risk of misconfigurations. You can take it a step further by exposing high-level utility functions such as encrypt_tenant_data() and decrypt_tenant_data(), hiding the underlying complexity while promoting secure and consistent usage patterns across teams.
Conclusion
In this post, we explored an efficient approach to managing encryption keys in a multi-tenant SaaS environment through centralization. We examined common challenges faced by growing SaaS providers, including key proliferation, rising costs, and operational complexity across multiple AWS accounts and services. The solution, centralizing key management, uses AWS best practices for IAM role delegation and cross-account access, enabling organizations to maintain security and compliance while reducing operational overhead. By implementing this approach, SaaS providers or large organizations facing similar challenges can effectively manage their encryption infrastructure as they scale, without compromising on security or increasing complexity.
About the authors
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
