Internet shutdowns have long been a tool in government toolboxes when it comes to silencing opposition and cutting off access from the outside world. The KeepItOn campaign by Access Now, a group that defends the digital rights of global Internet users, documented at least 182 Internet shutdowns in 34 countries in 2021. Many of these shutdowns occurred during public protests, elections, and wars as an extreme form of censorship in places like Afghanistan, Democratic Republic of the Congo, Ukraine, India, and Iran.
There are a range of ways governments block or slow communications, including throttling, IP blocking, DNS interference, mobile data shutoffs, and deep packet inspection, all with similar goals: exerting control over information.
Although Internet shutdowns are largely public, it is difficult to document and track the ways in which governments implement them. The shutdowns not only impact people’s ability to participate in civil and political life and the economy but also have grave consequences for trust in democratic institutions.
We have reported on these shutdowns in the past, and for Cloudflare Impact Week, we want to tell you more about how we work with civil society organizations to provide tools to track and document the scope of these disruptions. We want to support their critical work and provide the tools they need so they can demand accountability and condemn the use of shutdowns to silence dissent.
Radar Internet shutdown alerts for civil society
We launched Radar in 2020 to shine light on the Internet’s patterns, insights, threats, and trends based on aggregated data from our network. Once we launched Radar, we found that many civil society organizations and those who work in democracy-building use Radar to track trends in countries to better understand the rise and fall of Internet usage.
Internally, we had an alert system for potential Internet disruptions that we use as an early warning regarding shifts in network patterns and incidents. When we engaged with these organizations that use Radar to track Internet trends, we learned more about how our internal tool to identify traffic distributions could be useful for organizations that work with human rights defenders on the ground that are impacted by these shutdowns.
To determine the best way to provide a tool to alert organizations when Cloudflare has seen these disruptions, we spoke with organizations such as Access Now, Internews, The Carter Center, National Democratic Institute, Internet Society, and the International Foundation for Electoral Systems. After our conversations, we launched Radar Internet shutdown alerts in 2021 to provide alerts on when Cloudflare has detected significant drops in traffic with the hope that the information is used to document, track, and hold institutions accountable for these human rights violations.
Since 2021, we have been providing these alerts to civil society partners to track these shutdowns. As we have collected feedback to improve the alerts, we have seen many partners looking for more ways to integrate Radar and the alerts into their existing tracking mechanisms. With this, we announced Radar 2.0 with API access for free so academics, data sleuths, civil society, human rights organizations, and other web enthusiasts can analyze, visualize, and investigate Internet usage across the globe, based on data from our global network. In addition, we launched Cloudflare Radar Outage Center to archive Internet outages and make it easier for civil society organizations, journalists/news media, and impacted parties to track past shutdowns.
Highlighting the work of our civil society partners to track Internet shutdowns
We believe our job at Cloudflare is to build tools that improve privacy and security for a range of players on the Internet. With this, we want to highlight the work of our civil society partners. These organizations are pushing back against targeted shutdowns that inflict lasting damage to democracies around the world. Here are their stories.
Access Now’s #KeepItOn coalition was launched in 2016 to help unite and organize the efforts of activists and organizations across the world to end Internet shutdowns. It now represents more than 280 organizations from 105 countries across the globe. The goal of STOP Project (Shutdown Tracker Optimization Project) is ultimately to document and report shutdowns accurately, which requires diligent verification. Access Now regularly uses multiple sources to identify and understand the shutdown, the choice and combination of which depends on where and how the shutdown occurred.
The tracker uses both quantitative and qualitative data to record the number of Internet shutdowns in the world in a given year and to characterize the nature of the shutdowns, including their magnitude, scope, and causes.
Zach Rosson, #KeepItOn Data Analyst, Access Now, details, “Sometimes, we confirm an Internet shutdown through means such as technical measurement, while at other times we rely upon contextual information, such as news reports or personal accounts. We also work hard to document how a particular shutdown was ordered and how it impacted society, including why and how it happened.”
On how Access Now’s #KeepItOn coalition uses Cloudflare Radar, Rosson says, “We use Radar Internet shutdown alerts in both email and tweet form, as a trusted source to help verify a shutdown occurrence. These alerts and their underlying measurements are used as primary sources in our dataset when compiling shutdowns for our annual report, so they are used in an archival sense as well. Cloudflare Radar is sometimes the first place that we hear about a shutdown, which is quite useful in a rapid response context, since we can quickly mobilize to verify the shutdown and have strong evidence when advocating against it.”
The recorded instances of shutdowns include events reported through local or international news sources that are included in the dataset, from local actors through Access Now’s Digital Security Helpline or the #KeepItOn Coalition email list, or directly from telecommunication and Internet companies.
Rosson notes, “When it comes to Radar 2.0 and API, we plan to use these resources to speed up our response, verification, and publication of shutdown data as compiled from different sources. Thus, the Cloudflare Radar Outage Center (CROC) and related API endpoint will be very useful for us to access timely information on shutdowns, either through visual inspection of the CROC in the short term or through using the API to pull data into a centralized database in the long term.”
On the Internet Society Pulse platform, Susannah Gray, Director, Communications, Internet Society, explains that they strive to curate meaningful information around a government-mandated Internet shutdown by using data from multiple trusted sources, and making it available to everyone, everywhere in an easy-to-understand manner. ISOC does this by monitoring Internet traffic using various tools, including Radar. When they see something that might indicate that an Internet shutdown is in progress, they check if the shutdown meets their criteria. For a shutdown to appear on the Pulse Shutdowns Tracker it needs to meet all the following requirements. It must:
Be artificially induced, as evident from reputable sources, including government statements and orders.
ISOC uses many resources to track shutdowns. Gray explains, “Radar Internet shutdown alerts are incredibly useful for bringing incidents to our attention as they are happening. The easy access to the data provided helps us assess the nature of an outage. If an outage is established as a government-mandated shutdown, we often use screenshots of Radar charts on the Pulse shutdowns tracker incident page to help illustrate how traffic stopped flowing in and out of a country during the shutdown. We provide a link back to the Radar platform so that people interested in getting more in-depth data can find out more.”
ISOC’s aim has never been to be the first to report a government-mandated shutdown: instead, their mission is to report accurate and meaningful information about the shutdown and explore its impact on the economy and society.
Gray adds, “For Radar 2.0 and the API, we plan to use it as part of the data aggregation tool we are developing. This internal tool will combine several outage alert and monitoring tools and sources into one single system so that we are able to track incidents more efficiently.”
OONI is a nonprofit that measures Internet censorship, including the blocking of websites, instant messaging apps, and circumvention tools. Cloudflare Radar is one of the main public data sources that they use when examining reported Internet connectivity shutdowns. For example, OONI relied on Radar data when reporting on shutdowns in Iran amid ongoing protests. In 2022, the team launched the Measurement Aggregation Toolkit (MAT), which enables the public to track censorship worldwide and create their own charts based on real-time OONI data. OONI also forms partnerships with multiple digital rights organizations that use OONI tools and data to monitor and respond to censorship events in their regions.
Maria Xynou, OONI Research and Partnerships Director, explains “Cloudflare Radar is one of the main public data sources that OONI has referred to when examining reported internet connectivity shutdowns. Specifically, OONI refers to Cloudflare Radar to check whether the platform provides signals of a reported internet connectivity shutdown; compare Cloudflare Radar signals with those visible in other, relevant public data sources (such as IODA, and Google traffic data).”
Tracking the shutdowns of tomorrow
As we work with more organizations in the human rights space and learn how our global network can be used for good, we are eager to improve and create new tools to protect human rights in the digital age.
“The more you practice the art of thankfulness, the more you have to be thankful for.”
— Norman Vincent Peale, American author
The turkey. The sweet potatoes. The stuffing. The pumpkin pie. Yesterday, November 24, 2022, was Thanksgiving Day in the US. A time for families and loved ones to be together and thankful, according to the tradition. Last year, we saw how the US paused shopping (and browsing) for Thanksgiving. So, how was it this year? Not only did we see Internet traffic go down (by 13%) during Thanksgiving dinner, but it was much higher than usual the day before and the day after (the Black Friday effect… so far). There was also a clear, but short, Thanksgiving day effect on e-commerce DNS trends.
We’ll have to wait to see what Black Friday looks like.
Let’s start with Internet traffic at the time of Thanksgiving dinner. Although every family is different, a 2018 survey of US consumers showed that for 42% early afternoon (between 13:00 and 15:00 is the preferred time to sit at the table and start to dig in). But 16:00 seems to be the “correct time” — The Atlantic explains why.
That said, Cloudflare Radar shows that between 21:00 and 01:00 UTC (we use that as the standard timezone in Radar) there was a clear drop in Internet traffic, mostly between 21:00 and 22:00 UTC, when traffic dropped 13%, compared with the week before. That time period is “translated” for the East Coast to between 16:00 and 20:00 EST and for the West Coast the time between 13:00 to 17:00 PST. Similar to what we saw last year.
Radar also allows anyone to focus on the last 24 hours and check the traffic volume change compared with the previous period. The more granular view in the next graph shows not only the 13% drop during Thanksgiving dinner, but also the clear increase after. At around 01:00 EST (22:00 PST), traffic was 15% higher than the day before, and today, November 25, Black Friday morning (08:00 EST, 05:00 PST), was growing ~16% more in traffic at 09:00 EST (06:00 PST).
It’s a similar perspective when we look at the last seven days, a filter that also shows the night before Thanksgiving in the US, traffic was 15% higher than the week before at around 01:00-03:00 EST (22:00-00:00 PST). And there’s a general increase in traffic this week, probably related to the fact it is also “Black Friday Week” (more on e-commerce trends at the end).
In terms of Internet traffic growth (made by humans, not bots) in November, there’s a clear increase throughout the month, but mostly this week. The next chart aggregates traffic by day. So far, Tuesday, November 22, 2022, was the day of the month with most traffic in the US — +13% than what we saw on Tuesday, November 1.
It’s also clear in the previous graph that weekends in the US have less traffic, especially Saturdays, but that Thanksgiving Day was the one with less traffic of the past two weeks — 10% less traffic than the same day the week before.
We’ve been focused on human Internet traffic. Bots, on the other hand, are not that interested in the Thanksgiving and Black Friday, and there was actually more bot traffic in the US last week than in this one. So far.
To wrap up this Internet traffic section, let’s look at mobile device trends. In the last four weeks, we saw an average of 48% of Internet traffic in the US coming from mobile devices. But on Thanksgiving Day that average was 55%. That was actually the day in November when people in the US were most online using their mobile devices.
Here’s the view that shows the mobile percentage difference from the past two weeks, with an up to 9% increase (compared with the previous week) in mobile devices’ predominance in Internet traffic, between 10:00 and 16:00 EST (07:00-13:00 PST).
E-commerce interest: growing (but with a Thanksgiving dip)
Now, let’s look at DNS query trends (from our globally used 1.1.1.1 DNS resolver) to e-commerce websites in the US. First, the Thanksgiving Day effect.
Aggregating several e-commerce domains, we can see not only that there are several spikes in the last two weeks, but that during Thanksgiving, there was a clear dip in DNS traffic between 15:00 and 17:00 EST (12:00-14:00 PST). How much? At 17:00 EST, Thanksgiving Day, there was 13% less DNS traffic than in the previous week.
Using a smoothed seven day rolling average to those e-commerce domains (only in the US), the growth trend for the past 30 days is even more clear in the past two weeks (after a clear dip in early November). From November 13 to November 22, the rolling average grew ~5%.
Last year, Cyber Monday was the biggest day for online shopping, in terms of DNS queries that we saw. Next week, we’ll see how it was this year.
Japan: A different kind of Thanksgiving
Also this week, Japan had its Labor Thanksgiving, an annual public holiday that was celebrated on Wednesday, November 23, 2022. And there was also a clear impact, but because, in Japan, this is a day full of events held throughout the country, there was an increase in traffic during the day. How much?
The peak was at around 01:00 UTC (10:00 in local time), when Internet traffic was 60% higher than in the previous week (and it continued to remain high during Labor Thanksgiving Day).
You can check Cloudflare Radar, but also our Twitter account where we continue to see country patterns related to the FIFA World Cup in Qatar (Internet traffic does shift, depending on the country, when national teams are playing), but also e-commerce DNS trends.
Today we’re introducing Cloudflare Radar’s route leak data and API so that anyone can get information about route leaks across the Internet. We’ve built a comprehensive system that takes in data from public sources and Cloudflare’s view of the Internet drawn from our massive global network. The system is now feeding route leak data on Cloudflare Radar’s ASN pages and via the API.
This blog post is in two parts. There’s a discussion of BGP and route leaks followed by details of our route leak detection system and how it feeds Cloudflare Radar.
About BGP and route leaks
Inter-domain routing, i.e., exchanging reachability information among networks, is critical to the wellness and performance of the Internet. The Border Gateway Protocol (BGP) is the de facto routing protocol that exchanges routing information among organizations and networks. At its core, BGP assumes the information being exchanged is genuine and trust-worthy, which unfortunately is no longer a valid assumption on the current Internet. In many cases, networks can make mistakes or intentionally lie about the reachability information and propagate that to the rest of the Internet. Such incidents can cause significant disruptions of the normal operations of the Internet. One type of such disruptive incident is route leaks.
We consider route leaks as the propagation of routing announcements beyond their intended scope (RFC7908). Route leaks can cause significant disruption affecting millions of Internet users, as we have seen in many past notable incidents. For example, in June 2016 a misconfiguration in a small network in Pennsylvania, US (AS396531 – Allegheny Technologies Inc) accidentally leaked a Cloudflare prefix to Verizon, which proceeded to propagate the misconfigured route to the rest of its peers and customers. As a result, the traffic of a large portion of the Internet was squeezed through the limited-capacity links of a small network. The resulting congestion caused most of Cloudflare traffic to and from the affected IP range to be dropped.
A similar incident in November 2018 caused widespread unavailability of Google services when a Nigerian ISP (AS37282 – Mainone) accidentally leaked a large number of Google IP prefixes to its peers and providers violating the valley-free principle.
These incidents illustrate not only that route leaks can be very impactful, but also the snowball effects that misconfigurations in small regional networks can have on the global Internet.
Despite the criticality of detecting and rectifying route leaks promptly, they are often detected only when users start reporting the noticeable effects of the leaks. The challenge with detecting and preventing route leaks stems from the fact that AS business relationships and BGP routing policies are generally undisclosed, and the affected network is often remote to the root of the route leak.
In the past few years, solutions have been proposed to prevent the propagation of leaked routes. Such proposals include RFC9234 and ASPA, which extends the BGP to annotate sessions with the relationship type between the two connected AS networks to enable the detention and prevention of route leaks.
An alternative proposal to implement similar signaling of BGP roles is through the use of BGP Communities; a transitive attribute used to encode metadata in BGP announcements. While these directions are promising in the long term, they are still in very preliminary stages and are not expected to be adopted at scale soon.
At Cloudflare, we have developed a system to detect route leak events automatically and send notifications to multiple channels for visibility. As we continue our efforts to bring more relevant data to the public, we are happy to announce that we are starting an open data API for our route leak detection results today and integrate results to Cloudflare Radar pages.
Route leak definition and types
Before we jump into how we design our systems, we will first do a quick primer on what a route leak is, and why it is important to detect it.
> A route leak is the propagation of routing announcement(s) beyond their intended scope.
The intended scope is often concretely defined as inter-domain routing policies based on business relationships between Autonomous Systems (ASes). These business relationships are broadly classified into four categories: customers, transit providers, peers and siblings, although more complex arrangements are possible.
In a customer-provider relationship the customer AS has an agreement with another network to transit its traffic to the global routing table. In a peer-to-peer relationship two ASes agree to free bilateral traffic exchange, but only between their own IPs and the IPs of their customers. Finally, ASes that belong under the same administrative entity are considered siblings, and their traffic exchange is often unrestricted. The image below illustrates how the three main relationship types translate to export policies.
By categorizing the types of AS-level relationships and their implications on the propagation of BGP routes, we can define multiple phases of a prefix origination announcements during propagation:
upward: all path segments during this phase are customer to provider
peering: one peer-peer path segment
downward: all path segments during this phase are provider to customer
An AS path that follows valley-free routing principle will have upward, peering, downward phases, all optional but have to be in that order. Here is an example of an AS path that conforms with valley-free routing.
> A multihomed AS learns a route from one upstream ISP and simply propagates it to another upstream ISP (the turn essentially resembling a hairpin). Neither the prefix nor the AS path in the update is altered.
An AS path that contains a provider-customer and customer-provider segment is considered a type 1 leak. The following example: AS4 → AS5 → AS6 forms a type 1 leak.
Type 1 is the most recognized type of route leaks and is very impactful. In many cases, a customer route is preferable to a peer or a provider route. In this example, AS6 will likely prefer sending traffic via AS5 instead of its other peer or provider routes, causing AS5 to unintentionally become a transit provider. This can significantly affect the performance of the traffic related to the leaked prefix or cause outages if the leaking AS is not provisioned to handle a large influx of traffic.
In June 2015, Telekom Malaysia (AS4788), a regional ISP, leaked over 170,000 routes learned from its providers and peers to its other provider Level3 (AS3549, now Lumin). Level3 accepted the routes and further propagated them to its downstream networks, which in turn caused significant network issues globally.
Type 2: Lateral ISP-ISP-ISP Leak
Type 2 leak is defined as propagating routes obtained from one peer to another peer, creating two or more consecutive peer-to-peer path segments.
Here is an example: AS3 → AS4 → AS5 forms a type 2 leak.
However, in the real world, it is not unusual to see multiple small peering networks exchanging routes and passing on to each other. Legit business reasons exist for having this type of network path. We are less concerned about this type of route leak as compared to type 1.
Type 3 and 4: Provider routes to peer; peer routes to provider
These two types involve propagating routes from a provider or a peer not to a customer, but to another peer or provider. Here are the illustrations of the two types of leaks:
As in the previously mentioned example, a Nigerian ISP who peers with Google accidentally leaked its route to its provider AS4809, and thus generated a type 4 route leak. Because routes via customers are usually preferred to others, the large provider (AS4809) rerouted its traffic to Google via its customer, i.e. the leaking ASN, overwhelmed the small ISP and took down Google for over one hour.
Route leak summary
So far, we have looked at the four types of route leaks defined in RFC7908. The common thread of the four types of route leaks is that they’re all defined using AS-relationships, i.e., peers, customers, and providers. We summarize the types of leaks by categorizing the AS path propagation based on where the routes are learned from and propagate to. The results are shown in the following table.
Routes from / propagates to
To provider
To peer
To customer
From provider
Type 1
Type 3
Normal
From peer
Type 4
Type 2
Normal
From customer
Normal
Normal
Normal
We can summarize the whole table into one single rule: routes obtained from a non-customer AS can only be propagated to customers.
Note: Type 5 and type 6 route leaks are defined as prefix re-origination and announcing of private prefixes. Type 5 is more closely related to prefix hijackings, which we plan to expand our system to as the next steps, while type 6 leaks are outside the scope of this work. Interested readers can refer to sections 3.5 and 3.6 of RFC7908 for more information.
The Cloudflare Radar route leak system
Now that we know what a route leak is, let’s talk about how we designed our route leak detection system.
From a very high level, we compartmentalize our system into three different components:
Raw data collection module: responsible for gathering BGP data from multiple sources and providing BGP message stream to downstream consumers.
Leak detection module: responsible for determining whether a given AS-level path is a route leak, estimate the confidence level of the assessment, aggregating and providing all external evidence needed for further analysis of the event.
Storage and notification module: responsible for providing access to detected route leak events and sending out notifications to relevant parties. This could also include building a dashboard for easy access and search of the historical events and providing the user interface for high-level analysis of the event.
Data collection module
There are three types of data input we take into consideration:
Historical: BGP archive files for some time range in the past
a. RouteViews and RIPE RIS BGP archives
Semi-real-time: BGP archive files as soon as they become available, with a 10-30 minute delay.
a. RouteViews and RIPE RIS archives with data broker that checks new files periodically (e.g. BGPKIT Broker)
Real-time: true real-time data sources
a. RIPE RIS Live
b. Cloudflare internal BGP sources
For the current version, we use the semi-real-time data source for the detection system, i.e., the BGP updates files from RouteViews and RIPE RIS. For data completeness, we process data from all public collectors from these two projects (a total of 63 collectors and over 2,400 collector peers) and implement a pipeline that’s capable of handling the BGP data processing as the data files become available.
For data files indexing and processing, we deployed an on-premises BGPKIT Broker instance with Kafka feature enabled for message passing, and a custom concurrent MRT data processing pipeline based on BGPKIT Parser Rust SDK. The data collection module processes MRT files and converts results into a BGP messages stream at over two billion BGP messages per day (roughly 30,000 messages per second).
Route leak detection
The route leak detection module works at the level of individual BGP announcements. The detection component investigates one BGP message at a time, and estimates how likely a given BGP message is a result of a route leak event.
We base our detection algorithm mainly on the valley-free model, which we believe can capture most of the notable route leak incidents. As mentioned previously, the key to having low false positives for detecting route leaks with the valley-free model is to have accurate AS-level relationships. While those relationship types are not publicized by every AS, there have been over two decades of research on the inference of the relationship types using publicly observed BGP data.
While state-of-the-art relationship inference algorithms have been shown to be highly accurate, even a small margin of errors can still incur inaccuracies in the detection of route leaks. To alleviate such artifacts, we synthesize multiple data sources for inferring AS-level relationships, including CAIDA/UCSD’s AS relationship data and our in-house built AS relationship dataset. Building on top of the two AS-level relationships, we create a much more granular dataset at the per-prefix and per-peer levels. The improved dataset allows us to answer the question like what is the relationship between AS1 and AS2 with respect to prefix P observed by collector peer X. This eliminates much of the ambiguity for cases where networks have multiple different relationships based on prefixes and geo-locations, and thus helps us reduce the number of false positives in the system. Besides the AS-relationships datasets, we also apply the AS Hegemony dataset from IHR IIJ to further reduce false positives.
Route leak storage and presentation
After processing each BGP message, we store the generated route leak entries in a database for long-term storage and exploration. We also aggregate individual route leak BGP announcements and group relevant leaks from the same leak ASN within a short period together into route-leak events. The route leak events will then be available for consumption by different downstream applications like web UIs, an API, or alerts.
Route leaks on Cloudflare Radar
At Cloudflare, we aim to help build a better Internet, and that includes sharing our efforts on monitoring and securing Internet routing. Today, we are releasing our route leak detection system as public beta.
Starting today, users going to the Cloudflare Radar ASN pages will now find the list of route leaks that affect that AS. We consider that an AS is being affected when the leaker AS is one hop away from it in any direction, before or after.
The Cloudflare Radar ASN page is directly accessible via https://radar.cloudflare.com/as{ASN}. For example, one can navigate to https://radar.cloudflare.com/as174 to view the overview page for Cogent AS174. ASN pages now show a dedicated card for route leaks detected relevant to the current ASN within the selected time range.
There is a lot more we are planning to do with route-leak detection. More features like a global view page, route leak notifications, more advanced APIs, custom automations scripts, and historical archive datasets will begin to ship on Cloudflare Radar over time. Your feedback and suggestions are also very important for us to continue improving on our detection results and serve better data to the public.
Furthermore, we will continue to expand our work on other important topics of Internet routing security, including global BGP hijack detection (not limited to our customer networks), RPKI validation monitoring, open-sourcing tools and architecture designs, and centralized routing security web gateway. Our goal is to provide the best data and tools for routing security to the communities so that we can build a better and more secure Internet together.
In the meantime, we opened a Radar room on our Developers Discord Server. Feel free to join and talk to us; the team is eager to receive feedback and answer questions.
Visit Cloudflare Radar for more Internet insights. You can also follow us on Twitter for more Radar updates.
Radar 2.0 was built on the learnings of Radar 1.0 and was launched last month during Cloudflare’s Birthday Week as a complete product revamp. We wanted to make it easier for our users to find insights and navigate our data, and overall provide a better and faster user experience.
We’re building a Supercloud. Cloudflare’s products now include hundreds of features in networking, security, access controls, computing, storage, and more.
This blog will explain how we built the new Radar from an engineering perspective. We wanted to do this to demonstrate that anyone could build a somewhat complex website that involves demanding requirements and multiple architectural layers, do it on top of our stack, and how easy it can be.
Hopefully, this will inspire other developers to switch from traditional software architectures and build their applications using modern, more efficient technologies.
High level architecture
The following diagram is a birds-eye view of the Radar 2.0 architecture. As you can see, it’s divided into three main layers:
The Core layer is where we keep our data lake, data exploration tools, and backend API.
The Cloudflare network layer is where we host and run Radar and serve the public APIs.
The Client layer is essentially everything else that runs in your browser. We call it the Radar Web app.
As you can see, there are Cloudflare products everywhere. They provide the foundational resources to host and securely run our code at scale, but also other building blocks necessary to run the application end to end.
By having these features readily available and tightly integrated into our ecosystem and tools, at the distance of a click and a few lines of code, engineering teams don’t have to reinvent the wheel constantly and can use their time on what is essential: their app logic.
Let’s dig in.
Cloudflare Pages
Radar 2.0 is deployed using Cloudflare Pages, our developer-focused website hosting platform. In the early days, you could only host static assets on Pages, which was helpful for many use cases, including integrating with static site generators like Hugo, Jekyll, or Gatsby. Still, it wouldn’t solve situations where your application needs some sort of server-side computing or advanced logic using a single deployment.
Luckily Pages recently added support to run custom Workers scripts. With Functions, you can now run server-side code and enable any kind of dynamic functionality you’d typically implement using a separate Worker.
Cloudflare Pages Functions also allow you to use Durable Objects, KV, R2, or D1, just like a regular Worker would. We provide excellent documentation on how to do this and more in our Developer Documentation. Furthermore, the team wrote a blog on how to build a full-stack application that describes all the steps in detail.
Radar 2.0 needs server-side functions for two reasons:
To render Radar and run the server side of Remix.
To implement and serve our frontend API.
Remix and Server-side Rendering
We use Remix with Cloudflare Pages on Radar 2.0.
Remix follows a server/client model and works under the premise that you can’t control the user’s network, so web apps must reduce the amount of Javascript, CSS, and JSON they send through the wire. To do this, they move some of the logic to the server.
In this case, the client browser will get pre-rendered DOM components and the result of pre-fetched API calls with just the right amount of JSON, Javascript, and CSS code, rightfully adjusted to the UI needs. Here’s the technical explanation with more detail.
Typically, Remix would need a Node.js server to do all of this, but guess what: It can also run on Cloudflare Workers and Pages.
Here’s the code to get the Remix server running on Workers, using Cloudflare Pages:
import { createPagesFunctionHandler } from "@remix-run/cloudflare-pages";
import * as build from "@remix-run/dev/server-build";
const handleRequest = createPagesFunctionHandler({
build: {
...build,
publicPath: "/build/",
assetsBuildDirectory: "public/build",
},
mode: process.env.NODE_ENV,
getLoadContext: (context) => ({
...context.env,
CF: (context.request as any).cf as IncomingRequestCfProperties | undefined,
}),
});
const handler: ExportedHandler<Env> = {
fetch: async (req, env, ctx) => {
const r = new Request(req);
return handleRequest({
env,
params: {},
request: r,
waitUntil: ctx.waitUntil,
next: () => {
throw new Error("next() called in Worker");
},
functionPath: "",
data: undefined,
});
},
};
In Remix, routes handle changes when a user interacts with the app and changes it (clicking on a menu option, for example). A Remix route can have a loader, an action and a default export. The loader handles API calls for fetching data (GET method). The action handles submissions to the server (POST, PUT, PATCH, DELETE methods) and returns the response. The default export handles the UI code in React that’s returned for that route. A route without a default export returns only data.
Because Remix runs both on the server and the client, it can get smart and know what can be pre-fetched and computed server-side and what must go through the network connection, optimizing everything for performance and responsiveness.
Here’s an example of a Radar route, simplified for readability, for the Outage Center page.
Another project porting their app to Remix is Cloudflare TV. This is how their metrics looked before and after the changes.
Radar’s Desktop Lighthouse score is now nearly 100% on Performance, Accessibility, Best Practices, and SEO.
Another Cloudflare product that we use extensively on Radar 2.0 is Speed. In particular, we want to mention the Early Hints feature. Early Hints is a new web standard that defines a new HTTP 103 header the server can use to inform the browser which assets will likely be needed to render the web page while it’s still being requested, resulting in dramatic load times improvements.
Radar has two APIs. The backend which has direct access to our data sources, and the frontend, which is available on the Internet.
Backend API
The backend API was written using Python, Pandas and FastAPI and is protected by Cloudflare Access, JWT tokens and an authenticated origin pull (AOP) configuration. Using Python allows anyone on the team, engineers or data scientists, to collaborate easily and contribute to improving and expanding the API, which is great. Our data science team uses JupyterHub and Jupyter Notebooks as part of their data exploration workflows, which makes prototyping and reusing code, algorithms and models particularly easy and fast.
It then talks to the upstream frontend API via a Strawberry based GraphQL server. Using GraphQL makes it easy to create complex queries, giving internal users and analysts the flexibility they need when building reports from our vast collection of data.
Frontend API
We built Radar’s frontend API on top of Cloudflare Workers. This worker has two main functions:
It fetches data from the backend API using GraphQL, and then transforms it.
It provides a public REST API that anyone can use, including Radar.
Using a worker in front of our core API allows us to easily add and separate microservices, and also adds notable features like:
Cloudflare’s Cache API allows finer control over what to cache and for how long and supports POST requests and customizable cache control headers, which we use.
Stale responses using R2. When the backend API cannot serve a request for some reason, and there’s a stale response cached, it’ll be served directly from R2, giving end users a better experience.
CSV and JSON output formats. The CSV format is convenient and makes it easier for data scientists, analysts, and others to use the API and consume our API data directly from other tools.
Open sourcing our OpenAPI 3 schema generator and validator
One last feature on the frontend API is OpenAPI 3 support. We automatically generate an OpenAPI schema and validate user input with it. This is done through a custom library that we built on top of itty-router, which we also use. Today we’re open sourcing this work.
itty-router-openapi provides an easy and compact OpenAPI 3 schema generator and validator for Cloudflare Workers. Check our GitHub repository for more information and details on how to use it.
Developer’s Documentation
Today we’re also launching our developer’s documentation pages for the Radar API where you can find more information about our data license, basic concepts, how to get started and the available API methods. Cloudflare Radar’s API is free, allowing academics, data sleuths and other web enthusiasts to investigate Internet usage across the globe, based on data from our global network.
To facilitate using our API, we also put together a Colab Notebook template that you can play with, copy and expand to your use case.
The Radar App
The Radar App is the code that runs in your browser. We’ve talked about Remix, but what else do we use?
Radar relies on a lot of data visualizations. Things like charts and maps are essential to us. We decided to build our reusable library of visualization components on top of two other frameworks: visx, a “collection of expressive, low-level visualization primitives for React,” D3, a powerful JavaScript library for manipulating the DOM based on data, and MapLibre, an open-source map visualization stack.
Here’s one of our visualization components in action. We call it the “PewPew map”.
And here’s the Remix React code for it, whenever we need to use it in a page:
Another change we made to Radar was switching our images and graphical assets to Scalable Vector Graphics. SVGs are great because they’re essentially a declarative graphics language. They’re XML text files with vectorial information. And so, they can be easily manipulated, transformed, stored, or indexed, and of course, they can be rendered at any size, producing beautiful, crisp results on any device and resolution.
SVGs are also extremely small and efficient in size compared to bitmap formats and support internationalization, making them easier to translate to other languages (localization), thus providing better accessibility.
Here’s an example of a Radar Bubble Chart, inspected, where you can see the SVG code and the <text/> strings embedded.
Cosmos
React Cosmos is a “sandbox for developing and testing UI components in isolation.” We wanted to use Cosmos with Radar 2.0 because it’s the perfect project for it:
It has a lot of visual components; some are complex and have many configuration options and features.
The components are highly reusable across multiple pages in different contexts with different data.
We have a multidisciplinary team; everyone can send a pull request and add or change code in the frontend.
Cosmos acts as a component library where you can see our palette of ready-to-use visualizations and widgets, from simple buttons to complex charts, and you play with their options in real-time and see what happens. Anyone can do it, not only designers or engineers but also other project stakeholders. This effectively improves team communications and makes contributing and iterating quickly.
Here’s a screenshot of our Cosmos in action:
Continuous integration and development
Continuous integration is important for any team doing modern software. Cloudflare Pages provides multiple options to work with CI tools using direct uploads, out of the box. The team has put up documentation and examples on how to do that with GitHub Actions, CircleCI, and Travis, but you can use others.
In our case, we use BitBucket and TeamCity internally to build and deploy our releases. Our workflow automatically builds, tests, and deploys Radar 2.0 within minutes on an approved PR and follow-up merge.
Furthermore, we have multiple environments to stage and test our releases before they go live to production. Our CI/CD setup makes it easy to switch from one environment to the other or quickly roll back any undesired deployment.
Radar 1.0 wasn’t particularly fast doing CI/CD, we confess. We had a few episodes when a quick fix could take some good 30 minutes from committing the code to deployment, and we felt frustrated about it.
So we invested a lot in ensuring that the new CI would be fast, efficient, and furious.
One cool thing we ended up doing was fast preview links on any commit pushed to the code repository. Using a combination of intelligent caching during builds and doing asynchronous tests when the commit is outside the normal release branches, we were able to shorten the deployment time to seconds.
This is the notification we get in our chat when anyone pushes code to any branch:
Anyone can follow a thread for a specific branch in the chat and get notified on new changes when they happen.
Blazing-fast builds, preview links and notifications are game-changers. An engineer can go from an idea or a quick fix to showing the result on a link to a product manager or another team member. Anyone can quickly click the link to see the changes on a fully working end-to-end version of Radar.
Accessibility and localization
Cloudflare is committed to web accessibility. Recently we announced how we upgraded Cloudflare’s Dashboard to adhere to industry accessibility standards, but this premise is valid for all our properties. The same is true for localization. In 2020, we internationalized our Dashboard and added support for new languages and locales.
Accessibility and localization go hand in hand and are both important, but they are also different. The Web Content Accessibility Guidelines define many best practices around accessibility, including using color and contrast, tags, SVGs, shortcuts, gestures, and many others. The A11Y project page is an excellent resource for learning more.
Localization, though, also known as L10n, is more of a technical requirement when you start a new project. It’s about making sure you choose the right set of libraries and frameworks that will make it easier to add new translations without engineering dependencies or code rewrites.
We wanted Radar to perform well on both fronts. Our design system takes Cloudflare’s design and brand guidelines seriously and adds as many A11Y good practices as possible, and the app is fully aware of localization strings across its pages and UI components.
Adding a new language is as easy as translating a single JSON file. Here’s a snippet of the en-US.json file with the default American English strings:
{
"abbr.asn": "Autonomous System Number",
"actions.chart.download.csv": "Download chart data in CSV",
"actions.chart.download.png": "Download chart in PNG Format",
"actions.chart.download.svg": "Download chart in SVG Format",
"actions.chart.download": "Download chart",
"actions.chart.maximize": "Maximize chart",
"actions.chart.minimize": "Minimize chart",
"actions.chart.share": "Share chart",
"actions.download.csv": "Download CSV",
"actions.download.png": "Download PNG",
"actions.download.svg": "Download SVG",
"actions.share": "Share",
"alert.beta.link": "Radar Classic",
"alert.beta.message": "Radar 2.0 is currently in Beta. You can still use {link} during the transition period.",
"card.about.cloudflare.p1": "Cloudflare, Inc. ({website} / {twitter}) is on a mission to help build a better Internet. Cloudflare's suite of products protects and accelerates any Internet application online without adding hardware, installing software, or changing a line of code. Internet properties powered by Cloudflare have all web traffic routed through its intelligent global network, which gets smarter with every request. As a result, they see significant improvement in performance and a decrease in spam and other attacks. Cloudflare was named to Entrepreneur Magazine's Top Company Cultures 2018 list and ranked among the World's Most Innovative Companies by Fast Company in 2019.",
"card.about.cloudflare.p2": "Headquartered in San Francisco, CA, Cloudflare has offices in Austin, TX, Champaign, IL, New York, NY, San Jose, CA, Seattle, WA, Washington, D.C., Toronto, Dubai, Lisbon, London, Munich, Paris, Beijing, Singapore, Sydney, and Tokyo.",
"card.about.cloudflare.title": "About Cloudflare",
...
You can expect us to release Radar in other languages soon.
Radar Reports and Jupyter notebooks
Radar Reports are documents that use data exploration and storytelling to analyze a particular theme in-depth. Some reports tend to get updates from time to time. Examples of Radar Reports are our quarterly DDoS Attack Trends, or the IPv6 adoption.
The source of these Reports is Jupyter Notebooks. Our Data Science team works on some use-case or themes with other stakeholders using our internal Jupyter Hub tool. After all the iteration and exploration are done, and the work is signed off, a notebook is produced.
A Jupyter Notebook is a JSON document containing text, source code, rich media such as images or charts, and other metadata. It is the de facto standard for presenting data science projects, and every data scientist uses it.
With Radar 1.0, converting a Jupyter Notebook to a Radar page was a lengthy and manual process implicating many engineering and design resources and causing much frustration to everyone involved. Even updating an already-published notebook would frequently cause trouble for us.
Radar 2.0 changed all of this. We now have a fully automated process that takes a Jupyter Notebook and, as long as it’s designed using a list of simple rules and internal guidelines, converts it automatically, hosts the resulting HTML and assets in an R2 bucket, and publishes it on the Reports page.
The conversion to HTML takes into account our design system and UI components, and the result is a beautiful document, usually long-form, perfectly matching Radar’s look and feel.
We will eventually open-source this tool so that anyone can use it.
More Cloudflare, less to worry about
We gave examples of using Cloudflare’s products and features to build your next-gen app without worrying too much about things that aren’t core to your business or logic. A few are missing, though.
Once the app is up and running, you must protect it from bad traffic and malicious actors. Cloudflare offers you DDoS, WAF, and Bot Management protection out of the box at a click’s distance.
For example, here are some of our security rules. This is traffic we don’t have to worry about in our app because Cloudflare detects it and acts on it according to our rules.
Another thing we don’t need to worry about is redirects from the old site to the new one. Cloudflare has a feature called Bulk Redirects, where you can easily create redirect lists directly on the dashboard.
It’s also important to mention that every time we talk about what you can do using our Dashboard, we’re, in fact, also saying you can do precisely the same using Cloudflare’s APIs. Our Dashboard is built entirely on top of them. And if you’re the infrastructure as code kind of person, we have you covered, too; you can use the Cloudflare Terraform provider.
Deploying and managing Workers, R2 buckets, or Pages sites is obviously scriptable too. Wrangler is the command-line tool to do this and more, and it goes the extra mile to allow you to run your full app locally, emulating our stack, on your computer, before deploying.
Final words
We hope you enjoyed this Radar team write-up and were inspired to build your next app on top of our Supercloud. We will continue improving and innovating on Radar 2.0 with new features, share our findings and open-sourcing our tools with you.
In the meantime, we opened a Radar room on our Developers Discord Server. Feel free to join it and ask us questions; the team is eager to receive feedback and discuss web technology with you.
You can also follow us on Twitter for more Radar updates.
Through Cloudflare’s Impact programs, we provide cyber security products to help protect access to authoritative voting information and the security of sensitive voter data. Two core programs in this space are the Athenian Project, dedicated to protecting state and local governments that run elections, and Cloudflare for Campaigns, a project with a suite of Cloudflare products to secure political campaigns’ and state parties’ websites and internal teams.
However, the weeks ahead of the elections, and Election Day itself, were not entirely devoid of attacks. Using data from Cloudflare Radar, which showcases global Internet traffic, attack, and technology trends and insights, we can explore traffic patterns, attack types, and top attack sources associated with both Athenian Project and Cloudflare for Campaigns participants.
For both programs, overall traffic volume unsurprisingly ramped up as Election Day approached. SQL Injection (SQLi) and HTTP Anomaly attacks were the two largest categories of attacks mitigated by Cloudflare’s Web Application Firewall (WAF), and the United States was the largest source of observed attacks — see more on this last point below.
Below, we explore the trends seen across both customer sets from October 1, 2022, through Election Day on November 8.
Athenian Project
Throughout October, daily peak traffic volumes effectively doubled over the course of the month, with a weekday/weekend pattern also clearly visible. However, significant traffic growth is visible on Monday, November 7, and Tuesday, November 8 (Election Day), with Monday’s peak just under 2x October’s peaks, while Tuesday saw two peaks, one just under 4x higher than October peaks, while the other was just over 4x higher. Zooming in, the first peak was at 1300 UTC (0800 Eastern time, 0500 Pacific time), while the second was at 0400 UTC (2300 Eastern time, 2000 Pacific time). The first one appears to be aligned with the polls opening on the East Coast, while the second appears to be aligned with the time that the polls closed on the West Coast.
However, aggregating the traffic here presents a somewhat misleading picture. While both spikes were due to increased traffic across multiple customer sites, the second one was exacerbated by a massive increase in traffic for a single customer. Regardless, the increased traffic clearly shows that voters turned to local government sites around Election Day.
Despite this increase in overall traffic, attack traffic mitigated by Cloudflare’s Web Application Firewall (WAF) remained remarkably consistent throughout October and into November, as seen in the graph below. The obvious exception was an attack that occurred on Monday, October 10. This attack targeted a single Athenian Project participant, and was mitigated by rate limiting the requests.
SQL injection (SQLi) attacks saw significant growth in volume in the week and a half ahead of Election Day, along with an earlier significant spike on October 24. While the last weekend in October (October 29 and 30) saw significant SQLi attack activity, the weekend of November 5 and 6 was comparatively quiet. However, those attacks ramped up again heading into and on Election Day, as seen in the graph below.
Attempted attacks mitigated with the HTTP Anomaly ruleset also ramped up in the week ahead of Election Day, though to a much lesser extent than SQLi attacks. As the graph below shows, the biggest spikes were seen on October 31/November 1, and just after midnight UTC on November 4 (late afternoon to early evening in the US). Related request volume also grew heading into Election Day, but without significant short-duration spikes. There is also a brief but significant attack clearly visible on the graph on October 10. However, it occurred several hours after the rate limited attack referenced above — it is not clear if the two are related.
The distribution of attacks over the surveyed period from October 1 through November 9 shows that those categorized as SQLi and HTTP Anomaly were responsible for just over two-thirds of WAF-mitigated requests. Nearly 14% were categorized as “Software Specific,” which includes attacks related to specific CVEs. The balance of the attacks were mitigated by WAF rules in categories including File Inclusion, XSS (Cross Site Scripting), Directory Traversal, and Command Injection.
Media reports suggest that foreign adversaries actively try to interfere with elections in the United States. While this may be the case, analysis of the mitigated attacks targeting Athenian Project customers found that over 95% of the mitigated requests (attacks) came from IP addresses that geolocate to the United States. However, that does not mean that the attackers themselves are necessarily located in the country, but rather that they appear to be using compromised systems and proxies within the United States to launch their attacks against these sites protected by Cloudflare.
Cloudflare for Campaigns
In contrast to Athenian Project participants, traffic to candidate sites that are participants in Cloudflare for Campaigns began to grow several weeks ahead of Election Day. The graph below shows a noticeable increase (~50%) in peak traffic volumes starting on October 12, with an additional growth (50-100%) starting a week later. Traffic to these sites appeared to quiet a bit toward the end of October, but saw significant growth again heading into, and during, Election Day.
However, once again, this aggregate traffic data presents something of a misleading picture, as one candidate site saw multiple times more traffic than the other participating sites. While those other sites saw similar shifts in traffic as well, they were dwarfed by those experienced by the outlier site.
The WAF-mitigated traffic trend for campaign sites followed a similar pattern to the overall traffic. As the graph below shows, attack traffic also began to increase around October 19, with a further ramp near the end of the month. The October 27 spike visible in the graph was due to an attack targeting a single customer’s site, and was addressed using “Security Level” mitigation techniques, which uses IP reputation information to decide if and how to present challenges for incoming requests.
The top two rule categories, HTTP Anomaly and SQLi, together accounted for nearly three-quarters of the mitigated requests, and Directory Traversal attacks were just under 10% of mitigated requests for this customer set. The HTTP Anomaly and Directory Traversal percentages were higher than those for attacks targeting Athenian Project participants, while the SQLi percentage was slightly lower.
Once again, a majority of the WAF-mitigated attacks came from IP addresses in the United States. However, among Cloudflare for Campaigns participants, the United States only accounted for 55% of attacks, significantly lower than the 95% seen for Athenian Project participants. The balance is spread across a long tail of countries, with allies including Germany, Canada, and the United Kingdom among the top five. As noted above, however, the attackers may be elsewhere, and are using botnets or other compromised systems in these countries to launch attacks.
Improving security with data
We are proud to be trusted by local governments, campaigns, state parties, and voting rights organizations to protect their websites and provide uninterrupted access to information and trusted election results. Sharing information about the threats facing these websites helps us further support their valuable work by enabling them, and other participants in the election space, to take proactive steps to improve site security.
Brasil, sei lá Ou o meu coração se engana Ou uma terra igual não há — From Tom Jobim’s song, Brasil Nativo
Brazil’s recent presidential election got significant attention from both global and national media outlets, not only because of the size of the country, but also because of premature allegations of electoral fraud. The first round of the Brazilian 2022 general election was held on October 2, and the runoff was held on Sunday, October 30. With 124 million votes counted, former president Lula da Silva (2003-2010) won with 50.9% of the votes, beating incumbent Jair Bolsonaro, who had 49.1% of the votes.
The final results of the elections as published by the official Tribunal Super Eleitoral, with more than 124 million votes counted.)
Using Cloudflare’s data, we can explore the impact that this election had on Internet traffic patterns in Brazil, as well as interest in content from election-related websites, news organizations, social media platforms, and video platforms.
Here are a few highlights: while the runoff generated much more interest to election related websites (we actually have a view to DNS queries, a proxy to websites), the first round showed bigger increases in traffic to news organizations.
For the candidate’s domains, Lula’s win had the higher impact.
Also: official results came earlier on the runoff than the first round, and spikes in traffic were higher earlier that day (October 30).
(Note: we’re using local times — that means UTC-3, that is related to the more populated regions of Brazil — in this blog, although some charts have x-axis UTC).
Let’s start by looking at general Internet traffic in Brazil.
On election days, traffic goes down (during the day)
Using Cloudflare Radar, we can see something that has also been observed in other countries that hold Sunday elections: when most people are getting outside to vote, Internet traffic goes down (in comparison with previous Sundays). We saw this in the two rounds of the Presidential elections in France back in April 2022, in Portugal’s legislative elections in January 2022 and now, in Brazil.
We can also compare Sundays in October. There were five weekends. The two that had elections show the same pattern of lower traffic during the day, as seen in the previous chart. Comparing the two election days, there was a bigger drop in traffic on October 30 (down 21% at around 18:00 local time), than on October 2 (down 10% at around 20:00). Related or not, there was a bigger turnout on the runoff (124 million votes) than on the first round (123 million). Here’s the view on October 30:
And here’s October 2:
A more clear view in comparing the October weekends, and where you can see how the October 2 and 30 Sundays have the same pattern and different from the others three of the month, is this one (bear in mind that the x-axis is showing UTC time, it’s -3 hours in Brazil):
If we look at the main network providers (ASNs) in Brazil, the trend is the same. Claro (AS28573) also shows the drop in traffic on October 30, as does Telefonica (AS27699):
Here’s Telefonica:
We observed a similar impact from the October 30 runoff election to traffic from different states in Brazil, including São Paulo, Rio de Janeiro, Rio Grande do Norte, Minas Gerais, and Bahia.
Mobile device usage greater on weekends (and on election days)
When we look at the share of Brazil’s Internet traffic from mobile devices during October, we find that the highest percentages were on October 2 (first round of the elections, 66.3%), October 9 (66.4%) and October 30 (runoff election, 65%). We’ve seen this in other elections, an increase in mobile device traffice, so this seems to follow the same trend.
This chart also shows how mobile device usage in Brazil is at its highest on the weekends (all the main spikes for percentage of mobile devices are over the weekend, and more on Sundays).
Now, let’s look at anonymized and aggregated DNS traffic data from our 1.1.1.1 resolver. This data provides a proxy for traffic to, and thus interest in, different categories of sites from users in Brazil around the election.
Election-related sites: higher interest in the runoff
Brazil has government websites related to elections, but also its own Tribunal Superior Eleitoral (Electoral Superior Court) that includes a website and app with live updates on the results of the elections for everyone to check. Looking at those related domains and using mean hourly traffic in September as a baseline, we can see that the October 2 first round spiked to 16x more DNS queries at 20:00 local time. However, DNS query traffic during the runoff election peaked at 18:00 local time on October 30 with 17.4x more DNS traffic as compared to the September baseline.
We can look more closely at each one of those two election days. On October 2, traffic had its first significant increase at around 17:00 local time, reaching 15x more requests to election-related domains as compared to the September baseline. This initial peak occurred at the same time the polling stations were closing. However, the peak that day, at 16x above baseline, was reached at 20:00 local time, as seen in the figure below.
On Sunday, October 30, 2022, the pattern is similar, although the peak was reached earlier, given that results started to arrive earlier than on the first round. The peak was reached at around 18:00 local time, with request traffic 17.4x above baseline.
As seen in the figure below, Lula first led in the official results at 18:45 local time, with votes from 67% of the polling stations counted at that time. Around 20:00 Lula was considered the winner (the peak seen in the previous chart was at that time).
Candidate websites: in the end, winner takes all?
For Lula-related domains, there are clear spikes around the first round of elections on October 2. A 13x spike was observed on October 1 at around 21:00 local time. Two notable spikes were observed on October 2 — one at 16.7x above baseline at 09:00 local time, and the other at 10.7x above baseline at 21:00 local time. During the October 30 runoff election, only one clear spike was observed. The spike, at 16.7x above baseline, occurred at around 20:00, coincident with the time Lula was being announced as the winner.
For Bolsonaro-related domains, we observed a different pattern. Increased traffic as compared to the baseline is visible in the days leading up to the first round election, reaching 10x on September 30. On October 2, a 8x spike above baseline was seen at 18:00 local time. However, the two most significant spikes seen over the course of the month were observed on October 16, at 20x above baseline, a few hours after the first Lula-Bolsonaro television debate, and on October 25, at around 20:00, at 22x above baseline. That was the last week of campaigning before the October 30 runoff and when several polling predictions were announced. The second and last Bolsonaro-Lula debate was on October 28, and there’s a spike at 22:00 to Lula’s websites, and a smaller but also clear one at 21:00 to Bolsonaro’s websites).
News websites: more interest in the first round
With official election results being available more rapidly, DNS traffic for Brazilian news organization websites peaked much earlier in the evening than what we saw in France, for example, where more definitive election results arrived much later on election day. But another interesting trend here is how the first round, on October 2, had 9.1x more DNS traffic (compared with the September baseline), than what we saw during the runoff on October 30 (6.1x).
The way the results arrived faster also had an impact on the time of the peak, occurring at around 19:00 local time on October 30, as compared to around 20:00 on October 2.
At 19:45 local time on October 30, Lula was already the winner with more than 98% of the votes counted. After 20:00 there was a clear drop in DNS traffic to news organizations.
On October 2, it was only around 22:00 that it became official that there would be a runoff between Lula and Bolsonaro. Peak request volume was reached at 20:00 (9x), but traffic remained high (8x) at around 21:00 and until 22:00, like the following chart shows:
Conclusion: Real world events impact the Internet
Cloudflare Radar, our tool for Internet insights, can provide a unique perspective on how major global or national events impact the Internet. It is interesting to not only see that a real world event can impact Internet traffic (and different types of websites) for a whole country, but also see how much that impact is represented at specific times. It’s all about human behavior at relevant moments in time, like elections as a collective event is.
Cloudflare operates in more than 275 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. In many cases, these disruptions can be attributed to a physical event, while in other cases, they are due to an intentional government-directed shutdown. In this post, we review selected Internet disruptions observed by Cloudflare during the third quarter of 2022, supported by traffic graphs from Cloudflare Radar and other internal Cloudflare tools, and grouped by associated cause or common geography. The new Cloudflare Radar Outage Center provides additional information on these, and other historical, disruptions.
Government directed shutdowns
Unfortunately, for the last decade, governments around the world have turned to shutting down the Internet as a means of controlling or limiting communication among citizens and with the outside world. In the third quarter, this was an all too popular cause of observed disruptions, impacting countries and regions in Africa, the Middle East, Asia, and the Caribbean.
Iraq
As mentioned in our Q2 summary blog post, on June 27, the Kurdistan Regional Government in Iraq began to implement twice-weekly (Mondays and Thursday) multi-hour regional Internet shutdowns over the following four weeks, intended to prevent cheating on high school final exams. As seen in the figure below, these shutdowns occurred as expected each Monday and Thursday through July 21, with the exception of July 21. They impacted three governorates in Iraq, and lasted from 0630–1030 local time (0330–0730 UTC) each day.
In Cuba, an Internet disruption was observed between 0055-0150 local time (0455-0550 UTC) on July 15 amid reported anti-government protests in Los Palacios and Pinar del Rio.
Closing out the quarter, another significant disruption was observed in Cuba, reportedly in response to protests over the lack of electricity in the wake of Hurricane Ian. A complete outage is visible in the figure below between 2030 on September 29 and 0315 on September 30 local time (0030-0715 UTC on September 30).
Afghanistan
Telecommunications services were reportedly shut down in part of Kabul, Afghanistan on the morning of August 8. The figure below shows traffic dropping starting around 0930 local time (0500 UTC), recovering 11 hours later, around 2030 local time (1600 UTC).
Protests in Freetown, Sierra Leone over the rising cost of living likely drove the Internet disruptions observed within the country on August 10 & 11. The first one occurred between 1200-1400 local time (1200-1400 UTC) on August 10. While this outage is believed to have been government directed as a means of quelling the protests, Zoodlabs, which manages Sierra Leone Cable Limited, claimed that the outage was the result of “emergency technical maintenance on some of our international routes”.
A second longer outage was observed between 0100-0730 local time (0100-0730 UTC) on August 11, as seen in the figure below. These shutdowns follow similar behavior in years past, where Internet connectivity was shut off following elections within the country.
In Somaliland, local authorities reportedly cut off Internet service on August 11 ahead of scheduled opposition demonstrations. The figure below shows a complete Internet outage in Woqooyi Galbeed between 0645-1355 local time (0345-1055 UTC.)
At a network level, the observed outage was due to a loss of traffic from AS37425 (SomCable) and AS37563 (Somtel), as shown in the figures below. Somtel is a mobile services provider, while SomCable is focused on providing wireline Internet access.
India
India is no stranger to government-directed Internet shutdowns, taking such action hundreds of times over the last decade. This may be changing in the future, however, as the country’s Supreme Court ordered the Ministry of Electronics and Information Technology (MEITY) to reveal the grounds upon which it imposes or approves Internet shutdowns. Until this issue is resolved, we will continue to see regional shutdowns across the country.
One such example occurred in Assam, where mobile Internet connectivity was shut down to prevent cheating on exams. The figure below shows that these shutdowns were implemented twice daily on August 21 and August 28. While the shutdowns were officially scheduled to take place between 1000-1200 and 1400-1600 local time (0430-0630 and 0830-1030 UTC), some providers reportedly suspended connectivity starting in the early morning.
In addition to multi-hour outages in Sanadij and Tehran province on September 19 and 21 that were covered in a blog post, three mobile network providers — AS44244 (Irancell), AS57218 (RighTel), and AS197207 (MCCI) — implemented daily Internet “curfews”, generally taking place between 1600 and midnight local time (1230-2030 UTC), although the start times varied on several days. These regular shutdowns are clearly visible in the figure below, and continued into early October.
As noted in the blog post, access to DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) services was also blocked in Iran starting on September 20, and in a move that is likely related, connections over HTTP/3 and QUIC were blocked starting on September 22, as shown in the figure below from Cloudflare Radar.
Natural disasters
Natural disasters such as earthquakes and hurricanes wreak havoc on impacted geographies, often causing loss of life, as well as significant structural damage to buildings of all types. Infrastructure damage is also extremely common, with widespread loss of both electrical power and telecommunications infrastructure.
Papua New Guinea
On September 11, a 7.6 magnitude earthquake struck Papua New Guinea, resulting in landslides, cracked roads, and Internet connectivity disruptions. Traffic to the country dropped by 26% just after 1100 local time (0100 UTC) . The figure below shows that traffic volumes remained lower into the following day as well. An announcement from PNG DataCo, a local provider, noted that the earthquake “has affected the operations of the Kumul Submarine Cable Network (KSCN) Express Link between Port Moresby and Madang and the PPC-1 Cable between Madang and Sydney.” This damage, they stated, resulted in the observed outage and degraded service.
Several major hurricanes plowed their way up the east coast of North America in late September, causing significant damage, resulting in Internet disruptions. On September 18, island-wide power outages caused by Hurricane Fiona disrupted Internet connectivity on Puerto Rico. As the figure below illustrates, it took over 10 days for traffic volumes to return to expected levels. Luma Energy, the local power company, kept customers apprised of repair progress through regular updates to its Twitter feed.
Two days later, Hurricane Fiona slammed the Turks and Caicos islands, causing flooding and significant damage, as well as disrupting Internet connectivity. The figure below shows traffic starting to drop below expected levels around 1245 local time (1645 UTC) on September 20. Recovery took approximately a day, with traffic returning to expected levels around 1100 local time (1500 UTC) on September 21.
Continuing to head north, Hurricane Fiona ultimately made landfall in the Canadian province of Nova Scotia on September 24, causing power outages and disrupting Internet connectivity. The figure below shows that the most significant impact was seen in Nova Scotia. As Nova Scotia Power worked to restore service to customers, traffic volumes gradually increased, as seen in the figure below. By September 29, traffic volumes on the island had returned to normal levels.
Hurricane Ian
On September 28, Hurricane Ian made landfall in Florida, and was the strongest hurricane to hit Florida since Hurricane Michael in 2018. With over four million customers losing power due to damage from the storm, a number of cities experienced associated Internet disruptions. Traffic from impacted cities dropped significantly starting around 1500 local time (1900 UTC), and as the figure below shows, recovery has been slow, with traffic levels still not back to pre-storm volumes more than two weeks later.
In addition to power outages caused by earthquakes and hurricanes, a number of other power outages caused multi-hour Internet disruptions during the third quarter.
Iran
A reported power outage in a key data center building disrupted Internet connectivity for customers of local ISP Shatel in Iran on July 25. As seen in the figure below, traffic dropped significantly at approximately 0715 local time (0345 UTC). Recovery began almost immediately, with traffic nearing expected levels by 0830 local time (0500 UTC).
Venezuela
Electrical issues frequently disrupt Internet connectivity in Venezuela, and the independent @vesinfiltro Twitter account tracks these events closely. One such example occurred on August 9, when electrical issues disrupted connectivity across multiple states, including Mérida, Táchira, Barinas, Portuguesa, and Estado Trujillo. The figure below shows evidence of two disruptions, the first around 1340 local time (1740 UTC) and the second a few hours later, starting at around 1615 local time (2015 UTC). In both cases, traffic volumes appeared to recover fairly quickly.
On September 5, a power outage in Oman impacted energy, aviation, and telecommunications services. The latter is evident in the figure below, which shows the country’s traffic volume dropping nearly 60% when the outage began just before 1515 local time (0915 UTC). Although authorities claimed that “the electricity network would be restored within four hours,” traffic did not fully return to normal levels until 0400 local time on September 6 (2200 UTC on September 5) the following day, approximately 11 hours later.
Ukraine
Over the last seven-plus months of war in Ukraine, we have observed multiple Internet disruptions due to infrastructure damage and power outages related to the fighting. We have covered these disruptions in our first and second quarter summary blog posts, and continue to do so on our @CloudflareRadar Twitter account as they occur. Power outages were behind Internet disruptions observed in Kharkiv on September 11, 12, and 13.
The figure below shows that the first disruption started around 2000 local time (1700 UTC) on September 11. This near-complete outage lasted just over 12 hours, with traffic returning to normal levels around 0830 local time (0530 UTC) on the 12th. However, later that day, another partial outage occurred, with a 50% traffic drop seen at 1330 local time (1030 UTC). This one was much shorter, with recovery starting approximately an hour later. Finally, a nominal disruption is visible at 0800 local time (0500 UTC) on September 13, with lower than expected traffic volumes lasting for around five hours.
Cable damage
Damage to both terrestrial and submarine cables have caused many Internet disruptions over the years. The recent alleged sabotage of the sub-sea Nord Stream natural gas pipelines has brought an increasing level of interest from European media (including Swiss and French publications) around just how important submarine cables are to the Internet, and an increasing level of concern among policymakers about the safety of these cable systems and the potential impact of damage to them. However, the three instances of cable damage reviewed below are all related to terrestrial cable.
Iran
On August 1, a reported “fiber optic cable” problem caused by a fire in a telecommunications manhole disrupted connectivity across multiple network providers, including AS31549 (Aria Shatel), AS58224 (TIC), AS43754 (Asiatech), AS44244 (Irancell), and AS197207 (MCCI). The disruption started around 1215 local time (0845 UTC) and lasted for approximately four hours. Because it impacted a number of major wireless and wireline networks, the impact was visible at a country level as well, as seen in the figure below.
Pakistan
Cable damage due to heavy rains and flooding caused several Internet disruptions in Pakistan in August. The first notable disruption occurred on August 19, starting around 0700 local time (0200 UTC) and lasted just over six and a half hours. On August 22, another significant disruption is also visible, starting at 2250 local time (1750 UTC), with a further drop at 0530 local time (0030 UTC) on the 23rd. The second more significant drop was brief, lasting only 45 minutes, after which traffic began to recover.
Haiti
Amidst protests over fuel price hikes, fiber cuts in Haiti caused Internet outages on multiple network providers. Starting at 1500 local time (1900 UTC) on September 14, traffic on AS27759 (Access Haiti) fell to zero. According to a (translated) Twitter post from the provider, they had several fiber optic cables that were cut in various areas of the country, and blocked roads made it “really difficult” for their technicians to reach the problem areas. Repairs were eventually made, with traffic starting to increase again around 0830 local time (1230 UTC) on September 15, as shown in the figure below.
Access Haiti provides AS27774 (Haiti Networking Group) with Internet connectivity (as an “upstream” provider), so the fiber cut impacted their connectivity as well, causing the outage shown in the figure below.
Technical problems
As a heading, “technical problems” can be a catch-all, referring to multiple types of issues, including misconfigurations and routing problems. However, it is also sometimes the official explanation given by a government or telecommunications company for an observed Internet disruption.
Rogers
Arguably the most significant Internet disruption so far this year took place on AS812 (Rogers), one of Canada’s largest Internet service providers. At around 0845 UTC on July 8, a near complete loss of traffic was observed, as seen in the figure below.
The figure below shows that small amounts of traffic were seen from the network over the course of the outage, but it took nearly 24 hours for traffic to return to normal levels.
A notice posted by the Rogers CEO explained that “We now believe we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction early Friday morning. We disconnected the specific equipment and redirected traffic, which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.” A Cloudflare blog post covered the Rogers outage in real-time, highlighting related BGP activity and small increases of traffic.
Chad
A four-hour near-complete Internet outage took place in Chad on August 12, occurring between 1045 and 1300 local time (0945 to 1400 UTC). Authorities in Chad said that the disruption was due to a “technical problem” on connections between Sudachad and networks in Cameroon and Sudan.
Unknown
In many cases, observed Internet disruptions are attributed to underlying causes thanks to statements by service providers, government officials, or media coverage of an associated event. However, for some disruptions, no published explanation or associated event could be found.
On August 11, a multi-hour outage impacted customers of US telecommunications provider Centurylink in states including Colorado, Iowa, Missouri, Montana, New Mexico, Utah, and Wyoming, as shown in the figure below. The outage was also visible in a traffic graph for AS209, the associated autonomous system.
Starlink
On August 30, satellite Internet provider suffered a global service disruption, lasting between 0630-1030 UTC as seen in the figure below.
Conclusion
As part of Cloudflare’s Birthday Week at the end of September, we launched the Cloudflare Radar Outage Center (CROC). The CROC is a section of our new Radar 2.0 site that archives information about observed Internet disruptions. The underlying data that powers the CROC is also available through an API, enabling interested parties to incorporate data into their own tools, sites, and applications. For regular updates on Internet disruptions as they occur and other Internet trends, follow @CloudflareRadar on Twitter.
Welcome to our DDoS Threat Report for the third quarter of 2022. This report includes insights and trends about the DDoS threat landscape – as observed across Cloudflare’s global network.
Multi-terabit strong DDoS attacks have become increasingly frequent. In Q3, Cloudflare automatically detected and mitigated multiple attacks that exceeded 1 Tbps. The largest attack was a 2.5 Tbps DDoS attack launched by a Mirai botnet variant, aimed at the Minecraft server, Wynncraft. This is the largest attack we’ve ever seen from the bitrate perspective.
It was a multi-vector attack consisting of UDP and TCP floods. However, Wynncraft, a massively multiplayer online role-playing game Minecraft server where hundreds and thousands of users can play on the same server, didn’t even notice the attack, since Cloudflare filtered it out for them.
The 2.5 Tbps DDoS attack that targeted Wynncraft — launched by Mirai
General DDoS attack trends
Overall this quarter, we’ve seen:
An increase in DDoS attacks compared to last year.
Longer-lasting volumetric attacks, a spike in attacks generated by the Mirai botnet and its variants.
Surges in attacks targeting Taiwan and Japan.
Application-layer DDoS attacks
HTTP DDoS attacks increased by 111% YoY, but decreased by 10% QoQ.
HTTP DDoS attacks targeting Taiwan increased by 200% QoQ; attacks targeting Japan increased by 105% QoQ.
Reports of Ransom DDoS attacks increased by 67% YoY and 15% QoQ.
Network-layer DDoS attacks
L3/4 DDoS attacks increased by 97% YoY and 24% QoQ.
L3/4 DDoS attacks by Mirai botnets increased by 405% QoQ.
The Gaming / Gambling industry was the most targeted by L3/4 DDoS attacks including a massive 2.5 Tbps DDoS attack.
This report is based on DDoS attacks that were automatically detected and mitigated by Cloudflare’s DDoS Protection systems. To learn more about how it works, check out this deep-dive blog post.
Ransom attacks
Ransom DDoS attacks are attacks where the attacker demands a ransom payment, usually in the form of Bitcoin, to stop/avoid the attack. In Q3, 15% of Cloudflare customers that responded to our survey reported being targeted by HTTP DDoS attacks accompanied by a threat or a ransom note. This represents a 15% increase QoQ and 67% increase YoY of reported ransom DDoS attacks.
Distribution of Ransom DDoS attacks by quarter
Diving into Q3, we can see that since June 2022, there was a steady decline in reports of ransom attacks. However, in September, the reports of ransom attacks spiked again. In the month of September, almost one out of every four respondents reported receiving a ransom DDoS attack or threat — the highest month in 2022 so far.
Distribution of Ransom DDoS attacks by month
How we calculate Ransom DDoS attack trends Our systems constantly analyze traffic and automatically apply mitigation when DDoS attacks are detected. Each DDoS’d customer is prompted with an automated survey to help us better understand the nature of the attack and the success of the mitigation. For over two years, Cloudflare has been surveying attacked customers. One of the questions in the survey asks the respondents if they received a threat or a ransom note demanding payment in exchange to stop the DDoS attack. Over the past year, on average, we collected 174 responses per quarter. The responses of this survey are used to calculate the percentage of Ransom DDoS attacks.
Application-layer DDoS attacks
Application-layer DDoS attacks, specifically HTTP DDoS attacks, are attacks that usually aim to disrupt a web server by making it unable to process legitimate user requests. If a server is bombarded with more requests than it can process, the server will drop legitimate requests and – in some cases – crash, resulting in degraded performance or an outage for legitimate users.
Application-layer DDoS attack trends
When we look at the graph below, we can see a clear trend of approximately 10% decrease in attacks each quarter since 2022 Q1. However, despite the downward trend, when comparing Q3 of 2022 to Q3 of 2021, we can see that HTTP DDoS attacks still increased by 111% YoY.
Distribution of HTTP DDoS attacks by quarter
When we dive into the months of the quarter, attacks in September and August were fairly evenly distributed; 36% and 35% respectively. In July, the amount of attacks was the lowest for the quarter (29%).
Distribution of HTTP DDoS attacks by month in 2022 Q3
Application-layer DDoS attacks by industry
By bucketing the attacks by our customers’ industry of operation, we can see that HTTP applications operated by Internet companies were the most targeted in Q3. Attacks on the Internet industry increased by 131% QoQ and 300% YoY.
The second most attacked industry was the Telecommunications industry with an increase of 93% QoQ and 2,317% (!) YoY. In third place was the Gaming / Gambling industry with a more conservative increase of 17% QoQ and 36% YoY.
Top industries targeted by HTTP DDoS attacks in 2022 Q3
Application-layer DDoS attacks by target country
Bucketing attacks by our customers’ billing address gives us an understanding of which countries are more attacked. HTTP applications operated by US companies were the most targeted in Q3. US-based websites saw an increase of 60% QoQ and 105% YoY in attacks targeting them. After the US, was China with a 332% increase QoQ and an 800% increase YoY.
Looking at Ukraine, we can see that attacks targeting Ukrainian websites increased by 67% QoQ but decreased by 50% YoY. Furthermore, attacks targeting Russian websites increased by 31% QoQ and 2,400% (!) YoY.
In East Asia, we can see that attacks targeting Taiwanese companies increased by 200% QoQ and 60% YoY, and attacks targeting Japanese companies increased by 105% QoQ.
Top countries targeted by HTTP DDoS attacks in 2022 Q3
When we zoom in on specific countries, we can identify the below trends that may reveal interesting insights regarding the war in Ukraine and geopolitical events in East Asia:
In Ukraine, we see a surprising change in the attacked industries. Over the past two quarters, Broadcasting, Online Media and Publishing companies were targeted the most in what appeared to be an attempt to silence information and make it unavailable to civilians. However, this quarter, those industries dropped out of the top 10 list. Instead, the Marketing & Advertising industry took the lead (40%), followed by Education companies (20%), and Government Administration (8%).
In Russia, attacks on the Banking, Financial Services and Insurance (BFSI) industry continue to persist (25%). Be that as it may, attacks on the BFSI sector still decreased by 44% QoQ. In second place is the Events Services industry (20%), followed by Cryptocurrency (16%), Broadcast Media (13%), and Retail (11%). A significant portion of the attack traffic came from Germany-based IP addresses, and the rest were globally distributed.
In Taiwan, the two most attacked industries were Online Media (50%) and Internet (23%). Attacks to those industries were globally distributed indicating the usage of botnets.
In Japan, the most attacked industry was Internet/Media & Internet (52%), Business Services (12%), and Government – National (11%).
Application-layer DDoS attack traffic by source country
Before digging into specific source country metrics, it is important to note that while country of origin is interesting, it is not necessarily indicative of where the attacker is located. Oftentimes with DDoS attacks, they are launched remotely, and attackers will go to great lengths to hide their actual location in an attempt to avoid being caught. If anything, it is indicative of where botnet nodes are located. With that being said, by mapping the attacking IP address to their location, we can understand where attack traffic is coming from.
After two consecutive quarters, China replaced the US as the main source of HTTP DDoS attack traffic. In Q3, China was the largest source of HTTP DDoS attack traffic. Attack traffic from China-registered IP addresses increased by 29% YoY and 19% QoQ. Following China was India as the second-largest source of HTTP DDoS attack traffic — an increase of 61% YoY. After India, the main sources were the US and Brazil.
Looking at Ukraine, we can see that this quarter there was a drop in attack traffic originating from Ukrainian and Russian IP addresses — a decrease of 29% and 11% QoQ, respectively. However, YoY, attack traffic from within those countries still increased by 47% and 18%, respectively.
Another interesting data point is that attack traffic originating from Japanese IP addresses increased by 130% YoY.
Top source countries of HTTP DDoS attacks in 2022 Q3
Network-layer DDoS attacks
While application-layer attacks target the application (Layer 7 of the OSI model) running the service that end users are trying to access (HTTP/S in our case), network-layer attacks aim to overwhelm network infrastructure (such as in-line routers and servers) and the Internet link itself.
Network-layer DDoS attack trends
In Q3, we saw a large surge in L3/4 DDoS attacks — an increase of 97% YoY and a 24% QoQ. Furthermore, when we look at the graph we can see a clear trend, over the past three quarters, of an increase in attacks.
Distribution of L3/4 DDoS attacks by quarter
Drilling down into the quarter, it’s apparent that the attacks were, for the most part, evenly distributed throughout the quarter — with a slightly larger share for July.
Distribution of L3/4 DDoS attacks by month in 2022 Q3
Network-layer DDoS attacks by Industry
The Gaming / Gambling industry was hit by the most L3/4 DDoS attacks in Q3. Almost one out of every five bytes Cloudflare ingested towards Gaming / Gambling networks was part of a DDoS attack. This represents a whopping 381% increase QoQ.
The second most targeted industry was Telecommunications — almost 6% of bytes towards Telecommunications networks were part of DDoS attacks. This represents a 58% drop from the previous quarter where Telecommunications was the top most attacked industry by L3/4 DDoS attacks.
Following were the Information Technology and Services industry along with the Software industry. Both saw significant growth in attacks — 89% and 150% QoQ, respectively.
Top industries targeted by L3/4 DDoS attacks in 2022 Q3
Network-layer DDoS attacks by target country
In Q3, Singapore-based companies saw the most L3/4 DDoS attacks — over 15% of all bytes to their networks were associated with a DDoS attack. This represents a dramatic 1,175% increase QoQ.
The US comes in second after a 45% decrease QoQ in attack traffic targeting US networks. In third, China, with a 62% QoQ increase. Attacks on Taiwan companies also increased by 200% QoQ.
Top countries targeted by L3/4 DDoS attacks in 2022 Q3
Network-layer DDoS attacks by ingress country
In Q3, Cloudflare’s data centers in Azerbaijan saw the largest percentage of attack traffic. More than a third of all packets ingested there were part of a L3/4 DDoS attack. This represents a 44% increase QoQ and a huge 59-fold increase YoY.
Similarly, our data centers in Tunisia saw a dramatic increase in attack packets – 173x the amount in the previous year. Zimbabwe and Germany also saw significant increases in attacks.
Zooming into East Asia, we can see that our data centers in Taiwan saw an increase of attacks — 207% QoQ and 1,989% YoY. We saw similar numbers in Japan where attacks increased by 278% QoQ and 1,921% YoY.
Looking at Ukraine, we actually see a dip in the amount of attack packets we observed in our Ukraine-based and Russia-based data centers — 49% and 16% QoQ, respectively.
Top Cloudflare data center locations with the highest percentage of DDoS attack traffic in 2022 Q3
Attack vectors & Emerging threats
An attack vector is the method used to launch the attack or the method of attempting to achieve denial-of-service. With a combined share of 71%, SYN floods and DNS attacks remain the most popular DDoS attack vectors in Q3.
Top attack vectors in 2022 Q3
Last quarter, we saw a resurgence of attacks abusing the CHARGEN protocol, the Ubiquity Discovery Protocol, and Memcached reflection attacks. While the growth in Memcached DDoS attacks also slightly grew (48%), this quarter, there was a more dramatic increase in attacks abusing the BitTorrent protocol (1,221%), as well as attacks launched by the Mirai botnet and its variants.
BitTorrent DDoS attacks increased by 1,221% QoQ The BitTorrent protocol is a communication protocol that’s used for peer to peer file sharing. To help the BitTorrent clients find and download the files efficiently, BitTorrent clients may use BitTorrent Trackers or Distributed Hash Tables (DHT) to identify the peers that are seeding the desired file. This concept can be abused to launch DDoS attacks. A malicious actor can spoof the victim’s IP address as a seeder IP address within Trackers and DHT systems. Then clients would request the files from those IPs. Given a sufficient number of clients requesting the file, it can flood the victim with more traffic than it can handle.
Mirai DDoS attacks increased by 405% QoQ Mirai is malware that infects smart devices that run on ARC processors, turning them into a network of bots that can be used to launch DDoS attacks. This processor runs a stripped-down version of the Linux operating system. If the default username-and-password combo is not changed, Mirai is able to log in to the device, infect it, and take over. The botnet operator can instruct the botnet to launch a flood of UDP packets at the victim’s IP address to bombard them.
Top emerging threats in 2022 Q3
Network-layer DDoS attacks by Attack Rates & Duration
While Terabit-strong attacks are becoming more frequent, they are still the outliers. The majority of attacks are tiny (in terms of Cloudflare scale). Over 95% of attacks peaked below 50,000 packets per second (pps) and over 97% below 500 Megabits per second (Mbps). We call this “cyber vandalism”.
What is cyber vandalism? As opposed to “classic” vandalism where the purpose is to cause deliberate destruction of or damage to public or private physical property — such as graffiti on the side of a building — in the cyberworld, cyber vandalism is the act of causing deliberate damage to Internet properties. Today the source codes for various botnets are available online and there are a number of free tools that can be used to launch a flood of packets. By directing those tools to Internet properties, any script-kid can use those tools to launch attacks against their school during exam season or any other website they desire to take down or disrupt. This is as opposed to organized crime, Advanced Persistent Threat actors, and state-level actors that can launch much larger and sophisticated attacks.
Distribution of DDoS attacks by bitrate in 2022 Q3
Similarly, most of the attacks are very short and end within 20 minutes (94%). This quarter we did see an increase of 9% in attacks of 1-3 hours, and a 3% increase in attacks over 3 hours — but those are still the outliers.
QoQ change in the duration of DDoS attacks in 2022 Q3
Even with the largest attacks, such as the 2.5 Tbps attack we mitigated earlier this quarter, and the 26M request per second attack we mitigated back in the summer, the peak of the attacks were short-lived. The entire 2.5 Tbps attack lasted about 2 minutes, and the peak of the 26M rps attack only 15 seconds. This emphasizes the need for automated, always-on solutions. Security teams can’t respond quick enough. By the time the security engineer looks at the PagerDuty notification on their phone, the attack has subsided.
Summary
Attacks may be initiated by humans, but they are executed by bots — and to play to win, you must fight bots with bots. Detection and mitigation must be automated as much as possible, because relying solely on humans puts defenders at a disadvantage. Cloudflare’s automated systems constantly detect and mitigate DDoS attacks for our customers, so they don’t have to.
Over the years, it has become easier, cheaper, and more accessible for attackers and attackers-for-hire to launch DDoS attacks. But as easy as it has become for the attackers, we want to make sure that it is even easier – and free – for defenders of organizations of all sizes to protect themselves against DDoS attacks of all types. We’ve been providing unmetered and unlimited DDoS protection for free to all of our customers since 2017 — when we pioneered the concept.
Cloudflare’s mission is to help build a better Internet. A better Internet is one that is more secure, faster, and reliable for everyone – even in the face of DDoS attacks.
The Internet is a living organism. Technology changes, shifts in human behavior, social events, intentional disruptions, and other occurrences change the Internet in unpredictable ways, even to the trained eye.
Cloudflare Radar has long been the place to visit for accessing data and getting unique insights into how people and organizations are using the Internet across the globe, as well as those unpredictable changes to the Internet.
One of the most popular features on Radar has always been the “Most Popular Domains,” with both global and country-level perspectives. Domain usage signals provide a proxy for user behavior over time and are a good representation of what people are doing on the Internet.
Today, we’re going one step further and launching a new dataset called Radar Domain Rankings (Beta). Domain Rankings is based on aggregated 1.1.1.1 resolver data that is anonymized in accordance with our privacy commitments. The dataset aims to identify the top most popular domains based on how people use the Internet globally, without tracking individuals’ Internet use.
There are a few reasons why we’re doing this now. One is obviously to improve our Radar features with better data and incorporate new learnings. But also, ranking lists are used all over the Internet in all sorts of systems. One of the most used and trusted sources of domain rankings was Alexa, but that service was recently deprecated. We believe we are in a good position to provide a strong alternative.
Let’s see how we built it.
Differences in domain names
Before we dig into the data science behind Domain Rankings, it’s important to understand what a domain and DNS are. Internet domain names are human-readable dot-separated letters, digits and hyphens that correspond to a network resource, like a server or a website. However, your computer and applications don’t know what to do with a domain name; they need IP addresses to send and receive information over the network. DNS is the system that converts, or resolves, a domain name into an IP address. Think of it as an Internet phonebook for domain names.
Note: This is a simplification. A new standard called Internationalized Domain Names, or IDN, allows using Unicode strings in domain names.
Each dot defines a new hierarchy level, reading right to left. Domains can have multiple levels of depth. The highest level corresponds to country code top-level domains (ccTLDs) like .uk, .fr or .pt, or generic top-level domains (gTLDs) like .com, .org, or .net. These are normally assigned to and managed by either country-level entities or administrative organizations operating a registry.
Then there are the second-level domains like cloudflare.com or google.com. These are normally purchased and registered by individuals or organizations, which are then free to create and manage as many hostnames and hierarchy levels as they want.
Unfortunately, however, there are exceptions. For instance, many countries use second-level domain registration. One such example is the United Kingdom, where commercial domains can only be registered under the .co.uk hierarchy. That’s why Google in the UK isn’t google.uk, but rather google.co.uk.
But that’s not all. Some countries use 3rd level domain registrations. One example is Japan, which offers Regional Domain registration under cities like *.aisai.aichi.jp.
Projects like the Public Suffix List are a good starting point for understanding the variations involved, and how they affect validations and assumptions in other systems, such as cookies in web browsers.
Domain Rankings takes some of this nuance into account to inform the definition of our current ruleset:
We boil everything down to second-level domains, such as cloudflare.com or google.com.
However, if the second level is .edu, .com, .org, .gov, .net, .gov, .net, .co or .mil, then we use third-level domains.
We don’t distinguish between what we think is a website or an infrastructure system. A domain represents an Internet-available resource.
We will also semi-automate, curate and maintain a list of domains that map to popular platforms and services in the future. Example: fb.audio, fb.com, fb.watch, all map to a “facebook” platform.
Defining popularity
Definitions are important. We established what we consider a domain, but what does domain popularity mean exactly? Our research showed that the volume of traffic generated to a given domain doesn’t really work as a proxy for what we perceive as popular. Instead, Domain Rankings looks at the size of the population of users that look up a domain per unit of time. The more people who are interested in a domain, the more popular it is.
Sounds pretty straightforward, right? Well, it’s not. Our databases don’t have cookies, IPs, or other tracking artifacts, and we strip information that leads to identifying an individual from all of our data, by design.
The good news, however, is that we do a very good job at identifying automated traffic (for instance, you can read about Bot Management and how we use Machine Learning to detect bots in HTTP traffic in our blog) and we found we could develop a reasonable proxy for the unique users metric without sacrificing privacy (using other data points that we store for a limited period of time, like the ASN and high-level geolocation information of the request or the Cloudflare data center that served it).
Domain Rankings’ popularity metric is best described as the estimated relative size of the user population that accesses a domain over some period of time.
Our approach
We announced 1.1.1.1, our privacy-first consumer DNS resolver in 2018, and over the years it’s grown to become one of the top DNS services in the world. 1.1.1.1 is also part of a Research Agreement with APNIC in which we collaborate with them doing public research and DNS data insights.
The data we collect from it honors our privacy commitments, and is aggregated and stripped of any information that could lead to identifying or tracking users. We conducted a privacy examination by a Big Four accounting firm to determine whether the 1.1.1.1 resolver was effectively configured to meet our privacy commitments. You can read more about it in this blog, and the full report is publicly available on our compliance page.
Even without this personally identifying information, the resulting collection is vast and representative of Internet activity.
The 1.1.1.1 service is used in many ways. Regular (human) Internet users use it as their DNS resolver, either because they explicitly configured it in their devices, or their ISP did, or because they use WARP, or their browser uses 1.1.1.1 under the hood. However, servers and cloud infrastructure, IoT devices, home routers, and bots also use 1.1.1.1 extensively, which creates a lot of challenges for us when trying to identify human traffic.
We’ve been using DNS data to calculate the top and trending domains found on both the global and country pages on Cloudflare Radar. It’s been quite a learning experience trying to improve these lists. We have implemented aggregations, counts, filters, handling exceptions, and tried reducing noise, and yet they’re far from perfect. We felt that there had to be a better way.
We’ve spent the last six months building a variety of machine learning models to help us predict the rank of a domain.
Building the model was no easy feat. We experimented with multiple regression types first, to know exactly what the model was doing, and then more complex algorithms to get better performance. We played with different datasets, changed the population groups, variables (features), and combinations of variables, and used synthetic data.
After evaluation, one of our first conclusions was that building a model that could produce good results for the highest ranked domains and the long tail would be difficult.
The paper “A Long Way to the Top: Significance, Structure, and Stability of Internet Top Lists” describes this problem well. “The ranking of domains in the long tail should be based on significantly smaller and hence less reliable numbers.” Talking to our Research Team who submitted the collaboration paper “Toppling Top Lists: Evaluating the Accuracy of Popular Website Lists” to IMC 2022, got us to the same conclusion: the most popular domains (like google.com and facebook.com) have feature values disproportionally higher than the lower-ranked domains.
Therefore, we selected the two models that performed best. One model was trained on the population with the highest feature values, uses more features, and is used to generate the ordered top 100 domain list. A second model was trained on a more general group of domains, uses fewer features, and is used to get the top one million most popular domains, which we then divide into ranking buckets.
These buckets are ranked, but each bucket’s contents are intentionally unordered. For example, the second bucket of 10,000 most popular domains includes the set of domains that rank from 10,001 to 20,000, but give no further indication of the individual ranking of domains in that bucket. Given the size of some of these buckets and the window of time we use to populate them, they will inherently be exposed to more instability, too. We feel this is a good compromise between the described natural uncertainties of our long tail model and providing a reasonable idea of how close to the top a domain is.
Results
It’s important to mention there is no global view that can establish the perfect rank, and there’s no easy mechanism to confirm if a ranking is, ultimately, good. Data-driven results are always subject to some bias and skewing, related to the context of the organizations and systems that collect them. Sometimes all that can be done is to be transparent about potential sources of bias. The geographical distribution of customers and users, product characteristics, platform features, and behavioral diversity play an essential role in the final result. We are presenting the Cloudflare view, what we see.
Having said this, Cloudflare sits in a privileged position and handles a significant amount of Internet traffic. We have plenty of signals we can extract from our aggregated data, and believe that makes it possible to generate high quality domain rankings.
Domain Rankings are available today. You can head up to the Domains page and check it out:
Ordered list of the top 100 most popular domains globally and per country, based on our first model. Last 24 hours, updated daily.
Unordered global most popular domains datasets divided into buckets of the following sizes: 200, 500, 1,000, 2,000, 5,000, 10,000, 20,000, 50,000, 100,000, 200,000, 500,000, 1,000,000. Last 7 days, updated weekly.
Next steps
We will keep improving Domain Rankings and monitoring the results. Anyone can access them on Cloudflare Radar, read the results, and download the CSV files.
Feel free to explore our Domain Rankings and share feedback with us.
Historically, Cloudflare has covered large-scale Internet outages with timely blog posts, such as those published for Iran, Sudan, Facebook, and Syria. While we still explore such outages on the Cloudflare blog, throughout 2022 we have ramped up our monitoring of Internet outages around the world, posting timely information about those outages to @CloudflareRadar on Twitter.
Furthermore, this initial release is also laying the groundwork for the CROC to become a first stop and key resource for civil society organizations, journalists/news media, and impacted parties to get information on, or corroboration of, reported or observed Internet outages.
What information does the CROC contain?
At launch, the CROC includes summary information about observed outage events. This information includes:
Location: Where was the outage?
ASN: What autonomous system experienced a disruption in connectivity?
Type: How broad was the outage? Did connectivity fail nationwide, or at a sub-national level? Did just a single network provider have an outage?
Scope: If it was a sub-national/regional outage, what state or city was impacted? If it was a network-level outage, which one?
Cause: Insight into the likely cause of the outage, based on publicly available information. Historically, some have been government directed shutdowns, while others are caused by severe weather or natural disasters, or by infrastructure issues such as cable cuts, power outages, or filtering/blocking.
Start time: When did the outage start?
End time: When did the outage end?
Using the CROC
Radar pages, including the main landing page, include a card displaying information about the most recently observed outage, along with a link to the CROC. The CROC will also be linked from the left-side navigation bar
Within the CROC, we have tried to keep the interface simple and easily understandable. Based on the selected time period, the global map highlights locations where Internet outages have been observed, along with a tooltip showing the number of outages observed during that period. Similarly, the table includes information (as described above) about each observed outage, along with a link to more information. The linked information may be a Twitter post, a blog post, or a custom Radar graph.
As mentioned in the Radar 2.0 launch blog post, we launched an associated API alongside the new site. Outage information is available through this API as well — in fact, the CROC is built on top of this API. Interested parties, including civil society organizations, data journalists, or others, can use the API to integrate the available outage data with their own data sets, build their own related tools, or even develop a custom interface.
Information about the related API endpoint and how to access it can be found in the Cloudflare API documentation.
We recognize that some users may want to download the whole list of observed outages for local consumption and analysis. They can do so by clicking the “Download CSV” link below the table.
The status page the Internet needs (coming soon)
Today’s launch of the Cloudflare Radar Outage Center is just the beginning, as we plan to improve it over time. This includes increased automation of outage detection, enabling us to publish more timely information through both the API and the CROC tool, which is important for members of the community that track and respond to Internet outages. We are also exploring how we can use synthetic monitoring in combination with other network-level performance and availability information to detect outages of popular consumer and business applications/platforms.
And anyone who uses a cloud platform provider (such as AWS) will know that those companies’ status pages take a surprisingly long time to update when there’s an outage. It’s very common to experience difficulty accessing a service, see hundreds of messages on Twitter and message boards about a service being down, only to go to the cloud platform provider’s status page and see everything green and “All systems normal”.
For the last few months we’ve been monitoring the performance of cloud platform providers to see if we can detect when they go down and provide our own, real time status page for them. We believe we can and Cloudflare Radar Outage Center will be extended to include cloud service providers and give the Internet the status page it needs.
If you have questions about the CROC, or suggestions for features that you would like to see, please reach out to us on Twitter at @CloudflareRadar.
Cloudflare Radar was launched two years ago to give everyone access to the Internet trends, patterns and insights Cloudflare uses to help improve our service and protect our customers.
Until then, these types of insights were only available internally at Cloudflare. However, true to our mission of helping build a better Internet, we felt everyone should be able to look behind the curtain and see the inner workings of the Internet. It’s hard to improve or understand something when you don’t have clear visibility over how it’s working.
On Cloudflare Radar you can find timely graphs and visualizations on Internet traffic, security and attacks, protocol adoption and usage, and outages that might be affecting the Internet. All of these can be narrowed down by timeframe, country, and Autonomous System (AS). You can also find interactive deep dive reports on important subjects such as DDoS and the Meris Botnet. It’s also possible to search for any domain name to see details such as SSL usage and which countries their visitors are coming from.
Since launch, Cloudflare Radar has been used by NGOs to confirm the Internet disruptions their observers see in the field, by journalists looking for Internet trends related to an event in a country of interest or at volume of cyberattacks as retaliation to political sanctions, by analysts looking at the prevalence of new protocols and technologies, and even by brand PR departments using Cloudflare Radar data to analyze the online impact of a major sports event.
Cloudflare Radar has clearly become an important tool for many and, most importantly, we find it has helped shed light on parts of the Internet that deserve more attention and investment.
Introducing Cloudflare Radar 2.0
What has made Cloudflare Radar so valuable is that the data and insights it contains are unique and trustworthy. Cloudflare Radar shows aggregate data from across the massive spectrum of Internet traffic we see every day, presenting you with datasets you won’t find elsewhere.
However, there were still gaps. Today, on the second anniversary of Cloudflare Radar, we are launching Cloudflare Radar 2.0 in beta. It will address three common pieces of feedback from users:
Ease of finding insights and data. The way information was structured on Cloudflare Radar made finding information daunting for some people. We are redesigning Cloudflare Radar so that it becomes a breeze.
Number of insights. We know many users have wanted to see insights about other important parts of the Internet, such as email. We have also completely redesigned the Cloudflare Radar backend so that we can quickly add new insights over the coming months (including insights into email).
Sharing insights.The options for sharing Cloudflare Radar insights were limited. We will now provide you the options you want, including downloadable and embeddable graphs, sharing to social media platforms, and an API.
Finding insights and data
On a first visit to the redesigned Cloudflare Radar homepage one will notice:
Prominent and intuitive filtering capabilities on the top bar. A global search bar is also coming soon.
Content navigation on the sidebar.
Content cards showing glanceable and timely information.
The content you find on the homepage are what we call “quick bytes”. Those link you to more in-depth content for that specific topic, which can also be found through the sidebar navigation.
At the top of the page you can search for a country, autonomous system number (ASN), domain, or report to navigate to a home page for that specific content. For example, the domain page for google.com:
The navigation sidebar allows you to find more detailed insights and data related to Traffic, Security & Attacks, Adoption & Usage, and Domains. (We will be adding additional topic areas in the future.) It also gives you quick access to the Cloudflare Radar Outage Center, a tool for tracking Internet disruptions around the world and to which we are dedicating a separate blog post, and to Radar Reports, which are interactive deep dive reports on important subjects such as DDoS and the Meris Botnet.
Within these topic pages (such as the one for Adoption & Usage shown above), you will find the quick bytes for the corresponding topic at the top, and quick bytes for related topics on the right. The quick bytes on the right allow you to quickly glance at and navigate to related sections.
In the middle of the page are the more detailed charts for the topic you’re exploring.
Sharing insights
Cloudflare Radar’s reason to exist is to make Internet insights available to everyone, but historically we haven’t been as flexible as our users would want. People could download a snapshot of the graph, but not much more.
With Cloudflare Radar 2.0 we will be introducing three major new ways of using Radar insights and data:
Social share. Cloudflare Radar 2.0 charts have a more modern and clean look and feel, and soon you’ll be able to share them directly on the social media platform of your choice. No more dealing with low quality screenshots.
Embeddable charts. The beautiful charts will also be able to be embedded directly into your webpage or blog – it will work just like a widget, always showing up-to-date information.
API. If you like the data on Cloudflare Radar but want to manipulate it further for analysis, visualization, or for posting your own charts, you’ll have the Cloudflare Radar API available to you starting today.
Note: The API is available today. To use the Cloudflare API you need an API token or key (more details here). Embedding charts and sharing directly to social are new features to be released later this year.
Technology changes
Cloudflare Radar 2.0 was built on a new technology stack; we will write a blog post about why and how we did it soon. A lot changed: we now have proper GraphQL data endpoints and a public API, the website runs on top of Cloudflare Pages and Workers, and we’re finally doing server-side rendering using Remix. We adopted SVG whenever possible, built our reusable data visualization components system, and are using Cosmos for visual TDD. These foundational changes will provide a better UX/UI to our users and give us speed when iterating and improving Cloudflare Radar in the future.
We hope you find this update valuable, and recommend you keep an eye on radar.cloudflare.com to see the new insights and topics we’ll be adding regularly. If you have any feedback, please send it to us through the Cloudflare Community.
Published reports indicate that the growing protests have resulted in at least eight deaths. Iran has a history of restricting Internet connectivity in response to protests, taking such steps in May 2022, February 2021, and November 2019. They have taken a similar approach to the current protests, including disrupting Internet connectivity, blocking social media platforms, and blocking DNS. The impact of these actions, as seen through Cloudflare’s data, are reviewed below.
On September 21, Internet disruptions started to become more widespread, with mobile networks effectively shut down nationwide. (Iran is a heavily mobile-centric country, with Cloudflare Radar reporting that 85% of requests are made from mobile devices.) Internet traffic from Iran Mobile Communications Company (AS197207) started to decline around 1530 UTC, and remained near zero until it started to recover at 2200 UTC, returning to “normal” levels by the end of the day.
Internet traffic from RighTel (AS57218) began to decline around 1630 UTC. After an outage lasting more than 12 hours, traffic returned at 0510 UTC.
Internet traffic from MTN Irancell (AS44244) began to drop just before 1700 UTC. After a 12-hour outage, traffic began recovering at 0450 UTC.
The impact of these disruptions is also visible when looking at traffic at both a regional and national level. In Tehran Province, HTTP request volume declined by approximately 70% around 1600 UTC, and continued to drop for the next several hours before seeing a slight recovery at 2200 UTC, likely related to the recovery also seen at that time on AS197207.
Similarly, Internet traffic volumes across the whole country began to decline just after 1600 UTC, falling approximately 40%. Nominal recovery at 2200 UTC is visible in this view as well, again likely from the increase in traffic from AS197207. More aggressive traffic growth is visible starting around 0500 UTC, after the remaining two mobile network providers came back online.
In analyzing DNS requests to Cloudflare’s resolver for domains associated with leading social media platforms, we observe that requests for instagram.com hostnames drop sharply at 1310 UTC, remaining lower for the rest of the day, except for a significant unexplained spike in requests between 1540 and 1610 UTC. Request volumes for hostnames associated with other leading social media platforms did not appear to be similarly affected.
In addition, it was reported that access to WhatsApp had also been blocked in Iran. This can be seen in resolution requests to Cloudflare’s resolver for whatsapp.com hostnames. The graph below shows a sharp decline in query traffic at 1910 UTC, dropping to near zero.
The Open Observatory for Network Interference (OONI), an organization that measures Internet censorship, reported in a Tweet that the cloudflare-dns.com domain name, used for DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) connections to Cloudflare’s DNS resolver, was blocked in Iran on September 20. This is clearly evident in the graph below, with resolution volume over DoH and DoT dropping to zero at 1940 UTC. The OONI tweet also noted that the 1.1.1.1 IP address “remains blocked on most networks.” The trend line for resolution over TCP or UDP (on port 53) in the graph below suggests that the IP address is not universally blocked, as there are still resolution requests reaching Cloudflare.
Interested parties can use Cloudflare Radar to monitor the impact of such government-directed Internet disruptions, and can follow @CloudflareRadar on Twitter for updates on Internet disruptions as they occur.
The James Webb Telescope reveals emerging stellar nurseries and individual stars in the Carina Nebula that were previously obscured. Credits: NASA, ESA, CSA, and STScI. Full image here.
“Somewhere, something incredible is waiting to be known.” — Carl Sagan
In the past few years, space technology and travel have been trending with increased attention and endeavors (including private ones). In our 2021 Year in Review we showed how NASA and SpaceX flew higher, at least in terms of interest on the Internet.
This week, NASA in collaboration with the European Space Agency (ESA) and the Canadian Space Agency (CSA), released the first images from the James Webb Telescope (JWST) which conducts infrared astronomy to “reveal the unseen universe”.
Webb’s First Deep Field is the first operational image taken by the James Webb Space Telescope, depicting a galaxy cluster with a distance of 5.12 billion light-years from Earth. Revealed to the public on 11 July 2022. Credits: NASA, ESA, CSA, and STScI. Full image here.
So, let’s dig into something we really like here at Cloudflare, checking how real life and human interest has an impact on the Internet. In terms of general Internet traffic in the US, Radar shows us that there was an increase both on July 11 and July 12, compared to the previous week (bear in mind that July 4, the previous Monday, was the Independence Day holiday in the US).
Next, we look at DNS request trends to get a sense of traffic to Internet properties (and using from this point on EST time in all the charts). Let’s start with the cornucopia of NASA, ESA and other websites (there are many, some dedicated just to the James Webb Telescope findings).
There are two clear spikes in the next chart. The first was around the time the first galaxy cluster infrared image was announced by Joe Biden, on Monday, July 11, 2022 (at 17:00), with traffic rising 13x higher than in the previous week. There was also a 5x spike at 01:00 EST that evening. The second spike was higher and longer and happened during Tuesday, July 12, 2022, when more images were revealed. Tuesday’s peak was at 10:00, with traffic being 19x higher than in the previous week — traffic was higher than 10x between 09:00 and 13:00.
The first image was presented by US president at around 17:00 on July 11. DNS traffic was 1.5x higher to White House-related websites than any time in the preceding month.
Conclusion: space, the final frontier
As we saw in 2021, space projects and announcements continue to have a clear impact on the Internet, in this case in our DNS request view of Internet traffic. So far, what the James Webb Telescope images are showing us is a glimpse of a never-before-seen picture of parts of the universe (there’s no lack of excitement in Cloudflare’s internal chat groups).
You can keep an eye on these and other trends using Cloudflare Radar and follow @CloudflareRadar on Twitter — recently we covered extensively Canada’s Internet outage.
An outage at one of the largest ISPs in Canada, Rogers Communications, started earlier today, July 8, 2022, and is ongoing (eight hours and counting), and is impacting businesses and consumers. At the time of writing, we are seeing a very small amount of traffic from Rogers, but we are only seeing residual traffic, and nothing close to a full recovery to normal traffic levels.
Based on what we’re seeing and similar incidents in the past, we believe this is likely to be an internal error, not a cyber attack.
Cloudflare Radar shows a near complete loss of traffic from Roger’s ASN, AS812, that started around 08:45 UTC (all times in this blog are UTC).
What happened?
Cloudflare data shows that there was a clear spike in BGP (Border Gateway Protocol) updates after 08:15, reaching its peak at 08:45.
BGP is a mechanism to exchange routing information between networks on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver each network packet to its final destination. Without BGP, the Internet routers wouldn’t know what to do, and the Internet wouldn’t exist.
The Internet is literally a network of networks, or for the maths fans, a graph, with each individual network a node in it, and the edges representing the interconnections. All of this is bound together by BGP. BGP allows one network (say Rogers) to advertise its presence to other networks that form the Internet. Rogers is not advertising its presence, so other networks can’t find Roger’s network and so it is unavailable.
A BGP update message informs a router of changes made to a prefix (a group of IP addresses) advertisement or entirely withdraws the prefix. In this next chart, we can see that at 08:45 there was a withdrawal of prefixes from Roger’s ASN.
Since then, at 14:30, attempts seem to be made to advertise their prefixes again. This maps to us seeing a slow increase in traffic again from Rogers’ end users.
The graph below, which shows the prefixes we were receiving from Rogers in Toronto, clearly shows the withdrawal of prefixes around 08:45, and the slow start in recovery at 14:30, with another round of withdraws at around 15:45.
Outages happen more regularly than people think. This week we did an Internet disruptions overview for Q2 2022 where you can get a better sense of that, and on how collaborative and interconnected the Internet (the network of networks) is. And not so long ago Facebook had an hours long outage where BGP updates showed Facebook itself disappearing from the Internet.
Follow @CloudflareRadar on Twitter for updates on Internet disruptions as they occur, and find up-to-date information on Internet trends using Cloudflare Radar.
Welcome to our 2022 Q2 DDoS report. This report includes insights and trends about the DDoS threat landscape — as observed across the global Cloudflare network. An interactive version of this report is also available on Radar.
In Q2, we’ve seen some of the largest attacks the world has ever seen including a 26 million request per second HTTPS DDoS attacks that Cloudflare automatically detected and mitigated. Furthermore, attacks against Ukraine and Russia continue, whilst a new Ransom DDoS attack campaign emerged.
The Highlights
Ukrainian and Russian Internet
The war on the ground is accompanied by attacks targeting the spread of information.
Broadcast Media companies in the Ukraine were the most targeted in Q2 by DDoS attacks. In fact, all the top five most attacked industries are all in online/Internet media, publishing, and broadcasting.
In Russia on the other hand, Online Media drops as the most attacked industry to the third place. Making their way to the top, Banking, Financial Services and Insurance (BFSI) companies in Russia were the most targeted in Q2; almost 45% of all application-layer DDoS attacks targeted the BFSI sector. Cryptocurrency companies in Russia were the second most attacked.
We’ve seen a new wave of Ransom DDoS attacks by entities claiming to be the Fancy Lazarus.
In June 2022, ransom attacks peaked to the highest of the year so far: one out of every five survey respondents who experienced a DDoS attack reported being subject to a Ransom DDoS attack or other threats.
Overall in Q2, the percent of Ransom DDoS attacks increased by 11% QoQ.
Application-layer DDoS attacks
In 2022 Q2, application-layer DDoS attacks increased by 72% YoY.
Organizations in the US were the most targeted, followed by Cyprus, Hong Kong, and China. Attacks on organizations in Cyprus increased by 166% QoQ.
The Aviation & Aerospace industry was the most targeted in Q2, followed by the Internet industry, Banking, Financial Services and Insurance, and Gaming / Gambling in fourth place.
Network-layer DDoS attacks
In 2022 Q2, network-layer DDoS attacks increased by 109% YoY. Attacks of 100 Gbps and larger increased by 8% QoQ, and attacks lasting more than 3 hours increased by 12% QoQ.
The top attacked industries were Telecommunications, Gaming / Gambling and the Information Technology and Services industry.
Organizations in the US were the most targeted, followed by China, Singapore, and Germany.
This report is based on DDoS attacks that were automatically detected and mitigated by Cloudflare’s DDoS Protection systems. To learn more about how it works, check out this deep-dive blog post.
A note on how we measure DDoS attacks observed over our network
To analyze attack trends, we calculate the “DDoS activity” rate, which is either the percentage of attack traffic out of the total traffic (attack + clean) observed over our global network, or in a specific location, or in a specific category (e.g., industry or billing country). Measuring the percentages allows us to normalize data points and avoid biases reflected in absolute numbers towards, for example, a Cloudflare data center that receives more total traffic and likely, also more attacks.
Ransom Attacks
Our systems constantly analyze traffic and automatically apply mitigation when DDoS attacks are detected. Each DDoS’d customer is prompted with an automated survey to help us better understand the nature of the attack and the success of the mitigation.
For over two years now, Cloudflare has been surveying attacked customers — one question on the survey being if they received a threat or a ransom note demanding payment in exchange to stop the DDoS attack.
The number of respondents reporting threats or ransom notes in Q2 increased by 11% QoQ and YoY. During this quarter, we’ve been mitigating Ransom DDoS attacks that have been launched by entities claiming to be the Advanced Persistent Threat (APT) group “Fancy Lazarus”. The campaign has been focusing on financial institutions and cryptocurrency companies.
The percentage of respondents reported being targeted by a ransom DDoS attack or that have received threats in advance of the attack.
Drilling down into Q2, we can see that in June one out of every five respondents reported receiving a ransom DDoS attack or threat — the highest month in 2022, and the highest since December 2021.
Application-layer DDoS attacks
Application-layer DDoS attacks, specifically HTTP DDoS attacks, are attacks that usually aim to disrupt a web server by making it unable to process legitimate user requests. If a server is bombarded with more requests than it can process, the server will drop legitimate requests and — in some cases — crash, resulting in degraded performance or an outage for legitimate users.
Application-layer DDoS attacks by month
In Q2, application-layer DDoS attacks increased by 72% YoY.
Overall, in Q2, the volume of application-layer DDoS attacks increased by 72% YoY, but decreased 5% QoQ. May was the busiest month in the quarter. Almost 41% of all application-layer DDoS attacks took place in May, whereas the least number of attacks took place in June (28%).
Application-layer DDoS attacks by industry
Attacks on the Aviation and Aerospace industry increased by 493% QoQ.
In Q2, Aviation and Aerospace was the most targeted industry by application-layer DDoS attacks. After it, was the Internet industry, Banking, Financial Institutions and Insurance (BFSI) industry, and in fourth place the Gaming / Gambling industry.
Ukraine and Russia cyberspace
Media and publishing companies are the most targeted in Ukraine.
As the war in Ukraine continues on the ground, in the air and on the water, so does it continue in cyberspace. Entities targeting Ukrainian companies appear to be trying to silence information. The top five most attacked industries in the Ukraine are all in broadcasting, Internet, online media, and publishing — that’s almost 80% of all DDoS Attacks targeting Ukraine.
On the other side of the war, the Russian Banks, Financial Institutions and Insurance (BFSI) companies came under the most attacks. Almost 45% of all DDoS attacks targeted the BFSI sector. The second most targeted was the Cryptocurrency industry, followed by Online media.
In both sides of the war, we can see that the attacks are highly distributed, indicating the use of globally distributed botnets.
Application-layer DDoS attacks by source country
In Q2, attacks from China shrank by 78%, and attacks from the US shrank by 43%.
To understand the origin of the HTTP attacks, we look at the geolocation of the source IP address belonging to the client that generated the attack HTTP requests. Unlike network-layer attacks, source IP addresses cannot be spoofed in HTTP attacks. A high percentage of DDoS activity in a given country doesn’t mean that that specific country is launching the attacks but rather indicates the presence of botnets operating from within the country’s borders.
For the second quarter in a row, the United States tops the charts as the main source of HTTP DDoS attacks. Following the US is China in second place, and India and Germany in the third and fourth. Even though the US remained in the first place, attacks originating from the US shrank by 48% QoQ while attacks from other regions grew; attacks from India grew by 87%, from Germany by 33%, and attacks from Brazil grew by 67%.
Application-layer DDoS attacks by target country
In order to identify which countries are targeted by the most HTTP DDoS attacks, we bucket the DDoS attacks by our customers’ billing countries and represent it as a percentage out of all DDoS attacks.
HTTP DDoS attacks on US-based countries increased by 67% QoQ pushing the US back to the first place as the main target of application-layer DDoS attacks. Attacks on Chinese companies plunged by 80% QoQ dropping it from the first place to the fourth. Attacks on Cyprus increase by 167% making it the second most attacked country in Q2. Following Cyprus is Hong Kong, China, and the Netherlands.
Network-layer DDoS attacks
While application-layer attacks target the application (Layer 7 of the OSI model) running the service that end users are trying to access (HTTP/S in our case), network-layer attacks aim to overwhelm network infrastructure (such as in-line routers and servers) and the Internet link itself.
Network-layer DDoS attacks by month
In Q2, network-layer DDoS attacks increased by 109% YoY, and volumetric attacks of 100 Gbps and larger increased by 8% QoQ.
In Q2, the total amount of network-layer DDoS attacks increased by 109% YoY and 15% QoQ. June was the busiest month of the quarter with almost 36% of the attacks occurring in June.
Network-layer DDoS attacks by industry
In Q2, attacks on Telecommunication companies grew by 66% QoQ.
For the second consecutive quarter, the Telecommunications industry was the most targeted by network-layer DDoS attacks. Even more so, attacks on Telecommunication companies grew by 66% QoQ. The Gaming industry came in second place, followed by Information Technology and Services companies.
Network-layer DDoS attacks by target country
Attacks on US networks grew by 95% QoQ.
In Q2, the US remains the most attacked country. After the US came China, Singapore and Germany.
Network-layer DDoS attacks by ingress country
In Q2, almost a third of the traffic Cloudflare observed in Palestine and a fourth in Azerbaijan was part of a network-layer DDoS attack.
When trying to understand where network-layer DDoS attacks originate, we cannot use the same method as we use for the application-layer attack analysis. To launch an application-layer DDoS attack, successful handshakes must occur between the client and the server in order to establish an HTTP/S connection. For a successful handshake to occur, the attacks cannot spoof their source IP address. While the attacker may use botnets, proxies, and other methods to obfuscate their identity, the attacking client’s source IP location does sufficiently represent the attack source of application-layer DDoS attacks.
On the other hand, to launch network-layer DDoS attacks, in most cases, no handshake is needed. Attackers can spoof the source IP address in order to obfuscate the attack source and introduce randomness into the attack properties, which can make it harder for simple DDoS protection systems to block the attack. So if we were to derive the source country based on a spoofed source IP, we would get a ‘spoofed country’.
For this reason, when analyzing network-layer DDoS attack sources, we bucket the traffic by the Cloudflare data center locations where the traffic was ingested, and not by the (potentially) spoofed source IP to get an understanding of where the attacks originate from. We are able to achieve geographical accuracy in our report because we have data centers in over 270 cities around the world. However, even this method is not 100% accurate, as traffic may be back hauled and routed via various Internet Service Providers and countries for reasons that vary from cost reduction to congestion and failure management.
Palestine jumps from the second to the first place as the Cloudflare location with the highest percentage of network-layer DDoS attacks. Following Palestine is Azerbaijan, South Korea, and Angola.
To view all regions and countries, check out the interactive map.
Attack vectors
In Q2, DNS attacks increased making it the second most frequent attack vector.
An attack vector is a term used to describe the method that the attacker uses to launch their DDoS attack, i.e., the IP protocol, packet attributes such as TCP flags, flooding method, and other criteria.
In Q2, 53% of all network-layer attacks were SYN floods. SYN floods remain the most popular attack vector. They abuse the initial connection request of the stateful TCP handshake. During this initial connection request, servers don’t have any context about the TCP connection as it is new and without the proper protection may find it hard to mitigate a flood of initial connection requests. This makes it easier for the attacker to consume an unprotected server’s resources.
After the SYN floods are attacks targeting DNS infrastructure, RST floods again abusing TCP connection flow, and generic attacks over UDP.
Emerging threats
In Q2, the top emerging threats included attacks over CHARGEN, Ubiquiti and Memcached.
Identifying the top attack vectors helps organizations understand the threat landscape. In turn, this may help them improve their security posture to protect against those threats. Similarly, learning about new emerging threats that may not yet account for a significant portion of attacks, can help mitigate them before they become a significant force.
In Q2, the top emerging threats were amplification attacks abusing the Character Generator Protocol (CHARGEN), amplification attacks reflecting traffic off of exposed Ubiquiti devices, and the notorious Memcached attack.
Abusing the CHARGEN protocol to launch amplification attacks
In Q2, attacks abusing the CHARGEN protocol increased by 378% QoQ.
Initially defined in RFC 864 (1983), the Character Generator (CHARGEN) protocol is a service of the Internet Protocol Suite that does exactly what it says it does – it generates characters arbitrarily, and it doesn’t stop sending them to the client until the client closes the connection. Its original intent was for testing and debugging. However, it’s rarely used because it can so easily be abused to generate amplification/reflection attacks.
An attacker can spoof the source IP of their victim and fool supporting servers around the world to direct a stream of arbitrary characters “back” to the victim’s servers. This type of attack is amplification/reflection. Given enough simultaneous CHARGEN streams, the victim’s servers, if unprotected, would be flooded and unable to cope with legitimate traffic — resulting in a denial of service event.
Amplification attacks exploiting the Ubiquiti Discovery Protocol
In Q2, attacks over Ubiquity increased by 327% QoQ.
Ubiquiti is a US-based company that provides networking and Internet of Things (IoT) devices for consumers and businesses. Ubiquiti devices can be discovered on a network using the Ubiquiti Discovery protocol over UDP/TCP port 10001.
Similarly to the CHARGEN attack vector, here too, attackers can spoof the source IP to be the victim’s IP address and spray IP addresses that have port 10001 open. Those would then respond to the victim and essentially flood it if the volume is sufficient.
Memcached DDoS attacks
In Q2, Memcached DDoS attacks increased by 287% QoQ.
Memcached is a database caching system for speeding up websites and networks. Similarly to CHARGEN and Ubiquiti, Memcached servers that support UDP can be abused to launch amplification/reflection DDoS attacks. In this case, the attacker would request content from the caching system and spoof the victim’s IP address as the source IP in the UDP packets. The victim will be flooded with the Memcache responses which can be amplified by a factor of up to 51,200x.
Network-layer DDoS attacks by attack rate
Volumetric attacks of over 100 Gbps increase by 8% QoQ.
There are different ways of measuring the size of an L3/4 DDoS attack. One is the volume of traffic it delivers, measured as the bit rate (specifically, terabits per second or gigabits per second). Another is the number of packets it delivers, measured as the packet rate (specifically, millions of packets per second).
Attacks with high bit rates attempt to cause a denial-of-service event by clogging the Internet link, while attacks with high packet rates attempt to overwhelm the servers, routers, or other in-line hardware appliances. These devices dedicate a certain amount of memory and computation power to process each packet. Therefore, by bombarding it with many packets, the appliance can be left with no further processing resources. In such a case, packets are “dropped,” i.e., the appliance is unable to process them. For users, this results in service disruptions and denial of service.
Distribution by packet rate
The majority of network-layer DDoS attacks remain below 50,000 packets per second. While 50 kpps is on the lower side of the spectrum at Cloudflare scale, it can still easily take down unprotected Internet properties and congest even a standard Gigabit Ethernet connection.
When we look at the changes in the attack sizes, we can see that packet-intensive attacks above 50 kpps decreased in Q2, resulting in an increase of 4% in small attacks.
Distribution by bitrate
In Q2, most of the network-layer DDoS attacks remain below 500 Mbps. This too is a tiny drop in the water at Cloudflare scale, but can very quickly shut down unprotected Internet properties with less capacity or at the very least cause congestion for even a standard Gigabit Ethernet connection.
Interestingly enough, large attacks between 500 Mbps and 100 Gbps decreased by 20-40% QoQ, but volumetric attacks above 100 Gbps increased by 8%.
Network-layer DDoS attacks by duration
In Q2, attacks lasting over three hours increased by 9%.
We measure the duration of an attack by recording the difference between when it is first detected by our systems as an attack and the last packet we see with that attack signature towards that specific target.
In Q2, 52% of network-layer DDoS attacks lasted less than 10 minutes. Another 40% lasted 10-20 minutes. The remaining 8% include attacks ranging from 20 minutes to over three hours.
One important thing to keep in mind is that even if an attack lasts only a few minutes, if it is successful, the repercussions could last well beyond the initial attack duration. IT personnel responding to a successful attack may spend hours and even days restoring their services.
While most of the attacks are indeed short, we can see an increase of over 15% in attacks ranging between 20-60 minutes, and a 12% increase of attacks lasting more than three hours.
Short attacks can easily go undetected, especially burst attacks that, within seconds, bombard a target with a significant number of packets, bytes, or requests. In this case, DDoS protection services that rely on manual mitigation by security analysis have no chance in mitigating the attack in time. They can only learn from it in their post-attack analysis, then deploy a new rule that filters the attack fingerprint and hope to catch it next time. Similarly, using an “on-demand” service, where the security team will redirect traffic to a DDoS provider during the attack, is also inefficient because the attack will already be over before the traffic routes to the on-demand DDoS provider.
It’s recommended that companies use automated, always-on DDoS protection services that analyze traffic and apply real-time fingerprinting fast enough to block short-lived attacks.
Summary
Cloudflare’s mission is to help build a better Internet. A better Internet is one that is more secure, faster, and reliable for everyone — even in the face of DDoS attacks. As part of our mission, since 2017, we’ve been providing unmetered and unlimited DDoS protection for free to all of our customers. Over the years, it has become increasingly easier for attackers to launch DDoS attacks. But as easy as it has become, we want to make sure that it is even easier — and free — for organizations of all sizes to protect themselves against DDoS attacks of all types.
Not using Cloudflare yet? Start now with our Free and Pro plans to protect your websites, or contact us for comprehensive DDoS protection for your entire network using Magic Transit.
Cloudflare operates in more than 270 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. In many cases, these disruptions can be attributed to a physical event, while in other cases, they are due to an intentional government-directed shutdown. In this post, we review selected Internet disruptions observed by Cloudflare during the second quarter of 2022, supported by traffic graphs from Cloudflare Radar and other internal Cloudflare tools, and grouped by associated cause or common geography.
Optic outages
This quarter, we saw the usual complement of damage to both terrestrial and submarine fiber-optic cables, including one that impacted multiple countries across thousands of miles, and another more localized outage that was due to an errant rodent.
Comcast
On April 25, Comcast subscribers in nearly 20 southwestern Florida cities experienced an outage, reportedly due to a fiber cut. The traffic impact of this cut is clearly visible in the graph below, with Cloudflare traffic for these cities dropping to zero between 1915–2050 UTC (1515–1850 local time).
Not only did the fiber cut force a significant number of Comcast subscribers offline, but it also impacted the types of traffic observed across Comcast as a whole. The graphs below illustrate the mix of mobile vs desktop clients, as well as IPv4 vs. IPv6 request volume across AS7922, Comcast’s primary autonomous system. During the brief disruption period, the percentage of Comcast traffic from mobile devices increased, while desktop devices dropped, and the percentage of IPv4 traffic dropped, with a corresponding increase in IPv6 traffic share.
South Africa
On the morning of May 17, Telkom SA, a South African telecommunications provider, tweeted an “important notice” to customers, noting that “Damage to a Fibre cable was detected on the Telkom network around 8:00am on Tuesday, 17 May 2022.” and outlining the impacted services and geographies. The graphs below show the impact to Cloudflare traffic from the Telkom autonomous system in three South African provinces. The top graph shows the impact to traffic in Gauteng, while the lower graph shows the impact in Limpopo and North West. Across all three, traffic falls at 0600 UTC (0800 local time), recovering around 1300 UTC (1500 local time). Telkom SA did not provide any additional information on where the fiber cut occurred or what caused it.
Venezuela
Although unconfirmed, a fiber cut was suspected to be the cause of an Internet disruption experienced by CANTV subscribers in Venezuela on May 19, the latest of several such incidents affecting that provider. Although the fiber cut reportedly impacted subscribers in multiple states, the most significant impact was measured in Falcón, as shown in the graph below. In this state, traffic dropped precipitously at 1800 UTC (1400 local time), finally recovering approximately 24 hours later.
AAE-1 & SMW-5
Just after 1200 UTC on Tuesday, June 7, the Africa-Asia-Europe-1 (AAE-1) and SEA-ME-WE-5 (SMW-5) submarine cables suffered cable cuts, impacting Internet connectivity for millions of Internet users across multiple countries in the Middle East and Africa, as well as thousands of miles away in Asia. Although specific details are sparse, the cable damage reportedly occurred in Egypt – both of the impacted cables land in Abu Talat and Zafarana, which also serve as landing points for a number of other submarine cables.
Finally, on June 13, a beaver was responsible for an outage that impacted Internet users in British Columbia, Canada. According to a published report, a beaver gnawed its way through a tree, causing it to fall on both power lines and a Telus fiber optic cable. The damage to the fiber optic cable affected connectivity customers in over a dozen communities across British Columbia, including those using CityWest (AS18988), a utility company that uses the Telus cable. In the graph below, the impact of the damage to the fiber optic cable is clearly visible, with no traffic to Cloudflare from CityWest subscribers in British Columbia between 1800 UTC on June 7 until 0310 UTC on June 8 (1100–2010 local time).
School’s in, Internet’s out
Nationwide Internet shutdowns have, unfortunately, become a popular approach taken by authoritarian regimes over the past half dozen years to prevent cheating on secondary school exams. It is not clear that this heavy-handed tactic is actually effective in preventing cheating, but the associated damage to the national economies has been estimated to be in the tens to hundreds of millions of US dollars, depending on the duration and frequency of the shutdowns.
This year, governments in Sudan and Syria implemented a number of multi-hour shutdowns in late May into June, while Algeria’s government appears to have resorted to more targeted content blocking. Additional details on these Internet disruptions can be found in the recent “Exam time means Internet disruptions in Syria, Sudan and Algeria” blog post.
Starting on May 30, Syria implemented the first of four nationwide Internet shutdowns, the last of which occurred on June 12, as seen in the graph below. Interestingly, we have observed that these shutdowns tend to be “asymmetric” in nature — that is, inbound traffic (into the country) is disabled, but egress traffic (from the country) remains. One effect of this is visible as spikes in the DNS graph below. During three of the four shutdowns, requests to Cloudflare’s 1.1.1.1 resolver from clients in Syria increased because DNS queries were able to exit the country, but responses couldn’t return, leading to retry floods.
In Sudan, daily shutdowns were implemented 0530-0830 UTC (0730–1030 local time) between June 11 and June 22, except for June 17. (It isn’t clear why that date was skipped.) The graph below shows that these shutdowns were nationwide, but not complete, as traffic from the country did not drop to zero.
In Algeria, exams took place June 12 through June 16. In the past, the country has implemented nationwide shutdowns, but after recognizing the enormous cost to the economy, the government has apparently chosen an alternate tactic this year. The graph below shows nominal drops in country-level traffic during the two times each day that the exams took place—0730–1000 UTC (0830–1100 am local time) and 1330–1600 UTC (1430–1700 local time). These drops in traffic are likely more indicative of a content-blocking approach, instead of a broad Internet shutdown.
On June 27, the Kurdistan Regional Government in Iraq began to implement twice-weekly (Mondays and Thursday) multi-hour regional Internet shutdowns, expected to last for a four-week period. The shutdowns are intended to prevent cheating on high school final exams, according to a published report, and are scheduled for 0630–1030 am local time (0330–0730 UTC). The graph below shows the impact to traffic from three governorates in Kurdistan, with traffic dropping to near zero in all three areas during the duration of the shutdowns.
Government-guided
In addition to shutting down the Internet to prevent cheating on exams, governments have also been known to use shutdowns as a tool to limit or control communication around elections, rallies, protests, etc. During the second quarter, we observed several such shutdowns of note.
A month and a half later, on May 25, an Internet disruption was observed in Pakistan amid protests led by the country’s former Prime Minister. The disruption lasted only two hours, and was limited in scope — it was not a nationwide shutdown. (Telecom providers claimed that it was due to problems with a web filtering system.) At a national level, the impact of the disruption is visible as a slight drop in traffic.
In the cities of Lahore and Karachi, the disruption is visible a little more clearly, as is the rapid recovery in traffic.
The impact of the disruption is most evident at a network level, as seen in the graphs below. Cyber Internet Services (AS9541) saw a modest drop in traffic, while Mobilink (AS45669) experienced a near complete outage.
Closing out the quarter, a communications blackout, including an Internet shutdown, was imposed in Sudan on June 30 as protestors staged rallies against the country’s military leadership. This shutdown follows similar disruptions seen in October 2021 after the military toppled the transitional government and attempted to limit protests, as well the shutdowns seen earlier in June as the government attempted to prevent cheating on exams. The graphs below show that the shutdown started at 0600 UTC (0800 local time) and initially ended almost 12 hours later at 1740 UTC (1940 local time). Connectivity returned for approximately three hours, with traffic again dropping to near-zero levels again around 2040 UTC (2240 local time). This second outage remained active at the end of the day.
In addition to fiber/cable cuts, as discussed above, problems with other infrastructure, whether due to fires, electrical issues, or maintenance, can also disrupt Internet services.
Around 2030 local time on April 6 (0030 UTC on April 7), a fire erupted at the Costa Sur generation plant, one of the largest power plants in Puerto Rico, resulting in a widespread power outage across the island territory. This island-wide outage caused a significant interruption to Internet services, clearly visible in Cloudflare traffic data. The graph below shows that as the power failed, traffic from Puerto Rico immediately fell by more than half. The regular diurnal pattern remained in place, albeit at lower levels, over the next three days, with traffic returning to “normal levels” three days later. By April 10, Luma Energy reported that it had restored electrical power to 99.7% of its 1.5M customers.
The impact of the Internet service disruption is also fairly significant when viewed at a network level. The graphs below show traffic for Datacom Caribe/Claro (AS10396) and Liberty Cablevision of Puerto Rico (AS14638). At Datacom Caribe/Claro, traffic immediately fell by more than half, while Liberty Cablevision traffic declined approximately 85%.
On the evening of May 3, Swiss telecom provider Swisscom tweeted that there had been an interruption to Internet service following maintenance work. A published report noted that the interruption occurred between 2223–2253 local time (2023–2053 UTC), and the graph below shows a complete loss of traffic, but quick recovery, during that 30-minute window. Beyond citing maintenance work, Swisscom did not provide any additional details about the Internet disruption.
On May 6, the government disrupted Internet connectivity in Khuzestan province, reportedly in response to mass protests around shortages of bread and water. It was reported that mobile data had been cut off locally, and that fixed connectivity speeds were significantly reduced. To this end, we observed a drop in traffic for Irancell (AS44244) (a mobile network provider) in Khuzestan starting around 1000 UTC as seen in the graph below.
A similar disruption affecting Irancell, occurring amid reports of ongoing protests in the country, was observed on May 12, with lower peak traffic during the day, and a further drop around 1800 UTC.
Mobile provider Mobinnet (AS50810) experienced a multi-hour Internet disruption on May 14, lasting from 1230–1530 UTC (1700–2000 local time). According to a tweet from Mobinnet, the disruption was due to a “widespread cyber attack of foreign origin”.
Ukraine
Now more than four months into the war in Ukraine, the Internet continues to be an active battlefield, with ongoing Internet outages in multiple cities and across multiple networks. However, we want to highlight here two similar events observed during the second quarter.
The Russian-occupied city of Kherson experienced a near-complete Internet outage between 1600 UTC (1900 local time) on April 30 and 0430 UTC (0730 local time) on May 4. According to social media posts from Ukraine’s vice Prime-Minister Mykhailo Fedorov and the State Service of Special Communications and Information Protection, the outage was caused by “interruptions of fiber-optic trunk lines and disconnection from the power supply of equipment of operators in the region”. The graph below shows effectively no traffic for Kherson for approximately 24 hours after the disruption began, followed by a nominal amount of traffic for the next several days.
Around the time that the nominal amount of traffic returned, we also observed a shift in routing for an IPv4 prefix announced by AS47598 (Khersontelecom). As shown in the table below, prior to the outage, it reached the Internet through several other Ukrainian network providers, including AS12883, AS3326, and AS35213. However, as traffic returned, its routing path now showed a Russian network, AS201776 (Miranda) as the upstream provider. The path through Miranda also includes AS12389 (Rostelecom), which bills itself as “the largest digital services provider in Russia”.
Peer AS
Last Update
AS Path
AS1299 (TWELVE99 Arelion, fka Telia Carrier)
5/1/2022 16:02:26
1299 12389 201776 47598
AS6777 (AMS-IX-RS)
4/28/2022 11:23:33
12883 47598
As the disruption ended on May 4, we observed updates to Khersontelecom’s routing path that enabled it to return to reaching the global Internet through non-Russian upstream providers.
A month later, on May 30, we again observed a significant Internet disruption in Kherson starting at 1435 UTC (1735 local time). And once again, we observed updated routing for Khersontelecom, as it shifted from Ukrainian upstream providers to Russian ones. As of the end of June, the Internet disruption in Kherson and the routing through Russian upstream providers both remain firmly in place, although the loss of traffic has not been nearly as significant as the April/May disruption.
Peer AS
Last Update
AS Path
AS4775 (Globe Telecoms)
5/30/2022 13:56:22
4775 1273 12389 201776 47598
AS9002 (RETN-AS)
5/30/2022 09:58:16
9002 3326 47598
Conclusion
This post is by no means an exhaustive review of the Internet outages, shutdowns, and disruptions that have occurred throughout the second quarter. Some were extremely brief or limited in scope, while others were observed but had no known or publicly conjectured underlying cause. Having said that, it is important to bring increased visibility to these events so that the community can share information on what is happening, why it happened, and what the impact was — human, financial, or otherwise.
Follow @CloudflareRadar on Twitter for updates on Internet disruptions as they occur, and find up-to-date information on Internet trends using Cloudflare Radar.
Here’s a short list of recent technical blog posts to give you something to read today.
Internet Explorer, we hardly knew ye
Microsoft has announced the end-of-life for the venerable Internet Explorer browser. Here we take a look at the demise of IE and the rise of the Edge browser. And we investigate how many bots on the Internet continue to impersonate Internet Explorer versions that have long since been replaced.
Live-patching security vulnerabilities inside the Linux kernel with eBPF Linux Security Module
Looking for something with a lot of technical detail? Look no further than this blog about live-patching the Linux kernel using eBPF. Code, Makefiles and more within!
Hertzbleed explained
Feeling mathematical? Or just need a dose of CPU-level antics? Look no further than this deep explainer about how CPU frequency scaling leads to a nasty side channel affecting cryptographic algorithms.
Early Hints update: How Cloudflare, Google, and Shopify are working together to build a faster Internet for everyone
The HTTP standard for Early Hints shows a lot of promise. How much? In this blog post, we dig into data about Early Hints in the real world and show how much faster the web is with it.
Private Access Tokens: eliminating CAPTCHAs on iPhones and Macs with open standards
Dislike CAPTCHAs? Yes, us too. As part of our program to eliminate captures there’s a new standard: Private Access Tokens. This blog shows how they work and how they can be used to prove you’re human without saying who you are.
Optimizing TCP for high WAN throughput while preserving low latency
Network nerd? Yeah, me too. Here’s a very in depth look at how we tune TCP parameters for low latency and high throughput.
It is once again exam time in Syria, Sudan, and Algeria, and with it, we find these countries disrupting Internet connectivity in an effort to prevent cheating on these exams. As they have done over the past several years, Syria and Sudan are implementing multi-hour nationwide Internet shutdowns. Algeria has also taken a similar approach in the past, but this year appears to be implementing more targeted website/application blocking.
Syria
Syria has been implementing Internet shutdowns across the country since 2011, but exam-related shutdowns have only been in place since 2016. In 2021, exams took place between May 31 and June 22, with multi-hour shutdowns observed on each of the exam days.
This year, the first shutdown was observed on May 30, with subsequent shutdowns (to date) seen on June 2, 6, and 12. In the Cloudflare Radar graph below, traffic for Syria drops to zero while the shutdowns are active. According to Internet Society Pulse, several additional shutdowns are expected through June 21. Each takes place between 02000530 UTC (0500–0830 local time). According to a published report, the current exam cycle covers more than 500,000 students for basic and general secondary education certificates.
Consistent with shutdowns observed in prior years, Syria is once again implementing them in an asymmetric fashion – that is, inbound traffic is disabled, but egress traffic remains. This is clearly visible in request traffic from Syria to Cloudflare’s 1.1.1.1 DNS resolver. As the graph below shows, queries from clients in Syria are able to exit the country and reach Cloudflare, but responses can’t return, leading to retry floods, visible as spikes in the graph.
Last year, the Syrian Minister of Education noted that, for the first time, encryption and surveillance technologies would be used in an effort to curtail cheating, with an apparent promise to suspend Internet shutdowns in the future if these technologies proved successful.
Sudan
Sudan is also no stranger to nationwide Internet shutdowns, with some last lasting for multiple weeks. Over the last several years, Sudan has also implemented Internet shutdowns during secondary school exams in an effort to limit cheating or leaking of exam questions. (We covered the 2021 round of shutdowns in a blog post.)
According to a schedule published by digital rights organization AccessNow, this year’s Secondary Certificate Exams will be taking place in Sudan daily between June 11–22, except June 17. As of this writing, near-complete shutdowns have been observed on June 11, 12, and 13 between 0530-0830 UTC (0730-1030 local time), as seen in the graph below. The timing of these shutdowns aligns with a communication reportedly sent to subscribers of telecommunications services in the country, which stated “In implementation of the decision of the Attorney General, the Internet service will be suspended during the Sudanese certificate exam sessions from 8 in the morning until 11 in the morning.”
It is interesting to note that the shutdown, while nationwide, does not appear to be complete. The graph below shows that Cloudflare continues to see a small volume of HTTP requests from Sudatel during the shutdown periods. This is not completely unusual, as Sudatel may have public sector, financial services, or other types of customers that remain online.
Algeria
Since 2018, Algeria has been shutting down the Internet nationwide during baccalaureate exams, following widespread cheating in 2016 that saw questions leaked online both before and during tests. These shutdowns reportedly cost businesses across the country an estimated 500 million Algerian Dinars (approximately $3.4 million USD) for every hour the Internet was unavailable. In 2021, there were two Internet shutdowns each day that exams took place—the first between 0700–1100 UTC (0800–1200 local time), and the second between 1330–1600 UTC (1430–1700 local time).
Perhaps recognizing the economic damage caused by these Internet shutdowns, this year the Algerian Minister of National Education announced that there would be no Internet shutdowns on exam days.
Thus far, it appears that this has been the case. However, it appears that the Algerian government has shifted to a content blocking-based approach, instead of a wide-scale Internet shutdown. The Cloudflare Radar graph below shows two nominal drops in country-level traffic during the two times on June 13 that the exams took place—0730–1000 UTC (0830–1100 local time) and 1330–1600 UTC (1430–1700 local time), similar to last year’s timing.
The disruptions are also visible in traffic graphs for several major Algerian network providers, as shown below.
Analysis of additional Cloudflare data further supports the hypothesis that Algeria is blocking access to specific websites and applications, rather than shutting down the Internet completely.
As described in a previous blog post, Network Error Logging (NEL) is a browser-based reporting system that allows users’ browsers to report connection failures to an endpoint specified by the webpage that failed to load. Below, a graph of NEL reports from browsers in Algeria shows clear spikes during the times (thus far) that the exams have taken place, with report levels significantly lower and more consistent during other times of the day.
Conclusion
In addition to Syria, Sudan, and Algeria, countries including India, Jordan, Iraq, Uzbekistan, and Ethiopia have shut down or limited access to the Internet as exams took place. It is unclear whether these brute-force methods are truly effective at preventing cheating on these exams. However, it is clear that the impact of these shutdowns goes beyond students, as they impose a significant financial cost on businesses within the affected countries as they lose Internet access for multiple hours a day over the course of several weeks.
If you want to follow the remaining scheduled disruptions for these countries, you can see live data on the Cloudflare Radar pages for Syria, Sudan, and Algeria.
Just after 1200 UTC on Tuesday, June 7, the Africa-Asia-Europe-1 (AAE-1) and SEA-ME-WE-5 (SMW-5) submarine cables suffered cable cuts. The damage reportedly occurred in Egypt, and impacted Internet connectivity for millions of Internet users across multiple countries in the Middle East and Africa, as well as thousands of miles away in Asia. In addition, Google Cloud Platform and OVHcloud reported connectivity issues due to these cable cuts.
The impact
Data from Cloudflare Radar showed significant drops in traffic across the impacted countries as the cable damage occurred, recovering approximately four hours later as the cables were repaired.
It appears that Saudi Arabia may have also been affected by the cable cut(s), but the impact was much less significant, and traffic recovered almost immediately.
In the graphs above, we show that Ethiopia was one of the impacted countries. However, as it is landlocked, there are obviously no submarine cable landing points within the country. The Afterfibre map from the Network Startup Resource Center (NSRC) shows that that fiber in Ethiopia connects to fiber in Somalia, which experienced an impact. In addition, Ethio Telecom also routes traffic through network providers in Kenya and Djibouti. Djibouti Telecom, one of these providers, in turn peers with larger global providers like Telecom Italia (TI) Sparkle, which is one of the owners of SMW5.
In addition to impacting end-user connectivity in the impacted countries, the cable cuts also reportedly impacted cloud providers including Google Cloud Platform and OVHcloud. In their incident report, Google Cloud noted“Google Cloud Networking experienced increased packet loss for egress traffic from Google to the Middle East, and elevated latency between our Europe and Asia Regions as a result, for 3 hours and 12 minutes, affecting several related products including Cloud NAT, Hybrid Connectivity and Virtual Private Cloud (VPC). From preliminary analysis, the root cause of the issue was a capacity shortage following two simultaneous fiber-cuts.” OVHcloud noted that “Backbone links between Marseille and Singapore are currently down” and that “Upon further investigation, our Network OPERATION teams advised that the fault was related to our partner fiber cuts.”
When concurrent disruptions like those highlighted above are observed across multiple countries in one or more geographic areas, the culprit is often a submarine cable that connects the impacted countries to the global Internet. The impact of such cable cuts will vary across countries, largely due to the levels of redundancy that they may have in place. That is, are these countries solely dependent on an impacted cable for global Internet connectivity, or do they have redundant connectivity across other submarine or terrestrial cables? Additionally, the location of the country relative to the cable cut will also impact how connectivity in a given country may be affected. Due to these factors, we didn’t see a similar impact across all of the countries connected to the AAE-1 and SMW5 cables.
What happened?
Specific details are sparse, but as noted above, the cable damage reportedly occurred in Egypt – both of the impacted cables land in Abu Talat and Zafarana, which also serve as landing points for a number of other submarine cables. According to a 2021 article in Middle East Eye, “There are 10 cable landing stations on Egypt’s Mediterranean and Red Sea coastlines, and some 15 terrestrial crossing routes across the country.” Alan Mauldin, research director at telecommunications research firm TeleGeography, notes that routing cables between Europe and the Middle East to India is done via Egypt, because there is the least amount of land to cross. This places the country in a unique position as a choke point for international Internet connectivity, with damage to infrastructure locally impacting the ability of millions of people thousands of miles away to access websites and applications, as well as impacting connectivity for leading cloud platform providers.
As the graphs above show, traffic returned to normal levels within a matter of hours, with tweets from telecommunications authorities in Pakistan and Oman also noting that Internet services had returned to their countries. Such rapid repairs to submarine cable infrastructure are unusual, as repair timeframes are often measured in days or weeks, as we saw with the cables damaged by the volcanic eruption in Tonga earlier this year. This is due to the need to locate the fault, send repair ships to the appropriate location, and then retrieve the cable and repair it. Given this, the damage to these cables likely occurred on land, after they came ashore.
Keeping content available
By deploying in data centers close to end users, Cloudflare helps to keep traffic local, which can mitigate the impact of catastrophic events like cable cuts, while improving performance, availability, and security. Being able to deliver content from our network generally requires first retrieving it from an origin, and with end users around the world, Cloudflare needs to be able to reach origins from multiple points around the world at the same time. However, a customer origin may be reachable from some networks but not from others, due to a cable cut or some other network disruption.
In September 2021, Cloudflare announced Orpheus, which provides reachability benefits for customers by finding unreachable paths on the Internet in real time, and guiding traffic away from those paths, ensuring that Cloudflare will always be able to reach an origin no matter what is happening on the Internet.
Conclusion
Because the Internet is an interconnected network of networks, an event such as a cable cut can have a ripple effect across the whole Internet, impacting connectivity for users thousands of miles away from where the incident occurred. Users may be unable to access content or applications, or the content/applications may suffer from reduced performance. Additionally, the providers of those applications may experience problems within their own network infrastructure due to such an event.
For network providers, the impact of such events can be mitigated through the use of multiple upstream providers/peers, and diverse physical paths for critical infrastructure like submarine cables. Cloudflare’s globally deployed network can help content and application providers ensure that their content and applications remain available and performant in the face of network disruptions.
“I declare before you all that my whole life, whether it be long or short, shall be devoted to your service and the service of our great imperial family to which we all belong.” — Queen Elizabeth II birthday speech, April 21, 1947
When Elizabeth II’s reign started, on February 6, 1952 (the coronation was on June 2, 1953), the Turing machine had already been proposed (1936), and with that the basis for computer science. ARPANET, which became the technical foundation of the Internet, was still a dream that came to fruition in the late 60s — the World Wide Web is from 1989 and in 2014 we celebrated its Silver Jubilee. So, with that in mind, let’s answer the question: did the 2022 celebrations of the first British monarch with a 70th anniversary on the throne have an impact on the UK’s Internet traffic?
First, some details about the Platinum Jubilee. There was a four-day bank holiday (June 2-5) in the UK for the celebration that included parades and pageants, and several ceremonies. There was a Big Jubilee Lunch in many communities on Sunday, June 5, and more than 16,000 street parties (pubs and bars were also allowed to stay open for extra two hours). In events like these, not only there’s a lot to do outside, but also to see on the television and that impacts the Internet — we saw it during the Eurovision 2022 final.
Looking at Cloudflare Radar’s data from the UK, we can see that this past weekend clearly had less Internet traffic compared to recent weekends, so people were less online during the daytime, when the Jubilee was being celebrated. Here’s the chart with the previous four weekends of the UK’s Internet traffic:
That lower traffic trend is most clear on Saturday, June 4, at 20:00 local time, when traffic was 23% lower than on the previous Saturday, and on Sunday, June 5, at 15:00, when traffic was 25% lower than on the previous Sunday. The weather was actually sunnier on the previous weekend, May 28-29, but people did seem to have many reasons (related to the Jubilee) to go outside or at least be less online.
Looking at the full picture of when the four-day bank holiday started, Thursday, June 2, 2022, until Sunday, June 5, there’s a clear trend of less traffic through all of those days, which is not unusual, at least for Thursday and Friday, considering that holidays usually have traffic more similar to weekends than weekdays.
No surprise, when there’s a holiday, or it’s the weekend people tend to use their mobile devices more to access the Internet, and that was clearly what we saw in the UK since Thursday, June 2, mobile traffic (green line) was always prevalent compared to desktop traffic (blue line) since then.
On the weekdays before June 2, we can see that Internet traffic by mobile devices only stands out after 18:00 (before that, with people working, desktop took the lead).
From Canada to New Zealand
There are several other commonwealth countries that also had relevant events to celebrate since June 2 and through the past weekend for the Queen’s Platinum Jubilee. Canada is one with several activities throughout the country, including free admission to museums and historic sites, park parties and concerts.
Related to the Jubilee celebrations or not, Internet traffic in Canada was lower this past weekend than in the previous one. Saturday, June 4, at 22:00 in Toronto traffic was 13% lower than in the previous period, and throughout the day that was also the case. On Sunday, traffic was only lower during daytime, especially around 12:00 in Toronto, when it was 15% lower than in the previous Sunday. That was the time of the Jubilee Pageant, in central London (in the next charts, times are in UTC).
Something similar can be seen in terms of lower traffic this weekend in Australia:
And also New Zealand:
Royal family and news websites (Boris Johnson’s no confidence vote included)
Here we’re looking at DNS request trends to get a sense of traffic to Internet properties. First, we can see that websites concerning the UK Royal family and the Jubilee were clearly seeing more traffic after Wednesday, June 1 (the day before the four-day bank holiday). The three biggest spikes were: Wednesday evening, when traffic was 777% higher at 22:00 (compared to the previous week); the next morning (08:00), when it rose 1060%; and on Saturday evening (21:00) it got 1043% more traffic.
UK-based news websites (TV broadcasters and newspapers) also covered Queen Elizabeth II’s Platinum Jubilee extensively over the extended weekend. And there are three big highlights/spikes from the past few days regarding media outlets’ websites, but only two seem to be related to the Jubilee or the bank holidays.
We can see that the biggest spike in traffic (75% more than the previous period) was the night before the Jubilee four-day bank holiday started. Then, Sunday afternoon when the London Jubilee Pageant was ending, there was another spike (25% higher).
But the day with more sustained traffic from the last 14 days was actually Monday, June 6. That was the day that Boris Johnson, the British prime minister, won a no confidence vote in the UK’s Parliament. There was a clear first spike at around 08:00, when the news that a vote of no confidence would take place on that day broke, and a much bigger one at 21:00 (68% higher), when the final result of the vote was announced.
Social media and messaging trends
Social media trends show a similar pattern to Internet traffic in general, but it’s interesting to see that Thursday, June 2, the first day of the extended weekend, was the one of the full 14 days we’re looking at with less DNS traffic to those platforms.
Messaging, on the other hand, had consistently much lower traffic during the four-day bank holiday, even compared to the previous weekend. Saturday, June 4, was the day with less messaging DNS traffic, at least of the two weeks period we’re observing. At 11:00 Saturday, traffic was 18% lower than in the previous period, the same level of lower DNS requests at 15:00 and through most of the day.
Conclusion: celebrations and events ‘move’ the Internet
When there’s a big country-wide celebration going on, especially one that has a lot of outdoor events and activities, Internet patterns do change. That happens, in this case, for a monarch whose reign began in 1952, when there wasn’t any Internet (it took more than 40 years for the network of networks that can connect us all on Earth to reach its more popular global form).
You can keep an eye on these and other trends using Cloudflare Radar.
The collective thoughts of the interwebz
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.
