All posts by David Belson

Launching email security insights on Cloudflare Radar

Post Syndicated from David Belson original https://blog.cloudflare.com/email-security-insights-on-cloudflare-radar


During 2021’s Birthday Week, we announced our Email Routing service, which allows users to direct different types of email messages (such as marketing, transactional, or administrative) to separate accounts based on criteria such as the recipient’s address or department. Its capabilities and the volume of messages routed have grown significantly since launch.

Just a few months later, on February 23, 2022, we announced our intent to acquire Area 1 Security to protect users from phishing attacks in email, web, and network environments. Since the completion of the acquisition on April 1, 2022, Area 1’s email security capabilities have been integrated into Cloudflare’s secure access service edge (SASE) solution portfolio, and now processes tens of millions of messages daily.

Processing millions of email messages each day on behalf of our customers gives us a unique perspective on the threats posed by malicious emails, spam volume, the adoption of email authentication methods like SPF, DMARC, and DKIM, and the use of IPv4/IPv6 and TLS by email servers. Today, we are launching a new Email Security section on Cloudflare Radar to share these perspectives with you. The insights in this new section can help you better understand the state of email security as viewed across various metrics, as well as understanding real-time trends in email-borne threats. (For instance, correlating an observed increase within your organization in messages containing malicious links with a similar increase observed by Cloudflare.) Below, we review the new metrics that are now available on Radar.

Tracking malicious email

As Cloudflare’s email security service processes email messages on behalf of customers, we are able to identify and classify offending messages as malicious. As examples, malicious emails may attempt to trick recipients into sharing personal information like login details, or the messages could attempt to spread malware through embedded images, links, or attachments. The new Email Security section on Cloudflare Radar now provides insight at a global level into the aggregate share of processed messages that we have classified as malicious over the selected timeframe. During February 2024, as shown in the figure below, we found that an average of 2.1% of messages were classified as being malicious. Spikes in malicious email volume were seen on February 10 and 11, accounting for as much as 29% of messages. These spikes occurred just ahead of the Super Bowl, in line with previous observations of increases in malicious email volume in the week ahead of the game. Other notable (but lower) spikes were seen on February 13, 15, 17, 24, and 25. The summary and time series data for malicious email share are available through the Radar API.

Threat categorization

The Cloudflare Radar 2023 Year in Review highlighted some of the techniques used by attackers when carrying out attacks using malicious email messages. As noted above, these can include links or attachments leading to malware, as well as approaches like identity deception, where the message appears to be coming from a trusted contact, and brand impersonation, where the message appears to be coming from a trusted brand. In analyzing malicious email messages, Cloudflare’s email security service categorizes the threats that it finds these messages contain. (Note that a single message can contain multiple types of threats — the sender could be impersonating a trusted contact while the body of the email contains a link leading to a fake login page.)

Based on these assessments, Cloudflare Radar now provides insights into trends observed across several different groups of threat types including “Attachment”, “Link”, “Impersonation”, and “Other”. “Attachment” groups individual threat types where the attacker has attached a file to the email message, “Link” groups individual threat types where the attacker is trying to get the user to click on something, and “Impersonation” groups individual threat types where the attacker is impersonating a trusted brand or contact. The “Other” grouping includes other threat types not covered by the previous three.

During February 2024 for the “Link” grouping, as the figure below illustrates, link-based threats were unsurprisingly the most common, and were found in 58% of malicious emails. Since the display text for a link (i.e., hypertext) in HTML can be arbitrarily set, attackers can make a URL appear as if it links to a benign site when, in fact, it is actually malicious. Nearly a third of malicious emails linked to something designed to harvest user credentials. The summary and time series data for these threat categories are available through the Radar API.

For the “Attachment” grouping, during February 2024, nearly 13% of messages were found to have a malicious attachment that when opened or executed in the context of an attack, includes a call-to-action (e.g. lures target to click a link) or performs a series of actions set by an attacker. The share spiked several times throughout the month, reaching as high as 70%. The attachments in nearly 6% of messages attempted to download additional software (presumably malware) once opened.

If an email message appears to be coming from a trusted brand, users may be more likely to open it and take action, like checking the shipping status of a package or reviewing a financial transaction. During February 2024, on average, over a quarter of malicious emails were sent by attackers attempting to impersonate well-known brands. Similar to other threat categories, this one also saw a number of significant spikes, reaching as high as 88% of February 17. Just over 18% of messages were found to be trying to extort users in some fashion. It appears that such campaigns were very active in the week ahead of Valentine’s Day (February 14), although the peak was seen on February 15, at over 95% of messages.

Identity deception occurs when an attacker or someone with malicious intent sends an email claiming to be someone else, whether through use of a similar-looking domain or display name manipulation. This was the top threat category for the “Other” grouping, seen in over 36% of malicious emails during February 2024. The figure below shows three apparent “waves” of the use of this technique — the first began at the start of the month, the second around February 9, and the third around February 20. Over 11% of messages were categorized as malicious because of the reputation of the network (autonomous system) that they were sent from; some network providers are well-known sources of malicious and unwanted email.

Dangerous domains

Top-level domains, also known as TLDs, are found in the right-most portion of a hostname. For example, radar.cloudflare.com is in the .com generic Top Level Domain (gTLD), while bbc.co.uk is in the .uk country code Top Level Domain (ccTLD). As of February 2024, there are nearly 1600 Top Level Domains listed in the IANA Root Zone Database. Over the last 15 years or so, several reports have been published that look at the “most dangerous TLDs” — that is, which TLDs are most favored by threat actors. The “top” TLDs in these reports are often a mix of ccTLDs from smaller counties and newer gTLDs. On Radar, we are now sharing our own perspective on these dangerous TLDs, highlighting those where we have observed the largest shares of malicious and spam emails. The analysis is based on the sending domain’s TLD, found in the From: header of an email message. For example, if a message came from [email protected], then example.com is the sending domain, and .com is the associated TLD.

On Radar, users can view shares of spam and malicious email, and can also filter by timeframe and “type” of TLD, with options to view all (the complete list), ccTLDs (country codes), or “classic” TLDs (the original set of gTLDs specified in RFC 1591). Note that spam percentages shown here may be lower than those published in other industry analyses. Cloudflare cloud email security customers may be performing initial spam filtering before messages arrive at Cloudflare for processing, resulting in a lower percentage of messages characterized as spam by Cloudflare.

Looking back across February 2024, we found that new gTLD associates and the ccTLD zw (Zimbabwe) were the TLDs with domains originating the largest shares of malicious email, at over 85% each. New TLDs academy, directory, and bar had the largest shares of spam in email sent by associated domains, at upwards of 95%.

TLDs with the highest percentage of malicious email in February 2024
TLDs with the highest percentage of spam email in February 2024

The figure below breaks out ccTLDs, where we found that at least half of the messages coming from domains in zw (Zimbabwe, at 85%) and bd (Bangladesh, at 50%) were classified as malicious. While the share of malicious email vastly outweighed the share of spam seen from zw domains, it was much more balanced in bd and pw (Palau). A total of 80 ccTLDs saw fewer than 1% of messages classified as malicious in February 2024.

ccTLDs with the highest percentage of malicious email in February 2024

Among the “classic” TLDs, we can see that the shares of both malicious emails and spam are relatively low. Perhaps unsurprisingly, as the largest TLD, com has the largest shares of both in February 2024. Given the restrictions around registering int and gov domains, it is interesting to see that even 2% of the messages from associated domains are classified as malicious.

Classic TLDs with the highest percentage of malicious email in February 2024.

The reasons that some TLDs are responsible for a greater share of malicious and/or spam email vary — some may have loose or non-existent registration requirements, some may be more friendly to so-called “domain tasting”, and some may have particularly low domain registration fees.The malicious and spam summary shares per TLD are available through the Radar API.

Adoption of email authentication methods

SPF, DKIM, and DMARC are three email authentication methods and when used together, they help prevent spammers, phishers, and other unauthorized parties from sending emails on behalf of a domain they do not own.

Sender Policy Framework (SPF) is a way for a domain to list all the servers they send emails from, with SPF records in the DNS listing the IP addresses of all the servers that are allowed to send emails from the domain. Mail servers that receive an email message can check it against the SPF record before passing it on to the recipient’s inbox. DomainKeys Identified Mail (DKIM) enables domain owners to automatically “sign” emails from their domain with a digital “signature” that uses cryptography to mathematically verify that the email came from the domain. Domain-based Message Authentication Reporting and Conformance (DMARC) tells a receiving email server what to do, given the results after checking SPF and DKIM. A domain’s DMARC policy, stored in DMARC records, can be set in a variety of ways, instructing mail servers to quarantine emails that fail SPF or DKIM (or both), to reject such emails, or to deliver them.

These authentication methods have recently taken on increased importance, as both Google and Yahoo! have announced that during the first quarter of 2024, as part of a more aggressive effort to reduce spam, they will require bulk senders to follow best practices that include implementing stronger email authentication using standards like SPF, DKIM, and DMARC. When a given email message is evaluated against these three methods, the potential outcomes are PASS, FAIL, and NONE. The first two are self-explanatory, while NONE means that there was no associated SPF/DKIM/DMARC policy associated with the message’s sending domain.

Reviewing the average shares across February 2024, we find that over 93% of messages passed SPF authentication, while just 2.7% failed. When considering this metric, FAIL is the outcome of greater interest because SPF is easier to spoof than DKIM, and also because failure may be driven by “shadow IT” situations, such as when a company’s Marketing department uses a third party to send email on behalf of the company, but fails to add that third party to the associated SPF records. An average of 88.5% of messages passed DKIM evaluation in February, while just 2.1% failed. For DKIM, the focus should be on PASS, as there are potential non-malicious reasons that a given signature may fail to verify. For DMARC, 86.5% of messages passed authentication, while 4.2% failed, and the combination of PASS and FAIL is the focus, as the presence of an associated policy is of greatest interest for this metric, and whether the message passed or failed less so. For all three methods in this section, NONE indicates the lack of an associated policy. SPF (summary, time series), DKIM (summary, time series), and DMARC (summary, time series) data is available through the Radar API.

Protocol usage

Cloudflare has long evangelized IPv6 adoption, although it has largely been focused on making Web resources available via this not-so-new version of the protocol. However, it’s also important that other Internet services begin to support and use IPv6, and this is an area where our recent research shows that providers may be lacking.

Through analysis of inbound connections from senders’ mail servers to Cloudflare’s email servers, we can gain insight into the distribution of these connections across IPv4 and IPv6. Looking at this distribution for February 2024, we find that 95% of connections were made over IPv4, while only 5% used IPv6. This distribution is in sharp contrast to the share of IPv6 requests for IPv6-capable (dual stacked) Web content, which was 37% for the same time period. The summary and time series data for IPv4/v6 distribution are available through the Radar API.

Cloudflare has also been a long-time advocate for secure connections, launching Universal SSL during 2014’s Birthday Week, to enable secure connections between end users and Cloudflare for all of our customers’ sites (which numbered ~2 million at the time). Over the last 10 years, SSL has completed its evolution to TLS, and although many think of TLS as only being relevant for Web content, possibly due to years of being told to look for the 🔒 padlock in our browser’s address bar, TLS is also used to encrypt client/server connections across other protocols including SMTP (email), FTP (file transfer), and XMPP (messaging).

Similar to the IPv4/v6 analysis discussed above, we can also calculate the share of inbound connections to Cloudflare’s email servers that are using TLS. Messages are encrypted in transit when the connection is made over TLS, while messages sent over unencrypted connections can potentially be read or modified in transit. Fortunately, the vast majority of messages received by Cloudflare’s email servers are made over encrypted connections, with just 6% sent unencrypted during February 2024. The summary and time series data for TLS usage are available through the Radar API.

Conclusion

Although younger Internet users may eschew email in favor of communicating through a variety of messaging apps, email remains an absolutely essential Internet service, relied on by individuals, enterprises, online and offline retailers, governments, and more. However, because email is so ubiquitous, important, and inexpensive, it has also become an attractive threat vector. Cloudflare’s email routing and security services help customers manage and secure their email, and Cloudflare Radar’s new Email Security section can help security researchers, email administrators, and other interested parties understand the latest trends around threats found in malicious email, sources of spam and malicious email, and the adoption of technologies designed to prevent abuse of email.

If you have any questions about this new section, you can contact the Cloudflare Radar team at [email protected] or on social media at @CloudflareRadar (X/Twitter), cloudflare.social/@radar (Mastodon), and radar.cloudflare.com (Bluesky).

Tune in for more news, announcements and thought-provoking discussions! Don’t miss the full Security Week hub page.

Cloudflare 2023 Year in Review

Post Syndicated from David Belson original http://blog.cloudflare.com/radar-2023-year-in-review/


Cloudflare 2023 Year in Review

The 2023 Cloudflare Radar Year in Review is our fourth annual review of Internet trends and patterns observed throughout the year at both a global and country/region level across a variety of metrics. Below, we present a summary of key findings, and then explore them in more detail in subsequent sections.

Key findings

  • Global Internet traffic grew 25%, in line with peak 2022 growth. Major holidays, severe weather, and intentional shutdowns clearly impacted Internet traffic. 🔗
  • Google was again the most popular general Internet service, with 2021 leader TikTok falling to fourth place. OpenAI was the most popular service in the emerging Generative AI category, and Binance remained the most popular Cryptocurrency service. 🔗
  • Globally, over two-thirds of mobile device traffic was from Android devices. Android had a >90% share of mobile device traffic in over 25 countries/regions; peak iOS mobile device traffic share was 66%. 🔗
  • Global traffic from Starlink nearly tripled in 2023. After initiating service in Brazil in mid-2022, Starlink traffic from that country was up over 17x in 2023. 🔗
  • Google Analytics, React, and HubSpot were among the most popular technologies found on top websites. 🔗
  • Globally, nearly half of web requests used HTTP/2, with 20% using HTTP/3. 🔗
  • NodeJS was the most popular language used for making automated API requests. 🔗
  • Googlebot was responsible for the highest volume of request traffic to Cloudflare in 2023. 🔗

Connectivity & Speed

  • Over 180 Internet outages were observed around the world in 2023, with many due to government-directed regional and national shutdowns of Internet connectivity. 🔗
  • Aggregated across 2023, only a third of IPv6-capable requests worldwide were made over IPv6. In India, however, that share reached 70%. 🔗
  • The top 10 countries all had measured average download speeds above 200 Mbps, with Iceland showing the best results across all four measured Internet quality metrics. 🔗
  • Over 40% of global traffic comes from mobile devices. In more than 80 countries/regions, the majority of traffic comes from mobile devices. 🔗

Security

  • Just under 6% of global traffic was mitigated by Cloudflare’s systems as being potentially malicious or for customer-defined reasons. In the United States, 3.65% of traffic was mitigated, while in South Korea, it was 8.36%. 🔗
  • A third of global bot traffic comes from the United States, and over 11% of global bot traffic comes from Amazon Web Services. 🔗
  • Globally, Finance was the most attacked industry, but the timing of spikes in mitigated traffic and the target industries varied widely throughout the year and around the world. 🔗
  • Even as an older vulnerability, Log4j remained a top target for attacks during 2023. However, HTTP/2 Rapid Reset emerged as a significant new vulnerability, beginning with a flurry of record-breaking attacks. 🔗
  • 1.7% of TLS 1.3 traffic is using post-quantum encryption. 🔗
  • Deceptive links and extortion attempts were two of the most common types of threats found in malicious email messages. 🔗
  • Routing security, measured as the share of RPKI valid routes, improved globally during 2023. Significant growth was observed in countries including Saudi Arabia, the United Arab Emirates, and Vietnam. 🔗

Introduction

Cloudflare Radar launched in September 2020, and in the blog post that announced its availability, we talked about how its intent was to “shine a light on the Internet’s patterns”. Cloudflare’s network currently spans more than 310 cities in over 120 countries/regions, serving an average of over 50 million HTTP(S) requests per second for millions of Internet properties, in addition to handling over 70 million DNS requests per second on average. The data generated by this massive global footprint and scale, combined with data from complementary Cloudflare tools, enables Radar to provide unique near-real time perspectives on the patterns and trends we observe across the Internet. For the last several years (2020, 2021, 2022), we’ve been aggregating these insights into an annual Year In Review, shining a light on the Internet’s patterns over the course of that year. The new Cloudflare Radar 2023 Year In Review continues that tradition, featuring interactive charts, graphs, and maps you can use to explore notable Internet trends observed throughout this past year.

The 2023 Year In Review is organized into three sections: Traffic Insights & Trends, Connectivity & Speed, and Security. We have incorporated several new metrics this year, and have endeavored to keep underlying methodologies consistent with last year wherever possible. Website visualizations shown at a weekly granularity cover the period from January 2 through November 26, 2023. Trends for over 180 countries/regions are available on the website, with some smaller or less populated locations excluded due to insufficient data. Note that some of the metrics are presented only as a worldwide view, and will not be shown if a country/region is selected. Because of the control plane and analytics outage that occurred November 2-4, traffic data for relevant metrics has been interpolated for that three-day period.

Below, we provide an overview of the content contained within the major Year In Review sections (Traffic Insights & Trends, Connectivity & Speed, and Security), along with notable observations and key findings. In addition, we have also published a companion blog post that specifically explores trends seen across Top Internet Services.

However, the notable observations and key findings contained within this post only skim the surface of the unique insights that can be found in the Year in Review website, which we strongly encourage you to visit to explore the data in more detail and look at trends for your country/region. As you do so, we encourage you to consider how the trends presented within these blog posts and the website’s various sections impact your business or organization, and to think about how these insights can inform actions that you can take to improve user experience or enhance your security posture in the future.

Traffic Insights & Trends

Global Internet traffic grew 25%, in line with peak 2022 growth. Major holidays, severe weather, and intentional shutdowns clearly impacted Internet traffic.

Twenty-five years ago, Worldcom executives claimed that Internet traffic was doubling every 100 days (3.5 months). A quarter-century later, we know that these claims were unrealistically aggressive, but it is clear that the Internet is growing quickly as more and more devices are connected, consuming content from a growing universe of websites, applications, and services.

To determine the traffic trends over time, we first established a baseline, calculated as the average daily traffic volume (excluding bot traffic) over the second full calendar week (January 8-14) of 2023. We chose the second calendar week to allow time for people to get back into their “normal” routines (school, work, etc.) after the winter holidays and New Year’s Day. The percent change shown in our traffic trends chart is calculated relative to the baseline value, and represents a seven-day trailing average — it does not represent absolute traffic volume for a country/region. The seven-day averaging is done to smooth the sharp changes seen with a daily granularity. A trend line for 2022 is shown for comparison purposes.

Our data shows that globally, Internet traffic grew 25% in 2023, with nominal initial growth accelerating during the second half of the year. Overall, the pattern is similar to that observed in 2022 (excepting last year’s late February spike), and peak growth for the year is just slightly above the peak growth level seen in 2022. Traffic patterns in Canada were also rather consistent year-over-year, exhibiting similar seasonality, and peak growth above 30% in both 2022 and 2023. In many countries, the 2022 trend line shows a clear drop in traffic heading into the Christmas holiday, with a slight rebound ahead of New Year’s Day. It will be interesting to see if traffic follows this pattern in 2023 as well.

Global Internet traffic growth in 2023, compared with 2022
Internet traffic growth in Canada in 2023, compared with 2022 

Comparisons with 2022 traffic trends helps make the impact of major holidays on Internet traffic more visible. For example, in Muslim countries including Indonesia, Turkey, and the United Arab Emirates, the celebration of Eid-Ul-Fitr, the festival marking the end of the fast of Ramadan, is visible as a noticeable drop in traffic around April 21-23, 2023, just before a similar drop visible in the 2022 trend line during last year’s celebration on May 2-3. In Italy, a drop in traffic is clearly visible around Pasqua di Resurrezione and Lunedì dell’Angelo (Easter Sunday and Monday) on April 9-10, one week ahead of a similar drop in traffic in 2022

Internet traffic growth in Indonesia in 2023, compared with 2022 

In addition, extended disruptions to Internet connectivity are also clearly visible within the traffic trend charts. Examples include Mauritania, where government-directed shutdowns occurred from March 6-12 and May 30 – June 6, and Gabon, where a shutdown was in place from August 26-30, as well as Guam, where Super Typhoon Mawar caused a multi-week drop in traffic starting on May 24.

Internet traffic growth in Mauritania in 2023, compared with 2022 
Internet traffic growth in Guam in 2023, compared with 2022 

Google was again the most popular general Internet service, with 2021 leader TikTok falling to fourth place. OpenAI was the most popular service in the emerging Generative AI category, and Binance remained the most popular Cryptocurrency service.

One of the most popular sections of the Year In Review over the last several years has been the exploration of the most popular Internet services, both generally and across a number of categories. These rankings of service popularity are based on analysis of anonymized query data of traffic to our 1.1.1.1 public DNS resolver from millions of users around the world. Although DNS resolution operates at a domain level, domains that belong to a single Internet service are grouped together for the purposes of these rankings.

In the overall category, Google once again held the top spot, owing in part to its broad portfolio of services as well as the popularity of the Android mobile operating system. In addition to perennial categories like e-commerce, video streaming, and messaging, this year we also looked at Generative AI, which has been on a meteoric rise in 2023. In this category, OpenAI held the top spot, building on the success and popularity of ChatGPT, which it launched only a year ago. And despite the turmoil seen in the cryptocurrency space this year, Binance remained the most popular service in that category.

We explore these categorical rankings, as well as trends seen by specific services, in more detail in a separate blog post.

Globally, over two-thirds of mobile device traffic was from Android devices. Android had a >90% share of mobile device traffic in over 25 countries/regions; peak iOS mobile device traffic share was 66%.

Apple’s iOS and Google’s Android are the two leading operating systems used on mobile devices, and analysis of information in the user agent reported with each request allows us to gain insight into the distribution of traffic by client operating system throughout the year. Given the wide range of both devices and price points for Android devices, it is not surprising that Android is responsible for the majority of mobile device traffic when aggregated globally.

Globally, over two-thirds of mobile device traffic was from Android devices. The split is in line with Android/iOS usage observed in 2022. When looking at the countries/regions with the highest levels of Android usage, we find Bangladesh and Papua New Guinea at the top of the list, both with over 95% of mobile device traffic coming from Android devices. Looking more closely at other countries that see particularly high levels of Android usage, it is interesting to note that they are largely in Africa, Oceania/Asia, and South America, and that many have lower levels of gross national income per capita. This is presumably where the availability of lower priced “budget” phones plays to Android’s advantage from an adoption perspective.

In contrast, while the share of mobile device traffic from iOS at a country/region level never tops 70%, many of the countries with an iOS share over 50%, including Denmark, Australia, Japan, and Canada, have comparatively higher gross national income per capita, which likely speaks to a greater ability to afford higher priced devices.

Mobile device traffic operating system distribution across selected countries

SpaceX’s Starlink high-speed satellite Internet service has continued to rapidly grow its footprint since launching in 2019, making high performance Internet connections available in many countries/regions that were previously unserved or underserved by traditional wired or wireless broadband. The current leader in the space, in the future it will be joined by Amazon’s Project Kuiper service, which launched its first two test satellites this year, as well as Eutelsat OneWeb, which grew its satellite constellation in 2023 as well.

To track the growth in usage and availability of Starlink’s service, we analyzed aggregate Cloudflare traffic volumes associated with the service’s autonomous system (AS14593) throughout 2023. Although Starlink is not yet available globally, we did see traffic growth across a number of countries/regions. The request volume shown on the trend line in the chart represents a seven-day trailing average. A trend line for 2022 is shown for comparison purposes, and is scaled to the maximum value across 2022 and 2023.

Globally, we saw Starlink traffic more than triple this year. In the United States, traffic from Starlink was up over 2.5x, and grew over 17x in Brazil. In countries where Starlink turned up service in 2023, including Kenya, the Philippines, and Zambia, we saw traffic grow rapidly once the service became available.

Starlink traffic growth in Brazil, compared with 2022

Google Analytics, React, and HubSpot were among the most popular technologies found on top websites.

Modern websites are complex productions, relying on a mix of frameworks, platforms, services, and tools, and the developer community is responsible for making them coexist with one another to deliver a seamless experience. Using the Cloudflare Radar URL Scanner, which we launched in March 2023, we scanned websites associated with the top 5000 domains to identify the most popular technologies and services used across a dozen different categories, including (but not limited to) Analytics, where Google Analytics was by far the most widely used; JavaScript Frameworks, where React had a commanding lead; and Marketing Automation providers, where leader HubSpot was closely followed by several competitors.

Top website technologies, JavaScript frameworks category

Globally, nearly half of web requests used HTTP/2, with 20% using HTTP/3.

HTTP (HyperText Transfer Protocol) is the core protocol that the web relies upon. HTTP/1.0 was first standardized in 1996, HTTP/1.1 in 1999, and HTTP/2 in 2015. The most recent version, HTTP/3, was completed in 2022, and runs on top of QUIC, a new transport protocol. On the client side, HTTP/3 support is enabled by default in the latest versions of desktop and mobile Google Chrome and Mozilla Firefox, and for a portion of Apple Safari users. HTTP/3 is available for free for all Cloudflare customers, though not every customer chooses to enable it.

Using QUIC allows HTTP/3 to deliver improved performance by mitigating the effects of packet loss and network changes, as well as establishing connections more quickly. It also provides encryption by default, mitigating the risk of attacks. Websites and applications that remain on older versions of HTTP miss out on these benefits.

Analysis of the HTTP version negotiated for each request allows us to gain insight into the distribution of traffic by the various versions of the protocol aggregated throughout the year. (“HTTP/1.x” aggregates requests made over HTTP/1.0 and HTTP/1.1.) At a global level, 20% of requests were made over the latest version, HTTP/3. Another third of requests were made over the comparatively ancient HTTP/1.x versions, while HTTP/2 remained dominant, and accounted for the 47% balance.

Global HTTP version traffic distribution

Looking at the version distribution geographically, we found a number of Asian countries, including Nepal, Thailand, Malaysia, and Sri Lanka among those with highest rates of HTTP/3 usage, although these rates did not exceed 35%. In contrast, more than half of the requests from ten countries, including Ireland, Albania, Finland, and China, were made over HTTP/1.x during 2023.

NodeJS was the most popular language used for making automated API requests.

In addition, as developers increasingly use automated API calls to power dynamic websites and applications, we can use our unique visibility into Web traffic to identify the top languages these API clients are written in. Looking at API-related requests determined to not be coming from a person using a browser or native mobile application, we applied heuristics to help identify the language used to build the client.

Our analysis found that almost 15% of automated API requests are made by NodeJS clients, with Go, Java, Python, and .NET holding smaller shares.

Top languages used to make automated API calls

Googlebot was responsible for the highest volume of request traffic to Cloudflare in 2023.

Cloudflare Radar enables users to see Internet traffic trends at a country/region or network level over a selected period of time. However, we wanted to zoom out a bit, and look at the traffic Cloudflare saw from the entire IPv4 Internet over the course of the entire year. Hilbert curves, as “continuous space-filling curves”, have properties that are useful for visualizing the Internet’s IPv4 address space.

Using a Hilbert curve visualization, we can visualize aggregated request traffic (over IPv4) to Cloudflare from January 1st through November 26th, 2023. In order to make the amount of data used for the visualization manageable, IP addresses are aggregated at a /20 level, meaning that at the highest zoom level, each cell represents traffic from 4096 IPv4 addresses. (The sheer size of the IPv6 address space would make associated traffic very hard to see in such a visualization, especially as such a small amount has been allocated for assignment by the Regional Internet Registries.)

Within the visualization, IP addresses are grouped by ownership, and for much of the IP address space shown there, a mouseover at the default zoom level will show the Regional Internet Registry (RIR) that the address block belongs to. However, there are also a number of blocks that were assigned prior to the existence of the RIR system, and for these, they are labeled with the name of the organization that owns them. Progressive zooming ultimately shows the autonomous system and country/region that the IP address block is associated with, as well as its share of traffic relative to the maximum. (If a country/region is selected, only the IP address blocks associated with that location are visible.) Overall traffic shares are indicated by shading based on a color scale, and although a number of large unshaded blocks are visible, this does not necessarily mean that the associated address space is unused, but rather that it may be used in a way that does not generate traffic to Cloudflare.

Hilbert curve showing aggregated 2023 traffic to Cloudflare across the IPv4 Internet

Areas of higher request volume, indicated by warmer orange/red shading, are visibly scattered throughout the plot, but the IP address block that had the maximum request volume to Cloudflare during 2023 was 66.249.64.0/20, which belongs to Google. This IP address block is one of several used by the Googlebot web crawler, which is a likely explanation for the high request volume, given the number of web properties on Cloudflare’s network.

Zoomed view of the Hilbert curve showing the IPv4 address block that generated the highest volume of requests

It is hard to do this visualization justice with a short summary and static screenshot. To explore it in more detail, we encourage you to go to the Year in Review website and explore it by dragging and zooming to move around the IPv4 Internet.

Connectivity & Speed

Over 180 Internet outages were observed around the world in 2023, with many due to government-directed regional and national shutdowns of Internet connectivity.

During 2023, we have written frequently about Internet outages, whether due to technical issues, government-directed shutdowns, or geopolitical conflict, as well as infrastructure resilience issues (including fiber cuts, power outages, and severe weather) highlighted in our quarterly summaries. The impacts of these outages can be significant, including significant economic losses and severely limited communications. The Cloudflare Radar Outage Center tracks these Internet outages, and uses Cloudflare traffic data for insights into their scope and duration.

Some of these outages seen through the year were short-lived, lasting just a couple of hours, while others have stretched on for multiple months. In the latter category, localized government-directed shutdowns in Manipur, India and Amhara, Ethiopia have lasted over seven and four months respectively (as of early December). In the former category, Iraq frequently experienced multi-hour nationwide Internet shutdowns intended to prevent cheating on academic exams — these contribute to the clustering visible in the timeline during June, July, and August.

Within the timeline on the Year in Review website, mousing over a dot will display metadata about that outage, and clicking on it will open a page with additional information. If a country/region is selected, only outages for that country will be displayed.

Internet outages were observed around the world during 2023

Aggregated across 2023, only a third of IPv6-capable requests worldwide were made over IPv6. In India, however, that share reached 70%.

IPv6 has been around in some fashion since 1998, with an expanded address space that better supports the universe of Internet-connected devices that has grown exponentially over the last quarter-century. And over that time, available IPv4 space has been exhausted, leading connectivity providers to resort to solutions like Network Address Translation, and cloud and hosting providers to acquire blocks of IPv4 address space for as much as $50 per address. IPv6 also brings a number of other benefits to network providers, and if implemented correctly, adoption should be transparent from an end user perspective.

Cloudflare has been a vocal and active advocate for IPv6 stretching all the way back to our first birthday in 2011, when we announced our Automatic IPv6 Gateway, which enabled free IPv6 support for all of our customers. Just a few years later, we enabled IPv6 support by default for all of our customers. (Although it is enabled by default, not all customers choose to keep it enabled for a variety of reasons.) However, this support is only half of the equation for driving IPv6 adoption, as end user connections need to support it as well. (Technically, it is a bit more complex than that, but those are the two foundational requirements.) Analysis of the IP version used for each request made to Cloudflare allows us to gain insight into the distribution of traffic by the various versions of the protocol, aggregated throughout the year.

Thanks to near-complete IPv6 adoption by Indian telecommunications provider Reliance Jio, 70% of dual-stacked requests from Indian users were made via IPv6. India was followed closely by Malaysia, where 66% of dual-stacked requests were made over IPv6 during 2023, thanks to strong IPv6 adoption rates across leading Internet providers within the country. Other countries that saw more than half of dual-stacked requests, on average, made over IPv6 include Saudi Arabia, Vietnam, Greece, France, Uruguay, and Thailand. In contrast, there were on the order of 40 countries/regions where less than 1% of dual-stacked requests were made over IPv6 during 2023. Lagging adoption across such a large cohort of countries/regions 25 years after IPv6 was first published as a draft standard is quite surprising.

IPv4/IPv6 traffic distribution in India

The top 10 countries all had measured average download speeds above 200 Mbps, with Iceland showing the best results across all four measured Internet quality metrics.

Even when they are not facing Internet outages, users around the world are often contending with poor performance on their Internet connections, whether due to low speeds, high latency, or a combination of these factors. Although Internet providers continue to evolve their service portfolios to offer increased connection speeds and reduced latency in order to support growth in use cases like online gaming and videoconferencing, consumer adoption is often mixed due to cost, availability, or other issues. By aggregating the results of speed.cloudflare.com tests taken during 2023, we can get a geographic perspective on connection quality metrics including average download and upload speeds, and average idle and loaded latencies, as well as the distribution of the measurements.

In Iceland, over 85% of all Internet connections are over fiber, and this is reflected in its ranking as the country with the best overall Internet quality metrics, as speed test results show that providers there deliver the highest average speeds (282.5 Mbps download, 179.9 Mbps upload) and lowest average latencies (9.6 ms idle, 77.1 ms loaded). The histogram below shows that while there is a large cluster of download speeds between 0–100 Mbps, there were also a significant number of tests that measured even higher speeds, including some in excess of 1 Gbps.

Western European countries including Spain, Portugal, and Denmark also ranked among the top 10 across multiple Internet quality metrics.

Download and upload speed test result distribution in Iceland

Over 40% of global traffic comes from mobile devices. In more than 80 countries/regions, the majority of traffic comes from mobile devices.

Over the last 15 years or so, mobile devices have become increasingly ubiquitous, becoming indispensable in both our personal and professional lives, thanks in large part to their ability to enable us to access the Internet from nearly anywhere at any time. In some countries/regions, mobile devices primarily connect to the Internet via Wi-Fi, while others are “mobile first”, where Internet access is primarily through 4G/5G services.

Analysis of information contained with the user agent reported with each request to Cloudflare enables us to categorize it as coming from a mobile, desktop, or other type of device. Aggregating this categorization throughout the year at a global level, we found that 42% of traffic came from mobile devices, with 58% coming from desktop devices such as laptops and “classic” PCs. These traffic shares were in line with those measured in 2022. 79% of traffic came from mobile devices in Zambia, making it the country with the largest mobile device traffic share in 2023. Other countries/regions that had more than 50% of traffic come from mobile devices were concentrated in the Middle East/Africa, the Asia Pacific Region, and South/Central America. In contrast, Finland had one of the highest shares of desktop device traffic, at 80%.

Desktop and mobile device traffic distribution across selected countries

Security

Security

Just under 6% of global traffic was mitigated by Cloudflare’s systems as being potentially malicious or for customer-defined reasons. In the United States, 3.65% of traffic was mitigated, while in South Korea, it was 8.36%.

Malicious bots are often used to attack websites and applications. To protect customers from these threats, Cloudflare mitigates (blocks) this attack traffic using DDoS mitigation techniques or Web Application Firewall (WAF) Managed Rules. However, customers may also choose to have Cloudflare mitigate traffic using other techniques for a variety of other reasons, such as rate-limiting requests, or blocking all traffic from a given location, even if it isn’t malicious. Analyzing traffic to Cloudflare’s network seen throughout 2023, we looked at the overall share that was mitigated (for any reason), as well as the share that was mitigated as a DDoS attack or by WAF Managed Rules.

Overall, just under 6% of global traffic was mitigated by Cloudflare’s systems as being potentially malicious or for customer-defined reasons, while only around 2% of it saw DDoS/Managed WAF mitigations. Some countries, such as Bermuda, saw the percentages for the two metrics track very closely, while other countries, like Pakistan and South Africa showed much larger gaps between their trend lines.

Mitigated traffic trends in Pakistan

A third of global bot traffic comes from the United States, and over 11% of global bot traffic comes from Amazon Web Services.

Bot traffic describes any non-human Internet traffic, and monitoring bot traffic levels can help site and application owners spot potentially malicious activity. Of course, bots can be helpful too, and Cloudflare maintains a list of verified bots to help keep the Internet healthy. Verified bots include those used for things like search engine indexing, performance testing, and availability monitoring. Regardless of intent, we wanted to look at where bot traffic was coming from, and we can use the IP address of a request to identify the network (autonomous system) and country/region associated with the bot making the request. Perhaps unsurprisingly, we found that cloud platforms were among the leading sources of bot traffic. This is likely due to the ease of automating the provisioning/teardown of compute resources and the relatively low cost of doing so, the distributed geographic footprint of cloud platforms, and the availability of high-bandwidth connections.

Globally, nearly 12% of bot traffic comes from Amazon Web Services, and over 7% from Google. Some of it comes from consumer ISPs as well, with U.S. broadband provider Comcast originating over 1.5% of global bot traffic. A disproportionate amount of bot traffic originates from the United States, responsible for nearly a third of global bot traffic, four times that of Germany, which originates just 8%. Within the United States, Amazon’s total share of bot traffic just edges out Google’s.

Global bot traffic distribution by source network
Global bot traffic distribution by source country

Globally, Finance was the most attacked industry, but the timing of spikes in mitigated traffic and the target industries varied widely throughout the year and around the world.

The industries targeted by attacks often shift over time, depending on the intent of the attackers. They may be trying to cause financial harm by attacking ecommerce sites during a busy shopping period, or they may be trying to make a political statement by attacking government-related sites. To identify industry-targeted attack activity during 2023, we analyzed mitigated traffic for customers that had an associated industry and vertical within their customer record. Mitigated traffic was aggregated weekly by source country/region across 18 target industries.

At a global level, Finance organizations were the most attacked over the course of the year, though we saw a significant amount of volatility from week-to-week. Interestingly, some clustering was evident, as Finance, which includes organizations that provide websites and applications for mobile payments, investments/trading, and cryptocurrency, was also a top target for a number of European countries, including Austria, Switzerland, France, the United Kingdom, Ireland, Italy, and the Netherlands, as well as in North America, for Canada, the United States, and Mexico. The Health industry, which includes companies that make exercise equipment, as well medical testing device manufacturers, was a top target across multiple African countries, including Benin, Côte d’Ivoire, Cameroon, Ethiopia, Senegal, and Somalia.

Overall, however, the year started slowly, with no industry seeing more than 8% of traffic being mitigated. As the first quarter progressed, Professional Services and News/Media/Publications organizations saw spikes in the share of mitigated traffic later in January, with Health jumping in mid-February and Law & Government organizations seeing a sharp increase in mitigated traffic in early March. Customers in the Arts/Entertainment/Recreation industry classification were apparently targeted by a multi-week attack campaign, with more than 20% of traffic mitigated during the weeks of March 26, April 2, and April 9. The overall peak during the year was experienced by the Professional Services industry, which saw a mitigated traffic share of 38.4% for the week of August 6, nearly twice its January spike. The timing of spikes and the industries experiencing those spikes varied widely across countries/regions.

Global mitigated traffic share by industry, week of August 6, 2023

Even as an older vulnerability, Log4j remained a top target for attacks during 2023. However, HTTP/2 Rapid Reset emerged as a significant new vulnerability, beginning with a flurry of record-breaking attacks.

In August 2023, we published a blog post that explored traffic seen by Cloudflare for the most commonly exploited vulnerabilities of 2022, as listed in a joint Cybersecurity Advisory. These included vulnerabilities in the Log4j Java-based logging utility, Microsoft Exchange, Atlassian’s Confluence platform, VMWare, and F5’s BIG-IP traffic management system. Although these are older vulnerabilities, attackers continued to actively target and exploit them throughout 2023, in part because organizations are frequently slow to follow the recommendations outlined in the Cybersecurity Advisory. We updated the analysis done for our blog post to include just the attack activity seen in 2023.

Attack activity by vulnerability varied by location, and in some, attacks targeted only a subset of the vulnerabilities. Aggregated worldwide, attack volume targeting Log4j consistently dwarfed that seen for the other vulnerabilities, and saw spikes during the last week of October and mid-late November; attack activity targeting Atlassian vulnerabilities increased in late July and trended slowly higher through the rest of the year. At a country/region level, Log4j was generally the most targeted vulnerability. In countries including France, Germany, India, and the United States, associated attack volume remained at a significant level throughout the year, while in other countries/regions, these attacks are most visible as infrequent, short-lived spikes within a country/region’s graphs, punctuating otherwise low levels of attack volume.

Global attack activity trends for commonly exploited vulnerabilities

We also expect that through 2024, attackers will continue to target the HTTP/2 Rapid Reset vulnerability disclosed in October. The vulnerability (see CVE-2023-44487 for details) abuses an underlying weakness in the request cancellation feature of the HTTP/2 protocol, leading to resource exhaustion on the target web/proxy server. Between the end of August and the beginning of October, we saw a number of attacks targeting this vulnerability. Across this set of attacks, the average attack rate was 30M requests per second (rps), with nearly 90 peaking above 100M rps, and the largest one hitting 201M rps. This largest attack was nearly 3x bigger than our previous biggest attack on record.

One notable concern about this vulnerability is that the attacker was able to generate such a large attack with a botnet consisting of just 20,000 compromised systems. This is much smaller than some of the largest botnets today, which comprise hundreds of thousands or millions of hosts. With average web traffic estimated to be between 1–3 billion requests per second, attacks using this method could conceivably focus an entire web’s worth of requests on a few unsuspecting targets.

HTTP/2 Rapid Reset campaign of hyper-volumetric DDoS attacks

1.7% of TLS 1.3 traffic is using post-quantum encryption

Post-quantum refers to a new set of cryptographic techniques that can protect data from adversaries with the ability to capture and store today’s data for decryption by sufficiently powerful quantum computers in the future. The Cloudflare Research team has been exploring post-quantum cryptography since 2017.

In October 2022, we enabled post-quantum key agreement at our edge by default, but use of it requires that the browser support it as well. Google’s Chrome browser started to slowly enable support in August 2023, and we expect its support will continue to grow in 2024, and that other browsers will add support over time as well. In September 2023, we announced general availability of post-quantum cryptography for both inbound and outbound connections and for many internal services, and expect to finish upgrading all internal services by the end of 2024.

After first enabling support in August, Chrome began ramping the number of browsers (version 116 and later) that use post-quantum cryptography, resulting in gradual growth leading to the significant increase seen on November 8. These actions helped push the share of TLS 1.3 traffic using post-quantum encryption to 1.7% at the end of November. As this ramp continues with future Chrome updates, and as other browsers add support for post-quantum encryption, we expect this share to continue to grow rapidly in 2024.

Growth trends in post-quantum encrypted TLS 1.3 traffic

Deceptive links and extortion attempts were two of the most common types of threats found in malicious email messages.

As the #1 business application, email represents a very attractive entry point into enterprise networks for attackers. Targeted malicious emails may attempt to impersonate an otherwise legitimate sender, try to get the user to click on a deceptive link, or contain a dangerous attachment, among other types of threats. Cloudflare Area 1 Email Security protects customers from email-based attacks, including those carried out through targeted malicious email messages. Over the course of 2023, an average of 2.65% of emails analyzed by Cloudflare Area 1 were found to be malicious. Aggregated at a weekly level, spikes to over 3.5%, 4.5%, and over 5% were seen in early February, early September, and late October respectively.

Global malicious email share trends

When carrying out attacks using malicious email messages, attackers use a variety of techniques, which we refer to as threat categories. These categories are defined and explored in detail in Cloudflare’s 2023 phishing threats report. Analysis of malicious emails shows that messages may contain multiple types of threats, highlighting the need for a comprehensive email security solution. Exploring threat activity trends for these categories, aggregated weekly across the year, we found that as much as 80% of them contained deceptive links.

However, it appears that attackers may have started to shift strategies in August, as the percentage of emails containing deceptive links began to fall while the share proposing to extort the recipient began to increase. By the end of October, and into November, the two threat categories had traded places, with nearly 80% of analyzed malicious emails containing an extortion threat, while only 20% contained deceptive links, as seen towards the right side of the graph below. However, this extortion campaign may have been short-lived, as its percentage fell almost as quickly as it rose. Identity deception and credential harvesting were also commonly identified threats, though the share of emails they were found in gradually declined over the course of the year.

Global threat category trends for malicious emails

Routing security, measured as the share of RPKI valid routes, improved globally during 2023. Significant growth was observed in countries including Saudi Arabia, the United Arab Emirates, and Vietnam.

Border Gateway Protocol (BGP) is the routing protocol for the Internet, communicating routes between networks, enabling traffic to flow between source and destination. However, because it relies on trust between networks, incorrect information shared between peers, whether done so intentionally or not, can send traffic to the wrong place, potentially with malicious results. Resource Public Key Infrastructure (RPKI) is a cryptographic method of signing records that associate a BGP route announcement with the correct originating AS number. In simple terms, it provides a way of ensuring that the information being shared originally came from a network that is allowed to do so. (Note that this is only half of the challenge of implementing routing security, as network providers also need to validate these signatures and filter out invalid announcements.) In the United States, the federal government recognizes the importance of routing security, with the Federal Communications Commission holding a “Border Gateway Protocol Security Workshop” on July 31.

Cloudflare has been a strong proponent of routing security, from being a founding participant in the MANRS CDN and Cloud Programme, to releasing an RPKI toolkit for network operators, to providing a public tool that enables users to test whether their Internet provider has implemented BGP safely, to presenting at this summer’s FCC workshop.

Building on the July release of the new Routing page on Cloudflare Radar, we analyzed data from RIPE NCC’s RPKI daily archive to determine the share of RPKI valid routes (as opposed to those route announcements that are invalid or whose status is unknown)  and how that share has changed over the course of 2023. Since the start of the year, the global share of RPKI valid routes grew to nearly 45%, up six percentage points from the end of 2022. At a country/region level, we are looking at routes announced by autonomous systems associated with the given country/region. In the United States, the increased FCC attention on routing security is arguably warranted, as less than a third of the routes are RPKI valid. Although this is significantly better than South Korea, where less than 1% of announced routes are RPKI valid, it trails Vietnam significantly, where the share increased 35 percentage points during the first half of the year to 90%.

RPKI valid route growth in Vietnam

Conclusion

In the Cloudflare Radar 2023 Year In Review, we have attempted to provide a snapshot of the Internet, as dynamic as it is, through trend graphs and summary statistics, providing unique perspectives on Internet traffic, Internet quality, and Internet security, and how key metrics across these areas vary around the world.

As we said in the introduction, we strongly encourage you to visit the Cloudflare Radar 2023 Year In Review website and explore the trends relevant to metrics, countries/regions, and industries of interest, and to consider how they impact your organization so that you are appropriately prepared for 2024.

If you have any questions, you can contact the Cloudflare Radar team at [email protected] or on social media at @CloudflareRadar (X/Twitter), cloudflare.social/@radar (Mastodon), and radar.cloudflare.com (Bluesky).

Acknowledgements

As we noted last year, it truly is a team effort to produce the data, website, and content for our annual Year in Review, and I’d like to acknowledge those team members that contributed to this year’s effort. Thank you to: Sabina Zejnilovic, Jorge Pacheco, Carlos Azevedo (Data Science); Arun Chintalapati, Reza Mohammady (Design); Vasco Asturiano, Nuno Pereira, Tiago Dias (Front End Development); João Tomé (Most popular Internet services); and Davide Marquês, Paula Tavares, Celso Martinho (Project/Engineering Management) as well as countless other colleagues for their answers, edits, and ideas.

Q3 2023 Internet disruption summary

Post Syndicated from David Belson original http://blog.cloudflare.com/q3-2023-internet-disruption-summary/

This post is also available in Deutsch, Français and Español.

Q3 2023 Internet disruption summary

Cloudflare operates in more than 300 cities in over 100 countries, where we interconnect with over 12,500 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions.

We have been publishing these summaries since the first quarter of 2022, and over that time, the charts on Cloudflare Radar have evolved. Many of the traffic graphs in early editions of this summary were screenshots from the relevant traffic pages on Radar. Late last year, we launched the ability to download graphs, and earlier this year, to embed dynamic graphs, and these summaries have taken advantage of those capabilities where possible. Sharp-eyed readers may notice an additional evolution in some of the graphs below: yellow highlighting indicating an observed “traffic anomaly”. Identification of such anomalies, along with the ability to be notified about them, as well as a timeline enhancement (embedded below) to the Cloudflare Radar Outage Center, were launched as part of Birthday Week at the end of September. More information on these new features can be found in our announcement blog post.

As we have seen in previous quarters, Iraq pursued an aggressive plan of government-directed Internet shutdowns intended to prevent cheating on exams, and several other African countries implemented politically motivated shutdowns. Damage to several submarine cables, as well as planned maintenance to others, caused Internet disruptions across a number of countries during the third quarter. Natural disasters, including wildfires and an earthquake, caused issues with connectivity, as did power outages in multiple countries. An acknowledged cyberattack resulted in a major US university intentionally disconnecting from the Internet, while a number of other major Internet providers acknowledged problems on their networks without ever disclosing the root cause of those problems.

(Note that the Internet disruptions related to the Israel/Palestine conflict are not covered in this post, as they began on October 7 in Q4 of 2023. Disruptions related to this conflict are being tracked, with additional insights found on the Cloudflare blog and @CloudflareRadar on X/Twitter.)

Government directed

Because the Internet has become a critical communications tool, Internet shutdowns are often used by governments as a means of controlling communication both within a country and with the outside world. These government-directed shutdowns are imposed for a variety of reasons, including during periods of civil unrest and protests around elections, and as a deterrent against cheating during exams.

Iraq

As we have discussed in past summaries, Internet shutdowns are used by some governments in an attempt to prevent cheating on national high school or baccalaureate exams. These shutdowns have a nationwide impact, and it isn’t clear whether they are ultimately successful at mitigating cheating. As we have also discussed in the past, such shutdowns frequently occur in Iraq, and that was certainly the case during the third quarter, with rounds of shutdowns occurring during all three months.

The first round of exam-related Internet shutdowns during the quarter in Iraq was a continuation of a set that started in June, and continued on into July, targeting cheating on 9th and 12th grade exams. On ten days between July 4 and July 17, Internet connectivity was shut down on AS203214 (HulumTele), AS59588 (ZAINAS-IQ), AS199739 (Earthlink), AS203735 (Capacities-LTD), AS51684 (ASIACELL), and AS58322 (Halasat) in Iraq (except for the Kurdistan Region) between 04:00 – 08:00 local time (01:00 – 05:00 UTC).

During the second week of August, several networks in the Kurdistan region of Iraq again implemented daily exam-related Internet shutdowns due to a second round of exams for 12th grade students. These shutdowns took place between 06:00 – 08:00 local time (03:00 – 05:00 UTC), and impacted AS21277 (Newroz Telecom), AS48492 (IQ-Online), and AS59625 (KorekTel) from August 6-13. These two hour shutdowns were similar to those seen in the region in June.

A second round of 9th grade exams in August drove a week of Internet shutdowns across Iraq (except the Kurdistan region) between August 21 and August 29. Connectivity was shut down between 04:00 – 08:00 local time (01:00 – 05:00 UTC) across the same networks impacted by the shutdowns implemented in July.

Following the second round of 9th grade exams in August, the second round of 12th grade exams in Iraq (except the Kurdistan region) occurred in September, and with these exams, came yet another round of Internet shutdowns. Impacting the same set of network providers as the previous two months, these shutdowns occurred between September 17-30. However, while they started at the same time (04:00 local time, 01:00 UTC), they were shorter than previous rounds, ending an hour earlier (07:00 local time, 04:00 UTC).

Senegal

On July 31, following the arrest of the Senegalese opposition leader, the Senegalese Ministry of Communication, Telecommunications and the Digital Economy once again ordered the disconnection of mobile Internet connectivity in Senegal as shown in the communiqué below. These disruptions to mobile Internet access were visible on two of the four network providers within the country: AS37196 (Sudatel Senegal) and AS37649 (Tigo/Free).

As shown in the graphs below, the shutdowns began mid-morning local time, generally between 08:00 and 10:00, from July 31 through August 5, and ended early the next morning, generally between midnight and 02:00. The final shutdown on August 5 was an exception, ending at 22:00 local time on both networks. (Senegal is UTC+0, so the local times are the same as UTC.)

Ethiopia

Following days of clashes between the federal military and local militia, mobile Internet connectivity was shut down in Amhara, Ethiopia. Cloudflare saw traffic to the region drop around 21:00 local time (18:00 UTC) on August 2. This is the second time that authorities have shut down mobile Internet connectivity in Amhara in 2023 — the first time was on April 6 after protests broke out following the federal government’s move to disband regional security forces. (Note that the country is no stranger to Internet shutdowns, as they have taken such action multiple times over the last several years.) Despite calls to restore connectivity, mobile Internet remained unavailable through the end of the third quarter, as seen in the figure below.

Gabon

On August 26, following contentious presidential elections in Gabon, Internet connectivity was shut down in order to “prevent the spread of calls for violence”. As shown in the figure below, traffic began to fall just before 17:00 local time (16:00 UTC), and remained at zero through approximately 07:30 local time (06:30 UTC) on August 30. Connectivity was restored hours after military officers seized power in the country, placing President Ali Bongo under house arrest and naming a new leader after the country’s election body announced Bongo had won a third term.

Cable cuts

Cameroon

On July 7, an X/Twitter post from Cameroon Telecommunications alerted subscribers to disruptions to voice and data services, with a subsequent post nearly six hours later noting that services had been re-established. Although these posts did not provide details on the cause of the disruption, a Facebook post from the operator included an attached communiqué explaining that “The optical fibre has been severed by road maintenance operations, causing major disruptions in the delivery of our services.” The figure below shows the impact of this fiber damage, with traffic from AS15964 (CAMNET-AS) dropping sharply around 11:30 local time (10:30 UTC), and returning to expected levels by 18:00 local time (17:00 UTC).

Liberia

Damage to the Africa Coast to Europe (ACE) submarine cable disrupted Internet connectivity in Liberia on July 28. A Facebook post from the Liberia Telecommunications Authority (LTA) noted “The Liberia Telecommunications Authority(LTA) announces the temporary interruption of all nationwide Internet services due to the breakdown of the Africa Coast to Europe Cable in Ivory Coast.” and also highlighted that the ACE cable serves as the “sole source of internet connectivity between Europe and Liberia”. The figure below shows a near complete loss of traffic starting at 13:00 local time (13:00 UTC) and gradually recovering over the next several hours, returning to expected levels by 17:00 local time (17:00 UTC).

Togo, Benin, Namibia, and the Republic of Congo (Brazzaville)

On August 6, the West African Cable System (WACS) and the South Atlantic 3 (SAT–3) undersea cables were damaged by an undersea landslide in the Congo Canyon, located at the mouth of the Congo River. The damage to the cables impacted Internet connectivity in Togo, Benin, Namibia, and the Republic of Congo (Brazzaville). Social media posts from Telecom Namibia and Canalbox Congo alerted subscribers that connectivity would be impacted as a result of the damage to the cables.

Cable repair ship CS Leon Thevenin was called upon to perform repairs, but it took several weeks for it to arrive at the site of the damage, and then additional time to perform the repairs, which were reportedly completed on September 6. Network operators in impacted countries were able to shift some traffic to alternate cables, such as Google’s Equiano cable, which went live in February 2023.

As such, the graphs below illustrate that there was not a complete loss of traffic for impacted countries. To that end, traffic in Togo appeared to recover several weeks before the cable repairs were completed. The full impact is harder to see in the graphs for Benin, Namibia, and the Republic of Congo (Brazzaville) because the selected timeframe is long enough to force data aggregation at a daily level, but it is clearly visible in graphs covering shorter periods of time (with data aggregation at an hourly level) during the weeks after the cable cut occurred.

South Sudan

Highlighting the interconnected nature of the Internet, fiber cuts in Uganda caused a brief Internet disruption for customers on MTN South Sudan (AS37594) on August 14, occurring between 13:00 – 15:00 local time (11:00 – 13:00 UTC), and impacting an estimated 438,000 users. An X/Twitter post from the provider that afternoon told subscribers “We sincerely apologize for the network issues experienced over the last couple of hours. It was due to multiple fiber cuts in Uganda.

Cyberattack

University of Michigan

On August 27, a “significant security concern” led the University of Michigan to shut down the Internet on the Ann Arbor, Flint and Dearborn campuses. Although the shutdown occurred at the start of the new school year, classes continued as scheduled, but an announcement posted by the University detailed the impact of disconnecting from the Internet, including potential delays in financial aid refunds and the unavailability of certain campus systems. The impact of the disconnection can be seen in the figure below, appearing as a significant drop in traffic starting just before 14:00 local time (18:00 UTC) on August 27, and lasting until just after 08:00 local time (12:00 UTC) on August 30 on AS36375 (UMICH-AS-5), the primary autonomous system used by the University of Michigan.

Fire

Lahaina, Hawaii

In early August, a series of wildfires broke out in the state of Hawaii, predominantly on the island of Maui. The town of Lahaina was one of the hardest hit, with the fires killing nearly 100 people, as well as destroying homes, businesses, and infrastructure, causing power outages and disrupting Internet connectivity. The graph below shows traffic to Cloudflare from Lahaina dropping to near zero around 21:00 local time on August 7 (07:00 UTC on August 8), and remaining at minimal levels through August 30. Some recovery of Internet traffic can be seen through the end of September as cleanup and repairs progressed, and as wireless operators deployed temporary network assets in an effort to restore some service capacity.

Earthquake

Morocco

At 23:11 local time on September 8 (22:11 UTC), a magnitude 6.8 earthquake occurred in Morocco, centered 79 kilometers (49 miles) southwest of Marrakesh. Nearly 3,000 deaths were reported as a result of the quake, and significant damage was reported, including the collapse of schools, houses, and historic buildings. Power outages and infrastructure damage also impacted Internet connectivity in the region, leading to largely localized disruptions.

The country-level graph below shows a nominal loss of traffic in Morocco after the earthquake, remaining slightly lower than expected for approximately four days. However, the impacts are more evident at a regional level, with the earthquake causing an immediate 64% drop in traffic in Marrkesh-Safi, a 64% loss in Souss-Massa, and a 49% decline in Casablanca-Settat. Peak traffic levels in these regions remained slightly lower than those seen in previous weeks for several days after the earthquake occurred.

Power outages

Curaçao

On July 27, a malfunction at a major Aqualectra Utility power distribution center resulted in 70% of neighborhoods in Curaçao losing power. The power outage resulted in an island-wide Internet disruption. As seen in the graph below, Internet traffic fell sharply at around 12:30 local time (16:30 UTC), remaining largely flat for approximately five hours before starting to recover around 17:30 local time (21:30 UTC). The start of the recovery aligns with the timing of a Facebook post made at 18:00 local time by Aqualectra Utility noting that “55% of Curaçao’s power supply has been restored.” The ongoing traffic increase is in line with additional neighborhoods having power restored, with traffic returning to expected levels by around 22:00 local time (2:00 UTC on July 28).

Brazil

A widespread power outage in Brazil starting at 08:30 local time (11:30 UTC) on August 15 resulted in a nominal disruption to Internet traffic within the country. Although the power outage represented a loss of approximately 27% of the total electric load at the time it occurred, the impact to the country’s Internet traffic was much lower, as seen in the graph below. Traffic returned to expected levels by around 11:30 local time (14:30 UTC).

Kenya

A “system disturbance” at 21:45 local time (18:45 UTC) on August 25 led to “loss of bulk power supply to various parts of the country” in Kenya, according to an X/Twitter post from Kenya Power. The impact of the power outage is visible in the graph below, with traffic dropping as power is lost. Subsequent updates from Kenya Power on August 26 (1, 2, 3) highlighted the progress made in restoring electricity across the country. Internet traffic from the country returned to expected levels by 03:00 on August 27 (00:00 UTC).

French Guiana

An 11-hour Internet disruption in French Guiana on August 27 was the result of a power outage caused by “a problem that occurred at the energy evacuation station which connects Petit-Saut to the Kourou-Saint-Laurent line”. The power outage caused a nationwide drop in Internet traffic between 11:00 local time (14:00 UTC) and 22:00 local time (01:00 UTC on August 28), visible in the graph below.

Tunisia

A fire at the Tunisian Company of Electricity and Gas power station in Rades, Ben Arous Governorate caused a widespread power outage in Tunisia, resulting in an Internet disruption starting at 01:00 local time (00:00 UTC) on September 20. Traffic remained lower than expected for approximately five hours, as shown in the graph below, in line with a published report that noted “The unexpected outage lasted for over four hours in some areas of the country.

Barbados

A September 21 Facebook post from The Barbados Light & Power Company Limited noted that the company was aware of an outage affecting customers, and that they were “working to promptly and safely restore power in the shortest time possible.” This outage resulted in a significant drop in Internet traffic from the country starting at 11:30 local time (15:30 UTC). A subsequent Facebook post from the utility company at 20:00 local time (00:00 UTC on September 22) noted that power had been restored to all customers. Ahead of full power restoration, Internet traffic had returned to expected levels around 17:00 local time (21:00 UTC).

Maintenance

Guinea

La Guinéenne de la Large Bande, also known as GUILAB, is the company responsible for managing the capacity allocated to the country of Guinea on the Africa Coast to Europe (ACE) submarine cable. According to a (translation of the) communiqué posted by the company on Facebook, planned maintenance on the cable would be taking place between 22:00 on July 14 and 06:18 “sharp” on July 15 (22:00 on July 14 and 06:18 on July 15 UTC). This maintenance resulted in a complete Internet outage in Guinea, as seen in the graph below. It appears that the ACE submarine cable is Guinea’s sole international Internet connection, with no other backup submarine or terrestrial connectivity.

Palau

Just a few days later, planned maintenance to another submarine cable took Palau, an island country in the western Pacific, completely offline for several days. According to a press release from the Palau National Communications Corporation (PNCC) posted to their Facebook page, “BSCC (Belau Submarine Cable Corporation) has been notified that an emergency repair will be undertaken on the SEA – US cable network in Guam from Tuesday, July 18th 7:00 a.m. Palau time, and expected to be completed 5:00 p.m. Saturday, July 22nd. … For safety reasons, repairs can only be undertaken when the cable is not powered. Since BSCC’s Palau Cable Network No 1 connects to SEA – US for onward transport to Guam, BSCC will be unable to provide service for the duration of the repair. BSCC will be unable to provide any international connectivity for Palau. The only available international connection will be via PNCC satellite connection, which will provide limited capacity compared to normal cable service.

The graph below shows that Cloudflare did not see any appreciable traffic from Palau’s backup satellite connection during the duration of the repairs, as traffic dropped to zero at 07:00 local time on July 18 (22:00 UTC on July 17), and remained there until around 18:00 local time on July 21 (09:00 UTC), as the repairs were completed earlier than expected. A PNCC press release confirmed this early completion, noting “PNCC is pleased to inform the public that Internet and Mobile Data services for our customers have been restored, due to the early completion today of the emergency repairs on the SEA-US Submarine Cable System, our main off-island internet connection.

Unspecified issues

Spectrum (Charter Communications)

At 14:03 Eastern Time (18:03 UTC) on August 17, the X/Twitter support account for Spectrum, a brand of US-based Internet service provider Charter Communications, posted a statement that noted “We are aware of an outage affecting customers in Alabama, Georgia and Tennessee. We apologize for the inconvenience and are working to resolve as quickly as possible. Thank you.” The graphs below show the varied impacts to traffic seen from Spectrum (AS20115) across the listed states, as well as Texas, which wasn’t initially cited by Spectrum as having an issue, though customers quickly called it out.

A near complete outage was observed in Tennessee between 12:30 – 14:00 local time (17:30 – 19:00 UTC), while a brief drop in traffic at 12:00 local time (17:00 UTC) and quick recovery ahead of another drop at 13:30 local time (18:30 UTC) was seen in Alabama. Georgia also saw an initial drop in traffic at 13:00 local time (17:00 UTC) ahead of a larger fall at 14:30 local time (18:30 UTC), while traffic from Texas only experienced a decline at 13:30 local time (18:30 UTC). Traffic volumes from all four impacted states recovered within several hours — approximately three hours after the initial post, Spectrum’s support account statedWe have received confirmation repairs have been completed and services have been restored to affected customers in the Alabama, Georgia and Tennessee area.

On September 12, satellite Internet service provider SpaceX Starlink experienced a brief but complete outage. The graph below shows traffic from AS14593 (SPACEX-STARLINK) dropping at 23:15 UTC, but quickly recovering, returning to normal within 90 minutes. At 00:33 UTC on September 13, Starlink shared an X/Twitter post stating “Starlink is currently in a network outage, and we are actively implementing a solution. We appreciate your patience, we’ll share an update once this issue is resolved” and just over an hour later, posted “The network issue has been fully resolved”.

Sky UK

During the evening (UTC) of September 19, numerous complaints could be found on social media about a nationwide outage across the United Kingdom on Sky Broadband (AS5607). A sharp drop in traffic from Sky Broadband can be seen in the graph below starting at 21:00 UTC, but a full outage did not appear to have taken place. Traffic volumes below expected levels lasted until approximately 01:00 UTC on September 20. While the issue was acknowledged by Sky’s support account on X/Twitter, no root cause for the disruption was ever provided.

Conclusion

As we’ve noted in past quarterly summaries, this report is intended as a summary overview of observed disruptions, and not an exhaustive or complete list of issues that have occurred during the quarter. Some disruptions not covered here were visible in our data, but never acknowledged by the impacted provider, while others were reported by industry colleagues based on their measurement methodologies, but not clearly obvious in our traffic graphs.

As we indicated above, the Cloudflare Radar Outage Center now includes information on observed traffic anomalies as well as verified outages. Interested users can subscribe to notifications for both anomalies and outages — our blog post includes more information on how to do so.

Visit Cloudflare Radar for additional insights around Internet disruptions. Follow us on social media at @CloudflareRadar (Twitter), cloudflare.social/@radar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

Traffic anomalies and notifications with Cloudflare Radar

Post Syndicated from David Belson original http://blog.cloudflare.com/traffic-anomalies-notifications-radar/

Traffic anomalies and notifications with Cloudflare Radar

Traffic anomalies and notifications with Cloudflare Radar

We launched the Cloudflare Radar Outage Center (CROC) during Birthday Week 2022 as a way of keeping the community up to date on Internet disruptions, including outages and shutdowns, visible in Cloudflare’s traffic data. While some of the entries have their genesis in information from social media posts made by local telecommunications providers or civil society organizations, others are based on an internal traffic anomaly detection and alerting tool. Today, we’re adding this alerting feed to Cloudflare Radar, showing country and network-level traffic anomalies on the CROC as they are detected, as well as making the feed available via API.

Building on this new functionality, as well as the route leaks and route hijacks insights that we recently launched on Cloudflare Radar, we are also launching new Radar notification functionality, enabling you to subscribe to notifications about traffic anomalies, confirmed Internet outages, route leaks, or route hijacks. Using the Cloudflare dashboard’s existing notification functionality, users can set up notifications for one or more countries or autonomous systems, and receive notifications when a relevant event occurs. Notifications may be sent via e-mail or webhooks — the available delivery methods vary according to plan level.

Traffic anomalies

Internet traffic generally follows a fairly regular pattern, with daily peaks and troughs at roughly the same volumes of traffic. However, while weekend traffic patterns may look similar to weekday ones, their traffic volumes are generally different. Similarly, holidays or national events can also cause traffic patterns and volumes to differ significantly from “normal”, as people shift their activities and spend more time offline, or as people turn to online sources for information about, or coverage of, the event. These traffic shifts can be newsworthy, and we have covered some of them in past Cloudflare blog posts (King Charles III coronation, Easter/Passover/Ramadan, Brazilian presidential elections).

However, as you also know from reading our blog posts and following Cloudflare Radar on social media, it is the more drastic drops in traffic that are a cause for concern. Some are the result of infrastructure damage from severe weather or a natural disaster like an earthquake and are effectively unavoidable, but getting timely insights into the impact of these events on Internet connectivity is valuable from a communications perspective. Other traffic drops have occurred when an authoritarian government orders mobile Internet connectivity to be shut down, or shuts down all Internet connectivity nationwide. Timely insights into these types of anomalous traffic drops are often critical from a human rights perspective, as Internet shutdowns are often used as a means of controlling communication with the outside world.

Over the last several months, the Cloudflare Radar team has been using an internal tool to identify traffic anomalies and post alerts for followup to a dedicated chat space. The companion blog post Gone Offline: Detecting Internet Outages goes into deeper technical detail about our traffic analysis and anomaly detection methodologies that power this internal tool.

Many of these internal traffic anomaly alerts ultimately result in Outage Center entries and Cloudflare Radar social media posts. Today, we’re extending the Cloudflare Radar Outage Center and publishing information about these anomalies as we identify them. As shown in the figure below, the new Traffic anomalies table includes the type of anomaly (location or ASN), the entity where the anomaly was detected (country/region name or autonomous system), the start time, duration, verification status, and an “Actions” link, where the user can view the anomaly on the relevant entity traffic page or subscribe to a notification. (If manual review of a detected anomaly finds that it is present in multiple Cloudflare traffic datasets and/or is visible in third-party datasets, such as Georgia Tech’s IODA platform, we will mark it as verified. Unverified anomalies may be false positives, or related to Netflows collection issues, though we endeavor to minimize both.)

Traffic anomalies and notifications with Cloudflare Radar

In addition to this new table, we have updated the Cloudflare Radar Outage Center map to highlight where we have detected anomalies, as well as placing them into a broader temporal context in a new timeline immediately below the map. Anomalies are represented as orange circles on the map, and can be hidden with the toggle in the upper right corner. Double-bordered circles represent an aggregation across multiple countries, and zooming in to that area will ultimately show the number of anomalies associated with each country that was included in the aggregation. Hovering over a specific dot in the timeline displays information about the outage or anomaly with which it is associated.

Traffic anomalies and notifications with Cloudflare Radar

Internet outage information has been available via the Radar API since we launched the Outage Center and API in September 2022, and traffic anomalies are now available through a Radar API endpoint as well. An example traffic anomaly API request and response are shown below.

Example request:

curl --request GET \ --url https://api.cloudflare.com/client/v4/radar/traffic_anomalies \ --header 'Content-Type: application/json' \ --header 'X-Auth-Email: '

Example response:

{
  "result": {
    "trafficAnomalies": [
      {
        "asnDetails": {
          "asn": "189",
          "locations": {
            "code": "US",
            "name": "United States"
          },
          "name": "LUMEN-LEGACY-L3-PARTITION"
        },
        "endDate": "2023-08-03T23:15:00Z",
        "locationDetails": {
          "code": "US",
          "name": "United States"
        },
        "startDate": "2023-08-02T23:15:00Z",
        "status": "UNVERIFIED",
        "type": "LOCATION",
        "uuid": "55a57f33-8bc0-4984-b4df-fdaff72df39d",
        "visibleInDataSources": [
          "string"
        ]
      }
    ]
  },
  "success": true
}

Notifications overview

Timely knowledge about Internet “events”, such as drops in traffic or routing issues, are potentially of interest to multiple audiences. Customer service or help desk agents can use the information to help diagnose customer/user complaints about application performance or availability. Similarly, network administrators can use the information to better understand the state of the Internet outside their network. And civil society organizations can use the information to inform action plans aimed at maintaining communications and protecting human rights in areas of conflict or instability. With the new notifications functionality also being launched today, you can subscribe to be notified about observed traffic anomalies, confirmed Internet outages, route leaks, or route hijacks, at a country or autonomous system level. In the following sections, we discuss how to subscribe to and configure notifications, as well as the information contained within the various types of notifications.

Subscribing to notifications

Note that you need to log in to the Cloudflare dashboard to subscribe to and configure notifications. No purchase of Cloudflare services is necessary — just a verified email address is required to set up an account. While we would have preferred to not require a login, it enables us to take advantage of Cloudflare’s existing notifications engine, allowing us to avoid having to dedicate time and resources to building a separate one just for Radar. If you don’t already have a Cloudflare account, visit https://dash.cloudflare.com/sign-up to create one. Enter your username and a unique strong password, click “Sign Up”, and follow the instructions in the verification email to activate your account. (Once you’ve activated your account, we also suggest activating two-factor authentication (2FA) as an additional security measure.)

Once you have set up and activated your account, you are ready to start creating and configuring notifications. The first step is to look for the Notifications (bullhorn) icon – the presence of this icon means that notifications are available for that metric — in the Traffic, Routing, and Outage Center sections on Cloudflare Radar. If you are on a country or ASN-scoped traffic or routing page, the notification subscription will be scoped to that entity.

Traffic anomalies and notifications with Cloudflare Radar
Look for this icon in the Traffic, Routing, and Outage Center sections of Cloudflare Radar to start setting up notifications.
Traffic anomalies and notifications with Cloudflare Radar
In the Outage Center, click the icon in the “Actions” column of an Internet outages table entry to subscribe to notifications for the related location and/or ASN(s). Click the icon alongside the table description to subscribe to notifications for all confirmed Internet outages.
Traffic anomalies and notifications with Cloudflare Radar
In the Outage Center, click the icon in the “Actions” column of a Traffic anomalies table entry to subscribe to notifications for the related entity. Click the icon alongside the table description to subscribe to notifications for all traffic anomalies.
Traffic anomalies and notifications with Cloudflare Radar
On country or ASN traffic pages, click the icon alongside the description of the traffic trends graph to subscribe to notifications for traffic anomalies or Internet outages impacting the selected country or ASN.
Traffic anomalies and notifications with Cloudflare Radar
Traffic anomalies and notifications with Cloudflare Radar
On country or ASN routing pages, click the icon alongside the description to subscribe to notifications for route leaks or origin hijacks related to the selected country or ASN.
Traffic anomalies and notifications with Cloudflare Radar
Traffic anomalies and notifications with Cloudflare Radar
Within the Route Leaks or Origin Hijacks tables on the routing pages, click the icon in a table entry to subscribe to notifications for route leaks or origin hijacks for referenced countries and/or ASNs. 

After clicking a notification icon, you’ll be taken to the Cloudflare login screen. Enter your username and password (and 2FA code if required), and once logged in, you’ll see the Add Notification page, pre-filled with the key information passed through from the referring page on Radar, including relevant locations and/or ASNs. (If you are already logged in to Cloudflare, then you’ll be taken directly to the Add Notification page after clicking a notification icon on Radar.) On this page, you can name the notification, add an optional description, and adjust the location and ASN filters as necessary. Enter an email address for notifications to be sent to, or select an established webhook destination (if you have webhooks enabled on your account).

Traffic anomalies and notifications with Cloudflare Radar

Click “Save”, and the notification is added to the Notifications Overview page for the account.

Traffic anomalies and notifications with Cloudflare Radar

You can also create and configure notifications directly within Cloudflare, without starting from a link on Radar a Radar page. To do so, log in to Cloudflare, and choose “Notifications” from the left side navigation bar. That will take you to the Notifications page shown below. Click the “Add” button to add a new notification.

Traffic anomalies and notifications with Cloudflare Radar

On the next page, search for and select “Radar” from the list of Cloudflare products for which notifications are available.

Traffic anomalies and notifications with Cloudflare Radar

On the subsequent “Add Notification” page, you can create and configure a notification from scratch. Event types can be selected in the “Notify me for:” field, and both locations and ASNs can be searched for and selected within the respective “Filtered by (optional)” fields. Note that if no filters are selected, then notifications will be sent for all events of the selected type(s). Add one or more emails to send notifications to, or select a webhook target if available, and click “Save” to add it to the list of notifications configured for your account.

Traffic anomalies and notifications with Cloudflare Radar

It is worth mentioning that advanced users can also create and configure notifications through the Cloudflare API Notification policies endpoint, but we will not review that process within this blog post.

Notification messages

Example notification email messages are shown below for the various types of events. Each contains key information like the type of event, affected entities, and start time — additional relevant information is included depending on the event type. Each email includes both plaintext and HTML versions to accommodate multiple types of email clients. (Final production emails may vary slightly from those shown below.)

Traffic anomalies and notifications with Cloudflare Radar
Internet outage notification emails include information about the affected entities, a description of the cause of the outage, start time, scope (if available), and the type of outage (Nationwide, Network, Regional, or Platform), as well as a link to view the outage in a Radar traffic graph.
Traffic anomalies and notifications with Cloudflare Radar
Traffic anomaly notification emails simply include information about the affected entity and a start time, as well as a link to view the anomaly in a Radar traffic graph.
Traffic anomalies and notifications with Cloudflare Radar
BGP hijack notification emails include information about the hijacking and victim ASNs, affected IP address prefixes, the number of BGP messages (announcements) containing leaked routes, the number of peers announcing the hijack, detection timing, a confidence level on the event being a true hijack, and relevant tags, as well as a link to view details of the hijack event on Radar.
Traffic anomalies and notifications with Cloudflare Radar
BGP route leak notification emails include information about the AS that the leaked routes were learned from, the AS that leaked the routes, the AS that received and propagated the leaked routes, the number of affected prefixes, the number of affected origin ASes, the number of BGP route collector peers that saw the route leak, and detection timing, as well as a link to view details of the route leak event on Radar.

If you are sending notifications to webhooks, you can integrate those notifications into tools like Slack. For example, by following the directions in Slack’s API documentation, creating a simple integration took just a few minutes and results in messages like the one shown below.

Traffic anomalies and notifications with Cloudflare Radar

Conclusion

Cloudflare’s unique perspective on the Internet provides us with near-real-time insight into unexpected drops in traffic, as well as potentially problematic routing events. While we’ve been sharing these insights with you over the past year, you had to visit Cloudflare Radar to figure out if there were any new “events”. With the launch of notifications, we’ll now automatically send you information about the latest events that you are interested in.

We encourage you to visit Cloudflare Radar to familiarize yourself with the information we publish about traffic anomalies, confirmed Internet outages, BGP route leaks, and BGP origin hijacks. Look for the notification icon on the relevant graphs and tables on Radar, and go through the workflow to set up and subscribe to notifications. (And don’t forget to sign up for a Cloudflare account if you don’t have one already.) Please send us feedback about the notifications, as we are constantly working to improve them, and let us know how and where you’ve integrated Radar notifications into your own tools/workflows/organization.

Follow Cloudflare Radar on social media at @CloudflareRadar (Twitter), cloudflare.social/@radar (Mastodon), and radar.cloudflare.com (Bluesky).

Traffic anomalies and notifications with Cloudflare Radar

Q2 2023 Internet disruption summary

Post Syndicated from David Belson original http://blog.cloudflare.com/q2-2023-internet-disruption-summary/

Q2 2023 Internet disruption summary

This post is also available in Deutsch, Français, 日本語, 简体中文, 繁體中文 and 한국어.

Q2 2023 Internet disruption summary

Cloudflare operates in more than 300 cities in over 100 countries, where we interconnect with over 12,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions.

The second quarter of 2023 was a particularly busy one for Internet disruptions, and especially for government-directed Internet shutdowns. During the quarter, we observed many brief disruptions, but also quite a few long-lived ones. In addition to the government-directed Internet shutdowns, we also observed partial or complete outages due to severe weather, cable damage, power outages, general or unspecified technical problems, cyberattacks, military action, and infrastructure maintenance.

As we have noted in the past, this post is intended as a summary overview of observed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter.

Government directed

Late spring often marks the start of a so-called “exam season” in several Middle Eastern and African countries, where students sit for a series of secondary school exams. In an attempt to prevent cheating on these exams, governments in the countries have taken to implementing wide-scale Internet shutdowns covering time periods just before and during the exams. We have covered these shutdowns in the past, including Sudan and Syria in 2021 and Syria, Sudan, and Algeria in 2022. This year, we saw governments in Iraq, Algeria, and Syria taking such actions.

Iraq

In the weeks prior to the start of this year’s shutdowns, it was reported that the Iraqi Ministry of Communications had announced it had refused a request from the Ministry of Education to impose an Internet shutdown during the exams as part of efforts to prevent cheating. Unfortunately, this refusal was short-lived, with shutdowns ultimately starting two weeks later.

In Iraq, two sets of shutdowns were observed: one impacted networks nationwide, except for the Kurdistan Region, while the other impacted networks within the Kurdistan Region. The former set of shutdowns were related to 9th and 12th grade exams, and were scheduled to occur from June 1 through July 15, between 04:00 and 08:00 local time (01:00 – 05:00 UTC). The graphs below show that during June, shutdowns took place on June 1, 4, 6, 8, 11, 13, 15, 17, 21, 22, 24, 25, and 26, resulting in significant disruptions to Internet connectivity. The shutdowns were implemented across a number of network providers, including AS203214 (HulumTele), AS59588 (Zain), AS199739 (Earthlink), AS203735 (Net Tech), AS51684 (Asiacell), and AS58322 (Halasat). The orange-highlighted areas in the graphs below show traffic on each network provider dropping to zero during the shutdowns.

As noted above, exam-related Internet shutdowns were also implemented in the Kurdistan region of Iraq. One report quoted the Minister of Education of the Kurdistan Regional Government as stating "The Internet will be turned off as needed during exams, but just like in previous years, the period of the Internet shutdown will not be lengthy, but rather short.” To that end, the observed shutdowns generally lasted about two hours, occurring between 06:30 and 08:30 local time (03:30 – 05:30 UTC) on June 3, 6, 10, 13, 17, and 24. The graphs below show the impact across three network providers in the region: AS21277 (Newroz Telecom), AS48492 (IQ Online), and AS59625 (KorekTel).

Additional details about both sets of Internet shutdowns in Iraq can be found in our June 13 blog post: Exam-related Internet shutdowns in Iraq and Algeria put connectivity to the test.

Algeria

2023 marks the sixth year that Algeria has disrupted Internet connectivity to prevent cheating during nationwide exams. In 2022, we noted that “it appears that the Algerian government has shifted to a content blocking-based approach, instead of a wide-scale Internet shutdown.” It appears that the same approach was taken this year, as we again observed two nominal drops in traffic during each of the exam days, rather than a complete loss of traffic. These traffic shifts were observed on mobile network providers AS33779 (Ooredoo/Wataniya), AS327931 (Djezzy/Optimum), and AS327712 (Mobilis/Telecom Algeria). The first disruption takes place between 08:00 – 12:00 local time (07:00 – 11:00 UTC), with the second occurring between 14:00 – 17:00 local time (13:00 – 16:00 UTC).

Syria

After implementing four exam-related Internet shutdowns in 2022, this year saw just two. On June 25 and 26, Internet shutdowns took place between 05:00 – 08:30 local time (02:00 – 05:30 UTC). Syrian Telecom (AS29256), the government-affiliated telecommunications company, informed subscribers in a Facebook post that the Internet would be cut off at the request of the Ministry of Education.

Senegal

In Senegal, violent protests over the sentencing of opposition leader Ousmane Sonko to jail led the government to restrict access to platforms including WhatsApp, Facebook, Twitter, Instagram, TikTok, Telegram, Signal, and YouTube. On June 4, the Senegal Ministry of Communication issued a statement temporarily suspending mobile Internet access, with a followup statement on June 6 ending the suspension. These disruptions to mobile Internet access were visible on two network providers within the country: AS37196 (Sudatel Senegal) and AS37649 (Tigo/Free).

As shown in the graphs below, the shutdowns on Sudatel Senegal occurred from 15:00 local time on June 3 through 01:00 local time on June 5, and then again from 13:00 local time on June 5 until 01:00 local time on June 6. The three shutdowns seen on Tigo/Free took place between 15:30 – 19:00 local time on June 3, from 13:45 local time on June 4 until 02:05 local time on June 5, and from 13:05 local time on June 5 through 01:00 local time on June 6. (Senegal is UTC+0, so the local times are the same as UTC.)

Mauritania

In Mauritania, authorities cut off mobile Internet services after protests over the death of a young man in police custody. The shutdown began at 23:00 local time on May 30, and lasted six days, with connectivity returning at 23:00 local time on June 6. (Mauritania is UTC+0, so the local times are the same as UTC.) The graphs below show a near complete loss of Internet traffic during that period from AS37541 (Chinguitel) and AS37508 (Mattel), two mobile network providers within the country.

Pakistan

On May 9, Imran Khan, former Prime Minister of Pakistan was arrested on corruption charges. Following the arrest, violent protests erupted in several cities, leading the government of Pakistan to order the shutdown of mobile Internet services, as well as the blocking of several social media platforms. The figures below show the impact of the ordered shutdown to traffic on four mobile network providers within the country: AS24499 (Telenor Pakistan), AS59257 (China Mobile Pak), AS45669 (Mobilink/Jazz), and AS56167 (Ufone/PTML). The ordered shutdown caused a complete loss of Internet traffic from these networks that started at 22:00 local time (17:00 UTC) on May 9 at Telenor and China Mobile Pakistan, 18:00 local time (13:00 UTC) on Mobilink/Jazz, and 01:00 local time on May 10 (20:00 UTC on May 9) at Ufone/PTML. Traffic was restored at 22:00 local time (17:00 UTC) on May 12.

Looking at Cloudflare Radar’s recently launched Internet Quality page for Pakistan during the duration of the shutdown, we observed that median latency within Pakistan dropped slightly after mobile networks were shut down, shown in the graph below. Prior to the shutdown, median latency (as observed to Cloudflare and a set of other providers) was in the 90-100ms range, while afterward, it averaged closer to 75ms. This may be a result of users shifting to lower latency fixed broadband connections – several fixed broadband providers in the country experienced increased traffic volumes while the mobile networks were unavailable.

Additional details about the mobile network shutdowns, content blocking, and the impact at an administrative unit and city level can be found in our May 12 blog post Cloudflare’s view of Internet disruptions in Pakistan.

India

Internet shutdowns are unfortunately frequent in India, with digital rights organization Access Now reporting at least 84 shutdowns within the country in 2022. The shutdowns are generally implemented at a more local level, and often last for a significant amount of time. One such shutdown took place in the northeastern Indian state of Manipur starting on May 3 after the escalation of ethnic conflict, and was reportedly intended to “thwart the design and activities of anti-national and anti-social elements… by stopping the spread of disinformation and false rumours'' and the likelihood of “serious disturbances to the entire peaceful coexistence of the communities and maintenance of public order”. Mobile data services were initially suspended for a five-day period, with the suspension continually extended through additional templated orders issued every five days.

The graphs below show the impact of the ordered shutdown to traffic from two major network providers in Manipur. Traffic from both AS45609 (Airtel) and AS9829 (BSNL) fell significantly around 18:00 local time (12:30 UTC) on May 4. Traffic on Airtel has remained low, and continued to drop further through the end of June. Traffic on BSNL showed slight signs of recovery starting in early June, but remains extremely low.

The shutdown order remains in place as of the time of this writing (late July).

Q2 2023 Internet disruption summary
Q2 2023 Internet disruption summary

Severe weather

Guam

On May 24, “Super Typhoon” Mawar wreaked havoc on the US territory of Guam, causing widespread physical damage after it made landfall, taking down trees, buildings, power lines, and communications infrastructure across the island. One result of this damage was a significant disruption to Internet connectivity, as shown in the country-level graph below. Restoration efforts started almost immediately, with Guam Power Authority, Docomo Pacific, and GTA Teleguam all posting regular status updates on their websites and/or social media accounts.

Among the two Internet providers, GTA Teleguam (AS9246) was largely able to complete service restoration in June, with traffic returning to pre-storm levels around June 17, as seen in the graph below. In fact, in a June 20 Facebook post they noted that “As of today, a majority of our wireless network cell sites are operational.” However, recovery at Docomo Pacific (AS3605) is taking significantly longer. The graph below shows that as of the end of June, traffic remained significantly below pre-storm levels.

Cable damage

Bolivia

On June 19, COTAS, a Bolivian telecommunications company, posted an update on their Facebook page that alerted users that a fiber optic cable had been cut in the town of Pongo. As seen in the graphs below, this cut significantly disrupted Internet connectivity across COTAS and two other network providers in the country: AS25620 (COTAS), AS27839 (Comteco), and AS52495 (Cotel) between 13:00 – 18:00 local time (17:00 –  22:00 UTC).

The Gambia

Gamtel, the state telecommunications company in The Gambia, notified subscribers via a Twitter post on June 7 of a localized fiber cut, and then of additional cable cuts on June 8. These fiber cuts disrupted Internet connectivity on AS25250 (Gamtel) between 14:00 local time on June 7 and 00:00 local time on June 9, with traffic volumes down as much as 80% as compared to the previous period. (The Gambia is UTC+0, so the local times are the same as UTC.)

Philippines

An advisory posted on Twitter by Philippines telecommunications provider PLDT at 18:43 local time (10:43 UTC) on June 5 stated “One of our submarine cable partners confirms a loss in some of its internet bandwidth capacity, and thus causing slower Internet browsing. We are working with our partners to provide alternate capacity that would restore the browsing experience in the next few hours.” The traffic graph below shows a minor disruption to Internet traffic for AS9299 (PLDT) starting around 14:00 local time (06:00 UTC), and the “slower Internet browsing” noted by PLDT is evident in the Internet quality graphs below, with increased latency and decreased bandwidth evident around that same time. PLDT stated in a subsequent tweet that as of 06:22 local time on June 6 (22:22 UTC on June 5), “Our submarine cable partner confirms supplementing additional capacity, restoring browser experience.

Power outages

Curaçao

Aqualectra is the primary utility company in Curaçao, providing water and power services. On June 8, they posted a series of alerts to their Facebook page (1, 2, 3, 4) regarding a power outage impacting “all neighborhoods”, caused by a malfunction in one of the main power cables connected to the substation at Parera. This loss of power impacted Internet connectivity on the island, with a significant loss of traffic observed at a country level, as seen in the graph below, as well as across several Internet service providers, including AS11081 (UTS), AS52233 (Columbus Communications), and AS27660 (Curaçao Telecom). A followup Facebook post dated 01:25 local time on June 9 (05:25 UTC) confirmed the restoration of power to all neighborhoods.

Portugal

A power outage at an Equinix data center in Prior Velho (near Lisbon) on the afternoon of June 6 affected local utilities, banking services, and court networks, according to published reports (1, 2). Portuguese Internet service provider MEO was also impacted by the power outage, which caused a drop in traffic for AS3243 (MEO-RESIDENCIAL) and AS15525 (MEO-EMPRESAS), seen in the graphs below. The disruptions caused by the power outage also impacted connectivity quality within Portugal, as the Radar Internet quality graphs below highlight – a concurrent drop in bandwidth and increase in latency is visible, indicating that end users likely experienced poorer performance during that period of time.

Q2 2023 Internet disruption summary
Q2 2023 Internet disruption summary

Botswana

A countrywide power outage in Botswana on May 19 caused an Internet disruption that lasted about 90 minutes, from 10:45 until 12:15 local time (08:45 – 10:15 UTC), visible in the graph below. A tweet from Botswana Power Corporation provided public notice of the incident, but did not include a root cause.

Barbados

On April 4, The Barbados Light & Power Company tweeted an “Outage Notice”, stating “We are aware that our customers across the island are currently without electricity.” Posted at 11:46 local time (15:46 UTC), the notice comes shortly after a significant drop in traffic was observed country-wide, indicating that the power outage also impacted Internet connectivity across the country. After posting several additional updates throughout the day, a final update posted at 18:00 local time (22:00 UTC) indicated that power had been restored to 100% of impacted customers. The graph below shows that traffic took several additional hours to return to normal levels. (Note that the orange highlighting in the graph represents the duration of the disruption, and the red shading is related to an internal data collection issue.)

Technical problems

Namibia

A seven-hour Internet disruption observed in Namibia on June 15 and 16 was caused by unspecified “technical challenges” faced by Telecom Namibia. According to a tweet from the provider, “Telecom Namibia experienced technical challenges on its fixed and mobile data services on Thursday leading to intermittent Internet connectivity.” The impact of these challenges is visible in both the country- and network-level traffic graphs below.

Solomon Islands

Unspecified “technical reasons” also disrupted mobile Internet connectivity for Our Telekom customers in the Solomon Islands on April 26 and 27. An April 26 Facebook post from Our Telekom simply stated “Our mobile data network is currently down due to technical reasons.” The graphs below show a significant drop in traffic for AS45891 (Our Telekom/SBT) between 06:30 local time on April 27 (19:30 UTC on April 26) and 17:00 local time on April 27 (06:00 UTC). The loss of mobile traffic from Our Telekom also impacted traffic at a country level, as the graph shows a similar disruption for the Solomon Islands.

With an increasingly global service footprint, disruptions observed on SpaceX Starlink potentially impact users across multiple countries around the world. Just before midnight UTC on April 7, Internet traffic seen from AS14593 (SpaceX-Starlink) began to decline significantly. The disruption was short-lived, with traffic returning to expected levels within two hours. According to a Twitter post from Elon Musk, CEO of SpaceX, the problem was “Caused by expired ground station cert” (an expired digital certificate associated with one or more Starlink ground stations, likely preventing communication between the satellite constellation and the ground station(s)).

Madagascar

In Madagascar, a “problem with the backbone”, reported by Telma Madagascar, caused a loss of as much as two-thirds of Internet traffic between 09:15 – 14:00 local time (06:15 – 11:00 UTC) on April 7. The graphs below show that the backbone issue disrupted traffic at a national level, as well as for AS37054 (Telma Madagascar).

United Kingdom

On April 4, UK Internet provider Virgin Media suffered multiple service disruptions that impacted Internet connectivity for broadband customers. The first outage started just before 01:00 local time (midnight UTC)l, lasting until approximately 09:00 local time (08:00 UTC). The second outage started around 16:00 local time (15:00 UTC), with traffic volumes going up and down over the next several hours before appearing to stabilize around 21:30 local time (20:30 UTC).

Virgin Media’s Twitter account acknowledged the early morning disruption several hours after it began, postingWe’re aware of an issue that is affecting broadband services for Virgin Media customers as well as our contact centres. Our teams are currently working to identify and fix the problem as quickly as possible and we apologise to those customers affected.A subsequent post after service restoration noted “We’ve restored broadband services for customers but are closely monitoring the situation as our engineers continue to investigate. We apologise for any inconvenience caused.

However, the second disruption was acknowledged on Virgin Media’s Twitter account much more rapidly, with a post at 16:25 UTC stating “Unfortunately we have seen a repeat of an earlier issue which is causing intermittent broadband connectivity problems for some Virgin Media customers. We apologise again to those impacted, our teams are continuing to work flat out to find the root cause of the problem and fix it.

Although no additional details have been shared via social media by Virgin Media about the outages or their resolution, some additional information was shared via Twitter by an apparent customer, who posted “Virgin Media engineers re-seated fibre cards and reset hub equipment to restore service. TTL was extended as a workaround to maintain stability whilst a permanent fix is implemented.

Additional details about the Virgin Media outage can be found in our April 4 blog post: Cloudflare’s view of the Virgin Media outage in the UK.

Cyberattacks

Ukraine

As we have covered in past blog posts, the physical war between Russia and Ukraine also has a very active online component, with traffic shifts, cyberattacks, and traffic rerouting all observed since the conflict began in February 2022. In early May 2022, we observed traffic from several Ukrainian network providers being rerouted through AS201776 (Miranda Media), a Crimean-based, Russian-controlled network operator. (This rerouting is discussed in more detail in two blog posts: Tracking shifts in Internet connectivity in Kherson, Ukraine and One year of war in Ukraine: Internet trends, attacks, and resilience.)

A little more than a year later, on May 26, we observed an Internet outage at Miranda Media. Traffic started to fall around 16:30 local time (13:30 UTC), dropping to zero around 18:15 local time (15:15 UTC). The outage disrupted connectivity on the Crimean Peninsula and parts of occupied Ukraine and lasted until approximately 06:00 local time on May 27 (03:00 UTC). Published reports (1,2) suggest that the outage was due to a cyberattack targeting Miranda Media, reportedly carried out by Ukrainian hacktivists.

Russia

Russian satellite provider Dozor Teleport, whose customers include Russia’s Ministry of Defense, ships of the Northern Fleet, Russian energy firm Gazprom, remote oil fields, the Bilibino nuclear power plant, the Federal Security Service (FSB), Rosatom, and other organizations, experienced a multi-hour outage on June 29. The outage, which occurred between 01:30 – 17:30 UTC, was reportedly the result of a cyberattack that at least two groups claimed responsibility for.

Military action

Chad

Multiple Internet disruptions occurred in Chad on April 23 and 24, impacting several Internet providers, and were ultimately visible at a country level as well. As seen in the graphs below, the outages occurred from 04:30 – 06:00 local time (03:30 – 05:00 UTC) and 15:00 – 20:00 local time (14:00 – 19:00 UTC) on April 23, and 04:00 – 08:30 local time (03:00 – 07:30 UTC) on April 24. The impacted network providers in Chad included AS327802 (Millicom Chad), AS327756 (Airtel Chad), AS328594 (Sudat Chad), and AS327975 (ILNET-TELECOM). The outages were reportedly caused by damage to fiber infrastructure that links Chad with neighboring Cameroon and Sudan, with the latter experiencing Internet service disruptions amid clashes between the Sudanese Armed Forces (SAF) and Rapid Support Forces (RSF).

Sudan

As noted above, military action in Sudan disrupted Internet connectivity in Chad in April. Starting in mid-April, multiple Internet outages were observed at major Sudanese Internet providers, three of which are shown in the graphs below. The fighting in the country has led to fuel shortages and power cuts, ultimately disrupting Internet connectivity.

AS15706 (Sudatel) experienced complete Internet outages from 03:00 on April 23 to 17:00 on April 24 local time (01:00 on April 23 to 15:00 on April 24 UTC) and again from 03:00 on April 25 until 01:00 on April 28 local time (01:00 on April 25 to 23:00 on April 27 UTC). Internet connectivity on AS36972 (MTN) was disrupted between 03:00 and 12:00 local time on April 16 (01:00 – 10:00 UTC) and again between 20:00 on April 27 until 02:00 on April 29 (18:00 on April 27 to 00:00 on April 29). After a nominal multi-day recovery, a long-term near complete outage started on May 5, lasting for multiple weeks. Similar to MTN, multiple extended outages were also observed on AS33788 (Kanar Telecommunication). After seeing a significant drop in traffic midday on April 19, a near complete outage is visible between 12:00 on April 21 and 01:00 on April 29 (10:00 on April 21 to 23:00 on April 28 UTC), with a very brief minor recovery late in the day on April 24. A longer duration outage began around 00:00 local time on May 11 (22:00 on May 10 UTC), also lasting for multiple weeks.

Additional details about the Internet disruptions in Sudan can be found in our May 2 blog post: Effects of the conflict in Sudan on Internet patterns.

Maintenance

Togo, Republic of Congo (Brazzaville), Burkina Faso

Repair work on the West Africa Cable System (WACS) submarine cable disrupted Internet connectivity across multiple countries, including Togo, Republic of Congo (Brazzaville), and Burkina Faso on April 6. According to the Google translation of a Facebook post from Canalbox Congo, the repair work was likely to cause “very strong disruptions on Internet connections with the risk of a total outage”. (Canalbox (GVA) is an African telecommunications operator that provides services across multiple countries in Africa.)

The graph below for AS36924 (GVA-Canalbox) shows three overlapping outage annotations, with each related to a disruption observed on that autonomous system (network) in one of the impacted countries. In the Republic of Congo (Brazzaville), a significant traffic disruption is visible between 16:15 – 23:15 local time (15:15 – 22:15 UTC). In Burkina Faso, the disruption happened earlier and was less severe, taking place between 09:15 – 18:00 local time (09:15 – 18:00 UTC), with a similar impact in Togo, where traffic was disrupted between 11:00 – 23:15 local time (11:00 – 23:15 UTC).

Conclusion

Because of how tightly interwoven the Internet has become with commerce, financial services, and everyday life around the world, any disruption to Internet connectivity ultimately carries an economic impact. The providers impacted by disruptions caused by unexpected or unavoidable events such as cable cuts or severe weather generally try to minimize the scope and duration of such disruptions, ultimately limiting the economic impact. However, in the case of government-directed Internet shutdowns, the damage to the economy is ultimately self-inflicted. The Internet Society’s new Net Loss Calculator now provides a way to quantify this damage, enabling the public, advocacy groups, and governments themselves to understand the potential cost of an Internet shutdown from gross domestic product (GDP), foreign direct investment (FDI), and unemployment perspectives.

Visit Cloudflare Radar for additional insights around Internet disruptions. Follow us on social media at @CloudflareRadar (Twitter), cloudflare.social/@radar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

Introducing the Cloudflare Radar Internet Quality Page

Post Syndicated from David Belson original http://blog.cloudflare.com/introducing-radar-internet-quality-page/

Introducing the Cloudflare Radar Internet Quality Page

Introducing the Cloudflare Radar Internet Quality Page

Internet connections are most often marketed and sold on the basis of "speed", with providers touting the number of megabits or gigabits per second that their various service tiers are supposed to provide. This marketing has largely been successful, as most subscribers believe that "more is better”. Furthermore, many national broadband plans in countries around the world include specific target connection speeds. However, even with a high speed connection, gamers may encounter sluggish performance, while video conference participants may experience frozen video or audio dropouts. Speeds alone don't tell the whole story when it comes to Internet connection quality.

Additional factors like latency, jitter, and packet loss can significantly impact end user experience, potentially leading to situations where higher speed connections actually deliver a worse user experience than lower speed connections. Connection performance and quality can also vary based on usage – measured average speed will differ from peak available capacity, and latency varies under loaded and idle conditions.

The new Cloudflare Radar Internet Quality page

A little more than three years ago, as residential Internet connections were strained because of the shift towards working and learning from home due to the COVID-19 pandemic, Cloudflare announced the speed.cloudflare.com speed test tool, which enabled users to test the performance and quality of their Internet connection. Within the tool, users can download the results of their individual test as a CSV, or share the results on social media. However, there was no aggregated insight into Cloudflare speed test results at a network or country level to provide a perspective on connectivity characteristics across a larger population.

Today, we are launching these long-missing aggregated connection performance and quality insights on Cloudflare Radar. The new Internet Quality page provides both country and network (autonomous system) level insight into Internet connection performance (bandwidth) and quality (latency, jitter) over time. (Your Internet service provider is likely an autonomous system with its own autonomous system number (ASN), and many large companies, online platforms, and educational institutions also have their own autonomous systems and associated ASNs.) The insights we are providing are presented across two sections: the Internet Quality Index (IQI), which estimates average Internet quality based on aggregated measurements against a set of Cloudflare & third-party targets, and Connection Quality, which presents peak/best case connection characteristics based on speed.cloudflare.com test results aggregated over the previous 90 days. (Details on our approach to the analysis of this data are presented below.)

Users may note that individual speed test results, as well as the aggregate speed test results presented on the Internet Quality page will likely differ from those presented by other speed test tools. This can be due to a number of factors including differences in test endpoint locations (considering both geographic and network distance), test content selection, the impact of “rate boosting” by some ISPs, and testing over a single connection vs. multiple parallel connections. Infrequent testing (on any speed test tool) by users seeking to confirm perceived poor performance or validate purchased speeds will also contribute to the differences seen in the results published by the various speed test platforms.

And as we announced in April, Cloudflare has partnered with Measurement Lab (M-Lab) to create a publicly-available, queryable repository for speed test results. M-Lab is a non-profit third-party organization dedicated to providing a representative picture of Internet quality around the world. M-Lab produces and hosts the Network Diagnostic Tool, which is a very popular network quality test that records millions of samples a day. Given their mission to provide a publicly viewable, representative picture of Internet quality, we chose to partner with them to provide an accurate view of your Internet experience and the experience of others around the world using openly available data.

Connection speed & quality data is important

While most advertisements for fixed broadband and mobile connectivity tend to focus on download speeds (and peak speeds at that), there’s more to an Internet connection, and the user’s experience with that Internet connection, than that single metric. In addition to download speeds, users should also understand the upload speeds that their connection is capable of, as well as the quality of the connection, as expressed through metrics known as latency and jitter. Getting insight into all of these metrics provides a more well-rounded view of a given Internet connection, or in aggregate, the state of Internet connectivity across a geography or network.

The concept of download speeds are fairly well understood as a measure of performance. However, it is important to note that the average download speeds experienced by a user during common Web browsing activities, which often involves the parallel retrieval of multiple smaller files from multiple hosts, can differ significantly from peak download speeds, where the user is downloading a single large file (such as a video or software update), which allows the connection to reach maximum performance. The bandwidth (speed) available for upload is sometimes mentioned in ISP advertisements, but doesn’t receive much attention. (And depending on the type of Internet connection, there’s often a significant difference between the available upload and download speeds.) However, the importance of upload came to the forefront in 2020 as video conferencing tools saw a surge in usage as both work meetings and school classes shifted to the Internet during the COVID-19 pandemic. To share your audio and video with other participants, you need sufficient upload bandwidth, and this issue was often compounded by multiple people sharing a single residential Internet connection.

Latency is the time it takes data to move through the Internet, and is measured in the number of milliseconds that it takes a packet of data to go from a client (such as your computer or mobile device) to a server, and then back to the client. In contrast to speed metrics, lower latency is preferable. This is especially true for use cases like online gaming where latency can make a difference between a character’s life and death in the game, as well as video conferencing, where higher latency can cause choppy audio and video experiences, but it also impacts web page performance. The latency metric can be further broken down into loaded and idle latency. The former measures latency on a loaded connection, where bandwidth is actively being consumed, while the latter measures latency on an “idle” connection, when there is no other network traffic present. (These specific loaded and idle definitions are from the device’s perspective, and more specifically, from the speed test application’s perspective. Unless the speed test is being performed directly from a router, the device/application doesn't have insight into traffic on the rest of the network.) Jitter is the average variation found in consecutive latency measurements, and can be measured on both idle and loaded connections. A lower number means that the latency measurements are more consistent. As with latency, Internet connections should have minimal jitter, which helps provide more consistent performance.

Our approach to data analysis

The Internet Quality Index (IQI) and Connection Quality sections get their data from two different sources, providing two different (albeit related) perspectives. Under the hood they share some common principles, though.

IQI builds upon the mechanism we already use to regularly benchmark ourselves against other industry players. It is based on end user measurements against a set of Cloudflare and third-party targets, meant to represent a pattern that has become very common in the modern Internet, where most content is served from distribution networks with points of presence spread throughout the world. For this reason, and by design, IQI will show worse results for regions and Internet providers that rely on international (rather than peering) links for most content.

IQI is also designed to reflect the traffic load most commonly associated with web browsing, rather than more intensive use. This, and the chosen set of measurement targets, effectively biases the numbers towards what end users experience in practice (where latency plays an important role in how fast things can go).

For each metric covered by IQI, and for each ASN, we calculate the 25th percentile, median, and 75th percentile at 15 minute intervals. At the country level and above, the three calculated numbers for each ASN visible from that region are independently aggregated. This aggregation takes the estimated user population of each ASN into account, biasing the numbers away from networks that source a lot of automated traffic but have few end users.

The Connection Quality section gets its data from the Cloudflare Speed Test tool, which exercises a user's connection in order to see how well it is able to perform. It measures against the closest Cloudflare location, providing a good balance of realistic results and network proximity to the end user. We have a presence in 285 cities around the world, allowing us to be pretty close to most users.

Similar to the IQI, we calculate the 25th percentile, median, and 75th percentile for each ASN. But here these three numbers are immediately combined using an operation called the trimean — a single number meant to balance the best connection quality that most users have, with the best quality available from that ASN (users may not subscribe to the best available plan for a number of reasons).

Because users may choose to run a speed test for different motives at different times, and also because we take privacy very seriously and don’t record any personally identifiable information along with test results, we aggregate at 90-day intervals to capture as much variability as we can.

At the country level and above, the calculated trimean for each ASN in that region is aggregated. This, again, takes the estimated user population of each ASN into account, biasing the numbers away from networks that have few end users but which may still have technicians using the Cloudflare Speed Test to assess the performance of their network.

The new Internet Quality page includes three views: Global, country-level, and autonomous system (AS). In line with the other pages on Cloudflare Radar, the country-level and AS pages show the same data sets, differing only in their level of aggregation. Below, we highlight the various components of the Internet Quality page.

Global

Introducing the Cloudflare Radar Internet Quality Page

The top section of the global (worldwide) view includes time series graphs of the Internet Quality Index metrics aggregated at a continent level. The time frame shown in the graphs is governed by the selection made in the time frame drop down at the upper right of the page, and at launch, data for only the last three months is available. For users interested in examining a specific continent, clicking on the other continent names in the legend removes them from the graph. Although continent-level aggregation is still rather coarse, it still provides some insight into regional Internet quality around the world.

Introducing the Cloudflare Radar Internet Quality Page

Further down the page, the Connection Quality section presents a choropleth map, with countries shaded according to the values of the speed, latency, or jitter metric selected from the drop-down menu. Hovering over a country displays a label with the country’s name and metric value, and clicking on the country takes you to the country’s Internet Quality page. Note that in contrast to the IQI section, the Connection Quality section always displays data aggregated over the previous 90 days.

Country-level

Within the country-level page (using Canada as an example in the figures below), the country’s IQI metrics over the selected time frame are displayed. These time series graphs show the median bandwidth, latency, and DNS response time within a shaded band bounded at the 25th and 75th percentile and represent the average expected user experience across the country, as discussed in the Our approach to data analysis section above.

Introducing the Cloudflare Radar Internet Quality Page
Introducing the Cloudflare Radar Internet Quality Page
Introducing the Cloudflare Radar Internet Quality Page

Below that is the Connection Quality section, which provides a summary view of the country’s measured upload and download speeds, as well as latency and jitter, over the previous 90 days. The colored wedges in the Performance Summary graph are intended to illustrate aggregate connection quality at a glance, with an “ideal” connection having larger upload and download wedges and smaller latency and jitter wedges. Hovering over the wedges displays the metric’s value, which is also shown in the table to the right of the graph.

Introducing the Cloudflare Radar Internet Quality Page

Below that, the Bandwidth and Latency/Jitter histograms illustrate the bucketed distribution of upload and download speeds, and latency and jitter measurements. In some cases, the speed histograms may show a noticeable bar at 1 Gbps, or 1000 ms (1 second) on the latency/jitter histograms. The presence of such a bar indicates that there is a set of measurements with values greater than the 1 Gbps/1000 ms maximum histogram values.

Introducing the Cloudflare Radar Internet Quality Page

Autonomous system level

Within the upper-right section of the country-level page, a list of the top five autonomous systems within the country is shown. Clicking on an ASN takes you to the Performance page for that autonomous system. For others not displayed in the top five list, you can use the search bar at the top of the page to search by autonomous system name or number. The graphs shown within the AS level view are identical to those shown at a country level, but obviously at a different level of aggregation. You can find the ASN that you are connected to from the My Connection page on Cloudflare Radar.

Exploring connection performance & quality data

Digging into the IQI and Connection Quality visualizations can surface some interesting observations, including characterizing Internet connections, and the impact of Internet disruptions, including shutdowns and network issues. We explore some examples below.

Characterizing Internet connections

Verizon FiOS is a residential fiber-based Internet service available to customers in the United States. Fiber-based Internet services (as opposed to cable-based, DSL, dial-up, or satellite) will generally offer symmetric upload and download speeds, and the FiOS plans page shows this to be the case, offering 300 Mbps (upload & download), 500 Mbps (upload & download), and “1 Gig” (Verizon claims average wired speeds between 750-940 Mbps download / 750-880 Mbps upload) plans. Verizon carries FiOS traffic on AS701 (labeled UUNET due to a historical acquisition), and in looking at the bandwidth histogram for AS701, several things stand out. The first is a rough symmetry in upload and download speeds. (A cable-based Internet service provider, in contrast, would generally show a wide spread of download speeds, but have upload speeds clustered at the lower end of the range.) Another is the peaks around 300 Mbps and 750 Mbps, suggesting that the 300 Mbps and “1 Gig” plans may be more popular than the 500 Mbps plan. It is also clear that there are a significant number of test results with speeds below 300 Mbps. This is due to several factors: one is that Verizon also carries lower speed non-FiOS traffic on AS701, while another is that erratic nature of in-home WiFi often means that the speeds achieved on a test will be lower than the purchased service level.

Introducing the Cloudflare Radar Internet Quality Page

Traffic shifts drive latency shifts

On May 9, 2023, the government of Pakistan ordered the shutdown of mobile network services in the wake of protests following the arrest of former Prime Minister Imran Khan. Our blog post covering this shutdown looked at the impact from a traffic perspective. Within the post, we noted that autonomous systems associated with fixed broadband networks saw significant increases in traffic when the mobile networks were shut down – that is, some users shifted to using fixed networks (home broadband) when mobile networks were unavailable.

Examining IQI data after the blog post was published, we found that the impact of this traffic shift was also visible in our latency data. As can be seen in the shaded area of the graph below, the shutdown of the mobile networks resulted in the median latency dropping about 25% as usage shifted from higher latency mobile networks to lower latency fixed broadband networks. An increase in latency is visible in the graph when mobile connectivity was restored on May 12.

Introducing the Cloudflare Radar Internet Quality Page

Bandwidth shifts as a potential early warning sign

On April 4, UK mobile operator Virgin Media suffered several brief outages. In examining the IQI bandwidth graph for AS5089, the ASN used by Virgin Media (formerly branded as NTL), indications of a potential problem are visible several days before the outages occurred, as median bandwidth dropped by about a third, from around 35 Mbps to around 23 Mbps. The outages are visible in the circled area in the graph below. Published reports indicate that the problems lasted into April 5, in line with the lower median bandwidth measured through mid-day.

Introducing the Cloudflare Radar Internet Quality Page

Submarine cable issues cause slower browsing

On June 5, Philippine Internet provider PLDT Tweeted an advisory that noted “One of our submarine cable partners confirms a loss in some of its internet bandwidth capacity, and thus causing slower Internet browsing.” IQI latency and bandwidth graphs for AS9299, a primary ASN used by PLDT, shows clear shifts starting around 06:45 UTC (14:45 local time). Median bandwidth dropped by half, from 17 Mbps to 8 Mbps, while median latency increased by 75% from 37 ms to around 65 ms. 75th percentile latency also saw a significant increase, nearly tripling from 63 ms to 180 ms coincident with the reported submarine cable issue.

Introducing the Cloudflare Radar Internet Quality Page
Introducing the Cloudflare Radar Internet Quality Page

Conclusion

Making network performance and quality insights available on Cloudflare Radar supports Cloudflare’s mission to help build a better Internet. However, we’re not done yet – we have more enhancements planned. These include making data available at a more granular geographical level (such as state and possibly city), incorporating AIM scores to help assess Internet quality for specific types of use cases, and embedding the Cloudflare speed test directly on Radar using the open source JavaScript module.

In the meantime, we invite you to use speed.cloudflare.com to test the performance and quality of your Internet connection, share any country or AS-level insights you discover on social media (tag @CloudflareRadar on Twitter or @[email protected] on Mastodon), and explore the underlying data through the M-Lab repository or the Radar API.

Exam-related Internet shutdowns in Iraq and Algeria put connectivity to the test

Post Syndicated from David Belson original http://blog.cloudflare.com/exam-internet-shutdowns-iraq-algeria/

Exam-related Internet shutdowns in Iraq and Algeria put connectivity to the test

Exam-related Internet shutdowns in Iraq and Algeria put connectivity to the test

Over the last several years, governments in a number of countries in the Middle East/Northern Africa (MENA) region have taken to implementing widespread nationwide shutdowns in an effort to prevent cheating on nationwide academic exams. Although it is unclear whether such shutdowns are actually successful in curbing cheating, it is clear that they take a financial toll on the impacted countries, with estimated losses in the millions of US dollars.

During the first two weeks of June 2023, we’ve seen Iraq implementing a series of multi-hour shutdowns that will reportedly occur through mid-July, as well as Algeria taking similar actions to prevent cheating on baccalaureate exams. Shutdowns in Syria were reported to begin on June 7, but there’s been no indication of them in traffic data as of this writing (June 13). These actions echo those taken in Iraq, Syria, Sudan, and Algeria in 2022 and in Syria and Sudan in 2021.

(Note: The interactive graphs below have been embedded directly into the blog post using a new Cloudflare Radar feature. This post is best viewed in landscape mode when on a mobile device.)

Iraq

Iraq had reportedly committed on May 15 to not implementing Internet shutdowns during the 2023 exam season, with a now unavailable page on the Iraqi Ministry of Communications web site (although captured in the Internet Archive’s Wayback Machine) noting (via Google Translate) “Her Excellency the Minister of Communications, Dr. Hayam Al-Yasiri: We rejected a request to cut off the internet service during the ministerial exams.” However, that commitment was apparently short-lived, as the first shutdown was implemented just a couple of weeks later, on June 1. The shutdowns observed across Iraq thus far have impacted networks and localities nationwide, with the exception of the autonomous Kurdistan region. However, networks in that region have experienced their own set of connectivity restrictions due to local exams.

In Iraq, the impact of the shutdowns between 04:00 – 08:00 local time (01:00 – 05:00 UTC) is clearly visible at a country level, as seen in the figure below.

The impact is, of course, also visible in the network-level graphs shown below, with traffic dropping to or near zero during each of the four-hour shutdown windows.

The shutdowns are also visible in the BGP announcement activity from the impacted networks, with spikes in announcement volume clearly visible around the shutdown windows each day that they have occurred. The announcement activity represents withdrawals ahead of the shutdown, removing routes to prefixes within the network, effectively cutting them off from the Internet, and updates after the shutdown period has ended, restoring the previously withdrawn routes, effectively reconnecting the prefixes to the Internet. (Additional announcement activity may also be visible for periods outside of the scheduled shutdowns, and is likely unrelated.)

While the shutdowns discussed above didn’t impact the Kurdistan region of Iraq, that region has also chosen to implement their own shutdowns. In the Kurdistan region, exams started June 3, we saw shorter traffic disruptions across three local network providers on June 3, 6, 10, and 13. The disruptions lasted from 06:30 – 07:30 local time (03:30 to 04:30 UTC) on the 3rd, and 06:40 – 08:30 local time (03:30 to 05:30 UTC) on the 6th, 10th, and 13th). Impacted regions include Erbil, Sulaymaniyah, and Duhok.

BGP announcement activity for the impacted networks in the Kurdistan region did not show the same patterns as those observed on the other Iraqi network providers discussed above.

Both sets of shutdowns in Iraq are also visible in traffic to Cloudflare’s 1.1.1.1 DNS resolver, although they highlight a difference in usage between the autonomous Kurdistan region and the rest of the country. The “totalTcpUdp” graph (blue line) below shows requests made to the resolver over UDP or TCP on port 53, the standard port used for DNS requests. The “totalDoHDoT” graph (orange line) below shows requests made to the resolver using DNS-over-HTTPS or DNS-over-TLS using port 443 or 853 respectively.

In the “totalTcpUdp” graph, we can see noticeable drops in traffic that align with the dates and times where we observed the traffic disruptions across Kurdistani networks. This drop in DNS traffic, combined with the lack of BGP announcement activity, suggests that the Internet disruptions in the Kurdistan region may be implemented as widespread blocking of Internet traffic, rather than routing-based shutdowns.

Exam-related Internet shutdowns in Iraq and Algeria put connectivity to the test

In the “totalDoHDoT” graph below, we can see noticeable drops in traffic that align with the dates and times where we observed the traffic disruptions in the rest of Iraq.

Exam-related Internet shutdowns in Iraq and Algeria put connectivity to the test

It isn’t immediately clear why there is a difference in the use of 1.1.1.1 between the two parts of the country.

Algeria

In Algeria, it appears that the country is following a similar pattern as that seen in 2021 and 2022, with two multi-hour Internet disruptions each day. Also similar to last year, it appears that they are pursuing a content blocking-based approach, instead of the wide-scale Internet shutdowns implemented in 2021, as impacted networks are not experiencing complete outages, nor do they show patterns of BGP announcement activity.

A published report indicates that two Internet disruptions will be implemented each day between June 11 and June 15. The first takes place between 08:00 – 12:00 local time (07:00 – 11:00 UTC), with the second occurring between 14:00 – 17:00 local time (13:00 – 16:00 UTC). These disruptions are visible in the shaded areas of the network-level graphs below as two distinct drops in traffic each day.

Conclusion

In cooperation with the Internet Society and Lebanese digital rights organization SMEX, digital rights organization Access Now is coordinating a #NoExamShutdown campaign across social media platforms. The campaign calls on MENA governments to end the practice of Internet shutdowns during exams, and aims to highlight how these shutdowns undermine human rights and disrupt essential social, political, economic, and cultural activities. Cloudflare Radar will continue to bring visibility to these, and other similar Internet disruptions, as they occur. You can follow them through the Cloudflare Radar Outage Center, or by following Cloudflare Radar on Twitter or Mastodon.

Examining HTTP/3 usage one year on

Post Syndicated from David Belson original http://blog.cloudflare.com/http3-usage-one-year-on/

Examining HTTP/3 usage one year on

Examining HTTP/3 usage one year on

In June 2022, after the publication of a set of HTTP-related Internet standards, including the RFC that formally defined HTTP/3, we published HTTP RFCs have evolved: A Cloudflare view of HTTP usage trends. One year on, as the RFC reaches its first birthday, we thought it would be interesting to look back at how these trends have evolved over the last year.

Our previous post reviewed usage trends for HTTP/1.1, HTTP/2, and HTTP/3 observed across Cloudflare’s network between May 2021 and May 2022, broken out by version and browser family, as well as for search engine indexing and social media bots. At the time, we found that browser-driven traffic was overwhelmingly using HTTP/2, although HTTP/3 usage was showing signs of growth. Search and social bots were mixed in terms of preference for HTTP/1.1 vs. HTTP/2, with little-to-no HTTP/3 usage seen.

Between May 2022 and May 2023, we found that HTTP/3 usage in browser-retrieved content continued to grow, but that search engine indexing and social media bots continued to effectively ignore the latest version of the web’s core protocol. (Having said that, the benefits of HTTP/3 are very user-centric, and arguably offer minimal benefits to bots designed to asynchronously crawl and index content. This may be a key reason that we see such low adoption across these automated user agents.) In addition, HTTP/3 usage across API traffic is still low, but doubled across the year. Support for HTTP/3 is on by default for zones using Cloudflare’s free tier of service, while paid customers have the option to activate support.

HTTP/1.1 and HTTP/2 use TCP as a transport layer and add security via TLS. HTTP/3 uses QUIC to provide both the transport layer and security. Due to the difference in transport layer, user agents usually require learning that an origin is accessible using HTTP/3 before they'll try it. One method of discovery is HTTP Alternative Services, where servers return an Alt-Svc response header containing a list of supported Application-Layer Protocol Negotiation Identifiers (ALPN IDs). Another method is the HTTPS record type, where clients query the DNS to learn the supported ALPN IDs. The ALPN ID for HTTP/3 is "h3" but while the specification was in development and iteration, we added a suffix to identify the particular draft version e.g., "h3-29" identified draft 29. In order to maintain compatibility for a wide range of clients, Cloudflare advertised both "h3" and "h3-29". However, draft 29 was published close to three years ago and clients have caught up with support for the final RFC. As of late May 2023, Cloudflare no longer advertises h3-29 for zones that have HTTP/3 enabled, helping to save several bytes on each HTTP response or DNS record that would have included it. Because a browser and web server typically automatically negotiate the highest HTTP version available, HTTP/3 takes precedence over HTTP/2.

In the sections below, “likely automated” and “automated” traffic based on Cloudflare bot score has been filtered out for desktop and mobile browser analysis to restrict analysis to “likely human” traffic, but it is included for the search engine and social media bot analysis. In addition, references to HTTP requests or HTTP traffic below include requests made over both HTTP and HTTPS.

Overall request distribution by HTTP version

Examining HTTP/3 usage one year on

Aggregating global web traffic to Cloudflare on a daily basis, we can observe usage trends for HTTP/1.1, HTTP/2, and HTTP/3 across the surveyed one year period. The share of traffic over HTTP/1.1 declined from 8% to 7% between May and the end of September, but grew rapidly to over 11% through October. It stayed elevated into the new year and through January, dropping back down to 9% by May 2023. Interestingly, the weekday/weekend traffic pattern became more pronounced after the October increase, and remained for the subsequent six months. HTTP/2 request share saw nominal change over the year, beginning around 68% in May 2022, but then starting to decline slightly in June. After that, its share didn’t see a significant amount of change, ending the period just shy of 64%. No clear weekday/weekend pattern was visible for HTTP/2. Starting with just over 23% share in May 2022, the percentage of requests over HTTP/3 grew to just over 30% by August and into September, but dropped to around 26% by November. After some nominal loss and growth, it ended the surveyed time period at 28% share. (Note that this graph begins in late May due to data retention limitations encountered when generating the graph in early June.)

API request distribution by HTTP version

Examining HTTP/3 usage one year on

Although API traffic makes up a significant amount of Cloudflare’s request volume, only a small fraction of those requests are made over HTTP/3. Approximately half of such requests are made over HTTP/1.1, with another third over HTTP/2. However, HTTP/3 usage for APIs grew from around 6% in May 2022 to over 12% by May 2023. HTTP/3’s smaller share of traffic is likely due in part to support for HTTP/3 in key tools like curl still being considered as “experimental”. Should this change in the future, with HTTP/3 gaining first-class support in such tools, we expect that this will accelerate growth in HTTP/3 usage, both for APIs and overall as well.

Mitigated request distribution by HTTP version

Examining HTTP/3 usage one year on

The analyses presented above consider all HTTP requests made to Cloudflare, but we also thought that it would be interesting to look at HTTP version usage by potentially malicious traffic, so we broke out just those requests that were mitigated by one of Cloudflare’s application security solutions. The graph above shows that the vast majority of mitigated requests are made over HTTP/1.1 and HTTP/2, with generally less than 5% made over HTTP/3. Mitigated requests appear to be most frequently made over HTTP/1.1, although HTTP/2 accounted for a larger share between early August and late November. These observations suggest that attackers don’t appear to be investing the effort to upgrade their tools to take advantage of the newest version of HTTP, finding the older versions of the protocol sufficient for their needs. (Note that this graph begins in late May 2022 due to data retention limitations encountered when generating the graph in early June 2023.)

HTTP/3 use by desktop browser

As we noted last year, support for HTTP/3 in the stable release channels of major browsers came in November 2020 for Google Chrome and Microsoft Edge, and April 2021 for Mozilla Firefox. We also noted that in Apple Safari, HTTP/3 support needed to be enabled in the “Experimental Features” developer menu in production releases. However, in the most recent releases of Safari, it appears that this step is no longer necessary, and that HTTP/3 is now natively supported.

Examining HTTP/3 usage one year on

Looking at request shares by browser, Chrome started the period responsible for approximately 80% of HTTP/3 request volume, but the continued growth of Safari dropped it to around 74% by May 2023. A year ago, Safari represented less than 1% of HTTP/3 traffic on Cloudflare, but grew to nearly 7% by May 2023, likely as a result of support graduating from experimental to production.

Examining HTTP/3 usage one year on

Removing Chrome from the graph again makes trends across the other browsers more visible. As noted above, Safari experienced significant growth over the last year, while Edge saw a bump from just under 10% to just over 11% in June 2022. It stayed around that level through the new year, and then gradually dropped below 10% over the next several months. Firefox dropped slightly, from around 10% to just under 9%, while reported HTTP/3 traffic from Internet Explorer was near zero.

As we did in last year’s post, we also wanted to look at how the share of HTTP versions has changed over the last year across each of the leading browsers. The relative stability of HTTP/2 and HTTP/3 seen over the last year is in some contrast to the observations made in last year’s post, which saw some noticeable shifts during the May 2021 – May 2022 timeframe.

Examining HTTP/3 usage one year on
Examining HTTP/3 usage one year on
Examining HTTP/3 usage one year on
Examining HTTP/3 usage one year on

In looking at request share by protocol version across the major desktop browser families, we see that across all of them, HTTP/1.1 share grows in late October. Further analysis indicates that this growth was due to significantly higher HTTP/1.1 request volume across several large customer zones, but it isn’t clear why this influx of traffic using an older version of HTTP occurred. It is clear that HTTP/2 remains the dominant protocol used for content requests by the major browsers, consistently accounting for 50-55% of request volume for Chrome and Edge, and ~60% for Firefox. However, for Safari, HTTP/2’s share dropped from nearly 95% in May 2022 to around 75% a year later, thanks to the growth in HTTP/3 usage.

HTTP/3 share on Safari grew from under 3% to nearly 18% over the course of the year, while its share on the other browsers was more consistent, with Chrome and Edge hovering around 40% and Firefox around 35%, and both showing pronounced weekday/weekend traffic patterns. (That pattern is arguably the most pronounced for Edge.) Such a pattern becomes more, yet still barely, evident with Safari in late 2022, although it tends to vary by less than a percent.

HTTP/3 usage by mobile browser

Mobile devices are responsible for over half of request volume to Cloudflare, with Chrome Mobile generating more than 25% of all requests, and Mobile Safari more than 10%. Given this, we decided to explore HTTP/3 usage across these two key mobile platforms.

Examining HTTP/3 usage one year on
Examining HTTP/3 usage one year on

Looking at Chrome Mobile and Chrome Mobile Webview (an embeddable version of Chrome that applications can use to display Web content), we find HTTP/1.1 usage to be minimal, topping out at under 5% of requests. HTTP/2 usage dropped from 60% to just under 55% between May and mid-September, but then bumped back up to near 60%, remaining essentially flat to slightly lower through the rest of the period. In a complementary fashion, HTTP/3 traffic increased from 37% to 45%, before falling just below 40% in mid-September, hovering there through May. The usage patterns ultimately look very similar to those seen with desktop Chrome, albeit without the latter’s clear weekday/weekend traffic pattern.

Perhaps unsurprisingly, the usage patterns for Mobile Safari and Mobile Safari Webview closely mirror those seen with desktop Safari. HTTP/1.1 share increases in October, and HTTP/3 sees strong growth, from under 3% to nearly 18%.

Search indexing bots

Exploring usage of the various versions of HTTP by search engine crawlers/bots, we find that last year’s trend continues, and that there remains little-to-no usage of HTTP/3. (As mentioned above, this is somewhat expected, as HTTP/3 is optimized for browser use cases.) Graphs for Bing & Baidu here are trimmed to a period ending April 1, 2023 due to anomalous data during April that is being investigated.

Examining HTTP/3 usage one year on

GoogleBot continues to rely primarily on HTTP/1.1, which generally comprises 55-60% of request volume. The balance is nearly all HTTP/2, although some nominal growth in HTTP/3 usage sees it peaking at just under 2% in March.

Examining HTTP/3 usage one year on

Through January 2023, around 85% of requests from Microsoft’s BingBot were made via HTTP/2, but dropped to closer to 80% in late January. The balance of the requests were made via HTTP/1.1, as HTTP/3 usage was negligible.

Examining HTTP/3 usage one year on
Examining HTTP/3 usage one year on

Looking at indexing bots from search engines based outside of the United States, Russia’s YandexBot appears to use HTTP/1.1 almost exclusively, with HTTP/2 usage generally around 1%, although there was a period of increased usage between late August and mid-November. It isn’t clear what ultimately caused this increase. There was no meaningful request volume seen over HTTP/3. The indexing bot used by Chinese search engine Baidu also appears to strongly prefer HTTP/1.1, generally used for over 85% of requests. However, the percentage of requests over HTTP/2 saw a number of spikes, briefly reaching over 60% on days in July, November, and December 2022, as well as January 2023, with several additional spikes in the 30% range. Again, it isn’t clear what caused this spiky behavior. HTTP/3 usage by BaiduBot is effectively non-existent as well.

Social media bots

Similar to Bing & Baidu above, the graphs below are also trimmed to a period ending April 1.

Examining HTTP/3 usage one year on

Facebook’s use of HTTP/3 for site crawling and indexing over the last year remained near zero, similar to what we observed over the previous year. HTTP/1.1 started the period accounting for under 60% of requests, and except for a brief peak above it in late May, usage of HTTP/1.1 steadily declined over the course of the year, dropping to around 30% by April 2023. As such, use of HTTP/2 increased from just over 40% in May 2022 to over 70% in April 2023. Meta engineers confirmed that this shift away from HTTP/1.1 usage is an expected gradual change in their infrastructure's use of HTTP, and that they are slowly working towards removing HTTP/1.1 from their infrastructure entirely.

Examining HTTP/3 usage one year on

In last year’s blog post, we noted that “TwitterBot clearly has a strong and consistent preference for HTTP/2, accounting for 75-80% of its requests, with the balance over HTTP/1.1.” This preference generally remained the case through early October, at which point HTTP/2 usage began a gradual decline to just above 60% by April 2023. It isn’t clear what drove the week-long HTTP/2 drop and HTTP/1.1 spike in late May 2022. And as we noted last year, TwitterBot’s use of HTTP/3 remains non-existent.

Examining HTTP/3 usage one year on

In contrast to Facebook’s and Twitter’s site crawling bots, HTTP/3 actually accounts for a noticeable, and growing, volume of requests made by LinkedIn’s bot, increasing from just under 1% in May 2022 to just over 10% in April 2023. We noted last year that LinkedIn’s use of HTTP/2 began to take off in March 2022, growing to approximately 5% of requests. Usage of this version gradually increased over this year’s surveyed period to 15%, although the growth was particularly erratic and spiky, as opposed to a smooth, consistent increase. HTTP/1.1 remained the dominant protocol used by LinkedIn’s bots, although its share dropped from around 95% in May 2022 to 75% in April 2023.

Conclusion

On the whole, we are excited to see that usage of HTTP/3 has generally increased for browser-based consumption of traffic, and recognize that there is opportunity for significant further growth if and when it starts to be used more actively for API interactions through production support in key tools like curl. And though disappointed to see that search engine and social media bot usage of HTTP/3 remains minimal to non-existent, we also recognize that the real-time benefits of using the newest version of the web’s foundational protocol may not be completely applicable for asynchronous automated content retrieval.

You can follow these and other trends in the “Adoption and Usage” section of Cloudflare Radar at https://radar.cloudflare.com/adoption-and-usage, as well as by following @CloudflareRadar on Twitter or https://cloudflare.social/@radar on Mastodon.

Cloudflare’s view of Internet disruptions in Pakistan

Post Syndicated from David Belson original http://blog.cloudflare.com/cloudflares-view-of-internet-disruptions-in-pakistan/

Cloudflare’s view of Internet disruptions in Pakistan

Cloudflare’s view of Internet disruptions in Pakistan

On Tuesday, May 9, Imran Khan, former Prime Minister of Pakistan was arrested on corruption charges. Following the arrest, violent protests erupted in several cities, leading the government of Pakistan to order the shutdown of mobile Internet services, as well as the blocking of several social media platforms. Below, we examine the impact of these shutdowns at a national and local level, as seen through Cloudflare traffic data. In addition, we illustrate how Pakistanis appear to be turning to Cloudflare’s 1.1.1.1 resolver in an attempt to maintain access to the open Internet.

Since Tuesday, May 9, peak traffic levels aggregated at a country level (as measured by HTTP request volume) have been declining, down nearly 30% during the first several days of the mobile Internet shutdowns. The lowest traffic levels (nadirs of the graph) have also declined, dropping by as much as one-third as well. In the sections below, we drill down into this traffic loss, looking at outages at a network level, and the impact of those outages at an administrative unit and city level.

Cloudflare’s view of Internet disruptions in Pakistan

The mobile network shutdowns have also impacted the profile of traffic that Cloudflare sees from Pakistan. In analyzing traffic from desktop devices vs. mobile devices, we observed a 60% drop in request volume from mobile devices, while desktop traffic request volume remained fairly consistent. Peak mobile device traffic share dropped from 70% to 43%.

Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan

Cloudflare uses a bot score assigned to each request to indicate how likely it is that the request came from a bot or a human user. Since these shutdowns began, peak human request volume has dropped by 40%, while bot traffic has remained relatively consistent.

Cloudflare’s view of Internet disruptions in Pakistan

Mobile network shutdowns

On Wednesday, May 10, the Pakistan Telecommunication Authority (PTA) announced that Internet services would remain suspended across the country for an “indefinite” period, responding to a directive from the Ministry of the Interior to block mobile broadband services. As a result of the shutdowns associated with this directive, Cloudflare observed outages on the four major mobile providers within the country:

Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan

Although Pakistan has high mobile Internet usage, it appears that fixed broadband Internet connections are readily used as a backup when mobile connectivity becomes unavailable. Autonomous systems associated with fixed broadband networks saw significant increases in traffic when the mobile networks were shut down.

Nationwide providers PTCL (AS17557) and Cybernet (AS9541) saw higher peak traffic volumes as compared to a week prior starting at 17:00 UTC (22:00 local time) on May 9.

Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan

Smaller local providers Nayatel (AS23674) and Wateen Telecom (AS38264) also saw higher peak traffic levels starting around 16:00 UTC (21:00 local time) on May 9.

Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan

Interestingly, median latency within Pakistan also dropped slightly after mobile networks were shut down. Prior to the shutdown, median latency (as observed to Cloudflare and a set of other providers) was in the 90-100ms range, while afterwards, it has averaged closer to 75ms. This may be a result of users shifting to lower latency fixed broadband connections, as discussed above.

Cloudflare’s view of Internet disruptions in Pakistan

Administrative unit-level disruptions

Because the mobile network providers that were affected by the shutdown directive provide services nationwide, we also observed an impact to traffic across multiple administrative units within the country. None of these locations has experienced a complete outage, but peak traffic levels have clearly been declining in comparison to previous days.

Gilgit-Baltistan experienced the largest loss, where peak traffic has fallen nearly 60%. In Sindh, peak traffic is down around 35%, followed by Khyber Pakhtunkhwa, where it is down 30%. Islamabad and Azad Jammu and Kashmir have seen peak traffic declines of ~20%.

Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan

City-level disruptions

The impact of the mobile network shutdowns is also visible at a more local level, with lower peak traffic levels clearly visible in four cities. The significant traffic loss has been in Peshawar (Khyber Pakhtunkhwa), which has dropped nearly 55% from prior days. Faisalabad (Punjab), Karachi (Sindh), and Multan (Punjab) have all seen peak traffic drop approximately 40%.

Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan

Content blocking

In addition to the government-directed mobile network shutdowns, Pakistan’s authorities have also ordered Internet service providers to block access to social media platforms including Facebook, Instagram, YouTube, and Twitter. Testing by the Open Observatory for Network Interference (OONI), an Internet censorship measurement organization, suggests that this blocking is using a combination of TLS-level interference and DNS-based blocking. When the latter occurs in a country, Cloudflare’s 1.1.1.1 DNS resolver often sees an increase in request volume from the country as users seek ways to continue to access the open Internet.

Over the last several days, as expected, 1.1.1.1 request volume from Pakistan has increased, up approximately 40%. Peak request volume for the blocked social media platforms has also increased. Traffic for facebook.com saw a significant increase starting around 14:00 UTC (19:00 local time) on May 9, with peak request volume more than doubling. Request volume for instagram.com, also owned by Facebook parent Meta, also began to increase around the same time, and has grown nearly 50%. Requests for twitter.com began to spike around 08:00 UTC (13:00 local time) on May 9, growing as much as 150% that afternoon. Request volume for youtube.com also spiked on May 9, increasing by approximately 40%. And like twitter.com, request volume on May 10 was higher than earlier in the week, but lower than the spike seen the previous day.

Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan
Cloudflare’s view of Internet disruptions in Pakistan

Conclusion

Because of the ubiquity of Internet connectivity and social media tools in everyday life, Internet shutdowns and website blocking ultimately come with a significant human and financial cost. The mobile network shutdowns in Pakistan have impacted tens of thousands of “gig workers” and freelancers that depend on mobile connectivity. Many point-of-sale terminals in the country also depend on mobile connectivity, with transactions through Pakistan’s main digital payment systems fell by around 50% after the shutdowns were put into place. Telecommunications operators within Pakistan have estimated the extent of the financial damage thus far to be Rs. 820 million (approximately $2.8 million USD).

Use Cloudflare Radar to monitor the impact of such government-directed Internet disruptions, and follow @CloudflareRadar on Twitter for updates on Internet disruptions as they occur.

Internet disruptions overview for Q1 2023

Post Syndicated from David Belson original https://blog.cloudflare.com/q1-2023-internet-disruption-summary/

Internet disruptions overview for Q1 2023

Internet disruptions overview for Q1 2023

Cloudflare operates in more than 285 cities in over 100 countries, where we interconnect with over 11,500 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions.

We entered 2023 with Internet disruptions due to causes that ran the gamut, including several government-directed Internet shutdowns, cyclones, a massive earthquake, power outages, cable cuts, cyberattacks, technical problems, and military action. As we have noted in the past, this post is intended as a summary overview of observed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter.

Government directed

Iran

Over the last six-plus months, government-directed Internet shutdowns in Iran have largely been in response to protests over the death of Mahsa Amini while in police custody. While these shutdowns are still occurring in a limited fashion, a notable shutdown observed in January was intended to prevent cheating on academic exams. Internet shutdowns with a similar purpose have been observed across a number of other countries, and have also occurred in Iran in the past. Access was restricted across AS44244 (Irancell) and AS197207 (MCCI), with lower traffic levels observed in Alborz Province, Fars, Khuzestan, and Razavi Khorasan between 08:00 to 11:30 local time (04:30 to 08:00 UTC) on January 19.

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Mauritania

On March 6, Internet traffic across the three major mobile network providers in Mauritania was disrupted amid a search for four jihadist prisoners that escaped from prison. Starting around 10:00 local time (10:00 UTC), a drop in traffic was observed at AS37541 (Chinguitel), AS29544 (Mauritel), and AS37508 (Mattel), as well as at a country level. The Internet disruption lasted for multiple days, with traffic starting to recover around 13:45 local time (13:45 UTC) on March 12, after Mauritanian authorities reported that three of the escapees had been killed, with the fourth detained after a shootout.

Internet disruptions overview for Q1 2023

Punjab, India

A shutdown of mobile Internet connectivity in Punjab, India began on March 19, ordered by the local government amid concerns of protest-related violence. Although the initial shutdown was ordered to take place between March 18, 12:00 local time and March 19, 12:00 local time, it was extended several times, ultimately lasting for three days. Traffic for AS38266 (Vodafone India), AS45271 (Idea Cellular Limited), AS45609 (Bharti Mobility), and AS55836 (Reliance Jio Infocomm) began to fall around 12:30 local time (07:00 UTC) on March 18, recovering around 12:30 local time (07:00 UTC) on March 21. However, it was subsequently reported that connectivity remained shut down in some districts until March 23 or 24.

Internet disruptions overview for Q1 2023

Cable cuts

Bolivia

Bolivian ISP Cometco (AS27839) reported on January 12 that problems with international fiber links were causing degradation of Internet service. Traffic from the network dropped by approximately 80% starting around 16:00 local time (20:00 UTC) before returning to normal approximately eight hours later. It isn’t clear whether the referenced international fiber links were terrestrial connections to neighboring countries, or issues with submarine cables several network hops upstream. As a landlocked country, Bolivia is not directly connected to any submarine cables.

Internet disruptions overview for Q1 2023

Anguilla

On February 18, a Facebook post from the Government of Anguilla noted that there was a “Telecommunications Outage affecting both service providers, FLOW & DIGICEL.” The accompanying graphic noted that the outage was due to a “subsea fiber break”. Although not confirmed, the break likely occurred on the Eastern Caribbean Fiber System (ECFS), as this is the only submarine cable system that Anguilla is connected to. The figures below show a clear drop in traffic around 09:00 local time (13:00 UTC) in Anguilla and at AS2740 (Caribbean Cable Communications, acquired by Digicel) and to a lesser extent at AS11139 (Cable & Wireless, parent company of Flow Anguilla). The disruption lasted for over two days, with traffic returning to normal levels around 15:00 local time (19:00 UTC) on February 20, corroborated by a follow-on Facebook post from the Government of Anguilla.

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Bangladesh

A brief connectivity disruption was observed on Bangladeshi provider Grameenphone on February 23, between 11:45-14:00 local time (05:45-08:00 UTC). According to a Facebook post from Grameenphone, the outage was caused by fiber cuts due to road maintenance.

Internet disruptions overview for Q1 2023

Venezuela

Venezuela, and more specifically, AS8048 (CANTV), are no stranger to Internet disruptions, seeing several (Q1, Q2) during 2022, as well as others in previous years. During the last couple of days in February, a few small outages were observed on CANTV’s network in several Venezuelan states. However, a more significant near-complete outage occurred on February 28, starting around midnight local time (04:00 UTC), and lasting for the better part of the day, with traffic recovering at 17:30 local time (21:30 UTC). A Tweet posted the morning of February 28 by CANTV referenced an outage in their fiber optic network, which was presumably the cause.

Internet disruptions overview for Q1 2023

Power outages

Pakistan

A country-wide power outage in Pakistan on January 23 impacted more than 220 million people, and resulted in a significant drop in Internet traffic being observed in the country. The power outage began at 07:34 local time (02:34 UTC), with Internet traffic starting to drop almost immediately. The figure below shows that traffic volumes dropped as much as 50% from normal levels before recovering around 04:15 local time on January 24 (23:15 UTC on January 23). This power outage was reportedly due to a “sudden drop in the frequency of the power transmission system”, which led to a “widespread breakdown”. Nationwide power outages have also occurred in Pakistan in January 2021, May 2018, and January 2015.

Internet disruptions overview for Q1 2023

Bermuda

BELCO, the power company servicing the island of Bermuda, tweeted about a mass power outage affecting the island on February 3, and linked to their outage map so that customers could track restoration efforts. BELCO’s tweet was posted at 16:10 local time (20:10 UTC), approximately one hour after a significant drop was observed in Bermuda’s Internet traffic. The power outage, and subsequent Internet disruption, lasted over five hours, as BELCO later tweeted that “As of 9.45 pm [00:45 UTC, February 4], all circuits have been restored.

Internet disruptions overview for Q1 2023

Argentina

Soaring temperatures in Argentina triggered a large-scale power outage across the country that resulted in multi-hour Internet disruption on March 1. Internet traffic dropped by approximately one-third during the disruption, which lasted from 16:30 to 19:30 local time (19:30 to 22:30 UTC). Cities that experienced visible impacts to Internet traffic during the power outage included Buenos Aires, Cordoba, Mendoza, and Tucuman.

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Kenya

Just a few days later on March 4, Kenya Power issued a Customer Alert at 18:25 local time (15:25 UTC) regarding a nationwide power outage, noting that it had “lost bulk power supply to various parts of the country due to a system disturbance.” The alert came approximately an hour after the country’s Internet traffic dropped significantly. A subsequent tweet dated midnight local time (21:00 UTC) claimed that “electricity supply has been restored to all areas countrywide” and the figure below shows that traffic levels returned to normal levels shortly thereafter.

Internet disruptions overview for Q1 2023

Earthquake

Turkey

On February 5, a magnitude 7.8 earthquake occurred 23 km east of Nurdağı, Turkey, leaving many thousands dead and injured. The quake, which occurred at 04:17 local time (01:17 UTC), was believed to be the strongest to hit Turkey since 1939. The widespread damage and destruction resulted in significant disruptions to Internet connectivity in multiple areas of the country, as shown in the figures below. Although Internet traffic volumes were relatively low because it was so early in the morning, the graphs show it dropping even further at or around the time of the earthquake. Nearly half a day later, traffic volumes in selected locations were between 63-94% lower than at the same time the previous week. A month later, after several aftershocks, traffic volumes had mostly recovered, although some regions were still struggling to recover.

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Weather

New Zealand

Called the “country’s biggest weather event in a century”, Cyclone Gabrielle wreaked havoc on northern New Zealand, including infrastructure damage and power outages impacting tens of thousands of homes. As a result, regions including Gisborne and Hawkes Bay experienced Internet disruptions that lasted several days, starting at 00:00 local time on February 14 (11:00 UTC on February 13). The figures below show that in both regions, peak traffic volume returned to pre-cyclone levels around February 19.

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Vanuatu

Later in February, Cyclone Judy hit the South Pacific Ocean nation of Vanuatu, the South Pacific Ocean nation made up of roughly 80 islands that stretch 1,300 kilometers. The Category 4 cyclone damaged homes and caused power outages, resulting in a significant drop in the country’s Internet traffic. On February 28, Vanuatu’s traffic dropped by nearly 80% as the cyclone struck, and as seen in the figure below, it took nearly two weeks for traffic to recover to the levels seen earlier in February.

Internet disruptions overview for Q1 2023

Malawi

Cyclone Freddy, said to be the longest-lasting, most powerful cyclone on record, hit Malawi during the weekend of March 11-12, and into Monday, March 13. The resulting damage disrupted Internet connectivity in the east African country, with traffic dropping around 11:00 local time (09:00 UTC) on March 13. The disruption lasted for over two days, with traffic levels recovering around 21:00 local time (19:00 UTC) on March 15.

Internet disruptions overview for Q1 2023

Technical problems

South Africa

Just before 07:00 local time (05:00 UTC) on February 1, South African service provider RSAWEB initially tweeted about a problem that they said was impacting their cloud and VOIP platforms. However, in several subsequent tweets, they noted that the problem was also impacting internal systems, as well as fiber and mobile connectivity. The figure below shows traffic for RSAWEB dropping at 06:30 local time (04:30 UTC), a point at which it would normally be starting to increase for the day. Just before 16:00 local time (14:00 UTC), RSAWEB tweeted “…engineers are actively working on restoring services post the major incident. Customers who experienced no connectivity may see some services restoring.” The figure below shows a sharp increase in traffic around that time with gradual growth through the evening. However, full restoration of services across all of RSAWEB’s impacted platforms took a full week, according to a February 8 tweet.

Internet disruptions overview for Q1 2023

Italy

An unspecified “international interconnectivity problem” impacting Telecom Italia caused a multi-hour Internet disruption in Italy on February 5. At a country level, a nominal drop in traffic is visible in the figure below starting around 11:45 local time (10:45 UTC) with some volatility visible in the lower traffic through 17:15 local time (16:15 UTC). However, the impact of the problem is more obvious in the traffic graphs for AS3269 and AS16232, both owned by Telecom Italia. Both graphs show a more significant loss of traffic, as well as greater volatility through the five-plus hour disruption.

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Myanmar

A fire at an exchange office of MPT (Myanma Posts and Telecommunications) on February 7 disrupted Internet connectivity for customers of the Myanmar service provider. A Facebook post from MPT informed customers that “We are currently experiencing disruptions to our MPT’s services including MPT’s call centre, fiber internet, mobile internet and mobile and telephone communications.” The figure below shows the impact of this disruption on MPT-owned AS9988 and AS45558, with traffic dropping significantly at 10:00 local time (03:30 UTC). Significant recovery was seen by 22:00 local time (15:30 local time).

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Republic of the Congo (Brazzaville)

Congo Telecom tweeted a “COMMUNIQUÉ” on March 15, alerting users to a service disruption due to a “network incident”. The impact of this disruption is clearly visible at a country level, with traffic dropping sharply at 00:45 local time (23:45 on March 14 UTC), driven by complete outages at MTN Congo and Congo Telecom, as seen in the graphs below. While traffic at MTN Congo began to recover around 08:00 local time (07:00 UTC), Congo Telecom’s recovery took longer, with traffic beginning to increase around 17:00 local time (16:00 UTC). Congo Telecom tweeted on March 16 that the nationwide Internet outage had been resolved. MTN Congo did not acknowledge the disruption on social media, and neither company provided more specific information about the reported “network incident”.

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Lebanon

Closing out March, disruptions observed at AS39010 (Terranet) and AS42334 (Mobi) in Lebanon may have been related to a strike at upstream provider Ogero Telecom, common to both networks. A published report quoted the Chairman of Ogero commenting on the strike, “We are heading to a catastrophe if a deal is not found with the government: the network will completely stop working as our generators will gradually run out of fuel. Lebanon completely relies on Ogero for its bandwidth, leaving no one exempt from a blackout.” Traffic at both Terranet and Mobi dropped around 05:00 local time (03:00 UTC) on March 29, with the disruption lasting approximately 4.5 hours, as traffic recovered at 09:30 local time (07:30 UTC).

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Cyberattacks

South Korea

On January 29, South Korean Internet provider LG Uplus suffered two brief Internet disruptions which were reportedly caused by possible DDoS attacks. The first disruption occurred at 03:00 local time (18:00 UTC on January 28), and the second occurred at 18:15 local time (09:15 UTC). The disruptions impacted traffic on AS17858 and AS3786, both owned by LG. The company was reportedly hit by a second pair of DDoS attacks on February 4.

Internet disruptions overview for Q1 2023

Guam

In a March 17 tweet posted at 11:30 local time (01:30 UTC), Docomo Pacific reported an outage affecting multiple services, with a subsequent tweet noting that “Early this morning, a cyber security incident occurred and some of our servers were attacked”. This outage is visible at a country level in Guam, seen as a significant drop in traffic starting around 10:00 local time (00:00 UTC) in the figure below. However, in the graph below for AS3605 (ERX-KUENTOS/Guam Cablevision/Docomo Pacific), the cited outage results in a near-complete loss of traffic starting around 05:00 local time (19:00 on March 16 UTC). Traffic returned to normal levels by 18:00 local time on March 18 (08:00 UTC).

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Ukraine/Military Action

In February, the conflict in Ukraine entered its second year, and over this past year, we have tracked its impact on the Internet, highlighting traffic shifts, attacks, routing changes, and connectivity disruptions. In the fourth quarter of 2022, a number of disruptions were related to attacks on electrical infrastructure, and this pattern continued into the first quarter of 2023.

One such disruption occurred in Odessa on January 27, amid news of Russian airstrikes on local energy infrastructure. As seen in the figure below, Internet traffic in Odessa usually begins to climb just before 08:00 local time (06:00 UTC), but failed to do so that morning after several energy infrastructure facilities near Odessa were hit and damaged. Traffic remained lower than levels seen the previous week for approximately 18 hours.

Internet disruptions overview for Q1 2023

Power outages resulting from Russian attacks on energy generation and distribution facilities on March 9 resulted in disruptions to Internet connectivity in multiple locations around Ukraine. As seen in the figures below, traffic dropped below normal levels after 02:00 local time (00:00 UTC) on March 9. Traffic in Kharkiv fell over 50% as compared to previous week, while in Odessa, traffic fell as much as 60%. In Odessa, Mykolaiv, and Kirovohrad Oblast, traffic recovered by around 08:00 local time (06:00 UTC), while in Kharkiv, the disruption lasted nearly two days, returning to normal levels around 23:45 local time (21:45 UTC) on Friday, March 10.

Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023
Internet disruptions overview for Q1 2023

Conclusion

The first quarter of 2023 seemed to be particularly active from an Internet disruption perspective, but hopefully it is not a harbinger of things to come through the rest of the year. This is especially true of government-directed shutdowns, which occurred fairly regularly through 2022. To that end, civil society organization Access Now recently published their Internet shutdowns in 2022 report, finding that In 2022, governments and other actors disrupted the internet at least 187 times across 35 countries. Cloudflare Radar is proud to support Access Now’s #KeepItOn initiative, using our data to help illustrate the impact of Internet shutdowns and other disruptions.

To follow Internet disruptions as they occur, check the Cloudflare Radar Outage Center (CROC) or the Radar API. On social media, follow @CloudflareRadar on Twitter or cloudflare.social/@radar on Mastodon.

Cloudflare’s view of the Virgin Media outage in the UK

Post Syndicated from David Belson original https://blog.cloudflare.com/virgin-media-outage-april-4-2023/

Cloudflare’s view of the Virgin Media outage in the UK

Just after midnight (UTC) on April 4, subscribers to UK ISP Virgin Media (AS5089) began experiencing an Internet outage, with subscriber complaints multiplying rapidly on platforms including Twitter and Reddit.

Cloudflare Radar data shows Virgin Media traffic dropping to near-zero around 00:30 UTC, as seen in the figure below. Connectivity showed some signs of recovery around 02:30 UTC, but fell again an hour later. Further nominal recovery was seen around 04:45 UTC, before again experiencing another complete outage between around 05:45-06:45 UTC, after which traffic began to recover, reaching expected levels around 07:30 UTC.

After the initial set of early-morning disruptions, Virgin Media experienced another round of issues in the afternoon. Cloudflare observed instability in traffic from Virgin Media’s network (called an autonomous system in Internet jargon) AS5089 starting around 15:00 UTC, with a significant drop just before 16:00 UTC. However in this case, it did not appear to be a complete outage, with traffic recovering approximately a half hour later.

Cloudflare’s view of the Virgin Media outage in the UK

Virgin Media’s Twitter account acknowledged the early morning disruption several hours after it began, posting responses stating “We’re aware of an issue that is affecting broadband services for Virgin Media customers as well as our contact centres. Our teams are currently working to identify and fix the problem as quickly as possible and we apologise to those customers affected.” Further responses after service restoration noted “We’ve restored broadband services for customers but are closely monitoring the situation as our engineers continue to investigate. We apologise for any inconvenience caused.”

However, the second disruption was acknowledged on Virgin Media’s Twitter account much more rapidly, with a post at 16:25 UTC stating “Unfortunately we have seen a repeat of an earlier issue which is causing intermittent broadband connectivity problems for some Virgin Media customers. We apologise again to those impacted, our teams are continuing to work flat out to find the root cause of the problem and fix it.”

At the time of the outages, www.virginmedia.com, which includes the provider’s status page, was unavailable. As seen in the figure below, a DNS lookup for the hostname resulted in a SERVFAIL error, indicating that the lookup failed to return a response. This is because the authoritative nameservers for virginmedia.com are listed as ns{1-4}.virginmedia.net, and these nameservers are all hosted within Virgin Media’s network (AS5089) and thus are not accessible during the outage.

Cloudflare’s view of the Virgin Media outage in the UK

Although Virgin Media has not publicly released a root cause for the series of disruptions that its network has experienced, looking at BGP activity can be instructive.

BGP is a mechanism to exchange routing information between networks on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver each network packet to its final destination. Without BGP, the Internet routers wouldn’t know what to do, and the Internet wouldn’t exist.

The Internet is literally a network of networks, or for math fans, a graph, with each individual network a node in it, and the edges representing the interconnections. All of this is bound together by BGP, which allows one network (Virgin Media, for instance) to advertise its presence to other networks that form the Internet. When Virgin Media is not advertising its presence, other networks can’t find its network and it becomes effectively unavailable.

BGP announcements inform a router of changes made to the routing of a prefix (a group of IP addresses) or entirely withdraws the prefix, removing it from the routing table. The figure below shows aggregate BGP announcement activity from AS5089 with spikes that align with the decreases and increases seen in the traffic graph above, suggesting that the underlying cause may in fact be BGP-related, or related to problems with core network infrastructure.

Cloudflare’s view of the Virgin Media outage in the UK

We can drill down further to break out the observed activity between BGP announcements (dark blue) and withdrawals (light blue) seen in the figure below, with key activity coincident with the loss and return of traffic. An initial set of withdrawals are seen just after midnight, effectively removing Virgin Media from the Internet resulting in the initial outage.

A set of announcements occurred just before 03:00 UTC, aligning with the nominal increase in traffic noted above, but those were followed quickly by another set of withdrawals. A similar announcement/withdrawal exchange was observed at 05:00 and 05:30 UTC respectively, before a final set of announcements restored connectivity at 07:00 UTC.

Things remained relatively stable through the morning into the afternoon, before another set of withdrawals presaged the afternoon’s connectivity problems, with a spike of withdrawals at 15:00 UTC, followed by additional withdrawal/announcement exchanges over the next several hours.

Cloudflare’s view of the Virgin Media outage in the UK

Conclusion

Track ongoing traffic trends for Virgin Media on Cloudflare Radar, and follow us on Twitter and Mastodon for regular updates.

A look at Internet traffic trends during Super Bowl LVII

Post Syndicated from David Belson original https://blog.cloudflare.com/super-bowl-lvii/

A look at Internet traffic trends during Super Bowl LVII

A look at Internet traffic trends during Super Bowl LVII

The Super Bowl has been happening since the end of the 1966 season, the same year that the ARPANET project, which gave birth to the Internet, was initiated. Around 20 years ago, 50% of the US population were Internet users, and that number is now around 92%. So, it’s no surprise that interest in an event like Super Bowl LVII resulted in a noticeable dip in Internet traffic in the United States at the time of the game’s kickoff, dropping to around 5% lower than the previous Sunday. During the game, Rihanna’s halftime show also caused a significant drop in Internet traffic across most states, with Pennsylvania and New York feeling the biggest impact, but messaging and video platforms saw a surge of traffic right after her show ended.

In this blog post, we will dive into who the biggest winners were among Super Bowl advertisers, as well as examine how traffic to food delivery services, social media and sports and betting websites changed during the game. In addition, we look at traffic trends seen at city and state levels during the game, as well as email threat volume across related categories in the weeks ahead of the game.

Cloudflare Radar uses a variety of sources to provide aggregate information about Internet traffic and attack trends. In this blog post, as we did last year and the year before, we use DNS name resolution data from our 1.1.1.1 resolver to estimate traffic to websites. We can’t see who visited the websites mentioned, or what anyone did on the websites, but DNS can give us an estimate of the interest generated by the ads or across a set of sites in the categories listed above.

Ads: are URLs no longer cool?

In contrast to Super Bowl commercials of the past 25 years, many of this year’s advertisements didn’t include a URL, possibly suggesting strong confidence by brands in their search engine results placement, or an assumption that the viewer would engage with the brand through an app on their phone, rather than a website. To that end, several ads did include an app store-related call to action, encouraging the viewer to download the associated mobile app. And possibly in an effort to capitalize on the success of Coinbase’s QR code commercial during Super Bowl LVI, a number of brands, including Toyota, Michelob Ultra, and Mr. Peanut included QR codes as a way for viewers to get additional information or see more.

As we did last year, we again tracked DNS request traffic to our 1.1.1.1 resolver in United States data centers for domains associated with the advertised products or brands. Traffic growth is plotted against a baseline calculated as the mean request volume for the associated domains between 1200-1500 EST on Sunday, February 12 (Super Bowl Sunday.) Although over 50 brands advertised during the game, the brands highlighted below were chosen because their advertisements drove some of the largest percentage traffic spikes, as well as one interesting tale.

BlueMoon

Although the commercial initially seemed to be for sibling beer brands Coors Light and Miller Lite, there was a twist at the end, This twist was only fitting, as the ad was actually for Blue Moon, which is often served with a twist of orange on the rim of the glass. Although beer ads don’t usually drive significant traffic spikes, this one did, reaching 76,400% above baseline for Blue Moon’s site. Coors Light saw a 275% bump in DNS traffic coincident with the ad, while Miller Lite grew 120%. However, traffic for Coors and Miller was fairly volatile at other times during the game.

A look at Internet traffic trends during Super Bowl LVII

LimitBreak

Although last year’s advertisements included a number of cryptocurrency-related brands, they were all but absent from this year’s slate of ads. The closest we got during this year’s game was a commercial from LimitBreak, which describes itself as “bringing the free-to-play gaming experience to Web3 and beyond”, in which it promoted a giveaway of thousands of its Dragon series NFTs. This ad featured a QR code and a URL, and given the nearly 54,000% increase in DNS traffic observed, both were effective means of driving traffic to the LimitBreak website.

A look at Internet traffic trends during Super Bowl LVII

Temu

Upstart mobile shopping app Temu purchased multiple Super Bowl ad slots to promote its “shop like a billionaire” campaign, urging viewers to download its mobile app. As seen in the graph below, these advertisements drove spikes in traffic, and continued engagement, each time they ran. The first airing at 19:16 EST drove a 222% spike over baseline in DNS traffic. However, the second airing at 21:12 EST apparently resulted in significantly more interest, driving a 475% traffic increase. A third airing at 22:20 EST reached 169% over baseline, with another one just after that reaching over 200%.

A look at Internet traffic trends during Super Bowl LVII

Dunkin’

In early January, Boston-area media blew up with the news that local celebrity Ben Affleck was spotted working the drive-through window at one of the coffee chain’s Medford locations, raising some speculation that he was filming a Super Bowl commercial. That speculation turned out to be true, as the commercial aired at 18:53 EST. But the commercial had a side effect: DNS traffic for dunkin.com, associated with DunkinWorks (a small personal coaching and training business), spiked 8,000% when the commercial aired, as shown in the graph below. (It isn’t clear what drove the later three spikes for dunkin.com, as the advertisement didn’t air again nationally during the remainder of the game.) We can only hope that the dunkin.com system administrators were fueled with plenty of coffee and donuts as they dealt with the rapid growth in traffic.

A look at Internet traffic trends during Super Bowl LVII

Site categories: touchdowns bring attention

As we saw last year, there are two factors that bring a surge of traffic to the websites of Super Bowl participants: touchdowns and winning. However, nothing is more impactful than the sweet taste of victory. Both the Kansas City Chiefs’ and Philadelphia Eagles’ websites experienced a surge in DNS traffic just before the game started, as compared to a baseline calculated as the mean request volume for the associated domains between 12:00-15:00 EST on Sunday, February 12 (Super Bowl Sunday.). The Eagles website had its peak just around the time of the kickoff, with 828% growth over baseline, and continued to grow more rapidly than traffic to the Chiefs’ website until 20:55 EST, when traffic to chiefs.com began to pull ahead.

What happened at that time? That was the moment of the Chiefs’ third touchdown of the game, when DNS traffic to the team’s website had its first peak of the evening, at 514% above baseline. There was a clear spike during another Chiefs touchdown at 21:42 EST, at 454% above baseline, but that was nothing compared to the end of the game, when the Kansas City Chiefs were once again, after their 2019 victory, the winners. At 22:15 EST, when the game ended, DNS traffic to the Chiefs’ website was 871% higher, and peaked 10 minutes later at 890%, as compared to the baseline. At this same time, DNS traffic for the Eagles’ website dropped significantly. As we saw last year as well, winning the Super Bowl clearly drives increased traffic to the victor’s website.

A look at Internet traffic trends during Super Bowl LVII

Sports websites trends also followed the in-game events. There was a clear spike to approximately 90% above baseline when the game started at 18:30 EST, with further growth to 120% over baseline at 19:00 EST during the Kansas City Chiefs’ first touchdown. There were also clear spikes at 21:30 and 21:40 EST coinciding with the two more Chiefs touchdowns. The Super Bowl peak for these websites was reached during the final break at 22:00 EST, reaching 145% above baseline, just before the Chiefs’ game-winning field goal. After a brief drop as the game ended, there was an additional spike to 134%.

A look at Internet traffic trends during Super Bowl LVII

Rihanna’s impact on messaging and social media sites

What happened following Rihanna’s performance during the Super Bowl halftime show? As the game resumed, we saw a clear increase in traffic for messaging websites, with a first peak right after the end of the show at around 20:45 EST, 22% over baseline. The biggest peak, however, was when the game ended. At 22:15 EST, DNS traffic for messaging sites was 30% higher than the earlier baseline.

A look at Internet traffic trends during Super Bowl LVII

Rihanna’s announcement of her second pregnancy, which made news after her performance, also impacted traffic to social media platforms. After a small increase when halftime started, there was a clear drop during Rihanna’s show, followed by a jump from 6% below baseline back to 0% right after the show. An additional 3% of traffic growth was reached during the final break at 22:00 EST, just before the Kansas City Chiefs’ winning field goal. After a brief drop, traffic reached 2% above baseline as the game ended.

A look at Internet traffic trends during Super Bowl LVII

Is halftime also a time for rewatching ads?

The arrival of halftime at 20:21 EST also brought a surge in DNS traffic for video platforms. The first peak was reached at 18:00 EST, before the game started, at 12% above baseline. The peak during halftime was reached at 20:25 EST with 13% growth above baseline, suggesting that viewers may have been looking at that time to Super Bowl related videos or just using the time to browse those platforms.

A look at Internet traffic trends during Super Bowl LVII

Food delivery websites saw flat to lower DNS traffic just before the game as compared to the earlier baseline, suggesting that food orders were placed/scheduled earlier in the afternoon, hours before the game. At kickoff, traffic was 19% below baseline, but there was a clear spike at the time of the first break and right after the first Kansas City touchdown at 18:55 EST. After falling again during the game, there was a small increase in traffic observed just after the game ended.

A look at Internet traffic trends during Super Bowl LVII

What about betting sites? They expected a big day during the Super Bowl, given that more states have recently legalized gambling on sports. The peak was reached at 19:00 EST, as DNS traffic reached 295% over baseline, when the Chiefs had their first touchdown, The first Eagles touchdown, minutes before, resulted in a 233% spike. The lowest traffic for betting sites during the Super Bowl was during the halftime show. In the second half of the game, two other clear spikes in traffic are visible. The first was at 20:55 EST at 167% above baseline when the Chiefs pulled ahead with a touchdown, and then a jump to 278% over baseline when the game ended.

A look at Internet traffic trends during Super Bowl LVII

Rihanna runs this town city

While the so-called NFL cities across the country are loyal to their local teams, looking at traffic trends across cities from both conferences makes it clear that fans everywhere find joy, not division, in the unknown pleasures of a good halftime show. The drop visible in both graphs below between 20:30-20:50 EST coincides with Rihanna’s return to live performance, as she last performed live in January 2018. Based on the observed drop in traffic, viewers apparently turned away from their computers and devices, giving their attention to Rihanna, or at least stopped their general Internet surfing during the halftime show. As the graphs show, traffic recovered as soon as halftime was over.

A look at Internet traffic trends during Super Bowl LVII
A look at Internet traffic trends during Super Bowl LVII

Zooming in to individual cities, we examined the traffic patterns observed in both Philadelphia and Kansas City. While both teams have fans across the country, we can use their home cities as a proxy. In this case, we compared normalized Internet traffic levels between 17:00-22:30 EST on Super Bowl Sunday (February 12) with the same time frame on the prior Sunday (February 5).

In Kansas City last Sunday, traffic volumes remained fairly consistent across the surveyed time period. However, on Super Bowl Sunday, traffic levels were initially similar, but by the start of the game were 84% lower than the same time the previous week. Slight drops in traffic are visible coincident with Chiefs touchdowns, but don’t stand out from the overall noisiness of the graph. The graph reached its nadir at 22:13 EST when the Chiefs broke the tie and kicked the game-winning field goal, with the significant drop in traffic likely due to an increased shift in focus towards the outcome of the game, even by those that hadn’t previously been paying close attention.

A look at Internet traffic trends during Super Bowl LVII

As the graph below shows, last Sunday saw Internet traffic in Philadelphia gradually decline as the evening wore on. On Super Bowl Sunday, traffic started out slightly lower than the week prior, and also diverged as game time approached, reaching nearly 50% lower at kickoff. As the Eagles took an early lead, their first touchdown resulted in a noticeable drop in traffic from Philadelphia, seen at 18:52 EST, less than 10 minutes after the start of the game. Visible drops in traffic are also coincident with the Eagles’ other three touchdowns, although they don’t stand out against the volatility of the graph. Traffic began to drop towards the end of the game, as the tie score added tension, and reached its lowest point when it became clear that the Eagles were not going to emerge victorious in Super Bowl LVII.

A look at Internet traffic trends during Super Bowl LVII

In addition to looking at traffic impacts at a city level, we can also zoom out to examine Internet traffic trends in the Super Bowl states. Arizona, which hosted the big game at State Farm Stadium in Glendale, saw a drop in state-level traffic starting around 13:00 EST. At the time of the kickoff, traffic was 25% lower than the previous Sunday, but the biggest impact was during the wildly popular halftime show by Rihanna. At 20:30 EST, traffic was 29% lower than the same time on the previous Sunday. After the game ended, traffic levels returned to normal around 23:30 EST.

A look at Internet traffic trends during Super Bowl LVII

In Pennsylvania, home of the Philadelphia Eagles, traffic began to dip after 15:00 EST and reached its first low point around kickoff, when it was 28% lower than the previous Sunday. Just like in Arizona, the biggest difference was during Rihanna’s halftime show, when it was a whopping 33% lower than usual. However, just a few minutes after the game ended at 22:30 EST, traffic returned to normal.

A look at Internet traffic trends during Super Bowl LVII

What about the winning team’s state of Missouri? There, traffic started to decrease only after 17:00 EST and was actually higher than the previous Sunday before that point. With the kickoff came a clear drop, resulting in 28% less traffic than the previous Sunday at the same time. Traffic increased a bit heading towards halftime, but dropped again during Rihanna’s show, when it was 30% lower than usual. The biggest drop in traffic, not surprisingly, was during the exciting moment of the Kansas City Chiefs’ winning field goal. At 22:15 EST, traffic was 33% lower than the previous Sunday. However, after 22:50 EST, Internet traffic in Missouri was back on the fast track, with traffic increasing to levels higher than the previous Sunday.

A look at Internet traffic trends during Super Bowl LVII

Rihanna’s halftime performance had a clear impact on Internet traffic at a state level, which dropped across all states with NFL teams at the time of her show. Below we take a closer look at the most populous states, among which Pennsylvania, New York and Arizona were winners, with the largest traffic declines. The impacts in Pennsylvania and Arizona are shown above, and the graph below shows the traffic trends seen in New York.

A look at Internet traffic trends during Super Bowl LVII

California, Texas, Florida, and New York all had their fair share of Internet traffic dropping before and throughout the game, but it was during the halftime show when things really got interesting. At the time of Rihanna’s performance, Internet traffic in California was 24% lower than the previous Sunday, while in Texas it was 21% below a week earlier, and Florida also saw a 21% drop. Meanwhile, New York had a clear 30% decrease in traffic during the show and, as shown above, Pennsylvania took the cake with a 33% drop. Illinois, Ohio, Georgia, North Carolina, and Michigan were close behind with 23%, 27%, 22%, 25%, and 22% drops respectively.

This seems to be a clear indication that the Super Bowl in general, but also the much-anticipated halftime shows, and the winning celebrations, all have a massive impact on the Internet, causing a noticeable dip in Internet traffic, especially in the state of the winning team.

Do email spammers and scammers take advantage of “The Big Game”?

Spammers and scammers will frequently try to take advantage of the popularity of major events when running their campaigns, hoping the tie-in will entice the user to open the message and click on a malicious link, or visit a malicious website where they give up a password or credit card number. Cloudflare Area 1 Email Security analyzed the subject lines of email messages processed by the service in the weeks leading up to the Super Bowl to identify malicious, suspicious, and spam messages across four topic areas: Super Bowl/football, sports gambling, sports media/websites, and food delivery.

As the “regular” season NFL games wrapped up, Super Bowl and football themed email threat volume remained relatively low. However, campaigns clearly picked up between January 23-29 as the message count grew sevenfold. However, campaigns kicked into high gear once the Chiefs and Eagles were headed to the Super Bowl, as the number of identified messages between January 30 and February 5 was nearly six times higher than the previous week. These campaigns quickly ended in the week before the big game, though, as Super Bowl and football themed suspicious, malicious, and spam email volume dropped by nearly 90%.

A look at Internet traffic trends during Super Bowl LVII

Overall, the number of sports gambling themed subject lines remained fairly low over the survey period. This is somewhat surprising, given that an increasing number of US states have recently legalized betting on sporting events. Interestingly, the trend was highest at the beginning of the year, although that first week was too late to capture potential interest in college football “bowl” games. However, the weeks ahead of the NFL conference championship games (January 23-29) and the Super Bowl (February 6-12) saw message volume increase to levels nearly 2.5x higher than previous weeks.

A look at Internet traffic trends during Super Bowl LVII

Sports media and website themed suspicious, malicious, and spam email messages apparently don’t draw the clicks, because the volume of such messages seen by Cloudflare Area 1 has remained extremely low since the start of the year, but peaked during the week of January 23-29. And although lower in volume, the observed trends were similar to those seen for sports gambling, with peaks during the same weeks.

A look at Internet traffic trends during Super Bowl LVII

For many people, the Super Bowl is less about the football game than it is about the commercials and the food, and the growth of food delivery services over the last few years have made it easier to ensure that the snacks and libations never run out during the game. Scammers and spammers have apparently learned to take advantage of this hunger, as food delivery themed email messages saw the highest counts across the four categories reviewed here. Peak message counts were seen the weeks of January 2-8 and January 30-February 5. Message volume the weeks following these peaks fell by over 50% in both cases.

A look at Internet traffic trends during Super Bowl LVII

Conclusion

As we have seen time and again, advertising during the Super Bowl can drive significant traffic spikes, and apparently this holds true even if a URL isn’t included as a call to action within the commercial. In addition, the trends observed during the game remain a clear reminder that human behavior drives Internet traffic, especially when the halftime show features a popular singer that last performed live five years ago.

Visit Cloudflare Radar for up to date Internet traffic and attack trends, and follow the Cloudflare Radar Twitter and Mastodon accounts for regular insights on Internet events.

Internet disruptions overview for Q4 2022

Post Syndicated from David Belson original https://blog.cloudflare.com/q4-2022-internet-disruption-summary/

Internet disruptions overview for Q4 2022

Internet disruptions overview for Q4 2022

Cloudflare operates in more than 250 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions.

While Internet disruptions are never convenient, online interest in the 2022 World Cup in mid-November and the growth in online holiday shopping in many areas during November and December meant that connectivity issues could be particularly disruptive. Having said that, the fourth quarter appeared to be a bit quieter from an Internet disruptions perspective, although Iran and Ukraine continued to be hotspots, as we discuss below.

Government directed

Multi-hour Internet shutdowns are frequently used by authoritarian governments in response to widespread protests as a means of limiting communications among protestors, as well preventing protestors from sharing information and video with the outside world. During the fourth quarter Cuba and Sudan again implemented such shutdowns, while Iran continued the series of “Internet curfews” across mobile networks it started in mid-September, in addition to implementing several other regional Internet shutdowns.

Cuba

In late September, Hurricane Ian knocked out power across Cuba. While officials worked to restore service as quickly as possible, some citizens responded to perceived delays with protests that were reportedly the largest since anti-government demonstrations over a year earlier. In response to these protests, the Cuban government reportedly cut off Internet access several times. A shutdown on September 29-30 was covered in the Internet disruptions overview for Q3 2022, and the impact of the shutdown that occurred on October 1 (UTC) is shown in the figure below. The timing of this one was similar to the previous one, taking place between 1900 on September 30 and 0245 on October 1 (0000-0745 UTC on October 1).

Internet disruptions overview for Q4 2022

Sudan

October 25 marked the first anniversary of a coup in Sudan that derailed the country’s transition to civilian rule, and thousands of Sudanese citizens marked the anniversary by taking to the streets in protest. Sudan’s government has a multi-year history of shutting down Internet access during times of civil unrest, and once again implemented an Internet shutdown in response to these protests. The figure below shows a near complete loss of Internet traffic from Sudan on October 25 between 0945-1740 local time (0745 – 1540 UTC).

Internet disruptions overview for Q4 2022

Iran

As we covered in last quarter’s blog post, the Iranian government implemented daily Internet “curfews”, generally taking place between 1600 and midnight local time (1230-2030 UTC) across three mobile network providers — AS44244 (Irancell), AS57218 (RighTel), and AS197207 (MCCI) — in response to protests surrounding the death of Mahsa Amini. These multi-hour Internet curfew shutdowns continued into early October, with additional similar outages also observed on October 8, 12 and 15 as seen in the figure below. (The graph’s line for AS57218 (Rightel), the smallest of the three mobile providers, suggests that the shutdowns on this network were not implemented after the end of September.)

Internet disruptions overview for Q4 2022

In addition to the mobile network shutdowns, several regional Internet disruptions were also observed in Iran during the fourth quarter, two of which we review below. The first was in Sanandaj, Kurdistan Province on October 26, where a complete Internet shutdown was implemented in response to demonstrations marking the 40th day since the death of Mahsa Amini. The figure below shows a complete loss of traffic starting at 1030 local time (0700 UTC), with the outage lasting until 0805 local time on October 27 (0435 UTC). In December, a province-level Internet disruption was observed starting on December 18, lasting through December 25.

Internet disruptions overview for Q4 2022
Kurdistan Province, Iran. (Source: Map data ©2023 Google, MapaGISrael)

Internet disruptions overview for Q4 2022
Internet disruptions overview for Q4 2022

The Internet disruptions that have taken place in Iran over the last several months have had a significant economic impact on the country. A December post from Filterwatch shared concerns stated in a letter from mobile operator Rightel:

The letter, signed by the network’s Managing Director Yasser Rezakhah, states that “during the past few weeks, the company’s resources and income have significantly decreased during Internet shutdowns and other restrictions, such as limiting Internet bandwidth from 21 September. They have also caused a decrease in data use from subscribers, decreasing data traffic by around 50%.” The letter also states that the “continued lack of compensation for losses could lead to bankruptcy.”

The post also highlighted economic concerns shared by Iranian officials:

Some Iranian officials have expressed concern about the cost of Internet shutdowns, including Valiollah Bayati, MP for Tafresh and Ashtian in Markazi province. In a public session in Majles (parliament), he stated that continued Internet shutdowns have led to the closure of many jobs and people are worried, the government and the President must provide necessary measures.

Statistics in an article on news site enthkhab.ir provide a more tangible view of the local economic impact, stating (via Google Translate):

Since the 30th of Shahrivar month and with the beginning of the government disruption in the Internet, the country’s businesses have been damaged daily at least 50 million tomans and at most 500 million tomans. More than 41% of companies have lost 25-50% of their income during this period, and about 47% have had more than 50% reduction in sales. A review of the data of the research assistant of the country’s tax affairs organization shows that the Internet outage in Iran has caused 3000 billion tomans of damage per day. That is, the cost of 3 months of Internet outage in Iran is equal to 43% of one year’s oil revenue of the country ($25 billion).

Power outages

Bangladesh, October 4

Over 140 million people in Bangladesh were left without electricity on October 4 as the result of a reported grid failure caused by a failure by power distribution companies to follow instructions from the National Load Dispatch Centre to shed load. The resultant power outage resulted in an observed drop in Internet traffic from the country, starting at 1405 local time (0805 UTC), as shown in the figure below. The disruption lasted approximately seven hours, with traffic returning to expected levels around 1900 local time (1500 UTC).

Internet disruptions overview for Q4 2022

Pakistan

Over a week later, a similar issue in Pakistan caused power outages across the southern part of the country, including Sindh, Punjab, and Balochistan. The power outages were caused by a fault in the national grid’s southern transmission system, reportedly due to faulty equipment and sub-standard maintenance. As expected, the power outages resulted in disruptions to Internet connectivity, and the figure below illustrates the impact observed in Sindh, where traffic dropped nearly 30% as compared to the previous week starting at 0935 local time (0435 UTC) on October 6. The disruption lasted over 15 hours, with traffic returning to expected levels at 0100 on October 7 (2000 UTC on October 6).

Internet disruptions overview for Q4 2022
Sindh, Pakistan (Source: Map data ©2023 Google)

Internet disruptions overview for Q4 2022

Kenya

On November 24, a Tweet from Kenya Power at 1525 local time noted that they had “lost bulk power supply to various parts of the country due to a system disturbance”. A subsequent Tweet published just over six hours later at 2150 local time stated that “normal power supply has been restored to all parts of the country.” The time stamps on these notifications align with the loss of Internet traffic visible in the figure below, which lasted between 1500-2050 local time (1200-1750 UTC).

Internet disruptions overview for Q4 2022

United States (Moore County, North Carolina)

On December 3, two electrical substations in Moore County, North Carolina were targeted by gunfire, with the resultant damage causing localized power outages that took multiple days to resolve. The power outages reportedly began just after 1900 local time (0000 UTC on December 4), resulting in the concurrent loss of Internet traffic from communities within Moore County, as seen in the figure below.

Internet traffic within the community of West End appeared to return midday (UTC) on December 5, but that recovery was apparently short-lived, as it fell again during the afternoon of December 6. In Pinehurst, traffic began to slowly recover after about a day, but returned to more normal levels around 0800 local time (1300 UTC) on December 7.

Internet disruptions overview for Q4 2022
West End and Pinehurst, North Carolina. (Source: Map data ©2023 Google)

Internet disruptions overview for Q4 2022

Ukraine

The war in Ukraine has been going on since February 24, and Cloudflare has covered the impact of the war on the country’s Internet connectivity in a number of blog posts across the year (March, March, April, May, June, July, October, December). Throughout the fourth quarter of 2022, Russian missile strikes caused widespread damage to electrical infrastructure, resulting in power outages and disruptions to Internet connectivity. Below, we highlight several examples of the Internet disruptions observed in Ukraine during the fourth quarter, but they are just a few of the many disruptions that occurred.

On October 20, the destruction of several power stations in Kyiv resulted in a 25% drop in Internet traffic from Kyiv City as compared to the two previous weeks. The disruption began around 0900 local time (0700 UTC).

Internet disruptions overview for Q4 2022
Kviv City, Ukraine. (Source: Map data ©2023 Google)

Internet disruptions overview for Q4 2022

On November 23, widespread power outages after Russian strikes caused a nearly 50% decrease in Internet traffic in Ukraine, starting just after 1400 local time (1200 UTC). This disruption lasted for nearly a day and a half, with traffic returning to expected levels around 2345 local time on November 24 (2145 UTC).

Internet disruptions overview for Q4 2022

On December 16, power outages resulting from Russian air strikes targeting power infrastructure caused country-level Internet traffic to drop around 13% at 0915 local time (0715 UTC), with the disruption lasting until midnight local time (2200 UTC). However, at a network level, the impact was more significant, with AS13188 (Triolan) seeing a 70% drop in traffic, and AS15895 (Kyivstar) a 40% drop, both shown in the figures below.

Internet disruptions overview for Q4 2022
Internet disruptions overview for Q4 2022
Internet disruptions overview for Q4 2022

Cable cuts

Shetland Islands, United Kingdom

The Shetland Islands are primarily dependent on the SHEFA-2 submarine cable system for Internet connectivity, connecting through the Scottish mainland. Late in the evening of October 19, damage to this cable knocked the Shetland Islands almost completely offline. At the time, there was heightened concern about the potential sabotage of submarine cables due to the reported sabotage of the Nord Stream natural gas pipelines in late September, but authorities believed that this cable damage was due to errant fishing vessels, and not sabotage.

The figure below shows that the impact of the damage to the cable was relatively short-lived, compared to the multi-day Internet disruptions often associated with submarine cable cuts. Traffic dropped just after 2300 local time (2200 UTC) on October 19, and recovered 14.5 hours later, just after 1430 local time (1330 UTC) on October 20.

Internet disruptions overview for Q4 2022
Shetland Islands, United Kingdom. (Source: Map data ©2023 GeoBasis-DE/BKG (©2009), Google)

Internet disruptions overview for Q4 2022

Natural disasters

Solomon Islands

Earthquakes frequently cause infrastructure damage and power outages in affected areas, resulting in disruptions to Internet connectivity. We observed such a disruption in the Solomon Islands after a magnitude 7.0 earthquake occurred near there on November 22. The figure below shows Internet traffic from the country dropping significantly at 1300 local time (0200 UTC), and recovering 11 hours later at around 2000 local time (0900 UTC).

Internet disruptions overview for Q4 2022

Technical problems

Kyrgyzstan

On October 24, a three-hour Internet disruption was observed in Kyrgyzstan lasting between 1100-1400 local time (0500-0800 UTC), as seen in the figure below. According to the country’s Ministry of Digital Development, the issue was caused by “an accident on one of the main lines that supply the Internet”, but no additional details were provided regarding the type of accident or where it had occurred.

Internet disruptions overview for Q4 2022

Australia (Aussie Broadband)

Customers of Australian broadband Internet provider Aussie Broadband in Victoria and New South Wales suffered brief Internet disruptions on October 27. As shown in the figure below, AS4764 (Aussie Broadband) traffic from Victoria dropped by approximately 40% between 1505-1745 local time (0405-0645 UTC). A similar, but briefer, loss of traffic from New South Wales was also observed, lasting between 1515-1550 local time (0415-0450 UTC). A representative of Aussie Broadband provided insight into the underlying cause of the disruption, stating “A config change was made which was pushed out through automation to the DHCP servers in those states. … The change has been rolled back but getting the sessions back online is taking time for VIC, and we are now manually bringing areas up one at a time.”

Internet disruptions overview for Q4 2022
Victoria and New South Wales, Australia. (Source: Map data ©2023 Google)

Internet disruptions overview for Q4 2022

Haiti

In Haiti, customers of Internet service provider Access Haiti experienced disrupted service for more than half a day on November 9. The figure below shows that Internet traffic for AS27759 (Access Haiti) fell precipitously around midnight local time (0500 UTC), remaining depressed until 1430 local time (1930 UTC), at which time it recovered quickly. A Facebook post from Access Haiti explained to customers that “Due to an intermittent outage on one of our international circuits, our network is experiencing difficulties that cause your Internet service to slow down.” While Access Haiti didn’t provide additional details on which international circuit was experiencing an outage, submarinecablemap.com shows that two submarine cables provide international Internet connectivity to Haiti — the Bahamas Domestic Submarine Network (BDSNi), which connects Haiti to the Bahamas, and Fibralink, which connects Haiti to the Dominican Republic and Jamaica.

Internet disruptions overview for Q4 2022

Unknown

Many Internet disruptions can be easily tied to an underlying cause, whether through coverage in the press, a concurrent weather or natural disaster event, or communication from an impacted provider. However, the causes of other observed disruptions remain unknown as the impacted providers remain silent about what caused the problem.

United States (Wide Open West)

On November 15, customers of Wide Open West, an Internet service provider with a multi-state footprint in the United States, experienced an Internet service disruption that lasted a little over an hour. The figure below illustrates the impact of the disruption in Alabama and Michigan on AS12083 (Wide Open West), with traffic dropping at 1150 local time (1650 UTC) and recovering just after 1300 local time (1800 UTC).

Internet disruptions overview for Q4 2022

Cuba

Cuba is no stranger to Internet disruptions, whether due to government-directed shutdowns (such as the one discussed above), fiber cuts, or power outages. However, no underlying cause was ever shared for the seven-hour disruption in the country’s Internet traffic observed between 2345 on November 25 and 0645 on November 26 local time (0445-1145 UTC on November 26). Traffic was down as much as 75% from previous levels during the disruption.

Internet disruptions overview for Q4 2022

As a provider of low earth orbit (LEO) satellite Internet connectivity services, disruptions to SpaceX Starlink’s service can have a global impact. On November 30, a disruption was observed on AS14593 (SPACEX-STARLINK) between 2050-2130 UTC, with traffic volume briefly dropping to near zero. Unfortunately, Starlink did not acknowledge the incident, nor did they provide any reason for the disruption.

Internet disruptions overview for Q4 2022

Conclusion

Looking back at the Internet disruptions observed during 2022, a number of common themes can be found. In countries with more authoritarian governments, the Internet is often weaponized as a means of limiting communication within the country and with the outside world through network-level, regional, or national Internet shutdowns. As noted above, this approach was used aggressively in Iran during the last few months of the year.

Internet connectivity quickly became a casualty of war in Ukraine. Early in the conflict, network-level outages were common, and some Ukrainian networks ultimately saw traffic re-routed through upstream Russian Internet service providers. Later in the year, as electrical power infrastructure was increasingly targeted by Russian attacks, widespread power outages resulted in multi-hour disruptions of Internet traffic across the country.

While the volcanic eruption in Tonga took the country offline for over a month due to its reliance on a single submarine cable for Internet connectivity, the damage caused by earthquakes in other countries throughout the year resulted in much shorter and more limited disruptions.

And while submarine cable issues can impact multiple countries along its route, the advent of services with an increasingly global footprint like SpaceX Starlink mean that service disruptions will ultimately have a much broader impact. (Starlink’s subscriber base is comparatively small at the moment, but it currently has a service footprint in over 30 countries around the world.)

To follow Internet disruptions as they occur, check the Cloudflare Radar Outage Center (CROC) and follow @CloudflareRadar on Twitter. To review those disruptions observed earlier in 2022, refer to the Q1, Q2, and Q3 Internet disruptions overview blog posts.

Cloudflare Radar 2022 Year in Review

Post Syndicated from David Belson original https://blog.cloudflare.com/radar-2022-year-in-review/

Cloudflare Radar 2022 Year in Review

Cloudflare Radar 2022 Year in Review

In 2022, with nearly five billion people around the world (as well as an untold number of “bots”) using the Internet, analyzing aggregate data about this usage can uncover some very interesting trends. To that end, we’re excited to present the Cloudflare Radar 2022 Year In Review, featuring interactive charts, graphs, and maps you can use to explore notable Internet trends observed throughout this past year. The Year In Review website is part of Cloudflare Radar, which celebrated its second birthday in September with the launch of Radar 2.0.

We have organized the trends we observed around three different topic areas: Traffic, Adoption, and Security. The content covered within each of these areas is described in more detail in their respective sections below. Building on the 2021 Year In Review, we have incorporated several additional metrics this year, and have also improved the underlying methodology. (As such, the charts are not directly comparable to develop insights into year-over-year changes.)

Website visualizations shown at a weekly granularity cover the period from January 2 through November 26, 2022 (the start of the first full week of the year through the end of the last full week of November). We plan to update the underlying data sets through the end of the year in early 2023. Trends for nearly 200 locations are available on the website, with some smaller or less populated locations excluded due to insufficient data.

Before we jump in, we urge anyone who prefers to see the headline stats up front and to explore the data themselves to go ahead and visit the website. Anyone who wants a more lengthy, but curated set of observations should continue reading below. Regardless, we encourage you to consider how the trends presented within this post and the website’s various sections impact your business or organization, and to think about how these insights can inform actions that you can take to improve user experience or enhance your security posture.

Traffic

Cloudflare Radar 2022 Year in Review

Anyone following recent technology headlines might assume that the Internet’s decades-long trend of incredible growth would have finally begun to falter. In times like these, data is key. Our data indicates that global Internet traffic, which grew at 23% this year, is as robust as ever.

To determine the traffic trends over time, we first established a baseline, calculated as the average daily traffic volume (excluding bot traffic) over the second full calendar week (January 9-15) of 2022. We chose the second calendar week to allow time for people to get back into their “normal” routines (school, work, etc.) after the winter holidays and New Year’s Day. The percent change shown on the trend lines in our charts are calculated relative to the baseline value, and represents a seven-day trailing average — it does not represent absolute traffic volume for a location. The seven-day averaging is done to smooth the sharp changes seen with a daily granularity.

In addition to calculating traffic growth, our 1.1.1.1 public DNS resolver and broad global customer base enables us to have a unique view into online activity. This includes insights into the most popular types of Internet content and the most popular Internet services in general and across specific categories, as well as the impact of bots. Of course, none of this matters if connectivity is unavailable, so we also drill down into major Internet disruptions observed in 2022.

After an initial dip, worldwide Internet traffic saw nominal growth coinciding with the 2022 Olympic Winter Games in Beijing, but slipped again in the weeks after their conclusion. After a couple of months of slight growth, traffic again dipped below baseline heading into July. However, after reaching that nadir, Internet traffic experienced a fairly consistent rate of growth through the back part of the year. An upwards inflection at the end of November is visible in the worldwide traffic graph as well as the traffic graphs of a number of locations. Traffic analysis showed that this increase resulted from the convergence of early holiday shopping traffic (to e-commerce sites) with the run-up to and early days of FIFA World Cup Qatar 2022.

Cloudflare Radar 2022 Year in Review

The An Update on Cloudflare’s assistance to Ukraine blog post published during Impact Week looked at the conflict from an attack perspective. Viewing Ukraine through an Internet traffic lens provides unique insights into the impacts of the war’s damage and destruction to Internet connectivity within the country. After starting the year with some nominal traffic growth, that trend was quickly reversed once the Russian invasion began on February 24, with traffic quickly falling as infrastructure was damaged and the populace focused on finding safety and shelter. Although traffic started to grow again after that initial steep decline, drops in May and June appear to be correlated with significant outages observed by Cloudflare. After returning to growth during August, several additional disruptions were visible in September, October, and November coincident with widespread power outages across the country resulting from Russian attacks.

Cloudflare Radar 2022 Year in Review

Reliable electric power is critical for reliable Internet connectivity, both for the core network infrastructure in data centers, as well as for last-mile infrastructure like cell towers and Wi-Fi routers, as well as laptops, cellphones, and other devices used to access the Internet. For several years, the residents of Puerto Rico have struggled to contend with an unreliable electric grid, resulting in frequent power outages and slow restoration times. In 2022, the island suffered two multi-day power outages that clearly impacted otherwise strong traffic growth. In April, a fire at a power plant caused an outage that lasted three days, disrupting Internet connectivity during that period. In September, widespread power outages resulting from damage from Hurricane Fiona resulted in a rapid drop in Internet traffic with the disruption lasting over a week until power restoration work and infrastructure repair was completed.

Cloudflare Radar 2022 Year in Review

Top categories

Cloudflare’s global customer base spans a range of industry categories, including technology, e-commerce, and entertainment, among others. Analysis of the traffic to our customers’ websites and applications reveals which categories of content were most popular throughout the year, and can be broken out by user location. The domains associated with each customer zone have one or more associated categories — these can be viewed on Cloudflare Radar. To calculate the distribution of traffic across the set of categories for each location, we divided the number of requests for domains associated with a given category seen over the course of a week by the total number of requests mapped to a category seen over that week, filtering out bot traffic. If a domain is associated with multiple categories, then the associated request was included in the aggregate count for each category. The chart shows how the distribution of requests across the selected categories changes over the course of the year.

Globally, sites in the Technology category were the most popular, accounting for approximately one-third of traffic throughout the year. The next most popular category was Business & Economy, which drove approximately 15% of traffic. Shopping & Auctions also saw a bump in traffic in November, as consumers began their holiday shopping.

Cloudflare Radar 2022 Year in Review

In sharp contrast to other Asian countries, in South Korea, Internet Communication was consistently the second most popular category during the year. Elsewhere, Internet Communication was occasionally among the top five, but usually within the top 10. Internet Communication was followed closely by Entertainment and Business & Economy. The former saw multiple periods of increased traffic through the year, in contrast to other categories, which saw traffic share remain fairly consistent over time.

Traffic distribution in Turkey represented a rare departure from most other locations around the world. Although Technology started the year as the most popular category, its popularity waned during the back half of the year, ending below Shopping & Auctions and Society & Lifestyle. These latter two saw gradual growth starting in September, and posted larger increases in November. Business & Economy and Entertainment sites were comparatively less popular here, in contrast to many other locations.

Armenia’s traffic distribution also ran counter to that seen in most other locations. Entertainment was the most popular category for nearly the entire year, except for the final week of November. Technology was generally the second most popular category, although it was surpassed by Gambling several times throughout the year. However, Gambling saw its popularity fall significantly in November, as it was surpassed by the Shopping & Auctions and Business & Economy categories.

The luxury of being a popular Internet service is that the service’s brand becomes very recognizable, so it will be no surprise that Google was #1 in our General ranking.

Top 10 — General, late 2022 ranking
1. Google
2. Facebook
3. Apple, TikTok (tie)
5. YouTube
6. Microsoft
7. Amazon Web Services
8. Instagram
9. Amazon
10. iCloud, Netflix, Twitter, Yahoo (tie)

Last year TikTok was at the top of our ranking. However, the results between the two years aren’t comparable. As part of our launch of Radar 2.0, we introduced improvements to our domain ranking algorithms, and this year’s rankings are based on those new algorithms. In addition, this year we have grouped domains that all belong to a single Internet service. For example, Google operates google.com, google.pt, and mail.google.com among others, so we aggregated the popularity of each domain under a single “Google” Internet service for simplicity. However, while Meta operates both Facebook and Instagram, consumers typically perceive those brands as distinct, so we decided to group domains associated with those services separately.

Zooming out from our General top 10, the anonymized DNS query data from our 1.1.1.1 public DNS resolver reflects traffic from millions of users around the world, enabling us to offer category specific rankings as well. While you can view them all in the “Most popular Internet services” section of our Year in Review website, we’ve decided to highlight a few of our favorite observations below.

Cryptocurrencies always seem to have as much promise as they have controversy. We couldn’t help but be curious about which cryptocurrency services were the most popular. But before jumping into the Top 10, let’s double-click on one that fell out of the running: FTX. Known as the third largest cryptocurrency exchange in the world, our popularity ranking shows it hovered around 9th place for most of the year. That is, until it filed for bankruptcy in November. At that point, there is a precipitous drop, which also appears to coincide with reports that FTX disabled its users’ ability to make cryptocurrency withdrawals. Moving back to the Top 10, the two other major cryptocurrency exchanges, Binance and Coinbase, ranked #1 and #3 respectively and don’t appear to have been adversely impacted by FTX in our rankings.

Cloudflare Radar 2022 Year in Review

The universe has been the hottest place to be since the beginning of time, but some suggest that we’ll all soon be in the metaverse. If that’s true, then the question becomes “Whose metaverse?”. Last year, Facebook changed its name to Meta as it poured billions of dollars into the space, so we were curious about the impact of their efforts on the metaverse landscape one year later. With Meta’s Oculus offering their initial foray into the metaverse, our data indicates that while its popularity saw tangible improvements, rising from 10th to 5th in the back half of the year, Roblox is clearly the champion of the metaverse arena. It is fascinating to see this smaller challenger dominating Oculus, which is operated by Meta, a company ~18x larger in market capitalization. We are excited to check back at the end of 2023 to see whether Oculus’ ascent of the rankings topples Roblox, or if the smaller player retains the crown.

Cloudflare Radar 2022 Year in Review

Facebook’s transition to Meta, however, does not appear to have impacted its popularity as a social media platform. Within our ranking of the top social media platforms, Facebook held the top position throughout the year. TikTok and Snapchat also held steady in their places among the top five. Instagram and Twitter traded places several times mid-year, but the photo and video sharing app ultimately knocked Twitter from 3rd place in August. More active volatility was seen in the bottom half of the top 10, as LinkedIn, Discord, and Reddit frequently shifted between sixth, seventh, and eighth position in the rankings.

Cloudflare Radar 2022 Year in Review

While those are the most popular sites today, over the last 20+ years, the landscape of social media platforms has been quite dynamic, with new players regularly emerging. Some gained a foothold and became successful, while others became a footnote of Internet history. Although it has actually been around since 2016, Mastodon emerged as the latest potential disruptor in the space. In a landscape where the top social media platforms operate closed-source, centralized platforms, Mastodon offers free, open source software to allow anyone to start their own social networking platform, built around a decentralized architecture, and easily federated with others.

Aggregating the domain names used by 400 top Mastodon instances, this cohort started the year hovering around the #200 rank of most popular services overall. Its position in the overall rankings steadily improved throughout the year, hitting an inflection point in November, moving up about 60 positions. This trend appears to be driven by a spike in interest and usage of Mastodon, which we elaborate on in the Adoption section below.

Cloudflare Radar 2022 Year in Review

Bot traffic

Bot traffic describes any non-human traffic to a website or an app. Some bots are useful, such as those that monitor site and application availability or search engine bots that index content for search, and Cloudflare maintains a list of verified bots known to perform such services. However, visibility into other non-verified bot activity is just as, if not more, important as they may be used to perform malicious activities, such as breaking into user accounts or scanning the web for exposed vulnerabilities to exploit. To calculate bot traffic percentages, we used the bot score assigned to each request to identify those made by bots, and then divided the total number of daily requests from these bots by the total number of daily requests. These calculations were done both globally and on a per-location basis. The line shown in the trends graph represents a seven-day trailing average. For the top 10 chart, we calculated the average bot percentage on a monthly basis per location, and then ranked the locations by percentage. The chart illustrates the ranking by month, and how those rankings change across the year.

Globally, bots generally accounted for between 30-35% of traffic over the course of the year. Starting January at around 35%, the percentage of bot traffic dropped by nearly a quarter through the end of February, but then reclaimed some of that loss, staying just above 30% through October. A slight downward trend is evident at the start of November, due to human traffic increasing while bot traffic remained fairly consistent. Despite a couple of nominal spikes/drops, the global trend exhibited fairly low volatility overall throughout the year.

Cloudflare Radar 2022 Year in Review

While around one-third of global traffic was from bots, two locations stood out with bot traffic percentages double the global level. Except for two brief mid-year spikes, just under 70% of traffic from Ireland was classified as bot-driven. Similarly, in Singapore, bot traffic consistently ranged between 60-70% across the year. Bots account for the majority share of traffic from these locations due to the presence of local “regions” from multiple cloud platform providers in each. Because doing so is easily automated and free/inexpensive, attackers will frequently spin up ephemeral instances in these clouds in order to launch high volume attacks, such as we saw with the “Mantis” attack in June. (Internal traffic analysis indicates that a significant portion of traffic for these two geographies is from cloud provider networks and that the vast majority of traffic we see from these networks is classified as bot traffic.)

Cloudflare Radar 2022 Year in Review

Cloudflare Radar 2022 Year in Review

The top 10 list of locations with the highest percentage of bot traffic saw a fair amount of movement throughout the year, with four different locations holding the top slot at some point during the year, although Turkmenistan spent the most time at the top of the list. Overall, 17 locations held a spot among the top 10 at some point during 2022, with greater concentrations in Europe and Asia.

Internet outages

Although the metrics included in the 2022 Year In Review were ultimately driven by Internet traffic to Cloudflare from networks and locations around the world, there are, unfortunately, times when traffic is disrupted. These disruptions can have a number of potential causes, including natural disasters and extreme weather, fiber optic cable cuts, or power outages. However, they can also happen when authoritarian governments order Internet connectivity to be shutdown at a network, regional, or national level.

We saw examples of all of these types of Internet disruptions, and more, during 2022, and aggregated coverage of them in quarterly overview blog posts. With the launch of Radar 2.0 in September, we also began to catalog them on the Cloudflare Radar Outage Center. These disruptions are most often visible as drops in Cloudflare traffic from a given network, region, or country. The 2022 Year In Review website illustrates where these disruptions occurred throughout the year. Some notable outages observed during 2022 are highlighted below.

One of the most significant Internet disruptions of the year took place on AS812 (Rogers), one of Canada’s largest Internet service providers. During the morning of July 8, a near complete loss of traffic was observed, and it took nearly 24 hours for traffic volumes to return to normal levels. A Cloudflare blog post covered the Rogers outage in real-time as the provider attempted to restore connectivity. Data from APNIC estimates that as many as five million users were directly affected, while press coverage noted that the outage also impacted phone systems, retail point of sale systems, automatic teller machines, and online banking services. According to a notice posted by the Rogers CEO, the outage was attributed to “a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction”.

Cloudflare Radar 2022 Year in Review

In late September, protests and demonstrations erupted across Iran in response to the death of Mahsa Amini. Amini was a 22-year-old woman from the Kurdistan Province of Iran, and was arrested on September 13 in Tehran by Iran’s “morality police”, a unit that enforces strict dress codes for women. She died on September 16 while in police custody. Iran’s government is no stranger to using Internet shutdowns as a means of limiting communication with the outside world, and in response to these protests and demonstrations, Internet connectivity across the country experienced multiple waves of disruptions.

Three of the major mobile network providers — AS44244 (Irancell), AS57218 (RighTel), and AS197207 (MCCI) — started implementing daily Internet “curfews” on September 21, generally taking place between 1600 and midnight local time (1230-2030 UTC), although the start times varied on several days. These regular shutdowns lasted into early October, with several more ad-hoc disruptions taking place through the middle of the month, as well as other more localized shutdowns of Internet connectivity. Over 75 million users were impacted by these shutdowns, based on subscriber figures for MCCI alone.

Cloudflare Radar 2022 Year in Review

Cable cuts are also a frequent cause of Internet outages, with an old joke among network engineers that suggested that backhoes were the Internet’s natural enemy. While backhoes may be a threat to terrestrial fiber-optic cable, natural disasters can wreak havoc on submarine cables.

A prime example took Tonga offline earlier this year, when the Hunga Tonga–Hunga Ha’apai volcanic eruption damaged the submarine cable connecting Tonga to Fiji, resulting in a 38-day Internet outage. After the January 14 eruption, only minimal Internet traffic (via limited satellite services) was seen from Tonga. On February 22, Digicel announced that the main island was back online after initial submarine cable repairs were completed, but it was estimated that repairs to the domestic cable, connecting outlying islands, could take an additional six to nine months. We saw rapid growth in traffic from Tonga once the initial cable repairs were completed.

Cloudflare Radar 2022 Year in Review

The war in Ukraine is now ten months old, and throughout the time it has been going on, multiple networks across the country have experienced outages. In March, we observed outages in Mariupol and other cities where fighting was taking place. In late May, an extended Internet disruption began in Kherson, coincident with AS47598 (Khersontelecom) starting to route traffic through Russian network provider AS201776 (MIranda), rather than a Ukrainian upstream. And in October, widespread power outages disrupted Internet connectivity in Kharkiv, Lviv, Kyiv, Poltova Oblast, and Zhytomyr. These outages and others were covered in more detail in the quarterly Internet disruption overview blog posts, as well as several other Ukraine-specific blog posts.

Adoption

Cloudflare Radar 2022 Year in Review

Working with millions of websites and applications accessed by billions of people as well as providing an industry-leading DNS resolver service gives Cloudflare a unique perspective on the adoption of key technologies and platforms. SpaceX Starlink was frequently in the news this year, and we observed a 15x increase in traffic from the satellite Internet service provider. Social networking platform Mastodon was also in the news this year, and saw significant growth in interest as well.

IPv6 remains increasingly important as connected device growth over the last decade has exhausted available IPv4 address space, but global adoption remained around 35% across the year. And as the Internet-connected population continues to grow, many of those people are using mobile devices as their primary means of access. To that end, we also explore mobile device usage trends across the year.

Starlink adoption

Internet connectivity through satellites in geostationary orbit (GEO) has been around for a number of years, but services have historically been hampered by high latency and slower speeds. However, the launch of SpaceX Starlink’s Low Earth Orbit (LEO) satellite Internet service in 2019 and subsequent expansion of the satellite constellation has made high performance Internet connections available in many locations that were previously unserved or underserved by traditional wired or wireless broadband. To track the growth in usage and availability of Starlink’s service, we analyzed aggregate Cloudflare traffic volumes associated with the service’s autonomous system (AS14593) throughout 2022. Although Starlink is not yet available globally, we did see traffic growth across a number of locations. The request volume shown on the trend line in the chart represents a seven-day trailing average.

Damage from the war in Ukraine has disrupted traditional wired and wireless Internet connectivity since the invasion started in late February. Starlink made headlines that month after the company activated service within the country, and the necessary satellite Internet terminals became more widely available. Within days, Cloudflare began to see Starlink traffic, with volume growing consistently throughout the year.

Cloudflare Radar 2022 Year in Review

Latent interest in the service was also apparent in a number of locations where traffic grew quickly after Starlink announced availability. One such example is Romania, which was included in Starlink’s May announcement of an expanded service footprint, and which saw rapid traffic growth after the announcement.

Cloudflare Radar 2022 Year in Review

And in the United States, where Starlink has provided service since launch, traffic grew more than 10x through the end of November. Service enhancements announced during the year, like the ability to get Internet connectivity from moving vehicles, boats, and planes will likely drive additional traffic growth in the future.

Cloudflare Radar 2022 Year in Review

Mastodon interest

Above, we showed that Mastodon hit an inflection point in its popularity during the last few months of 2022. To better understand how interest in Mastodon evolved during 2022, we analyzed aggregate 1.1.1.1 request volume data for the domain names associated with 400 top Mastodon instances, looking at aggregate request volume by location. The request volume shown on the trend line in the chart represents a seven-day trailing average.

Although interest in Mastodon clearly accelerated over the last few months of the year, this interest was unevenly distributed throughout the world as we saw little to no traffic across many locations. Graphs for those locations are not included within the Year In Review website. However, because Mastodon has been around since 2016, it built a base of early adopters over the last six years before being thrust into the spotlight in 2022.

Those early adopters are visible at a global level, as we see a steady volume of resolver traffic for the analyzed Mastodon instance domain names through the first nine months of the year, with the timing of the increase visible in late April aligning with the announcement that Elon Musk had reached a deal to acquire Twitter for $44 billion. The slope of the graph clearly shifted in October as it became increasingly clear that the acquisition would close shortly, with additional growth into November after the deal was completed. This growth is likely due to a combination of existing but dormant Mastodon accounts once again becoming active, and an influx of new users.

Cloudflare Radar 2022 Year in Review

The traffic pattern observed for the United States appears fairly similar to the global pattern, with traffic from an existing set of users seeing massive growth starting in late October as well.

Cloudflare Radar 2022 Year in Review

Although the core Mastodon software was developed by a programmer living in Germany, and the associated organization is incorporated as a German not-for-profit, it didn’t appear to have any significant home field advantage. Query volume for Germany was relatively low throughout most of the year, and only started to rapidly increase at the end of October, similar to behavior observed in a number of other countries.

Cloudflare Radar 2022 Year in Review

IPv6 adoption

Although IPv6 has been around for nearly a quarter-century, adoption has been relatively slow over that time. However, with the exhaustion of available IPv4 address space and the growth in connected and mobile devices, IPv6 plays a critical role in the future of the Internet. Cloudflare has enabled customers to deliver content over IPv6 since our first birthday, back in 2011, and we have evolved support in several ways since that time. Analysis of traffic to the Cloudflare network provides us with insights into IPv6 adoption across the Internet.

On a global basis, IPv6 adoption hovered around the 35% mark throughout the year, with nominal growth evident in the trend line shown in the graph. While it is encouraging to see one of every three requests for dual stacked content being made over IPv6, this adoption rate demonstrates a clear opportunity for improvement.

To calculate IPv6 adoption for each location, we identified the set of customer zones that had IPv6 enabled (were “dual stacked”) during 2022, and then divided the daily request count for the zones over IPv6 by the daily sum of IPv4 and IPv6 requests for the zones, filtering out bot traffic in both cases. The line shown in the trends graph represents a seven-day trailing average. For the top 10 chart, we calculated the average IPv6 adoption level on a monthly basis per location, and then ranked the locations by percentage. The chart illustrates the ranking by month, and how those rankings change across the year.

Cloudflare Radar 2022 Year in Review

One location that has seized that opportunity is India, which recorded the highest IPv6 adoption rate throughout the year. After seeing more than 70% adoption through July, it began to drop slightly in late summer, losing a couple of percentage points over the subsequent months.

One key driver behind India’s leadership in this area is IPv6 support from Jio, India’s largest mobile network operator, as well as being a provider of fiber-to-the-home broadband connectivity. They aggressively started their IPv6 journey in late 2015, and now much of Jio’s core network infrastructure is IPv6-only, while customer-facing mobile and fiber connections are dual-stacked.

Cloudflare Radar 2022 Year in Review

Also heading in the right direction are the more than 60 locations around the world that saw IP adoption rates more than double this year. One of the largest increases was seen in the European country of Georgia, which grew more than 3,500% to close out the year at 10% adoption thanks to rapid growth across February and March at Magticom, a leading Georgian telecommunications provider.

Many of the other locations in this set also experienced large gains over a short period of time, likely due to a local network provider enabling subscriber support for IPv6. While significant gains seen in over a quarter of the total surveyed locations is certainly a positive sign, it must be noted that over 50 are under 10% adoption, with more than half of those remaining well under 1%, even after seeing adoption more than double. Internet service providers around the world continue to add or improve IPv6 support for their subscribers, but many have low to non-existent adoption rates, presenting significant opportunity to improve in the future.

Cloudflare Radar 2022 Year in Review

As noted above, India had the highest level of IPv6 adoption through 2022. In looking at the remainder of the top 10 list, Saudi Arabia and Malaysia traded places several times during the year as the locations with the second and third-highest adoption rates, at just under 60% and around 55% respectively. The United States appeared towards the bottom of the top 10 list during the first quarter, but ranked lower for the remainder of the year. Belgium proved to be the most consistent, holding the fourth-place spot from March through November, with around 55% IPv6 adoption. Overall, a total of 14 locations appeared among the top 10 at some point during the year.

Mobile device usage

Each year, mobile devices become more and more powerful, and are increasingly being used as the primary onramp to the Internet in many places. In fact, in some parts of the world, so-called “desktop” devices (which includes laptop form factors) are the exception for Internet access, not the rule.

Analysis of the information included with each content request enables us to classify the type of device (mobile or desktop) used to make the request. To calculate the percentage of mobile device usage by location, we divided the number of requests made by mobile devices over the course of a week by the total number of requests seen that week, filtering out bot traffic in both cases. For the top 10 chart, we ranked the locations by the calculated percentage. The chart illustrates the ranking by month, and how those rankings change across the year.

In looking at the top 10 chart, we note that Iran and Sudan held the top two slots for much of the year, bookended by Yemen in January and Mauritania in November. Below the top two spots, however, significant volatility is clear throughout the year within the rest of the top 10. However, this movement was actually concentrated across a relatively small percentage range, with just five to ten percentage points separating the top and bottom ranked locations, depending on the week. The top ranked locations generally saw 80-85% of traffic from mobile devices, while the bottom ranked locations saw 75-80% of traffic from mobile devices.

This analysis reinforces the importance of mobile connectivity in Iran, and underscores why mobile network providers were targeted for Internet shutdowns in September and October, as discussed above. (And the shutdowns subsequently explain why Iran disappears from the top 10 list after September.)

Security

Cloudflare Radar 2022 Year in Review

Improving Internet security is a key part of Cloudflare’s drive to help build a better Internet. One way we do that is by protecting customer websites, applications, and network infrastructure from malicious traffic and attacks. Because malicious actors regularly use a variety of techniques and approaches in launching their attacks, we have a number of products within our security solution portfolio that provide customers with flexibility around how they handle these attacks. Below, we explore insights derived from the attack mitigation we do on behalf of customers, including how we are mitigating attacks, what kinds of websites and applications attacks are targeting, and where these attacks appear to be coming from. In addition, with the acquisition of Area 1 earlier in 2022, we are presenting insight into where malicious email originates from. Analysis of this data highlights that there is very much no “one size fits all” security solution, as attackers use a wide variety of techniques, frequently shifting between them. As such, having a broad but flexible portfolio of security solutions at the ready is critical for CISOs and CIOs.

Mitigation sources

Depending on the approach taken by an attacker, and the type of content being targeted, one attack mitigation technique may be preferable over another. Cloudflare refers to these techniques as “mitigation sources”, and they include popular tools and techniques like Web Application Firewall (WAF) and DDoS Mitigation (DDoS), but also lesser known ones like IP Reputation (IPR), Access Rules (AR), Bot Management (BM), and API Shield (APIS). Examining the distribution of mitigation sources applied by location can help us better understand the types of attacks originating from those locations. To calculate the percentage of mitigated traffic associated with each mitigation source by location, we divided the total number of daily mitigated requests for each source by the total number of mitigated requests seen that day. Bot traffic is included in these calculations, given that many attacks originate from bots. A single request can be mitigated by multiple techniques, and here we consider the last technique that mitigated the request.

Across many locations, IP Reputation, Bot Management, and Access Rules accounted for small amounts of mitigated traffic throughout the year, with the volumes varying by country. However, in other locations, IP Reputation and Access Rules were responsible for larger amounts of mitigated traffic, possibly indicating those places had more of their traffic being blocked outright. A number of countries saw a rapid and significant increase in DDoS mitigated traffic during January to the 80-90% range, followed by a rapid drop to the 10-20% range. In that vein, DDoS Mitigation and WAF percentage shifts were frequently very spiky, with only occasional sustained periods of relatively consistent percentages.

Overall, DDoS Mitigation and WAF were the two most frequently used techniques to address attacks. The former’s share on a global basis was highest in mid-January, growing to nearly 80%, while the latter’s peak was during February, when it accounted for almost 60% of mitigated traffic. A spike in the usage of Access Rules is clearly visible in August, related to similar spikes observed for the United States, United Arab Emirates, and Malaysia.

Cloudflare Radar 2022 Year in Review

Although Access Rules accounted for as much as 20% of mitigated traffic from the United States in August, it saw much lower usage throughout the balance of the year. DDoS Mitigation was the primary technique used to mitigate attack traffic coming from the United States, responsible for over 80% of such traffic during the first quarter, though it steadily declined through August. In a complimentary fashion, WAF drove only ~20% of mitigated traffic early in the year, but that volume steadily grew and had tripled through August. Interestingly, the growth in Access Rules usage followed rapid growth and then similarly rapid decline in WAF, possibly suggesting that more targeted rules were implemented to augment the managed rules applied by the Web Application Firewall against US-originated attacks.

Cloudflare Radar 2022 Year in Review

Access Rules and IP Reputation were applied more frequently to mitigate attack traffic coming from Germany, with Bot Management also seeing increased usage in February, March, and June. However, except for periods in February and July, DDoS Mitigation drove the bulk of mitigated traffic, generally ranging between 60-80%. WAF mitigation was clearly most significant during February, with 70-80% of mitigated traffic, and July, at around 60%.

Cloudflare Radar 2022 Year in Review

In mitigating attacks coming from Japan, it is interesting to see a couple of notable spikes in Bot Management. In March, it was briefly responsible for upwards of 40% of mitigated traffic, with another spike that was half as big in June. Access Rules also maintained a consistent presence in the graph, with around 5% of mitigated traffic through August, but slightly less in the following months. In dealing with Japanese attack traffic, WAF & DDoS Mitigation frequently traded positions as the largest source of mitigated traffic, although there was no clear pattern or apparent cycle. Both reached as much as 90% of mitigated traffic at times throughout the year – WAF in February and DDoS Mitigation in March. DDoS Mitigation’s periods of “dominance” tended to be more sustained, lasting for several weeks, but were punctuated by brief WAF spikes.

Cloudflare Radar 2022 Year in Review

WAF rules

As noted above, Cloudflare’s WAF is frequently used to mitigate application layer attacks. There are hundreds of individually managed rules that can be applied by the WAF depending on the characteristics of the mitigated request, but these rules can be grouped into over a dozen types. Examining the distribution of WAF rules by location can help us better understand the techniques that attacks coming from that location are using. (For example, are attackers trying to inject SQL code into a form field, or exploit a published CVE?) To calculate the distribution of WAF mitigated traffic across the set of rule types for each location, we divided the number of requests mitigated by a particular type of WAF rule seen over the course of a week by the total number of WAF mitigated requests seen over that week. A single request can be mitigated by multiple rules and here we consider the last rule in a sequence that mitigated the request. The chart shows how the distribution of mitigated requests across the selected rule types changes over the course of the year. Bot traffic is included in these calculations.

At a worldwide level, during the first few months of the year, approximately half of HTTP requests blocked by our Managed WAF Rules contained HTTP anomalies, such as malformed method names, null byte characters in headers, non-standard ports, or content length of zero with a POST request. During that period, Directory Traversal and SQL Injection (SQLi) rules both accounted for just over 10% of mitigated requests as well. Attackers began to further vary their approach starting in May, as Cross Site Scripting (XSS) and File Inclusion both grew to over 10% of mitigations, while HTTP anomalies dropped to below 30%. Use of Software Specific rules grew above 10% in July, as attackers apparently ramped their efforts to exploit vendor-specific vulnerabilities. Broken Authentication and Command Injection rulesets also saw some growth in activity during the last several months, suggesting that attackers increased their efforts to find vulnerabilities in login/authentication systems or to execute commands on vulnerable systems in an attempt to gain access.

Cloudflare Radar 2022 Year in Review

Although HTTP Anomaly was the most frequently applied rule when mitigations are aggregated at a global level, there were a number of locations where it held the top spot only briefly, if at all, as discussed below.

Attacks originating in Australia were WAF-mitigated using a number of rulesets, with the most applied ruleset changing frequently during the first half of the year. In contrast to the global overview, HTTP Anomaly was the top ruleset for only a single week in February, when it accounted for just over 30% of mitigations. Otherwise, attacks were most frequently mitigated with Software Specific, Directory Traversal, File Inclusion, and SQLi rules, generally accounting for 25-35% of mitigations. This pattern shifted starting in July, though, as Directory Traversal attacks became the most common, staying that way through the balance of the year. After peaking in June, SQLi attacks became significantly less common, rapidly falling and staying below 10% of mitigations.

WAF mitigations of attacks originating in Canada also demonstrated a pattern that differed from the global one. Although the HTTP Anomaly ruleset started the year accounting for approximately two thirds of mitigated requests, it was half that by the end of January, and saw significant volatility throughout the balance of the year. SQLi mitigations of Australian traffic effectively saw an opposite pattern, starting the year below 10% of mitigations but growing rapidly, accounting for 60% or more of mitigated traffic at multiple times throughout the year. Interestingly, SQLi attacks from Canada appeared to come in multi-week waves, becoming the most applied ruleset during those waves, and then receding for a brief period.

For attacks originating in Switzerland, the HTTP Anomaly ruleset was never the most frequently invoked, although it remained among the top five throughout the year. Instead, Directory Traversal and XSS rules were most frequently used, accounting for as much as 40% of mitigations. Directory Traversal most consistently held the top spot, though XSS attacks were the most prevalent during August. SQLi attacks saw peaks in April, July/August, and then again at the end of November. The Software Specific ruleset also breakout growth in September to as much as 20% of mitigated requests.

Target categories

Above, we discussed how traffic distribution across a set of categories provides insights into the types of content that users are most interested in. By performing similar analysis through a mitigation lens, we can gain insights into the types of websites and applications that are being most frequently targeted by attackers. To calculate the distribution of mitigated traffic across the set of categories for each location, we divided the number of mitigated requests for domains associated with a given category seen over the course of a week by the total number of requests mapped to that category during that week. The chart shows how the distribution of mitigated requests across each category changes over the course of the year. (As such, percentages will not sum to 100%). Bot traffic is included in these calculations. The percentage of traffic that was mitigated as an attack varied widely across industries and originating locations. In some places, a nominal percentage of traffic across all categories was mitigated, while in others, multiple categories experienced spikes in mitigated traffic at multiple times during 2022.

When aggregated at a global level, there was significant variance over the course of the year in the industry categories that attracted the most attacks as a fraction of their overall traffic. Through January and February, Technology sites had the largest percentage of mitigated requests, ranging between 20-30%. After that, a variety of categories moved in and out of the top slot, with none holding it for more than a few weeks. The biggest spike in attacks was targeted at Travel sites in mid-April, when more than half of the category’s traffic was mitigated. Coincident with the start of the 2022 World Cup in the last week of November, Gambling and Entertainment sites saw the largest percentages of mitigated traffic.

Cloudflare Radar 2022 Year in Review

For attacks coming from the United Kingdom, Technology sites consistently saw around 20% of mitigated traffic through the year. During those times that it was not the most mitigated category, half a dozen other categories topped the list. Travel sites experienced two significant bursts of attacks, with nearly 60% of traffic mitigated in April, and nearly 50% in October. Other categories, including Government & Politics, Real Estate, Religion, and Education had the largest shares of mitigated traffic at various times throughout the year. UK-originated attacks on Entertainment sites jumped significantly in late November, with 40% of traffic mitigated at the end of the month.

Similar to the trends seen at the global level, Technology sites accounted for the largest percentage of mitigated attacks from the United States in January and February, clocking in between 30-40%. After that, attackers shifted their focus to target other industry categories. In mid-April, Travel sites had over 60% of requests mitigated as attacks. However, starting in May, Gambling sites most frequently had the highest percentage of traffic being mitigated, generally ranging between 20-40%, but spiking up to 70% in late October/early November.

In contrast, significantly smaller percentages of traffic across the surveyed categories from Japan was mitigated as attacks throughout 2022. Most categories saw mitigation shares of less than 10%, although there were a number of brief spikes observed at times. In late March, traffic to sites in the Government & Politics category briefly jumped to a nearly 80% mitigation share, while Travel sites spiked to nearly 70% of requests mitigated as attacks, similar to the behavior seen in other locations. In late June, Religion sites had a mitigation share of over 60%, and a couple of months later, Gambling sites experienced a rapid increase in mitigated traffic, reaching just over 40%. These attacks targeting Gambling sites then receded for a few months before starting to aggressively increase again in October.

Phishing email sources

Phishing emails are ultimately intended to trick users into providing attackers with login credentials for important websites and applications. At a consumer level, this could include an e-commerce site or banking application, while for businesses, this could include code repositories or employee information systems. For customers protected by Cloudflare Area 1 Email Security, we can identify the location that these phishing emails are being sent from. IP address geolocation is used to identify origination location, and the aggregate email counts apply to emails processed by Area 1 only. For the top 10 chart, we aggregated the number of phishing emails seen on a weekly basis per location, and then ranked the locations by phishing email volume. The chart illustrates the ranking by week, and how those rankings change across the year.

Reviewing the top 10 list, we find that the United States was the top source of phishing emails observed by Area 1 during 2022. It held the top spot for nearly the entire year, ceding it only once to Germany in November. The balance of the top 10 saw a significant amount of volatility over time, with a total of 23 locations holding a spot in the rankings for at least one month during the year. These locations were well-distributed geographically across the Americas, Europe, and Asia, highlighting that no one region of the world is a greater threat than others. Obviously, distrusting or rejecting all email originating from these locations is not a particularly practical response, but applying additional scrutiny can help keep your organization, and the Internet, safer.

Conclusion

Attempting to concisely summarize our “year in review” observations is challenging, especially as we only looked at trends in this blog post across a small fraction of the nearly 200 locations included in the website’s visualizations. Having said that, we will leave you with the following brief thoughts:

  • Attack traffic comes from everywhere, with constantly shifting targets, using widely varied techniques. Ensure that your security solutions provider offers a comprehensive portfolio of services to help keep your sites, applications, and infrastructure safe.
  • Internet service providers around the world need to improve support for IPv6 — it is no longer a “new” technology, and available IPv4 address space will become both increasingly scarce and increasingly expensive. Support for IPv6 needs to become the default going forward.
  • Internet shutdowns are being increasingly used by governments to limit communications within a country, as well as limiting communications with the rest of the world. As the United Nations stated in a May 2022 report, “Blanket shutdowns in particular inherently impose unacceptable consequences for human rights and should never be imposed.”

As we said in the introduction, we encourage you to visit the full Cloudflare Radar 2022 Year In Review website and explore the trends relevant to locations and industries of interest, and to consider how they impact your organization so that you are appropriately prepared for 2023.

If you have any questions, you can contact the Cloudflare Radar team at [email protected] or on Twitter at @CloudflareRadar.

Acknowledgements

It truly took a village to produce the Cloudflare Radar 2022 Year In Review, and we would be remiss if we didn’t acknowledge the contributions of colleagues that were instrumental in making this project possible. Thank you to: Sabina Zejnilovic, Carlos Azevedo, Jorge Pacheco (Data Science); Ricardo Baeta, Syeef Karim (Design); Nuno Pereira, Tiago Dias, Junior Dias de Oliveira (Front End Development); João Tomé (Most popular Internet services); and Davide Marques, Paula Tavares, Celso Martinho (Project/Engineering Management).

2022 US midterm elections attack analysis

Post Syndicated from David Belson original https://blog.cloudflare.com/2022-us-midterm-elections-attack-analysis/

2022 US midterm elections attack analysis

2022 US midterm elections attack analysis

Through Cloudflare’s Impact programs, we provide cyber security products to help protect access to authoritative voting information and the security of sensitive voter data. Two core programs in this space are the Athenian Project, dedicated to protecting state and local governments that run elections, and Cloudflare for Campaigns, a project with a suite of Cloudflare products to secure political campaigns’ and state parties’ websites and internal teams.

However, the weeks ahead of the elections, and Election Day itself, were not entirely devoid of attacks. Using data from Cloudflare Radar, which showcases global Internet traffic, attack, and technology trends and insights, we can explore traffic patterns, attack types, and top attack sources associated with both Athenian Project and Cloudflare for Campaigns participants.

For both programs, overall traffic volume unsurprisingly ramped up as Election Day approached. SQL Injection (SQLi) and HTTP Anomaly attacks were the two largest categories of attacks mitigated by Cloudflare’s Web Application Firewall (WAF), and the United States was the largest source of observed attacks — see more on this last point below.

Below, we explore the trends seen across both customer sets from October 1, 2022, through Election Day on November 8.

Athenian Project

Throughout October, daily peak traffic volumes effectively doubled over the course of the month, with a weekday/weekend pattern also clearly visible. However, significant traffic growth is visible on Monday, November 7, and Tuesday, November 8 (Election Day), with Monday’s peak just under 2x October’s peaks, while Tuesday saw two peaks, one just under 4x higher than October peaks, while the other was just over 4x higher. Zooming in, the first peak was at 1300 UTC (0800 Eastern time, 0500 Pacific time), while the second was at 0400 UTC (2300 Eastern time, 2000 Pacific time). The first one appears to be aligned with the polls opening on the East Coast, while the second appears to be aligned with the time that the polls closed on the West Coast.

However, aggregating the traffic here presents a somewhat misleading picture. While both spikes were due to increased traffic across multiple customer sites, the second one was exacerbated by a massive increase in traffic for a single customer. Regardless, the increased traffic clearly shows that voters turned to local government sites around Election Day.

2022 US midterm elections attack analysis

Despite this increase in overall traffic, attack traffic mitigated by Cloudflare’s Web Application Firewall (WAF) remained remarkably consistent throughout October and into November, as seen in the graph below. The obvious exception was an attack that occurred on Monday, October 10. This attack targeted a single Athenian Project participant, and was mitigated by rate limiting the requests.

2022 US midterm elections attack analysis

SQL injection (SQLi) attacks saw significant growth in volume in the week and a half ahead of Election Day, along with an earlier significant spike on October 24. While the last weekend in October (October 29 and 30) saw significant SQLi attack activity, the weekend of November 5 and 6 was comparatively quiet. However, those attacks ramped up again heading into and on Election Day, as seen in the graph below.

2022 US midterm elections attack analysis

Attempted attacks mitigated with the HTTP Anomaly ruleset also ramped up in the week ahead of Election Day, though to a much lesser extent than SQLi attacks. As the graph below shows, the biggest spikes were seen on October 31/November 1, and just after midnight UTC on November 4 (late afternoon to early evening in the US). Related request volume also grew heading into Election Day, but without significant short-duration spikes. There is also a brief but significant attack clearly visible on the graph on October 10. However, it occurred several hours after the rate limited attack referenced above — it is not clear if the two are related.

2022 US midterm elections attack analysis

The distribution of attacks over the surveyed period from October 1 through November 9 shows that those categorized as SQLi and HTTP Anomaly were responsible for just over two-thirds of WAF-mitigated requests. Nearly 14% were categorized as “Software Specific,” which includes attacks related to specific CVEs. The balance of the attacks were mitigated by WAF rules in categories including File Inclusion, XSS (Cross Site Scripting), Directory Traversal, and Command Injection.

2022 US midterm elections attack analysis

Media reports suggest that foreign adversaries actively try to interfere with elections in the United States. While this may be the case, analysis of the mitigated attacks targeting Athenian Project customers found that over 95% of the mitigated requests (attacks) came from IP addresses that geolocate to the United States. However, that does not mean that the attackers themselves are necessarily located in the country, but rather that they appear to be using compromised systems and proxies within the United States to launch their attacks against these sites protected by Cloudflare.

2022 US midterm elections attack analysis

Cloudflare for Campaigns

In contrast to Athenian Project participants, traffic to candidate sites that are participants in Cloudflare for Campaigns began to grow several weeks ahead of Election Day. The graph below shows a noticeable increase (~50%) in peak traffic volumes starting on October 12, with an additional growth (50-100%) starting a week later. Traffic to these sites appeared to quiet a bit toward the end of October, but saw significant growth again heading into, and during, Election Day.

However, once again, this aggregate traffic data presents something of a misleading picture, as one candidate site saw multiple times more traffic than the other participating sites. While those other sites saw similar shifts in traffic as well, they were dwarfed by those experienced by the outlier site.

2022 US midterm elections attack analysis

The WAF-mitigated traffic trend for campaign sites followed a similar pattern to the overall traffic. As the graph below shows, attack traffic also began to increase around October 19, with a further ramp near the end of the month. The October 27 spike visible in the graph was due to an attack targeting a single customer’s site, and was addressed using “Security Level” mitigation techniques, which uses IP reputation information to decide if and how to present challenges for incoming requests.

2022 US midterm elections attack analysis

The top two rule categories, HTTP Anomaly and SQLi, together accounted for nearly three-quarters of the mitigated requests, and Directory Traversal attacks were just under 10% of mitigated requests for this customer set. The HTTP Anomaly and Directory Traversal percentages were higher than those for attacks targeting Athenian Project participants, while the SQLi percentage was slightly lower.

2022 US midterm elections attack analysis

Once again, a majority of the WAF-mitigated attacks came from IP addresses in the United States. However, among Cloudflare for Campaigns participants, the United States only accounted for 55% of attacks, significantly lower than the 95% seen for Athenian Project participants. The balance is spread across a long tail of countries, with allies including Germany, Canada, and the United Kingdom among the top five. As noted above, however, the attackers may be elsewhere, and are using botnets or other compromised systems in these countries to launch attacks.

2022 US midterm elections attack analysis

Improving security with data

We are proud to be trusted by local governments, campaigns, state parties, and voting rights organizations to protect their websites and provide uninterrupted access to information and trusted election results. Sharing information about the threats facing these websites helps us further support their valuable work by enabling them, and other participants in the election space, to take proactive steps to improve site security.

Learn more about how to apply to the Athenian Project, and check out Cloudflare Radar for real-time insights into Internet traffic, attack trends, and more.

Internet disruptions overview for Q3 2022

Post Syndicated from David Belson original https://blog.cloudflare.com/q3-2022-internet-disruption-summary/

Internet disruptions overview for Q3 2022

Internet disruptions overview for Q3 2022

Cloudflare operates in more than 275 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. In many cases, these disruptions can be attributed to a physical event, while in other cases, they are due to an intentional government-directed shutdown. In this post, we review selected Internet disruptions observed by Cloudflare during the third quarter of 2022, supported by traffic graphs from Cloudflare Radar and other internal Cloudflare tools, and grouped by associated cause or common geography. The new Cloudflare Radar Outage Center provides additional information on these, and other historical, disruptions.

Government directed shutdowns

Unfortunately, for the last decade, governments around the world have turned to shutting down the Internet as a means of controlling or limiting communication among citizens and with the outside world. In the third quarter, this was an all too popular cause of observed disruptions, impacting countries and regions in Africa, the Middle East, Asia, and the Caribbean.

Iraq

As mentioned in our Q2 summary blog post, on June 27, the Kurdistan Regional Government in Iraq began to implement twice-weekly (Mondays and Thursday) multi-hour regional Internet shutdowns over the following four weeks, intended to prevent cheating on high school final exams. As seen in the figure below, these shutdowns occurred as expected each Monday and Thursday through July 21, with the exception of July 21. They impacted three governorates in Iraq, and lasted from 0630–1030 local time (0330–0730 UTC) each day.

Internet disruptions overview for Q3 2022
Erbil, Sulaymaniyah, and Duhok Governorates, Iraq. (Source: Map data ©2022 Google, Mapa GISrael)
Internet disruptions overview for Q3 2022

Cuba

In Cuba, an Internet disruption was observed between 0055-0150 local time (0455-0550 UTC) on July 15 amid reported anti-government protests in Los Palacios and Pinar del Rio.

Internet disruptions overview for Q3 2022
Los Palacios and Pinar del Rio, Cuba. (Source: Map data ©2022 INEGI)
Internet disruptions overview for Q3 2022

Closing out the quarter, another significant disruption was observed in Cuba, reportedly in response to protests over the lack of electricity in the wake of Hurricane Ian. A complete outage is visible in the figure below between 2030 on September 29 and 0315 on September 30 local time (0030-0715 UTC on September 30).

Internet disruptions overview for Q3 2022

Afghanistan

Telecommunications services were reportedly shut down in part of Kabul, Afghanistan on the morning of August 8. The figure below shows traffic dropping starting around 0930 local time (0500 UTC), recovering 11 hours later, around 2030 local time (1600 UTC).

Internet disruptions overview for Q3 2022
Kabul, Afghanistan. (Source: Map data ©2022 Google)
Internet disruptions overview for Q3 2022

Sierra Leone

Protests in Freetown, Sierra Leone over the rising cost of living likely drove the Internet disruptions observed within the country on August 10 & 11. The first one occurred between 1200-1400 local time (1200-1400 UTC) on August 10. While this outage is believed to have been government directed as a means of quelling the protests, Zoodlabs, which manages Sierra Leone Cable Limited, claimed that the outage was the result of “emergency technical maintenance on some of our international routes”.

A second longer outage was observed between 0100-0730 local time (0100-0730 UTC) on August 11, as seen in the figure below. These shutdowns follow similar behavior in years past, where Internet connectivity was shut off following elections within the country.

Internet disruptions overview for Q3 2022
Freetown, Sierra Leone (Source: Map data ©2022 Google, Inst. Geogr. Nacional)
Internet disruptions overview for Q3 2022

Region of Somaliland

In Somaliland, local authorities reportedly cut off Internet service on August 11 ahead of scheduled opposition demonstrations. The figure below shows a complete Internet outage in Woqooyi Galbeed between 0645-1355 local time (0345-1055 UTC.)

Internet disruptions overview for Q3 2022
Woqooyi Galbeed, Region of Somaliland. (Source: Map data ©2022 Google, Mapa GISrael)
Internet disruptions overview for Q3 2022

At a network level, the observed outage was due to a loss of traffic from AS37425 (SomCable) and AS37563 (Somtel), as shown in the figures below. Somtel is a mobile services provider, while SomCable is focused on providing wireline Internet access.

Internet disruptions overview for Q3 2022
Internet disruptions overview for Q3 2022

India

India is no stranger to government-directed Internet shutdowns, taking such action hundreds of times over the last decade. This may be changing in the future, however, as the country’s Supreme Court ordered the Ministry of Electronics and Information Technology (MEITY) to reveal the grounds upon which it imposes or approves Internet shutdowns. Until this issue is resolved, we will continue to see regional shutdowns across the country.

One such example occurred in Assam, where mobile Internet connectivity was shut down to prevent cheating on exams. The figure below shows that these shutdowns were implemented twice daily on August 21 and August 28. While the shutdowns were officially scheduled to take place between 1000-1200 and 1400-1600 local time (0430-0630 and 0830-1030 UTC), some providers reportedly suspended connectivity starting in the early morning.

Internet disruptions overview for Q3 2022
Assam, India. (Source: Map data ©2022 Google, TMap Mobility)
Internet disruptions overview for Q3 2022

Iran

In late September, protests and demonstrations have erupted across Iran in response to the death of Mahsa Amini. Amini was a 22-year-old woman from the Kurdistan Province of Iran, and was arrested on September 13, 2022, in Tehran by Iran’s “morality police”, a unit that enforces strict dress codes for women. She died on September 16 while in police custody. In response to these protests and demonstrations, Internet connectivity across the country experienced multiple waves of disruptions.

In addition to multi-hour outages in Sanadij and Tehran province on September 19 and 21 that were covered in a blog post, three mobile network providers — AS44244 (Irancell), AS57218 (RighTel), and AS197207 (MCCI) — implemented daily Internet “curfews”, generally taking place between 1600 and midnight local time (1230-2030 UTC), although the start times varied on several days. These regular shutdowns are clearly visible in the figure below, and continued into early October.

Internet disruptions overview for Q3 2022
Sanandij and Tehran, Iran. (Source: Map data ©2022 Google)
Internet disruptions overview for Q3 2022

As noted in the blog post, access to DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) services was also blocked in Iran starting on September 20, and in a move that is likely related, connections over HTTP/3 and QUIC were blocked starting on September 22, as shown in the figure below from Cloudflare Radar.

Internet disruptions overview for Q3 2022

Natural disasters

Natural disasters such as earthquakes and hurricanes wreak havoc on impacted geographies, often causing loss of life, as well as significant structural damage to buildings of all types. Infrastructure damage is also extremely common, with widespread loss of both electrical power and telecommunications infrastructure.

Papua New Guinea

On September 11, a 7.6 magnitude earthquake struck Papua New Guinea, resulting in landslides, cracked roads, and Internet connectivity disruptions. Traffic to the country dropped by 26% just after 1100 local time (0100 UTC) . The figure below shows that traffic volumes remained lower into the following day as well. An announcement from PNG DataCo, a local provider, noted that the earthquake “has affected the operations of the Kumul Submarine Cable Network (KSCN) Express Link between Port Moresby and Madang and the PPC-1 Cable between Madang and Sydney.” This damage, they stated, resulted in the observed outage and degraded service.

Internet disruptions overview for Q3 2022

Mexico

Just over a week later, a 7.6 magnitude earthquake struck the Colima-Michoacan border region in Mexico at 1305 local time (1805 UTC). As shown in the figure below, traffic dropped over 50% in the impacted states immediately after the quake occurred, but recovered fairly quickly, returning to normal levels by around 1600 local time (2100 UTC).

Internet disruptions overview for Q3 2022
Earthquake epicenter, 35 km SW of Aguililla, Mexico. (Source: Map data ©2022 INEGI)
Internet disruptions overview for Q3 2022

Hurricane Fiona

Several major hurricanes plowed their way up the east coast of North America in late September, causing significant damage, resulting in Internet disruptions. On September 18, island-wide power outages caused by Hurricane Fiona disrupted Internet connectivity on Puerto Rico. As the figure below illustrates, it took over 10 days for traffic volumes to return to expected levels. Luma Energy, the local power company, kept customers apprised of repair progress through regular updates to its Twitter feed.

Internet disruptions overview for Q3 2022

Two days later, Hurricane Fiona slammed the Turks and Caicos islands, causing flooding and significant damage, as well as disrupting Internet connectivity. The figure below shows traffic starting to drop below expected levels around 1245 local time (1645 UTC) on September 20. Recovery took approximately a day, with traffic returning to expected levels around 1100 local time (1500 UTC) on September 21.

Internet disruptions overview for Q3 2022

Continuing to head north, Hurricane Fiona ultimately made landfall in the Canadian province of Nova Scotia on September 24, causing power outages and disrupting Internet connectivity. The figure below shows that the most significant impact was seen in Nova Scotia. As Nova Scotia Power worked to restore service to customers, traffic volumes gradually increased, as seen in the figure below. By September 29, traffic volumes on the island had returned to normal levels.

Internet disruptions overview for Q3 2022

Hurricane Ian

On September 28, Hurricane Ian made landfall in Florida, and was the strongest hurricane to hit Florida since Hurricane Michael in 2018. With over four million customers losing power due to damage from the storm, a number of cities experienced associated Internet disruptions. Traffic from impacted cities dropped significantly starting around 1500 local time (1900 UTC), and as the figure below shows, recovery has been slow, with traffic levels still not back to pre-storm volumes more than two weeks later.

Internet disruptions overview for Q3 2022
Sarasota, Naples, Fort Myers, Cape Coral, North Port, Port Charlotte, Punta Gorda, and Marco Island, Florida. (Source: Map data ©2022 Google, INEGI)
Internet disruptions overview for Q3 2022

Power outages

In addition to power outages caused by earthquakes and hurricanes, a number of other power outages caused multi-hour Internet disruptions during the third quarter.

Iran

A reported power outage in a key data center building disrupted Internet connectivity for customers of local ISP Shatel in Iran on July 25. As seen in the figure below, traffic dropped significantly at approximately 0715 local time (0345 UTC). Recovery began almost immediately, with traffic nearing expected levels by 0830 local time (0500 UTC).

Internet disruptions overview for Q3 2022

Venezuela

Electrical issues frequently disrupt Internet connectivity in Venezuela, and the independent @vesinfiltro Twitter account tracks these events closely. One such example occurred on August 9, when electrical issues disrupted connectivity across multiple states, including Mérida, Táchira, Barinas, Portuguesa, and Estado Trujillo. The figure below shows evidence of two disruptions, the first around 1340 local time (1740 UTC) and the second a few hours later, starting at around 1615 local time (2015 UTC). In both cases, traffic volumes appeared to recover fairly quickly.

Internet disruptions overview for Q3 2022
Mérida, Táchira, Barinas, Portuguesa, and Estado Trujillo, Venezuela. (Source: Map data ©2022 Google. INEGI)
Internet disruptions overview for Q3 2022

Oman

On September 5, a power outage in Oman impacted energy, aviation, and telecommunications services. The latter is evident in the figure below, which shows the country’s traffic volume dropping nearly 60% when the outage began just before 1515 local time (0915 UTC). Although authorities claimed that “the electricity network would be restored within four hours,” traffic did not fully return to normal levels until 0400 local time on September 6 (2200 UTC on September 5) the following day, approximately 11 hours later.

Internet disruptions overview for Q3 2022

Ukraine

Over the last seven-plus months of war in Ukraine, we have observed multiple Internet disruptions due to infrastructure damage and power outages related to the fighting. We have covered these disruptions in our first and second quarter summary blog posts, and continue to do so on our @CloudflareRadar Twitter account as they occur. Power outages were behind Internet disruptions observed in Kharkiv on September 11, 12, and 13.

The figure below shows that the first disruption started around 2000 local time (1700 UTC) on September 11. This near-complete outage lasted just over 12 hours, with traffic returning to normal levels around 0830 local time (0530 UTC) on the 12th. However, later that day, another partial outage occurred, with a 50% traffic drop seen at 1330 local time (1030 UTC). This one was much shorter, with recovery starting approximately an hour later. Finally, a nominal disruption is visible at 0800 local time (0500 UTC) on September 13, with lower than expected traffic volumes lasting for around five hours.

Internet disruptions overview for Q3 2022

Cable damage

Damage to both terrestrial and submarine cables have caused many Internet disruptions over the years. The recent alleged sabotage of the sub-sea Nord Stream natural gas pipelines has brought an increasing level of interest from European media (including Swiss and French publications) around just how important submarine cables are to the Internet, and an increasing level of concern among policymakers about the safety of these cable systems and the potential impact of damage to them. However, the three instances of cable damage reviewed below are all related to terrestrial cable.

Iran

On August 1, a reported “fiber optic cable” problem caused by a fire in a telecommunications manhole disrupted connectivity across multiple network providers, including AS31549 (Aria Shatel), AS58224 (TIC), AS43754 (Asiatech), AS44244 (Irancell), and AS197207 (MCCI). The disruption started around 1215 local time (0845 UTC) and lasted for approximately four hours. Because it impacted a number of major wireless and wireline networks, the impact was visible at a country level as well, as seen in the figure below.

Internet disruptions overview for Q3 2022

Pakistan

Cable damage due to heavy rains and flooding caused several Internet disruptions in Pakistan in August. The first notable disruption occurred on August 19, starting around 0700 local time (0200 UTC) and lasted just over six and a half hours. On August 22, another significant disruption is also visible, starting at 2250 local time (1750 UTC), with a further drop at 0530 local time (0030 UTC) on the 23rd. The second more significant drop was brief, lasting only 45 minutes, after which traffic began to recover.

Internet disruptions overview for Q3 2022

Haiti

Amidst protests over fuel price hikes, fiber cuts in Haiti caused Internet outages on multiple network providers. Starting at 1500 local time (1900 UTC) on September 14, traffic on AS27759 (Access Haiti) fell to zero. According to a (translated) Twitter post from the provider, they had several fiber optic cables that were cut in various areas of the country, and blocked roads made it “really difficult” for their technicians to reach the problem areas. Repairs were eventually made, with traffic starting to increase again around 0830 local time (1230 UTC) on September 15, as shown in the figure below.

Internet disruptions overview for Q3 2022

Access Haiti provides AS27774 (Haiti Networking Group) with Internet connectivity (as an “upstream” provider), so the fiber cut impacted their connectivity as well, causing the outage shown in the figure below.

Internet disruptions overview for Q3 2022

Technical problems

As a heading, “technical problems” can be a catch-all, referring to multiple types of issues, including misconfigurations and routing problems. However, it is also sometimes the official explanation given by a government or telecommunications company for an observed Internet disruption.

Rogers

Arguably the most significant Internet disruption so far this year took place on AS812 (Rogers), one of Canada’s largest Internet service providers. At around 0845 UTC on July 8, a near complete loss of traffic was observed, as seen in the figure below.

Internet disruptions overview for Q3 2022

The figure below shows that small amounts of traffic were seen from the network over the course of the outage, but it took nearly 24 hours for traffic to return to normal levels.

Internet disruptions overview for Q3 2022

A notice posted by the Rogers CEO explained that “We now believe we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction early Friday morning. We disconnected the specific equipment and redirected traffic, which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.” A Cloudflare blog post covered the Rogers outage in real-time, highlighting related BGP activity and small increases of traffic.

Chad

A four-hour near-complete Internet outage took place in Chad on August 12, occurring between 1045 and 1300 local time (0945 to 1400 UTC). Authorities in Chad said that the disruption was due to a “technical problem” on connections between Sudachad and networks in Cameroon and Sudan.

Internet disruptions overview for Q3 2022

Unknown

In many cases, observed Internet disruptions are attributed to underlying causes thanks to statements by service providers, government officials, or media coverage of an associated event. However, for some disruptions, no published explanation or associated event could be found.

On August 11, a multi-hour outage impacted customers of US telecommunications provider Centurylink in states including Colorado, Iowa, Missouri, Montana, New Mexico, Utah, and Wyoming, as shown in the figure below. The outage was also visible in a traffic graph for AS209, the associated autonomous system.

Internet disruptions overview for Q3 2022
Internet disruptions overview for Q3 2022

On August 30, satellite Internet provider suffered a global service disruption, lasting between 0630-1030 UTC as seen in the figure below.

Internet disruptions overview for Q3 2022

Conclusion

As part of Cloudflare’s Birthday Week at the end of September, we launched the Cloudflare Radar Outage Center (CROC). The CROC is a section of our new Radar 2.0 site that archives information about observed Internet disruptions. The underlying data that powers the CROC is also available through an API, enabling interested parties to incorporate data into their own tools, sites, and applications. For regular updates on Internet disruptions as they occur and other Internet trends, follow @CloudflareRadar on Twitter.

The status page the Internet needs: Cloudflare Radar Outage Center

Post Syndicated from David Belson original https://blog.cloudflare.com/announcing-cloudflare-radar-outage-center/

The status page the Internet needs: Cloudflare Radar Outage Center

The status page the Internet needs: Cloudflare Radar Outage Center

Historically, Cloudflare has covered large-scale Internet outages with timely blog posts, such as those published for Iran, Sudan, Facebook, and Syria. While we still explore such outages on the Cloudflare blog, throughout 2022 we have ramped up our monitoring of Internet outages around the world, posting timely information about those outages to @CloudflareRadar on Twitter.

The new Cloudflare Radar Outage Center (CROC), launched today as part of Radar 2.0, is intended to be an archive of this information, organized by location, type, date, etc.

Furthermore, this initial release is also laying the groundwork for the CROC to become a first stop and key resource for civil society organizations, journalists/news media, and impacted parties to get information on, or corroboration of, reported or observed Internet outages.

The status page the Internet needs: Cloudflare Radar Outage Center

What information does the CROC contain?

At launch, the CROC includes summary information about observed outage events. This information includes:

  • Location: Where was the outage?
  • ASN: What autonomous system experienced a disruption in connectivity?
  • Type: How broad was the outage? Did connectivity fail nationwide, or at a sub-national level? Did just a single network provider have an outage?
  • Scope: If it was a sub-national/regional outage, what state or city was impacted? If it was a network-level outage, which one?
  • Cause: Insight into the likely cause of the outage, based on publicly available information. Historically, some have been government directed shutdowns, while others are caused by severe weather or natural disasters, or by infrastructure issues such as cable cuts, power outages, or filtering/blocking.
  • Start time: When did the outage start?
  • End time: When did the outage end?

Using the CROC

Radar pages, including the main landing page, include a card displaying information about the most recently observed outage, along with a link to the CROC. The CROC will also be linked from the left-side navigation bar

The status page the Internet needs: Cloudflare Radar Outage Center

Within the CROC, we have tried to keep the interface simple and easily understandable. Based on the selected time period, the global map highlights locations where Internet outages have been observed, along with a tooltip showing the number of outages observed during that period. Similarly, the table includes information (as described above) about each observed outage, along with a link to more information. The linked information may be a Twitter post, a blog post, or a custom Radar graph.

The status page the Internet needs: Cloudflare Radar Outage Center
The status page the Internet needs: Cloudflare Radar Outage Center

As mentioned in the Radar 2.0 launch blog post, we launched an associated API alongside the new site. Outage information is available through this API as well — in fact, the CROC is built on top of this API. Interested parties, including civil society organizations, data journalists, or others, can use the API to integrate the available outage data with their own data sets, build their own related tools, or even develop a custom interface.

Information about the related API endpoint and how to access it can be found in the Cloudflare API documentation.

We recognize that some users may want to download the whole list of observed outages for local consumption and analysis. They can do so by clicking the “Download CSV” link below the table.

The status page the Internet needs (coming soon)

Today’s launch of the Cloudflare Radar Outage Center is just the beginning, as we plan to improve it over time. This includes increased automation of outage detection, enabling us to publish more timely information through both the API and the CROC tool, which is important for members of the community that track and respond to Internet outages. We are also exploring how we can use synthetic monitoring in combination with other network-level performance and availability information to detect outages of popular consumer and business applications/platforms.

And anyone who uses a cloud platform provider (such as AWS) will know that those companies’ status pages take a surprisingly long time to update when there’s an outage. It’s very common to experience difficulty accessing a service, see hundreds of messages on Twitter and message boards about a service being down, only to go to the cloud platform provider’s status page and see everything green and “All systems normal”.

For the last few months we’ve been monitoring the performance of cloud platform providers to see if we can detect when they go down and provide our own, real time status page for them. We believe we can and Cloudflare Radar Outage Center will be extended to include cloud service providers and give the Internet the status page it needs.

The status page the Internet needs: Cloudflare Radar Outage Center

If you have questions about the CROC, or suggestions for features that you would like to see, please reach out to us on Twitter at @CloudflareRadar.

Protests spur Internet disruptions in Iran

Post Syndicated from David Belson original https://blog.cloudflare.com/protests-internet-disruption-ir/

Protests spur Internet disruptions in Iran

Protests spur Internet disruptions in Iran

Over the past several days, protests and demonstrations have erupted across Iran in response to the death of Mahsa Amini. Amini was a 22-year-old woman from the Kurdistan Province of Iran, and was arrested on September 13, 2022, in Tehran by Iran’s “morality police”, a unit that enforces strict dress codes for women. She died on September 16 while in police custody.

Published reports indicate that the growing protests have resulted in at least eight deaths. Iran has a history of restricting Internet connectivity in response to protests, taking such steps in May 2022, February 2021, and November 2019. They have taken a similar approach to the current protests, including disrupting Internet connectivity, blocking social media platforms, and blocking DNS. The impact of these actions, as seen through Cloudflare’s data, are reviewed below.

Impact to Internet traffic

In the city of Sanandij in the Kurdistan Province, several days of anti-government protests took place after the death of Mahsa Amini. In response, the government reportedly disrupted Internet connectivity there on September 19. This disruption is clearly visible in the graph below, with traffic on TCI (AS58224), Iran’s fixed-line incumbent operator, in Sanandij dropping to zero between 1630 and 1925 UTC, except for a brief spike evident between 1715 and 1725 UTC.

Protests spur Internet disruptions in Iran

On September 21, Internet disruptions started to become more widespread, with mobile networks effectively shut down nationwide. (Iran is a heavily mobile-centric country, with Cloudflare Radar reporting that 85% of requests are made from mobile devices.) Internet traffic from Iran Mobile Communications Company (AS197207) started to decline around 1530 UTC, and remained near zero until it started to recover at 2200 UTC, returning to “normal” levels by the end of the day.

Protests spur Internet disruptions in Iran

Internet traffic from RighTel (AS57218) began to decline around 1630 UTC. After an outage lasting more than 12 hours, traffic returned at 0510 UTC.

Protests spur Internet disruptions in Iran

Internet traffic from MTN Irancell (AS44244) began to drop just before 1700 UTC. After a 12-hour outage, traffic began recovering at 0450 UTC.

Protests spur Internet disruptions in Iran

The impact of these disruptions is also visible when looking at traffic at both a regional and national level. In Tehran Province, HTTP request volume declined by approximately 70% around 1600 UTC, and continued to drop for the next several hours before seeing a slight recovery at 2200 UTC, likely related to the recovery also seen at that time on AS197207.

Protests spur Internet disruptions in Iran

Similarly, Internet traffic volumes across the whole country began to decline just after 1600 UTC, falling approximately 40%. Nominal recovery at 2200 UTC is visible in this view as well, again likely from the increase in traffic from AS197207. More aggressive traffic growth is visible starting around 0500 UTC, after the remaining two mobile network providers came back online.

Protests spur Internet disruptions in Iran

DNS blocking

In addition to shutting down mobile Internet providers within the country, Iran’s government also reportedly blocked access to social media platform Instagram, as well as blocking access to DNS-over-HTTPS from open DNS resolver services including Quad9, Google’s 8.8.8.8, and Cloudflare’s 1.1.1.1. Analysis of requests originating in Iran to 1.1.1.1 illustrates the impacts of these blocking attempts.

In analyzing DNS requests to Cloudflare’s resolver for domains associated with leading social media platforms, we observe that requests for instagram.com hostnames drop sharply at 1310 UTC, remaining lower for the rest of the day, except for a significant unexplained spike in requests between 1540 and 1610 UTC. Request volumes for hostnames associated with other leading social media platforms did not appear to be similarly affected.

Protests spur Internet disruptions in Iran

In addition, it was reported that access to WhatsApp had also been blocked in Iran. This can be seen in resolution requests to Cloudflare’s resolver for whatsapp.com hostnames. The graph below shows a sharp decline in query traffic at 1910 UTC, dropping to near zero.

Protests spur Internet disruptions in Iran

The Open Observatory for Network Interference (OONI), an organization that measures Internet censorship, reported in a Tweet that the cloudflare-dns.com domain name, used for DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) connections to Cloudflare’s DNS resolver, was blocked in Iran on September 20. This is clearly evident in the graph below, with resolution volume over DoH and DoT dropping to zero at 1940 UTC. The OONI tweet also noted that the 1.1.1.1 IP address “remains blocked on most networks.” The trend line for resolution over TCP or UDP (on port 53) in the graph below suggests that the IP address is not universally blocked, as there are still resolution requests reaching Cloudflare.

Protests spur Internet disruptions in Iran

Interested parties can use Cloudflare Radar to monitor the impact of such government-directed Internet disruptions, and can follow @CloudflareRadar on Twitter for updates on Internet disruptions as they occur.

Internet disruptions overview for Q2 2022

Post Syndicated from David Belson original https://blog.cloudflare.com/q2-2022-internet-disruption-summary/

Internet disruptions overview for Q2 2022

Internet disruptions overview for Q2 2022

Cloudflare operates in more than 270 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. In many cases, these disruptions can be attributed to a physical event, while in other cases, they are due to an intentional government-directed shutdown. In this post, we review selected Internet disruptions observed by Cloudflare during the second quarter of 2022, supported by traffic graphs from Cloudflare Radar and other internal Cloudflare tools, and grouped by associated cause or common geography.

Optic outages

This quarter, we saw the usual complement of damage to both terrestrial and submarine fiber-optic cables, including one that impacted multiple countries across thousands of miles, and another more localized outage that was due to an errant rodent.

Comcast

On April 25, Comcast subscribers in nearly 20 southwestern Florida cities experienced an outage, reportedly due to a fiber cut. The traffic impact of this cut is clearly visible in the graph below, with Cloudflare traffic for these cities dropping to zero between 1915–2050 UTC (1515–1850 local time).

Internet disruptions overview for Q2 2022

Not only did the fiber cut force a significant number of Comcast subscribers offline, but it also impacted the types of traffic observed across Comcast as a whole. The graphs below illustrate the mix of mobile vs desktop clients, as well as IPv4 vs. IPv6 request volume across AS7922, Comcast’s primary autonomous system. During the brief disruption period, the percentage of Comcast traffic from mobile devices increased, while desktop devices dropped, and the percentage of IPv4 traffic dropped, with a corresponding increase in IPv6 traffic share.

Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022

South Africa

On the morning of May 17, Telkom SA, a South African telecommunications provider, tweeted an “important notice” to customers, noting that “Damage to a Fibre cable was detected on the Telkom network around 8:00am on Tuesday, 17 May 2022.” and outlining the impacted services and geographies. The graphs below show the impact to Cloudflare traffic from the Telkom autonomous system in three South African provinces. The top graph shows the impact to traffic in Gauteng, while the lower graph shows the impact in Limpopo and North West. Across all three, traffic falls at 0600 UTC (0800 local time), recovering around 1300 UTC (1500 local time). Telkom SA did not provide any additional information on where the fiber cut occurred or what caused it.

Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022

Venezuela

Although unconfirmed, a fiber cut was suspected to be the cause of an Internet disruption experienced by CANTV subscribers in Venezuela on May 19, the latest of several such incidents affecting that provider. Although the fiber cut reportedly impacted subscribers in multiple states, the most significant impact was measured in Falcón, as shown in the graph below. In this state, traffic dropped precipitously at 1800 UTC (1400 local time), finally recovering approximately 24 hours later.

Internet disruptions overview for Q2 2022

AAE-1 & SMW-5

Just after 1200 UTC on Tuesday, June 7, the Africa-Asia-Europe-1 (AAE-1) and SEA-ME-WE-5 (SMW-5) submarine cables suffered cable cuts, impacting Internet connectivity for millions of Internet users across multiple countries in the Middle East and Africa, as well as thousands of miles away in Asia. Although specific details are sparse, the cable damage reportedly occurred in Egypt – both of the impacted cables land in Abu Talat and Zafarana, which also serve as landing points for a number of other submarine cables.

The Cloudflare Radar graphs below illustrate the impact of these cable cuts across Africa, Asia, and the Middle East. Given that the associated traffic disruption only lasted several hours, the damage to these cables likely occurred on land, after they came ashore. More details on this event can be found in the “AAE-1 & SMW5 cable cuts impact millions of users across multiple countries” blog post.

Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022

Castor canadensis

Finally, on June 13, a beaver was responsible for an outage that impacted Internet users in British Columbia, Canada. According to a published report, a beaver gnawed its way through a tree, causing it to fall on both power lines and a Telus fiber optic cable. The damage to the fiber optic cable affected connectivity customers in over a dozen communities across British Columbia, including those using CityWest (AS18988), a utility company that uses the Telus cable. In the graph below, the impact of the damage to the fiber optic cable is clearly visible, with no traffic to Cloudflare from CityWest subscribers in British Columbia between 1800 UTC on June 7 until 0310 UTC on June 8 (1100–2010 local time).

Internet disruptions overview for Q2 2022

School’s in, Internet’s out

Nationwide Internet shutdowns have, unfortunately, become a popular approach taken by authoritarian regimes over the past half dozen years to prevent cheating on secondary school exams. It is not clear that this heavy-handed tactic is actually effective in preventing cheating, but the associated damage to the national economies has been estimated to be in the tens to hundreds of millions of US dollars, depending on the duration and frequency of the shutdowns.

This year, governments in Sudan and Syria implemented a number of multi-hour shutdowns in late May into June, while Algeria’s government appears to have resorted to more targeted content blocking. Additional details on these Internet disruptions can be found in the recent “Exam time means Internet disruptions in Syria, Sudan and Algeria” blog post.

Starting on May 30, Syria implemented the first of four nationwide Internet shutdowns, the last of which occurred on June 12, as seen in the graph below. Interestingly, we have observed that these shutdowns tend to be “asymmetric” in nature — that is, inbound traffic (into the country) is disabled, but egress traffic (from the country) remains. One effect of this is visible as spikes in the DNS graph below. During three of the four shutdowns, requests to Cloudflare’s 1.1.1.1 resolver from clients in Syria increased because DNS queries were able to exit the country, but responses couldn’t return, leading to retry floods.

Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022

In Sudan, daily shutdowns were implemented 0530-0830 UTC (0730–1030 local time) between June 11 and June 22, except for June 17. (It isn’t clear why that date was skipped.) The graph below shows that these shutdowns were nationwide, but not complete, as traffic from the country did not drop to zero.

Internet disruptions overview for Q2 2022

In Algeria, exams took place June 12 through June 16. In the past, the country has implemented nationwide shutdowns, but after recognizing the enormous cost to the economy, the government has apparently chosen an alternate tactic this year. The graph below shows nominal drops in country-level traffic during the two times each day that the exams took place—0730–1000 UTC (0830–1100 am local time) and 1330–1600 UTC (1430–1700 local time). These drops in traffic are likely more indicative of a content-blocking approach, instead of a broad Internet shutdown.

Internet disruptions overview for Q2 2022

On June 27, the Kurdistan Regional Government in Iraq began to implement twice-weekly (Mondays and Thursday) multi-hour regional Internet shutdowns, expected to last for a four-week period. The shutdowns are intended to prevent cheating on high school final exams, according to a published report, and are scheduled for 0630–1030 am local time (0330–0730 UTC). The graph below shows the impact to traffic from three governorates in Kurdistan, with traffic dropping to near zero in all three areas during the duration of the shutdowns.

Internet disruptions overview for Q2 2022

Government-guided

In addition to shutting down the Internet to prevent cheating on exams, governments have also been known to use shutdowns as a tool to limit or control communication around elections, rallies, protests, etc. During the second quarter, we observed several such shutdowns of note.

On April 10, following the blocking of social networks, VPN providers, and cloud platforms, the government of Turkmenistan implemented a near complete Internet shutdown, starting at 1400 UTC. Apparently related to criticism over the recent presidential election, the disruption lasted nearly 40 hours, as traffic started to return around 0700 UTC on April 12. The graphs below show the impact of the shutdown at a country level, as well as at two major network providers within the country, Telephone Network of Ashgabat CJSC (AS51495) and TurkmenTelecom (AS20661).

Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022

A month and a half later, on May 25, an Internet disruption was observed in Pakistan amid protests led by the country’s former Prime Minister. The disruption lasted only two hours, and was limited in scope — it was not a nationwide shutdown. (Telecom providers claimed that it was due to problems with a web filtering system.) At a national level, the impact of the disruption is visible as a slight drop in traffic.

Internet disruptions overview for Q2 2022

In the cities of Lahore and Karachi, the disruption is visible a little more clearly, as is the rapid recovery in traffic.

Internet disruptions overview for Q2 2022

The impact of the disruption is most evident at a network level, as seen in the graphs below. Cyber Internet Services (AS9541) saw a modest drop in traffic, while Mobilink (AS45669) experienced a near complete outage.

Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022

Closing out the quarter, a communications blackout, including an Internet shutdown, was imposed in Sudan on June 30 as protestors staged rallies against the country’s military leadership. This shutdown follows similar disruptions seen in October 2021 after the military toppled the transitional government and attempted to limit protests, as well the shutdowns seen earlier in June as the government attempted to prevent cheating on exams. The graphs below show that the shutdown started at 0600 UTC (0800 local time) and initially ended almost 12 hours later at 1740 UTC (1940 local time). Connectivity returned for approximately three hours, with traffic again dropping to near-zero levels again around 2040 UTC (2240 local time). This second outage remained active at the end of the day.

As a complete nationwide shutdown, the impact is also visible in the loss of traffic at major local Internet providers including MTN, Sudatel, Kanartel, and Sudanese Mobile Telephone (SDN Mobitel / ZAIN).

Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022

Infrastructure issues

In addition to fiber/cable cuts, as discussed above, problems with other infrastructure, whether due to fires, electrical issues, or maintenance, can also disrupt Internet services.

Around 2030 local time on April 6 (0030 UTC on April 7), a fire erupted at the Costa Sur generation plant, one of the largest power plants in Puerto Rico, resulting in a widespread power outage across the island territory. This island-wide outage caused a significant interruption to Internet services, clearly visible in Cloudflare traffic data. The graph below shows that as the power failed, traffic from Puerto Rico immediately fell by more than half. The regular diurnal pattern remained in place, albeit at lower levels, over the next three days, with traffic returning to “normal levels” three days later. By April 10, Luma Energy reported that it had restored electrical power to 99.7% of its 1.5M customers.

Internet disruptions overview for Q2 2022

The impact of the Internet service disruption is also fairly significant when viewed at a network level. The graphs below show traffic for Datacom Caribe/Claro (AS10396) and Liberty Cablevision of Puerto Rico (AS14638). At Datacom Caribe/Claro, traffic immediately fell by more than half, while Liberty Cablevision traffic declined approximately 85%.

Internet disruptions overview for Q2 2022
Internet disruptions overview for Q2 2022

On the evening of May 3, Swiss telecom provider Swisscom tweeted that there had been an interruption to Internet service following maintenance work. A published report noted that the interruption occurred between 2223–2253 local time (2023–2053 UTC), and the graph below shows a complete loss of traffic, but quick recovery, during that 30-minute window. Beyond citing maintenance work, Swisscom did not provide any additional details about the Internet disruption.

Internet disruptions overview for Q2 2022

Iran

Iran has a history of both nationwide and regional Internet shutdowns, as well as connectivity disruptions due to infrastructure damage.

On May 6, the government disrupted Internet connectivity in Khuzestan province, reportedly in response to mass protests around shortages of bread and water. It was reported that mobile data had been cut off locally, and that fixed connectivity speeds were significantly reduced. To this end, we observed a drop in traffic for Irancell (AS44244) (a mobile network provider) in Khuzestan starting around 1000 UTC as seen in the graph below.

Internet disruptions overview for Q2 2022

A similar disruption affecting Irancell, occurring amid reports of ongoing protests in the country, was observed on May 12, with lower peak traffic during the day, and a further drop around 1800 UTC.

Internet disruptions overview for Q2 2022

Near-complete Internet outages were observed on multiple Iranian network providers on May 9 between 1300–1440 UTC (1730–1910 local time), as illustrated in the graph below. Impacted providers included Atrin Information & Communications Technology Company (AS39650), AryaSat (AS43343), Ariana Gostar Spadana (AS48309), and Pirooz Leen (AS51759). All of these networks share Fanaptelecom (AS24631) as an upstream provider, which, as the graph shows, was also experiencing an outage. No root cause for the Fanaptelecom outage was available.

Internet disruptions overview for Q2 2022

Mobile provider Mobinnet (AS50810) experienced a multi-hour Internet disruption on May 14, lasting from 1230–1530 UTC (1700–2000 local time). According to a tweet from Mobinnet, the disruption was due to a “widespread cyber attack of foreign origin”.

Internet disruptions overview for Q2 2022

Ukraine

Now more than four months into the war in Ukraine, the Internet continues to be an active battlefield, with ongoing Internet outages in multiple cities and across multiple networks. However, we want to highlight here two similar events observed during the second quarter.

The Russian-occupied city of Kherson experienced a near-complete Internet outage between 1600 UTC (1900 local time) on April 30 and 0430 UTC (0730 local time) on May 4. According to social media posts from Ukraine’s vice Prime-Minister Mykhailo Fedorov and the State Service of Special Communications and Information Protection, the outage was caused by “interruptions of fiber-optic trunk lines and disconnection from the power supply of equipment of operators in the region”. The graph below shows effectively no traffic for Kherson for approximately 24 hours after the disruption began, followed by a nominal amount of traffic for the next several days.

Internet disruptions overview for Q2 2022

Around the time that the nominal amount of traffic returned, we also observed a shift in routing for an IPv4 prefix announced by AS47598 (Khersontelecom). As shown in the table below, prior to the outage, it reached the Internet through several other Ukrainian network providers, including AS12883, AS3326, and AS35213. However, as traffic returned, its routing path now showed a Russian network, AS201776 (Miranda) as the upstream provider. The path through Miranda also includes AS12389 (Rostelecom), which bills itself as “the largest digital services provider in Russia”.

Peer AS Last Update AS Path
AS1299 (TWELVE99 Arelion, fka Telia Carrier) 5/1/2022 16:02:26 1299 12389 201776 47598
AS6777 (AMS-IX-RS) 4/28/2022 11:23:33 12883 47598

As the disruption ended on May 4, we observed updates to Khersontelecom’s routing path that enabled it to return to reaching the global Internet through non-Russian upstream providers.

Peer AS Last Update AS Path
AS174 (COGENT-174) 5/4/2022 05:56:27 174 3326 3326 3326 47598
AS1273 (CW Vodafone Group PLC) 5/4/2022 03:11:25 1273 12389 201776 47598

Additional details about this outage and re-routing event can be found in the “Tracking shifts in Internet connectivity in Kherson, Ukraine” blog post.

A month later, on May 30, we again observed a significant Internet disruption in Kherson starting at 1435 UTC (1735 local time). And once again, we observed updated routing for Khersontelecom, as it shifted from Ukrainian upstream providers to Russian ones. As of the end of June, the Internet disruption in Kherson and the routing through Russian upstream providers both remain firmly in place, although the loss of traffic has not been nearly as significant as the April/May disruption.

Internet disruptions overview for Q2 2022

Peer AS Last Update AS Path
AS4775 (Globe Telecoms) 5/30/2022 13:56:22 4775 1273 12389 201776 47598
AS9002 (RETN-AS) 5/30/2022 09:58:16 9002 3326 47598

Conclusion

This post is by no means an exhaustive review of the Internet outages, shutdowns, and disruptions that have occurred throughout the second quarter. Some were extremely brief or limited in scope, while others were observed but had no known or publicly conjectured underlying cause. Having said that, it is important to bring increased visibility to these events so that the community can share information on what is happening, why it happened, and what the impact was — human, financial, or otherwise.

Follow @CloudflareRadar on Twitter for updates on Internet disruptions as they occur, and find up-to-date information on Internet trends using Cloudflare Radar.

Internet Explorer, we hardly knew ye

Post Syndicated from David Belson original https://blog.cloudflare.com/internet-explorer-retired/

Internet Explorer, we hardly knew ye

Internet Explorer, we hardly knew ye

On May 19, 2021, a Microsoft blog post announced that “The future of Internet Explorer on Windows 10 is in Microsoft Edge” and that “the Internet Explorer 11 desktop application will be retired and go out of support on June 15, 2022, for certain versions of Windows 10.” According to an associated FAQ page, those “certain versions” include Windows 10 client SKUs and Windows 10 IoT. According to data from Statcounter, Windows 10 currently accounts for over 70% of desktop Windows market share on a global basis, so this “retirement” impacts a significant number of Windows systems around the world.

As the retirement date for Internet Explorer 11 has recently passed, we wanted to explore several related usage trends:

  • Is there a visible indication that use is declining in preparation for its retirement?
  • Where is Internet Explorer 11 still in the heaviest use?
  • How does the use of Internet Explorer 11 compare to previous versions?
  • How much Internet Explorer traffic is “likely human” vs. “likely automated”?
  • How do Internet Explorer usage patterns compare with those of Microsoft Edge, its replacement?

The long goodbye

Publicly released in January 2020, and automatically rolled out to Windows users starting in June 2020, Chromium-based Microsoft Edge has become the default browser for the Windows platform, intended to replace Internet Explorer. Given the two-year runway, and Microsoft’s May 2021 announcement, we would expect to see Internet Explorer traffic decline over time as users shift to Edge.

Looking at global request traffic to Cloudflare from Internet Explorer versions between January 1 and June 20, 2022, we see in the graph below that peak request volume for Internet Explorer 11 has declined by approximately one-third over that period. The clear weekly usage pattern suggests higher usage in the workplace than at home, and the nominal decline in traffic year-to-date suggests that businesses are not rushing to replace Internet Explorer with Microsoft Edge. However, we expect traffic from Internet Explorer 11 to drop more aggressively as Microsoft rolls out a two-phase plan to redirect users to Microsoft Edge, and then ultimately disable Internet Explorer. Having said that, we do not expect Internet Explorer 11 traffic to ever fully disappear for several reasons, including Microsoft Edge’s “IE Mode” representing itself as Internet Explorer 11, the ongoing usage of Internet Explorer 11 on Windows 8.1 and Windows 7 (which were out of scope for the retirement announcement), and automated (bot) traffic masquerading as Internet Explorer 11.

Internet Explorer, we hardly knew ye

It is also apparent in the graph above that traffic from earlier versions of Internet Explorer has never fully disappeared. (In fact, we still see several million requests each day from clients purporting to be Internet Explorer 2, which was released in November 1995 — over a quarter-century ago.) After version 11, Internet Explorer 7, first released in October 2006 and last updated in May 2009, generates the next largest volume of requests. Traffic trends for this version have remained relatively consistent. Internet Explorer 9 was the next largest traffic generator through late May, when Internet Explorer 6 seemed to stage a comeback. (Internet Explorer 7 saw a slight bump in traffic at that time as well.)

Where is Internet Explorer 11 used?

Perhaps unsurprisingly, the United States has accounted for the largest volume of Internet Explorer 11 requests year-to-date. Similar to the global observation above, daily peak request traffic has declined by approximately one-third. With request volume approximately one-fourth that seen in the United States, Japan ostensibly has the next largest Internet Explorer 11 user base. (And published reports note that Internet Explorer’s retirement is likely to cause Japan headaches ‘for months’” because local businesses and government agencies didn’t take action in the months ahead of the event.)

Internet Explorer, we hardly knew ye

However, unusual shifts in Brazil’s request volume, seen in the graph above, are particularly surprising. For several weeks in January, Internet Explorer 11 traffic from the country appears to quadruple, with the same behavior seen from early May through mid-June, as well as a significant spike in March. Classifying the request traffic by bot score, as shown in the graph below, makes it clear that the observed increases are the result of automated (bot) traffic presenting itself as coming from Internet Explorer 11.

Internet Explorer, we hardly knew ye

Further, analyzing this traffic to see what percentage of requests were mitigated by Cloudflare’s Web Application Firewall, we find that the times when the mitigation percentage increased, as shown in the graph below, align very closely with the periods where we observed the higher levels of automated (bot) traffic. This suggests that the spikes in Internet Explorer 11 traffic coming from Brazil that were seen over the last six months were from a botnet presenting itself as that version of the browser.

Internet Explorer, we hardly knew ye

Bot or not

Building on the Brazil analysis, breaking out the traffic for each version by associated bot score can help us better understand the residual traffic from long-deprecated versions of Internet Explorer shown above. For requests with a bot score that characterizes the traffic as “likely human”, the graph below shows clear weekly traffic patterns for versions 11 and 7, suggesting that the traffic is primarily driven by systems primarily in use on weekdays, likely by business users. For Internet Explorer 7, that traffic pattern becomes more evident starting in mid-February, after a significant decline in associated request volume.

Internet Explorer, we hardly knew ye

Interestingly, that decline in “likely human” Internet Explorer 7 request volume aligns with an increase in “likely automated” (bot) request volume for that version, visible in the graph below. Given that the “likely human” traffic didn’t appear to migrate to another version of Internet Explorer, the shift may be related to improvements to the machine learning model that powers bot detection that were rolled out in the January/February time frame. It is also interesting to note that “likely automated” request volume for both Internet Explorer 11 and 7 has been extremely similar since mid-March. It is not immediately clear why this is the case.

Internet Explorer, we hardly knew ye

We can also use this data to understand what percentage of the traffic from a given version of Internet Explorer is likely to be automated (coming from bots). The graph below highlights the ratios for Internet Explorer 11 and 7. For version 11, we can see that the percentage has grown from around 60% at the start of 2022 to around 80% in June. For version 7, it starts the year in the 40% range, and more than doubles to over 80% in February and remains consistent at that level.

Internet Explorer, we hardly knew ye

However, when we look at firewall mitigated traffic percentages, we don’t see the same clear alignment of trends as was visible for Brazil, as discussed above. In addition, only a fraction of the “likely automated” traffic was mitigated, suggesting that the automated traffic is split between being generated by bots and other non-malicious tools, such as performance testing.

Internet Explorer, we hardly knew ye

Internet Explorer versions 6 & 9 were also discussed above, with respect to driving the largest volume of requests. However, when we examine the “likely automated” request ratios for these two browsers, we find that most of their traffic appears to be bot-driven. Internet Explorer 6 started 2022 at around 80%, growing to 95% in June. In contrast, Internet Explorer 9 starts the year around 90%, drops to 60% at the end of January, and then gradually increases back to the 75-80% range.

Internet Explorer, we hardly knew ye

As Internet Explorer 6’s “likely automated” traffic has increased, the fraction of it that was mitigated has increased as well. The small bumps visible in the graph above align with the larger spikes in the graph below, potentially due to brief bursts of bot activity. In contrast, mitigated Internet Explorer 9 traffic has remained relatively consistent, even as its automated request percentage dropped and then gradually increased.

Internet Explorer, we hardly knew ye

For the oldest, long-deprecated versions of Internet Explorer, automated traffic frequently comprises more than 80% of request volume, reaching 100% on multiple days year-to-date. Mitigated traffic generally amounted to under 30% of request volume, although Internet Explorer 2 frequently increased to the 50% range, spiking as high as 90%.

Edging into the future

As Microsoft stated, “the future of Internet Explorer on Windows 10 is in Microsoft Edge.” Given that, we wanted to understand the usage patterns of Microsoft Edge. Similar to the analysis above, we looked at request volumes for the last ten versions of the browser year-to-date. The graph below clearly illustrates strong enterprise usage of edge, with weekday peaks, and lower traffic on the weekends. In addition, the four-week major release cycle cadence is clearly evident, with a long tail of usage extending across eight weeks due to enterprise customers who need an extended timeline to manage updates.

Internet Explorer, we hardly knew ye

Having said that, in analyzing the split by bot score for these Edge versions, we note that only around 80% of requests are classified as “likely human” for about eight weeks after a given version is released, after which it gradually tapers to around 60%. The balance is classified as “likely automated”, suggesting that those who develop bots and other automated processes recognize the value in presenting their user agents as the latest version of Microsoft’s web browser. For Edge, there does not appear to be any meaningful correlation between firewall mitigated traffic percentages and “likely automated” traffic percentages or the traffic cycles visible in the graph above.

Conclusion

Analyzing traffic trends from deprecated versions of Internet Explorer brought to mind the “I’m not dead yet” scene from Monty Python and the Holy Grail with these older versions of the browser claiming to still be alive, at least from a traffic perspective. However, categorizing this traffic to better understand the associated bot/human split showed that the majority of Internet Explorer traffic seen by Cloudflare, including for Internet Explorer 11, is apparently not coming from actual browser clients installed on user systems, but rather from bots and other automated processes. For the automated traffic, analysis of firewall mitigation activity shows that the percentage likely coming from malicious bots varies by version.

As Microsoft executes its planned two-phase approach for actively moving users off of Internet Explorer, it will be interesting to see how both request volumes and bot/human splits for the browser change over time – check back later this year for an updated analysis.