Tag Archives: Consumer Services

Fresh insights from old data: corroborating reports of Turkmenistan IP unblocking and firewall testing

Post Syndicated from Luke Valenta original https://blog.cloudflare.com/fresh-insights-from-old-data-corroborating-reports-of-turkmenistan-ip/

Here at Cloudflare, we frequently use and write about data in the present. But sometimes understanding the present begins with digging into the past.  

We recently learned of a 2024 turkmen.news article (available in Russian) that reports Turkmenistan experienced “an unprecedented easing in blocking,” causing over 3 billion previously-blocked IP addresses to become reachable. The same article reports that one of the reasons for unblocking IP addresses was that Turkmenistan may have been testing a new firewall. (The Turkmen government’s tight control over the country’s Internet access is well-documented.) 

Indeed, Cloudflare Radar shows a surge of requests coming from Turkmenistan around the same time, as we’ll show below. But we had an additional question: Does the firewall activity show up on Radar, as well? Two years ago, we launched the dashboard on Radar to give a window into the TCP connections to Cloudflare that close due to resets and timeouts. These stand out because they are considered ungraceful mechanisms to close TCP connections, according to the TCP specification. 

In this blog post, we go back in time to share what Cloudflare saw in connection resets and timeouts. We must remind our readers that, as passive observers, there are limitations on what we can glean from the data. For example, our data can’t reveal attribution. Even so, the ability to observe our environment can be insightful. In a recent example, our visibility into resets and timeouts helped corroborate reports of large-scale blocking and traffic tampering by Russia.

Turkmenistan requests where there were none before

Let’s look first at the number of requests, since those should increase if IP addresses are unblocked. In mid-June 2024 Cloudflare started receiving a noticeable increase in HTTP requests, consistent with reports of Turkmenistan unblocking IPs.


Source: Cloudflare Radar

Overall TCP resets and timeouts

The Transmission Control Protocol (TCP) is a lower-layer mechanism used to create a connection between clients and servers, and also carries 70% of HTTP traffic to Cloudflare. A TCP connection works much like a telephone call between humans, who follow graceful conventions to end a call—and who are acutely aware when conventions are broken if a call ends abruptly. 

TCP also defines conventions to end the connection gracefully, and we developed mechanisms to detect when they don’t. An ungraceful end is triggered by a reset instruction or a timeout. Some are due to benign artifacts of software design or human user behaviours. However, sometimes they are exploited by third parties to close connections in everything from school and enterprise firewalls or software, to zero-rating on mobile plans, to nation-state filtering.

When we look at connections from Turkmenistan, we see that on June 13, 2024, the combined proportion of the four coloured regions increases; each coloured region represents ungraceful ends at a distinct stage of the connection lifetime. In addition to the combined increase, the relative proportions between stages (or colours) changes as well.


Source: Cloudflare Radar

Further changes appeared in the weeks that followed. Among them are an increase in Post-PSH (orange) anomalies starting around July 4; a reduction in Post-ACK (light blue) anomalies around July 13; and an increase in anomalies later in connections (green) starting July 22.


Source: Cloudflare Radar

The shifts above could be explained by a large firewall system. It’s important to keep in mind that data in each of the connection stages (captured by the four coloured regions in the graphs) can be explained by browser implementations or user actions. However, the scale of the data would need a great number of browsers or users doing the same thing to show up. Similarly, individual changes in behaviour would be lost unless they occur in large numbers at the same time.

Digging down to individual networks

We’ve learned that it can be helpful to look at the data for individual networks to reveal common patterns between different networks in different regions operated by single entities

Looking at individual networks within Turkmenistan, trends and timelines appear more pronounced. July 22 in particular sees greater proportions of anomalies associated with the Server Name Indication, or domain name, rather than the IP address (dark blue), although the connection stage where the anomalies appear varies by individual network.

The general Turkmenistan trends are largely mirrored in connections from AS20661 (TurkmenTelecom), indicating that this autonomous system (AS) accounts for a large proportion of Turkmenistan’s traffic to Cloudflare’s network. There is a notable reduction in Post-ACK (light blue) anomalies starting around July 26.


Source: Cloudflare Radar

A different picture emerges from AS51495 (Ashgabat City Telephone Network). Post-ACK anomalies almost completely disappear on July 12, corresponding with an increase in anomalies during the Post-PSH stage. An increase of anomalies in the Later (green) connection stage on July 22 is apparent for this AS as well.


Source: Cloudflare Radar

Finally, for AS59974 (Altyn Asyr), you can see below that there is a clear spike in Post-ACK anomalies starting July 22. This is the stage of the connection where a firewall could have seen the SNI, and chooses to drop the packets immediately, so they never reach Cloudflare’s servers.


Source: Cloudflare Radar

Timeouts and resets in context, never isolation

We’ve previously discussed how to use the resets and timeouts data because, while useful, it can also be misinterpreted. Radar’s data on resets and timeouts is unique among operators, but in isolation it’s incomplete and subject to human bias.

Take the figure above for AS59974 where Post-ACK (light blue) anomalies markedly increased on July 22. The Radar view is proportional, meaning that the increase in proportion could be explained by greater numbers of anomalies – but could also be explained, for example, by a smaller number of valid requests. Indeed, looking at the HTTP request levels for the same AS, there was a similarly pronounced drop starting on the same day, as shown below. 


Source: Cloudflare Radar

If we look at the same two graphs before July 22, however, rates of reset and timeout anomalies do not appear to mirror the very large shifts up and down in HTTP requests.

Looking ahead can also mean looking behind

These charts from Radar above offer a way to analyze news events from a different angle, by looking at requests and TCP connection resets and timeouts. Does this data tell us definitively that new firewalls were being tested in Turkmenistan? No. But the trends in the data are consistent with what we could expect to see if that were the case.

If thinking about ways to use the resets and timeouts data going forward, we’d encourage also looking at the data in retrospect—or even further past to improve context.

A natural question might be, for example, “If Turkmenistan stopped blocking IPs in mid-2024, what did the data say beforehand?” The figure below captures October and November 2023. (The red-shaded region contains missing data due to the Nov. 2 Cloudflare control plane and metrics outage.) Signals about the Internet in Turkmenistan were evolving well before the news article that prompted us to look.


Source: Cloudflare Radar

What’s next?

To learn more, see our guide about how to use the resets and timeouts data available on Radar, as well as the technical details about our third-party tampering measurement and some perspectives by a former intern who helped drive the study. 

We’re proud to offer a unique view of TCP connection anomalies on Radar. It’s a testament to the long-lived benefits that emerge when approaching Internet measurement as a science. In keeping with the open spirit of science, we’ve also shared how we detect and log resets and timeouts so that others can reproduce the observability on their servers, whether by hobbyists or other large operators.

Online outages: Q3 2025 Internet disruption summary

Post Syndicated from David Belson original https://blog.cloudflare.com/q3-2025-internet-disruption-summary/

In the third quarter, we observed Internet disruptions with a wide variety of known causes, as well as several with no definitive or published cause. Once again, we unfortunately saw a number of government-directed shutdowns, including exam-related shutdowns in Sudan, Syria, and Iraq. Cable cuts, both submarine and terrestrial, caused Internet outages, including one caused by a stray bullet. A rogue contractor, among other events, caused power outages that impacted Internet connectivity. Damage from an earthquake and a fire caused service disruptions, as did a targeted cyberattack. And a myriad of technical issues, including issues with China’s Great Firewall, resulted in traffic losses across multiple countries.

As we have noted in the past, this post is intended as a summary overview of observed and confirmed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter. A larger list of detected traffic anomalies is available in the Cloudflare Radar Outage Center. These anomalies are detected through significant deviations from expected traffic patterns observed across our network. Note that both bytes-based and request-based traffic graphs are used within the post to illustrate the impact of the observed disruptions — the choice of metric to include was generally made based on which better illustrated the impact of the disruption.

Government-directed shutdowns

Sudan

Regular drops in traffic from Sudan were observed between 12:00-15:00 UTC (14:00-17:00 local time) each day from July 7-10. Partial outages were observed at Sudatel (AS15706), and near-complete outages at SDN Mobitel (AS36998) and MTN Sudan (AS36972). Similar drops were also seen in traffic to our 1.1.1.1 DNS resolver from these impacted ASNs.

We have observed Sudan implementing government-directed Internet shutdowns in the past (2021, 2022), and given that the timing aligns with the last four days of postponed 2024 secondary school certificate examinations, in addition to fitting the pattern of short-duration disruptions repeating across multiple days, we believe that these drops in traffic were exam-related shutdowns as well. 

Syria

In our second quarter post, we covered the cellular connectivity-focused exam-related Internet shutdowns that Syria chose to implement this year in an effort to limit their impact. During the second quarter, the shutdowns associated with the “Basic Education Certificate” took place on June 21, 24, and 29 between 05:15 – 06:00 UTC (08:15 – 09:00 local time). Exams and associated shutdowns for the “Secondary Education Certificate” were scheduled to take place between July 12 and August 3, and during that period, we observed six additional Internet disruptions in Syria on July 12, 17, 21, 28, 31, and August 3, as shown in the graph below.

At the end of the exam period, the Syrian Ministry of Education posted a Telegram message that was presumably intended to justify the shutdowns, and the focus on cellular connectivity. Translated, it said in part:

As part of its efforts to ensure the integrity of the examination process, and in coordination with relevant authorities, the Ministry of Education was able to uncover organized exam cheating networks in three examination centers in Lattakia Governorate. These networks used advanced electronic technologies and devices in their attempt to manipulate the exam process.

The network was seized in cooperation with the Lattakia Education Directorate, following close monitoring and detection of suspicious attempts. It was found that members of the network used small earphones, wireless communication devices, and mobile phones equipped with advanced transmission and reception technologies, which contradict educational values and violate the integrity of the examination process and the principle of justice.

Venezuela 

A slightly more unusual government directed shutdown took place in Venezuela on August 18 when Venezuelan provider SuperCable (AS22313) ceased service. An X post from Venezuelan industry watcher VE sin Filtro published a notification from CONATEL, the National Commission of Telecommunications in Venezuela, that notified SuperCable that as of March 14, 2025, its authority to operate in the country had been revoked, and established a 60 day transition period so that users could find another provider. Another X post from VE sin Filtro shared an email that SuperCable subscribers received from the company announcing the end of the service and, and noted that half an hour after the email was sent, subscribers were left without Internet connectivity. Traffic began to fall at 15:00 UTC (11:00 local time), and was gone after 15:30 UTC (11:30 local time). Connectivity remained shut down through the end of the quarter.

Interestingly, we did not see a corresponding full loss of announced IP address space when traffic disappeared. However, such full losses did occur between August 19-21, and again briefly on September 16. The number of announced /24s (blocks of 256 IPv4 addresses) fell from 95 to 63 on September 25, and remained at that level through the end of the quarter.

Iraq

Similar to Syria, we covered the latest rounds of exam-related Internet shutdowns in Iraq in our second quarter blog post. In that post, we noted that the shutdowns in the main part of the country ran until July 3 for preparatory school exams, and through July 6 in the Kurdistan region. These can be seen in the graph below.

The Kurdistan Regional Government in Iraq ordered Internet services to be suspended on August 23 between 03:30 and 04:45 UTC (6:30-7:45 local time), and again every Saturday, Monday, and Wednesday until September 8 to prevent cheating on the second round of grade 12 exams. Similar to last quarter, KNET (AS206206), Newroz Telecom (AS21277), IQ Online (AS48492), and KorekTel (AS59625) were impacted by the ordered shutdowns.

In the main part of the country, starting on August 26, the latest round of Internet shutdowns for high school exams began, scheduled through September 13, taking place between 03:00-05:00 UTC (06:00-08:00 local time). Networks impacted by these shutdowns included Earthlink (AS199739), Asiacell (AS51684), Zainas (AS59588), Halasat (AS58322), and HulumTele (AS203214).

Afghanistan

In mid-September, the Taliban ordered the shutdown of fiber optic Internet connectivity in multiple provinces across Afghanistan, as part of a drive to “prevent immorality”. It was the first such ban issued since the Taliban took full control of the country in August 2021. As many as 15 provinces experienced shutdowns, and these regional shutdowns blocked Afghani students from attending online classes, impacted commerce and banking, and limited access to government agencies and institutions such as passport and registration offices, customs offices.

Less than two weeks later, just after 11:30 UTC (16:00 local time) on Monday, September 29, 2025, subscribers of wired Internet providers in Afghanistan experienced a brief service interruption, lasting until just before 12:00 UTC (16:30 local time). Mobile providers Afghan Wireless (AS38472) and Etisalat (AS131284) remained available during that period. However, just after 12:30 UTC (17:00 local time), the Internet was completely shut down, taking the country completely offline.

These shutdowns are reviewed in more detail in our September 30 blog post, Nationwide Internet shutdown in Afghanistan extends localized disruptions. Connectivity was restored around 11:45 UTC (16:15 local time) on October 1.

Fiber optic cable damage

Dominican Republic

On July 7, a post on X from Claro alerted subscribers to a service disruption caused by damage to two fiber optic cables. According to a subsequent post, one was damaged by work being done by CORAAVEGA (La Vega Water And Sewerage Corporation) and the other by work being done by the Dominican Electric Transmission Company. As a result of the damage, traffic from Claro (AS6400) began to drop just before 16:00 UTC (12:00 local time), falling just over two-thirds compared to the prior week. Claro’s technicians were able to quickly locate the faults and repair them, with traffic recovering around 18:00 UTC (14:00 local time).

Angola

Between 12:45-15:45 UTC (13:45-16:45 local time) on July 19, users in Angola experienced an Internet disruption, with Unitel Angola (AS37119) experiencing as much as a 95% drop in traffic as compared to the previous week, and Connectis (AS327932) suffering a complete outage. According to an X post from Unitel Angola, it “was caused by a disruption at our partner Angola Cables, resulting from public road works that affected the national fiber optic interconnections.

However, the timing of the disruption coincided with protests over the rise in diesel fuel prices, and local non-governmental organizations disputed Unitel Angola’s explanation, claiming that it was actually due to a government-directed Internet shutdown. Multiple Angolan network providers experienced a drop in announced IP address space during the period the Internet disruption occurred, and analysis of routing information for these networks finds that they share Angola Cables (AS37468) as an upstream provider, lending some credence to the explanation from Unitel Angola.

Haiti

Digicel Haiti (AS27653) is no stranger to Internet disruptions caused by damage to both terrestrial and submarine cables, experiencing such problems during the first and second quarters of 2025, as well as first, second, and third quarters of 2024. The most recent such disruption occurred on August 26, when they experienced two different cuts on their fiber optic infrastructure, according to an X post from the company’s Director General. Traffic dropped by approximately 80% during the disruption, which lasted from 19:30-23:00 UTC (15:30-19:00 UTC).

Pakistan & United Arab Emirates

Telegeography’s Submarine Cable Map shows that the Red Sea has a high density of submarine cables that carry data between Europe, Africa, and Asia. Cuts to these cables can significantly impact connectivity, ranging from increased latency on international connections to complete outages. The impacts may only affect a single country, or they may disrupt multiple countries connected to a damaged cable. On September 6, Pakistan Telecom (AS17557) posted a message on X that stated “We would like to inform that submarine cable cuts have occurred in Saudi waters near Jeddah, impacting partial bandwidth capacity on SMW4 and IMEWE systems. As a result, internet users in Pakistan may experience some service degradation during peak hours.” (Initial reporting that the cable cuts occurred near Jeddah were apparently incorrect, as the damage occurred in Yemeni waters.)

Looking at the impact in Pakistan, we observed traffic drop by 25-30% in Sindh and Punjab between 12:00-20:00 UTC (17:00 – 01:00 local time).

In the United Arab Emirates, Etisalat alerted customers via a post on X that they “may experience slowness in data services due to an interruption in the international submarine cables.” Between 11:00-22:00 UTC (15:00-02:00 local time) on September 6, traffic from AS8966 (Etisalat) dropped as much as 28%.

Also in the UAE, service provider du (AS15802) told their customers via a post on X that “You may experience some slowness in our data services due to an International submarine cable cut.” This slowness is visible in Radar’s Internet quality metrics for the network between 11:00-22:00 UTC (15:00-02:00 local time) on September 6, with median bandwidth dropping by more than half, from 25 Mbps to as low as 9.8 Mbps, and median latency doubling from 30 ms to over 60 ms.

The graphs below provide another view of the impact of the cable cuts, based on Cloudflare network probes between New Delhi (del-c) to London (lhr-a) and Bombay (bom-c) to Frankfurt (fra-a). For the former pair of data centers, mean latency grew by approximately 20%, and for the latter pair, by approximately 30%, starting around 23:00 UTC on September 5. (The stable latency line at the bottom of both graphs represents probes going over the Cloudflare backbone, which was not impacted by the cable cuts.)



Texas, United States

Fiber optic cables are frequently damaged by errant ship anchors (submarine) or construction equipment (terrestrial), but on September 26, a stray bullet damaged a cable in the Dallas, Texas area, disrupting Internet connectivity for Spectrum (AS11427) customers. Spectrum acknowledged the service interruption in a post on X, followed by another post four and a half hours later stating that the issue had been resolved. Although neither post cited the bullet as the cause of the disruption, news reports attributed the claim to a Spectrum spokesperson. Overall, the disruption was fairly nominal, lasting for just two hours between 18:00-20:00 UTC (13:00-15:00 local time), with traffic dropping less than 25% as compared to the prior week.

South Africa

“Major cable breaks” disrupted Internet connectivity for customers of Telkom (AS37457) in South Africa on September 27. Although Telkom acknowledged the initial service disruption and its subsequent resolution in posts on X, it didn’t provide any information about the cause in these posts. However, it apparently later issued a statement, stating “Telkom confirms that mobile voice and data services, which were disrupted earlier on Saturday due to major cable breaks, have now been fully restored nationwide.” The disruption lasted six hours, from 08:00-14:00 UTC (10:00-16:00 local time), with traffic dropping as much as 50% as compared to the previous week.

Power outages cause Internet disruptions

Tanzania

A reported power outage at one of Airtel Tanzania’s data centers on July 1 resulted in a multi-hour disruption in connectivity for its mobile customers. The service interruption occurred between 11:30-18:00 UTC (14:30-21:00 local time), with traffic dropping on Airtel Tanzania (AS37133) by as much as 40% as compared to the previous week.

Czech Republic

According to the Industry and Trade Ministry in the Czech Republic, a fallen power cable caused a widespread power outage on July 4. This power outage impacted Internet connectivity within the country, with traffic dropping by as much as 32%. Traffic fell just after the power outage began at 10:00 UTC (12:00 local time), and although it was “nearly fully resolved” by 16:00 UTC (18:00 local time), traffic did not return to expected levels until closer to 20:00 UTC (22:00 local time). This trailing traffic recovery aligns with a published report that noted “While ČEPS, the national transmission system operator, restored full grid functionality by mid-afternoon, tens of thousands remained without electricity into the evening.

St. Vincent and the Grenadines

On St. Vincent and the Grenadines, the St Vincent Electricity Services Limited (VINLEC) stated in a Facebook post that a “system failure” caused a power outage that affected customers on mainland St. Vincent. According to VINLEC, the system failed at approximately 11:30 local time on August 16 (03:30 UTC on August 17), and power was restored to all customers just after 04:00 local time on August 17 (08:00 UTC). During the four-hour power outage, which also disrupted Internet connectivity, traffic dropped by as much as 80% below expected levels.

Curaçao

In Curaçao, a series of Facebook posts from Aqualectra, the island’s water and power company, confirmed that there was a power outage, and provided updates on the progress towards restoration. The impact of the power outage to Internet connectivity was visible in traffic disruptions across several Internet service providers, including Flow (AS52233) and UTS (AS11081). The observed disruptions lasted for most of the day, with traffic dropping around 06:45 UTC (02:45 local time) and recovering to expected levels around 23:45 UTC (19:45 local time). During the disruption, the country’s traffic dropped by over 80% as compared to the previous week, with Flow experiencing a near complete outage.

Cuba

Wide-scale power outages occur all too frequently in Cuba, and when power is lost, Internet connectivity follows. We have covered many such events in this series of blog posts over the last several years, and the latest occurred on September 10. That morning, an X post from the Unión Eléctrica de Cuba reported the collapse of the national electric power system at 09:14 local time (13:14 UTC) following the unexpected shutdown of the Antonio Guiteras Thermoelectric Power Plant (CTE). The island’s Internet traffic dropped by nearly 60% (as compared to expected levels) almost immediately, and remained lower than normal for over a day, returning to expected levels around 17:15 UTC on September 11 (13:15 local time) when the Ministerio de Energía y Minas de Cuba posted on X that the national electric system had been restored.

Gibraltar

A contractor cutting through three high voltage cables caused a nationwide power outage in Gibraltar on September 16, according to a Facebook post from the Gibraltar government. This power outage resulted in a disruption to Internet traffic between 11:15-18:30 UTC (13:15-20:30 local time), falling as low as 80% below the previous week.

Earthquake

Kamchatka Peninsula, Russia

A magnitude 8.8 earthquake struck the Kamchatka Peninsula in Russia at 23:24 UTC on July 29 (11:24 local time on July 30), and was powerful enough to trigger tsunami warnings for Japan, Alaska, Hawaii, Guam, and other Russian regions. The graphs below show that there was an immediate impact to Internet traffic across several networks in the region, including Rostelecom (AS12389) and InterkamService (AS42742), where traffic dropped by 75% or more. While traffic started to recover almost immediately across both providers, traffic on Rostelecom approached expected levels much more quickly than on InterkamService.

Targeted cyberattack

Yemen

A cyberattack targeting Houthi-controlled YemenNet (AS30873) on August 11 briefly disrupted connectivity across the network in Yemen. A significant drop in traffic occurred at around 14:15 UTC (17:15 local time), recovering by 15:00 UTC (18:00 local time). This observed drop in traffic aligns with the reported timing and duration of the attack, which was focused on YemenNet’s ADSL infrastructure.

The attack also apparently impacted YemenNet’s routing, as announced IPv4 address space began to decline as the attack commenced. Although the attack ended within an hour after it started, announced address space remained depressed for approximately an additional hour, reaching as low as 510 /24s (blocks of 256 IPv4 addresses) being announced, down from a “steady state” of 870 /24s.

Fire causes infrastructure damage

Egypt

A fire at the Ramses Central Exchange in Cairo, Egypt on July 7 disrupted telecommunications services for a number of providers with infrastructure in the facility. The fire broke out in a Telecom Egypt equipment room, and impacted connectivity across multiple providers, including Etisalat (AS36992), Mobinil (AS37069), Orange Egypt (AS24863), and Vodafone Egypt (AS24835). Internet traffic across these providers initially dropped at 14:30 UTC (17:30 local time). Recovery to expected levels varied across the providers, with Etisalat recovering by July 9, Vodafone and Mobinil by July 10, and Orange Egypt on July 11.

On July 10, Telecom Egypt announced that services affected by the fire had been restored, after operations were transferred to alternative exchanges.

Technical problems

Starlink

Global satellite Internet service provider Starlink (AS14593) acknowledged a July 24 network outage through a post on X. The Vice President of Network Engineering at SpaceX explained, in a subsequent X post, that “The outage was due to failure of key internal software services that operate the core network.

Traffic initially dropped around 19:15 UTC, and the disruption lasted approximately 2.5 hours. The impact of the Starlink outage was particularly noticeable in countries including Yemen and Sudan, where traffic dropped by approximately 50%, as well as in Zimbabwe, South Sudan, and Chad.

China

At around 16:30 UTC on August 19 (00:30 local time on August 20), we observed an anomalous 25% drop in China’s Internet traffic. Our analysis of related metrics found that this disruption caused a drop in the share of IPv4 traffic, as well as a spike in the share of HTTP traffic (meaning that HTTPS traffic share had fallen), as shown in the graphs below.

Further analysis also found the share of TCP connections terminated in the Post SYN stage doubled during the observed outage, from 39% to 78%, as shown below. The cause of these unusual observations was ultimately uncovered by a Great Firewall Report blog post, which stated, in part: “Between approximately 00:34 and 01:48 (Beijing Time, UTC+8) on August 20, 2025, the Great Firewall of China (GFW) exhibited anomalous behavior by unconditionally injecting forged TCP RST+ACK packets to disrupt all connections on TCP port 443. This incident caused massive disruption of the Internet connections between China and the rest of the world. … The responsible device does not match the fingerprints of any known GFW devices, suggesting that the incident was caused by either a new GFW device or a known device operating in a novel or misconfigured state.” This explanation is consistent with the anomalies visible in the Radar graphs.

Pakistan

Subscribers of Nayatel (AS23674) experienced an approximately 90 minute disruption to Internet connectivity on September 24, due to a reported outage at an upstream provider. Traffic dropped as much as 57% between around 09:15-10:45 UTC (14:15-15:45 local). Transworld (AS38193) is one of several upstream providers to Nayatel, and a more significant drop in traffic is visible for that network, lasting from around 09:15-12:15 UTC (14:15-17:15 local time). The Nayatel disruption was likely less significant than the one seen at Transworld because Transworld is upstream of only a portion of the prefixes originated by Nayatel — traffic from other Nayatel prefixes was carried by other providers that remained available.

No definitive cause

Iran

Several weeks after experiencing a full Internet shutdown, Iran again experienced a sudden drop in Internet traffic around 21:00 UTC on July 5 (00:30 local time on July 6), with traffic falling 80% as compared to the prior week. While most of the “unknown” disruptions covered in this series of posts are observed but have no associated acknowledgement or explanation, this disruption had multiple competing explanations.

A published report noted “IRNA, Iran’s official news agency, cited the state-run Telecommunications Infrastructure Company, reporting a national-level disruption in international connectivity that affected most internet service providers Saturday night. Yet government officials have not publicly addressed the cause.” However, posts from civil society groups that follow Internet connectivity in Iran (net4people, FilterWatch) suggested that the disruption was again due to an intentional shutdown. And a post thread on X referenced, and disputed, a claim that the disruption was due to a DDoS attack. Unfortunately, no definitive root cause for this disruption could be found.

Colombia

Customers of Claro Colombia experienced an Internet disruption that lasted just over 30 minutes on August 6, with traffic falling two-thirds or more as compared to the prior week between 16:45 – 17:20 UTC. The disruption affected multiple ASNs owned by Claro, including AS10620, AS14080, and AS26611. (The Telmex Colombia and Comcel names shown in the graphs below are historical – Telmex and Comcel merged in 2012 and have operated under the Claro brand since then.) Claro did not acknowledge the disruption on social media, nor did it provide any explanation for it.

Pakistan

A near-complete outage at Pakistani backbone provider PTCL (AS17557) caused traffic from the network provider to drop 90% at 16:10 UTC (21:10 local time) on August 19. PTCL acknowledged the issue in a post on X, noting “We are currently facing data connectivity challenges on our PTCL and Ufone services.” Although they published a subsequent post several hours later after service was restored, they did not provide any additional information about the cause of the outage. However, one published report claimed “The disruption was primarily caused by a technical fault in PTCL’s fiber optic infrastructure.” while another report claimed “According to industry sources, the internet disruption in Pakistan may be connected to a technical fault in the fiber optic backbone or issues with main internet providers responsible for international online traffic.

Interestingly, traffic from PTCL to Cloudflare’s 1.1.1.1 DNS resolver spiked as the outage began, and the share of requests made over UDP grew from 94% to 99%. In addition, routing data shows that there was also a small drop in announced IPv4 address space coincident with the outage. However, these additional observations do not necessarily confirm a “technical fault in PTCL’s fiber optic infrastructure” as the ultimate cause of the disruption.

South Africa

To their credit, South African provider RSAWEB (AS37053) quickly acknowledged an issue with their FTTx and Enterprise connectivity on September 10, but neither their initial post nor subsequent updates provided any information on the cause of the problem. Whatever the cause, it resulted in a near-complete loss of Internet traffic from RSAWEB between 15:00 and 16:30 UTC (17:00 – 18:30 local time).

Routing data also shows a loss of just two announced /24 address blocks concurrent with the outage, dropping from 470 to 468. Unless all of RSAWEB’s outbound traffic was flowing through this limited amount of IP address space, it seems unusual that the withdrawal of just 512 IPv4 addresses from the=e routing table would have such a significant impact on the network’s traffic.

SpaceX Starlink

After experiencing a brief disruption in July due to a software failure, Starlink (AS14593) suffered another short disruption between 04:00-05:00 UTC on September 15. Although Starlink generally acknowledges disruptions to their global network on their X account, and often providing a root cause, in this case they apparently published an acknowledgement on X, but deleted it after the issue was resolved. In addition to the drop in traffic, we observed a concurrent drop in announced IPv4 address space and spike in BGP announcements (likely withdrawals), suggesting that the disruption may have been caused by a network-related issue.

Conclusion

The recent launch of regional traffic insights on Radar brings yet another perspective to our ability to investigate observed Internet traffic anomalies. We can now drill down at regional and network levels, as well as exploring the impact across DNS traffic, connection bandwidth and latency, TCP connection tampering, and announced IP address space, helping us understand the impact of such events. And while these blog posts feature graphs from Radar and the Radar Data Explorer, the underlying data is available from our rich API. You can use the API to retrieve data to do your own local monitoring or analysis, or the Radar MCP server to incorporate Radar data into your AI tools.

The Cloudflare Radar team is constantly monitoring for Internet disruptions, sharing our observations on the Cloudflare Radar Outage Center, via social media, and in posts on blog.cloudflare.com. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

How a volunteer-run wildfire site in Portugal stayed online during DDoS attacks

Post Syndicated from João Tomé original https://blog.cloudflare.com/wildfire-fogos-pt-portugal-ddos-attack/

On July 31, 2025, just as Portugal entered the peak of another intense wildfire season, João Pina, also known as Tomahock, received an automated alert from Cloudflare. His volunteer-run project, fogos.pt, now a trusted source of real-time wildfire information for millions across Portugal, was under attack.


One of the several alerts fogos.pt received related to the DDoS attack

What started in 2015 as a late-night side project with friends around a dinner table in Aveiro has grown into a critical public resource. During wildfires, the site is where firefighters, journalists, citizens, and even government agencies go to understand what’s happening on the ground. Over the years, fogos.pt has evolved from parsing PDFs into visual maps to a full-featured app and website with historical data, weather overlays, and more. It’s also part of Project Galileo, Cloudflare’s initiative to protect vulnerable but important public interest sites at no cost.

Wildfires are not just a Portuguese challenge. They are frequent across southern Europe (Spain, Greece, currently also under alert), California, Australia, and in Canada, which in 2023 faced record-setting fires. In all these cases, reliable information can be crucial, sometimes life-saving. Other organizations offering similar public services can also apply to join Project Galileo to receive protection and handle heavy traffic.

A side project that became a national reference

Fogos.pt began with a simple question: why was fire data only available in hard-to-read PDF documents? João and a group of friends, including volunteer firefighters, decided to build something better. They pulled the data, geolocated the fire reports, and visualized them on a map.

Soon, thousands of people were using it. Then tens of thousands. Today, fogos.pt is integrated into official communications, including mentions from the Portuguese government on social media and direct links from the national wildfire information portal (SGIFR.gov.pt).

In 2018, fogos.pt formally joined forces with VOST Portugal, a digital volunteer organization that was early on also part of our Project Galileo — whose story was also featured in an earlier case study. João Pina is also a co-founder of VOSTPT. Together, they created a complementary model: fogos.pt provides data and the platform; VOSTPT validates and communicates it to the public in real-time during emergencies.

It’s an operation run entirely by volunteers, with no funding, no formal team — just passion, and the help of partners.


Homepage of fogos.pt on August 20, 2025, highlighting a major wildfire near Piódão in central Portugal.

Under attack during fire season

On July 31 and August 1, 2025, two Distributed Denial of Service (DDoS) attacks targeted fogos.pt. Cloudflare automatically detected and mitigated both attacks.

July 31 attack:
• Duration: 7 minutes
• Peak: 33,000 requests per second at 11:27 UTC
• Bandwidth: 1.7 Gbps (Max)

How the attack looks like in requests per second:


August 1 attack:
• Duration: 5 minutes
• Peak: 31,000 requests per second at 10:24 UTC
• Bandwidth: 849 Mbps (Max)

How the attack looks like in requests per second from our perspective:


By Cloudflare’s standards, these were small. For comparison, last year we mitigated an attack exceeding 700,000 requests per second against a high-profile US election campaign site. But for an civic project like fogos.pt, even tens of thousands of requests per second — if unprotected — can be enough to take services offline at the worst possible time.

Attackers typically use three main methods for DDoS attacks:

  • IoT devices: hacked cameras, routers, or smart gadgets sending traffic.

  • Proxies: open or misconfigured servers, residential proxy networks, or anonymity tools that hide attackers’ IPs.

  • Cloud machines: compromised or rented servers from cloud providers.

The July 31 attack likely relied on open proxies, with much of the traffic arriving unencrypted (a common sign of proxy-based attacks). The August 1 attack, in contrast, came largely from cloud machines, matching patterns we see from botnets that exploit cloud infrastructure.

These attacks were blocked without disruption. Cloudflare’s autonomous mitigation systems kicked in, and email alerts were automatically sent to João and the team. No downtime, no manual intervention required.

The role of Project Galileo: traffic surges

Fogos.pt has used Cloudflare’s free services since the beginning, starting with DNS and gradually expanding to DDoS mitigation, caching, rate limiting, and more. The site joined Project Galileo, which protects journalists, human rights defenders, and public service projects, to get stronger, upgraded features and service at no cost.

“Without Cloudflare, the site would have gone down many times during fire season,” says João Pina. “We use almost every product — but protection against attacks is critical.”


August 11, 2025, detail the area of interest of a wildfire in central Portugal. 

Traffic to fogos.pt surges when wildfires hit the news or get mentioned by authorities. These spikes can bring tens of thousands of visitors per day. And as attention grows, so does the risk. Attacks can be used to silence or disrupt critical services, or simply as distractions for more malicious activity. In August 2025, the site often had close to 60,000 people browsing at the same time, with around 40,000 being the norm across the web and app services.


In just two weeks (with an August 15 peak of almost 70 million requests), fogos.pt handled over 550 million requests (more than 25 million per day) 9 TB of data transfer, nearly 100 million page views, 15 million visits, and 240 million API calls. A massive load for a volunteer-run project, as the next screenshot from the fogos.pt team shows:


In a time when timely wildfire updates can mean the difference between safety and danger, keeping the site online is essential. 

Built by community, supported by allies

Fogos.pt is a reminder of what’s possible when public service meets technology, and why we launched Project Galileo: to protect the digital infrastructure that keeps people informed and safe. Built with no formal funding or full-time team, it runs on volunteers, partners, and a shared sense of purpose, an authenticity that João Pina believes is why it works, and why it matters.

And while this story is about Portugal, wildfires are a global challenge. Other organizations providing critical public services can also apply to join Project Galileo and receive this protection.

From a dinner-table idea by an engineer to critical national infrastructure, fogos.pt shows the Internet at its best. Cloudflare is proud to help protect it.

Shutdown season: the Q2 2025 Internet disruption summary

Post Syndicated from David Belson original https://blog.cloudflare.com/q2-2025-internet-disruption-summary/

Cloudflare’s network currently spans more than 330 cities in over 125 countries, and we interconnect with over 13,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions at both a local and national level, as well as at a network level.

As we have noted in the past, this post is intended as a summary overview of observed and confirmed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter. A larger list of detected traffic anomalies is available in the Cloudflare Radar Outage Center. Note that both bytes-based and request-based traffic graphs are used within the post to illustrate the impact of the observed disruptions — the choice of metric was generally made based on which better illustrated the impact of the disruption.

In our Q1 2025 summary post, we noted that we had not observed any government-directed Internet shutdowns during the quarter. Unfortunately, that forward progress was short-lived — in the second quarter of 2025, we observed shutdowns in Libya, Iran, Iraq, Syria, and Panama. The Internet’s reliance on a stable electric grid was made abundantly clear during the quarter, with a massive power outage impacting Spain and Portugal disrupting connectivity within those countries. Fiber optic cable cuts impacted providers in Haiti and Malawi, major North American providers saw technical problems disrupt Internet traffic, and a Russian provider was once again targeted by a significant cyberattack, knocking the network offline. Unfortunately, official attribution of an Internet outage’s root cause isn’t always available — and we observed several significant, yet unexplained, Internet outages during the quarter.

Government-directed shutdowns

Libya

On May 16, Internet disruptions were observed across multiple Libyan network providers, with connectivity reportedly shut down in response to public protests against the Government of National Unity. Starting at 13:30 UTC (15:30 local time), traffic dropped by more than 50% as compared to the prior week at Libyan International Company for Technology (AS329129), Giga Communication (AS328539), Aljeel Aljadeed for Technology (AS37284), and Awal Telecom (AS328733), with the latter experiencing a complete outage. Lower traffic volumes were observed until around 00:00 UTC (02:00 local time), with traffic restoration occurring within an hour or so on either side. Giga Communication (AS328539) experienced a second disruption on May 17 between 02:00 – 11:30 UTC (04:00 – 13:30 local time).

Iran

Multiple Internet shutdowns occurred in Iran in June following Israel’s initial attacks on the country’s nuclear sites. The first, on June 13, occurred between 07:15 – 09:45 UTC (10:45 – 13:15 local time). Iran’s Ministry of Communications issued a statement that announced the shutdown: “In light of the country’s special circumstances and based on the measures taken by the competent authorities, temporary restrictions have been imposed on the country’s Internet. It is obvious that these restrictions will be lifted once normal conditions are restored.” This shutdown order impacted network providers including FanapTelecom (AS24631), Rasana (AS205647 and AS31549), MCCI (AS197207), and TCI (AS58224), as well as others.

On June 17, Internet connectivity was again restricted, this time reportedly in an effort to “ward off cyber attacks”, according to a government spokesperson. This second round of shutdowns began at 17:30 local time (14:00 UTC), impacting multiple networks. Traffic recovered at 15:30 UTC (19:00 local time) on FanapTelecom (AS24631) and Pars Online (AS16322), at 20:00 UTC (23:30 local time) on MCCI (AS197207) and IranCell (AS44244), at 22:00 UTC on June 17 (01:30 on June 18 local time) on RighTel (AS57218), and at 06:00 UTC on June 18 (09:30 local time) on Rasana (AS31549 and AS205647).

During these initial Internet shutdowns, incoming Internet traffic was reportedly also blocked, and user access was limited to Iran’s domestic “National Information Network” (NIN).

Just a day later, on June 18, an extended third shutdown was put into place, this one lasting from 12:50 UTC (16:20 local time) through 05:00 UTC (08:30 local time) on June 25. Once again, the shutdown was reportedly implemented as a means of protecting against cyberattacks, with a government spokesperson commenting “We have previously stated that if necessary, we will certainly switch to a national internet and restrict global internet access. Security is our main concern, and we are witnessing cyberattacks on the country’s critical infrastructure and disruptions in the functioning of banks. Many of the enemy’s drones are managed and controlled via the internet, and a large amount of information is exchanged this way. A cryptocurrency exchange was also hacked, and considering all these issues, we have decided to impose Internet restrictions.” This shutdown resulted in a near-complete loss of traffic through 02:00 UTC (05:30 local time) on June 21, when some traffic recovery was observed, though at levels remaining well-below pre-shutdown volumes. Traffic from this partial recovery settled into a consistent cycle for several days, until returning to expected levels on June 25. The same network providers impacted by the previous shutdowns were affected by this one as well.

Iraq

Consistent with measures taken over the past several years (2024, 2023, 2022), governments in Iraq again implemented regular Internet shutdowns in an effort to prevent cheating on national exams. (We say “governments” here because the shutdowns took place both in the main part of the country and in the Iraqi Kurdistan region in the northern part of the country.)

Occurring between 03:00 – 05:00 UTC (06:00 – 08:00 local time) at the request of the Ministry of Education, the shutdowns in the main part of the country started on May 20 and ran through June 4 for middle school exams, and from June 14 until July 3 for preparatory school exams. Network providers that implemented the shutdowns included Earthlink (AS199739), Asiacell (AS51684), Zainas (AS59588), Halasat (AS58322), and HulumTele (AS203214).

In the Kurdistan region, the shutdowns began June 1, and ran through July 6, taking place between 03:30 – 04:30 UTC (06:30 – 07:30 local time) on Wednesdays and Sundays. Network providers that implemented the shutdowns included IQ Online (AS48492), KorekTel (AS59625), Newroz Telecom (AS21277), and KNET (AS206206).

Syria

As Iraq does, Syria also implements nationwide Internet shutdowns to prevent cheating on exams, and has been doing so for several years (2021, 2022, 2023, 2024). However, in contrast to previous years, in 2025, the government only ordered the cutoff of cellular connectivity, with a published statement noting (translated) “As part of our commitment to ensuring the integrity of public examinations and safeguarding the future of our dear students, and based on our national responsibility to secure a fair and transparent examination environment, a temporary cellular communications blackout will be implemented in areas near examination centers across the Syrian Arab Republic. … The cellular communications blackout will be implemented exclusively within the narrowest possible geographical and timeframe, during the time students are in exam halls.

During the second quarter, the shutdowns associated with the “Basic Education Certificate” took place on June 21, 24, and 29 between 05:15 – 06:00 UTC (08:15 – 09:00 local time). Exams and associated shutdowns for the “Secondary Education Certificate” are scheduled to take place between July 12 and August 3.

Because these shutdowns only impacted mobile connectivity, they only resulted in a partial drop in announced IP address space, as opposed to a more complete loss as seen in previous years.

Panama

On June 21, an X post from ASEP Panamá (the telecommunications regulating agency) announced that (translated) “…in compliance with Cabinet Decree No. 27 of June 20, 2025, and by formal instruction from the Ministry of Government, the temporary suspension of mobile telephony and residential internet services in the province of Bocas del Toro has been coordinated.” The suspension, according to the post, was supposed to be in place until June 25, however a subsequent X post noted that it would be extended until Sunday, June 29, 2025.

The suspension of Internet connectivity was implemented in response to protests and demonstrations against reforms to the Social Security Fund, retirement, and pensions, specifically in the province of Bocas del Toro.

The graph below shows an effective loss of traffic from Cable Onda (AS18809) in Bocas Del Toro, Panama around 03:30 UTC on June 21 (22:30 local time on June 20), recovering around 06:00 UTC (01:00 local time) on June 30. The recovery is in line with the final related X post from ASEP, which noted (translated) “… Internet and cellular telephone services in the province of Bocas del Toro have been restored as of 12:01 a.m. on Monday, June 30…”.

Power outages lead to Internet outages

Portugal & Spain

The big power outage story during the second quarter was the massive outage across much of Portugal and Spain on April 28. The impact of the event was covered in detail in the How the April 28, 2025, power outage in Portugal and Spain impacted Internet traffic and connectivity blog post, which explored shifts in traffic at a country/network/regional level, as well as how the power outage impacted network quality and announced IP address space.

In Portugal, Internet traffic dropped as the power grid failed — when compared with the previous week, traffic fell ~50 % immediately and within five hours it was ~90% below the week before.


In Spain, Internet traffic dropped as the power grid failed, with traffic immediately dropping by around 60% as compared to the previous week, falling to approximately 80% below the previous week within the next five hours.


In both countries, traffic returned to expected levels around 01:00 local time (midnight UTC) on April 29. More details about the outage can be found in the blog post linked above.

Morocco

It appears that Morocco may have also been impacted in some fashion by the Portugal/Spain power outage, or at least Orange Maroc was. In a post on X, the provider stated (translated) “Internet traffic has been disrupted following a massive power outage in Spain and Portugal, which is affecting international connections.Traffic from the network (AS36925) fell sharply around 12:00 UTC (13:00 local time), 90 minutes after the power outage began, with a full outage beginning around 15:00 UTC (16:00 local time). Traffic returned to expected levels around 23:30 UTC on April 28 (00:30 local time on April 29).

Puerto Rico

Genera PR, a power company in Puerto Rico, posted on X on April 16 that they had (translated) “…experienced a massive power outage across the island due to the unexpected shutdown of all generating plants, including those of Genera PR and other private generators. This situation has caused a significant disruption to electrical service…Luma Energy, the private power company that is responsible for power distribution and power transmission in Puerto Rico, published their own X post that stated (translated) “Approximately at 12:40pm, an event was recorded that affects the service island-wide.

Although the reported power outage was “massive” and “island-wide”, it did not have an outsized impact on Puerto Rico’s Internet traffic, which initially dropped by about 40%. Over the next several days, both companies published multiple updates to their X accounts detailing the progress being made in restoring service. By 15:00 UTC (11:00 local time) on April 18, traffic had returned to expected levels, in line with a post from Luma Energy that noted (translated) “As of 10:00 a.m. on April 18, and thanks to LUMA’s extraordinary response and the tireless efforts of the island’s workforce—in coordination with the Puerto Rico government and generating companies—LUMA has restored electric service to 1,450,367 customers, representing 98.8% of total customers, in less than 38 hours since the island-wide outage began.

As seen in the graphs below, the power outage not only impacted end-user connectivity, driving the observed drop in traffic, but also had some impact on local Internet infrastructure, with some disturbance visible to announced IP address space.

Saint Kitts and Nevis

A Facebook post from SKELEC (The St. Kitts Electricity Company) on May 9 alerted customers on St. Kitts and Nevis that “…a fault developed at our Needsmust Power Plant resulting in an island wide outage. Restoration has begun, and complete restoration will be in two hours.” The post was published at 17:31 UTC (13:31 local time), approximately 30 minutes after the island’s Internet traffic initially dropped. Traffic recovery initially began around 17:45 UTC (13:45 local time), well within the two-hour estimate for complete power restoration. However, Internet traffic did not fully return to expected levels until 20:15 UTC (16:15 local time).

North Macedonia

On May 18, it was reported that “High voltages in the regional 400 kV network amid low consumption caused a short-term outage in North Macedonia‘s 110 kV transmission network…”, according to state-owned power company MEPSO. While the outage reportedly impacted most of the country, MEPSO also noted that the country’s power supply was normalized within an hour after the outage began. Although brief, the power outage caused the country’s Internet traffic to drop by nearly 60% as compared to the previous week during the disruption, which occurred between 03:00 – 04:45 UTC (05:00 – 06:45 local time).

Maldives

On June 1, Internet traffic in the Maldives dropped by nearly half as compared to the previous week when a widespread power outage affected the Greater Malé region. Local Internet service providers including Ooredoo and Dhiraagu took to social media to warn subscribers of potential interruptions to both fixed and mobile broadband connections. At a country level, Internet traffic was disrupted between 07:30 – 13:00 UTC (12:30 – 18:00 local time).

The power outage also had a nominal impact on Internet infrastructure, as announced IPv4 address space saw a nominal drop (from 355 to 350 /24s) that began shortly after the initial drop in traffic was observed, but returned to normal as the disruption ended.

Curaçao

A near-complete Internet outage at provider Flow Curaçao (AS52233) on June 14-15 sparked outrage and demands for answers by the country’s telecommunications regulator. Flow’s Internet traffic dropped significantly at 18:00 UTC (14:00 local time) on June 14, falling further in the following hours. Signs of recovery became visible around 11:00 UTC (07:00 local time) on June 15, with more complete recovery occurring at 14:00 UTC (10:00 local time). A Facebook post from Flow Barbados, posted on June 18, referenced a local disruption that began on June 14, but pointed at a commercial power outage at one of their key regional network facilities in Curaçao, which was likely the driver of this Internet outage.

Fiber optic cable damage

Digicel Haiti

Two instances of damage to its fiber optic infrastructure caused a complete Internet outage at Digicel Haiti (AS27653) as of 21:00 UTC (17:00 local time) on May 28, according to a (translated) X post from the company’s Director General. The cable damage took the network completely off the Internet, as announced IPv4 and IPv6 address space also dropped to zero. Digicel Haiti remained offline until 00:45 on May 29 (20:45 local time on May 28), when both traffic and announced IP address space returned to expected levels.

Airtel Malawi

Airtel Malawi (AS37440) experienced a 90-minute Internet outage on June 24, caused by ongoing vandalism on their fiber network. Although traffic effectively disappeared between 12:30 – 14:00 UTC (14:30 – 16:00 local time), the network remained at least partially online as at least some of the network’s IPv4 address space continued to be announced to the Internet.  Announced IPv6 address space, however, fell to zero during the duration of the outage.

Technical problems

Bell Canada

A router update gone awry disrupted Internet service for Bell Canada (AS577) customers in Ontario and Quebec on May 21. An initial X post from the provider, posted at 13:52 UTC (09:52 local time), alerted customers to the service interruption. The post trailed the start of the disruption by approximately a half hour, as traffic dropped around 13:15 UTC (09:15 local time), falling by as much as 70% as compared to the same time a week prior. Request traffic to Cloudflare’s 1.1.1.1 DNS Resolver also saw a significant drop. A negligible decline in announced IPv4 address space was also observed.

The disruption was fairly short-lived, with traffic returning to expected levels just an hour later. A subsequent X post confirmed that services had been fully restored by 15:00 UTC (11:00 local time), with another post noting that the initial update had been rolled back quickly to restore service. 

Lumen/CenturyLink 

Across parts of the United States, Lumen/CenturyLink (AS209) customers experienced a widespread Internet service disruption on June 19. Traffic volumes dropped by over 50% as compared to the prior week starting around 21:45 UTC. The disruption only lasted a couple of hours, with traffic returning to normal by 00:00 UTC on June 20.

Social media posts from affected subscribers suggested that the problem might have been DNS related, as those that switched their DNS resolver to Cloudflare’s 1.1.1.1 were once again able to access the Internet. The graph below shows that traffic to 1.1.1.1 from Lumen/CenturyLink exceeded levels seen the previous week as the disruption began, and remained elevated through June 20. Problems with an Internet service provider’s DNS resolver can appear to subscribers like an Internet outage, as they become unable to access anything requiring a DNS lookup (effectively, all Internet resources), ultimately resulting in a drop in traffic to those resources (from the affected user base), as seen in the graph above.

Cyberattack impact

ASVT (Russia)

Russian Internet provider ASVT (AS8752) was reportedly targeted by a major DDoS attack that resulted in a multi-day complete Internet outage. This attack followed one targeting Russian provider Nodex (AS29329) in March, which also caused a complete service outage. Reaching 70.07 Gbps/6.92 million packets/second, the attack caused traffic to drop to near zero around 05:00 UTC on May 28 (08:00 Moscow time), with the effective outage lasting for approximately 10 hours. Although traffic began to return around 15:00 UTC (18:00 Moscow time), it remained below expected levels throughout the following week.

Interestingly, query volume to Cloudflare’s 1.1.1.1 DNS Resolver from ASVT saw a rapid increase as traffic began to return after the initial outage, and remained elevated throughout the duration of the disruption. It isn’t clear whether the increase could be related to problems with ASVT’s native DNS resolver during the attack, forcing users to seek alternative resolvers, or if it could be related to ASVT subscribers seeking ways around damage from the attack.

Unexplained disruptions

Telia Finland (April 1)

According to a (now unavailable) “Disturbance bulletin” and an associated X post from Telia Finland (AS1759), the company acknowledged that “A widespread disruption has been detected in the operation of mobile network data connections and fixed broadband.” The widespread disruption resulted in a brief near-complete outage for subscribers between 06:30 – 07:15 UTC (09:30 – 10:15 local time).

Telia Finland did not disclose the cause of the disruption, but it is clear that it impacted IPv4 connectivity, as seen in the graph below showing announced IPv4 address space. (Announced IPv6 address space did not see any change.) This loss of IPv4 connectivity resulted in a concurrent spike in the share of traffic from Telia Finland over IPv6 — normally below 5%, it spiked above 30% during the disruption. Request traffic to Cloudflare’s 1.1.1.1 resolver from Telia Finland also spiked at that time.

SkyCable

Around 19:15 UTC on May 7 (03:15 local time on May 8), subscribers of SkyCable (AS23944) in the Philippines experienced a complete Internet outage. Internet traffic from the network dropped to zero, as did announced IPv4 address space. The disruption lasted until 03:00 UTC on May 8 (11:00 local time), and SkyCable did not publish any information regarding the cause of the eight-hour service outage.

TrueMove H

On May 22, Thai mobile provider TrueMove H (AS132061) suffered a nationwide outage, impacting connectivity for subscribers. The provider acknowledged and apologized for the disruption, but did not provide an official reason for the outage. (An article in the local press reported “that the outage was caused by technical errors on True’s computer servers” and also stated that others suggested that “the problem might have been caused by an error on True’s DNS servers”.)

At 03:00 UTC (10:00 local time), traffic initially dropped by over 80% as compared to the prior week. Almost immediately, traffic began to slowly recover, and returned to expected levels around 08:00 UTC (15:00 local time). A brief partial drop in announced IPv4 address space was also observed during the first hour of the disruption.

Digicel Haiti

Two days after experiencing an outage due to cable damage, Digicel Haiti (AS27653) experienced another complete outage on May 30. In contrast to the previous outage, no additional information about this one was published on social media by Digicel Haiti or its Director General. The network effectively disappeared from the Internet at 14:15 UTC (10:15 local time), with both traffic and announced IP address space (IPv4 & IPv6) dropping to zero. The outage lasted nearly three hours, with traffic and announced IP space all returning around 17:00 UTC (13:00 local time).

Syria

On June 10, an Internet outage in Syria reportedly affected the ADSL landline network across multiple provinces. Traffic dropped by as much as two-thirds below the same time the previous week at 08:15 UTC (11:15 local time), with the disruption lasting two hours. Announced IPv4 address space also fell during the course of the outage, indicating a potential infrastructure issue. However, as seen below, request volume from Syria to Cloudflare’s 1.1.1.1 DNS resolver was also elevated during the outage. This behavior has been observed in the past during government-directed shutdowns of Internet connectivity in Syria, when traffic can leave the country, but not return. There was no other indication that this outage was due to an intentional shutdown, but no official explanation for the disruption was available.

Conclusion

Government-directed Internet shutdowns returned with a vengeance in the second quarter, and that trend continues into the third quarter, though the latest ones have been exam-related, and not driven by protests. And while power-outage related Internet disruptions have frequently been observed in the past, often in smaller countries with less stable infrastructure, the massive outage in Spain and Portugal on April 28 reminds us that much like the Internet, electrical infrastructure is often interconnected across countries, meaning that problems in one can potentially cause significant problems in others.

The Cloudflare Radar team is constantly monitoring for Internet disruptions, sharing our observations on the Cloudflare Radar Outage Center, via social media, and in posts on blog.cloudflare.com. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

New year, no shutdowns: the Q1 2025 Internet disruption summary

Post Syndicated from David Belson original https://blog.cloudflare.com/q1-2025-internet-disruption-summary/

Cloudflare’s network spans more than 330 cities in over 125 countries, where we interconnect with over 13,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions at both a local and national level, as well as at a network level.

As we have noted in the past, this post is intended as a summary overview of observed and confirmed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter. A larger list of detected traffic anomalies is available in the Cloudflare Radar Outage Center. Note that both bytes-based and request-based traffic graphs are used within the post to illustrate the impact of the observed disruptions — the choice of metric was generally made based on which better illustrated the impact of the disruption.

In the first quarter of 2025, we observed a significant number of Internet disruptions due to cable damage and power outages. Severe storms caused outages in Ireland and Réunion, and an earthquake caused ongoing connectivity issues in Myanmar. Russian networks were taken offline by a reported cyberattack and purported technical problems, while a fire took a telecom provider in Haiti offline briefly. In Q4 2024, we observed only a single government-directed Internet shutdown, and this quarter, no such shutdowns were observed. Unfortunately, this is an unusual occurrence, and in the three-year history of this blog post series, has only occurred previously in Q4 2023 and Q1 2022

Submarine and terrestrial cable damage

Pakistan

Just after the new year, Internet connectivity in Pakistan was disrupted by a fault in the AAE-1 submarine cable. According to a January 2 alert published on social media by the Pakistan Telecommunications Authority, the cable fault occurred near Qatar, and would likely impact user experience across the country. Because there are seven submarine cables carrying international Internet traffic to/from Pakistan, the loss of AAE-1 did not cause an observable outage. However, the impact of the disruption was visible in the bandwidth and latency graphs for Pakistan. On January 2 and 3, median latency peaked at around 125 ms, up from a pre-disruption median of approximately 80 ms. Concurrent drops in bandwidth were observed, with media download speeds dropping to around 6 Mbps from a pre-disruption media of around 9 Mbps. In an “Important Update” posted to their Instagram account, Pakistan Telecom (PTCL, AS17557) also highlighted the potential for “slow browsing” — the Internet Quality graphs for that network show similarly-timed shifts in median bandwidth and latency.

Pakistan is currently connected to seven submarine cables, with two additional connections on the way in 2026. This connection diversity means that damage to or an issue with one cable will likely have minimal impact on Internet availability within the country, as traffic can be re-routed across other paths.

Syria

According to an announcement from the Syrian Ministry of Communications, a widespread Internet outage spanning January 23-24 was caused by sabotage that damaged two fiber optic cables that run along the highway between Damascus and Homs. The graphs below show that both HTTP and DNS request traffic from Syria dropped to near zero between 00:30 and 03:30 local time on January 24 (21:30 on January 23 – 00:30 on January 24 UTC). Traffic began recovering shortly thereafter, and returned to expected levels by 09:00 local time (06:00 UTC). Announced IPv4 address space for the country, almost exclusively from Syria Telecom (AS29256), also saw an approximately 90% drop during this period, which suggests that these fiber cuts caused a significant amount of Syria Telecom’s network to become unreachable during the incident.

Echoing the disruption above, Syria experienced another Internet outage on March 25, again caused by sabotage that damaged fiber optic cables. According to an announcement from the Syrian Ministry of Communications, the damage occurred in the Maaloula and Hasiya regions, resulting in a near complete outage between 03:00 – 13:15 local time (00:00 – 10:15 UTC). Similar to the January outage, the graphs below show a near complete loss of HTTP request traffic and a significant loss of announced IPv4 address space.

Somewhat paradoxically, DNS request volume from Syria was elevated during this outage, in contrast to the behavior observed during the January event. It isn’t clear what drove the additional traffic to Cloudflare’s 1.1.1.1 DNS resolver in this case.

Nepal

In early February, several Internet providers in Nepal saw services disrupted when Indian provider Bharti Airtel (AS9498) went offline. AS23752 (Nepal Telecom), AS17501 (Worldlink Communications), AS139922 (Dishhome Fibernet), AS45650 (Vianet Communications), and AS38565 (Ncell), who all have Airtel as an upstream provider or peer, saw traffic disrupted between 21:00 – 22:30 local time (15:15 – 16:45 UTC) on February 2.

Published reports disagree on the underlying cause of the Airtel issue, with one source claiming that it was related to an ongoing payment dispute, while another claims that it was due to reported fiber cuts in Airtel’s network.

Show less

Widespread power outages

Angola

Eleven provinces in Angola lost electrical power on January 6 due to an interruption in the North and Center Interconnected System, according to the National Electricity Transmission Network (RNT). The widespread power outage disrupted Internet connectivity across the country, leading to a drop in traffic between 14:45 – 22:00 local time (13:45 – 21:00 UTC). Published reports said that RNT was investigating the cause of the power outage, but no subsequent information was available confirming a specific cause.

Sri Lanka

Monkey business at the Pandura electrical substation caused an island-wide power outage in Sri Lanka on February 9. More seriously, a monkey coming into contact with a grid transformer caused the power outage, which resulted in a multi-hour disruption to Internet traffic from the country. Traffic initially dropped around 11:30 local time (06:00 UTC), and recovered by around 21:00 local time (15:30 UTC). The graph below for AS18001 (Dialog), a major Sri Lankan network services provider, illustrates the impact on traffic.

Chile

On February 25, a massive power outage in Chile reportedly impacted 98.5% of the country. A published report noted that there was an interruption in the power supply from Arica to the Los Lagos region, caused by a disconnection of the 500 kV transmission system in the Norte Chico. The power outage resulted in an immediate and significant drop in Internet traffic, as seen at a country level, as well at a network level, as shown in the graphs below. Traffic initially fell at around 14:15 local time (18:15 UTC) and recovered to expected levels approximately 12 hours later, around 02:00 local time (06:00 UTC). It was reported that as of an hour after traffic had recovered, approximately 94% of customers had power restored.

Honduras

A ground fault at the 15 de Septiembre electrical substation in El Salvador was reportedly the cause of a power outage that resulted in a multi-hour Internet disruption in Honduras on March 1. The Regional Operator Entity (OER) stated that the failure occurred at 09:22 local time (15:22 UTC), which resulted in traffic from the country dropping by about half. The disruption to Internet connectivity was relatively short-lived, as traffic returned to expected levels approximately two hours later.

Cuba

According to an X post from @EnergiasMinasCub (the Cuban state agency responsible for promoting the sustainable development of the country’s energy, geological, and mining sectors), at around 20:15 local time on March 14 (00:15 UTC on March 15) “a failure at the Diezmero substation caused a significant loss of generation in the west of #Cuba and with it the failure of the National Electric System, SEN”. This widespread power outage resulted in an immediate drop in request traffic from Cuba. Over the following two days, X posts from @EnergiasMinasCub, @OSDE_UNE (the Cuban Electric Union), and @ETECSA_Cuba (the Cuban Telecommunications Company) kept impacted subscribers apprised of the status of ongoing repairs. Traffic levels returned to expected levels around 20:00 local time on March 16 (00:00 on March 17 UTC), two full days after the incident began.

Panama

An explosion and fire at the La Chorrera Thermoelectric Power Plant in Panama caused a massive power outage across the country, starting at 23:40 local time on March 15 (04:40 on March 16 UTC). As expected, traffic dropped immediately, as seen in the HTTP and DNS request graphs below. However, recovery was fairly swift, as the electric system saw 75% recovery by 03:00 local time (08:00 UTC), with full restoration completed at 06:08 local time (11:08 UTC). Traffic volumes began to increase after power was restored.

Severe weather

Ireland

Storm Éowyn wreaked havoc on Ireland in late January, knocking out power and water, causing property damage, and limiting air and train travel. The storm’s impacts also disrupted Internet connectivity, as we observed traffic from Connacht and Ulster fall by 75% as compared to the previous week at 06:30 local time (06:30 UTC) on January 24. As recovery from the storm progressed over the next several days, Internet traffic gradually recovered as well, with traffic in the two provinces reaching levels near those seen the prior week by mid-day on January 28.


Réunion

Cyclone Garance made landfall over the French territory of Réunion at ~10:00 local time (06:00 UTC) on February 28. Damage from the storm’s 100+ miles/hour (160+ km/hour) winds caused power outages and infrastructure damage, resulting in disruptions to Internet connectivity. The most significant impacts to traffic were observed in the hours after the storm made landfall, but it took several days before traffic returned to expected levels, reaching that point around 08:00 local time (04:00 UTC) on March 4.

Earthquake

Myanmar

On March 28 at 12:50 local time (06:20 UTC), a magnitude 7.7 earthquake shook Myanmar, resulting in power outages and fuel shortages. Almost immediately, traffic dropped by around 40% at a country level. Regionally, traffic from Nay Pyi Taw dropped 97% as compared to the previous week, Mandalay fell 90%, Ayeyarwady lost 88%, Bago 50%, and Shan State was down 38%.

While recovery efforts stretch into April, regular traffic patterns and volumes bounced back within days, as seen in the HTTP and DNS request graphs below.

However, at a network level, recovery has been mixed. Both AS134840 (MCCL) and AS136442 (Oceanwave) saw significant drops in traffic after the earthquake occurred, and traffic remained disrupted on both networks through the end of the first quarter. Peak traffic on MCCL has increased slightly, but nearly two weeks on, remains significantly lower than pre-earthquake levels. Traffic on Oceanwave saw steady growth after the initial disruption, and as of this writing is approaching pre-earthquake peaks. (It is unclear what caused the significant spike in request traffic seen from Oceanwave on April 3-4.) In contrast to these two providers, traffic from AS163255 (Mytel) saw a significantly smaller disruption, and a significantly faster recovery, as did traffic from AS135300 (Myanmar Broadband Telecom).

Cyberattack

Russia

On January 7, Russian Internet provider Nodex (AS29329) said in a post on Russian social media platform VKontakte (translated) “Dear Subscribers, our technical staff is still working on restoring the network. The process is painstaking and long. We express our deep gratitude to those who support us in this difficult moment! This is really important for us. Let me remind you that our network was attacked by Ukrainian hackers, which resulted in its complete failure. At the moment, its functioning is being restored. There will be communication. When, is still unknown.” The Ukrainian Cyber Alliance, a community of pro-Ukraine cyber activists formed in 2016, claimed responsibility for the attack in a Telegram post.

The “complete failure” of the Nodex network is visible in the traffic graph below, where Internet traffic from the network began to drop after 03:00 local time (00:00 UTC) on January 7, reaching zero around 05:30 local time (02:30 UTC). Traffic from the network remained essentially non-existent until around 14:00 local time (11:00 UTC) the next day, recovering fairly quickly after that. Announced IPv4 address space fell by two-thirds at the same time that traffic volume dropped to zero, but recovered at 21:20 local time (18:20 UTC).

Fire damage

Los Angeles, California

Between January 7-9, during the early days of the 2025 Southern California wildfires — which affected the Palisades and Eaton areas in Los Angeles — there were clear Internet disruptions in at least 13 Los Angeles neighborhoods. According to Cloudflare’s data, traffic drops of over 50% compared to the previous week were especially noticeable in cities like Pacific Palisades, Altadena, Malibu, Temple City, and Monrovia, among others. In the weeks that followed, traffic remained significantly lower than before the fires, particularly in Pacific Palisades and Altadena, reflecting the devastation in those areas. However, traffic recovery occurred significantly sooner in Malibu, Temple City, and Monrovia, although peak traffic levels remain somewhat below pre-fire levels. 






Haiti

On January 15, an X post from the Director General of Digicel Haiti (AS27653) stated (translated) “Dear customers, last night at 8:30 pm we suffered damage to 2 of our international fiber optic cables caused by a fire in the metropolitan area. At 10:30 am a 3rd outage affected all international services, Internet and Moncash. Our teams are mobilized to resolve the problem as quickly as possible.” These fires ultimately caused two complete Internet outages for Digicel Haiti’s customers, as seen in the graphs below.

Both traffic and announced IP address space (IPv4 & IPv6) dropped to zero between 20:30 – 21:45 local time on January 14 (01:30 – 02:45 on January 15 UTC) and again between 10:15 – 11:00 local time on January 15 (15:15 – 16:00 UTC).

Technical problems

Russia

On January 14, multiple Russian networks, including AS8359 (MTS), AS12389 (Rostelecom), AS16345 (Beeline), AS31133 (MegaFon), and AS203451 (K-telecom) all experienced a brief disruption in connectivity. As the traffic graphs below show, Internet traffic from these networks dropped by around 80% between 14:00 – 14:30 UTC. According to a statement from Roskomnadzor,  “A brief connectivity issue was identified. Network operations were promptly restored.” However, industry observers suggested that the cause may have been due to an update to the so-called “Russian firewall” that failed and was quickly rolled back.

Georgia

Subscribers to Magticom (AS16010), one of the largest Internet providers in Georgia, experienced a complete outage on January 27. Request traffic and announced IP address space disappeared at 21:25 local time (17:25 UTC), recovering at 01:55 local time on January 28 (21:55 UTC). A (translated) Facebook post from Magticom explained that the company’s Internet connectivity comes through “channels from Europe” and that “damage was reported in Turkey, where heavy snowfall and avalanche risks have prevented the partner company’s technical teams from reaching the affected area”. Further, it noted that on the backup channel, “suspicious damage was reported at three points on the Georgian side, in the territory of Adjara…” Magticom’s published start and end times for the outage align with the loss and recovery of traffic and announced IP address space observed in Cloudflare data. 

France

Subscribers of Bouygues Telecom (AS5410) in France experienced a brief disruption to their Internet connectivity on March 11. According to a (translated) X post from the provider, “Following a technical incident between 5 a.m. and 7 a.m. you may have encountered difficulties using your services.” As seen in the request traffic graphs below, a drop in traffic is visible between 05:00 – 06:45 local time (04:00 – 05:45 UTC), aligning with the provider’s stated timeframe. Bouyges Telecom did not provide any subsequent details around the cause of the “technical incident”.

Unknown cause

Syria

Major Internet outages and disruptions in Syria are generally well documented, such as the cable cuts discussed above. However, on February 3, a multi-hour disruption was observed in the country, but no underlying cause was ever publicly disclosed. Starting approximately 14:00 local time (11:00 UTC), traffic from the country dropped by approximately 80%, along with a ~60% drop in announced IPv4 address space. Both traffic and announced IP address space returned to expected levels around 23:00 local time (20:00 UTC). The outage was confirmed in an X post from Syrian Television.

Conclusion

While the single government-directed shutdown last quarter, and the lack of such shutdowns this quarter, is an encouraging trend, we expect that it will be short-lived if countries like Iraq and Syria once again take such measures to prevent cheating on nationwide exams. As always, we encourage governments to recognize the collateral damage of such actions, and suggest that they explore alternative solutions to this problem.

The Cloudflare Radar team is constantly monitoring for Internet disruptions, sharing our observations on the Cloudflare Radar Outage Center, via social media, and in posts on blog.cloudflare.com. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

A diversity of downtime: the Q4 2024 Internet disruption summary

Post Syndicated from David Belson original https://blog.cloudflare.com/q4-2024-internet-disruption-summary/

Cloudflare’s network spans more than 330 cities in over 120 countries, where we interconnect with over 13,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions at both a local and national level, as well as at a network level.

As we have noted in the past, this post is intended as a summary overview of observed and confirmed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter. A larger list of detected traffic anomalies is available in the Cloudflare Radar Outage Center.

In the third quarter we covered quite a few government-directed Internet shutdowns, including many intended to prevent cheating on exams. In the fourth quarter, however, we only observed a single government-directed shutdown, this one related to protests. Terrestrial cable cuts impacted connectivity in two African countries. As we have seen multiple times before, both unexpected power outages and rolling power outages following military action resulted in Internet disruptions. Violent storms and an earthquake predictably caused Internet outages in the affected countries. And unexpected issues with maintenance efforts caused outages at two European providers, while Verizon customers in several US states experienced a brief but unexplained outage.

Cable cuts

Rwanda

On October 1, local mobile provider MTN Rwanda (AS36890) published a post on X alerting subscribers of a double fiber cut in Tanzania and Uganda that may impact connection quality. As a result of these fiber cuts, Internet traffic began to drop sharply after 12:45 local time (10:45 UTC), with a full outage visible between 13:15 – 13:30 local time (11:15 – 11:30 UTC). Traffic then began to rapidly recover, recovering to expected levels around 19:00 local time (17:00 UTC). Several hours later, MTN Rwanda published a followup post confirming that all services had been restored.

The African Undersea and Terrestrial Fibre Optic Cables (AfTerFibre) map shows that in addition to connecting with networks to the north and south in Tanzania and Uganda, it appears that connections are also available through networks to the west in the Democratic Republic of the Congo (DRC). However, MTN Rwanda’s upstream providers and/or peers may not be routing traffic through DRC-based networks, meaning that they couldn’t be used as a backup path when the apparently simultaneous fiber cuts occurred.

Niger

On November 30, local mobile provider Airtel Niger (AS37531) posted a thread of messages on X apologizing for Internet service disruptions, explaining that (translated) “Indeed, due to a simultaneous interruption on the national optical fiber on the Niamey-Dosso, Niamey-Balleyara exits, our internet services are completely interrupted throughout the territory, beyond our control.” These simultaneous fiber cuts resulted in a near complete outage between 17:30 local time (16:30 UTC) on November 29 until 19:45 local time (18:45 UTC) on November 30.

It seems unusual that the message thread was not posted until after the outage was resolved. It is possible that Airtel Niger themselves had no backup connectivity, and could not post an update until connectivity was restored. Alternately, given that the first post of the thread starts with “[COMMUNIQUÉ IMPORTANT📢]” (“[IMPORTANT PRESS RELEASE 📢 ]”), it is possible that the alert and apology was communicated through more official channels, such as Airtel’s website, in a timely manner, with the thread on X simply a follow-up once Internet services were again available.

Power outages

Cuba 

Instability in a country’s electrical infrastructure often causes widespread power outages, which, in turn, disrupt Internet connectivity. This happened on October 18 in Cuba, where a post on X from the Ministry of Energy and Mines of Cuba noted (translated) “Following the unexpected departure of the Antonio Guiteras CTE, the National Electricity System was completely disconnected at 11 a.m. today. The Unión Eléctrica is working on its restoration.” The power outage caused Internet traffic within the country to drop by more than half within minutes (15:15 UTC). Connectivity was disrupted for approximately three-and-a-half days, as it returned to expected levels around 23:00 local time on October 21 (03:00 UTC on October 22).

The Ministry posted several status updates on October 19 and 20, covering the work being done to restore power across the country. A final X post on October 22 signaled the end of the power outage, proclaiming (translated) “At 02:44 pm the National Electric System was synchronized.

Several weeks later, power issues again impacted Internet connectivity in Cuba. On November 6, the Electrical Union of Cuba (Uníon Eléctrica) posted on X that (translated) “14:48 hours. Strong winds caused by the intense Hurricane Rafael, cause the disconnection of the National Electric System. Contingency protocols are applied.” The timing of this post aligns with a sharp decline in traffic observed from Cuba, which fell sharply around 14:30 local time (19:30 UTC). Over the following days, after Hurricane Rafael passed the island, the Uníon Eléctrica posted numerous updates on the restoration of electrical service. Internet traffic appeared to return to expected levels around 13:00 local time (18:00 UTC) on November 9, although full restoration of electrical services took several days longer.

On December 4, Cuba suffered its third nationwide power outage in as many months. Early that morning, the Ministry of Energy and Mines posted on X that (translated) “At 2:08 this morning, the Electrical System, SEN, was disconnected when the Antonio Guiteras thermoelectric plant went out due to the automatic tripping.” The loss of this electrical power due to the failure of this generation plant caused a significant drop in Internet traffic from Cuba, falling approximately 60% as compared to the previous week at just before 02:15 local time (07:15 UTC). Traffic recovered to expected levels almost a day later at around 00:30 local time (05:30 UTC). This timing aligns with a follow-on X post from the Ministry that announced that all units had been synchronized, signaling a restoration of electrical service.

Guadeloupe

An article published in The Guardian on October 25 noted that “The French Caribbean island of Guadeloupe has been left entirely without power after striking workers seized control of the territory’s power station.” Workers entered the power station’s command room “and caused an emergency shutdown of all the engines”, according to the article. The power outage caused by this “emergency shutdown” resulted in traffic dropping nearly 70% as compared to the previous week at 08:30 local time (12:30 UTC). Although “restored electricity supply for the 230,000 affected households was expected at 3 pm local time (19:00 UTC) at best”, it appears that recovery took significantly longer than expected, as Internet traffic did not return to expected levels until around 22:00 local time on October 26 (02:00 UTC on October 27) . A press release from the government at 11:00 local time (15:00 UTC) on October 26 gave an update on the recovery efforts, noting (translated) “160,000 users have had their electricity restored. The restoration of service for the 70,000 customers still cut off is continuing, with a return to normal expected over the weekend.” It also noted that “76% of Orange subscribers have been able to regain their network connection. 1,800 homes are still without internet.

Kenya

Power outages in Kenya resulted in multiple Internet disruptions during both the second and third quarters of 2024. A similar event occurred during the fourth quarter as well. An X post from Kenya Power contained a “Customer Alert” issued at 01:28 local time on December 18 (22:28 UTC on December 17) that informed customers that “We are experiencing a widespread power outage affecting most of the country, except parts of Western and North Rift regions.” This outage caused Internet traffic from the country to drop by over 70% starting just after midnight local time on December 18 (21:00 UTC on December 17). On December 18 at 07:35 local time (04:35 UTC), an update from Kenya Power posted to X reported that power had been restored to all affected areas. Internet traffic from the country had recovered to near expected levels by that time as well.

Natural disasters

United States, Florida

At 20:30 local time on October 9 (00:30 UTC on October 10), Hurricane Milton made landfall in Florida as a Category 3 storm. Damage from Milton was extensive, including flooding, downed trees and power lines, and damage to homes and businesses. The power outages and other infrastructure damage caused by the storm, coupled with evacuation from impacted areas, resulted in a notable Internet disruption at a state level. As seen in the graph below, peak traffic levels on October 10, after Milton’s arrival, were approximately 40% lower than the preceding days. As recovery and restoration efforts began over the following days, and as evacuees returned to home, school, and work, the state’s Internet traffic began to gradually increase.


This gradual recovery is also visible in the series of maps below, which illustrate cities where Internet traffic was over 50% lower than the same time the prior week, with snapshots taken at 09:00 local time (13:00 UTC) on October 10, 11, and 14. On October 10, over 70 cities had significantly lower traffic, while on October 14, it was just over 10 cities.


Mayotte

On December 14, Cyclone Chido caused significant destruction on the French territory of Mayotte in the Indian Ocean. Power, water, and communications infrastructure were all damaged, as well as homes and public facilities. Over three dozen people were killed, with thousands more injured. With such widespread devastation, Internet traffic from the country was also impacted, as would be expected. Chido made landfall in Mayotte early in the morning on December 14, and traffic dropped sharply around 09:00 local time (06:00 UTC), causing a near-complete Internet outage. After extremely slow growth over the following week, a diurnal pattern is once again visible, with peak traffic levels continuing to gradually increase through the end of the month. As of the third week of January 2025, Mayotte’s Internet traffic continues to slowly increase, but remains well below pre-Chido levels.

Vanuatu

A magnitude 7.3 earthquake struck 24 km WNW of Port-Vila, Vanuatu at 17:46 local time (01:47 UTC) on December 17. Internet traffic from the country dropped sharply almost immediately, falling nearly 90% compared to the previous week. A significant drop in announced IPv4 address space was also observed, suggesting that damage from the earthquake took core network provider infrastructure offline as well. Recovery was slow, with Internet traffic not returning to expected levels until around 23:00 local time (12:00 UTC) on December 26.

An editorial published on The Maritime Executive website highlights that Vanuatu is currently reliant on the Interchange Cable Network 1 (ICN1) submarine cable connection to Fiji for international Internet connectivity. The editorial states that “A fire at the cable landing station temporarily interrupted the power supply, disabling internet traffic. The connection was restored 10 days later…” The resolution of the power outage at the cable landing station roughly aligns with traffic returning to expected levels, suggesting that this was a significant driver of the drop in traffic seen from Vanuatu after the earthquake. Starlink’s satellite Internet service provides some nominal redundancy, as the company announced service availability on October 7. The TAMTAM submarine cable, connecting Vanuatu to New Caledonia, is expected to be ready for service in 2026 — once available, it will provide additional redundancy for Internet connectivity. 

Government directed

Mozambique

On October 25 in Mozambique, mobile Internet connectivity across multiple providers was shut down after protests against the re-election of the ruling Frelimo party became violent. Starting around 13:00 local time (11:00 UTC), significant drops in traffic were observed across AS30619 (Telecomiuncacoes de Mocambique), AS37342 (Movitel), and AS37223 (Vodacom). Both Vodacom and Movitel experienced near complete outages almost immediately, while some traffic remained on Telecomiuncacoes de Mocambique until just before 02:00 local time (00:00 UTC) on October 26. Connectivity was restored the morning of October 26, as traffic returned around 08:00 local time (06:00 UTC). However, after connectivity returned, some social media platforms and messaging applications remained unavailable.

Just over a week later, on November 3, subscribers on these mobile networks experienced another Internet shutdown. At around 20:30 local time (18:30 UTC) traffic dropped significantly on each of these networks, with connectivity disrupted for nearly 12 hours before recovering around 08:00 (06:00 UTC) the morning of November 4. Similar shutdowns (“Internet curfews”) were observed November 4-5 and November 6-7 on all three networks, and November 7-8 on Movitel and Vodacom. According to a published report, the country’s Minister of Transport and Communications “admitted that Internet access was restricted in order ‘to avoid the destruction of the country’”, but shifted blame to the impacted services providers, claiming that when they note misuse of their services, they can take the initiative of interrupting the services, as part of their “civil responsibility” to safeguard “the stability and welfare of the population”.

Military action

Syria

An Internet disruption observed in Syria on November 9 may have been caused by damage from an Israeli airstrike near Aleppo and Idlib reported to have taken place earlier that morning. Internet traffic from the country dropped by about 80% at around 04:00 local time (01:00 UTC), with announced IP address space from the country falling significantly at that time as well. The disruption lasted approximately four hours, with traffic and announced IP address space returning to expected levels around 08:00 local time (05:00 UTC). 

Internal analysis of city-level Internet traffic shows a similar disruption in Aleppo, suggesting that it may have been caused by the airstrike.


Ukraine

Russian missile strikes on November 17 targeting electrical power infrastructure in Ukraine resulted in rolling power outages in multiple regions across the country. As we have seen multiple times throughout the nearly three-year-old conflict, these power outages result in disruptions to Internet traffic, impacting both service provider infrastructure and subscriber connectivity.

During the period between 07:30 local time (05:30 UTC) on November 17 and 02:00 local time (00:00 UTC) on November 23, we observed lower Internet traffic as compared to the previous week in Odessa, Zaporizhzhia, Mykolaiv, and Sumy. Traffic in Odessa initially dropped on November 17 by around 50% as compared to the prior week, while on November 18, traffic dropped by over 20% in the other regions. Traffic largely recovered in Odessa by November 21, while the other regions took several additional days.





Similar attacks took place just a few days later, with additional Russian airstrikes again targeting electrical infrastructure in Ukraine. Once again, Ukrainian officials implemented emergency power outages, which impacted Internet traffic in multiple areas across the country. Starting around 07:00 local time (05:00 UTC) on November 28, we observed traffic drop by as much as 65% as compared to the previous week in Kherson Oblast, Mykolaiv, Ternopil Oblast, Rivna, and Lviv. Traffic remained lower over the next several days, but appears to have generally recovered by December 1.






Maintenance

Switzerland, Salt Mobile

According to the image below, which replaced the homepage of Swiss provider Salt Mobile (AS15796), reported maintenance took the network completely offline early in the morning of December 3.


The outage lasted nearly three hours, with observed traffic at or near zero, between 01:25 and 04:20 local time (00:25 – 03:20 UTC). 

Greenland, Tusass A/S

A December 10 update from Tusass A/S (AS8818, formerly TeleGreenland) explained why the provider experienced a complete Internet outage between 02:30 and 05:15 local time (04:30 – 07:15 UTC) that morning. The post noted “This happened because preventive maintenance was to be done on the connections in Canada between 02:00 and 06:00 last night, but with a combined fault on our connection to Denmark we lost nationwide connectivity. Fortunately, the fault on the connection to Denmark occurred on land, and therefore easy to repair.” The graphs below show that for the duration of the outage, traffic from the network dropped to zero, no IPv6 address space was announced, and the volume of announced IPv4 address space fell by 94%.

According to Telegeography’s Submarine Cable Map, the Greenland Connect cable system connects Greenland to Newfoundland, Canada. It is possible that the fault on the connection to Denmark may have occurred on the Greenland-to-Iceland segment of the Greenland Connect cable system; the Iceland-to-Denmark connection is made over the DANICE submarine cable.

Unknown

United States, Verizon

Very early in the morning of November 12, some subscribers of Verizon’s Fios Internet service experienced a disruption to their Internet connectivity. A post to the Outages mailing list noted that a major multi-state Verizon Fios outage began at 12:28am EST, impacting Virginia, Washington DC, Maryland, and New Jersey, as well as parts of eastern Pennsylvania. Traffic from AS701, the autonomous system used by Verizon for their Fios service, dropped by approximately 30% around 00:30 Eastern time (05:30 UTC). At a state level, traffic from AS701 dropped between 50-70% in Pennsylvania, Delaware, Maryland, and Washington DC.

A subsequent post on the Outages mailing list stated that the outage was resolved everywhere at 3:23am EST (08:23 UTC). Nearly six hours after the outage ended, Verizon Support published a post on X acknowledging the issue, stating “A network issue early this morning disrupted service for some Verizon Fios customers in the Northeast for a short period of time. As soon as the issue was identified, our engineering teams quickly restored the service.” However, they did not provide any information on what ultimately caused the service disruption.

Conclusion

In addition to the outages and disruptions covered above, resilient Internet connectivity meant that two Baltic Sea cable cuts that occurred on November 17 and 18 had minimal impact. Whether accidental or sabotage, the security and resiliency of submarine cable infrastructure continues to be an important topic. The security and resilience of terrestrial cable infrastructure, as well as other critical Internet infrastructure, must also remain top of mind to help speed recovery from storms, earthquakes, military action, and power outages.

The Cloudflare Radar team is constantly monitoring for Internet disruptions, sharing our observations on the Cloudflare Radar Outage Center, via social media, and in posts on blog.cloudflare.com. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

Cloudflare’s perspective of the October 30 OVHcloud outage

Post Syndicated from Bryton Herdes original https://blog.cloudflare.com/cloudflare-perspective-of-the-october-30-2024-ovhcloud-outage

On October 30, 2024, cloud hosting provider OVHcloud (AS16276) suffered a brief but significant outage. According to their incident report, the problem started at 13:23 UTC, and was described simply as “An incident is in progress on our backbone infrastructure.” OVHcloud noted that the incident ended 17 minutes later, at 13:40 UTC. As a major global cloud hosting provider, some customers use OVHcloud as an origin for sites delivered by Cloudflare — if a given content asset is not in our cache for a customer’s site, we retrieve the asset from OVHcloud.

We observed traffic starting to drop at 13:21 UTC, just ahead of the reported start time. By 13:28 UTC, it was approximately 95% lower than pre-incident levels. Recovery appeared to start at 13:31 UTC, and by 13:40 UTC, the reported end time of the incident, it had reached approximately 50% of pre-incident levels.


Traffic from OVHcloud (AS16276) to Cloudflare

Cloudflare generally exchanges most of our traffic with OVHcloud over peering links. However, as shown below, peered traffic volume during the incident fell significantly. It appears that some small amount of traffic briefly began to flow over transit links from Cloudflare to OVHcloud due to sudden changes in which Cloudflare data centers we were receiving OVHcloud requests. (Peering is a direct connection between two network providers for the purpose of exchanging traffic. Transit is when one network pays an intermediary network to carry traffic to the destination network.) 


Because we peer directly, we exchange most traffic over our private peering sessions with OVHcloud. Instead, we found OVHcloud routing to Cloudflare dropped entirely for a few minutes, then switched to just a single Internet Exchange port in Amsterdam, and finally normalized globally minutes later.

As the graphs below illustrate, we normally see the largest amount of traffic from OVHcloud in our Frankfurt and Paris data centers, as OVHcloud has large data center presences in these regions. However, in that shift to transit, and the shift to an Amsterdam Internet Exchange peering point, we saw a spike in traffic routed to our Amsterdam data center. We suspect the routing shifts are the earliest signs of either internal BGP reconvergence, or general network recovery within AS12676, starting with their presence nearest our Amsterdam peering point.


The postmortem published by OVHcloud noted that the incident was caused by “an issue in a network configuration mistakenly pushed by one of our peering partner[s]” and that “We immediately reconfigured our network routes to restore traffic.” One possible explanation for the backbone incident may be a BGP route leak by the mentioned peering partner, where OVHcloud could have accepted a full Internet table from the peer and therefore overwhelmed their network or the peering partner’s network with traffic, or caused unexpected internal BGP route updates within AS12676.

Upon investigating what route leak may have caused this incident impacting OVHcloud, we found evidence of a maximum prefix-limit threshold being breached on our peering with Worldstream (AS49981) in Amsterdam. 

Oct 30 13:16:53  edge02.ams01 rpd[9669]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 141.101.65.53 (External AS 49981) changed state from Established to Idle (event PrefixLimitExceeded) (instance master)

As the number of received prefixes exceeded the limits configured for our peering session with Worldstream, the BGP session automatically entered an idle state. This prevented the route leak from impacting Cloudflare’s network. In analyzing BGP Monitoring Protocol (BMP) data from AS49981 prior to the automatic session shutdown, we were able to confirm Worldstream was sending advertisements with AS paths that contained their upstream Tier 1 transit provider.

During this time, we also detected over 500,000 BGP announcements from AS49981, as Worldstream was announcing routes to many of their peers, visible on Cloudflare Radar.


Worldsteam later posted a notice on their status page, indicating that their network experienced a route leak, causing routes to be unintentionally advertised to all peers:

“Due to a configuration error on one of the core routers, all routes were briefly announced to all our peers. As a result, we pulled in more traffic than expected, leading to congestion on some paths. To address this, we temporarily shut down these BGP sessions to locate the issue and stabilize the network. We are sorry for the inconvenience.”

We believe Worldstream also leaked routes on an OVHcloud peering session in Amsterdam, which caused today’s impact.

Conclusion

Cloudflare has written about impactful route leaks before, and there are multiple methods available to prevent BGP route leaks from impacting your network. One is setting max prefix-limits for a peer, so the BGP session is automatically torn down when a peer sends more prefixes than they are expected to. Other forward-looking measures are Autonomous System Provider Authorization (ASPA) for BGP, where Resource Public Key Infrastructure (RPKI) helps protect a network from accepting BGP routes with an invalid AS path, or RFC9234, which prevents leaks by tying strict customer and provider relationships to BGP updates. For improved Internet resilience, we recommend that network operators follow recommendations defined within MANRS for Network Operators.

Forced offline: the Q3 2024 Internet disruption summary

Post Syndicated from David Belson original https://blog.cloudflare.com/q3-2024-internet-disruption-summary

Cloudflare’s network spans more than 330 cities in over 120 countries, where we interconnect with over 13,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. Thanks to Cloudflare Radar functionality released earlier this year, we can explore the impact from a routing perspective, as well as a traffic perspective, at both a network and location level.

As we have noted in the past, this post is intended as a summary overview of observed and confirmed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter. 

A larger list of detected traffic anomalies is available in the Cloudflare Radar Outage Center.

Having said that, the third quarter of 2024 was particularly active, with quite a few significant Internet disruptions. Unfortunately, governments continued to impose nationwide Internet shutdowns intended to prevent cheating on exams. Damage to both terrestrial and submarine cables impacted Internet connectivity across Africa and in other parts of the world. Damage caused by an active hurricane season caused Internet outages across the Caribbean and in multiple parts of the United States. Because Internet connectivity is dependent on reliable electrical power, both planned and unplanned power outages in South America and Africa resulted in multi-hour Internet disruptions. Military action continued to cause Internet outages in affected countries, as did infrastructure maintenance, fire, and a purported cyberattack. The quarter also saw several noteworthy Internet disruptions that did not have verified causes.

Government Directed

Over the past several years, we have seen multiple governments around the world implement Internet shutdowns in response to protests within their countries. Some shutdowns are more targeted, affecting only (a subset of) mobile Internet providers, while others are more aggressive, effectively cutting off Internet connectivity at a national level. In addition, we all too frequently see governments implement nationwide multi-hour Internet shutdowns in an effort to prevent students from cheating on national exams. Unfortunately, governments were active in both respects during the third quarter, as we observed multiple government directed Internet shutdowns. Several were covered in our August 1 blog post, A recent spate of Internet disruptions.

Bangladesh

Violent student protests in Bangladesh against quotas in government jobs and rising unemployment rates led the government to order the nationwide shutdown of mobile Internet connectivity on July 18, reportedly to “ensure the security of citizens.” This government-directed shutdown ultimately became a near-complete Internet outage for the country, as broadband networks were taken offline as well. At a country level, Internet traffic in Bangladesh dropped to near zero just before 21:00 local time (15:00 UTC). Announced IP address space from the country dropped to near zero at that time as well, meaning that nearly every network in the country was disconnected from the Internet.

Traffic and announced IP address space at a national level began to recover around 18:00 local time (12:00 UTC) on July 23, and continued over the next several days, as fixed broadband connectivity was restored, with mobile connectivity returning on July 28. The initial restoration was characterized as a “trial run”, prioritizing banking, commercial sectors, technology firms, exporters, outsourcing providers and media outlets, according to the state minister for post, telecommunication and information technology.

Ahead of this nationwide shutdown, we observed outages across several Bangladeshi network providers, perhaps foreshadowing what was to come. At AS24389 (Grameenphone), a complete Internet outage started at 01:30 local time on July 18 (19:30 UTC on July 17), with a total loss of both Internet traffic and announced IP address space.

The outage at AS25245 (Banglalink) started at 02:15 local time on July 18 (20:15 UTC on July 17) as both Internet traffic and announced IP address space dropped to zero.

At AS24432 (Robi Axiata), an Internet outage was observed starting around 06:30 local time on July 18 (00:30 UTC), with both Internet traffic and announced IP address space disappearing at that time.

Internet traffic at AS58715 (Earth Telecommunication) began to fall at 18:00 local time on July 18 (12:00 UTC), reaching zero four hours later. Announced IP address space began to fall at 21:00 local time (15:00 UTC), and was completely gone by 21:25 local time (15:25 UTC).

AS63526 (Carnival Internet) was one of the last to fall before the complete shutdown, losing traffic at 20:45 local time (14:45 UTC), and seeing all of its announced IP address space withdrawn over the following hour.

These mobile connectivity outages lasted from July 18 through July 28. Just a few days after connectivity was restored, additional clashes between police and protestors drove the government to order mobile Internet connectivity to be shut down again. As shown in the graphs below, traffic on these mobile network providers dropped between 13:30 and 14:15 local time (07:30 to 08:15 UTC) on Sunday, August 4.

These protests ultimately led the government to order a full Internet shutdown in the country, with both traffic and announced IP address space dropping precipitously around 10:30 local time (04:30 UTC) on Monday, August 5. However, the shutdown appeared to be short-lived, as broadband connectivity began to recover around 13:20 local time (07:20 UTC), with mobile connectivity being restored around 14:00 local time (08:00 UTC).

Iraqi Kurdistan

Both Iraq and Iraqi Kurdistan (the autonomous Kurdistan region in the northern part of the country) regularly implement government directed Internet shutdowns to prevent cheating on secondary and baccalaureate exams. Within Iraqi Kurdistan, we observed two sets of exam-related Internet shutdowns during the third quarter. The impacts of the shutdowns are visible on traffic from networks that operate within the region, as well as on the country-level graphs for Iraq.

The first round of shutdowns occurred in July, impacting AS59625 (KorekTel), AS21277 (Newroz Telecom), AS48492 (IQ Online), and AS206206 (KNET) between 06:00 – 08:00 local time (03:00 – 05:00 UTC) on July 3, 7, 10, and 14. This is consistent with shutdowns observed in the second quarter, as well as in June 2023. None of the impacted networks experienced a drop in announced IP address space during these shutdowns.

The second set of shutdowns in Iraqi Kurdistan took place across multiple days during the back half of August. On August 17, 19, 21, 24, 26, 28, and 31, all four network providers were again impacted, as seen in the graphs below, with traffic dropping between 06:00 – 08:00 local time (03:00 – 05:00 UTC).

Iraq

In Iraq, a second round of exams for 12th graders resulted in over two weeks of regular Internet shutdowns across the country occurring between 06:00 – 08:00 local time (03:00 – 05:00 UTC) on multiple days between August 29 and September 16, intended to prevent cheating on second ministerial exams for secondary education. Both HTTP traffic and announced IP address space from Iraq dropped during these shutdowns, as seen in the graphs below.

(Note that the red annotation bar visible on September 11 & 12 on both the country and network-level graphs below highlights an internal data pipeline issue, and is not associated with an Internet shutdown in Iraq.)

This round of government-directed shutdowns impacted multiple local network providers, including AS58322 (Halasat), AS51684 (AsiaCell), AS203214 (HulumTele), AS199739 (Earthlink), and AS59588 (ZAINAS). In reviewing the distribution of mobile device and desktop traffic at a network level, gaps were observed during the shutdowns on AS58322 and AS199739, and to a lesser extent, AS203214, suggesting that these networks were completely offline, while AS56184 and AS59588 remained at least partially online. (This is also corroborated by complete or partial loss of announced IP address space across these networks during the shutdowns.)

Syria

A first round of exam-related Internet shutdowns took place in Syria earlier this year, between May 26 and June 13, and were discussed in our Exam-ining recent Internet shutdowns in Syria, Iraq, and Algeria blog post. A second set of exams, and the associated Internet shutdowns requested by the Ministry of Education, began on July 25 and ran through August 8, as specified in the schedule published by Syrian Telecom on its Facebook page.

The length of the shutdowns varied by day — they all began at 07:00 local time (04:00 UTC), but the end times ranged between 09:45 -10:30 local time (06:45 – 07:30 UTC). The graphs below show the impact at a country level, as well as to AS29256 (Syrian Telecom), the primary telecommunications provider within the country.

These shutdowns were also covered in our August 1 blog post, A recent spate of Internet disruptions.

Mauritania

On August 12, a round of baccalaureate exams began in Mauritania, and in an effort to prevent student cheating on the exams, the government instituted multiple Internet shutdowns that impacted several major mobile providers. Two shutdowns were observed on August 12, between 08:00 – 12:00 local time (08:00 – 12:00 UTC) and between 15:00 – 19:00 local time (15:00 – 19:00 UTC), and an additional one was observed on August 13, between 08:00 – 12:30 local time (08:00 – 12:30 UTC). Impacted network providers included AS37508 (Mattel), AS37541 (Chinguitel), and AS29544 (Mauritel). Announced IP address space for these networks remained unchanged during the shutdown periods, suggesting that that mobile subscriber connectivity was disabled, as opposed to the networks effectively being disconnected from the Internet, as we have seen in other countries.

Exam-related Internet shutdowns are, unfortunately, not new to Mauritania, as authorities in the country also implemented them between 2017 and 2020.

Cable cuts

Eswatini (Swaziland)

On July 14, MTN Eswatini (AS327765) informed customers via a post on X that “connection to the internet and data services is currently intermittent, because of fiber cable breaks resulting from wildfires.” This apparent connection disruption was visible in Cloudflare Radar between 19:30 and 20:15 local time (17:30 and 18:15 UTC).

Cameroon

In Cameroon, a fiber cut that occurred on August 4 during sanitation work disrupted mobile connectivity for Cameroon Telecommunications (AS15964 (Camtel)) customers for over half a day. According to a (translated) post on X from Camtel, “We inform you that due to the sanitation work carried out in the city of Yaoundé, at the place called Cradat, our Voice and Data services have been temporarily interrupted on the entire mobile network.” The observed disruption occurred between 03:00 – 16:30 local time (02:00 – 15:30 UTC). Although it initially started during a time when traffic was lower overnight anyway, both request and bytes traffic remained lower than the same time a week prior during the duration of the disruption.

Liberia

The Liberia Telecommunications Authority posted an announcement to their Facebook page on August 21 noting that “We have been informed by the CCL that the ACE Cable is experiencing interruptions.” (The Africa Coast to Europe (ACE) submarine cable connects multiple countries along the West Coast of Africa to Portugal and Europe.) The announcement further noted that the first signs of interruption occurred at 01:00 local time (and UTC), and that Lonestar Cell MTN (AS37410) was among the providers that had been “gravely affected” by the cut.

We observed traffic on Lonestar Cell MTN dropping just after 01:00, in line with the announcement. The network experienced a complete outage lasting over a day and a half, before traffic started to recover at 14:00 local time (and UTC) on August 22. In a Facebook post on August 22, Lonestar Cell MTN confirmed that Internet service had been restored, and that customer accounts would be credited with 500 MB of data for free.

Niger

A September 7 post on X from Airtel Niger alerted customers to Internet service disruptions caused by cuts on international fiber optic cables. As a land-locked country, Niger is dependent on terrestrial connections to networks in neighboring countries, but it isn’t clear which connection or country Airtel Niger’s post was referencing.

Two significant Internet disruptions were observed around the time of Airtel Niger’s post that we believe are related to the referenced fiber cuts. The first occurred between 18:00 – 21:00 local time (17:00 – 20:00 UTC) on September 6, visible at a country level and at a network level as well on AS37531 (Airtel Niger) and AS37233 (Orange Niger / Zamani Telecom). The second disruption occurred between 10:45 – 12:00 local time (09:45 – 11:00 UTC) on September 7, visible at a country level as well as on those two networks. 

Haiti

Internet disruptions related to submarine cable failures often take a significant amount of time to resolve because of the challenges repair crews face in getting to, and accessing, the damaged portion of the cable, as it is frequently located deep underwater in the middle of an ocean. A September 14 submarine cable failure that impacted Digicel Haiti (AS27653) lasted for over a week for a similar, but slightly different, reason.

A significant loss of traffic on Digicel Haiti was first observed at 08:00 local time (12:00 UTC) on September 14. On September 16, Digicel Haiti posted a press release confirming that since September 14, a failure had been detected on an international submarine cable belonging to Cable and Wireless, and that the cable damage occurred at Kaliko Beach Club (the property is reportedly used as a cable entry point). Digicel noted that their technicians went to the scene of the damage immediately, but were denied access, apparently because of a business dispute dating back to 2021. The release also explained that technical teams had taken temporary steps to ensure the continuity of essential services, which prevented the incident from resulting in a complete loss of connectivity. On September 22, a subsequent press release posted by Digicel Haiti announced the restoration of Internet services as of 02:00 local time (06:00 UTC), and referenced vandalism as the cause of the cable damage.

Kyrgyzstan

Reported damage to the “backbone wire” or “main cable” of an upstream provider resulted in a brief Internet outage for Kyrgyzstan Internet provider Megacom (AS50223) of September 25. AS12389 (Rostelecom) is listed as Megacom’s only upstream provider.

The outage lasted for only an hour, between 15:45 and 16:45 local time (09:45 – 10:45 UTC), dropping both traffic and announced IP address space to zero. At a country level, traffic dropped as much as 72% as compared to the previous week. Given the complete loss of both traffic and IP address space, the damage likely occurred on the connection between Megacom and Rostelecom.

Severe weather

An active hurricane season during July, August, and September resulted in infrastructure damage caused by multiple hurricanes disrupting Internet connectivity in multiple places across the Caribbean and Southeastern United States.

Grenada & Saint Vincent and the Grenadines

At the start of the third quarter, Grenada and Saint Vincent and the Grenadines both suffered significant damage from Hurricane Beryl, reportedly causing destruction of infrastructure, buildings, agriculture, and the natural environment.

On July 1, traffic from Grenada dropped significantly at 10:00 local time (14:00 UTC), just ahead of landfall on Grenada’s Carriacou Island. The most significant impacts to traffic were seen for approximately the first 24 hours, though traffic did not return to expected pre-storm levels until around 10:00 local time (14:00 UTC) on July 5.

Internet traffic in Saint Vincent and the Grenadines was also disrupted by Hurricane Beryl, also falling at 10:00 local time (14:00 UTC). Similar to Grenada, the most significant impact was seen in the first 24 hours, with consistent gradual recovery seen after that time. However, traffic did not return to expected pre-storm levels until July 11.

Jamaica

As Hurricane Beryl continued across the Caribbean, it passed Jamaica on July 3. The associated damage that it caused impacted Internet connectivity on the island, with traffic dropping significantly around 14:00 local time (19:00 UTC). As the graph below shows, the disruption was preceded by higher than normal traffic volumes, presumably due to residents looking for information about Beryl. The disruption lasted nearly a week, with traffic returning to expected levels on July 10.

U.S. Virgin Islands

The following month, damage from Tropical Storm Ernesto caused power outages across the U.S. Virgin Islands, resulting in disruptions to Internet connectivity. Traffic from the islands dropped precipitously at 22:00 local time on August 13 (02:00 UTC on August 14) and remained lower for over two days, before returning to expected pre-storm levels around 11:00 local time (15:00 UTC) on August 16.

Bermuda

Over the course of the following few days, Ernesto strengthened from a tropical storm into a hurricane, but had weakened by the time it hit Bermuda on August 16/17. In this case, damage was reportedly limited to power outages, downed trees, and flooding, but even this limited damage disrupted Internet connectivity on the island. As the storm made landfall on the island, traffic levels dropped over 80% at 22:00 local time on August 16 (01:00 UTC on August 17). Traffic levels remained depressed for about two and a half days, recovering to expected levels around 09:00 local time (12:00 UTC) on August 19.

Nepal

Heavy rains in Nepal at the end of September resulted in flooding and landslides across much of the country, which in turn resulted in power outages and Internet disruptions. One such disruption believed to be associated with the impacts of the storm was observed on September 28, when AS23752 (Nepal Telecom), AS45650 (Vianet), AS139922 (Dishhome), and AS17501 (Worldlink) all saw traffic drop 50 – 70% between 14:15 – 16:00 local time (08:30 – 10:15 UTC).

United States

A disruption to traffic from AS11427 (Charter Communications/Spectrum) in Texas that occurred between 12:30 and 19:30 local time on July 9 (17:30 – 00:30 UTC) was caused by “a third-party infrastructure issue caused by the impact of Hurricane Beryl”, according to a July 9 post on X from the provider. Spectrum acknowledged the issue shortly after it began, and followed up again after service had been restored.

Hurricane Helene made landfall in northern Florida as a Category 4 storm late in the evening (local time) on September 26, and over the following hours and days, continued north through Georgia, South Carolina, and North Carolina, and into Tennessee. Even as it weakened, it caused historic flooding and damage to roads, homes, power lines, and telecommunications infrastructure. Below, we review the traffic impacts observed at a state level in three of the most impacted states, as well as exploring the impact at a network level for selected providers. (Doug Madory at Kentik published an excellent blog post exploring the impact of Helene from the perspective of their data, and the networks referenced below were informed by that post.)

Georgia

Helene entered Georgia early morning on Friday, September 27, and by midday (local time), peak traffic was approximately 20% lower than peak levels seen in the days ahead of the storm. (The lower peaks on September 28 & 29 are likely due to it being a weekend.) At a state level, peak traffic remained lower over the following week, with more recovery seen heading into the week of October 6.


One of the most significantly impacted network providers in Georgia was AS11240 (ATC Broadband), which saw traffic start to drop around 22:00 local time on September 26 (02:00 UTC on September 27). Subscribers and customers experienced a near complete outage until around 08:00 local time on September 30 (12:00 UTC), when traffic volumes slowly started to recover. The normal diurnal traffic pattern became more clear in the following days, with peak traffic levels continuing to increase over the next week as well.

Other network providers in Georgia that experienced significant impacts include AS400511 (Clearwave Fiber), AS394473 (Brantley Telephone Company), AS40285 (Northland Cable Television), AS15313 (Pembroke Telephone Company), and AS397118 (Glenwood Telephone Company).

South Carolina

The midday traffic peak on September 27 in South Carolina was just 65% of the preceding days, with the peaks remaining lower over the following two weekend days. Traffic remained somewhat lower during the week following Helene, with peak increases becoming more evident the week of October 6.


At AS19212 (Piedmont Rural Telephone) in South Carolina, traffic began to fall rapidly around midnight local time on September 27 (04:00 UTC), reaching a state of near complete outage over the next eight hours. A gradual recovery is visible over the following several days, with a more regular pattern becoming evident on October 1, with rapid growth over the following week, accelerating towards the end of the week.

Other network providers in South Carolina, including AS397068 (Carolina Connect), AS10279 (West Carolina Communications), AS20222 & AS21898 (TruVista), and AS14615 (Rock Hill Telephone), also experienced significant disruptions to connectivity in the wake of Helene.

North Carolina

Although a drop in traffic is visible in the graph for North Carolina on September 27, it occurs after a midday peak in line with previous days, and the magnitude is not as significant as that seen in South Carolina and Georgia. Traffic peaks over the following week are in line with the week preceding Helene’s arrival, with higher peaks seen the week of October 6.


North Carolina providers AS53488 (Morris Broadband) and AS53274 (Skyrunner) both experienced multi-day disruptions, likely related to damage from Helene. However, these disruptions took Morris Broadband completely offline several times over the course of a week — the announced IP address space graph below shows three distinct drops to zero, aligning with outages visible in the traffic graph, when the network was effectively disconnected from the Internet. A similar but less severe pattern was seen at Skyrunner, which lost 75-80% of announced IP address space for a two-day period covering September 27-29, aligning with an outage visible in the associated traffic graph.

Other impacted network providers in North Carolina included AS22191 (Wilkes Communications) and AS23118 (Skyline Telephone).

Power outages

Venezuela

A nationwide power outage in Venezuela on August 30 was, according to President Nicolás Maduro, the result of an attack on the Guri Reservoir, Venezuela’s largest hydroelectric project. A published report indicated that all 24 of the country’s states reported a total or partial loss of electricity supply. The loss of power unsurprisingly caused an Internet disruption, with country-level traffic dropping 82%, starting around 04:45 local time (08:45 UTC). Traffic began to increase as electricity returned to various parts of the country throughout the day, and returned to expected levels just after midnight local time on August 31 (04:00 UTC). 

Kenya

On August 30, Kenya Power Care posted a Customer Alert on its Facebook page, issued at 21:57 local time (18:57 UTC), stating that “We have lost power supply to various parts of the country except North Rift region and sections of Western region.” Approximately a half hour before that alert, Kenya’s Internet traffic began to drop, falling as much as 61%. Just two hours later, Kenya Power Care posted a follow up, stating “Following the partial outage affecting several parts of the country this evening, we are pleased to report that power supply has now been restored to the entire Western region, as well as parts of Central Rift, South Nyanza, and Nairobi regions.” However, traffic did not return to expected levels for several more hours, taking until 06:00 local time (03:00 UTC).

A week later, on September 6, Kenya Power Care posted another similar Customer Alert, noting that “We are experiencing a power outage affecting several parts of the country, except sections of North Rift and Western regions.” This alert was issued at 09:20 local time (06:20 UTC), and follows a drop in Internet traffic that started around 09:00 local time (06:00 UTC). Traffic dropped approximately 45% during this power outage, and returned to expected levels around 16:00 local time (13:00 UTC). Traffic recovery aligns with a subsequent Customer Alert posted on Facebook, where Kenya Power Care stated “We are glad to report that normal electricity supply was restored across the country as at 3:49pm”.

A statement from Energy and Petroleum Cabinet Secretary Opiyo Wandayi, shared on Facebook by Kenya Power Care, explained the cause of the power outage: “Today, Friday 6th September 2024 at 8.56 am, the 220kV High Voltage Loiyangalani transmission line tripped at Suswa substation while evacuating 288MW from Lake Turkana Wind Power (LTWP) plant. This was followed by a trip on the Ethiopia – Kenya 500kV DC interconnector that was then carrying 200MW, resulting to a total loss of 488MW…” 

Ecuador

According to a (translated) September 7 post on X from CENACE, the national electricity operator in Ecuador, “We inform the public that due to a fault in the Molino substation bar, which is connected to the Paute generation, there has been a power outage in some provinces of the country. Cenace’s technical team, in coordination with the distribution companies, is working to gradually restore electrical service. It is estimated that it will take 3 to 4 hours maximum for the supply to return to normal.” The post was published at 09:53 local time (14:53 UTC), approximately an hour after Internet traffic from the country began to drop. Traffic returned to expected levels just under four hours later, at around 12:30 local time (17:30 UTC), in line with CENACE’s predicted time for power to be fully restored.

On September 18/19, the first of several planned nightly power outages to enable needed grid maintenance in Ecuador disrupted Internet connectivity. Traffic dropped by over 60% as compared to the same time the prior week starting around 21:30 local (02:30 UTC), with the power outages reportedly scheduled for 22:00 – 06:00 local time. Internet traffic recovered to expected levels around 06:00 local time (11:00 UTC) as power was restored. Similar power cuts were reportedly planned from September 23 to September 27, but these power outages did not appear to impact traffic levels in Ecuador as compared to the previous week

Senegal

Senegal’s power company, Senelec, posted a communiqué on X on September 12 that stated (translated) “Senelec informs its valued customers that an incident that occurred this morning at the Hann substation resulted in the loss of the OMVS interconnected network and disruptions to electricity distribution.” This disruption to electricity distribution also resulted in a disruption to Internet traffic, which dropped sharply at 13:00 local time (13:00 UTC), falling as much as 80%. Traffic recovered to expected levels by 20:00 local time (20:00 UTC) around the same time that Senelec posted a followup about the incident that stated (translated) “Effective restoration of electricity supply in all localities.

Maintenance

Syria

As we discussed above, Internet users in Syria were impacted by an exam-related Internet shutdown from 07:00 – 10:15 local time (04:00 – 07:15 UTC) on July 30. However, just an hour after connectivity was restored, another disruption occurred, as seen in both the traffic and announced IP address space graphs below. According to a (translated) Facebook post from Syrian Telecom, “…during the periodic maintenance of one of the air conditioners in one of the technical halls, an explosion occurred, which caused the internet circuits to be temporarily out of service.” Traffic remained depressed for approximately eight hours, recovering to expected levels around 19:00 local time (16:00 UTC).

Cyberattack

Russia

Roskomnadzor, Russia’s Internet regulate, blamed a brief disruption in traffic observed in Russia and on AS12389 (Rostelecom) on August 21 on a distributed denial-of-service (DDoS) attack that targeted Russian telecommunications operators. The disruption was brief, lasting from around 13:45 until 14:30 Moscow time (10:45 – 11:30 UTC). Roskomnadzor subsequently statedAs of 3 PM Moscow time, the attack has been repelled, and services are operating normally.” The disruption reportedly impacted messaging services Telegram and WhatsApp, as well as Wikipedia, Yandex, VKontakte, telecom support services, and mobile banking apps. Some experts questioned the official explanation, suggesting instead that the disruption was due to centralized interference from Roskomnadzor.

Military action

Palestine

We have covered Internet disruptions related to the ongoing conflict in Gaza multiple times since October 2023, both on Cloudflare Radar’s presence on X, and on the Cloudflare blog (1, 2, 3). In many of these cases, Paltel (AS12975) has posted notices on social media regarding service disruptions and outages. On September 8, Paltel posted a message on its Facebook page, stating (translated) “We regret to announce the suspension of home internet services in the central and southern areas of the Gaza Strip, due to the ongoing aggression.

Within the Gaza, Rafah, Deir al-Balah Governorates, we observed a sharp drop in traffic at 18:00 local time (16:00 UTC). The impact appeared to be most significant in Rafah and Deir al-Balah. Traffic returned to expected levels around 23:00 local time (21:00 UTC), and Paltel confirmed the service restoration in a subsequent Facebook post, stating (translated) “We would like to announce the return of home Internet services in central and southern Gaza Strip to the way it was before it was interrupted hours ago.




Lebanon

Israeli airstrikes targeting the Lebanese capital of Beirut on September 28 likely knocked local network provider Solidere (AS42852) offline for several hours. The graph below shows a loss of traffic starting around 12:15 local time (10:15 UTC), at the same time a complete loss of announced IP address space occurred. Most of Solidere’s IP address space started to get announced again at 14:45 local time (12:45 UTC), and a slight increase in traffic was seen at that time as well. Traffic levels fully recovered just after 18:00 local time (16:00 UTC), and announced IP address space had stabilized by that time as well. 

Fire

Algeria

A fire near a data center in Blida Province, Algeria disrupted connectivity on AS327931 (Djezzy) at 13:00 and local time (12:00 UTC) on July 24. According to a (translated) X post from Djezzy, “Djezzy announced fluctuations in its services in some areas of the country, as it was a victim of a fire that broke out on Wednesday, July 24, 2024, in a warehouse of one of the companies located near its technical center in the state of Blida.” The post from Djezzy predicted that “97% of the sites will be restored by around 3 am [July 25]”, but traffic did not return to expected levels until the end of the day on July 25.

Unknown

United States

On Monday, September 30, customers on Verizon’s mobile network in multiple cities across the United States reported experiencing a loss of connectivity. Impacted phones showed “SOS” instead of the usual bar-based signal strength indicator, and customers complained of an inability to make or receive calls on their mobile devices. Although initial reports of connectivity problems started around 09:00 ET (13:00 UTC), we didn’t see a noticeable change in request volume at an ASN level until about two hours later. AS6167 (CELLCO) is the autonomous system used by Verizon for its mobile network.

Just before 12:00 ET (16:00 UTC), Verizon published a social media post acknowledging the problem, stating “We are aware of an issue impacting service for some customers. Our engineers are engaged, and we are working quickly to identify and solve the issue.” As the graph below shows, a slight decline (-5%) in HTTP traffic as compared to traffic at the same time a week prior is first visible around 11:00 ET (15:00 UTC), and request volume fell as much as 9% below expected levels at 13:45 ET (17:45 UTC).

Media reports listed cities including Chicago, Indianapolis, New York City, Atlanta, Cincinnati, Omaha, Phoenix, Denver, Minneapolis, Seattle, Los Angeles, and Las Vegas as being most impacted. Traffic graphs illustrating the impacts seen in these cities can be found in our Impact of Verizon’s September 30 outage on Internet traffic blog post.

Traffic appeared to return to expected levels around 17:15 ET (21:15 UTC). At 19:18 ET (23:18 UTC), a social media post from Verizon noted “Verizon engineers have fully restored today’s network disruption that impacted some customers. Service has returned to normal levels.

Pakistan

On July 31, Pakistan experienced a wide-scale Internet disruption that lasted approximately two hours, between 13:30 – 15:30 local time (08:30 – 10:30 UTC). Traffic only dropped ~45% at a country level, but AS17557 (PTCL) experienced a near complete loss of traffic, while traffic at AS24499 (Telenor Pakistan) dropped nearly 90%. Together, the two network providers serve an estimated nine million users, and are among the top five Internet service providers in the country.

The actual cause of the disruption is disputed. It was reported that the Pakistan Telecommunication Authority (PTA) attributed the disruptions to a technical glitch in the international submarine cable affecting the Pakistan Telecommunication Company Limited (PTCL) network. However, another published report noted “According to our sources, the government’s latest firewall edition to block the content was misconfigured, resulting in Internet connectivity disruption.” Additional details can be found in our August 1 blog post, A recent spate of Internet disruptions.

United Kingdom

On August 14, subscribers of UK service provider Vodafone (AS25135) reported problems accessing both mobile and landline Internet connections. Starting around 11:00 local time (10:00 UTC), we observed traffic starting to drop, ultimately falling 43% below the same time the prior week. The disruption was fairly short-lived, as traffic returned to expected levels by 13:30 local time (12:30 UTC). Vodafone did not acknowledge the issue on social media, nor did it provide a public explanation for what caused the disruption.

Conclusion

Although Internet disruptions observed during the third quarter had a variety of underlying causes, those caused by power outages due to aging or insufficiently maintained electrical infrastructure are worth highlighting. Of course, widespread power outages always create a massive inconvenience for impacted populations, but over the last several years, as communication, entertainment, commerce, and more have become increasingly reliant on the Internet, the impact of these outages has become even more significant, because losing electrical power largely means losing Internet connectivity. Although mobile connectivity may still be available in some cases, it is decidedly not a complete replacement, not to mention that mobile devices will eventually need to be recharged. While addressing the underlying infrastructure issues require non-trivial amounts of time, resources, and money, governments appear to be taking steps towards doing so.

Visit Cloudflare Radar for additional insights around Internet disruptions, routing issues, Internet traffic trends, security and attacks, and Internet quality. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via e-mail.

Impact of Verizon’s September 30 outage on Internet traffic

Post Syndicated from David Belson original https://blog.cloudflare.com/impact-of-verizons-september-30-outage-on-internet-traffic

On Monday, September 30, customers on Verizon’s mobile network in multiple cities across the United States reported experiencing a loss of connectivity. Impacted phones showed “SOS” instead of the usual bar-based signal strength indicator, and customers complained of an inability to make or receive calls on their mobile devices.

AS6167 (CELLCO) is the autonomous system used by Verizon for its mobile network. To better understand how the outage impacted Internet traffic on Verizon’s network, we took a look at HTTP request volume from AS6167 independent of geography, as well as traffic from AS6167 in various cities that were reported to be the most significantly impacted.

Although initial reports of connectivity problems started around 09:00 ET (13:00 UTC), we didn’t see a noticeable change in request volume at an ASN level until about two hours later. Just before 12:00 ET (16:00 UTC), Verizon published a social media post acknowledging the problem, stating “We are aware of an issue impacting service for some customers. Our engineers are engaged and we are working quickly to identify and solve the issue.

As the Cloudflare Radar graph below shows, a slight decline (-5%) in HTTP traffic as compared to traffic at the same time a week prior is first visible around 11:00 ET (15:00 UTC). Request volume fell as much as 9% below expected levels at 13:45 ET (17:45 UTC).

Just after 17:00 ET (21:00 UTC), Verizon published a second social media post noting, in part, “Verizon engineers are making progress on our network issue and service has started to be restored.” Request volumes returned to expected levels around the same time, surpassing the previous week’s levels at 17:15 ET (21:15 UTC). At 19:18 ET (23:18 UTC), a social media post from Verizon noted “Verizon engineers have fully restored today’s network disruption that impacted some customers. Service has returned to normal levels.”


Media reports listed cities including Chicago, Indianapolis, New York City, Atlanta, Cincinnati, Omaha, Phoenix, Denver, Minneapolis, Seattle, Los Angeles, and Las Vegas as being most impacted. In addition to looking at comparative traffic trends across the whole Verizon Wireless network, we also compared request volumes in the listed cities to the same time a week prior (September 23).

Declines in request traffic starting around 11:00 ET (15:00 UTC) are clearly visible in cities including Los Angeles, Seattle, Omaha, Denver, Phoenix, Minneapolis, Indianapolis, and Chicago. In contrast to other cities, Omaha’s request volume was already trending lower than last week heading into today’s outage, but its graph clearly shows the impact of today’s disruption as well. Omaha’s difference in traffic was the most significant, down approximately 30%, while other cities saw declines in the 10-20% range. 









Request traffic from Las Vegas initially appeared to exhibit a bit of volatility around 11:00 ET (15:00 UTC), but continues to track fairly closely to last week’s levels before exceeding them starting at 16:00 ET (20:00 UTC). Cincinnati was tracking slightly above last week’s request volume before the outage began, and tracked closely to the prior week during the outage period.



We observed week-over-week traffic increases during the outage period in New York and Atlanta. However, in both cities, traffic was already slightly above last week’s levels, and that trend continued throughout the day. 



Based on our observations, it appears that voice services on Verizon’s network may have been more significantly impacted than data services, as we saw some declines in request traffic across impacted cities, but none experienced full outages.

As of this writing (19:15 ET, 23:15 UTC), no specific information has been made available by Verizon regarding the root cause of the network problems.

Q2 2024 Internet disruption summary

Post Syndicated from David Belson original https://blog.cloudflare.com/q2-2024-internet-disruption-summary


Cloudflare’s network spans more than 320 cities in over 120 countries, where we interconnect with over 13,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. Thanks to Cloudflare Radar functionality released earlier this year, we can explore the impact from a routing perspective, as well as a traffic perspective, at both a network and location level.

As we have seen in previous years, nationwide exams take place across several MENA countries in the second quarter, and with them come government directed Internet shutdowns. Cable cuts, both terrestrial and submarine, caused Internet outages across a number of countries, with the ACE submarine cable being a particular source of problems. Maintenance, power outages, and technical problems also disrupted Internet connectivity, as did unknown issues. And as we have frequently seen in the two-plus years since the conflict began, Internet connectivity in Ukraine suffers as a result of Russian attacks.

As we have noted in the past, this post is intended as a summary overview of observed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter.

Government directed

Syria, Algeria, Iraq

Each spring, governments in several countries in the Middle East and North Africa (MENA) region order local telecommunications providers to shut down or disrupt Internet connectivity across the country in an effort to prevent students from cheating on national secondary and high school exams. These shutdowns/disruptions generally occur for several hours per day over a multi-week period. We covered such events in 2023, 2022, and 2021, as they occurred in locations including Syria, Sudan, Algeria, and Iraq.

In June, we published Exam-ining recent Internet shutdowns in Syria, Iraq, and Algeria, which examined the daily Internet shutdowns that took place in Iraq and Syria, as well as the two multi-hour daily disruptions in Algeria, which appeared to be pursuing a content blocking strategy, rather than a full nationwide shutdown. The post examined the impact that these shutdowns have on Internet traffic, and also analyzed routing information and traffic from other Cloudflare services in an effort to better understand how these shutdowns are being implemented.

In addition to the shutdowns covered in the previously referenced blog post, Iraq implemented a second round of shutdowns that started on June 23, and ran through at least July 14. Some of these shutdowns impacted the same set of networks seen in the first round, and some impacted networks in the autonomous Kurdistan region in the north.

Among the latter set, AS206206 (Kurdistan Net), AS59625 (Korek Telecom), AS48492 (IQ-Online), and AS21277 (Newroz Telecom) all implemented shutdowns on June 23, June 26, June 30, July 3, July 7, and July 10, between 06:00 – 08:00 local time (03:00 – 05:00 UTC).

Outside the autonomous Kurdistan region, networks including AS59588 (Zainas), AS199739 (Earthlink), AS203214 (HulumTele), AS51684 (Asiacell), and AS58322 (Halasat) implemented Internet shutdowns between 06:00 – 08:00 local time (03:00 – 05:00 UTC) on June 23, June 24, June 26, June 27, June 29, June 30, July 1, and July 2.

Both sets of shutdowns reviewed above appeared to have followed the same approach as the first round covered in the earlier blog post.

Kenya, Burundi, Uganda, Rwanda, Tanzania

Concerns over a potential Internet shutdown during planned protests against tax increases proposed in “Finance Bill 2024” by the Kenyan government led to the publication of a joint statement signed by multiple organizations. The statement strongly urged the Kenyan government to refrain from enforcing any

Internet shutdowns or information controls, and highlighted the “disastrous economic effects” such a move could have. In response, the Communications Authority of Kenya issued a press release stating that “For the avoidance of doubt, the Authority has no intention whatsoever to shut down Internet traffic or interfere with the quality of connectivity. Such actions would be a betrayal of the Constitution as a whole, the freedom of expression in particular and our own ethos.

As protests escalated on June 25, Internet traffic in Kenya dropped at 16:30 local time (13:30 UTC). Initially, this outage was thought to be due to issues with one or more undersea cables that provide international connectivity to the country, with the potential cause supported by social media posts from Safaricom and Airtel.

Similar concurrent drops in Internet traffic were observed in Burundi, Uganda, Rwanda, and Tanzania, as shown below. Issues with submarine cables connected to one country can impact Internet connectivity in other countries if there is a dependency on that country/cable for upstream Internet connectivity. As such, the observed disruptions in those four countries were not that unusual. To that end, a (subsequently deleted) post on X from MTN Uganda noted: “Our esteemed customers, We are experiencing a degraded service on all our internet services due to an outage caused by our connectivity supply through Kenya. Our technical teams and partners are working jointly to resolve the issue in the shortest time possible. In the interim, we kindly advise our customers to use *165# to access Mobile Money and other app based services. Thank you.

However, other participants in the Internet infrastructure community in Africa called the undersea cable outage explanation into question. Kyle Spencer, Executive Director of the Uganda Internet eXchange Point, posted on X that “I am told the Kenyan government ordered sea cable landing stations to disconnect circuits.” Ben Roberts, Group CTIO at Liquid Intelligent Technologies (a pan-African network infrastructure provider), postedNo cables are damaged this week.” In addition, outages on undersea cables are rarely, if ever, resolved in a matter of hours, as this disruption was – they frequently last for days or weeks.

On June 26, Safaricom’s CEO claimed “This outage was occasioned by reduced bandwidth on some cables that carry Internet traffic”, contradicting the company’s original claim. No additional information was forthcoming from Airtel or the Communications Authority of Kenya, but as noted above, some within the industry believe that the disruption that impacted connectivity in Kenya, Burundi, Uganda, Rwanda, and Tanzania was directed by the government of Kenya, and was not caused by submarine cable outages.

Cable cuts

Haiti

At 17:36 local time (21:36 UTC) on April 28, Digicel Haiti posted an “important note” on X that stated in part (translated) “On April 27, 2024, the company suffered several attacks on its international optical infrastructure in the Drouya area on National Road #1. The optical fiber was damaged by the impact of cartridges after the armed clashes in the area for a few days. It affected several services such as internet (data), SMS, MonCash and international calling. For now, we are happy to inform the population that all services are restored to 100%.” The graph below shows the impact of the fiber damage, with AS27653 (Digicel Haiti) suffering an Internet outage lasting nearly 24 hours, from around 17:30 local time (21:30 UTC) on April 27 through approximately 16:00 local time (20:00 UTC) on April 28, after which traffic quickly recovered.

Then on May 3, The Director General of Digicel Haiti posted on X that (translated) “Digicel is informing the general public that it suffered two more damages to its international fiber infrastructure at 2am this morning. We have restored Moncash services, SMS, and Fiber Optic connections. Our crews are already on their way to address the apparent landslide in the Canaan area.” The disruption caused by this fiber damage lasted for approximately eight hours, between 02:15 – 10:30 local time (06:15 – 14:30 UTC), and as seen in the graph below, appeared to have a nominal impact on traffic.

Kenya, Madagascar, Malawi, Mozambique, Rwanda, Tanzania, Uganda

On Sunday, May 12, issues with the EASSy and Seacom submarine cables again disrupted connectivity to East Africa, impacting a number of countries previously affected by a set of cable cuts that occurred nearly three months earlier. Insight into these earlier cable cuts and the initial impact of May’s cable damage was covered in our East African Internet connectivity again impacted by submarine cable cuts blog post.

Traffic levels across a number of the impacted countries dropped just before 11:00 local time (08:00 UTC).  The magnitude of the initial impact varied by country, with traffic initially dropping by 10-25% in Kenya, Uganda, Madagascar, and Mozambique, while traffic in Rwanda, Malawi, and Tanzania dropped by one-third or more than compared to the previous week. The overall impact was most significant in Tanzania, Madagascar, and Rwanda, as seen in the graphs below. Traffic returned to expected levels at various times over the following week, ranging from a day and a half later (May 13) in Kenya to a week later (May 19) in Rwanda.

Repairs to the EASSy and Seacom cables were completed on May 31. Repairs to the cables damaged in February were ongoing as of July 9, as their location in a war zone complicates repair efforts.

Chad

A reported fiber optic cable cut in Cameroon disrupted Internet connectivity for customers of Moov Africa TChad on May 25. The outage lasted three hours, between 15:15 -18:15 local time (14:15 – 17:15 UTC), with the impact visible at a country level as well. Routing was disrupted too, as the number of IPv4 /24 prefixes (256 IPv4 addresses) announced by Moov Africa Tchad fell from eight to three during the disruption.

The event was similar to one that occurred on January 10, when Moov Africa Tchad and country-level traffic was disrupted for over 12 hours “due to a cut in the optical fiber coming from Cameroon through which Chad has access to the Internet”. During that event, significant volatility was also observed from a routing perspective, as the volume of announced IPv4 address space shifted frequently at a network and country level during the disruption. As we noted last quarter, as a landlocked country, Chad is dependent on terrestrial Internet connections to/through neighboring countries, and the AfTerFibre cable map illustrates Chad’s reliance on limited cable paths through Cameroon and Sudan.

Gambia, Mauritania, Senegal

A reported “network interruption” on the Africa Coast to Europe (ACE) submarine cable disrupted traffic across networks in the Gambia, Mauritania, and Senegal on June 5. AS25250 (Gamtel), AS29544 (Mauritel), and AS37649 (Free/Tigo) all saw traffic drop around 23:00 local time (23:00 UTC). As seen in the graphs below, the outage lasted for nearly 11 hours, with traffic recovering just 10:00 local time on June 6 (10:00 UTC). Mauritel saw a near complete outage, while Gamtel and Free/Tigo saw less severe impacts, possibly because they were able to shift traffic to back up links.

Maintenance

Guinea, Gambia, Sierra Leone, Liberia

Above, we discussed an unexpected network interruption on the ACE submarine cable that caused outages across multiple countries on June 5. However, two months earlier, a planned outage for repair work on the cable also disrupted connectivity across multiple African countries. A communiqúe issued by the Ministry of Posts, Telecommunications and the Digital Economy in Guinea noted in part (translated) “…the ACE (Africa Coast to Europe) network will undergo a planned outage on April 8, 2024, between midnight and 2:00 a.m. morning in the following countries: Guinea, Senegal, Gambia, Sierra Leone and Liberia. This total outage of approximately 2 hours will affect Internet traffic and international calls.

The graphs below show the impact to traffic in the listed countries for the planned two-hour repair window, though it appears that traffic did not return fully to expected levels after the repair window concluded – it is unclear why it remained slightly depressed. In addition, despite being listed as one of the impacted countries, no impact to traffic was observed in Senegal.

Guinea

Rounding out a trifecta of entries about the ACE submarine cable, planned maintenance work on the cable by GUILAB reportedly caused a multi-hour outage at AS37461 (Orange Guinea) and at a country level as well, lasting from 12:15 – 15:45 local time (12:15 – 15:45 UTC). (GUILAB is the company in charge of managing the capacity allocated to Guinea on the ACE submarine cable.) The maintenance work was reported by Orange Guinea in two X posts (1, 2), although these posts were subsequently deleted.

Power outage

Kenya

At 18:30 local time (15:30 UTC) on May 2, Kenya Power posted a “Power Outage Alert” on X that stated “At 5:40 PM (EAT) today, Thursday, 2nd May 2024, we experienced a system disturbance on the grid, resulting in power supply disruption in most parts of the country.” The graph below shows the resultant impact on Internet connectivity in the country, with traffic dropping sharply between 17:30 – 17:45 local time (14:30 – 14:45 UTC). The drop in traffic lasted until approximately 21:30 local time (18:30 UTC), the same time that Kenya Power posted a “Power Supply Restoration” notice on X, highlighting the restoration of power to parts of the country. Although the post-outage spike seen in the graph would suggest pent-up demand for online content, a longer-term view of Kenya’s Internet traffic shows traffic peaks at the same time (22:00 local time, 19:00 UTC) during the preceding two days as well.

Ecuador

A nationwide power outage in Ecuador on June 19 impacted hospitals, homes, and the subway, in addition to causing a major disruption to Internet connectivity. The graph below shows Ecuador’s Internet traffic dropping sharply just after 15:00 local time (20:00 UTC). A post on X from Public Works Minister Roberto Luque explained (translated) “The immediate report that we received from CENACE is that there is a failure in the transmission line that caused a cascade disconnection, so there is no energy service on a national scale.” A subsequent post pointed at a lack of investment in the underlying systems, and noted that as of 18:41 pm local time (23:41 UTC), “95% of the energy has already been restored”. After the initial sharp drop, traffic began to recover fairly quickly, and was effectively back to expected levels by the stated time.

Albania, Bosnia, Montenegro

A sudden increase in power consumption related to increased usage due to high temperatures, as well electrical systems being impacted by the heat, caused a widespread power outage across Montenegro, Bosnia, and Montenegro on June 21. The outage reportedly originated in Montenegro after a 400-kilowatt transmission line exploded. While power outages are generally more localized to a single country, or region within a country, power distribution systems are linked across Balkan countries as part of the Trans-Balkan Electricity Corridor.

Published reports (MSN, Reuters) noted that electrical networks went down 12:00 – 13:00 local time (10:00 – 11:00 UTC), and that electricity suppliers in the impacted countries started restoring power by mid-afternoon, and had it largely restored by the evening. The graphs below show traffic from Albania, Bosnia, and Montenegro starting to drop around 12:00 local time (10:00 UTC), reaching its nadir in Albania and Bosnia at 12:30 local time (10:30 UTC) and at 13:00 local time (11:00 UTC) in Montenegro. Traffic recovered gradually over the next several hours as power was restored, returning to expected levels by 15:30 local time (13:30 UTC).

Croatia was reportedly impacted by the power outage as well, but no adverse impact to traffic at a country level is visible during the timeframe that connectivity in the other countries was disrupted.

Military action

Ukraine

During the two-plus years of the Russia-Ukraine conflict, Ukraine’s power grid has been a frequent target for Russian air attacks. When damage to Ukraine’s electrical power infrastructure occurs as a result of these attacks, Internet connectivity is also disrupted. Attacks on May 21 caused power outages across a number of areas in Ukraine. The most significant impact was in Sumy, where traffic dropped as low as 82% below the previous week at 00:00 on May 22 local time (21:00 UTC). As the graphs below illustrate, traffic was also lower than the previous week for several hours in Kyiv, Kharkiv, and Vinnytsia, with traffic returning to expected levels by around 08:00 local time (05:00 UTC) on May 22.

\

Technical problems

Malaysia

As we’ve covered in previous quarterly posts, Internet outages and disruptions aren’t always due to significant wide-scale events like severe weather, power outages, or cable cuts. Sometimes more mundane technical issues can cause problems when users try to access the Internet. One example of this occurred on April 15 in Malaysia, when customers of Time Internet experienced a network outage for nearly two hours. The company explained the reason for the outage in a contrite post on their Facebook page, stating in part “This Internet service outage was by far the worst in our history – affecting approximately 40% of our customers. … At 5.38pm today, both our primary and secondary Secure DNS servers became unreachable. This means that any browser or service requiring a DNS address resolution was not able to reach its intended site.” Because subscribers could not reach Time Internet’s DNS resolvers, they were unable to resolve hostnames for Internet services, sites, and applications, including those delivered by Cloudflare. This resulted in the drop in traffic seen in the graph below, which started just after 17:00 local time (05:00 UTC), and began to recover approximately an hour later. The company did not provide any additional information on what caused the DNS servers to fail.

Nepal

In Nepal, a number of local Internet service providers including AS45650 (Vianet) and AS139922 (Dishhome) rely on Indian provider Bharti Airtel for upstream connectivity, enabling them to reach the rest of the Internet. A published report underscores the reliance, noting “Nepali ISPs buy 70 percent of their internet from Airtel.

On April 25, these ISPs warned that their services could be interrupted because the Nepali government had not provided them with foreign exchange services that would enable them to pay bandwidth vendors such as Airtel, whom they reportedly owed USD $30 million to. On May 1, Airtel informed the delinquent Nepali providers that Internet services may be interrupted at any time due to the overdue payment, and on May 2, Airtel took that step. The graphs below show Vianet’s traffic dropping to near zero at 16:15 local time (10:30 UTC), recovering to expected levels six hours later. An hour later, at 17:15 local time (11:30 UTC), Dishhome’s traffic dropped significantly, though not as severely as Vianet’s. Dishhome’s traffic also recovered approximately six hours later.

Dishhome may not have experienced a near-complete outage like Vianet did because Bharti Airtel is one of four upstream providers used by its parent company, whereas Bharti Airtel is one of Vianet’s two upstream providers.

A month later, on June 3, AS45650 (Vianet) and AS17501 (Worldlink) in Nepal experienced Internet disruptions that were reportedly caused by routing issues on Bharti Airtel’s network. On Worldlink, a drop in traffic occurred between 12:15 – 14:00 local time (06:30 – 08:15 UTC), while on Vianet, the loss of traffic took place between 12:15 – 13:15 local time (06:30 – 07:30 local time).

Unknown

Most of the Internet disruptions covered in this blog post series have a known root cause, whether admitted/stated by the impacted provider(s) or closely associated with a real world event (severe weather, power outage, etc.) However, other disruptions are observed and even publicized by the impacted provider, but no underlying reason for the outage is ever made public.

Malaysia

On May 21, CelcomDigi (AS10030) posted on X that it was experiencing an outage on its network, and that it was working to resolve the issue as soon as possible. However. just 12 minutes later, it published a second post stating that it had fully restored Celcom Internet service. These posts were made at 21:35 and 21:47 local time (13:35 and 13:47) respectively. However, as the graph below shows, traffic volumes had returned to expected levels over an hour earlier, as the observed Internet disruption on Celcom’s network lasted between 18:00 – 20:15 local time (10:00 – 12:15 UTC). (Note that the second disruption shown in the graph below was due to an internal Cloudflare data pipeline issue, and not any sort of problem with Celcom’s network.)

Starlink

SpaceX Starlink’s satellite Internet service is unique in that it has an international subscriber base, so outages on its network have a more wide-reaching impact than issues with an ISP that covers a single country. At 01:59 UTC on May 29, Starlink shared on X that it was currently experiencing a network outage, and that it was actively implementing a solution. Twenty-eight minutes later, it postedThe network issue has been fully resolved.” This brief outage is visible in the graph below as a slight dip in traffic. However, what is particularly interesting is the spike in traffic to Cloudflare from Starlink’s network following the resolution of the outage. The sharp increase and rapid decline of the traffic curve after service was restored suggests that it may be related to an automated connectivity check of some kind, rather than pent-up user demand for content.

Chad

A near-complete Internet outage was observed in Chad on June 5 between 08:15 – 12:00 local time (07:15 – 11:00 UTC), as seen in the graph below. Routing was also impacted, as the number of IPv4 /24 address blocks (256 IPv4 addresses) announced by network providers in the country dropped by as much as 75% during the outage.

A news item covering the outage noted that only Starlink subscribers retained Internet access during the outage. It also noted that Chad has faced recurring Internet disruptions since 2016, either because of problems with fiber-optic cables, or due to government directed shutdowns in the name of national security. It is unclear what ultimately caused this particular outage.

India

With an estimated subscriber base in excess of over 460 million, any Internet disruption affecting Reliance Jio’s network (AS55836) is going to have a widespread impact across India. On June 18, Reliance Jio experienced two disruptions that occurred between 13:15 – 17:15 local time (07:45 – 11:45 UTC). Each disruption lasted less than an hour, and dropped traffic levels to approximately half of those seen at the same time a week prior. Both mobile and fiber connectivity was affected, and no additional information has been provided by Reliance Jio regarding the root cause of the connectivity issues.

Conclusion

As we become increasingly dependent on reliable Internet connectivity, we must recognize that that connectivity is itself reliant on a complex and interconnected foundation of physical, technical, and political factors. A failure in any one of these foundational components, whether due to a cable cut, power outage, misconfiguration, or government action, can have a significant impact, disrupting Internet connectivity for millions of users, potentially across multiple countries. While the resilience and reliability of the physical and technical components can be improved through redundancy and best practices, political factors have arguably proven to be the hardest to address. However, organizations like AccessNow, through their #KeepItOn campaign, mobilize people, communities, and civil society actors globally to fight against government-directed Internet shutdowns, which can have significant financial consequences.

Visit Cloudflare Radar for additional insights around Internet disruptions, routing issues, Internet traffic trends, security and attacks, and Internet quality. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via e-mail.

Exam-ining recent Internet shutdowns in Syria, Iraq, and Algeria

Post Syndicated from David Belson original https://blog.cloudflare.com/syria-iraq-algeria-exam-internet-shutdown


The practice of cheating on exams (or at least attempting to) is presumably as old as the concept of exams itself, especially when the results of the exam can have significant consequences for one’s academic future or career. As access to the Internet became more ubiquitous with the growth of mobile connectivity, and communication easier with an assortment of social media and messaging apps, a new avenue for cheating on exams emerged, potentially facilitating the sharing of test materials or answers. Over the last decade, some governments have reacted to this perceived risk by taking aggressive action to prevent cheating, ranging from targeted DNS-based blocking/filtering to multi-hour nationwide shutdowns across multi-week exam periods.

Syria and Iraq are well-known practitioners of the latter approach, and we have covered past exam-related Internet shutdowns in Syria (2021, 2022, 2023) and Iraq (2022, 2023) here on the Cloudflare blog. It is now mid-June 2024, and exams in both countries took place over the last several weeks, and with those exams, regular nationwide Internet shutdowns. In addition, Baccalaureate exams also took place in Algeria, and we have written about related Internet disruptions there in the past (2022, 2023). However, in contrast to the single daily shutdowns in Syria and Iraq, the Algerian government opted instead for two multi-hour disruptions each day – one in the morning, one in the afternoon – and appears to be pursuing a content blocking strategy, rather than a full nationwide shutdown.

As we have done in past year’s posts, we will examine the impact that these shutdowns have on Internet traffic, but also analyze routing information and traffic from other Cloudflare services in an effort to better understand how these shutdowns are being implemented.

Syria

The Syrian Telecom Company, to their credit, publishes an exam schedule on social media, with the image below published to their Facebook page. The English version was created by applying Google Translate to the image. The schedule shows the date & time of each Internet shutdown (“disconnection”), in addition to the subject(s) of that day’s exam(s). In 2024, exams started on May 26, and went through June 13.

In Syria, AS29256 (Syrian Telecom) is effectively the Internet, as shown in the table below. While there are a few other autonomous systems (ASNs/ASes) registered in Syria, there are only two that currently announce IP address space to the public Internet. As such, the trends seen at a country level for Syria reflect those seen for AS29256, and this is clearly evident in the traffic graphs below.

Nationwide Internet shutdowns in Syria began on May 26, taking place for varying multi-hour periods from Sunday to Thursday for three consecutive weeks. The graphs below show Internet traffic from the country, as well as AS29256, dropping to zero during the scheduled shutdowns.

In addition, graphs from the Cloudflare Radar Routing pages for Syria and AS29256 show the number of IPv4 and IPv6 prefixes being announced country-wide and by AS29256 dropping to at or near zero during each shutdown. This ultimately means that there is no Internet path back to systems (IP addresses) connected to Syrian Telecom. Below, we explore why this is important and problematic.

As has been observed in the past, the shutdowns in Syria are asymmetrical. That is, traffic can exit the country (via AS29256), but there are no paths for responses to return. The impact of this approach is clearly evident in traffic to Cloudflare’s 1.1.1.1 DNS Resolver. We continue to see traffic to the resolver when the shutdowns take place, and in fact, we see the traffic spike during the shutdowns, as the graph below shows.

If we dig into traffic to 1.1.1.1 by protocol, we can see that it is driven by requests over UDP port 53, the standard port used for DNS requests over UDP and TCP. (Given the request pattern, that also appears to be the primary way that we see traffic to the resolver from Syria.)

If we remove the UDP line from the graph, we see that request volume for DNS over TCP port 53, as well as DNS over HTTPS (DoH) and DNS over TLS (DoT), all drops to zero during the shutdowns.

Similarly, we can clearly see the shutdowns in HTTP(S) request-based traffic graphs as well, since HTTP(S) is also a TCP-based protocol.

Why do we see this impact? With DNS over UDP, the client simply makes a request to the resolver – no multi-step handshake is involved, as with TCP. So in this case, 1.1.1.1 is receiving these requests, but as shown above, there’s no path for the response to reach the client. Because it hasn’t received a response, the client retries the request, and this flood of retries is manifested as the spike seen in the graphs above.

However, as we see above, request volume for DNS over TCP, as well as DoH, DoT, and HTTP(S) (which all use TCP), falls to zero during the shutdowns. The lack of a path back to the client means that the TCP 3-way handshake can’t complete, and thus we don’t see DNS requests over these protocols.

In looking at 1.1.1.1 Resolver request volume from Syria for popular social media and messaging applications, we can see traffic for facebook.com most closely matches the spikes shown above. Removing facebook.com from the graph, we can also see similar, though more limited, increases for domains used by popular messaging applications WhatsApp, Signal, and Telegram. Facebook and WhatsApp are reportedly the most popular social media and messaging applications in Syria.

Although we have focused on the analysis of traffic to Cloudflare’s DNS resolver, and the patterns seen within that traffic, it is also worth highlighting an interesting pattern observed in traffic to Cloudflare’s Authoritative DNS platform. (DNS resolvers act as a middleman between clients, such as a laptop or phone, and an authoritative DNS server. Authoritative DNS servers contain information specific to the domain names they serve, including IP addresses and other types of records.)

The graph below shows bits/second traffic from Syria for Cloudflare’s authoritative DNS service on June 13. (Similar patterns were observed during the other days when shutdowns occurred, but data volume limits the ability to create a graph showing an extended period of time.) In this graph, we can see that at the start of the shutdown (03:00 UTC), traffic rises sharply, effectively plateaus for the duration of the shutdown, and then returns to normal levels. We believe that the traffic pattern illustrated here could be the result of some local resolvers in Syria having the IP addresses for our authoritative DNS servers cached, and are making requests to them. The increased traffic level could be because they are retrying their queries after not receiving responses, but in a less aggressive fashion than the client applications driving the resolver traffic spikes shown above.

In summary, Syria appears to be implementing their Internet shutdowns not through filtering, but rather by simply not announcing their IP address space for the duration of the shutdown, thereby preventing any responses from returning to the originating requestor, whether client application, web browser, or local DNS resolver.

Iraq

On May 19, the Iraqi Ministry of Communication posted an update that stated (translated) “The Ministry of Communications would like to note that the Internet service will be cut off for two hours during the general exams for intermediate studies, from six in the morning until eight in the morning, based on higher directives and at the request of the Ministry of Education.” The post came nearly a year after the Iraqi Ministry of Communication refused a request from the Ministry of Education to shut down the Internet during the baccalaureate exams as part of efforts to prevent cheating. On May 20, the Iraqi Ministry of Education posted the schedule for the upcoming set of exams to its Facebook page.

Iraq has a much richer network service provider environment than Syria does, with over 150 autonomous systems (ASNs) registered in the country and announcing IP address space, compared to just two ASNs (both Syrian Telecom) in Syria announcing IP address space. Although traffic in Iraq is generally concentrated among the larger providers, shutdowns are rarely “complete” at a country level because not every autonomous system (network provider) in the country implements a shutdown. (This is due in part to the autonomous Kurdistan region in the north, which often implements similar shutdowns on their own schedule. Network providers in this region are included in Iraq’s country-level graphs.)

We can see this in a Cloudflare Radar traffic graph that shows the shutdowns at a country level, where traffic is dropping by around 87% during each multi-hour shutdown. In addition to the five networks also shown here (AS203214 (HulumTele), AS199739 (Earthlink), AS58322 (Halasat), AS51684 (Asiacell), and AS59588 (Zainas)), further analysis finds more than 30 where we observed a complete loss of traffic during the shutdowns, with a number of them downstream of these providers.

In contrast to Syria, the changes to announced IP address space during the shutdowns are much less severe in Iraq. Several of the shutdowns are correlated with a drop of ~20-25% in announced IPv4 address space, while a few others saw a drop closer to just 2%.

At an ASN level, the changes in announced address space were mixed – AS59588 (Zainas), AS199739 (Earthlink), and AS51684 (Asiacell) experienced a significant loss, while AS203214 (HulumTele) and AS58322 (Halasat) experienced little to no change.

Similar to Syria, we can also look at 1.1.1.1 resolver traffic data to better understand how the shutdowns are being implemented. The country-level graphs below suggest that UDP traffic patterns are not visibly changing, suggesting that responses from the resolver are, in fact, getting back to the clients. However, this likely isn’t the case, and such a conclusion is at least in part an artifact of the graph’s time frame and hourly granularity, as well as the inclusion of resolver traffic from Kurdish network providers (ASNs). The shutdowns are more clearly evident in the DNS-over-TCP and DNS-over-HTTPS graphs below, as well as in the graph for HTTP(S) request traffic (both mobile & desktop), which is also TCP-based. In these graphs, the troughs on days that shutdowns occurred generally dip lower than those on the days that the Internet remained available.

In looking at authoritative DNS traffic from Iraq during a shutdown (for June 13 as an example day, as above), we see evidence of a decline in traffic during the time the shutdown occurs.

The decline in authoritative DNS traffic is more evident at an ASN level, such as in the graph below for AS203214 (Hulum), effectively confirming that UDP traffic is not getting through here either.

Considering the traffic, 1.1.1.1 Resolver, and authoritative DNS observations reviewed here, it suggests that the Internet shutdowns taking place in Iraq are more complex than Syria’s, as it appears that both UDP and TCP traffic are unable to egress from impacted network providers. As not all impacted network providers are showing a complete loss of announced IP address space during the shutdowns, Iraq is taking a different approach to disrupting Internet connectivity. Although analysis of our data doesn’t provide a definitive conclusion, there are several likely options, and network providers in the country may be combining several. These options revolve around:

  1. IP: Block packets from reaching IP addresses. This may be done by withdrawing prefix announcements from the routing table (a brute force approach) or by blocking access to specific IP addresses, such as those associated with a specific application or service (a more surgical approach).
  2. Connection: Block connections based on SNI/HTTP headers, or other application data. If a network or on-path device is able to observe the server name (or other relevant headers/data), then the connection can be terminated.
  3. DNS: Operators of private or ‘internal’ DNS resolvers, offered by ISPs and enterprise environments for use by their own users, can apply content restrictions, blocking the resolution of hostnames associated with websites and other applications.

The consequences of these options are covered in more detail in a blog post. In addition, applying them at common network chokepoints, such as AS212330 (IRAQIXP) or AS208293 (AlSalam State Company, associated with the Iraqi Ministry of Communications), can disrupt connectivity at multiple downstream ISPs, without those providers necessarily having to take action themselves.

Algeria

As we noted in blog posts in 2022 and 2023, Algeria has a history of disrupting Internet connectivity during Baccalaureate exams. This has been taking place since 2018, following widespread cheating in 2016 that saw questions leaked online both before and during tests. On March 13, the Algerian Ministry of Education announced that the Baccalaureate exams would be held June 9-13. As expected, Internet disruptions were observed both country-wide and at a network level. Similar to previous years, two disruptions were observed each day. The first one began at 08:00 local time (07:00 UTC), and except for June 9, lasted three hours, ending at 11:00 local time (10:00 UTC). (On June 9, it lasted until 13:00 local time (12:00 UTC).) The second one began between 14:00-14:30 local time (13:00-13:30 UTC), and lasted until 16:00-17:00 local time (15:00-16:00 UTC) – the end time varied by day.

As seen in the graphs below, the impact to traffic was fairly nominal, suggesting that wide scale Internet shutdowns similar to those seen in Syria were not being implemented. While this is in line with 2023’s pronouncement by the Minister of Education that there would be no Internet shutdown on exam days, a number of posts on X complained of broader cuts to Internet connectivity.

Similar to the analysis above of the shutdowns in Syria and Iraq, we can also review changes to announced IP address space to better understand how connectivity was being disrupted. In this case, as the graphs below show, no meaningful changes to announced IPv4 address space were observed during the days the Baccalaureate exams were given. As such, the observed drops in traffic were not caused by routing changes.

In the HTTP(S) request traffic graph below, the twice-daily disruptions are highlighted, with the morning one appearing as a nominal drop in traffic, and the afternoon one causing a more severe decline. (The graph shows request traffic aggregated at a country level, but the graphs for the ASNs listed above also show similar patterns.)

In addition, similar patterns are observed in 1.1.1.1 resolver traffic at a country and ASN level, but only for DNS over TCP, DNS over TLS, and DNS over HTTPS, all of which leverage TCP. In the graph below showing only resolver traffic over UDP, there’s no clear evidence of disruptions. However, in the graph that shows resolver traffic over HTTPS, TCP, and TLS, a slight perturbation is visible in the morning, as traffic begins to rise for the day, and a sharper decrease is visible in the afternoon, with both disruptions aligning with the twice daily drops in traffic discussed above.

These observations support the conjecture that the Algerian government is likely taking a more nuanced approach to restricting access to content, interfering in some fashion with TCP-based traffic. The conjecture is also supported by an internal tool that helps to understand connection tampering that is based on research co-designed and developed by members of the Cloudflare Research team. We will be launching insights into TCP connection tampering on Cloudflare Radar later in 2024 and, in the meantime, technical details can be found in the peer-reviewed paper titled Global, Passive Detection of Connection Tampering.

The graph below, taken from the internal tool, highlights observed TCP connection tampering in connections from Algeria during the week that the Baccalaureate exams took place. While some baseline level of post-ACK and post-PSH tampering is consistently visible, we see significant increases in post-ACK twice a day during the exam period, at the times that align with the shifts in traffic discussed above. Technical descriptions of post-ACK and post-PSH tampering can be found in the Cloudflare Radar glossary, but in short, tampering post-ACK means an established TCP connection to Cloudflare’s server has been abruptly ended by one or more RST packets before the server sees data packets. Although clients do use RSTs, clients are more likely to close connections with a FIN (as specified by the RFC). The RST method can also be used by middleboxes that  (i) sees the data packet, then (ii) drops the data packet, then (iii) sends an RST to the server to force the server to close the connection (and very likely another RST to the client too for the same reason). Tampering post-PSH means that something on the path, like a middlebox, (i) saw something it didn’t like on an established connection, then (ii) permitted the data to pass but then, (iii) it sends the RST to force endpoints to close the connection.

Looking beyond Cloudflare-sourced data, aggregated test results from the Open Observatory of Network Interference (OONI) also show evidence of anomalous behavior. Using OONI Probe, a mobile and desktop app, can probe for potential blocking of websites, instant messaging apps, and censorship circumvention tools. Examining test results from users in Algeria for popular messaging platforms WhatsApp, Telegram, Signal, and Facebook Messenger for the first two weeks of June, we clearly see the appearance of test results marked as “Anomaly” starting on June 9. (OONI defines “Anomaly” results as “Measurements that provided signs of potential blocking”.) OONI Tor test results also show a similar “Anomaly” pattern. Anomalous traffic patterns are also visible for Google Web Search, YouTube, and GMail.

Although the analysis of these observations and data sets doesn’t provide us with specific details around exactly how the observed Internet disruptions are being implemented, it strongly supports the supposition that network providers in Algeria are, in some fashion, interfering with TCP connections, but not blocking them outright nor shutting down their networks completely. Given that popular messaging platforms, Google properties, Cloudflare’s 1.1.1.1 DNS resolver, and some number of Cloudflare customer sites all appear to be impacted, it suggests that a list of hostnames are being targeted for disruption/interference, either by the SNI or the destination IP address.

Conclusion

Perhaps recognizing the broad negative impact that brute-force nationwide Internet shutdowns have as a response to cheating on exams, some governments appear to be turning to more nuanced techniques, such as content blocking or connection tampering. However, because these are widely applied as well, they are arguably just as disruptive as a full nationwide Internet shutdown. The cause of full shutdowns, such as those seen in Syria, are arguably easier to diagnose than the disruptions to connectivity seen in Iraq and Algeria, which appear to use approaches that are hard to specifically identify from the outside.

Visit Cloudflare Radar for additional insights around these, and other, Internet disruptions. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.