All posts by David Belson

Exam time means Internet disruptions in Syria, Sudan and Algeria

Post Syndicated from David Belson original https://blog.cloudflare.com/syria-sudan-algeria-exam-internet-shutdown/

Exam time means Internet disruptions in Syria, Sudan and Algeria

Exam time means Internet disruptions in Syria, Sudan and Algeria

It is once again exam time in Syria, Sudan, and Algeria, and with it, we find these countries disrupting Internet connectivity in an effort to prevent cheating on these exams. As they have done over the past several years, Syria and Sudan are implementing multi-hour nationwide Internet shutdowns. Algeria has also taken a similar approach in the past, but this year appears to be implementing more targeted website/application blocking.

Syria

Syria has been implementing Internet shutdowns across the country since 2011, but exam-related shutdowns have only been in place since 2016. In 2021, exams took place between May 31 and June 22, with multi-hour shutdowns observed on each of the exam days.

This year, the first shutdown was observed on May 30, with subsequent shutdowns (to date) seen on June 2, 6, and 12. In the Cloudflare Radar graph below, traffic for Syria drops to zero while the shutdowns are active. According to Internet Society Pulse, several additional shutdowns are expected through June 21. Each takes place between 02000530 UTC (0500–0830 local time). According to a published report, the current exam cycle covers more than 500,000 students for basic and general secondary education certificates.

Exam time means Internet disruptions in Syria, Sudan and Algeria

Consistent with shutdowns observed in prior years, Syria is once again implementing them in an asymmetric fashion – that is, inbound traffic is disabled, but egress traffic remains. This is clearly visible in request traffic from Syria to Cloudflare’s 1.1.1.1 DNS resolver. As the graph below shows, queries from clients in Syria are able to exit the country and reach Cloudflare, but responses can’t return, leading to retry floods, visible as spikes in the graph.

Exam time means Internet disruptions in Syria, Sudan and Algeria

Last year, the Syrian Minister of Education noted that, for the first time, encryption and surveillance technologies would be used in an effort to curtail cheating, with an apparent promise to suspend Internet shutdowns in the future if these technologies proved successful.

Sudan

Sudan is also no stranger to nationwide Internet shutdowns, with some last lasting for multiple weeks. Over the last several years, Sudan has also implemented Internet shutdowns during secondary school exams in an effort to limit cheating or leaking of exam questions. (We covered the 2021 round of shutdowns in a blog post.)

According to a schedule published by digital rights organization AccessNow, this year’s Secondary Certificate Exams will be taking place in Sudan daily between June 11–22, except June 17. As of this writing, near-complete shutdowns have been observed on June 11, 12, and 13 between 0530-0830 UTC (0730-1030 local time), as seen in the graph below. The timing of these shutdowns aligns with a communication reportedly sent to subscribers of telecommunications services in the country, which stated “In implementation of the decision of the Attorney General, the Internet service will be suspended during the Sudanese certificate exam sessions from 8 in the morning until 11 in the morning.”

Exam time means Internet disruptions in Syria, Sudan and Algeria

It is interesting to note that the shutdown, while nationwide, does not appear to be complete. The graph below shows that Cloudflare continues to see a small volume of HTTP requests from Sudatel during the shutdown periods. This is not completely unusual, as Sudatel may have public sector, financial services, or other types of customers that remain online.

Exam time means Internet disruptions in Syria, Sudan and Algeria

Algeria

Since 2018, Algeria has been shutting down the Internet nationwide during baccalaureate exams, following widespread cheating in 2016 that saw questions leaked online both before and during tests. These shutdowns reportedly cost businesses across the country an estimated 500 million Algerian Dinars (approximately $3.4 million USD) for every hour the Internet was unavailable. In 2021, there were two Internet shutdowns each day that exams took place—the first between 0700–1100 UTC (0800–1200 local time), and the second between 1330–1600 UTC (1430–1700 local time).

This year, more than 700,000 students will sit for the baccalaureate exams between June 12-16.

Perhaps recognizing the economic damage caused by these Internet shutdowns, this year the Algerian Minister of National Education announced that there would be no Internet shutdowns on exam days.

Thus far, it appears that this has been the case. However, it appears that the Algerian government has shifted to a content blocking-based approach, instead of a wide-scale Internet shutdown. The Cloudflare Radar graph below shows two nominal drops in country-level traffic during the two times on June 13 that the exams took place—0730–1000 UTC (0830–1100 local time) and 1330–1600 UTC (1430–1700 local time), similar to last year’s timing.

Exam time means Internet disruptions in Syria, Sudan and Algeria

The disruptions are also visible in traffic graphs for several major Algerian network providers, as shown below.

Exam time means Internet disruptions in Syria, Sudan and Algeria
Exam time means Internet disruptions in Syria, Sudan and Algeria
Exam time means Internet disruptions in Syria, Sudan and Algeria

Analysis of additional Cloudflare data further supports the hypothesis that Algeria is blocking access to specific websites and applications, rather than shutting down the Internet completely.

As described in a previous blog post, Network Error Logging (NEL) is a browser-based reporting system that allows users’ browsers to report connection failures to an endpoint specified by the webpage that failed to load. Below, a graph of NEL reports from browsers in Algeria shows clear spikes during the times (thus far) that the exams have taken place, with report levels significantly lower and more consistent during other times of the day.

Exam time means Internet disruptions in Syria, Sudan and Algeria

Conclusion

In addition to Syria, Sudan, and Algeria, countries including India, Jordan, Iraq, Uzbekistan, and Ethiopia have shut down or limited access to the Internet as exams took place. It is unclear whether these brute-force methods are truly effective at preventing cheating on these exams. However, it is clear that the impact of these shutdowns goes beyond students, as they impose a significant financial cost on businesses within the affected countries as they lose Internet access for multiple hours a day over the course of several weeks.

If you want to follow the remaining scheduled disruptions for these countries, you can see live data on the Cloudflare Radar pages for Syria, Sudan, and Algeria.

AAE-1 & SMW5 cable cuts impact millions of users across multiple countries

Post Syndicated from David Belson original https://blog.cloudflare.com/aae-1-smw5-cable-cuts/

AAE-1 & SMW5 cable cuts impact millions of users across multiple countries

AAE-1 & SMW5 cable cuts impact millions of users across multiple countries

Just after 1200 UTC on Tuesday, June 7, the Africa-Asia-Europe-1 (AAE-1) and SEA-ME-WE-5 (SMW-5) submarine cables suffered cable cuts. The damage reportedly occurred in Egypt, and impacted Internet connectivity for millions of Internet users across multiple countries in the Middle East and Africa, as well as thousands of miles away in Asia. In addition, Google Cloud Platform and OVHcloud reported connectivity issues due to these cable cuts.

The impact

Data from Cloudflare Radar showed significant drops in traffic across the impacted countries as the cable damage occurred, recovering approximately four hours later as the cables were repaired.

AAE-1 & SMW5 cable cuts impact millions of users across multiple countries
AAE-1 & SMW5 cable cuts impact millions of users across multiple countries
AAE-1 & SMW5 cable cuts impact millions of users across multiple countries
AAE-1 & SMW5 cable cuts impact millions of users across multiple countries
AAE-1 & SMW5 cable cuts impact millions of users across multiple countries
AAE-1 & SMW5 cable cuts impact millions of users across multiple countries
AAE-1 & SMW5 cable cuts impact millions of users across multiple countries

It appears that Saudi Arabia may have also been affected by the cable cut(s), but the impact was much less significant, and traffic recovered almost immediately.

AAE-1 & SMW5 cable cuts impact millions of users across multiple countries

In the graphs above, we show that Ethiopia was one of the impacted countries. However, as it is landlocked, there are obviously no submarine cable landing points within the country. The Afterfibre map from the Network Startup Resource Center (NSRC) shows that that fiber in Ethiopia connects to fiber in Somalia, which experienced an impact. In addition, Ethio Telecom also routes traffic through network providers in Kenya and Djibouti. Djibouti Telecom, one of these providers, in turn peers with larger global providers like Telecom Italia (TI) Sparkle, which is one of the owners of SMW5.

In addition to impacting end-user connectivity in the impacted countries, the cable cuts also reportedly impacted cloud providers including Google Cloud Platform and OVHcloud. In their incident report, Google Cloud noted “Google Cloud Networking experienced increased packet loss for egress traffic from Google to the Middle East, and elevated latency between our Europe and Asia Regions as a result, for 3 hours and 12 minutes, affecting several related products including Cloud NAT, Hybrid Connectivity and Virtual Private Cloud (VPC). From preliminary analysis, the root cause of the issue was a capacity shortage following two simultaneous fiber-cuts.” OVHcloud noted that “Backbone links between Marseille and Singapore are currently down” and that “Upon further investigation, our Network OPERATION teams advised that the fault was related to our partner fiber cuts.”

When concurrent disruptions like those highlighted above are observed across multiple countries in one or more geographic areas, the culprit is often a submarine cable that connects the impacted countries to the global Internet. The impact of such cable cuts will vary across countries, largely due to the levels of redundancy that they may have in place. That is, are these countries solely dependent on an impacted cable for global Internet connectivity, or do they have redundant connectivity across other submarine or terrestrial cables? Additionally, the location of the country relative to the cable cut will also impact how connectivity in a given country may be affected. Due to these factors, we didn’t see a similar impact across all of the countries connected to the AAE-1 and SMW5 cables.

What happened?

Specific details are sparse, but as noted above, the cable damage reportedly occurred in Egypt – both of the impacted cables land in Abu Talat and Zafarana, which also serve as landing points for a number of other submarine cables. According to a 2021 article in Middle East Eye, “There are 10 cable landing stations on Egypt’s Mediterranean and Red Sea coastlines, and some 15 terrestrial crossing routes across the country.” Alan Mauldin, research director at telecommunications research firm TeleGeography, notes that routing cables between Europe and the Middle East to India is done via Egypt, because there is the least amount of land to cross. This places the country in a unique position as a choke point for international Internet connectivity, with damage to infrastructure locally impacting the ability of millions of people thousands of miles away to access websites and applications, as well as impacting connectivity for leading cloud platform providers.

As the graphs above show, traffic returned to normal levels within a matter of hours, with tweets from telecommunications authorities in Pakistan and Oman also noting that Internet services had returned to their countries. Such rapid repairs to submarine cable infrastructure are unusual, as repair timeframes are often measured in days or weeks, as we saw with the cables damaged by the volcanic eruption in Tonga earlier this year. This is due to the need to locate the fault, send repair ships to the appropriate location, and then retrieve the cable and repair it. Given this, the damage to these cables likely occurred on land, after they came ashore.

Keeping content available

By deploying in data centers close to end users, Cloudflare helps to keep traffic local, which can mitigate the impact of catastrophic events like cable cuts, while improving performance, availability, and security. Being able to deliver content from our network generally requires first retrieving it from an origin, and with end users around the world, Cloudflare needs to be able to reach origins from multiple points around the world at the same time. However, a customer origin may be reachable from some networks but not from others, due to a cable cut or some other network disruption.

In September 2021, Cloudflare announced Orpheus, which provides reachability benefits for customers by finding unreachable paths on the Internet in real time, and guiding traffic away from those paths, ensuring that Cloudflare will always be able to reach an origin no matter what is happening on the Internet.

Conclusion

Because the Internet is an interconnected network of networks, an event such as a cable cut can have a ripple effect across the whole Internet, impacting connectivity for users thousands of miles away from where the incident occurred. Users may be unable to access content or applications, or the content/applications may suffer from reduced performance. Additionally, the providers of those applications may experience problems within their own network infrastructure due to such an event.

For network providers, the impact of such events can be mitigated through the use of multiple upstream providers/peers, and diverse physical paths for critical infrastructure like submarine cables. Cloudflare’s globally deployed network can help content and application providers ensure that their content and applications remain available and performant in the face of network disruptions.

Internet disruptions overview for Q1 2022

Post Syndicated from David Belson original https://blog.cloudflare.com/q1-2022-internet-disruption-summary/

Internet disruptions overview for Q1 2022

Internet disruptions overview for Q1 2022

Cloudflare operates in more than 250 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. In many cases, these disruptions can be attributed to a physical event, while in other cases, they are due to an intentional government-directed shutdown. In this post, we review selected Internet disruptions observed by Cloudflare during the first quarter of 2022, supported by traffic graphs from Cloudflare Radar and other internal Cloudflare tools, and grouped by associated cause.

Plate tectonics

Internet outages caused by “earth movers” are more frequently caused by errant backhoes. However, two Internet disruptions in the first quarter were caused by more significant earth movement — a volcanic eruption and an earthquake.

The first impacted connectivity on the island nation of Tonga, when the Hunga Tonga–Hunga Ha’apai volcanic eruption damaged the submarine cable connecting Tonga to Fiji, resulting in a 38 day Internet outage. After the January 14 eruption, only minimal Internet traffic (via satellite services) was seen from Tonga. On February 22, Digicel announced that the main island was back online after initial submarine cable repairs were completed – the immediate return of traffic is clearly visible in the figure below. However, it was estimated that repairs to the domestic cable, connecting outlying islands, could take an additional six to nine months.

Internet disruptions overview for Q1 2022

The second disruption, caused by a 7.3 magnitude earthquake off the coast of central Japan on March 16, was significantly shorter, and had a significantly smaller impact. The earthquake occurred around 1436 UTC, causing power outages that resulted in a loss of Internet connectivity in cities including Tokyo for several hours, as seen in the figure below. Almost exactly 11 years prior, a magnitude 8.9 earthquake also had a nominal impact on Internet connectivity in Japan, that time apparently due to damage to subsea cable systems.

Internet disruptions overview for Q1 2022

Infrastructure damage

Internet resilience is, of course, heavily dependent on the resilience of the underlying physical infrastructure, including data centers, terrestrial fiber, and submarine cables. Damage to this infrastructure often disrupts Internet connectivity.

Early in the morning of January 5, the Gambia was completely isolated from the global Internet. As evident in the figure below, the incident lasted over eight hours, between 0117 and 0945 UTC. According to a press release from GAMTEL, after the failure of the primary link (damage to the ACE submarine cable), traffic was routed onto two backup links through Senegal. However, these backup links also failed because they converged in a location that was ultimately identified as a single point of failure.

Internet disruptions overview for Q1 2022

Around 2130 UTC on January 20, Internet traffic to Yemen dropped to near zero, as shown in the figure below, after ongoing airstrikes reportedly hit a telecommunications building in Al-Hudaydah where the FALCON undersea cable lands. The outage lasted four days, finally recovering around 2100 on January 24. The outage primarily affected YemenNet (Public Telecommunication Corporation), the state-owned telecommunications provider.

Internet disruptions overview for Q1 2022

On March 1, Tasmania suffered a 6.5-hour Internet outage after two of the three submarine cables (Basslink, Bass Strait-1, Bass Strait-2) connecting it to the Australian mainland were cut.

According to a published report, one of the cuts was on the Victorian (mainland) end, and the other on the Tasmanian side, with both cuts caused by “third parties”.  A significant reduction in traffic between 0130 – 0800 UTC is visible in the figure below.

Internet disruptions overview for Q1 2022

A reported Telecom Infrastructure Company (TIC) data center fire caused a four-hour Internet disruption in Iran on March 4. Telecom Infrastructure Company (TIC) is the monopoly provider of telecom infrastructure to all public and private operators in Iran. As the figure below shows, Internet traffic to the country dropped by approximately 20% at 0640 UTC, and recovered around 1030 UTC.

Internet disruptions overview for Q1 2022

On March 15, ETECSA, the Cuban state telecommunications company, reported that a fiber optic cable had been cut on a public road in the capital that morning. The impact of this fiber cut on Internet traffic to Cuba and ETECSA is visible in the figures below, starting just after 1200 UTC, lasting for over six hours.

Internet disruptions overview for Q1 2022
Internet disruptions overview for Q1 2022

Although initially believed to be the result of a power outage (all too common in Venezuela), a March 24 Internet disruption in the country was ultimately due to a fiber cut. Internet traffic to CANTV customers in multiple Venezuelan states dropped significantly between 1140 and 1740 UTC, as seen in the figure below. In addition to this disruption, VE sin Filtro reported a number of additional multi-hour, multi-state Internet disruptions in Venezuela during the first quarter.

Internet disruptions overview for Q1 2022

On March 31, Internet traffic to Telenor Pakistan dropped 60% between 0600-0745 UTC, as shown in the figure below. According to Telenor Pakistan responses to hundreds of customer complaints lodged via Twitter, the disruption was due to multiple fiber-optic cable cuts in several locations. Just after 1800 UTC, Telenor Pakistan Tweeted that services had been fully restored.

Internet disruptions overview for Q1 2022

Power outages

In addition to the physical infrastructure, reliable electrical power is also critical for resilient Internet connectivity. At a provider level, loss of power can take key data centers and routers offline, impacting connectivity for customers and other connected networks. Consumer power outages can take home/business routers and connected devices offline, forcing users onto mobile connectivity, assuming that is/remains available.

The interconnected electrical grids of Kazakhstan, Uzbekistan, and Kyrgyzstan all suffered outages on January 24 after Kazakhstan’s North-South power line was disconnected due to “emergency imbalances”. These power outages caused multi-hour Internet disruptions across all three countries starting around 0600 UTC, as the figures below show. The impact to traffic in Kazakhstan appeared to be fairly minor, while traffic declined significantly in Uzbekistan and Kyrgyzstan and took longer to recover.

Internet disruptions overview for Q1 2022
Internet disruptions overview for Q1 2022
Internet disruptions overview for Q1 2022

A power outage across multiple counties and cities in Taiwan starting around 0100 UTC on March 3 caused a brief #Internet disruption. The figure below shows a nominal initial drop in traffic, though traffic remained lower throughout the next several hours. The power outage was reportedly caused by human negligence during annual repairs of a generator at the Hsinta power plant.

Internet disruptions overview for Q1 2022

In addition to the fiber cut discussed above, Cuba’s Internet suffered a second disruption on March 24. A Tweet from ETECSA stated that a power failure had caused a disruption to voice service, SMS, and mobile data. Analysis of Internet traffic for both Cuba and ETECSA finds that the disruption started around 1230 UTC, and lasted for approximately 90 minutes, as shown in the figures below.

Internet disruptions overview for Q1 2022
Internet disruptions overview for Q1 2022

DDoS attack

Although distributed denial of service (DDoS) attacks often target web or application servers in an attempt to knock a given website or application offline, such attacks that target network infrastructure can have more widespread impact, not only restricting access to sites and applications hosted within that network, but also disrupting connectivity for users attached to the network.

Such a DDoS attack targeted AS8867 (E-Gov – Tehila Project) in Israel on March 14. The figure below shows that Internet traffic to that ASN began to decline just before 1530 UTC. A published report notes that the websites of the interior, health, justice and welfare ministries, as well as that of the Prime Minister’s office, were all taken offline as a result of the attack.

Internet disruptions overview for Q1 2022

Unspecified technical causes

As discussed above, the underlying technical or physical causes of Internet disruptions are often easily identified, frequently thanks to social media or other communications from the impacted network providers. However, sometimes disruptions are observed that are correlated with a real-world (often political) event with no specified technical or physical cause, while other times disruptions are observed but are both uncorrelated and unattributed.

In Kazakhstan, an Internet disruption began on January 5 amid mass protests against sudden increases in energy prices. Starting around 1030 UTC, traffic from Kazakhstan dropped to near zero. The figure below shows that traffic returned to a regular diurnal pattern on January 11, but several apparent restorations of connectivity are also visible during the six-day disruption. These brief periods of connectivity appeared to align with televised speeches or announcements from the Zazakh president.

Internet disruptions overview for Q1 2022

In Burkina Faso, heavy gunfire related to an army mutiny was reported early in the morning of January 23. A significant drop in traffic from the country was observed in Cloudflare Radar starting around 0915 UTC, with Orange, FasoNet, and Telecel Faso all seeing lower traffic volumes. As the figure below shows, the disruption lasted for nearly a day and a half, recovering around 2000 UTC on January 24.

Internet disruptions overview for Q1 2022

Just after 2200 UTC on March 15, Yemen experienced a significant, albeit brief, Internet disruption, lasting just 30 minutes. As the figures below show, the disruption was primarily due to an issue at YemenNet. A published report claims that the disruption was due to a deliberate act by the Houthi coup militia.

Internet disruptions overview for Q1 2022
Internet disruptions overview for Q1 2022

Russian invasion of Ukraine

The Russian invasion of Ukraine has now been going on for over a month. In some cases, Internet connectivity has been collateral damage from the kinetic military action, while in other cases, targeted attacks on network providers and power outages have disrupted connectivity. Technicians from Ukrainian service providers have been risking their lives to keep the country online, and have been largely successful – Cloudflare Radar traffic data for Ukraine shows that as of the end of March, peak traffic levels are at 85-90% of pre-invasion peaks. An earlier blog post provides additional details about Internet traffic patterns observed in Ukraine during the first week after the conflict began.

Below we highlight just a few significant disruptions observed on major Ukrainian network providers in March.

Two brief outages were observed at Ukrtelecom during the second week of March, shown in the figure below. The first, on March 8, lasted for just over two hours, while the second one, on March 10, lasted for approximately 40 minutes. No root cause has been reported for these disruptions.

Internet disruptions overview for Q1 2022

Later in the month, on March 28, Ukrtelecom experienced a ~15 hour outage, lasting from 0800 UTC to approximately 0100 on March 29, as seen in the figure below. A Twitter thread from the State Service of Special Communications and Information Protection of Ukraine explained that the outage was caused by “a powerful cyberattack” against Ukrtelecom’s infrastructure, and that “In order to preserve its network infrastructure and to continue providing services to Ukraine’s Armed Forces and other military formations as well as to the customers, Ukrtelecom has temporarily limited providing its services to the majority of private users and business-clients.A LinkedIn post from Ukrtelecom also highlights the non-stop work that the company has been doing to re-establish telecommunications services in impacted regions across the country.

Internet disruptions overview for Q1 2022

The figure below shows that around 2100 UTC on March 9, Ukrainian Internet service provider Triolan suffered a significant disruption, reportedly resulting from a cyber attack. Traffic began to gradually return after approximately 10 hours.

Internet disruptions overview for Q1 2022

Conclusion

Despite occasional connectivity disruptions, the Internet remains remarkably resilient. This resiliency is increasingly critical as the Internet finds its way into more and more areas of everyday life around the world. In addition to providing a suite of solutions that support that resiliency, we use the data exhaust from these solutions to monitor Internet reliability, availability, security, and performance.

Follow @CloudflareRadar on Twitter for updates on Internet disruptions as they occur, and find up-to-date information on Internet trends using Cloudflare Radar.

DNSSEC issues take Fiji domains offline

Post Syndicated from David Belson original https://blog.cloudflare.com/dnssec-issues-fiji/

DNSSEC issues take Fiji domains offline

DNSSEC issues take Fiji domains offline

On the morning of March 8, a post to Hacker News stated that “All .fj domains have gone offline”, listing several hostnames in domains within the Fiji top level domain (known as a ccTLD) that had become unreachable. Commenters in the associated discussion thread had mixed results in being able to reach .fj hostnames—some were successful, while others saw failures. The fijivillage news site also highlighted the problem, noting that the issue also impacted Vodafone’s M-PAiSA app/service, preventing users from completing financial transactions.

The impact of this issue can be seen in traffic to Cloudflare customer zones in the .com.fj second-level domain. The graph below shows that HTTP traffic to these zones dropped by approximately 40% almost immediately starting around midnight UTC on March 8. Traffic volumes continued to decline throughout the rest of the morning.

DNSSEC issues take Fiji domains offline

Looking at Cloudflare’s 1.1.1.1 resolver data for queries for .com.fj hostnames, we can also see that error volume associated with those queries climbs significantly starting just after midnight as well. This means that our resolvers encountered issues with the answers from .fj servers.

DNSSEC issues take Fiji domains offline

This observation suggests that the problem was strictly DNS related, rather than connectivity related—Cloudflare Radar does not show any indication of an Internet disruption in Fiji coincident with the start of this problem.

DNSSEC issues take Fiji domains offline

It was suggested within the Hacker News comments that the problem could be DNSSEC related. Upon further investigation, it appears that may be the cause. In verifying the DNSSEC record for the .fj ccTLD, shown in the dig output below, we see that it states EDE: 9 (DNSKEY Missing): 'no SEP matching the DS found for fj.'

kdig fj. soa +dnssec @1.1.1.1 
;; ->>HEADER<<- opcode: QUERY; status: SERVFAIL; id: 12710
;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 1
 
;; EDNS PSEUDOSECTION:
;; Version: 0; flags: do; UDP size: 1232 B; ext-rcode: NOERROR
;; EDE: 9 (DNSKEY Missing): 'no SEP matching the DS found for fj.'
 
;; QUESTION SECTION:
;; fj.                          IN      SOA
 
;; Received 73 B
;; Time 2022-03-08 08:57:41 EST
;; From 1.1.1.1@53(UDP) in 17.2 ms

Extended DNS Error 9 (EDE: 9) is defined as “A DS record existed at a parent, but no supported matching DNSKEY record could be found for the child.” The Cloudflare Learning Center article on DNSKEY and DS records explains this relationship:

The DS record is used to verify the authenticity of child zones of DNSSEC zones. The DS key record on a parent zone contains a hash of the KSK in a child zone. A DNSSEC resolver can therefore verify the authenticity of the child zone by hashing its KSK record, and comparing that to what is in the parent zone’s DS record.

Ultimately, it appears that around midnight UTC, the .fj zone started to be signed with a key that was not in the root zone DS, possibly as the result of a scheduled rollover that happened without checking that the root zone was updated first by IANA, which updates the root zone. (IANA owns contact with the TLD operators, and instructs the Root Zone Publisher on the changes to make in the next version of the root zone.)

DNSSEC problems as the root cause of the observed issue align with the observation in the Hacker News comments that some were able to access .fj websites, while others were not. Users behind resolvers doing strict DNSSEC validation would have seen an error in their browser, while users behind less strict resolvers would have been able to access the sites without a problem.

Conclusion

Further analysis of Cloudflare resolver metrics indicates that the problem was resolved around 1400 UTC, when the DS was updated. When DNSSEC is improperly configured for a single domain name, it can cause problems accessing websites or applications in that zone. However, when the misconfiguration occurs at a ccTLD level, the impact is much more significant. Unfortunately, this seems to occur all too often.

(Thank you to Ólafur Guðmundsson for his DNSSEC expertise.)

How The Gambia lost access to the Internet for more than 8 hours

Post Syndicated from David Belson original https://blog.cloudflare.com/the-gambia-without-internet/

How The Gambia lost access to the Internet for more than 8 hours

How The Gambia lost access to the Internet for more than 8 hours

Internet outages are more common than most people think, and may be caused by misconfigurations, power outages, extreme weather, or infrastructure damage. Note that such outages are distinct from state-imposed shutdowns that also happen all too frequently, generally used to deal with situations of unrest, elections or even exams.

On the morning of January 4, 2022, citizens of The Gambia woke up to a country-wide Internet outage. Gamtel (the main state-owned telecommunications company of the West Africa country), announced that it happened due to “technical issues on the backup links” — we elaborate more on this below.

Cloudflare Radar shows that the outage had a significant impact on Internet traffic in the country and started after 01:00 UTC (which is the same local time), lasting until ~09:45 — a disruption of over 8 hours.

How The Gambia lost access to the Internet for more than 8 hours

Looking at  BGP (Border Gateway Protocol) updates from Gambian ASNs around the time of the outage, we see a clear spike at 01:10 UTC. These update messages are BGP signaling that the Gambian ASNs are no longer routable.

How The Gambia lost access to the Internet for more than 8 hours

It is important to know that BGP is a mechanism to exchange routing information between autonomous systems (networks) on the Internet. The routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn’t know what to do, and the Internet wouldn’t work. As we saw in our blog post in 2021 about how Facebook disappeared from the Internet, the Internet is literally a network of networks, and it’s bound together by BGP.

The Gambia’s Internet access is solely dependent on a single provider, Gamtel. Because The Gambia’s international Internet connectivity via the ACE submarine cable was unavailable, it was reliant on the “backup links” referenced above – terrestrial connectivity via Senegal and the provider Sonatel. This is visible in BGP data. If we look at the ASNs that are allocated to networks in The Gambia (AS25250, AS37309, AS37503, AS37552, AS37524, AS37323, AS328488, AS328140), and put those into a regular expression on BGP routing tools like route-views as so:

route-views>show ip bgp regexp .*_(25250|37309|37503|37552|37524|37323|328488|328140)

We are able to see all the possible upstream ASN paths from these networks to the rest of the Internet.

Looking at the “Path” results, we see that AS8346 (Sonatel) and AS25250 (Gamtel) are in the path for all the Gambian networks.

How The Gambia lost access to the Internet for more than 8 hours

Visualized, you can see the dependency on this network path for The Gambia’s Internet access.

How The Gambia lost access to the Internet for more than 8 hours

No interruptions were seen in Sonatel (AS8346), so this indicates that the single network path between Sonatel and Gamtel (AS25250) is a critical point for connectivity. A failure in either of these networks could result in The Gambia going offline again.

Yesterday’s outage in The Gambia outage illustrates something we frequently reference here in the blog: the Internet is literally a network of networks. A significant amount of  Internet traffic is carried by a complex network of undersea fiber-optic cables that connect countries and continents — all the cable systems used have landing points in two or more countries. So a problem in one country can easily affect others.

Going back to The Gambia, Gamtel explained in a January 5, 2022, press release that there was “a primary link failure at ACE” — the cable system that serves 24 countries, from Europe to Africa. “The ACE cable repair is expected to be completed in mid-January, 2022,” explained the company.

How The Gambia lost access to the Internet for more than 8 hours
The full ACE (Africa Coast to Europe) submarine cable system. From NSRC

The “backup failure” here was “due to a faulty card at Toubakota, in Senegal”. That problem affects “both the Karang and Seleti links [points of cable connections from Senegal to The Gambia] as both North and South links converges there”. “Thus, the reason for the complete isolation on the Sonatel link”, concludes Gamtel.

Recognizing the critical importance of reliable Internet connectivity, The Gambia Public Utilities Regulatory Authority also issued a statement noting “The Authority, operators, MOICI, and the Government are exploring other options of making sure that the Gambia has a second fibre cable backup considering the impact that these failures are having on our national security, economy, and social activities.”