Cloudflare’s view of the Rogers Communications outage in Canada

Post Syndicated from João Tomé original https://blog.cloudflare.com/cloudflares-view-of-the-rogers-communications-outage-in-canada/

Cloudflare’s view of the Rogers Communications outage in Canada

Cloudflare’s view of the Rogers Communications outage in Canada

An outage at one of the largest ISPs in Canada, Rogers Communications, started earlier today, July 8, 2022, and is ongoing (eight hours and counting), and is impacting businesses and consumers. At the time of writing, we are seeing a very small amount of traffic from Rogers, but we are only seeing residual traffic, and nothing close to a full recovery to normal traffic levels.

Based on what we’re seeing and similar incidents in the past, we believe this is likely to be an internal error, not a cyber attack.

Cloudflare Radar shows a near complete loss of traffic from Roger’s ASN, AS812, that started around 08:45 UTC (all times in this blog are UTC).

Cloudflare’s view of the Rogers Communications outage in Canada

What happened?

Cloudflare data shows that there was a clear spike in BGP (Border Gateway Protocol) updates after 08:15, reaching its peak at 08:45.

Cloudflare’s view of the Rogers Communications outage in Canada

BGP is a mechanism to exchange routing information between networks on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver each network packet to its final destination. Without BGP, the Internet routers wouldn’t know what to do, and the Internet wouldn’t exist.

The Internet is literally a network of networks, or for the maths fans, a graph, with each individual network a node in it, and the edges representing the interconnections. All of this is bound together by BGP. BGP allows one network (say Rogers) to advertise its presence to other networks that form the Internet. Rogers is not advertising its presence, so other networks can’t find Roger’s network and so it is unavailable.

A BGP update message informs a router of changes made to a prefix (a group of IP addresses) advertisement or entirely withdraws the prefix. In this next chart, we can see that at 08:45 there was a withdrawal of prefixes from Roger’s ASN.

Cloudflare’s view of the Rogers Communications outage in Canada

Since then, at 14:30, attempts seem to be made to advertise their prefixes again. This maps to us seeing a slow increase in traffic again from Rogers’ end users.

Cloudflare’s view of the Rogers Communications outage in Canada

The graph below, which shows the prefixes we were receiving from Rogers in Toronto, clearly shows the withdrawal of prefixes around 08:45, and the slow start in recovery at 14:30, with another round of withdraws at around 15:45.

Cloudflare’s view of the Rogers Communications outage in Canada

Outages happen more regularly than people think. This week we did an Internet disruptions overview for Q2 2022 where you can get a better sense of that, and on how collaborative and interconnected the Internet (the network of networks) is. And not so long ago Facebook had an hours long outage where BGP updates showed Facebook itself disappearing from the Internet.

Follow @CloudflareRadar on Twitter for updates on Internet disruptions as they occur, and find up-to-date information on Internet trends using Cloudflare Radar.