All posts by Celso Martinho

What happened on the Internet during the Facebook outage

Post Syndicated from Celso Martinho original https://blog.cloudflare.com/during-the-facebook-outage/

What happened on the Internet during the Facebook outage

It’s been a few days now since Facebook, Instagram, and WhatsApp went AWOL and experienced one of the most extended and rough downtime periods in their existence.

When that happened, we reported our bird’s-eye view of the event and posted the blog Understanding How Facebook Disappeared from the Internet where we tried to explain what we saw and how DNS and BGP, two of the technologies at the center of the outage, played a role in the event.

In the meantime, more information has surfaced, and Facebook has published a blog post giving more details of what happened internally.

As we said before, these events are a gentle reminder that the Internet is a vast network of networks, and we, as industry players and end-users, are part of it and should work together.

In the aftermath of an event of this size, we don’t waste much time debating how peers handled the situation. We do, however, ask ourselves the more important questions: “How did this affect us?” and “What if this had happened to us?” Asking and answering these questions whenever something like this happens is a great and healthy exercise that helps us improve our own resilience.

Today, we’re going to show you how the Facebook and affiliate sites downtime affected us, and what we can see in our data.

1.1.1.1

1.1.1.1 is a fast and privacy-centric public DNS resolver operated by Cloudflare, used by millions of users, browsers, and devices worldwide. Let’s look at our telemetry and see what we find.

First, the obvious. If we look at the response rate, there was a massive spike in the number of SERVFAIL codes. SERVFAILs can happen for several reasons; we have an excellent blog called Unwrap the SERVFAIL that you should read if you’re curious.

In this case, we started serving SERVFAIL responses to all facebook.com and whatsapp.com DNS queries because our resolver couldn’t access the upstream Facebook authoritative servers. About 60x times more than the average on a typical day.

What happened on the Internet during the Facebook outage

If we look at all the queries, not specific to Facebook or WhatsApp domains, and we split them by IPv4 and IPv6 clients, we can see that our load increased too.

As explained before, this is due to a snowball effect associated with applications and users retrying after the errors and generating even more traffic. In this case, 1.1.1.1 had to handle more than the expected rate for A and AAAA queries.

What happened on the Internet during the Facebook outage

Here’s another fun one.

DNS vs. DoT and DoH. Typically, DNS queries and responses are sent in plaintext over UDP (or TCP sometimes), and that’s been the case for decades now. Naturally, this poses security and privacy risks to end-users as it allows in-transit attacks or traffic snooping.

With DNS over TLS (DoT) and DNS over HTTPS, clients can talk DNS using well-known, well-supported encryption and authentication protocols.

Our learning center has a good article on “DNS over TLS vs. DNS over HTTPS” that you can read. Browsers like Chrome, Firefox, and Edge have supported DoH for some time now, WAP uses DoH too, and you can even configure your operating system to use the new protocols.

When Facebook went offline, we saw the number of DoT+DoH SERVFAILs responses grow by over x300 vs. the average rate.

What happened on the Internet during the Facebook outage
What happened on the Internet during the Facebook outage
What happened on the Internet during the Facebook outage

So, we got hammered with lots of requests and errors, causing traffic spikes to our 1.1.1.1 resolver and causing an unexpected load in the edge network and systems. How did we perform during this stressful period?

Quite well. 1.1.1.1 kept its cool and continued serving the vast majority of requests around the famous 10ms mark. An insignificant fraction of p95 and p99 percentiles saw increased response times, probably due to timeouts trying to reach Facebook’s nameservers.

What happened on the Internet during the Facebook outage

Another interesting perspective is the distribution of the ratio between SERVFAIL and good DNS answers, by country. In theory, the higher this ratio is, the more the country uses Facebook. Here’s the map with the countries that suffered the most:

What happened on the Internet during the Facebook outage

Here’s the top twelve country list, ordered by those that apparently use Facebook, WhatsApp and Instagram the most:

Country SERVFAIL/Good Answers ratio
Turkey 7.34
Grenada 4.84
Congo 4.44
Lesotho 3.94
Nicaragua 3.57
South Sudan 3.47
Syrian Arab Republic 3.41
Serbia 3.25
Turkmenistan 3.23
United Arab Emirates 3.17
Togo 3.14
French Guiana 3.00

Impact on other sites

When Facebook, Instagram, and WhatsApp aren’t around, the world turns to other places to look for information on what’s going on, other forms of entertainment or other applications to communicate with their friends and family. Our data shows us those shifts. While Facebook was going down, other services and platforms were going up.

To get an idea of the changing traffic patterns we look at DNS queries as an indicator of increased traffic to specific sites or types of site.

Here are a few examples.

Other social media platforms saw a slight increase in use, compared to normal.

What happened on the Internet during the Facebook outage

Traffic to messaging platforms like Telegram, Signal, Discord and Slack got a little push too.

What happened on the Internet during the Facebook outage

Nothing like a little gaming time when Instagram is down, we guess, when looking at traffic to sites like Steam, Xbox, Minecraft and others.

What happened on the Internet during the Facebook outage

And yes, people want to know what’s going on and fall back on news sites like CNN, New York Times, The Guardian, Wall Street Journal, Washington Post, Huffington Post, BBC, and others:

What happened on the Internet during the Facebook outage

Attacks

One could speculate that the Internet was under attack from malicious hackers. Our Firewall doesn’t agree; nothing out of the ordinary stands out.

What happened on the Internet during the Facebook outage

Network Error Logs

Network Error Logging, NEL for short, is an experimental technology supported in Chrome. A website can issue a Report-To header and ask the browser to send reports about network problems, like bad requests or DNS issues, to a specific endpoint.

Cloudflare uses NEL data to quickly help triage end-user connectivity issues when end-users reach our network. You can learn more about this feature in our help center.

If Facebook is down and their DNS isn’t responding, Chrome will start reporting NEL events every time one of the pages in our zones fails to load Facebook comments, posts, ads, or authentication buttons. This chart shows it clearly.​​

What happened on the Internet during the Facebook outage

WARP

Cloudflare announced WARP in 2019, and called it “A VPN for People Who Don’t Know What V.P.N. Stands For” and offered it for free to its customers. Today WARP is used by millions of people worldwide to securely and privately access the Internet on their desktop and mobile devices. Here’s what we saw during the outage by looking at traffic volume between WARP and Facebook’s network:

What happened on the Internet during the Facebook outage

You can see how the steep drop in Facebook ASN traffic coincides with the start of the incident and how it compares to the same period the day before.

Our own traffic

People tend to think of Facebook as a place to visit. We log in, and we access Facebook, we post. It turns out that Facebook likes to visit us too, quite a lot. Like Google and other platforms, Facebook uses an army of crawlers to constantly check websites for data and updates. Those robots gather information about websites content, such as its titles, descriptions, thumbnail images, and metadata. You can learn more about this on the “The Facebook Crawler” page and the Open Graph website.

Here’s what we see when traffic is coming from the Facebook ASN, supposedly from crawlers, to our CDN sites:

What happened on the Internet during the Facebook outage

The robots went silent.

What about the traffic coming to our CDN sites from Facebook User-Agents? The gap is indisputable.

What happened on the Internet during the Facebook outage

We see about 30% of a typical request rate hitting us. But it’s not zero; why is that?

We’ll let you know a little secret. Never trust User-Agent information; it’s broken. User-Agent spoofing is everywhere. Browsers, apps, and other clients deliberately change the User-Agent string when they fetch pages from the Internet to hide, obtain access to certain features, or bypass paywalls (because pay-walled sites want sites like Facebook to index their content, so that then they get more traffic from links).

Fortunately, there are newer, and privacy-centric standards emerging like User-Agent Client Hints.

Core Web Vitals

Core Web Vitals are the subset of Web Vitals, an initiative by Google to provide a unified interface to measure real-world quality signals when a user visits a web page. Such signals include Largest Contentful Paint (LCP), First Input Delay (FID), and Cumulative Layout Shift (CLS).

We use Core Web Vitals with our privacy-centric Web Analytics product and collect anonymized data on how end-users experience the websites that enable this feature.

One of the metrics we can calculate using these signals is the page load time. Our theory is that if a page includes scripts coming from external sites (for example, Facebook “like” buttons, comments, ads), and they are unreachable, its total load time gets affected.

We used a list of about 400 domains that we know embed Facebook scripts in their pages and looked at the data.

What happened on the Internet during the Facebook outage

Now let’s look at the Largest Contentful Paint. LCP marks the point in the page load timeline when the page’s main content has likely loaded. The faster the LCP is, the better the end-user experience.

What happened on the Internet during the Facebook outage

Again, the page load experience got visibly degraded.

The outcome seems clear. The sites that use Facebook scripts in their pages took 1.5x more time to load their pages during the outage, with some of them taking more than 2x the usual time. Facebook’s outage dragged the performance of  some other sites down.

Conclusion

When Facebook, Instagram, and WhatsApp went down, the Web felt it. Some websites got slower or lost traffic, other services and platforms got unexpected load, and people lost the ability to communicate or do business normally.

Uganda’s January 13, 2021 Internet Shut Down

Post Syndicated from Celso Martinho original https://blog.cloudflare.com/uganda-january-13-2021-internet-shut-down/

Uganda's January 13, 2021 Internet Shut Down

Two days ago, through its communications regulator, Uganda’s government ordered the “Suspension Of The Operation Of Internet Gateways” the day before the country’s general election. This action was confirmed by several users and journalists who got access to the letter sent to Internet providers. In other words, the government effectively cut off Internet access from the population to the rest of the world.

On Cloudflare Radar, we want to help anyone understand what happens on the Internet. We are continually monitoring our network and exposing insights, threats, and trends based on the aggregated data that we see.

Uganda’s unusual traffic patterns quickly popped up in our charts. Our 7-day change in Internet Traffic chart in Uganda shows a clear drop to near zero starting around 1900 local time, when the providers received the letter.

Uganda's January 13, 2021 Internet Shut Down

This is also obvious in the Application-level Attacks chart.

Uganda's January 13, 2021 Internet Shut Down

The traffic drop was also confirmed by the Uganda Internet eXchange point, a place where many providers exchange their data traffic, on their public statistics page.

Uganda's January 13, 2021 Internet Shut Down

We keep an eye on traffic levels and BGP routing to our edge network, and are able to see which networks carry traffic to and from Uganda and their relative traffic levels. The cutoff is clear in those statistics also. Each colored line is a different network inside Uganda (such as ISPs, mobile providers, etc.)

Uganda's January 13, 2021 Internet Shut Down

We will continue to keep an eye on traffic levels from Uganda and update the blog when we see significant changes. At the time of writing, Internet access appears to be still cut off.

Holiday Season Update from Lisbon

Post Syndicated from Celso Martinho original https://blog.cloudflare.com/holiday-season-update-from-lisbon/

Holiday Season Update from Lisbon

Holiday Season Update from Lisbon

It’s the end of the year, so we thought it would be a great time to give you an update on how we’re doing and what we’re planning for 2021. If you’re reading this, you know we like to share everything we do at Cloudflare, including how the organization is evolving.

In July, John Graham-Cumming wrote a blog post entitled Cloudflare’s first year in Lisbon. and showed how we went from an announcement, just a few months before, to an entirely bootstrapped and fully functional office. At the time, despite a ramping pandemic, the team was already hard at work doing a fantastic job scaling up and solidifying our presence here.

A few weeks later, in August, I proudly joined the team.

The first weeks

Cloudflare is, by any standard, a big company. There’s a lot you need to learn, many people you need to get to know first, and a lot of setup steps you need to get through before you’re in a position to do actual real productive work.

Joining the company during COVID was challenging. I felt just as excited as I was scared. We were (and still are) fully working from home, I didn’t have a team to work with in person. A setup like this surely looks daunting, even for experienced people.

But here’s the thing. Cloudflare isn’t just any company. We’re unparalleled because we masterfully combine scale, ambition, talent, product, vision, values, and culture in a way that’s very difficult to replicate and maintain at any other company.

We’re big, but we move fast. We’re over 1,600 working together, but it feels like a cohesive group. We’re distributed across multiple offices and continents, often working in teams with members from different time zones, but we don’t notice it. We have tools, documentation, and methodologies, but they don’t get in the way of our “shipping products” mantra. There are product owners, teams for specific features, but we all hold ownership for everyone’s work.

I felt all of this right after my orientation week. The warm welcome, the regular check-ins to say hello and see how I was doing, and everyone’s urge to make sure I was adjusting and getting all the help I needed, giving me advice, introducing me to other colleagues. Cloudflarians take genuine pride in making sure everyone feels at home. You can learn more about this experience from a Story Time segment John did with me.

Where do we stand

Cloudflare Lisbon has come a long way. We now have 74 incredibly talented people working or joining in areas such as Engineering, Security, Infrastructure, Customer Support, People, Places, Product Management, Emerging Technologies or Accounting, and growing fast.

Although the pandemic didn’t help our plans, especially those related to growing and physically working in our brand new office on Praça Marquês de Pombal, it didn’t slow us down either. November and December alone, 15 people joined the team. We’re gaining momentum.

More interestingly, we have a super diverse team in Lisbon, and we couldn’t be prouder of it. We’re putting action ahead of words and actively contributing to create more opportunities for women in technology and to attract people to work in Portugal regardless of their country of origin.

Our discussions on whether “Pastéis de Nata” is best served with or without cinnamon, our holiday traditions, Portuguese music, coffee, our frequent virtual Pub Quizzes, escape room events, and of course, the comments on shirtless Marcelo are now routine. They are evidence that we feel like a group working together, having fun while growing.

Returning to Portugal?

We live in unusual and contemplative times. Many of our emigrants living outside the country are considering returning home to Portugal and our office in Lisbon is proof of this growing movement. Portuguese returnees represent roughly 10% of our team.

The Portuguese Government has an initiative called “Programa Regressar,” where they provide tax benefits and financial assistance to support emigrants and their families returning to Portugal.

While this is great, we think it’s not enough. Moving you and your family to another country is a life-changing event. Although things like patriotism, cost of living, and tax incentives play an essential role in the personal decision process, skilled and talented people will also be looking for a great workplace and a meaningful, ambitious company to join.

This is where Cloudflare can help you. We can provide you the best of the two worlds. Living in a beautiful country, your home, while working in a world-class company, solving big problems at scale on a mission to help build a better Internet with a unique culture. Furthermore, we support your return, and we’re ready to help you in any way we can.

The future

Cloudflare is serious about its presence in Portugal. We’re going to continue growing and investing in highly skilled talent for our Lisbon office and making it one of Cloudflare’s top locations, alongside San Francisco, Austin, Singapore, and London.

Currently, we have 28 open positions in Lisbon, and you can expect new ones to open over the upcoming weeks. Some are for teams based in Lisbon, like Data Insights and Cloudflare Radar (we’re doubling in 2021), while others will join different projects, some of which have teams distributed across multiple offices.

If you decide to apply, there are many resources you can use to learn about Cloudflare and improve your chances of snatching your dream job. Here are a few:

  • Cloudflare Careers Day: Meet the Engineering Team with Isabel Rodrigues, Jen Langdon, and me.
  • Our Blog. We share an unusual amount of information about our infrastructure and products, our technical decisions, architecture, and our approach to solving complex, large-scale technical challenges.
  • Our Official Github Page. We have open-source encoded all over in our DNA, and we like to give back to the community whenever possible. Cloudflare has over 300 public projects that you can explore, try them yourself, or fork.
  • Our Developers Website, where you can learn about our products, the way they work, their features, and APIs. Speaking of APIs, take a look at cloudflare-go and flarectl.
  • Check our community Forum, ask us questions; we’re always there for you, you’ll be surprised. Follow us on Twitter.
  • Cloudflare TV airs excellent content all the time. You can check our schedule for numerous live segments with the team and guests or re-run past segments. We also have a “Best of” archive.
  • Finally, you can try our products. As part of our mission and values, we offer very generous free tiers to individual users and small startups. You can try our CDN features, DDoS, Workers (100,000 requests per day, with Workers KV included), and even Access for Teams (with Argo tunnel included, for companies or households under 50 seats), at no cost.

We’re a highly ambitious, large-scale technology company with a soul. Fundamental to our mission to help build a better Internet is protecting the free and open Internet. Cloudflare powers Internet requests for ~16% of the Fortune 1,000 and serves 20 million HTTP requests per second on average.