This week, a group of US lawmakers introduced the Stopping Grinch Bots Act — new legislation that could stop holiday hoarders on the Internet. This inspired us to put a spin on a Dr. Seuss classic:
Each person on the Internet liked Christmas a lot But the Grinch Bot, built by the scalper did not! The Grinch Bot hated Christmas! The whole Christmas season! Now, please don’t ask why. No one quite knows the reason.
Cloudflare stops billions of bad bots every day. As you might have guessed, we see all types of attacks, but none is more painful than a Grinch Bot attack. Join us as we take a closer look at this notorious holiday villain…
25 days seconds of Christmas
What is the Grinch Bot? Technically speaking, it’s just a program running on a computer, making automated requests that reach different websites. We’ve come to refer to these requests as “bots” on the Internet. Bots move quickly, leveraging the efficiency of computers to carry out tasks at scale. The Grinch Bot is a very special type that satisfies two conditions:
It only pursues online inventory, attempting to purchase items before humans can complete their orders.
It only operates during the holiday season.
Now, attackers use bots to perform these tasks all year long. But in these winter months, we like to use the term “Grinch Bot” as seasonal terminology.
The Grinch Bot strikes first around Black Friday. It knows that the best discounts come around Thanksgiving, and it loves to get a good deal. Exclusive items are always the first to go, so attackers use the Grinch Bot to cut every (virtual) line and checkpoint. Cloudflare detected nearly 1.5 trillion bot requests on Black Friday. That’s about half of all our traffic; but more on this in a bit.
The Grinch Bot strikes again on Cyber Monday. As shoppers find gifts for their loved ones, bots are ten steps ahead — selecting “add to cart” automatically. Many bots have payment details ready (perhaps even stolen from your account!).
The Grinch Bot will buy 500 pairs of Lululemon joggers before you even get one. And it’ll do so in seconds.
Nearly 44% of traffic comes from bad bots
The Grinch Bot has friends working throughout the year, putting pressure on security teams and moving undetected. 43.8% of Internet traffic comes from these bots. When the holidays arrive, the Grinch Bot can ask its friends how to attack the largest sites. They have already been testing tactics for months.
In response, many sites block individual IP addresses, groups of devices, or even entire countries. Other sites use Rate Limiting to reduce traffic volume. At Cloudflare, we’ve advocated not only for Rate Limiting, but also for a more sophisticated approach known as Bot Management, which dynamically identifies threats as they appear. Here’s a look at bot traffic before the holidays (1H 2021):
When we looked at bot traffic on Black Friday, we found that it had surged to nearly 50%.Cloudflare Radar showed data close to 55% (if you want to include the good bots as well). Businesses tell us this is the most vulnerable time of the year for their sites.
Over 300 billion bots…
Bots are highly effective at scale. While humans can purchase one or two items within a few minutes, bots can purchase far more inventory with little effort.
During the year, Cloudflare observed over 300 billion bots try to “add to cart.” How did we find this? We ran our bot detection engines on every endpoint that contains the word “cart.” Keep in mind, most bots are stopped before they can even view item details. There are trillions of inventory hoarding bots that were caught earlier in their efforts by our Bot Management and security solutions.
Even worse, some bots want to steal your holiday funds. They skip the ecommerce sites and head right for your bank, where they test stolen credentials and try to break into your account. 71% of login traffic comes from bots:
Bots operate at such an immense scale that they occasionally succeed. When this happens, they can break into accounts, retrieve your credit card information, and begin a holiday shopping spree.
Deck the halls with JS Challenges
We hate CAPTCHAs almost as much as we hate the Grinch Bot, so we built JS challenges as a lightweight, non-interactive alternative:
Not surprisingly, we issue more JS Challenges when more bots reach our network. These challenges are traditionally a middle ground between taking no action and completely blocking requests. They offer a chance for suspicious looking requests to prove their legitimacy. Cloudflare issued over 35 billion JS Challenges over the shopping weekend.
Even more impressive, however, is the number of threats blocked around this time. On Black Friday, Cloudflare blocked over 150 billion threats:
While we expected the Grinch Bot to make its move on Friday, we did not expect it to recede as it did on Cyber Monday. Bot traffic decreased as the shopping weekend continued. We like to think the Grinch Bot spent its time furiously trying to avoid blocks and JS Challenges, but eventually gave up.
Saving the Internet (and Christmas)
While large retailers can afford to purchase bot solutions, not every site is so fortunate. We decided to fix that.
Cloudflare’s Bot Fight Mode is a completely free tool that stops bots. You can activate it with one click, drawing on our advanced detection engines to protect your site. It’s easy:
And Bot Fight Mode doesn’t just stop bots — it makes them pay. We unleash a tarpit challenge that preoccupies each bot with nonsense puzzles, ultimately handing bot operators a special gift: a massive server bill. We even plant trees to offset the carbon emissions of these expensive challenges. In fact, with so many bots stopped in the snow, there’s really just one thing left to say…
Every person on the Internet, the tall and the small, Called out with joy that their shopping didn’t stall! He hadn’t stopped Christmas from coming! It came! Somehow or other, it came just the same! And the Grinch Bot, with his grinch feet ice-cold in the snow, Stood puzzling and puzzling. “How could it be so?”
November comes, the temperatures start to get colder for most of the planet’s population (87% live in the Northern Hemisphere) and many are also starting to prepare for the festive season. That also brings significant changes in Internet traffic, most notably the online shopping kind of traffic.
So, what were the November days that e-commerce websites had the most traffic in the US and what about worldwide? Is humanity using more mobile Internet at this time? And what are the most popular days online — is Black Friday the winner?
Let’s start with e-commerce — we added a chart to Radar that shows trends for e-commerce by country. The worldwide trend is pretty evident: Cyber Monday, the day for supposedly last-minute discounts, was the clear winner.
Worldwide most popular days for e-commerce
#1. Cyber Monday, November 29.
#2. Monday, November 23.
#3. Black Friday, November 26 — November 24 is pretty close to Black Friday. All in all a very good week in terms of e-commerce traffic.
US: November e-commerce traffic ‘rain’
When we focus on the United States, the country that instituted Black Friday (the day after US Thanksgiving has since become a “retail bonanza” in other countries), the trend is a little different when we look to the full month of November.
US most popular days for e-commerce
#1. Cyber Monday, November 29.
#2. Monday, November 2.
#3. Sunday, November 1.
The Black Friday week definitely had a big impact on e-commerce traffic, but besides the clear winner, Cyber Monday, the podium was actually completed with the first two days in November. Those days have a big traffic peak, but the Black Friday week has more sustained traffic over five days.
When we look just at last week, Black Friday isn’t actually the most popular day, it’s Monday, November 22 — that isn’t surprising given that shoppers also “returned to stores” on Black Friday 2021 and didn’t do everything online.
Despite this, Black Friday 2021 had definitely more sustained traffic throughout the day. The line in the next chart stays up on November 26 (Black Friday) for several hours after 12:00 UTC, early morning in the US, more than in the previous days.
For example, when we look at the 00:00 UTC mark in those red circles (19:00 US East Coast time; 16:00 US West Coast time), Black Friday evening was the most popular evening of the week — even more than November 22. In the past few days, only Cyber Monday had (a lot) more traffic than Black Friday.
And we can also notice the “pause” in online shopping for Thanksgiving Day (we wrote a blog post about that).
2021: How about the UK, France, Germany or India?
With our new Radar tool for e-commerce websites, everyone can see the trends for their country looking back to the previous seven or 30 days. We can give some interesting examples by looking at some countries.
In the UK, for example, the most popular day was Black Friday, followed by Cyber Monday.
In Germany, Black Friday 2021, followed by Cyber Monday, were the most popular days although there’s a bigger traffic peak on November 2.
In the neighbourhood, ‘down’ in France, the most popular days for e-commerce were Thursday, November 18, and Tuesday, November 23. Those days were even bigger than Black Friday or Cyber Monday — there’s also a clear sustained increase in traffic in the Black Friday week.
Now let’s ‘travel’ to India, the fastest growing online retail market in the world, which also had the Black Friday week as the best week of the month for online shopping. Cyber Monday was the most popular day, followed by Wednesday, November 24, and also Black Friday.
One exception seems to be Japan. The start of the Black Friday week and the end of the previous week were the better periods for online shopping traffic — November 18, 23 and 20 were much better days than Black Friday or Cyber Monday.
The mobile traffic percentage rose by the end of November
So how about November? If we look at the worldwide trend, it’s pretty clear that after Sunday, November 22, the mobile traffic percentage went up — Internet traffic from mobile devices represented 55% of the total in the past week.
We can also see in the next chart that Black Friday, November 26, saw an increase of more than 4% in the mobile traffic percentage, compared to the same period of the previous month. So, people were using their mobile devices a lot more to go online — 4% more.
Now let’s go to the US, where Thanksgiving (as we explained before) had a big influence on Internet traffic. That trend is even more pronounced, specifically on Thanksgiving day, November 25 (mobile traffic percentage grew more than 6%), but also on Black Friday, November 26. At the weekend mobile traffic went back down.
And remember: you can keep an eye on Cloudflare Radar to monitor how we see Internet traffic globally and in every country.
Cloudflare Radar launched as part of last year’s Birthday Week. We described it as a“newspaper for the Internet”, that gives“any digital citizen the chance to see what’s happening online [which] is part of our pursuit to help build a better, more informed, Internet”.
Since then, we have made considerable strides, including adding dedicated pages to cover how key events such as the UEFA Euro 2020 Championship and the Tokyo Olympics shaped Internet usage in participating countries, and added a Radar section for interactive deep-dive reports on topics such as DDoS.
Today, Radar has four main sections:
Main page with near real-time information about global Internet usage.
Internet usage details by country (see, for example, Portugal).
Domain insights, where searching for a domain returns traffic, registration and certificate information about it.
Deep-dive reports on complex and often underreported topics.
Cloudflare’s global network spans more than 250 cities in over 100 countries. Because of this, we have the unique ability to see both macro and micro trends happening online, including insights on how traffic is flowing around the world or what type of attacks are prevalent in a certain country.
Radar Maps will make this information even richer and easier to consume.
Introducing Radar Maps
Starting today, Radar has two new data visualizations to help us share more insights from our data and represent what’s happening on the Internet.
Geographical distribution of application-level attacks
Note: The identified location of the devices involved in the attack may not be the actual location of the people performing the attack.
Geographical distribution of application-level attacks, in both directions
Cyber threats are more common than ever. In the third quarter of 2021 Cloudflare blocked an average of 76 billion cyber threats each day and had visibility over many more. Helping build a better Internet also means giving people more visibility over our data. That’s why we’ve made a near real-time view of the types of attacks, protocol distribution, and attack volume over time available on Radar from day one.
Now we’re adding a geographical representation of origin and target of such attacks using two new visualizations.
First, we have a global map drawing near real-time directional lines of the attacks, also known as a “pew pew” map — thank you, 1983 and WarGames.
Second, we have Sankey diagrams that are great for representing how strongly the attacks are flowing from one country to the other.
We hope you like what we’ve built with our new Radar Maps. Radar, unlike any other insights platform out there, is totally built on Cloudflare components and our edge computing platform — Workers and Workers KV. This gives us new and unique ways of representing data at scale. So do keep checking back radar.cloudflare.com to see the Internet evolving in (near) real-time.
Meris first got our attention due to an exceptionally large 17.2 million requests per second (rps) DDoS attack that it launched against one of our customers. This attack, along with subsequent attacks originated by the Meris botnet, was automatically detected and mitigated by our DDoS protection systems. Cloudflare customers, even ones on the free plan, are protected against Meris attacks.
Over the past months, we’ve been tracking and analyzing the activity of the Meris botnet. Some main highlights include:
Meris targets approximately 50 different websites every single day with a daily average of 104 unique DDoS attacks.
More than 33% of all Meris DDoS attack traffic targeted China-based websites.
More than 12% of all websites that were attacked by Meris are operated by US-based companies.
View more Meris attack insights and trends in the interactive Radar dashboard.
So what is Meris?
Meris (Latvian for plague) is the name of an active botnet behind a series of recent DDoS attacks that have targeted thousands of websites around the world. It was originally detected in late June 2021 by QRator in joint research they conducted with Yandex. Their initial research identified 30,000 to 56,000 bots, but they estimated that the numbers are actually much higher, in the ballpark of 250,000 bots.
The Meris botnet is formed of infected routers and networking hardware manufactured by the Latvian company MikroTik. According to MikroTik’s blog, the attackers exploited a vulnerability in the router’s operating system (RouterOS) which enabled attackers to gain unauthenticated remote access to read and write arbitrary files (CVE-2018-14847).
RouterOS is the router operating system that’s used by MikroTik’s routers and the RouterBOARD hardware product family, which can also be used to turn any PC into a router. Administration of RouterOS can be done either via direct SSH connection or by using a configuration utility called WinBox. The vulnerability itself was possible due to a directory traversal vulnerability in the WinBox interface with RouterOS.
Directory traversal is a type of exploit that allows attackers to travel to the parent directories to gain access to the operating system’s file system, a method and structure of how data is stored and retrieved in the operating system. Once they gain access to the file system, attackers can then read the existing files that administer the router and write files directly into the file system to administer the routers to their botnet needs.
While the vulnerability was patched after its detection back in 2018, it’s still being exploited in compromised devices that do not use the patched RouterOS versions, or that use the default usernames and passwords. MicroTik has advised its customers to upgrade their devices’ OS version, to only allow access to the devices via secure IPsec, and to inspect for any abnormalities such as unknown SOCKS proxy settings and scripts.
To launch volumetric attacks, the botnet uses HTTP pipelining which allows it to send multiple requests over a single connection, thus increasing its total attack throughput. Furthermore, in an attempt to obfuscate the attack source, the botnet uses open SOCKS proxies to proxy their attack traffic to the target.
Cloudflare’s DDoS protection systems automatically detect and mitigate Meris attacks. One of the mitigation actions that the system can choose to use is the ‘Connection Close’ action which eliminates the risk of HTTP pipelining and helps slow down attackers. Additionally, as part of Cloudflare’s threat intelligence suite, we provide a Managed IP List of Open SOCKS Proxies that customers can use as part of their firewall rules — to block, challenge or rate-limit traffic that arrives via SOCKS proxies.
How does Meris compare to Mirai?
About five years ago, Mirai (Japanese for future) — the infamous botnet that infected hundreds of thousands of IoT devices — launched record-breaking DDoS attacks against websites.
There have been many variants of the Mirai botnet since its source code was leaked. One version of Mirai, called Moobot, was detected last year when it attacked a Cloudflare customer with a 654 Gbps DDoS attack. Another variant recently made a resurgence when it targeted Cloudflare customers with over a dozen UDP and TCP based DDoS attacks that peaked multiple times above 1 Tbps, with a max peak of approximately 1.2 Tbps.
While Mirai infected IoT devices with low computational power, Meris is a swarm of routers that have significantly higher processing power and data transfer capabilities than IoT devices, making them much more potent in causing harm at a larger scale to web properties that are not protected by sophisticated cloud-based DDoS mitigation.
Tracking the Meris botnet attacks
Since the appearance of Meris, Cloudflare’s systems automatically detected and mitigated Meris attacks using the existing mitigation rules. During our analysis of the Meris botnet attacks, our security experts noticed the attack vectors adapt to try and bypass Cloudflare’s defenses. Needless to say, they were not successful. But we wanted to stay many steps ahead of attackers — and so our engineers deployed additional rules that mitigate Meris attacks even more comprehensively. A side effect of these mitigation rules is that it also provides us with more granular threat intelligence on the Meris attacks.
Since we deployed the new rules in early August, we’ve seen Meris launch an average of 104 DDoS attacks on Cloudflare customers every day. The highest figure we’ve seen was on September 6, when Meris was used to launch 261 unique attacks against Cloudflare customers.
Overall, Meris targets about 50 different websites and applications every single day. Although the average attack peaked at 106K rps, the median attack size was actually smaller at 17.6K rps. The largest attack we’ve seen was 17.2M rps and that occurred in July. In the graph below, you can see the daily highest requests per second rate after we deployed the new rules. Since then, the largest attack we’ve seen was 16.7M rps, which took place on August 19.
Meris used to target Banks, Financial Services, and Insurance companies
Over the past few months, the industry that received the most attack traffic from the Meris botnet was the Banking, Financial Services, and Insurance (BFSI) industry
Following the BFSI industry, the most attacked industries were the Publishing, Gaming/Gambling, and IT Services industries. And while BFSI was the number one most attacked industry when considering the Meris DDoS activity rate, it only came in fourth place when considering the percentage of targeted websites.
In terms of the percentage of targeted websites, the Computer Software industry came in first place. Almost 4% of all impacted websites were of Computer Software companies protected by Cloudflare, followed by Gaming/Gambling and IT Services with 3% and 2%, respectively.
Besides the total breakdowns shown above, we can also view the top industries the botnet attacked over time to understand the changing trends. These trends may be tied to political events, new video game releases, sporting events, or any other global or local public interest events.
Off the top, we can already see the two largest peaks on August 9 and August 29 — mainly on the Computer Software, Gaming/Gambling, and IT industries. Another interesting peak occurred on August 14 against Cryptocurrency providers.
In late August, the botnet was pointed against gambling and casino websites, generating attacks at rates of hundreds of thousands to millions of requests per second. A second significant wave against the same industry was launched in early September.
Meris targets websites in China, Australia, and US
Similarly to the analysis of the top industries, we can calculate the Meris DDoS activity rate per target country to identify which countries came under the most attacks. In total, China-based companies saw the largest amount of DDoS attacks. More than 33% of all requests generated by Meris were destined for China-based companies that are protected by Cloudflare. Australia came in second place, and the US in third.
On the other hand, when we look at the number of websites that were targeted by Meris, the US came in first place. More than 12% of all websites that were targeted by Meris are operated by US-based companies. China came in second place with 5.6% and Russia in third with 4.4%.
Over time, we can see how the attacks on the top countries change. Similarly to the per-industry breakdown, we can also see two large peaks. The first one occurred on the same spike as the per-industry breakdown on August 9. However, the second one here occurred on September 1.
Although only tens of thousands of bots have been detected per attack, it is estimated that there are roughly 250,000 bots worldwide. As indicated above, the botnet is formed of MikroTik routers. Using the source IP address of the routers, we’re able to identify the origin country of the bots to paint a geographical representation of the bots’ presence and growth over time.
The change in the location of the bots doesn’t necessarily indicate that the botnet is growing or shrinking. It could also be that different bot groups are activated from time to time to spread the load of the attacks while attempting not to get caught.
At the beginning of August, the majority of the bots were located in Brazil. But by the end of August, that number plummeted to a single digit percentage close to zero. Meanwhile, the number of infected devices grew in the United States. From the beginning of September, the number of bots was significantly higher in the US, Russia, India, Indonesia, and China.
Cloudflare operates autonomous DDoS protection systems that automatically detect and mitigate DDoS attacks of all types, including attacks launched by Meris and Mirai. These systems are also customizable, and Cloudflare customers can tweak and tune their DDoS protection settings as needed with the HTTP DDoS Managed Rulesetand the L3/4 DDoS Managed Ruleset.
It’s been a few days now since Facebook, Instagram, and WhatsApp went AWOL and experienced one of the most extended and rough downtime periods in their existence.
When that happened, we reported our bird’s-eye view of the event and posted the blog Understanding How Facebook Disappeared from the Internet where we tried to explain what we saw and how DNS and BGP, two of the technologies at the center of the outage, played a role in the event.
In the meantime, more information has surfaced, and Facebook has published a blog post giving more details of what happened internally.
As we said before, these events are a gentle reminder that the Internet is a vast network of networks, and we, as industry players and end-users, are part of it and should work together.
In the aftermath of an event of this size, we don’t waste much time debating how peers handled the situation. We do, however, ask ourselves the more important questions: “How did this affect us?” and “What if this had happened to us?” Asking and answering these questions whenever something like this happens is a great and healthy exercise that helps us improve our own resilience.
Today, we’re going to show you how the Facebook and affiliate sites downtime affected us, and what we can see in our data.
1.1.1.1
1.1.1.1 is a fast and privacy-centric public DNS resolver operated by Cloudflare, used by millions of users, browsers, and devices worldwide. Let’s look at our telemetry and see what we find.
First, the obvious. If we look at the response rate, there was a massive spike in the number of SERVFAIL codes. SERVFAILs can happen for several reasons; we have an excellent blog called Unwrap the SERVFAIL that you should read if you’re curious.
In this case, we started serving SERVFAIL responses to all facebook.com and whatsapp.com DNS queries because our resolver couldn’t access the upstream Facebook authoritative servers. About 60x times more than the average on a typical day.
If we look at all the queries, not specific to Facebook or WhatsApp domains, and we split them by IPv4 and IPv6 clients, we can see that our load increased too.
As explained before, this is due to a snowball effect associated with applications and users retrying after the errors and generating even more traffic. In this case, 1.1.1.1 had to handle more than the expected rate for A and AAAA queries.
Here’s another fun one.
DNS vs. DoT and DoH. Typically, DNS queries and responses are sent in plaintext over UDP (or TCP sometimes), and that’s been the case for decades now. Naturally, this poses security and privacy risks to end-users as it allows in-transit attacks or traffic snooping.
With DNS over TLS (DoT) and DNS over HTTPS, clients can talk DNS using well-known, well-supported encryption and authentication protocols.
Our learning center has a good article on “DNS over TLS vs. DNS over HTTPS” that you can read. Browsers like Chrome, Firefox, and Edge have supported DoH for some time now, WAP uses DoH too, and you can even configure your operating system to use the new protocols.
When Facebook went offline, we saw the number of DoT+DoH SERVFAILs responses grow by over x300 vs. the average rate.
So, we got hammered with lots of requests and errors, causing traffic spikes to our 1.1.1.1 resolver and causing an unexpected load in the edge network and systems. How did we perform during this stressful period?
Quite well. 1.1.1.1 kept its cool and continued serving the vast majority of requests around the famous 10ms mark. An insignificant fraction of p95 and p99 percentiles saw increased response times, probably due to timeouts trying to reach Facebook’s nameservers.
Another interesting perspective is the distribution of the ratio between SERVFAIL and good DNS answers, by country. In theory, the higher this ratio is, the more the country uses Facebook. Here’s the map with the countries that suffered the most:
Here’s the top twelve country list, ordered by those that apparently use Facebook, WhatsApp and Instagram the most:
Country
SERVFAIL/Good Answers ratio
Turkey
7.34
Grenada
4.84
Congo
4.44
Lesotho
3.94
Nicaragua
3.57
South Sudan
3.47
Syrian Arab Republic
3.41
Serbia
3.25
Turkmenistan
3.23
United Arab Emirates
3.17
Togo
3.14
French Guiana
3.00
Impact on other sites
When Facebook, Instagram, and WhatsApp aren’t around, the world turns to other places to look for information on what’s going on, other forms of entertainment or other applications to communicate with their friends and family. Our data shows us those shifts. While Facebook was going down, other services and platforms were going up.
To get an idea of the changing traffic patterns we look at DNS queries as an indicator of increased traffic to specific sites or types of site.
Here are a few examples.
Other social media platforms saw a slight increase in use, compared to normal.
Traffic to messaging platforms like Telegram, Signal, Discord and Slack got a little push too.
Nothing like a little gaming time when Instagram is down, we guess, when looking at traffic to sites like Steam, Xbox, Minecraft and others.
And yes, people want to know what’s going on and fall back on news sites like CNN, New York Times, The Guardian, Wall Street Journal, Washington Post, Huffington Post, BBC, and others:
Attacks
One could speculate that the Internet was under attack from malicious hackers. Our Firewall doesn’t agree; nothing out of the ordinary stands out.
Network Error Logs
Network Error Logging, NEL for short, is an experimental technology supported in Chrome. A website can issue a Report-To header and ask the browser to send reports about network problems, like bad requests or DNS issues, to a specific endpoint.
Cloudflare uses NEL data to quickly help triage end-user connectivity issues when end-users reach our network. You can learn more about this feature in our help center.
If Facebook is down and their DNS isn’t responding, Chrome will start reporting NEL events every time one of the pages in our zones fails to load Facebook comments, posts, ads, or authentication buttons. This chart shows it clearly.
WARP
Cloudflare announced WARP in 2019, and called it “A VPN for People Who Don’t Know What V.P.N. Stands For” and offered it for free to its customers. Today WARP is used by millions of people worldwide to securely and privately access the Internet on their desktop and mobile devices. Here’s what we saw during the outage by looking at traffic volume between WARP and Facebook’s network:
You can see how the steep drop in Facebook ASN traffic coincides with the start of the incident and how it compares to the same period the day before.
Our own traffic
People tend to think of Facebook as a place to visit. We log in, and we access Facebook, we post. It turns out that Facebook likes to visit us too, quite a lot. Like Google and other platforms, Facebook uses an army of crawlers to constantly check websites for data and updates. Those robots gather information about websites content, such as its titles, descriptions, thumbnail images, and metadata. You can learn more about this on the “The Facebook Crawler” page and the Open Graph website.
Here’s what we see when traffic is coming from the Facebook ASN, supposedly from crawlers, to our CDN sites:
The robots went silent.
What about the traffic coming to our CDN sites from Facebook User-Agents? The gap is indisputable.
We see about 30% of a typical request rate hitting us. But it’s not zero; why is that?
We’ll let you know a little secret. Never trust User-Agent information; it’s broken. User-Agent spoofing is everywhere. Browsers, apps, and other clients deliberately change the User-Agent string when they fetch pages from the Internet to hide, obtain access to certain features, or bypass paywalls (because pay-walled sites want sites like Facebook to index their content, so that then they get more traffic from links).
Fortunately, there are newer, and privacy-centric standards emerging like User-Agent Client Hints.
Core Web Vitals
Core Web Vitals are the subset of Web Vitals, an initiative by Google to provide a unified interface to measure real-world quality signals when a user visits a web page. Such signals include Largest Contentful Paint (LCP), First Input Delay (FID), and Cumulative Layout Shift (CLS).
We use Core Web Vitals with our privacy-centric Web Analytics product and collect anonymized data on how end-users experience the websites that enable this feature.
One of the metrics we can calculate using these signals is the page load time. Our theory is that if a page includes scripts coming from external sites (for example, Facebook “like” buttons, comments, ads), and they are unreachable, its total load time gets affected.
We used a list of about 400 domains that we know embed Facebook scripts in their pages and looked at the data.
Now let’s look at the Largest Contentful Paint. LCP marks the point in the page load timeline when the page’s main content has likely loaded. The faster the LCP is, the better the end-user experience.
Again, the page load experience got visibly degraded.
The outcome seems clear. The sites that use Facebook scripts in their pages took 1.5x more time to load their pages during the outage, with some of them taking more than 2x the usual time. Facebook’s outage dragged the performance of some other sites down.
Conclusion
When Facebook, Instagram, and WhatsApp went down, the Web felt it. Some websites got slower or lost traffic, other services and platforms got unexpected load, and people lost the ability to communicate or do business normally.
Attackers continue targeting VoIP infrastructure around the world. In our blog from last week, May I ask who’s calling, please? A recent rise in VoIP DDoS attacks, we reviewed how the SIP protocol works, ways it can be abused, and how Cloudflare can help protect against attacks on VoIP infrastructure without impacting performance.
Cloudflare’s network stands in front of some of the largest, most performance-sensitive voice and video providers in the world, and is uniquely well suited to mitigating attacks on VoIP providers.
Because of the sustained attacks we are observing, we are sharing details on recent attack patterns, what steps they should take before an attack, and what to do after an attack has taken place.
Below are three of the most common questions we’ve received from companies concerned about attacks on their VoIP systems, and Cloudflare’s answers.
Question #1: How is VoIP infrastructure being attacked?
The attackers primarily use off-the-shelf booter services to launch attacks against VoIP infrastructure. The attack methods being used are not novel, but the persistence of the attacker and their attempts to understand the target’s infrastructure are.
Attackers have used various attack vectors to probe the existing defenses of targets and try to infiltrate any existing defenses to disrupt VoIP services offered by certain providers. In some cases, they have been successful. HTTP attacks against API gateways and the corporate websites of the providers have been combined with network-layer and transport-layer attack against VoIP infrastructures. Examples:
TCP floods targeting stateful firewalls These are being used in “trial-and-error” type attacks. They are not very effective against telephony infrastructure specifically (because it’s mostly UDP) but very effective at overwhelming stateful firewalls.
UDP floods targeting SIP infrastructure Floods of UDP traffic that have no well-known fingerprint, aimed at critical VoIP services. Generic floods like this may look like legitimate traffic to unsophisticated filtering systems.
UDP reflection targeting SIP infrastructure These methods, when targeted at SIP or RTP services, can easily overwhelm Session Border Controllers (SBCs) and other telephony infrastructure. The attacker seems to learn enough about the target’s infrastructure to target such services with high precision.
SIP protocol-specific attacks Attacks at the application layer are of particular concern because of the higher resource cost of generating application errors vs filtering on network devices.
Question #2: How should I prepare my organization in case our VoIP infrastructure is targeted?
Deploy an always-on DDoS mitigation service Cloudflare recommends the deployment of always-on network level protection, like Cloudflare Magic Transit, prior to your organization being attacked.
Do not rely on reactive on-demand SOC-based DDoS Protection services that require humans to analyze attack traffic — they take too long to respond. Instead, onboard to a cloud service that has sufficient network capacity and automated DDoS mitigation systems.
Cloudflare has effective mitigations in place for the attacks seen against VoIP infrastructure, including for sophisticated TCP floods and SIP specific attacks.
Enforce a positive security model Block TCP on IP/port ranges that are not expected to receive TCP, instead of relying on on-premise firewalls that can be overwhelmed. Block network probing attempts (e.g. ICMP) and other packets that you don’t normally expect to see.
Build custom mitigation strategies Work together with your DDoS protection vendor to tailor mitigation strategies to your workload. Every network is different, and each poses unique challenges when integrating with DDoS mitigation systems.
Educate your employees Train all of your employees to be on the lookout for ransom demands. Check email, support tickets, form submissions, and even server access logs. Ensure employees know to immediately report ransom demands to your Security Incident Response team.
Question #3: What should I do if I receive a ransom/threat?
Do not to pay the ransom Paying the ransom only encourages bad actors—and there’s no guarantee that they won’t attack your network now or later.
Notify Cloudflare We can help ensure your website and network infrastructure are safeguarded against these attacks.
Notify local law enforcement They will also likely request a copy of the ransom letter that you received.
Cloudflare is here to help
With over 100 Tbps of network capacity, a network architecture that efficiently filters traffic close to the source, and a physical presence in over 250 cities, Cloudflare can help protect critical VoIP infrastructure without impacting latency, jitter, and call quality. Test results demonstrate a performance improvement of 36% on average across the globe for a real customer network using Cloudflare Magic Transit.
Some of the largest voice and video providers in the world rely on Cloudflare to protect their networks and ensure their services remain online and fast. We stand ready to help.
Talk to a Cloudflare specialist to learn more. Under attack? Contact our hotline to speak with someone immediately.
“Facebook can’t be down, can it?”, we thought, for a second.
Today at 1651 UTC, we opened an internal incident entitled “Facebook DNS lookup returning SERVFAIL” because we were worried that something was wrong with our DNS resolver 1.1.1.1. But as we were about to post on our public status page we realized something else more serious was going on.
Social media quickly burst into flames, reporting what our engineers rapidly confirmed too. Facebook and its affiliated services WhatsApp and Instagram were, in fact, all down. Their DNS names stopped resolving, and their infrastructure IPs were unreachable. It was as if someone had “pulled the cables” from their data centers all at once and disconnected them from the Internet.
How’s that even possible?
Meet BGP
BGP stands for Border Gateway Protocol. It’s a mechanism to exchange routing information between autonomous systems (AS) on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn’t know what to do, and the Internet wouldn’t work.
The Internet is literally a network of networks, and it’s bound together by BGP. BGP allows one network (say Facebook) to advertise its presence to other networks that form the Internet. As we write Facebook is not advertising its presence, ISPs and other networks can’t find Facebook’s network and so it is unavailable.
The individual networks each have an ASN: an Autonomous System Number. An Autonomous System (AS) is an individual network with a unified internal routing policy. An AS can originate prefixes (say that they control a group of IP addresses), as well as transit prefixes (say they know how to reach specific groups of IP addresses).
Cloudflare’s ASN is AS13335. Every ASN needs to announce its prefix routes to the Internet using BGP; otherwise, no one will know how to connect and where to find us.
In this simplified diagram, you can see six autonomous systems on the Internet and two possible routes that one packet can use to go from Start to End. AS1 → AS2 → AS3 being the fastest, and AS1 → AS6 → AS5 → AS4 → AS3 being the slowest, but that can be used if the first fails.
At 1658 UTC we noticed that Facebook had stopped announcing the routes to their DNS prefixes. That meant that, at least, Facebook’s DNS servers were unavailable. Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com.
route-views>show ip bgp 185.89.218.0/23
% Network not in table
route-views>
route-views>show ip bgp 129.134.30.0/23
% Network not in table
route-views>
Meanwhile, other Facebook IP addresses remained routed but weren’t particularly useful since without DNS Facebook and related services were effectively unavailable:
route-views>show ip bgp 129.134.30.0
BGP routing table entry for 129.134.0.0/17, version 1025798334
Paths: (24 available, best #14, table default)
Not advertised to any peer
Refresh Epoch 2
3303 6453 32934
217.192.89.50 from 217.192.89.50 (138.187.128.158)
Origin IGP, localpref 100, valid, external
Community: 3303:1004 3303:1006 3303:3075 6453:3000 6453:3400 6453:3402
path 7FE1408ED9C8 RPKI State not found
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
route-views>
We keep track of all the BGP updates and announcements we see in our global network. At our scale, the data we collect gives us a view of how the Internet is connected and where the traffic is meant to flow from and to everywhere on the planet.
A BGP UPDATE message informs a router of any changes you’ve made to a prefix advertisement or entirely withdraws the prefix. We can clearly see this in the number of updates we received from Facebook when checking our time-series BGP database. Normally this chart is fairly quiet: Facebook doesn’t make a lot of changes to its network minute to minute.
But at around 15:40 UTC we saw a peak of routing changes from Facebook. That’s when the trouble began.
If we split this view by routes announcements and withdrawals, we get an even better idea of what happened. Routes were withdrawn, Facebook’s DNS servers went offline, and one minute after the problem occurred, Cloudflare engineers were in a room wondering why 1.1.1.1 couldn’t resolve facebook.com and worrying that it was somehow a fault with our systems.
With those withdrawals, Facebook and its sites had effectively disconnected themselves from the Internet.
DNS gets affected
As a direct consequence of this, DNS resolvers all over the world stopped resolving their domain names.
➜ ~ dig @1.1.1.1 facebook.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;facebook.com. IN A
➜ ~ dig @1.1.1.1 whatsapp.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;whatsapp.com. IN A
➜ ~ dig @8.8.8.8 facebook.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;facebook.com. IN A
➜ ~ dig @8.8.8.8 whatsapp.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;whatsapp.com. IN A
This happens because DNS, like many other systems on the Internet, also has its routing mechanism. When someone types the https://facebook.com URL in the browser, the DNS resolver, responsible for translating domain names into actual IP addresses to connect to, first checks if it has something in its cache and uses it. If not, it tries to grab the answer from the domain nameservers, typically hosted by the entity that owns it.
If the nameservers are unreachable or fail to respond because of some other reason, then a SERVFAIL is returned, and the browser issues an error to the user.
Again, our learning center provides a good explanation on how DNS works.
Due to Facebook stopping announcing their DNS prefix routes through BGP, our and everyone else’s DNS resolvers had no way to connect to their nameservers. Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses.
But that’s not all. Now human behavior and application logic kicks in and causes another exponential effect. A tsunami of additional DNS traffic follows.
This happened in part because apps won’t accept an error for an answer and start retrying, sometimes aggressively, and in part because end-users also won’t take an error for an answer and start reloading the pages, or killing and relaunching their apps, sometimes also aggressively.
This is the traffic increase (in number of requests) that we saw on 1.1.1.1:
So now, because Facebook and their sites are so big, we have DNS resolvers worldwide handling 30x more queries than usual and potentially causing latency and timeout issues to other platforms.
Fortunately, 1.1.1.1 was built to be Free, Private, Fast (as the independent DNS monitor DNSPerf can attest), and scalable, and we were able to keep servicing our users with minimal impact.
The vast majority of our DNS requests kept resolving in under 10ms. At the same time, a minimal fraction of p95 and p99 percentiles saw increased response times, probably due to expired TTLs having to resort to the Facebook nameservers and timeout. The 10 seconds DNS timeout limit is well known amongst engineers.
Impacting other services
People look for alternatives and want to know more or discuss what’s going on. When Facebook became unreachable, we started seeing increased DNS queries to Twitter, Signal and other messaging and social media platforms.
We can also see another side effect of this unreachability in our WARP traffic to and from Facebook’s affected ASN 32934. This chart shows how traffic changed from 15:45 UTC to 16:45 UTC compared with three hours before in each country. All over the world WARP traffic to and from Facebook’s network simply disappeared.
The Internet
Today’s events are a gentle reminder that the Internet is a very complex and interdependent system of millions of systems and protocols working together. That trust, standardization, and cooperation between entities are at the center of making it work for almost five billion active users worldwide.
Update
At around 21:00 UTC we saw renewed BGP activity from Facebook’s network which peaked at 21:17 UTC.
This chart shows the availability of the DNS name ‘facebook.com’ on Cloudflare’s DNS resolver 1.1.1.1. It stopped being available at around 15:50 UTC and returned at 21:20 UTC.
Undoubtedly Facebook, WhatsApp and Instagram services will take further time to come online but as of 22:28 UTC Facebook appears to be reconnected to the global Internet and DNS working again.
Earlier this summer, Cloudflare’s autonomous edge DDoS protection systems automatically detected and mitigated a 17.2 million request-per-second (rps) DDoS attack, an attack almost three times larger than any previous one that we’re aware of. For perspective on how large this attack was: Cloudflare serves over 25 million HTTP requests per second on average. This refers to the average rate of legitimate traffic in 2021 Q2. So peaking at 17.2 million rps, this attack reached 68% of our Q2 average rps rate of legitimate HTTP traffic.
Comparison graph of Cloudflare’s average request per second rate versus the DDoS attack
Automated DDoS mitigation with Cloudflare’s autonomous edge
This attack, along with the additional attacks provided in the next sections, were automatically detected and mitigated by our autonomous edge DDoS protection systems. The system is powered by our very own denial of service daemon (dosd). Dosd is a home-grown software-defined daemon. A unique dosd instance runs in every server in each one of our data centers around the world. Each dosd instance independently analyzes traffic samples out-of-path. Analyzing traffic out-of-path allows us to scan asynchronously for DDoS attacks without causing latency and impacting performance. DDoS findings are also shared between the various dosd instances within a data center, as a form of proactive threat intelligence sharing.
Once an attack is detected, our systems generate a mitigation rule with a real-time signature that matches the attack patterns. The rule is propagated to the most optimal location in the tech stack. As an example, a volumetric HTTP DDoS attack may be blocked at L4 inside the Linux iptables firewall instead of at L7 inside the L7 reverse proxy which runs in the user space. Mitigating lower in the stack, e.g. dropping the packets at L4 instead of responding with a 403 error page in L7, is more cost-efficient. It reduces our edge CPU consumption and intra-data center bandwidth utilization — thus helping us mitigate large attacks at scale without impacting performance.
This autonomous approach, along with our network’s global scale and reliability, allow us to mitigate attacks that reach 68% of our average per-second-rate, and higher, without requiring any manual mitigation by Cloudflare personnel, nor causing any performance degradation.
The resurgence of Mirai and new powerful botnets
This attack was launched by a powerful botnet, targeting a Cloudflare customer in the financial industry. Within seconds, the botnet bombarded the Cloudflare edge with over 330 million attack requests.
Graph of 17.2M rps attack
The attack traffic originated from more than 20,000 bots in 125 countries around the world. Based on the bots’ source IP addresses, almost 15% of the attack originated from Indonesia and another 17% from India and Brazil combined. Indicating that there may be many malware infected devices in those countries.
Distribution of the attack sources by top countries
Volumetric attacks increase
This 17.2 million rps attack is the largest HTTP DDoS attack that Cloudflare has ever seen to date and almost three times the size of any other reported HTTP DDoS attack. This specific botnet, however, has been seen at least twice over the past few weeks. Just last week it also targeted a different Cloudflare customer, a hosting provider, with an HTTP DDoS attack that peaked just below 8 million rps.
Graph of 8M rps attack
Two weeks before, a Mirai-variant botnet launched over a dozen UDP and TCP based DDoS attacks that peaked multiple times above 1 Tbps, with a max peak of approximately 1.2 Tbps. And while the first HTTP attacks targeted Cloudflare customers on the WAF/CDN service, the 1+ Tbps network-layer attacks targeted Cloudflare customers on the Magic Transit and Spectrum services. One of these targets was a major APAC-based Internet services, telecommunications and hosting provider. The other was a gaming company. In all cases, the attacks were automatically detected and mitigated without human intervention.
Graph of Mirai botnet attack peaking at 1.2 Tbps
The Mirai botnet started with roughly 30K bots and slowly shrinked to approximately 28K. However, despite losing bots from its fleet, the botnet was still able to generate impressive volumes of attack traffic for short periods. In some cases, each burst lasted only a few seconds.
These attacks join the increase in Mirari-based DDoS attacks that we’ve observed on our network over the past weeks. In July alone, L3/4 Mirai attacks increased by 88% and L7 attacks by 9%. Additionally, based on the current August per-day average of the Mirai attacks, we can expect L7 Mirai DDoS attacks and other similar botnet attacks to increase by 185% and L3/4 attacks by 71% by the end of the month.
Graph of change in Mirai based DDoS attacks by month
Back to the Mirai
Mirai, which means ‘future’ in Japanese, is a codename for malware that was first discovered in 2016 by MalwareMustDie, a non-profit security research workgroup. The malware spreads by infecting Linux-operated devices such as security cameras and routers. It then self-propagates by searching for open Telnet ports 23 and 2323. Once found, it then attempts to gain access to vulnerable devices by brute forcing known credentials such as factory default usernames and passwords. Later variants of Mirai also took advantage of zero-day exploits in routers and other devices. Once infected, the devices will monitor a Command & Control (C2) server for instructions on which target to attack.
Diagram of Botnet operator controlling the botnet to attack websites
How to protect your home and business
While the majority of attacks are small and short, we continue to see these types of volumetric attacks emerging more often. It’s important to note that these volumetric short burst attacks can be especially dangerous for legacy DDoS protection systems or organizations without active, always-on cloud-based protection.
Furthermore, while the short duration may say something about the botnet’s capability to deliver sustained levels of traffic over time, it can be challenging or impossible for humans to react to it in time. In such cases, the attack is over before a security engineer even has time to analyze the traffic or activate their stand-by DDoS protection system. These types of attacks highlight the need for automated, always-on protection.
How to protect your business and Internet properties
Follow our preventive best practices, to ensure that both your Cloudflare settings and your origin server settings are optimized. As an example, make sure that you allow only traffic from Cloudflare’s IP range. Ideally, ask your upstream Internet Service Provider (ISP) to apply an access control list (ACL), otherwise, attackers may target your servers’ IP addresses directly and bypass your protection.
Recommendations on how to protect your home and IoT appliances
Change the default username and password of any device that is connected to the Internet such as smart cameras and routers. This will reduce the risk that malware such as Mirai can gain access to your router and IoT devices.
Protect your home against malware with Cloudflare for Families. Cloudflare for Families is a free service that automatically blocks traffic from your home to malicious websites and malware communication.
Recent weeks have witnessed massive ransomware and ransom DDoS (Distributed Denial of Service) attack campaigns that interrupted aspects of critical infrastructure around the world, including one of the largest petroleum pipeline system operators, and one of the world’s biggest meat processing companies. Earlier this quarter, more than 200 organizations across Belgium, including the government and parliament websites and other services, were also DDoS’d.
And when most of the United States were celebrating Independence Day on July 4, hundreds of US companies were hit by a ransomware attack demanding 70 million USD in Bitcoin. Attackers known to be affiliated with REvil, a Russian ransomware group, exploited multiple previously unknown vulnerabilities in IT management software. The targets included schools, small public-sector bodies, travel and leisure organizations, and credit unions, to name a few. While the threat of ransomware and ransom DDoS is not new (read our posts on ransomware and ransom DDoS from 2021 Q1), the latest attacks on Internet properties ranging from wineries, professional sports teams, ferry services and hospitals has brought them from just being background noise to front page headlines affecting our day-to-day lives. In fact, recent attacks have propelled ransomware and DDoS to the top of US President Biden’s national security agenda.
The DDoS attack trends observed over Cloudflare’s network in 2021 Q2 paint a picture that reflects the overall global cyber threat landscape. Here are some highlights.
Over 11% of our surveyed customers who were targeted by a DDoS attack reported receiving a threat or ransom letter threatening in advance, in the first six months of this year. Emergency onboarding of customers under an active DDoS attack increased by 41.8% in 2021 H1 compared to 2020 H2.
HTTP DDoS attacks targeting government administration/public sector websites increased by 491%, making it the second most targeted industry after Consumer Services whose DDoS activity increased by 684% QoQ.
China remains the country with the most DDoS activity originating from within their borders — 7 out of every 1,000 HTTP requests originating from China were part of an HTTP DDoS attack targeting websites, and more than 3 out of every 100 bytes that were ingested in our data centers in China were part of a network-layer DDoS attack.
Emerging threats included amplification DDoS attacks that abused the Quote of the Day (QOTD) protocol which increased by 123% QoQ. Additionally, as the adoption of QUIC protocol continues to increase, so do attacks over QUIC — registering a whopping 109% QoQ surge in 2021 Q2. The number of network-layer DDoS attacks in the range of 10-100 Gbps increased by 21.4% QoQ. One customer that was attacked is Hypixel, an American gaming company. Hypixel remained online with no downtime and no performance penalties to their gamer users, even when under an active DDoS attack campaign larger than 620 Gbps. Read their story here.
To view all DDoS attack insights across all regions and industries worldwide, visit Cloudflare’s interactive Radar DDoS dashboard.
Application-layer DDoS attacks
Application-layer DDoS attacks, specifically HTTP DDoS attacks, are attacks that usually aim to disrupt an HTTP server by making it unable to process legitimate user requests. If a server is bombarded with more requests than it can process, the server will drop legitimate requests or even crash resulting in performance penalties or a denial of service event for legitimate users.
DDoS activity per market industry
When we analyze attacks, we calculate the ‘DDoS activity’ rate, which is the percentage of attack traffic out of the total traffic (attack + clean). This allows us to normalize the data points and avoid biases towards, for example, a larger data center that naturally handles more traffic and therefore also more attacks.
In 2021 Q2, Consumer Services was the most targeted industry followed by Government Administration and Marketing & Advertising.
DDoS activity per source country
To understand the origin of the HTTP attacks we observed over Cloudflare’s network, we look at the source IP address of the client generating the attack HTTP requests. Unlike network-layer attacks, source IPs cannot be spoofed in HTTP attacks. A high DDoS activity rate in a given country indicates large botnets operating from within.
China and the US remain in the first and second places, respectively, regarding the percentage of DDoS activity originating from within their territories. In China, more than 7 out of every 1,000 HTTP requests were part of an HTTP DDoS attack, while in the US almost 5 out of 1,000 HTTP requests were part of an attack.
DDoS activity per target country
In order to identify which countries the targets of the DDoS attacks resided in, we break down the DDoS activity by our customers’ billing countries. Note that Cloudflare does not charge for attack traffic and has pioneered providing unmetered and unlimited DDoS protection since 2017. By cross-referencing the attack data with our customers’ billing country, we can identify which countries were attacked the most.
Data observed in 2021 Q2 suggest that organizations in the US and China were the most targeted by HTTP DDoS attacks. In fact, one out of every 200 HTTP requests destined to US-based organizations was part of a DDoS attack.
Network-layer DDoS attacks
While application-layer attacks strike the application (Layer 7 of the OSI model) running the service end users are trying to access, network-layer attacks target network infrastructure (such as in-line routers and other network servers) and the Internet link itself.
The chart above shows the distribution of network-layer DDoS attacks in 2021 Q2.
Distribution of attacks by size (packet rate and bit rate)
There are different ways of measuring the size of a L3/4 DDoS attack. One is the volume of traffic it delivers, measured as the bit rate (specifically, gigabits-per-second). Another is the number of packets it delivers, measured as the packet rate (specifically, packets-per-second). Attacks with high bit rates attempt to saturate the Internet link, while attacks with high packet rates attempt to overwhelm the servers, routers or other in-line hardware appliances.
The distribution of attacks by their size (in bit rate) and month is shown below. As observed in the chart, all attacks over 300 Gbps were observed in the month of June.
In terms of bit rate, attacks under 500 Mbps constituted a majority of all DDoS attacks observed in 2021 Q2.
Similarly, looking from the lens of packet rate, nearly 94% of attacks were under 50K pps. Even though attacks from 1-10M pps constituted only 1% of all DDoS attacks observed, this number is 27.5% higher than that observed in the previous quarter, suggesting that larger attacks are not diminishing either — but rather increasing.
Note that while attacks under 500 Mbps and 50K pps might seem ‘small’ compared to other headline-making large attacks, they are often sufficient to create major disruptions for Internet properties that are not protected by an always-on, automated cloud-based DDoS protection service. Moreso, many organisations have uplinks provided by their service providers with a bandwidth capacity smaller than 1 Gbps. Assuming their public-facing network interface also serves legitimate traffic, DDoS attacks smaller than 500 Mbps are often capable of taking down exposed Internet properties.
Distribution by attack duration
Cloudflare continues to see a large percentage of DDoS attacks that last under an hour. In Q2, over 97% of all DDoS attacks lasted less than an hour.
Short burst attacks may attempt to cause damage without being detected by DDoS detection systems. DDoS services that rely on manual analysis and mitigation may prove to be useless against these types of attacks because they are over before the analyst even identifies the attack traffic.
Alternatively, the use of short attacks may be used to probe the cyber defenses of the target. Load-testing tools and automated DDoS tools, that are widely available on the dark web, can generate short bursts of a SYN flood, for example, and then follow up with another short attack using a different attack vector. This allows attackers to understand the security posture of their targets before they decide to launch larger attacks at larger rates and longer durations — which come at a cost.
In other cases, attackers generate small DDoS attacks as proof and warning to the target organization of the attacker’s ability to cause real damage later on. It’s often followed by a ransom email to the target organization, demanding payment to avoid suffering an attack that could more thoroughly cripple network infrastructure.
This highlights the need for an always on, automated DDoS protection approach. DDoS protection services that rely on manual re-routing, analysis and mitigation may prove to be useless against these types of attacks because they are over before the analyst can even identify the attack traffic.
Distribution of attacks by attack vectors
An attack vector is the term used to describe the method that the attacker utilizes in their attempt to cause a denial of service event.
As observed in previous quarters, attacks utilizing SYN floods and UDP-based protocols remain the most popular methods by attackers.
What is a SYN flood attack? It’s a DDoS attack that exploits the very foundation of the TCP protocol. A stateful TCP connection between a client and a server begins with a 3-way TCP handshake. The client sends an initial connection request packet with a synchronize flag (SYN). The server responds with a packet that contains a synchronized acknowledgment flag (SYN-ACK). Finally, the client responds with an acknowledgment (ACK) packet. At this point, a connection is established and data can be exchanged until the connection is closed. This stateful process can be abused by attackers to cause denial of service events.
By repeatedly sending SYN packets, the attacker attempts to overwhelm a server or the router’s connection table that tracks the state of TCP connections. The router replies with a SYN-ACK packet, allocates a certain amount of memory for each given connection, and falsely waits for the client to respond with the final ACK. Given a sufficient number of connections occupying the router’s memory, the router is unable to allocate further memory for legitimate clients, causing the router to crash or preventing it from handling legitimate client connections, i.e., a denial of service event.
Emerging threats
Emerging threats included amplification DDoS attacks that abuse the Quote of the Day (QOTD) service which increased by 123% QoQ. QOTD was defined in RFC-865 (1983) and can be sent over either the UDP or TCP protocols. It was originally designed for debugging and as a measurement tool, with no specific syntax for the quote. The RFC does however recommend the use of ASCII characters and to limit the length to 512 characters.
Furthermore, we’ve seen a 107% increase QoQ in UDP Portmap and Echo attacks — all of which are really old attack vectors. This may indicate attackers digging up old methods and attack tools to try and overcome protection systems. As we’ve seen in previous quarters, the adoption of the QUIC protocol continues to increase. Consequently, so do attacks over QUIC, or more specifically floods and amplification attacks of non-QUIC traffic in places where we’d expect to see QUIC traffic. In 2021 Q2, these types of attacks increased by 109% QoQ. This continued trend may indicate that attackers are attempting to abuse the QUIC-designated ports and gateways into organizations’ networks — searching for vulnerabilities and security holes.
DDoS activity by Cloudflare data center country
In 2021 Q2, our data center in Haiti observed the largest percentage of network-layer DDoS attack traffic, followed by Brunei (almost 3 out of every 100 packets were part of an attack) and China.
Note that when analyzing network-layer DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations where the traffic was ingested, and not by the source IP. The reason for this is that, when attackers launch network-layer attacks, they can spoof the source IP address in order to obfuscate the attack source and introduce randomness into the attack properties, which may make it harder for simple DDoS protection systems to block the attack. Hence, if we were to derive the source country based on a spoofed source IP, we would get a spoofed country. Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of Cloudflare’s data center in which the attack was observed. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities around the world.
A note on ransomware and ransom DDoS — a growing global threat
The last few weeks have seen a resurgence of ransom-driven cyber threats: ransomware and ransom DDoS (RDDoS).
So what is ransomware and ransom DDoS, and how are they different?
Ransomware is malicious software that encrypts an organization’s systems and databases, rendering them inaccessible and unusable. Malware is usually introduced into an organization’s systems via phishing emails — tricking employees to click on a link or download a file. Once the malware is installed on the employee’s device, it encrypts the device and can propagate to the entire network of the organization’s servers and employee devices. The attacker will demand money, usually in the form of Bitcoin, in exchange for decrypting the organization’s systems and granting them access back to their systems.
Unlike a ransomware attack, a ransom DDoS attack does not encrypt a company’s systems; it aims to knock them offline if the ransom is not paid. What makes ransom DDoS attacks even more dangerous is that they do not require the attacker to gain access to a business’s internal systems to execute the attack. However, with a strong DDoS protection strategy in place, a ransom DDoS attack has little to no effect on businesses.
Ransomware and ransom DDoS threats are impacting most industries across the globe — the financial industry, transportation, oil and gas, consumer goods, and even education and healthcare.
Entities claiming to be ‘Fancy Lazarus’, ‘Fancy Bear’, ‘Lazarus Group’, and ‘REvil’ are once again launching ransomware and ransom-DDoS attacks against organizations’ websites and network infrastructure unless a ransom is paid before a given deadline. In the case of DDoS threats, prior to the ransom note, a small DDoS attack is usually launched as a form of demonstration. The demonstration attack is typically over UDP, lasting roughly 30-120 minutes.
The ransom note is typically sent to the common group email aliases of the company that are publicly available online such as noc@, support@, help@, legal@, abuse@, etc. In several cases, it has ended up in spam. In other cases, we’ve seen employees disregard the ransom note as spam, increasing the organization’s response time which resulted in further damage to their online properties.
Cloudflare’s recommendation for organizations that receive a threat or ransom note:
Do not panic, and we recommend you do not pay the ransom: Paying ransom only encourages and funds bad actors. There’s also no guarantee that you won’t be attacked again anyway.
Contact local law enforcement: Be ready to provide a copy of the ransom letter you received and any other logs or packet captures.
Activate an effective DDoS protection strategy: Cloud-based DDoS protection can be quickly onboarded in the event of an active threat, and with a team of security experts on your side, risks can be mitigated quickly and effectively.
Here’s a short video by Cloudflare CTO, John Graham-Cumming addressing the threat of ransom DDoS attacks.
Cloudflare protects Hypixel against a massive DDoS attack campaign
At Cloudflare, our teams have been exceptionally busy this past quarter rapidly onboarding (onto our Magic Transit service) a multitude of new and existing customers that have either received a ransom letter or were under an active DDoS attack.
One such customer is Hypixel Inc, the development studio behind the world’s largest Minecraft minigame server. With over 24M total unique logins to date and a world record 216,000+ concurrent players on PC, the Hypixel team works hard to add value to the experience of millions of players across the globe.
The gaming industry is often subject to some of the largest volumetric DDoS attacks — and as a marquee brand, Hypixel attracts more than its fair share. Uptime and high performance are fundamental to the functioning of Hypixel’s servers. Any perceived downtime or noticeable lag could result in an exodus of gamers.
When Hypixel was under a massive DDoS attack campaign, they turned to Cloudflare to extend their services with Cloudflare to include Magic Transit, Cloudflare’s BGP-based DDoS protection service for network infrastructure. After rapidly onboarding them overnight, Cloudflare was automatically able to detect and mitigate DDoS attacks targeting their network — several of which were well over 620 Gbps. The DDoS attack comprised mostly TCP floods and UDP amplification attacks. In the graph, the various colors represent the multiple Cloudflare systems that contribute to detecting and mitigating the multi-vector attack — emphasising the value of our multi-layered DDoS approach.
Even as attack patterns changed in real-time, Magic Transit shielded Hypixel’s network. In fact, because all their clean traffic routed over Cloudflare’s high performing low-latency network, Hypixel’s users noticed no change in gamer experience — even during an active volumetric DDoS attack.
During the attack campaign, Cloudflare automatically detected and mitigated over 5,000 DDoS attacks: 53% were ACK floods, 39% were UDP-based attacks and 8% SYN floods.
“We had several attacks of well over 620 Gbps with no impact at all on our players. Their gaming experience remained uninterrupted and fast, thanks to Cloudflare Magic Transit.” – Simon Collins-Laflamme, CEO, Hypixel Inc.
Hypixel’s journey with Cloudflare began with them employing Cloudflare Spectrum to help protect their gaming infrastructure against DDoS attacks. As their user base grew, they adopted additional Cloudflare products to bolster the robustness and resilience of all of their critical infrastructure. Today, they use multiple Cloudflare products including CDN, Rate Limiting, Spectrum, Argo Smart Routing, and Load Balancing to build and secure infrastructure that provides gamers around the world the real-time gaming experiences they need.
Get holistic protection against cyber attacks of any kind
DDoS attacks constitute just one facet of the many cyber threats organizations are facing today. As businesses shift to a Zero Trust approach, network and security buyers will face larger threats related to network access, and a continued surge in the frequency and sophistication of bot-related and ransomware attacks.
A key design tenet while building products at Cloudflare is integration. Cloudflare One is a solution that uses a Zero Trust security model to provide companies a better way to protect devices, data, and applications — and is deeply integrated with our existing platform of security and DDoS solutions.
In fact, Cloudflare offers an integrated solution that comprises an all-star cast featuring the following to name a few:
DDoS: LEADER in Forrester Wave™ for DDoS Mitigation Solutions, Q1 20211
WAF: Cloudflare is a CHALLENGER in the 2020 Gartner Magic Quadrant for Web Application Firewall (receiving the highest placement in the ‘Ability to Execute’)2
Zero Trust: Cloudflare is a LEADER in the Omdia Market Radar: Zero-Trust Access Report, 20203
Web protection: Innovation leader in the Global Holistic Web Protection Market for 2020 by Frost & Sullivan4
Cloudflare’s global (and growing) network is uniquely positioned to deliver DDoS protection and other security, performance, and reliability services with unparalleled scale, speed, and smarts.
As the election season has ramped down and the new Presidential Administration begins, we think it’s important to assess whether there are lessons we can draw from our experience helping to provide cybersecurity services for those involved in the 2020 U.S. elections.
Cloudflare built the Athenian Project – our project to provide free services to state and local election websites – around the idea that access to the authoritative voting information offered by state and local governments is key to a functioning democracy and that Cloudflare could play an important role in ensuring that election-related websites are protected from cyberattacks intended to disrupt that access. Although the most significant challenges in this election cycle fell outside the realm of cybersecurity, the 2020 election certainly validated the importance of having access to definitive sources of authoritative election information.
We were pleased that the robust cybersecurity preparations we saw for the 2020 U.S. election appeared to be successful. From the Cloudflare perspective, we had the opportunity to witness firsthand the benefits of having access to free cybersecurity services provided to organizations that promote accurate voting information and election results, state and local governments conducting elections, and federal U.S candidates running for office. As we protect many entities in the election space, we have the ability to identify, learn and analyze attack trends targeted at these sites that provide authoritative election information. We hope that we will continue to be able to assist researchers, policymakers and security experts looking to support best practices to protect the integrity of the electoral process.
Supporting free and fair elections
Many state and local governments bolstered their security postures ahead of the 2020 elections. There have been partnerships between governments, organizations, and private companies assisting election officials with the tools and expertise on best ways to secure the democratic process. Additionally, the spread of COVID-19 has prompted unprecedented challenges on how citizens can vote safely and securely.
Before the 2020 U.S. election, we detailed much of the activity targeting those in the election space to prepare for election day. To the relief of security experts, there were no significant publicly reported cybersecurity incidents as Chris Krebs, Director of the Cybersecurity and Infrastructure Security Agency during the 2020 election described it as “just another Tuesday on the Internet.” On November 12, 2020, a joint statement from the leading election security organizations stated “The November 3rd election was the most secure in American history . . . [T]here is no evidence that any voting system deleted or lost votes, changed votes, or was in any way compromised.”
At Cloudflare, we had a team of over 50 employees monitoring and addressing any issues to ensure we were providing our highest level of support to those working in the election space. It is important to note that our services do not protect electronic voting boxes or ballot counters; instead, Cloudflare services provide protection to websites, applications, and APIs. But we do protect many websites that provide pertinent information on the electoral process in the United States. This includes a wide range of players in the election space that facilitate voter registration, provide information on polling places, and publish election results. Since the 2016 election, state and local government websites that provide information such as voter registration, polling places, and election results, which have been increasingly targeted with cyberattacks.
Protecting organizations in the election space with Project Galileo
We launched Project Galileo in 2014 to provide a free set of security services to a range of vulnerable groups on the Internet such as human rights organizations, journalists and social justice organizations. Under the project, we currently protect more than 1,400 organizations working in regions all over the world with many organizations that work towards providing accurate voting information, tackling voter suppression, providing resources on voting rights and publishing election results. Cloudflare works with a variety of different types of non-governmental entities under Project Galileo, but we generally put them into two groups: participants, who are granted the benefits of Project Galileo, and partners, who work with us to identify other organizations who might be worth supporting. Our partners are typically larger civil society organizations and high profile NGOs, who work with entities who might benefit from our services and decide who should receive Cloudflare protections under the project.
Many of these organizations need cybersecurity protections well before election day. Belmont University is a private, four-year university located in Nashville, Tennessee. Shortly after the University was selected to be the site of the third and final 2020 U.S. Presidential Debate, the University reached out to Cloudflare asking for assistance. As part of the support for the debate, Belmont launched a new website to provide a centralized space for volunteers, media, and the community to prepare and organize the debate.
The project was quickly accepted to Project Galileo and we worked with Paul Chenoweth, Web Programming Service Manager for Belmont University to tackle concerns over server capacity, visitor traffic, site security, and analytics. Chenoweth explains, “We faced a number of web site challenges in 2008 when the university hosted the Town Hall Presidential Debate and with a totally new set of conditions in 2020, we did not know what to expect. We were worried about our site being taken down by malicious actors but also by unpredictable surges in traffic to the site. The Cloudflare team helped us create firewall rules, lock down our origin, and provided support during the Presidential debate.” Due to the spread of COVID-19, the debate website was the primary source of information for media registration, volunteer applications, and the event calendar for more than 40 themed virtual education events for the community. Overall, the university saw a 5x increase in traffic and blocked more than 80,000 malicious HTTP requests targeting their site.
Read stories from these organizations and Project Galileo here.
Under Project Galileo, we provide powerful cybersecurity tools to assist organizations such as Vote America, U.S. Vote Foundation, Decision Desk HQ, and many more working in the election space to identify and mitigate attacks targeting their web infrastructure. Along with protection from malicious DDoS attacks, our services also help with large influxes of unexpected traffic as organizations tend to see traffic spikes during voter registration deadlines. During the months leading up to elections, many of these organizations provided up to date information on the changing voting processes due to COVID-19. During the ballot count, many organizations posted election results online as state and local governments began reporting official numbers.
Many of the election-related organizations under Project Galileo allow you to register to vote, view the status of your voting ballot, and much more. States often hold their state and presidential primaries on different dates with the earliest primaries for 2020 held in March with 24 states and June with 23 states. When looking at cyberattacks against election organizations during the elections, the Cloudflare WAF blocked more than 10 million attacks in 2020. We can see that the WAF mitigated a majority of attacks during these two months, as many states held elections and voter registration deadlines.
Protecting election websites with the Athenian Project
In 2017, we launched the Athenian Project to provide our highest level of service to U.S. state and local governments running elections. This includes county board of election websites, Secretaries of State, and many smaller municipalities that register citizens to vote and publish election results. Under the Athenian Project, we protect more than 275 election entities in 30 states. In the past year, we onboarded more than 100 government election sites in preparation for the November 3rd election.
Read stories from state and local governments protected under the Athenian project here.
During the month leading up to elections, we had a team of engineers ready to assist state and local governments looking for help protecting their websites from cyberattacks. We onboarded Solano County in California, who engaged with our team on the best way to secure their election resources as we approached November 3rd. “The right to a free and fair election is one of the most basic civil rights we enjoy as Americans; it is a right upon which many of our foundational civil rights depend. Creating the conditions for transparent, clear, and truthful communications about the process and outcomes of elections is crucial to maintain the public trust in our electoral process”, saysTim Flanagan, Chief Information Officer for Solano County. In a few hours, we onboarded the county to Cloudflare and implemented best-practices tailored for election entities that use our services under the Athenian Project. “Cloudflare’s services added additional layers of security to our web presence that raised confidence in our ability to assure County’s residents that our election results were trustworthy.”
Starting in November, we saw traffic to government election sites increase as many people looked for polling places or how to contact local election officials. We also saw those traffic spikes after election day, as many election websites post periodic updates as the counting of ballots ensues. We reported many of these traffic spikes in the Election Dashboard with Cloudflare Radar.
For cyberattacks targeting government election websites, we found a majority of attacks before election day and primarily in September with about 50 million HTTPS requests blocked by the web application firewall.
From November 4 to November 11, the WAF mitigated 16,304,656 malicious requests to sites under the Athenian Project. During this time, many state and local governments were counting ballots and posting election results to their websites. A majority of attacks were blocked by the managed ruleset in the WAF – a set of rules curated by Cloudflare engineers to block against common vulnerabilities – including SQLi, cross-site scripting and cross-site forgery requests. These are not sophisticated attacks that we see, but hackers looking for vulnerabilities to access or modify sensitive information. For example, file inclusion is an attack targeting web applications to upload malware to steal or modify the content of the site.
Protecting Political Campaigns in 2020
In January 2020, we launched Cloudflare for Campaigns, a suite of free security services to federal campaigns with our partnership with Defending Digital Campaigns. During the course of the year, we onboarded 75 campaigns ranging from House, Senate, and Presidential candidates running for election in 2020. At Cloudflare, we have a range of campaigns that use our services ranging from free up to our Enterprise level plan. Overall, we protected more than 450 candidate sites running for federal office in 2020.
In 2020, the average number of attacks on U.S. campaign websites on Cloudflare per month was about 13 million. When comparing attacks against political campaigns and government election sites, we saw more DDoS attacks rather than hackers trying to exploit website vulnerabilities. As depicted below, campaigns used Cloudflare’s layer 7 DDoS protection that automatically monitors and mitigates large DDoS attacks, alongside rate-limiting to mitigate malicious traffic. For election websites, it’s clear that hackers tried to exploit common website vulnerabilities that were blocked by the WAF and firewall rules, with the goal of gaining access to internal systems rather than make the site inaccessible like we see in DDoS attacks.
Lessons learned and how we move forward
We learned a lot from preparing for the 2020 U.S. election while engaging with those in the election space and learned to be flexible in the face of the unexpected. We learned that COVID-19 had impacted many of these groups at a disportionate rate. For example, organizations that work in promoting online voter registration were well suited for the move to online that we found ourselves in during COVID-19. For political candidates, they had to adapt to moving campaign events and outreach to an online environment rather than the traditional campaign operations of door-knocking and large fundraising events. This move online meant that campaigns needed to pay more attention to digital risks.
We also learned as we approached the November election that the election space involves a range of players. Protecting elections requires not only working with governments to secure their websites for the unexpected, but also working with campaigns and non-profit organizations who work on election-related issues. We appreciated the fact that Cloudflare has many different projects that support a range of players working in promoting trust in the electoral process, giving us the flexibility to protect them. Many of these players need different levels of support and assistance with how to properly protect their web infrastructure from cyberattacks, and having a range of projects offering a different level of plans and support, helped us in finding the best way to protect them. We were able to provide a free set of services to a wide range of players each with separate goals but a common mission: providing authoritative information to build trust in the electoral process.
Both the awareness of the importance of election security and election security itself has improved since the 2016 election. We have seen the benefits of sharing information across many partners, organizations, and local players. To help prepare state and local governments for elections, we conducted webinars and security tunings sessions for many of these election players. In the case of state and local governments we protect under the Athenian Project, as we conducted more security training, we saw many participants recommend others in their state to ensure they were protected as well. For example, a week before the general election, the Wisconsin Election Commission sent an election security reminder with resources on how to mitigate a DDoS attack with Cloudflare to county and municipal clerks across Wisconsin.
At Cloudflare, we worked with a variety of government agencies to share threat information that we saw targeted against these participants. Days before the November 3rd election, we were invited to the last meeting conducted by the Cybersecurity and Infrastructure Security Agency to share threats data we had seen against government election websites and how they could be mitigated to more than 200 general election stakeholders, including counties across the United States.
Weeks after the election, I spoke with Stacy Mahaney, the Chief Information Officer at the Missouri Secretary of State, which is currently protected under the Athenian Project. His comment aptly summarized Cloudflare’s security practices. “Security is like an onion. Every layer of security that you add protects against various layers of attack or exposure. We were able to add layers to our security defenses with Cloudflare. The more layers you add, the more difficult it is for attackers to succeed in making voters question the trust of the democratic process that we work to protect every day.” Information security is about prevention and detection and is a continual process that involves monitoring, training, and threat analysis. By adding more layers including tools such as a web application firewall, 2FA, SSL encryption, authentication protocols, and security awareness training, it makes it more difficult for hackers to penetrate through the security layers.
Although cybersecurity experts concluded that the 2020 election was one of the safest in the history of elections, the work is not done yet. Not only will future U.S. election cycles begin again soon, but election security is a global concern that benefits from the involvement of experienced players with appropriate expertise. The longer we engage with those working with those in the election space, the more we learn the best ways to protect their web infrastructure and internal teams. We look forward to continuing our work to protect resources in the voting process and help build trust in democratic institutions.
DDoS attack trends in the final quarter of 2020 defied norms in many ways. For the first time in 2020, Cloudflare observed an increase in the number of large DDoS attacks. Specifically, the number of attacks over 500Mbps and 50K pps saw a massive uptick.
In addition, attack vectors continued to evolve, with protocol-based attacks seeing a 3-10x increase compared to the prior quarter. Attackers were also more persistent than ever — nearly 9% of all attacks observed between October and December lasted more than 24 hours.
Below are additional noteworthy observations from the fourth quarter of 2020, which the rest of this blog explores in greater detail.
Number of attacks: For the first time in 2020, the total number of attacks observed in Q4 decreased compared to the prior quarter.
Attack duration: 73% of all attacks observed lasted under an hour, a decrease from 88% in Q3.
Attack vectors: While SYN, ACK, and RST floods continued to be the dominant attack vectors deployed, attacks over NetBIOS saw a whopping 5400% increase, followed by those over ISAKMP and SPSS.
Global DDoS activity: Our data centers in Mauritius, Romania, and Brunei recorded the highest percentages of DDoS activity relative to non-attack traffic.
Additional attack tactics:Ransom DDoS (RDDoS) attacks continue to target organizations around the world as criminal groups attempt to extort a ransom in the form of Bitcoin under a threat of a DDoS attack.
Number of attacks
For the first time in 2020, the total number of network layer DDoS attacks we observed decreased compared to the previous quarter. Q4 constituted 15% of all attacks observed in 2020, compared to Q3’s 48%. In fact, the total number of attacks in Q4 was less than that seen in the month of September alone by a whopping 60%. On a monthly basis, December was Q4’s busiest month for attackers.
Attack rates
There are different ways of measuring an L3/4 DDoS attack’s size. One is the volume of traffic it delivers, or its ‘bit rate’ (measured in gigabits-per-second). Another is the number of packets it delivers, or its ‘packet rate’ (measured in packets-per-second). Attacks with high bit rates attempt to saturate last-mile network links of the target, and attacks with high packet rates attempt to overwhelm routers or other in-line hardware devices.
In Q4, as in previous quarters, the majority of attacks were quite small — under 1 Gbps and 1M pps, specifically. This trend is not surprising, since most attacks are launched by amateur attackers using tools that are easy to use and cost a few dollars at most. Small attacks may also serve as a smokescreen to distract security teams from other kinds of cyberattacks, or to test a network’s existing defense mechanisms.
However, the overall popularity of small attacks didn’t tell the whole story in Q4. Attacks over 500Mbps and 50K pps constituted a larger percentage of total attacks than they did in previous quarters. In fact, the number of attacks over 100 Gbps increased by 10x from Q3, and those over 10M pps increased by 3.6x.
One unique large attack Cloudflare observed was an ACK flood DoS attack that was automatically detected and mitigated by Cloudflare’s systems. What was unique about this attack was not the max packet rate, but the attack method that appears to have been borrowed from the world of acoustics.
As can be seen in the graph above, the attack’s packet rate followed a wave-shaped pattern for over 19 hours. It seems as though the attacker was inspired by an acoustics concept called beat. For this reason, we codenamed this attack “Beat”. In acoustics, a beat is a term that is used to describe an interference of two different wave frequencies. You can read more about the Beat attack in our blog post: Beat – An Acoustics Inspired DDoS Attack
Whether packet intensive or bit intensive, the increase in large DDoS attacks is a disturbing trend. It indicates that attackers are getting more brazen, and are using tools that allow them to launch larger attacks. What’s worse, often larger attacks have implications to not just target the network, but also intermediary service providers that serve the target network downstream.
Attack Duration
73% of attacks in Q4 ‘20 lasted for under an hour. On the other end of the spectrum, nearly 9% of attacks lasted over 24 hrs (compared to a mere 1.5% in Q3 ’20). This increase reinforces the need for a real-time, always-on defense system to protect against attacks of every size and duration.
Attack vectors
An ‘attack vector’ is a term used to describe the attack method. The most popular method, SYN floods, constituted nearly 42% of all attacks observed in Q3, followed by ACK, RST, and UDP-based DDoS attacks. This is relatively consistent with observations from previous quarters. However, ACK attacks jumped from ninth place in Q3 to second place — a 13x increase quarter-over-quarter— dethroning RST attacks from second place.
Top emerging threats
While TCP based attacks like SYN and RST floods remain popular, UDP-protocol specific attacks such as NetBIOS and ISAKMP-based DDoS attacks are seeing an explosion compared to the prior quarter.
NetBIOS is a protocol that allows applications on separate machines to communicate and access shared resources over a local area network, and ISAKMP is a protocol used to establish Security Associations (SAs) and cryptographic keys when setting up an IPsec VPN connection (IPsec uses the Internet Key Exchange (IKE) protocol to ensure secure connections and will authenticate and encrypt packets of data sent over an Internet Protocol (IP) network.)
Cloudflare continues to see protocol based attacks — and indeed, multi-vector attacks — deployed to attempt to bring networks down. As the complexity of attacks elevates, adequate DDoS protection needs to be put in place to keep organizations secure and online at all times.
Global DDoS activity
To understand where these attacks come from, we look at the Cloudflare edge network data centers where the traffic was ingested, rather than the location of the source IP. The reason? When attackers launch L3/4 attacks, they can spoof the source IP address in order to obfuscate their attack’s source.
In this report, we also measure the attack traffic observed at a Cloudflare data center relative to the non-attack traffic observed at the same data center for geo-based distribution. This gives us more accuracy in our endeavor to pinpoint geographic locations that are observing more threats than others. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities, in more than 100 countries around the world.
Looking at Q4 metrics, we observed interesting insights — our data centers in Mauritius, Romania, and Brunei recorded the highest percentages of attack traffic relative to non-attack traffic. Specifically, between 4.4% and 4.9% of all traffic in those countries came from DDoS attacks. Another way of saying this is that almost 5 out of every 100 bytes was part of attack traffic. These observations indicate increased botnet activities in those countries.
What might explain the comparatively high incidence of DDoS attacks in these countries? While it’s impossible to say for sure, here are some possibilities for the top two countries on the list:
Mauritius – In August 2020, a state of environmental emergency was declared in Mauritius after a ship carrying nearly 4,000 tons of fuel cracked its hull. The oil spill ignited anti-government protests calling for the resignation of the prime minister. Since then, the government has suspended the parliament twice, and has also been accused of suppressing local media and independent reporting covering the incident. Even five months after, following a series of human-rights scandals, the protests continue. The events in Mauritius may be linked to the increased DDoS activity.
Source: wikipedia
Romania – Two events may be behind the increased DDoS activity in Romania. Romania recently held parliamentary elections which ended on December 6, 2020. In addition, the EU announced on December 9th that Romania will host their new cyber security research hub, the European Cybersecurity Industrial, Technology and Research Competence Centre (ECCC). Another possible explanation is that Romania is the country with the cheapest super-fast broadband Internet in the world — making it easier for anyone to launch volumetric attacks from within Romania.
🎉#Bucharest to be the host of the future EU #Cyber Centre🎉Hub for high tech and innovation, featuring a thriving digital ecosystem, dynamic and young, Romania’s capital will take this task in a responsible and dedicated manner, to the benefit of the entire European Union👏👏👏
Ransom-based attacks continue to plague organizations
In our previous quarterly DDoS report, we noted a rise in extortion and ransom-based DDoS (RDDoS) attacks around the world. In a RDDoS attack, a malicious party threatens a person or organization with a cyberattack that could knock their networks, websites, or applications offline for a period of time, unless the person or organization pays a ransom. You can read more about RDDoS attacks here.
In Q4 ‘20, this disturbing trend continued. Organizations large and small came to Cloudflare asking for help in keeping their network infrastructure online while they figured out how to respond to ransom notes. Read this story of what a Fortune Global 500 company did when they received a ransom note, and about their recommendations for organizations.
Cloudflare continues to closely monitor this trend. If you receive a threat:
Do not panic — we recommend you to not pay the ransom: Paying the ransom only encourages bad actors and finances illegal activities — and there’s no guarantee attackers won’t attack your network anyway.
Notify local law enforcement: They will also likely request a copy of the ransom letter that you received.
Contact Cloudflare: We can help ensure your website and network infrastructure are safeguarded from these ransom attacks.
Cloudflare DDoS Protection
Cloudflare provides comprehensive L3-L7 DDoS protection. In 2017, we pioneered the elimination of the industry standard surge pricing for DDoS attacks, providing customers with unmetered and unlimited DDoS protection. Since then, we’ve onboarded thousands of customers of all sizes — including Wikimedia, Panasonic, and Discord — that use Cloudflare to protect and accelerate their Internet properties. Why do they choose Cloudflare? Three main reasons:
1. No scrubs Cloudflare doesn’t operate scrubbing centers as we believe that the scrubbing center model is a flawed approach to DDoS protection. Scrubbing centers cause delays and cost too much to build and run. What’s more, DDoS attacks are asymmetric — attackers have more available bandwidth than a single scrubbing center will ever be able to handle.
Cloudflare’s network is architected so that every machine in every data center performs DDoS mitigation. Doing this at the edge is the only way to mitigate at scale without impacting performance. Our Anycast-based architecture makes our capacity equivalent to our DDoS scrubbing capacity, the largest in the market at 51 Tbps. This means Cloudflare detects and mitigates DDoS attacks close to the source of attack. Better yet, Cloudflare’s global threat intelligence acts like an immune system for the Internet — employing our machine learning models to learn from and mitigate attacks against any customer to protect them all.
2. It’s about time Most organizations are in some stage of their journey from on-prem to the cloud. The threat landscape, functional requirements, and scale of business applications are evolving faster than ever before, and the volume and sophistication of network attacks are already straining the defensive capabilities of even the most advanced enterprises. One concern many enterprises have when adopting the cloud is added latency for applications. Most cloud-based DDoS protection services rely on specialized data centers aka “scrubbing centers” for DDoS mitigation. Backhauling traffic to those data centers can add significant latency depending on its location relative to the destination server.
This problem compounds when an organization uses different providers for different networking functions. When traffic must hop from provider to provider, latency can be measured in hundreds of milliseconds.
Cloudflare’s distributed geographical presence ensures that attacks are globally detected and mitigated in under 3 seconds on average — making it one of the fastest in the industry.
3. It’s not just about DDoS DDoS attacks constitute just one facet of the many cyber threats organizations are facing today. As businesses shift to a Zero Trust approach, network and security buyers will face larger threats related to network access, and a continued surge in the frequency and sophistication of bot-related attacks.
A key design tenet while building products at Cloudflare is integration. Cloudflare One is a solution that uses a Zero Trust security model to provide companies a better way to protect devices, data, and applications — and is deeply integrated with our existing platform of security and DDoS solutions.
To learn more about Cloudflare’s DDoS solution contact us or get started today by signing up on our dashboard.
DDoS attacks are surging — both in frequency and sophistication. After doubling from Q1 to Q2, the total number of network layer attacks observed in Q3 doubled again — resulting in a 4x increase in number compared to the pre-COVID levels in the first quarter. Cloudflare also observed more attack vectors deployed than ever — in fact, while SYN, RST, and UDP floods continue to dominate the landscape, we saw an explosion in protocol specific attacks such as mDNS, Memcached, and Jenkins DoS attacks.
Here are other key network layer DDoS trends we observed in Q3:
Majority of the attacks are under 500 Mbps and 1 Mpps — both still suffice to cause service disruptions
We continue to see a majority of attacks be under 1 hr in duration
Ransom-driven DDoS attacks (RDDoS) are on the rise as groups claiming to be Fancy Bear, Cozy Bear and the Lazarus Group extort organizations around the world. As of this writing, the ransom campaign is still ongoing. See a special note on this below.
Number of attacks
The total number of L3/4 DDoS attacks we observe on our network continues to increase substantially, as indicated in the graph below. All in all, Q3 saw over 56% of all attacks this year — double that of Q2, and four times that of Q1. In addition, the number of attacks per month increased throughout the quarter.
While September witnessed the largest number of attacks overall, August saw the most large attacks (over 500Mbps). Ninety-one percent of large attacks in Q3 took place in that month—while monthly distribution of other attack sizes was far more even.
While the total number of attacks between 200-300 Gbps decreased in September, we saw more global attacks on our network in Q3. This suggests the increase in the use of distributed botnets to launch attacks. In fact, in early July, Cloudflare witnessed one of the largest-ever attacks on our network — generated by Moobot, a Mirai-based botnet. The attack peaked at 654 Mbps and originated from 18,705 unique IP addresses, each believed to be a Moobot-infected IoT device. The attack campaign lasted nearly 10 days, but the customer was protected by Cloudflare, so they observed no downtime or service degradation.
Attack size (bit rate and packet rate)
There are different ways of measuring a L3/4 DDoS attack’s size. One is the volume of traffic it delivers, measured as the bit rate (specifically, Gigabits-per-second). Another is the number of packets it delivers, measured as the packet rate (specifically, packets-per-second). Attacks with high bit rates attempt to saturate the Internet link, and attacks with high packet rates attempt to overwhelm the routers or other in-line hardware devices.
In Q3, most of the attacks we observed were smaller in size. In fact, over 87% of all attacks were under 1 Gbps. This represents a significant increase from Q2, when roughly 52% of attacks were that small. Note that, even ‘small’ attacks of under 500 Mbps are many times sufficient to create major disruptions for Internet properties that are not protected by a Cloud based DDoS protection service. Many organizations have uplinks provided by their ISPs that are far less than 1 Gbps. Assuming their public facing network interface also serves legitimate traffic, you can see how even these ‘small’ DDoS attacks can easily take down Internet properties.
This trend holds true for attack packet rates. In Q3, 47% of attacks were under 50k pps — compared to just 19% in Q2.
Smaller attacks can indicate that amateur attackers may be behind the attacks — using tools easily available to generate attacks on exposed IPs/ networks. Alternatively, small attacks may serve as a smokescreen to distract security teams from other kinds of cyberattacks that might be taking place simultaneously.
Attack duration
In terms of length, very short attacks were the most common attack type observed in Q3, accounting for nearly 88% of all attacks. This observation is in line with our prior reports — in general, Layer 3/4 DDoS attacks are getting shorter in duration.
Short burst attacks may attempt to cause damage without being detected by DDoS detection systems. DDoS services that rely on manual analysis and mitigation may prove to be useless against these types of attacks because they are over before the analyst even identifies the attack traffic.
Alternatively, the use of short attacks may be used to probe the cyber defenses of the target. Load-testing tools and automated DDoS tools, that are widely available on the dark web, can generate short bursts of, say, a SYN flood, and then following up with another short attack using an alternate attack vector. This allows attackers to understand the security posture of their targets before they decide to potentially launch larger attacks at larger rates and longer durations – which come at a cost.
In other cases, attackers generate small DDoS attacks as proof and warning to the target organization of the attacker’s ability to cause real damage later on. It’s often followed by a ransom note to the target organization, demanding payment so as to avoid suffering an attack that could more thoroughly cripple network infrastructure.
Whatever their motivation, DDoS attacks of any size or duration are not going away anytime soon. Even short DDoS attacks cause harm, and having an automated real-time defense mechanism in place is critical for any online business.
Attack vectors
SYN floods constituted nearly 65% of all attacks observed in Q3, followed by RST floods and UDP floods in second and third places. This is relatively consistent with observations from previous quarters, highlighting the DDoS attack vector of choice by attackers.
While TCP based attacks like SYN and RST floods continue to be popular, UDP-protocol specific attacks such as mDNS, Memcached, and Jenkins are seeing an explosion compared to the prior quarter.
Multicast DNS (mDNS) is a UDP-based protocol that is used in local networks for service/device discovery. Vulnerable mDNS servers respond to unicast queries originating outside of the local network, which are ‘spoofed’ (altered) with the victim’s source address. This results in amplification attacks. In Q3, we noticed an explosion of mDNS attacks — specifically, we saw a 2,680% increase compared to the previous quarter.
This was followed by Memcached and Jenkins attacks. Memcached is a Key Value database. Requests can be made over the UDP protocol with a spoofed source address of the target. The size of the Value stored in the requested Key will affect the amplification factor, resulting in a DDoS amplification attack. Similarly, Jenkins, NTP, Ubiquity and the other UDP based protocols have seen a dramatic increase over the quarter due to its UDP stateless nature. A vulnerability in the older version (Jenkins 2.218 and earlier) aided the launch of DDoS attacks. This vulnerability was fixed in Jenkins 2.219 by disabling UDP multicast/ broadcast messages by default. However there are still many vulnerable and exposed devices that run UDP based services which are being harnessed to generate volumetric amplification attacks.
Attack by country
Looking at country-based distribution, the United States observed the most number of L3/4 DDoS attacks, followed by Germany and Australia. Note that when analyzing L3/4 DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations where the traffic was ingested, and not by the location of the source IP. The reason is when attackers launch L3/4 attacks they can spoof the source IP address in order to obfuscate the attack source. If we were to derive the country based on a spoofed source IP, we would get a spoofed country. Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of Cloudflare’s data center in which the attack was observed. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities around the world.
Africa
Asia Pacific & Oceania
Europe
Middle East
North America
South America
United States
A note on recent ransom-driven DDoS attacks
Over the past months, Cloudflare has observed another disturbing trend — a rise in extortion and ransom-based DDoS (RDDoS) attacks targeting organizations around the world. While RDDoS threats do not always result in an actual attack, the cases seen in recent months show that attacker groups are willing to carry out the threat, launching large scale DDoS attacks that can overwhelm organizations that lack adequate protection. In some cases, the initial teaser attack may be sufficient to cause impact if not protected by a Cloud based DDoS protection service.
In a RDDoS attack, a malicious party threatens a person or organization with a cyberattack that could knock their networks, websites, or applications offline for a period of time, unless the person or organization pays a ransom. You can read more about RDDoS attacks here.
Entities claiming to be Fancy Bear, Cozy Bear, and Lazarus have been threatening to launch DDoS attacks against organizations’ websites and network infrastructure unless a ransom is paid before a given deadline. Additionally, an initial ‘teaser’ DDoS attack is usually launched as a form of demonstration before parallel to the ransom email. The demonstration attack is typically a UDP reflection attack using a variety of protocols, lasting roughly 30 minutes in duration (or less).
What to do if you receive a threat:
Do not panic and we recommend you to not pay the ransom: Paying the ransom only encourages bad actors, finances illegal activities —and there’s no guarantee that they won’t attack your network now or later.
Notify local law enforcement: They will also likely request a copy of the ransom letter that you received.
Contact Cloudflare: We can help ensure your website and network infrastructure are safeguarded from these ransom attacks.
Cloudflare DDoS protection is different
On-prem hardware/cloud-scrubbing centers can’t address the challenges of modern volumetric DDoS attacks. Appliances are easily overwhelmed by large DDoS attacks, Internet links quickly saturate, and rerouting traffic to cloud scrubbing centers introduces unacceptable latency penalties. Our cloud-native, always-on, automated DDoS protection approach solves problems that traditional cloud signaling approaches were originally created to address.
Cloudflare’s mission is to help build a better Internet, which grounds our DDoS approach and is why in 2017, we pioneered unmetered DDoS mitigation for all of our customers on all plans including the free plan. We are able to provide this level of protection because every server on our network can detect & block threats, enabling us to absorb attacks of any size/kind, with no latency impact. This architecture gives us unparalleled advantages compared to any other vendor.
51 Tbps of DDoS mitigation capacity and under 3 sec TTM: Every data center in Cloudflare’s network detects and mitigates DDoS attacks. Once an attack is identified, the Cloudflare’s local data center mitigation system (dosd) generates and applies a dynamically crafted rule with a real-time signature — and mitigates attacks in under 3 seconds globally on average. This 3-second Time To Mitigate (TTM) is one of the fastest in the industry. Firewall rules and “proactive”/static configurations take effect immediately.
Fast performance included: Cloudflare is architected so that customers do not incur a latency penalty as a result of attacks. We deliver DDoS protection from every Cloudflare data center (instead of legacy scrubbing centers or on-premise hardware boxes) which allows us to mitigate attacks closest to the source. Cloudflare analyzes traffic out-of-path ensuring that our DDoS mitigation solution doesn’t add any latency to legitimate traffic. The rule is applied at the most optimal place in the Linux stack for a cost efficient mitigation, ensuring no performance penalty.
Global Threat Intelligence: Like an immune system, our network learns from/mitigates attacks against any customer to protect them all. With threat intelligence (TI), it automatically blocks attacks and is employed in customer facing features (Bot Fight mode, Firewall Rules & Security Level). Users create custom rules to mitigate attacks based on traffic attribute filters, threat & bot scores generated using ML models (protecting against bots/botnets/DDoS).
Now that we’re a long way through 2020, let’s take a look at automated traffic, which makes up almost 40% of total Internet traffic.
This blog post is a high-level overview of bot traffic on Cloudflare’s network. Cloudflare offers a comprehensive Bot Management tool for Enterprise customers, along with an effective free tool called Bot Fight Mode. Because of the tremendous amount of traffic that flows through our network each day, Cloudflare is in a unique position to analyze global bot trends.
In this post, we will cover the basics of bot traffic and distinguish between automated requests and other human requests (What Is A Bot?). Then, we’ll move on to a global overview of bot traffic around the world (A RoboBird’s Eye View, A Bot Day and Bots All Over The World), and dive into North American traffic (A Look into North American Traffic). Lastly, we’ll finish with an overview of how the coronavirus pandemic affected global traffic, and we’ll take a deeper look at European traffic (Bots During COVID-19 In Europe).
On average, Cloudflare processes 18 million HTTP requests every second. This is a great opportunity to understand how bots shape the Internet, how much infrastructure is dedicated to these automated requests, and why our customers need a great bot management solution.
What Is A Bot?
Cloudflare groups traffic into four bot-related categories:
Our goal is to stop malicious and unwanted bots from harming our customers, while giving customers the opportunity to control how other automated traffic is managed.
We label each request that comes into Cloudflare with a “bot score” 1 through 99, where a lower score means that a request probably came from a bot. A higher score means that a request probably came from a human. This score is available in our Firewall, logs, and Workers, giving customers the flexibility to act on any score.
Cloudflare also maintains a challenge platform that customers can choose to deploy on suspected bots. You’ll recognize these as CAPTCHA challenges or JavaScript challenges. In fact, having the score available in Firewall Rules means that customers can take any action they choose. This platform can be used for mitigation, ensuring that unwanted traffic is stopped in its tracks.
To learn more about how Bot Management interacts with our firewall, check out our support page.
We track successes and failures during these challenges, which ultimately allows us to improve our detection systems. Assuming that our challenges are solvable by humans, effective detections should have low solve rates, given that they are usually presented to bots.
Verified bots are registered in an internal verified bot directory. These good bots power search engines and monitoring tools. Good bots enable our customers’ web pages to be found by search engines, for example.
For known non-verified bots (such as a scraper using a simple curl library), we keep a similar directory that is managed by our heuristics engine. If not otherwise verified, we consider requests caught by this engine to be definitely automated.
Our machine learning engine provides another way to identify potential bots. This engine identifies requests with a high probability of automation and marks them as likely automated. This detection mechanism benefits from models built on data from our global network.
If a request is not marked as automated, we mark it as likely human and pass along the bot score from our machine learning system.
We also have a behavioral analysis engine and a JavaScript detections engine. You can learn more about these systems by checking out Alex Bocharov’s previous post on Cloudflare Bot Management.
The two bot definitions for automated traffic are somewhat complementary. Requests caught by heuristic detections will not count towards machine learning detections. Requests that are reliably caught by our machine learning detections won’t need to be registered in our known heuristics bot directory. Because of this, we combine these two together when we discuss “automated traffic” in general.
A RoboBird’s Eye View
Data from this piece comes from information about Cloudflare’s customers, analyzed between January 15, 2020 and July 31, 2020.
First, let’s get a basic understanding of the traffic on our network.
Figure 1.1 Traffic type on Cloudflare’s network.
Figure 1.1 has a global breakdown regarding classification; 60.6% of traffic is likely human, 19.3% is likely automated, 18.1% is definitely automated and only 2.1% is from verified bots. In total, 39.5% of requests we score come from some kind of bot.
A Bot Day
Regular traffic fluctuates throughout the day. Do bots follow suit? Let’s check. Figure 2.1 represents traffic deviation from the average hourly traffic. An increase of 10% would mean that the hour is 10% busier than the average hour (measuring requests per hour). We include the total overall traffic in this chart to serve as a comparison to other types of traffic.
Figure 2.1 Hourly traffic as a deviation from the average hour.Figure 2.2 Bot classification over an average day.
We can clearly see a difference between human traffic and bot traffic. Human traffic varies heavily, but predictably, throughout the day. We can see a 15% decrease in human traffic early in the day, between midnight and 05:00 UTC, corresponding to the end of business hours in the Americas, and up to a 25% increase during business hours, 14:00 to 17:00 UTC, where traffic is highest. Conversely, bot traffic is more consistent. Slow hours still see a smaller drop than overall traffic, and busy hours are less busy. The difference between good and bad bots is also apparent: good bots are even more consistent, with small fluctuations in hourly traffic.
But why would this happen? A large portion of bots, good and bad, perform the same task across the Internet. Bad bots may be scraping websites or looking to infect unprotected machines, and they will do this with little intervention from human operators. Good bots could be doing some of these operations, but less frequently and in a more targeted fashion. A good bot scraping a website may be doing so to add it to a search engine, while a bad bot will do the same thing at a much higher rate, for other reasons.
A lot of bots follow business hours. For example, sneaker bots—focused on nabbing exclusive items from sneaker stores—will naturally be active when new products launch.
This difference in volume does not mean that our classifications are affected: our scores remain consistent throughout the day, as Figure 2.1 shows.
Figure 2.3 Daily traffic as a deviation from the average day. Grouped by day of week.Figure 2.4 Bot classification over an average week.
We can also see that good bots don’t take weekends off. Weekdays and weekends have fairly marked differences for most traffic, but good bots keep a consistent schedule. Whereas a typical weekday is slightly above average, we can see a drop of about 4% in overall traffic. This does not fully apply to verified bots, which only see a small 1% drop in traffic.
Bots All Over The World
Now that we’ve taken a look at global traffic, let’s dig a little deeper.
Different regions have distinct traffic landscapes regarding automated traffic.
Figure 3.1Traffic type by region.
Figure 3.1 breaks down traffic by region, letting us peek into where each type of traffic comes from. North America stands out as a major automated traffic source; over 50% of definitely automated traffic comes from there, and they also contribute almost 80% of all verified bot traffic. Europe makes up the second largest chunk of traffic, followed by Asia.
Figure 3.2 Traffic classification within each region.
Looking at regional breakdown of traffic in Figure 3.2, we can see just how much North American traffic is automated, well above the global average.
A Look into North American Traffic
As the largest source of automated traffic, North America deserves a closer look.
First, we’ll start with a breakdown of each country.
Figure 3.3 Percentage of traffic within North America.
Most of our requests in North America come from just three countries—the United States, Canada and Mexico. These account for 98% of all requests from North America, 97% of all requests from likely human sources and 100% of requests from verified bots. The United States alone accounts for 88% of total requests, 82% of requests from likely human sources, 96% of requests from definitely automated sources, 88% of requests from likely automated traffic sources and 98% of requests from verified bot.
However, this alone does not mean that the United States has an unusual amount of activity. These countries have a combined population of roughly 497 million people. The United States accounts for 66.5% of that, Mexico 25.9% and Canada 7.6%. With this context, we can see that the United States is overrepresented in terms of raw requests, but underrepresented in terms of how much of that traffic is likely to be human. Conversely, Canadian traffic is more likely to be human.
Let’s take another look at each country.
Figure 3.4 Percentage of traffic within each country.
Over half of the traffic from the United States is automated in some way, which is a clear departure from trends in Mexico and Canada.
American Bots
So far, we’ve seen how much the United States contributes to automated traffic. If we want to go deeper, a good place to start is by understanding how these bots get online. We can do this by examining the networks from which the traffic originates. Networks are identified by Autonomous System Numbers, or ASNs. These form the backbone of the Internet infrastructure.
Think of these as Internet Service Providers, but facing inward towards the network instead of outward towards end consumers. ISPs like Comcast and Verizon are examples of residential ASNs, where we expect mostly human traffic. Cloud providers such as Google and Amazon are also ASNs, but targeted towards cloud services. We expect most of these requests to be automated in some way.
Looking at traffic on the ASN level is important because we can identify cloud-based traffic, or traffic using residential proxies, among others.
Let’s take a look at which ASNs are associated with visitors in the United States. We’ll restrict ourselves to “eyeball” traffic, which is the term we use for requests coming from site visitors.
Figure 4.1 Top ASN in the United States.
From figure 4.1 we can clearly see the impact that cloud services have on traffic; 11.5% of all eyeball traffic comes from Amazon and Google.
Figure 4.2 Top ASN in the United States for verified bot traffic.
Verified bots operate in a different landscape, coming from cloud providers such as Amazon, Google, Microsoft, Advanced Hosting and Wowrack.
Figure 4.3 Top ASN in the United States for likely and definitely automated traffic.
Automated traffic has a variety of ASNs. Cloud providers such as Amazon, Google and Microsoft make up the 30% of automated traffic. Comcast also makes up a significant portion of traffic at 4.8%, indicating that some bots come from residential services.
Bots During COVID-19 In Europe
Lockdowns and limits on public events came as a consequence of the ongoing coronavirus pandemic. Many people have been working from home, and even those who do not have this option are using the Internet in new ways. Overall, this has meant that Cloudflare’s network has grown tremendously.
But how does this impact bot traffic? First let’s get an idea of how it impacted traffic in general. Countries were impacted by the virus at different times, so we expect to see differences, right?
Figure 5.1Total traffic across all regions.
Figure 5.1 has just the traffic increase. Globally, we are seeing an average increase of 10%, while North America saw an increase of over 40% compared to the beginning of the year. Some regions did not change much, such as Africa and Asia, while others, such as Europe saw an increased period, but has since normalized to previous levels.
Let’s look at a few countries, so we can understand what this looks like.
Figure 5.2 Daily traffic evolution for Italy, the United Kingdom and Portugal, overlaid with Europe.
Figure 5.2 shows daily traffic relative to January 15, when data collection started. For comparison, we have overall European traffic, and three selected countries: Italy, the United Kingdom and Portugal. Italy was picked because it was one of the first countries in Europe to face the worst of the coronavirus and enact lockdown measures. The United Kingdom took another strategy, with an initial focus on herd immunity, and enacted measures later than the others. Portugal is somewhere in between, locking down later than Italy, in slightly different circumstances.
At the beginning of the year, traffic kept stable and fluctuations kept in line with the European average. As lockdown measures began, traffic increased. Italy was first out of these countries, rising a few weeks before the others, and keeping well above average. Eventually, all countries saw a growth in traffic, followed by a stabilization. Italy seems to have adjusted to a normal, with its growth in line with the European average. Portugal has also stabilized, but with busier weekdays. Conversely, the United Kingdom showed no signs of stopping, exceeding a growth of 40% compared to the beginning of the year.
Figure 5.3 Daily definitely automated traffic evolution for Italy, the United Kingdom and Portugal, overlaid with Europe.
Definitely automated traffic did not have that much of a pronounced variation. Italian traffic kept steady throughout, and Portugal had a rather large increase. The biggest one, however, was the United Kingdom, which tripled its initial count.
Figure 5.4 Verified bot traffic evolution for Italy, the United Kingdom and Portugal, overlaid with Europe.
Verified bot traffic is steady, except in Italy, with a massive increase between March and May. What could be the cause of this? Are these a few zones, getting a massive number of requests?
Figure 5.5 Verified bot traffic in Italy for the top 10 000 zones, relative to January 15th 2020.
Well, no. If we only examine the top 10,000 zones (by total verified bot requests), we can still see a massive increase in traffic for other zones. So, what’s happening?
Let’s look at user agents. We can separate the top 10 user agents during the bump, and see how they evolve over time.
Figure 5.6 Verified bot traffic in Italy for the top 10 user agents, relative to January 15th 2020.
We can see that these 10 user agents are responsible for the majority of verified traffic coming from Italy.
Figure 5.7 Verified bot traffic in Italy for the top user agent, relative to January 15 2020.
In fact, most of this increase is from a single user agent. This instance of Google image proxy anonymizes image requests from Gmail, which explains its popularity.
Where does this increase come from? Did this bot suddenly appear and disappear?
Not quite. One thing to keep in mind when dealing with bots is that they cross borders easily. As a proxy service, this bot is making calls on behalf of the end user – people opening emails. These requests will originate from a data center, which can be anywhere in the world. To see this in action, let’s take a look at traffic for this bot in a few select countries.
Figure 5.8. Countries of origin for GoogleImageProxy.
We can see that the global average barely budges. It appears that Google may be moving image proxy traffic between data centers and during the period we observed above that traffic was coming from Italy.
Summary
With Cloudflare’s global reach, we’re in a position to understand how bots behave.
The first half of 2020 saw a massive increase in web traffic of around 35% since the beginning of the year, driven by the ongoing coronavirus pandemic, and some bots have taken advantage of it.
We explained how bot management works for our customers, and how we distinguish between likely automated and human traffic.
We showed an overview of how much of our global traffic is automated, and how bots change their behavior throughout the day and the week. Notably, 39.4% of all traffic Cloudflare processes comes from a suspected automated source.
A regional overview of automated traffic lets us know which regions were the source of traffic from likely automated agents. North America, Europe and Asia were the primary sources of traffic, and also of automated traffic in particular.
We then focused on North America, where the majority of automated traffic originates. The United States alone accounted for the majority of requests, over half of which come from automated sources.
To explore this further, we briefly dived into ASN traffic in the United States, so we could see where these requests were coming from. ASNs like Comcast and AT&T were the top ASNs for overall traffic, but unsurprisingly, data centers like Google and Amazon AWS were the main drivers of automated traffic.
Finally, we examined how the coronavirus has impacted traffic in Europe, with a deeper dive on Italian traffic. This led to some interesting insights on verified bot traffic, which saw a massive increase in Italy for a few months.
This post is a small peek into bot management at Cloudflare. In the future, we hope to expand this series of blog posts on bot management, exposing even more insights about bots on the Internet.
The collective thoughts of the interwebz
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.
