Earlier this summer, Cloudflare’s autonomous edge DDoS protection systems automatically detected and mitigated a 17.2 million request-per-second (rps) DDoS attack, an attack almost three times larger than any previous one that we’re aware of. For perspective on how large this attack was: Cloudflare serves over 25 million HTTP requests per second on average. This refers to the average rate of legitimate traffic in 2021 Q2. So peaking at 17.2 million rps, this attack reached 68% of our Q2 average rps rate of legitimate HTTP traffic.
Automated DDoS mitigation with Cloudflare’s autonomous edge
This attack, along with the additional attacks provided in the next sections, were automatically detected and mitigated by our autonomous edge DDoS protection systems. The system is powered by our very own denial of service daemon (dosd). Dosd is a home-grown software-defined daemon. A unique dosd instance runs in every server in each one of our data centers around the world. Each dosd instance independently analyzes traffic samples out-of-path. Analyzing traffic out-of-path allows us to scan asynchronously for DDoS attacks without causing latency and impacting performance. DDoS findings are also shared between the various dosd instances within a data center, as a form of proactive threat intelligence sharing.
Once an attack is detected, our systems generate a mitigation rule with a real-time signature that matches the attack patterns. The rule is propagated to the most optimal location in the tech stack. As an example, a volumetric HTTP DDoS attack may be blocked at L4 inside the Linux iptables firewall instead of at L7 inside the L7 reverse proxy which runs in the user space. Mitigating lower in the stack, e.g. dropping the packets at L4 instead of responding with a 403 error page in L7, is more cost-efficient. It reduces our edge CPU consumption and intra-data center bandwidth utilization — thus helping us mitigate large attacks at scale without impacting performance.
This autonomous approach, along with our network’s global scale and reliability, allow us to mitigate attacks that reach 68% of our average per-second-rate, and higher, without requiring any manual mitigation by Cloudflare personnel, nor causing any performance degradation.
The resurgence of Mirai and new powerful botnets
This attack was launched by a powerful botnet, targeting a Cloudflare customer in the financial industry. Within seconds, the botnet bombarded the Cloudflare edge with over 330 million attack requests.
The attack traffic originated from more than 20,000 bots in 125 countries around the world. Based on the bots’ source IP addresses, almost 15% of the attack originated from Indonesia and another 17% from India and Brazil combined. Indicating that there may be many malware infected devices in those countries.
Volumetric attacks increase
This 17.2 million rps attack is the largest HTTP DDoS attack that Cloudflare has ever seen to date and almost three times the size of any other reported HTTP DDoS attack. This specific botnet, however, has been seen at least twice over the past few weeks. Just last week it also targeted a different Cloudflare customer, a hosting provider, with an HTTP DDoS attack that peaked just below 8 million rps.
Two weeks before, a Mirai-variant botnet launched over a dozen UDP and TCP based DDoS attacks that peaked multiple times above 1 Tbps, with a max peak of approximately 1.2 Tbps. And while the first HTTP attacks targeted Cloudflare customers on the WAF/CDN service, the 1+ Tbps network-layer attacks targeted Cloudflare customers on the Magic Transit and Spectrum services. One of these targets was a major APAC-based Internet services, telecommunications and hosting provider. The other was a gaming company. In all cases, the attacks were automatically detected and mitigated without human intervention.
The Mirai botnet started with roughly 30K bots and slowly shrinked to approximately 28K. However, despite losing bots from its fleet, the botnet was still able to generate impressive volumes of attack traffic for short periods. In some cases, each burst lasted only a few seconds.
These attacks join the increase in Mirari-based DDoS attacks that we’ve observed on our network over the past weeks. In July alone, L3/4 Mirai attacks increased by 88% and L7 attacks by 9%. Additionally, based on the current August per-day average of the Mirai attacks, we can expect L7 Mirai DDoS attacks and other similar botnet attacks to increase by 185% and L3/4 attacks by 71% by the end of the month.
Back to the Mirai
Mirai, which means ‘future’ in Japanese, is a codename for malware that was first discovered in 2016 by MalwareMustDie, a non-profit security research workgroup. The malware spreads by infecting Linux-operated devices such as security cameras and routers. It then self-propagates by searching for open Telnet ports 23 and 2323. Once found, it then attempts to gain access to vulnerable devices by brute forcing known credentials such as factory default usernames and passwords. Later variants of Mirai also took advantage of zero-day exploits in routers and other devices. Once infected, the devices will monitor a Command & Control (C2) server for instructions on which target to attack.
How to protect your home and business
While the majority of attacks are small and short, we continue to see these types of volumetric attacks emerging more often. It’s important to note that these volumetric short burst attacks can be especially dangerous for legacy DDoS protection systems or organizations without active, always-on cloud-based protection.
Furthermore, while the short duration may say something about the botnet’s capability to deliver sustained levels of traffic over time, it can be challenging or impossible for humans to react to it in time. In such cases, the attack is over before a security engineer even has time to analyze the traffic or activate their stand-by DDoS protection system. These types of attacks highlight the need for automated, always-on protection.
How to protect your business and Internet properties
Follow our preventive best practices, to ensure that both your Cloudflare settings and your origin server settings are optimized. As an example, make sure that you allow only traffic from Cloudflare’s IP range. Ideally, ask your upstream Internet Service Provider (ISP) to apply an access control list (ACL), otherwise, attackers may target your servers’ IP addresses directly and bypass your protection.
Recommendations on how to protect your home and IoT appliances
Change the default username and password of any device that is connected to the Internet such as smart cameras and routers. This will reduce the risk that malware such as Mirai can gain access to your router and IoT devices.
Protect your home against malware with Cloudflare for Families. Cloudflare for Families is a free service that automatically blocks traffic from your home to malicious websites and malware communication.
Recent weeks have witnessed massive ransomware and ransom DDoS (Distributed Denial of Service) attack campaigns that interrupted aspects of critical infrastructure around the world, including one of the largest petroleum pipeline system operators, and one of the world’s biggest meat processing companies. Earlier this quarter, more than 200 organizations across Belgium, including the government and parliament websites and other services, were also DDoS’d.
And when most of the United States were celebrating Independence Day on July 4, hundreds of US companies were hit by a ransomware attack demanding 70 million USD in Bitcoin. Attackers known to be affiliated with REvil, a Russian ransomware group, exploited multiple previously unknown vulnerabilities in IT management software. The targets included schools, small public-sector bodies, travel and leisure organizations, and credit unions, to name a few. While the threat of ransomware and ransom DDoS is not new (read our posts on ransomware and ransom DDoS from 2021 Q1), the latest attacks on Internet properties ranging from wineries, professional sports teams, ferry services and hospitals has brought them from just being background noise to front page headlines affecting our day-to-day lives. In fact, recent attacks have propelled ransomware and DDoS to the top of US President Biden’s national security agenda.
The DDoS attack trends observed over Cloudflare’s network in 2021 Q2 paint a picture that reflects the overall global cyber threat landscape. Here are some highlights.
Over 11% of our surveyed customers who were targeted by a DDoS attack reported receiving a threat or ransom letter threatening in advance, in the first six months of this year. Emergency onboarding of customers under an active DDoS attack increased by 41.8% in 2021 H1 compared to 2020 H2.
HTTP DDoS attacks targeting government administration/public sector websites increased by 491%, making it the second most targeted industry after Consumer Services whose DDoS activity increased by 684% QoQ.
China remains the country with the most DDoS activity originating from within their borders — 7 out of every 1,000 HTTP requests originating from China were part of an HTTP DDoS attack targeting websites, and more than 3 out of every 100 bytes that were ingested in our data centers in China were part of a network-layer DDoS attack.
Emerging threats included amplification DDoS attacks that abused the Quote of the Day (QOTD) protocol which increased by 123% QoQ. Additionally, as the adoption of QUIC protocol continues to increase, so do attacks over QUIC — registering a whopping 109% QoQ surge in 2021 Q2. The number of network-layer DDoS attacks in the range of 10-100 Gbps increased by 21.4% QoQ. One customer that was attacked is Hypixel, an American gaming company. Hypixel remained online with no downtime and no performance penalties to their gamer users, even when under an active DDoS attack campaign larger than 620 Gbps. Read their story here.
To view all DDoS attack insights across all regions and industries worldwide, visit Cloudflare’s interactive Radar DDoS dashboard.
Application-layer DDoS attacks
Application-layer DDoS attacks, specifically HTTP DDoS attacks, are attacks that usually aim to disrupt an HTTP server by making it unable to process legitimate user requests. If a server is bombarded with more requests than it can process, the server will drop legitimate requests or even crash resulting in performance penalties or a denial of service event for legitimate users.
DDoS activity per market industry
When we analyze attacks, we calculate the ‘DDoS activity’ rate, which is the percentage of attack traffic out of the total traffic (attack + clean). This allows us to normalize the data points and avoid biases towards, for example, a larger data center that naturally handles more traffic and therefore also more attacks.
In 2021 Q2, Consumer Services was the most targeted industry followed by Government Administration and Marketing & Advertising.
DDoS activity per source country
To understand the origin of the HTTP attacks we observed over Cloudflare’s network, we look at the source IP address of the client generating the attack HTTP requests. Unlike network-layer attacks, source IPs cannot be spoofed in HTTP attacks. A high DDoS activity rate in a given country indicates large botnets operating from within.
China and the US remain in the first and second places, respectively, regarding the percentage of DDoS activity originating from within their territories. In China, more than 7 out of every 1,000 HTTP requests were part of an HTTP DDoS attack, while in the US almost 5 out of 1,000 HTTP requests were part of an attack.
DDoS activity per target country
In order to identify which countries the targets of the DDoS attacks resided in, we break down the DDoS activity by our customers’ billing countries. Note that Cloudflare does not charge for attack traffic and has pioneered providing unmetered and unlimited DDoS protection since 2017. By cross-referencing the attack data with our customers’ billing country, we can identify which countries were attacked the most.
Data observed in 2021 Q2 suggest that organizations in the US and China were the most targeted by HTTP DDoS attacks. In fact, one out of every 200 HTTP requests destined to US-based organizations was part of a DDoS attack.
Network-layer DDoS attacks
While application-layer attacks strike the application (Layer 7 of the OSI model) running the service end users are trying to access, network-layer attacks target network infrastructure (such as in-line routers and other network servers) and the Internet link itself.
Distribution of attacks by size (packet rate and bit rate)
There are different ways of measuring the size of a L3/4 DDoS attack. One is the volume of traffic it delivers, measured as the bit rate (specifically, gigabits-per-second). Another is the number of packets it delivers, measured as the packet rate (specifically, packets-per-second). Attacks with high bit rates attempt to saturate the Internet link, while attacks with high packet rates attempt to overwhelm the servers, routers or other in-line hardware appliances.
The distribution of attacks by their size (in bit rate) and month is shown below. As observed in the chart, all attacks over 300 Gbps were observed in the month of June.
In terms of bit rate, attacks under 500 Mbps constituted a majority of all DDoS attacks observed in 2021 Q2.
Similarly, looking from the lens of packet rate, nearly 94% of attacks were under 50K pps. Even though attacks from 1-10M pps constituted only 1% of all DDoS attacks observed, this number is 27.5% higher than that observed in the previous quarter, suggesting that larger attacks are not diminishing either — but rather increasing.
Note that while attacks under 500 Mbps and 50K pps might seem ‘small’ compared to other headline-making large attacks, they are often sufficient to create major disruptions for Internet properties that are not protected by an always-on, automated cloud-based DDoS protection service. Moreso, many organisations have uplinks provided by their service providers with a bandwidth capacity smaller than 1 Gbps. Assuming their public-facing network interface also serves legitimate traffic, DDoS attacks smaller than 500 Mbps are often capable of taking down exposed Internet properties.
Distribution by attack duration
Cloudflare continues to see a large percentage of DDoS attacks that last under an hour. In Q2, over 97% of all DDoS attacks lasted less than an hour.
Short burst attacks may attempt to cause damage without being detected by DDoS detection systems. DDoS services that rely on manual analysis and mitigation may prove to be useless against these types of attacks because they are over before the analyst even identifies the attack traffic.
Alternatively, the use of short attacks may be used to probe the cyber defenses of the target. Load-testing tools and automated DDoS tools, that are widely available on the dark web, can generate short bursts of a SYN flood, for example, and then follow up with another short attack using a different attack vector. This allows attackers to understand the security posture of their targets before they decide to launch larger attacks at larger rates and longer durations — which come at a cost.
In other cases, attackers generate small DDoS attacks as proof and warning to the target organization of the attacker’s ability to cause real damage later on. It’s often followed by a ransom email to the target organization, demanding payment to avoid suffering an attack that could more thoroughly cripple network infrastructure.
This highlights the need for an always on, automated DDoS protection approach. DDoS protection services that rely on manual re-routing, analysis and mitigation may prove to be useless against these types of attacks because they are over before the analyst can even identify the attack traffic.
Distribution of attacks by attack vectors
An attack vector is the term used to describe the method that the attacker utilizes in their attempt to cause a denial of service event.
As observed in previous quarters, attacks utilizing SYN floods and UDP-based protocols remain the most popular methods by attackers.
What is a SYN flood attack? It’s a DDoS attack that exploits the very foundation of the TCP protocol. A stateful TCP connection between a client and a server begins with a 3-way TCP handshake. The client sends an initial connection request packet with a synchronize flag (SYN). The server responds with a packet that contains a synchronized acknowledgment flag (SYN-ACK). Finally, the client responds with an acknowledgment (ACK) packet. At this point, a connection is established and data can be exchanged until the connection is closed. This stateful process can be abused by attackers to cause denial of service events.
By repeatedly sending SYN packets, the attacker attempts to overwhelm a server or the router’s connection table that tracks the state of TCP connections. The router replies with a SYN-ACK packet, allocates a certain amount of memory for each given connection, and falsely waits for the client to respond with the final ACK. Given a sufficient number of connections occupying the router’s memory, the router is unable to allocate further memory for legitimate clients, causing the router to crash or preventing it from handling legitimate client connections, i.e., a denial of service event.
Emerging threats included amplification DDoS attacks that abuse the Quote of the Day (QOTD) service which increased by 123% QoQ. QOTD was defined in RFC-865 (1983) and can be sent over either the UDP or TCP protocols. It was originally designed for debugging and as a measurement tool, with no specific syntax for the quote. The RFC does however recommend the use of ASCII characters and to limit the length to 512 characters.
Furthermore, we’ve seen a 107% increase QoQ in UDP Portmap and Echo attacks — all of which are really old attack vectors. This may indicate attackers digging up old methods and attack tools to try and overcome protection systems. As we’ve seen in previous quarters, the adoption of the QUIC protocol continues to increase. Consequently, so do attacks over QUIC, or more specifically floods and amplification attacks of non-QUIC traffic in places where we’d expect to see QUIC traffic. In 2021 Q2, these types of attacks increased by 109% QoQ. This continued trend may indicate that attackers are attempting to abuse the QUIC-designated ports and gateways into organizations’ networks — searching for vulnerabilities and security holes.
DDoS activity by Cloudflare data center country
In 2021 Q2, our data center in Haiti observed the largest percentage of network-layer DDoS attack traffic, followed by Brunei (almost 3 out of every 100 packets were part of an attack) and China.
Note that when analyzing network-layer DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations where the traffic was ingested, and not by the source IP. The reason for this is that, when attackers launch network-layer attacks, they can spoof the source IP address in order to obfuscate the attack source and introduce randomness into the attack properties, which may make it harder for simple DDoS protection systems to block the attack. Hence, if we were to derive the source country based on a spoofed source IP, we would get a spoofed country. Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of Cloudflare’s data center in which the attack was observed. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities around the world.
A note on ransomware and ransom DDoS — a growing global threat
The last few weeks have seen a resurgence of ransom-driven cyber threats: ransomware and ransom DDoS (RDDoS).
So what is ransomware and ransom DDoS, and how are they different?
Ransomware is malicious software that encrypts an organization’s systems and databases, rendering them inaccessible and unusable. Malware is usually introduced into an organization’s systems via phishing emails — tricking employees to click on a link or download a file. Once the malware is installed on the employee’s device, it encrypts the device and can propagate to the entire network of the organization’s servers and employee devices. The attacker will demand money, usually in the form of Bitcoin, in exchange for decrypting the organization’s systems and granting them access back to their systems.
Unlike a ransomware attack, a ransom DDoS attack does not encrypt a company’s systems; it aims to knock them offline if the ransom is not paid. What makes ransom DDoS attacks even more dangerous is that they do not require the attacker to gain access to a business’s internal systems to execute the attack. However, with a strong DDoS protection strategy in place, a ransom DDoS attack has little to no effect on businesses.
Ransomware and ransom DDoS threats are impacting most industries across the globe — the financial industry, transportation, oil and gas, consumer goods, and even education and healthcare.
Entities claiming to be ‘Fancy Lazarus’, ‘Fancy Bear’, ‘Lazarus Group’, and ‘REvil’ are once again launching ransomware and ransom-DDoS attacks against organizations’ websites and network infrastructure unless a ransom is paid before a given deadline. In the case of DDoS threats, prior to the ransom note, a small DDoS attack is usually launched as a form of demonstration. The demonstration attack is typically over UDP, lasting roughly 30-120 minutes.
The ransom note is typically sent to the common group email aliases of the company that are publicly available online such as [email protected], [email protected], [email protected], [email protected], [email protected], etc. In several cases, it has ended up in spam. In other cases, we’ve seen employees disregard the ransom note as spam, increasing the organization’s response time which resulted in further damage to their online properties.
Cloudflare’s recommendation for organizations that receive a threat or ransom note:
Do not panic, and we recommend you do not pay the ransom: Paying ransom only encourages and funds bad actors. There’s also no guarantee that you won’t be attacked again anyway.
Contact local law enforcement: Be ready to provide a copy of the ransom letter you received and any other logs or packet captures.
Activate an effective DDoS protection strategy: Cloud-based DDoS protection can be quickly onboarded in the event of an active threat, and with a team of security experts on your side, risks can be mitigated quickly and effectively.
Here’s a short video by Cloudflare CTO, John Graham-Cumming addressing the threat of ransom DDoS attacks.
Cloudflare protects Hypixel against a massive DDoS attack campaign
At Cloudflare, our teams have been exceptionally busy this past quarter rapidly onboarding (onto our Magic Transit service) a multitude of new and existing customers that have either received a ransom letter or were under an active DDoS attack.
One such customer is Hypixel Inc, the development studio behind the world’s largest Minecraft minigame server. With over 24M total unique logins to date and a world record 216,000+ concurrent players on PC, the Hypixel team works hard to add value to the experience of millions of players across the globe.
The gaming industry is often subject to some of the largest volumetric DDoS attacks — and as a marquee brand, Hypixel attracts more than its fair share. Uptime and high performance are fundamental to the functioning of Hypixel’s servers. Any perceived downtime or noticeable lag could result in an exodus of gamers.
When Hypixel was under a massive DDoS attack campaign, they turned to Cloudflare to extend their services with Cloudflare to include Magic Transit, Cloudflare’s BGP-based DDoS protection service for network infrastructure. After rapidly onboarding them overnight, Cloudflare was automatically able to detect and mitigate DDoS attacks targeting their network — several of which were well over 620 Gbps. The DDoS attack comprised mostly TCP floods and UDP amplification attacks. In the graph, the various colors represent the multiple Cloudflare systems that contribute to detecting and mitigating the multi-vector attack — emphasising the value of our multi-layered DDoS approach.
Even as attack patterns changed in real-time, Magic Transit shielded Hypixel’s network. In fact, because all their clean traffic routed over Cloudflare’s high performing low-latency network, Hypixel’s users noticed no change in gamer experience — even during an active volumetric DDoS attack.
During the attack campaign, Cloudflare automatically detected and mitigated over 5,000 DDoS attacks: 53% were ACK floods, 39% were UDP-based attacks and 8% SYN floods.
“We had several attacks of well over 620 Gbps with no impact at all on our players. Their gaming experience remained uninterrupted and fast, thanks to Cloudflare Magic Transit.” – Simon Collins-Laflamme, CEO, Hypixel Inc.
Hypixel’s journey with Cloudflare began with them employing Cloudflare Spectrum to help protect their gaming infrastructure against DDoS attacks. As their user base grew, they adopted additional Cloudflare products to bolster the robustness and resilience of all of their critical infrastructure. Today, they use multiple Cloudflare products including CDN, Rate Limiting, Spectrum, Argo Smart Routing, and Load Balancing to build and secure infrastructure that provides gamers around the world the real-time gaming experiences they need.
Get holistic protection against cyber attacks of any kind
DDoS attacks constitute just one facet of the many cyber threats organizations are facing today. As businesses shift to a Zero Trust approach, network and security buyers will face larger threats related to network access, and a continued surge in the frequency and sophistication of bot-related and ransomware attacks.
A key design tenet while building products at Cloudflare is integration. Cloudflare One is a solution that uses a Zero Trust security model to provide companies a better way to protect devices, data, and applications — and is deeply integrated with our existing platform of security and DDoS solutions.
In fact, Cloudflare offers an integrated solution that comprises an all-star cast featuring the following to name a few:
DDoS: LEADER in Forrester Wave™ for DDoS Mitigation Solutions, Q1 20211
WAF: Cloudflare is a CHALLENGER in the 2020 Gartner Magic Quadrant for Web Application Firewall (receiving the highest placement in the ‘Ability to Execute’)2
Zero Trust: Cloudflare is a LEADER in the Omdia Market Radar: Zero-Trust Access Report, 20203
Web protection: Innovation leader in the Global Holistic Web Protection Market for 2020 by Frost & Sullivan4
Cloudflare’s global (and growing) network is uniquely positioned to deliver DDoS protection and other security, performance, and reliability services with unparalleled scale, speed, and smarts.
As the election season has ramped down and the new Presidential Administration begins, we think it’s important to assess whether there are lessons we can draw from our experience helping to provide cybersecurity services for those involved in the 2020 U.S. elections.
Cloudflare built the Athenian Project – our project to provide free services to state and local election websites – around the idea that access to the authoritative voting information offered by state and local governments is key to a functioning democracy and that Cloudflare could play an important role in ensuring that election-related websites are protected from cyberattacks intended to disrupt that access. Although the most significant challenges in this election cycle fell outside the realm of cybersecurity, the 2020 election certainly validated the importance of having access to definitive sources of authoritative election information.
We were pleased that the robust cybersecurity preparations we saw for the 2020 U.S. election appeared to be successful. From the Cloudflare perspective, we had the opportunity to witness firsthand the benefits of having access to free cybersecurity services provided to organizations that promote accurate voting information and election results, state and local governments conducting elections, and federal U.S candidates running for office. As we protect many entities in the election space, we have the ability to identify, learn and analyze attack trends targeted at these sites that provide authoritative election information. We hope that we will continue to be able to assist researchers, policymakers and security experts looking to support best practices to protect the integrity of the electoral process.
Supporting free and fair elections
Many state and local governments bolstered their security postures ahead of the 2020 elections. There have been partnerships between governments, organizations, and private companies assisting election officials with the tools and expertise on best ways to secure the democratic process. Additionally, the spread of COVID-19 has prompted unprecedented challenges on how citizens can vote safely and securely.
Before the 2020 U.S. election, we detailed much of the activity targeting those in the election space to prepare for election day. To the relief of security experts, there were no significant publicly reported cybersecurity incidents as Chris Krebs, Director of the Cybersecurity and Infrastructure Security Agency during the 2020 election described it as “just another Tuesday on the Internet.” On November 12, 2020, a joint statement from the leading election security organizations stated “The November 3rd election was the most secure in American history . . . [T]here is no evidence that any voting system deleted or lost votes, changed votes, or was in any way compromised.”
At Cloudflare, we had a team of over 50 employees monitoring and addressing any issues to ensure we were providing our highest level of support to those working in the election space. It is important to note that our services do not protect electronic voting boxes or ballot counters; instead, Cloudflare services provide protection to websites, applications, and APIs. But we do protect many websites that provide pertinent information on the electoral process in the United States. This includes a wide range of players in the election space that facilitate voter registration, provide information on polling places, and publish election results. Since the 2016 election, state and local government websites that provide information such as voter registration, polling places, and election results, which have been increasingly targeted with cyberattacks.
Protecting organizations in the election space with Project Galileo
We launched Project Galileo in 2014 to provide a free set of security services to a range of vulnerable groups on the Internet such as human rights organizations, journalists and social justice organizations. Under the project, we currently protect more than 1,400 organizations working in regions all over the world with many organizations that work towards providing accurate voting information, tackling voter suppression, providing resources on voting rights and publishing election results. Cloudflare works with a variety of different types of non-governmental entities under Project Galileo, but we generally put them into two groups: participants, who are granted the benefits of Project Galileo, and partners, who work with us to identify other organizations who might be worth supporting. Our partners are typically larger civil society organizations and high profile NGOs, who work with entities who might benefit from our services and decide who should receive Cloudflare protections under the project.
Many of these organizations need cybersecurity protections well before election day. Belmont University is a private, four-year university located in Nashville, Tennessee. Shortly after the University was selected to be the site of the third and final 2020 U.S. Presidential Debate, the University reached out to Cloudflare asking for assistance. As part of the support for the debate, Belmont launched a new website to provide a centralized space for volunteers, media, and the community to prepare and organize the debate.
The project was quickly accepted to Project Galileo and we worked with Paul Chenoweth, Web Programming Service Manager for Belmont University to tackle concerns over server capacity, visitor traffic, site security, and analytics. Chenoweth explains, “We faced a number of web site challenges in 2008 when the university hosted the Town Hall Presidential Debate and with a totally new set of conditions in 2020, we did not know what to expect. We were worried about our site being taken down by malicious actors but also by unpredictable surges in traffic to the site. The Cloudflare team helped us create firewall rules, lock down our origin, and provided support during the Presidential debate.” Due to the spread of COVID-19, the debate website was the primary source of information for media registration, volunteer applications, and the event calendar for more than 40 themed virtual education events for the community. Overall, the university saw a 5x increase in traffic and blocked more than 80,000 malicious HTTP requests targeting their site.
Read stories from these organizations and Project Galileo here.
Under Project Galileo, we provide powerful cybersecurity tools to assist organizations such as Vote America, U.S. Vote Foundation, Decision Desk HQ, and many more working in the election space to identify and mitigate attacks targeting their web infrastructure. Along with protection from malicious DDoS attacks, our services also help with large influxes of unexpected traffic as organizations tend to see traffic spikes during voter registration deadlines. During the months leading up to elections, many of these organizations provided up to date information on the changing voting processes due to COVID-19. During the ballot count, many organizations posted election results online as state and local governments began reporting official numbers.
Many of the election-related organizations under Project Galileo allow you to register to vote, view the status of your voting ballot, and much more. States often hold their state and presidential primaries on different dates with the earliest primaries for 2020 held in March with 24 states and June with 23 states. When looking at cyberattacks against election organizations during the elections, the Cloudflare WAF blocked more than 10 million attacks in 2020. We can see that the WAF mitigated a majority of attacks during these two months, as many states held elections and voter registration deadlines.
Protecting election websites with the Athenian Project
In 2017, we launched the Athenian Project to provide our highest level of service to U.S. state and local governments running elections. This includes county board of election websites, Secretaries of State, and many smaller municipalities that register citizens to vote and publish election results. Under the Athenian Project, we protect more than 275 election entities in 30 states. In the past year, we onboarded more than 100 government election sites in preparation for the November 3rd election.
Read stories from state and local governments protected under the Athenian project here.
During the month leading up to elections, we had a team of engineers ready to assist state and local governments looking for help protecting their websites from cyberattacks. We onboarded Solano County in California, who engaged with our team on the best way to secure their election resources as we approached November 3rd. “The right to a free and fair election is one of the most basic civil rights we enjoy as Americans; it is a right upon which many of our foundational civil rights depend. Creating the conditions for transparent, clear, and truthful communications about the process and outcomes of elections is crucial to maintain the public trust in our electoral process”, saysTim Flanagan, Chief Information Officer for Solano County. In a few hours, we onboarded the county to Cloudflare and implemented best-practices tailored for election entities that use our services under the Athenian Project. “Cloudflare’s services added additional layers of security to our web presence that raised confidence in our ability to assure County’s residents that our election results were trustworthy.”
Starting in November, we saw traffic to government election sites increase as many people looked for polling places or how to contact local election officials. We also saw those traffic spikes after election day, as many election websites post periodic updates as the counting of ballots ensues. We reported many of these traffic spikes in the Election Dashboard with Cloudflare Radar.
For cyberattacks targeting government election websites, we found a majority of attacks before election day and primarily in September with about 50 million HTTPS requests blocked by the web application firewall.
From November 4 to November 11, the WAF mitigated 16,304,656 malicious requests to sites under the Athenian Project. During this time, many state and local governments were counting ballots and posting election results to their websites. A majority of attacks were blocked by the managed ruleset in the WAF – a set of rules curated by Cloudflare engineers to block against common vulnerabilities – including SQLi, cross-site scripting and cross-site forgery requests. These are not sophisticated attacks that we see, but hackers looking for vulnerabilities to access or modify sensitive information. For example, file inclusion is an attack targeting web applications to upload malware to steal or modify the content of the site.
Protecting Political Campaigns in 2020
In January 2020, we launched Cloudflare for Campaigns, a suite of free security services to federal campaigns with our partnership with Defending Digital Campaigns. During the course of the year, we onboarded 75 campaigns ranging from House, Senate, and Presidential candidates running for election in 2020. At Cloudflare, we have a range of campaigns that use our services ranging from free up to our Enterprise level plan. Overall, we protected more than 450 candidate sites running for federal office in 2020.
In 2020, the average number of attacks on U.S. campaign websites on Cloudflare per month was about 13 million. When comparing attacks against political campaigns and government election sites, we saw more DDoS attacks rather than hackers trying to exploit website vulnerabilities. As depicted below, campaigns used Cloudflare’s layer 7 DDoS protection that automatically monitors and mitigates large DDoS attacks, alongside rate-limiting to mitigate malicious traffic. For election websites, it’s clear that hackers tried to exploit common website vulnerabilities that were blocked by the WAF and firewall rules, with the goal of gaining access to internal systems rather than make the site inaccessible like we see in DDoS attacks.
Lessons learned and how we move forward
We learned a lot from preparing for the 2020 U.S. election while engaging with those in the election space and learned to be flexible in the face of the unexpected. We learned that COVID-19 had impacted many of these groups at a disportionate rate. For example, organizations that work in promoting online voter registration were well suited for the move to online that we found ourselves in during COVID-19. For political candidates, they had to adapt to moving campaign events and outreach to an online environment rather than the traditional campaign operations of door-knocking and large fundraising events. This move online meant that campaigns needed to pay more attention to digital risks.
We also learned as we approached the November election that the election space involves a range of players. Protecting elections requires not only working with governments to secure their websites for the unexpected, but also working with campaigns and non-profit organizations who work on election-related issues. We appreciated the fact that Cloudflare has many different projects that support a range of players working in promoting trust in the electoral process, giving us the flexibility to protect them. Many of these players need different levels of support and assistance with how to properly protect their web infrastructure from cyberattacks, and having a range of projects offering a different level of plans and support, helped us in finding the best way to protect them. We were able to provide a free set of services to a wide range of players each with separate goals but a common mission: providing authoritative information to build trust in the electoral process.
Both the awareness of the importance of election security and election security itself has improved since the 2016 election. We have seen the benefits of sharing information across many partners, organizations, and local players. To help prepare state and local governments for elections, we conducted webinars and security tunings sessions for many of these election players. In the case of state and local governments we protect under the Athenian Project, as we conducted more security training, we saw many participants recommend others in their state to ensure they were protected as well. For example, a week before the general election, the Wisconsin Election Commission sent an election security reminder with resources on how to mitigate a DDoS attack with Cloudflare to county and municipal clerks across Wisconsin.
At Cloudflare, we worked with a variety of government agencies to share threat information that we saw targeted against these participants. Days before the November 3rd election, we were invited to the last meeting conducted by the Cybersecurity and Infrastructure Security Agency to share threats data we had seen against government election websites and how they could be mitigated to more than 200 general election stakeholders, including counties across the United States.
Weeks after the election, I spoke with Stacy Mahaney, the Chief Information Officer at the Missouri Secretary of State, which is currently protected under the Athenian Project. His comment aptly summarized Cloudflare’s security practices. “Security is like an onion. Every layer of security that you add protects against various layers of attack or exposure. We were able to add layers to our security defenses with Cloudflare. The more layers you add, the more difficult it is for attackers to succeed in making voters question the trust of the democratic process that we work to protect every day.” Information security is about prevention and detection and is a continual process that involves monitoring, training, and threat analysis. By adding more layers including tools such as a web application firewall, 2FA, SSL encryption, authentication protocols, and security awareness training, it makes it more difficult for hackers to penetrate through the security layers.
Although cybersecurity experts concluded that the 2020 election was one of the safest in the history of elections, the work is not done yet. Not only will future U.S. election cycles begin again soon, but election security is a global concern that benefits from the involvement of experienced players with appropriate expertise. The longer we engage with those working with those in the election space, the more we learn the best ways to protect their web infrastructure and internal teams. We look forward to continuing our work to protect resources in the voting process and help build trust in democratic institutions.
DDoS attack trends in the final quarter of 2020 defied norms in many ways. For the first time in 2020, Cloudflare observed an increase in the number of large DDoS attacks. Specifically, the number of attacks over 500Mbps and 50K pps saw a massive uptick.
In addition, attack vectors continued to evolve, with protocol-based attacks seeing a 3-10x increase compared to the prior quarter. Attackers were also more persistent than ever — nearly 9% of all attacks observed between October and December lasted more than 24 hours.
Below are additional noteworthy observations from the fourth quarter of 2020, which the rest of this blog explores in greater detail.
Number of attacks: For the first time in 2020, the total number of attacks observed in Q4 decreased compared to the prior quarter.
Attack duration: 73% of all attacks observed lasted under an hour, a decrease from 88% in Q3.
Attack vectors: While SYN, ACK, and RST floods continued to be the dominant attack vectors deployed, attacks over NetBIOS saw a whopping 5400% increase, followed by those over ISAKMP and SPSS.
Global DDoS activity: Our data centers in Mauritius, Romania, and Brunei recorded the highest percentages of DDoS activity relative to non-attack traffic.
Additional attack tactics:Ransom DDoS (RDDoS) attacks continue to target organizations around the world as criminal groups attempt to extort a ransom in the form of Bitcoin under a threat of a DDoS attack.
Number of attacks
For the first time in 2020, the total number of network layer DDoS attacks we observed decreased compared to the previous quarter. Q4 constituted 15% of all attacks observed in 2020, compared to Q3’s 48%. In fact, the total number of attacks in Q4 was less than that seen in the month of September alone by a whopping 60%. On a monthly basis, December was Q4’s busiest month for attackers.
There are different ways of measuring an L3/4 DDoS attack’s size. One is the volume of traffic it delivers, or its ‘bit rate’ (measured in gigabits-per-second). Another is the number of packets it delivers, or its ‘packet rate’ (measured in packets-per-second). Attacks with high bit rates attempt to saturate last-mile network links of the target, and attacks with high packet rates attempt to overwhelm routers or other in-line hardware devices.
In Q4, as in previous quarters, the majority of attacks were quite small — under 1 Gbps and 1M pps, specifically. This trend is not surprising, since most attacks are launched by amateur attackers using tools that are easy to use and cost a few dollars at most. Small attacks may also serve as a smokescreen to distract security teams from other kinds of cyberattacks, or to test a network’s existing defense mechanisms.
However, the overall popularity of small attacks didn’t tell the whole story in Q4. Attacks over 500Mbps and 50K pps constituted a larger percentage of total attacks than they did in previous quarters. In fact, the number of attacks over 100 Gbps increased by 10x from Q3, and those over 10M pps increased by 3.6x.
One unique large attack Cloudflare observed was an ACK flood DoS attack that was automatically detected and mitigated by Cloudflare’s systems. What was unique about this attack was not the max packet rate, but the attack method that appears to have been borrowed from the world of acoustics.
As can be seen in the graph above, the attack’s packet rate followed a wave-shaped pattern for over 19 hours. It seems as though the attacker was inspired by an acoustics concept called beat. For this reason, we codenamed this attack “Beat”. In acoustics, a beat is a term that is used to describe an interference of two different wave frequencies. You can read more about the Beat attack in our blog post: Beat – An Acoustics Inspired DDoS Attack
Whether packet intensive or bit intensive, the increase in large DDoS attacks is a disturbing trend. It indicates that attackers are getting more brazen, and are using tools that allow them to launch larger attacks. What’s worse, often larger attacks have implications to not just target the network, but also intermediary service providers that serve the target network downstream.
73% of attacks in Q4 ‘20 lasted for under an hour. On the other end of the spectrum, nearly 9% of attacks lasted over 24 hrs (compared to a mere 1.5% in Q3 ’20). This increase reinforces the need for a real-time, always-on defense system to protect against attacks of every size and duration.
An ‘attack vector’ is a term used to describe the attack method. The most popular method, SYN floods, constituted nearly 42% of all attacks observed in Q3, followed by ACK, RST, and UDP-based DDoS attacks. This is relatively consistent with observations from previous quarters. However, ACK attacks jumped from ninth place in Q3 to second place — a 13x increase quarter-over-quarter— dethroning RST attacks from second place.
Top emerging threats
While TCP based attacks like SYN and RST floods remain popular, UDP-protocol specific attacks such as NetBIOS and ISAKMP-based DDoS attacks are seeing an explosion compared to the prior quarter.
NetBIOS is a protocol that allows applications on separate machines to communicate and access shared resources over a local area network, and ISAKMP is a protocol used to establish Security Associations (SAs) and cryptographic keys when setting up an IPsec VPN connection (IPsec uses the Internet Key Exchange (IKE) protocol to ensure secure connections and will authenticate and encrypt packets of data sent over an Internet Protocol (IP) network.)
Cloudflare continues to see protocol based attacks — and indeed, multi-vector attacks — deployed to attempt to bring networks down. As the complexity of attacks elevates, adequate DDoS protection needs to be put in place to keep organizations secure and online at all times.
Global DDoS activity
To understand where these attacks come from, we look at the Cloudflare edge network data centers where the traffic was ingested, rather than the location of the source IP. The reason? When attackers launch L3/4 attacks, they can spoof the source IP address in order to obfuscate their attack’s source.
In this report, we also measure the attack traffic observed at a Cloudflare data center relative to the non-attack traffic observed at the same data center for geo-based distribution. This gives us more accuracy in our endeavor to pinpoint geographic locations that are observing more threats than others. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities, in more than 100 countries around the world.
Looking at Q4 metrics, we observed interesting insights — our data centers in Mauritius, Romania, and Brunei recorded the highest percentages of attack traffic relative to non-attack traffic. Specifically, between 4.4% and 4.9% of all traffic in those countries came from DDoS attacks. Another way of saying this is that almost 5 out of every 100 bytes was part of attack traffic. These observations indicate increased botnet activities in those countries.
What might explain the comparatively high incidence of DDoS attacks in these countries? While it’s impossible to say for sure, here are some possibilities for the top two countries on the list:
Mauritius – In August 2020, a state of environmental emergency was declared in Mauritius after a ship carrying nearly 4,000 tons of fuel cracked its hull. The oil spill ignited anti-government protests calling for the resignation of the prime minister. Since then, the government has suspended the parliament twice, and has also been accused of suppressing local media and independent reporting covering the incident. Even five months after, following a series of human-rights scandals, the protests continue. The events in Mauritius may be linked to the increased DDoS activity.
🎉#Bucharest to be the host of the future EU #Cyber Centre🎉Hub for high tech and innovation, featuring a thriving digital ecosystem, dynamic and young, Romania’s capital will take this task in a responsible and dedicated manner, to the benefit of the entire European Union👏👏👏
Ransom-based attacks continue to plague organizations
In our previous quarterly DDoS report, we noted a rise in extortion and ransom-based DDoS (RDDoS) attacks around the world. In a RDDoS attack, a malicious party threatens a person or organization with a cyberattack that could knock their networks, websites, or applications offline for a period of time, unless the person or organization pays a ransom. You can read more about RDDoS attacks here.
In Q4 ‘20, this disturbing trend continued. Organizations large and small came to Cloudflare asking for help in keeping their network infrastructure online while they figured out how to respond to ransom notes. Read this story of what a Fortune Global 500 company did when they received a ransom note, and about their recommendations for organizations.
Cloudflare continues to closely monitor this trend. If you receive a threat:
Do not panic — we recommend you to not pay the ransom: Paying the ransom only encourages bad actors and finances illegal activities — and there’s no guarantee attackers won’t attack your network anyway.
Notify local law enforcement: They will also likely request a copy of the ransom letter that you received.
Contact Cloudflare: We can help ensure your website and network infrastructure are safeguarded from these ransom attacks.
Cloudflare DDoS Protection
Cloudflare provides comprehensive L3-L7 DDoS protection. In 2017, we pioneered the elimination of the industry standard surge pricing for DDoS attacks, providing customers with unmetered and unlimited DDoS protection. Since then, we’ve onboarded thousands of customers of all sizes — including Wikimedia, Panasonic, and Discord — that use Cloudflare to protect and accelerate their Internet properties. Why do they choose Cloudflare? Three main reasons:
1. No scrubs Cloudflare doesn’t operate scrubbing centers as we believe that the scrubbing center model is a flawed approach to DDoS protection. Scrubbing centers cause delays and cost too much to build and run. What’s more, DDoS attacks are asymmetric — attackers have more available bandwidth than a single scrubbing center will ever be able to handle.
Cloudflare’s network is architected so that every machine in every data center performs DDoS mitigation. Doing this at the edge is the only way to mitigate at scale without impacting performance. Our Anycast-based architecture makes our capacity equivalent to our DDoS scrubbing capacity, the largest in the market at 51 Tbps. This means Cloudflare detects and mitigates DDoS attacks close to the source of attack. Better yet, Cloudflare’s global threat intelligence acts like an immune system for the Internet — employing our machine learning models to learn from and mitigate attacks against any customer to protect them all.
2. It’s about time Most organizations are in some stage of their journey from on-prem to the cloud. The threat landscape, functional requirements, and scale of business applications are evolving faster than ever before, and the volume and sophistication of network attacks are already straining the defensive capabilities of even the most advanced enterprises. One concern many enterprises have when adopting the cloud is added latency for applications. Most cloud-based DDoS protection services rely on specialized data centers aka “scrubbing centers” for DDoS mitigation. Backhauling traffic to those data centers can add significant latency depending on its location relative to the destination server.
This problem compounds when an organization uses different providers for different networking functions. When traffic must hop from provider to provider, latency can be measured in hundreds of milliseconds.
Cloudflare’s distributed geographical presence ensures that attacks are globally detected and mitigated in under 3 seconds on average — making it one of the fastest in the industry.
3. It’s not just about DDoS DDoS attacks constitute just one facet of the many cyber threats organizations are facing today. As businesses shift to a Zero Trust approach, network and security buyers will face larger threats related to network access, and a continued surge in the frequency and sophistication of bot-related attacks.
A key design tenet while building products at Cloudflare is integration. Cloudflare One is a solution that uses a Zero Trust security model to provide companies a better way to protect devices, data, and applications — and is deeply integrated with our existing platform of security and DDoS solutions.
To learn more about Cloudflare’s DDoS solution contact us or get started today by signing up on our dashboard.
DDoS attacks are surging — both in frequency and sophistication. After doubling from Q1 to Q2, the total number of network layer attacks observed in Q3 doubled again — resulting in a 4x increase in number compared to the pre-COVID levels in the first quarter. Cloudflare also observed more attack vectors deployed than ever — in fact, while SYN, RST, and UDP floods continue to dominate the landscape, we saw an explosion in protocol specific attacks such as mDNS, Memcached, and Jenkins DoS attacks.
Here are other key network layer DDoS trends we observed in Q3:
Majority of the attacks are under 500 Mbps and 1 Mpps — both still suffice to cause service disruptions
We continue to see a majority of attacks be under 1 hr in duration
Ransom-driven DDoS attacks (RDDoS) are on the rise as groups claiming to be Fancy Bear, Cozy Bear and the Lazarus Group extort organizations around the world. As of this writing, the ransom campaign is still ongoing. See a special note on this below.
Number of attacks
The total number of L3/4 DDoS attacks we observe on our network continues to increase substantially, as indicated in the graph below. All in all, Q3 saw over 56% of all attacks this year — double that of Q2, and four times that of Q1. In addition, the number of attacks per month increased throughout the quarter.
While September witnessed the largest number of attacks overall, August saw the most large attacks (over 500Mbps). Ninety-one percent of large attacks in Q3 took place in that month—while monthly distribution of other attack sizes was far more even.
While the total number of attacks between 200-300 Gbps decreased in September, we saw more global attacks on our network in Q3. This suggests the increase in the use of distributed botnets to launch attacks. In fact, in early July, Cloudflare witnessed one of the largest-ever attacks on our network — generated by Moobot, a Mirai-based botnet. The attack peaked at 654 Mbps and originated from 18,705 unique IP addresses, each believed to be a Moobot-infected IoT device. The attack campaign lasted nearly 10 days, but the customer was protected by Cloudflare, so they observed no downtime or service degradation.
Attack size (bit rate and packet rate)
There are different ways of measuring a L3/4 DDoS attack’s size. One is the volume of traffic it delivers, measured as the bit rate (specifically, Gigabits-per-second). Another is the number of packets it delivers, measured as the packet rate (specifically, packets-per-second). Attacks with high bit rates attempt to saturate the Internet link, and attacks with high packet rates attempt to overwhelm the routers or other in-line hardware devices.
In Q3, most of the attacks we observed were smaller in size. In fact, over 87% of all attacks were under 1 Gbps. This represents a significant increase from Q2, when roughly 52% of attacks were that small. Note that, even ‘small’ attacks of under 500 Mbps are many times sufficient to create major disruptions for Internet properties that are not protected by a Cloud based DDoS protection service. Many organizations have uplinks provided by their ISPs that are far less than 1 Gbps. Assuming their public facing network interface also serves legitimate traffic, you can see how even these ‘small’ DDoS attacks can easily take down Internet properties.
This trend holds true for attack packet rates. In Q3, 47% of attacks were under 50k pps — compared to just 19% in Q2.
Smaller attacks can indicate that amateur attackers may be behind the attacks — using tools easily available to generate attacks on exposed IPs/ networks. Alternatively, small attacks may serve as a smokescreen to distract security teams from other kinds of cyberattacks that might be taking place simultaneously.
In terms of length, very short attacks were the most common attack type observed in Q3, accounting for nearly 88% of all attacks. This observation is in line with our prior reports — in general, Layer 3/4 DDoS attacks are getting shorter in duration.
Short burst attacks may attempt to cause damage without being detected by DDoS detection systems. DDoS services that rely on manual analysis and mitigation may prove to be useless against these types of attacks because they are over before the analyst even identifies the attack traffic.
Alternatively, the use of short attacks may be used to probe the cyber defenses of the target. Load-testing tools and automated DDoS tools, that are widely available on the dark web, can generate short bursts of, say, a SYN flood, and then following up with another short attack using an alternate attack vector. This allows attackers to understand the security posture of their targets before they decide to potentially launch larger attacks at larger rates and longer durations – which come at a cost.
In other cases, attackers generate small DDoS attacks as proof and warning to the target organization of the attacker’s ability to cause real damage later on. It’s often followed by a ransom note to the target organization, demanding payment so as to avoid suffering an attack that could more thoroughly cripple network infrastructure.
Whatever their motivation, DDoS attacks of any size or duration are not going away anytime soon. Even short DDoS attacks cause harm, and having an automated real-time defense mechanism in place is critical for any online business.
SYN floods constituted nearly 65% of all attacks observed in Q3, followed by RST floods and UDP floods in second and third places. This is relatively consistent with observations from previous quarters, highlighting the DDoS attack vector of choice by attackers.
While TCP based attacks like SYN and RST floods continue to be popular, UDP-protocol specific attacks such as mDNS, Memcached, and Jenkins are seeing an explosion compared to the prior quarter.
Multicast DNS (mDNS) is a UDP-based protocol that is used in local networks for service/device discovery. Vulnerable mDNS servers respond to unicast queries originating outside of the local network, which are ‘spoofed’ (altered) with the victim’s source address. This results in amplification attacks. In Q3, we noticed an explosion of mDNS attacks — specifically, we saw a 2,680% increase compared to the previous quarter.
This was followed by Memcached and Jenkins attacks. Memcached is a Key Value database. Requests can be made over the UDP protocol with a spoofed source address of the target. The size of the Value stored in the requested Key will affect the amplification factor, resulting in a DDoS amplification attack. Similarly, Jenkins, NTP, Ubiquity and the other UDP based protocols have seen a dramatic increase over the quarter due to its UDP stateless nature. A vulnerability in the older version (Jenkins 2.218 and earlier) aided the launch of DDoS attacks. This vulnerability was fixed in Jenkins 2.219 by disabling UDP multicast/ broadcast messages by default. However there are still many vulnerable and exposed devices that run UDP based services which are being harnessed to generate volumetric amplification attacks.
Attack by country
Looking at country-based distribution, the United States observed the most number of L3/4 DDoS attacks, followed by Germany and Australia. Note that when analyzing L3/4 DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations where the traffic was ingested, and not by the location of the source IP. The reason is when attackers launch L3/4 attacks they can spoof the source IP address in order to obfuscate the attack source. If we were to derive the country based on a spoofed source IP, we would get a spoofed country. Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of Cloudflare’s data center in which the attack was observed. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities around the world.
Asia Pacific & Oceania
A note on recent ransom-driven DDoS attacks
Over the past months, Cloudflare has observed another disturbing trend — a rise in extortion and ransom-based DDoS (RDDoS) attacks targeting organizations around the world. While RDDoS threats do not always result in an actual attack, the cases seen in recent months show that attacker groups are willing to carry out the threat, launching large scale DDoS attacks that can overwhelm organizations that lack adequate protection. In some cases, the initial teaser attack may be sufficient to cause impact if not protected by a Cloud based DDoS protection service.
In a RDDoS attack, a malicious party threatens a person or organization with a cyberattack that could knock their networks, websites, or applications offline for a period of time, unless the person or organization pays a ransom. You can read more about RDDoS attacks here.
Entities claiming to be Fancy Bear, Cozy Bear, and Lazarus have been threatening to launch DDoS attacks against organizations’ websites and network infrastructure unless a ransom is paid before a given deadline. Additionally, an initial ‘teaser’ DDoS attack is usually launched as a form of demonstration before parallel to the ransom email. The demonstration attack is typically a UDP reflection attack using a variety of protocols, lasting roughly 30 minutes in duration (or less).
What to do if you receive a threat:
Do not panic and we recommend you to not pay the ransom: Paying the ransom only encourages bad actors, finances illegal activities —and there’s no guarantee that they won’t attack your network now or later.
Notify local law enforcement: They will also likely request a copy of the ransom letter that you received.
Contact Cloudflare: We can help ensure your website and network infrastructure are safeguarded from these ransom attacks.
Cloudflare DDoS protection is different
On-prem hardware/cloud-scrubbing centers can’t address the challenges of modern volumetric DDoS attacks. Appliances are easily overwhelmed by large DDoS attacks, Internet links quickly saturate, and rerouting traffic to cloud scrubbing centers introduces unacceptable latency penalties. Our cloud-native, always-on, automated DDoS protection approach solves problems that traditional cloud signaling approaches were originally created to address.
Cloudflare’s mission is to help build a better Internet, which grounds our DDoS approach and is why in 2017, we pioneered unmetered DDoS mitigation for all of our customers on all plans including the free plan. We are able to provide this level of protection because every server on our network can detect & block threats, enabling us to absorb attacks of any size/kind, with no latency impact. This architecture gives us unparalleled advantages compared to any other vendor.
51 Tbps of DDoS mitigation capacity and under 3 sec TTM: Every data center in Cloudflare’s network detects and mitigates DDoS attacks. Once an attack is identified, the Cloudflare’s local data center mitigation system (dosd) generates and applies a dynamically crafted rule with a real-time signature — and mitigates attacks in under 3 seconds globally on average. This 3-second Time To Mitigate (TTM) is one of the fastest in the industry. Firewall rules and “proactive”/static configurations take effect immediately.
Fast performance included: Cloudflare is architected so that customers do not incur a latency penalty as a result of attacks. We deliver DDoS protection from every Cloudflare data center (instead of legacy scrubbing centers or on-premise hardware boxes) which allows us to mitigate attacks closest to the source. Cloudflare analyzes traffic out-of-path ensuring that our DDoS mitigation solution doesn’t add any latency to legitimate traffic. The rule is applied at the most optimal place in the Linux stack for a cost efficient mitigation, ensuring no performance penalty.
Global Threat Intelligence: Like an immune system, our network learns from/mitigates attacks against any customer to protect them all. With threat intelligence (TI), it automatically blocks attacks and is employed in customer facing features (Bot Fight mode, Firewall Rules & Security Level). Users create custom rules to mitigate attacks based on traffic attribute filters, threat & bot scores generated using ML models (protecting against bots/botnets/DDoS).
This blog post is a high-level overview of bot traffic on Cloudflare’s network. Cloudflare offers a comprehensive Bot Management tool for Enterprise customers, along with an effective free tool called Bot Fight Mode. Because of the tremendous amount of traffic that flows through our network each day, Cloudflare is in a unique position to analyze global bot trends.
In this post, we will cover the basics of bot traffic and distinguish between automated requests and other human requests (What Is A Bot?). Then, we’ll move on to a global overview of bot traffic around the world (A RoboBird’s Eye View, A Bot Day and Bots All Over The World), and dive into North American traffic (A Look into North American Traffic). Lastly, we’ll finish with an overview of how the coronavirus pandemic affected global traffic, and we’ll take a deeper look at European traffic (Bots During COVID-19 In Europe).
On average, Cloudflare processes 18 million HTTP requests every second. This is a great opportunity to understand how bots shape the Internet, how much infrastructure is dedicated to these automated requests, and why our customers need a great bot management solution.
What Is A Bot?
Cloudflare groups traffic into four bot-related categories:
Our goal is to stop malicious and unwanted bots from harming our customers, while giving customers the opportunity to control how other automated traffic is managed.
We label each request that comes into Cloudflare with a “bot score” 1 through 99, where a lower score means that a request probably came from a bot. A higher score means that a request probably came from a human. This score is available in our Firewall, logs, and Workers, giving customers the flexibility to act on any score.
To learn more about how Bot Management interacts with our firewall, check out our support page.
We track successes and failures during these challenges, which ultimately allows us to improve our detection systems. Assuming that our challenges are solvable by humans, effective detections should have low solve rates, given that they are usually presented to bots.
Verified bots are registered in an internal verified bot directory. These good bots power search engines and monitoring tools. Good bots enable our customers’ web pages to be found by search engines, for example.
For known non-verified bots (such as a scraper using a simple curl library), we keep a similar directory that is managed by our heuristics engine. If not otherwise verified, we consider requests caught by this engine to be definitely automated.
Our machine learning engine provides another way to identify potential bots. This engine identifies requests with a high probability of automation and marks them as likely automated. This detection mechanism benefits from models built on data from our global network.
If a request is not marked as automated, we mark it as likely human and pass along the bot score from our machine learning system.
The two bot definitions for automated traffic are somewhat complementary. Requests caught by heuristic detections will not count towards machine learning detections. Requests that are reliably caught by our machine learning detections won’t need to be registered in our known heuristics bot directory. Because of this, we combine these two together when we discuss “automated traffic” in general.
A RoboBird’s Eye View
Data from this piece comes from information about Cloudflare’s customers, analyzed between January 15, 2020 and July 31, 2020.
First, let’s get a basic understanding of the traffic on our network.
Figure 1.1 has a global breakdown regarding classification; 60.6% of traffic is likely human, 19.3% is likely automated, 18.1% is definitely automated and only 2.1% is from verified bots. In total, 39.5% of requests we score come from some kind of bot.
A Bot Day
Regular traffic fluctuates throughout the day. Do bots follow suit? Let’s check. Figure 2.1 represents traffic deviation from the average hourly traffic. An increase of 10% would mean that the hour is 10% busier than the average hour (measuring requests per hour). We include the total overall traffic in this chart to serve as a comparison to other types of traffic.
We can clearly see a difference between human traffic and bot traffic. Human traffic varies heavily, but predictably, throughout the day. We can see a 15% decrease in human traffic early in the day, between midnight and 05:00 UTC, corresponding to the end of business hours in the Americas, and up to a 25% increase during business hours, 14:00 to 17:00 UTC, where traffic is highest. Conversely, bot traffic is more consistent. Slow hours still see a smaller drop than overall traffic, and busy hours are less busy. The difference between good and bad bots is also apparent: good bots are even more consistent, with small fluctuations in hourly traffic.
But why would this happen? A large portion of bots, good and bad, perform the same task across the Internet. Bad bots may be scraping websites or looking to infect unprotected machines, and they will do this with little intervention from human operators. Good bots could be doing some of these operations, but less frequently and in a more targeted fashion. A good bot scraping a website may be doing so to add it to a search engine, while a bad bot will do the same thing at a much higher rate, for other reasons.
A lot of bots follow business hours. For example, sneaker bots—focused on nabbing exclusive items from sneaker stores—will naturally be active when new products launch.
This difference in volume does not mean that our classifications are affected: our scores remain consistent throughout the day, as Figure 2.1 shows.
We can also see that good bots don’t take weekends off. Weekdays and weekends have fairly marked differences for most traffic, but good bots keep a consistent schedule. Whereas a typical weekday is slightly above average, we can see a drop of about 4% in overall traffic. This does not fully apply to verified bots, which only see a small 1% drop in traffic.
Bots All Over The World
Now that we’ve taken a look at global traffic, let’s dig a little deeper.
Different regions have distinct traffic landscapes regarding automated traffic.
Figure 3.1 breaks down traffic by region, letting us peek into where each type of traffic comes from. North America stands out as a major automated traffic source; over 50% of definitely automated traffic comes from there, and they also contribute almost 80% of all verified bot traffic. Europe makes up the second largest chunk of traffic, followed by Asia.
Looking at regional breakdown of traffic in Figure 3.2, we can see just how much North American traffic is automated, well above the global average.
A Look into North American Traffic
As the largest source of automated traffic, North America deserves a closer look.
First, we’ll start with a breakdown of each country.
Most of our requests in North America come from just three countries—the United States, Canada and Mexico. These account for 98% of all requests from North America, 97% of all requests from likely human sources and 100% of requests from verified bots. The United States alone accounts for 88% of total requests, 82% of requests from likely human sources, 96% of requests from definitely automated sources, 88% of requests from likely automated traffic sources and 98% of requests from verified bot.
However, this alone does not mean that the United States has an unusual amount of activity. These countries have a combined population of roughly 497 million people. The United States accounts for 66.5% of that, Mexico 25.9% and Canada 7.6%. With this context, we can see that the United States is overrepresented in terms of raw requests, but underrepresented in terms of how much of that traffic is likely to be human. Conversely, Canadian traffic is more likely to be human.
Let’s take another look at each country.
Over half of the traffic from the United States is automated in some way, which is a clear departure from trends in Mexico and Canada.
So far, we’ve seen how much the United States contributes to automated traffic. If we want to go deeper, a good place to start is by understanding how these bots get online. We can do this by examining the networks from which the traffic originates. Networks are identified by Autonomous System Numbers, or ASNs. These form the backbone of the Internet infrastructure.
Think of these as Internet Service Providers, but facing inward towards the network instead of outward towards end consumers. ISPs like Comcast and Verizon are examples of residential ASNs, where we expect mostly human traffic. Cloud providers such as Google and Amazon are also ASNs, but targeted towards cloud services. We expect most of these requests to be automated in some way.
Looking at traffic on the ASN level is important because we can identify cloud-based traffic, or traffic using residential proxies, among others.
Let’s take a look at which ASNs are associated with visitors in the United States. We’ll restrict ourselves to “eyeball” traffic, which is the term we use for requests coming from site visitors.
From figure 4.1 we can clearly see the impact that cloud services have on traffic; 11.5% of all eyeball traffic comes from Amazon and Google.
Verified bots operate in a different landscape, coming from cloud providers such as Amazon, Google, Microsoft, Advanced Hosting and Wowrack.
Automated traffic has a variety of ASNs. Cloud providers such as Amazon, Google and Microsoft make up the 30% of automated traffic. Comcast also makes up a significant portion of traffic at 4.8%, indicating that some bots come from residential services.
Bots During COVID-19 In Europe
Lockdowns and limits on public events came as a consequence of the ongoing coronavirus pandemic. Many people have been working from home, and even those who do not have this option are using the Internet in new ways. Overall, this has meant that Cloudflare’s network has grown tremendously.
But how does this impact bot traffic? First let’s get an idea of how it impacted traffic in general. Countries were impacted by the virus at different times, so we expect to see differences, right?
Figure 5.1 has just the traffic increase. Globally, we are seeing an average increase of 10%, while North America saw an increase of over 40% compared to the beginning of the year. Some regions did not change much, such as Africa and Asia, while others, such as Europe saw an increased period, but has since normalized to previous levels.
Let’s look at a few countries, so we can understand what this looks like.
Figure 5.2 shows daily traffic relative to January 15, when data collection started. For comparison, we have overall European traffic, and three selected countries: Italy, the United Kingdom and Portugal. Italy was picked because it was one of the first countries in Europe to face the worst of the coronavirus and enact lockdown measures. The United Kingdom took another strategy, with an initial focus on herd immunity, and enacted measures later than the others. Portugal is somewhere in between, locking down later than Italy, in slightly different circumstances.
At the beginning of the year, traffic kept stable and fluctuations kept in line with the European average. As lockdown measures began, traffic increased. Italy was first out of these countries, rising a few weeks before the others, and keeping well above average. Eventually, all countries saw a growth in traffic, followed by a stabilization. Italy seems to have adjusted to a normal, with its growth in line with the European average. Portugal has also stabilized, but with busier weekdays. Conversely, the United Kingdom showed no signs of stopping, exceeding a growth of 40% compared to the beginning of the year.
Definitely automated traffic did not have that much of a pronounced variation. Italian traffic kept steady throughout, and Portugal had a rather large increase. The biggest one, however, was the United Kingdom, which tripled its initial count.
Verified bot traffic is steady, except in Italy, with a massive increase between March and May. What could be the cause of this? Are these a few zones, getting a massive number of requests?
Well, no. If we only examine the top 10,000 zones (by total verified bot requests), we can still see a massive increase in traffic for other zones. So, what’s happening?
Let’s look at user agents. We can separate the top 10 user agents during the bump, and see how they evolve over time.
We can see that these 10 user agents are responsible for the majority of verified traffic coming from Italy.
In fact, most of this increase is from a single user agent. This instance of Google image proxy anonymizes image requests from Gmail, which explains its popularity.
Where does this increase come from? Did this bot suddenly appear and disappear?
Not quite. One thing to keep in mind when dealing with bots is that they cross borders easily. As a proxy service, this bot is making calls on behalf of the end user – people opening emails. These requests will originate from a data center, which can be anywhere in the world. To see this in action, let’s take a look at traffic for this bot in a few select countries.
We can see that the global average barely budges. It appears that Google may be moving image proxy traffic between data centers and during the period we observed above that traffic was coming from Italy.
With Cloudflare’s global reach, we’re in a position to understand how bots behave.
The first half of 2020 saw a massive increase in web traffic of around 35% since the beginning of the year, driven by the ongoing coronavirus pandemic, and some bots have taken advantage of it.
We explained how bot management works for our customers, and how we distinguish between likely automated and human traffic.
We showed an overview of how much of our global traffic is automated, and how bots change their behavior throughout the day and the week. Notably, 39.4% of all traffic Cloudflare processes comes from a suspected automated source.
A regional overview of automated traffic lets us know which regions were the source of traffic from likely automated agents. North America, Europe and Asia were the primary sources of traffic, and also of automated traffic in particular.
We then focused on North America, where the majority of automated traffic originates. The United States alone accounted for the majority of requests, over half of which come from automated sources.
To explore this further, we briefly dived into ASN traffic in the United States, so we could see where these requests were coming from. ASNs like Comcast and AT&T were the top ASNs for overall traffic, but unsurprisingly, data centers like Google and Amazon AWS were the main drivers of automated traffic.
Finally, we examined how the coronavirus has impacted traffic in Europe, with a deeper dive on Italian traffic. This led to some interesting insights on verified bot traffic, which saw a massive increase in Italy for a few months.
This post is a small peek into bot management at Cloudflare. In the future, we hope to expand this series of blog posts on bot management, exposing even more insights about bots on the Internet.
In the first quarter of 2020, within a matter of weeks, our way of life shifted. We’ve become reliant on online services more than ever. Employees that can are working from home, students of all ages and grades are taking classes online, and we’ve redefined what it means to stay connected. The more the public is dependent on staying connected, the larger the potential reward for attackers to cause chaos and disrupt our way of life. It is therefore no surprise that in Q1 2020 (January 1, 2020 to March 31, 2020) we reported an increase in the number of attacks—especially after various government authority mandates to stay indoors—shelter-in-place went into effect in the second half of March.
In Q2 2020 (April 1, 2020 to June 30, 2020), this trend of increasing DDoS attacks continued and even accelerated:
The number of L3/4 DDoS attacks observed over our network doubled compared to that in the first three months of the year.
The scale of the largest L3/4 DDoS attacks increased significantly. In fact, we observed some of the largest attacks ever recorded over our network.
We observed more attack vectors being deployed and attacks were more geographically distributed.
The number of global L3/4 DDoS attacks in Q2 doubled
Gatebot is Cloudflare’s primary DDoS protection system. It automatically detects and mitigates globally distributed DDoS attacks. A global DDoS attack is an attack that we observe in more than one of our edge data centers. These attacks are usually generated by sophisticated attackers employing botnets in the range of tens of thousand to millions of bots.
Sophisticated attackers kept Gatebot busy in Q2. The total number of global L3/4 DDoS attacks that Gatebot detected and mitigated in Q2 doubled quarter over quarter. In our Q1 DDoS report, we reported a spike in the number and size of attacks. We continue to see this trend accelerate through Q2; over 66% of all global DDoS attacks in 2020 occurred in the second quarter (nearly 100% increase). May was the busiest month in the first half of 2020, followed by June and April. Almost a third of all L3/4 DDoS attacks occurred in May.
In fact, 63% of all L3/4 DDoS attacks that peaked over 100 Gbps occurred in May. As the global pandemic continued to heighten around the world in May, attackers were especially eager to take down websites and other Internet properties.
Small attacks continue to dominate in numbers as big attacks get bigger in size
A DDoS attack’s strength is equivalent to its size—the actual number of packets or bits flooding the link to overwhelm the target. A ‘large’ DDoS attack refers to an attack that peaks at a high rate of Internet traffic. The rate can be measured in terms of packets or bits. Attacks with high bit rates attempt to saturate the Internet link, and attacks with high packet rates attempt to overwhelm the routers or other in-line hardware devices.
Similar to Q1, the majority of L3/4 DDoS attacks that we observed in Q2 were also relatively ‘small’ with regards to the scale of Cloudflare’s network. In Q2, nearly 90% of all L3/4 DDoS attacks that we saw peaked below 10 Gbps. Small attacks that peak below 10 Gbps can still easily cause an outage to most of the websites and Internet properties around the world if they are not protected by a cloud-based DDoS mitigation service.
Similarly, from a packet rate perspective, 76% of all L3/4 DDoS attacks in Q2 peaked up to 1 million packets per second (pps). Typically, a 1 Gbps Ethernet interface can deliver anywhere between 80k to 1.5M pps. Assuming the interface also serves legitimate traffic, and that most organizations have much less than a 1 Gbps interface, you can see how even these ‘small’ packet rate DDoS attacks can easily take down Internet properties.
In terms of duration, 83% of all attacks lasted between 30 to 60 minutes. We saw a similar trend in Q1 with 79% of attacks falling in the same duration range. This may seem like a short duration, but imagine this as a 30 to 60 minute cyber battle between your security team and the attackers. Now it doesn’t seem so short. Additionally, if a DDoS attack creates an outage or service degradation, the recovery time to reboot your appliances and relaunch your services can be much longer; costing you lost revenue and reputation for every minute.
In Q2, we saw the largest DDoS attacks on our network, ever
This quarter, we saw an increasing number of large scale attacks; both in terms of packet rate and bit rate. In fact, 88% of all DDoS attacks in 2020 that peaked above 100 Gbps were launched after shelter-in-place went into effect in March. Once again, May was not just the busiest month with the most number of attacks, but also the greatest number of large attacks above 100 Gbps.
From the packet perspective, June took the lead with a whopping 754 million pps attack. Besides that attack, the maximum packet rates stayed mostly consistent throughout the quarter with around 200 million pps.
The 754 million pps attack was automatically detected and mitigated by Cloudflare. The attack was part of an organized four-day campaign that lasted from June 18 to the 21. As part of the campaign, attack traffic from over 316,000 IP addresses targeted a single Cloudflare IP address.
Cloudflare’s DDoS protection systems automatically detected and mitigated the attack, and due to the size and global coverage of our network, there was no impact to performance. A global interconnected network is crucial when mitigating large attacks in order to be able to absorb the attack traffic and mitigate it close to the source, whilst also continuing serving legitimate customer traffic without inducing latency or service interruptions.
The United States is targeted with the most attacks
When we look at the L3/4 DDoS attack distribution by country, our data centers in the United States received the most number of attacks (22.6%), followed by Germany (4.4%), Canada (2.7%) and Great Britain (2.6%).
However when we look at the total attack bytes mitigated by each Cloudflare data center, the United States still leads (34.9%), but followed by Hong Kong (6.6%), Russia (6.5%), Germany (4.5%) and Colombia (3.7%). The reason for this change is due to the total amount of bandwidth that was generated in each attack. For instance, while Hong Kong did not make it to the top 10 list due to the relatively small number of attacks that was observed in Hong Kong (1.8%), the attacks were highly volumetric and generated so much attack traffic that pushed Hong Kong to the 2nd place.
When analyzing L3/4 DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations and not by the location of the source IP. The reason is when attackers launch L3/4 attacks they can ‘spoof’ (alter) the source IP address in order to obfuscate the attack source. If we were to derive the country based on a spoofed source IP, we would get a spoofed country. Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of Cloudflare’s data center in which the attack was observed. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities around the world.
57% of all L3/4 DDoS attacks in Q2 were SYN floods
An attack vector is a term used to describe the attack method. In Q2, we observed an increase in the number of vectors used by attackers in L3/4 DDoS attacks. A total of 39 different types of attack vectors were used in Q2, compared to 34 in Q1. SYN floods formed the majority with over 57% in share, followed by RST (13%), UDP (7%), CLDAP (6%) and SSDP (3%) attacks.
SYN flood attacks aim to exploit the handshake process of a TCP connection. By repeatedly sending initial connection request packets with a synchronize flag (SYN), the attacker attempts to overwhelm the router’s connection table that tracks the state of TCP connections. The router replies with a packet that contains a synchronized acknowledgment flag (SYN-ACK), allocates a certain amount of memory for each given connection and falsely waits for the client to respond with a final acknowledgment (ACK). Given a sufficient number of SYNs that occupy the router’s memory, the router is unable to allocate further memory for legitimate clients causing a denial of service.
No matter the attack vector, Cloudflare automatically detects and mitigates stateful or stateless DDoS attacks using our 3 pronged protection approach comprising of our home-built DDoS protection systems:
Gatebot – Cloudflare’s centralized DDoS protection systems for detecting and mitigating globally distributed volumetric DDoS attacks. Gatebot runs in our network’s core data center. It receives samples from every one of our edge data centers, analyzes them and automatically sends mitigation instructions when attacks are detected. Gatebot is also synchronized to each of our customers’ web servers to identify its health and triggers accordingly, tailored protection.
dosd (denial of service daemon) – Cloudflare’s decentralized DDoS protection systems. dosd runs autonomously in each server in every Cloudflare data center around the world, analyzes traffic, and applies local mitigation rules when needed. Besides being able to detect and mitigate attacks at super fast speeds, dosd significantly improves our network resilience by delegating the detection and mitigation capabilities to the edge.
flowtrackd (flow tracking daemon) – Cloudflare’s TCP state tracking machine for detecting and mitigating the most randomized and sophisticated TCP-based DDoS attacks in unidirectional routing topologies. flowtrackd is able to identify the state of a TCP connection and then drops, challenges or rate-limits packets that don’t belong to a legitimate connection.
In addition to our automated DDoS protection systems, Cloudflare also generates real-time threat intelligence that automatically mitigates attacks. Furthermore, Cloudflare provides its customers firewall, rate-limiting and additional tools to further customize and optimize their protection.
Cloudflare DDoS mitigation
As Internet usage continues to evolve for businesses and individuals, expect DDoS tactics to adapt as well. Cloudflare protects websites, applications, and entire networks from DDoS attacks of any size, kind, or level of sophistication.
Our customers and industry analysts recommend our comprehensive solution for three main reasons:
Network scale: Cloudflare’s 37 Tbps network can easily block attacks of any size, type, or level of sophistication. The Cloudflare network has a DDoS mitigation capacity that is higher than the next four competitors—combined.
Time-to-mitigation: Cloudflare mitigates most network layer attacks in under 10 seconds globally, and immediate mitigation (0 seconds) when static rules are preconfigured. With our global presence, Cloudflare mitigates attacks close to the source with minimal latency. In some cases, traffic is even faster than over the public Internet.
Threat intelligence: Cloudflare’s DDoS mitigation is powered by threat intelligence harnessed from over 27 million Internet properties on it. Additionally, the threat intelligence is incorporated into customer facing firewalls and tools in order to empower our customers.
Cloudflare is uniquely positioned to deliver DDoS mitigation with unparalleled scale, speed, and smarts because of the architecture of our network. Cloudflare’s network is like a fractal—every service runs on every server in every Cloudflare data center that spans over 200 cities globally. This enables Cloudflare to detect and mitigate attacks close to the source of origin, no matter the size, source, or type of attack.
You can also join an upcoming live webinar where we will be discussing these trends, and strategies enterprises can implement to combat DDoS attacks and keep their networks online and fast. You can register here.
As we wrapped up the first quarter of 2020, we set out to understand if and how DDoS attack trends have shifted during this unprecedented time of global shelter in place. Since then, traffic levels have increased by over 50% in many countries, but have DDoS attacks increased as well?
Traffic increases are often observed during holiday seasons. During holidays, people may spend more time online; whether shopping, ordering food, playing online games or a myriad of other online activities. This higher usage translates into higher revenue per minute for the companies that provide those various online services.
Downtime or service degradation during these peak times could result in user churn and loss of significant revenue in a very short time. ITIC estimates that the average cost of an outage is $5,600 per minute, which extrapolates to well over $300K per hour. It is therefore no surprise that attackers capitalize on the opportunity by launching a higher number of DDoS attacks during the holiday seasons.
The current pandemic has a similar cause and effect. People are forced to stay home. They have become more reliant on online services to accomplish their daily tasks which has generated a surge in the Internet traffic and DDoS attacks.
The Rise of Smaller, Shorter Attacks
Most of the attacks that we observed in Q1 2020 were relatively small, as measured by their bit rates. As shown in the figure below, in Q1 2020, 92% of the attacks were under 10 Gbps, compared to 84% in Q4 2019.
Diving deeper, an interesting shift can be observed in the distribution of attacks below 10 Gbps in Q1, as compared to the previous quarter. In Q4, 47% of network-layer DDoS attacks peaked below 500 Mbps, whereas in Q1 they increased to 64%.
From a packet rate perspective, the majority of the attacks peaked below 1 million packets per second (pps). This rate, along with their bit rate, indicates that attackers are no longer focusing their efforts and resources to generate high-rate floods — bits or packets per second.
However, it’s not only the packet and bit rates that are decreasing, but also the attack durations. The figure below illustrates that 79% of DDoS attacks in Q1 lasted between 30 to 60 minutes, compared to 60% in Q4, which represents a 19% increase.
These three trends could be explained by the following:
Launching DDoS attacks is cheap and you don’t need much technical background. DDoS-as-a-service tools have provided a possible avenue for bad actors with little to no technical expertise to launch DDoS attacks quickly, easily, in a cost-effective manner and with limited bandwidth. According to Kaspersky, DDoS attack services can cost as little as $5 for a 300-second attack (5 minutes). Additionally, amateur attackers can also easily leverage free tools to generate floods of packets. As we’ll see in the next section, 13.5% of all DDoS attacks in Q1 were generated using variations of the publicly available Mirai code.
While an attack under 10 Gbps might seem small, it can still be enough to affect underprotected Internet properties. Smaller and quicker attacks might prove to deliver a high ROI for attackers to extort a ransom from companies in lieu of not disrupting the availability of the Internet property.
Larger Attacks Still Persist, Albeit in Smaller Numbers
While the majority of the attacks were under 10 Gbps, larger attacks are still prevalent. The below graph shows a trend in the largest bit-rate of network-layer DDoS attacks that Cloudflare has observed and mitigated in Q4 2019 and Q1 2020. The largest attack for the quarter was observed during March and peaked just above 550 Gbps.
If At First You Don’t Succeed, Try, Try Again
A persistent attacker is one that does not give up when their attacks fail; they try and try again. They launch multiple attacks on their target, often utilizing multiple attack vectors. In the Q4 2019 holiday season, attackers persisted and launched as many as 523 DDoS attacks in one day against a single Cloudflare IP. Each Cloudflare IP under attack was targeted by as many as 4.6 DDoS attacks every day on average.
During Q1, as the world entered COVID-19 lockdown, we observed a significant increase in the number of attacks compared to the monthly average. The last time we saw such an increase was in the Q4 2019 holiday season. However, an interesting difference is that attackers seem less persistent now than during the holidays. In Q1 2020, the average persistence rate dropped as low as 2.2 attacks per Cloudflare IP address per day, with a maximum of 311 attacks on a single IP; 40% less than the previous holiday quarter.
Throughout the past two quarters, the average number of attack vectors employed in DDoS attacks per IP per day has been mostly steady at approximately 1.4, with a maximum of 10.
Over the past quarter, we’ve seen over 34 different types of attack vectors on L3/4. ACK attacks formed the majority (50.1%) in Q1, followed by SYN attacks with 16.6%, and in third place, Mirai, which still represents a significant portion of the attacks (15.4%). Together, SYN & ACK DDoS attacks (TCP) form 66% of all L3/4 attack vectors in Q1.
Top Attack Vectors
All Attack Vectors
Percent in Q1
A Crisis is Unfortunately Sometimes a Malevolent Opportunity
The number of DDoS attacks in March 2020 increased as compared to January and February. Attackers found the crisis period to be an opportune time to launch an increased number of DDoS attacks, as illustrated below.
Furthermore, as various government authorities started mandating lockdowns and shelter-in-place orders, attackers resorted to increasing the number of large-sized attacks in the latter half of March. There were 55% more attacks observed in the second half of month (March 16-31) as compared to the first half (March 1-15). Additionally, 94% of attacks peaking at 300-400 Gbps were launched in the month of March.
Stop DDoS attacks, Large or Small, Closer To The Source
With the ever shifting DDoS landscape, it is important to have a DDoS protection solution which is comprehensive and adaptive. In context with the attack insights illustrated above, here’s how Cloudflare stays ahead of these shifts to protect our customers.
As attacks shrink in rate and duration, Time To Mitigate SLAs as long as 15 minutes provided by legacy vendors are just not practical anymore. Cloudflare mitigates network layer DDoS attacks under 10 seconds in most cases, which is especially critical for the increasingly shorter attacks. Read more about the recent enhancements to our DDoS detection and mitigation systems that allow us to automatically detect and mitigate DDoS attacks so quickly at scale.
An increasing number of DDoS attacks are localized, which implies that legacy DDoS solutions which adopt a scrubbing center approach are not a feasible solution, as they are limited in their global coverage as well as act as a choke point, as DDoS traffic needs to be hauled back and forth from them. Cloudflare’s unique distributed architecture empowers every one of its data centers, spanning across 200 cities globally, to provide full DDoS mitigation capabilities.
Large distributed volumetric attacks still exist and are employed by resourceful attackers when the opportunity is rife. An attack exceeding 1 Tbps can be expected in the future, so the ability to mitigate large DDoS attacks is a key aspect of today’s DDoS solution. Cloudflare has one of the most interconnected networks in the world with a capacity of over 35 Tbps which allows it to mitigate even the largest DDoS attacks. This massive network capacity concomitant with the globally distributed architecture allows Cloudflare to mitigate attacks, both small and large, closer to the source.
The proliferation of DDoS attacks of varying size, duration, and persistence has made DDoS protection a foundational part of every business and organization’s online presence. However, there are key considerations including network capacity, management capabilities, global distribution, alerting, reporting and support that security and risk management technical professionals need to evaluate when selecting a DDoS protection solution.
Gartner’s view of the DDoS solutions; How did Cloudflare fare?
Gartner recently published the report Solution Comparison for DDoS Cloud Scrubbing Centers (ID G00467346), authored by Thomas Lintemuth, Patrick Hevesi and Sushil Aryal. This report enables customers to view a side-by-side solution comparison of different DDoS cloud scrubbing centers measured against common assessment criteria. If you have a Gartner subscription, you can view the report here. Cloudflare has received the greatest number of ‘High’ ratings as compared to the 6 other DDoS vendors across 23 assessment criteria in the report.
The vast landscape of DDoS attacks
From our perspective, the nature of DDoS attacks has transformed, as the economics and ease of launching a DDoS attack has changed dramatically. With a rise in cost-effective capabilities of launching a DDoS attack, we have observed a rise in the number of under 10 Gbps DDoS network-level attacks, as shown in the figure below. Even though 10 Gbps from an attack size perspective does not seem that large, it is large enough to significantly affect a majority of the websites existing today.
At the same time, larger-sized DDoS attacks are still prevalent and have the capability of crippling the availability of an organization’s infrastructure. In March 2020, Cloudflare observed numerous 300+ Gbps attacks with the largest attack being 550 Gbps in size.
In the report Gartner also observes a similar trend, “In speaking with the vendors for this research, Gartner discovered a consistent theme: Clients are experiencing more frequent smaller attacks versus larger volumetric attacks.” In addition, they also observe that “For enterprises with Internet connections up to and exceeding 10 Gbps, frequent but short attacks up to 10 Gbps are still quite disruptive without DDoS protection. Not to say that large attacks have gone away. We haven’t seen a 1-plus Tbps attack since spring 2018, but attacks over 500 Gbps are still common.”
Gartner recommends in the report to “Choose a provider that offers scrubbing capacity of three times the largest documented volumetric attack on your continent.”
From an application-level DDoS attack perspective an interesting DDoS attack observed and mitigated by Cloudflare last year, is shown below. This HTTP DDoS attack had a peak of 1.4M requests per second, which isn’t highly rate-intensive. However, the fact that the 1.1M IPs from which the attack originated were unique and not spoofed made the attack quite interesting. The unique IP addresses were actual clients who were able to complete a TCP and HTTPS handshake.
Harness the full power of Cloudflare’s DDoS protection
Cloudflare’s cloud-delivered DDoS solution provides key features that enable security professionals to protect their organizations and customers against even the most sophisticated DDoS attacks. Some of the key features and benefits include:
Massive network capacity: With over 35 Tbps of network capacity, Cloudflare ensures that you are protected against even the most sophisticated and largest DDoS attacks. Cloudflare’s network capacity is almost equal to the total scrubbing capacity of the other 6 leading DDoS vendors combined.
Globally distributed architecture: Having a few scrubbing centers globally to mitigate DDoS attacks is an outdated approach. As DDoS attacks scale and individual attacks originate from millions of unique IPs worldwide, it’s important to have a DDoS solution that mitigates the attack at the source rather than hauling traffic to a dedicated scrubbing center. With every one of our data centers across 200 cities enabled with full DDoS mitigation capabilities, Cloudflare has more points of presence than the 6 leading DDoS vendors combined.
Fast time to mitigation: Automated edge-analyzed and edge-enforced DDoS mitigation capabilities allows us to mitigate attacks at unprecedented speeds. Typical time to mitigate a DDoS attack is less than 10s.
Integrated security: A key design tenet while building products at Cloudflare is integration. Our DDoS solution integrates seamlessly with other product offerings including WAF, Bot Management, CDN and many more. A comprehensive and integrated security solution to bolster the security posture while aiding performance. No tradeoffs between security and performance!
Unmetered and unlimited mitigation: Cloudflare offers unlimited and unmetered DDoS mitigation. This eliminates the legacy concept of ‘Surge Pricing,’ which is especially painful when a business is under duress and experiencing a DDoS attack. This enables you to avoid unpredictable costs from traffic.
Whether you’re part of a large global enterprise, or use Cloudflare for your personal site, we want to make sure that you’re protected and also have the visibility that you need. DDoS Protection is included as part of every Cloudflare service. Enterprise-level plans include advanced mitigation, detailed reporting, enriched logs, productivity enhancements and fine-grained controls. Enterprise Plan customers also receive access to dedicated customer success and solution engineering.
On Cloudflare’s 8th birthday in 2017, we announced free unmetered DDoS Protection as part of all of our plans, regardless if you’re an independent blogger using WordPress on Cloudflare’s Free plan or part of a large enterprise operating global network infrastructures. Our DDoS protection covers attack vectors on Layers 3-7; whether highly distributed and volumetric (rate-intensive) or small and sneaky. We protect over 26 million Internet properties, and at this scale, identifying small and sneaky DDoS attacks can be challenging, especially at L7. In this post, we discuss this challenge along with trends that we’ve seen, interesting DDoS attacks, and how we’ve responded to them so that you don’t have to worry.
Let’s Talk Trends
When analyzing attacks on the Cloudflare network, we’ve seen a steady decline in the proportion of L3/L4 DDoS attacks that exceed a rate of 30 Gbps in recent months. From September 2019 to March 2020, attacks peaking over 30 Gbps decreased by 82%, and in March 2020, more than 95% of all network-layer DDoS attacks peaked below 30 Gbps. Over the same time period, the average size of a DDoS attack has also steadily decreased by 53%, to just 11.88 Gbps. Yet, very large attacks have not disappeared: we’re still seeing attacks with intensive rates peaking at 330 Gbps on average and up to 400 millions packets per second. Some of our customers are being targeted with as many as 890 DDoS attacks in a single day and 1,750 DDoS attacks in a month.
As the average rate of these L3/L4 attacks has decreased, they have become more localized and less geographically distributed. Increasingly, we’re seeing attacks hit just one or two of our data centers, which means that these hyper-localized attacks were launched in the catchment of the data center, otherwise our Anycast network would have spread the attack surface across our global fleet of data centers. Counterintuitively, these hyper-localized floods can be more difficult to detect on a global scale as the attack samples get diluted when aggregated from all of our data centers in the core. Therefore we’ve had to change our tactics and systems to roll with the change in attacker behavior.
Keeping things interesting in the penthouse floor of the OSI Model, over the same time period we’ve also observed some of the most rate-intensive and highly distributed L7 HTTP DDoS attacks we’ve ever seen. These attacks have pushed our engineering teams to invent even more efficient and intelligent ways to defend our network and our customers at scale. Let’s take a look at some of these trends and attacks.
Before we released dosd late last year, the primary automated system responsible for protecting Cloudflare and our customers against distributed rate-intensive attacks was Gatebot. Gatebot works by ingesting samples of flow data from routers and samples of HTTP requests from servers. It then analyzes these samples for anomalies, and when attacks are detected, pushes mitigation instructions automatically to the edge.
Gatebot requires a lot of computational power to analyze these samples, and correlate them across all the data centers, so it runs centrally in our “core” data centers, rather than at the edge. It does a terrific job at mitigating large attacks, and on average stops over 4,000 L3/L4 DDoS attacks every month.
Edge Analyzed, Edge Enforced Mitigations
The persistent increase we’ve observed in smaller, more localized attacks was one of the main factors that drove us to develop a new, complementary system to Gatebot. We call this new system our denial of service daemon, or “dosd”, and this past month alone it mitigated 281,746 L3/4 DDoS attacks. This figure is roughly 6 times greater than what Gatebot dropped over the same period, thanks to dosd’s ability to detect smaller network attacks that would previously have flown under the radar (or taken longer to mitigate).
To complement the computationally heavy, centralized deployments of Gatebot, dosd was architected as a decentralized system that runs on every single server in every one of our data centers. Each instance detects and mitigates attacks independent of the other instances, or any sort of centralized data center whatsoever. As a result, the system is much faster than Gatebot, and can detect and mitigate attacks within 0-3 seconds (and less than 10 seconds on average). The speed of dosd enables it to generate real-time rules to quickly protect our customers at the data center. Then Gatebot, which samples traffic globally, can determine a mitigation that applies to all data centers if needed. In such a case, Gatebot will push rules to the data centers which will take priority over dosd’s rules.
dosd is also a leaner piece of software, consumes less memory and CPU, and significantly improves the resiliency of our network by removing the need to communicate with our core data centers to mitigate attacks. dosd detects and mitigates attacks using a similar logic to Gatebot’s methods, but in the scope of a single server, across a subset of servers in the same data center, or even across the entire data center.
Changing L7 Trends
Example #1 – Highly Distributed DDoS Attack Targeting A Customer Website
In July 2019, Cloudflare mitigated an HTTP DDoS attack that peaked at 1.4M requests per second. While this isn’t the most rate-intensive attack that we’ve seen, what is interesting is that the attack originated from almost 1.1M unique IP addresses. These were actual clients with the ability to complete a TCP and HTTPS handshake, they were not spoofed IP addresses. As it turns out, responding (rather than dropping at the network level) to over a million clients at a max rate of 1.4M requests per second can be quite costly.
Example #2 – Rate-Intensive DDoS Attack Targeting A Customer Website
The second attack took place in September 2019. We mitigated an HTTP DDoS attack that peaked and persisted just below 5M requests per second for a little over an hour. What’s interesting is the sustained capability of the attacker to reach those rates from only 371K unique IPs (also not spoofed).
These attacks highlighted to us what needed to be optimized and consequently drove us to improve our L7 mitigations even more so, and significantly reduced the cost of mitigating an attack.
Using IP Jails to Reduce the Cost of Mitigation
With the goal of reducing the computational cost to Cloudflare of mitigating rate-intensive attacks, we recently rolled out a new Gatebot capability called IP Jails. IP Jails excelsat efficiently mitigating extremely rate-intensive and distributed HTTP DDoS attacks. It is triggered when an attack exceeds a certain request rate and then pushes the mitigation from the application layer (L7 in the OSI model) to the transport layer (L4). Therefore instead of responding with an error or challenge page from the proxy, we simply drop the connection for that IP. Mitigating at L4 is more computationally efficient, it reduces our CPU and memory consumption in addition to saving bandwidth. It allows us to keep mitigating the largest of attacks without sacrificing performance.
IP Jails in action
In the first graph below, you can see an HTTP flood peaking just below 8M rps before the IPs are ‘jailed’ for misbehaving. In the second graph, you can see that same attack being dropped as packets at L4.
The flood requests generated over 130 Gbps in responses. IP Jails slashed it by a factor of 10.
Similarly, you can see a spike in the attack mitigation CPU usage which then drops back to normal after IP Jails kicks in.
Using Origin Errors to Catch Low-Rate Attacks
We see one or two of these rate-intensive attacks every month. But the vast majority of attacks we observe are mostly of a lower request rate, trying to sneak under the radar. To tackle these low-rate attacks better, last month we completed the rollout of a new capability that synchronizes Gatebot’s detection sensitivity with our customers’ origin server health. Gatebot uses the origin’s error response codes as an additional adaptive feedback signal.
However, when we take a step back and think about what a DDoS attack is actually, we usually think of a malicious actor that targets traffic at a specific website or IP address with the intent to degrade performance or cause an outage. However, malicious attackers are not the only threats to your applications availability.
As the migration of functionality to the edge increases, the cloud becomes smarter and more powerful, which often allows administrators to scale down their origin servers and infrastructure leaving the origin server weaker and under-configured. Evidently, there are many cases where an origin was taken down by small floods of traffic that were neither malicious nor generated with bad intentions. These floods may be generated by an overly excited good bot or even faulty client applications calling home too frequently. Fixing a home-sick client application or strengthening a server can be lengthy and costly processes during which the origin remains susceptible. Consequently, if a website is taken offline, no matter the reason, the end-users still experience it as if it were an attack.
Therefore this new capability not only protects our customers against DDoS attacks, but also protects the origin against all kinds of unwanted floods. It is designed to protect every one of our customers; big or small. It’s available on all of our plans including the Free plan.
When an origin responds to Cloudflare with an increasing rate of errors from the 500 range (Internal Server Error), Gatebot initiates automatically and analyzes traffic to reduce or eliminate the impact on the origin even faster than before. The current error rate is also compared to the average error rate to minimize false positives. Once an attack is detected, dynamically generated, ephemeral mitigation rules are propagated to Cloudflare’s edge data centers to mitigate the flood. Mitigation rules may use a block action (403), rate-limit (429), or even a challenge based on the fingerprint logic and confidence.
In March 2020, we mitigated 812 HTTP DDoS attacks on average every day, and approximately 20,000 HTTP DDoS attacks in total.
Don’t Take Our Word For It, See For Yourself
Whether it’s Gatebot or dosd that mitigated L3/4 DDoS attacks, you can see both types of attack events for yourself in our new Network Analytics dashboard.
Today this dashboard provides Magic Transit & BYOIP customers real-time visibility into L3/4 traffic and DDoS attacks, and in the future we plan to expand access to customers of our other products.
Whether you’re part of a large global enterprise, or use Cloudflare for your personal site on the Free plan, we want to make sure that you’re protected and also have the visibility that you need.
DDoS Protection is included as part of every Cloudflare service; from Magic Transit at L3, through Spectrum at L4, to the WAF/CDN service at L7. Our mission is to help build a better Internet – and this means a safer, faster, and more reliable Internet. For everyone.
If you’re a Cloudflare customer of any plan (Free, Pro, Business or Enterprise), these new protections are now enabled by default at no additional charge.
The collective thoughts of the interwebz
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.