During the Holocaust, and in the events that led to it, the Nazis exterminated one third of the European Jewish population. Six million Jews, along with countless other members of minority and disability groups, were murdered because the Nazis believed they were inferior.
Cloudflare’s Project Galileo provides free protection to at-risk groups across the world including Holocaust educational and remembrance websites. During the past year alone, Cloudflare mitigated over a quarter of a million cyber threats launched against Holocaust-related websites.
Antisemitism and the Final Solution
In the Second World War and the years leading up to it, antisemitism served as the foundation of racist laws and fueled violent Pogroms against Jews. The tipping point was a night of violence known as the Kristallnacht (“Night of Broken Glass”). Jews and other minority groups were outlawed, dehumanized, persecuted and killed. Jewish businesses were boycotted, Jewish books burned and synagogues destroyed. Jews, Roma and other “enemies of the Reich” were forced into closed ghettos and concentration camps. Finally, as part of the Final Solution for the Jewish Question, Germany outlined a policy to deliberately and systematically exterminate the Jewish race in what came to be known as the Holocaust.
As part of the Final Solution, the Nazis deployed mobile killing units. Jews were taken to forests near their villages, forced to dig mass graves, undress, and then shot — falling into the mass graves they dug. This was the first step. However, this was “inefficient”. More “efficient” solutions were engineered using deadly gas. Eventually, six main extermination camps were established. They were extremely “efficient” at exterminating humans. Initially, the Nazis experimented with gas vans for mass extermination. Later, they built and operated gas chambers which could kill more humans and do it faster. After being gassed, prisoners would load the bodies into ovens in crematoriums to be burned. In one of the larger death camps, Auschwitz-Birkenau, more than one million Jews were murdered — some 865,000 were gassed and burned on arrival.
It is through education that we will defeat bigotry and racism, and we will do our part at Cloudflare — through education and by supporting Holocaust educational organizations.
“Our response to ignorance must be education” – United Nations Secretary-General António Guterres
Supporting Holocaust educational organizations with Project Galileo
As part of Project Galileo, we currently provide free security and performance products to more than 1,500 organizations in 111 countries. These organizations are targeted by cyber attacks due to their critical work. These groups include human rights defenders, independent media and journalists, and organizations that work in strengthening democracy. Among them are organizations dedicated to educating about the horrors of the Holocaust, and preserving and telling the stories of the victims and survivors of the Holocaust to younger and future generations.
Cyber attacks on Holocaust-related websites
Over the past year, we’ve seen cyber attacks on Holocaust-related websites gradually increase throughout the year. These attacks include mostly application-layer attacks that were automatically detected and mitigated by Cloudflare’s Web Application Firewall and DDoS Protection systems.
In May 2021, cyber attacks on Holocaust-related websites peaked as they increased by 263% compared to their monthly average.
Applying to Project Galileo
Cloudflare’s mission is to help build a better Internet. Part of this mission includes protecting free expression online for vulnerable groups.
The Internet can be a powerful tool in this matter. However, organizations often face attacks from powerful and entrenched opponents, yet operate on limited budgets and lack the resources to secure themselves against malicious traffic intended to silence them. If they are silenced, the Internet stops fulfilling its promise.
To combat the threats, Cloudflare’s Project Galileo provides robust security and performance products for at-risk public interest websites at no cost. Application to Project Galileo is open to any vulnerable public interest website. You can apply via our partners or apply directly to Project Galileo if you don’t have any affiliation with our trusted partners.
A note from Cloudflare’s Jewish employees
Many of us, like myself, are descendants of Holocaust survivors. My grandparents fled from Nazi-occupied Poland to survive. Sadly, my grandparents — as other elderly survivors, are no longer with us. Many of us have faced antisemitism in various forms. Together, we are part of Cloudflare’s Employee Resource Group for Cloudflare’s Jewish community: Judeoflare. We have a responsibility to make sure the world remembers and never forgets the atrocities of the Holocaust and what racism and antisemitism can lead to.
We’re excited to announce that customers using our Free plan can now get real-time alerts about HTTP DDoS attacks that were automatically detected and mitigated by Cloudflare. The real-time DDoS alerts were originally announced over a year ago but were made available to customers on the Pro plan or higher. This announcement extends the DDoS alerts feature to Free plan users. You can read the original announcement blog post here.
What is a DDoS attack?
A Distributed Denial of Service (DDoS) attack is a cyber-attack that attempts to disrupt your online business. Whether your business relies on VoIP servers, UDP-based gaming servers, or HTTP servers, DDoS attacks can be used to disrupt any type of Internet property, server, or network.
In this blog post, we’ll focus on DDoS attacks that target HTTP servers. Whether your HTTP server is powering a mobile app, an eCommerce website, an API gateway, or any other HTTP application, if an attacker sends you more requests than it can handle, your server won’t be able to serve your real users. A flood of requests can cause service disruptions or even take your entire server offline. DDoS attacks can have real-world consequences such as a blow to your revenue and reputation.
How Cloudflare detects and mitigates DDoS attacks
Protecting your server against DDoS attacks requires two main capabilities:
The bandwidth to absorb both your users’ requests and the attack requests
The ability to differentiate between your users’ requests and the attack requests
Using our home-grown systems, we do just that, regardless of the size, frequency and duration of the attacks. All Cloudflare customers, including those using the Free plan, are protected by our unmetered DDoS mitigation commitment.
To protect against DDoS attacks, first, we route your traffic to our network of data centers. Our network spans more than 250 cities in over 100 countries around the world. Its capacity is over 100 Tbps — fifty times larger than the largest attack we’ve ever seen. Our bandwidth is more than enough to absorb both your users’ traffic and attack traffic.
Cloudflare’s global network
Cloudflare’s global network
Second, once your traffic reaches our data centers, it goes through state-of-the-art analysis mechanisms that constantly scan for DDoS attacks. Once an attack is detected, a real-time mitigation rule is automatically generated to surgically mitigate the attack requests based on the attack pattern, whilst leaving your users’ requests untouched. Using the HTTP DDoS Managed Ruleset you can customize the settings of the mitigation system to tailor it to your needs and specific traffic patterns.
Not sure what to do? That’s ok. For the most part, you won’t need to do anything and our system will automatically keep your servers protected. You can read more about it in our Get Started guide or in the original blog post. If you’re interested, you can also read more about how our mitigation system works in this technical blog post: A deep-dive into Cloudflare’s autonomous edge DDoS protection
Configuring a DDoS alert
Once our system detects and mitigates a DDoS attack, you’ll receive a real-time alert. To receive an alert, make sure you, first, configure a notification policy by following these steps:
Cloudflare’s mission is to help build a better Internet, and it guides everything we do. As part of this mission, we believe that a better Internet is one where enterprise-grade DDoS protection is available for everyone, not just bigger organizations.
Furthermore, we’ve also made our DDoS Managed Ruleset available for everyone to make sure that even non-paying customers can tailor and optimize their DDoS protection settings. Taking a step further, we want all of our users to be able to react as fast as possible when needed. This is why we’re providing real-time alerts for free. Knowledge is power, and notifying our users of attacks in real-time empowers them to ensure their website is safe, available, and performant.
The first half of 2021 witnessed massive ransomware and ransom DDoS attack campaigns that interrupted aspects of critical infrastructure around the world (including one of the largest petroleum pipeline system operators in the US) and a vulnerability in IT management software that targeted schools, public sector, travel organizations, and credit unions, to name a few.
The second half of the year recorded a growing swarm of one of the most powerful botnets deployed (Meris) and record-breaking HTTP DDoS attacks and network-layer attacks observed over the Cloudflare network. This besides the Log4j2 vulnerability (CVE-2021-44228) discovered in December that allows an attacker to execute code on a remote server — arguably one of the most severe vulnerabilities on the Internet since both Heartbleed and Shellshock.
Prominent attacks such as the ones listed above are but a few examples that demonstrate a trend of intensifying cyber-insecurity that affected everyone, from tech firms and government organizations to wineries and meat processing plants.
Here are some DDoS attack trends and highlights from 2021 and Q4 ‘21 specifically:
In December alone, one out of every three survey respondents reported being targeted by a ransom DDoS attack or threatened by the attacker.
Application-layer DDoS attacks
The Manufacturing industry was the most attacked in Q4 ’21, recording a whopping 641% increase QoQ in the number of attacks. The Business Services and Gaming/Gambling industries were the second and third most targeted industries by application-layer DDoS attacks.
For the fourth time in a row this year, China topped the charts with the highest percentage of attack traffic originating from its networks.
Q4 ’21 was the busiest quarter for attackers in 2021. In December 2021 alone, there were more than all the attacks observed in Q1 and Q2 ’21 separately.
While the majority of attacks were small, terabit-strong attacks became the new norm in the second half of 2021. Cloudflare automatically mitigated dozens of attacks peaking over 1 Tbps, with the largest one peaking just under 2 Tbps — the largest we’ve ever seen.
Attacks originating from Moldova quadrupled in Q4 ’21 QoQ, making it the country with the highest percentage of network-layer DDoS activity.
SYN floods and UDP floods were the most frequent attack vectors while emerging threats such as SNMP attacks increased by nearly 5,800% QoQ.
This report is based on DDoS attacks that were automatically detected and mitigated by Cloudflare’s DDoS Protection systems. To learn more about how it works, check out this deep-dive blog post.
A note on how we measure DDoS attacks observed over our network
To analyze attack trends, we calculate the “DDoS activity” rate, which is the percentage of attack traffic out of the total traffic (attack + clean) observed over our global network. Measuring attack numbers as a percentage of the total traffic observed allows us to normalize data points and avoid biases reflected in absolute numbers towards, for example, a Cloudflare data center that receives more total traffic and likely, also more attacks.
An interactive version of this report is available on Cloudflare Radar.
Ransom Attacks
Our systems constantly analyze traffic and automatically apply mitigation when DDoS attacks are detected. Each DDoS’d customer is prompted with an automated survey to help us better understand the nature of the attack and the success of the mitigation.
For over two years now, Cloudflare has been surveying attacked customers — one question on the survey being if they received a ransom note demanding payment in exchange to stop the DDoS attack. Q4 ’21 recorded the highest survey responses ever that indicated ransom threats — ransom attacks increased by 29% YoY and 175% QoQ. More specifically, one out of every 4.5 respondents (22%) reported receiving a ransom letter demanding payment by the attacker.
The percentage of respondents reported being targeted by a ransom DDoS attack or that have received threats in advance of the attack.
When we break it down by month, we can see that December 2021 topped the charts with 32% of respondents reporting receiving a ransom letter — that’s nearly one out of every three surveyed respondents.
Application-layer DDoS attacks
Application-layer DDoS attacks, specifically HTTP DDoS attacks, are attacks that usually aim to disrupt a web server by making it unable to process legitimate user requests. If a server is bombarded with more requests than it can process, the server will drop legitimate requests and — in some cases — crash, resulting in degraded performance or an outage for legitimate users.
Application-layer DDoS attacks by industry
In Q4, DDoS attacks on Manufacturing companies increased by 641% QoQ, and DDoS attacks on the Business Services industry increased by 97%.
When we break down the application-layer attacks targeted by industry, the Manufacturing, Business Services, and Gaming/Gambling industries were the most targeted industries in Q4 ’21.
Application-layer DDoS attacks by source country
To understand the origin of the HTTP attacks, we look at the geolocation of the source IP address belonging to the client that generated the attack HTTP requests. Unlike network-layer attacks, source IP addresses cannot be spoofed in HTTP attacks. A high percentage of DDoS activity in a given country usually indicates the presence of botnets operating from within the country’s borders.
For the fourth quarter in a row, China remains the country with the highest percentage of DDoS attacks originating from within its borders. More than three out of every thousand HTTP requests that originated from Chinese IP addresses were part of an HTTP DDoS attack. The US remained in second place, followed by Brazil and India.
Application-layer DDoS attacks by target country
In order to identify which countries are targeted by the most HTTP DDoS attacks, we bucket the DDoS attacks by our customers’ billing countries and represent it as a percentage out of all DDoS attacks.
For the third consecutive time this year, organizations in the United States were targeted by the most HTTP DDoS attacks, followed by Canada and Germany.
Network-layer DDoS attacks
While application-layer attacks target the application (Layer 7 of the OSI model) running the service that end users are trying to access, network-layer attacks aim to overwhelm network infrastructure (such as in-line routers and servers) and the Internet link itself.
Cloudflare thwarts an almost 2 Tbps attack
In November, our systems automatically detected and mitigated an almost 2 Tbps DDoS attack. This was a multi-vector attack combining DNS amplification attacks and UDP floods. The entire attack lasted just one minute. The attack was launched from approximately 15,000 bots running a variant of the original Mirai code on IoT devices and unpatched GitLab instances.
Network-layer DDoS attacks by month
December was the busiest month for attackers in 2021.
Q4 ‘21 was the busiest quarter in 2021 for attackers. Over 43% of all network-layer DDoS attacks took place in the fourth quarter of 2021. While October was a relatively calmer month, in November, the month of the Chinese Singles’ Day, the American Thanksgiving holiday, Black Friday, and Cyber Monday, the number of network-layer DDoS attacks nearly doubled. The number of observed attacks increased towards the final days of December ’21 as the world prepared to close out the year. In fact, the total number of attacks in December alone was higher than all the attacks in Q2 ’21 and almost equivalent to all attacks in Q1 ’21.
Network-layer DDoS attacks by attack rate
While most attacks are still relatively ‘small’ in size, terabit-strong attacks are becoming the norm.
There are different ways of measuring the size of an L3/4 DDoS attack. One is the volume of traffic it delivers, measured as the bit rate (specifically, terabits per second or gigabits per second). Another is the number of packets it delivers, measured as the packet rate (specifically, millions of packets per second).
Attacks with high bit rates attempt to cause a denial-of-service event by clogging the Internet link, while attacks with high packet rates attempt to overwhelm the servers, routers, or other in-line hardware appliances. These devices dedicate a certain amount of memory and computation power to process each packet. Therefore, by bombarding it with many packets, the appliance can be left with no further processing resources. In such a case, packets are “dropped,” i.e., the appliance is unable to process them. For users, this results in service disruptions and denial of service.
The distribution of attacks by their size (in bit rate) and month is shown below. As seen in the graph above, the majority of attacks took place in December. However, the graph below illustrates that larger attacks, over 300 Gbps in size, took place in November. Most of the attacks between 5-20 Gbps took place in December.
Distribution by packet rate
An interesting correlation Cloudflare has observed is that when the number of attacks increases, their size and duration decrease. In the first two-thirds of 2021, the number of attacks was relatively small, and correspondingly, their rates increased, e.g., in Q3 ’21, attacks ranging from 1-10 million packets per second (mpps) increased by 196%. In Q4 ’21, the number of attacks increased and Cloudflare observed a decrease in the size of attacks. 91% of all attacks peaked below 50,000 packets per second (pps) — easily sufficient to take down unprotected Internet properties.
Larger attacks of over 1 mpps decreased by 48% to 28% QoQ, while attacks peaking below 50K pps increased by 2.36% QoQ.
Distribution by bit rate
Similar to the trend observed in packet-intensive attacks, the amount of bit-intensive attacks shrunk as well. While attacks over 1 Tbps are becoming the norm, with the largest one we’ve ever seen peak just below 2 Tbps, the majority of attacks are still small and peaked below 500 Mbps (97.2%).
In Q4 ’21, larger attacks of all ranges above 500 Mbps saw massive decreases ranging from 35% to 57% for the larger 100+ Gbps attacks.
Network-layer DDoS attacks by duration
Most attacks remain under one hour in duration, reiterating the need for automated always-on DDoS mitigation solutions.
We measure the duration of an attack by recording the difference between when it is first detected by our systems as an attack and the last packet we see with that attack signature towards that specific target. In the last quarter of 2021, 98% of all network-layer attacks lasted less than one hour. This is very common as most of the attacks are short-lived. Even more so, a trend we’ve seen is that when the number of attacks increases, as in this quarter, their rate and duration decreases.
Short attacks can easily go undetected, especially burst attacks that, within seconds, bombard a target with a significant number of packets, bytes, or requests. In this case, DDoS protection services that rely on manual mitigation by security analysis have no chance in mitigating the attack in time. They can only learn from it in their post-attack analysis, then deploy a new rule that filters the attack fingerprint and hope to catch it next time. Similarly, using an “on-demand” service, where the security team will redirect traffic to a DDoS provider during the attack, is also inefficient because the attack will already be over before the traffic routes to the on-demand DDoS provider.
It’s recommended that companies use automated, always-on DDoS protection services that analyze traffic and apply real-time fingerprinting fast enough to block short-lived attacks.
Attack vectors
SYN floods remain attackers’ favorite method of attack, while attacks over SNMP saw a massive surge of almost 5,800% QoQ.
An attack vector is a term used to describe the method that the attacker uses to launch their DDoS attack, i.e., the IP protocol, packet attributes such as TCP flags, flooding method, and other criteria.
For the first time in 2021, the percentage of SYN flood attacks significantly decreased. Throughout 2021, SYN floods accounted for 54% of all network-layer attacks on average. While still grabbing first place as the most frequent vector, its share dropped by 38% QoQ to 34%.
However, it was a close-run for SYN attacks and UDP attacks. A UDP flood is a type of denial-of-service attack in which a large number of User Datagram Protocol (UDP) packets are sent to a targeted server with the aim of overwhelming that device’s ability to process and respond. Oftentimes, the firewall protecting the targeted server can also become exhausted as a result of UDP flooding, resulting in a denial-of-service to legitimate traffic. Attacks over UDP jumped from fourth place in Q3 ’21 to second place in Q4 ’21, with a share of 32% of all network-layer attacks — a 1,198% increase in QoQ.
In third place came the SNMP underdog that made a massive leap with its first time 2021 appearance in the top attack vectors.
Emerging threats
When we look at emerging attack vectors — which helps us understand what new vectors attackers are deploying to launch attacks — we observe a massive spike in SNMP, MSSQL, and generic UDP-based DDoS attacks.
Both SNMP and MSSQL attacks are used to reflect and amplify traffic on the target by spoofing the target’s IP address as the source IP in the packets used to trigger the attack.
Simple Network Management Protocol (SNMP) is a UDP-based protocol that is often used to discover and manage network devices such as printers, switches, routers, and firewalls of a home or enterprise network on UDP well-known port 161. In an SNMP reflection attack, the attacker sends out a large number of SNMP queries while spoofing the source IP address in the packet as the targets to devices on the network that, in turn, reply to that target’s address. Numerous responses from the devices on the network results in the target network being DDoSed.
Similar to the SNMP amplification attack, the Microsoft SQL (MSSQL) attack is based on a technique that abuses the Microsoft SQL Server Resolution Protocol for the purpose of launching a reflection-based DDoS attack. The attack occurs when a Microsoft SQL Server responds to a client query or request, attempting to exploit the Microsoft SQL Server Resolution Protocol (MC-SQLR), listening on UDP port 1434.
Network-layer DDoS attacks by country
Attacks originating from Moldova quadrupled, making it the country with the highest percentage of network-layer DDoS activity.
When analyzing network-layer DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations where the traffic was ingested, and not by the source IP. The reason for this is that, when attackers launch network-layer attacks, they can spoof the source IP address in order to obfuscate the attack source and introduce randomness into the attack properties, which can make it harder for simple DDoS protection systems to block the attack. Hence, if we were to derive the source country based on a spoofed source IP, we would get a spoofed country.
Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of the Cloudflare data center in which the attack was observed. We are able to achieve geographical accuracy in our report because we have data centers in over 250 cities around the world.
To view all regions and countries, check out the interactive map.
Summary
Cloudflare’s mission is to help build a better Internet. A better Internet is one that is more secure, faster, and reliable for everyone — even in the face of DDoS attacks. As part of our mission, since 2017, we’ve been providing unmetered and unlimited DDoS protection for free to all of our customers. Over the years, it has become increasingly easier for attackers to launch DDoS attacks. To counter the attacker’s advantage, we want to make sure that it is also easy and free for organizations of all sizes to protect themselves against DDoS attacks of all types.
After initially providing our customers control over the HTTP-layer DDoS protection settings earlier this year, we’re now excited to extend the control our customers have to the packet layer. Using these new controls, Cloudflare Enterprise customers using the Magic Transit and Spectrum services can now tune and tweak their L3/4 DDoS protection settings directly from the Cloudflare dashboard or via the Cloudflare API.
The new functionality provides customers control over two main DDoS rulesets:
Network-layer DDoS Protectionruleset — This ruleset includes rules to detect and mitigate DDoS attacks on layer 3/4 of the OSI model such as UDP floods, SYN-ACK reflection attacks, SYN Floods, and DNS floods. This ruleset is available for Spectrum and Magic Transit customers on the Enterprise plan.
Advanced TCP Protectionruleset — This ruleset includes rules to detect and mitigate sophisticated out-of-state TCP attacks such as spoofed ACK Floods, Randomized SYN Floods, and distributed SYN-ACK Reflection attacks. This ruleset is available for Magic Transit customers only.
A Distributed Denial of Service (DDoS) attack is a type of cyberattack that aims to disrupt the victim’s Internet services. There are many types of DDoS attacks, and they can be generated by attackers at different layers of the Internet. One example is the HTTP flood. It aims to disrupt HTTP application servers such as those that power mobile apps and websites. Another example is the UDP flood. While this type of attack can be used to disrupt HTTP servers, it can also be used in an attempt to disrupt non-HTTP applications. These include TCP-based and UDP-based applications, networking services such as VoIP services, gaming servers, cryptocurrency, and more.
To defend organizations against DDoS attacks, we built and operate software-defined systems that run autonomously. They automatically detect and mitigate DDoS attacks across our entire network. You can read more about our autonomous DDoS protection systems and how they work in our deep-dive technical blog post.
Unmetered and unlimited DDoS Protection
The level of protection that we offer is unmetered and unlimited — It is not bounded by the size of the attack, the number of the attacks, or the duration of the attacks. This is especially important these days because as we’ve recently seen, attacks are getting larger and more frequent. Consequently, in Q3, network-layer attacks increased by 44% compared to the previous quarter. Furthermore, just recently, our systems automatically detected and mitigated a DDoS attack that peaked just below 2 Tbps — the largest we’ve seen to date.
Mirai botnet launched an almost 2 Tbps DDoS attack
You can think of our autonomous DDoS protection systems as groups (rulesets) of intelligent rules. There are rulesets of HTTP DDoS Protection rules, Network-layer DDoS Protection rules and Advanced TCP Protection rules. In this blog post, we will cover the latter two rulesets. We’ve already covered the former in the blog post How to customize your HTTP DDoS protection settings.
Cloudflare L3/4 DDoS Managed Rules
In the Network-layer DDoS Protection rulesets, each rule has a unique set of conditional fingerprints, dynamic field masking, activation thresholds, and mitigation actions. These rules are managed (by Cloudflare), meaning that the specifics of each rule is curated in-house by our DDoS experts. Before deploying a new rule, it is first rigorously tested and optimized for mitigation accuracy and efficiency across our entire global network.
In the Advanced TCP Protection ruleset, we use a novel TCP state classification engine to identify the state of TCP flows. The engine powering this ruleset is flowtrackd — you can read more about it in our announcement blog post. One of the unique features of this system is that it is able to operate using only the ingress (inbound) packet flows. The system sees only the ingress traffic and is able to drop, challenge, or allow packets based on their legitimacy. For example, a flood of ACK packets that don’t correspond to open TCP connections will be dropped.
How attacks are detected and mitigated
Sampling
Initially, traffic is routed through the Internet via BGP Anycast to the nearest Cloudflare edge data center. Once the traffic reaches our data center, our DDoS systems sample it asynchronously allowing for out-of-path analysis of traffic without introducing latency penalties. The Advanced TCP Protection ruleset needs to view the entire packet flow and so it sits inline for Magic Transit customers only. It, too, does not introduce any latency penalties.
Analysis & mitigation
The analysis for the Advanced TCP Protection ruleset is straightforward and efficient. The system qualifies TCP flows and tracks their state. In this way, packets that don’t correspond to a legitimate connection and its state are dropped or challenged. The mitigation is activated only above certain thresholds that customers can define.
The analysis for the Network-layer DDoS Protection ruleset is done using data streaming algorithms. Packet samples are compared to the conditional fingerprints and multiple real-time signatures are created based on the dynamic masking. Each time another packet matches one of the signatures, a counter is increased. When the activation threshold is reached for a given signature, a mitigation rule is compiled and pushed inline. The mitigation rule includes the real-time signature and the mitigation action, e.g., drop.
Example
As a simple example, one fingerprint could include the following fields: source IP, source port, destination IP, and the TCP sequence number. A packet flood attack with a fixed sequence number would match the fingerprint and the counter would increase for every packet match until the activation threshold is exceeded. Then a mitigation action would be applied.
However, in the case of a spoofed attack where the source IP addresses and ports are randomized, we would end up with multiple signatures for each combination of source IP and port. Assuming a sufficiently randomized/distributed attack, the activation thresholds would not be met and mitigation would not occur. For this reason, we use dynamic masking, i.e. ignoring fields that may not be a strong indicator of the signature. By masking (ignoring) the source IP and port, we would be able to match all the attack packets based on the unique TCP sequence number regardless of how randomized/distributed the attack is.
Configuring the DDoS Protection Settings
For now, we’ve only exposed a handful of the Network-layer DDoS protection rules that we’ve identified as the ones most prone to customizations. We will be exposing more and more rules on a regular basis. This shouldn’t affect any of your traffic.
Overriding the sensitivity level and mitigation action
For the Network-layer DDoS Protection ruleset, for each of the available rules, you can override the sensitivity level (activation threshold), customize the mitigation action, and apply expression filters to exclude/include traffic from the DDoS protection system based on various packet fields. You can create multiple overrides to customize the protection for your network and your various applications.
Configuring expression fields for the DDoS Managed Rules to match on
In the past, you’d have to go through our support channels to customize the rules. In some cases, this may have taken longer to resolve than desired. With today’s announcement, you can tailor and fine-tune the settings of our autonomous edge system by yourself to quickly improve the accuracy of the protection for your specific network needs.
For the Advanced TCP Protection ruleset, for now, we’ve only exposed the ability to enable or disable it as a whole in the dashboard. To enable or disable the ruleset per IP prefix, you must use the API. At this time, when initially onboarding to Cloudflare, the Cloudflare team must first create a policy for you. After onboarding, if you need to change the sensitivity thresholds, use Monitor mode, or add filter expressions you must contact Cloudflare Support. In upcoming releases, this too will be available via the dashboard and API without requiring help from our Support team.
Pre-existing customizations
If you previously contacted Cloudflare Support to apply customizations, your customizations have been preserved, and you can visit the dashboard to view the settings of the Network-layer DDoS Protection ruleset and change them if you need. If you require any changes to your Advanced TCP Protection customizations, please reach out to Cloudflare Support.
If so far you didn’t have the need to customize this protection, there is no action required on your end. However, if you would like to view and customize your DDoS protection settings, follow this dashboard guide or review the API documentation to programmatically configure the DDoS protection settings.
Helping Build a Better Internet
At Cloudflare, everything we do is guided by our mission to help build a better Internet. The DDoS team’s vision is derived from this mission: our goal is to make the impact of DDoS attacks a thing of the past. Our first step was to build the autonomous systems that detect and mitigate attacks independently. Done. The second step was to expose the control plane over these systems to our customers (announced today). Done. The next step will be to fully automate the configuration with an auto-pilot feature — training the systems to learn your specific traffic patterns to automatically optimize your DDoS protection settings. You can expect many more improvements, automations, and new capabilities to keep your Internet properties safe, available, and performant.
Earlier this week, Cloudflare automatically detected and mitigated a DDoS attack that peaked just below 2 Tbps — the largest we’ve seen to date. This was a multi-vector attack combining DNS amplification attacks and UDP floods. The entire attack lasted just one minute. The attack was launched from approximately 15,000 bots running a variant of the original Mirai code on IoT devices and unpatched GitLab instances.
DDoS attack peaking just below 2 Tbps
Network-layer DDoS attacks increased by 44%
Last quarter, we saw multiple terabit-strong DDoS attacks and this attack continues this trend of increased attack intensity. Another key finding from our Q3 DDoS Trends report was that network-layer DDoS attacks actually increased by 44% quarter-over-quarter. While the fourth quarter is not over yet, we have, again, seen multiple terabit-strong attacks that targeted Cloudflare customers.
DDoS attacks peaking at 1-1.4 Tbps
How did Cloudflare mitigate this attack?
To begin with, our systems constantly analyze traffic samples “out-of-path” which allows us to asynchronously detect DDoS attacks without causing latency or impacting performance. Once the attack traffic was detected (within sub-seconds), our systems generated a real-time signature that surgically matched against the attack patterns to mitigate the attack without impacting legitimate traffic.
Once generated, the fingerprint is propagated as an ephemeral mitigation rule to the most optimal location in the Cloudflare edge for cost-efficient mitigation. In this specific case, as with most L3/4 DDoS attacks, the rule was pushed in-line into the Linux kernel eXpress Data Path (XDP) to drop the attack packet at wirespeed.
A conceptual diagram of Cloudflare’s DDoS protection systems
Cloudflare’s mission is to help build a better Internet — one that is secure, faster, and more reliable for everyone. The DDoS team’s vision is derived from this mission: our goal is to make the impact of DDoS attacks a thing of the past. Whether it’s the Meris botnet that launched some of the largest HTTP DDoS attacks on record, the recent attacks on VoIP providers or this Mirai-variant that’s DDoSing Internet properties, Cloudflare’s network automatically detects and mitigates DDoS attacks. Cloudflare provides a secure, reliable, performant, and customizable platform for Internet properties of all types.
For more information about Cloudflare’s DDoS protection, reach out to us or have a go with a hands-on evaluation of Cloudflare’s Free plan here.
Attackers continue targeting VoIP infrastructure around the world. In our blog from last week, May I ask who’s calling, please? A recent rise in VoIP DDoS attacks, we reviewed how the SIP protocol works, ways it can be abused, and how Cloudflare can help protect against attacks on VoIP infrastructure without impacting performance.
Cloudflare’s network stands in front of some of the largest, most performance-sensitive voice and video providers in the world, and is uniquely well suited to mitigating attacks on VoIP providers.
Because of the sustained attacks we are observing, we are sharing details on recent attack patterns, what steps they should take before an attack, and what to do after an attack has taken place.
Below are three of the most common questions we’ve received from companies concerned about attacks on their VoIP systems, and Cloudflare’s answers.
Question #1: How is VoIP infrastructure being attacked?
The attackers primarily use off-the-shelf booter services to launch attacks against VoIP infrastructure. The attack methods being used are not novel, but the persistence of the attacker and their attempts to understand the target’s infrastructure are.
Attackers have used various attack vectors to probe the existing defenses of targets and try to infiltrate any existing defenses to disrupt VoIP services offered by certain providers. In some cases, they have been successful. HTTP attacks against API gateways and the corporate websites of the providers have been combined with network-layer and transport-layer attack against VoIP infrastructures. Examples:
TCP floods targeting stateful firewalls These are being used in “trial-and-error” type attacks. They are not very effective against telephony infrastructure specifically (because it’s mostly UDP) but very effective at overwhelming stateful firewalls.
UDP floods targeting SIP infrastructure Floods of UDP traffic that have no well-known fingerprint, aimed at critical VoIP services. Generic floods like this may look like legitimate traffic to unsophisticated filtering systems.
UDP reflection targeting SIP infrastructure These methods, when targeted at SIP or RTP services, can easily overwhelm Session Border Controllers (SBCs) and other telephony infrastructure. The attacker seems to learn enough about the target’s infrastructure to target such services with high precision.
SIP protocol-specific attacks Attacks at the application layer are of particular concern because of the higher resource cost of generating application errors vs filtering on network devices.
Question #2: How should I prepare my organization in case our VoIP infrastructure is targeted?
Deploy an always-on DDoS mitigation service Cloudflare recommends the deployment of always-on network level protection, like Cloudflare Magic Transit, prior to your organization being attacked.
Do not rely on reactive on-demand SOC-based DDoS Protection services that require humans to analyze attack traffic — they take too long to respond. Instead, onboard to a cloud service that has sufficient network capacity and automated DDoS mitigation systems.
Cloudflare has effective mitigations in place for the attacks seen against VoIP infrastructure, including for sophisticated TCP floods and SIP specific attacks.
Enforce a positive security model Block TCP on IP/port ranges that are not expected to receive TCP, instead of relying on on-premise firewalls that can be overwhelmed. Block network probing attempts (e.g. ICMP) and other packets that you don’t normally expect to see.
Build custom mitigation strategies Work together with your DDoS protection vendor to tailor mitigation strategies to your workload. Every network is different, and each poses unique challenges when integrating with DDoS mitigation systems.
Educate your employees Train all of your employees to be on the lookout for ransom demands. Check email, support tickets, form submissions, and even server access logs. Ensure employees know to immediately report ransom demands to your Security Incident Response team.
Question #3: What should I do if I receive a ransom/threat?
Do not to pay the ransom Paying the ransom only encourages bad actors—and there’s no guarantee that they won’t attack your network now or later.
Notify Cloudflare We can help ensure your website and network infrastructure are safeguarded against these attacks.
Notify local law enforcement They will also likely request a copy of the ransom letter that you received.
Cloudflare is here to help
With over 100 Tbps of network capacity, a network architecture that efficiently filters traffic close to the source, and a physical presence in over 250 cities, Cloudflare can help protect critical VoIP infrastructure without impacting latency, jitter, and call quality. Test results demonstrate a performance improvement of 36% on average across the globe for a real customer network using Cloudflare Magic Transit.
Some of the largest voice and video providers in the world rely on Cloudflare to protect their networks and ensure their services remain online and fast. We stand ready to help.
Talk to a Cloudflare specialist to learn more. Under attack? Contact our hotline to speak with someone immediately.
We’re excited to announce the availability of the HTTP DDoS Managed Ruleset. This new feature allows Cloudflare customers to independently tailor their HTTP DDoS protection settings. Whether you’re on the Free plan or the Enterprise plan, you can now tweak and optimize the settings directly from within the Cloudflare dashboard or via API.
We expect that in most cases, Cloudflare customers won’t need to customize any settings. Our mission is to make DDoS disruptions a thing of the past, with no customer overhead. To achieve this mission we’re constantly investing in our automated detection and mitigation systems. In some rare cases, there is a need to make some configuration changes, and so now, Cloudflare customers can customize those protection mechanisms independently. The next evolutionary step is to make those settings learn and auto-tune themselves for our customers, based on their unique traffic patterns. Zero-touch DDoS protection at scale.
Unmetered DDoS Protection
Back in 2017, we announced that we will never kick a customer off of our network because they face large attacks, even if they are not paying us at all (i.e., using the Free plan). Furthermore, we committed to never charge a customer for DDoS attack traffic — no matter the size and duration of the attack. Just this summer, our systems automatically detected and mitigated one of the largest DDoS attacks of all time. As opposed to other vendors, Cloudflare customers don’t need to request a service credit for the attack traffic: we simply exclude DDoS traffic from our billing systems. This is done automatically, just like our attack detection and mitigation mechanisms.
Autonomous DDoS Protection
Our unmetered DDoS protection commitment is possible due to our ongoing investment in our network and technology stack. The global coverage and capacity of our network allows us to absorb the largest attacks ever recorded, without manual intervention. Using BGP Anycast, traffic is routed to the closest Cloudflare edge data center as a form of global inter-data center load balancing. Traffic is then load balanced efficiently inside the data center between servers with the help of Unimog, our home-grown L4 load balancer, to ensure that traffic arrives at the least loaded server. Then, each server scans for malicious traffic and, if detected, applies mitigations in the most optimal location in the tech stack. Each server detects and mitigates attacks completely autonomously without requiring any centralized consensus, and shares details with each other using multicast. This is done using our proprietary autonomous edge detection and mitigation system, and this is how we’re able to continue offering unmetered DDoS protection for free at the scale we operate at.
Configurable DDoS Protection
Our autonomous systems use a set of dynamic rules that scan for attack patterns, known attack tools, suspicious patterns, protocol violations, requests causing large amounts of origin errors, excessive traffic hitting the origin/cache, and additional attack vectors. Each rule has a predefined sensitivity level and default action that varies based on the rule’s confidence that the traffic is indeed part of an attack.
But how do we determine those confidence levels? The answer to that depends on each specific rule and what that rule is looking for. Some rules look for the patterns in HTTP attributes that are generated by known attack tools and botnets, known protocol violations and other general suspicious patterns andtraffic abnormalities. If a given rule is searching for the HTTP patterns of known attack tools, then once found, the likelihood (i.e., confidence) that this traffic is part of an attack is high, and we can therefore safely block all the traffic that matches that rule. However, in other cases, the detected patterns or abnormal activity might resemble an attack but can actually be caused by faulty applications that generate abnormal HTTP calls, misbehaving API clients that flood their origin server, and even legitimate traffic that naively violates protocol standards. In those cases, we might want to rate-limit the traffic that matches the rule or serve a challenge action to verify and allow legitimate users in while blocking bad bots and attackers.
Configuring the DDoS Protection Settings
In the past, you’d have to go through our support channels to customize any of the default actions and sensitivity levels. In some cases, this may have taken longer to resolve than desired. With today’s release, you can tailor and fine-tune the settings of our autonomous edge system by yourself to quickly improve the accuracy of the protection for your specific application needs.
If you previously contacted Cloudflare Support to apply customizations, the DDoS Ruleset has been set to Essentially off or Low for your zone, based on your existing customization. You can visit the dashboard to view the settings and change them if you need.
If you’ve requested to exclude or bypass the mitigations for specific HTTP attributes or IPs, or if you’ve requested a significantly high threshold that requires Cloudflare approval, then those customizations are still active but may not yet be visible in the dashboard.
If you haven’t experienced this issue previously, there is no action required on your end. However, if you would like to customize your DDoS protection settings, go directly to the DDoS tab or follow these steps:
Next to HTTP DDoS attack protection, click Configure.
In Ruleset configuration, select the action and sensitivity values for all the rules in the HTTP DDoS Managed Ruleset.
Alternatively, follow the API documentation to programmatically configure the DDoS protection settings.
In the configuration page, you can select a different Action and Sensitivity Level to override all the DDoS protection rules as a group of rules (i.e., the “ruleset”).
Alternatively, you can click Browse Rules to override specific rules, rather than all of them as a set of rules.
Mitigation Action
The mitigation action defines what action to take when the mitigation rule is applied. Our systems constantly analyze traffic and track potentially malicious activity. When certain request-per-second thresholds exceed the configured sensitivity level, a mitigation rule with a dynamically generated signature will be applied to mitigate the attack. The default mitigation action can vary according to the specific rule. A rule with less confidence may apply a Challenge action as a form of soft mitigation, and a rule with a Block action is applied when there is higher confidence that the traffic is part of an attack — as a form of a stricter mitigation action.
The available values for the action are:
Block
Challenge (CAPTCHA)
Log
Use Rule Defaults
Some examples of when you may want to change the mitigation action include:
Safer onboarding: You’re onboarding a new HTTP application which has odd traffic patterns, naively violates protocol violations or causes spiky behavior. In this case, you can set the action to Log and see what traffic our systems flag. Afterwards, you can make the necessary changes to the sensitivity levels as required and switch the mitigation action back to the default.
Stricter mitigation: A DDoS attack has been detected but a Rate-limit or Challenge action have been applied due to the rule’s default logic. However, in this specific case, you’re sure that this is malicious traffic, so you can change the action to Block for a more complete mitigation.
Mitigation Sensitivity
The sensitivity level defines when the mitigation rule is applied. Our systems constantly analyze traffic and track potentially malicious activity. When certain request-per-second thresholds are crossed, a mitigation rule with a dynamically generated signature will be applied to mitigate the attack. Toggling the sensitivity levels allows you to define when the mitigation is applied. The higher the sensitivity, the faster the mitigation is applied. The available values for sensitivity are:
High (default)
Medium
Low
Essentially Off
Essentially Off means that we’ve set an exceptionally low sensitivity level so in most cases traffic won’t be mitigated for you. However, attack traffic will be mitigated at exceptional levels to ensure the safety and stability of the Cloudflare network.
Some examples of when you may want to change the sensitivity action include:
Avoid impact to legitimate traffic: One of the rules has applied mitigation to your legitimate traffic due to a suspicious pattern. In this case, you may want to reduce the rule sensitivity to avoid recurrence of the issue and negative impact to your traffic.
Legacy applications: One of your legacy HTTP applications is violating protocol standards, or you may have mistakenly introduced a bug into your mobile application/API client. These cases may result in abnormal traffic activity that may be flagged by our systems. In such a case, you can select the Essentially Off sensitivity level until you’ve resolved the issue on your end, to avoid false positives.
Overriding Specific Rules
As mentioned above, you can also select a specific rule to override its action and sensitivity levels. The per-rule override takes priority over the ruleset override.
When configuring per-rule overrides, you’ll see that some rules have a DDoS Dynamic action. This means that the mitigation is multi-staged and will apply different mitigation actions depending on various factors including attack type, request characteristics, and various other factors. This dynamic action can also be overridden if you choose to do so.
DDoS Attack Analytics
When a DDoS attack is detected and mitigated, you’ll receive a real-time DDoS alert (if you’ve configured one) and you’ll be able to view the attack in the Firewall analytics dashboard. The attack details and the rule ID that was triggered will also be displayed in the Activity log as part of each HTTP request log that was mitigated.
Helping Build a Better Internet
At Cloudflare, everything we do is guided by our mission to help build a better Internet. A significant part of that mission is to make DDoS downtime and service disruptions a thing of the past. By giving our users the visibility and tools they need in order to understand and improve their DDoS protection, we’re hoping to make another step towards a better Internet.
Do you have feedback about the user interface or anything else? In the new DDoS tab, we’ve added a link to provide feedback, so you too can help shape the future of Cloudflare’s DDoS protection Managed Rules.
Earlier this summer, Cloudflare’s autonomous edge DDoS protection systems automatically detected and mitigated a 17.2 million request-per-second (rps) DDoS attack, an attack almost three times larger than any previous one that we’re aware of. For perspective on how large this attack was: Cloudflare serves over 25 million HTTP requests per second on average. This refers to the average rate of legitimate traffic in 2021 Q2. So peaking at 17.2 million rps, this attack reached 68% of our Q2 average rps rate of legitimate HTTP traffic.
Comparison graph of Cloudflare’s average request per second rate versus the DDoS attack
Automated DDoS mitigation with Cloudflare’s autonomous edge
This attack, along with the additional attacks provided in the next sections, were automatically detected and mitigated by our autonomous edge DDoS protection systems. The system is powered by our very own denial of service daemon (dosd). Dosd is a home-grown software-defined daemon. A unique dosd instance runs in every server in each one of our data centers around the world. Each dosd instance independently analyzes traffic samples out-of-path. Analyzing traffic out-of-path allows us to scan asynchronously for DDoS attacks without causing latency and impacting performance. DDoS findings are also shared between the various dosd instances within a data center, as a form of proactive threat intelligence sharing.
Once an attack is detected, our systems generate a mitigation rule with a real-time signature that matches the attack patterns. The rule is propagated to the most optimal location in the tech stack. As an example, a volumetric HTTP DDoS attack may be blocked at L4 inside the Linux iptables firewall instead of at L7 inside the L7 reverse proxy which runs in the user space. Mitigating lower in the stack, e.g. dropping the packets at L4 instead of responding with a 403 error page in L7, is more cost-efficient. It reduces our edge CPU consumption and intra-data center bandwidth utilization — thus helping us mitigate large attacks at scale without impacting performance.
This autonomous approach, along with our network’s global scale and reliability, allow us to mitigate attacks that reach 68% of our average per-second-rate, and higher, without requiring any manual mitigation by Cloudflare personnel, nor causing any performance degradation.
The resurgence of Mirai and new powerful botnets
This attack was launched by a powerful botnet, targeting a Cloudflare customer in the financial industry. Within seconds, the botnet bombarded the Cloudflare edge with over 330 million attack requests.
Graph of 17.2M rps attack
The attack traffic originated from more than 20,000 bots in 125 countries around the world. Based on the bots’ source IP addresses, almost 15% of the attack originated from Indonesia and another 17% from India and Brazil combined. Indicating that there may be many malware infected devices in those countries.
Distribution of the attack sources by top countries
Volumetric attacks increase
This 17.2 million rps attack is the largest HTTP DDoS attack that Cloudflare has ever seen to date and almost three times the size of any other reported HTTP DDoS attack. This specific botnet, however, has been seen at least twice over the past few weeks. Just last week it also targeted a different Cloudflare customer, a hosting provider, with an HTTP DDoS attack that peaked just below 8 million rps.
Graph of 8M rps attack
Two weeks before, a Mirai-variant botnet launched over a dozen UDP and TCP based DDoS attacks that peaked multiple times above 1 Tbps, with a max peak of approximately 1.2 Tbps. And while the first HTTP attacks targeted Cloudflare customers on the WAF/CDN service, the 1+ Tbps network-layer attacks targeted Cloudflare customers on the Magic Transit and Spectrum services. One of these targets was a major APAC-based Internet services, telecommunications and hosting provider. The other was a gaming company. In all cases, the attacks were automatically detected and mitigated without human intervention.
Graph of Mirai botnet attack peaking at 1.2 Tbps
The Mirai botnet started with roughly 30K bots and slowly shrinked to approximately 28K. However, despite losing bots from its fleet, the botnet was still able to generate impressive volumes of attack traffic for short periods. In some cases, each burst lasted only a few seconds.
These attacks join the increase in Mirari-based DDoS attacks that we’ve observed on our network over the past weeks. In July alone, L3/4 Mirai attacks increased by 88% and L7 attacks by 9%. Additionally, based on the current August per-day average of the Mirai attacks, we can expect L7 Mirai DDoS attacks and other similar botnet attacks to increase by 185% and L3/4 attacks by 71% by the end of the month.
Graph of change in Mirai based DDoS attacks by month
Back to the Mirai
Mirai, which means ‘future’ in Japanese, is a codename for malware that was first discovered in 2016 by MalwareMustDie, a non-profit security research workgroup. The malware spreads by infecting Linux-operated devices such as security cameras and routers. It then self-propagates by searching for open Telnet ports 23 and 2323. Once found, it then attempts to gain access to vulnerable devices by brute forcing known credentials such as factory default usernames and passwords. Later variants of Mirai also took advantage of zero-day exploits in routers and other devices. Once infected, the devices will monitor a Command & Control (C2) server for instructions on which target to attack.
Diagram of Botnet operator controlling the botnet to attack websites
How to protect your home and business
While the majority of attacks are small and short, we continue to see these types of volumetric attacks emerging more often. It’s important to note that these volumetric short burst attacks can be especially dangerous for legacy DDoS protection systems or organizations without active, always-on cloud-based protection.
Furthermore, while the short duration may say something about the botnet’s capability to deliver sustained levels of traffic over time, it can be challenging or impossible for humans to react to it in time. In such cases, the attack is over before a security engineer even has time to analyze the traffic or activate their stand-by DDoS protection system. These types of attacks highlight the need for automated, always-on protection.
How to protect your business and Internet properties
Follow our preventive best practices, to ensure that both your Cloudflare settings and your origin server settings are optimized. As an example, make sure that you allow only traffic from Cloudflare’s IP range. Ideally, ask your upstream Internet Service Provider (ISP) to apply an access control list (ACL), otherwise, attackers may target your servers’ IP addresses directly and bypass your protection.
Recommendations on how to protect your home and IoT appliances
Change the default username and password of any device that is connected to the Internet such as smart cameras and routers. This will reduce the risk that malware such as Mirai can gain access to your router and IoT devices.
Protect your home against malware with Cloudflare for Families. Cloudflare for Families is a free service that automatically blocks traffic from your home to malicious websites and malware communication.
In late 2020, a major Fortune Global 500 company was targeted by a Ransom DDoS (RDDoS) attack by a group claiming to be the Lazarus Group. Cloudflare quickly onboarded them to the Magic Transit service and protected them against the lingering threat. This extortion attempt was part of wider ransom campaigns that have been unfolding throughout the year, targeting thousands of organizations around the world. Extortionists are threatening organizations with crippling DDoS attacks if they do not pay a ransom.
Throughout 2020, Cloudflare onboarded and protected many organizations with Magic Transit, Cloudflare’s DDoS protection service for critical network infrastructure, the WAF service for HTTP applications, and the Spectrum service for TCP/UDP based applications — ensuring their business’s availability and continuity.
Unwinding the attack timeline
I spoke with Daniel (a pseudonym) and his team, who work at the Incident Response and Forensics team at the aforementioned company. I wanted to learn about their experience, and share it with our readers so they could learn how to better prepare for such an event. The company has requested to stay anonymous and so some details have been omitted to ensure that. In this blog post, I will refer to them as X.
Initially, the attacker sent ransom emails to a handful of X’s publicly listed email aliases such as press@, shareholder@, and hostmaster@. We’ve heard from other customers that in some cases, non-technical employees received the email and ignored it as being spam which delayed the incident response team’s time to react by hours. However, luckily for X, a network engineer that was on the email list of the hostmaster@ alias saw it and immediately forwarded it to Daniel’s incident response team.
In the ransom email, the attackers demanded 20 bitcoin and gave them a week to pay up, or else a second larger attack would strike, and the ransom would increase to 30 bitcoin. Daniel says that they had a contingency plan ready for this situation and that they did not intend to pay. Paying the ransom funds illegitimate activities, motivates the attackers, and does not guarantee that they won’t attack anyway.
…Please perform a google search of “Lazarus Group” to have a look at some of our previous work. Also, perform a search for “NZX” or “New Zealand Stock Exchange” in the news. You don’t want to be like them, do you?…
The current fee is 20 Bitcoin (BTC). It’s a small price to pay for what will happen if your whole network goes down. Is it worth it? You decide!…
If you decide not to pay, we will start the attack on the indicated date and uphold it until you do. We will completely destroy your reputation and make sure your services will remain offline until you pay…
–An excerpt of the ransom note
The contingency plan
Upon receiving the email from the network engineer, Daniel called him and they started combing through the network data — they noticed a significant increase in traffic towards one of their global data centers. This attacker was not playing around, firing gigabits per second towards a single server. The attack, despite just being a proof of intention, saturated the Internet uplink to that specific data center, causing a denial of service event and generating a series of failure events.
This first “teaser” attack came on a work day, towards the end of business hours as people were already wrapping up their day. At the time, X was not protected by Cloudflare and relied on an on-demand DDoS protection service. Daniel activated the contingency plan which relied on the on-demand scrubbing center service.
Daniel contacted their DDoS protection service. It took them over 30 minutes to activate the service and redirect X’s traffic to the scrubbing center. Activating the on-demand service caused networking failures and resulted in multiple incidents for X on various services — even ones that were not under attack. Daniel says hindsight is 2020 and he realized that an always-on service would have been much more effective than on-demand, reactionary control that takes time to implement, after the impact is felt. The networking failures amplified the one-hour attack resulting in incidents lasting much longer than expected.
Onboarding to Cloudflare
Following the initial attack, Daniel’s team reached out to Cloudflare and wanted to onboard to our automated always-on DDoS protection service, Magic Transit. The goal was to onboard to it before the second attack would strike. Cloudflare explained the pre-onboarding steps, provided details on the process, and helped onboard X’s network in a process Daniel defined as “quite painless and very professional. The speed and responsiveness were impressive. One of the key differentiation is the attack and traffic analytics that we see that our incumbent provider couldn’t provide us. We’re seeing attacks we never knew about being mitigated automatically.”
The attackers promised a second, huge attack which never happened. Perhaps it was just an empty threat, or it could be that the attackers detected that X is protected by Cloudflare which deterred them and they, therefore, decided to move on to their next target?
Recommendations for organizations
I asked Daniel if he has any recommendations for businesses so they can learn from his experience and be better prepared, should they be targeted by ransom attacks:
1. Utilize an automated always-on DDoS protection service
Do not rely on reactive on-demand SOC-based DDoS Protection services that require humans to analyze attack traffic. It just takes too long. Don’t be tempted to use an on-demand service: “you get all of the pain and none of the benefits”. Instead, onboard to a cloud service that has sufficient network capacity and automated DDoS mitigation systems.
2. Work with your vendor to build and understand your threat model
Work together with your DDoS protection vendor to tailor mitigation strategies to your workload. Every network is different, and each poses unique challenges when integrating with DDoS mitigation systems.
3. Create a contingency plan and educate your employees
Be prepared. Have plans ready and train your teams on them. Educate all of your employees, even the non-techies, on what to do if they receive a ransom email. They should report it immediately to your Security Incident Response team.
Cloudflare customers need not worry as they are protected. Enterprise customers can reach out to their account team if they are being extorted in order to review and optimize their security posture if needed. Customers on all other plans can reach out to our support teams and learn more about how to optimize your Cloudflare security configuration.
On the week of Black Friday, Cloudflare automatically detected and mitigated a unique ACK DDoS attack, which we’ve codenamed “Beat”, that targeted a Magic Transit customer. Usually, when attacks make headlines, it’s because of their size. However, in this case, it’s not the size that is unique but the method that appears to have been borrowed from the world of acoustics.
Acoustic inspired attack
As can be seen in the graph below, the attack’s packet rate follows a wave-shaped pattern for over 8 hours. It seems as though the attacker was inspired by an acoustics concept called beat. In acoustics, a beat is a term that is used to describe an interference of two different wave frequencies. It is the superposition of the two waves. When the two waves are nearly 180 degrees out of phase, they create the beating phenomenon. When the two waves merge they amplify the sound and when they are out of sync they cancel one another, creating the beating effect.
Beat DDoS Attack
Acedemo.org has a nice tool where you can create your own beat wave. As you can see in the screenshot below, the two waves in blue and red are out of phase and the purple wave is their superposition, the beat wave.
It looks like the attacker launched a flood of packets where the rate of the packets is determined by the equation of the beat wave: y‘beat=y1+y2. The two equations y1 and y2 represent the two waves.
Each equation is expressed as
where fi is the frequency of each wave and t is time.
Therefore, the packet rate of the attack is determined by manipulation of the equation
to achieve a packet rate that ranges from ~18M to ~42M pps.
To get to the scale of this attack we will need to multiply y‘beat by a certain variable a and also add a constant c, giving us ybeat=ay‘beat+c. Now, it’s been a while since I played around with equations, so I’m only going to try and get an approximation of the equation.
By observing the attack graph, we can guesstimate that
by playing around with desmos’s cool graph visualizer tool, if we set f1=0.0000345 and f2=0.00003455 we can generate a graph that resembles the attack graph. Plotting in those variables, we get:
Now this formula assumes just one node firing the packets. However, this specific attack was globally distributed, and if we assume that each node, or bot in this botnet, was firing an equal amount of packets at an equal rate, then we can divide the equation by the size of the botnet; the number of bots b. Then the final equation is something in the form of:
In the screenshot below, g = f 1. You can view this graph here.
Beating the drum
The attacker may have utilized this method in order to try and overcome our DDoS protection systems (perhaps thinking that the rhythmic rise and fall of the attack would fool our systems). However, flowtrackd, our unidirectional TCP state tracking machine, detected it as being a flood of ACK packets that do not belong to any existing TCP connection. Therefore, flowtrackd automatically dropped the attack packets at Cloudflare’s edge.
The attacker was beating the drum for over 19 hours with an amplitude of ~7 Mpps, a wavelength of ~4 hours, and peaking at ~42 Mpps. During the two days in which the attack took place, Cloudflare systems automatically detected and mitigated over 700 DDoS attacks that targeted this customer. The attack traffic accumulated at almost 500 Terabytes out of a total of 3.6 Petabytes of attack traffic that targeted this single customer in November alone. During those two days, the attackers utilized mainly ACK floods, UDP floods, SYN floods, Christmas floods (where all of the TCP flags are ‘lit’), ICMP floods, and RST floods.
The challenge of TCP based attacks
TCP is a stateful protocol, which means that in some cases, you’d need to keep track of a TCP connection’s state in order to know if a packet is legitimate or part of an attack, i.e. out of state. We were able to provide protection against out-of-state TCP packet attacks for our “classic” WAF/CDN service and Spectrum service because in both cases Cloudflare serves as a reverse-proxy seeing both ingress and egress traffic.
However, when we launched Magic Transit, which relies on an asymmetric routing topology with a direct server return (DSR), we couldn’t utilize our existing TCP connection tracking systems.
And so, being a software-defined company, we’re able to write code and spin up software when and where needed — as opposed to vendors that utilize dedicated DDoS protection hardware appliances. And that is what we did. We built flowtrackd, which runs autonomously on each server at our network’s edge. flowtrackd is able to classify the state of TCP flows by analyzing only the ingress traffic, and then drops, challenges, or rate-limits attack packets that do not correspond to an existing flow.
flowtrackd works together with our two additional DDoS protection systems, dosd and Gatebot, to assure our customers are protected against DDoS attacks, regardless of their size or sophistication — in this case, serving as a noise-canceling system to the Beat attack; reducing the headaches for our customers.
Read more about how our DDoS protection systems work here.
DDoS attacks are surging — both in frequency and sophistication. After doubling from Q1 to Q2, the total number of network layer attacks observed in Q3 doubled again — resulting in a 4x increase in number compared to the pre-COVID levels in the first quarter. Cloudflare also observed more attack vectors deployed than ever — in fact, while SYN, RST, and UDP floods continue to dominate the landscape, we saw an explosion in protocol specific attacks such as mDNS, Memcached, and Jenkins DoS attacks.
Here are other key network layer DDoS trends we observed in Q3:
Majority of the attacks are under 500 Mbps and 1 Mpps — both still suffice to cause service disruptions
We continue to see a majority of attacks be under 1 hr in duration
Ransom-driven DDoS attacks (RDDoS) are on the rise as groups claiming to be Fancy Bear, Cozy Bear and the Lazarus Group extort organizations around the world. As of this writing, the ransom campaign is still ongoing. See a special note on this below.
Number of attacks
The total number of L3/4 DDoS attacks we observe on our network continues to increase substantially, as indicated in the graph below. All in all, Q3 saw over 56% of all attacks this year — double that of Q2, and four times that of Q1. In addition, the number of attacks per month increased throughout the quarter.
While September witnessed the largest number of attacks overall, August saw the most large attacks (over 500Mbps). Ninety-one percent of large attacks in Q3 took place in that month—while monthly distribution of other attack sizes was far more even.
While the total number of attacks between 200-300 Gbps decreased in September, we saw more global attacks on our network in Q3. This suggests the increase in the use of distributed botnets to launch attacks. In fact, in early July, Cloudflare witnessed one of the largest-ever attacks on our network — generated by Moobot, a Mirai-based botnet. The attack peaked at 654 Mbps and originated from 18,705 unique IP addresses, each believed to be a Moobot-infected IoT device. The attack campaign lasted nearly 10 days, but the customer was protected by Cloudflare, so they observed no downtime or service degradation.
Attack size (bit rate and packet rate)
There are different ways of measuring a L3/4 DDoS attack’s size. One is the volume of traffic it delivers, measured as the bit rate (specifically, Gigabits-per-second). Another is the number of packets it delivers, measured as the packet rate (specifically, packets-per-second). Attacks with high bit rates attempt to saturate the Internet link, and attacks with high packet rates attempt to overwhelm the routers or other in-line hardware devices.
In Q3, most of the attacks we observed were smaller in size. In fact, over 87% of all attacks were under 1 Gbps. This represents a significant increase from Q2, when roughly 52% of attacks were that small. Note that, even ‘small’ attacks of under 500 Mbps are many times sufficient to create major disruptions for Internet properties that are not protected by a Cloud based DDoS protection service. Many organizations have uplinks provided by their ISPs that are far less than 1 Gbps. Assuming their public facing network interface also serves legitimate traffic, you can see how even these ‘small’ DDoS attacks can easily take down Internet properties.
This trend holds true for attack packet rates. In Q3, 47% of attacks were under 50k pps — compared to just 19% in Q2.
Smaller attacks can indicate that amateur attackers may be behind the attacks — using tools easily available to generate attacks on exposed IPs/ networks. Alternatively, small attacks may serve as a smokescreen to distract security teams from other kinds of cyberattacks that might be taking place simultaneously.
Attack duration
In terms of length, very short attacks were the most common attack type observed in Q3, accounting for nearly 88% of all attacks. This observation is in line with our prior reports — in general, Layer 3/4 DDoS attacks are getting shorter in duration.
Short burst attacks may attempt to cause damage without being detected by DDoS detection systems. DDoS services that rely on manual analysis and mitigation may prove to be useless against these types of attacks because they are over before the analyst even identifies the attack traffic.
Alternatively, the use of short attacks may be used to probe the cyber defenses of the target. Load-testing tools and automated DDoS tools, that are widely available on the dark web, can generate short bursts of, say, a SYN flood, and then following up with another short attack using an alternate attack vector. This allows attackers to understand the security posture of their targets before they decide to potentially launch larger attacks at larger rates and longer durations – which come at a cost.
In other cases, attackers generate small DDoS attacks as proof and warning to the target organization of the attacker’s ability to cause real damage later on. It’s often followed by a ransom note to the target organization, demanding payment so as to avoid suffering an attack that could more thoroughly cripple network infrastructure.
Whatever their motivation, DDoS attacks of any size or duration are not going away anytime soon. Even short DDoS attacks cause harm, and having an automated real-time defense mechanism in place is critical for any online business.
Attack vectors
SYN floods constituted nearly 65% of all attacks observed in Q3, followed by RST floods and UDP floods in second and third places. This is relatively consistent with observations from previous quarters, highlighting the DDoS attack vector of choice by attackers.
While TCP based attacks like SYN and RST floods continue to be popular, UDP-protocol specific attacks such as mDNS, Memcached, and Jenkins are seeing an explosion compared to the prior quarter.
Multicast DNS (mDNS) is a UDP-based protocol that is used in local networks for service/device discovery. Vulnerable mDNS servers respond to unicast queries originating outside of the local network, which are ‘spoofed’ (altered) with the victim’s source address. This results in amplification attacks. In Q3, we noticed an explosion of mDNS attacks — specifically, we saw a 2,680% increase compared to the previous quarter.
This was followed by Memcached and Jenkins attacks. Memcached is a Key Value database. Requests can be made over the UDP protocol with a spoofed source address of the target. The size of the Value stored in the requested Key will affect the amplification factor, resulting in a DDoS amplification attack. Similarly, Jenkins, NTP, Ubiquity and the other UDP based protocols have seen a dramatic increase over the quarter due to its UDP stateless nature. A vulnerability in the older version (Jenkins 2.218 and earlier) aided the launch of DDoS attacks. This vulnerability was fixed in Jenkins 2.219 by disabling UDP multicast/ broadcast messages by default. However there are still many vulnerable and exposed devices that run UDP based services which are being harnessed to generate volumetric amplification attacks.
Attack by country
Looking at country-based distribution, the United States observed the most number of L3/4 DDoS attacks, followed by Germany and Australia. Note that when analyzing L3/4 DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations where the traffic was ingested, and not by the location of the source IP. The reason is when attackers launch L3/4 attacks they can spoof the source IP address in order to obfuscate the attack source. If we were to derive the country based on a spoofed source IP, we would get a spoofed country. Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of Cloudflare’s data center in which the attack was observed. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities around the world.
Africa
Asia Pacific & Oceania
Europe
Middle East
North America
South America
United States
A note on recent ransom-driven DDoS attacks
Over the past months, Cloudflare has observed another disturbing trend — a rise in extortion and ransom-based DDoS (RDDoS) attacks targeting organizations around the world. While RDDoS threats do not always result in an actual attack, the cases seen in recent months show that attacker groups are willing to carry out the threat, launching large scale DDoS attacks that can overwhelm organizations that lack adequate protection. In some cases, the initial teaser attack may be sufficient to cause impact if not protected by a Cloud based DDoS protection service.
In a RDDoS attack, a malicious party threatens a person or organization with a cyberattack that could knock their networks, websites, or applications offline for a period of time, unless the person or organization pays a ransom. You can read more about RDDoS attacks here.
Entities claiming to be Fancy Bear, Cozy Bear, and Lazarus have been threatening to launch DDoS attacks against organizations’ websites and network infrastructure unless a ransom is paid before a given deadline. Additionally, an initial ‘teaser’ DDoS attack is usually launched as a form of demonstration before parallel to the ransom email. The demonstration attack is typically a UDP reflection attack using a variety of protocols, lasting roughly 30 minutes in duration (or less).
What to do if you receive a threat:
Do not panic and we recommend you to not pay the ransom: Paying the ransom only encourages bad actors, finances illegal activities —and there’s no guarantee that they won’t attack your network now or later.
Notify local law enforcement: They will also likely request a copy of the ransom letter that you received.
Contact Cloudflare: We can help ensure your website and network infrastructure are safeguarded from these ransom attacks.
Cloudflare DDoS protection is different
On-prem hardware/cloud-scrubbing centers can’t address the challenges of modern volumetric DDoS attacks. Appliances are easily overwhelmed by large DDoS attacks, Internet links quickly saturate, and rerouting traffic to cloud scrubbing centers introduces unacceptable latency penalties. Our cloud-native, always-on, automated DDoS protection approach solves problems that traditional cloud signaling approaches were originally created to address.
Cloudflare’s mission is to help build a better Internet, which grounds our DDoS approach and is why in 2017, we pioneered unmetered DDoS mitigation for all of our customers on all plans including the free plan. We are able to provide this level of protection because every server on our network can detect & block threats, enabling us to absorb attacks of any size/kind, with no latency impact. This architecture gives us unparalleled advantages compared to any other vendor.
51 Tbps of DDoS mitigation capacity and under 3 sec TTM: Every data center in Cloudflare’s network detects and mitigates DDoS attacks. Once an attack is identified, the Cloudflare’s local data center mitigation system (dosd) generates and applies a dynamically crafted rule with a real-time signature — and mitigates attacks in under 3 seconds globally on average. This 3-second Time To Mitigate (TTM) is one of the fastest in the industry. Firewall rules and “proactive”/static configurations take effect immediately.
Fast performance included: Cloudflare is architected so that customers do not incur a latency penalty as a result of attacks. We deliver DDoS protection from every Cloudflare data center (instead of legacy scrubbing centers or on-premise hardware boxes) which allows us to mitigate attacks closest to the source. Cloudflare analyzes traffic out-of-path ensuring that our DDoS mitigation solution doesn’t add any latency to legitimate traffic. The rule is applied at the most optimal place in the Linux stack for a cost efficient mitigation, ensuring no performance penalty.
Global Threat Intelligence: Like an immune system, our network learns from/mitigates attacks against any customer to protect them all. With threat intelligence (TI), it automatically blocks attacks and is employed in customer facing features (Bot Fight mode, Firewall Rules & Security Level). Users create custom rules to mitigate attacks based on traffic attribute filters, threat & bot scores generated using ML models (protecting against bots/botnets/DDoS).
Today we’re announcing the availability of DDoS attack alerts. The alerts are available for free for all Cloudflare’s customers on paid plans.
Unmetered DDoS protection
Last week we celebrated Cloudflare’s 10th birthday in what we call Birthday Week. Every year, on each day of Birthday Week, we announce a new product with the goal of helping make the Internet a better place — one that is safer and faster. To do that, over the years we’ve democratized many products that were previously only available to large enterprises by making them available for free (or at very low cost) to all. For example, on Cloudflare’s 7th birthday in 2017, we announced free unmetered DDoS protection as part of every Cloudflare product and every plan, including the free plan.
DDoS attacks aim to take down websites or online services and make them unavailable to the public. We wanted to make sure that every organization and every website is available and accessible, regardless if they can or can’t afford enterprise-grade DDoS protection. This has been a core part of our mission. We’ve been heavily investing in our DDoS protection capabilities over the last 10 years, and we will continue to do so in the future.
Real-time DDoS attack alerts
I’ve recently published a few blogs that provide a look under the hood of our DDoS protection systems. These systems run autonomously, they detect and mitigate attacks without any human intervention. As was the case with the 654 Gbps attack in July, and the 754 Mpps attack in June. We’ve been successful at blocking DDoS attacks and also providing our users with important analytics and insights about the attacks, but our customers also want to be notified in real-time when they are targeted by DDoS attacks.
So today, we’re excited to announce the availability of DDoS alerts. The current delivery methods by Cloudflare plan type are listed in the table below. Additional delivery methods will be made available in the future.
Delivery methods by plan
Delivery method
Plan
Free
Pro
Business
Enterprise
Email
❌
✅
✅
✅
PagerDuty
❌
❌
✅
✅
There are two types of DDoS alerts: HTTP DDoS alerts and L3/4 DDoS alerts. Whether you are eligible to one or both depends on the Cloudflare services that you are subscribed to. The table below lists the alert types by the Cloudflare service.
Alert types by service
Alert type
Service
WAF/CDN
Spectrum
Spectrum BYOIP
Magic Transit
HTTP DDoS alerts
✅
❌
❌
❌
L3/4 DDoS alerts
Coming soon
Coming soon
✅
✅
Creating a DDoS alert policy
In order to receive alerts on DDoS attacks that target your Cloudflare-protected Internet property, you must first create a notification policy. That’s fast and easy:
In the Account Home page, navigate to the Notifications tab
In the Notifications card, click Create
Give your notification a name, add an optional description, and the email addresses of the recipients.
If you are on the Business plan or higher, you’ll need to connect to PagerDuty before creating the alert policy. Once you’ve done so, you’ll have the option to send the alert to your PagerDuty service.
Receive the alert, view the attack, and give feedback
When developing and designing the alert template, we interviewed many of our customers to understand what information is important to them, what would make the alert useful and easy to understand. We’ve intentionally made the alert short. The email subject is also straightforward: DDoS Attack Detected, and it will only be sent from our official email address: [email protected][dot]com. Add this email to your list of trusted email addresses to assure you don’t miss the alerts.
The alert includes the following information:
A short description of what happened
The date and time the attack was initially detected and mitigated by our systems
The attack type
The max rate of the attack when the alert was triggered
The attack target
The attack may be ongoing when you receive the alert and so we also include a link to view the attack in the Cloudflare dashboard and also a link to provide feedback on the protection and visibility.
We’d love to get your feedback!
We’d love your feedback on our DDoS protection solution. When you receive a DDoS alert, you’ll be provided with a link to submit your feedback. Measuring user satisfaction helps us build better products. Your feedback helps us measure user satisfaction for Cloudflare’s DDoS protection and the attack analytics that we provide in the dashboard. User satisfaction rates are one of the main Key Performance Indicators (KPIs) for our DDoS protection service that we monitor closely. So give your feedback, and help us make DDoS protection better for everyone.
Not a Cloudflare customer yet? Sign up to get started.
On July 3, Cloudflare’s global DDoS protection system, Gatebot, automatically detected and mitigated a UDP-based DDoS attack that peaked at 654 Gbps. The attack was part of a ten-day multi-vector DDoS campaign targeting a Magic Transit customer and was mitigated without any human intervention. The DDoS campaign is believed to have been generated by Moobot, a Mirai-based botnet. No downtime, service degradation, or false positives were reported by the customer.
Moobot Targets 654 Gbps towards a Magic Transit Customer
Over those ten days, our systems automatically detected and mitigated over 5,000 DDoS attacks against this one customer, mainly UDP floods, SYN floods, ACK floods, and GRE floods. The largest DDoS attack was a UDP flood and lasted a mere 2 minutes. This attack targeted only one IP address but hit multiple ports. The attack originated from 18,705 unique IP addresses, each believed to be a Moobot-infected IoT device.
Attack Distribution by Country – From 100 countries
The attack was observed in Cloudflare’s data centers in 100 countries around the world. Approximately 89% of the attack traffic originated from just 10 countries with the US leading at 41%, followed by South Korea and Japan in second place (12% each), and India in third (10%). What this likely means is that the malware has infected at least 18,705 devices in 100 countries around the world.
Attack Distribution by Country – Top 10
Moobot – Self Propagating Malware
‘Moobot’ sounds like a cute name, but there’s nothing cute about it. According to Netlab 360, Moobot is the codename of a self-propagating Mirai-based malware first discovered in 2019. It infects IoT (Internet of Things) devices using remotely exploitable vulnerabilities or weak default passwords. IoT is a term used to describe smart devices such as security hubs and cameras, smart TVs, smart speakers, smart lights, sensors, and even refrigerators that are connected to the Internet.
Once a device is infected by Moobot, control of the device is transferred to the operator of the command and control (C2) server, who can issue commands remotely such as attacking a target and locating additional vulnerable IoT devices to infect (self-propagation).
Self-propagation – The self-propagation module is in charge of the botnet’s growth. After an IoT device is infected, it randomly scans the Internet for open telnet ports and reports back to the C2 server. Once the C2 server gains knowledge of open telnet ports around the world, it tries to leverage known vulnerabilities or brute force its way into the IoT devices with common or default credentials.
Self-propagation
Synchronized attacks – The C2 server orchestrates a coordinated flood of packets or HTTP requests with the goal of creating a denial of service event for the target’s website or service.
Synchronized attacks
The botnet operator may use multiple C2 servers in various locations around the world in order to reduce the risk of exposure. Infected devices may be assigned to different C2 servers varying by region and module; one server for self-propagation and another for launching attacks. Thus if a C2 server is compromised and taken down by law enforcement authorities, only parts of the botnet are deactivated.
Why this attack was not successful
This is the second large scale attack in the past few months that we observed on Cloudflare’s network. The previous one peaked at 754M packets per second and attempted to take down our routers with a high packet rate. Despite the high packet rate, the 754Mpps attack peaked at a mere 253 Gbps.
As opposed to the high packet rate attack, this attack was a high bit rate attack, peaking at 654 Gbps. Due to the high bit rates of this attack, it seems as though the attacker tried (and failed) to cause a denial of service event by saturating our Internet link capacity. So let’s explore why this attack was not successful.
Avoiding link saturation & keeping appliances running
Cloudflare’s global network capacity is over 42 Tbps and growing. Our network spans more than 200 cities in over 100 countries, including 17 cities in mainland China. It interconnects with over 8,800 networks globally, including major ISPs, cloud services, and enterprises. This level of interconnectivity along with the use of Anycast ensures that our network can easily absorb even the largest attacks.
The Cloudflare Network
After traffic arrives at an edge data center, it is then load-balanced efficiently using our own Layer 4 load-balancer that we built, Unimog, which uses our appliances’ health and other metrics to load-balance traffic intelligently within a data center to avoid overwhelming any single server.
Besides the use of Anycast for inter-data center load balancing and Unimog for intra-data center load balancing, we also utilize various forms of traffic engineering in order to deal with sudden changes in traffic loads across our network. We utilize both automatic and manual traffic engineering methods that can be employed by our 24/7/365 Site Reliability Engineering (SRE) team.
These combined factors significantly reduce the likelihood of a denial of service event due to link saturation or appliances being overwhelmed — and as seen in this attack, no link saturation occurred.
Detecting & Mitigating DDoS attacks
Once traffic arrives at our edge, it encounters our three software-defined DDoS protection systems:
Gatebot – Cloudflare’s centralized DDoS protection systems for detecting and mitigating globally distributed volumetric DDoS attacks. Gatebot runs in our network’s core data center. It receives samples from every one of our edge data centers, analyzes them, and automatically sends mitigation instructions when attacks are detected. Gatebot is also synchronized to each of our customers’ web servers to identify its health and triggers mitigation accordingly.
dosd (denial of service daemon) – Cloudflare’s decentralized DDoS protection systems. dosd runs autonomously in each server in every Cloudflare data center around the world, analyzing traffic and applying local mitigation rules when needed. Besides being able to detect and mitigate attacks at super-fast speeds, dosd significantly improves our network resilience by delegating the detection and mitigation capabilities to the edge.
flowtrackd (flow tracking daemon) – Cloudflare’s TCP state tracking machine for detecting and mitigating the most randomized and sophisticated TCP-based DDoS attacks in unidirectional routing topologies (such as the case for Magic Transit). flowtrackd is able to identify the state of a TCP connection and then drops, challenges, or rate-limits packets that don’t belong to a legitimate connection.
Cloudflare DDoS Protection Lifecycle
The three DDoS protection systems collect traffic samples in order to detect DDoS attacks. The types of traffic data that they sample include:
Packet fields such as the source IP, source port, destination IP, destination port, protocol, TCP flags, sequence number, options, and packet rate.
HTTP request metadata such as HTTP headers, user agent, query-string, path, host, HTTP method, HTTP version, TLS cipher version, and request rate.
HTTP response metrics such as error codes returned by customers’ origin servers and their rates.
Our systems then crunch these sample data points together to form a real-time view of our network’s security posture and our customer’s origin server health. They look for attack patterns and traffic anomalies. When found, a mitigation rule with a dynamically crafted attack signature is generated in real-time. Rules are propagated to the most optimal place for cost-effective mitigation. For example, an L7 HTTP flood might be dropped at L4 to reduce the CPU consumption.
Rules that are generated by dosd and flowtrackd are propagated within a single data center for rapid mitigation. Gatebot’s rules are propagated to all of the edge data centers which then take priority over dosd’s rules for an even and optimal mitigation. Even if the attack is detected in a subset of edge data centers, Gatebot propagates the mitigation instructions to all of Cloudflare’s edge data centers — effectively sharing the threat intelligence across our network as a form of proactive protection.
In the case of this attack, in each edge data center, dosd generated rules to mitigate the attack promptly. Then as Gatebot received and analyzed samples from the edge, it determined that this was a globally distributed attack. Gatebot propagated unified mitigation instructions to the edge, which prepared each and every one of our 200+ data centers to tackle the attack as the attack traffic may shift to a different data center due to Anycast or traffic engineering.
No inflated bills
DDoS attacks obviously pose the risk of an outage and service disruption. But there is another risk to consider — the cost of mitigation. During these ten days, more than 65 Terabytes of attack traffic were generated by the botnet. However, as part of Cloudflare’s unmetered DDoS protection guarantee, Cloudflare mitigated and absorbed the attack traffic without billing the customer. The customer doesn’t need to submit a retroactive credit request. Attack traffic is automatically excluded from our billing system. We eliminated the financial risk.
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
