All posts by Omer Yoachimik

Beat – An Acoustics Inspired DDoS Attack

Post Syndicated from Omer Yoachimik original https://blog.cloudflare.com/beat-an-acoustics-inspired-ddos-attack/

Beat - An Acoustics Inspired DDoS Attack

Beat - An Acoustics Inspired DDoS Attack

On the week of Black Friday, Cloudflare automatically detected and mitigated a unique ACK DDoS attack, which we’ve codenamed “Beat”, that targeted a Magic Transit customer. Usually, when attacks make headlines, it’s because of their size. However, in this case, it’s not the size that is unique but the method that appears to have been borrowed from the world of acoustics.

Acoustic inspired attack

As can be seen in the graph below, the attack’s packet rate follows a wave-shaped pattern for over 8 hours. It seems as though the attacker was inspired by an acoustics concept called beat. In acoustics, a beat is a term that is used to describe an interference of two different wave frequencies. It is the superposition of the two waves. When the two waves are nearly 180 degrees out of phase, they create the beating phenomenon. When the two waves merge they amplify the sound and when they are out of sync they cancel one another, creating the beating effect.

Beat - An Acoustics Inspired DDoS Attack
Beat DDoS Attack

Acedemo.org has a nice tool where you can create your own beat wave. As you can see in the screenshot below, the two waves in blue and red are out of phase and the purple wave is their superposition, the beat wave.

Beat - An Acoustics Inspired DDoS Attack
Source: https://academo.org/demos/wave-interference-beat-frequency/ 

Reverse engineering the attack

It looks like the attacker launched a flood of packets where the rate of the packets is determined by the equation of the beat wave: ybeat=y1+y2. The two equations y1 and y2 represent the two waves.

Each equation is expressed as

Beat - An Acoustics Inspired DDoS Attack

where fi is the frequency of each wave and t is time.

Therefore, the packet rate of the attack is determined by manipulation of the equation

Beat - An Acoustics Inspired DDoS Attack

to achieve a packet rate that ranges from ~18M to ~42M pps.

To get to the scale of this attack we will need to multiply ybeat by a certain variable a and also add a constant c, giving us ybeat=aybeat+c. Now, it’s been a while since I played around with equations, so I’m only going to try and get an approximation of the equation.

By observing the attack graph, we can guesstimate that

Beat - An Acoustics Inspired DDoS Attack

by playing around with desmos’s cool graph visualizer tool, if we set f1=0.0000345 and f2=0.00003455 we can generate a graph that resembles the attack graph. Plotting in those variables, we get:

Beat - An Acoustics Inspired DDoS Attack

Now this formula assumes just one node firing the packets. However, this specific attack was globally distributed, and if we assume that each node, or bot in this botnet, was firing an equal amount of packets at an equal rate, then we can divide the equation by the size of the botnet; the number of bots b. Then the final equation is something in the form of:

Beat - An Acoustics Inspired DDoS Attack

In the screenshot below, g = f 1. You can view this graph here.

Beat - An Acoustics Inspired DDoS Attack

Beating the drum

The attacker may have utilized this method in order to try and overcome our DDoS protection systems (perhaps thinking that the rhythmic rise and fall of the attack would fool our systems). However, flowtrackd, our unidirectional TCP state tracking machine, detected it as being a flood of ACK packets that do not belong to any existing TCP connection. Therefore, flowtrackd automatically dropped the attack packets at Cloudflare’s edge.

The attacker was beating the drum for over 19 hours with an amplitude of ~7 Mpps, a wavelength of ~4 hours, and peaking at ~42 Mpps. During the two days in which the attack took place, Cloudflare systems automatically detected and mitigated over 700 DDoS attacks that targeted this customer. The attack traffic accumulated at almost 500 Terabytes out of a total of 3.6 Petabytes of attack traffic that targeted this single customer in November alone. During those two days, the attackers utilized mainly ACK floods, UDP floods, SYN floods, Christmas floods (where all of the TCP flags are ‘lit’), ICMP floods, and RST floods.

The challenge of TCP based attacks

TCP is a stateful protocol, which means that in some cases, you’d need to keep track of a TCP connection’s state in order to know if a packet is legitimate or part of an attack, i.e. out of state. We were able to provide protection against out-of-state TCP packet attacks for our “classic” WAF/CDN service and Spectrum service because in both cases Cloudflare serves as a reverse-proxy seeing both ingress and egress traffic.

Beat - An Acoustics Inspired DDoS Attack

However, when we launched Magic Transit, which relies on an asymmetric routing topology with a direct server return (DSR), we couldn’t utilize our existing TCP connection tracking systems.

Beat - An Acoustics Inspired DDoS Attack

And so, being a software-defined company, we’re able to write code and spin up software when and where needed — as opposed to vendors that utilize dedicated DDoS protection hardware appliances. And that is what we did. We built flowtrackd, which runs autonomously on each server at our network’s edge. flowtrackd is able to classify the state of TCP flows by analyzing only the ingress traffic, and then drops, challenges, or rate-limits attack packets that do not correspond to an existing flow.

Beat - An Acoustics Inspired DDoS Attack

flowtrackd works together with our two additional DDoS protection systems, dosd and Gatebot, to assure our customers are protected against DDoS attacks, regardless of their size or sophistication — in this case, serving as a noise-canceling system to the Beat attack; reducing the headaches for our customers.

Read more about how our DDoS protection systems work here.

Network-layer DDoS attack trends for Q3 2020

Post Syndicated from Omer Yoachimik original https://blog.cloudflare.com/network-layer-ddos-attack-trends-for-q3-2020/

Network-layer DDoS attack trends for Q3 2020

Network-layer DDoS attack trends for Q3 2020

DDoS attacks are surging — both in frequency and sophistication. After doubling from Q1 to Q2, the total number of network layer attacks observed in Q3 doubled again — resulting in a 4x increase in number compared to the pre-COVID levels in the first quarter. Cloudflare also observed more attack vectors deployed than ever — in fact, while SYN, RST, and UDP floods continue to dominate the landscape, we saw an explosion in protocol specific attacks such as mDNS, Memcached, and Jenkins DoS attacks.

Here are other key network layer DDoS trends we observed in Q3:

  • Majority of the attacks are under 500 Mbps and 1 Mpps — both still suffice to cause service disruptions
  • We continue to see a majority of attacks be under 1 hr in duration
  • Ransom-driven DDoS attacks (RDDoS) are on the rise as groups claiming to be Fancy Bear, Cozy Bear and the Lazarus Group extort organizations around the world. As of this writing, the ransom campaign is still ongoing. See a special note on this below.

Number of attacks

The total number of L3/4 DDoS attacks we observe on our network continues to increase substantially, as indicated in the graph below. All in all, Q3 saw over 56% of all attacks this year — double that of Q2, and four times that of Q1. In addition, the number of attacks per month increased throughout the quarter.

Network-layer DDoS attack trends for Q3 2020

While September witnessed the largest number of attacks overall, August saw the most large attacks (over 500Mbps). Ninety-one percent of large attacks in Q3 took place in that month—while monthly distribution of other attack sizes was far more even.

Network-layer DDoS attack trends for Q3 2020

While the total number of attacks between 200-300 Gbps decreased in September, we saw more global attacks on our network in Q3. This suggests the increase in the use of distributed botnets to launch attacks. In fact, in early July, Cloudflare witnessed one of the largest-ever attacks on our network — generated by Moobot, a Mirai-based botnet. The attack peaked at 654 Mbps and originated from 18,705 unique IP addresses, each believed to be a Moobot-infected IoT device. The attack campaign lasted nearly 10 days, but the customer was protected by Cloudflare, so they observed no downtime or service degradation.

Attack size (bit rate and packet rate)

There are different ways of measuring a L3/4 DDoS attack’s size. One is the volume of traffic it delivers, measured as the bit rate (specifically, Gigabits-per-second). Another is the number of packets it delivers, measured as the packet rate (specifically, packets-per-second). Attacks with high bit rates attempt to saturate the Internet link, and attacks with high packet rates attempt to overwhelm the routers or other in-line hardware devices.

In Q3, most of the attacks we observed were smaller in size. In fact, over 87% of all attacks were under 1 Gbps. This represents a significant increase from Q2, when roughly 52% of attacks were that small.  Note that, even ‘small’ attacks of under 500 Mbps are many times sufficient to create major disruptions for Internet properties that are not protected by a Cloud based DDoS protection service. Many organizations have uplinks provided by their ISPs that are far less than 1 Gbps. Assuming their public facing network interface also serves legitimate traffic, you can see how even these ‘small’ DDoS attacks can easily take down Internet properties.

Network-layer DDoS attack trends for Q3 2020

This trend holds true for attack packet rates. In Q3, 47% of attacks were under 50k pps — compared to just 19% in Q2.

Network-layer DDoS attack trends for Q3 2020

Smaller attacks can indicate that amateur attackers may be behind the attacks — using tools easily available to generate attacks on exposed IPs/ networks. Alternatively, small attacks may serve as a smokescreen to distract security teams from other kinds of cyberattacks that might be taking place simultaneously.

Attack duration

Network-layer DDoS attack trends for Q3 2020

In terms of length, very short attacks were the most common attack type observed in Q3, accounting for nearly 88% of all attacks. This observation is in line with our prior reports — in general, Layer 3/4 DDoS attacks are getting shorter in duration.

Short burst attacks may attempt to cause damage without being detected by DDoS detection systems. DDoS services that rely on manual analysis and mitigation may prove to be useless against these types of attacks because they are over before the analyst even identifies the attack traffic.

Alternatively, the use of short attacks may be used to probe the cyber defenses of the target. Load-testing tools and automated DDoS tools, that are widely available on the dark web, can generate short bursts of, say, a SYN flood, and then following up with another short attack using an alternate attack vector. This allows attackers to understand the security posture of their targets before they decide to potentially launch larger attacks at larger rates and longer durations – which come at a cost.

In other cases, attackers generate small DDoS attacks as proof and warning to the target organization of the attacker’s ability to cause real damage later on. It’s often followed by a ransom note to the target organization, demanding payment so as to avoid suffering an attack that could more thoroughly cripple network infrastructure.

Whatever their motivation, DDoS attacks of any size or duration are not going away anytime soon. Even short DDoS attacks cause harm, and having an automated real-time defense mechanism in place is critical for any online business.

Attack vectors

SYN floods constituted nearly 65% of all attacks observed in Q3, followed by RST floods and UDP floods in second and third places. This is relatively consistent with observations from previous quarters, highlighting the DDoS attack vector of choice by attackers.

While TCP based attacks like SYN and RST floods continue to be popular, UDP-protocol specific attacks such as mDNS, Memcached, and Jenkins are seeing an explosion compared to the prior quarter.

Network-layer DDoS attack trends for Q3 2020
Network-layer DDoS attack trends for Q3 2020

Multicast DNS (mDNS) is a UDP-based protocol that is used in local networks for service/device discovery. Vulnerable mDNS servers respond to unicast queries originating outside of the local network, which are ‘spoofed’ (altered) with the victim’s source address. This results in amplification attacks. In Q3, we noticed an explosion of mDNS attacks — specifically, we saw a 2,680% increase compared to the previous quarter.

This was followed by Memcached and Jenkins attacks. Memcached is a Key Value database. Requests can be made over the UDP protocol with a spoofed source address of the target. The size of the Value stored in the requested Key will affect the amplification factor, resulting in a DDoS amplification attack. Similarly, Jenkins, NTP, Ubiquity and the other UDP based protocols have seen a dramatic increase over the quarter due to its UDP stateless nature. A vulnerability in the older version (Jenkins 2.218 and earlier) aided the launch of DDoS attacks. This vulnerability was fixed in Jenkins 2.219 by disabling UDP multicast/ broadcast messages by default. However there are still many vulnerable and exposed devices that run UDP based services which are being harnessed to generate volumetric amplification attacks.

Attack by country

Network-layer DDoS attack trends for Q3 2020
Network-layer DDoS attack trends for Q3 2020

Looking at country-based distribution, the United States observed the most number of L3/4 DDoS attacks, followed by Germany and Australia. Note that when analyzing L3/4 DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations where the traffic was ingested, and not by the location of the source IP. The reason is when attackers launch L3/4 attacks they can spoof the source IP address in order to obfuscate the attack source. If we were to derive the country based on a spoofed source IP, we would get a spoofed country. Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of Cloudflare’s data center in which the attack was observed. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities around the world.

Africa

Network-layer DDoS attack trends for Q3 2020

Asia Pacific & Oceania

Network-layer DDoS attack trends for Q3 2020

Europe

Network-layer DDoS attack trends for Q3 2020

Middle East

Network-layer DDoS attack trends for Q3 2020

North America

Network-layer DDoS attack trends for Q3 2020

South America

Network-layer DDoS attack trends for Q3 2020

United States

Network-layer DDoS attack trends for Q3 2020

A note on recent ransom-driven DDoS attacks

Over the past months, Cloudflare has observed another disturbing trend — a rise in extortion and ransom-based DDoS (RDDoS) attacks targeting organizations around the world. While RDDoS threats do not always result in an actual attack, the cases seen in recent months show that attacker groups are willing to carry out the threat, launching large scale DDoS attacks that can overwhelm organizations that lack adequate protection. In some cases, the initial teaser attack may be sufficient to cause impact if not protected by a Cloud based DDoS protection service.

In a RDDoS attack, a malicious party threatens a person or organization with a cyberattack that could knock their networks, websites, or applications offline for a period of time, unless the person or organization pays a ransom. You can read more about RDDoS attacks here.

Entities claiming to be Fancy Bear, Cozy Bear, and Lazarus have been threatening to launch DDoS attacks against organizations’ websites and network infrastructure unless a ransom is paid before a given deadline. Additionally, an initial ‘teaser’ DDoS attack is usually launched as a form of demonstration before parallel to the ransom email. The demonstration attack is typically a UDP reflection attack using a variety of protocols, lasting roughly 30 minutes in duration (or less).

What to do if you receive a threat:

  1. Do not panic and we recommend you to not pay the ransom: Paying the ransom only encourages bad actors, finances illegal activities —and there’s no guarantee that they won’t attack your network now or later.
  2. Notify local law enforcement: They will also likely request a copy of the ransom letter that you received.
  3. Contact Cloudflare: We can help ensure your website and network infrastructure are safeguarded from these ransom attacks.

Cloudflare DDoS protection is different

On-prem hardware/cloud-scrubbing centers can’t address the challenges of modern volumetric DDoS attacks. Appliances are easily overwhelmed by large DDoS attacks, Internet links quickly saturate, and rerouting traffic to cloud scrubbing centers introduces unacceptable latency penalties. Our cloud-native, always-on, automated DDoS protection approach solves problems that traditional cloud signaling approaches were originally created to address.

Cloudflare’s mission is to help build a better Internet, which grounds our DDoS approach and is why in 2017, we pioneered unmetered DDoS mitigation for all of our customers on all plans including the free plan. We are able to provide this level of protection because every server on our network can detect & block threats, enabling us to absorb attacks of any size/kind, with no latency impact. This architecture gives us unparalleled advantages compared to any other vendor.

  • 51 Tbps of DDoS mitigation capacity and under 3 sec TTM: Every data center in Cloudflare’s network detects and mitigates DDoS attacks. Once an attack is identified, the Cloudflare’s local data center mitigation system (dosd) generates and applies a dynamically crafted rule with a real-time signature — and mitigates attacks in under 3 seconds globally on average. This 3-second Time To Mitigate (TTM) is one of the fastest in the industry. Firewall rules and “proactive”/static configurations take effect immediately.
  • Fast performance included:  Cloudflare is architected so that customers do not incur a latency penalty as a result of attacks. We deliver DDoS protection from every Cloudflare data center (instead of legacy scrubbing centers or on-premise hardware boxes) which allows us to mitigate attacks closest to the source. Cloudflare analyzes traffic out-of-path ensuring that our DDoS mitigation solution doesn’t add any latency to legitimate traffic. The rule is applied at the most optimal place in the Linux stack for a cost efficient mitigation, ensuring no performance penalty.
  • Global Threat Intelligence: Like an immune system, our network learns from/mitigates attacks against any customer to protect them all. With threat intelligence (TI), it automatically blocks attacks and is employed in customer facing features (Bot Fight mode, Firewall Rules & Security Level). Users create custom rules to mitigate attacks based on traffic attribute filters, threat & bot scores generated using ML models (protecting against bots/botnets/DDoS).

To learn more about Cloudflare’s DDoS solution contact us or get started.

Know When You’ve Been DDoS’d

Post Syndicated from Omer Yoachimik original https://blog.cloudflare.com/announcing-ddos-alerts/

Know When You’ve Been DDoS’d

Know When You’ve Been DDoS’d

Today we’re announcing the availability of DDoS attack alerts. The alerts are available for free for all Cloudflare’s customers on paid plans.

Unmetered DDoS protection

Last week we celebrated Cloudflare’s 10th birthday in what we call Birthday Week. Every year, on each day of Birthday Week, we announce a new product with the goal of helping make the Internet a better place — one that is safer and faster. To do that, over the years we’ve democratized many products that were previously only available to large enterprises by making them available for free (or at very low cost) to all. For example, on Cloudflare’s 7th birthday in 2017, we announced free unmetered DDoS protection as part of every Cloudflare product and every plan, including the free plan.

DDoS attacks aim to take down websites or online services and make them unavailable to the public. We wanted to make sure that every organization and every website is available and accessible, regardless if they can or can’t afford enterprise-grade DDoS protection. This has been a core part of our mission. We’ve been heavily investing in our DDoS protection capabilities over the last 10 years, and we will continue to do so in the future.

Real-time DDoS attack alerts

I’ve recently published a few blogs that provide a look under the hood of our DDoS protection systems. These systems run autonomously, they detect and mitigate attacks without any human intervention. As was the case with the 654 Gbps attack in July, and the 754 Mpps attack in June. We’ve been successful at blocking DDoS attacks and also providing our users with important analytics and insights about the attacks, but our customers also want to be notified in real-time when they are targeted by DDoS attacks.

So today, we’re excited to announce the availability of DDoS alerts. The current delivery methods by Cloudflare plan type are listed in the table below. Additional delivery methods will be made available in the future.

Delivery methods by plan

Delivery method Plan
Free Pro Business Enterprise
Email
PagerDuty

There are two types of DDoS alerts: HTTP DDoS alerts and L3/4 DDoS alerts. Whether you are eligible to one or both depends on the Cloudflare services that you are subscribed to. The table below lists the alert types by the Cloudflare service.

Alert types by service

Alert type Service
WAF/CDN Spectrum Spectrum BYOIP Magic Transit
HTTP DDoS alerts
L3/4 DDoS alerts Coming soon Coming soon

Creating a DDoS alert policy

In order to receive alerts on DDoS attacks that target your Cloudflare-protected Internet property, you must first create a notification policy. That’s fast and easy:

  1. Log in to your Cloudflare account dashboard: https://dash.cloudflare.com
  2. In the Account Home page, navigate to the Notifications tab
  3. In the Notifications card, click Create
  4. Give your notification a name, add an optional description, and the email addresses of the recipients.
Know When You’ve Been DDoS’d

If you are on the Business plan or higher, you’ll need to connect to PagerDuty before creating the alert policy. Once you’ve done so, you’ll have the option to send the alert to your PagerDuty service.

Receive the alert, view the attack, and give feedback

When developing and designing the alert template, we interviewed many of our customers to understand what information is important to them, what would make the alert useful and easy to understand. We’ve intentionally made the alert short. The email subject is also straightforward: DDoS Attack Detected, and it will only be sent from our official email address: [email protected][dot]com. Add this email to your list of trusted email addresses to assure you don’t miss the alerts.

The alert includes the following information:

  1. A short description of what happened
  2. The date and time the attack was initially detected and mitigated by our systems
  3. The attack type
  4. The max rate of the attack when the alert was triggered
  5. The attack target

The attack may be ongoing when you receive the alert and so we also include a link to view the attack in the Cloudflare dashboard and also a link to provide feedback on the protection and visibility.

Know When You’ve Been DDoS’d

We’d love to get your feedback!

We’d love your feedback on our DDoS protection solution. When you receive a DDoS alert, you’ll be provided with a link to submit your feedback. Measuring user satisfaction helps us build better products. Your feedback helps us measure user satisfaction for Cloudflare’s DDoS protection and the attack analytics that we provide in the dashboard. User satisfaction rates are one of the main Key Performance Indicators (KPIs) for our DDoS protection service that we monitor closely. So give your feedback, and help us make DDoS protection better for everyone.

Not a Cloudflare customer yet? Sign up to get started.

Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps

Post Syndicated from Omer Yoachimik original https://blog.cloudflare.com/moobot-vs-gatebot-cloudflare-automatically-blocks-botnet-ddos-attack-topping-at-654-gbps/

Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps

On July 3, Cloudflare’s global DDoS protection system, Gatebot, automatically detected and mitigated a UDP-based DDoS attack that peaked at 654 Gbps. The attack was part of a ten-day multi-vector DDoS campaign targeting a Magic Transit customer and was mitigated without any human intervention. The DDoS campaign is believed to have been generated by Moobot, a Mirai-based botnet. No downtime, service degradation, or false positives were reported by the customer.

Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps
Moobot Targets 654 Gbps towards a Magic Transit Customer

Over those ten days, our systems automatically detected and mitigated over 5,000 DDoS attacks against this one customer, mainly UDP floods, SYN floods, ACK floods, and GRE floods. The largest DDoS attack was a UDP flood and lasted a mere 2 minutes. This attack targeted only one IP address but hit multiple ports. The attack originated from 18,705 unique IP addresses, each believed to be a Moobot-infected IoT device.

Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps
Attack Distribution by Country – From 100 countries

The attack was observed in Cloudflare’s data centers in 100 countries around the world. Approximately 89% of the attack traffic originated from just 10 countries with the US leading at 41%, followed by South Korea and Japan in second place (12% each), and India in third (10%). What this likely means is that the malware has infected at least 18,705 devices in 100 countries around the world.

Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps
Attack Distribution by Country – Top 10

Moobot – Self Propagating Malware

‘Moobot’ sounds like a cute name, but there’s nothing cute about it. According to Netlab 360, Moobot is the codename of a self-propagating Mirai-based malware first discovered in 2019. It infects IoT (Internet of Things) devices using remotely exploitable vulnerabilities or weak default passwords. IoT is a term used to describe smart devices such as security hubs and cameras, smart TVs, smart speakers, smart lights, sensors, and even refrigerators that are connected to the Internet.

Once a device is infected by Moobot, control of the device is transferred to the operator of the command and control (C2) server, who can issue commands remotely such as attacking a target and locating additional vulnerable IoT devices to infect (self-propagation).

Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps

Moobot is a Mirai-based botnet, and has similar capabilities (modules) as Mirai:

  1. Self-propagation – The self-propagation module is in charge of the botnet’s growth. After an IoT device is infected, it randomly scans the Internet for open telnet ports and reports back to the C2 server. Once the C2 server gains knowledge of open telnet ports around the world, it tries to leverage known vulnerabilities or brute force its way into the IoT devices with common or default credentials.
Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps
Self-propagation
  1. Synchronized attacks – The C2 server orchestrates a coordinated flood of packets or HTTP requests with the goal of creating a denial of service event for the target’s website or service.
Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps
Synchronized attacks

The botnet operator may use multiple C2 servers in various locations around the world in order to reduce the risk of exposure. Infected devices may be assigned to different C2 servers varying by region and module; one server for self-propagation and another for launching attacks. Thus if a C2 server is compromised and taken down by law enforcement authorities, only parts of the botnet are deactivated.

Why this attack was not successful

This is the second large scale attack in the past few months that we observed on Cloudflare’s network. The previous one peaked at 754M packets per second and attempted to take down our routers with a high packet rate. Despite the high packet rate, the 754Mpps attack peaked at a mere 253 Gbps.

As opposed to the high packet rate attack, this attack was a high bit rate attack, peaking at 654 Gbps. Due to the high bit rates of this attack, it seems as though the attacker tried (and failed) to cause a denial of service event by saturating our Internet link capacity. So let’s explore why this attack was not successful.

Cloudflare’s global network capacity is over 42 Tbps and growing. Our network spans more than 200 cities in over 100 countries, including 17 cities in mainland China. It interconnects with over 8,800 networks globally, including major ISPs, cloud services, and enterprises. This level of interconnectivity along with the use of Anycast ensures that our network can easily absorb even the largest attacks.

Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps
The Cloudflare Network

After traffic arrives at an edge data center, it is then load-balanced efficiently using our own Layer 4 load-balancer that we built, Unimog, which uses our appliances’ health and other metrics to load-balance traffic intelligently within a data center to avoid overwhelming any single server.

Besides the use of Anycast for inter-data center load balancing and Unimog for intra-data center load balancing, we also utilize various forms of traffic engineering in order to deal with sudden changes in traffic loads across our network. We utilize both automatic and manual traffic engineering methods that can be employed by our 24/7/365 Site Reliability Engineering (SRE) team.

These combined factors significantly reduce the likelihood of a denial of service event due to link saturation or appliances being overwhelmed — and as seen in this attack, no link saturation occurred.

Detecting & Mitigating DDoS attacks

Once traffic arrives at our edge, it encounters our three software-defined DDoS protection systems:

  1. Gatebot – Cloudflare’s centralized DDoS protection systems for detecting and mitigating globally distributed volumetric DDoS attacks. Gatebot runs in our network’s core data center. It receives samples from every one of our edge data centers, analyzes them, and automatically sends mitigation instructions when attacks are detected. Gatebot is also synchronized to each of our customers’ web servers to identify its health and triggers mitigation accordingly.
  2. dosd (denial of service daemon) – Cloudflare’s decentralized DDoS protection systems. dosd runs autonomously in each server in every Cloudflare data center around the world, analyzing traffic and applying local mitigation rules when needed. Besides being able to detect and mitigate attacks at super-fast speeds, dosd significantly improves our network resilience by delegating the detection and mitigation capabilities to the edge.
  3. flowtrackd (flow tracking daemon) – Cloudflare’s TCP state tracking machine for detecting and mitigating the most randomized and sophisticated TCP-based DDoS attacks in unidirectional routing topologies (such as the case for Magic Transit). flowtrackd is able to identify the state of a TCP connection and then drops, challenges, or rate-limits packets that don’t belong to a legitimate connection.
Moobot vs. Gatebot: Cloudflare Automatically Blocks Botnet DDoS Attack Topping At 654 Gbps
Cloudflare DDoS Protection Lifecycle

The three DDoS protection systems collect traffic samples in order to detect DDoS attacks. The types of traffic data that they sample include:

  1. Packet fields such as the source IP, source port, destination IP, destination port, protocol, TCP flags, sequence number, options, and packet rate.
  2. HTTP request metadata such as HTTP headers, user agent, query-string, path, host, HTTP method, HTTP version, TLS cipher version, and request rate.
  3. HTTP response metrics such as error codes returned by customers’ origin servers and their rates.

Our systems then crunch these sample data points together to form a real-time view of our network’s security posture and our customer’s origin server health. They look for attack patterns and traffic anomalies. When found, a mitigation rule with a dynamically crafted attack signature is generated in real-time. Rules are propagated to the most optimal place for cost-effective mitigation. For example, an L7 HTTP flood might be dropped at L4 to reduce the CPU consumption.

Rules that are generated by dosd and flowtrackd are propagated within a single data center for rapid mitigation. Gatebot’s rules are propagated to all of the edge data centers which then take priority over dosd’s rules for an even and optimal mitigation. Even if the attack is detected in a subset of edge data centers, Gatebot propagates the mitigation instructions to all of Cloudflare’s edge data centers — effectively sharing the threat intelligence across our network as a form of proactive protection.

In the case of this attack, in each edge data center, dosd generated rules to mitigate the attack promptly. Then as Gatebot received and analyzed samples from the edge, it determined that this was a globally distributed attack. Gatebot propagated unified mitigation instructions to the edge, which prepared each and every one of our 200+ data centers to tackle the attack as the attack traffic may shift to a different data center due to Anycast or traffic engineering.

No inflated bills

DDoS attacks obviously pose the risk of an outage and service disruption. But there is another risk to consider — the cost of mitigation. During these ten days, more than 65 Terabytes of attack traffic were generated by the botnet. However, as part of Cloudflare’s unmetered DDoS protection guarantee, Cloudflare mitigated and absorbed the attack traffic without billing the customer. The customer doesn’t need to submit a retroactive credit request. Attack traffic is automatically excluded from our billing system. We eliminated the financial risk.