Efficiently Scaling kOps clusters with Amazon EC2 Spot Instances

Post Syndicated from Pranaya Anshu original https://aws.amazon.com/blogs/compute/efficiently-scaling-kops-clusters-with-amazon-ec2-spot-instances/

This post is written by Carlos Manzanedo Rueda, WW SA Leader for EC2 Spot, and Brandon Wagner, Senior Software Development Engineer for EC2.

This post focuses on how you can leverage recently released tools to optimize your usage of Amazon EC2 Spot Instances on Kubernetes Operations (kOps) clusters. Spot Instances let you utilize unused capacity in the AWS cloud for up to 90% off compared to On-Demand prices, and they are a great fit for fault-tolerant, containerized applications. kOps is an open source project providing a cohesive toolset for provisioning, operating, and deleting Kubernetes clusters in the cloud.

Even with customers such as Snap Inc., Babylon Health, and Fidelity Investments telling us how Amazon Elastic Kubernetes Service (EKS) is essential for running their containerized workloads, we appreciate that there are scenarios where using Amazon EC2 instances and kOps are a viable alternative. At AWS, we understand “one size does not fit all.” While we encourage Kubernetes users to contribute their feedback to the AWS container roadmap so that we can improve our services, we also would like to reduce heavy lifting and simplify Spot best practices integration in kOps clusters.

To simplify the integration of Spot Instances in kOps clusters, in January of 2021 we introduced a new kops toolbox command: kops toolbox instance-selector. The utility is distributed as part of the standard kOps distribution. Moreover, it simplifies the creation of kOps Instance Groups by configuring them with full adherence to Spot Instances best practices.

Handling Spot interruption notifications in Kubernetes

Let’s quickly recap Spot best practices. Spot Instances perform exactly like any other EC2 Instances, except that in exchange for their discounted price, they can be interrupted with a two-minute warning when EC2 must reclaim capacity. Applications running on Spot can typically recover from transient interruptions by simply starting a new instance. Spot best practices involve measures such as diversifying into as many Spot capacity pools as possible, choosing the right Spot allocation strategy, and utilizing Spot integrated services. These handle the Spot Instances lifecycles for you. This blog post on handling Spot interruptions dives deeper into AWS’s EC2 Spot best practices.

In Kubernetes, to handle spot termination and rebalance recommendation events (both explained in this blog post on proactively managing Spot Instance lifecycle), we utilize the AWS open-source project AWS Node Termination Handler. We will be deploying the Node Termination Handler as a kOps managed addon, which simplifies its setup and configuration.

The Node Termination Handler ensures that the Kubernetes control plane responds appropriately to events that can make EC2 instances unavailable. It can be operated in two different modes: Instance Metadata Service (IMDS), deployed as a DaemonSet, or Queue Processor, deployed as a Deployment Controller. We recommend running it in Queue Processor mode. The Queue Processor controller continuously monitors an Amazon Simple Queue Service (SQS) queue for events received from Amazon EventBridge. This can lead to node termination in your cluster. When one of these events is received, the Node Termination Handler notifies the Kubernetes control plane to cordon and drain the node that is about to be interrupted. Then, the kubelet sends a SIGTERM signal to the Pods and containers running on the node. This lets your application proceed with a graceful termination – one of the recommended best practices of a Twelve-Factor App.

The kOps managed addon will let you configure the Node Termination Handler within your kOps cluster spec and, more importantly, manage provisioning the necessary infrastructure for you.

To deploy the AWS Node Termination Handler, we start by editing our cluster spec:

kops edit cluster --name ${KOPS_CLUSTER_NAME}

We append the nodeTerminationHandler configuration to the spec node:

spec:
  nodeTerminationHandler:
    enabled: true
    enableSQSTerminationDraining: true
    managedASGTag: "aws-node-termination-handler/managed"

Finally, we deploy the changes made to our cluster configuration:

kops update cluster --name ${KOPS_CLUSTER_NAME} –-state {KOPS_STATE_STORE} --yes --admin

${KOPS_CLUSTER_NAME} refers to the environment variable containing the cluster name, and ${KOPS_STATE_STORE} indicates the Amazon Simple Storage Service (S3) bucket – or kOps State Store – where kOps configuration is stored.

To check that your Node Termination Handler deployment was successful, you can execute:

kops get deployment aws-node-termination-handler -n kube-system

Instance Flexibility and Diversification

Diversification and selection of multiple instances types is essential to acquire and maintain Spot capacity, as well as to successfully replace interrupted instances with others from different pools. When running kOps on AWS, this is implemented by utilizing Amazon EC2 Auto Scaling. Amazon Auto Scaling group’s capacity-optimized allocation strategy ensures that Spot capacity is provisioned from the optimal pools, thereby reducing the chances of Spot terminations.

Simplifying adoption of Spot Best practices on kOps

Before the kops toolbox instance-selector, you would have to setup Spot best practices on kOps manually. This involved writing a stub file following the InstanceGroup specification and examples, and then implementing every best practice, including finding every pool that qualifies for our workload.

The new functionality in kops toolbox instance-selector simplifies InstanceGroup creation by moving the focus of kOps users and administrators from this manual configuration over to simply selecting the vCPUs and Memory requirements for their application (or a base instance type), and then letting kops toolbox instance-selector define the right configuration. Behind the scenes, it utilizes a library allowing it to plug into the feature-set of Amazon EC2 instance selector. At its core, ec2 instance selector helps you select compatible instance types for your application to run on. Utilize ec2 instance selector CLI or library when automating your configurations. In the case of kOps, the integration already comes in the kops toolbox.

For example, let’s say your cluster runs stateless, fault tolerant applications that are CPU/Memory bound and have a ratio of vCPU to Memory requiring at least 1vCPU : 4GB of RAM. You can run the following command in order to acquire cluster spot capacity:

kops toolbox instance-selector "spot-group-" \
  --usage-class spot --flexible --cluster-autoscaler \
  --vcpus-to-memory-ratio="1:4" \
  --ig-count 2

Let’s focus first on the command, and later cover its output. You can get a list of parameters and default values by running: kops toolbox instance-selector –help. A few default parameters weren’t passed in the command above, but they will be set to sane defaults, such as the maximum and minimum number of instances in the Instance Group. The parameter –flexible refers to our request to provide a group of flexible instance types spanning multiple generations.

Once you’ve defined the InstanceGroups, start them up by using the command:

kops update cluster \
–state=${KOPS_STATE_STORE} \
–name=${KOPS_CLUSTER_NAME} \
–yes –admin

The two commands above define and create a request for spot capacity from a flexible and diversified pool set, which meet the criteria to provide at least 4GB of RAM for each vCPU. The command creates not just one, but two node groups named “spot-group-1” and “spot-group-2” (–ig-count 2).

Now, let’s check the contents of the configuration file generated by kops toolbox instance-selector. To preview a configuration without making changes, add –dry-run –output yaml.

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-08-11T10:22:16Z"
  labels:
    kops.k8s.io/cluster: spot-kops-cluster.k8s.local
  name: spot-group-1
spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/enabled: "1"
    k8s.io/cluster-autoscaler/spot-kops-cluster.k8s.local: "1"
    kops.k8s.io/instance-selector: "1"
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200716
  machineType: m3.xlarge
  maxSize: 15
  minSize: 2
  mixedInstancesPolicy:
    instances:
    - m3.xlarge
    - m4.xlarge
    - m5.xlarge
    - m5a.xlarge
    - t2.xlarge
    - t3.xlarge
    onDemandAboveBase: 0
    onDemandBase: 0
    spotAllocationStrategy: capacity-optimized
  nodeLabels:
    kops.k8s.io/instancegroup: spot-group-1
  role: Node
  subnets:
  - eu-west-1a
  - eu-west-1b
  - eu-west-1c
...

The configuration above lists one of the groups created by kops toolbox instance-selector in the previous example. The second group will have a very similar make-up and format, except that it will refer to instances such as: r3.xlarge, r4.xlarge, r5.xlarge, and r5a.xlarge in the mixedInstancesPolicy section. By defining the parameter –usage-class to Spot, the configuration created by kops toolbox instance-selector will add the tags identifying this Auto Scaling group as a Spot group. When the nodes are initialized, kOps controller will identify the nodes as Spot and add the label node-role.kubernetes.io/spot-worker=true. Therefore, at a later stage, we can apply placement logic to our cluster by using nodeSelector and affinity. The configuration above adheres to the definition of kOps support for mixed Instance Groups in AWS, and adds all of the right cloudLabels in order to integrate and implement not only with Spot best practices, but also with Cluster Autoscaler Auto-Discovery configuration best practices.

Kubernetes Cluster Autoscaler is a Kubernetes controller that dynamically adjusts the cluster size. According to a 2020 survey by Cloud Native Computing Foundation (CNCF), 70% of Kubernetes workloads plan to autoscale their stateless applications. Dynamically scaling applications and clusters is also a great practice for optimizing your system costs in situations where capacity is unnecessary, as well as for scaling out accordingly in order to meet business demands. If there are Pods that can’t be scheduled due to insufficient resources, then Cluster Autoscaler will issue a Scale-out action. When there are nodes in the cluster that have been under-utilized for a configurable period of time, Cluster Autoscaler will Scale-in the cluster, and even down-scale to 0 instances when applications don’t need to be run.

On Scale-out operations, Cluster Autoscaler evaluates a set of node groups. When Cluster Autoscaler runs on AWS, node groups are implemented by using Auto Scaling groups (referring to the same instance group as a kOps Instance Group). Therefore, to calculate the number of nodes to scale-out, Cluster Autoscaler assumes that every instance in a node group has the same number of vCPUs and memory size.

By creating two node groups, you apply two diversification levels. You diversify within each node group by using an Auto Scaling group with Mixed Instance Policies and capacity-optimized allocation strategy. Then, to increase the pool range you can leverage, you add more than one node group, while still adhering to the best practices required by Cluster Autoscaler.

While we’ve been focusing on Spot Instances, the parameter –usage-class can be utilized to get OnDemand instances instead of Spot. In the next example, let’s say we would like to get On-Demand capacity in order to train complex deep learning models that will take hours to run. To train our models, we need instances that have at least one GPU with 16GB of RAM on instances that have at least 32GB Ram and 8 vCPUs.

kops toolbox instance-selector "ondemand-gpu-group" \
  --gpus-min 1 --gpu-memory-total-min 16gb --memory-min 32gb --vcpus 8\
  --node-count-max 4 --node-count-min 4 --cpu-architecture amd64

The command above, followed by kops update cluster –state=${KOPS_STATE_STORE} –name=${KOPS_CLUSTER_NAME} –yes can be utilized to produce a configuration and create a nodegroup with the right requirements. This could be created at the start of the training procedure, and then – once the training is done and the capacity is no longer needed – you could automate the nodegroup removal with the following command:

kops delete instancegroup ondemand-gpu-group --name ${KOPS_CLUSTER_NAME} –yes

Conclusions

We believe the best way to run Kubernetes on AWS is by using Amazon EKS. However, scenarios may exist where kOps is utilized in AWS. By using the kOps managed add-on to install aws-node-termination-handler and kops toolbox instance-selector, it is easier than ever to apply Spot best practices to Kubernetes workloads on kOps, and cost-optimize fault-tolerant, stateless applications. These tools let kOps workloads gracefully terminate applications, as well as proactively handle the replacement of instances that are at an elevated risk of termination. kops toolbox instance-selector leverages Amazon ec2-instance-selector in order to simplify the creation of Instance Group configurations adhering to Spot Instances best practices, implementing instance type flexibility, and utilizing capacity-optimized allocation strategy.

By adhering to these best practices to reduce the frequency of Spot interruptions, we will optimize not only the cost, but also our Spot Instances selection. This will enable us to acquire capacity at a massive scale if necessary.

To start using the tools we have described, follow along this step-by-step tutorial. Also, head over to the kops toolbox documentation to learn more about the ways in which you can use it.

Looking back at 2021, looking forward at 2022 (Libre Arts)

Post Syndicated from original https://lwn.net/Articles/880811/rss

Here is a
comprehensive look
on the Libre Arts site at the current state of free
software for creative
artists.

The other reason is that, with a project like GIMP, it’s hard to do
just one thing. The team is constantly bombarded with requests that
are mostly doable once you have a team of 10 to 20 full-time
developers, which is light years away from where GIMP is now. Which
results in a lot of running around between under-the-hood work, UX
fixes, featurettes, better file formats support etc. So you give
everyone a little of what they want but you end up delaying an
actual release because the big stuff still needs to happen.

Dev corrupts NPM libs ‘colors’ and ‘faker’ breaking thousands of apps(Bleeping Computer)

Post Syndicated from original https://lwn.net/Articles/880809/rss

Bleeping Computer reports
on the latest NPM mess: the developer of the “faker” module deleted the
code and its development history from GitHub (with a force push), replaced
it with a malicious alternative, and broke dependencies for numerous
applications.

The reason behind this mischief on the developer’s part appears to
be retaliation—against mega-corporations and commercial consumers
of open-source projects who extensively rely on cost-free and
community-powered software but do not, according to the developer,
give back to the community.

GitHub has evidently called this action a violation of its terms of
service and disabled the owner’s account; NPM has restored a previous
version of the code.

Dev corrupts NPM libs ‘colors’ and ‘faker’ breaking thousands of apps (Bleeping Computer)

Post Syndicated from original https://lwn.net/Articles/880809/rss

Bleeping Computer reports
on the latest NPM mess: the developer of the “faker” module deleted the
code and broke dependencies for numerous applications.

The reason behind this mischief on the developer’s part appears to
be retaliation—against mega-corporations and commercial consumers
of open-source projects who extensively rely on cost-free and
community-powered software but do not, according to the developer,
give back to the community.

GitHub has evidently called the deletion a violation of its terms of
service and since restored the code, an action which raises interesting
questions of its own.

The 2021 Naughty and Nice Lists: Cybersecurity Edition

Post Syndicated from Jesse Mack original https://blog.rapid7.com/2022/01/10/the-2021-naughty-and-nice-lists-cybersecurity-edition/

The 2021 Naughty and Nice Lists: Cybersecurity Edition

Editor’s note: We had planned to publish our Hacky Holidays blog series throughout December 2021 – but then Log4Shell happened, and we dropped everything to focus on this major vulnerability that impacted the entire cybersecurity community worldwide. Now that it’s 2022, we’re feeling in need of some holiday cheer, and we hope you’re still in the spirit of the season, too. Throughout January, we’ll be publishing Hacky Holidays content (with a few tweaks, of course) to give the new year a festive start. So, grab an eggnog latte, line up the carols on Spotify, and let’s pick up where we left off.

It’s not just Santa who gets to have all the fun — we in the security community also love to make our lists and check them twice. That’s why we asked some of our trusty cybersecurity go-to’s who and what they’d place on their industry-specific naughty and nice lists, respectively, for 2021. Here’s who the experts we talked to would like to give a super-stuffed stocking filled with tokens of gratitude — and who’s getting a lump of coal.

The nice list

Call me boring, but I am pretty stoked about the Minimum Viable Security Product (MVSP), the vendor-neutral checklist for vetting third-party companies. It has questions like whether a vendor performs annual comprehensive penetration testing on systems, complies with local laws and regulations like GDPR, implemented single sign-on, applies security patches on a frequent basis, maintains a list of sensitive data types that the application is expected to process, keeps an up-to-date data flow diagram indicating how sensitive data reaches the systems, and whether vendors have layered perimeter controls or entry and exit logs for physical security. Its success depends on people using it, and this industry tends to be allergic to checklists, but it strikes me as super important. – Fahmida Y. Rashid, award-winning infosec journalist

Editor’s note: Check out our Security Nation podcast episode with Chris John Riley on his work helping develop MVSP.

All of the security researchers that have focused their research and efforts to identify vulnerabilities and security issues within IoT technology over the last year. Their effort have helped bring focus to these issues which has led to improvements in product and processes in the IoT industry. – Deral Heiland, IoT Research Lead at Rapid7

Increased federal government focus on securing critical infrastructure. Examples: pipeline and rail cybersecurity directives, energy sector sprints, cybersecurity funding in the infrastructure package. – Harley Geiger, Senior Director of Public Policy at Rapid7

Huntress Labs and the Reddit r/msp board for their outstanding, tireless support for those responding to the Kaseya mass ransomware attack. While the attack was devastating, the community coalesced to help triage and recover, showing the power we have as defenders and protectors when we all work together. – Bob Rudis, Chief Security Data Scientist at Rapid7

The January 20th swearing-in of Biden is on the nice list, not because of who won but the fact that the election worked. We’ve talked an excessive amount about election security, but the reality is, there was no big deal. It was a largely unremarkable election even in the abnormal environments of the pandemic and the cyber. Election computers will continue to be wildly insecure, but since we’ve got paper trails, it won’t really matter. – Rob Graham, CEO of Errata Security

The naughty list

The Colonial Pipeline and Kaseya attacks are far above any other “naughty” case. They affected millions of people around the world. However, like the big things from past years, I think it’ll be solved by lots of small actions by individuals rather than some big Government or Corporation initiative. No big action was needed to solve notPetya or Mirai; no big action will be needed here. Those threatened will steadily (albeit slowly) respond. – Rob Graham, CEO of Errata Security

Microsoft, bar none. They bungled response to many in-year critical vulnerabilities, putting strain on already beat up teams of protectors, causing many organizations to suffer at the mercy of attackers. Everything from multiple, severe Exchange vulnerabilities, to unfixable print spooler flaws, to being the #1 cloud document service for hosting malicious content. – Bob Rudis, Chief Security Data Scientist at Rapid7

The whole Pegasus spyware from NSO Group is bad news start to finish, but the fact that the ruler of United Arab Emirates used the spyware on his wife in a custody battle? That was just flabbergasting. We talk about stalkerware and other types of spyware — but when you have something like Pegasus just showing up on individual phones, that is downright frightening. – Fahmida Y. Rashid, award-winning infosec journalist

All manufacturers of IoT technology that have not heeded the warnings, taken advantages of the work done by IoT security researchers to improve their product security, or made efforts to build and improve their internal and external process for reporting and remediating security vulnerabilities within their products. – Deral Heiland, IoT Research Lead at Rapid7

Apparent lack of urgency to provide support and phase in requirements for healthcare cybersecurity, despite ransomware proliferation during the pandemic. – Harley Geiger, Senior Director of Public Policy at Rapid7

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

More Hacky Holidays blogs

Security updates for Monday

Post Syndicated from original https://lwn.net/Articles/880807/rss

Security updates have been issued by Debian (ghostscript and roundcube), Fedora (gegl04, mbedtls, and mediawiki), openSUSE (kubevirt, virt-api-container, virt-controller-container, virt-handler-container, virt-launcher-container, virt-operator-container), SUSE (kubevirt, virt-api-container, virt-controller-container, virt-handler-container, virt-launcher-container, virt-operator-container and libvirt), and Ubuntu (apache2).

DDoS Attack Trends for Q4 2021

Post Syndicated from Omer Yoachimik original https://blog.cloudflare.com/ddos-attack-trends-for-2021-q4/

DDoS Attack Trends for Q4 2021

This post is also available in 日本語, Deutsch, Français, Español.

DDoS Attack Trends for Q4 2021

The first half of 2021 witnessed massive ransomware and ransom DDoS attack campaigns that interrupted aspects of critical infrastructure around the world (including one of the largest petroleum pipeline system operators in the US) and a vulnerability in IT management software that targeted schools, public sector, travel organizations, and credit unions, to name a few.

The second half of the year recorded a growing swarm of one of the most powerful botnets deployed (Meris) and record-breaking HTTP DDoS attacks and network-layer attacks observed over the Cloudflare network. This besides the Log4j2 vulnerability (CVE-2021-44228) discovered in December that allows an attacker to execute code on a remote server — arguably one of the most severe vulnerabilities on the Internet since both Heartbleed and Shellshock.

Prominent attacks such as the ones listed above are but a few examples that demonstrate a trend of intensifying cyber-insecurity that affected everyone, from tech firms and government organizations to wineries and meat processing plants.

Here are some DDoS attack trends and highlights from 2021 and Q4 ‘21 specifically:

Ransom DDoS attacks

  • In Q4, ransom DDoS attacks increased by 29% YoY and 175% QoQ.
  • In December alone, one out of every three survey respondents reported being targeted by a ransom DDoS attack or threatened by the attacker.

Application-layer DDoS attacks

  • The Manufacturing industry was the most attacked in Q4 ’21, recording a whopping 641% increase QoQ in the number of attacks. The Business Services and Gaming/Gambling industries were the second and third most targeted industries by application-layer DDoS attacks.
  • For the fourth time in a row this year, China topped the charts with the highest percentage of attack traffic originating from its networks.
  • A new botnet called the Meris botnet emerged in mid-2021 and continued to bombard organizations around the world, launching some of the largest HTTP attacks on record — including a 17.2M rps attack that Cloudflare automatically mitigated.

Network-layer DDoS attacks

  • Q4 ’21 was the busiest quarter for attackers in 2021. In December 2021 alone, there were more than all the attacks observed in Q1 and Q2 ’21 separately.
  • While the majority of attacks were small, terabit-strong attacks became the new norm in the second half of 2021. Cloudflare automatically mitigated dozens of attacks peaking over 1 Tbps, with the largest one peaking just under 2 Tbps — the largest we’ve ever seen.
  • Q4 ’21, and November specifically, recorded a persistent ransom DDoS campaign against VoIP providers around the world.
  • Attacks originating from Moldova quadrupled in Q4 ’21 QoQ, making it the country with the highest percentage of network-layer DDoS activity.
  • SYN floods and UDP floods were the most frequent attack vectors while emerging threats such as SNMP attacks increased by nearly 5,800% QoQ.

This report is based on DDoS attacks that were automatically detected and mitigated by Cloudflare’s DDoS Protection systems. To learn more about how it works, check out this deep-dive blog post.

A note on how we measure DDoS attacks observed over our network

To analyze attack trends, we calculate the “DDoS activity” rate, which is the percentage of attack traffic out of the total traffic (attack + clean) observed over our global network. Measuring attack numbers as a percentage of the total traffic observed allows us to normalize data points and avoid biases reflected in absolute numbers towards, for example, a Cloudflare data center that receives more total traffic and likely, also more attacks.

An interactive version of this report is available on Cloudflare Radar.

Ransom Attacks

Our systems constantly analyze traffic and automatically apply mitigation when DDoS attacks are detected. Each DDoS’d customer is prompted with an automated survey to help us better understand the nature of the attack and the success of the mitigation.

For over two years now, Cloudflare has been surveying attacked customers — one question on the survey being if they received a ransom note demanding payment in exchange to stop the DDoS attack. Q4 ’21 recorded the highest survey responses ever that indicated ransom threats — ransom attacks increased by 29% YoY and 175% QoQ. More specifically, one out of every 4.5 respondents (22%) reported receiving a ransom letter demanding payment by the attacker.

DDoS Attack Trends for Q4 2021
The percentage of respondents reported being targeted by a ransom DDoS attack or that have received threats in advance of the attack.

When we break it down by month, we can see that December 2021 topped the charts with 32% of respondents reporting receiving a ransom letter — that’s nearly one out of every three surveyed respondents.

DDoS Attack Trends for Q4 2021

Application-layer DDoS attacks

Application-layer DDoS attacks, specifically HTTP DDoS attacks, are attacks that usually aim to disrupt a web server by making it unable to process legitimate user requests. If a server is bombarded with more requests than it can process, the server will drop legitimate requests and — in some cases — crash, resulting in degraded performance or an outage for legitimate users.

DDoS Attack Trends for Q4 2021

Application-layer DDoS attacks by industry

In Q4, DDoS attacks on Manufacturing companies increased by 641% QoQ, and DDoS attacks on the Business Services industry increased by 97%.

When we break down the application-layer attacks targeted by industry, the Manufacturing, Business Services, and Gaming/Gambling industries were the most targeted industries in Q4 ’21.

DDoS Attack Trends for Q4 2021

Application-layer DDoS attacks by source country

To understand the origin of the HTTP attacks, we look at the geolocation of the source IP address belonging to the client that generated the attack HTTP requests. Unlike network-layer attacks, source IP addresses cannot be spoofed in HTTP attacks. A high percentage of DDoS activity in a given country usually indicates the presence of botnets operating from within the country’s borders.

For the fourth quarter in a row, China remains the country with the highest percentage of DDoS attacks originating from within its borders. More than three out of every thousand HTTP requests that originated from Chinese IP addresses were part of an HTTP DDoS attack. The US remained in second place, followed by Brazil and India.

DDoS Attack Trends for Q4 2021

Application-layer DDoS attacks by target country

In order to identify which countries are targeted by the most HTTP DDoS attacks, we bucket the DDoS attacks by our customers’ billing countries and represent it as a percentage out of all DDoS attacks.

For the third consecutive time this year, organizations in the United States were targeted by the most HTTP DDoS attacks, followed by Canada and Germany.

DDoS Attack Trends for Q4 2021

Network-layer DDoS attacks

While application-layer attacks target the application (Layer 7 of the OSI model) running the service that end users are trying to access, network-layer attacks aim to overwhelm network infrastructure (such as in-line routers and servers) and the Internet link itself.

Cloudflare thwarts an almost 2 Tbps attack

In November, our systems automatically detected and mitigated an almost 2 Tbps DDoS attack. This was a multi-vector attack combining DNS amplification attacks and UDP floods. The entire attack lasted just one minute. The attack was launched from approximately 15,000 bots running a variant of the original Mirai code on IoT devices and unpatched GitLab instances.

DDoS Attack Trends for Q4 2021

Network-layer DDoS attacks by month

December was the busiest month for attackers in 2021.

Q4 ‘21 was the busiest quarter in 2021 for attackers. Over 43% of all network-layer DDoS attacks took place in the fourth quarter of 2021. While October was a relatively calmer month, in November, the month of the Chinese Singles’ Day, the American Thanksgiving holiday, Black Friday, and Cyber Monday, the number of network-layer DDoS attacks nearly doubled. The number of observed attacks increased towards the final days of December ’21 as the world prepared to close out the year. In fact, the total number of attacks in December alone was higher than all the attacks in Q2 ’21 and almost equivalent to all attacks in Q1 ’21.

DDoS Attack Trends for Q4 2021

Network-layer DDoS attacks by attack rate

While most attacks are still relatively ‘small’ in size, terabit-strong attacks are becoming the norm.

There are different ways of measuring the size of an L3/4 DDoS attack. One is the volume of traffic it delivers, measured as the bit rate (specifically, terabits per second or gigabits per second). Another is the number of packets it delivers, measured as the packet rate (specifically, millions of packets per second).

Attacks with high bit rates attempt to cause a denial-of-service event by clogging the Internet link, while attacks with high packet rates attempt to overwhelm the servers, routers, or other in-line hardware appliances. These devices dedicate a certain amount of memory and computation power to process each packet. Therefore, by bombarding it with many packets, the appliance can be left with no further processing resources. In such a case, packets are “dropped,” i.e., the appliance is unable to process them. For users, this results in service disruptions and denial of service.

The distribution of attacks by their size (in bit rate) and month is shown below. As seen in the graph above, the majority of attacks took place in December. However, the graph below illustrates that larger attacks, over 300 Gbps in size, took place in November. Most of the attacks between 5-20 Gbps took place in December.

DDoS Attack Trends for Q4 2021

Distribution by packet rate

An interesting correlation Cloudflare has observed is that when the number of attacks increases, their size and duration decrease. In the first two-thirds of 2021, the number of attacks was relatively small, and correspondingly, their rates increased, e.g., in Q3 ’21, attacks ranging from 1-10 million packets per second (mpps) increased by 196%. In Q4 ’21, the number of attacks increased and Cloudflare observed a decrease in the size of attacks. 91% of all attacks peaked below 50,000 packets per second (pps) — easily sufficient to take down unprotected Internet properties.

DDoS Attack Trends for Q4 2021

Larger attacks of over 1 mpps decreased by 48% to 28% QoQ, while attacks peaking below 50K pps increased by 2.36% QoQ.

DDoS Attack Trends for Q4 2021

Distribution by bit rate

Similar to the trend observed in packet-intensive attacks, the amount of bit-intensive attacks shrunk as well. While attacks over 1 Tbps are becoming the norm, with the largest one we’ve ever seen peak just below 2 Tbps, the majority of attacks are still small and peaked below 500 Mbps (97.2%).

DDoS Attack Trends for Q4 2021

In Q4 ’21, larger attacks of all ranges above 500 Mbps saw massive decreases ranging from 35% to 57% for the larger 100+ Gbps attacks.

DDoS Attack Trends for Q4 2021

Network-layer DDoS attacks by duration

Most attacks remain under one hour in duration, reiterating the need for automated always-on DDoS mitigation solutions.

We measure the duration of an attack by recording the difference between when it is first detected by our systems as an attack and the last packet we see with that attack signature towards that specific target. In the last quarter of 2021, 98% of all network-layer attacks lasted less than one hour. This is very common as most of the attacks are short-lived. Even more so, a trend we’ve seen is that when the number of attacks increases, as in this quarter, their rate and duration decreases.

DDoS Attack Trends for Q4 2021

Short attacks can easily go undetected, especially burst attacks that, within seconds, bombard a target with a significant number of packets, bytes, or requests. In this case, DDoS protection services that rely on manual mitigation by security analysis have no chance in mitigating the attack in time. They can only learn from it in their post-attack analysis, then deploy a new rule that filters the attack fingerprint and hope to catch it next time. Similarly, using an “on-demand” service, where the security team will redirect traffic to a DDoS provider during the attack, is also inefficient because the attack will already be over before the traffic routes to the on-demand DDoS provider.

It’s recommended that companies use automated, always-on DDoS protection services that analyze traffic and apply real-time fingerprinting fast enough to block short-lived attacks.

Attack vectors

SYN floods remain attackers’ favorite method of attack, while attacks over SNMP saw a massive surge of almost 5,800% QoQ.

An attack vector is a term used to describe the method that the attacker uses to launch their DDoS attack, i.e., the IP protocol, packet attributes such as TCP flags, flooding method, and other criteria.

For the first time in 2021, the percentage of SYN flood attacks significantly decreased. Throughout 2021, SYN floods accounted for 54% of all network-layer attacks on average. While still grabbing first place as the most frequent vector, its share dropped by 38% QoQ to 34%.

However, it was a close-run for SYN attacks and UDP attacks. A UDP flood is a type of denial-of-service attack in which a large number of User Datagram Protocol (UDP) packets are sent to a targeted server with the aim of overwhelming that device’s ability to process and respond. Oftentimes, the firewall protecting the targeted server can also become exhausted as a result of UDP flooding, resulting in a denial-of-service to legitimate traffic. Attacks over UDP jumped from fourth place in Q3 ’21 to second place in Q4 ’21, with a share of 32% of all network-layer attacks — a 1,198% increase in QoQ.

In third place came the SNMP underdog that made a massive leap with its first time 2021 appearance in the top attack vectors.

DDoS Attack Trends for Q4 2021

Emerging threats

When we look at emerging attack vectors — which helps us understand what new vectors attackers are deploying to launch attacks — we observe a massive spike in SNMP, MSSQL, and generic UDP-based DDoS attacks.

Both SNMP and MSSQL attacks are used to reflect and amplify traffic on the target by spoofing the target’s IP address as the source IP in the packets used to trigger the attack.

Simple Network Management Protocol (SNMP) is a UDP-based protocol that is often used to discover and manage network devices such as printers, switches, routers, and firewalls of a home or enterprise network on UDP well-known port 161. In an SNMP reflection attack, the attacker sends out a large number of SNMP queries while spoofing the source IP address in the packet as the targets to devices on the network that, in turn, reply to that target’s address. Numerous responses from the devices on the network results in the target network being DDoSed.

Similar to the SNMP amplification attack, the Microsoft SQL (MSSQL) attack is based on a technique that abuses the Microsoft SQL Server Resolution Protocol for the purpose of launching a reflection-based DDoS attack. The attack occurs when a Microsoft SQL Server responds to a client query or request, attempting to exploit the Microsoft SQL Server Resolution Protocol (MC-SQLR), listening on UDP port 1434.

DDoS Attack Trends for Q4 2021

Network-layer DDoS attacks by country

Attacks originating from Moldova quadrupled, making it the country with the highest percentage of network-layer DDoS activity.

When analyzing network-layer DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations where the traffic was ingested, and not by the source IP. The reason for this is that, when attackers launch network-layer attacks, they can spoof the source IP address in order to obfuscate the attack source and introduce randomness into the attack properties, which can make it harder for simple DDoS protection systems to block the attack. Hence, if we were to derive the source country based on a spoofed source IP, we would get a spoofed country.

Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of the Cloudflare data center in which the attack was observed. We are able to achieve geographical accuracy in our report because we have data centers in over 250 cities around the world.

DDoS Attack Trends for Q4 2021
DDoS Attack Trends for Q4 2021

To view all regions and countries, check out the interactive map.

Summary

Cloudflare’s mission is to help build a better Internet. A better Internet is one that is more secure, faster, and reliable for everyone — even in the face of DDoS attacks. As part of our mission, since 2017, we’ve been providing unmetered and unlimited DDoS protection for free to all of our customers. Over the years, it has become increasingly easier for attackers to launch DDoS attacks. To counter the attacker’s advantage, we want to make sure that it is also easy and free for organizations of all sizes to protect themselves against DDoS attacks of all types.

Not using Cloudflare yet? Start now.

Много сме, жалки сме!

Post Syndicated from Николай Марченко original https://bivol.bg/%D0%BC%D0%BD%D0%BE%D0%B3%D0%BE-%D1%81%D0%BC%D0%B5-%D0%B6%D0%B0%D0%BB%D0%BA%D0%B8-%D1%81%D0%BC%D0%B5.html

понеделник 10 януари 2022


Хич не обичам да съм назидателен, когато става въпрос за безобидни дивотии и детинщини. Не бях най-кроткият тийнейджър на света. Бил съм, били са ме, били сме се. Понякога съм…

Pluton is not (currently) a threat to software freedom

Post Syndicated from original https://mjg59.dreamwidth.org/58125.html

At CES this week, Lenovo announced that their new Z-series laptops would ship with AMD processors that incorporate Microsoft’s Pluton security chip. There’s a fair degree of cynicism around whether Microsoft have the interests of the industry as a whole at heart or not, so unsurprisingly people have voiced concerns about Pluton allowing for platform lock-in and future devices no longer booting non-Windows operating systems. Based on what we currently know, I think those concerns are understandable but misplaced.

But first it’s helpful to know what Pluton actually is, and that’s hard because Microsoft haven’t actually provided much in the way of technical detail. The best I’ve found is a discussion of Pluton in the context of Azure Sphere, Microsoft’s IoT security platform. This, in association with the block diagrams on page 12 and 13 of this slidedeck, suggest that Pluton is a general purpose security processor in a similar vein to Google’s Titan chip. It has a relatively low powered CPU core, an RNG, and various hardware cryptography engines – there’s nothing terribly surprising here, and it’s pretty much the same set of components that you’d find in a standard Trusted Platform Module of the sort shipped in pretty much every modern x86 PC. But unlike Titan, Pluton seems to have been designed with the explicit goal of being incorporated into other chips, rather than being a standalone component. In the Azure Sphere case, we see it directly incorporated into a Mediatek chip. In the Xbox Series devices, it’s incorporated into the SoC. And now, we’re seeing it arrive on general purpose AMD CPUs.

Microsoft’s announcement says that Pluton can be shipped in three configurations:as the Trusted Platform Module; as a security processor used for non-TPM scenarios like platform resiliency; or OEMs can choose to ship with Pluton turned off. What we’re likely to see to begin with is the former – Pluton will run firmware that exposes a Trusted Computing Group compatible TPM interface. This is almost identical to the status quo. Microsoft have required that all Windows certified hardware ship with a TPM for years now, but for cost reasons this is often not in the form of a separate hardware component. Instead, both Intel and AMD provide support for running the TPM stack on a component separate from the main execution cores on the system – for Intel, this TPM code runs on the Management Engine integrated into the chipset, and for AMD on the Platform Security Processor that’s integrated into the CPU package itself.

So in this respect, Pluton changes very little; the only difference is that the TPM code is running on hardware dedicated to that purpose, rather than alongside other code. Importantly, in this mode Pluton will not do anything unless the system firmware or OS ask it to. Pluton cannot independently block the execution of any other code – it knows nothing about the code the CPU is executing unless explicitly told about it. What the OS can certainly do is ask Pluton to verify a signature before executing code, but the OS could also just verify that signature itself. Windows can already be configured to reject software that doesn’t have a valid signature. If Microsoft wanted to enforce that they could just change the default today, there’s no need to wait until everyone has hardware with Pluton built-in.

The two things that seem to cause people concerns are remote attestation and the fact that Microsoft will be able to ship firmware updates to Pluton via Windows Update. I’ve written about remote attestation before, so won’t go into too many details here, but the short summary is that it’s a mechanism that allows your system to prove to a remote site that it booted a specific set of code. What’s important to note here is that the TPM (Pluton, in the scenario we’re talking about) can’t do this on its own – remote attestation can only be triggered with the aid of the operating system. Microsoft’s Device Health Attestation is an example of remote attestation in action, and the technology definitely allows remote sites to refuse to grant you access unless you booted a specific set of software. But there are two important things to note here: first, remote attestation cannot prevent you from booting whatever software you want, and second, as evidenced by Microsoft already having a remote attestation product, you don’t need Pluton to do this! Remote attestation has been possible since TPMs started shipping over two decades ago.

The other concern is Microsoft having control over the firmware updates. The context here is that TPMs are not magically free of bugs, and sometimes these can have security consequences. One example is Infineon TPMs producing weak RSA keys, a vulnerability that could be rectified by a firmware update to the TPM. Unfortunately these updates had to be issued by the device manufacturer rather than Infineon being able to do so directly. This meant users had to wait for their vendor to get around to shipping an update, something that might not happen at all if the machine was sufficiently old. From a security perspective, being able to ship firmware updates for the TPM without them having to go through the device manufacturer is a huge win.

Microsoft’s obviously in a position to ship a firmware update that modifies the TPM’s behaviour – there would be no technical barrier to them shipping code that resulted in the TPM just handing out your disk encryption secret on demand. But Microsoft already control the operating system, so they already have your disk encryption secret. There’s no need for them to backdoor the TPM to give them something that the TPM’s happy to give them anyway. If you don’t trust Microsoft then you probably shouldn’t be running Windows, and if you’re not running Windows Microsoft can’t update the firmware on your TPM.

So, as of now, Pluton running firmware that makes it look like a TPM just isn’t a terribly interesting change to where we are already. It can’t block you running software (either apps or operating systems). It doesn’t enable any new privacy concerns. There’s no mechanism for Microsoft to forcibly push updates to it if you’re not running Windows.

Could this change in future? Potentially. Microsoft mention another use-case for Pluton “as a security processor used for non-TPM scenarios like platform resiliency”, but don’t go into any more detail. At this point, we don’t know the full set of capabilities that Pluton has. Can it DMA? Could it play a role in firmware authentication? There are scenarios where, in theory, a component such as Pluton could be used in ways that would make it more difficult to run arbitrary code. It would be reassuring to hear more about what the non-TPM scenarios are expected to look like and what capabilities Pluton actually has.

But let’s not lose sight of something more fundamental here. If Microsoft wanted to block free operating systems from new hardware, they could simply mandate that vendors remove the ability to disable secure boot or modify the key databases. If Microsoft wanted to prevent users from being able to run arbitrary applications, they could just ship an update to Windows that enforced signing requirements. If they want to be hostile to free software, they don’t need Pluton to do it.

comment count unavailable comments

Лобистът Тони Подеста обслужва бизнеси свързвани с Пеевски. Отворил фирма с Мондешки

Post Syndicated from Екип на Биволъ original https://bivol.bg/podesta-lobbying-peevski.html

събота 8 януари 2022


Лобистът на Демократическата партия Антъни Подеста е подписал лобистки договори на стойност над 2 милиона долара с български компании, чиито собственици се свързват с интересите на санкционирания по глобалния закон “Магнитски” олигарх Делян…

The collective thoughts of the interwebz

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close