DDoS threat report for 2023 Q2

2023-07-18 Omer Yoachimik

Post Syndicated from Omer Yoachimik original http://blog.cloudflare.com/ddos-threat-report-2023-q2/

Welcome to the second DDoS threat report of 2023. DDoS attacks, or distributed denial-of-service attacks, are a type of cyber attack that aims to disrupt websites (and other types of Internet properties) to make them unavailable for legitimate users by overwhelming them with more traffic than they can handle — similar to a driver stuck in a traffic jam on the way to the grocery store.

We see a lot of DDoS attacks of all types and sizes and our network is one of the largest in the world spanning more than 300 cities in over 100 countries. Through this network we serve over 63 million HTTP requests per second at peak and over 2 billion DNS queries every day. This colossal amount of data gives us a unique vantage point to provide the community access to insightful DDoS trends.

For our regular readers, you might notice a change in the layout of this report. We used to follow a set pattern to share our insights and trends about DDoS attacks. But with the landscape of DDoS threats changing as DDoS attacks have become more powerful and sophisticated, we felt it's time for a change in how we present our findings. So, we'll kick things off with a quick global overview, and then dig into the major shifts we're seeing in the world of DDoS attacks.

Reminder: an interactive version of this report is also available on Cloudflare Radar. Furthermore, we’ve also added a new interactive component that will allow you to dive deeper into attack activity in each country or region.

The DDoS landscape: a look at global patterns

The second quarter of 2023 was characterized by thought-out, tailored and persistent waves of DDoS attack campaigns on various fronts, including:

Multiple DDoS offensives orchestrated by pro-Russian hacktivist groups REvil, Killnet and Anonymous Sudan against Western interest websites.
An increase in deliberately engineered and targeted DNS attacks alongside a 532% surge in DDoS attacks exploiting the Mitel vulnerability (CVE-2022-26143). Cloudflare contributed to disclosing this zero-day vulnerability last year.
Attacks targeting Cryptocurrency companies increased by 600%, as a broader 15% increase in HTTP DDoS attacks was observed. Of these, we’ve noticed an alarming escalation in attack sophistication which we will cover more in depth.

Additionally, one of the largest attacks we’ve seen this quarter was an ACK flood DDoS attack which originated from a Mirai-variant botnet comprising approximately 11K IP addresses. The attack targeted an American Internet Service Provider. It peaked at 1.4 terabit per seconds (Tbps) and was automatically detected and mitigated by Cloudflare’s systems.

Despite general figures indicating an increase in overall attack durations, most of the attacks are short-lived and so was this one. This attack lasted only two minutes. However, more broadly, we’ve seen that attacks exceeding 3 hours have increased by 103% QoQ.

Now having set the stage, let’s dive deeper into these shifts we’re seeing in the DDoS landscape.

Hacktivist alliance dubbed “Darknet Parliament” aims at Western banks and SWIFT network

On June 14, Pro-Russian hacktivist groups Killnet, a resurgence of REvil and Anonymous Sudan announced that they have joined forces to execute “massive” cyber attacks on the Western financial system including European and US banks, and the US Federal Reserve System. The collective, dubbed “Darknet Parliament”, declared its first objective was to paralyze SWIFT (Society for Worldwide Interbank Financial Telecommunication). A successful DDoS attack on SWIFT could have dire consequences because it's the main service used by financial institutions to conduct global financial transactions.

Beyond a handful of publicized events such as the Microsoft outage which was reported by the media, we haven’t observed any novel DDoS attacks or disruptions targeting our customers. Our systems have been automatically detecting and mitigating attacks associated with this campaign. Over the past weeks, as many as 10,000 of these DDoS attacks were launched by the Darknet Parliament against Cloudflare-protected websites (see graph below).

Despite the hacktivists’ statements, Banking and Financial Services websites were only the ninth most attacked industry — based on attacks we’ve seen against our customers as part of this campaign.

The most attacked industries were Computer Software, Gambling & Casinos and Gaming. Telecommunications and Media outlets came in fourth and fifth, respectively. Overall, the largest attack we witnessed in this campaign peaked at 1.7 million requests per second (rps) and the average was 65,000 rps.

For perspective, earlier this year we mitigated the largest attack in recorded history peaking at 71 million rps. So these attacks were very small compared to Cloudflare scale, but not necessarily for an average website. Therefore, we shouldn’t underestimate the damage potential on unprotected or suboptimally configured websites.

Sophisticated HTTP DDoS attacks

An HTTP DDoS attack is a DDoS attack over the Hypertext Transfer Protocol (HTTP). It targets HTTP Internet properties such as websites and API gateways. Over the past quarter, HTTP DDoS attacks increased by 15% quarter-over-quarter (QoQ) despite a 35% decrease year-over-year (YoY).

Additionally, we've observed an alarming uptick in highly-randomized and sophisticated HTTP DDoS attacks over the past few months. It appears as though the threat actors behind these attacks have deliberately engineered the attacks to try and overcome mitigation systems by adeptly imitating browser behavior very accurately, in some cases, by introducing a high degree of randomization on various properties such as user agents and JA3 fingerprints to name a few. An example of such an attack is provided below. Each different color represents a different randomization feature.

Furthermore, in many of these attacks, it seems that the threat actors try to keep their attack rates-per-second relatively low to try and avoid detection and hide amongst the legitimate traffic.

This level of sophistication has previously been associated with state-level and state-sponsored threat actors, and it seems these capabilities are now at the disposal of cyber criminals. Their operations have already targeted prominent businesses such as a large VoIP provider, a leading semiconductor company, and a major payment & credit card provider to name a few.

Protecting websites against sophisticated HTTP DDoS attacks requires intelligent protection that is automated and fast, that leverages threat intelligence, traffic profiling and Machine Learning/statistical analysis to differentiate between attack traffic and user traffic. Moreover, even increasing caching where applicable can help reduce the risk of attack traffic impacting your origin. Read more about DDoS protection best practices here.

DNS Laundering DDoS attacks

The Domain Name System, or DNS, serves as the phone book of the Internet. DNS helps translate the human-friendly website address (e.g. www.cloudflare.com) to a machine-friendly IP address (e.g. 104.16.124.96). By disrupting DNS servers, attackers impact the machines’ ability to connect to a website, and by doing so making websites unavailable to users.

Over the past quarter, the most common attack vector was DNS-based DDoS attacks — 32% of all DDoS attacks were over the DNS protocol. Amongst these, one of the more concerning attack types we’ve seen increasing is the DNS Laundering attack which can pose severe challenges to organizations that operate their own authoritative DNS servers.

The term “Laundering” in the DNS Laundering attack name refers to the analogy of money laundering, the devious process of making illegally-gained proceeds, often referred to as "dirty money," appear legal. Similarly, in the DDoS world, a DNS Laundering attack is the process of making bad, malicious traffic appear as good, legitimate traffic by laundering it via reputable recursive DNS resolvers.

In a DNS Laundering attack, the threat actor will query subdomains of a domain that is managed by the victim’s DNS server. The prefix that defines the subdomain is randomized and is never used more than once or twice in such an attack. Due to the randomization element, recursive DNS servers will never have a cached response and will need to forward the query to the victim’s authoritative DNS server. The authoritative DNS server is then bombarded by so many queries until it cannot serve legitimate queries or even crashes all together.

From the protection point of view, the DNS administrators can’t block the attack source because the source includes reputable recursive DNS servers like Google’s 8.8.8.8 and Cloudflare’s 1.1.1.1. The administrators also cannot block all queries to the attacked domain because it is a valid domain that they want to preserve access to legitimate queries.

The above factors make it very challenging to distinguish legitimate queries from malicious ones. A large Asian financial institution and a North American DNS provider are amongst recent victims of such attacks. An example of such an attack is provided below.

Similar to the protection strategies outlined for HTTP applications, protecting DNS servers also requires a precise, fast, and automated approach. Leveraging a managed DNS service or a DNS reverse proxy such as Cloudflare’s can help absorb and mitigate the attack traffic. For those more sophisticated DNS attacks, a more intelligent solution is required that leverages statistical analysis of historical data to be able to differentiate between legitimate queries and attack queries.

The rise of the Virtual Machine Botnets

As we’ve previously disclosed, we are witnessing an evolution in botnet DNA. The era of VM-based DDoS botnets has arrived and with it hyper-volumetric DDoS attacks. These botnets are comprised of Virtual Machines (VMs, or Virtual Private Servers, VPS) rather than Internet of Things (IoT) devices which makes them so much more powerful, up to 5,000 times stronger.

Because of the computational and bandwidth resources that are at the disposal of these VM-based botnets, they’re able to generate hyper-volumetric attacks with a much smaller fleet size compared to IoT-based botnets.

These botnets have executed one largest recorded DDoS attacks including the 71 million request per second DDoS attack. Multiple organizations including an industry-leading gaming platform provider have already been targeted by this new generation of botnets.

Cloudflare has proactively collaborated with prominent cloud computing providers to combat these new botnets. Through the quick and dedicated actions of these providers, significant components of these botnets have been neutralized. Since this intervention, we have not observed any further hyper-volumetric attacks yet, a testament to the efficacy of our collaboration.

While we already enjoy a fruitful alliance with the cybersecurity community in countering botnets when we identify large-scale attacks, our goal is to streamline and automate this process further. We extend an invitation to cloud computing providers, hosting providers, and other general service providers to join Cloudflare’s free Botnet Threat Feed. This would provide visibility into attacks originating within their networks, contributing to our collective efforts to dismantle botnets.

“Startblast”: Exploiting Mitel vulnerabilities for DDoS attacks

In March 2023, we disclosed a zero-day vulnerability (CVE-2022-26143), named TP240PhoneHome, which was identified in the Mitel MiCollab business phone system, exposing the system to UDP amplification DDoS attacks.

This exploit operates by reflecting traffic off vulnerable servers, amplifying it in the process, with a factor as high as 220 billion percent. The vulnerability stems from an unauthenticated UDP port exposed to the public Internet, which could allow malicious actors to issue a 'startblast' debugging command, simulating a flurry of calls to test the system.

As a result, for each test call, two UDP packets are sent to the issuer, enabling an attacker to direct this traffic to any IP and port number to amplify a DDoS attack. Despite the vulnerability, only a few thousand of these devices are exposed, limiting the potential scale of attack, and attacks must run serially, meaning each device can only launch one attack at a time.

Overall, in the past quarter, we’ve seen additional emerging threats such as DDoS attacks abusing the TeamSpeak3 protocol. This attack vector increased by a staggering 403% this quarter.

TeamSpeak, a proprietary voice-over-Internet Protocol (VoIP) that runs over UDP to help gamers talk with other gamers in real time. Talking instead of just chatting can significantly improve a gaming team’s efficiency and help them win. DDoS attacks that target TeamSpeak servers may be launched by rival groups in an attempt to disrupt their communication path during real-time multiplayer games and thus impact their team’s performance.

DDoS hotspots: The origins of attacks

Overall, HTTP DDoS attacks increased by 15% QoQ despite a 35% decrease YoY. Additionally, network-layer DDoS attacks decreased this quarter by approximately 14%.

In terms of total volume of attack traffic, the US was the largest source of HTTP DDoS attacks. Three out of every thousand requests we saw were part of HTTP DDoS attacks originating from the US. China came in second place and Germany in third place.

Some countries naturally receive more traffic due to various factors such as market size, and therefore more attacks. So while it’s interesting to understand the total amount of attack traffic originating from a given country, it is also helpful to remove that bias by normalizing the attack traffic by all traffic to a given country.

When doing so, we see a different pattern. The US doesn’t even make it into the top ten. Instead, Mozambique, Egypt and Finland take the lead as the source countries of the most HTTP DDoS attack traffic relative to all of their traffic. Almost a fifth of all HTTP traffic originating from Mozambique IP addresses were part of DDoS attacks.

Using the same calculation methodology but for bytes, Vietnam remains the largest source of network-layer DDoS attacks (aka L3/4 DDoS attacks) for the second consecutive quarter — and the amount even increased by 58% QoQ. Over 41% of all bytes that were ingested in Cloudflare’s Vietnam data centers were part of L3/4 DDoS attacks.

Industries under attack: examining DDoS attack targets

When examining HTTP DDoS attack activity in Q2, Cryptocurrency websites were targeted with the largest amount of HTTP DDoS attack traffic. Six out of every ten thousand HTTP requests towards Cryptocurrency websites behind Cloudflare were part of these attacks. This represents a 600% increase compared to the previous quarter.

After Crypto, Gaming and Gambling websites came in second place as their attack share increased by 19% QoQ. Marketing and Advertising websites not far behind in third place with little change in their share of attacks.

However, when we look at the amount of attack traffic relative to all traffic for any given industry, the numbers paint a different picture. Last quarter, Non-profit organizations were attacked the most — 12% of traffic to Non-profits were HTTP DDoS attacks. Cloudflare protects more than 2,271 Non-profit organizations in 111 countries as part of Project Galileo which celebrated its ninth anniversary this year. Over the past months, an average of 67.7 million cyber attacks targeted Non-profits on a daily basis.

Overall, the amount of DDoS attacks on Non-profits increased by 46% bringing the percentage of attack traffic to 17.6%. However, despite this growth, the Management Consulting industry jumped to the first place with 18.4% of its traffic being DDoS attacks.

When descending the layers of the OSI model, the Internet networks that were most targeted belonged to the Information Technology and Services industry. Almost every third byte routed to them were part of L3/4 DDoS attacks.

Surprisingly enough, companies operating in the Music industry were the second most targeted industry, followed by Broadcast Media and Aviation & Aerospace.

Top attacked industries: a regional perspective

Cryptocurrency websites experienced the highest number of attacks worldwide, while Management Consulting and Non-profit sectors were the most targeted considering their total traffic. However, when we look at individual regions, the situation is a bit different.

Africa

The Telecommunications industry remains the most attacked industry in Africa for the second consecutive quarter. The Banking, Financial Services and Insurance (BFSI) industry follows as the second most attacked. The majority of the attack traffic originated from Asia (35%) and Europe (25%).

Asia

For the past two quarters, the Gaming and Gambling industry was the most targeted industry in Asia. In Q2, however, the Gaming and Gambling industry dropped to second place and Cryptocurrency took the lead as the most attacked industry (~50%). Substantial portions of the attack traffic originated from Asia itself (30%) and North America (30%).

Europe

For the third consecutive quarter, the Gaming & Gambling industry remains the most attacked industry in Europe. The Hospitality and Broadcast Media industries follow not too far behind as the second and third most attacked. Most of the attack traffic came from within Europe itself (40%) and from Asia (20%).

Latin America

Surprisingly, half of all attack traffic targeting Latin America was aimed at the Sporting Goods industry. In the previous quarter, the BFSI was the most attacked industry. Approximately 35% of the attack traffic originated from Asia, and another 25% originated from Europe.

Middle East

The Media & Newspaper industries were the most attacked in the Middle East. The vast majority of attack traffic originated from Europe (74%).

North America

For the second consecutive quarter, Marketing & Advertising companies were the most attacked in North America (approximately 35%). Manufacturing and Computer Software companies came in second and third places, respectively. The main sources of the attack traffic were Europe (42%) and the US itself (35%).

Oceania

This quarter, the Biotechnology industry was the most attacked. Previously, it was the Health & Wellness industry. Most of the attack traffic originated from Asia (38%) and Europe (25%).

Countries and regions under attack: examining DDoS attack targets

When examining the total volume of attack traffic, last quarter, Israel leaped to the front as the most attacked country. This quarter, attacks targeting Israeli websites decreased by 33% bringing it to the fourth place. The US takes the lead again as the most attacked country, followed by Canada and Singapore.

If we normalize the data per country and region and divide the attack traffic by the total traffic, we get a different picture. Palestine jumps to the first place as the most attacked country. Almost 12% of all traffic to Palestinian websites were HTTP DDoS attacks.

Last quarter, we observed a striking deviation at the network layer, with Finnish networks under Cloudflare's shield emerging as the primary target. This surge was likely correlated with the diplomatic talks that precipitated Finland's formal integration into NATO. Roughly 83% of all incoming traffic to Finland comprised cyberattacks, with China a close second at 68% attack traffic.

This quarter, however, paints a very different picture. Finland has receded from the top ten, and Chinese Internet networks behind Cloudflare have ascended to the first place. Almost two-thirds of the byte streams towards Chinese networks protected by Cloudflare were malicious. Following China, Switzerland saw half of its inbound traffic constituting attacks, and Turkey came third, with a quarter of its incoming traffic identified as hostile.

Ransom DDoS attacks

Occasionally, DDoS attacks are carried out to extort ransom payments. We’ve been surveying Cloudflare customers over three years now, and have been tracking the occurrence of Ransom DDoS attack events.

Unlike Ransomware attacks, where victims typically fall prey to downloading a malicious file or clicking on a compromised email link which locks, deletes or leaks their files until a ransom is paid, Ransom DDoS attacks can be much simpler for threat actors to execute. Ransom DDoS attacks bypass the need for deceptive tactics such as luring victims into opening dubious emails or clicking on fraudulent links, and they don't necessitate a breach into the network or access to corporate resources.

Over the past quarter, reports of Ransom DDoS attacks decreased. One out of ten respondents reported being threatened or subject to Ransom DDoS attacks.

Wrapping up: the ever-evolving DDoS threat landscape

In recent months, there's been an alarming escalation in the sophistication of DDoS attacks. And even the largest and most sophisticated attacks that we’ve seen may only last a few minutes or even seconds — which doesn’t give a human sufficient time to respond. Before the PagerDuty alert is even sent, the attack may be over and the damage is done. Recovering from a DDoS attack can last much longer than the attack itself — just as a boxer might need a while to recover from a punch to the face that only lasts a fraction of a second.

Security is not one single product or a click of a button, but rather a process involving multiple layers of defense to reduce the risk of impact. Cloudflare's automated DDoS defense systems consistently safeguard our clients from DDoS attacks, freeing them up to focus on their core business operations. These systems are complemented by the vast breadth of Cloudflare capabilities such as firewall, bot detection, API protection and even caching which can all contribute to reducing the risk of impact.

The DDoS threat landscape is evolving and increasingly complex, demanding more than just quick fixes. Thankfully, with Cloudflare's multi-layered defenses and automatic DDoS protections, our clients are equipped to navigate these challenges confidently. Our mission is to help build a better Internet, and so we continue to stand guard, ensuring a safer and more reliable digital realm for all.

Methodologies

How we calculate Ransom DDoS attack insights

Cloudflare’s systems constantly analyze traffic and automatically apply mitigation when DDoS attacks are detected. Each attacked customer is prompted with an automated survey to help us better understand the nature of the attack and the success of the mitigation. For over two years, Cloudflare has been surveying attacked customers. One of the questions in the survey asks the respondents if they received a threat or a ransom note. Over the past two years, on average, we collected 164 responses per quarter. The responses of this survey are used to calculate the percentage of Ransom DDoS attacks.

How we calculate geographical and industry insights

Source country
At the application-layer, we use the attacking IP addresses to understand the origin country of the attacks. That is because at that layer, IP addresses cannot be spoofed (i.e., altered). However, at the network layer, source IP addresses can be spoofed. So, instead of relying on IP addresses to understand the source, we instead use the location of our data centers where the attack packets were ingested. We’re able to get geographical accuracy due to our large global coverage in over 285 locations around the world.

Target country
For both application-layer and network-layer DDoS attacks, we group attacks and traffic by our customers’ billing country. This lets us understand which countries are subject to more attacks.

Target industry
For both application-layer and network-layer DDoS attacks, we group attacks and traffic by our customers’ industry according to our customer relations management system. This lets us understand which industries are subject to more attacks.

Total volume vs. percentage
For both source and target insights, we look at the total volume of attack traffic compared to all traffic as one data point. Additionally, we also look at the percentage of attack traffic towards or from a specific country, to a specific country or to a specific industry. This gives us an “attack activity rate” for a given country/industry which is normalized by their total traffic levels. This helps us remove biases of a country or industry that normally receives a lot of traffic and therefore a lot of attack traffic as well.

How we calculate attack characteristics
To calculate the attack size, duration, attack vectors and emerging threats, we bucket attacks and then provide the share of each bucket out of the total amount for each dimension. On the new Radar component, these trends are calculated by number of bytes instead. Since attacks may vary greatly in number of bytes from one another, this could lead to trends differing between the reports and the Radar component.

General disclaimer and clarification

When we describe ‘top countries’ as the source or target of attacks, it does not necessarily mean that that country was attacked as a country, but rather that organizations that use that country as their billing country were targeted by attacks. Similarly, attacks originating from a country does not mean that that country launched the attacks, but rather that the attack was launched from IP addresses that have been mapped to that country. Threat actors operate global botnets with nodes all over the world, and in many cases also use Virtual Private Networks and proxies to obfuscate their true location. So if anything, the source country could indicate the presence of exit nodes or botnet nodes within that country.

Informe sobre las amenazas DDoS en el 2º trimestre de 2023

2023-07-18 Omer Yoachimik

Post Syndicated from Omer Yoachimik original http://blog.cloudflare.com/ddos-threat-report-2023-q2-es-es/

Informe sobre las amenazas DDoS en el 2º trimestre de 2023

Te damos la bienvenida al segundo informe sobre amenazas DDoS de 2023. Los ataques DDoS, o ataques de denegación de servicio distribuido, son un tipo de ciberataque cuyo objetivo es sobrecargar de tráfico sitios web (y otros tipos de propiedades de Internet) para interrumpir el funcionamiento normal y que los usuarios legítimos no puedan acceder a ellos, lo mismo que cuando un conductor está atrapado en un atasco de camino al supermercado.

Observamos muchos ataques DDoS de diferentes tipos y tamaños, y nuestra red es una de las mayores del mundo, ya que abarca más de 300 ciudades en más de 100 países. A través de esta red atendemos más de 63 millones de solicitudes HTTP por segundo durante picos de tráfico y más de 2 billones de consultas de DNS cada día. Esta ingente cantidad de datos nos ofrece una perspectiva privilegiada para dar a conocer a la comunidad tendencias reveladoras sobre los ataques DDoS.

Nuestros lectores habituales quizá noten un cambio en el diseño de este informe. Solíamos seguir un patrón fijo para compartir nuestras percepciones y tendencias sobre los ataques DDoS. Sin embargo, creemos que ha llegado el momento de cambiar la forma de presentar nuestras conclusiones en vista de los cambios observados en el panorama de las amenazas DDoS conforme avanzan en potencia y sofisticación. Así pues, empezaremos con una rápida visión global y, a continuación, profundizaremos en los principales cambios que estamos observando en el mundo de los ataques DDoS.

Recordatorio: puedes consultar la versión interactiva de este informe en Cloudflare Radar. Además, hemos añadido un nuevo elemento interactivo que te permitirá analizar la actividad de los ataques en cada país o región.

Panorama de los ataques DDoS: análisis de los patrones globales

El 2º trimestre de 2023 se caracterizó por oleadas de campañas de ataques DDoS persistentes, que se concibieron y adaptaron para dirigirse a varios frentes. Destacamos:

Numerosas ofensivas DDoS orquestadas por los grupos hacktivistas prorrusos REvil, Killnet y Anonymous Sudan contra sitios web de interés en países occidentales.
Un aumento de los ataques DNS dirigidos y diseñados intencionadamente, junto con un incremento del 532 % en los ataques DDoS contra la vulnerabilidad Mitel (CVE-2022-26143). Cloudflare contribuyó a revelar esta vulnerabilidad de día cero el año pasado.
Los ataques contra empresas de criptomonedas se dispararon un 600 %, al tiempo que se observó un aumento generalizado del 15 % en los ataques DDoS HTTP. Hemos observado una escalada alarmante en la sofisticación de este tipo de ataques, que trataremos más en profundidad.

Además, uno de los mayores ataques que hemos observado este trimestre fue un ataque DDoS de inundación ACK que se originó en una variante de la botnet Mirai y que comprendía aproximadamente 11 000 direcciones IP. El ataque iba dirigido a un proveedor de acceso a Internet estadounidense y alcanzó un pico de 1,4 terabits por segundo (TB/s), pero los sistemas de Cloudflare pudieron detectarlo y mitigarlo.

A pesar de que las cifras generales indican un aumento en la duración global de los ataques, la mayoría de ellos fueron de corta duración como este, ya que solo duró dos minutos. Sin embargo, en términos más generales, hemos observado que los ataques de más de 3 horas aumentaron un 103 % en términos intertrimestrales.

Con este escenario, profundicemos en estos cambios que estamos observando en el panorama de los ataques DDoS.

La alianza hacktivista apodada "Darknet Parliament" amenaza a bancos occidentales y la red SWIFT

El 14 de junio, los grupos hacktivistas prorrusos Killnet, un resurgimiento de REvil y Anonymous Sudan anunciaron su unión para ejecutar ciberataques "masivos" contra el sistema financiero occidental, incluidos bancos europeos y estadounidenses, y el Sistema de la Reserva Federal de Estados Unidos. El colectivo, apodado "Darknet Parliament", declaró que su primer objetivo era paralizar la red SWIFT (Sociedad para las Telecomunicaciones Financieras Interbancarias Mundiales). Un ataque DDoS llevado a cabo con éxito contra el sistema SWIFT podría tener consecuencias nefastas, ya que es el principal servicio utilizado por las instituciones financieras para realizar transacciones mundiales.

Aparte de una serie de sucesos que se han hecho públicos, como la interrupción de Microsoft de la que se hicieron eco los medios de comunicación, no hemos observado ningún ataque DDoS novedoso ni interrupciones dirigidas a nuestros clientes. Nuestros sistemas han estado detectando y mitigando automáticamente los ataques asociados a esta campaña. En las últimas semanas, Darknet Parliament ha sido el autor de hasta 10 000 ataques DDoS contra sitios web protegidos por Cloudflare (véase el gráfico siguiente).

A pesar de las declaraciones formuladas por los hacktivistas, los sitios web de banca y servicios financieros solo fueron el noveno sector más afectado, según los ataques que hemos observado contra nuestros clientes en el marco de esta campaña.

Los principales blancos de ataque fueron los sectores de software informático, apuestas y casinos y videojuegos. El sector de las telecomunicaciones y los medios de comunicación ocuparon el cuarto y quinto lugar, respectivamente. En general, el mayor ataque que presenciamos en esta campaña alcanzó un máximo de 1,7 millones de solicitudes por segundo y la media fue de 65 000 de solicitudes por segundo.

Poniendo estas cifras en perspectiva, a principios de este año mitigamos el mayor ataque registrado en la historia, que alcanzó un pico de 71 millones de solicitudes por segundo. Por tanto, los ataques que hemos mencionado fueron muy pequeños en comparación con la escala de Cloudflare, pero no necesariamente para un sitio web medio. Por consiguiente, no debemos subestimar el potencial de daño en sitios web con una protección o configuración deficientes.

Ataques DDoS HTTP sofisticados

Un ataque DDoS HTTP es un ataque DDoS a través del protocolo de transferencia de hipertexto (HTTP). Se dirige a propiedades HTTP de Internet, como sitios web y puertas de enlace de API. En el último trimestre, los ataques DDoS HTTP se incrementaron un 15 % intertrimestral, a pesar de que descendieron un 35 % respecto al mismo periodo del año pasado.

Además, hemos observado un incremento alarmante de ataques DDoS HTTP sofisticados con un alto grado de aleatoriedad en los últimos meses. Parece como si los ciberdelincuentes que están detrás de estos ataques los hubieran diseñado intencionadamente para eludir los sistemas de mitigación, imitando de forma eficaz el comportamiento del navegador con mucha precisión. En algunos casos, presentan un alto grado de aleatoriedad en varias propiedades como los agentes de usuario y las huellas JA3, por nombrar algunas. A continuación, mostramos un ejemplo de un ataque de este tipo. Cada color representa una función de aleatoriedad distinta.

Por otra parte, en muchos de estos ataques, parece que los ciberdelincuentes intentan mantener la velocidad de ataque por segundo relativamente baja para tratar evitar la detección y ocultarse entre el tráfico legítimo.

Este nivel de sofisticación solía asociarse con ciberdelincuentes a nivel estatal y patrocinados por el Estado. Ahora parece que estas capacidades están al alcance de los ciberdelincuentes, que ya han dirigido sus ataques a empresas destacadas, como un gran proveedor de VoIP, una empresa líder en semiconductores y un importante proveedor de servicios de pago y tarjetas de crédito, entre otros.

La protección de los sitios web contra ataques DDoS HTTP sofisticados requiere una defensa inteligente, automatizada y rápida, que utilice la información sobre amenazas, la elaboración de perfiles de tráfico y el análisis estadístico/de aprendizaje automático para diferenciar entre los ataques de tráfico y el tráfico de los usuarios. Además, incluso el aumento del almacenamiento en caché, cuando proceda, puede ayudar a reducir el riesgo de que el tráfico de ataque afecte a tu servidor de origen. Consulta más información sobre las prácticas recomendadas de protección contra DDoS aquí.

Ataques DDoS de blanqueo de DNS

El sistema de nombres de dominio, o DNS, funciona como la guía telefónica de Internet. El DNS ayuda a traducir la dirección de un sitio web legible por humanos (p. ej., www.cloudflare.com) a una dirección IP legible para la máquina (p. ej., 104.16.124.96). Cuando los atacantes interrumpen los servidores DNS, afectan a la capacidad de las máquinas para conectarse a un sitio web, y al hacerlo impiden que los usuarios accedan a los sitios web.

En el último trimestre, los ataques DDoS a través del DNS representaron el vector de ataque más común. El 32 % de todos los ataques DDoS se produjeron a través del protocolo DNS. Entre ellos, uno de los ataques en auge más preocupantes es el ataque de blanqueo de DNS (DNS Laundering), que puede plantear graves problemas a las organizaciones que gestionan sus propios servidores DNS autoritativos.

El término "blanqueo" en el nombre del ataque "DNS Laundering" hace referencia a la analogía del blanqueo de dinero, el tortuoso proceso de hacer que las ganancias obtenidas ilegalmente, comúnmente conocidas como "dinero negro", parezcan legales. Del mismo modo, en el mundo de los ataques DDoS, un ataque de blanqueo de DNS es el proceso de hacer que el tráfico malicioso parezca tráfico legítimo, blanqueándolo a través de resolvedores de DNS recursivo de confianza.

En un ataque de blanqueo de DNS, el ciberdelincuente consultará subdominios de un dominio gestionado por el servidor DNS de la víctima. El prefijo que define el subdominio es aleatorio y nunca se utiliza más de una o dos veces en un ataque de este tipo. Debido al componente de aleatoriedad, los servidores DNS recursivos nunca tendrán una respuesta en caché y tendrán que reenviar la consulta al servidor DNS autoritativo de la víctima. Entonces, el servidor DNS autoritativo recibe tal bombardeo de consultas que no puede atender consultas legítimas, e incluso se bloquea por completo.

Desde el punto de vista de la protección, los administradores de DNS no pueden bloquear el origen del ataque porque este incluye servidores DNS recursivos de confianza, como el 8.8.8.8 de Google y el 1.1.1.1 de Clouflare. Los administradores tampoco pueden bloquear todas las consultas al dominio atacado porque es un dominio válido y quieren preservar el acceso a las consultas legítimas.

Los factores anteriores hacen que sea muy difícil distinguir las consultas legítimas de las malintencionadas. Una gran institución financiera asiática y un proveedor de DNS norteamericano son dos de las últimas víctimas de este tipo de ataques. A continuación, mostramos un ejemplo de un ataque de este tipo.

Al igual que las estrategias de protección descritas para las aplicaciones HTTP, la protección de los servidores DNS también requiere un enfoque preciso, rápido y automatizado. La utilización de un servicio DNS gestionado o un proxy inverso DNS como el de Cloudflare puede ayudar a absorber y mitigar los ataques de tráfico. Para los ataques DNS más sofisticados, se requiere una solución más inteligente que use el análisis estadístico de los datos históricos para poder diferenciar entre consultas legítimas y consultas de ataque.

El auge de las botnets en máquinas virtuales

Como hemos revelado anteriormente, estamos siendo testigos de una evolución en el ADN de las botnets. Ha llegado la era de las botnets DDoS en máquinas virtuales y, con ella, los ataques DDoS hipervolumétricos. Estas botnets se componen de máquinas virtuales (VM) o servidores privados virtuales (VPS) en lugar de dispositivos del Internet de las cosas (IoT), lo que multiplica por 5 000 su eficacia.

Debido a los recursos informáticos y de ancho de banda de que disponen estas botnets basadas en máquinas virtuales, son capaces de generar ataques hipervolumétricos con una flota mucho menor en comparación con las botnets basadas en dispositivos IoT.

Estas botnets han ejecutado uno de los mayores ataques DDoS registrados, incluido el ataque DDoS de 71 millones de solicitudes por segundo. Numerosas organizaciones, incluido un proveedor de plataformas de videojuegos líder del sector, ya han sido blanco de esta nueva generación de botnets.

Cloudflare ha colaborado proactivamente con destacados proveedores de informática en la nube para hacer frente estas nuevas botnets. Gracias a la intervención rápida y dedicada de estos proveedores, se han neutralizado componentes significativos de estas amenazas. Desde esta intervención, aún no hemos observado ningún otro ataque hipervolumétrico, lo que demuestra la eficacia de nuestra colaboración.

Si bien ya compartimos una alianza fructífera con la comunidad de la ciberseguridad para contrarrestar las botnets cuando identificamos ataques a gran escala, nuestro objetivo es agilizar y automatizar aún más este proceso. Extendemos una invitación a los proveedores de informática en la nube, proveedores de alojamiento y otros proveedores de servicios generales para que se unan a Botnet Threat Feed de Cloudflare de manera gratuita. Esta solución ofrece visibilidad de los ataques originados en sus redes, lo que contribuirá a nuestros esfuerzos comunes para desmantelar las botnets.

"Startblast": abuso de las vulnerabilidades de Mitel para lanzar ataques DDoS

En marzo de 2022, revelamos una vulnerabilidad de día cero (CVE-2022-26143), denominada TP240PhoneHome, que se identificó en el sistema de telefonía empresarial Mitel MiCollab y que expuso al sistema a ataques DDoS de amplificación UDP.

Esta vulnerabilidad funciona reflejando el tráfico de los servidores expuestos y es capaz de amplificar el tráfico de ataque en un factor de 220 000 millones por ciento. La vulnerabilidad se deriva de un puerto UDP no autenticado expuesto a la red pública de Internet, que podría permitir a ciberdelincuentes emitir un comando de depuración "startblast", simulando una avalancha de llamadas para probar el sistema.

Como resultado, por cada llamada de prueba, se envían dos paquetes UDP al emisor, lo que permite a un atacante dirigir este tráfico a cualquier dirección IP y número de puerto para amplificar un ataque DDoS. A pesar de la vulnerabilidad, solo unos pocos miles de estos dispositivos están expuestos, lo que limita la escala potencial del ataque. Además, los ataques se deben ejecutar en serie, lo que significa que cada dispositivo solo puede lanzar un ataque a la vez.

En general, en el último trimestre hemos observado otras amenazas emergentes como los ataques DDoS que abusan del protocolo TeamSpeak3. Este vector de ataque aumentó un asombroso 403 % este trimestre.

TeamSpeak, una aplicación patentada de protocolo de voz sobre Internet (VoIP), funciona sobre UDP para ayudar a los jugadores a hablar con otros jugadores en tiempo real. Hablar en lugar de solo chatear puede mejorar significativamente la eficacia de un equipo de jugadores y ayudarles a ganar. Grupos rivales pueden lanzar ataques DDoS contra servidores de TeamSpeak en un intento de interrumpir su vía de comunicación durante las partidas multijugador en tiempo real y afectar así al rendimiento de su equipo.

Puntos de acceso de los ataques DDoS: el origen de los ataques

En general, los ataques DDoS HTTP se alzaron un 15 % en términos intertrimestrales, pese a que disminuyeron un 35 % respecto al mismo periodo del año pasado. Además, los ataques DDoS a la capa de red se contrajeron un 14 % aproximadamente durante el trimestre en revisión.

En términos de volumen total de ataques de tráfico, EE. UU. fue el principal origen de ataques DDoS HTTP. Tres de cada mil solicitudes que observamos formaban parte de ataques DDoS HTTP originados en EE. UU. China ocupó el segundo lugar y Alemania el tercero.

Algunos países reciben de por sí más tráfico debido a diversos factores, como el tamaño del mercado, y por tanto más ataques. Por tanto, aunque es interesante comprender la cantidad total de ataques de tráfico originados en un país determinado, también es útil eliminar ese sesgo normalizando el ataque de tráfico por todo el tráfico dirigido a un país determinado.

Al hacerlo, observamos un patrón diferente. EE. UU. ni siquiera figura entre los diez primeros puestos. En su lugar, Mozambique, Egipto y Finlandia toman la delantera como los países donde se originó el mayor volumen de ataques de tráfico DDoS HTTP en relación con todo su tráfico. Casi una quinta parte de todo el tráfico HTTP procedente de direcciones IP de Mozambique formaba parte de ataques DDoS.

Si utilizamos la misma metodología de cálculo, pero para los bytes, observamos que Vietnam sigue siendo el principal origen de ataques DDoS a la capa de red (también conocidos como ataques DDoS a las capas 3 y 4) por segundo trimestre consecutivo, y la cantidad incluso aumentó un 58 % intertrimestral. Más del 41 % de todos los bytes que absorbieron los centros de datos de Cloudflare en Vietnam formaban parte de ataques DDoS a las capas 3 y 4.

Sectores blanco de ataques: análisis de los objetivos de los ataques DDoS

Cuando analizamos la actividad de ataques DDoS HTTP en el segundo trimestre, observamos que los sitios web de criptomonedas fueron blanco de la mayor cantidad de ataques de tráfico DDoS HTTP. Seis de cada diez mil solicitudes HTTP hacia sitios web de criptomonedas que confían en Cloudflare formaron parte de estos ataques. Esta cifra se ha disparado un 600 % en comparación con el trimestre anterior.

Por detrás de las criptomonedas, los sitios web de videojuegos y apuestas ocuparon el segundo lugar, cuyo porcentaje de ataque aumentó un 19 % respecto al trimestre anterior. Los sitios web de marketing y publicidad les siguieron de cerca en tercer lugar, si bien apenas hubo cambios en su porcentaje de ataques.

Sin embargo, si nos fijamos en la cantidad de ataques de tráfico en relación con todo el tráfico de un sector determinado, las cifras muestran un panorama diferente. El trimestre pasado, las organizaciones sin ánimo de lucro fueron las más afectadas. Los ataques DDoS HTTP representaron el 12 % del tráfico dirigido a estas organizaciones. Cloudflare protege a más de 2271 organizaciones sin ánimo de lucro en 111 países como parte del proyecto Galileo, que celebró su noveno aniversario este año . En los últimos meses, una media de 67,7 millones de ciberataques se dirigieron diariamente a este tipo de organizaciones.

En general, la cantidad de ataques DDoS contra organizaciones sin ánimo de lucro se alzó un 46 %, con lo que el porcentaje de ataques de tráfico alcanzó el 17,6 %. Sin embargo, a pesar de este crecimiento, el sector de la consultoría de gestión saltó al primer puesto teniendo en cuenta que los ataques DDoS representaron un 18,4 % de su tráfico.

Cuando descendemos por las capas del modelo OSI, observamos que las redes de Internet más afectadas pertenecían a los sectores de las tecnologías de la información y los servicios. Casi uno de cada tres bytes dirigidos a estos sectores formaban parte de ataques DDoS a las capas 3 y 4.

Sorprendentemente, las empresas del sector de la música fueron el segundo mayor blanco de ataques, seguidas por el sector de medios audiovisuales, y la industria aeronáutica y aeroespacial.

Principales sectores blanco de ataques: análisis desde una perspectiva regional

Los sitios web de criptomonedas experimentaron el mayor número de ataques en todo el mundo, mientras que el sector de consultoría de gestión y las organizaciones sin ánimo de lucro fueron los más afectados teniendo en cuenta su tráfico total. Sin embargo, si observamos las regiones individuales, la situación es un poco diferente.

África

El sector de las telecomunicaciones siguió siendo el más afectado por los ataques DDoS en África por segundo trimestre consecutivo. El sector de servicios bancarios, financieros y seguros (BFSI) ocupó el segundo lugar. La mayor parte de los ataques de tráfico se originó en Asia (35 %) y Europa (25 %).

Asia

Durante los dos últimos trimestres, el sector de los videojuegos y las apuestas fue el peor parado en Asia. En el 2º trimestre, sin embargo, bajó al segundo puesto y las criptomonedas encabezaron la lista como el sector más afectado (aproximadamente el 50 %). Una parte importante de los ataques de tráfico se originó en la propia Asia (30 %) y Norteamérica (30 %).

Europa

Por tercer trimestre consecutivo, el sector de los videojuegos sigue siendo el peor parado en Europa. Le siguieron de cerca los sectores de la hostelería y los medios audiovisuales, que ocuparon el segundo y el tercer puesto como sectores más afectados. La mayor parte de los ataques de tráfico se originó dentro de la propia Europa (40 %) y Asia (20 %).

Latinoamérica

Sorprendentemente, la mitad de los ataques de tráfico dirigidos a Latinoamérica tuvo como objetivo al sector de los artículos deportivos. En el trimestre anterior, el sector BFSI fue el principal blanco de ataques. Aproximadamente el 35 % de los ataques de tráfico procedieron de Asia, y otro 25% de Europa.

Oriente Medio

Los sectores de medios de comunicación y prensa fueron objeto del mayor número de ataques en Oriente Próximo. La gran mayoría de los ataques de tráfico se originaron en Europa (74 %).

Norteamérica

Por segundo trimestre consecutivo, las empresas de marketing y publicidad fueron las más afectadas en Norteamérica (aproximadamente el 35 %). Las empresas manufactureras y de software informático ocuparon el segundo y tercer lugar, respectivamente. Los principales orígenes de los ataques de tráfico fueron Europa (42 %) y EE. UU. (35 %).

Oceanía

Este trimestre, el sector de la biotecnología fue el que recibió el mayor número de ataques. Anteriormente, fue el sector de la salud y el bienestar. La mayor parte de los ataques de tráfico procedieron de Asia (38 %) y Europa (25 %).

Países y regiones blanco de ataques: análisis de los objetivos de los ataques DDoS

Si analizamos el volumen total de los ataques de tráfico, Israel encabezó la lista de los países más afectados el trimestre pasado. Este trimestre, los ataques dirigidos a sitios web israelíes disminuyeron un 33 %, bajando así a la cuarta posición. EE. UU. vuelve a tomar la delantera como país más afectado, seguido de Canadá y Singapur.

Si normalizamos los datos por países y regiones y dividimos el tráfico de ataque por el tráfico total, obtenemos una imagen diferente. Palestina saltó al primer puesto como país más afectado. Casi el 12 % de todo el tráfico a sitios web palestinos fueron ataques DDoS HTTP.

El trimestre pasado, observamos una sorprendente desviación en la capa de red, cuando las redes finlandesas que usan las soluciones de protección de Cloudflare fueron el objetivo principal de los ataques. Este aumento estuvo probablemente relacionado con las conversaciones diplomáticas que precipitaron la integración formal de Finlandia en la OTAN. Los ciberataques representaron aproximadamente el 83 % de todo el tráfico entrante a Finlandia, seguido de cerca por China, con un 68 % de ataques de tráfico.

Este trimestre, sin embargo, muestra un panorama muy diferente. Finlandia no estuvo en los diez primeros puestos, y las redes chinas protegidas por Cloudflare estuvieron a la cabeza. Casi dos tercios de los flujos de bytes hacia redes chinas protegidas por Cloudflare eran maliciosos. Suiza se coló en segunda posición, donde la mitad del tráfico entrante formó parte de ataques. Turquía ocupó el tercer lugar, donde una cuarta parte de su tráfico entrante se identificó como hostil.

Ataques DDoS de rescate

En ocasiones, los ataques DDoS se llevan a cabo para extorsionar el pago de rescates. Llevamos más de tres años encuestando a los clientes de Cloudflare y haciendo un seguimiento de los casos de ataques DDoS de rescate.

A diferencia de los ataques de ransomware, en los que las víctimas suelen caer en la trampa y descargan un archivo malicioso o hacen clic en un enlace de correo electrónico en riesgo que bloquea, elimina o filtra sus archivos hasta que se paga un rescate, los ataques DDoS de rescate pueden ser mucho más sencillos de ejecutar para los ciberdelincuentes. Los ataques DDoS de rescate no necesitan hacer uso de tácticas engañosas, como atraer a las víctimas para que abran correos electrónicos dudosos o hagan clic en enlaces fraudulentos, y tampoco necesitan aprovechar una brecha en la red ni acceder a los recursos corporativos.

En el último trimestre, se observó una disminución de las denuncias de ataques DDoS de rescate. Uno de cada diez encuestados declaró haber sufrido amenazas o ataques DDoS de rescate.

Conclusión: el panorama de las amenazas DDoS en constante evolución

En los últimos meses, se ha producido una escalada alarmante en la sofisticación de los ataques DDoS. Incluso los ataques más grandes y sofisticados que hemos observado pueden durar solo unos minutos o incluso segundos, lo que no da a un humano tiempo suficiente para responder. Antes incluso de que se envíe la alerta PagerDuty, el ataque puede haber terminado con consecuencias nefastas. La recuperación de un ataque DDoS puede durar mucho más que el propio ataque, igual que un boxeador puede necesitar un tiempo para recuperarse de un puñetazo en la cara que solo dura una fracción de segundo.

La seguridad no es un único producto ni se activa con solo hacer clic en un botón, es más bien un proceso que implica numerosas capas de protección para reducir el riesgo de impacto. Los sistemas automatizados de protección contra ataques DDoS de Cloudflare protegen sistemáticamente a nuestros clientes de los ataques DDoS, permitiéndoles centrarse en sus operaciones empresariales principales. Estos sistemas se complementan con la amplia gama de funciones de Cloudflare, como firewalls, detección de bots, protección de API e incluso almacenamiento en caché, que pueden contribuir a reducir el riesgo de impacto.

El panorama de las amenazas DDoS está evolucionando y es cada vez más complejo, lo que exige algo más que soluciones rápidas. Afortunadamente, con las soluciones de protección multicapa y protección DDoS automáticas de Cloudflare, nuestros clientes están bien preparados para afrontar estos retos con confianza. Nuestra misión es ayudar a mejorar Internet, por lo que seguimos en guardia para garantizar un mundo digital más seguro y fiable para todos.

Metodologías

Cómo calculamos las perspectivas de los ataques DDoS de rescate

Los sistemas de Cloudflare analizan constantemente el tráfico y aplican soluciones de mitigación de forma automática cuando se detectan ataques DDoS. Cada víctima de un ataque recibe una encuesta automatizada que nos ayuda a comprender mejor la naturaleza del ataque y el éxito de las medidas de mitigación. Durante más de dos años, hemos formulado las preguntas de esta encuesta a aquellos clientes que han sido víctimas de ataques. Una de ellas es si han recibido una amenaza o nota de rescate. En los dos últimos años, hemos recopilado una media de 164 respuestas por trimestre, que utilizamos para calcular el porcentaje de ataques DDoS de rescate.

Cómo calculamos la información geográfica y sectorial

País de origen
En la capa de aplicación, utilizamos las direcciones IP enemigas para conocer el país de origen de los ataques. Esto se debe a que, en esa capa, las direcciones IP no se pueden suplantar (es decir, alterar). Sin embargo, en la capa de red, las direcciones IP de origen sí se pueden suplantar. Así que, en lugar de basarnos en las direcciones IP para conocer el origen, utilizamos la ubicación de nuestros centros de datos donde se detectaron los paquetes de ataque. Podemos obtener precisión geográfica gracias a nuestra amplia cobertura global en más de 285 ubicaciones de todo el mundo.

País objetivo
Tanto para los ataques DDoS a la capa de aplicación como a la capa de red, agrupamos los ataques y el tráfico según el país de facturación de nuestros clientes. Esta metodología nos permite comprender qué países son objeto de más ataques.

Sector objetivo
Tanto para los ataques DDoS a la capa de aplicación como a la capa de red, agrupamos los ataques y el tráfico según el sector de nuestros clientes, de acuerdo con nuestro sistema de gestión de relaciones con los clientes. Esta metodología nos permite saber qué sectores son objeto de más ataques.

Volumen total vs. porcentaje
En cuanto a la información sobre los países de origen y países objetivo, observamos el volumen total de tráfico de ataque comparado con todo el tráfico como un punto de datos. Además, también observamos el porcentaje de ataques de tráfico hacia o desde un país concreto, a un país específico o a un sector determinado. Este análisis nos ofrece la "velocidad de la actividad de ataque" para un país/sector determinado, normalizada por sus niveles de tráfico total. De esta manera, eliminamos los sesgos de un país o sector que normalmente recibe mucho tráfico y, por tanto, también muchos ataques de tráfico.

Cómo calculamos las características de los ataques
Para calcular el tamaño, la duración y los vectores de ataque, así como las amenazas emergentes, agrupamos los ataques en categorías y luego establecemos la proporción de cada categoría respecto a la cantidad total de cada aspecto. En el nuevo elemento de Radar, estas tendencias se calculan en cambio por número de bytes. Como los ataques pueden variar mucho en número de bytes entre sí, esto podría dar lugar a tendencias diferentes entre los informes y el elemento de Radar.

Exención de responsabilidad y aclaración general

Cuando describimos los "principales países" como el origen o el objetivo de ataques, no significa necesariamente que ese país haya sido atacado como país, sino que las organizaciones que utilizan ese país como país de facturación fueron objeto de ataques. Del mismo modo, los ataques originados en un país no significan que ese país lanzara los ataques, sino que el ataque se lanzó desde direcciones IP que han sido asignadas a ese país. Los ciberdelincuentes operan botnets globales con nodos en todo el mundo, y en muchos casos también utilizan redes privadas virtuales y proxies para ocultar su verdadera ubicación. Así que, en todo caso, el país de origen podría indicar la presencia de nodos de salida o nodos de botnets dentro de ese país.

THG Podcast: Hot Dogs and Ketchup

2023-07-18 The History Guy: History Deserves to Be Remembered

Post Syndicated from The History Guy: History Deserves to Be Remembered original https://www.youtube.com/watch?v=vz0Udz-AGAY

Managing Risk Across Hybrid Environments with Executive Risk View

2023-07-18 Pauline Logan

Post Syndicated from Pauline Logan original https://blog.rapid7.com/2023/07/18/managing-risk-across-hybrid-environments-with-executive-risk-view/

Managing Risk Across Hybrid Environments with Executive Risk View

Over the last decade or so, organizations of all shapes and sizes across all industries have been going through a seismic shift in the way they engage with their customers and deliver their solutions to the market. These new delivery models are often underpinned by cloud services, which can change the composition of an organization’s IT environment drastically.

As part of this digital transformation, and in turn cloud adoption, many administrators have moved from maintaining a few hundred or so physical servers in their on-premises environment to running thousands and thousands of cloud instances spread across hundreds of cloud accounts—which are much more complex and ephemeral in nature.

The Modern Attack Surface is Expanding

Whether the impetus for this transformation is an attempt to maintain or gain a competitive advantage, or even as a result of mergers and acquisition, security teams are forced to play catch-up to harden a rapidly expanding attack surface. This expanding attack surface means that security teams need to evolve the scope and approach of their vulnerability management programs, and because they’re already playing catch-up, these teams are often asked to adapt their programs on the fly.

Making matters worse, many of the tools and processes used by teams to manage and secure those workloads aren’t able to keep up with the pace of innovation. Plus, many organizations have given DevOps teams self-service access to the underlying infrastructure that their teams need to innovate quickly, making it even more difficult for the security team to keep up with the ever-changing environment.

Adapting Your Vulnerability Management Program to the Cloud Requires a Different Approach

Assessing and reducing risk across on-premises and cloud environments can be complex and cumbersome, often requiring significant time and manual effort to aggregate, analyze and prioritize a plethora of risk signals. Practitioners are often forced to context switch between multiple tools and exert manual effort to normalize data and translate security findings into meaningful risk metrics that the business can understand. As a result, many teams struggle with blind spots resulting from gaps in data, or too much noise being surfaced without the ability to effectively prioritize remediation efforts and drive accountability across the organization. To effectively manage risk across complex and dynamic hybrid environments, security teams must adapt their programs and take a fundamentally different approach.

Managing Risk Across Hybrid Environments with Executive Risk View

As is the case with traditional on-premises environments, you need to first achieve and maintain full visibility of your environment. You also need to keep track of how the environment changes over time, and how that change impacts your risk posture. Doing this in an ephemeral environment can be tricky, because in the cloud things can (and will) change on a minute to minute basis. Traditional agent-based vulnerability management tools are too cumbersome to manage and simply won’t scale in the way modern environments require. Agentless solutions deliver the real-time visibility and change management capabilities that today’s cloud and hybrid environments require.

Once you establish real-time and continuous visibility, you need to assess your environment for risk, understanding your organization’s current risk posture. You’re going to need a way to effectively prioritize risk, and make sure your teams are focusing on the most pressing and impactful issues based on exploitability as well as potential impact to your business and customers.

Finally, once you’ve gotten to a point where you can identify which risk signals need your attention first, you’ll want to remediate them as quickly and comprehensively as possible. When you’re operating at the speed of the cloud, this means you’re likely going to be relying on some form of automation, whether that’s automating repetitive processes, or even having a security solution take action to remediate vulnerabilities on your behalf. Of course, you’ll need to be measuring and tracking progress throughout this process, and you’ll need a way to communicate the progress you and your team is making to improve your risk posture with trending analysis over time.

So, as you can see, it’s not that “what” security teams need to do is significantly different, but “how” they go about it has to change, because traditional approaches just won’t work. The challenge is that this isn’t an either/or scenario. Organizations that are operating in a hybrid environment need to adapt their programs to be able to manage and report on risk in on-premises and cloud environments simultaneously and holistically. If not, security leaders will struggle to make informed decisions on how to effectively plan their budgets and allocate resources to ensure that cloud migration doesn’t have a negative impact on its risk posture.

Manage Risk in Hybrid Environments with Executive Risk View

Executive Risk View, now generally available in Rapid7’s Cloud Risk Complete offering, provides security leaders with the comprehensive visibility and context needed to track total risk across both cloud and on-premises assets to better understand organizational risk posture and trends.

With Executive Risk View, customers can:

Achieve a complete view of risk across their hybrid environments to effectively communicate risk across the organization and track progress.
Establish a consistent definition of risk across their organization, aggregating insights and normalizing scores from on-premises and cloud assessments.
Take a data-driven approach to decision making, capacity planning and drive accountability for risk reduction across the entire business.

Sounds too good to be true? You can see it in action for yourself in a new guided product tour we recently posted on our website! In addition to taking the tour, you can find more information on Executive Risk View in the docs page.

Disabling Self-Driving Cars with a Traffic Cone

2023-07-18 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/07/disabling-self-driving-cars-with-a-traffic-cone.html

You can disable a self-driving car by putting a traffic cone on its hood:

The group got the idea for the conings by chance. The person claims a few of them walking together one night saw a cone on the hood of an AV, which appeared disabled. They weren’t sure at the time which came first; perhaps someone had placed the cone on the AV’s hood to signify it was disabled rather than the other way around. But, it gave them an idea, and when they tested it, they found that a cone on a hood renders the vehicles little more than a multi-ton hunk of useless metal. The group suspects the cone partially blocks the LIDAR detectors on the roof of the car, in much the same way that a human driver wouldn’t be able to safely drive with a cone on the hood. But there is no human inside to get out and simply remove the cone, so the car is stuck.

Delightfully low-tech.

How to Know Your Neighbors: How To Talk to People, Episode 5

2023-07-18 The Atlantic

Post Syndicated from The Atlantic original https://www.youtube.com/watch?v=UWK1OXu9GhY

Comic for 2023.07.18 – Edible Panties

2023-07-18 Explosm.net

Post Syndicated from Explosm.net original https://explosm.net/comics/edible-panties

New Cyanide and Happiness Comic

Optimize AWS Config for AWS Security Hub to effectively manage your cloud security posture

2023-07-17 Nicholas Jaeger

Post Syndicated from Nicholas Jaeger original https://aws.amazon.com/blogs/security/optimize-aws-config-for-aws-security-hub-to-effectively-manage-your-cloud-security-posture/

AWS Security Hub is a cloud security posture management service that performs security best practice checks, aggregates security findings from Amazon Web Services (AWS) and third-party security services, and enables automated remediation. Most of the checks Security Hub performs on AWS resources happen as soon as there is a configuration change, giving you nearly immediate visibility of non-compliant resources in your environment, compared to checks that run on a periodic basis. This near real-time finding and reporting of non-compliant resources helps you to quickly respond to infrastructure misconfigurations and reduce risk. Security Hub offers these continuous security checks through its integration with the AWS Config configuration recorder.

By default, AWS Config enables recording for more than 300 resource types in your account. Today, Security Hub has controls that cover approximately 60 of those resource types. If you’re using AWS Config only for Security Hub, you can optimize the configuration of the configuration recorder to track only the resources you need, helping to reduce the costs related to monitoring those resources in AWS Config and the amount of data produced, stored, and analyzed by AWS Config. This blog post walks you through how to set up and optimize the AWS Config recorder when it is used for controls in Security Hub.

Using AWS Config and Security Hub for continuous security checks

When you enable Security Hub, you’re alerted to first enable resource recording in AWS Config, as shown in Figure 1. AWS Config continually assesses, audits, and evaluates the configurations and relationships of your resources on AWS, on premises, and in other cloud environments. Security Hub uses this capability to perform change-initiated security checks. Security Hub checks that use periodic rules don’t depend on the AWS Config recorder. You must enable AWS Config resource recording for all the accounts and in all AWS Regions where you plan to enable Security Hub standards and controls. AWS Config charges for the configuration items that are recorded, separately from Security Hub.

Figure 1: Security Hub alerts you to first enable resource recording in AWS Config

When you get started with AWS Config, you’re prompted to set up the configuration recorder, as shown in Figure 2. AWS Config uses the configuration recorder to detect changes in your resource configurations and capture these changes as configuration items. Using the AWS Config configuration recorder not only allows for continuous security checks, it also minimizes the need to query for the configurations of the individual services, saving your service API quotas for other use cases. By default, the configuration recorder records the supported resources in the Region where the recorder is running.

Note: While AWS Config supports the configuration recording of more than 300 resource types, some Regions support only a subset of those resource types. To learn more, see Supported Resource Types and Resource Coverage by Region Availability.

Figure 2: Default AWS Config settings

Optimizing AWS Config for Security Hub

Recording global resources as well as current and future resources in AWS Config is more than what is necessary to enable Security Hub controls. If you’re using the configuration recorder only for Security Hub controls, and you want to cost optimize your use of AWS Config or reduce the amount of data produced, stored, and analyzed by AWS Config, you only need to record the configurations of approximately 60 resource types, as described in AWS Config resources required to generate control findings.

Set up AWS Config, optimized for Security Hub

We’ve created an AWS CloudFormation template that you can use to set up AWS Config to record only what’s needed for Security Hub. You can download the template from GitHub.

This template can be used in any Region that supports AWS Config (see AWS Services by Region). Although resource coverage varies by Region (Resource Coverage by Region Availability), you can still use this template in every Region. If a resource type is supported by AWS Config in at least one Region, you can enable the recording of that resource type in all Regions supported by AWS Config. For the Regions that don’t support the specified resource type, the recorder will be enabled but will not record any configuration items until AWS Config supports the resource type in the Region.

Security Hub regularly releases new controls that might rely on recording additional resource types in AWS Config. When you use this template, you can subscribe to Security Hub announcements with Amazon Simple Notification Service (SNS) to get information about newly released controls that might require you to update the resource types recorded by AWS Config (and listed in the CloudFormation template). The CloudFormation template receives periodic updates in GitHub, but you should validate that it’s up to date before using it. You can also use AWS CloudFormation StackSets to deploy, update, or delete the template across multiple accounts and Regions with a single operation. If you don’t enable the recording of all resources in AWS Config, the Security Hub control, Config.1 AWS Config should be enabled, will fail. If you take this approach, you have the option to disable the Config.1 Security Hub control or suppress its findings using the automation rules feature in Security Hub.

Customizing for your use cases

You can modify the CloudFormation template depending on your use cases for AWS Config and Security Hub. If your use case for AWS Config extends beyond your use of Security Hub controls, consider what additional resource types you will need to record the configurations of for your use case. For example, AWS Firewall Manager, AWS Backup, AWS Control Tower, AWS Marketplace, and AWS Trusted Advisor require AWS Config recording. Additionally, if you use other features of AWS Config, such as custom rules that depend on recording specific resource types, you can add these resource types in the CloudFormation script. You can see the results of AWS Config rule evaluations as findings in Security Hub.

Another customization example is related to the AWS Config configuration timeline. By default, resources evaluated by Security Hub controls include links to the associated AWS Config rule and configuration timeline in AWS Config for that resource, as shown in Figure 3.

Figure 3: Link from Security Hub control to the configuration timeline for the resource in AWS Config

The AWS Config configuration timeline, as illustrated in Figure 4, shows you the history of compliance changes for the resource, but it requires the AWS::Config::ResourceCompliance resource type to be recorded. If you need to track changes in compliance for resources and use the configuration timeline in AWS Config, you must add the AWS::Config::ResourceCompliance resource type to the CloudFormation template provided in the preceding section. In this case, Security Hub may change the compliance of the Security Hub managed AWS Config rules, which are recorded as configuration items for the AWS::Config::ResourceCompliance resource type, incurring additional AWS Config recorder charges.

Figure 4: Config resource timeline

Summary

You can use the CloudFormation template provided in this post to optimize the AWS Config configuration recorder for Security Hub to reduce your AWS Config costs and to reduce the amount of data produced, stored, and analyzed by AWS Config. Alternatively, you can run AWS Config with the default settings or use the AWS Config console or scripts to further customize your configuration to fit your use case. Visit Getting started with AWS Security Hub to learn more about managing your security alerts.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

Build a serverless log analytics pipeline using Amazon OpenSearch Ingestion with managed Amazon OpenSearch Service

2023-07-17 Hajer Bouafif

Post Syndicated from Hajer Bouafif original https://aws.amazon.com/blogs/big-data/build-a-serverless-log-analytics-pipeline-using-amazon-opensearch-ingestion-with-managed-amazon-opensearch-service/

In this post, we show how to build a log ingestion pipeline using the new Amazon OpenSearch Ingestion, a fully managed data collector that delivers real-time log and trace data to Amazon OpenSearch Service domains. OpenSearch Ingestion is powered by the open-source data collector Data Prepper. Data Prepper is part of the open-source OpenSearch project. With OpenSearch Ingestion, you can filter, enrich, transform, and deliver your data for downstream analysis and visualization. OpenSearch Ingestion is serverless, so you don’t need to worry about scaling your infrastructure, operating your ingestion fleet, and patching or updating the software.

For a comprehensive overview of OpenSearch Ingestion, visit Amazon OpenSearch Ingestion, and for more information about the Data Prepper open-source project, visit Data Prepper.

In this post, we explore the logging infrastructure for a fictitious company, AnyCompany. We explore the components of the end-to-end solution and then show how to configure OpenSearch Ingestion’s main parameters and how the logs come in and out of OpenSearch Ingestion.

Solution overview

Consider a scenario in which AnyCompany collects Apache web logs. They use OpenSearch Service to monitor web access and identify possible root causes to error logs of type 4xx and 5xx. The following architecture diagram outlines the use of every component used in the log analytics pipeline: Fluent Bit collects and forwards logs; OpenSearch Ingestion processes, routes, and ingests logs; and OpenSearch Service analyzes the logs.

The workflow contains the following stages:

Generate and collect – Fluent Bit collects the generated logs and forwards them to OpenSearch Ingestion. In this post, you create fake logs that Fluent Bit forwards to OpenSearch Ingestion. Check the list of supported clients to review the required configuration for each client supported by OpenSearch Ingestion.
Process and ingest – OpenSearch Ingestion filters the logs based on response value, processes the logs using a grok processor, and applies conditional routing to ingest the error logs to an OpenSearch Service index.
Store and analyze – We can analyze the Apache httpd error logs using OpenSearch Dashboards.

Prerequisites

To implement this solution, make sure you have the following prerequisites:

An AWS account
An active AWS Cloud9 environment on an Amazon Elastic Compute Cloud (Amazon EC2) instance
Pre-installed Docker compose in the AWS Cloud9 environment
AWS managed temporary credentials disabled in AWS Cloud9
An active OpenSearch Service domain

Configure OpenSearch Ingestion

First, you define the appropriate AWS Identity and Access Management (IAM) permissions to write to and from OpenSearch Ingestion. Then you set up the pipeline configuration in the OpenSearch Ingestion. Let’s explore each step in more detail.

Configure IAM permissions

OpenSearch Ingestion works with IAM to secure communications into and out of OpenSearch Ingestion. You need two roles, authenticated using AWS Signature V4 (SigV4) signed requests. The originating entity requires permissions to write to OpenSearch Ingestion. OpenSearch Ingestion requires permissions to write to your OpenSearch Service domain. Finally, you must create an access policy using OpenSearch Service’s fine-grained access control, which allows OpenSearch Ingestion to create indexes and write to them in your domain.

The following diagram illustrates the IAM permissions to allow OpenSearch Ingestion to write to an OpenSearch Service domain. Refer to Setting up roles and users in Amazon OpenSearch Ingestion to get more details on roles and permissions required to use OpenSearch Ingestion.

In the demo, you use the AWS Cloud9 EC2 instance profile’s credentials to sign requests sent to OpenSearch Ingestion. You use Fluent Bit to fetch the credentials and assume the role you pass in the aws_role_arn you configure later.

Create an ingestion role (called IngestionRole) to allow Fluent Bit to ingest the logs into your pipeline.

Create a trust relationship to allow Fluent Bit to assume the ingestion role, as shown in the following code. Fluent Bit attempts to fetch the credentials in the following order. In configuring the access policy for this role, you grant permission for the osis:Ingest.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "{your-account-id}"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Create a pipeline role (called PipelineRole) with a trust relationship for OpenSearch Ingestion to assume that role. The domain-level access policy of the OpenSearch domain grants the pipeline role access to the domain.

Finally, configure your domain’s security plugin to enable OpenSearch Ingestion’s assumed role to create indexes and write data to the domain.

In this demo, the OpenSearch Service domain uses fine-grained access control for authentication, so you need to map the OpenSearch Ingestion pipeline role to the OpenSearch backend role all_access. For instructions, refer to Step 2: Include the pipeline role in the domain access policy page.

Create the pipeline in OpenSearch Ingestion

To create an OpenSearch Ingestion pipeline, complete the following steps:

On the OpenSearch Service console, choose Pipelines in the navigation pane.
Choose Create pipeline.
For Pipeline name, enter a name.

Input the minimum and maximum Ingestion OpenSearch Compute Units (Ingestion OCUs). In this example, we use the default pipeline capacity settings of minimum 1 Ingestion OCU and maximum 4 Ingestion OCUs.

Each OCU is a combination of approximately 8 GB of memory and 2 vCPUs that can handle an estimated 8 GiB per hour. OpenSearch Ingestion supports up to 96 OCUs, and it automatically scales up and down based on your ingest workload demand.

In the Pipeline configuration section, configure Data Prepper to process your data by choosing the appropriate blueprint configuration template on the Configuration blueprints menu. For this post, we choose AWS-LogAggregationWithConditionalRouting.

The OpenSearch Ingestion pipeline configuration consists of four sections:

Source – This is the input component of a pipeline. It defines the mechanism through which a pipeline consumes records. In this post, you use the http_source plugin and provide the Fluent Bit output URI value within the path attribute.
Processors – This represents an intermediate processing to filter, transform, and enrich your input data. Refer to Supported plugins for more details on the list of operations supported in OpenSearch Ingestion. In this post, we use the grok processor COMMONAPACHELOG, which matches input logs against the common Apache log pattern and makes it easy to query in OpenSearch Service.
Sink – This is the output component of a pipeline. It defines one or more destinations to which a pipeline publishes records. In this post, you define an OpenSearch Service domain and index as sink.
Route – This is the part of a processor that allows the pipeline to route the data into different sinks based on specific conditions. In this example, you create four routes based in the response field value of the log. If the response field value of the log line matches 2xx or 3xx, the log is sent to the OpenSearch Service index aggregated_2xx_3xx. If the response field value matches 4xx, the log is sent to the index aggregated_4xx. If the response field value matches 5xx, the log is sent to the index aggregated_5xx.

Update the blueprint based on your use case. The following code shows an example of the pipeline configuration YAML file:

version: "2"
log-aggregate-pipeline:
  source:
    http:
      # Provide the FluentBit output URI value.
      path: "/log/ingest"
  processor:
    - date:
        from_time_received: true
        destination: "@timestamp"
    - grok:
        match:
          log: [ "%{COMMONAPACHELOG_DATATYPED}" ]
  route:
    - 2xx_status: "/response >= 200 and /response < 300"
    - 3xx_status: "/response >= 300 and /response < 400"
    - 4xx_status: "/response >= 400 and /response < 500"
    - 5xx_status: "/response >= 500 and /response < 600"
  sink:
    - opensearch:
        # Provide an AWS OpenSearch Service domain endpoint
        hosts: [ "{your-domain-endpoint}" ]
        aws:
          # Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com
          sts_role_arn: "arn:aws:iam::{your-account-id}:role/PipelineRole"
          # Provide the region of the domain.
          region: "{AWS_Region}"
        index: "aggregated_2xx_3xx"
        routes:
          - 2xx_status
          - 3xx_status
    - opensearch:
        # Provide an AWS OpenSearch Service domain endpoint
        hosts: [ "{your-domain-endpoint}"  ]
        aws:
          # Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com
          sts_role_arn: "arn:aws:iam::{your-account-id}:role/PipelineRole"
          # Provide the region of the domain.
          region: "{AWS_Region}"
        index: "aggregated_4xx"
        routes:
          - 4xx_status
    - opensearch:
        # Provide an AWS OpenSearch Service domain endpoint
        hosts: [ "{your-domain-endpoint}"  ]
        aws:
          # Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com
          sts_role_arn: "arn:aws:iam::{your-account-id}:role/PipelineRole"
          # Provide the region of the domain.
          region: "{AWS_Region}"
        index: "aggregated_5xx"
        routes:
          - 5xx_status

Provide the relevant values for your domain endpoint, account ID, and Region related to your configuration.

Check the health of your configuration setup by choosing Validate pipeline when you finish the update.

When designing a production workload, deploy your pipeline within a VPC. For instructions, refer to Securing Amazon OpenSearch Ingestion pipelines within a VPC.

For this post, select Public access under Network.

In the Log publishing options section, select Publish to CloudWatch logs and Create new group.

OpenSearch Ingestion uses the log levels of INFO, WARN, ERROR, and FATAL. Enabling log publishing helps you monitor your pipelines in production.

Choose Next and Create pipeline.
Select the pipeline and choose View details to see the progress of the pipeline creation.

Wait until the status changes to Active to start using the pipeline.

Send logs to the OpenSearch Ingestion pipeline

To start sending logs to the OpenSearch Ingestion pipeline, complete the following steps:

On the AWS Cloud9 console, create a Fluent Bit configuration file and update the following attributes:
- Host – Enter the ingestion URL of your OpenSearch Ingestion pipeline.
- aws_service – Enter osis.
- aws_role_arn – Enter the ARN of the IAM role IngestionRole.

The following code shows an example of the Fluent-bit.conf file:

[SERVICE]
    parsers_file          ./parsers.conf
    
[INPUT]
    name                  tail
    refresh_interval      5
    path                  /var/log/*.log
    read_from_head        true
[FILTER]
    Name parser
    Key_Name log
    Parser apache
[OUTPUT]
    Name http
    Match *
    Host {Ingestion URL}
    Port 443
    URI /log/ingest
    format json
    aws_auth true
    aws_region {AWS_region}
    aws_role_arn arn:aws:iam::{your-account-id}:role/IngestionRole
    aws_service osis
    Log_Level trace
    tls On

In the AWS Cloud9 environment, create a docker-compose YAML file to deploy Fluent Bit and Flog containers:

version: '3'
services:
  fluent-bit:
    container_name: fluent-bit
    image: docker.io/amazon/aws-for-fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - ./apache-logs:/var/log
  flog:
    container_name: flog
    image: mingrammer/flog
    command: flog -t log -f apache_common -o web/log/test.log -w -n 100000 -d 1ms -p 1000
    volumes:
      - ./apache-logs:/web/log

Before you start the Docker containers, you need to update the IAM EC2 instance role in AWS Cloud9 so it can sign the requests sent to OpenSearch Ingestion.

For demo purposes, create an IAM service-linked role and choose EC2 under Use case to allow the AWS Cloud9 EC2 instance to call OpenSearch Ingestion on your behalf.
Add the OpenSearch Ingestion policy, which is the same policy you used with IngestionRole.
Add the AdministratorAccess permission policy to the role as well.

Your role definition should look like the following screenshot.

After you create the role, go back to AWS Cloud9, select your demo environment, and choose View details.
On the EC2 instance tab, choose Manage EC2 instance to view the details of the EC2 instance attached to your AWS Cloud9 environment.

On the Amazon EC2 console, replace the IAM role of your AWS Cloud9 EC2 instance with the new role.
Open a terminal in AWS Cloud9 and run the command docker-compose up.

Check the output in the terminal—if everything is working correctly, you get status 200.

Fluent Bit collects logs from the /var/log repository in the container and pushes the data to the OpenSearch Ingestion pipeline.

Open OpenSearch Dashboards, navigate to Dev Tools, and run the command GET _cat/indices to validate that the data has been delivered by OpenSearch Ingestion to your OpenSearch Service domain.

You should see the three indexes created: aggregated_2xx_3xx, aggregated_4xx, and aggregated_5xx.

Now you can focus on analyzing your log data and reinvent your business without having to worry about any operational overhead regarding your ingestion pipeline.

Best practices for monitoring

You can monitor the Amazon CloudWatch metrics made available to you to maintain the right performance and availability of your pipeline. Check the list of available pipeline metrics related to the source, buffer, processor, and sink plugins.

Navigate to the Metrics tab for your specific OpenSearch Ingestion pipeline to explore the graphs available to each metric, as shown in the following screenshot.

In your production workloads, make sure to configure the following CloudWatch alarms to notify you when the pipeline metrics breach a specific threshold so you can promptly remediate each issue.

Managing cost

While OpenSearch Ingestion automatically provisions and scales the OCUs for your spiky workloads, you only pay for the compute resources actively used by your pipeline to ingest, process, and route data. Therefore, setting up a maximum capacity of Ingestion OCUs allows you to handle your workload peak demand while controlling cost.

For production workloads, make sure to configure a minimum of 2 Ingestion OCUs to ensure 99.9% availability for the ingestion pipeline. Check the sizing recommendations and learn how OpenSearch Ingestion responds to workload spikes.

Clean up

Make sure you clean up unwanted AWS resources created during this post in order to prevent additional billing for these resources. Follow these steps to clean up your AWS account:

On the AWS Cloud9 console, choose Environments in the navigation pane.
Select the environment you want to delete and choose Delete.
On the OpenSearch Service console, choose Domains under Managed clusters in the navigation pane.
Select the domain you want to delete and choose Delete.
Select Pipelines under Ingestion in the navigation pane.
Select the pipeline you want to delete and on the Actions menu, choose Delete.

Conclusion

In this post, you learned how to create a serverless ingestion pipeline to deliver Apache access logs to an OpenSearch Service domain using OpenSearch Ingestion. You learned the IAM permissions required to start using OpenSearch Ingestion and how to use a pipeline blueprint instead of creating a pipeline configuration from scratch.

You used Fluent Bit to collect and forward Apache logs, and used OpenSearch Ingestion to process and conditionally route the log data to different indexes in OpenSearch Service. For more examples about writing to OpenSearch Ingestion pipelines, refer to Sending data to Amazon OpenSearch Ingestion pipelines.

Finally, the post provided you with recommendations and best practices to deploy OpenSearch Ingestion pipelines in a production environment while controlling cost.

Follow this post to build your serverless log analytics pipeline, and refer to Top strategies for high volume tracing with Amazon OpenSearch Ingestion to learn more about high volume tracing with OpenSearch Ingestion.

About the authors

Hajer Bouafif is an Analytics Specialist Solutions Architect at Amazon Web Services. She focuses on OpenSearch Service and helps customers design and build well-architected analytics workloads in diverse industries. Hajer enjoys spending time outdoors and discovering new cultures.

Francisco Losada is an Analytics Specialist Solutions Architect based out of Madrid, Spain. He works with customers across EMEA to architect, implement, and evolve analytics solutions at AWS. He advocates for OpenSearch, the open-source search and analytics suite, and supports the community by sharing code samples, writing content, and speaking at conferences. In his spare time, Francisco enjoys playing tennis and running.

Muthu Pitchaimani is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search applications and solutions. Muthu is interested in the topics of networking and security, and is based out of Austin, Texas.

Active Exploitation of Multiple Adobe ColdFusion Vulnerabilities

2023-07-17 Caitlin Condon

Post Syndicated from Caitlin Condon original https://blog.rapid7.com/2023/07/17/etr-active-exploitation-of-multiple-adobe-coldfusion-vulnerabilities/

Active Exploitation of Multiple Adobe ColdFusion Vulnerabilities

Rapid7 managed services teams have observed exploitation of Adobe ColdFusion in multiple customer environments. The attacks our team has responded to thus far appear to be chaining CVE-2023-29298, a Rapid7-discovered access control bypass in ColdFusion that was disclosed on July 11, with an additional vulnerability. The behavior our teams are observing appears to be consistent with a zero-day exploit published (and then subsequently taken down) by Project Discovery circa July 12.

Background

On Tuesday, July 11, Adobe released fixes for several vulnerabilities affecting ColdFusion, including a Rapid7-discovered access control bypass vulnerability (CVE-2023-29298) that we disclosed in coordination with the vendor. On July 13, Rapid7 managed services teams began observing exploitation of Adobe ColdFusion in multiple customer environments. Based on available evidence, threat actors appear to be exploiting CVE-2023-29298 in conjunction with a secondary vulnerability. The behavior our teams are observing appears to be consistent with CVE-2023-38203, which was published and then subsequently taken down by Project Discovery circa July 12.

It’s highly likely that Project Discovery thought they were publishing an n-day exploit for CVE-2023-29300 in their July 12 blog post. Adobe published a fix for CVE-2023-29300, which is a deserialization vulnerability that allows for arbitrary code execution, on July 11. In actuality, what Project Discovery had detailed was a new zero-day exploit chain that Adobe fixed in an out-of-band update on July 14.

The patch for CVE-2023-29300 implements a denylist of classes that cannot be deserialized by the Web Distributed Data eXchange (WDDX) data that forms part of some requests to ColdFusion. Adobe is likely unable to remove this WDDX functionality completely, as that would break all the things that rely on it, so instead of prohibiting deserialization of WDDX data, they implement a denylist of Java class paths that cannot be deserialized (so an attacker cannot specify a deserialization gadget located in these class paths).

The Project Discovery authors evidently figured out a gadget that worked (i.e., a class that is not on Adobe’s denylist and can be used as a deserialization gadget to achieve remote code execution) based on the class com.sun.rowset.JdbcRowSetImpl. The Project Discovery team probably did not realize their discovery was a new zero-day vulnerability and (we assume) took down their blog while Adobe fixed the flaw. On Friday July 14, Adobe published an out-of-band patch for CVE-2023-38203 — a new deserialization vulnerability. The only thing this patch does is add the class path !com.sun.rowset.** to the denylist, breaking the exploit Project Discovery had published on July 12.

Incomplete fix for CVE-2023-29298

Rapid7 researchers determined earlier today that the fix Adobe provided for CVE-2023-29298 on July 11 is incomplete, and that a trivially modified exploit still works against the latest version of ColdFusion (released July 14). We have notified Adobe that their patch is incomplete. There is currently no mitigation for CVE-2023-29298, but the exploit chain Rapid7 is observing in the wild relies on a secondary vulnerability for full execution on target systems. Therefore, updating to the latest available version of ColdFusion that fixes CVE-2023-38203 should still prevent the attacker behavior our MDR team is observing.

Affected Products

The following versions of ColdFusion are vulnerable to both CVE-2023-29298 and CVE-2023-38203:

Adobe ColdFusion 2023 Update 1
Adobe ColdFusion 2021 Update 7 and below
Adobe ColdFusion 2018 Update 17 and below

The latest versions of ColdFusion are below, and contain the July 14 out-of-band patch for CVE-2023-38203. Note that these are still vulnerable to CVE-2023-29298:

Adobe ColdFusion 2023 Update 2
Adobe ColdFusion 2021 Update 8
Adobe ColdFusion 2018 Update 18

Observed attacker behavior

Rapid7 has observed POST requests (see example below) in IIS logs that were sent to file accessmanager.cfc in order to leverage this exploit.

Active Exploitation of Multiple Adobe ColdFusion Vulnerabilities

The attackers then executed encoded PowerShell commands on an endpoint in order to create a webshell to gain access to the endpoint. The webshell is typically observed in \wwwroot\CFIDE directory: .\ColdFusion11\cfusion\wwwroot\CFIDE\ckeditr.cfm

Additionally, Rapid7 observed cURL commands to the following Burpsuite URL, along with nltest /domain_trusts related activity in order to query the domain controller. : hXXp://rlgt1hin2gdk2p3teyhuetitrkxblg95.oastify[.]com

IOCs

IP addresses:
62.233.50[.]13
5.182.36[.]4
195.58.48[.]155

Domains:

oastify[.]com
ckeditr[.]cfm (SHA256 08D2D815FF070B13A9F3B670B2132989C349623DB2DE154CE43989BB4BBB2FB1)

Mitigation guidance

Adobe ColdFusion customers should immediately update to the latest version of ColdFusion and block domain oastify[.]com. As of July 17, the latest versions of ColdFusion are in APSB23-41 here.

Adobe’s July 14 advisory also explicitly notes the following, which ColdFusion customers may want to consider implementing in addition to applying the latest updates:

“Note: If you become aware of any package with a deserialization vulnerability in the future, use the serialfilter.txt file in <cfhome>/lib to denylist the package (eg: !org.jroup.**;).”

Rapid7 customers

InsightVM and Nexpose customers can assess their exposure to CVE-2023-29298 and CVE-2023-38203 with a vulnerability check available in the July 17 content release. Note that the previous vulnerability check for CVE-2023-29298 has been updated to reflect that the fix is incomplete.

InsightIDR and Managed Detection & Response customers have existing detection coverage through Rapid7’s expansive library of detection rules. The following detection rules are deployed and alerting on post-exploitation activity related to this vulnerability:

Webshell – Possible ColdFusion Webshell In Command Line (deployed: March, 2023)
Attacker Tool – PowerShell -noni -ep -nop Flags (deployed: August, 2019)
Attacker Technique – PowerShell Download Cradles (deployed: January, 2019)
PowerShell – Obfuscated Script (deployed: March, 2018)
Suspicious Process – Burpsuite Related Domain in Command Line (deployed: October 2020)

Managed Detection & Response customers please note: If the Rapid7 MDR team detects suspicious activity in your environment, your Customer Advisor will reach out to you directly.

AWS Week in Review – Updates on Amazon FSx for NetApp ONTAP, AWS Lambda, eksctl, Karpetner, and More – July 17, 2023

2023-07-17 Channy Yun

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/aws-week-in-review-updates-on-amazon-fsx-for-netapp-ontap-aws-lambda-eksctl-karpetner-and-more-july-17-2023/

The Data Centered: Eastern Oregon, a five-part mini-documentary series looking at the real-life impact of the more than $15 billion investment AWS has made in the local community, and how the company supports jobs, generates economic growth, provides skills training and education, and unlocks opportunities for local businesses suppliers.

Last week, I watched a new episode introducing the Data Center Technician training program offered by AWS to train people with little or no previous technical experience in the skills they need to work in data centers and other information technology (IT) roles. This video reminded me of my first days of cabling and transporting servers in data centers. Remember, there are still people behind cloud computing.

Last Week’s Launches
Here are some launches that got my attention:

Amazon FSx for NetApp ONTAP Updates – Jeff Barr introduced Amazon FSx for NetApp ONTAP support for SnapLock, an ONTAP feature that gives you the power to create volumes that provide write once read many (WORM) functionality for regulatory compliance and ransomware protection. In addition, FSx for NetApp ONTAP now supports IPSec encryption of data in transit and two additional monitoring and troubleshooting capabilities that you can use to monitor file system events and diagnose network connectivity.

AWS Lambda detects and stops recursive loops in Lambda functions – In certain scenarios, due to resource misconfiguration or code defects, a processed event might be sent back to the same service or resource that invoked the Lambda function. This can cause an unintended recursive loop and result in unintended usage and costs for customers. With this launch, Lambda will stop recursive invocations between Amazon SQS, Lambda, and Amazon SNS after 16 recursive calls. For more information, refer to our documentation or the launch blog post.

Email notification

Amazon CloudFront supports for 3072-bit RSA certificates – You can now associate their 3072-bit RSA certificates with CloudFront distributions to enhance communication security between clients and CloudFront edge locations. To get started, associate a 3072-bit RSA certificate with your CloudFront distribution using console or APIs. There are no additional fees associated with this feature. For more information, please refer to the CloudFront Developer Guide.

Running GitHub Actions with AWS CodeBuild – Two weeks ago, AWS CodeBuild started to support GitHub Actions. You can now define GitHub Actions steps directly in the BuildSpec and run them alongside CodeBuild commands. Last week, the AWS DevOps Blog published the blog post about using the Liquibase GitHub Action for deploying changes to an Amazon Aurora database in a private subnet. You can learn how to integrate AWS CodeBuild and nearly 20,000 GitHub Actions developed by the open source community.

CodeBuild configuration showing the GitHub repository URL

Amazon DynamoDB local version 2.0 – You can develop and test applications by running Amazon DynamoDB local in your local development environment without incurring any additional costs. The new 2.0 version allows Java developers to use DynamoDB local to work with Spring Boot 3 and frameworks such as Spring Framework 6 and Micronaut Framework 4 to build modernized, simplified, and lightweight cloud-native applications.

For a full list of AWS announcements, be sure to keep an eye on the What’s New at AWS page.

Open Source Updates
Last week, we introduced new open source projects and significant roadmap contributions to the Jupyter community.

New joint maintainership between Weaveworks and AWS for eksctl – Now the eksctl open source project has been moved from the Weaveworks GitHub organization to a new top level GitHub organization—eksctl-io—that will be jointly maintained by Weaveworks and AWS moving forward. The eksctl project can now be found on GitHub.

Karpenter now supports Windows containers – Karpenter is an open source flexible, high-performance Kubernetes node provisioning and management solution that you can use to quickly scale Amazon EKS clusters. With the launch of version 0.29.0, Karpenter extends the automated node provisioning support to Windows containers running on EKS. Read this blog post for a step-by-step guide on how to get started with Karpenter for Windows node groups.

Updates in Amazon Aurora and Amazon OpenSearch Service – Following the announcement of updates to the PostgreSQL database in May by the open source community, we’ve updated Amazon Aurora PostgreSQL-Compatible Edition to support PostgreSQL 15.3, 14.8, 13.11, 12.15, and 11.20. These releases contain product improvements and bug fixes made by the PostgreSQL community, along with Aurora-specific improvements. You can also run OpenSearch version 2.7 in Amazon OpenSearch Service. With OpenSearch 2.7 (also released in May), we’ve made several improvements to observability, security analytics, index management, and geospatial capabilities in OpenSearch Service.

To learn about weekly updates for open source at AWS, check out the latest AWS open source newsletter by Ricardo.

Upcoming AWS Events
Check your calendars and sign up for these AWS events:

AWS Storage Day on August 9 – Join a one-day virtual event that will help you to better understand AWS storage services and make the most of your data. Register today.

AWS Global Summits – Sign up for the AWS Summit closest to your city: Hong Kong (July 20), New York City (July 26), Taiwan (August 2-3), São Paulo (August 3), and Mexico City (August 30).

AWS Community Days – Join a community-led conference run by AWS user group leaders in your region: Malaysia (July 22), Philippines (July 29-30), Colombia (August 12), and West Africa (August 19).

AWS re:Invent 2023 – Join us to hear the latest from AWS, learn from experts, and connect with the global cloud community. Registration is now open.

You can browse all upcoming AWS-led in-person and virtual events, and developer-focused events such as AWS DevDay.

Take the AWS Blog Customer Survey
We’re focused on improving our content to provide a better customer experience, and we need your feedback to do so. Take our survey to share insights regarding your experience on the AWS Blog.

This survey is hosted by an external company. AWS handles your information as described in the AWS Privacy Notice. AWS will own the data gathered via this survey and will not share the information collected with survey respondents.

That’s all for this week. Check back next Monday for another Week in Review!

— Channy

This post is part of our Week in Review series. Check back each week for a quick roundup of interesting news and announcements from AWS!

AWS Fargate Enables Faster Container Startup using Seekable OCI

2023-07-17 Donnie Prakoso

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/aws-fargate-enables-faster-container-startup-using-seekable-oci/

While developing with containers is becoming an increasingly popular way for deploying and scaling applications, there are still areas where improvements can be made. One of the main issues with scaling containerized applications is the long startup time, especially during scale up when newer instances need to be added. This issue can have a negative impact on the customer experience, for example when a website needs to scale out to serve additional traffic.

A research paper shows that container image downloads account for 76 percent of container startup time, but on average only 6.4 percent of the data is needed for the container to start doing useful work. Starting and scaling out containerized applications requires downloading container images from a remote container registry. This may introduce a non-trivial latency, as the entire image must be downloaded and unpacked before the applications can be started.

One solution to this problem is lazy loading (also known as asynchronous loading) container images. This approach downloads data from the container registry in parallel with the application startup, such as stargz-snapshotter, a project that aims to improve the overall container start time.

Last year, we introduced Seekable OCI (SOCI), a technology open sourced by Amazon Web Services (AWS) that enables container runtimes to implement lazy loading the container image to start applications faster without modifying the container images. As part of that effort, we open sourced SOCI Snapshotter, a snapshotter plugin that enables lazy loading with SOCI in containerd.

AWS Fargate Support for SOCI
Today, I’m excited to share that AWS Fargate now supports Seekable OCI (SOCI), which helps applications deploy and scale out faster by enabling containers to start without waiting to download the entire container image. At launch, this new capability is available for Amazon Elastic Container Service (Amazon ECS) applications running on AWS Fargate.

Here’s a quick look to show how AWS Fargate support for SOCI works:

SOCI works by creating an index (SOCI index) of the files within an existing container image. This index is a key enabler to launching containers faster, providing the capability to extract an individual file from a container image without having to download the entire image. Your applications no longer need to wait to complete pulling and unpacking a container image before your applications start running. This allows you to deploy and scale out applications more quickly and reduce the rollout time for application updates.

A SOCI index is generated and stored separately from the container images. This means that your container images don’t need to be converted to use SOCI, therefore not breaking secure hash algorithm (SHA)-based security, such as container image signing. The index is then stored in the registry alongside the container image. At release, AWS Fargate support for SOCI works with Amazon Elastic Container Registry (Amazon ECR).

When you use Amazon ECS with AWS Fargate to run your SOCI-indexed containerized images, AWS Fargate automatically detects if a SOCI index for the image exists and starts the container without waiting for the entire image to be pulled. This also means that AWS Fargate will still continue to run container images that don’t have SOCI indexes.

Let’s Get Started
There are two ways to create SOCI indexes for container images.

Use AWS SOCI Index Builder – AWS SOCI Index Builder is a serverless solution for indexing container images in the AWS Cloud. This AWS CloudFormation stack deploys an Amazon EventBridge rule to identify Amazon ECR action events and invoke an AWS Lambda function to match the defined filter. Then, another AWS Lambda function generates and pushes SOCI indexes to repositories in the Amazon ECR registry.
Create SOCI indexes manually – This approach provides more flexibility on in how the SOCI indexes are created, including for existing container images in Amazon ECR repositories. To create SOCI indexes, you can use the soci CLI provided by the soci-snapshotter project.

The AWS SOCI Index Builder provides you with an automated process to get started and build SOCI indexes for your container images. The sociCLI provides you with more flexibility around index generation and the ability to natively integrate index generation in your CI/CD pipelines.

In this article, I manually generate SOCI indexes using the soci CLI from the soci-snapshotter project.

Create a Repository and Push Container Images
First, I create an Amazon ECR repository called pytorch-socifor my container image using AWS CLI.

$ aws ecr create-repository --region us-east-1 --repository-name pytorch-soci

I keep the Amazon ECR URI output and define it as a variable to make it easier for me to refer to the repository in the next step.

$ ECRSOCIURI=xyz.dkr.ecr.us-east-1.amazonaws.com/pytorch-soci:latest

For the sample application, I use a PyTorch training (CPU-based) container image from AWS Deep Learning Containers. I use the nerdctl CLI to pull the container image because, by default, the Docker Engine stores the container image in the Docker Engine image store, not the containerd image store.

$ SAMPLE_IMAGE="763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.5.1-cpu-py36-ubuntu16.04" 
$ aws ecr get-login-password --region us-east-1 | sudo nerdctl login --username AWS --password-stdin xyz.dkr.ecr.ap-southeast-1.amazonaws.com
$ sudo nerdctl pull --platform linux/amd64 $SAMPLE_IMAGE

Then, I tag the container image for the repository that I created in the previous step.

$ sudo nerdctl tag $SAMPLE_IMAGE $ECRSOCIURI

Next, I need to push the container image into the ECR repository.

$ sudo nerdctl push $ECRSOCIURI

At this point, my container image is already in my Amazon ECR repository.

Create SOCI Indexes
Next, I need to create SOCI index.

A SOCI index is an artifact that enables lazy loading of container images. A SOCI index consists of 1) a SOCI index manifest and 2) a set of zTOCs. The following image illustrates the components in a SOCI index manifest, and how it refers to a container image manifest.

The SOCI index manifest contains the list of zTOCs and a reference to the image for which the manifest was generated. A zTOC, or table of contents for compressed data, consists of two parts:

TOC, a table of contents containing file metadata and the corresponding offset in the decompressed TAR archive.
zInfo, a collection of checkpoints representing the state of the compression engine at various points in the layer.

To learn more about the concept and term, please visit soci-snapshotter Terminology page.

Before I can create SOCI indexes, I need to install the sociCLI. To learn more about how to install the soci, visit Getting Started with soci-snapshotter.

To create SOCI indexes, I use the soci create command.

$ sudo soci create $ECRSOCIURI
layer sha256:4c6ec688ebe374ea7d89ce967576d221a177ebd2c02ca9f053197f954102e30b -> ztoc skipped
layer sha256:ab09082b308205f9bf973c4b887132374f34ec64b923deef7e2f7ea1a34c1dad -> ztoc skipped
layer sha256:cd413555f0d1643e96fe0d4da7f5ed5e8dc9c6004b0731a0a810acab381d8c61 -> ztoc skipped
layer sha256:eee85b8a173b8fde0e319d42ae4adb7990ed2a0ce97ca5563cf85f529879a301 -> ztoc skipped
layer sha256:3a1b659108d7aaa52a58355c7f5704fcd6ab1b348ec9b61da925f3c3affa7efc -> ztoc skipped
layer sha256:d8f520dcac6d926130409c7b3a8f77aea639642ba1347359aaf81a8b43ce1f99 -> ztoc skipped
layer sha256:d75d26599d366ecd2aa1bfa72926948ce821815f89604b6a0a49cfca100570a0 -> ztoc skipped
layer sha256:a429d26ed72a85a6588f4b2af0049ae75761dac1bb8ba8017b8830878fb51124 -> ztoc skipped
layer sha256:5bebf55933a382e053394e285accaecb1dec9e215a5c7da0b9962a2d09a579bc -> ztoc skipped
layer sha256:5dfa26c6b9c9d1ccbcb1eaa65befa376805d9324174ac580ca76fdedc3575f54 -> ztoc skipped
layer sha256:0ba7bf18aa406cb7dc372ac732de222b04d1c824ff1705d8900831c3d1361ff5 -> ztoc skipped
layer sha256:4007a89234b4f56c03e6831dc220550d2e5fba935d9f5f5bcea64857ac4f4888 -> ztoc sha256:0b4d78c856b7e9e3d507ac6ba64e2e2468997639608ef43c088637f379bb47e4
layer sha256:089632f60d8cfe243c5bc355a77401c9a8d2f415d730f00f6f91d44bb96c251b -> ztoc sha256:f6a16d3d07326fe3bddbdb1aab5fbd4e924ec357b4292a6933158cc7cc33605b
layer sha256:f18dd99041c3095ade3d5013a61a00eeab8b878ba9be8545c2eabfbca3f3a7f3 -> ztoc sha256:95d7966c964dabb54cb110a1a8373d7b88cfc479336d473f6ba0f275afa629dd
layer sha256:69e1edcfbd217582677d4636de8be2a25a24775469d677664c8714ed64f557c3 -> ztoc sha256:ac0e18bd39d398917942c4b87ac75b90240df1e5cb13999869158877b400b865

From the above output, I can see that sociCLI created zTOCs for four layers, which and this means only these four layers will be lazily pulled and the other container image layers will be downloaded in full before the container image starts. This is because there is less of a launch time impact in lazy loading very small container image layers. However, you can configure this behavior using the --min-layer-size flag when you run soci create.

Verify and Push SOCI Indexes
The soci CLI also provides several commands that can help you to review the SOCI Indexes that have been generated.

To see a list of all index manifests, I can run the following command.

$ sudo soci index list

DIGEST                                                                     SIZE    IMAGE REF                                                                                   PLATFORM       MEDIA TYPE                                    CREATED
sha256:ea5c3489622d4e97d4ad5e300c8482c3d30b2be44a12c68779776014b15c5822    1931    xyz.dkr.ecr.us-east-1.amazonaws.com/pytorch-soci:latest                                     linux/amd64    application/vnd.oci.image.manifest.v1+json    10m4s ago
sha256:ea5c3489622d4e97d4ad5e300c8482c3d30b2be44a12c68779776014b15c5822    1931    763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.5.1-cpu-py36-ubuntu16.04    linux/amd64    application/vnd.oci.image.manifest.v1+json    10m4s ago

While optional, if I need to see the list of zTOC, I can use the following command.

$ sudo soci ztoc list
DIGEST                                                                     SIZE        LAYER DIGEST
sha256:0b4d78c856b7e9e3d507ac6ba64e2e2468997639608ef43c088637f379bb47e4    2038072     sha256:4007a89234b4f56c03e6831dc220550d2e5fba935d9f5f5bcea64857ac4f4888
sha256:95d7966c964dabb54cb110a1a8373d7b88cfc479336d473f6ba0f275afa629dd    11442416    sha256:f18dd99041c3095ade3d5013a61a00eeab8b878ba9be8545c2eabfbca3f3a7f3
sha256:ac0e18bd39d398917942c4b87ac75b90240df1e5cb13999869158877b400b865    36277264    sha256:69e1edcfbd217582677d4636de8be2a25a24775469d677664c8714ed64f557c3
sha256:f6a16d3d07326fe3bddbdb1aab5fbd4e924ec357b4292a6933158cc7cc33605b    10152696    sha256:089632f60d8cfe243c5bc355a77401c9a8d2f415d730f00f6f91d44bb96c251b

This series of zTOCs contains all of the information that SOCI needs to find a given file in a layer. To review the zTOC for each layer, I can use one of the digest sums from the preceding output and use the following command.

$ sudo soci ztoc info sha256:0b4d78c856b7e9e3d507ac6ba64e2e2468997639608ef43c088637f379bb47e4
{
  "version": "0.9",
  "build_tool": "AWS SOCI CLI v0.1",
  "size": 2038072,
  "span_size": 4194304,
  "num_spans": 33,
  "num_files": 5552,
  "num_multi_span_files": 26,
  "files": [
    {
      "filename": "bin/",
      "offset": 512,
      "size": 0,
      "type": "dir",
      "start_span": 0,
      "end_span": 0
    },
    {
      "filename": "bin/bash",
      "offset": 1024,
      "size": 1037528,
      "type": "reg",
      "start_span": 0,
      "end_span": 0
    }

---Trimmed for brevity---

Now, I need to use the following command to push all SOCI-related artifacts into the Amazon ECR.

$ PASSWORD=$(aws ecr get-login-password --region us-east-1)
$ sudo soci push --user AWS:$PASSWORD $ECRSOCIURI

If I go to my Amazon ECR repository, I can verify the index is created. Here, I can see that two additional objects are listed alongside my container image: a SOCI Index and an Image index. The image index allows AWS Fargate to look up SOCI indexes associated with my container image.

Understanding SOCI Performance
The main objective of SOCI is to minimize the required time to start containerized applications. To measure the performance of AWS Fargate lazy loading container images using SOCI, I need to understand how long it takes for my container images to start with SOCI and without SOCI.

To understand the duration needed for each container image to start, I can use metrics available from the DescribeTasks API on Amazon ECS. The first metric is createdAt, the timestamp for the time when the task was created and entered the PENDING state. The second metric is startedAt, the time when the task transitioned from the PENDING state to the RUNNING state.

For this, I have created another Amazon ECR repository using the same container image but without generating a SOCI index, called pytorch-without-soci. If I compare these container images, I have two additional objects in pytorch-soci(an image index and a SOCI index) that don’t exist in pytorch-without-soci.

Deploy and Run Applications
To run the applications, I have created an Amazon ECS cluster called demo-pytorch-soci-cluster, a VPC and the required ECS task execution role. If you’re new to Amazon ECS, you can follow Getting started with Amazon ECS to be more familiar with how to deploy and run your containerized applications.

Now, let’s deploy and run both the container images with FARGATE as the launch type. I deﬁne five tasks for each pytorch-sociand pytorch-without-soci.

$ aws ecs \ 
    --region us-east-1 \ 
    run-task \ 
    --count 5 \ 
    --launch-type FARGATE \ 
    --task-definition arn:aws:ecs:us-east-1:XYZ:task-definition/pytorch-soci \ 
    --cluster socidemo 

$ aws ecs \ 
    --region us-east-1 \ 
    run-task \ 
    --count 5 \ 
    --launch-type FARGATE \ 
    --task-definition arn:aws:ecs:us-east-1:XYZ:task-definition/pytorch-without-soci \ 
    --cluster socidemo

After a few minutes, there are 10 running tasks on my ECS cluster.

After verifying that all my tasks are running, I run the following script to get two metrics: createdAt and startedAt.

#!/bin/bash
CLUSTER=<CLUSTER_NAME>
TASKDEF=<TASK_DEFINITION>
REGION="us-east-1"
TASKS=$(aws ecs list-tasks \
    --cluster $CLUSTER \
    --family $TASKDEF \
    --region $REGION \
    --query 'taskArns[*]' \
    --output text)

aws ecs describe-tasks \
    --tasks $TASKS \
    --region $REGION \
    --cluster $CLUSTER \
    --query "tasks[] | reverse(sort_by(@, &createdAt)) | [].[{startedAt: startedAt, createdAt: createdAt, taskArn: taskArn}]" \
    --output table

Running the above command for the container image without SOCI indexes — pytorch-without-soci— produces following output:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                                   DescribeTasks                                                                                   |
+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+
|             createdAt            |             startedAt             |                                                  taskArn                                                   |
+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+
|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:09.856000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/dcdf19b6e66444aeb3bc607a3114fae0   |
|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:09.459000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/9178b75c98ee4c4e8d9c681ddb26f2ca   |
|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:21.645000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/7da51e036c414cbab7690409ce08cc99   |
|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:00.606000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/5ee8f48194874e6dbba75a5ef753cad2   |
|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:02.461000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/58531a9e94ed44deb5377fa997caec36   |
+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+

From the average aggregated delta time (between startedAt and createdAt) for each task, the pytorch-without-soci (without SOCI indexes) successfully ran after 129 seconds.

Next, I’m running same command but for pytorch-sociwhich comes with SOCI indexes.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                                   DescribeTasks                                                                                   |
+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+
|             createdAt            |             startedAt             |                                                  taskArn                                                   |
+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+
|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:44:51.076000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/c57d8cff6033494b97f6fd0e1b797b8f   |
|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:44:52.212000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/6d168f9e99324a59bd6e28de36289456   |
|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:45:05.443000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/4bdc43b4c1f84f8d9d40dbd1a41645da   |
|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:44:50.618000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/43ea53ea84154d5aa90f8fdd7414c6df   |
|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:44:50.777000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/0731bea30d42449e9006a5d8902756d5   |
+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+

Here, I see my container image with SOCI-enabled — pytorch-soci — was started 60 seconds after being created.

This means that running my sample application with SOCI indexes on AWS Fargate is approximately 50 percent faster compared to running without SOCI indexes.

It’s recommended to benchmark the startup and scaling-out time of your application with and without SOCI. This helps you to have a better understanding of how your application behaves and if your applications benefit from AWS Fargate support for SOCI.

Customer Voices
During the private preview period, we heard lots of feedback from our customers about AWS Fargate support for SOCI. Here’s what our customers say:

Autodesk provides critical design, make, and operate software solutions across the architecture, engineering, construction, manufacturing, media, and entertainment industries. “SOCI has given us a 50% improvement in startup performance for our time-sensitive simulation workloads running on Amazon ECS with AWS Fargate. This allows our application to scale out faster, enabling us to quickly serve increased user demand and save on costs by reducing idle compute capacity. The AWS Partner Solution for creating the SOCI index is easy to configure and deploy.” – Boaz Brudner, Head of Innovyze SaaS Engineering, AI and Architecture, Autodesk.

Flywire is a global payments enablement and software company, on a mission to deliver the world’s most important and complex payments. “We run multi-step deployment pipelines on Amazon ECS with AWS Fargate which can take several minutes to complete. With SOCI, the total pipeline duration is reduced by over 50% without making any changes to our applications, or the deployment process. This allowed us to drastically reduce the rollout time for our application updates. For some of our larger images of over 750MB, SOCI improved the task startup time by more than 60%.”, Samuel Burgos, Sr. Cloud Security Engineer, Flywire.

Virtuoso is a leading software corporation that makes functional UI and end-to-end testing software. “SOCI has helped us reduce the lag between demand and availability of compute. We have very bursty workloads which our customers expect to start as fast as possible. SOCI helps our ECS tasks spin-up 40% faster, allowing us to quickly scale our application and reduce the pool of idle compute capacity, enabling us to deliver value more efficiently. Setting up SOCI was really easy. We opted to use the quick-start AWS Partner’s solution with which we could leave our build and deployment pipelines untouched.”, Mathew Hall, Head of Site Reliability Engineering, Virtuoso.

Things to Know
Availability — AWS Fargate support for SOCI is available in all AWS Regions where Amazon ECS, AWS Fargate, and Amazon ECR are available.

Pricing — AWS Fargate support for SOCI is available at no additional cost and you will only be charged for storing the SOCI indexes in Amazon ECR.

Get Started — Learn more about benefits and how to get started on the AWS Fargate Support for SOCI page.

Happy building.
— Donnie

[Lost Bots] S03 E04 A Security Leader’s Playbook for the C-suite

2023-07-17 Amy Hunt

Post Syndicated from Amy Hunt original https://blog.rapid7.com/2023/07/17/lost-bots-s03-e04-a-security-leaders-playbook-for-the-c-suite/

[Lost Bots] S03 E04 A Security Leader’s Playbook for the C-suite

In a special two-part “Lost Bots,” hosts Jeffrey Gardner and Stephen Davis talk about presenting cybersecurity results up the org chart. Both have handled C-suite and board communications and have lots of lessons learned.

Part 1 is about the style of a presentation: the point, the delivery, the storytelling. Gardner believes anyone can be great because he’s “an extreme introvert” himself. He shares a ton of wisdom about how to structure your presentation and really own the room with confidence. About halfway through, the ideas start coming fast and furious.

Part 2 brings it together with a deep dive into metrics (and an extraordinary bowtie on Mr. Davis, seriously). Metrics aren’t your story, but they do prove it true. The episode with one thing you must take away and remember: you’re not there to sell more security, you’re there to help stakeholders make well-informed business decisions. When that purpose is clear, some things get simpler.

QNAP TBS-574TX E1.S Thunderbolt 4 10GbE NAS Spotted

2023-07-17 Eric Smith

Post Syndicated from Eric Smith original https://www.servethehome.com/qnap-tbs-574tx-e1-s-thunderbolt-4-10gbe-nas-spotted/

We spotted the 5-bay E1.S QNAP TBS-574TX NAS at Computex 2023 that has two Thunderbolt 4 ports and both 10GbE and 2.5GbE

The post QNAP TBS-574TX E1.S Thunderbolt 4 10GbE NAS Spotted appeared first on ServeTheHome.

[$] Debian looks forward to 2038

2023-07-17

Post Syndicated from original https://lwn.net/Articles/938149/

On January 19, 2038, the time_t value used on many 32-bit Linux
systems will overflow and wrap around, causing those systems to believe
they have returned to 1970 and wonder why they feel like they have heard Déjà
Vu before. Much work has gone into preparing many layers of the
system for this event, but not all distributions have completed their
preparations. One of those is Debian but, as was seen in a conversation in
May, the Debian developers are now grappling with the problem in a serious
way. Along the way, they appear to have made an interesting decision
regarding which systems will (or will not) be updated.

A developer’s guide to prompt engineering and LLMs

2023-07-17 Albert Ziegler

Post Syndicated from Albert Ziegler original https://github.blog/2023-07-17-prompt-engineering-guide-generative-ai-llms/

In a blog post authored back in 2011, Marc Andreessen warned that, “Software is eating the world.” Over a decade later, we are witnessing the emergence of a new type of technology that’s consuming the world with even greater voracity: generative artificial intelligence (AI). This innovative AI includes a unique class of large language models (LLM), derived from a decade of groundbreaking research, that are capable of out-performing humans at certain tasks. And you don’t have to have a PhD in machine learning to build with LLMs—developers are already building software with LLMs with basic HTTP requests and natural language prompts.

In this article, we’ll tell the story of GitHub’s work with LLMs to help other developers learn how to best make use of this technology. This post consists of two main sections: the first will describe at a high level how LLMs function and how to build LLM-based applications. The second will dig into an important example of an LLM-based application: GitHub Copilot code completions.

Others have done an impressive job of cataloging our work from the outside. Now, we’re excited to share some of the thought processes that have led to the ongoing success of GitHub Copilot.

Let’s jump in.

Everything you need to know about prompt engineering in 1600 tokens or less

You know when you’re tapping out a text message on your phone, and in the middle of the screen just above the keypad, there’s a button you can click to accept a suggested next word? That’s pretty much what an LLM is doing—but at scale.

A GIF show autocomplete functionalities in iOS. — An example of iMessage’s text prediction feature.

Instead of text on your phone, an LLM works to predict the next best group of letters, which are called “tokens.” And in the same way that you can keep tapping that middle button to complete your text message, the LLM completes a document by predicting the next word. It will continue to do that over and over, and it will only stop once it has reached a maximum threshold of tokens or once it has encountered a special token that signals “Stop! This is the end of the document.”

There’s an important difference, though. The language model in your phone is pretty simple—it’s basically saying, “Based only upon the last two words entered, what is the most likely next word?” In contrast, an LLM produces an output that’s more akin to being “based upon the full content of every document ever known to exist in the public domain, what is the most likely next token in your document?” By training such a large, well-architected model on an enormous dataset, an LLM can almost appear to have common sense such as understanding that a glass ball sitting on a table might roll off and shatter.

A screenshot of ChatGPT answering a question about the danger of setting a round glass ball on a small table. — Example of an LLM’s awareness or “common sense” due to its training.

But be warned: LLMs will also sometimes confidently produce information that isn’t real or true, which are typically called “hallucinations” or “fabulations.” LLMs can also appear to learn how to do things they weren’t initially trained to do. Historically, natural language models have been created for one-off tasks, like classifying the sentiment of a tweet, extracting the business entities from an email, or identifying similar documents, but now you can ask AI tools like ChatGPT to perform a task that it was never trained to do.

A screenshot of ChatGPT answering a prompt to create a chicken-based limerick. — John conversing with ChatGPT about serious things.

Building applications using LLMs

A document completion engine is a far cry from the amazing proliferation of LLM applications that are springing up every day, running the gamut from conversational search, writing assistants, automated IT support, and code completion tools, like GitHub Copilot. But how is it possible that all of these tools can come from what is effectively a document completion tool? The secret is any application that uses an LLM is actually mapping between two domains: the user domain and the document domain.

A graphic showing how LLMs work and the processes behind them to determine context before giving an answer. — Diagram of the user flow when communicating with an LLM, in this case, Dave’s user flow.

On the left is the user. His name is Dave, and he has a problem. It’s the day of his big World Cup watch party, and the Wi-Fi is out. If they don’t get it fixed soon, he’ll be the butt of his friends’ jokes for years. Dave calls his internet provider and gets an automated assistant. Ugh! But imagine that we are implementing the automated assistant as an LLM application. Can we help him?

The key here is to figure out how to convert from user domain into document domain. For one thing, we will need to transcribe the user’s speech into text. As soon as the automated support agent says “Please state the nature of your cable-related emergency,” Dave blurts out:

Oh it’s awful! It’s the World Cup finals. My TV was connected to my Wi-Fi, but I bumped the counter and the Wi-Fi box fell off and broke! Now, we can’t watch the game.

At this point, we have text, but it’s not of much use. Maybe you would imagine that this was part of a story and continue it, “I guess, I’ll call up my brother and see if we can watch the game with him.” An LLM with no context will similarly create the continuation of Dave’s story. So, let’s give the LLM some context and establish what type of document this is:

### ISP IT Support Transcript:

The following is a recorded conversation between an ISP customer, Dave Anderson, and Julia Jones, IT support expert. This transcript serves as an example of the excellent support provided by Comcrash to its customers.

*Dave: Oh it's awful! This is the big game day. My TV was connected to my Wi-Fi, but I bumped the counter and the Wi-Fi box fell off and broke! Now we can't watch the game.
*Julia:

Now, if you found this pseudo document on the ground, how would you complete it? Based on the extra context, you would see that Julia is an IT support expert, and apparently a really good one. You would expect the next words to be sage advice to help Dave with his problem. It doesn’t matter that Julia doesn’t exist, and this wasn’t a recorded conversation—what matters is that these extra words offer more context for what a completion might look like. An LLM does the same exact thing. After reading this partial document, it will do its best to complete Julia’s dialogue in a helpful manner.

But there’s more we can do to make the best document for the LLM. The LLM doesn’t know a whole lot about cable TV troubleshooting. (Well, it has read every manual and IT document ever published online, but stay with me here). Let’s assume that its knowledge is lacking in this particular domain. One thing we can do is search for extra content that might help Dave and place it into the document. Let’s assume that we have a complaints search engine that allows us to find documentation that has been helpful in similar situations in the past. Now, all we have to do is weave this information into our pseudo document in a natural place.

Continuing from above:

*Julia:(rifles around in her briefcase and pulls out the perfect documentation for Dave's request)
Common internet connectivity problems ...
<...here we insert 1 page of text that comes from search results against our customer support history database...>
(After reading the document, Julia makes the following recommendation)
*Julia:

Now, given this full body of text, the LLM is conditioned to make use of the implanted documentation, and in the context of “a helpful IT expert,” the model will generate a response. This reply takes into account the documentation as well as Dave’s specific request.

The last step is to move from the document domain into the user’s problem domain. For this example, that means just converting text to voice. And since this is effectively a chat application, we would go back and forth several times between the user and the document domain, making the transcript longer each time.

This, at the core of the example, is prompt engineering. In the example, we crafted a prompt with enough context for the AI to produce the best possible output, which in this case was providing Dave with helpful information to get his Wi-Fi up and running again. In the next section, we’ll take a look at how we at GitHub have refined our prompt engineering techniques for GitHub Copilot.

The art and science of prompt engineering

Converting between the user domain and document domain is the realm of prompt engineering—and since we’ve been working on GitHub Copilot for over two years, we’ve started to identify some patterns in the process.

These patterns have helped us formalize a pipeline, and we think it is an applicable template to help others better approach prompt engineering for their own applications. Now, we’ll demonstrate how this pipeline works by examining it in the context of GitHub Copilot, our AI pair programmer.

The prompt engineering pipeline for GitHub Copilot

From the very beginning, GitHub Copilot’s LLMs have been built on AI models from OpenAI that have continued to get better and better. But what hasn’t changed is the answer to the central question of prompt engineering: what kind of document is the model trying to complete?

The OpenAI models we use have been trained to complete code files on GitHub. Ignoring some filtering and stratification steps that don’t really change the prompt engineering game, this distribution is pretty much that of individual file contents according to the most recent commit to main at data collection time.

The document completion problem the LLM solves is about code, and GitHub Copilot’s task is all about completing code. But the two are very different.

Here are some examples:

Most files committed to main are finished. For one, they usually compile. Most of the time the user is typing, the code does not compile because of incompletions that will be fixed before a commit is pushed.
The user might even write their code in hierarchical order, method signatures first, then bodies rather than line by line or in a mixed style.
Writing code means jumping around. In particular, people’s edits often require them to jump up in the document and make a change there, for example, adding a parameter to a function. Strictly speaking, if Codex suggests using a function that has not been imported yet, no matter how much sense it might make, that’s a mistake. But as a GitHub Copilot suggestion, it would be useful.

The issue is that merely predicting the most likely continuation based on the text in front of the cursor to make a GitHub Copilot suggestion would be a wasted opportunity. That’s because it ignores an incredible wealth of context. We can use that context to guide the suggestion, like metadata, the code below the cursor, the content of imports, the rest of the repository, or issues, and create a strong prompt for the AI assistant.

Software development is a deeply interconnected, multimodal challenge, and the more of that complexity we can tame and present to the model, the better your completions are going to be.

Step 1: Gathering context

GitHub Copilot lives in the context of an IDE such as Visual Studio Code (VS Code), and it can use whatever it can get the IDE to tell it—only if the IDE is quick about it though. In an interactive environment like GitHub Copilot, every millisecond matters. GitHub Copilot promises to take care of the common coding tasks, and if it wants to do that, it needs to display its solution to the developer before they have started to write more code in their IDE. Our rough heuristics say that for every additional 10 milliseconds we take to come up with a suggestion, the chance it’ll arrive in time decreases by one percent.

So, what can we say quickly? Well, here’s an example. Consider this suggestion to a simple piece of Python:

A developer prompting GitHub Copilot to write a simple function in Python to compute Fibonacci numbers.

Wrong! Turns out the user actually wanted to write Ruby, like this:

A developer using GitHub Copilot to write a simple function to compute Fibonacci numbers in Ruby.

The two languages have similar enough syntax so that only a couple of lines can be ambiguous, especially when it’s toward the beginning of the file where much of what we encounter are boilerplate comments. But modern IDEs such as VS Code typically know what language the user is writing in. That makes language mix ups especially annoying to the user because they break the implicit expectation that “the computer should know” (after all, most IDEs highlight language syntax).

So, let’s put the language metadata into our pile of context we might want to include. In fact, let’s add the whole filename too. If it’s available, it usually implies the language through its extension, and additionally sets the tone for what to expect in that file—small, easy pieces of information that won’t turn the tide but are helpful to include.

On the other end of the spectrum, there’s the rest of the repository. Say you’ve got a file that defines an abstract class DataReader. And you have another that defines a subclass CsvReader. And you’re now writing a new file defining another subclass SqlReader. Chances are that to write the new file, you’ll want to check out both existing files as well because they communicate useful background into what you need to implement and how to do it. Typically, developers keep such files open in different tabs and switch to remind themselves of definitions, examples, similar patterns, or tests.

If the content of those two files is useful to you, chances are it would be useful to the AI as well. So, let’s add it as context! After all, the IDE knows what other files from the repository are open as tabs in the same window. The repository might have hundreds or even thousands of files, but only some will be open, and that is a strong hint that they might be useful to what they’re doing right now. Of course, “some” can mean a lot of things, so we don’t consider any more than the 20 most recent tabs.

Step 2: Snippeting

Irrelevant information in an LLM’s context decreases its accuracy. Additionally, source code tends to be long, so even a single file is not guaranteed to fit completely into an LLM’s context window (a problem that occurs roughly a fifth of the time). So, unless the user is very frugal about their tab usage, we simply cannot include all the tabs.

It’s important to be selective about what code to include from other files, so we cut files into (hopefully) natural, overlapping snippets that are no longer than 60 lines. Of course, we don’t want to actually include all overlapping snippets—that’s why we score them and take only the best. In this case, the “score” is meant to reflect relevance. To determine a snippet’s score, we use the Jaccard similarity, a stat that can be used to gauge the similarity or diversity of sample sets. (It’s also super fast to compute, which is great for reducing latency.)

Step 3: Dressing them up

Now we have some context we’d like to pass on to the model. But how? Codex and other models don’t offer an API where you can add other files, or where you can specify the document’s language and filename for that matter. They complete one single document. As mentioned above, you’ll need to inject your context into that document in a natural way.

The path and name might be easiest. Many files start with a preamble that gives some metadata, like author, project name, or filename. So, we’ll pretend this is happening here as well, and add a line at the very top that reads something like # filepath: foo/bar.py or // filepath: foo.bar.js, depending on comment syntax in the file’s language.

Sometimes the path isn’t known, like with new files that haven’t yet been saved. Even then, we could try to at least specify the language, provided the IDE is aware of it. For many languages, we have the opportunity to include shebang lines like #!/usr/bin/python or #!/usr/bin/node. That’s a neat trick that works pretty well at warding against mistaken language identity. But it’s also a bit dangerous since files with shebang lines are a biased subpopulation of all code. So, let’s do it for short files where the danger of mistaken language identity is high, and avoid it for larger or named files.

If comments work as a delivery system for tiny nuggets of information, like path or language, we can also make them work as delivery systems for the chunky deep dives that are 60 lines of related code.

Comments are versatile, and commented-out code exists all over GitHub. Let’s look at some of the most common examples:

Old code that doesn’t apply anymore
Deleted features
Earlier versions of current code
Example code specifically left there for documentation purposes
Code lifted from other parts of the codebase

Let’s take our inspiration from the last group of examples. Familiarity with groups (1) – (3) makes things a bit easier on the model, but our snippets aim to emulate groups (4) and (5):

# compare this snippet from utils/concatenate.py:

# def crazy_concat(a, b):

# return str(a) + str(b)[::-1]

Note that including the file name and path of the snippet source can be useful. And combined with the current file’s path, this might guide completions referencing imports.

Step 4: Prioritization

So far, we have grabbed many pieces of context from many sources: the text directly above the cursor, text below the cursor, text in other files, and metadata like language and file path.

In the vast majority of cases (around 95%), we have to make the tough choice of what we can or cannot include.

We make that choice by thinking of the items we might include as “wishes.” Each time we uncover a piece of context, like a commented out snippet from an open tab, we make a wish. Wishes come with some priority attached, for example, the shebang lines have rather low priorities. Snippets with a low similarity score are barely higher. In contrast, the lines directly above the cursor have maximum priority. Wishes also come with a desired position in the document. The shebang line needs to be the very first item, while the text directly above the cursor comes last—it should directly precede the LLM’s completion.

The fastest way of selecting which wishes to fill and which ones to discard is by sorting that wishlist by priority. Then, we can keep deleting the lowest priority wishes until what remains fits in the context window. We then sort again by the intended order in the document and paste everything together.

Step 5: The AI does its thing

Now that we’ve assembled an informative prompt, it’s time for the AI to come up with a useful completion. We have always faced a very delicate tradeoff here—GitHub Copilot needs to use a highly capable model because quality makes all the difference between a useful suggestion and a distraction. But at the same time, it needs to be a model capable of speed, because latency makes all the difference between a useful suggestion and not being able to provide a suggestion at all.

So, which AI should we choose to “do its thing” on the completion task: the fastest or the most accurate one? It’s hard to know in advance, so OpenAI developed a fleet of models in collaboration with GitHub. We put two different models in front of developers but found that people got the most mileage (in terms of accepted and retained completions) out of the much faster model. Since then, further optimizations have increased model speed significantly, so that the current version of GitHub Copilot is backed by an even more capable model.

Step 6: Now, over to you!

The generative AI produces a string, and if it’s not stopped, it keeps on producing and will keep going until it predicts the end of the file. That would waste time and compute resources, so you need to set up “stop” criteria.

The most common stop criterion is actually looking for the first line break. In many situations, it seems likely that a software developer wants the current line to be finished, but not more. But some of the most magical contributions by GitHub Copilot are when it suggests multiple lines of code all at once.

Multi-line completions feel natural when they’re about a single semantic unit, such as the body of a function, an if-branch, or a class. GitHub Copilot looks for cases where such a block is being started, either because the developer has just written the start, such as the header, if guard, or class declaration, or is currently writing the start. If the block body appears to be empty, it will attempt to make a suggestion for it, and only stop when the block appears to be done.

This is the point when the suggestion gets surfaced to the coder. And the rest, as they say, is ~~history~~ 10x development.

If you’re interested in learning more about prompt engineering in general and how you can refine your own techniques, check out our guide on getting started with GitHub Copilot.

Security updates for Monday

2023-07-17

Post Syndicated from original https://lwn.net/Articles/938375/

Security updates have been issued by Debian (gpac, iperf3, kanboard, kernel, and pypdf2), Fedora (ghostscript), SUSE (bind, bouncycastle, ghostscript, go1.19, go1.20, installation-images, kernel, mariadb, MozillaFirefox, MozillaFirefox-branding-SLE, php74, poppler, and python-Django), and Ubuntu (cups, linux-oem-6.1, and ruby2.3, ruby2.5, ruby2.7, ruby3.0, ruby3.1).

Best of The History Guy: Gotham

2023-07-17 The History Guy: History Deserves to Be Remembered

Post Syndicated from The History Guy: History Deserves to Be Remembered original https://www.youtube.com/watch?v=DEXgY31fHLs

Tracking Down a Suspect through Cell Phone Records

2023-07-17 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/07/tracking-down-a-suspect-through-cell-phone-records.html

Interesting forensics in connection with a serial killer arrest:

Investigators went through phone records collected from both midtown Manhattan and the Massapequa Park area of Long Island—two areas connected to a “burner phone” they had tied to the killings. (In court, prosecutors later said the burner phone was identified via an email account used to “solicit and arrange for sexual activity.” The victims had all been Craigslist escorts, according to officials.)

They then narrowed records collected by cell towers to thousands, then to hundreds, and finally down to a handful of people who could match a suspect in the killings.

From there, authorities focused on people who lived in the area of the cell tower and also matched a physical description given by a witness who had seen the suspected killer.

In that narrowed pool, they searched for a connection to a green pickup truck that a witness had seen the suspect driving, the sources said.

Investigators eventually landed on Heuermann, who they say matched a witness’ physical description, lived close to the Long Island cell site and worked near the New York City cell sites that captured the other calls.

They also learned he had often driven a green pickup truck, registered to his brother, officials said. But they needed more than just circumstantial evidence.

Investigators were able to obtain DNA from an immediate family member and send it to a specialized lab, sources said. According to the lab report, Heuermann’s family member was shown to be related to a person who left DNA on a burlap sack containing one of the buried victims.

There’s nothing groundbreaking here; it’s casting a wide net with cell phone geolocation data and then winnowing it down using other evidence and investigative techniques. And right now, those are expensive and time consuming, so only used in major crimes like murder (or, in this case, murders).

What’s interesting to think about is what happens when this kind of thing becomes cheap and easy: when it can all be done through easily accessible databases, or even when an AI can do the sorting and make the inferences automatically. Cheaper digital forensics means more digital forensics, and we’ll start seeing this kind of thing for even routine crimes. That’s going to change things.

Comic for 2023.07.17 – The Wheel

2023-07-17 Explosm.net

Post Syndicated from Explosm.net original https://explosm.net/comics/the-wheel

New Cyanide and Happiness Comic