Tag Archives: DFIR

What’s New in Rapid7 Detection & Response: Q3 2023 in Review

Post Syndicated from Margaret Wei original https://blog.rapid7.com/2023/10/05/whats-new-in-rapid7-detection-response-q3-2023-in-review/

What’s New in Rapid7 Detection & Response: Q3 2023 in Review

This post takes a look at some of the investments we’ve made throughout Q3 2023 to our Detection and Response offerings to provide advanced DFIR capabilities with Velociraptor, more flexibility with custom detection rules, enhancements to our dashboard and log search features, and more.

Stop attacks before they happen with Next-Gen Antivirus in Managed Threat Complete

As endpoint attacks become more elusive and frequent, we know security teams need reliable coverage to keep their organizations safe. To provide teams with protection from both known and unknown threats, we’ve released multilayered prevention with Next-Gen Antivirus in Managed Threat Complete. Available through the Insight Agent, you’ll get immediate coverage with no additional configurations or deployments. With Managed Next-Gen Antivirus you’ll be able to:

  • Block known and unknown threats early in the kill chain
  • Halt malware that’s built to bypass existing security controls
  • Maximize your security stack and ROI with existing Insight Agent
  • Leverage the expertise of our MDR team to triage and investigate these alerts

To see more on our Managed Next-Gen Antivirus offering, including a demo walkthrough, visit our Endpoint Hub Page here.

Achieve faster DFIR outcomes with Velociraptor now integrated into the Insight Platform

As security teams are facing more and more persistent threats on their endpoints, it’s crucial to have proactive security measures that can identify attacks early in the kill chain, and the ability to access detailed evidence to drive complete remediation. We’re excited to announce that InsightIDR Ultimate customers can now recognize the value of Velociraptor, Rapid7’s open-source DFIR framework, faster than ever with its new integration into the Insight Platform.

With no additional deployment or configurations required, InsightIDR customers can deploy Velociraptor through their existing Insight Agents for daily threat monitoring and hunting, swift threat response, and expanded threat detection capabilities. For more details, check out our recent blog post here.

What’s New in Rapid7 Detection & Response: Q3 2023 in Review

A view of Velociraptor in InsightIDR

Tailor alerts to your unique needs with Custom Detection Rules

We know every organization has unique needs when it comes to detections and alerting on threats. While InsightIDR provides over 3,000 out-of-the-box detection rules to detect malicious behaviors, we’ve added additional capabilities with Custom Detection Rules to offer teams the ability to author rules tailored to their own individual needs. With Custom Detection Rules, you will be able to:

  • Build upon Rapid7’s library of expertly curated detection rules by creating rules that uniquely fit your organization’s security needs
  • Use LEQL to write rule logic against a variety of data sources
  • Add grouping and threshold conditions to refine your rule logic over specific periods of time to decrease unnecessary noise
  • Assess the rules activity before it starts to trigger alerts for downstream teams
  • Group alerts by specific keys such as by user or by asset within investigations to reduce triage time
  • Create exceptions and view modification history as you would with out-of-the-box ABA detection rules
  • Attach InsightConnect automation workflows to your custom rules to mitigate manual tasks such as containing assets and enriching data, or set up notifications when detections occur
What’s New in Rapid7 Detection & Response: Q3 2023 in Review

Creating a Custom Detection Rule in InsightIDR

Enhanced Attacker Behavior Analytics (ABA) alert details in Investigations

Easily view information about your ABA alerts that are a part of an investigation with our updated Evidence panel. With these updates, you’ll see more information on alerts, including their source event data and detection rule logic that generated them. Additionally, the Evidence button has also been renamed to Alert Details to more accurately reflect its function.

New alert details include:

  • A brief description of the alert and a recommendation for triage
  • The detection rule logic that generated the alert and the corresponding key-value payload from your environment
  • The process tree, which displays details about the process that occurred when the alert was generated and the processes that occurred before and after (only for MDR customers)

Dashboard Improvements: Revamped card builder and a new heat map visualization

Our recently released revamped card builder provides more functionality to make it faster and easier to build dashboard cards. For a look at what’s new, check out the demo below.

The new calendar heat map visualization allows you to more easily visualize trends in your data over time so you can quickly spot trends and anomalies. To see this new visualization in action, check out the demo below.

Export data locally with new Log Search option

You now have more flexibility when it comes to exporting your log search data, making it easier to gather evidence related to incidents for additional searching, sharing with others in your organization, or gathering evidence associated with incidents.

With this update you can now:

  • Use edit key selection to define what columns to export to csv
  • Export results from a grouby/calculate query to a csv file

New event sources

  • Microsoft Internet Information Services (IIS): A web server that is used to exchange web content with internet users. Read the documentation
  • Amazon Security Lake: A security data lake service that allows customers to aggregate & manage security-related logs. Read the documentation
  • Salesforce Threat Detection: Uses machine learning to detect threats within a Salesforce organization. Read the documentation

A growing library of actionable detections

In Q3 2023 we added 530 new ABA detection rules. See them in-product or visit the Detection Library for actionable descriptions and recommendations.

Stay tuned!

As always, we’re continuing to work on exciting product enhancements and releases throughout the year. Keep an eye on our blog and release notes as we continue to highlight the latest in detection and response at Rapid7.

Unlock Broader Detections and Forensics with Velociraptor in Rapid7 XDR

Post Syndicated from Shanna Battaglia original https://blog.rapid7.com/2023/09/29/unlock-broader-detections-forensics-with-velociraptor-in-rapid7-xdr/

Unlock Broader Detections and Forensics with Velociraptor in Rapid7 XDR

Nearly 70% of companies that are breached are likely to get breached again within twelve months (CPO). Effective remediation and addressing attacks at the root is key to staying ahead of threats and recurring breaches on the endpoint. Strong Digital Forensics and Incident Response (DFIR) ready to go when any incident occurs is a critical piece of a security team’s toolkit and drives successful response and remediation.

With this in mind, we’re excited to announce the integration of Velociraptor, Rapid7’s leading open-source DFIR framework, into the Insight Platform for InsightIDR Ultimate users — all with no additional deployment or configurations required. Already utilized in the field by our Incident Response experts on behalf of Managed Detection and Response (MDR) customers, InsightIDR Ultimate users can now experience the power of Velociraptor, from daily threat monitoring and hunting to swift threat response.

Key benefits of Velociraptor in InsightIDR:

  • Hunt for threats and vulnerabilities on single endpoints or across the entire fleet conveniently within InsightIDR, enabling faster identification and remediation.
  • Leverage the latest security researcher and practitioner-contributed Artifacts (structured YAML files containing queries) from Velociraptor’s Exchange. This ensures you always have up-to-date coverage across the current threat landscape.
  • Monitor for threat activity as it occurs on the endpoint, and forward matching events to InsightIDR for in-depth investigations into potential threats.
  • Efficiently analyze all of your Velociraptor data inside of InsightIDR with the flexibility of custom Notebooks (used to track and post process hunts or collaborate on an investigation) or the visual navigation of the Virtual File System (a server side cache of the files on the endpoint).
  • Unlock expanded threat detection capabilities – like EVTX or ETW watcher plug-ins to monitor Windows event sources – with Velociraptor’s Client Monitoring feature and forward alert investigations into InsightIDR.
  • Get to the bottom of new threats quickly with the most up-to-date detections thanks to Velociraptor’s expressive query language (rather than code) that makes it faster and easier for teams to share custom detections with the open-source community.

Let’s walk through a potential scenario with InsightIDR and Velociraptor

Jo is a SOC analyst on a team that uses Rapid7 Insight products to monitor and respond to security incidents in a fleet of a few thousand endpoints. This morning, an email notification for a new InsightIDR investigation grabs Jo’s full attention – an endpoint just triggered an alert from a usually quiet detection rule named “Velociraptor – Alert.Windows.ETW.Powershell”, and the data in the preview immediately looks suspicious to Jo.

Jo jumps into InsightIDR to begin their investigation. The endpoint in question has only triggered this single alert so far. It was forwarded from Velociraptor, a recent addition to Jo’s toolset within the Rapid7 Insight Platform. The artifact was one of several they’d deployed to endpoints for continuous monitoring. While InsightIDR and the Sysmon integration could already detect suspicious Powershell commands when included in Process Start commandline arguments, this artifact adds visibility into Windows Powershell ETW provider for logged events.

Jo follows the link to Velociraptor, provided in the InsightIDR Investigation. The link goes directly to the Alert.Windows.ETW.Powershell event page for the endpoint that triggered the investigation. From here, Jo starts a KAPE triage collection on this endpoint. Then they start a hunt across the entire fleet for the indicators of compromise they’ve gathered so far.

Data from Jo’s collection flows into the UI for review. After noticing an unfamiliar grandparent for Powershell in the process list, Jo decides to quarantine the endpoint. They go back to their InsightIDR tab, search for the affected endpoint, and toggle its Quarantine flag. Now that endpoint can only communicate with the Rapid7 Platform, which now includes Velociraptor.  

Among the hundreds in the hunt’s scope, a handful of endpoints return matches for Jo’s initial queries. Jo recruits a few of their teammates to help triage the results and begin deeper investigation. Between Velociraptor’s powerful DFIR capabilities and the skills of Jo’s team, the intruder will be thwarted before completing their mission.  

After the incident response concludes, Jo reviews the new Velociraptor Artifacts they created to detect this new malicious activity. They decide that a few of these will be useful submissions to the Artifact Exchange. They also take the opportunity to browse other recent additions from the DIFR community and deploy a few to their Velociraptor instance.  

With Rapid7’s platform-hosted Velociraptor service, Jo’s team was able to skip another lengthy deployment process and leap right into monitoring and hunting for threats. They were also welcomed to the wider open-source Velociraptor community to share knowledge, threat intel, and DFIR techniques with the practitioners who help Velociraptor thrive.

Learn more about Velociraptor and our expanded endpoint protection.

Velociraptor 0.7.0 Release: Dig Deeper With Enhanced Client Search, Server Improvements and Expanded VQL Library

Post Syndicated from Mike Cohen original https://blog.rapid7.com/2023/08/31/untitled-7/

Velociraptor 0.7.0 Release: Dig Deeper With Enhanced Client Search, Server Improvements and Expanded VQL Library

Carlos Canto contributed to this article.

Rapid7 is thrilled to announce version 0.7.0 of Velociraptor is now LIVE and available for download.  The focus of this release was on improving user efficiency while also expanding and strengthening the library of VQL plug-ins and artifacts.

Let’s take a look at some of the interesting new features in detail.

GUI improvements

The GUI was updated in this release to improve user workflow and accessibility.

In previous versions, client information was written to the datastore in individual files (one file per client record). This works ok, as long as the number of clients is not too large and the filesystem is fast. This has become more critical as users are now deploying Velociraptor with larger deployment sizes, often in excess of 50k.

In this release, the client index was rewritten to store all client records in a single snapshot file, while managing this file in memory. This approach allows client searching to be extremely quick even for large numbers of clients well over 100k.

Additionally, it is now possible to display the total number of hits in each search giving a more comprehensive indication of the total number of clients.

Velociraptor 0.7.0 Release: Dig Deeper With Enhanced Client Search, Server Improvements and Expanded VQL Library

Paged table in Flows list

Velociraptor’s collections view shows the list of collections from the endpoint (or the server). Previously, the GUI limited this view to 100 previous collections. This meant that for heavily collected clients it was impossible to view older collections (without custom VQL).

In this release, the GUI was updated to include a paged table (with suitable filtering and sorting capabilities) so all collections can be accessed.

VQL Plugins and artifacts

Chrome artifacts

Version 0.7.0 added a leveldb parser and several artifacts around Chrome Session Storage. This allows analyzing data that is stored by Chrome locally for various web apps.

Lnk forensics

This release added a more comprehensive Lnk parser covering all known Lnk file features. You can access the Lnk file analysis using the `Windows.Forensics.Lnk` artifact.

Direct S3 accessor

Velociraptor’s accessors provide a way to apply the many plugins that operate on files to other domains. In particular, the glob() plugin allows searching the accessors for filename patterns.

In this release, Velociraptor adds an Amazon S3 accessor. This allows plugins to directly operate on S3 buckets. In particular the glob() plugin can be used to query bucket contents and read files from various buckets. This capability opens the door for sophisticated automation around S3 buckets.

Volume Shadow Copies analysis

Windows Volume Shadow Service (VSS) is used to create snapshots of the drive at a specific point in time. Forensically, this can be very helpful as it captures a point-in-time view of the previous disk state (If the VSS is still around when we perform our analysis).

Velociraptor provides access to the different VSS volumes via the ntfs accessor, and many artifacts previously provided the ability to report files that differed between VSS snapshots.

In the 0.7.0 release, Velociraptor adds the ntfs_vss accessor. This accessor automatically considers different snapshots and deduplicates files that are identical in different snapshots. This makes it much easier to incorporate VSS analysis into your artifacts.

The SQLiteHunter project

Many artifacts consist of parsing SQLite files. For example, major browsers use SQLite files heavily. This release incorporates the SQLiteHunter artifact.

SQLiteHunter is a one stop shop for finding and analyzing SQLite files such as browser artifacts and OS internal files. Although the project started with SQLite files, it now automates a lot of artifacts such as WebCacheV01 parsing and the Windows Search Service – aka Windows.edb (which are ESE based parsers).

This one artifact combines and makes obsolete many distinct older artifacts.

More info can be found at https://github.com/Velocidex/SQLiteHunter.

Glob plugin improvements

The glob() plugin may be the most used plugin in VQL, as it allows for the efficient search of filenames in the filesystem. While the glob() plugin can accept a list of glob expressions so the filesystem walk can be optimized as much as possible, it was previously difficult to know why a particular reported file was chosen.

In this release, the glob() plugin reports the list of glob expressions that caused the match to be reported. This allows callers to more easily combine several file searches into the same plugin call.

URL style paths

In very old versions of Velociraptor, nested paths could be represented as URL objects. Until now, a backwards compatible layer was used to continue supporting this behavior. In the latest release, URL style paths are no longer supported.  Instead, use the pathspec() function to build proper OSPath objects.

Server improvements

Velociraptor offers automatic use of Let’s Encrypt certificates. However, Let’s Encrypt can only issue certificates for port 443. This means that the frontend service (which is used to communicate with clients) has to share the same port as the GUI port (which is used to serve the GUI application). This makes it hard to create firewall rules to filter access to the frontend and not to the GUI when used in this configuration.

In the 0.7.0 release, Velociraptor offers the GUI.allowed_cidr option. If specified, the list of CIDR addresses will specify the source IP acceptable to the server for connections to the GUI application (for example 192.168.1.0/24).

This filtering only applies to the GUI and forms an additional layer of security protecting the GUI application (in addition to the usual authentication methods).

Better handling of out of disk errors

Velociraptor can collect data very quickly and sometimes this results in a full disk. Previously, a full disk error could cause file corruption and data loss. In this release, the server monitors its free disk level and disables file writing when the disk is too full. This avoids data corruption when the disk fills up. When space is freed the server will automatically start writing again.

The offline collector

The offline collector is a pre-configured binary which can be used to automatically collect any artifacts into a ZIP file and optionally upload the file to a remote system like a cloud bucket or SMB share.

Previously, Velociraptor would embed the configuration file into the binary so it only needed to be executed (e.g. double clicked). While this method is still supported on Windows, it turned out that on MacOS this is no longer supported as binaries can not be modified after build. Even on Windows, embedding the configuration will invalidate the signature.

In this release, we added a generic collector:

Velociraptor 0.7.0 Release: Dig Deeper With Enhanced Client Search, Server Improvements and Expanded VQL Library

This collector will embed the configuration into a shell script instead of the Velociraptor binary. Users can then launch the offline collector using the unmodified official binary by specifying the --embedded_config flag:

velociraptor-v0.7.0-windows-amd64.exe -- --embedded_config Collector_velociraptor-collector

Velociraptor 0.7.0 Release: Dig Deeper With Enhanced Client Search, Server Improvements and Expanded VQL Library

While the method is required for MacOS, it can also be used for Windows in order to preserve the binary signature.

Conclusions

There are many more new features and bug fixes in the 0.7.0 release. If you’re interested in any of these new features, we welcome you to take Velociraptor for a spin by downloading it from our release page. It’s available for free on GitHub under an open-source license.

As always, please file bugs on the GitHub issue tracker or submit questions to our mailing list by emailing [email protected]. You can also chat with us directly on our Discord server.

Learn more about Velociraptor by visiting any of our web and social media channels below:

Finally, don’t forget to register for VeloCON 2023, taking place on Wednesday September 13, 2023.  VeloCON is a one-day virtual event which includes fascinating discussions, tech talks and the opportunity to get to know real members of the Velociraptor community.  It’s a forum to share experiences in using and developing Velociraptor to address the needs of the wider DFIR landscape and an opportunity to take a look ahead at the future of our platform.

Click here for more details and to register for the event.

Join us for VeloCON 2023: Digging Deeper Together!

Post Syndicated from Carlos Canto original https://blog.rapid7.com/2023/08/17/join-us-for-velocon-2023-digging-deeper-together/

September 13, 2023 at 9 am ET

Join us for VeloCON 2023: Digging Deeper Together!

Rapid7 is thrilled to announce that the 2nd annual VeloCON: Digging Deeper Together virtual summit will be held this September 13th at 9 am ET. Once again, the conference will be online and completely free!

VeloCON is a one-day event focused on the Velociraptor community. It’s a place to share experiences in using and developing Velociraptor to address the needs of the wider DFIR community and an opportunity to take a look ahead at the future of our platform.

This year’s event calls for even more of the stimulating and informative content that made last year’s VeloCON so much fun. Don’t miss your chance at being a part of the marquee event of the open-source DFIR calendar.

Registration is now OPEN!  Click here to register and get event updates and start time reminders.

Last year’s event was a tremendous success, with over 500 unique participants enjoying fascinating discussions, tech talks and the opportunity to get to know real members of our own community.

Leading Edge Panel

Rapid7 and the Velociraptor team have invited industry leading DFIR professionals, community advocates and thought leaders to host an exciting presentation panel.  Proposals underwent a thorough review process to select presentations of maximum interest to VeloCON attendees and the wider Velociraptor community.

VeloCON focuses on work that pushes the envelope of what is currently possible using Velociraptor. Potential topics to be addressed by the panel include, but are not limited to:

  • Use cases of Velociraptor in real investigations
  • Novel deployment modes to cater for specific requirements
  • Contributions to Velociraptor to address new capabilities
  • Potential future ideas and features that Velociraptor
  • Integration of Velociraptor with other tools/frameworks
  • Analysis and acquisition on novel Forensic Artifacts

Register Today

Please register for VeloCON 2023 by following this link.  You’ll be able to preview panelist bios as well as receive email confirmations and reminders as we get closer to the event.

Learn more about Velociraptor by visiting any of our web and social media channels below:

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode

Post Syndicated from Mike Cohen original https://blog.rapid7.com/2023/06/07/velociraptor-0-6-9-release-digging-even-deeper-with-smb-support-azure-storage-and-lockdown-server-mode/

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode

Carlos Canto contributed to this article.

Rapid7 is very excited to announce version 0.6.9 of Velociraptor is now LIVE and available for download.  Much of what went into this release was about expanding capabilities and improving workflows.

We’ll now explore some of the interesting new features in detail.

GUI Improvements

The GUI was updated in this release to improve user workflow and accessibility.

Table Filtering and Sorting
Previously, table filtering and sorting required a separate dialog. In this release, the filtering controls were moved to the header of each column making it more natural to use.

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
Filtering tables

VFS GUI Improvements

The VFS GUI allows the user to collect files from the endpoint in a familiar tree-based user interface. In previous versions, it was only possible to schedule a single download at a time. This proved problematic when the client was offline or transferring a large file, because the user had no way to kick off the next download until the first file was fully fetched.

In this release, the GUI was revamped to support multiple file downloads at the same time. Additionally it is now possible to schedule a file download by right clicking the download column in the file table and selecting “Download from client”.

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
Initiating file download in the VFS. Note multiple files can be scheduled at the same time, and the bottom details pane can be closed

Hex Viewer and File Previewer GUI

In release 0.6.9, a new hex viewer was introduced. This viewer makes it possible to quickly triage uploaded files from the GUI itself, implementing some common features:

  1. The file can be viewed as a hex dump or a strings-style output.
  2. The viewer can go to an arbitrary offset within the file, or page forward or backwards.
  3. The viewer can search forward or backwards in the file for a Regular Expression, String, or a Hex String.The hex viewer is available for artifacts that define a column of type preview_uploads including the File Upload table within the flow GUI.
Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
The hex viewer UI can be used to quickly inspect an uploaded file

Artifact Pack Import GUI Improvements

Velociraptor allows uploading an artifact pack – a simple Zip file containing artifact definitions. For example, the artifact exchange is simply a zip file with artifact definitions.

Previously, artifact packs could only be uploaded in their entirety and always had an “Exchange” prefix prepended. However, in this release the GUI was revamped to allow only some artifacts to be imported from the pack and to customize the prefix.

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
It is now possible to import specific artifacts in a pack

Direct SMB Support

Windows file sharing is implemented over the SMB protocol. Within the OS, accessing remote file shares happens transparently, for example by mapping the remote share to a drive using the  net use command or accessing a file name starting with a UNC path (e.g. \\ServerName\Share\File.exe).

While Velociraptor can technically also access UNC shares by using the usual file APIs and providing a UNC path, in reality this does not work because Velociraptor is running as the local System user. The system user normally does not have network credentials, so it can not map remote shares.

This limitation is problematic, because sometimes we need to access remote shares (e.g. to verify hashes, perform YARA scans etc). Until this release, the only workaround for this limitation was to install the Velociraptor user as a domain user account with credentials.

As of the 0.6.9 release, SMB is supported directly within the Velociraptor binary as an accessor. This means that all plugins that normally operate on files can also operate on a remote SMB share transparently.

Velociraptor does not rely on the OS to provide credentials to the remote share, instead credentials can be passed directly to the smb accessor to access the relevant smb server.

The new accessor can be used in any VQL that needs to use a file, but to make it easier there is a new artifact called Windows.Search.SMBFileFinder that allows for flexible file searches on an SMB share.

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
Searching a remote SMB share

Using SMB For Distributing Tools

Velociraptor can manage third-party tools within its collected artifacts by instructing the endpoint to download the tool from an external server or the velociraptor server itself.

It is sometimes convenient to download external tools from an external server (e.g. a cloud bucket) due to bandwidth considerations.

Previously, this server could only be a HTTP server, but in many deployments it is actually simpler to download external tools from an SMB share.

In this release, Velociraptor accepts an SMB URL as the Serve URL parameter within the tool configuration screen.

You can configure the remote share with read-only permissions (read these instructions for more details on configuring SMB).

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
Serving a third-party tool from an SMB server

The Offline Collector

The offline collector is a popular mode of running Velociraptor. In this mode, the artifacts to collect are pre-programmed into the collector, which stores the results in a zip file. The offline collector can be pre-configured to encrypt and upload the collection automatically to a remote server without user interaction, making it ideal for using remote agents or people to manually run the collector without needing further training.

In this release, the Velociraptor offline collector adds two more upload targets. It is now possible to upload to an SMB server and to Azure Blob Storage.

SMB Server Uploads
Because the offline collector is typically used to collect large volumes of data, it is beneficial to upload the data to a networked server close to the collected machine. This avoids cloud network costs and bandwidth limitations. It works very well in air gapped networks, as well.

You can now simply create a new share on any machine, by adding a local Windows user with password credentials, exporting a directory as a share, and adjusting the upload user’s permissions to only be able to write on the share and not read from it. It is now safe to embed these credentials in the offline collector, which can upload data but cannot read or delete other data.

Read the full instructions of how to configure the offline collector for SMB uploads.

Azure Blob Storage Service
Velociraptor can now upload collections to an Amazon S3 or Google Cloud Storage bucket. Many users requested direct support for Azure blob storage, which is now in 0.6.9.

Read about how to configure Azure for safe uploads. Similar to the other methods, credentials embedded in the offline collector can only be used to upload data and not read or delete data in the storage account.

Debugging VQL Queries

One of the points of feedback we received from our annual user survey was that although VQL is an extremely powerful language, users struggle with debugging and understanding how the query proceeds.

Unlike a more traditional programming language (e.g. Python), there is no debugger that allows users to pause execution and inspect variables, or add print statements to see what data is passed between parts of the query.

We took this feedback to heart and in release 0.6.9 the EXPLAIN keyword was introduced. The EXPLAIN keyword can be added before any SELECT in the VQL statement to place that SELECT statement into tracing mode.

As a recap, the general syntax of the VQL statement is:

SELECT vql_fun(X=1, Y=2), Foo, Bar
FROM plugin(A=1, B=2)
WHERE X = 1 

When a query is in tracing mode:

  1. All rows emitted from the plugin are logged with their types
  2. All parameters into any function are also logged
  3. When a row is filtered because it did not pass the WHERE clause this is also logged

This additional tracing information can be used to understand how data flows throughout the query.

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
Explaining a query reveals details information on how the VQL engine handles data flows

You can use the EXPLAIN statement in a notebook or within an artifact as collected from the endpoint (although be aware that it can lead to extremely verbose logging).

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
Inspect the details by clicking on the logs button

For example in the above query we can see:

The clients() plugin generates a row

  1. The timestamp() function received the last_seen_at value
  2. The WHERE condition rejected the row because the last_seen_at time was more than 60 seconds ago

Locking Down The Server

Another concern raised in our survey was the perceived risk of having Velociraptor permanently installed due to its high privilege and efficient scaling.

While this risk is not higher than any other domain-wide administration tool, in some deployment scenarios, Velociraptor does not need this level of access all the time. While in an incident response situation, however, it is necessary to promote Velociraptor’s level of access easily.

In the 0.6.9 release, Velociraptor has introduced lock down mode. When a server is locked down certain permissions are removed (even from administrators). The lockdown is set in the config file, helping to mitigate the risk of a Velociraptor server admin account compromise.

After initial deployment and configuration, the administrator can set the server in lockdown by adding the following configuration directive to the server.config.yaml and restarting the server:

lockdown: true

After the server is restarted the following permissions will be denied:

  • ARTIFACT_WRITER
  • SERVER_ARTIFACT_WRITER
  • COLLECT_CLIENT
  • COLLECT_SERVER
  • EXECVE
  • SERVER_ADMIN
  • FILESYSTEM_WRITE
  • FILESYSTEM_READ
  • MACHINE_STATE

Therefore it will still be possible to read existing collections, and continue collecting client monitoring data, but it will not be possible to edit artifacts or start new hunts or collections.

During an active IR, the server may be taken out of lockdown by removing the directive from the configuration file and restarting the service. Usually, the configuration file is only writable by root and the Velociraptor server process is running as a low privilege account that can not write to the config file. This combination makes it difficult for a compromised Velociraptor administrator account to remove the lockdown and use Velociraptor as a lateral movement vehicle.

Audit Events

Velociraptor maintains a number of log files over its operation, normally stored in the <filestore>/logs directory. While the logs are rotated and separated into different levels, the most important log type is the audit log which records auditable events. Within Velociraptor auditable events are security critical events such as:

  • Starting a new collection from a client
  • Creating a new hunt
  • Modifying an artifact
  • Updating the client monitoring configuration

Previous versions of Velociraptor simply wrote those events to the logging directory. However, the logging directory can be deleted if the server becomes compromised.

In 0.6.9 there are two ways to forward auditable events off the server

  1. Using remote syslog services
  2. Uploading to external log management systems e.g. Opensearch/Elastic using the Elastic.Events.Upload artifact.Additionally, auditable events are now emitted as part of the Server.Audit.Logs artifact so they can be viewed or searched in the GUI by any user.
Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
The server’s audit log is linked from the Welcome page
Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
Inspecting user activity through the audit log

Because audit events are available now as part of the server monitoring artifact, it is possible for users to develop custom VQL server monitoring artifacts to forward or respond to auditable events just like any other event on the client or the server. This makes it possible to forward events (e.g. to Slack or Discord) as demonstrated by the `Elastic.Events.Upload` artifact above.

Tool Definitions Can Now Specify An Expected Hash

Velociraptor supports pushing tools to external endpoints. A Velociraptor artifact can define an external tool, allowing the server to automatically fetch the tool and upload it to the endpoint.

Previously, the artifact could only specify the URL where the tool should be downloaded from. However, in this release, it is also possible to declare the expected hash of the tool. This prevents potential substitution attacks effectively by pinning the third-party binary hash.

While sometimes the upstream file may legitimately change (e.g. due to a patch), Velociraptor will not automatically accept the new file when the hash does not match the expected hash.

Velociraptor 0.6.9 Release: Digging Even Deeper with SMB Support, Azure Storage and Lockdown Server Mode
Mismatched hash

In the above example we modified the expected hash to be slightly different from the real tool hash. Velociraptor refuses to import the binary but provides a button allowing the user to accept this new hash instead. This should only be performed if the administrator is convinced the tool hash was legitimately updated.

Conclusions

There are many more new features and bug fixes in the 0.6.9 release. If you’re interested in any of these new features, we welcome you to take Velociraptor for a spin by downloading it from our release page. It’s available for free on GitHub under an open-source license.

As always, please file bugs on the GitHub issue tracker or submit questions to our mailing list by emailing [email protected]. You can also chat with us directly on our Discord server.

Learn more about Velociraptor by visiting any of our web and social media channels below:

If you want to master Velociraptor, consider joining us at a week-long Velociraptor training course held this year at the BlackHat USA 2023 Conference and delivered by the Velociraptor developers themselves.Details are here: https://docs.velociraptor.app/announcements/2023-trainings/

VeloCON 2023: Submissions Wanted!

Post Syndicated from Carlos Canto original https://blog.rapid7.com/2023/05/23/velocon-2023-submissions-wanted/

VeloCON 2023: Submissions Wanted!

Rapid7 is thrilled to announce that the 2nd annual VeloCON virtual summit will be held this September (date TBD), with times oriented to the continental USA time zones. Once again, the conference will be online and completely free!

VeloCON is a one-day event focused on the Velociraptor community. It’s a place to share experiences in using and developing Velociraptor to address the needs of the wider DFIR community and an opportunity to take a look ahead at the future of our platform.

This year’s event calls for even more of the stimulating and informative content that made last year’s VeloCON so much fun. Don’t miss your chance at being a part of this year’s marquee event of the open-source DFIR calendar.

The call for presentations closes Monday, July 17, 2023 (see details below).

Last year’s event was a tremendous success, with over 500 unique participants enjoying our lineup of fascinating discussions, tech talks and the opportunity to get to know real members of our own community.

Call for presentations (CFP)

VeloCON invites contributions in the form of a 30-45 minute presentation. We require a brief proposal (~500 words; not a paper). These proposals undergo a review process to select presentations of maximum interest to VeloCON attendees and the wider Velociraptor community and to filter out sales pitches.

VeloCON focuses on work that pushes the envelope of what is currently possible using Velociraptor. Potential topics to be addressed by submissions include, but are not limited to:

  • Use cases of Velociraptor in real investigations
  • Novel deployment modes to cater for specific requirements
  • Contributions to Velociraptor to address new capabilities
  • Potential future ideas and features that Velociraptor
  • Integration of Velociraptor with other tools/frameworks
  • Analysis and acquisition on novel Forensic Artifacts

Submission process

Please email your submission to [email protected] and include the following details:

  1. Your name and email address (if different from the sending email)
  2. Company/affiliation and title to be included on the agenda
  3. Presentation title
  4. A short abstract (~500 words) to be included in the agenda

Deadline

Submissions are due Monday, July 17, 2023 and a decision will be announced shortly afterwards.

The Velociraptor 2023 Annual Community Survey

Post Syndicated from Rapid7 original https://blog.rapid7.com/2023/05/10/the-velociraptor-2023-annual-community-survey/

The Velociraptor 2023 Annual Community Survey

By Dr. Mike Cohen & Carlos Canto

Velociraptor is an open-source project led and shaped by the community. Over the years, Velociraptor has become a real force in the field of DFIR, making it an obvious choice for many operational situations. Rapid7 is committed to continue making Velociraptor the premier open-source DFIR and security tool.

To learn more about how the tool is used in the community and what the community expectations are with regard to capabilities, features, and use cases the Velociraptor team distributed our first community survey in early 2023. We are using this information in order to shape future development direction, set priorities and develop our road map. We are grateful to the community members who took the time to respond.

As an open-source project, we depend on our community to contribute. There are many ways contributors can help the project, from developing code, to filing bugs, to improving documentation. One of the most important ways users can contribute is by providing valuable feedback through channels such as this survey, which helps to shape the future road map and new features.

We’re excited to share some of the responses we received in this blog post.

Who is the Velociraptor community?

Of the 213 survey respondents, the majority were analysts (57%) and managers (26%), indicating that most of the respondents are people who know and use Velociraptor frequently.

We also wanted to get a feel for the type of companies using Velociraptor. Users fell pretty evenly into company sizes, with about 30% of responses from small companies (less than 100 employees) and 20% of responses from very large companies of 10,000 employees or more.

These companies also came from a wide range of industries. While many were primarily in the information security fields such as managed security service providers (MSSPs), consultants, and cybersecurity businesses, we also saw a large number of responses from the government sector, the aerospace industries, education, banking/finance, healthcare, etc.

With such a wide range of users, we were interested in how often they use Velociraptor. About a third said they use Velociraptor frequently, another third use it occasionally, and the final third are in the process of evaluating and learning about the tool.

Velociraptor use cases

Velociraptor is a powerful tool with a wide feature set. We wanted to glimpse an idea of what features were most popular and how users prioritize these features. Specifically, we asked about the following main use cases:

Client monitoring and alerts (detection)
Velociraptor can collect client event queries focused on detection. This allows the client to autonomously monitor the endpoint and send back prioritized alerts when certain conditions are met.

→ 12% of users were actively using this feature to monitor endpoints.

Proactively hunting for indicators (threat intelligence)
Velociraptor’s unique ability to collect artifacts at scale from many systems can be combined with threat-intelligence information (such as hashes, etc.) to proactively hunt for compromises by known actors. This question was specifically related to hunting for threat-feed indicators, such as hashes, IP addresses, etc.

→ 16% of users were utilizing this feature.

Ongoing forwarding of events to another system
Velociraptor’s client monitoring queries can be used to simply forward events (such as ETW feeds).

→ 6% of users were utilizing this feature.

Collecting bulk files for analysis on another system (digital forensics)
Velociraptor can be used to collect bulk files from the endpoint for later analysis by other tools (for example, using the Windows.Collection.KapeFiles artifact).

→ 20% of users were using this feature regularly.

Parsing for indicators on the endpoint (digital forensics)
Velociraptor’s artifacts are used to directly parse files on the endpoint, quickly returning actionable high-value information without the need for lengthy post processing.

→ 21% of users use these types of queries.

Proactive hunting for indicators across many systems (incident response)
Velociraptor can hunt for artifacts from many endpoints at once.

→ 21% of users benefit from this capability.

We further asked for the relative importance of these features. Users most valued the ability to collect bulk files and hunt for artifacts across many systems, followed by the ability to directly parse artifacts on the endpoints.

Backwards compatibility

Some users deployed Velociraptor for limited-time engagements so they did not need backwards compatibility for stored data, as they wouldn’t be upgrading to major versions within the same deployment.

Other users required more stable data migration but were generally happy with removing backwards data compatibility, if necessary. For example, one response stated “I would rather you prioritize improvements over compatibility even if it breaks things.”

Another user explained: “In a typical Incident Response scenario, Digital Forensics data has a shelf life of a few weeks or months at best and I am comfortable with the convertibility and portability of much of the data that Velociraptor collects such that archival data can still be worked with even if newer versions of the server no longer support a deprecated format/archive. I think there will be workarounds if this becomes an issue for folks with mountains of legacy data that hasn’t been exported somewhere more meaningful for longer term storage and historical data analytic/intelligence purposes.”

Generally, most users indicated they rarely or never needed to go back to archived data and reanalyze.

Version compatibility

The Velociraptor support policy officially only supports clients and servers on the same release version. However, in reality it usually takes longer to upgrade clients than servers. While some users are able to upgrade clients promptly, many users estimate between 10-50% of deployed clients are a version (or more) older than the server. Therefore, the Velociraptor team needs to maintain some compatibility with older clients to allow time for users to upgrade their endpoints.

The offline collector

The offline collector gives users a way to use Velociraptor’s artifacts without needing to deploy a server. This feature is used exclusively by about 10% of users, while a further 30% of users employ it frequently.

Most users of the offline collection deploy it manually (50%). Deploying via another EDR tool or via Group Policy are also robust options. Some users have created custom wrappers to deploy the offline collector in the field. The offline collection supports directly uploading the collection to a cloud server using a number of methods.

The most popular upload method is to an AWS S3 bucket (30%) while the SFTP connector in the cloud or a custom SFTP server on a VM are also popular options (20% and 23%, respectively). Uploading directly to Google Cloud Storage is the least popular option at about 5%.

Manual copy methods were also popular, ranging from EDR-based copying to Zoom file copy.

Azure blob storage was a common request that Velociraptor currently does not support. Many responses indicate that SFTP is currently a workaround to the lack of direct Azure support. The Velociraptor team should prioritize supporting Azure blob storage.

Data analysis

Velociraptor supports collecting raw files (e.g. Event log files, $MFT etc.) for analysis in other tools. Alternatively, Velociraptor already contains extensive parsers for most forensic artifacts that can be used directly on the endpoint.

Most users do use the built-in forensic parsing and analysis artifacts (55%) but many users also collect raw files (e.g. via the Windows.Collection.KapeFiles artifact).

VQL artifacts

Velociraptor uses the Velociraptor Query Language to perform collections and analysis. The VQL is usually shared with the community via an artifact. Most users utilize the built-in artifacts as well as the artifact exchange. However, over 60% of users report they develop their own artifacts, as well. For those users who develop their own artifacts, we asked about limitations and difficulties in this process.

A common theme that arose was around debugging artifacts and the lack of a VQL debugger and better error reporting. Training and documentation were also pointed out as needing improvement. A suggestion was made to enhance documentation with more examples of how each VQL plugin can be used in practice.

In a related note, the Velociraptor team is running a training course at BlackHat 2023. Developers will impart detailed information on how to deploy Velociraptor and write effective custom VQL.

Role-based access controls

Velociraptor has a role-based access control (RBAC) mechanism where users can be assigned roles from administrator, to investigator, to read-only access provided by the reader role. Users generally found this feature useful—40% found it “moderately useful,” 20% “very useful” and 15% “extremely useful”.The main suggestions for improvements include:

  • Easier management through the GUI (as of version 0.6.8 all user ACLs are managed through the GUI)
  • Custom roles with more granular permissions
  • Better logging and auditing
  • The ability to allow a specific role to only run a pre-approved subset of artifacts
  • A way to only run signed/hashed VQL / prevent a malicious artifact being dropped on the server
  • Making it clearer what each permission grants the user

Multi-tenant support

Velociraptor offers a fully multi-tenanted mode, where organizations can be created or decommissioned quickly with minimal resource overhead. This feature is used by 25% of respondents, who are mainly consultants and service providers using it to support multiple customers. Some companies use multi-tenancy to separate different divisions or subsidiaries of the business.

Client monitoring and alerting

Velociraptor can run event queries on a client. These VQL queries run continuously and stream results to the server when certain conditions are met. Common use cases for these are to generate alerts and enhanced detection.

Some users deploy client monitoring artifacts frequently while others see it as an alternative to EDR tools, when these are available. The primary use-case breakdown was:

  • Detection (e.g. alert when an anomalous event occurs): 27% of users
  • Collection of client events (e.g. forward process event logs to an external system): 18% of users
  • Remediation (e.g. quarantine or remove files automatically): 15% of users

→ 30% of users do not use client monitoring at all.

The most common pain point with client monitoring is the lack of integrated alerting capability (an issue currently being worked on). Some useful feedback on this feature included:

  • Better support for integration with business tools (e.g., Teams, Slack, etc.)
  • Easier to manage event data
  • Not having to build a server side artifact for each client_event artifact
  • A dashboard that lists all alerts
  • An easier way to forward alerts based on severity
  • Lack of pre-built detection rules/packs—in other words, it would be easier to tune down, than to build up

The Quarantine feature

Velociraptor can quarantine an endpoint by collecting the Windows.Remediation.Quarantine artifact. This artifact tunes the firewall rules on the endpoint to block all external network communication while maintaining connectivity to the Velociraptor host. This allows for an endpoint to be isolated during investigation.

The feature is fairly popular—it was “sometimes used” by about 30% of users and “always used” by another 12%.

How is Velociraptor deployed?

Velociraptor is a very lightweight solution, typically taking a few minutes to provision a new deployment. For many of our users, Velociraptor is used in an incident response context on an as-needed basis (46%). Other users prefer a more permanent deployment (25%).

For larger environments, Velociraptor also supports multi-server configuration (13% of users), as well as the more traditional single-server deployment option (70% of users). While some users leverage very short-lived deployments of several days or less (13%), most users keep their deployment for several weeks (27%) to months or permanently (44%).

Velociraptor is designed to work efficiently with many endpoints. We recommend a maximum of 15-20k endpoints on a single server before switching to a multi-server architecture (although users reported success with larger deployment sizes on a single server). This level of performance is adequate in practice for the majority of users.

Many users run deployments of less than 250 endpoints (44%) while a further 40% of users deploy to less than 5,000 endpoints.

Approximately 10% of users have deployment sizes larger than 25,000 endpoints, with 2% of users over 100,000 endpoints.

Popular operating systems

Among Velociraptor’s supported operating systems, Windows 64-bit is the most popular (with 82% of users ranking it the most-deployed OS type), while Linux is the next most popular deployed endpoint OS. Mac is the third popular choice for Velociraptor’s users. Finally, 32-bit Windows systems are still prevalent, as well.

Resources and references

Velociraptor’s website at https://docs.velociraptor.app/ contains a wealth of reference material, training courses, and presentations. We also have an active YouTube channel with many instructional videos.

While some users ranked the website as “extremely useful” (25%), there is clearly room for improvement. 42% of users rated it as only “very useful” or “moderately useful” (28%).Suggestions for improvements included:

  • More in-depth YouTube videos breaking down the tool’s features with workflows
  • More detailed “how to” with practical examples
  • Improved documentation about functions and plugins, with a slightly more detailed explanation and a small example
  • Updates to the documentation to reflect the new versions and features

Testimonials

Finally, I wanted to share with you some of the testimonials that users wrote in the survey. We are humbled with the encouraging and positive words we read, and are excited to be making an impact on the DFIR field:

  • "I have to congratulate you and thank you for developing such an amazing tool. It’s the future of DFIR."
  • "Awesome product, can’t wait to use it in prod!"
  • "This is a game-changer for the DFIR industry. Keep up the great work."
  • "Keep the file system based backend, its simplicity makes chain of custody/court submissions possible."
  • "I thoroughly love Velociraptor. The team and community are absolutely fantastic. I would go as far as to say that Mike and Matthew Green are my favorite infosec gentlemen in the industry."
  • "Y’all are awesome. I feel like I was pretty critical, but that’s because this is an amazing software, and I want to see it continue to grow and improve."
  • "We have been deploying Velociraptor to client environments almost since it was released. Our DFIR business model is entirely centered around it and it works very well for us. It is a great solution that just keeps getting better and better."

Conclusions

This is our first Velociraptor community survey, and it has proven to be extremely useful. Since Velociraptor is a community-led, open-source project, we need an open feedback loop to our users. This helps us understand where things need improvement and which features should be prioritized.

At the same time, since Velociraptor is an open-source project, I hope this survey will inspire contributions from the community. We value all contributions, from code to documentation, testing, and bug reports.

Finally, for all of our US-based users, we hope to see you all in person this year at BlackHat 2023! Join us for an in-depth Velociraptor training and to geek out with VQL for 4 days, learning practical, actionable skills and supporting this open-source project.

Keep Digging!

Velociraptor Version 0.6.8 Available Now

Post Syndicated from Carlos Canto original https://blog.rapid7.com/2023/03/30/velociraptor-version-0-6-8-available-now/

A New Client-Server Communication Protocol, VFS GUI, and More Performance Upgrades Make This The Fastest and Most Scalable Velociraptor Yet

Velociraptor Version 0.6.8 Available Now

Rapid7 is excited to announce the release of version 0.6.8 of Velociraptor—an advanced, open-source digital forensics and incident response (DFIR) tool that enhances visibility into your organization’s endpoints. This release has been in development and testing for several months and features significant contributions and testing from our community. We are thrilled to share its powerful new features and improvements here today.

Performance Improvements

A big theme in the 0.6.8 release was about performance improvement, making Velociraptor faster, more efficient and more scalable (even more so than it currently is!).

New Client-Server Communication Protocol

When collecting artifacts from endpoints Velociraptor maintains a collection state (e.g. how many bytes were transferred?, how many rows? was the collection successful? etc). Previously tracking the collection was the task of the server, but this extra processing limited the total number of collections it could process.

In the 0.6.8 release, a new communication protocol was added to offload a lot of the collection tracking to the client itself. This reduces the amount of work on the server and allows more collections to be processed at the same time.

To maintain support with older clients, the server continues to use the older communication protocol with them—but will achieve the most improvement in performance once the newer clients are deployed.

New Virtual File System GUI

The VFS feature in Velociraptor allows users to interactively inspect directories and files on the endpoint, in a familiar tree-style user interface. The previous VFS view would store the entire directory listing in a single table for each directory. For very large directories like C:\Windows or C:\Windows\System32 (which typically have thousands of files) this would strain the browser leading to unusable UI.

In the latest release, the VFS GUI uses the familiar paged table and syncs this directory listing in a more efficient way. This improves performance significantly: for example, it is now possible and reasonable to perform a recursive directory sync on C:\Windows, on my system syncs over 250k files in less than 90 seconds.

Velociraptor Version 0.6.8 Available Now
Inspecting a large directory is faster with paging tables.


Since the VFS is now using the familiar paging table UI, it is also possible to filter, sort on any column using that same UI.

Faster Export Functionality

Velociraptor hunts and collections can be exported to a ZIP file for easy consumption in other tools. The 0.6.8 release improved the export code to make it much faster. Additionally, the GUI was improved to show how many files were exported into the zip, and other statistics.

Velociraptor Version 0.6.8 Available Now
Exporting collections is much faster!


Tracing Capability On Client Collections

We often get questions about what happened to a collection that seems to be hung? It is difficult to know why a collection seems to be unresponsive or stopped – it could mean the client was killed for some reason, (e.g. due to excessive memory use or a timeout).

Previously the only way to gather client side information was to collect a Generic.Client.Profile collection. This required running it at just the right time and did not guarantee that we would get helpful insight of what the query and the client binary were doing during the operation in question.
In the latest release it is now possible to specify a trace on any collection to automatically collect client side state as the collection is progressing.

Velociraptor Version 0.6.8 Available Now
Enabling trace on every collection increases visibility


Velociraptor Version 0.6.8 Available Now
Trace files contain debugging information


VQL Improvement – Disk Based Materialize Operator

The VQL LET ... <= operator is called the materializing LET operator because it expands the following query into a memory array which can be accessed cheaply multiple times.

While this is useful for small queries, it has proved dangerous in some cases, because users inadvertently attempted to materialize a very large query (e.g. a large glob() operation) dramatically increasing memory use. For example, the following query could cause problems in earlier versions.

LET X <= SELECT * FROM glob(globs=specs.Glob, accessor=Accessor)

In the latest release the VQL engine was improved to support a temp file based materialized operator. If the materialized query exceeds a reasonable level (by default 1000 rows), the engine will automatically switch away from memory based storage into file backed storage. Although file based storage is slower, memory usage is more controlled.

Ideally the VQL is rewritten to avoid this type of operation, but sometimes it is unavoidable, and in this case, file based materialize operations are essential to maintain stability and acceptable memory consumption.

New MSI Deployment Option

On Windows the recommended way to install Velociraptor is via an MSI package. The MSI package allows the software to be properly installed and uninstalled and it is also compatible with standard Windows software management procedures.

Previously however, building the MSI required using the WIX toolkit – a Windows only MSI builder which is difficult to run on other platforms. Operationally building with WIX complicates deployment procedures and requires using a complex release platform.

In the 0.6.8 release, a new method for repacking the official MSI package is now recommended. This can be done on any operating system and does not require WIX to be installed. Simply embed the client configuration file in the officially distributed MSI packages using the following command:

velociraptor-v0.6.8-rc1-linux-amd64 config repack --msi 
velociraptor-v0.6.8-rc1-windows-amd64.msi client.config.yaml 
output.msi

Velociraptor Version 0.6.8 Available Now
Repacking an MSI for windows distribution


Conclusion

If you’re interested in any of these new features, we welcome you to take Velociraptor for a spin by downloading it from our release page. It’s available for free on GitHub under an open-source license.
As always, please file bugs on the GitHub issue tracker or submit questions to our mailing list by emailing [email protected]. You can also chat with us directly on our Discord server.

Learn more about Velociraptor by visiting any of our web and social media channels below:

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever

Post Syndicated from Mike Cohen original https://blog.rapid7.com/2022/12/02/velociraptor-version-0-6-7-better-offline-collection-encryption-and-an-improved-ntfs-parser-dig-deeper-than-ever/

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever

By Mike Cohen and Carlos Canto

Rapid7 is excited to announce the release of version 0.6.7 of Velociraptor – an advanced, open-source digital forensics and incident response (DFIR) tool that enhances visibility into your organization’s endpoints. This release has been in development and testing for several months and features significant contributions from our community.  We are thrilled to share its powerful new features and improvements.

NTFS Parser changes

In this release, the NTFS parser was improved significantly. The main areas of development focused on better support for NTFS compressed and sparse files as well as improved path reconstruction.

In NTFS, there is a Master File Table (MFT) containing a record for each file on the filesystem. The MFT entry describes a file by attaching several attributes to it. Some of these are $FILE_NAME attributes representing the names of the file.

In NTFS, a file may have multiple names. Normally, files have a long file name and a short filename. Each $FILE_NAME record also contains a reference to the parent MFT entry of its directory.

When Velociraptor parses the MFT, it attempts to reconstruct the full path of each entry by traversing the parent MFT entry, recovering its name, etc. Previously, Velociraptor used one of the $FILE_NAME records (usually the long file name) to determine the parent MFT entry. However, this is not strictly correct, as each $FILE_NAME record can be a different parent directory. This surprising property of NTFS is called hard links.

You can play with this property using the fsutil program. The following adds a hard link to the program at C:/users/test/downloads/X.txt into a different directory.

C:> fsutil hardlink create c:\Users\Administrator\Y.txt c:\Users\Administrator\downloads\X.txtHardlink created for c:\Users\Administrator\Y.txt <<===>> c:\Users\Administrator\downloads\X.txt

The same file in NTFS can exist in multiple directories at the same time by use of hard links. The filesystem simply adds a new $FILE_NAME entry to the MFT entry for the file pointing at another parent directory MFT entry.

Therefore, when scanning the MFT, Velociraptor needs to report all possible directories in which each MFT entry can exist – there can be many such directories, since each directory can have its own hard links.

As a rule, an MFT Entry can represent many files in different directories!

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever
An example of the notepad MFT entry with its many hard links

Reassembling paths from MFT entries

When Velociraptor attempts to reassemble the path from an unallocated MFT entry, it might encounter an error where the parent MFT entry indicated has already been used for some other file or directory.

In previous versions, Velociraptor simply reported these parents as potential parts of the full path, since – for unallocated entries – the path reconstruction is best effort. This led to confusion among users with often nonsensical paths reported for unallocated entries.

In the latest release, Velociraptor is more strict in reporting parents of unallocated MFT entries, also ensuring that the MFT sequence numbers match. If the parent’s MFT entry sequence number does not match, Velociraptor’s path reconstruction indicates this as an error path.

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever
Unallocated MFT entries may have errors reconstructing a full path

In the above example, the parent’s MFT entry has a sequence number of 5, but we need a sequence number of 4 to match it. Therefore, the parent’s MFT entry is rejected and instead we report the error as the path.

The offline collection and encryption

Velociraptor’s offline collector is a pre-configured Velociraptor binary, which is designed to be a single shot acquisition tool. You can build an Offline Collector by following the documentation. The Offline Collector does not require access to the server, instead simply collecting the specified artifacts into a zip file (which can subsequently be uploaded to the cloud or simply shared with the DFIR experts for further analysis).

Previously, Velociraptor only supported encrypting the zip archive using a password. This is problematic because the password had to be embedded inside the collector configuration and so could be viewed by anyone with access to the binary.

In the latest release, Velociraptor supports asymmetric encryption to protect the acquisition zip file. There are two asymmetric schemes: X509 encryption and PGP encryption. Having asymmetric encryption improves security greatly because only the public key needs to be included in the collector configuration. Dumping the configuration from the collection is not sufficient to be able to decrypt the collected data – the corresponding private key is also required!

This is extremely important for forensic collections since these will often contain sensitive and PII information.

Using this new feature is also extremely easy: One simply selects the X509 encryption scheme during the configuration of the offline collector in the GUI.

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever
Configuring the offline collector for encryption

You can specify any X509 certificate here, but if you do not specify any, Velociraptor will use the server’s X509 certificate instead.

Velociraptor will generate a random password to encrypt the zip file, and then encrypt this password using the X509 certificate.

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever
The resulting encrypted container

Since the ZIP standard does not encrypt the file names, Velociraptor embeds a second zip called data.zip inside the container. The above illustrates the encrypted data zip file and the metadata file that describes the encrypted password.

Because the password used to encrypt the container is not known and needs to be derived from the X509 private key, we must use Velociraptor itself to decrypt the container (i.e. we can not use something like 7zip).

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever
Decrypting encrypted containers with the server&rsquo;s private key

Importing offline collections

Originally, the offline collector feature was designed as a way to collect the exact same VQL artifacts that Velociraptor allows in the usual client-server model in situations where installing the Velociraptor client was not possible. The same artifacts can be collected into a zip file.

As Velociraptor’s post processing capabilities improved (using notebooks and server side VQL to enrich the analysis), people naturally wanted to use Velociraptor to post process offline collections too.

Previously, Velociraptor did have the Server.Utils.ImportCollection artifact to allow an offline collection to be imported into Velociraptor. But this did not work well because the offline collector simply did not include enough information in the zip file to sufficiently emulate the GUI’s collection views.

In the recent release, the offline collector was updated to add more detailed information to the collection zip, allowing it to be easily imported.

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever
Exported zip archives now contain more information

Exporting and importing collections

Velociraptor has previously had the ability to export collections and hunts from the GUI directly, mainly so they can be processed by external tools.

But there was no way to import those collections back into the GUI. We just never imagined this would be a useful feature!

Recently, Eric Capuano from ReconInfosec shared some data from an exercise using Velociraptor. People wanted to import into their own Velociraptor installations so they could run notebook post processing on the data themselves.

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever
The OpenSoc challenge: https://twitter.com/eric_capuano/status/1559190056736378880

Our community has spoken though! This is a useful feature!

In the latest release, exported files from the GUI use the same container format at the offline collector, and therefore can be seamlessly imported into a different Velociraptor installation.

Handling of sparse files

When collecting files from the endpoint using the NTFS accessor, we quite often encounter sparse files. These are files with large unallocated holes in them. The most extreme sparse file is the USN Journal.

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever
Acquiring the USN journal

In the above example, the USN journal size is reported to be 1.3 GB but in reality only about 40 MB is occupied on disk. When collecting this file, Velociraptor only collects the real data and marks the file as sparse. The zip file will contain an index file which specifies how to reassemble the file into its original form.

While Velociraptor stores the file internally in an efficient way, when exporting the file for use by other tools, they might expect the file to be properly padded out (so that file offsets are correct).

Velociraptor now allows the user the choice of exporting an individual file in a padded form (with sparse regions padded). This can also be applied to the entire zip export in the GUI.

For very large sparse files, it makes no sense to pad so much data out – some USN journal files are in the TB region. So, Velociraptor implements a limit on padding of very sparse files.

Parsing user registry hives

Many Velociraptor artifacts simply parse keys and values from the registry to detect indicators. Velociraptor offers two methods of accessing the registry:

  1. Using the Windows APIs
  2. Employing the built-in raw registry parser to parse the hive files

While the first method is very intuitive and easy to use, it is often problematic. Using the APIs requires the user hive to be mounted. Normally, the user hive is only mounted when a user logs in. Therefore querying registry keys in the user hive will only work on users that are currently logged in at the time of the check and miss other users (which are not currently logged in so their hive is not mounted).

To illustrate this problem consider the Windows.Registry.Sysinternals.Eulacheck artifact which checks the keys in HKEY_USERS\*\Software\Sysinternals\* for the Sysinternals EULA value.

In previous versions of Velociraptor, this artifact simply used the windows API to check these keys/values and completely missed any users that were not logged in.

While this issue is known, users previously had to employ complex VQL to customize the query so it could search the raw NTUSER.DAT files in each user registry. This is more difficult to maintain since it requires two separate types of artifact for the same indicator.

With the advent of Velociraptor’s dead disk capabilities, it is possible to run a VQL query in a “virtualized” context consisting of a remapped environment. The end result is that the same VQL query can be used to run on raw registry hives. It is now trivial to apply the same generic registry artifact to a raw registry parse.

Velociraptor Version 0.6.7: Better Offline Collection, Encryption, and an Improved NTFS Parser Dig Deeper Than Ever
Remapping the raw registry hive to a regular registry artifact

All that is required to add raw registry capabilities to any registry artifact is:

  1. Import the Windows.Registry.NTUser artifact
  2. Use the MapRawRegistryHives helper function from that artifact to set up the mappings automaticallyCall the original registry query using the registry accessor. In the background this will be remapped to the raw registry accessor automatically

Conclusion

If you’re interested in the new features, take Velociraptor for a spin by downloading it from our release page. It’s available for free on GitHub under an open-source license.

As always, please file bugs on the GitHub issue tracker or submit questions to our mailing list by emailing [email protected]. You can also chat with us directly on our Discord server.

Learn more about Velociraptor by visiting any of our web and social media channels below: