Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-create-a-big-yellow-taxi-to-help-protect-amazon-workmail/

Your Organization’s Email May Be at Risk: Lessons from the Microsoft Exchange Online Intrusion and Guidance to Ensure Better Protection.

The recent compromise of Microsoft Exchange Online accounts serves as a stark reminder of the critical need for robust email security measures. In this high-profile incident, bad actors were able to bypass security controls and gain access to email accounts across multiple organizations. As the Cyber Safety Review Board’s investigation revealed, the successful intrusion occurred due to inadequate security controls, including insufficient audit logging and alert systems. Fortunately, the intrusion was detected by the U.S. State Department’s “Big Yellow Taxi” proactive monitoring and alert system.

This post explains how to implement a similar “Big Yellow Taxi” alert system for your Amazon WorkMail  environment using WorkMail’s audit logging capabilities. This blog will use several AWS services, including Amazon CloudWatch and Amazon Simple Notification Service (SNS), to send email alerts when unauthorized, failed access attempts are detected. This simple, near real-time alerting system provides your email and security administrators the information they need to investigate and address any potential security threats to protect your organization’s sensitive email communications.

The Microsoft Exchange Online Intrusion: A Stark Reminder

The recent Cyber Safety Review Board (CSRB) findings on the Summer 2023 Microsoft Exchange Online intrusion are deeply concerning for any organization using a cloud-hosted inbox service, including Amazon WorkMail. In this incident, the intruders leveraged a stolen Microsoft Services Account (MSA) signing key to forge authentication tokens, bypassing security controls and gaining access to the email accounts of high-profile government officials in multiple organizations. Critically, it was only the U.S. State Department’s proactive monitoring of the MailItemsAccessed log that first detected the intrusion which initiated a custom “Big Yellow Taxi” alert.

This scenario serves as a stark reminder that the most sophisticated email platforms can be vulnerable to determined threat actors. WorkMail is built on top of Amazon’s Simple Email Service (SES) and the core security architecture and practices of AWS, which help to ensure the security of your WorkMail Organization. However, it is up to you to ensure organization’s sensitive communications are not put at risk by following the best practice security measures called out in the documentation and blog posts, including:

• Take proper security measures, per the Amazon WorkMail documentation.
• Turn on WorkMail audit logging as described in this AWS blog.
• Periodically remind your users to practice good cybersecurity. They should never share usernames or passwords, and be careful not to leave private information on public computers.

As a reminder, while AWS is responsible for the security of the underlying AWS cloud infrastructure, the responsibility for securing your email data and access controls within Amazon WorkMail falls squarely on your organization, as outlined in AWS’s Shared Responsibility Model.

Securing and Monitoring Amazon WorkMail

Before you can create a custom “Big Yellow Taxi” alert you must first set up your Amazon WorkMail organization and have configured both Event Logging and Audit Logging. These steps are described in the Amazon WorkMail documentation.

Once logging is set up, the steps below will guide you in the creation of a “Big Yellow Taxi” alerting system that uses a CloudWatch Subscription Filter to continuously scan the WorkMail logs. Whenever the filter detects an Authentication log entry containing the phrase "access_granted:false" (indicating a failed login attempt), the CloudWatch Subscription Filter will automatically invoke an AWS Lambda function. The Lambda function then extracts the relevant details from the log data, constructs an alert notification, and dispatches it to an Amazon SNS topic configured to send email.

This simple, yet powerful alerting system helps ensure you’re alerted to failed login attempts in a timely manner, giving you the information you need to dig deeper to ensure your WorkMail Organization is not being targeted by a malicious actor. Let’s get started!

Building a WorkMail Monitoring & Alerting Mechanism

This sample solution uses a CloudWatch Subscription Filter (video) to monitor the WorkMail audit logs. The filter will scan the logs for a specific pattern that indicates a user has attempted to log in using an unauthorized method. For example, if you explicitly deny the use of the IMAP protocol in your WorkMail organization via Access control rules, the filter will detect such an attempt, invoking a Lambda function. This function will then construct an alert notification and dispatch it to an SNS topic configured to send an email, typically to your security admin’s mailing list.

1. When a message from the Log Group contains “access_granted:false” (indicating a match with the Access Control rule), the subscription filter will deliver the message to the AWS Lambda function.
2. The Lambda function extracts the user name and organization details from the ID received from CloudWatch.
3. The Lambda function concatenates this information and sends it to an SNS topic
4. The SNS topic delivers a notification email message to an admin’s mailbox.
   (note – the dashed lines in the diagram represent optional components that are not addressed in this specific scenario.)

Walkthrough

This blog will walk you thru the following steps in your AWS account needed to create your own “big-yellow-taxi” alerts from your WorkMail Audit Logs (perform all steps in the same AWS region as your WorkMail Organization):

1. Create an Amazon SNS topic and configure it to send email notifications to your email and security administrators when the alert condition is matched
2. Create an IAM Policy
3. Create an IAM Role to allow a custom Lambda to invoke SNS and WorkMail
4. Create a custom AWS Lambda function that will be invoked by the CloudWatch Subscription Filter. This function will do the following::
   1. Extract the relevant details from the WorkMail alert logs data, such as the user account, IP address, and timestamp of the failed login attempt.
   2. Construct a concise, informative alert notification.
   3. Publish the alert notification to an Amazon SNS topic.
5. Test the “big-yellow-taxi” alert by attempting unauthorized access attempts to your Amazon WorkMail environment.

Prerequisites

• An AWS account with permissions to create / modify WorkMail, SNS, Lambda and IAM.
  • You’ll need your AWS Account ID, which you can get from the AWS console or the AWS CLI.
• A fully set up Amazon WorkMail Organization.
  • You’ll need your WorkMail Organization’s ARN, which you can get from the AWS console or the AWS CLI.
• Your WorkMail Organization must be configured to deliver Access Control logs to CloudWatch.

Step 1 – Create Simple Notification Service (SNS) Topic

Create an SNS topic and configure it to send email notifications when a login occurs with an unauthorized method.

1. Open the AWS SNS Console in the same region as your WorkMail Organization
   1. Click Create topic
   2. Name: workmail-yellow-taxi
   3. Create topic
2. Create Subscription.
   1. Select the Topic you created in the step above.
   2. Select email as the protocol, and enter an email address to which you have access (you can modify this later if desired)
   3. Create subscription.
   4. Check your email and confirm the SNS subscription (this may take a minute and you may need to check your spam folder)
3. Make note of the SNS Topic ARN shown in the SNS topic Details panel, as you’ll need it in the next step.

Step 2 – Create an IAM policy

1. Open the AWS IAM Console
2. Create new IAM policy
   1. Name: workmail-yellow-taxi
   2. Copy (below), edit with your AWS ID, SNS arn and WorkMail Organization arn, and paste the updated JSON into the permission policy editor.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"workmail:DescribeOrganization",
"sns:Publish",
"workmail:DescribeUser"
],
"Resource": [
"arn:aws:sns:<aws-region-x>:000000000000:workmail-yellow-taxi",
"arn:aws:workmail:<aws-region-x>:000000000000:organization/m-03fdc11b0c8c42d699dfc65fbb683b6a"
]
}
]
}

Step 3 – Create an IAM role

1. Open the AWS IAM Console
   1. Create a new Role
   2. Select AWS service
      1. Use Case – choose Lambda
      2. Search for, and add “AWSLambdaBasicExecutionRole“
      3. Role Name: yellow-taxi-lambda
      4. Edit & Add permissions – Search for, and add:
         1. workmail-yellow-taxi
   3. Create role

Step 4 – Create an IAM role

1. Go to the AWS Lambda console (in the same region as your WorkMail Organization).
2. In the Services Search field, type Lambda
3. Click Create function
   1. Author from scratch
   2. Function name: WorkMailAudit
   3. Runtime: Python3.12
   4. Architecture: x86_64
   5. Permissions: select the role (created above) yellow-taxi-lambda.
   6. Create function
4. Copy the Lambda code (below) and paste it into the lambda_function editor (under the Lambda’s Code tab):

import json
import base64
import zlib
import boto3
from botocore.exceptions import ClientError
import re
import os
 
topic_arn = os.environ['SNS_ARN']
sns = boto3.client('sns')
workmail = boto3.client('workmail')
 
def send_sns_message(topic_arn, message):
 
    try:
        # Send Message
        response = sns.publish(
            TopicArn=topic_arn,
            Message=str(message)
        )
        print(f"Message delivered sucessfully! Message ID: {response['MessageId']}")
    except ClientError as e:
        print(f"Error to send message: {e}")
 
def get_workmail_user_and_organization_names(organization_id, user_id):
   
    try:
        # Getting user name
        user_response = workmail.describe_user(OrganizationId=organization_id, UserId=user_id)
        user_name = user_response['Name']
        
        # Getting organization name
        org_response = workmail.describe_organization(OrganizationId=organization_id.split('/')[-1])
        organization_name = org_response['Alias']
        
        return user_name, organization_name
    except Exception as e:
        print(f"Error to get information : {e}")
        return None, None
 
def lambda_handler(event, context):
   
    #uncode and decompress CloudWatch function
    workmail_log = zlib.decompress(base64.b64decode(event["awslogs"]["data"]), 16 + zlib.MAX_WBITS).decode('utf-8')
    log_json = json.loads(workmail_log)
   
    for log_event in log_json["logEvents"]:
        log_message = json.loads(log_event["message"])
        organization_arn = log_message["organization_arn"]
        
        # Regex to get Organization Id from log
        organization_id = (re.search(r'[^/]+$', organization_arn)).group(0)
        user_id = log_message["user_id"]
        
        # Get username from WorkMail Organization
        user_name, organization_name = get_workmail_user_and_organization_names(organization_id, user_id)
        payload = {
            "User": user_name,
            "Organization": organization_name,
            "Protocol": log_message["protocol"],
            "Source IP": log_message["source_ip"],
            "Event": "Login attempt using unauthorized method"
        }
 
    send_sns_message(topic_arn, payload)
   
    return {
        'statusCode': 200
    }

1. Click the Deploy button
2. Click the Configuration tab, click Environment Variables (left menu) and click edit
   Key = SNS_ARN | Value = <the arn of the SNS topic you created earlier)
3. Click Save

Step 5 – Verify the CloudWatch log you’ve configured in WorkMail

1. Open the AWS CloudWatch console for your WorkMail Organization
2. Click Logging settings
3. Click the Audit log settings tab
   1. Make note of the Amazon CloudWatch Log destination for the Log deliveries – Access Control logs

4. Create an Access Control rule in WorkMail create a rule to deny the use of the IMAP protocol (see below):

• • Describe the rule: DENY_IMAP
  • Effect: Deny
  • Apply the rule to requests that use IMAP
  • Create the rule

5. Create the CloudWatch subscription filter:

• Open the AWS CloudWatch console for your WorkMail Organization
• Click Log groups and select: WorkMail-yellow-taxi
• Click the subscription filter tab and Create a new Lambda subscription filter
  • Select the Lambda: WorkMailAudit
  • Log format: JSON
  • Subscription filter pattern: "\"access_granted\":false"
  • subscription filter name: workmail-yellow-taxi-filter
• Click Start streaming

Congratulations, you’ve set up your own Yellow-Taxi alert to send you an email if someone tries to login to a WorkMail account using a denied protocol (IMAP in this example).

Testing your Big Yellow Taxi alert

To test the new rule using the AWS CLI (you will need a valid WorkMail user’s login and password), you can attempt to login using IMAP, whcih you denied above.

1. Copy, edit & paste the openssl command into the AWS CLI (note – the imap address region needs to match your WorkMail region) to attempt a login using the prohibited IMAP protocol:

openssl s_client --connect imap.mail.us-west-2.awsapps.com:993


2. You’ll get the following back in the terminal:

OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ENABLE IDLE LITERAL+ AUTH=PLAIN] Amazon WorkMail IMAP Proxy

3. Type the LOGIN command using a valid WorkMail user’s email address and password

? LOGIN [email protected] user's_password

4. You’ll be denied access (because IMAP is set for “deny”)

Shortly after your failed attempt to login, you’ll receive an email at the address specified in the SNS topic from AmazonSNS with details about the failed attempt similar to the message shown below:

Conclusion

Just as the U.S. State Department’s “Big Yellow Taxi” custom alert rule proved instrumental in detecting the initial signs of the Microsoft Exchange Online compromise, this simple yet effective monitoring and alerting system will help ensure your organization receives timely notice of failed login attempts. These alerts allow you to investigate and respond to potential threats quickly. By actively monitoring your WorkMail environment, you will able to protect your organization’s sensitive email communications against this type of intrusion incident.

Next Steps

The time to act is now to ensure your WorkMail instance is fortified against emerging threats. Following the guidance in this blog post helps you strengthen your WorkMail security posture and protect your sensitive communications from malicious actors.

By implementing the CloudWatch Subscription Filter and AWS Lambda-based alerting system, you can establish a proactive “Big Yellow Taxi” monitoring solution for your Amazon WorkMail environment. This will provide your organization with the necessary visibility and alerting capabilities to quickly detect and respond to unauthorized access attempts, helping to safeguard your sensitive email data.

If you have questions or comments, join the AWS Community Forums to ask questions, share experiences, and learn from other AWS users who have implemented similar solutions for secure email sending from their web applications.

To learn more about Amazon WorkMail, or to create a no-cost 30-day test organization, see Amazon WorkMail.

About the Authors

Luis Miguel Flores dos Santos

Miguel is a Solutions Architect at AWS, boasting over a decade of expertise in solution architecture, encompassing both on-premises and cloud solutions. His focus lies on resilience, performance, and automation. Currently, he is delving into serverless computing. In his leisure time, he enjoys reading, riding motorcycles, and spending quality time with family and friends.

Andy Wong

Andy Wong is a Sr. Product Manager with the Amazon WorkMail team. He has 10 years of diverse experience in supporting enterprise customers and scaling start-up companies across different industries. Andy’s favorite activities outside of technology are soccer, tennis and free-diving.

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.