Tag Archives: Amazon Simple Email Service (SES)

AWS Weekly Roundup: Amazon EC2 F2 instances, Amazon Bedrock Guardrails price reduction, Amazon SES update, and more (December 16, 2024)

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-ec2-f2-instances-amazon-bedrock-guardrails-price-reduction-amazon-ses-update-and-more-december-16-2024/

The week after AWS re:Invent builds on the excitement and energy of the event and is a good time to learn more and understand how the recent announcements can help you solve your challenges. As usual, we have you covered with our top announcements of AWS re:Invent 2024 post.

You can now watch keynotes and sessions on the AWS Event YouTube channel. This year Andy Jassy, now President and CEO at Amazon, returned to re:Invent and shared some thoughts in these videos.

Drawing on experiences Amazon has had building distributed systems at massive scale, Werner Vogels, VP and CTO at Amazon, shared critical lessons and strategies he has learned for managing complex systems in his keynote.

Last week’s launches
Here are the launches that got my attention.

Amazon Elastic Compute Cloud (Amazon EC2) – A new generation of FPGA-powered instances (F2) is now available. In contrast to a purpose-built chip designed with a single function in mind and then hard-wired to implement it, a field programmable gate array (FPGA) can be programmed in the field, after it has been plugged in to a socket on a PC board. We’re also introducing Amazon EC2 High Memory U7i instances with 6TiB and 8TiB of memory. U7i instances are ideal to run large in-memory databases such as SAP HANA, Oracle, and SQL Server. Graviton-based 8th generation instances now support bandwidth configurations for Amazon VPC and Amazon EBS.

Amazon Bedrock Guardrails – We are reducing pricing by up to 85% to help you implement safeguards for your generative AI applications. Also, we’re adding multilingual capabilities with support for Spanish and French languages.

Amazon Simple Email Services (SES) – Now offers Global Endpoints for multi-region sending resilience and announces the availability of Deterministic Easy DKIM (DEED), a new form of global identity which simplifies the use of DomainKeys Identified Mail (DKIM) management.

AWS CloudFormation – An enhanced version of the AWS Secrets Manager transform introducing automatic AWS Lambda upgrades.

Amazon Lex – Launches new multilingual streaming speech recognition models that enhance recognition accuracy through two specialized groupings: a European-based model (for Portuguese, Catalan, French, Italian, German, and Spanish) and a Asia Pacific-based model (for Chinese, Korean, and Japanese).

Amazon Connect – Now supports push notifications for mobile chat on iOS and Android devices. In this way, you can be proactively notified as soon as there is a new message from an agent or chatbot, even when not actively chatting. You can now also configure holidays and other variances to your contact center hours of operation.

AWS Security Hub – Now supports automated security checks aligned to the Payment Card Industry Data Security Standard (PCI DSS) v4.0.1, a compliance framework that provides a set of rules and guidelines for safely handling credit and debit card information.

AWS Resource ExplorerSupports 59 new resource types including Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Kendra, AWS Identity and Access Management (IAM) Access Analyzer, and Amazon SageMaker.

Amazon SageMaker AI – Inference optimized Amazon EC2 G6e instances (powered by NVIDIA L40S Tensor Core GPUs) and P5e (powered by NVIDIA H200 Tensor Core GPUs) are now available on Amazon SageMaker.

Amazon Redshift – Now supports automatically and incrementally refreshable materialized views on tables in a zero-ETL integration. Previously, in this case, you had to run a full refresh.

AWS Toolkit for Visual Studio Code – Now includes Amazon CloudWatch Logs Live Tail, an interactive log streaming and analytics capability that provides real-time visibility into your logs and makes it easier to develop and troubleshoot applications.

Other AWS news
Here are some additional projects, blog posts, and news items that you might find interesting:

Build a managed transactional data lake with Amazon S3 Tables – Just introduced at re:Invent 2024, Amazon S3 Tables is the first cloud object store with built-in Apache Iceberg support and the easiest way to store tabular data at scale. This post on the AWS Storage Blog provides an overview of S3 Tables and an example of how to build a transactional data lake with S3 Tables using Apache Spark on Amazon EMR.

Introducing Cross-Region Connectivity for AWS PrivateLink – More information on this recent launch that can be used to share and access Amazon Virtual Private Cloud (Amazon VPC) endpoint services across different AWS Regions.

Marc Brooker, VP/Distinguished Engineer at AWS, shared on his personal blog a few posts about what Amazon Aurora DSQL is, how it works, and how to make the best use of it:

That’s all for this week. Check back next Monday for another Weekly Roundup!

Danilo

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Modernize email sending with Amazon Simple Email Service and Proofpoint SER

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/modernize-email-sending-with-amazon-simple-email-service-and-proofpoint-ser/

As organizations migrate to the cloud, managing email security and delivery becomes increasingly complex. This post explores how Amazon Simple Email Service (SES) and Proofpoint Secure Email Relay (SER) work together to provide a robust solution for modern email sending.

Shifting email workflows to the cloud

For many organizations, Simple Mail Transfer Protocol (SMTP) relay is a critical email delivery mechanism that facilitates the transmission of email messages between different domains and servers. When an email is sent to a recipient outside the sender’s domain, SMTP relay(s) ensures the message is routed correctly and delivered to the intended destination.

Traditionally, on-premises SMTP relay servers were used by messaging and security teams to manage email sending from on-premises applications on behalf of their organizations’ domains. This approach helps to:

  • Guard against brand damage and loss of sensitive data
  • Protect recipients from fraud
  • Provide straightforward control over email sending

However, as applications are modernized and migrated to the cloud, email sending has changed. Many organizations now outsource bulk email sending to cloud service providers such as Amazon SES. This shift has made it challenging for internal teams to regulate their organization’s email effectively.

Addressing modernization challenges

Amazon Web Services (AWS) is a popular choice for developing and modernizing applications, with Amazon SES offering a secure, scalable and cost effective service for applications to send email. To address the need for additional security and control over email, Proofpoint offers their popular Secure Email Relay (SER) on the AWS Marketplace.

Using Amazon SES with Proofpoint SER combines the convenience and features of Amazon SES with Proofpoint SER’s ability to:

  • Regulate and govern outbound application email
  • Support migration from on-premises relay to the cloud
  • Apply security controls to application emails

Email flow and security

SES Mail Manager allows emails to be conditionally routed from Amazon SES to Proofpoint SER. The process includes:

  1. Scanning emails with Proofpoint threat detection technologies
  2. Applying centrally managed Proofpoint SER policies
  3. Performing DKIM-signing and distributing DMARC compliant emails
  4. Sending mail to recipients

Two configuration options are available:

  1. Proofpoint SER handles distribution to final recipients (Figure 1)
  2. Emails are routed back to Amazon SES for distribution after Proofpoint SER processing (Figure 2)

In the first configuration, email is sent to SES and routed through a Mail Manager SMTP relay to Proofpoint SER where it is processed for threat detection, malware, spam, sensitive data, and more. In this configuration, Proofpoint SER distributes the email to recipients through a STMP relay or another email sending service beyond Proofpoint. Deliverability reporting, archiving, and other capabilities are left to Proofpoint or downstream providers.

SES to SER

Figure 1: Proofpoint SER applying security controls to application emails before sending.

In the second configuration, email is sent to SES which routes through a Mail Manager SMTP relay to Proofpoint SER to be processed for threat detection, malware, spam, sensitive data, and more. Proofpoint SER then sends the processed emails back to SES via Mail Manager STMP relay. Deliverability reporting, archiving and other capabilities such as Amazon Q Business can be provided by SES or other AWS services.

SES-SER-SESFigure 2. Security controls applied by Proofpoint SER before routing emails back to Amazon SES for distribution.

Conclusion

The integration of Amazon SES and Proofpoint SER offers a powerful solution for organizations looking to modernize their email sending processes while maintaining robust security. This collaboration allows businesses to leverage the scalability and convenience of cloud-based email services without compromising on control and protection.

Call to Action

Take the next step in modernizing your email infrastructure while enhancing security and control. To learn more about how Amazon SES and Proofpoint SER can benefit your organization:

How to Make Simple Email Service Resilient Across Two AWS Regions with Global Endpoints

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-make-simple-email-service-resilient-across-two-aws-regions-with-global-endpoints/

Introduction

Amazon Simple Email Service (SES) recently announced Global Endpoints, a major enhancement to its email sending features. This new capability improves the availability and reliability of SES API v2 email sending workloads by automatically distributing messages, in an active/active configuration across the Primary and Secondary AWS regions. When Global Endpoints detects degradation in either the Primary or Secondary SES region, the feature automatically shifts all traffic to the healthy region, no customer intervention is needed.

Multi-Region SES Configuration Challenges

Customers face significant difficulties in correctly implementing a multi-region setup or disaster recovery setup. The process requires careful curation of systems along the failover path and ensuring high availability of these systems. Ironically, the system designed to trigger failover can itself fail when needed most. For many SES customers, the effort required to design, build test, monitor and maintain a two-region SES system outweighs the benefits.

SES Global Endpoints eliminates the need for these complex, custom workarounds. The feature provides a straightforward solution for maintaining email sending during unexpected regional disruptions. Global Endpoints’ built-in safeguards ensure email infrastructure remains resilient when it matters most. Please note that at launch, Global Endpoints does not support SMTP or VPC endpoint access.

SES Global Endpoints: Key Technological Components

Global Endpoints utilizes four new SES components that work together to provide a seamless and reliable multi-region email sending experience:

  1. Multi-Region Endpoint (MREP) is a new type of SES endpoint that automatically distributes email traffic across two AWS regions.
  2. Deterministic Easy DKIM Identities (DEED) makes it easy to setup multi-region identities without having to make any DNS changes.
    1. Read the blog introducing SES DEED for more information.
  3. Updated AWS SES Console Tool walks you through the process and simplifies the duplication of Domain Identities, Configuration sets, and Sending limits across regions.
  4. Readiness Checks in the SES console verify uniformity between configurations of key resources in both the Parent and Secondary SES regions.

How SES Global Endpoints work

GE-MREP-healthy

Figure 1 – SES Global Endpoints with two healthy regions.

Global Endpoints are resources that distribute your SES outbound workloads across two AWS Regions. When you set up Global Endpoints, you select a Primary Region (where the actual Endpoint is created) and a Secondary Region. When configured, a new Global Endpoints resource, called “multi-region endpoint” (MREP) is created and managed. Developers will need to update their SES v2 API enabled applications and services to use their unique MREP as the entry point to SES for their email sending requests.

The Global Endpoints configuration requires that your sending domain identity(s) is verified in both the Primary and Secondary regions. SES Global Endpoints uses DEED to simplify this process. DEED is a new feature that generates consistent DKIM tokens across all AWS Regions based on a Parent Domain Identity that is configured with Easy DKIM. This consistency allows SES to automatically verify a domain identity in the Secondary region once it’s verified in the Primary region, without requiring additional DNS record updates. Customers do not need to make any additional changes other than activating the DEED identity type. When customers expand their sending workload’s geographic footprint, or reconfigure their Global Endpoints settings in the future, their DEED identities will continue to be available and managed automatically. You can learn more about DEED from this post.

Global Endpoints works alongside other SES services, such as Virtual Deliverability Manager (VDM). Once Global Endpoints are enabled, you’ll continue to see per-region data on email sending performance in VDM. If you’ve configured event destinations like CloudWatch, SNS, or Firehose, you can make use of those same monitoring and alerting tools in your second region as soon as you’re ready. As noted below, although Configuration Sets are automatically duplicated in the Secondary region, you must manually duplicate your SES event destinations in those Configuration Sets.

It is important to understand that Global Endpoints is not a failover solution for SES, it’s an active-active implementation; when no regional impairment exists, SES Global Endpoints distributes sending traffic across two AWS regions. Customers who use SES’ shared IP sending pool do not need to make any changes, Global Endpoints will utilize the SES shared IP pool in the Secondary region. Customers who use standard, dedicated IPs must manually set up equivalent number of dedicated IPs in the Secondary region. Once properly configured, Global Endpoints will keep the dedicated IPs warm in both regions as long as you use the MREP and maintain a steady sending volume.

For example, when SES’s regional health monitoring detects degradation in the Primary region (as shown in the diagram below), The MREP automatically shifts all traffic to the healthy region. This illustrates the need for matching configurations in both regions, since all traffic will be sent through a single region, in this example the Secondary region, as long as the impairment exists. When SES’s regional health monitoring detects the impaired region is back to normal, traffic is once again redistributed across both regions. Importantly, no customer intervention is needed; SES Global Endpoints automatically and dynamically monitors and manages the traffic distribution via the MREP.

GE-MREP-impaired

Figure 2 – SES Global Endpoints with one impaired region.

The key benefits of using Global Endpoints include:

  • Simplified multi-region configuration
  • Automatic routing between two regions with no delay
  • More resilient email sending

Global Endpoints: Setup and Use

Using the SES Console, the Global Endpoint setup process assists in duplicating the key artifacts and sending limits from your Primary Region to your Secondary Region. This process ensures that both regions have equivalent verified identities, configuration sets, and approved sending limits sufficient for all of the expected volume. Customers can manually duplicate these key artifacts using the SES v2API or CloudFormation, but we recommend using the SES console because these steps are simplified.

Once the Global Endpoint is ready, key resources duplicated and the application or service has been updated to use the new MREP, SES Global Endpoints automatically routes your outbound traffic evenly between your primary and secondary regions using the multi-region endpoint (MREP). If the MREP detects degraded performance in the Primary or Secondary region, it will automatically route all SES traffic to the healthy region until the impairment is resolved.

Preparing the Parent Region

The high-level steps to setup Global Endpoints using the SES Console are below.  Note – you must already have a primary SES region fully operational with at least 1 fully verified sending identity with production access before setting up Global Endpoints.

Create Global Endpoint

  1. Open the SES console in the Primary AWS region.
  2. In the navigation pane, choose Global Endpoints.
  3. Choose Create Global Endpoint.
  4. Select a Secondary Region from the dropdown menu. (Your Primary Region defaults to the region to which you signed into the console)
  5. Review the configuration and choose Create.

The creation process may take a few seconds. Once completed, the status of your Global Endpoint will change to “Ready.”

Global Endpoints "ready" status

Preparing the secondary region

Once your Global Endpoint is ready, you must now ensure that the your email sending configuration, including all its components (Domain Identities, Configuration sets, email templates, and sending limits), is consistent across the primary and secondary regions before utilizing the Global endpoint for sending emails. This alignment is crucial to avoid potential issues and ensure proper email delivery and tracking.

The new Region duplication feature in the SES console assists you by automatically duplicating resources and duplicating account-level settings from the primary to the secondary region, ensuring that both regions have equivalent configurations.

The high-level steps to setup the secondary region for use with Global Endpoints using the SES Console are below. If you’d like to use the AWS CLI to manually duplicate these resources, consult with the SES v2 API documentation.

Duplicating verified domain identities

Next you’ll use the Duplicate verified domain identities feature in the SES console to create one or more domain identities in the Secondary Region. SES will then use DEED to verify the domain identities in the Secondary Region. Note that DEED can only be used if the Primary Domain Identity is already configured with Easy DKIM. Domain identities that are verified with BYODKIM will need to be created manually in the Secondary Region, as DEED is not applicable in this case.

Important – It’s crucial that both Primary and Secondary Regions have the equivalent verified identities and configuration sets that you intend to send email with, along with matching sending limits to ensure proper functionality of the Global endpoint. Any discrepancies could cause delivery failures, diminished failover reliability, and missing metrics.

To duplicate identities from the SES console:

  1. On the Global endpoints page, choose the Global endpoint you want to duplicate by selecting it from the Name column.
  2. Under the Region duplication tab, choose Duplicate identities.
  3. Select the identities you want to duplicate followed by Confirm.

To duplicate configuration sets from the SES console:

  1. On the Global endpoints page, choose the Global endpoint you want to duplicate by selecting it from the Name column.
  2. From the Region duplication tab, choose Duplicate configuration sets.
  3. Select the configuration sets you want to duplicate followed by Confirm.

Important Notes:

  • When duplicating configuration sets across regions, Event destinations and Reputation settings require manual reconfiguration in the Secondary Region to match the Primary Region’s setup. Since SES event destinations are region-specific, you’ll need to manually recreate these configurations in each region. For cross-regional monitoring and event routing, you can refer to additional AWS documentation for services like CloudWatch (cross-region dashboards), SNS (cross-region message delivery), and EventBridge (cross-region event routing) to develop a comprehensive multi-region event strategy.
  • If you are using SES email template resources, those templates need to be manually duplicated into the Secondary Region (the console is currently unable to perform this action).
  • You must manually synchronize any changes made to the Parent Region’s configuration sets to the Secondary Region to maintain sending integrity. This includes adding or removing standard dedicated IPs to both regions to ensure either region has the required IPs to manage the expected throughput in the case of a regional event or impairment.

The Duplicate production limits feature allows you to:

  • Check if production limits are aligned between primary and secondary regions
  • Request a limit increase in the secondary region if needed

To duplicate production limits from the SES console:

  1. On the Global endpoints page, choose the Global endpoint you want to duplicate by selecting it from the Name column.
  2. From the Region duplication tab, verify the status in the Duplicate production limits card. If the status displays Sending limits not aligned, choose Duplicate production limits.
  3. The Service Quotas page opens in the secondary region where you can request increases to “Sending quota” and “Sending rate” to match the values from the primary region.

Note – SES checks if sending limits are aligned between regions and allows you to request limit increases in the secondary region if needed. We recommend that you request the maximum quota you’re eligible for in both regions. While email traffic is distributed amongst both regions under normal operating conditions, during a failover event, the full volume of email traffic will be sent to one region and its limits should be enough to handle the full volume load.

If any manual steps are required to complete the Global Endpoint creation, they will be shown in the SES Console:

manual steps warning

When the Global Endpoint is fully configured, a MREP will be created with an Endpoint ID (see below). You will use this new endpoint ID when re-configuring your SendEmail and SendBulkEmail API calls. (note – Global Endpoints MREP are only supported by SES APIv2. The feature is not available using SMTP or VPC endpoints).

Now you’re ready to send your first email through SES Global Endpoints and your MREP!

Once you’ve obtained the Endpoint ID of your Global endpoint (this is the MREP), you should update your applications’ SendEmail or SendBulkEmail API calls to include the Endpoint ID value for the -endpoint-id parameter.

Here’s an example of how to specify the Endpoint ID in a SendEmail API call using the AWS CLI (modify the from & to email addresses and the --endpoint-id accordingly):

aws sesv2 send-email \
--from-email-address "[[email protected]](mailto:[email protected])" \
--destination "ToAddresses=[[email protected]](mailto:[email protected])" \
--content "Subject={Data=Test
email,Charset=UTF-8},Body={Text={Data=This is a test email sent using Amazon SES
Global endpoints.,Charset=UTF-8}}" \
--endpoint-id "abcdef12.g3h"

The Global Endpoints console page provides summary observability metrics on the combined workload and a unified view of your email sending volume across both the primary and secondary regions. You can access these metrics through the Cross-region metrics tab on the Global endpoint details page in the SES console..

Conclusion

By using a SES Global Endpoint in their SES API v2 applications and services, SES customers benefit from uninterrupted email delivery during regional service issues. SES Global Endpoints automatically distributes sending workloads across two AWS Regions, significantly enhancing resilience against regional outages. The Global Endpoints feature maintains warmed-up dedicated IP addresses in both regions, when used, and automatically shifts traffic to the healthy region when the other is impaired, without requiring customer intervention. SES Global Endpoints eliminates the pain points typically associated with manually-built, multi-region SES sending systems.

Global Endpoint’s console tools provide quick and easy setup and includes readiness checks to identify and mitigate potential misconfigurations. These enhancements simplify the configuration and management process, making it easier for customers to maintain a robust email sending infrastructure.

Overall, SES Global Endpoints addresses customer needs for a more reliable and easily managed email sending system, automating critical processes and providing robust tools for setup and maintenance. This significant improvement to the email sending experience is expected to benefit AWS customers across various industries and use cases.

Call to Action

Get started with SES Global Endpoints today to enhance your email sending resilience!

  • Visit the AWS Console to enable this feature for your account
  • Review the comprehensive documentation for step-by-step guidance.
  • For personalized assistance, don’t hesitate to contact AWS Support or your AWS account team.

Elevate your email infrastructure now to ensure uninterrupted communication with your customers, even in the face of regional disruptions.

Simplify Multi-Region Email Sending with Simple Email Service’s Deterministic Easy DKIM

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/simplify-multi-region-email-sending-with-simple-email-services-deterministic-easy-dkim/

Introduction

Amazon Simple Email Service (SES) provides customers with a robust, scalable email solution to send large-scale, global email communications. The service offers customers many benefits, including scalability, high deliverability rates, cost effective pay-as-you go pricing, availability in over a dozen AWS regions, and tight integration with other AWS services.

We’ve heard from customers who want to synchronise domain identities across multiple AWS regions that it can be difficult to configure and maintain. These customers have shared that establishing and maintaining AWS region specific domain verifications can be confusing, time-consuming, and that it’s difficult to coordinate the many details across their users, customers and service providers.

In this post, we’ll explore the newly introduced SES feature called Deterministic Easy DKIM (aka “DEED”), which solves critical challenges in multi-region email identity management for SES customers. DEED generates consistent DKIM tokens across multiple AWS Regions based on a Parent (domain) Identity that is configured with SES Easy DKIM. DEED uses the Parent Identity’s Easy DKIM configuration to automatically provide the same DKIM signing configuration for a Replica (domain) Identity in a AWS Replica Region. With DEED, you only need to publish DNS records once for the Parent Identity. The Replica Identity will automatically use these same DNS records to verify domain ownership and manage DKIM signing. DEED streamlines multi-region email operations by simplifying DNS management and ensuring consistent DKIM signing across AWS regions, maintaining best-practice email authentication while reducing operational complexity.

Background and key challenges

SES introduced Easy DKIM over a decade ago as an innovative, streamlined solution to help customers create, verify, and manage domain identities with automated DKIM signing. Easy DKIM is simple to set-up, and works by generating pre-determined tokens that customers add to their DNS configurations. Once Easy DKIM is enabled, SES generates a public/private signing key for each domain identity and updates the verified identity’s CNAME public key. SES’ Easy DKIM simplifies email authentication by managing and rotating DKIM keys on behalf of customers.

Before DEED, customers who wanted to expand their SES email infrastructure across multiple regions faced complexity, operational challenges and a substantial administrative burden configuring and maintaining DKIM across AWS regions. This made it very difficult for customers to scale their email infrastructure efficiently, and often dissuaded customers from fully leveraging the potential of a multi-region email sending strategy. These challenges are more pronounced for Independent Software Vendors (ISVs) and email service providers using SES. Frequently these organizations don’t own or control the domains used by their end-customers, who must manually update their DNS entries each time the ISV expanded or shifted their SES sending infrastructure between AWS regions.

Solution Overview

Deterministic Easy DKIM (DEED) allows SES customers to set up email identities across multiple regions, leveraging an existing domain identity configuration in a single region without the need to make companion DNS changes in all other regions. This innovative approach eliminates the manual overhead of creating region-specific DNS entries, and provides a streamlined solution to configure and maintain global email infrastructure for organizations and ISVs alike.

The key benefits of using DEED include:

  • Simplified DNS Management – Publish DNS records once for the Parent Identity and the replica identity is automatically synced.
  • Easier Multi-Region Operations – Simplifies the process of expanding email sending operations to new AWS regions.
    Reduced Administrative Overhead – Manage DKIM configurations centrally from the Parent Identity.

Terminology:

To understand DEED, let’s explore the key terminology that underpins this innovative approach:

  • Deterministic – a process or system where the same input will always produce the same output, with no randomness or unpredictability involved. In other words, if all the starting conditions are known, the outcome can be precisely predicted or determined.
  • Parent Region – The original AWS Region where the primary email identity is initially established.
  • Parent Identity – A verified email identity configured with Easy DKIM that serves as the authoritative source for DKIM configuration across regions.
  • Replica Region – An AWS Region where an identical email identity is replicated without additional configuration.
  • Replica Identity – An identity that shares identical DNS configuration and DKIM signing configuration of a parent identity.
  • DEED Identity – Any identity that is used as either a parent identity or a replica identity. (When a new identity is created, it is initially treated as a regular (non-DEED) identity. However, once a replica is created, the identity is then considered a DEED identity.)

How DEED works

DEED flow

DEED is built on the existing Easy DKIM framework:

  1. Using Easy DKIM, SES generates a public-private key pair and automatically adds a DKIM signature to every message sent from a SES verified identity in the Parent Region.
  2. To accommodate authenticated SES sending from multiple AWS regions, DEED automatically synchronizes the signing keys from the Parent Region and Parent Identity to the Replica Region and Replica Identity.
  3. This automated process ensures that both the Parent and Replica Identities receive and use identical keys for DKIM signing, maintaining consistent authentication across different AWS regions.
  4. SES manages the complex process of key rotation across Parent Identity and Replica Identities, further simplifying email infrastructure management across multiple AWS regions.
  5. Replica Identities inherit the DKIM signing configuration of the parent identity. Because of this dependency, you cannot delete a Parent Identity until all Replica Identities are deleted.
  6. The receiving email server/service validates DKIM from DNS.

We recommend customers take advantage of DEED for single-Region sending, as this new capability is included, at no additional cost, in the base SES pricing.

Steps to setup SES DEED Replica in a second AWS region

These steps assume that you are already using SES in the Parent Region, and have a fully verified domain identity that is configured to use Easy DKIM. You can also use the AWS CLI.

Step 1 – Update the Parent Identity

  1. Login to the AWS SES Console in the Parent Region
  2. Click on the Identities link in the SES navigation panel (far left) and click on the verified identity you want to use in other AWS regions.
    1. The Parent Identity must have Easy DKIM enabled.
    2. You cannot create Replicas of Identities that use BYODKIM or self-signed identities.
  3. Click on the Authorization tab
  4. Click Create Policy and select Create custom policy from the drop-down menu.
  5. Name the policy (for example, DEED-example_com)
  6. Modify the IAM policy (below) with your AWS account ID and AWS region, and paste it into the IAM Policy document to grant permission on the Parent Identity to allow the desired Replica Region to replicate the Parent Identity’s DKIM Signing Attributes:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowDKIMReplication",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:root"
},
"Action": "ses:ReplicateEmailIdentityDKIMSigningKey",
"Resource": "arn:aws:ses:us-east-1:123456789124:identity/example.com",
"Condition": {
"ForAllValues:StringEquals": {
"ses:ReplicaRegion": ["us-west-2", "eu-west-1"]
}
}
}
]
}

Note – Use consistent IAM policies that all allow for DKIM replication across intended Replica Regions.

Step -2 – Setup DEED Replica Identity

  1. Login to the AWS SES Console in the Replica Region
  2. Click on the Identities link in the SES navigation panel (far left) and click Create identity
  3. Under Identity details, Identity type, click Domain
  4. Type the domain name exactly as it appears in the Parent Identity
  5. Under Verifying your domain, click Deterministic Easy DKIM
  6. Select the Parent Region in the Parent region drop-down
  7. Ensure DKIM Signature is Enabled
  8. Click Create identity

create DEED ID

  1. The Replica Identity in the Replica Region will now automatically synchronize DKIM with the Parent Identity in the Parent Region and your apps/services can be configured to use either Parent or Replica Region & Identity to send DKIM authenticated email.
  2. Repeat the process if you want to create additional Replica Regions.

You can verify that the replica identity was configured correctly with the parent identity’s DKIM signing configuration by using the get-email-identity command and specifying the Replica’s domain name and region:

`aws sesv2 get-email-identity --email-identity [example.com](http://example.com/) —region us-west-2`

The response will include the value of the Parent Region in the DomainSigningAttributesOrigin parameter, signifying that the Replica Identity has been successfully configured with the Parent Identity’s DKIM signing configuration:

{
  "DkimAttributes": {
    "SigningAttributesOrigin": "AWS_SES_US_EAST_1"
  }
}

Conclusion

Deterministic Easy DKIM (DEED) represents a significant leap forward in multi-region email management for Amazon SES users. By eliminating the need for manually configured region-specific DNS configurations, DEED streamlines the process of expanding email operations across multiple AWS regions. This innovation not only reduces administrative overhead but also opens up new possibilities for organizations and ISVs to implement robust, globally distributed email infrastructures. With DEED, businesses can now leverage the full potential of Amazon SES across regions, ensuring consistent authentication, improved disaster recovery, and optimal performance without the previous complexities of multi-region setup.

Call to Action

Are you ready to simplify your multi-region email strategy with Amazon SES and DEED? Take the next step in optimizing your email infrastructure:

  1. Explore the DEED feature in your Amazon SES console today.
  2. Set up a test environment to experience the seamless multi-region configuration firsthand.
  3. For more detailed information, check out our comprehensive documentation on implementing DEED.
  4. Have questions or need assistance? Reach out to our AWS support team or join the AWS community forums to connect with other users.

Don’t let regional boundaries limit your email capabilities. Embrace the power of Deterministic Easy DKIM and transform your global email strategy with Amazon SES. Start your DEED journey today and unlock new levels of efficiency and scalability in your email operations.

re:Invent 2024: Your Amazon SES and AWS End User Messaging guide to AWS’s biggest event

Post Syndicated from Will Quillin original https://aws.amazon.com/blogs/messaging-and-targeting/reinvent-2024-your-amazon-ses-aws-end-user-messaging-guide-to-awss-biggest-event/

On December 2, AWS re:Invent conference returns to Las Vegas for our 13th year. With the conference approaching, our tailored sessions for Amazon SES and AWS End User Messaging will make your re:Invent experience worthwhile.

Discover new ideas and reaffirm your AWS skills

BIZ205: Simplify your email with Amazon SES and Amazon Q*

Monday, Dec 2nd| 12:00 PM–2:00 PM PST

  • Leverage public information and Amazon Q to evaluate your company’s entire email presence.
  • Build a managed email infrastructure that extends beyond inboxes and into all email applications across networks.

*Note: This session’s title is out of date in the re:Invent catalog.

BIZ209: Unlock higher email deliverability rates with Amazon SES

Monday, Dec 2nd| 10:00 AM–11:00 AM PST

  • Learn to identify and resolve common email deliverability issues using Amazon Simple Email Service (SES) and Virtual Deliverability Manager.
  • Solve real-world scenarios using diagnostic tools.
  • Get step-by-step guidance to improve email delivery rates and ensure messages reach their intended recipients.

BIZ206: Implementing resilient omni-channel notifications with AWS

Monday, Dec 2nd| 8:00 AM–10:00 AM PST

  • Learn how to build a resilient omni-channel notification system with fallback options.
  • Discover how to use AWS services like Amazon API Gateway, AWS Lambda, AWS End User Messaging, and Amazon SES to create a solution that monitors message delivery and automatically creates channel fallback options.

BIZ311: Send high-volume one-time passwords via SMS and email

Wednesday, Dec 4th| 9:00 AM–10:00 AM PST

  • Set up one-time passwords (OTPs) for multi-factor authentication, payment verification, and account recovery.
  • Gain insights into configuring SMS and email for high-volume, international, and resilient OTP sending.

BIZ301: Integrating SMS capabilities with AWS End User Messaging

Monday, Dec 2nd| 2:30 PM–3:30 PM PST
Wednesday, Dec 4th| 4:00 PM–5:00 PM PST

  • Designed for developers and technical individuals looking to add SMS sending to their applications.
  • Learn the fundamentals of SMS and how to set up AWS End User Messaging.

If you have signed up for re:Invent, you can directly reserve your spot in the sessions here (login required).

Map out your schedule with re:Invent attendee guides

With sessions up and down the Las Vegas Strip, planning your agenda is critical to make the most of your time at re:Invent. Find specific attendee guides below:

Explore the session catalog to see available sessions, including chalk talks, workshops, builders’ sessions, and more.

Can’t make it in-person? Join the virtual re:Invent 2024 experience

If you’re unable to travel to Vegas this year, you can still watch remotely! All keynotes and leadership sessions will be livestreamed and available on demand. After the event, breakout sessions will be available to stream; closed captioning is available as well. There will also be other livestreamed events, like behind-the-scenes programming, commentary from special guests, and more.

Register now to take your AWS knowledge to the next level. Plan your time in Las Vegas to make the most of the available sessions, get inspired by the global cloud community, and learn the latest from AWS.

Check out the FAQs for answers to all your re:Invent questions.

See you at re:Invent 2024!

Build a Secure One-Time Password Architecture with AWS

Post Syndicated from Bruno Giorgini original https://aws.amazon.com/blogs/messaging-and-targeting/build-a-secure-one-time-password-architecture-with-aws/

In today’s digital landscape, where cyberattacks continue to grow more sophisticated, the need for robust security measures has never been more paramount. One-Time Passwords (OTPs) have long been a crucial component of multi-factor authentication. They provide an additional layer of security to protect user accounts from unauthorized access.

The landscape of OTP delivery is evolving rapidly. While organizations increasingly favor more secure, phishing-resistant methods like passwordless solutions and hardware security keys, many still rely on SMS-based OTPs or require time to transition to newer technologies.

For organizations already leveraging Okta as their identity provider, AWS offers a comprehensive guidance on implementing phone-based multi-factor authentication. The “AWS Guidance for Okta Phone-Based Multi-Factor Authentication on AWS” provides a detailed reference architecture and implementation steps for integrating Okta with AWS services to deliver OTPs via SMS or voice calls.

This blog post offers a comprehensive guide for implementing a reliable, multi-channel OTP solution using AWS services including Amazon DynamoDB, Amazon Simple Email Service (SES), and AWS End User Messaging.

By the end of this blog, you’ll understand how to generate, store, and deliver OTPs via email, SMS, and voice. You’ll also learn best practices for secure OTP implementation. This solution serves organizations that need to maintain SMS-based OTP capabilities.

Let’s explore how to build a secure, multi-channel OTP solution on AWS.

AWS End User Messaging is the new name for Amazon Pinpoint’s SMS, MMS, push, WhatsApp, and text to voice messaging capabilities.

The authentication flow

Let’s imagine a hypothetical scenario where a bank customer want’s to access his online account:

Alex, a customer of the XYZ financial institution, needed to access their online account. They initiated the login process and requested an OTP from the mobile or web application provided by the bank. Upon receiving the request, the bank’s server created a user-specific session to handle the OTP generation and verification. A unique one-time password was then generated and sent to Alex’s registered mobile number via SMS. Alex received the OTP on their phone and had three attempts to enter the correct code within a 10-minute timeframe. This security measure prevented unauthorized access to their account. If Alex couldn’t receive the SMS, they had another option. They could request the bank to send the same OTP to their registered email address, if they had one on file. If Alex entered the correct OTP, the login process would be successful, and they would be granted access to their online banking services. However, if they exceeded the three attempts in the 10-minute time limit, their ability to login to the account would be temporarily suspended for security reasons and Alex would have to call the bank to lift the suspension or wait 2 hours to retry again. The bank implements this multi-factor authentication process with an alternative email-based OTP delivery. This approach safeguards Alex’s sensitive financial information and enhances the security of digital banking services. It also provides a backup option if the primary SMS channel is unavailable.

OTP user flow

Prerequisites

To use the code examples provided in this blog post, you’ll need to have the following AWS resources in place:

  1. AWS Account: Sign up for an AWS account at AWS website if you don’t have one.
  2. Verified Email Address in Amazon SES: Enable email delivery of OTPs by verifying an email address in Amazon SES service.
  3. AWS End User Messaging Configuration: You’ll need to configure the necessary origination identity in the AWS End User Messaging service to deliver the OTPs via SMS or voice.

With these prerequisites in place, you’ll be ready to use the code examples provided in the following sections to implement your secure OTP solution.

Architecture:

OTP architecture

Flow Explained:

  1. The user initiates the process by requesting an OTP.
  2. The request is sent through Amazon API Gateway.
  3. AWS Lambda receives the request and processes it.
  4. AWS KMS is used to encrypt the OTP for secure storage.
  5. The encrypted OTP and related information are stored in Amazon DynamoDB.
  6. AWS End User Messaging is used to send the OTP to the user via SMS, email, or voice, Amazon SES is used to send the OTP over email.
  7. When the user receives the OTP, they enter it in the portal for verification. The verification process encrypts the value with the key from AWS KMS and goes through the same flow (API Gateway -> Lambda)
  8. The Lambda decrypts the OTP for verification using KMS and compares it with the stored value in DynamoDB, which is also decrypted using the same KMS key.

Typical architecture for a secure one-time password (OTP) solution would involve the following components:

  1. Front-end Application: The OTP functionality is typically exposed through a web or mobile application, which serves as the user-facing interface.
  2. API Gateway: The front-end application interacts with the OTP solution through an API Gateway. This gateway serves as the entry point, providing scalable and secure access to underlying services.
  3. AWS Lambda: The business logic for generating, storing, and verifying the OTPs is handled by one or more AWS Lambda functions. These serverless functions are responsible for the core OTP-related operations.
  4. AWS KMS: Encrypts the OTP submitted for verification by the customer on the client side. AWS Lambda then decrypts it before verifying it against the OTP stored in Amazon DynamoDB.
  5. Amazon DynamoDB: The generated OTP encrypted and associated metadata, such as creation timestamp and expiration, are securely stored in an Amazon DynamoDB table.
  6. AWS End User Messaging: Used to deliver the OTPs to the users through various communication channels, such as SMS, and voice.
  7. Amazon SES: Deliver the OTPs to the users via email.

In a production environment, it’s also important to consider the following security measures:

  • AWS WAF (Web Application Firewall): To protect the API Gateway from common web-based attacks, such as SQL injection and cross-site scripting (XSS).
  • Authentication and Authorization Services: Ensuring that the front-end application and users are properly authenticated and authorized before accessing the OTP-related functionality. Visit Control and manage access to REST APIs in API Gateway to view the available methods of managing access to Amazon API Gateway.

This architecture enables organizations to build a comprehensive and secure one-time password solution. It protects users’ sensitive information and offers a seamless authentication experience.

Generating OTPs

To generate the OTPs, the server used the pyotp (link) library in Python. This library provides a secure random number generator to create unique, hexadecimal-encoded tokens. The server-side generation ensures that the OTPs are truly random and unpredictable, a crucial requirement for effective one-time password authentication.

The server generates a 6-character hexadecimal OTP, creating approximately 16.8 million possible unique combinations. This approach keeps codes short and easy for users to enter while maintaining security. After generation, the server securely stores the OTP and sends it to the user through the chosen delivery channel (SMS, email, or voice).

Sample Code:

import secrets
import pyotp

def generate_otp():
    """
    Generates a secure one-time password using the pyotp library.
    
    Returns:
        str: The generated one-time password.
    """
    # Generate a random base32 secret - https://pyauth.github.io/pyotp/
    totp = pyotp.TOTP(pyotp.random_base32())
    
    # Use the Time-based One-Time Password (TOTP) algorithm to generate a 6-digit OTP
    return totp.now()

It’s important to note that the generated OTP values should be encrypted on the client-side before being sent to the server for storage. This can be achieved by using AWS Key Management Service (KMS) to securely encrypt the OTP values.

By encrypting the OTP values before storing them in the DynamoDB table, you can further enhance the security of the solution and protect against potential data breaches. The encrypted values ensure that even if the database is compromised, the raw OTP values are not directly accessible.

Next, the encrypted OTP values are stored in the DynamoDB table, along with necessary metadata to manage the OTP lifecycle. This metadata includes creation timestamp, expiration, and verification attempts. The specifics of this storage process are covered in the ‘Securely Storing OTPs’ section.

Securely Storing OTPs

Once generated, the OTPs will be stored in an Amazon DynamoDB table. DynamoDB is a fully managed NoSQL database service that provides reliable, high-performance data storage and retrieval, making it an ideal choice for our secure OTP solution. To store the OTPs, you’ll create a DynamoDB table with the user_id as the primary key. This approach ensures that the same user can’t generate multiple OTPs. The put_item() operation will fail if it encounters a duplicate user_id value. Depending your use case, you can change this to be a random id or a concatenation of the user id and a random id.

Once generated, the OTPs are stored in an Amazon DynamoDB table. DynamoDB, a fully managed NoSQL database service, provides reliable, high-performance data storage and retrieval, making it ideal for our secure OTP solution.

To store the OTPs, create a DynamoDB table with the user_id as the primary key. This approach allows for efficient retrieval of a user’s current OTP. When storing a new OTP for a user:

  1. If no existing entry is found for the user_id, a new item is created.
  2. If an entry already exists, it’s updated with the new OTP, effectively overwriting the old one.

This method ensures that each user has only one active OTP at a time, while still allowing users to request new OTPs when needed (for example, if the previous one expired).

Depending on your use case, you can modify the primary key to be a random id or a concatenation of the user id and a random id for additional security.

In addition to the user_id and otp_code, we’ll also include the following attributes:

    • creation_timestamp: The timestamp indicating when the OTP was generated. This is compared with the timestamp of each attempt to ensure all attempts fall within the allowed time window.
    • ttl: The Unix timestamp representing the time-to-live (TTL) for the OTP, after which the DynamoDB item will be automatically deleted. Set this value to 24 hours from the creation time. This allows for a reasonable cleanup period while ensuring expired OTPs are removed from the database.
    • attempts: The number of remaining verification attempts for the OTP.
    • verified: A boolean flag indicating whether the OTP has been successfully verified.
    • locked: A boolean flag indicating whether the user’s account has been locked due to exhausted verification attempts.

Sample Code:

import time 
import boto3 
from datetime import datetime, timedelta 

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('otp_main')
def store_otp(user_id, otp_code):
    """
    Stores the generated one-time password in an Amazon DynamoDB table with a creation timestamp, TTL, and remaining attempts.
    
    Args:
        user_id (str): The unique identifier for the user.
        otp_code (str): The generated one-time password.
    
    Returns:
        dict: The response from the DynamoDB put_item operation.
    """
    # Get the current timestamp
    creation_timestamp = datetime.now().isoformat()
    
    # Calculate the expiration time for the OTP (10 minutes from now)
    expiration_time = datetime.now() + timedelta(minutes=10)
    
    # Convert the expiration time to a Unix timestamp for the DynamoDB TTL
    ttl_value = int(time.mktime(expiration_time.timetuple()))
    
    # Store the OTP, creation timestamp, TTL, remaining attempts, and verification status in the DynamoDB table
    response = table.put_item(
        Item={
            'user_id': user_id,
            'otp_code': otp_code,
            'creation_timestamp': creation_timestamp,
            'ttl': ttl_value,
            'attempts': 3,
            'verified': False,
            'locked': False
        }
    )
    
    return response

We use the user_id as the primary key and store creation timestamp, TTL, remaining attempts, verification status, and account lock status. This approach ensures a secure and efficient OTP storage and retrieval process. This approach also allows for precise management of OTP expiration and account locking, as demonstrated in the Verifying OTPs section.

The encrypted OTP values are stored in the otp_code attribute. Encryption is performed on the client-side using a secure key management solution, like the AWS KMS client-side library. This ensures that the raw OTP values are never transmitted or stored in plain text, further enhancing the security of the solution.

Note: As an optional enhancement, you could use Amazon SQS with a visibility timeout set to the OTP validity period. A payload containing the user_id is sent to a Lambda function. After the visibility timeout, the function processes the SQS message and deletes the corresponding DynamoDB item. This approach provides greater precision compared to relying solely on DynamoDB TTL, though it adds complexity to the implementation. The current solution compares each verification attempt’s timestamp with the creation timestamp, ensuring that no attempts occur after the OTP has expired.

Delivering OTPs via Multiple Channels

Now that we have a secure way to generate and store the OTPs, it’s time to focus on delivering them to your users. Our solution leverages the AWS End User Messaging capabilities to provide a seamless and redundant OTP delivery experience across multiple communication channels.

Sending OTPs via SMS and Voice

AWS End User Messaging offers a versatile platform for OTP delivery across multiple channels, including email, SMS, voice calls, push notifications, and WhatsApp. This provides a redundant and convenient authentication experience for your users, ensuring they can receive their one-time passwords via their preferred method.

Sample Code:

import boto3

def send_otp_sms(mobile_number, otp_code, user_id, region_name):
    """
    Sends an OTP code to the user's mobile number using AWS End User Messaging SMS.
    
    Args:
        mobile_number (str): The phone number to send the OTP to.
        otp_code (str): The one-time password to be sent.
        user_id (str): The unique identifier for the user.
        region_name (str): The AWS region to use for the SESv2 client.
    
    Returns:
        dict: The response from the End User Messaging SMS send_text_message operation.
    """
    # Construct the SMS message with the OTP code
    message = f"""
    This is an AWS End User Messaging OTP message.

    Your one-time password is: {otp_code}.
    """
    
    try:
        # Create a new SMS-Voice v2 client
        aws_sms = boto3.client('pinpoint-sms-voice-v2')
        # Use the End User Messaging SMS client to send the SMS message
        response = aws_sms.send_text_message(
            DestinationPhoneNumber=mobile_number,
            MessageBody=message,
            MessageType='TRANSACTIONAL'
        )
        return {'StatusCode': 200, 'Response': response['MessageId']}
    except ClientError as e:
        error_message = e.response['Error']['Message']
        return {'StatusCode': 500, 'Response': error_message}

Sending OTPs via Email

To deliver OTPs via email, we’ll use the Amazon SES (Simple Email Service) SendEmail API. SES is a highly scalable and cost-effective email service. It can send notifications, alerts, and in our case, one-time passwords to users.

Sample Code:

import boto3
from botocore.exceptions import ClientError

def send_otp_email(user_id, email_address, otp_code, region_name):
    """
    Sends an OTP code to the user's email address using Amazon SESv2.
    
    Args:
        user_id (str): The unique identifier for the user.
        email_address (str): The email address to send the OTP to.
        otp_code (str): The one-time password to be sent.
        region_name (str): The AWS region to use for the SESv2 client.
    
    Returns:
        dict: The response from the SESv2 send_email operation.
    """
    try:
        # Create a new SESv2 client
        ses = boto3.client('sesv2', region_name=region_name)

        # Construct the email message with the OTP code
        message = "<p>Your one-time password is: </p> {otp_code}"
        html_body = message.format(otp_code=otp_code)

        # Use the SESv2 client to send the email
        response = aws_email.send_email(
            FromEmailAddress='[email protected]',
            Destination={
                'ToAddresses': [
                    email_address,
                ]
            },
            Content={
                'Simple': {
                    'Subject': {
                        'Charset': 'UTF-8',
                        'Data': 'Your AWS OTP code'
                    },
                    'Body': {
                        'Html': {
                            'Charset': 'UTF-8',
                            'Data': html_body
                        }
                    }
                }
            }
        )
        return {'StatusCode': 200, 'Response': response['MessageId']}
    except ClientError as e:
        error_message = e.response['Error']['Message']
        return {'StatusCode': 500, 'Response': error_message}

Verifying OTPs

The final piece of our secure OTP solution is the process of verifying the one-time passwords entered by your users. This is a crucial step in the authentication flow, as it ensures that only legitimate users are granted access to your applications or services.

The OTP verification logic is handled by a Lambda function that interacts directly with the DynamoDB table where the OTPs are stored. This Lambda function performs the following steps:

  1. Retrieve the stored OTP and its associated metadata from the DynamoDB table, using the user_id as the primary key. This metadata includes the creation timestamp and the number of remaining attempts.
  2. Decrypt the retrieved OTP value using the KMS client-side library, as the OTP was encrypted on the client side before being stored.
  3. Compare the unencrypted OTP value with the one entered by the user.
  4. Verify that the OTP has not expired by comparing the creation timestamp with the current time.
  5. If the OTP is valid and not expired, update the verification status in the DynamoDB table and delete the corresponding item.
  6. If the OTP is invalid or expired, deduct an attempt from the remaining attempts count stored in the DynamoDB table.
  7. If the remaining attempts count reaches zero, lock the user’s account and return an appropriate response.

Sample Code:

import boto3
from boto3.dynamodb.conditions import Key
from datetime import datetime, timedelta

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('otp_main')

def verify_otp(otp_entered, user_id):
    """
    Verifies the one-time password entered by the user against the stored OTP in DynamoDB.
    
    Args:
        otp_entered (str): The one-time password entered by the user.
        user_id (str): The unique identifier for the user.
    
    Returns:
        dict: The result of the OTP verification, containing the verification status and the OTP code.
    """
    try:
        # Query the DynamoDB table to find the stored OTP for the given user
        response = table.query(
            KeyConditionExpression=Key('user_id').eq(user_id)
        )
        
        if 'Items' in response and response['Items']:
            for item in response['Items']:
                # decrypt the retrieved OTP value using the KMS client-side library
                if str(otp_entered) == decrypt_otp(item['otp_code'], user_id):
                    # Check if the OTP has expired
                    creation_timestamp = datetime.fromisoformat(item['creation_timestamp'])
                    if datetime.now() - creation_timestamp < timedelta(minutes=10):
                        # Update the verification status and delete the DynamoDB item
                        update_item = table.update_item(
                            Key={'user_id': user_id},
                            UpdateExpression='SET verified = :verified',
                            ExpressionAttributeValues={':verified': True}
                        )
                        table.delete_item(Key={'user_id': user_id})
                        return {'result': True, 'otp': item['otp_code']}
                    else:
                        # Deduct an attempt from the remaining attempts count
                        update_item = table.update_item(
                            Key={'user_id': user_id},
                            UpdateExpression='SET attempts = attempts - :1',
                            ExpressionAttributeValues={':1': 1}
                        )
                        if item['attempts'] <= 0:
                            # Lock the account if the attempts are exhausted
                            update_item = table.update_item(
                                Key={'user_id': user_id},
                                UpdateExpression='SET locked = :locked',
                                ExpressionAttributeValues={':locked': True}
                            )
                        return {'result': False, 'otp': item['otp_code']}
                else:
                    # Handle invalid OTPs
                    pass
        
        # If the OTP is not found or does not match, return a failure result
        return {'result': False, 'otp': None}
    except Exception as e:
        error_message = str(e)
        return {'result': False, 'error': error_message}

def decrypt_otp(encrypted_otp, user_id):
    """
    decryptes the OTP value using the KMS client-side library.
    
    Args:
        encrypted_otp (str): The encrypted OTP value stored in DynamoDB.
        user_id (str): The unique identifier for the user.
    
    Returns:
        str: The unencrypted OTP value.
    """
    # decrypt the OTP value using the KMS client-side library
    return decrypt_using_kms(encrypted_otp, user_id)

 

In this implementation, a Lambda function handles the OTP verification logic. This ensures sensitive operations like OTP decryption and managing expiration and attempt counts occur in a secure, serverless environment.

Best Practices

As you implement a secure one-time password solution, it’s important to consider the following best practices:

OTP Message Best Practices

When delivering OTPs via email, SMS, or voice, clearly specify the sender and the content of the message. For example, the email subject and body, as well as the SMS or voice message, should include information like:

“This is a one-time password from [Company Name] for payment confirmation of your flight ABC123.”

Security Reminder in OTP Messages

Include a security reminder in the OTP message to encourage users to report any unauthorized access attempts. For example:

“If you did not request this OTP, please call [phone number] to report it.”

This helps raise user awareness and provides a clear course of action if they suspect their account has been compromised.

Configuration Set / Originating Identity

Include appropriate configuration sets and Context or EmailTags when using AWS End User Messaging services to deliver OTPs. This records message delivery events and traces them to your organization. Read more about Amazon SES and AWS End User Messaging configuration sets.
For example, in the send_otp_sms() and send_otp_email() functions, you should include the following parameters:

response = aws_sms.send_text_message(
    DestinationPhoneNumber=mobile_number,
    MessageBody=message,
    MessageType='TRANSACTIONAL',
    ConfigurationSetName='otp-config-set',
    OriginationNumber='+12345678901',
    Context={
        'user_id': user_id
    }
)
response = ses.send_email(
    FromEmailAddress='[email protected]',
    Destination={'ToAddresses': [email_address]},
    Content={
        # ...
    },
    ConfigurationSetName='otp-config-set',
    EmailTags=[
        {
            'Name': 'user_id',
            'Value': user_id
        }
    ]
)

Deleting OTPs After Verification

After a successful OTP verification, it’s recommended to delete the corresponding DynamoDB item. This helps maintain a clean and up-to-date database, reducing the risk of unauthorized access or potential data breaches.

Tracking Verification Attempts

Consider adding a column in the DynamoDB table to track the number of verification attempts for each OTP. This can help you implement rate-limiting and other security measures to prevent brute-force attacks.

Encrypting OTPs on the Client-side

As mentioned earlier, the OTP values should be encrypted on the client-side using a secure key management solution, such as the AWS KMS client-side library. This ensures that the raw OTP values are never transmitted or stored in plain text, further enhancing the security of the solution.

Following these best practices ensures your one-time password solution is secure and user-friendly. It also maintains necessary controls and traceability for production use cases.

Conclusion

In this guide, we’ve demonstrated a secure, multi-channel One-Time Password (OTP) solution using AWS services. You can now generate, store, and deliver OTPs via email, SMS, and voice channels using Amazon DynamoDB, Amazon SES, and AWS End User Messaging.

We’ve covered several important points throughout this process. We discussed using a secure random number generator and encrypting algorithms to generate and store OTPs. This ensures strong protection for your users’ sensitive information. By integrating with Amazon SES and AWS End User Messaging, you provide users with a convenient, redundant authentication experience through multiple channels.

This guide equips you with tools to maintain SMS-based OTP capabilities. However, it’s important to note the industry’s shift towards more secure, phishing-resistant authentication methods. These include passwordless solutions and hardware security keys. We encourage you to explore and implement these newer technologies as you develop your OTP solution.

Looking ahead, consider potential enhancements to this solution. Integrating support for standards like FIDO2 WebAuthn and Passkeys could allow seamless authentication without traditional OTPs. Keep these options as backup or alternative methods. Also, consider incorporating a mechanism to escalate users to live support for authentication issues.

Implement the secure OTP solution outlined in this guide and continuously update your authentication strategies. This approach ensures your organization remains equipped to protect users and assets from evolving digital threats.

Build a Two-Way Email-to-SMS Service with Amazon SES and Amazon End User Messaging

Post Syndicated from Cheng Wang original https://aws.amazon.com/blogs/messaging-and-targeting/build-a-two-way-email-to-sms-service-with-amazon-ses-and-amazon-end-user-messaging/

Introduction

Businesses and organizations today struggle to effectively communicate with their customers, employees, or other stakeholders across the diverse range of digital channels they now use. One common problem arises when the requirement to exchange information quickly and reliably extends beyond traditional email. This issue challenges organizations where recipients lack immediate access to email. This applies to field workers, remote teams, or customers who prefer to communicate via text messages. It is vital to bridge this gap between email and SMS communication for timely updates, urgent notifications, and seamless collaboration. However, separate management of these disparate channels independently proves cumbersome and leads to inefficiencies.

To address this challenge, one approach is to leverage Amazon Simple Email Service (SES) and Amazon End User Messaging services to create a robust, scalable, and cost-effective messaging system. This system seamlessly bridges the gap between email and SMS, enhances the reach and delivery of your messages and streamlines your communication workflows. Ultimately, this approach delivers a superior experience for your audience, ensuring that critical information reaches recipients through their preferred channels in a timely and efficient manner.

This blog post will delve into the step-by-step process of building a solution that enables both Email-to-SMS and SMS-to-Email communication. This solution allows you to send SMS messages using email and receive any SMS replies on the same email address you used to send the original message. Furthermore, you can continue the conversation by replying to the email you receive in response. By the end, you’ll have the knowledge and tools necessary to revolutionize your communication strategy and deliver a superior experience to your audience.

Here are some of the use cases for this solution:

  • Real estate agents can use this solution to send listing updates to clients via SMS, and then receive client inquiries and responses as emails.
  • Customer service team can leverage the Email to SMS functionality to proactively reach out to customers with important notifications. Customers are able to respond directly via SMS.
  • Retailers can use this solution to send order confirmations, shipping updates, and promotional offers to customers via SMS. Customers are able to respond with inquiries or feedback that are then received as emails.
  • Medical practices and hospitals can leverage the Email to SMS functionality to quickly notify patients of appointment reminders, prescription refills, or other time-sensitive information. Patients can then respond via SMS, which gets converted to an email that the healthcare staff can access.

Solution Overview

The following figure shows the high level architecture for this solution.

Figure 1: Two-Way Email-To-SMS architecture

  1. Email Users send an email to the email address formatted as “mobile-number@verified-domain”. Amazon SES email receiving receives the email and triggers a receipt rule.
  2. The email is published to Amazon Simple Notification Service (SNS) topic (EmailToSMS) based on the receipt rule action, which triggers an AWS Lambda function (ConvertEmailToSMS). The ConvertEmailToSMS Lambda function performs the following actions:
    1. Receives the event from SNS and constructs a text message using the email body content.
    2. The constructed message is then sent to the “mobile-number” in the destination email address using the “SendTextMessage” API from AWS End User Messaging SMS. This is achieved by using a phone number in AWS End User Messaging SMS as the origination identity.
    3. The SMS message ID and source email address are stored as items in the Amazon DynamoDB table (MessageTrackingTable). This tracks email addresses for replies from SMS users.
  3. Mobile Users receive the SMS, and they have the option to reply to the phone number with two-way SMS messaging
  4. AWS End User Messaging receives the incoming SMS message from the Mobile Users. It then publishes this message to a SNS topic (SMSToEmail) for two-way SMS integration, which triggers a Lambda function (ConvertSMSToEmail). The ConvertSMSToEmail Lambda function performs the following actions:
    1. Retrieves the item from “MessageTrackingTable” using “previousPublishedMessageId” (SMS message ID) from the SNS event, and locate the corresponding email address.
    2. Sends the SMS message body to the Email Users using SES. This step uses “mobile-number@verified-domain” as the source email address, and the email address retrieved from the previous step as the destination.
  5. Email Users receive the email, and they have the option to reply to the email to continue the conversation. Mobile Users will receive the latest reply from Email Users.

Walkthrough

This section will dive into the step-by step process for the deployment. There are 4 steps to deploy this solution:

  1. Configure SES verified identity for email receiving and sending.
  2. Deploy the CloudFormation stack for the Email to SMS functionality.
  3. Deploy the CloudFormation stack for the SMS to Email functionality.
  4. Set up two-way SMS messaging in AWS End User Messaging SMS.

Note: the Lambda code for this solution is developed based on phone numbers and long code as the supported origination identity in Australia. You need to adjust the Lambda code (“format_phone_number” function) accordingly for this to work in your country.

Refer to this GitHub repository for the solution source code.

Prerequisites

Prerequisites for this walkthrough:

  1. Administrator-level access to an AWS account
  2. A domain or subdomain that you own to create SES verified identity
  3. An origination identity that supports two-way messaging, following Choosing an origination identity for two-way messaging use cases. Simulator phone numbers are available if you are in the US
  4. A mobile phone to send and receive SMS
  5. An email address to send and receive emails

Step 1: Configure SES Verified Identity

Follow the steps outlined in Creating a domain identity to create a verified identity for your domain in your AWS account. Confirm your domain identity is in the “Verified” status before proceeding to the next step:

Figure 2: Verified Identity

Step 2: Deploy Email to SMS functionality

The following steps create a CloudFormation stack to deploy the required components for Email to SMS functionality:

  1. Sign in to your AWS account.
  2. Download the email-to-sms.yaml for creating the stack.
  3. Navigate to the AWS CloudFormation page.
  4. Choose Create stack, and then choose With new resources (standard).
  5. Choose Upload a template file and upload the CloudFormation template that you downloaded earlier: email-to-sms.yaml. Then choose Next.
  6. Enter the stack name Email-To-SMS.
  7. Enter the following values for the parameters:
    • RuleName: The name of your SES Rule Set and receipt rule.
    • Recipient1: Domain name used for recipient condition in the SES Rule Set.
    • Recipient2: Domain name used for recipient condition in the SES Rule Set if you need additional recipients.
    • PhoneNumberId: Phone number ID of the phone number to send SMS messages.
  8. Choose Next, and then optionally enter tags and choose Submit. Wait for the stack creation to finish.

Now you have the required components to convert email to text, and sending it as SMS to a phone number using AWS End User Messaging SMS.

Note: if required, modify the following code in email-to-sms.yaml to format your phone numbers accordingly:

def format_phone_number(email_address):

    # Extract the local part of the email address (before @)

    local_part = email_address.split('@')[0]   

    # Remove the leading '0' and add '+61' for phone number (Australia)

    if local_part.startswith('0'):

        formatted_number = '+61' + local_part[1:]

    return formatted_number

Step 3: Deploy SMS to Email functionality

The following steps create a CloudFormation stack to deploy the required components for SMS to Email functionality:

  1. Sign in to your AWS account.
  2. Download the sms-to-email.yaml for creating the stack.
  3. Navigate to the AWS CloudFormation page.
  4. Choose Create stack, and then choose With new resources (standard).
  5. Choose Upload a template file and upload the CloudFormation template that you downloaded earlier: sms-to-email.yaml. Then choose Next.
  6. Enter the stack name SMS-To-Email.
  7. Enter the following values for the parameters:
    • EmailDomain: The email domain to use for the SMS-to-Email function
  8. Choose Next, and then optionally enter tags and choose Submit. Wait for the stack creation to finish.

Note: if required, modify the following code in sms-to-email.yaml to format your phone numbers accordingly:

def format_phone_number(phone_number):

    # Replace the '+61' with '0' from the phone number (Australia)

    formatted_number = f"0{mobile_number[3:]}"

    return formatted_number

Step 4: Set up Two-Way Messaging in AWS End User Messaging SMS

Follow the steps 1 – 5 outlined in Set up two-way SMS messaging for a phone number in AWS End User Messaging SMS.

For step 6:

  • For Destination type, choose Amazon SNS.
  • Choose Existing SNS standard topic.
  • For Incoming messages destination, choose the SNS topic created from Step 3 (default topic name is SMSToEmailTopic).
  • For Two-way channel role, choose Use SNS topic policies.
  • Choose Save changes.

This allows your origination identity (phone number) to receive incoming messages, which is then published to an SNS topic and converted into emails by Lambda.

Testing

To test the solution, send an email with the destination address of “mobile-number@verified-domain”. You should receive a SMS on your mobile with the following information:

Note: AWS End User Messaging SMS has character limit for SMS messages depending on the type of characters the message contains. This solution takes the first 160 characters of the email body by default, you can adjust the EmailToSMS Lambda function as required.

Reply directly to the SMS, you should receive an email at the same email address that sent the original email, with the following information:

  • Subject: Re:mobile-number
  • Body: SMS message content
  • Source email address: mobile-number@verified-domain

If you are not receiving the email or SMS, check the Lambda CloudWatch logs for troubleshooting.

Cleaning up

To remove unneeded resources after testing the solution, follow these steps:

  1. In the CloudFormation console, delete the Email-To-SMS stack
  2. In the CloudFormation console, delete the SMS-To-Email stack
  3. If applicable, in Amazon SES, delete the verified identities
  4. If applicable, in AWS End User Messaging, release the unused phone numbers

Additional Consideration

Conclusion

This blog has explored how organizations can leverage AWS services to build a flexible, two-way communication solution bridging the gap between email and SMS channels. By integrating Amazon SES and Amazon End User Messaging, businesses can reach their audience through multiple channels, ensuring timely and effective delivery of critical messages.

The detailed guide provided the knowledge to create a scalable, cost-effective system tailored to evolving communication needs – whether facilitating email-to-SMS or SMS-to-email exchanges. This unified approach to email and SMS capabilities empowers companies to address the common challenge of managing disparate communication platforms, streamlining workflows and enhancing responsiveness.

If you run into issues or want to submit a feature request, use the New issue button under the issues tab in the GitHub repository.

How Amazon SES Mail Manager customers can prevent EchoSpoofing

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/how-ses-mail-manager-customers-can-prevent-echospoofing/

Customers not using Amazon SES Mail Manager, or those leveraging the authenticated SMTP functionality, are not at risk of EchoSpoofing. In such cases, no further action is required.

However, customers currently using or evaluating the unauthenticated SMTP relay feature of Amazon SES Mail Manager are strongly advised to review and implement the guidance provided in this blog post.

A new type of email spoofing attack

In July 2024, the researchers at Guardio Labs disclosed a new type of email spoofing (authentication bypass) attack they called “EchoSpoofing”. The attackers successfully sent spoofed emails by redirecting them through a virtual SMTP server, Office365 Exchange Online server, and a trusted third-party SMTP relay service. This path provided the fraudulent messages a means to pass standard authentication checks. Fortunately, the Guardio Labs researchers responsibly disclosed the issue to the targeted email security provider, leading to a speedy and effective remedy.

Unfortunately, before addressing the vulnerability completely, cybercriminals executed a series of sophisticated phishing campaigns. These campaigns involved sending millions of fraudulent emails that had valid Sender Policy Framework (SPF) and DomainKeys Identified Mail (DKIM) from well-known consumer brands.

The EchoSpoofing incident reminds both providers and customers to adopt a “trust-but-verify” approach to email security. This is especially true when mail routing functions have been, or are, in the process of being decoupled from single-tenant, on-premise (or cloud) email infrastructures.

As a leading provider of global managed email infrastructure, the Amazon SES service team went to work immediately after the Guardio Labs announcement. The Amazon SES team scrutinized the announcement and remediations undertaken by both Microsoft and the third-party SMTP relay service to fully understand the EchoSpoofing exploits and devised methods to swiftly safeguard Amazon SES customers.

Although we won’t delve into Guardio Labs’ in-depth analysis, it is crucial to grasp the attack’s main elements and examine how malicious individuals could exploit supposedly “secure” email relay systems. Amazon SES has taken steps to safeguard against EchoSpoofing and similar attacks, urging its customers to do the same.

Understanding the EchoSpoofing attack

The bad actors who implemented EchoSpoofing were able to send millions of well camouflaged malicious messages through the trusted delivery path of targeted organizations by preserving the SPF and DKIM attributes of the targeted sender’s domain. This greatly increased the likelihood of recipients trusting and acting upon the fake messages.

The attacker first set up a tenant in Microsoft Office365 and then delivered a spoofed email to that tenant, falsifying the from email headers. The attacker-controlled tenant was configured to relay the email to a security relay point linked to the forged sending identity. As the forged email came from trusted IP addresses belonging to Microsoft, the security relay point signed the message and relayed to the final recipients.

The attackers had amassed a large inventory of high-profile domains, and spread the EchoSpoof campaigns out across them to smooth the traffic and avoid sending spikes from any single domain. They carried out this attack for several months undetected, sending as many as 14 million messages per day, targeting the users of the compromised domains’ email services. This made the attack easy to automate at scale, difficult to detect via automated means, and highly successful in delivering malicious emails to unsuspecting recipients.

Guardio Labs’ discovery highlights the risks associated with an insecure SMTP relay model in a trusted domain configuration. This is of particular concern when permissive security policies allow fraudulent emails to be injected into the email flow without raising alarms.

The AWS shared security model for SES

As an email sender, Amazon SES is one of the largest on the internet, operating a worldwide fleet of trusted mail relays. Amazon SES maintains high IP reputations for this large fleet by maintaining a tight focus on robust, evolving security practices.

At AWS we operate under the shared security model. For Amazon SES, this means AWS takes responsibility for securing the underlying email delivery infrastructure, including the email servers, networks, and physical data centers. Customers take responsibility for securing their configurations, email content, sender authentication, and email lists they use with Amazon SES.

To ensure we meet our obligations in the shared security model, Amazon SES has recently added new features to Mail Manager SMTP relays that provide an increased level of protection to help guard against exploits like EchoSpoofing. Theses features are live in every AWS Region where Mail Manager is accessible.

We have outlined our recommendations and updated Amazon SES Mail Manager configurations in this blog to help customers meet their obligations and strengthen their Amazon SES email infrastructure against EchoSpoofing. As noted above, authenticated SMTP relays are not subject to this exploit.

Prevent EchoSpoofing when relaying email out of MailManager

If you need to relay email to a third party system that cannot enforce SMTP authentication, our recommendation is to limit access to the IP addresses used by Mail Manager in your region.

As of this writing, Mail Manager is generally available, and has its own IP range, in six commercial regions (below). As Amazon expands SES Mail Manager availability into more regions the IP ranges will be updated in the Amazon SES documentation .

SES Mail Manager regions

SES Mail Manager IP ranges as of 10/23/2024

When unable to enforce SMTP authentication, we recommend configuring your SMTP servers, or third party software for the new MIME header"X-MAIL-MANAGER-ORIGINATOR-ORG". This new Mail Manager header is now automatically inserted into messages relayed by Mail Manager. The X-MAIL-MANAGER-ORIGINATOR-ORG will be set to the customer’s unique SMTP relay ID, which can be found via the Mail Manager console or the ListSmtpRelays API.

In addition to added security, the MIME header feature can also be used in message search and filtering behaviors for a wide range of MIME header name:value pairs.

If the original email already contains an X-MAIL-MANAGER-ORIGINATOR-ORG header, it will be replaced with the last MailManager SMTP relay ID to relay the email. Here is an example of an email relayed by MailManager with the header:

MIME-Version: 1.0
From: [email protected]
To: [email protected]
Subject: Test
X-SES-REDIRECT-MESSAGE-ID: <[email protected]>
X-MAIL-MANAGER-ORIGINATOR-ORG: rl-usmoots8mgmfgfaeijckxhqx
X-SES-Outgoing: 2024.08.26-76.223.191.14

--===============1760803815732220490==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

This is a sample message. Have a nice day.
--===============1760803815732220490==--

This approach elevates your security posture because the IP access control lists on your third party system ensures only mail from Amazon SES is accepted, and the MIME header check can be trusted and checked.

Prevent EchoSpoofing when relaying into MailManager

When relaying email from a third party into Amazon SES Mail Manager, you will similarly need to configure an IP allowlisting, and if the email comes from a shared or cloud environment, you will need an additional header check to disambiguate among the multiple tenants it hosts. Those IPs and headers are provider specific, for example, emails coming from Office365 will have a header called X-OriginatorOrg.

You can use the rule editor screen in MailManager to configure the check in Mail Manager for the IPs and 3rd party headers before executing any action.

Mail Manager Rules Detail Page

The verification of a MIME header is not necessary if the third party relaying into MailManager uses an IP dedicated for your tenant. In that case, there is no possibility that an attacker tenant injects email into MailManager using your IP.

Conclusion

While the conditions that made the EchoSpoofing exploit were highly specific, they reminded us all of the importance of taking a proactive approach to email security.

The chances of your Amazon SES Mail Manager unauthenticated SMTP relay being compromised are low, but we highly advise you follow the recommendations in this blog post promptly. You’ll find more information in the Amazon SES documentation ( here ).

If you need help with securing your Amazon SES Mail Manager SMTP relay actions against EchoSpoofing, contact AWS support, or leave us a comment in community section of the blog post.

Call to Action:

If you are using SES Mail Manager’s unauthenticated SMTP relay today, follow the guidance in this blog to secure your email infrastructure today by configuring the recommended ACLs and MIME header verification in AWS SES Mail Manager.“

Stay ahead of emerging threats by subscribing to this AWS blog where we post the latest security updates as well as new features and interesting use cases for SES.

Join the conversation and share your best practices for email security with the AWS community.

Explore the new MIME header evaluation feature in AWS SES Mail Manager and share your creative use cases with us and the SES community via the community comments section of the blog.

About the Authors

Toby Weir-Jones

Toby Weir-Jones

Toby is a Principal Product Manager for Amazon SES and WorkMail. He joined AWS in January 2021 and has significant experience in both business and consumer information security products and services. His focus on email solutions at SES is all about tackling a product that everyone uses and finding ways to bring innovation and improved performance to one of the most ubiquitous IT tools.

Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

Leandro Batista Lameiro

Leandro Batista Lameiro

Leandro is a Sr. Software Dev Engineer at AWS.

Linzhou Zhong

Linzhou Zhong

Linzhou is a software engineer at AWS.

Leverage IAM Roles for email sending via SES from EC2 and eliminate a common credential risk

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/leverage-iam-roles-for-email-sending-via-ses-from-ec2-and-eliminate-a-common-credential-risk/

Sending automated transactional emails, such as account verifications and password resets, is a common requirement for web applications hosted on Amazon EC2 instances. Amazon SES provides multiple interfaces for sending emails, including SMTP, API, and the SES console itself. The type of SES credential you use with Amazon SES depends on the method through which you are sending the emails.

In this blog post, we describe how to leverage IAM roles for EC2 instances to securely send emails via the Amazon SES API, without the need to embed IAM credentials directly in the application code, link to a shared credentials file, or manage IAM credentials within the EC2 instance. By adopting the approach outlined in this blog, you can enhance security by eliminating the risk of credential exposure and simplify credential management for your web applications.

Solution Overview

Below we provide step-by-step instructions to configure an IAM role with SES permissions to use on your EC2 instance. This allows the EC2 hosted web application to securely send emails via Amazon SES without storing or managing IAM credentials within the EC2 instance. We present an option for running EC2 and SES in the same AWS account, as well as an option to accommodate running EC2 and SES in different AWS accounts. Both options offer a way to enhance security and simplify credential management.

Either option begins with creating an IAM role with SES permissions. Next, the IAM role is attached to your EC2 instance, providing it with the necessary permissions for SES without needing to embed IAM credentials in your application code or on a file in the EC2 instance. In option 2, we’ll add cross-account permissions that allow the code on the EC2 instance in account “A” to send email via the SES API in account “B”. We also provide a sample Python script that demonstrates how to send an email from your EC2 instance using the attached IAM role.

Option 1 – SES and EC2 are in a single AWS account

In a typical scenario where an EC2 instance is operating in the same AWS account as SES, the process of using an IAM role to send emails via SES is straightforward. In the steps below, you’ll configure and attach an IAM role to the EC2 instance. You’ll then update a sample Python script to use the permissions provided by the attached IAM role to send emails via SES. This direct access simplifies the SES sending process, as no explicit credential management is required in the code, nor do you need to include a shared credentials file on the EC2 instance.

Option_1-Single_AWS_Account

EC2 & SES in the same AWS Account

Prerequisites – single AWS account for EC2 and SES

  • A single AWS account in a region that supports SES
  • Verified domain or email identity in Amazon SES.
    • Make note of a verified sending email address here: ___________
  • EC2 instance (Linux) in running state
    • If you don’t have a EC2 instance create one (Linux)
  • Administrative Access to Amazon SES, IAM and EC2 consoles.
  • Access to a recipient email address to receive test emails from the python script.
    • Make note of a SES verified recipient email address to send test emails here: ___________

Step 1 – Create IAM Role for EC2 instance with SES Permissions

To start, create an IAM role that grants the necessary permissions to send emails using Amazon SES by following these steps:

  • Sign in to the AWS Management Console and open the IAM console.
  • In the navigation pane, choose “Roles,” and then choose “Create role.”
  • Choose the trusted entity type as “AWS service” and select “EC2” as the service that will use this role, then click ‘Next
  • Search for and select the “AmazonSESFullAccess” policy from the list (or create a custom policy with the necessary SES permissions), then click ‘Next’.
  • Provide a name for your role (e.g., EC2_SES_SendEmail_Role).
  • Click “Create role“.

Step 2 – Attach the IAM Role to EC2 instance.

Next, attach the IAM role to your EC2 instance:

  • Open the EC2 Management Console.
  • In the navigation pane, choose “Instances,” and select the running EC2 instance to which you want to attach the IAM role.
  • With the instance selected, choose “Actions,” then “Security,” and “Modify IAM role.
  • Choose the IAM role you created (EC2_SES_SendEmail_Role) from the drop-down menu and click “Update IAM role.”

Step 3 – Create a sample python script that sends emails from the EC2 instance with the attached role.

  • Now that your EC2 instance is configured with the necessary permissions, you can set up an example Python script to send emails via Amazon SES using the IAM Role. Here, we’re using the AWS SDK for Python (Boto3), a powerful and versatile library to interact with the SES API endpoint. Before running the example script, ensure that Python, pip (the package installer for Python), and the Boto3 library are installed on your EC2 instance:
    • Run the ‘python3 –version‘ command to check if Python is installed on your EC2 instance. If Python is installed, the version will be displayed, otherwise you’ll receive a ‘command not found’ or similar error message.
      • If python is not installed, run the command ‘sudo yum install python3 -y
    • Run the ‘pip3 --version‘ command to check if pip is installed on your EC2 instance. If pip3 is installed, is installed, the version will be displayed, otherwise you’ll receive a ‘command not found’ or similar error message.
      • If pip3 is not installed, run the command ‘sudo yum install python3-pip
    • Install the Boto3 Library which allows Python scripts to interact with AWS services including SES. Run the command ‘pip3 install boto3‘ to install (or update) Boto3 using pip.
  • Save the code below as a Python file named ‘sesemail.py‘ on your EC2 instance.
  • Edit 'sesemail.py‘ and replace the placeholder values of SENDER, RECIPIENT, and AWS_REGION with your values (see prerequisites). Do not modify any “” marks.

[copy]

import boto3
from botocore.exceptions import ClientError

SENDER = "[email protected]"
RECIPIENT = "[email protected]"
#CONFIGURATION_SET = "ConfigSet"
AWS_REGION = "us-west-2"
SUBJECT = "Amazon SES Test Email (SDK for Python) using IAM Role"
BODY_TEXT = ("Amazon SES Test (Python)\r\n"
             "This email was sent with Amazon SES using the "
             "AWS SDK for Python (Boto)."
            )
            
BODY_HTML = """<html>
<head></head>
<body>
  <h1>Amazon SES Test (SDK for Python) using IAM Role</h1>
  <p>This email was sent with
    <a href='https://aws.amazon.com/ses/'>Amazon SES</a> using the
    <a href='https://aws.amazon.com/sdk-for-python/'>
      AWS SDK for Python (Boto)</a>.</p>
</body>
</html>
            """            

CHARSET = "UTF-8"

client = boto3.client('ses',region_name=AWS_REGION)

try:
    response = client.send_email(
        Destination={
            'ToAddresses': [
                RECIPIENT,
            ],
        },
        Message={
            'Body': {
                'Html': {
                    'Charset': CHARSET,
                    'Data': BODY_HTML,
                },
                'Text': {
                    'Charset': CHARSET,
                    'Data': BODY_TEXT,
                },
            },
            'Subject': {
                'Charset': CHARSET,
                'Data': SUBJECT,
            },
        },
        Source=SENDER,
    )   
except ClientError as e:
    print(e.response['Error']['Message'])
else:
    print("Email sent! Message ID:"),
    print(response['MessageId'])
  • Run ‘python3 sesmail.py‘ to execute the Python script.
  • When ‘python3 sesmail.py‘ runs successfully, an email is sent to the RECIPIENT(check the inbox), and the command line will display the sent Message ID.


Option 2 – SES and EC2 are in different AWS accounts

In some scenarios, your EC2 instance might operate in a different AWS account than SES. Let’s call the EC2 AWS account “A” and SES AWS account “B”. Because the AWS resources in account A don’t automatically have permission to access AWS resources account B, we need some way to allow the code on EC2 to assume a role in the SES Account using the AWS Security Token Service (STS). This involves a method that generates temporary credentials that include an access key, secret access key, and session token, which are only valid for a limited time.

option-2

EC2 & SES in different AWS Accounts

In the steps below, you’ll configure and attach an IAM role to the EC2 instance in account “A” such that it can run an example Python script. This Python script can use the permissions provided by the attached IAM role to send emails via SES in account “B”. This approach leverages cross-account access and simplifies sending email from the EC2 in account A via SES in account B. As with Option 1, no explicit credential management is required in the code running on EC2, nor do you need to include a shared credentials file on the Ec2 instance.

Prerequisites – different AWS accounts for EC2 and SES (use cross-account access)

  • An AWS account “A” with:
    • EC2 instance (Linux) in running state. (If you don’t have a EC2 instance, create one using Amazon Linux)
    • Administrative Access to Amazon IAM and EC2 consoles.
    • Make note of your “A” AWS account ID here: ________________
  • An AWS account “B” with:
    • Verified domain (or email identity for testing only) in Amazon SES
      • Make note of a verified sending email address here: ___________
    • Administrative Access to Amazon SES and IAM consoles.
      • Make note of your “B” AWS account ID here: ________________
    • In the steps below, you will create a “SES_Role_for_account_A” role.
      • Make note of the ARN of the “SES_Role_for_account_A” role here: ___________
    • Access to a recipient email address to receive test emails from the python script.
      • Make note of a SES verified recipient email address to send test emails here: ___________

Step 1 – Create IAM Role in the SES “B” account

  • Sign in to the SES “B” account via the AWS Management Console and open the IAM console.
  • In the navigation pane, choose “Roles,” and then choose “Create role“.
  • Choose the trusted entity type as ‘AWS account’ and select ‘Another AWS account’.
  • Add the AWS account ID where your EC2 instance resides (AWS account “A” in the prerequisites) and click ‘Next’.
  • Search for and select the “AmazonSESFullAccess” policy or create a custom policy with the necessary SES permissions, then click ‘Next’.
  • Provide a name for your role (e.g., ‘SES_Role_for_account_A').
  • Click “Create role“.
  • Copy the arn for the new SES_Role_for_account_A (you’ll need the arn in the next step).

Step 2 – Create a IAM policy in the EC2 “A” account that allows this role to assume the SES_Role_for_account_A role you just created in the SES “B” Account.

  • Sign in to the EC2 “A” account via the AWS Management Console and open the IAM console.
  • In the navigation pane, choose “Policies,” and then choose “Create Policy”.
  • Choose the service as ‘EC2’ and select policy editor as JSON.
  • Copy the policy below, and in the policy editor, replace the Resource with the arn of theSES_Role_for_account_A in the SES account “B” (you created this in step 1).

[copy, paste into policy editor & replace the arn with SES_Role_for_account_A]

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::<SES_Account_ID>:role/<Role_Name>"
}
]
}

  • Click ‘Next’ and provide a name for your role (e.g., EC2_Policy_for_account_B).
  • Click ‘Create the Policy

Step 3 – Create an IAM role in the EC2 “A” account, and attach the previously created IAM policy (EC2_Policy_for_account_B) to it.

  • In the EC2 “A” account IAM console navigation pane, choose “Roles,” and then choose “Create role.”
  • Choose the trusted entity type as “AWS service” and select “EC2” as the service, then click ‘Next’.

  • Filter by type “customer managed”, search for (EC2_Policy_for_account_B) and select that policy and ‘Next’ (note – if you are using AWS Session Manger to remotely connect to your EC2 instance, you may need to add the “AmazonSSMManagedInstanceCore” policy to the role).

  • Provide a name for your role (e.g., EC2_SES_in_account_B_role).
  • Click “Create role“.

Step 4 – Attach the IAM Role (EC2_SES_in_account_B_role) to the EC2 instance in AWS account “A”.

  • Open the EC2 Management Console in AWS account “A”
  •  In the navigation pane, choose “Instances,” and select the instance to which you want to attach the EC2_SES_in_account_B_role IAM role.
  • With the instance selected, choose “Actions,” then “Security,” and “Modify IAM role.”

  • Choose the IAM role you created (EC2_SES_in_account_B_role) from the drop-down menu.
  • Click “Update IAM role.”

Step 5 – Create a sample python script that sends emails via SES in AWS account “B” from the EC2 instance in AWS account “A” using the EC2 attached role.

  1. Now that your EC2 instance is configured with the necessary permissions, you can set up an example Python script to send emails via Amazon SES in AWS Account “B” using the IAM Role on EC2 in AWS Account “A”. We’ll use the AWS SDK for Python (Boto3), a powerful and versatile library to interact with the SES API endpoint. Before running the example script, ensure that Python, pip (the package installer for Python), and the Boto3 library are installed on your EC2 instance:
    • Run the ‘python3 –version‘ command to check if Python is installed on your EC2 instance. If Python is installed, the version will be displayed, otherwise you’ll receive a ‘command not found’ or similar error message.
      • If python is not installed, run the command ‘sudo yum install python3 -y
    • Run the ‘pip3 --version‘ command to check if pip is installed on your EC2 instance. If pip3 is installed, is installed, the version will be displayed, otherwise you’ll receive a ‘command not found’ or similar error message.
      • If pip3 is not installed, run the command ‘sudo yum install python3-pip
    • Install the Boto3 Library which allows Python scripts to interact with AWS services including SES. Run the command ‘pip3 install boto3‘ to install (or update) Boto3 using pip.
  1. Save the code below as a Python file named cross_sesemail.py on your EC2 instance.
    4b. Edit cross_sesemail.py and replace the placeholder values of the ROLE_ARN with ARN of the SES_Role_for_account_A you created in SES Account “B” (see prerequisites), SENDER, RECIPIENT, and AWS_REGION with your values (see prerequisites). Do not modify any “” marks.

[copy, edit & replace the ROLE_ARN]

import boto3
from botocore.exceptions import ClientError

# Replace with your role ARN in SES Account
ROLE_ARN = "arn:aws:iam::<Account_ID>:role/<Role_Name>"

# Create an STS client
sts_client = boto3.client('sts')

# Assume the role
assumed_role = sts_client.assume_role(
    RoleArn=ROLE_ARN,
    RoleSessionName="SESSession"
)

# Extract the temporary credentials
credentials = assumed_role['Credentials']

# Create an SES client using the assumed role credentials
ses_client = boto3.client(
    'ses',
    region_name='us-west-2',
    aws_access_key_id=credentials['AccessKeyId'],
    aws_secret_access_key=credentials['SecretAccessKey'],
    aws_session_token=credentials['SessionToken']
)

# Email parameters
SENDER = "[email protected]"
RECIPIENT = "[email protected]"
SUBJECT = "Amazon SES Test (SDK for Python) using cross-account IAM Role"
BODY_TEXT = ("Amazon SES Test (Python)\r\n"
             "This email was sent with Amazon SES using the "
             "AWS SDK for Python (Boto) using IAM Role."
            )
BODY_HTML = """<html>
<head></head>
<body>
  <h1>Amazon SES Test (SDK for Python) using IAM Role</h1>
  <p>This email was sent with
    <a href='https://aws.amazon.com/ses/'>Amazon SES</a> using the
    <a href='https://aws.amazon.com/sdk-for-python/'>
      AWS SDK for Python (Boto)</a> using IAM Role.</p>
</body>
</html>
            """
CHARSET = "UTF-8"

# Send the email
try:
    response = ses_client.send_email(
        Destination={
            'ToAddresses': [RECIPIENT],
        },
        Message={
            'Body': {
                'Html': {
                    'Charset': CHARSET,
                    'Data': BODY_HTML,
                },
                'Text': {
                    'Charset': CHARSET,
                    'Data': BODY_TEXT,
                },
            },
            'Subject': {
                'Charset': CHARSET,
                'Data': SUBJECT,
            },
        },
        Source=SENDER,
    )
except ClientError as e:
    print(e.response['Error']['Message'])
else:
    print("Email sent! Message ID:"),
    print(response['MessageId'])
  • Run the python script python3 cross_sesemail.py. When the email is sent successfully, the command line output will display the message ID of the sent email, and the recipient will receive an email.


Conclusion:

By implementing IAM roles for EC2 instances with SES permissions, you can securely send emails via the SES APIs from your web applications without the need to store or manage IAM credentials within the EC2 instance or application code. This approach not only enhances security by eliminating the risk of credential exposure, but also simplifies the management of credentials. With the step-by-step guide provided in this blog post, you can easily configure IAM roles for your EC2 instances and start sending emails via the Amazon SES API in a secure and efficient manner, regardless of whether your EC2 and SES resources reside in the same or different AWS accounts.

Next Steps:

  1. Sign up for the AWS Free Tier and try out Amazon SES with IAM roles for EC2 instances as demonstrated in this blog post.
  2. Consult the AWS documentation on IAM Roles for Amazon EC2 and Amazon SES for more detailed instructions and best practices.
  3. Join the AWS Community Forums to ask questions, share experiences, and learn from other AWS users who have implemented similar solutions for secure email sending from their web applications.

About the Authors

Manas Murali M

Manas Murali M

Manas Murali M is a Cloud Support Engineer II at AWS and subject matter expert in Amazon Simple Email Service (SES) and Amazon CloudFront. With over 5 years of experience in the IT industry, he is passionate about resolving technical issues for customers. In his free time, he enjoys spending time with friends, traveling, and exploring emerging technologies.

zip

Zip

Zip is an Amazon Pinpoint and Amazon Simple Email Service Sr. Specialist Solutions Architect at AWS. Outside of work he enjoys time with his family, cooking, mountain biking and plogging.

Email Journaling with SES Mail Manager

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/email-journaling-with-ses-mail-manager/

Introduction to Journaling

Email journaling is the practice of preserving comprehensive records of all email communications within an organization. This approach stems from the need to maintain rigid, compliance-driven retention policies focused on auditing an entire organization’s email activities. Because journaled email messages are often required to satisfy on-demand audit and investigation requests, they must be readily searchable, making accessibility a key requirement. Reflecting legal and regulatory requirements, email journaling has historically required expensive, dedicated off-site storage and complex retrieval systems.

Amazon WorkMail is a managed business email service with flexible journaling capabilities that are configurable at both the individual mailbox and organization-wide level. With WorkMail, you can use custom rules to selectively preserve or redirect certain messages using granular journaling controls. This flexibility allows administrators to implement both traditional email journaling and configurations that you can customize to meet specific use cases.

Email journaling is used to capture and retain every email sent to and from an organization, primarily for compliance purposes. In contrast, email archiving is typically used to offload and store emails from an organization’s primary email system, often driven by inbox size limits and data backup or eDiscovery needs. While journaling focuses on preserving a consolidated record of communications separate from live mailboxes, archiving is a more selective process. Journaling is usually driven by regulatory, audit, and compliance requirements. As discussed in this blog post, you can use the Mail Manager archiving feature not only for selective email backup and optimization, but also to fulfill your email journaling requirements. You can learn more about email archiving with Mail Manager in this blog post.

Amazon Simple Email Service (SES) Mail Manager provides comprehensive tools that simplify managing large volumes of email communications within an organization. Mail Manager has a built-in archiving function which can be used as an inexpensive journaling solution for email systems like Amazon WorkMail. Mail Manager’s rules engine allows for the creation of rules that readily satisfy a wide range of email journaling requirements. Additionally, Mail Manager’s archiving capability supports multiple, concurrent archiving destinations that can be independently searched and exported on demand.

In this blog post, we discuss how Amazon WorkMail and Amazon Simple Email Service (SES) Mail Manager make email journaling easier to set up and use, more cost-effective and versatile. We’ll walk the reader through setting up email journaling for an Amazon WorkMail organization that uses SES Mail Manager’s routing, processing, and archiving features.

SES Mail Manager as Journaling Destination for WorkMail

For our purposes, we’ll assume you’ve already set up WorkMail as your mailbox provider, but the process described below will work with the journaling features of most 3rd party email solutions. If you want to explore Amazon WorkMail, visit the getting started documentation here.

In the following sections, we’ll describe how to configure WorkMail journaling to send full email journals to SES Mail Manager’s archives. We’ll define different retention periods for each archive to demonstrate how this solution can be used to meet both short and long-term retention requirements. Finally, we’ll use the AWS SES Mail Manager console to search, export, and manage the email journals and archives.

In our examples, we’ll use Amazon Route 53 to create a new domain called ‘journaling.solutions’ which we’ll configure to send all ‘@journaling.solutions’ emails to an SES Mail Manager Ingest endpoint. To begin, open the AWS Console, navigate to your WorkMail Organization’s settings, and click on the Journaling tab:

Organization settings Journaling tab

Organization settings Journaling tab

Click Edit, enable journaling, and provide a journaling email address (we’re using ‘[email protected]’) to receive journaled content. Provide a report email address, such as the admin email list, to receive journaling reports:

Provide a Journaling email address

Provide a Journaling email address

Open the AWS SES console in a new browser window, and navigate to Mail Manager’s Rule sets. Create a new rule set called ‘journaling-rule-demo’. Click Edit and create a new rule called “journal-all”, with an Archive action. Click the create an archive button and create an archive called ‘journaling-archive-demo’:

Create a new Rule Set called ‘Journaling-rule-demo’

Create a new Rule Set called ‘Journaling-rule-demo’

When creating Mail Manager archives, you have options to set the retention period from 3 months to permanent storage. You can also choose to encrypt your archived messages with your own KMS key. The configuration in our example is for permanent storage and shows the optional text field for using your own KMS key:

you can encrypt the archived messages with your own KMS key

you can encrypt the archived messages with your own KMS key

Traditional journaling calls for recording every email message to the journal, so for our ‘journal-all’ rule, we will not define filtering behaviors in the rule set. This will instruct Mail manager to send all emails for [email protected] to the journaling-archive-demo archive. It is worth noting that Mail Manager’s rule set can be configured to filter and independently process multiple recipient addresses. Consult the documentation to learn about other ways to customize Mail Manager for your use cases.

Next, create a new traffic policy, called journaling-traffic-demo, and configure it to reject any message not explicitly sent to the journaling destination address ([email protected]):

Create a new Traffic policy, called ‘Journaling-traffic-demo’

Create a new Traffic policy, called ‘Journaling-traffic-demo’

Create an open ingress endpoint called ‘journaling-demo-IG’, and select the ‘journaling-traffic-demo’ traffic policy and ‘journaling-rule-demo’ rule set:

Create an Open Ingress endpoint called ‘Journaling-demo-IG’,

Create an Open Ingress endpoint called ‘Journaling-demo-IG’,

After you press the create Ingest endpoint button, Mail Manager will create an Ingress endpoint and assign it a DNS A Record to be used in your DNS configurations to route email to Mail Manager:

Mail Manager Ingress endpoint DNS A Record to be used in your DNS configurations

Mail Manager Ingress endpoint DNS A Record to be used in your DNS configurations

From the General details page of the Ingress endpoint, copy the Ingress endpoint’s DNS A Record to your clipboard. Open a new browser window to your DNS provider’s MX configuration page (in our example below, we’re using AWS Route53). Edit the MX record for ‘journaling.solutions’ by pasting the Ingress endpoint A record. This configuration will route email sent to any address ‘@journaling.solutions’ to the Mail Manager’s Ingress endpoint for processing by the Traffic policy and Rule set:

Using AWS Route53 to edit MX record for ‘journaling.solutions’ to Ingress endpoint A record

Using AWS Route53 to edit MX record for ‘journaling.solutions’ to Ingress endpoint A record

To test your new journaling configuration, send several emails to several email addresses in your WorkMail organization (or the alternative inbox provider you configured in the first step). WorkMail (or your alternative inbox provider) will send a full record of all emails to the journaling destination address ([email protected]).

Wait a few minutes after sending the emails above, then open the AWS Mail Manager console’s archiving controls and search for messages sent in the last 12 hours:

AWS Mail Manager console’s archiving controls

AWS Mail Manager console’s archiving controls

The example above shows a search for all messages received in the “last 12 hours”, with no other filters specified. The results show every message inserted into the archive in this timeframe. You’ll see one entry where the from address is different (from toby@tegwj@…). This is an example of mail that was sent directly to the journaling destination address ([email protected]). This works because our traffic policy and rule set configurations don’t include any filters.

A cost effective solution at scale

Using Mail Manager as a journaling solution gives you more direct control over your costs than typical journaling services. While most journaling services in the market today charge a fixed rate per journaled mailbox, Mail Manager pricing is comprised of a monthly fixed fee per ingestion endpoint and consumption pricing for basic message handling, and the amount of data archived.

For example, imagine your organization has 250 mailboxes, each handling 50 messages per day. On a monthly basis this amounts to 375,000 messages. If we assume each message is 40 kilobytes in size, your organization is generating roughly 15 gigabytes of email per month. As you can see from the table below, the total cost in month 1 is about $140, or $0.56/mailbox.

|Item |Unit Price |Volume |Subtotal/Mo |
|— |— |— |— |
|Ingress Endpoint |$50/mo |1 |$50 |
|Core message processing |$0.15/1000 msgs |375 |$56.25 |
|Archive insertion/indexing |$2/GB (one-time) |15 |$30 |
|Archive storage |$0.19/GB/mo |15 |$2.85 |
|Subtotal: | | |$139.10 |
| |Monthly price per mailbox |$0.56 |

If the proposed email rate in our assumptions stays constant, the Mail Manager archive will grow by 15 gigabytes each month. After 36 months, the total monthly storage cost increases to $102.60. This results in a total monthly spend in month 36 of $238.85, or $0.96/mailbox/month.

Conclusion

In this blog post, we’ve explored how Amazon WorkMail and Amazon SES Mail Manager can provide a cost-effective and accessible solution for email journaling. By leveraging the flexible journaling capabilities of WorkMail and the archiving features of SES Mail Manager, organizations can easily satisfy rigorous compliance requirements around email retention and accessibility.

The combination of WorkMail’s journaling controls and SES Mail Manager’s rule-based archiving allows you to tailor your journaling solution to your specific needs. Whether you require short-term retention for audits or long-term preservation for legal and regulatory purposes, SES Mail Manager’s flexible archiving options have you covered with predictable and transparent costs that scale with your organization’s email volume.

If you’re looking for a modern, scalable, and cost-effective solution for your email journaling needs, we encourage you to explore the capabilities of Amazon SES Mail Manager. Get started today by visiting the AWS documentation and begin streamlining your email compliance and retention processes.

About the Authors

Toby Weir-Jones

Toby Weir-Jones

Toby is a Principal Product Manager for Amazon SES and WorkMail. He joined AWS in January 2021 and has significant experience in both business and consumer information security products and services. His focus on email solutions at SES is all about tackling a product that everyone uses and finding ways to bring innovation and improved performance to one of the most ubiquitous IT tools.

Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

Andy Wong

Andy Wong

Andy Wong is a Sr. Product Manager with the Amazon WorkMail team. He has 10 years of diverse experience in supporting enterprise customers and scaling start-up companies across different industries. Andy’s favorite activities outside of technology are soccer, tennis and free-diving.

Bruno Giorgini

Bruno Giorgini

Bruno Giorgini is a Senior Solutions Architect specializing in Pinpoint and SES. With over two decades of experience in the IT industry, Bruno has been dedicated to assisting customers of all sizes in achieving their objectives. When he is not crafting innovative solutions for clients, Bruno enjoys spending quality time with his wife and son, exploring the scenic hiking trails around the SF Bay Area.

How AWS powered Prime Day 2024 for record-breaking sales

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/how-aws-powered-prime-day-2024-for-record-breaking-sales/

The last Amazon Prime Day 2024 (July 17-18) was Amazon’s biggest Prime Day shopping event ever, with record sales and more items sold during the two-day event than any previous Prime Day event. Prime members shopped for millions of deals and saved billions across more than 35 categories globally.

I live in South Korea, but luckily I was staying in Seattle to attend the AWS Heroes Summit during Prime Day 2024. I signed up for a Prime membership and used Rufus, my new AI-powered conversational shopping assistant, to search for items quickly and easily. Prime members in the U.S. like me chose to consolidate their deliveries on millions of orders during Prime Day, saving an estimated 10 million trips. This consolidation results in lower carbon emissions on average.

We know from Jeff’s annual blog post that AWS runs the Amazon website and mobile app that makes these short-term, large scale global events feasible. (check out his 2016, 2017, 2019, 2020, 2021, 2022, and 2023 posts for a look back). Today I want to share top numbers from AWS that made my amazing shopping experience possible.

Prime Day 2024 – all the numbers
Here are some of the most interesting and/or mind-blowing metrics:

Amazon EC2 – Since many of Amazon.com services such as Rufus and Search use AWS artificial intelligence (AI) chips under the hood, Amazon deployed a cluster of over 80,000 Inferentia and Trainium chips for Prime Day. During Prime Day 2024, Amazon used over 250K AWS Graviton chips to power more than 5,800 distinct Amazon.com services (double that of 2023).

Amazon EBS – In support of Prime Day, Amazon provisioned 264 PiB of Amazon EBS storage in 2024, a 62 percent increase compared to 2023. When compared to the day before Prime Day 2024, Amazon.com performance on Amazon EBS jumped by 5.6 trillion read/write I/O operations during the event, or an increase of 64 percent compared to Prime Day 2023. Also, when compared to the day before Prime Day 2024, Amazon.com transferred an incremental 444 petabytes of data during the event, or an increase of 81 percent compared to Prime Day 2023.

Amazon Aurora – On Prime Day, 6,311 database instances running the PostgreSQL-compatible and MySQL-compatible editions of Amazon Aurora processed more than 376 billion transactions, stored 2,978 terabytes of data, and transferred 913 terabytes of data.

Amazon DynamoDB – DynamoDB powers multiple high-traffic Amazon properties and systems including Alexa, the Amazon.com sites, and all Amazon fulfillment centers. Over the course of Prime Day, these sources made tens of trillions of calls to the DynamoDB API. DynamoDB maintained high availability while delivering single-digit millisecond responses and peaking at 146 million requests per second.

Amazon ElastiCache – ElastiCache served more than quadrillion requests on a single day with a peak of over 1 trillion requests per minute.

Amazon QuickSight – Over the course of Prime Day 2024, one Amazon QuickSight dashboard used by Prime Day teams saw 107K unique hits, 1300+ unique visitors, and delivered over 1.6M queries.

Amazon SageMaker – SageMaker processed more than 145B inference requests during Prime Day.

Amazon Simple Email Service (Amazon SES) – SES sent 30 percent more emails for Amazon.com during Prime Day 2024 vs 2023, delivering 99.23 percent of those emails to customers.

Amazon GuardDuty – During Prime Day 2024, Amazon GuardDuty monitored nearly 6 trillion log events per hour, a 31.9% increase from the previous year’s Prime Day.

AWS CloudTrail – CloudTrail processed over 976 billion events in support of Prime Day 2024.

Amazon CloudFront – CloudFront handled a peak load of over 500 million HTTP requests per minute, for a total of over 1.3 trillion HTTP requests during Prime Day 2024, a 30 percent increase in total requests compared to Prime Day 2023.

Prepare to Scale
As Jeff noted in every year, rigorous preparation is key to the success of Prime Day and our other large-scale events. For example, 733 AWS Fault Injection Service experiments were run to test resilience and ensure Amazon.com remains highly available on Prime Day.

If you are preparing for a similar business-critical events, product launches, and migrations, I strongly recommend that you take advantage of newly-branded AWS Countdown, a support program designed for your project lifecycle to assess operational readiness, identify and mitigate risks, and plan capacity, using proven playbooks developed by AWS experts. For example, with additional help from AWS Countdown, Legal Zoom successfully migrated 450 servers with minimal issues and continues to leverage AWS Countdown Premium to streamline and expedite the launch of SaaS applications.

We look forward to seeing what other records will be broken next year!

Channy & Jeff;

Understanding Google Postmaster Tools (spam complaints) for Amazon SES email senders

Post Syndicated from Bruno Giorgini original https://aws.amazon.com/blogs/messaging-and-targeting/understanding-google-postmaster-tools-spam-complaints-for-amazon-ses-email-senders/

Introduction

Amazon Simple Email Service (SES) includes a robust set of built-in tools, such as the Virtual Deliverability Manager (VDM), to help senders ensure optimal email deliverability. Additionally, deliverability data from email service providers like Postmaster Tools by Google can provide invaluable insights for all sending domain owners, including those using SES for bulk or transactional email. Postmaster Tools offers detailed metrics on factors like delivery errors, spam rates, domain reputation, and recipient feedback for Gmail-hosted inboxes. Combining this external data with SES email sending events is critical for maintaining a healthy sender reputation. By leveraging both SES-native tools and resources like Postmaster Tools, senders can identify and address deliverability issues, ensuring their SES-powered emails reach intended recipients across providers.

Many, but not all, mailbox providers will send recipient feedback in the form of “complaints” that can each be attributed directly to the message that the recipient found to be objectionable. These complaints are available in the SES email sending event type “Complaint”. Gmail does not send spam complaint events because their priority is to protect the privacy of their users from the tracking techniques employed by spammers and data brokers. Gmail requires bulk senders to adopt “easy unsubscribe” mechanisms to reduce the need for their users to report messages as spam, and they will show spam complaint metrics in Postmaster Tools. This blog will show you how to maximize value in the spam complaint metric provided by Postmaster Tools.

Amazon SES now supports custom values in the Feedback-ID header in messages sent through SES. This feature provides additional details to help customers identify deliverability trends. Together with Postmaster Tools, customers can group complaints by identifiers of their choice, such as sender business unit or campaign ID. This makes it easier to track deliverability performance associated with independent workloads and campaigns, and accelerates troubleshooting when diagnosing complaint rates.

This image describes the flow for spam complaints to the email sender

Figure 1: Email Feedback Loop

This blog will guide you through implementing and using Feedback Loops within Postmaster Tools to identify email campaigns receiving high complaint volumes from Gmail users. It covers the history and background of feedback loops, the specific requirements for implementing them with Postmaster Tools, and practical examples using AWS CLI and Boto3 to send SES emails with the necessary Feedback-ID header. By the end, you’ll understand how to effectively set up and use Postmaster Tools to monitor and improve your SES email deliverability.

History and Background of FBLs

Traditional Feedback Loops (herein “FBLs”) have been a cornerstone of email deliverability for many years. Initially developed by Internet Service Providers (ISPs), FBLs serve as a mechanism for recipients to report spam complaints to the sender. This feedback is crucial for email service providers and senders to identify problematic email campaigns, take corrective actions, and maintain a healthy sender reputation.

FBLs operate by allowing recipients to mark emails as spam, which then sends a report to the sender’s email service provider. This report typically includes details about the email that triggered the complaint, enabling the sender to investigate and address any issues. By analyzing these reports, senders can refine their email lists, improve content, and ensure that their emails comply with best practices and regulatory requirements. Senders who receive a higher volume of spam complaints are more likely to be blocked or have their emails routed to the spam folder. While high spam complaints are not the sole reason for deliverability issues, they are often the underlying cause.

Postmaster Tools by Gmail is not a traditional FBL. Postmaster Tools will show complaint feedback metrics, but the complaints are not attributable to any individual recipient.

Requirements for using Postmaster Tools FBL with SES

The FBL helps identify campaigns with high complaint rates from Gmail users, specifically useful for email service providers to detect potential abuse of their services.

Note: Data in Postmaster Tools only applies to messages sent to personal Gmail accounts. A personal Gmail account is an account that ends in @gmail.comor @googlemail.com.

  • Implementation of FBL:
    • Feedback-ID Header: SES embeds a header called Feedback-ID containing parameters (Identifiers) uniquely identifying the account and SenderID (AmazonSES)
    • Header Format: The Feedback-ID header consists of four parameters, separated by colons:
      a:b:c:SenderId
      Where:
      • SenderId is a mandatory parameter that uniquely identifies the sender.
      • In the case of Amazon SES (Simple Email Service), the SenderId is always “AmazonSES” and cannot be overridden.
Header Parameter  Description
a  First parameter in the Feedback-ID header. SES users can customize through ses:feedback-id-a EmailTag
b  Second parameter in the Feedback-ID header. SES users can customize through ses:feedback-id-b EmailTag.
c  Third parameter in the Feedback-ID header. SES uses this to identify the sender account
SenderID  Fourth parameter in the Feedback-ID header. Mandatory parameter that uniquely identifies the sender. For Amazon SES, this is always “AmazonSES” and cannot be overridden.
  • Sender Data Handling:
    • DKIM signing by a sender-owned domain is required to prevent spoofing.
    • The domain must be added and verified in Postmaster Tools.
    • Complaint data is aggregated by distinct values on each of the 4 fields of Feedback-ID.
  • Feedback-ID header Requirements:
    • When sending emails through Amazon SES, users are limited to a single verified header value per traffic stream.
      • This means that the Feedback-ID header cannot contain an individualized value for each destination email address.
      • Instead, the Feedback-ID header needs to contain an identifier that can be used to match a larger campaign or batch of emails, rather than a unique value per recipient.
      • This constraint helps maintain a consistent sender reputation, improves deliverability monitoring and troubleshooting within tools like Postmaster Tools. The Feedback-ID acts as a grouping mechanism, rather than a per-message identifier
    • Identifiers must be unique and non-repetitive across fields.
  • Feedback-ID Example:
    • CampaignIDX:CustomerID2:1.us-west-2.TDQeKqHkSNfQztk25wIeVIGTuNmGDud4r1l7dUlxOio=:AmazonSES
      • Each Identifier is used to report spam percentages independently if unusual rates occur.
      • Amazon SES lets customers set the part a and part b of the Feedback-ID header using the EmailTag ses:feedback-id-a and ses:feedback-id-b
      • Amazon SES will combine these tags into a single Feedback-ID header with the format: Feedback-ID=a:b:region.accountId:AmazonSES

The next steps will cover what’s needed to leverage FBLs with SES.

Step 1 – Add Your Domain(s) To Google’s Postmaster Tools

  • In order to verify with Postmaster Tools that you’re authorized to track the feedback from your domain, you first need to register your ownership of the domain with Postmaster Tools by visiting https://gmail.com/postmaster/.
Verify a new domain with Google Postmaster Tools

Figure 2: Step 1 to verify a domain in Google Postmaster Tools

  • After entering in your domain, you’d be prompted to add a TXT record into your DNS configuration.
Step 2 to verify a domain in Google Postmaster Tools

Figure 3: Step 2 to verify a domain in Google Postmaster Tools

  • Update your sending domain(s) DNS records accordingly.
    • The example below specifies how to create the TXT record in Route53. If you’re using another DNS service provider, please refer to their documentation.
Create a new record in Route53

Figure 4: Create a new record in Route53

    • Navigate to the Route53 Console and click on Hosted zones , specify the hosted zone that contains the domain you want to verify and then Create record.
This image describe the creation of a TXT record including the value provided by Google Postmaster to verify the domain

Figure 5: Add a TXT record with the provided value for verification

    • Following the screenshot, create a TXT record type and paste the value assigned by Google for verification in step 2 here.
  • Go to Postmaster Tools and click on Verify. After successful verification of your domain in Postmaster Tools, you should see the Status column changed from Not Verified to Verified. You can verify your compliance status with the requirements in the Dashboard (2) link.
In this picture we show an example of how the domain would appear once verified in Google Postmaster Tools

Figure 6: Domain verified

  • Follow the recommendations provided in the Postmaster Tools dashboard to fully comply with the requirements (example below):
Email sender requirements

Figure 7: Email sender requirements compliance status recommendations

  • Once you have completed all the verification and configuration steps, you should see compliant checkmarks next to all available requirements (see example below):
Email sender requirements

Figure 8: Email sender requirements compliant status

Step 2 – Add Feedback-ID headers to your SES emails

  • Use this command line to send an email with Feedback-ID using the AWS CLI:
aws sesv2 send-email --from-email-address [email protected] \
   --destination '{"ToAddresses":["[email protected]"]}' \
   --content '{"Simple":{"Subject":{"Data":"Test Subject","Charset":"UTF-8"},"Body":{"Text":{"Data":"Test Data","Charset":"UTF-8"}}}}' \
   --email-tags '[{"Name": "ses:feedback-id-a","Value":"feedback-id-part-a-value"}]'

The values of ses:feedback-id-a and ses:feedback-id-b are specified using the --email-tags option.

  • Alternatively, use Boto3 to send an email with Feedback-ID with the following Python script:
import boto3
from botocore.exceptions import ClientError

def send_email(region_name):
    # Create a new SES client
    ses = boto3.client('sesv2', region_name=region_name)

    # Replace sender and recipient values
    SENDER = "Sender Name <[email protected]>"
    RECIPIENT = "[email protected]"
    CONFIGURATION_SET = "SES_Config_Set"
    SUBJECT = "Amazon SES Test (SDK for Python)"
    BODY_TEXT = "Amazon SES Test (Python)\r\nThis email was sent with Amazon SES using the AWS SDK for Python (Boto)."
    BODY_HTML = """<html>
    <head></head>
    <body>
      <h1>Amazon SES Test (SDK for Python)</h1>
      <p>This email was sent with
        <a href='https://aws.amazon.com/ses/'>Amazon SES</a> using the
        <a href='https://aws.amazon.com/sdk-for-python/'>
          AWS SDK for Python (Boto)</a>.</p>
    </body>
    </html>"""
    CHARSET = "UTF-8"

    try:
        # Send email
        response = ses.send_email(
            FromEmailAddress=SENDER,
            Destination={'ToAddresses': [RECIPIENT]},
            ConfigurationSetName=CONFIGURATION_SET,
            Content={
                "Simple": {
                    "Subject": {
                        "Charset": CHARSET,
                        "Data": SUBJECT
                    },
                    "Body": {
                        "Text": {
                            "Charset": CHARSET,
                            "Data": BODY_TEXT
                        },
                        "Html": {
                            "Charset": CHARSET,
                            "Data": BODY_HTML
                        }
                    },
                    "Headers": [
                        {
                            "Name": "List-Unsubscribe",
                            "Value": "<https://unsubscribe.example.email/[email protected]&topic=topic1>"
                        },
                        {
                            "Name": "List-Unsubscribe-Post",
                            "Value": "One-Click"
                        }
                    ]
                }
            },
            EmailTags=[
                {
                    'Name': 'ses:feedback-id-a',
                    'Value': 'campaign1'
                },
                {
                    'Name': 'ses:feedback-id-b',
                    'Value': 'line-of-business'
                }
            ] #the ses:feedback-id-a and ses:feedback-id-b are specified as a list using EmailTags
        )
        print("Email sent! Response:", response)
        print("Message ID:", response['MessageId'])

    except ClientError as e:
        print(e.response['Error']['Message'])

# Call the function to send the email
send_email(region_name='us-west-2')  # Specify the region here

Step 3 – Viewing FBL results in Postmaster Tools

In order to see any results in the Postmaster Tool dashboard (see examples below), you must send a substantial daily volume of email through the domain(s) you’ve registered. If you see the message “No Data to Display”, your reputation may already be too low, more likely the volume of email traffic sent since you configured the Postmaster tool is insufficient (return to the dashboard in later, after you’ve sent 1,000s of emails).

Figure 9: Feedback loop example image

Figure 9: Feedback loop example image

The image shows a section of the Postmaster Tools dashboard, specifically the Feedback Loop section. This dashboard provides insights into the spam complaint rates and the number of feedback loop identifiers flagged across a given time period, in this case, the last 120 days.

Conclusion

High-volume email senders should look to the combination of Amazon SES’ powerful framework for monitoring in concert with Postmaster Tools to improve and ensure email deliverability. Implementing the Feedback-ID header in your SES emails can significantly enhance your ability to track and troubleshoot deliverability issues. Use Postmaster Tools and the Feedback Loop via Feedback-ID headers in SES emails to gain detailed insights into complaint rates and other key metrics, enabling you to maintain a healthy sender reputation and ensure their emails reach the intended recipients.

Call to Action:

  1. Set Up Postmaster Tools for your sending domain(s)
  2. Verify Your Domain: Register and verify your domain with Postmaster Tools to access valuable insights and track your compliance status.
  3. Set Up Feedback-ID: Start embedding the Feedback-ID header in your emails sent via Amazon SES to take advantage of detailed complaint data and improve your email campaigns.
  4. Monitor and Adjust: Regularly check the Postmaster Tools dashboard to monitor your spam rates and feedback loop identifiers. Use this data to refine your email content and sending practices.
  5. Leverage AWS CLI and Boto3: Utilize the provided AWS CLI commands and Boto3 scripts to automate the process of sending emails with Feedback-ID headers, ensuring consistent and accurate tracking.

By following these steps, you can enhance your email deliverability, reduce spam complaints, and maintain a strong sender reputation. For more information on using Amazon SES and Google’s Postmaster Tools, refer to the Amazon SES Documentation and the Postmaster Tools Guide.

How to use Mail Manager to Archive Inbound Emails

Post Syndicated from Sesan Komaiya original https://aws.amazon.com/blogs/messaging-and-targeting/archiving-and-sending-to-final-smtp-server/

In today’s digital landscape, where email communication plays a vital role in business operations, Keeping your email archive secure, compliant, and retrievable is crucial for any business. However, managing the large volume of email data can lead to operational difficulties, including regulatory compliance, maintaining an audit trail, and preventing data loss. That’s where Amazon Simple Email Service (SES) Mail Manager’s email archiving feature comes in.

In this blog post, we will explore how Amazon SES Mail Manager’s email archiving and search features can improve email security and compliance. If you’re a newcomer to Mail Manager, look at this blog post on Amazon SES Mail Manager. It provides valuable information on important features, such as Ingress Endpoint, Traffic Policy, Rule Sets, and SMTP Relay.

Problem Statement:

Imagine a scenario where a critical email from a key client is buried deep within your organization’s email archives, and you need to retrieve it for an important audit. The challenge of ensuring your business remains compliant with stringent data retention policies across every email communication for thousands of employees for a certain period or permanently.

Solution explained:

Amazon SES Mail Manager Email archiving is a powerful tool that addresses many of the challenges organizations face dealing with the difficulty and expense of archiving email at scale. Compliance and regulatory frameworks like GDPR, HIPAA, and SOX often require email archiving, which is a common objective identified by customers needing to comply with those regulatory frameworks. For regulated businesses, failure to comply with email archiving regulations can result in severe financial penalties and reputational damage.

Amazon SES Mail Manager securely archives and safeguards your emails, providing easy search and export functionality. It provides full-time, enterprise-level archiving without increasing the storage requirements of your mailbox server. The feature provides a reliable and efficient solution to address most compliance requirements. By automatically archiving the types of emails you specify, the service ensures that your organization maintains a comprehensive audit trail of its communications, enabling quick retrieval and review as needed.

The email archiving feature of Mail Manager provides organizations the ability to archive email while in transit rather than archiving at the user’s mailbox. Many organizations prefer archiving in transit for email archiving to meet compliance requirements and maintain comprehensive records. If you would like to learn more about in transit archiving, visit this blog –  Email Archiving with Mail Manager: Why To Archive In Transit vs At The Mailbox.

How email flows with an Amazon SES Mail Manager Email Archiving

For instructional purposes in this blog post, we’ll focus on how you can introduce Mail Manager archiving into your existing email infrastructure. We’ll cover how to seamlessly integrate Mail Manager with reference architectures. Later in the blog, we are going to explore Mail Manager’s archiving capabilities, including search, export and retention policies.

Current setup

Our example organization has an existing mail server (it might be a on-premises Microsoft Exchange Server, Microsoft 365, Google Workspace, etc). Their DNS is configured to route all email directly to this mail server. There is currently no archiving capacity within the existing email infrastructure, when needed, archiving is handled by individual mailbox users and PST files. While this method is suitable for personal email archiving, it fails to meet the organizations’ security requirements and compliance standards.

Figure 1: Example organization’s existing inbound email workflow.

Email Archiving in transit

We are going to introduce Mail Manager into the current mail flow (see figure 1) to archive all incoming messages from our example enterprise’s email infrastructure.

Figure 2: Example organization’s proposed inbound email workflow, with Mail Manager archiving in-transit prior to delivery.

In the new architecture (see figure 2), we’ve introduced Mail Manager into the organization’s inbound email workflow. This new workflow leverages Mail Manager’s ability to archive either all inbound emails, or only those that match specified criteria. By using a Mail Manager Rule set, our example organization can selectively store and preserve emails that meet their configured criteria.

Mail Manager Email Archiving and Search and Export Capabilities

Mail Manager’s archive search capabilities are designed to be user-friendly and efficient. You can perform searches based on various criteria, such as sender, recipient, subject line, date range, or even specific keywords in the Subject line. The search results provide options to either export the search to Amazon Simple Storage Service (S3), or you can choose to download a single email.

Let’s explore using Mail Manager’s archive search to find a specific email by the sender’s address:

Figure 3: Mail Manager’s archive search interface

Once found, we can click on the results to review the email in the console:

Figure 4: Mail Manager’s archive search results

Once we’ve found the target email, it can be downloaded by clicking on “View details“. The image below shows an example message details page with information about the email, including message headers such as In-Reply-To, X-Original-Mailer and X-Mailer.

Figure 5: Mail Manager’s archived email detailed page

Mail Manager’s archive search history tab allows us to find archive searches created in the last 30 days, and view the search results, as shown in the image below:

Figure 6: Mail Manager’s archive search history

Mail Manager’s archive export history tab lists all of the archived email searches you exported to an Amazon S3 Bucket within the last 30 days.

Figure 7: Export History

Step by Step Setup:

Now that we have explained how Mail Manger can be inserted into our example organization’s email workflow to provide email archiving, let’s explore how you can implement Mail Manager’s archiving capabilities in your inbound email workflows. The following diagram (Figure 3) illustrates the overall structure and components involved in this architecture:

Figure 8: End-to-End Mail workflow

Follow the steps below to configure Mail Manager in your AWS account to implement this architecture:

  1. Log into the SES Console and select Mal Manager from the left navigation menu.
    1. Note, as of this writing, Mail Manager is generally available in the following AWS Regions: US East (N. Virginia), US West (Oregon), Europe (Ireland, Frankfurt), Asia Pacific (Tokyo, Sydney).
  2. Under Mail Manger navigation, create an archive (or multiple archives for different use cases)
    1. Enter a unique name in the Archive name field.
    2. (Optional) Select a retention period in the Retention period field to override the default retention period of 180 days.
    3. (Optional) You can encrypt your archive either by entering your own AWS KMS key into the KMS key ARN field, or by selecting Create new key.
    4. Choose Create archive.
  3. Under Mail Manger navigation, create a traffic policy to determine the email you want to block or allow.
    1. Create traffic policy.
    2. On the Create a traffic policy page, enter a unique name for your traffic policy.
    3. (Optional) If you want to discard any messages above a certain size, enter a value in bytes in the Maximum message size field.
    4. In Default action, choose whether the traffic policy is to either Allow or Deny (block) messages that fall outside of (are not addressed by) the conditions of your policy statements.
    5. Select Add new policy statement to create a statement for your traffic policy.
    6. Choose either Allow or Deny (block) for the action to be taken when the statement’s conditions are met.
    7. Build a condition by selecting an email protocol and a conditional operator for the value you enter. Select Add new condition if you want to add more conditions to this policy statement. To learn more about a condition property and its operators and valid values, see the Policy statement conditions reference.
    8. If you’re subscribed to an Email Add On, you’ll be able to select it here as an email protocol.
    9. If you want add more policy statements and conditions, repeat steps above.
    10. Select Create traffic policy.
  4. Under Mail Manger navigation, create a rule set to perform actions on the email you allow in.
    1. Create rule set and enter a unique name for your rule set.
    2. Create new rule on the edit page.
    3. In the Rule details sidebar, enter a unique name for your rule.
    4. Select Add new condition to create a condition that the message must match; or check the EXCEPT in the case of: box followed by Add new exception to create a condition that the message must not match.
    5. Build the condition or exception by selecting an email property and a conditional operator for the value you enter. Select Add new condition or Add new exception if you want to add more conditions or exceptions to this rule. To learn more about a condition property and its operators and valid values, see the Rule conditions reference.
    6. Select Add new action to define the action to be taken when the rule’s conditions are matched and/or exceptions are not matched. To add more actions to be taken, select Add new action. To learn more about actions and their parameters, see the Rule actions reference.
    7. Create an Archive rule. Save rule set
  5. Under Mail Manger navigation, Create your ingress endpoint and assign to it the traffic policy and rule set.
    1. choose Ingress endpoints under Mail Manager.
    2. On the Ingress endpoints page, select Create ingress endpoint.
    3. On the Create new ingress endpoint page, enter a unique name for your ingress endpoint.
    4. Choose whether it will be a Open or Authenticated endpoint.
    5. Select a traffic policy to determine the email you want to block or allow.
    6. Select a rule set containing the rule actions you want to perform on the email you allow in.
    7. Select Create ingress endpoint.
  6. Configure your environment to use the ingress endpoint.
    1. At the time you create an ingress endpoint, an “A” record for the endpoint will be generated and its value displayed on the ingress endpoint’s summary screen in the SES console. The way you use the value of this record depends on the type of endpoint you created and your use case.
    2. DNS providers have different procedures and interfaces for configuring email records. The key pieces of information you need to put into your DNS settings are listed in our documentation – https://docs.aws.amazon.com/ses/latest/dg/eb-ingress.html#eb-ingress-a-record
  7. Under Mail Manger navigation, create an SMTP Relay to send mail on to your existing mail server.
    1. choose SMTP relays under Mail Manager.
    2. On the SMTP relays page, select Create SMTP relay.
    3. On the Create SMTP relay page, enter a unique name for your SMTP relay.
    4. Depending on what type of SMTP Realy you want to configure, follow the respective instructions:
      1. inbound (non-authenticated)
      2. outbound (authenticated) SMTP relay
  8. Update your DNS MX records to point to your new Mail Manager’s ingress point, instead of the existing mail server.

Note: Make sure that you have tested the steps above in your development environment and that you understand the steps before deploying into your production environment.

Conclusion

Amazon SES Mail Manager’s email archiving capabilities are designed for organizations that are seeking to enhance the security, compliance, and audit-ability of their email communications. By seamlessly integrating this feature into their existing email infrastructure, organizations can now archive all inbound messages in transit, ensuring a comprehensive, tamper-proof record of their email activities. The powerful search and export functionality of Mail Manager makes it easy to quickly locate and access specific emails when needed, whether for compliance audits, legal requests, or internal investigations.

This level of email visibility and control is particularly crucial for organizations operating in highly regulated industries like government, healthcare and finance, where the stakes for non-compliance can be severe. Beyond the compliance benefits, Mail Manager’s email archiving also helps to alleviate the operational headaches and expenses associated with traditional in-house archiving systems. By offloading this responsibility to AWS, organizations can focus their resources on their core business priorities, while still maintaining the security and accessibility of their critical email data.

If you’re looking to strengthen your email security posture, simplify your compliance efforts, and improve the overall management of your email archives, we encourage you to explore how Amazon SES Mail Manager’s email archiving capabilities can be seamlessly integrated into your existing email infrastructure. Take the first step towards a more secure, compliant, and efficient email management solution by contacting us today.

About the Authors

Sesan Komaiya

Sesan Komaiya

Sesan is a Solutions Architect at Amazon Web Services. He works with a variety of customers, helping them with cloud adoption, cost optimization and emerging technologies. Sesan has over 15 year’s experience in Enterprise IT and has been at AWS for 5 years. In his free time, Sesan enjoys watching various sporting activities like Soccer, Tennis and Moto sport. He has 2 kids that also keeps him busy at home.

Alexey Kurbatsky

Alexey Kurbatsky

Alexey is a Senior Software Development Engineer at AWS, specializing in building distributed and scalable services. Outside of work, he enjoys exploring nature thru hiking as well as playing guitar.

Jesse Thompson
Jesse Thompson is an Email Deliverability Manager with the Amazon Simple Email Service team. His background is in enterprise development and operations, with a focus on email abuse mitigation and encouragement of authenticity practices with open standard protocols. Jesse’s favorite activity outside of technology is recreational curling.
Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

How to use SES Mail Manager SMTP Relay action to deliver inbound email to Google Workspace and Microsoft 365

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-use-ses-mail-manager-smtp-relay-action-to-deliver-inbound-email-to-google-workspace-and-microsoft-365/

Introduction

Customers often ask us if the Amazon Simple Email Service (SES) inbound capabilities they use with applications hosted on AWS infrastructure can also be used to process and automate employee email hosted on public services like Google Workspace and Microsoft 365. The answer has typically been “yes, but with some limitations”, as until now, SES inbound has been somewhat constrained by the fact that it didn’t support relaying messages for an existing domain. This limitation makes it very difficult to fully manage email flows across hybrid email environments.

Such conversations led the SES team to create Amazon Simple Email Service (SES) Mail Manager which offers a set of capabilities that simplify managing large volumes of email communications within an organization. Mail Manager’s rules set conditions and actions can optimize routing for improved delivery and communication flow, both for incoming and outgoing emails. Mail Manager’s email security features can be augmented by optional add-ons from industry-leading, vetted third-party providers. Flexible archiving features help organizations meet stringent compliance and record-keeping requirements.

In this blog, we position Mail Manager as a central ingress gateway for a fictitious company, Nutrition.co, that is based on real-world AWS customers. We discuss the customer challenges and explain how to configure Mail Manager’s SMTP Relay action to intercept, archive then deliver emails destined for employees’ Google Workspace hosted Gmail and Microsoft 365 hosted Exchange Online mailboxes. Similar mail flows can be used to process, automate and archive emails destined for their AWS hosted apps.

You can learn more about all of Mail Manager’s capabilities here.

Customer background and use case

Our fictitious company, Nutrition.co, is an online retail business with multiple employee departments, including administration, marketing, sales and fulfillment. The company has acquired several smaller rivals that use both Google Workspace and Microsoft 365 to host their employee inboxes, and plan to consolidate all users onto the same domain ( such as [email protected] and [email protected]). They also host several applications on Amazon Web Services (AWS) that use Amazon SES’ inbound capability to receive emails using a subdomain *customer-support*.nutrition.co, such as orders@*customer-support*.nutrition.co and returns@*customer-support*.nutrition.co.

Nutrition.co is looking for a solution to unify all their email domain routing, security and archiving processes onto one centralized management system to simplify their email infrastructure. They want an approach that provides more flexibility to control which addresses and domains are used for apps and automation as well as employee mail. They also want to enhance email compliance and governance with a flexible solution for screening, processing and archiving inbound emails to both employees and applications, before delivering those emails to recipient inboxes on Google Workspace and Microsoft 365 and applications hosted on AWS.

The SES Mail Manger based central ingress and egress gateway architecture we propose will allow Nutrition.co to manage their peer-to-peer and application-driven emails in one place, Amazon SES. It will simplify email security and management, and make it easy to unlock new cloud-enabled email use cases. The architecture can be modified to acommodate a wide variety of email infrastructure, including fully cloud hosted, on-premises, and hybrid mailbox hosting environments.

What is an Inbound SMTP Gateway?

An Inbound SMTP Gateway is an SMTP server that accepts inbound email via an Open Ingress Point, and then delivers those messages to another email environment’s inbound SMTP server. In the diagram below, Mail Manger is configured as an inbound SMTP Gateway:

Figure 1: Diagram of the inbound gateway mail flow to a mailbox hosting environment

Figure 1: Diagram of the inbound gateway mail flow to a mailbox hosting environment

“Inbound email” refers to email traffic flows where the originator of the message can be either a trusted (for example: the UK division of Nutrition.co) or an untrusted (for example: a Nutrition.co customer or vendor) entity. To send an email, the originating email system looks up the recipient domain’s MX record in the global DNS system to determine the address for the recipient’s inbound mail server. Once a connection is established on port 25, the originating server delivers the email message using the SMTP protocol typically using STARTTLS for transport level encryption. Inbound messages are typically authenticated using the SPF, DKIM, and DMARC industry standard protocols, which help ensure the messages are coming from the legitimate sender’s domain.

An Inbound SMTP gateway can act on messages, for example to process and/or archive, before passing them along to the end recipient’s email server. To learn more about archiving emails in transit, visit this blog.

Configuring Mail Manager as an Inbound SMTP Gateway

Before we can configure Mail Manager as an Inbound Gateway for Nutrition.co’s Google Workspace and Microsoft 365 hosted mailboxes, we need to “allow-list” Mail Manager in Nutrition.co’s Google Workspace and Microsoft 365 settings. Allow-listing in this context refers to configuring the hosted mailbox environments such that Mail Manager is not identified as the source of messages, but rather as an SMTP relay.

This configuration is necessary because the messages being relayed through Mail Manager originate from both trusted and untrusted senders. This mail flow will contain both wanted and, potentially, unwanted messages. Mail Manager is the intermediary, not the source of potentially unwanted email passing through Mail Manager’s Open Ingress Point before being relayed to the destination mailbox environment.

If Mail Manager is not allow-listed, inbound email that is relayed thru Mail Manager’s Open Ingress Point will fail SPF checks because the IP addresses of the intermediary server are not authorized by the domain’s SPF policy. Since DMARC relies on SPF, messages from intermediary mail servers will fail the domain’s DMARC policy if they are not signed with a domain-aligned DKIM signature.

Mailbox hosting environments and their anti-spam algorithms rely on SPF, DKIM and DMARC for authenticating different inbound mail flow configurations before making an assessment about the message’s disposition. Properly authenticated messages, if not otherwise identified as unwanted by recipients and their security administrator, are delivered to Inboxes. Messages that are not authenticated are more likely to be treated as spam. Messages from intermediary servers can sometimes be mistaken as spoofed or unwanted messages.

By allow-listing the egress IP addresses of the Mail Manager servers, Nutrition.co’s Google Workspace and Microsoft 365 hosting environments will be able to assess the correct SPF result when receiving inbound email from Mail Manager.

Note: Do not include Mail Manager’s IP addresses in the domain’s SPF policy, These IP addresses are shared by other Mail Manager customers so including them in the domain’s SPF policy can introduce a security risk.

Note: It is also possible to use DKIM and ARC for allow-listing mail streams, but Gmail and Exchange Online both support IP allow-listing.

Note: Nutrition.co’s Google Workspace and Microsoft 365 hosting environments may still make a spam assessment about the messages under the context that Mail Manager is not the original sender, but this is not common.

Figure 2: Diagram of the SES Mail Manager architecture to accept inbound email via an open Ingress endpoint and configured with a Rule set condition to relay messages with the SMTP Relay action.

Figure 2: Diagram of the SES Mail Manager architecture to accept inbound email via an open Ingress endpoint and configured with a Rule set condition to relay messages with the SMTP Relay action.

In the diagram above, the interaction points are as follows:

1. Email senders look in DNS to discover the MX record for example.com.
2. The value of the domain’s MX record is the A record of the Mail Manager Ingress endpoint. The Ingress endpoint is configured as an ‘open’ Ingress endpoint so that it can receive inbound email without requiring SMTP Auth
3. The Ingress endpoint traffic policy is configured to allow and deny traffic
4. The Rule Set conditions determine which messages are to be relayed
5. The SMTP Relay action relays messages for recipients that are SES verified identities

Configuring Mail Manager as an Inbound SMTP Gateway

Prerequisites

  • Access to the administrative console for Nutrition.co’s Google Workspace and Microsoft 365 hosted mailboxes
  • Access to the DNS zone hosting the MX records for the Nutrition.co’s domains

Step 1: Allow-list the regional Mail Manager IP addresses in Nutrition.co’s Google Workspace and Microsoft 365, and create the Mail Manager relay action(s) in AWS SES console.

  • If you do not configure the allow-list Nutrition.co’s Google Workspace and Microsoft 365 hosted, it may cause those mailbox providers to reject as spam or send to junk the emails replayed from your Mail Manager environment.

Step 1-a: Follow the instructions to allow-list Mail Manager to relay email to Nutrition.co’s Google Workspace and Microsoft 365 environments.

Step1-b: Create an SMTP relay for your mailbox hosting environment

* See Creating an SMTP relay in the SES console

Figure 3: Screenshot of an SMTP Relay rule action configured for Microsoft 365 Exchange Online inbound receiving

Figure 3: Screenshot of an SMTP Relay rule action configured for Microsoft 365 Exchange Online inbound receiving

Figure 4: Screenshot of an SMTP Relay rule action configured for Google Workspaces Gmail inbound receiving

Figure 4: Screenshot of an SMTP Relay rule action configured for Google Workspaces Gmail inbound receiving

Because Nutrition.co hosts email in both Google Workspace and Microsoft 365, we must create SMTP Relay actions for both.

Step 2: In SES console, verify Nutrition.co’s email domain, which is nutrition.co

SES needs to prove that Nutrition.co owns the domain of each of the recipient addresses before it will begin relaying inbound email. If you cannot verify ownership of the recipient email destinations, SES will not relay messages.

Follow the instructions to verify Nutrition.co’s SES domain identity for the recipient email addresses within Nutrition.co’s Google Workspace and Microsoft 365 environments. (*note that subdomains such as customer-support.nutrition.co inherit verification from the parent domain*).

Figure 5: Screenshot of a successfully verified domain in the SES console.

Figure 5: Screenshot of a successfully verified domain in the SES console.

Step 3: Configure Mail Manager with an Open Ingress Point and Rule Set Action to relay inbound email to the mailbox hosting environment.

Step 3-a: See Create a Traffic Policy to accept inbound email from the internet.

  • Default action: Allow
    (Optional) Add Policy statements, depending on your requirements. Choose the action to be taken when the filter conditions are met: Deny

    • While Nutrition.co does not want to apply additional security via the SMTP Relay gateway, Mail Manager supports both native capabilities and optional add-on subscriptions to 3rd party tools from vetted industry leaders such as Spamhaus and Abusix.
Figure 6: Screenshot of a traffic policy for accepting all email from the internet

Figure 6: Screenshot of a traffic policy for accepting all email from the internet

Step 3-b: Follow the instructions for creating rule sets and rules in the SES console.

  • Select the SMTP Relay that you created in Step 1-b and enable the **Preserve Mail From** option.
    • The ‘Preserve Mail From’ setting is necessary so that the mailbox provider can be configured to make the correct assessment of the message’s SPF policy evaluation, assuming that the allow-list configuration Step 1 is complete.
  • Add any conditions and exceptions for each rule, depending on your needs.
    • You may want to create a condition for the SMTP Relay rule so that only messages destined for recipients within your domain are relayed to the appropriate SMTP Relay action, and choose a different action for the recipients who are not hosted in your environment, such as the Archive action.
    • If you have both Google Workspace and Microsoft 365 configured as SMTP Relay destinations, you may combine the SMTP Relay actions in a single rule if the conditions are the same, or create them as separate rules if the conditions need to be different
Figure 7: A Mail Manager rule configured with an SMTP Relay action for Google Workspaces and another SMTP Relay actions for Microsoft 365

Figure 7: A Mail Manager rule configured with an SMTP Relay action for Google Workspaces and another SMTP Relay actions for Microsoft 365

Step 3-c: Follow the documentation for Creating an Ingress Point.

The Mail Manager Ingress point needs to be ‘Open“ for this use case because internet mail senders need to connect to port 25 and send without SMTP authentication for inbound mail flows.

  • Type: Open
    Traffic policy: Choose the traffic policy that you created step 3-a
    Rule set: Choose the rule set that you created in step 3-b

After saving the ingress endpoint settings, you should see something similar in the console.

Figure 8: Screenshot of an ‘open’ Mail Manager Ingress endpoint configured with a rule set and traffic policy

Figure 8: Screenshot of an ‘open’ Mail Manager Ingress endpoint configured with a rule set and traffic policy

Step 4. Verify your configuration and change your domain’s MX record

Once you have finished configuring Mail Manager with an Inbound Gateway configuration you will have:

  • An Open ingress point that does not require authentication and has an open traffic policy to allow messages from the internet.
  • A Rule set with SMTP Relay actions that will relay inbound messages to Google Workspace and/or Microsoft 365.

Step 4-a: Test your configuration

  • Ingress point: You can test that the Ingress endpoint receives email by using an SMTP capable client application, such as “openssl s_client” from a host that allows for outbound port 25 connections to the A Record of your Open Ingress Point (many ISPs and cloud infrastructure providers block port 25 by default to stop the proliferation of spam on the internet). If you get a “250 OK” response from the SMTP transaction, the Ingress point is configured correctly.
  • Rule set: You can test your Rule set by sending a message to your Ingress endpoint that has a recipient destination that is both a verified domain, and a domain that is hosted by your mailbox environment. You may want to add the Archive and/or Save to S3 rule actions to occur prior to SMTP Relay. This enables you to view message headers and diagnose issues that may occur during the SMTP relay to the mailbox hosting environments.
  • Final delivery: You can test the entire mail flow by looking at the received messages in your mailbox hosting environment.
    • How to look at received messages in a mailbox hosting environment
      • Google Workspace – From within the Gmail interface, find the message and open the message menu options.
      • Figure 9: Screenshot of Gmail’s interface for selecting message options
      • Choose “Show original”.
      • Screenshot of Gmail’s “Original message” summary showing SPF and DKIM passing and aligned with gmail.com, which was the source of the original message
      • Screenshot of the Gmail ‘Show original“ message headers. The Mail From address (also appears as the Return-path header, and envelope-from value in other headers) is preserved within the @gmail.com domain, and Gmail’s assessment of SPF correctly attributed the message as originating from 209.85.216.51 even though the message was relayed through 206.55.129.47. Since the 209.x.x.x address is in the SPF policy for gmail.com, the message passes SPF due to the allow-list configuration
      • (The Screenshot above shows the Gmail ‘Show original“ message headers. The Mail From address (also appears as the Return-path header, and envelope-from value in other headers) is preserved within the @gmail.com domain, and Gmail’s assessment of SPF correctly attributed the message as originating from 209.85.216.51 even though the message was relayed through 206.55.129.47. Since the 209.x.x.x address is in the SPF policy for gmail.com, the message passes SPF due to the allow-list configuration)
      • Microsoft 365 – From within the Outlook on the Web interface, find the message and open the message menu options.
      • Screenshot of Outlook on the Web’s interface for selecting message options
      • Choose “View message details”. You will see the message headers similar to the Gmail example above.

Step 4-b: Change the MX record for your domain.

Note: We recommend using a new subdomain so that you can test this mail flow configuration for a period of time prior to changing the MX record for the primary domain that is actively being used by end users and applications.

Once you have finished testing, you can change the MX record for the domain. The value of the MX record should be the **A Record** of the Open Ingress point along with the priority value.

Figure 13: A screenshot of an MX record configured in Amazon Route 53

Figure 13: A screenshot of an MX record configured in Amazon Route 53

Conclusion

In this blog post, we’ve explored how to leverage SES Mail Manager’s SMTP Relay action to simplify the handling of inbound email for organizations that use a mix of email hosting environments, specifically Google Workspace and Microsoft 365. By configuring Mail Manager as an inbound SMTP gateway, our fictitious customer, Nutrition.co was able to centralize the management of their email flows, enhance security through features like traffic policies and rule sets, and ensure compliance through flexible archiving.

The key steps involved setting up allow-listing in the Google Workspace and Microsoft 365 environments, creating SMTP relay configurations in Mail Manager, and updating Nutrition.co domain’s MX record to point to the Mail Manager ingress endpoint. This allowed Nutrition.co to seamlessly route inbound emails destined for both their cloud-hosted employee mailboxes and on-premises applications, processing and archiving the messages before final delivery.

The flexibility of Mail Manager’s SMTP Relay action makes it a powerful tool for organizations looking to unify their email infrastructure, especially in hybrid environments. By acting as a centralized ingress and egress gateway, Mail Manager can help streamline email management, improve security, and unlock new cloud-enabled email use cases. As email continues to be a critical communication channel, solutions like Mail Manager will become increasingly important for businesses looking to maximize the value of their email ecosystem.

Please visit AWS Re:Post to ask and find answers to questions about SES Mail Manager. Talk with your AWS account team if you are interested in exploring Mail Manager in more depth.

Additional blogs related to Mail Manager:

About the Authors

Jesse Thompson
Jesse Thompson is an Email Deliverability Manager with the Amazon Simple Email Service team. His background is in enterprise development and operations, with a focus on email abuse mitigation and encouragement of authenticity practices with open standard protocols. Jesse’s favorite activity outside of technology is recreational curling.
Alexey Kurbatsky

Alexey Kurbatsky

Alexey is a Senior Software Development Engineer at AWS, specializing in building distributed and scalable services. Outside of work, he enjoys exploring nature thru hiking as well as playing guitar.

Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

Email Archiving with Mail Manager: Why To Archive In Transit vs At The Mailbox

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/email-archiving-with-mail-manager-why-to-archive-in-transit-vs-at-the-mailbox/

When designing Amazon Simple Email Service’s (SES) Mail Manager, we often heard from customers about the “PST-file problem” inherent with user-side mailbox-based archiving. This occurs when, for a variety of reasons, end users decide to archive their emails to local PST files or other local storage. These PST files are fragile and easily corrupted. Furthermore, they are subject to the backup practices of individual workstations. Lastly, PST files are readily are portable and can be easily copied and moved outside the visibility of the email system and your IT and IP controls.

We developed Amazon Simple Email Service (SES) Mail Manager archiving features in response to this problem, and based on additional customer feedback: the need for consistent email retention behaviors, for all email. Customers also wanted the flexibility to determine which messages to archive, where to put them, and how long to retain those messages.

To make the feature applicable to the widest set of use cases, we designed Mail Manager to be able to archive any email traversing the SES service, not just those that have already been delivered to a user’s mailbox. This added flexibility ensures organizations can maintain a complete record of exactly those email communications they wish to preserve. Rather than require external tools to search and export Mail Manager’s archives, we built these functions directly into the SES console.

In fact, the entire Media Manager archiving solution is fully managed by SES within the customer’s Mail Manager account, reducing the operational overhead traditionally associated with email archiving and compliance.

Figure 1 - Mail Manager Archiving

Figure 1 – Mail Manager Archiving

At the core of the SES Mail Manager archiving solution is the ability to capture and retain any message, regardless of its source or destination, as it flows through the service’s rules engine. This design approach ensures that every email message traversing Mail Manager can be subject to archiving and retention policies, rather than requiring organizations to manage different systems and tools for mail flowing through mail servers, internal relays and other email infrastructure. The result is a unified, comprehensive compliance solution that provides visibility and control over an organization’s email archiving.

SES also published a detailed overview of the Archiving feature, which is available here: Archiving and sending to final SMTP server.

Archiving on its own isn’t an innovation; it’s an email primitive – an essential capability that can be used to enable other, more complex solutions. Historically, retention of email was configured as a function of your on-premises mail server, where your mailboxes themselves were resident. Personally-authored emails were considered the high-value material to retain, and adding archiving as a function of mailbox configurations was the simplest approach.

In practice, we find that the mail captured at the mailbox server, or end user’s inbox, represents only a fraction of of the mail a typical enterprise generates. As organizations grow, the number of applications generating Application To Person (A2P) messages tends to increase dramatically. Similarly, as corporate environments become more complex, SaaS-based solutions that are external to the primary email infrastructure often use email to update employees along with workflow-management systems. Much of that mail eludes archiving as it bypasses individual user mailboxes.

The SES strategy for archiving is to capture mail from anywhere, to anywhere, as long as it transits an ingress endpoint as part of your Mail Manager configuration. You have two choices: you can write those messages directly to an S3 bucket you control, and then ingest it into any other tool you like. Alternately, you can send messages into a managed archive within Mail Manager, and gain access to search, export, and configurable retention features. By default, SES configures retention for 6 months, but it’s adjustable up to permanent retention for customers who require it.

Mail Manager’s archiving feature captures any message which matches your rule, or all messages traversing any ingress endpoint. You can choose to write all messages to or from your senior leadership team into one archive, or you can organize by other envelope metadata. The rules operate the same way whether the message is A2P or Person to Person (P2P), ensuring uniform policies and retention options.

With Mail Manager’s managed archives, you pay for each gigabyte ingested, indexed, and available for search, and a separate storage fee for each gigabyte retained every month. Note that the storage fee includes both the raw content of the messages, and the size of the computed index required for search and export functions.

For messages you write to your S3 buckets, you also have the option to invoke an S3 trigger action that calls an Amazon Lambda to drive various automatation workflows. Regulated industries might want to write all messages to S3 to leverage S3’s glacier storage option for very long-term storage.

You can even split your workload between Mail Manager’s managed archive, for emails you are likely to need readily discoverable, and the Write to S3 option, for content which you don’t expect to ever need to search with granularity, but still needs to be archived to “check the box” for your retention policy. In fact, AWS encourages such a builder-oriented approach, because it rewards thoughtful decisions and resource utilization, and conforms to the broad goal of consumption-based pricing, which Mail Manager embraces fully at every step.

Figure 2 - Rule Set with conditions for archiving

Figure 2 – Rule Set with conditions for archivingMail Manager provides a more comprehensive, resilient archiving approach that increases both the overall scope of mail that can be captured, and the fidelity of the archived data. You don’t need any special adapters or plugins to capture mail from any source. All email that comes through your Mail Manager Ingress Endpoint can be archived.

Figure 3 - Create archive

Figure 3 – Create archive

Why not try Mail Manager today and experience the benefits of a centralized, scalable email archiving solution? You’ll pay only for the data you ingest and retain each month, without the fragility and visibility issues of user-managed archives. Visit the SES website to start your free trial of Mail Manager and take control of your organization’s critical email records. To start with Mail Manager, visit https://aws.amazon.com/ses/, click on Mail Manager, and set up your first workload today.

If you have any questions or need further guidance, feel free to reach out to us via the SES Forums or in the comments section of this blog post. We’re here to help you navigate the evolving email landscape and unlock the full potential of your Amazon SES investment.

About the Authors

Toby Weir-Jones

Toby Weir-Jones

Toby is a Principal Product Manager for Amazon SES and WorkMail. He joined AWS in January 2021 and has significant experience in both business and consumer information security products and services. His focus on email solutions at SES is all about tackling a product that everyone uses and finding ways to bring innovation and improved performance to one of the most ubiquitous IT tools.

Brett Ezell

Brett Ezell

Brett is an Amazon Pinpoint and Amazon Simple Email Service Specialist Solutions Architect at AWS. As a Navy veteran, he joined AWS in 2020 through an AWS technical military apprenticeship program. When he isn’t deep diving into solutions for customer challenges, Brett spends his time collecting vinyl, attending live music, and training at the gym. An admitted comic book nerd, he feeds his addiction every Wednesday by combing through his local shop for new books.

Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

How Amazon SES Mail Manager Elevates Email Security and Efficiency

Post Syndicated from Pavlos Ioannou Katidis original https://aws.amazon.com/blogs/messaging-and-targeting/how-amazon-ses-mail-manager-elevates-email-security-and-efficiency/

In today’s digital landscape, efficient and secure email management is essential for businesses facing the complexities of cyber threats and regulatory compliance. Companies are seeking ways to safeguard against unauthorized access and apply audit rules, while maintaining operational efficiency. Amazon SES Mail Manager is designed to meet these challenges, offering a suite of features that enhance both inbound and outbound email flows.

Mail Manager provides key components such as traffic policies for detailed email filtering, authenticated ingress endpoints that ensure emails are received only from verified senders, and customizable rule sets that enable administrators to precisely manage email traffic. These tools aim to bolster security and streamline the email management process.

The blog explores Mail Manager’s capabilities by demonstrating how each component works and can be utilized in practical business scenarios. Some common use cases include security, where Mail Manager blocks harmful emails based on IP ranges, TLS versions, and authentication checks while leveraging third-party security add-ons. Another use case is email archiving, where you can use Mail Manager to set up multiple archives with customizable retention periods and encryption, ensuring compliance and easy searchability.

Familiarize with some of mail manager’s key components below before proceeding with the customer use cases.

Mail manager components definition:

  • Ingress endpoints:
    • Open ingress endpoint: a SMTP endpoint responsible for accepting connections, and process SMTP conversation key infrastructure. It’s a key component that utilizes traffic polices and rules that you can configure to determine which emails should be allowed into your organization and which ones should be rejected.
    • Authenticated ingress endpoint: Mail sent to your domain has to come from authorized senders whom you’ve shared your SMTP credentials with, such as your on-premise email servers.
  • Traffic policies: let you determine the email you want to allow or block from your ingress endpoint. A traffic policy consists of one or more policy statements where you allow or deny traffic based on a variety of protocols including recipient address, sender IP address and TLS protocol version.
  • Rules sets: A Rule set is a container for an ordered set of rules you create to perform actions on your email. Each rule consists of conditions and rules.
  • Email add-ons: A suite of 3rd party applications that are seamlessly integrated with Amazon SES mail manager. Some of them are Trend Micro Virus Scanning, Abusix Mail intelligence and Spamhaus Domain Block List.

For a deep dive into Mail Manager’s capabilities, ready this blog.

Customer background and use case

Nutrition.co is an online retail business with multiple departments, including marketing, tech, and sales, that send and receive emails. Nutrition.co is looking for a solution to monitor both outbound and inbound emails and apply various controls such as filtering, message processing, and archiving. Nutrition.co uses Outlook as an enterprise mailbox environment for its employees.

Use case 1: Nutrition.co to the world

This use case focuses on the outbound email flow, where Nutrition.co employees are sending emails outside of Nutrition.co. Some of the requirements include the archival of all outbound emails originated by the marketing department, blocking any tech emails exceeding 1mb and scanning the email content of emails originated by sales. These controls should be centrally managed and provide flexibility to edit/create/delete new ones.

Solution: Each department will direct its outbound emails to an authenticated ingress endpoint by configuring an Exchange transport rule. These endpoints ensure that only authorized senders with SMTP credentials can send emails. Each ingress endpoint generates an A record, which is added as an MX record to the DNS provider for each department’s subdomain. Additionally, each ingress endpoint is associated with a specific traffic policy and rule set. According to Nutrition.co’s requirements, all connections between the departments and the ingress endpoints must use TLS 1.3 or higher. Emails that comply with the traffic policies are processed through distinct rule sets. Emails from marketing that comply with DKIM and SPF are first archived and then sent to the recipient via the Send to Internet action. Tech emails have their recipient’s address rewritten to a test email address, while emails from the sales department undergo content scanning before being sent to the final recipients via the Send to Internet action.

SES-Mail-Manager-Outbound

Use case 2: World to Nutrition.co

This use case focuses on the inbound email flow, where third parties send emails to Nutrition.co. Nutrition.co requires inbound emails to align with SPF and DKIM and have TLS 1.3 or higher to be archived. Emails originating from warehouse.com, Nutrition.co’s fulfilment partner, are containing customer order updates. These emails should be processed by Nutrition.co and accordingly update the customers’ order status database. Furthermore, warehouse.com emails should originate from a certain IP range, have TLS 1.3 or higher and align with SPF and DKIM.

Solution: Nutrition.co will use an open ingress endpoint without authentication for all inbound external emails. This is achieved by adding an MX record generated by Mail Manager upon the creation of the ingress endpoint. This ingress endpoint will be associated with a traffic policy that evaluates TLS. If the inbound email conforms to the traffic policy, it will proceed through the rule set condition and actions. The rule set condition is to align with SPF and DKIM and the actions are to be archived and then sent to the final recipient (Nutrition.co employee) via SMTP Relay. Emails containing parcel delivery updates from warehouse.com will be directed to a separate Nutrition.co subdomain, which routes all inbound emails to an authenticated ingress endpoint. Emails from warehouse.com with TLS 1.3 or higher will meet the traffic policy requirements. If they are SPF and DKIM aligned, they will be stored in a Nutrition.co Amazon S3 bucket as part of the rule set. Using Amazon S3 notifications, an AWS Lambda function is invoked upon receiving an email. This function processes the email payload, and performs an API call to update the Nutrition.co customers’ order status database.

SES-Mail-Manager-Inbound

Archiving inbound emails

In the following section, you will use AWS CloudShell and AWS CLI commands to create a traffic policy that rejects emails with TLS versions lower than 1.3, includes an open ingress endpoint, and establishes a ruleset that archives emails that are DKIM aligned.

Prerequisites: Own a domain and have access to its DNS provider, in order to add the MX record.

Navigate to the AWS Management Console and open CloudShell, find CloudShell availability here. Follow the steps below by copying and pasting the AWS CLI commands to the CloudShell terminal. Note that creating and configuring these resources, can also be done from the AWS Console.

# 1. Creating archive

ARCHIVE=$(aws mailmanager create-archive \
  --archive-name NutritionCo \
  --retention RetentionPeriod=THREE_MONTHS \
  --region ${AWS_REGION} \
  --tags Key=Company,Value=NutritionCo | jq -r '.ArchiveId') && echo $ARCHIVE

# 2. Creating traffic policy

TRAFFIC_POLICY=$(aws mailmanager --region ${AWS_REGION} create-traffic-policy \
  --traffic-policy-name ArchiveTrafficPolicy \
  --default-action DENY \
  --policy-statements '[
    {
      "Action": "ALLOW",
      "Conditions": [
        {
          "TlsExpression": {
            "Evaluate": {
              "Attribute": "TLS_PROTOCOL"
            },
            "Operator": "MINIMUM_TLS_VERSION",
            "Value": "TLS1_3"
          }
        }
      ]
    }
  ]'| jq -r '.TrafficPolicyId') && echo $TRAFFIC_POLICY

# 3. Creating Mailmanager RuleSet for archiving

RULE_SET=$(aws mailmanager --region ${AWS_REGION} create-rule-set \
  --rule-set-name ArchiveRuleSet \
  --rules '[
    {
      "Name": "Archive",
      "Actions": [
        {
          "Archive": {
            "TargetArchive": "'"${ARCHIVE}"'"
          }
        }
      ],
      "Conditions": [
        {
          "VerdictExpression": {
            "Evaluate": {
              "Attribute": "DKIM"
            },
            "Operator": "EQUALS",
            "Values": ["PASS"]
          }
        }
      ]
    }
  ]'| jq -r '.RuleSetId') && echo $RULE_SET

# 4. Create ingress endpoint

aws mailmanager --region ${AWS_REGION} create-ingress-point \
--ingress-point-name Archiving \
--type OPEN \
--traffic-policy-id ${TRAFFIC_POLICY} \
--rule-set-id ${RULE_SET}

To view the resources created above, navigate to the Amazon SES console > Mail Manager and view Traffic policies and Rule sets. Below, you can see the rule in edit mode.

Mail-Manager-RulesetNavigate to Amazon SES > Mail Manager > Ingress endpoint, select the ingress endpoint named Archiving and copy the ARecord, which looks like this <unique-id>.fips.wmjb.mail-manager-smtp.amazonaws.com – see screenshot below. Add this value to your MX record.

Mail-Manager-IngressEndpoint

To test if the MX record has been added successfully, open your local terminal and execute the command below:
nslookup -type=MX <your-domain.com>
The response should return the MX preference and mail exchanger containing the A record value.

Testing

To test if the inbound emails are archived successfully, send an email to an address within the domain for which you have added the MX record. Wait for 3-5 minutes to allow for email processing. Then, navigate to the AWS Management Console, go to Amazon SES, and select Mail Manager. Under Email Archiving, select NutritionCo under Archive and click on Search. This should return all the emails you have sent.

MailManager-Archive

Conclusion & Next steps

In this blog, we delved into the essential features of Amazon SES Mail Manager and its application in managing both inbound and outbound email flows. We explored key components such as traffic policies, authenticated ingress endpoints, and customizable rule sets that enhance security and operational efficiency. Through practical use cases, this blog demonstrates how these features can be implemented to meet the specific needs of a business like Nutrition.co. By leveraging Amazon SES Mail Manager, businesses can significantly enhance their email security and management processes, safeguarding against cyber threats while ensuring compliance and efficiency.

Continue exploring Mail Manager’s features such as SMTP relays and Email add-ons.

About the Authors

Pavlos Ioannou Katidis

Pavlos Ioannou Katidis

Pavlos Ioannou Katidis is an Amazon Pinpoint and Amazon Simple Email Service Senior Specialist Solutions Architect at AWS. He enjoys diving deep into customers’ technical issues and help in designing communication solutions. In his spare time, he enjoys playing tennis, watching crime TV series, playing FPS PC games, and coding personal projects.

Alexey Kiselev

Alexey Kiselev

Alexey Kiselev is a Senior SDE working on Amazon Email. Alexey has played a pivotal role in shaping the design, infrastructure, and delivery of MailManager. With years of experience, deep understanding of the industry and a passion for innovation he is enthusiast and a builder with a core area of interest on scalable and cost-effective email management and email security solutions.

The Four Pillars of Managing Email Reputation

Post Syndicated from Dustin Taylor original https://aws.amazon.com/blogs/messaging-and-targeting/the-four-pillars-of-email-reputation/

Introduction

A sender’s domain and IP reputation strongly indicate email deliverability success. Maintaining a high reputation ensures optimal recipient inboxing. This blog outlines how Amazon SES protects its network reputation to help customers deliver high-quality email consistently. Understanding sender reputation nuances across diverse mailbox providers can be challenging, making issue identification and root cause analysis difficult. We’ll explore SES’ approach to managing domain and IP reputation.

What are Domain and IP Reputation?

Domain and IP reputation are measured by mailbox providers to indicate how reputable a sender is based primarily on how recipients engage with their email. Mailbox providers have their own way of measuring reputation and typically consider indicators such as:

  • A history of the emails received from the domain/IP
  • The authentication technologies used during delivery (SPF, DKIM, DMARC)
  • The rate of user engagement for the messages
  • The rate of complaints generated by the messages
  • The rate of which the mailbox providers’ spam filter determines mail to be spam from a domain/IP

While not an exhaustive list, these are some of the inputs into the reputation of a sender. Of this list, 4 of the 5 have nothing to do with the body, or viewable content, of the email that is received. This illustrates how important it is to have effective processes in place to set up sending from your domain/IPs and the management of your email sending programs.

How does Amazon SES manage Domain and IP Reputation

Management of reputation requires a multi-faceted approach distilled into four distinct pillars: Prevention, Monitoring, Analysis, and Response. Let’s dive deeper into these four pillars to see how Amazon SES operates to protect sending reputation for our service and our customers.

Prevention

Prevention is arguably the most important of the four pillars of reputation management. Abuse, or misuse, is the leading cause of poor reputation. Abuse, or misuse, can be characterized as sending phishing emails, unsolicited emails, or aggressive sending practices ignoring user feedback or lack of engagement, but this is not an exhaustive list. Prevention of abuse is accomplished through customer education (blogs, public documentation, and customer correspondence), service terms, acceptable use policies, and strict rules on setup. These abuse prevention mechanisms aid in educating customers before they use SES on prohibited sending practices as well as providing guidance on email sending best practices. SES implements several mechanisms to mitigate abuse and misuse, including:

  • Production access reviews – Every customer must request access to send email outbound. This step plays two parts: 1\ giving customers an opportunity to test sending from and to verified identities and 2\ preventing malicious senders from being able to open an account and begin uninhibited sending of low-quality mail. Every customer requesting access to send via SES provides information on their sending practices and volume estimates. This information is used in three ways: first to ensure that a customer is following best practices for sending email, second to provide the appropriate limits needed for their business, and third to determine if a customer’s sending practices are a risk to other senders.
  • Restricted sending from only verified identities/domains – Every customer, must verify ownership of an email address or sending domain to send an email on SES. This can be done for email addresses by clicking a verification link or for domains by placing DNS records that SES is able to verify.
  • Daily volume and sending rate limits – SES applies sending limits to an account for the following reasons: prevention of reputational damage and limiting costs should a bug occur within a customer’s application, and limiting the damage an elusive bad actor may cause.

Monitoring

The second pillar of reputation management is accurately monitoring your sending performance. Amazon SES tracks metrics like bounces, complaints, abuse reports, and mailbox provider status codes. Establishing overall sending baselines is crucial to measure the impact of deliverability and reputation changes. Granular monitoring is equally important, including metrics at the account, domain, IP, and blocklist levels.

Having granular data regarding our customer’s sending performance gives SES, and our customers, the opportunity to identify mechanisms in which a customer’s sending can improve, or indicators of when a bad actor may intend to misuse SES. Some of the mechanisms that we use to reduce the risk of reputation degradation include:

  • Monitoring new customer activity closely – The riskiest time for SES is when a new customer begins using SES and we have no historical precedent for the mail they are attempting to send. While the overwhelming majority of our customers send quality email, it’s important to ensure that a customer that is onboarding exhibits good sending practices. A customer may be inexperienced in their sending practices and SES will notify customers early to aid them in improving their sending. This limits the damage that can be done to both SES reputation and that of our customer.
  • Monitor any customers that trend away from the baseline – SES looks to determine what customers are doing well and where they could improve. Should they be given access to send freely, or should there be restrictions?
  • Monitor high-performing customers as well as low-performing customers – For SES, it’s crucial to prevent events that can degrade our reputation, such as sender compromises, uploading purchased lists, or using unsolicited recipient lists. Thoroughly reviewing all customers is essential to avoid reputational degradation from unnoticed compromises or misused recipient lists.
  • Providing our customers with a way to monitor more than bounce and complaint rates – SES provides a feature called the Virtual Deliverability Manager (VDM) which gives customers the added insight into how their messages are received by mailbox providers. These insights are provided in a dashboard that customers can review and dig into problems at the domain level, and broken down by provider.

Analysis

The third of the pillars of reputation management is analysis. Understanding the history of a sender, normal behavior and trends, mailbox provider feedback patterns, and monitoring reputation from a reputation provider enables SES to build a picture of a sender. Lets speak on some specifics about each of these data points further.

  • Sending Behavior – Is this a new sender, or one with an established reputation? Do they have a history of previous bounce/complaint issues? What historical volume is sent?
    • Tip: Understanding the baseline or history of the sender gives you the ability to know when things have changed for the better or worse.
  • Mailbox provider feedback – Amazon SES reviews mailbox provider feedback patterns to analyze responses when sending mail. If normally all SES mail is received successfully and we begin to see a spike in throttles with a negative response message such as this one from Gmail:

    421-4.7.28 Gmail has detected an unusual rate of unsolicited mail originating from your DKIM domain [example.com 36]. To protect our users from spam, mail sent from your domain has been temporarily rate limited. For more information, go to https://support.google.com/mail/?p=UnsolicitedRateLimitError to review our Bulk Email Senders Guidelines. m25-20020ae9e019000000b0078edf1f4c40si26277545qkk.197 – gsmtp

    this could be the first sign of reputation degradation.

    • Tip: Mailbox provider feedback is a good data point for degradation, however this is a late sign as the damage has already occurred. More proactive measures should be in place to ensure this step doesn’t occur.
  • Using reputation providers – External feedback on Amazon SES reputation is critical to validate our processes and identify potential gaps. Selecting a reputation provider has helped SES close this gap.
    • Tip: As a mail provider, you rely on sending metrics and historical data for monitoring senders. However, you may not know how customers acquired their recipient lists – whether through confirmed opt-ins or purchased lists. Purchased lists risk your domain being blocklisted since recipients didn’t sign up for your mail. Lacking visibility into subscription workflows makes it hard to determine why blocklisting occurred. Refer to our FAQ for more on blocklists.

Response

The fourth of the pillars of reputation management is response. Understanding what to do when your reputation begins to show signs of decline is important. Some signals that show reputation declines are: low inbox rates, mail being throttled, mail being blocked, or external reputation tools showing poor reputation for your domain/IP. For Amazon SES, we take action to do the following:

  • Contact customers where metrics breach alarm thresholds.
  • Respond timely to signs of abuse or reputation degradation.
  • Stop sending based on continued, or high-risk, signals of abuse or reputation degradation.
  • Support customers in resolving sending issues to maintain the overall reputation of SES.

It is important to respond quickly to the signals of reputation degradation. The decision to impact a customer’s ability to send mail is not one that Amazon SES takes lightly. A decision to impact a customer’s ability to send mail is made when the quality of mail is abusive in nature (phishing) or if there are signals that the mail being sent is not well received by mailbox providers at scale. In some cases, a customer may not be aware that their sending patterns, practices, or content may be problematic. This can be due to a gap in monitoring, logging, or an issue with credentials being compromised. If the decision to impact a customer’s sending is made, a communication will be sent to that customer so that we can partner with them to resolve the issue.

Amazon SES doesn’t only make the decision to communicate with our customers when there is a problem. SES also communicates with customers, when appropriate, earlier in the reputation management cycle to warn of a negative trend in sending. This can be seen in the review periods that are triggered when increases in bounces, complaints, or mailbox provider feedback is seen. These review periods give SES customers the ability and time to understand the problem, and to work on fixes to avoid serious reputation impact. Being involved early in the discovery phase of a sending event improves the customer experience without the need to negatively impact sending.

Conclusion

Maintaining a positive sending reputation necessitates a diligent approach to prevent abusive emails. The four pillars outlined serve as guidelines to improve email quality: prevention, monitoring, analysis, and response. This is an iterative process that requires moving fluidly between pillars.

About the author:

Dustin Taylor

Dustin Taylor

Dustin is the Manager of anti-abuse and email deliverability for Amazon SES. His focus is both external and internal in helping improve inbox placement for SES customers and finding new ways to fight email abuse. In his off-time he enjoys going bass fishing and is a hobbyist woodworker.

Serverless IoT email capture, attachment processing, and distribution

Post Syndicated from Stacy Conant original https://aws.amazon.com/blogs/messaging-and-targeting/serverless-iot-email-capture-attachment-processing-and-distribution/

Many customers need to automate email notifications to a broad and diverse set of email recipients, sometimes from a sensor network with a variety of monitoring capabilities. Many sensor monitoring software products include an SMTP client to achieve this goal. However, managing email server infrastructure requires specialty expertise and operating an email server comes with additional cost and inherent risk of breach, spam, and storage management. Organizations also need to manage distribution of attachments, which could be large and potentially contain exploits or viruses. For IoT use cases, diagnostic data relevance quickly expires, necessitating retention policies to regularly delete content.

Solution Overview

This solution uses the Amazon Simple Email Service (SES) SMTP interface to receive SMTP client messages, and processes the message to replace an attachment with a pre-signed URL in the resulting email to its intended recipients. Attachments are stored separately in an Amazon Simple Storage Service (S3) bucket with a lifecycle policy implemented. This reduces the storage requirements of recipient email server receiving notification emails. Additionally, this solution leverages built-in anti-spam and security scanning capabilities to deal with spam and potentially malicious attachments while at the same time providing the mechanism by which pre-signed attachment links can be revoked should the emails be distributed to unintended recipients.

The solution uses:

  • Amazon SES SMTP interface to receive incoming emails.
  • Amazon SES receipt rule on a (sub)domain controlled by administrators, to store raw incoming emails in an Amazon S3 bucket.
  • AWS Lambda function, triggered on S3 ObjectCreated event, to process raw emails, extract attachments, replace each with pre-signed URL with configurable expiry, and send the processed emails to intended recipients.

Solution Flow Details:

  1. SMTP client transmits email content to an email address in a (sub) domain with MX record set to Amazon SES service’s regional endpoint.
  2. Amazon SES SMTP interface receives an email and forwards it to SES Receipt Rule(s) for processing.
  3. A matching Amazon SES Receipt Rule saves incoming email into an Amazon S3 Bucket.
  4. Amazon S3 Bucket emits an S3 ObjectCreated Event, and places the event onto the Amazon Simple Queue Services (SQS) queue.
  5. The AWS Lambda service polls the inbound messages’ SQS queue and feeds events to the Lambda function.
  6. The Lambda function, retrieves email files from the S3 bucket, parses the email sender/subject/body, saves attachments to a separate attachment S3 bucket (7), and replaces attachments with pre-signed URLs in the email body. The Lambda function then extracts intended recipient addresses from the email body. If the body contains properly formatted recipients list, email is then sent using SES API (9), otherwise a notice is posted to a fallback Amazon Simple Notification Service (SNS) Topic (8).
  7. The Lambda function saves extracted attachments, if any, into an attachments bucket.
  8. Malformed email notifications are posted to a fallback Amazon SNS Topic.
  9. The Lambda function invokes Amazon SES API to send the processed email to all intended recipient addresses.
  10. If the Lambda function is unable to process email successfully, the inbound message is placed on to the SQS dead-letter queue (DLQ) queue for later intervention by the operator.
  11. SES delivers an email to each recipients’ mail server.
  12. Intended recipients download emails from their corporate mail servers and retrieve attachments from the S3 pre-signed URL(s) embedded in the email body.
  13. An alarm is triggered and a notification is published to Amazon SNS Alarms Topic whenever:
    • More than 50 failed messages are in the DLQ.
    • Oldest message on incoming SQS queue is older than 3 minutes – unable to keep up with inbound messages (flooding).
    • The incoming SQS queue contains over 180 messages (configurable) over 5 minutes old.

Setting up Amazon SES

For this solution you will need an email account where you can receive emails. You’ll also need a (sub)domain for which you control the mail exchanger (MX) record. You can obtain your (sub)domain either from Amazon Route53 or another domain hosting provider.

Verify the sender email address

You’ll need to follow the instructions to Verify an email address for all identities that you use as “From”, “Source”, ” Sender”, or “Return-Path” addresses. You’ll also need to follow these instructions for any identities you wish to send emails to during initial testing while your SES account is in the “Sandbox” (see next “Moving out of the SES Sandbox” section).

Moving out of the SES Sandbox

Amazon SES accounts are “in the Sandbox” by default, limiting email sending only to verified identities. AWS does this to prevent fraud and abuse as well as protecting your reputation as an email sender. When your account leaves the Sandbox, SES can send email to any recipient, regardless of whether the recipient’s address or domain is verified by SES. However, you still have to verify all identities that you use as “From”, “Source”, “Sender”, or “Return-Path” addresses.
Follow the Moving out of the SES Sandbox instructions in the SES Developer Guide. Approval is usually within 24 hours.

Set up the SES SMTP interface

Follow the workshop lab instructions to set up email sending from your SMTP client using the SES SMTP interface. Once you’ve completed this step, your SMTP client can open authenticated sessions with the SES SMTP interface and send emails. The workshop will guide you through the following steps:

  1. Create SMTP credentials for your SES account.
    • IMPORTANT: Never share SMTP credentials with unauthorized individuals. Anyone with these credentials can send as many SMTP requests and in whatever format/content they choose. This may result in end-users receiving emails with malicious content, administrative/operations overload, and unbounded AWS charges.
  2. Test your connection to ensure you can send emails.
  3. Authenticate using the SMTP credentials generated in step 1 and then send a test email from an SMTP client.

Verify your email domain and bounce notifications with Amazon SES

In order to replace email attachments with a pre-signed URL and other application logic, you’ll need to set up SES to receive emails on a domain or subdomain you control.

  1. Verify the domain that you want to use for receiving emails.
  2. Publish a mail exchanger record (MX record) and include the Amazon SES inbound receiving endpoint for your AWS region ( e.g. inbound-smtp.us-east-1.amazonaws.com for US East Northern Virginia) in the domain DNS configuration.
  3. Amazon SES automatically manages the bounce notifications whenever recipient email is not deliverable. Follow the Set up notifications for bounces and complaints guide to setup bounce notifications.

Deploying the solution

The solution is implemented using AWS CDK with Python. First clone the solution repository to your local machine or Cloud9 development environment. Then deploy the solution by entering the following commands into your terminal:

python -m venv .venv
. ./venv/bin/activate
pip install -r requirements.txt

cdk deploy \
--context SenderEmail=<verified sender email> \
 --context RecipientEmail=<recipient email address> \
 --context ConfigurationSetName=<configuration set name>

Note:

The RecipientEmail CDK context parameter in the cdk deploy command above can be any email address in the domain you verified as part of the Verify the domain step. In other words, if the verified domain is acme-corp.com, then the emails can be [email protected], [email protected], etc.

The ConfigurationSetName CDK context can be obtained by navigating to Identities in Amazon SES console, selecting the verified domain (same as above), switching to “Configuration set” tab and selecting name of the “Default configuration set”

After deploying the solution, please, navigate to Amazon SES Email receiving in AWS console, edit the rule set and set it to Active.

Testing the solution end-to-end

Create a small file and generate a base64 encoding so that you can attach it to an SMTP message:

echo content >> demo.txt
cat demo.txt | base64 > demo64.txt
cat demo64.txt

Install openssl (which includes an SMTP client capability) using the following command:

sudo yum install openssl

Now run the SMTP client (openssl is used for the proof of concept, be sure to complete the steps in the workshop lab instructions first):

openssl s_client -crlf -quiet -starttls smtp -connect email-smtp.<aws-region>.amazonaws.com:587

and feed in the commands (replacing the brackets [] and everything between them) to send the SMTP message with the attachment you created.

EHLO amazonses.com
AUTH LOGIN
[base64 encoded SMTP user name]
[base64 encoded SMTP password]
MAIL FROM:[VERIFIED EMAIL IN SES]
RCPT TO:[VERIFIED EMAIL WITH SES RECEIPT RULE]
DATA
Subject: Demo from openssl
MIME-Version: 1.0
Content-Type: multipart/mixed;
 boundary="XXXXboundary text"

This is a multipart message in MIME format.

--XXXXboundary text
Content-Type: text/plain

Line1:This is a Test email sent to coded list of email addresses using the Amazon SES SMTP interface from openssl SMTP client.
Line2:Email_Rxers_Code:[ANYUSER1@DOMAIN_A,ANYUSER2@DOMAIN_B,ANYUSERX@DOMAIN_Y]:Email_Rxers_Code:
Line3:Last line.

--XXXXboundary text
Content-Type: text/plain;
Content-Transfer-Encoding: Base64
Content-Disposition: attachment; filename="demo64.txt"
Y29udGVudAo=
--XXXXboundary text
.
QUIT

Note: For base64 SMTP username and password above, use values obtained in Set up the SES SMTP interface, step 1. So for example, if the username is AKZB3LJAF5TQQRRPQZO1, then you can obtain base64 encoded value using following command:

echo -n AKZB3LJAF5TQQRRPQZO1 |base64
QUtaQjNMSkFGNVRRUVJSUFFaTzE=

This makes base64 encoded value QUtaQjNMSkFGNVRRUVJSUFFaTzE= Repeat same process for SMTP username and password values in the example above.

The openssl command should result in successful SMTP authentication and send. You should receive an email that looks like this:

Optimizing Security of the Solution

  1. Do not share DNS credentials. Unauthorized access can lead to domain control, potential denial of service, and AWS charges. Restrict access to authorized personnel only.
  2. Do not set the SENDER_EMAIL environment variable to the email address associated with the receipt rule. This address is a closely guarded secret, known only to administrators, and should be changed frequently.
  3. Review access to your code repository regularly to ensure there are no unauthorized changes to your code base.
  4. Utilize Permissions Boundaries to restrict the actions permitted by an IAM user or role.

Cleanup

To cleanup, start by navigating to Amazon SES Email receiving in AWS console, and setting the rule set to Inactive.

Once completed, delete the stack:

cdk destroy

Cleanup AWS SES Access Credentials

In Amazon SES Console, select Manage existing SMTP credentials, select the username for which credentials were created in Set up the SES SMTP interface above, navigate to the Security credentials tab and in the Access keys section, select Action -> Delete to delete AWS SES access credentials.

Troubleshooting

If you are not receiving the email or email is not being sent correctly there are a number of common causes of these errors:

  • HTTP Error 554 Message rejected email address is not verified. The following identities failed the check in region :
    • This means that you have attempted to send an email from address that has not been verified.
    • Please, ensure that the “MAIL FROM:[VERIFIED EMAIL IN SES]” email address sent via openssl matches the SenderEmail=<verified sender email> email address used in cdk deploy.
    • Also make sure this email address was used in Verify the sender email address step.
  • Email is not being delivered/forwarded
    • The incoming S3 bucket under the incoming prefix, contains file called AMAZON_SES_SETUP_NOTIFICATION. This means that MX record of the domain setup is missing. Please, validate that the MX record (step 2) of Verify your email domain with Amazon SES to receive emails section is fully configured.
    • Please ensure after deploying the Amazon SES solution, the created rule set was made active by navigating to Amazon SES Email receiving in AWS console, and set it to Active.
    • This may mean that the destination email address has bounced. Please, navigate to Amazon SES Suppression list in AWS console ensure that recipient’s email is not in the suppression list. If it is listed, you can see the reason in the “Suppression reason” column. There you may either manually remove from the suppression list or if the recipient email is not valid, consider using a different recipient email address.
AWS Legal Disclaimer: Sample code, software libraries, command line tools, proofs of concept, templates, or other related technology are provided as AWS Content or Third-Party Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content or Third-Party Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content or Third-Party Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content or Third-Party Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.

About the Authors

Tarek Soliman

Tarek Soliman

Tarek is a Senior Solutions Architect at AWS. His background is in Software Engineering with a focus on distributed systems. He is passionate about diving into customer problems and solving them. He also enjoys building things using software, woodworking, and hobby electronics.

Dave Spencer

Dave Spencer

Dave is a Senior Solutions Architect at AWS. His background is in cloud solutions architecture, Infrastructure as Code (Iac), systems engineering, and embedded systems programming. Dave’s passion is developing partnerships with Department of Defense customers to maximize technology investments and realize their strategic vision.

Ayman Ishimwe

Ayman Ishimwe

Ayman is a Solutions Architect at AWS based in Seattle, Washington. He holds a Master’s degree in Software Engineering and IT from Oakland University. With prior experience in software development, specifically in building microservices for distributed web applications, he is passionate about helping customers build robust and scalable solutions on AWS cloud services following best practices.

Dmytro Protsiv

Dmytro Protsiv

Dmytro is a Cloud Applications Architect for with Amazon Web Services. He is passionate about helping customers to solve their business challenges around application modernization.

Stacy Conant

Stacy Conant

Stacy is a Solutions Architect working with DoD and US Navy customers. She enjoys helping customers understand how to harness big data and working on data analytics solutions. On the weekends, you can find Stacy crocheting, reading Harry Potter (again), playing with her dogs and cooking with her husband.