Tag Archives: SES

How to use Mail Manager to Archive Inbound Emails

2024-06-25 Sesan Komaiya

Post Syndicated from Sesan Komaiya original https://aws.amazon.com/blogs/messaging-and-targeting/archiving-and-sending-to-final-smtp-server/

In today’s digital landscape, where email communication plays a vital role in business operations, Keeping your email archive secure, compliant, and retrievable is crucial for any business. However, managing the large volume of email data can lead to operational difficulties, including regulatory compliance, maintaining an audit trail, and preventing data loss. That’s where Amazon Simple Email Service (SES) Mail Manager’s email archiving feature comes in.

In this blog post, we will explore how Amazon SES Mail Manager’s email archiving and search features can improve email security and compliance. If you’re a newcomer to Mail Manager, look at this blog post on Amazon SES Mail Manager. It provides valuable information on important features, such as Ingress Endpoint, Traffic Policy, Rule Sets, and SMTP Relay.

Problem Statement:

Imagine a scenario where a critical email from a key client is buried deep within your organization’s email archives, and you need to retrieve it for an important audit. The challenge of ensuring your business remains compliant with stringent data retention policies across every email communication for thousands of employees for a certain period or permanently.

Solution explained:

Amazon SES Mail Manager Email archiving is a powerful tool that addresses many of the challenges organizations face dealing with the difficulty and expense of archiving email at scale. Compliance and regulatory frameworks like GDPR, HIPAA, and SOX often require email archiving, which is a common objective identified by customers needing to comply with those regulatory frameworks. For regulated businesses, failure to comply with email archiving regulations can result in severe financial penalties and reputational damage.

Amazon SES Mail Manager securely archives and safeguards your emails, providing easy search and export functionality. It provides full-time, enterprise-level archiving without increasing the storage requirements of your mailbox server. The feature provides a reliable and efficient solution to address most compliance requirements. By automatically archiving the types of emails you specify, the service ensures that your organization maintains a comprehensive audit trail of its communications, enabling quick retrieval and review as needed.

The email archiving feature of Mail Manager provides organizations the ability to archive email while in transit rather than archiving at the user’s mailbox. Many organizations prefer archiving in transit for email archiving to meet compliance requirements and maintain comprehensive records. If you would like to learn more about in transit archiving, visit this blog – Email Archiving with Mail Manager: Why To Archive In Transit vs At The Mailbox.

How email flows with an Amazon SES Mail Manager Email Archiving

For instructional purposes in this blog post, we’ll focus on how you can introduce Mail Manager archiving into your existing email infrastructure. We’ll cover how to seamlessly integrate Mail Manager with reference architectures. Later in the blog, we are going to explore Mail Manager’s archiving capabilities, including search, export and retention policies.

Current setup

Our example organization has an existing mail server (it might be a on-premises Microsoft Exchange Server, Microsoft 365, Google Workspace, etc). Their DNS is configured to route all email directly to this mail server. There is currently no archiving capacity within the existing email infrastructure, when needed, archiving is handled by individual mailbox users and PST files. While this method is suitable for personal email archiving, it fails to meet the organizations’ security requirements and compliance standards.

Figure 1: Example organization’s existing inbound email workflow.

Email Archiving in transit

We are going to introduce Mail Manager into the current mail flow (see figure 1) to archive all incoming messages from our example enterprise’s email infrastructure.

Figure 2: Example organization’s proposed inbound email workflow, with Mail Manager archiving in-transit prior to delivery.

In the new architecture (see figure 2), we’ve introduced Mail Manager into the organization’s inbound email workflow. This new workflow leverages Mail Manager’s ability to archive either all inbound emails, or only those that match specified criteria. By using a Mail Manager Rule set, our example organization can selectively store and preserve emails that meet their configured criteria.

Mail Manager Email Archiving and Search and Export Capabilities

Mail Manager’s archive search capabilities are designed to be user-friendly and efficient. You can perform searches based on various criteria, such as sender, recipient, subject line, date range, or even specific keywords in the Subject line. The search results provide options to either export the search to Amazon Simple Storage Service (S3), or you can choose to download a single email.

Let’s explore using Mail Manager’s archive search to find a specific email by the sender’s address:

Figure 3: Mail Manager’s archive search interface

Once found, we can click on the results to review the email in the console:

Figure 4: Mail Manager’s archive search results

Once we’ve found the target email, it can be downloaded by clicking on “View details“. The image below shows an example message details page with information about the email, including message headers such as In-Reply-To, X-Original-Mailer and X-Mailer.

Figure 5: Mail Manager’s archived email detailed page

Mail Manager’s archive search history tab allows us to find archive searches created in the last 30 days, and view the search results, as shown in the image below:

Figure 6: Mail Manager’s archive search history

Mail Manager’s archive export history tab lists all of the archived email searches you exported to an Amazon S3 Bucket within the last 30 days.

Figure 7: Export History

Step by Step Setup:

Now that we have explained how Mail Manger can be inserted into our example organization’s email workflow to provide email archiving, let’s explore how you can implement Mail Manager’s archiving capabilities in your inbound email workflows. The following diagram (Figure 3) illustrates the overall structure and components involved in this architecture:

Figure 8: End-to-End Mail workflow

Follow the steps below to configure Mail Manager in your AWS account to implement this architecture:

Log into the SES Console and select Mal Manager from the left navigation menu.
1. Note, as of this writing, Mail Manager is generally available in the following AWS Regions: US East (N. Virginia), US West (Oregon), Europe (Ireland, Frankfurt), Asia Pacific (Tokyo, Sydney).
Under Mail Manger navigation, create an archive (or multiple archives for different use cases)
1. Enter a unique name in the Archive name field.
2. (Optional) Select a retention period in the Retention period field to override the default retention period of 180 days.
3. (Optional) You can encrypt your archive either by entering your own AWS KMS key into the KMS key ARN field, or by selecting Create new key.
4. Choose Create archive.
Under Mail Manger navigation, create a traffic policy to determine the email you want to block or allow.
1. Create traffic policy.
2. On the Create a traffic policy page, enter a unique name for your traffic policy.
3. (Optional) If you want to discard any messages above a certain size, enter a value in bytes in the Maximum message size field.
4. In Default action, choose whether the traffic policy is to either Allow or Deny (block) messages that fall outside of (are not addressed by) the conditions of your policy statements.
5. Select Add new policy statement to create a statement for your traffic policy.
6. Choose either Allow or Deny (block) for the action to be taken when the statement’s conditions are met.
7. Build a condition by selecting an email protocol and a conditional operator for the value you enter. Select Add new condition if you want to add more conditions to this policy statement. To learn more about a condition property and its operators and valid values, see the Policy statement conditions reference.
8. If you’re subscribed to an Email Add On, you’ll be able to select it here as an email protocol.
9. If you want add more policy statements and conditions, repeat steps above.
10. Select Create traffic policy.
Under Mail Manger navigation, create a rule set to perform actions on the email you allow in.
1. Create rule set and enter a unique name for your rule set.
2. Create new rule on the edit page.
3. In the Rule details sidebar, enter a unique name for your rule.
4. Select Add new condition to create a condition that the message must match; or check the EXCEPT in the case of: box followed by Add new exception to create a condition that the message must not match.
5. Build the condition or exception by selecting an email property and a conditional operator for the value you enter. Select Add new condition or Add new exception if you want to add more conditions or exceptions to this rule. To learn more about a condition property and its operators and valid values, see the Rule conditions reference.
6. Select Add new action to define the action to be taken when the rule’s conditions are matched and/or exceptions are not matched. To add more actions to be taken, select Add new action. To learn more about actions and their parameters, see the Rule actions reference.
7. Create an Archive rule. Save rule set
Under Mail Manger navigation, Create your ingress endpoint and assign to it the traffic policy and rule set.
1. choose Ingress endpoints under Mail Manager.
2. On the Ingress endpoints page, select Create ingress endpoint.
3. On the Create new ingress endpoint page, enter a unique name for your ingress endpoint.
4. Choose whether it will be a Open or Authenticated endpoint.
5. Select a traffic policy to determine the email you want to block or allow.
6. Select a rule set containing the rule actions you want to perform on the email you allow in.
7. Select Create ingress endpoint.
Configure your environment to use the ingress endpoint.
1. At the time you create an ingress endpoint, an “A” record for the endpoint will be generated and its value displayed on the ingress endpoint’s summary screen in the SES console. The way you use the value of this record depends on the type of endpoint you created and your use case.
2. DNS providers have different procedures and interfaces for configuring email records. The key pieces of information you need to put into your DNS settings are listed in our documentation – https://docs.aws.amazon.com/ses/latest/dg/eb-ingress.html#eb-ingress-a-record
Under Mail Manger navigation, create an SMTP Relay to send mail on to your existing mail server.
1. choose SMTP relays under Mail Manager.
2. On the SMTP relays page, select Create SMTP relay.
3. On the Create SMTP relay page, enter a unique name for your SMTP relay.
4. Depending on what type of SMTP Realy you want to configure, follow the respective instructions:
  1. inbound (non-authenticated)
  2. outbound (authenticated) SMTP relay
Update your DNS MX records to point to your new Mail Manager’s ingress point, instead of the existing mail server.

Note: Make sure that you have tested the steps above in your development environment and that you understand the steps before deploying into your production environment.

Conclusion

Amazon SES Mail Manager’s email archiving capabilities are designed for organizations that are seeking to enhance the security, compliance, and audit-ability of their email communications. By seamlessly integrating this feature into their existing email infrastructure, organizations can now archive all inbound messages in transit, ensuring a comprehensive, tamper-proof record of their email activities. The powerful search and export functionality of Mail Manager makes it easy to quickly locate and access specific emails when needed, whether for compliance audits, legal requests, or internal investigations.

This level of email visibility and control is particularly crucial for organizations operating in highly regulated industries like government, healthcare and finance, where the stakes for non-compliance can be severe. Beyond the compliance benefits, Mail Manager’s email archiving also helps to alleviate the operational headaches and expenses associated with traditional in-house archiving systems. By offloading this responsibility to AWS, organizations can focus their resources on their core business priorities, while still maintaining the security and accessibility of their critical email data.

If you’re looking to strengthen your email security posture, simplify your compliance efforts, and improve the overall management of your email archives, we encourage you to explore how Amazon SES Mail Manager’s email archiving capabilities can be seamlessly integrated into your existing email infrastructure. Take the first step towards a more secure, compliant, and efficient email management solution by contacting us today.

About the Authors

Sesan Komaiya

Sesan is a Solutions Architect at Amazon Web Services. He works with a variety of customers, helping them with cloud adoption, cost optimization and emerging technologies. Sesan has over 15 year’s experience in Enterprise IT and has been at AWS for 5 years. In his free time, Sesan enjoys watching various sporting activities like Soccer, Tennis and Moto sport. He has 2 kids that also keeps him busy at home.

Alexey Kurbatsky

Alexey is a Senior Software Development Engineer at AWS, specializing in building distributed and scalable services. Outside of work, he enjoys exploring nature thru hiking as well as playing guitar.

Jesse Thompson is an Email Deliverability Manager with the Amazon Simple Email Service team. His background is in enterprise development and operations, with a focus on email abuse mitigation and encouragement of authenticity practices with open standard protocols. Jesse’s favorite activity outside of technology is recreational curling.

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

Mail Manager – Amazon SES introduces new email routing and archiving features

2024-05-22 Vinay Ujjini

Post Syndicated from Vinay Ujjini original https://aws.amazon.com/blogs/messaging-and-targeting/mail-manager-amazon-ses-introduces-new-email-routing-and-archiving-features/

Amazon Simple Email Service (SES) is a cloud-based email sending service provided by Amazon Web Services (AWS), handling both inbound and outbound email traffic for your applications. It allows users to send and receive email using SES’s reliable and cost-effective infrastructure without having to provision email servers yourself.

Managing multiple email workloads at scale can be a daunting task for organizations. From handling high volumes of emails to routing them efficiently, and ensuring uniform compliance with regulations, the challenges can be overwhelming. Managing different types of outbound emails, whether one-to-one user email, transactional or marketing emails generated from applications, also becomes challenging due to increasing concerns of security and compliance requirements. To help customers tackle these pain points, Amazon Web Services (AWS) has introduced a new feature to streamline inbound and outbound email management: SES Mail Manager.

The challenge: Managing different email flows efficiently with compliance and security in place

Efficiently routing and processing emails to the appropriate teams or systems while ensuring proper filtering, security, and compliance is a complex undertaking. Meanwhile, outbound email flows have become increasing complex. Besides emails being sent between users, more and more emails are generated from different types of applications. On top of that, keeping up security and compliance requirements is an ongoing task for all IT administrators and CISOs. Maintaining email integrations with existing business applications, providing scalability and redundancy to accommodate spikes, and facilitating long-term archiving and retrieval further compound the difficulties. Without a robust and scalable solution, organizations struggle to manage email communications effectively, hindering productivity and exposing themselves to risks.

Solution: Amazon SES Mail Manager

Amazon SES Mail Manager is a comprehensive solution with powerful set of email gateway features that strengthens your organization’s email infrastructure. It simplifies email workflow management and streamlines compliance control, while integrating seamlessly with your existing systems. Mail Manager consolidates all incoming and outgoing email through a single control point. This allows you to apply unified tools, rules, and delivery behaviors across your entire email flow. The centralized approach improves reliability, security, and flexibility.

Some key capabilities include connecting different business applications, automating inbound email processing, managing outgoing emails, enhancing compliance through email archiving, and efficiently controlling overall email traffic. It provides a centralized hub to optimize email infrastructure, simplify processes, ensure compliance, and maintain a high degree of reliability and security.

SES Inbound today

SES MailManager high level architecture

Mail Manager features

Ingress Endpoints: Customizable SMTP endpoints for receiving emails

Ingress endpoint is a key infrastructure component that utilizes filtering polices and rules that you can configure to determine which emails should be allowed into your organization and which ones should be rejected.

Amazon SES currently offers a way to receive incoming emails from the internet using its SMTP interface called SES Inbound. This provided a shared, regional SMTP endpoint that all SES customers could use to accept emails. Improved upon this, Mail Manager introduces a more flexible and powerful approach with different types of ingress endpoints to handling inbound email with Amazon SES.

Mail Manager now offers two options for customers: Open Ingress Endpoint, Authenticated Ingress Endpoint.

Open Ingress Endpoints allows you to create unique, customizable SMTP endpoints that give you control to accept or reject email messages tailored to your specific needs. Open Ingress Points do not require domain verification to receive inbound emails. Simply point your domain’s MX record to the newly created Ingress Endpoint, and it will start receiving emails for that domain.

Authenticated Ingress Endpoints in Mail Manager also enables a new capability – allowing SES to accept emails from trusted external SMTP servers for further processing. Users can create a Traffic Policy to configure trusted external SMTP servers with either type of Ingress Endpoints. What’s different about Authenticated Ingress Endpoints is, users need to use SMTP Authorization to send messages. Once provisioned, you obtain credentials to connect your existing email infrastructure to the Authenticated Ingress Endpoint as an outgoing email server.

For step-by-step guide on how to create this, refer to documentation.

Ingress Endpoints image

Traffic policies & policy statements

Traffic Policies enable fine-grained control over accepting or rejecting inbound email. A traffic policy is a container for policy statements that you assign to an ingress endpoint, so that it can sort the incoming mail by allowing or blocking specific types of email when the conditions of the policy statements are met. You have the option to set a maximum message size so that any email with a size greater, will immediately be discarded—this acts as a “first pass” filter when set. Next, you set either Allow or Deny as the default action that’s taken for email that falls outside of the conditions of your policy statements. This is the “catch all” action for the traffic policy.

Policy statements are also created with either an allow or block action that is taken when the statements’ conditions are met. You build the conditions by selecting an email protocol and a conditional operator for a value you enter that must be matched by the incoming message before the policy statement will allow or block it. The three conditions available in the policy statement are Recipient address, Sender IP range and TLS protocol version. Each policy statement can have multiple conditions.

MailManager Traffic Policy

Rule Sets & rules

Taking Control with Rule Sets:
After traffic policies permit certain messages into Mail Manager, customers use rule sets to apply custom processing logic for routing, optional functions, archiving, and delivery. You can add multiple Rules within a Rule Set. You can specify the order in which Rules within a Rule Set are evaluated, as well as the order of Actions within each Rule. Each Rule consists of:
Conditions: Criteria that an email must match for the Rule to be applied.
Exceptions: Criteria that, if matched, will exempt the email from the Rule.
Actions: The operations to be performed on emails that meet the Rule’s Conditions and don’t match any Exceptions.

Recipient-Oriented Processing:

Mail Manager processes emails in a recipient-oriented manner, meaning rules are applied separately for each recipient. In traditional email gateways, rules are typically applied at the message level, affecting all recipients uniformly. For instance, if a rule in a traditional gateway adds a header for emails addressed to [email protected], all recipients of that email will see the header. This can lead to unintended side effects where actions meant for one recipient affect others. With Mail Manager, only emails to [email protected] will have the header, ensuring rules are specific to each recipient.

Additionally, Mail Manager allows rules to be applied to all recipients when needed, such as using the Subject header as a rule condition. This flexibility provides greater precision and control in email processing, allowing rules administrators to tailor the application of rules to meet specific requirements for individual recipients or for the entire email.

By enabling both recipient-oriented and message-oriented approaches, it enhances privacy, compliance, and security by preventing unintended data exposure and ensuring actions are applied only where intended.

Flexible Conditions and Actions:
Mail Manager’s Rule Sets offer a powerful expression language for defining conditions based on various email properties, such as recipient address, TLS version, source IP, subject header, and more. Additionally, Rule Sets support a wide range of Actions, including:

Writing to Amazon S3 buckets
Sending outbound messages (leveraging SES’s SMTP capabilities)
Relaying emails to external SMTP servers
Archiving emails for long-term storage
Modifying recipient lists
Delivering to AWS WorkMail mailboxes

With these capabilities, Rule Sets enable you to build sophisticated, automated email processing workflows tailored to your organization’s needs.

RuleSets

MailManager Rules

SMTP Relay

Mail Manager’s SMTP Relay functionality allows you to integrate your inbound email processing workflows with external email infrastructure, such as on-premises Microsoft Exchange servers or third-party email gateways. Mail Manager’s SMTP Relay functionality allows you to integrate your email flows with appropriate servers based on predefined criteria, optimizing the journey of every email.

How SMTP Relay Works:

Define an SMTP Relay – First, you create an SMTP Relay resource within Mail Manager, specifying the details of the external SMTP server you want to relay emails to, such as the server hostname, port, and authentication credentials (if required).
Create a Rule with the SMTP Relay Action Next, within a Rule Set, you create a Rule that includes the “SMTP Relay” action, selecting the SMTP Relay resource you defined earlier.
Configure the Rule Conditions You then set the conditions for this Rule, determining which incoming emails should be relayed to the external SMTP server. For example, you could set a condition to relay all email destined for a specific domain (e.g., “@gmail.com”).
Assign the Rule Set to an Ingress Endpoint Finally, you assign the Rule Set containing this Rule to one or more Ingress Endpoints.

When an email matching the Rule’s conditions is received by the Ingress Endpoint, Mail Manager will automatically relay that email to the external SMTP server specified in the SMTP Relay resource.

Use Cases for SMTP Relay:

Processing layer for incoming emails: Relay incoming emails from Mail Manager after rules engine processing to your email server whether it’s on-premises or cloud email system.
Supporting hybrid and migration: In hybrid email environments where some mailboxes are hosted on-premises and others are in the cloud (e.g., Microsoft 365 or Google Workspace), SMTP relay allows for seamless communication between the two environments. During email migration projects, SMTP relay can be used to temporarily route emails between the old and new email platforms, ensuring that no messages are lost during the transition period.
Mailbox resilience: By terminating MX at Mail Manager, and then configuring rules for delivery to 1 or more mailbox providers, you can manage resilient mailbox delivery if your primary mailbox provider is impaired. No DNS propagation delays, just change the delivery rule and instantly fall into your other system.
Enforcement layer: Integrate Mail Manager with third-party email services or gateways by relaying emails to their SMTP endpoints, leveraging their capabilities to enforce additional policies or security measures while maintaining control with Mail Manager.
Inter-Server Communication: SMTP relay facilitates communication between different internal email servers or systems within the organization’s network, ensuring seamless delivery of emails across various domains or platforms.
Load balancing and redundancy: Distribute email traffic across multiple servers or gateways to optimize performance and resource utilization, ensuring high availability and fault tolerance.

With SMTP Relay, Mail Manager acts as a flexible email processing layer, allowing you to incorporate its powerful capabilities while maintaining and extending your current email infrastructure investments.

SMTP Relay screen

Email Archiving

As organizations face increasing regulatory and compliance requirements around email retention, Mail Manager provides a robust email archiving solution. The archiving feature allows you to securely store and easily search through your email data, ensuring you meet your archiving obligations.

How Email Archiving Works:

You create an archive resource within Mail Manager, specifying the desired retention period for your archived emails.
Create a Rule with the archive action within a Rule Set. Create a Rule that includes the “Archive” action, selecting the Archive resource you defined earlier.
You then set the conditions for this Rule, determining which incoming emails should be archived. For example, you could archive all emails sent to a specific department’s email alias.
Finally, you assign the Rule Set containing this archiving Rule to one or more Ingress Endpoints.

Now, when an email matching the Rule’s conditions is received by the Ingress Endpoint, Mail Manager will automatically archive a verbatim copy of that email to the designated Archive resource.

Mail Manager’s archiving capabilities offer several advantages for organizations:

Archiving stores email data in a secure, durable, and searchable archive, meeting regulatory requirements for email retention and enabling efficient audits.
Utilize powerful search filters to locate specific emails within your archive, and export search results for further analysis or legal purposes.
Reduce the storage burden on your mail servers by archiving emails to Mail Manager’s scalable and cost-effective storage solution.
Set customizable retention periods for your archives, ensuring important email data is preserved for as long as needed.

By integrating email archiving into your Mail Manager workflows, you can maintain a comprehensive, searchable, and compliant email archive without the hassle of managing additional infrastructure.

Email Archiving

Email Add-ons

Mail Manager offers a suite of specialized security tools, called Email Add-ons, that allow you to enhance your email security posture and tailor your inbound email workflows to your specific needs. Add-ons can be used as conditions within Traffic Policies to control which emails are allowed into your Ingress Endpoints, or as conditions within Rule Sets to determine the actions taken on specific email types. These Add-ons are certified security intelligence and enforcement solutions from vetted providers, ready to be integrated directly into your Mail Manager environment (e.g., Spamhaus Domain Block List, Abusix Mail Intelligence, Trend Micro Virus Scanning).

Email Add-Ons provide a flexible and modular approach to email security, enabling you to select and combine the solutions that best fit your unique use cases. Instead of investing in a monolithic product that may not fully align with your requirements, you can choose from a range of Add-ons and pay only for the capabilities you need, on a metered-price basis. Once you’ve subscribed to an Email Add-on from the Mail Manager console, you can seamlessly incorporate it into your email workflows.

Email Add-ons extend Mail Manager’s core threat intelligence and security enforcement features on a per-workload basis, ensuring you have the right level of protection without over-provisioning resources. Within the Mail Manager console, you can explore detailed product descriptions, key benefits, and pricing information for each Add-on, empowering you to make informed decisions.

Key benefits of Add-ons:

Immediate use: no separate setup/integration work required.
Cost effective: pay for only what is needed and consumed, turn on and off as required
Granular deployment via individual traffic policy or rule action

Conclusion:

Amazon SES Mail Manager introduces advanced email routing and archiving features, providing significant benefits to customers. With customizable SMTP endpoints and recipient-oriented rule processing, customers gain precise control over email traffic, ensuring that rules are applied specifically to each recipient. The enhanced traffic policies improve email security and compliance, while the robust SMTP relay functionality seamlessly integrates with existing systems, ensuring efficient email routing and processing. Mail Manager’s archiving capabilities help meet regulatory requirements and simplify data management. Overall, Mail Manager streamlines email operations, optimizes infrastructure, and enhances reliability, security, and compliance, offering a powerful solution for managing complex email workflows.

About the Authors:

Jessica Fan is a Senior Product Manager at AWS, striving to improve the experience for Amazon SES customers. Outside of work, she enjoys long distance running, biking and bouldering.

Vinay Ujjini is an Amazon Pinpoint and Amazon Simple Email Service Worldwide Principal Specialist Solutions Architect at AWS. He has been solving customer’s omni-channel challenges for over 15 years. He is an avid sports enthusiast and in his spare time, enjoys playing tennis & cricket.

Upgrade Your Email Tech Stack with Amazon SESv2 API

2024-03-15 Zip Zieper

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/upgrade-your-email-tech-stack-with-amazon-sesv2-api/

Amazon Simple Email Service (SES) is a cloud-based email sending service that helps businesses and developers send marketing and transactional emails. We introduced the SESv1 API in 2011 to provide developers with basic email sending capabilities through Amazon SES using HTTPS. In 2020, we introduced the redesigned Amazon SESv2 API, with new and updated features that make it easier and more efficient for developers to send email at scale.

This post will compare Amazon SESv1 API and Amazon SESv2 API and explain the advantages of transitioning your application code to the SESv2 API. We’ll also provide examples using the AWS Command-Line Interface (AWS CLI) that show the benefits of transitioning to the SESv2 API.

Amazon SESv1 API

The SESv1 API is a relatively simple API that provides basic functionality for sending and receiving emails. For over a decade, thousands of SES customers have used the SESv1 API to send billions of emails. Our customers’ developers routinely use the SESv1 APIs to verify email addresses, create rules, send emails, and customize bounce and complaint notifications. Our customers’ needs have become more advanced as the global email ecosystem has developed and matured. Unsurprisingly, we’ve received customer feedback requesting enhancements and new functionality within SES. To better support an expanding array of use cases and stay at the forefront of innovation, we developed the SESv2 APIs.

While the SESv1 API will continue to be supported, AWS is focused on advancing functionality through the SESv2 API. As new email sending capabilities are introduced, they will only be available through SESv2 API. Migrating to the SESv2 API provides customers with access to these, and future, optimizations and enhancements. Therefore, we encourage SES customers to consider the information in this blog, review their existing codebase, and migrate to SESv2 API in a timely manner.

Amazon SESv2 API

Released in 2020, the SESv2 API and SDK enable customers to build highly scalable and customized email applications with an expanded set of lightweight and easy to use API actions. Leveraging insights from current SES customers, the SESv2 API includes several new actions related to list and subscription management, the creation and management of dedicated IP pools, and updates to unsubscribe that address recent industry requirements.

One example of new functionality in SESv2 API is programmatic support for the SES Virtual Delivery Manager. Previously only addressable via the AWS console, VDM helps customers improve sending reputation and deliverability. SESv2 API includes vdmAttributes such as VdmEnabled and DashboardAttributes as well as vdmOptions. DashboardOptions and GaurdianOptions.

To improve developer efficiency and make the SESv2 API easier to use, we merged several SESv1 APIs into single commands. For example, in the SESv1 API you must make separate calls for createConfigurationSet, setReputationMetrics, setSendingEnabled, setTrackingOptions, and setDeliveryOption. In the SESv2 API, however, developers make a single call to createConfigurationSet and they can include trackingOptions, reputationOptions, sendingOptions, deliveryOptions. This can result in more concise code (see below).

SESv1-vs-SESv2

Another example of SESv2 API command consolidation is the GetIdentity action, which is a composite of SESv1 API’s GetIdentityVerificationAttributes, GetIdentityNotificationAttributes, GetCustomMailFromAttributes, GetDKIMAttributes, and GetIdentityPolicies. See SESv2 documentation for more details.

Why migrate to Amazon SESv2 API?

The SESv2 API offers an enhanced experience compared to the original SESv1 API. Compared to the SESv1 API, the SESv2 API provides a more modern interface and flexible options that make building scalable, high-volume email applications easier and more efficient. SESv2 enables rich email capabilities like template management, list subscription handling, and deliverability reporting. It provides developers with a more powerful and customizable set of tools with improved security measures to build and optimize inbox placement and reputation management. Taken as a whole, the SESv2 APIs provide an even stronger foundation for sending critical communications and campaign email messages effectively at a scale.

Migrating your applications to SESv2 API will benefit your email marketing and communication capabilities with:

New and Enhanced Features: Amazon SESv2 API includes new actions as well as enhancements that provide better functionality and improved email management. By moving to the latest version, you’ll be able to optimize your email sending process. A few examples include:
- Increase the maximum message size (including attachments) from 10Mb (SESv1) to 40Mb (SESv2) for both sending and receiving.
- Access key actions for the SES Virtual Deliverability Manager (VDM) which provides insights into your sending and delivery data. VDM provides near-realtime advice on how to fix the issues that are negatively affecting your delivery success rate and reputation.
- Meet Google & Yahoo’s June 2024 unsubscribe requirements with the SES v2 SendEmail action. For more information, see the “What’s New blog”
Future-proof Your Application: Avoid potential compatibility issues and disruptions by keeping your application up-to-date with the latest version of the Amazon SESv2 API via the AWS SDK.
Improve Usability and Developer Experience: Amazon SESv2 API is designed to be more user-friendly and consistent with other AWS services. It is a more intuitive API with better error handling, making it easier to develop, maintain, and troubleshoot your email sending applications.

Migrating to the latest SESv2 API and SDK positions customers for success in creating reliable and scalable email services for their businesses.

What does migration to the SESv2 API entail?

While SESv2 API builds on the v1 API, the v2 API actions don’t universally map exactly to the v1 API actions. Current SES customers that intend to migrate to SESv2 API will need to identify the SESv1 API actions in their code and plan to refactor for v2. When planning the migration, it is essential to consider several important considerations:

Customers with applications that receive email using SESv1 API’s CreateReceiptFilter, CreateReceiptRule or CreateReceiptRuleSet actions must continue using the SESv1 API client for these actions. SESv1 and SESv2 can be used in the same application, where needed.
We recommend all customers follow the security best practice of “least privilege” with their IAM policies. As such, customers may need to review and update their policies to include the new and modified API actions introduced in SESv2 before migrating. Taking the time to properly configure permissions ensures a seamless transition while maintaining a securely optimized level of access. See documentation.

Below is an example of an IAM policy with a user with limited allow privileges related to several SESv1 Identity actions only:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ses:VerifyEmailIdentity",
                "ses:Deleteldentity",
                "ses:VerifyDomainDkim",
                "ses:ListIdentities",
                "ses:VerifyDomainIdentity"
            ],
            "Resource": "*"
        }
    ]
}

When updating to SESv2, you need to update this user’s permissions with the SESv2 actions shown below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ses:CreateEmailIdentity",
                "ses:DeleteEmailIdentity",
                "ses:GetEmailIdentity",
                "ses:ListEmailIdentities"
            ],
            "Resource": "*"
        }
    ]
}

Examples of SESv1 vs. SESv2 APIs

Let’s look at a three examples that compare the SESv1 API with the SESv2 API.

LIST APIs

When listing identities in SESv1 list API, you need to specify type which requires multiple calls to API to list all resources:

aws ses list-identities --identity-type Domain
{
    "Identities": [
        "example.com"
    ]
}

aws ses list-identities --identity-type EmailAddress
{
    "Identities": [
        "[email protected]",
        "[email protected]",
        "[email protected]"
    ]
}

With SESv2, you can simply call a single API. Additionally, SESv2 also provides extended feedback:

aws sesv2 list-email-identities
{
    "EmailIdentities": [
        {
            "IdentityType": "DOMAIN",
            "IdentityName": "example.com",
            "SendingEnabled": false,
            "VerificationStatus": "FAILED"
        },
        {
            "IdentityType": "EMAIL_ADDRESS",
            "IdentityName": "[email protected]",
            "SendingEnabled": true,
            "VerificationStatus": "SUCCESS"
        },
        {
            "IdentityType": "EMAIL_ADDRESS",
            "IdentityName": "[email protected]",
            "SendingEnabled": false,
            "VerificationStatus": "FAILED"
        },
        {
            "IdentityType": "EMAIL_ADDRESS",
            "IdentityName": "[email protected]",
            "SendingEnabled": true,
            "VerificationStatus": "SUCCESS"
        }
    ]
}

CREATE APIs

With SESv1, creating email addresses or domains requires calling two different APIs:

aws ses verify-email-identity --email-address [email protected]

aws ses verify-domain-dkim --domain example.com
{
    "DkimTokens": [
        "mwmzhwhcebfh5kvwv7zahdatahimucqi",
        "dmlozjwrdbrjfwothoh26x6izvyts7qx",
        "le5fy6pintdkbxg6gdoetgbrdvyp664v"
    ]
}

With SESv2, we build an abstraction so you can call a single API. Additionally, SESv2 provides more detailed responses and feedback:

aws sesv2 create-email-identity --email-identity [email protected]
{
    "IdentityType": "EMAIL_ADDRESS",
    "VerifiedForSendingStatus": false
}

aws sesv2 create-email-identity --email-identity example.com
{
    "IdentityType": "DOMAIN",
    "VerifiedForSendingStatus": false,
    "DkimAttributes": {
        "SigningEnabled": true,
        "Status": "NOT_STARTED",
        "Tokens": [
            "mwmzhwhcebfh5kvwv7zahdatahimucqi",
            "dmlozjwrdbrjfwothoh26x6izvyts7qx",
            "le5fy6pintdkbxg6gdoetgbrdvyp664v"
        ],
        "SigningAttributesOrigin": "AWS_SES",
        "NextSigningKeyLength": "RSA_2048_BIT",
        "CurrentSigningKeyLength": "RSA_2048_BIT",
        "LastKeyGenerationTimestamp": "2024-02-23T15:01:53.849000+00:00"
    }
}

DELETE APIs

When calling delete- with SESv1, SES returns 200 (or no response), even if the identity was previously deleted or doesn’t exist:

 aws ses delete-identity --identity example.com

SESv2 provides better error handling and responses when calling the delete API:

aws sesv2 delete-email-identity --email-identity example.com

An error occurred (NotFoundException) when calling the DeleteEmailIdentity operation: Email identity example.com does not exist.

Hands-on with SESv1 API vs. SESv2 API

Below are a few examples you can use to explore the differences between SESv1 API and the SESv2 API. To complete these exercises, you’ll need:

AWS Account (setup) with enough permission to interact with the SES service via the CLI
Upgrade to the latest version of the AWS CLI (aws-cli/2.15.27 or greater)
SES enabled, configured and properly sending emails
A recipient email address with which you can check inbound messages (if you’re in the SES Sandbox, this email must be verified email identity). In the following examples, replace [email protected] with the verified email identity.
Your preferred IDE with AWS credentials and necessary permissions (you can also use AWS CloudShell)

Open the AWS CLI (or AWS CloudShell) and:

Create a test directory called v1-v2-test.
Create the following (8) files in the v1-v2-test directory:

destination.json (replace [email protected] with the verified email identity):

{ 
    "ToAddresses": ["[email protected]"] 
}

ses-v1-message.json

{
   "Subject": {
       "Data": "SESv1 API email sent using the AWS CLI",
       "Charset": "UTF-8"
   },
   "Body": {
       "Text": {
           "Data": "This is the message body from SESv1 API in text format.",
           "Charset": "UTF-8"
       },
       "Html": {
           "Data": "This message body from SESv1 API, it contains HTML formatting. For example - you can include links: <a class=\"ulink\" href=\"http://docs.aws.amazon.com/ses/latest/DeveloperGuide\" target=\"_blank\">Amazon SES Developer Guide</a>.",
           "Charset": "UTF-8"
       }
   }
}

ses-v1-raw-message.json (replace [email protected] with the verified email identity):

{
     "Data": "From: [email protected]\nTo: [email protected]\nSubject: Test email sent using the SESv1 API and the AWS CLI \nMIME-Version: 1.0\nContent-Type: text/plain\n\nThis is the message body from the SESv1 API SendRawEmail.\n\n"
}

ses-v1-template.json (replace [email protected] with the verified email identity):

{
  "Source":"SES Developer<[email protected]>",
  "Template": "my-template",
  "Destination": {
    "ToAddresses": [ "[email protected]"
    ]
  },
  "TemplateData": "{ \"name\":\"SESv1 Developer\", \"favoriteanimal\": \"alligator\" }"
}

my-template.json (replace [email protected] with the verified email identity):

{
  "Template": {
    "TemplateName": "my-template",
    "SubjectPart": "Greetings SES Developer, {{name}}!",
    "HtmlPart": "<h1>Hello {{name}},</h1><p>Your favorite animal is {{favoriteanimal}}.</p>",
    "TextPart": "Dear {{name}},\r\nYour favorite animal is {{favoriteanimal}}."
  }
}

ses-v2-simple.json (replace [email protected] with the verified email identity):

{
    "FromEmailAddress": "[email protected]",
    "Destination": {
        "ToAddresses": [
            "[email protected]"
        ]
    },
    "Content": {
        "Simple": {
            "Subject": {
                "Data": "SESv2 API email sent using the AWS CLI",
                "Charset": "utf-8"
            },
            "Body": {
                "Text": {
                    "Data": "SESv2 API email sent using the AWS CLI",
                    "Charset": "utf-8"
                }
            },
            "Headers": [
                {
                    "Name": "List-Unsubscribe",
                    "Value": "insert-list-unsubscribe-here"
                },
				{
                    "Name": "List-Unsubscribe-Post",
                    "Value": "List-Unsubscribe=One-Click"
                }
            ]
        }
    }
}

ses-v2-raw.json (replace [email protected] with the verified email identity):

{
     "FromEmailAddress": "[email protected]",
     "Destination": {
            "ToAddresses": [
                       "[email protected]"
              ]
       },
      "Content": {
             "Raw": {
                     "Data": "Subject: Test email sent using SESv2 API via the AWS CLI \nMIME-Version: 1.0\nContent-Type: text/plain\n\nThis is the message body from SendEmail Raw Content SESv2.\n\n"
              }
      }
}

ses-v2-tempate.json (replace [email protected] with the verified email identity):

{
     "FromEmailAddress": "[email protected]",
     "Destination": {
       "ToAddresses": [
         "[email protected]"
       ]
     },
     "Content": {
        "Template": {
          "TemplateName": "my-template",
          "TemplateData": "{ \"name\":\"SESv2 Developer\",\"favoriteanimal\":\"Dog\" }",
          "Headers": [
                {
                   "Name": "List-Unsubscribe",
                   "Value": "insert-list-unsubscribe-here"
                },
                {
                   "Name": "List-Unsubscribe-Post",
                   "Value": "List-Unsubscribe=One-Click"
                }
             ]
         }
     }
}

Perform the following commands using the SESv1 API:

send-email (simple):

In the CLI, (replace [email protected] with the verified email identity) run:

aws ses send-email --from [email protected] --destination file://destination.json --message file://ses-v1-message.json

The response will return a valid MessageID (signaling the action was successful). An email will be received by the verified email identity.

{
    "MessageId": "0100018dc7649400-Xx1x0000x-bcec-483a-b97c-123a4567890d-xxxxx"
}

send-raw-email:

In the CLI, run:

aws ses send-raw-email  --cli-binary-format raw-in-base64-out --raw-message file://ses-v1-raw-message.json

The response will return a valid MessageID (signaling the action was successful). An email will be received by the verified email identity.

{
   "MessageId": "0200018dc7649400-Xx1x1234x-bcec-483a-b97c-123a4567890d-
}

send templated mail:

In the CLI, run the following to create the template:

aws ses create-template  --cli-input-json file://my-template.json

In the CLI, run:

aws ses send-templated-email --cli-input-json file://ses-v1-template.json

The response will return a valid MessageID (signaling the action was successful). An email will be received by the verified email identity.

 {
    "MessageId": "0000018dc7649400-Xx1x1234x-bcec-483a-b97c-123a4567890d-xxxxx"
 }

Perform similar commands using the SESv2 API:

As mentioned above, customers who are using least privilege permissions with SESv1 API must first update their IAM policies before running the SESv2 API examples below. See documentation for more info.

As you can see from the .json files we created for SES v2 API (above), you can modify or remove sections from the .json files, based on the type of email content (simple, raw or templated) you want to send.

Please ensure you are using the latest version of the AWS CLI (aws-cli/2.15.27 or greater).

Send simple email

In the CLI, run:

aws sesv2 send-email --cli-input-json file://ses-v2-simple.json

The response will return a valid MessageID (signaling the action was successful). An email will be received by the verified email identity

{
    "MessageId": "0100018dc83ba7e0-7b3149d7-3616-49c2-92b6-00e7d574f567-000000"
}

Send raw email (note – if the only reason is to set custom headers, you don’t need to send raw email)

In the CLI, run:

aws sesv2 send-email --cli-binary-format raw-in-base64-out --cli-input-json file://ses-v2-raw.json

The response will return a valid MessageID (signaling the action was successful). An email will be received by the verified email identity.

{
    "MessageId": "0100018dc877bde5-fdff0df3-838e-4f51-8582-a05237daecc7-000000"
}

Send templated email

In the CLI, run:

aws sesv2 send-email --cli-input-json file://ses-v2-tempate.json

The response will return a valid MessageID (signaling the action was successful). An email will be received by the verified email identity.

{
    "MessageId": "0100018dc87fe72c-f2c547a1-2325-4be4-bf78-b91d6648cd12-000000"
}

Migrating your application code to SESv2 API

As you can see from the examples above, SESv2 API shares much of its syntax and actions with the SESv1 API. As a result, most customers have found they can readily evaluate, identify and migrate their application code base in a relatively short period of time. However, it’s important to note that while the process is generally straightforward, there may be some nuances and differences to consider depending on your specific use case and programming language.

Regardless of the language, you’ll need anywhere from a few hours to a few weeks to:

Update your code to use SESv2 Client and change API signature and request parameters
Update permissions / policies to reflect SESv2 API requirements
Test your migrated code to ensure that it functions correctly with the SESv2 API
Stage, test
Deploy

Summary

As we’ve described in this post, Amazon SES customers that migrate to the SESv2 API will benefit from updated capabilities, a more user-friendly and intuitive API, better error handling and improved deliverability controls. The SESv2 API also provide for compliance with the industry’s upcoming unsubscribe header requirements, more flexible subscription-list management, and support for larger attachments. Taken collectively, these improvements make it even easier for customers to develop, maintain, and troubleshoot their email sending applications with Amazon Simple Email Service. For these, and future reasons, we recommend SES customers migrate their existing applications to the SESv2 API immediately.

For more information regarding the SESv2 APIs, comment on this post, reach out to your AWS account team, or consult the AWS SESv2 API documentation:

About the Authors

Zip

Zip is an Amazon Pinpoint and Amazon Simple Email Service Sr. Specialist Solutions Architect at AWS. Outside of work he enjoys time with his family, cooking, mountain biking and plogging.

Vinay Ujjini

Vinay is an Amazon Pinpoint and Amazon Simple Email Service Worldwide Principal Specialist Solutions Architect at AWS. He has been solving customer’s omni-channel challenges for over 15 years. He is an avid sports enthusiast and in his spare time, enjoys playing tennis and cricket.

Dmitrijs Lobanovskis

Dmitrijs is a Software Engineer for Amazon Simple Email service. When not working, he enjoys traveling, hiking and going to the gym.

Amazon SES: Email Authentication and Getting Value out of Your DMARC Policy

2023-10-03 Bruno Giorgini

Post Syndicated from Bruno Giorgini original https://aws.amazon.com/blogs/messaging-and-targeting/email-authenctication-dmarc-policy/

Amazon SES: Email Authentication and Getting Value out of Your DMARC Policy

Introduction

For enterprises of all sizes, email is a critical piece of infrastructure that supports large volumes of communication. To enhance the security and trustworthiness of email communication, many organizations turn to email sending providers (ESPs) like Amazon Simple Email Service (Amazon SES). These ESPs allow users to send authenticated emails from their domains, employing industry-standard protocols such as the Sender Policy Framework (SPF) and DomainKeys Identified Mail (DKIM). Messages authenticated with SPF or DKIM will successfully pass your domain’s Domain-based Message Authentication, Reporting, and Conformance (DMARC) policy. This blog post will focus on the DMARC policy enforcement mechanism. The blog will explore some of the reasons why email may fail DMARC policy evaluation and propose solutions to fix any failures that you identify. For an introduction to DMARC and how to carefully choose your email sending domain identity, you can refer to Choosing the Right Domain for Optimal Deliverability with Amazon SES The relationship between DMARC compliance and email deliverability rates is crucial for organizations aiming to maintain a positive sender reputation and ensure successful email delivery. There are many advantages when organizations have this correctly setup, these include:

Improved Email Deliverability
Reduction in Email Spoofing and Phishing
Positive Sender Reputation
Reduced Risk of Email Marked as Spam
Better Email Engagement Metrics
Enhanced Brand Reputation

With this foundation, let’s explore the intricacies of DMARC and how it can benefit your organization’s email communication.

What is DMARC?

DMARC is a mechanism for domain owners to advertise SPF and DKIM protection and to tell receivers how to act if those authentication methods fail. The domain’s DMARC policy protects your domain from third parties attempting to spoof the domain in the “From” header of emails. Malicious email messages that aim to send phishing attempts using your domain will be subject to DMARC policy evaluation, which may result in their quarantine or rejection by the email receiving organization. This stringent policy ensures that emails received by email recipients are genuinely from the claimed sending domain, thereby minimizing the risk of people falling victim to email-based scams. Domain owners publish DMARC policies as a TXT record in the domain’s _dmarc.<domain> DNS record. For example, if the domain used in the “From” header is example.com, then the domain’s DMARC policy would be located in a DNS TXT record named _dmarc.example.com. The DMARC policy can have one of three policy modes:

A typical DMARC deployment of an existing domain will start with publishing "p=none". A none policy means that the domain owner is in a monitoring phase; the domain owner is monitoring for messages that aren’t authenticated with SPF and DKIM and seeks to ensure all email is properly authenticated
When the domain owner is comfortable that all legitimate use cases are properly authenticated with SPF and/or DKIM, they may change the DMARC policy to "p=quarantine". A quarantine policy means that messages which fail to produce a domain-aligned authenticated identifier via SPF or DKIM will be quarantined by the mail receiving organization. The mail receiving organization may filter these messages into Junk folders, or take another action that they feel best protects their recipients.
Finally, domain owners who are confident that all of the legitimate messages using their domain are authenticated with SPF or DKIM, may change the DMARC policy to "p=reject". A reject policy means that messages which fail to produce a domain-aligned authenticated identifier via SPF or DKIM will be rejected by the mail receiving organization.

The following are examples of a TXT record that contains a DMARC policy, depending on the desired policy (the ‘p’ tag):

	Name	Type	Value
1	_dmarc.example.com	TXT	“v=DMARC1;p=reject;rua=mailto:[email protected]”
2	_dmarc.example.com	TXT	“v=DMARC1;p=quarantine;rua=mailto:[email protected]”
3	_dmarc.example.com	TXT	“v=DMARC1;p=none;rua=mailto:[email protected]”

Table 1 – Example DMARC policy

This policy tells email providers to apply the DMARC policy to messages that fail to produce a DKIM or SPF authenticated identifier that is aligned to the domain in the “From” header. Alignment means that one or both of the following occurs:

The messages pass the SPF policy for the MAIL FROM domain and the MAIL FROM domain is the same as the domain in the “From” header, or a subdomain. Reference Using a custom MAIL FROM domain to learn more about how to send SPF aligned messages with SES.
The messages have a DKIM signature signed by a public key in DNS at a location within the domain of the “From” header. Reference Authenticating Email with DKIM in Amazon SES to learn more about how to send DKIM aligned messages with SES.

DMARC reporting

The rua tag in the domain’s DMARC policy indicates the location to which mail receiving organizations should send aggregate reports about messages that pass or fail SPF and DKIM alignment. Domain owners analyze these reports to discover messages which are using the domain in the “From” header but are not properly authenticated with SPF or DKIM. The domain owner will attempt to ensure that all legitimate messages are authenticated through analysis of the DMARC aggregate reports over time. Mail receiving organizations which support sending DMARC reports typically send these aggregated reports once per day, although these practices differ from provider to provider.

What does a typical DMARC deployment look like?

A DMARC deployment is the process of:

Ensuring that all emails using the domain in the “From” header are authenticated with DKIM and SPF domain-aligned identifiers. Focus on DKIM as the primary means of authentication.
Publishing a DMARC policy (none, quarantine, or reject) for the domain that reflects how the domain owner would like mail receiving organizations to handle unauthenticated email claiming to be from their domain.

New domains and subdomains

Deploying a DMARC policy is easy for organizations that have created a new domain or subdomain for the purpose of a new email sending use case on SES; for example email marketing, transaction emails, or one-time pass codes (OTP). These domains can start with the "p=reject" DMARC enforcement policy because the policy will not affect existing email sending programs. This strict enforcement is to ensure that there is no unauthenticated use of the domain and its subdomains.

Existing domains

For existing domains, a DMARC deployment is an iterative process because the domain may have a history of email sending by one or multiple email sending programs. It is important to gain a complete understanding of how the domain and its subdomains are being used for email sending before publishing a restrictive DMARC policy (p=quarantine or p=reject) because doing so would affect any unauthenticated email sending programs using the domain in the “From” header of messages. To get started with the DMARC implementation, these are a few actions to take:

Publish a p=none DMARC policy (sometimes referred to as monitoring mode), and set the rua tag to the location in which you would like to receive aggregate reports.
Analyze the aggregate reports. Mail receiving organizations will send reports which contain information to determine if the domain, and its subdomains, are being used for sending email, and how the messages are (or are not) being authenticated with a DKIM or SPF domain-aligned identifier. An easy to use analysis tool is the Dmarcian XML to Human Converter.
Avoid prematurely publishing a “p=quarantine” or “p=reject” policy. Doing so may result in blocked or reduced delivery of legitimate messages of existing email sending programs.

The image below illustrates how DMARC will be applied to an email received by the email receiving server and actions taken based on the enforcement policy:

Figure 1 – DMARC Flow

How do SPF and DKIM cause DMARC policies to pass

When you start sending emails using Amazon SES, messages that you send through Amazon SES automatically use a subdomain of amazonses.com as the default MAIL FROM domain. SPF evaluators will see that these messages pass the SPF policy evaluation because the default MAIL FROM domain has a SPF policy which includes the IP addresses of the SES infrastructure that sent the message. SPF authentication will result in an “SPF=PASS” and the authenticated identifier is the domain of the MAIL FROM address. The published SPF record applies to every message that is sent using SES regardless of whether you are using a shared or dedicated IP address. The amazonses.com SPF record lists all shared and dedicated IP addresses, so it is inclusive of all potential IP addresses that may be involved with sending email as the MAIL FROM domain. You can use ‘dig’ to look up the IP addresses that SES will use to send email:

dig txt amazonses.com | grep "v=spf1" amazonses.com. 850 IN TXT "v=spf1 ip4:199.255.192.0/22 ip4:199.127.232.0/22 ip4:54.240.0.0/18 ip4:69.169.224.0/20 ip4:23.249.208.0/20 ip4:23.251.224.0/19 ip4:76.223.176.0/20 ip4:54.240.64.0/19 ip4:54.240.96.0/19 ip4:52.82.172.0/22 ip4:76.223.128.0/19 -all"

Custom MAIL FROM domains

It is best practice for customers to configure a custom MAIL FROM domain, and not use the default amazonses.com MAIL FROM domain. The custom MAIL FROM domain will always be a subdomain of the customer’s verified domain identity. Once you configure the MAIL FROM domain, messages sent using SES will continue to result in an “SPF=PASS” as it does with the default MAIL FROM domain. Additionally, DMARC authentication will result in “DMARC=PASS” because the MAIL FROM domain and the domain in the “From” header are in alignment. It’s important to understand that customers must use a custom MAIL FROM domain if they want “SPF=PASS” to result in a “DMARC=PASS”.

For example, an Amazon SES-verified example.com domain will have the custom MAIL FROM domain “bounce.example.com”. The configured SPF record will be:

dig txt bounce.example.com | grep "v=spf1" "v=spf1 include:amazonses.com ~all"

Note: The chosen MAIL FROM domain could be any sub-domain of your choice. If you have the same domain identity configured in multiple regions, then you should create region-specific custom MAIL FROM domains for each region. e.g. bounce-us-east-1.example.com and bounce-eu-west-2.example.com so that asynchronously bounced messages are delivered directly to the region from which the messages were sent.

DKIM results in DMARC pass

For customers that establish Amazon SES Domain verification using DKIM signatures, DKIM authentication will result in a DKIM=PASS, and DMARC authentication will result in “DMARC=PASS” because the domain that publishes the DKIM signature is aligned to the domain in the “From” header (the SES domain identity).

DKIM and SPF together

Email messages are fully authenticated when the messages pass both DKIM and SPF, and both DKIM and SPF authenticated identifiers are domain-aligned. If only DKIM is domain-aligned, then the messages will still pass the DMARC policy, even if the SPF “pass” is unaligned. Mail receivers will consider the full context of SPF and DKIM when determining how they will handle the disposition of the messages you send, so it is best to fully authenticate your messages whenever possible. Amazon SES has taken care of the heavy lifting of the email authentication process away from our customers, and so, establishing SPF, DKIM and DMARC authentication has been reduced to a few clicks which allows SES customers to get started easily and scale fast.

Why is DMARC failing?

There are scenarios when you may notice that messages fail DMARC, whether your messages are fully authenticated, or partially authenticated. The following are things that you should look out for:

Email Content Modification

Sometimes email content is modified during the delivery to the recipients’ mail servers. This modification could be as a result of a security device or anti-spam agent along the delivery path (for example: the message Subject may be modified with an “[EXTERNAL]” warning to recipients). The modified message invalidates the DKIM signature which causes a DKIM failure. Remember, the purpose of DKIM is to ensure that the content of an email has not been tampered with during the delivery process. If this happens, the DKIM authentication will fail with an authentication error similar to “DKIM-signature body hash not verified“.

Solutions:

If you control the full path that the email message will traverse from sender to recipient, ensure that no intermediary mail servers modify the email content in transit.
Ensure that you configure a custom MAIL FROM domain so that the messages have a domain-aligned SPF identifier.
Keep the DMARC policy in monitoring mode (p=none) until these issues are identified/solved.

Email Forwarding

Email Forwarding There are multiple scenarios in which a message may be forwarded, and they may result in both/either SPF and DKIM failing to produce a domain-aligned authenticated identifier. For SPF, it means that the forwarding mail server is not listed in the MAIL FROM domain’s SPF policy. It is best practice for a forwarding mail server to avoid SPF failures and assume responsibility of mail handling for the messages it forwards by rewriting the MAIL FROM address to be in the domain controlled by the forwarding server. Forwarding servers that do not rewrite the MAIL FROM address pose a risk of impersonation attacks and phishing. Do not add the IP addresses of forwarding servers to your MAIL FROM domain’s SPF policy unless you are in complete control of all sources of mail being forwarded through this infrastructure. For DKIM, it means that the messages are being modified in some way that causes DKIM signature validation failure (see Email Content Modification section above). A responsible forwarding server will rewrite the MAIL FROM domain so that the messages pass SPF with a non-aligned authenticated identifier. These servers will attempt to forward the message without alteration in order to preserve DKIM signatures, but that is sometimes challenging to do in practice. In this scenario, since the messages carry no domain-aligned authenticated identifier, the messages will fail the DMARC policy.

Solution:

Email forwarding is an expected type of failure of which you will see in the DMARC aggregate reports. The domain owner must weigh the risk of causing forwarded messages to be rejected against the risk of not publishing a reject DMARC policy. Reference 8.6. Interoperability Considerations. Forwarding servers that wish to forward messages that they know will result in a DMARC failure will commonly rewrite the “From” header address of messages it forwards so that the messages pass a DMARC policy for a domain that the forwarding server is responsible for. The way to identify forwarding servers that rewrite the “From” header in this situation is to publish “p=quarantine pct=0 t=y” in your domain’s DMARC policy before publishing “p=reject”.

Multiple email sending providers are sending using the same domain

Multiple email sending providers: There are situations where an organization will have multiple business units sending email using the same domain, and these business units may be using an email sending provider other than SES. If neither SPF nor DKIM is configured with domain-alignment for these email sending providers, you will see DMARC failures in the DMARC aggregate report.

Solution:

Analyze the DMARC aggregate reports to identify other email sending providers, track down the business units responsible for each email sending program, and follow the instructions offered by the email sending provider about how to configure SPF and DKIM to produce a domain-aligned authenticated identifier.

What does a DMARC aggregate report look like?

The following XML example shows the general format of a DMARC aggregate report that you will receive from participating email service providers.

<?xml version="1.0" encoding="UTF-8" ?> 
<feedback> 
  <report_metadata> 
    <org_name>email-service-provider-domain.com</org_name> 
    <email>[email protected]</email> 
    <extra_contact_info>https://email-service-provider-domain.com/> 
    <report_id>620501112281841510</report_id> 
    <date_range> 
      <begin>1685404800</begin> 
      <end>1685491199</end> 
    </date_range> 
  </report_metadata> 
  <policy_published> 
    <domain>example.com</domain>
    <adkim>r</adkim> 
    <aspf>r</aspf> 
    <p>none</p> 
    <sp>none</sp> 
    <pct>100</pct> 
  </policy_published> 
  <record> 
    <row> 
      <source_ip>192.0.2.10</source_ip>
      <count>1</count> 
      <policy_evaluated> 
        <disposition>none</disposition> 
        <dkim>pass</dkim> 
        <spf>fail</spf> 
      </policy_evaluated> 
    </row> 
    <identifiers> 
      <header_from>example.com</header_from>
    </identifiers> 
    <auth_results> 
      <dkim> 
        <domain>example.com</domain> 
        <result>pass</result> 
        <selector>gm5h7da67oqhnr3ccji35fdskt</selector> 
      </dkim> 
      <dkim> 
        <domain>amazonses.com</domain> 
        <result>pass</result> 
        <selector>224i4yxa5dv7c2xz3womw6peua</selector> 
      </dkim> 
      <spf> 
        <domain>amazonses.com</domain> 
        <result>pass</result> 
      </spf> 
    </auth_results> 
  </record> 
</feedback>

How to address DMARC deployment for domains confirmed to be unused for email (dangling or otherwise)

Deploying DMARC for unused or dangling domains is a proactive step to prevent abuse or unauthorized use of your domain. Once you have confirmed that all subdomains being used for sending email have the desired DMARC policies, you can publish a ‘p=reject’ tag on the organizational domain, which will prevent unauthorized usage of unused subdomains without the need to publish DMARC policies for every conceivable subdomain. For more advanced subdomain policy scenarios, read the “tree walk” definitions in https://datatracker.ietf.org/doc/draft-ietf-dmarc-dmarcbis/

Conclusion:

In conclusion, DMARC is not only a technology but also a commitment to email security, integrity, and trust. By embracing DMARC best practices, organizations can protect their users, maintain a positive brand reputation, and ensure seamless email deliverability. Every message from SES passes SPF and DKIM for “amazonses.com”, but the authenticated identifiers are not always in alignment with the domain in the “From” header which carries the DMARC policy. If email authentication is not fully configured, your messages are susceptible to delivery issues like spam filtering, or being rejected or blocked by the recipient ESP. As a best practice, you can configure both DKIM and SPF to attain optimum deliverability while sending email with SES.

About the Authors

	Bruno Giorgini is a Senior Solutions Architect specializing in Pinpoint and SES. With over two decades of experience in the IT industry, Bruno has been dedicated to assisting customers of all sizes in achieving their objectives. When he is not crafting innovative solutions for clients, Bruno enjoys spending quality time with his wife and son, exploring the scenic hiking trails around the SF Bay Area.
	Jesse Thompson is an Email Deliverability Manager with the Amazon Simple Email Service team. His background is in enterprise IT development and operations, with a focus on email abuse mitigation and encouragement of authenticity practices with open standard protocols. Jesse’s favorite activity outside of technology is recreational curling.
	Sesan Komaiya is a Solutions Architect at Amazon Web Services. He works with a variety of customers, helping them with cloud adoption, cost optimization and emerging technologies. Sesan has over 15 year’s experience in Enterprise IT and has been at AWS for 5 years. In his free time, Sesan enjoys watching various sporting activities like Soccer, Tennis and Moto sport. He has 2 kids that also keeps him busy at home.
	Mudassar Bashir is a Solutions Architect at Amazon Web Services. He has over ten years of experience in enterprise software engineering. His interests include web applications, containerization, and serverless technologies. He works with different customers, helping them with cloud adoption strategies.
	Priya Singh is a Cloud Support Engineer at AWS and subject matter expert in Amazon Simple Email Service. She has a 6 years of diverse experience in supporting enterprise customers across different industries. Along with Amazon SES, she is a Cloudfront enthusiast. She loves helping customers in solving issues related to Cloudfront and SES in their environment.

How to list over 1000 email addresses from account-level suppression list

2023-06-29 vmgaddam

Post Syndicated from vmgaddam original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-list-over-1000-email-addresses-from-account-level-suppression-list/

Overview of solution

Amazon Simple Email Service (SES) offers an account-level suppression list, which assists customers in avoiding sending emails to addresses that have previously resulted in bounce or complaint events. This feature is designed to protect the sender’s reputation and enhance message delivery rates. There are various types of suppression lists available, including the global suppression List, account-level suppression list, and configuration set-level suppression. The account-level suppression list is owned and managed by the customer, providing them with control over their list and account reputation. Additionally, customers can utilize the configuration set-level suppression feature for more precise control over suppression list management, which overrides the account-level suppression list.

Maintaining a healthy sender reputation with email providers (such as Gmail, Yahoo, or Hotmail) increases the probability of emails reaching recipients’ inboxes instead of being marked as spam. One effective approach to uphold sender reputation involves refraining from sending emails to invalid email addresses and disinterested recipients.

The account-level suppression list can be managed using Amazon SES console or AWS CLI which provides an easy way to manage addresses including bulk actions to add or remove addresses.

Currently, If the account-level suppression list contains more than 1000 records, we need to use NextToken to obtain a complete list of email addresses in a paginated manner. If the email address you are looking for is not within the first 1000 records of the response, you won’t be able to obtain the information from the account-level suppression list with one single command. To list all the email addresses within the account-level suppression, we use Amazon SES ListSuppressedDestinations API. This API allows you to fetch the NextToken and pass it to a follow-up request in order to retrieve another page of results.

The code below creates a loop that makes multiple requests, in each iteration, the next token is replaced, aiding in retrieving all email addresses that have been added to the account-level suppression list.

Prerequisite

The code below can be used to run in your local machine or using AWS CloudShell As part of this blog spot, we will be using AWS CloudShell to fetch the list.

Note: Python 3 and Python 2 are both ready to use in the shell environment. Python 3 is now considered the default version of the programming language (support for Python 2 ended in January 2020).

1) An active AWS account.
2) User logged in to AWS management console must have “ses:ListSuppressedDestinations” permissions.

Walkthrough

Sign in to AWS management console and select the region where you are using Amazon SES
Launch AWS CloudShell
Save the code specified below as a file in your local environment. Example: List_Account_Level.py
Click Actions and Upload File (List_Account_Level.py)

Upload File to AWS CloudShell

5. Run Python code.

Python3 List_Account_Level.py >> Email_Addresses_List.json

6. The file Email_Addresses_List.json will be saved in current directory
7. To download the file – Click Actions and Download File providing File name Email_Addresses_List.json

Download File from AWS CloudShell

List the Email addresses in your Amazon SES account suppression list added to recent bounce or complaint event using Python.

We used the ListSuppressedDestinations operation in the SES API v2 to create a list with all the email addresses that are on your account-level suppression list for your account including bounces and complaints.

Note: SES account-level suppression list applies to your AWS account in the current AWS Region.

import boto3
from datetime import datetime
import json

def showTimestamp(results):
    updated_results = []
    for eachAddress in results:
        updated_address = eachAddress.copy()
        updated_address['LastUpdateTime'] = eachAddress['LastUpdateTime'].strftime("%m/%d/%Y, %H:%M:%S")
        updated_results.append(updated_address)
    return updated_results

def get_resources_from(supression_details):
    results = supression_details['SuppressedDestinationSummaries']
    next_token = supression_details.get('NextToken', None)
    return results, next_token

def main():
    client = boto3.client('sesv2')
    next_token = ''  # Variable to hold the pagination token
    results = []   # List for the entire resource collection
    # Call the `list_suppressed_destinations` method in a loop

    while next_token is not None:
        if next_token:
            suppression_response = client.list_suppressed_destinations(
                PageSize=1000,
                NextToken=next_token
            )
        else:
            suppression_response = client.list_suppressed_destinations(
                PageSize=1000
            )
        current_batch, next_token = get_resources_from(suppression_response)
        results += current_batch

    results = showTimestamp(results)

    print(json.dumps(results, indent=2, sort_keys=False))

if __name__ == "__main__":
    main()

Sample Response

Returns all of the email addresses and the output resembles the following example:

[{
    "EmailAddress": "[email protected]",
    "Reason": "BOUNCE",
    "LastUpdateTime": "04/30/2021, 15:43:01"
}, {
    "EmailAddress": "[email protected]",
    "Reason": "BOUNCE",
    "LastUpdateTime": "04/30/2021, 15:43:01"
}, {
    "EmailAddress": "[email protected]",
    "Reason": "BOUNCE",
    "LastUpdateTime": "04/30/2021, 15:43:01"
}, {
    "EmailAddress": "[email protected]",
    "Reason": "BOUNCE",
    "LastUpdateTime": "04/30/2021, 15:43:00"
}, {
    "EmailAddress": "[email protected]",
    "Reason": "COMPLAINT",
    "LastUpdateTime": "06/22/2023, 12:59:31"
}]

Cleaning up

The response file Email_Addresses_List.json will contain the list of all the email addresses on your account-level suppression list even if there are more than 1000 records. Please free to delete files that were created as part of the process if you no longer need them.

Conclusion

In this blog post, we explained listing of all email addresses if the account-level suppression list contains more than 1000 records using AWS CouldShell. Having complete list of email addresses will help you identify email addresses you are looking for and that are not included in the first 1000 records of the response. You can validate email address and determine who can receive email that can be removed from the account-level suppression list. This protect the sender reputations and improving delivery rates.

Follow-up

https://docs.aws.amazon.com/ses/latest/dg/sending-email-suppression-list.html
https://repost.aws/knowledge-center/ses-remove-email-from-suppresion-list

About the Author

Venkata Manoj Gaddam is Cloud Support Engineer II at AWS and Service Matter Expert in Amazon Simple Email Service (SES) and Amazon Simple Storage Service (S3). Along with Amazon SES and S3, he is AWS Snow Family enthusiast. In his free time, he enjoys hanging out with friends and traveling.

How to verify an email address in SES which does not have an inbox

2023-06-29 ajibho

Post Syndicated from ajibho original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-verify-an-email-address-in-ses-which-does-not-have-an-inbox/

Overview of solution

Amazon Simple Email Service (Amazon SES) is an email platform that provides a straightforward and cost-effective solution for sending and receiving emails using your own email addresses and domains.

One of the most common use cases for using separate verified from email address is in online retails/e-commerce platforms. Online/e-commerce platform need to send emails to their customers where the from address should look like “[email protected]. In these cases, the From addresses like [email protected] does not have inbox setup for receiving emails. Using the following solution, you can avoid setting up an inbox for the email identity while still verifying the email address for sending and receiving.

In order to send emails from SES using email/domain identity, we need to have the From email identity or domain verified in Amazon SES in a supported region. When verifying a domain,you have the option to use Easy DKIM or Bring Your Own DKIM(BYOD). For verifying an email address, you need to create an identity in Amazon SES for the respective region. Once the required email address identity is created, you will receive a verification link in your inbox. To successfully verify the email address, simply open the link in your browser. In this case, you would need to have inbox setup for email address to receive the verification link from [email protected].

Verifying a domain in Amazon SES allows you to send emails from any identity associated with that domain. For example, if you create and verify a domain identity called example.com, you don’t need to create separate subdomain identities for a.example.com, a.b.example.com, nor separate email address identities for [email protected], [email protected], and so on. Therefore, the settings for the domain remain the same for all From addresses and you cannot separate you sending activity. You can use this solution to verify the From address without setting up an inbox and differentiate sending activity and tracking based on settings. The benefits of having different email settings from the domain are mentioned below.

Benefits of verifying the email separately for the same domain:

1) When you verify the email along with your domain, you can keep the settings different for the two Identities. You can setup different Configuration sets, notifications and dedicated IP pools for the verified email. This separation enables you to manage domain and email settings independently.
2) You can have two separate emails for sending transaction ([email protected]) and Marketing emails ([email protected]). After assigning different configuration sets, you can monitor the bounces and complaints separately for the sender. A best practice here would be separating the Transactional and Marketing in sub domains. Having both types in the same domain can adversely affect the reputation for your domain, and reduce deliverability of your transactional emails.
3) Using different dedicated IP pools, you can separate the sending IPs for Marketing and transaction or any other emails. Thus, your IP reputation for one use case is not affected by any other emails.

Prerequisite

1) An active AWS account.
2) Administrative Access to the Amazon SES Console and Amazon Simple Storage Service(S3) console.
3) A verified identity (Domain) with an MX record for the domain pointing to a Receiving Endpoint in one of the following region in Amazon SES.

Region Name	Region	Receiving Endpoint
US East (N. Virginia)	us-east-1	inbound-smtp.us-east-1.amazonaws.com
US West (Oregon)	us-west-2	inbound-smtp.us-west-2.amazonaws.com
Europe (Ireland)	eu-west-1	inbound-smtp.eu-west-1.amazonaws.com

Solution walkthrough

In order to verify the email in SES, we need to verify the link send from Amazon SES in the email inbox. We will setup receiving rule set and add S3 bucket with required permissions to store emails from Amazon SES in S3 bucket. After receiving the email in S3 bucket, download the email to get the verification link. Open the verification link in a browser to complete the process.

Step 1 : How to setup SES Email Receiving Ruleset for S3 bucket

1) Open the Amazon SES console.
2) In the navigation pane, under Configuration, choose Email Receiving.
Email Receiving Rule set

3) To create a new rule set, choose Create a Rule Set, enter a rule set name, and then choose Create a Rule Set.
Note: If you create a new rule set, select the rule set, and then choose Set as Active Rule Set. Only one of your receipt rule sets can be the active rule set at any given time.

4) Choose Active Rule Set and Choose Create Rule.

Active Ruleset

5) Enter a unique rule name. If your use case requires TLS or spam and virus scanning, then choose Require TLS or Enable spam and virus scanning. To make this an active rule, select the Enabled checkbox. Choose Next.
Receiving Rule Setting

6) To receive emails for specific verified domain, click Add new recipient condition and enter the domain/email address. You can leave it blank and it will store for all the verified domain addresses with receiving setup.
Add recipient condition

7) Choose Add new action, and then choose Deliver to S3 bucket
Action Deliver to S3 bucket

8) Click on Create S3 bucket

9) Enter a unique S3 bucket name and click on ‘Create Bucket’
Note: S3 Bucket policy will be added automatically.
Provide Unique S3 bucket name

(Optional) Choose Message encryption for Amazon SES to use an Amazon Key Management Server (Amazon KMS) key to encrypt your emails.
(Optional) For SNS topic, select an Amazon Simple Notification Service (Amazon SNS) topic to notify you when Amazon SES delivers an email to the S3 bucket.
Add Action in Receiving rule set

10) Click Next and Create Rule.
Review and Create Ruleset

Step 2: Verifying email address in Amazon SES using S3

The following procedure shows you how to verify Email address in Amazon SES.
1) Open the Amazon SES console.
2) In the navigation pane, under Configuration, choose Verified identities.
3) Choose Create identity.
Create Verified Identity

4) Under Identity details, choose Email address as the identity type you want to create.
5) For Email address, enter the email address that you want to use. The email address must be an address that’s able to receive mail and that you have access to.
(Optional) If you want to Assign a default configuration set, select the check box.
6) To create your email address identity, choose Create identity. After it’s created, you should receive a verification email within five minutes from [email protected].

Create Verified identity and Enter
7) Open the Amazon S3 console.
Go to S3 bucket

8) Open the S3 Bucket that you configured to store the Amazon SES emails. Verify that the bucket contains the test email that you sent. It can take a few minutes for the test email to appear.
Select the Received Email in S3 bucket

9) Select the email/object received in S3 bucket. Click Download.
Download the received email/object

10) Open the Downloaded file in Notepad and copy the verification link under the Subject. Paste the link in your Browser and confirm it.
Open the Downloaded email in Notepad

11) Once the link is confirmed, you can check in SES console and confirm under verified identities that your email address is in verified Status.
Browser link after pasting the verification link

Verified Identity confirmation in SES console

Cleaning up:

You should have successfully verified email address in Amazon SES using S3 bucket. To avoid incurring any extra charges, remember to delete any resources created manually if you no longer need them for monitoring.

Steps for removing the resources:

1) Delete all the created/verified Identities.
2) Delete data regarding Amazon SES receiving Rules.
3) Delete data regarding Amazon S3 bucket.

Conclusion:

In this blog post, we explained the benefits of verifying a separate email address for the verified domain without setting up an inbox. Having separate identities for different use cases helps in efficient management of bounces, complaints, and delivery. You can setup different IP pools using configuration set for different use cases.

Follow-up:

https://aws.amazon.com/blogs/messaging-and-targeting/manage-incoming-emails-with-ses/
https://docs.aws.amazon.com/ses/latest/dg/receiving-email.html
https://repost.aws/knowledge-center/ses-receive-inbound-emails

About the Author

Ajinkya Bhoite is Cloud Support Engineer II in AWS and Service Matter Expert in Amazon Simple Email Service(SES). Along with Amazon SES, he is an Amazon S3 enthusiast. He loves helping customers in solving issues related to SES and S3 in their environment. He loves reading, writing and running but not in the same order. He has a fictional novel published on Amazon Kindle by the name Shiva Stone: Hampi’s Hidden treasure.

How to investigate what happened to the email that was sent via SES but was never received in recipient inbox

2023-06-27 singhzwz

Post Syndicated from singhzwz original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-investigate-what-happened-to-the-email-that-was-sent-via-ses-but-was-never-received-in-recipient-inbox/

How to investigate what happened to the email that was sent via SES but was never received in recipient inbox

Amazon Simple Email Service (SES) is an email platform that provides an easy, cost-effective way for you to send and receive email using your own email addresses and domains.

You can use SES for two major use cases. The first use case is marketing emails, which can include things such as special offers, newsletters, and product updates. The second use case is transactional emails, which can include things such as order confirmation, One Time Password (OTP), and registration confirmation.

One common issue we hear from our customers is that emails are not delivered to the recipient’s inbox. There are three areas in the flow where this could happen:

Emails were not sent successfully by your email application
Emails get dropped within SES after being received from your email application
Emails are dropped after leaving SES

In this blog post, we will explore each of these three scenarios.

Prerequisites:

To investigate each scenario, you need the following:

Understand How email sending works in Amazon SES
Enable verbose logging on your email application
Amazon SES – Set up notifications for bounces and complaints

Scenario 1: Emails were not sent successfully by your email application

The first step in the process is for your application to contact SES to pass it the message. This is one place where problems could occur.
Remember that for each message it successfully enqueues, SES returns a message ID that you can log on your side. An example message ID looks like 01000174f87e0fb4-81b97243-56b0-4cd1-90c7-d6987a00fdc1-000000.

Here is an example of Sendmail logs for successfully relaying the email to SES.

Jun 22 15:29:42 ip-172-12-12-12.ec2.internal sendmail[323257]: 35ABCDEF123123: to=<[email protected]>, delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=30123, relay=email-smtp.us-east-1.amazonaws.com. [3.95.123.123], dsn=2.0.0, stat=Sent (Ok 01000174f87e0fb4-81b97243-56b0-4cd1-90c7-d6987a00fdc1-000000)

To verify that your requests are successful, you can look for the message IDs. If SES never accepted your message, you can be sure it won’t be delivered! If you are able to log a message ID, it means that your email safely arrived at SES.

Actions to troubleshoot client side issue:

1. In order to test that the problem lies between your application and SES, you can try to send a test email from the SES console to an email address and domain where you are experiencing the issue. If that test email arrives on time, it’s a pretty good indicator that the error is occurring before your email even reaches SES.
2. Examine the logs at application side to confirm if the application has successfully passed the email to SES. A success response looks like the following example:

250 Ok 0000012345678e09-123a4cdc-b56c-78dd-b90e-d123be456789-000000

Some common errors include the following:

* Email address is not verified. The following identities failed the check in region region: identity1, identity2, identity3
* Account is paused
* Daily message quota exceeded
* Maximum sending rate exceeded
* Maximum SigV2 SMTP sending rate exceeded

Another common type of error is a connection timeout error. Here is an example of a Java exception that shows a connection timeout error to an SES endpoint:

Caused: com.sun.mail.util.MailConnectException: Couldn't connect to host, port: email-smtp.ap-southeast-2.amazonaws.com , 465; timeout 60000; nested exception is: java.net .ConnectException: Connection timed out (Connection timed out) at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2209 ) at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:740 )

The way that you observe this error message depends on the way that you call SES.

If you call the SES API directly, the action to interact with SES will return an error like MessageRejected or one of the errors specified in the Common Errors topic of the Amazon Simple Email Service API Reference.

If you call SES through its SMTP interface, the way that you experience the error depends on the application. Some applications might display a specific error message, and others might not. For a list of SMTP response codes that SES returns, see SMTP response codes returned by Amazon SES.

Scenario 2: Emails get dropped within SES after being received from your email application

Once SES accepts the message and provides a message ID, the next step is to process it and hand it over to the recipient ISP or the recipient mailbox provider. However, in the following scenarios SES can drop the email after receiving it:

Rendering Failure: This can happen when you use an email template to send email. Upon accepting the email from your email application, SES validates that the template data you send includes the required variables in the template. If the template data contains non-compliant variables or is missing variables, SES is unable to construct the email and drops the email. This is called a Rendering Failure.
Suppression list: If a recipient is on an SES suppression list, SES will drop the email to protect the sender reputation. There are two types of suppression lists: Account-level suppression list and global suppression list. To learn more about the two types of suppression lists, please refer to Managing lists and subscriptions in Amazon Simple Email Service.
Reject Event: SES accepted the email, but determined that it contained a virus and didn’t attempt to deliver it to the recipient’s mail server. SES drops the email and generates a Reject event.

Actions to troubleshoot such issues:

To identify the root cause of the three preceding scenarios, you need to set up Amazon SES event publishing. Please refer to the prerequisite section to set this up.

Scenario 1: Rendering Failure

Here is an example event publishing record for Rendering Failure

{
"eventType":"Rendering Failure",
"mail":{
"timestamp":"2018-01-22T18:43:06.197Z",
"source":"[email protected]",
"sourceArn":"arn:aws:ses:us-east-1:123456789012:identity/[email protected]",
"sendingAccountId":"123456789012",
"messageId":"EXAMPLE7c191be45-e9aedb9a-02f9-4d12-a87d-dd0099a07f8a-000000",
"destination":[
"[email protected]"
],
"headersTruncated":false,
"tags":{
"ses:configuration-set":[
"ConfigSet"
]
}
},
"failure":{
"errorMessage":"Attribute 'attributeName' is not present in the rendering data.",
"templateName":"MyTemplate"
}
}

In this example, the error message tells you that there is a missing variable attributeName in the template data. To learn more about troubleshooting rendering failure, please refer to Why are emails failing to deliver when I send Amazon SES emails using the SendTemplatedEmail operation?

Scenario 2: Suppression list

SES drops the email if the recipient is on an account-level suppression list or a global suppression list. The following is an example error message for recipient being on a global suppression list:

"diagnosticCode": "Amazon SES has suppressed sending to this address because it has a recent history of bouncing as an invalid address. For more information about how to remove an address from the suppression list, see the Amazon SES Developer Guide: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/remove-from-suppressionlist.html"

The example demonstrates that the email get bounced because the recipient is on the global suppression list. We recommend enabling SES account-level suppression list because the account-level suppression list supersedes the global suppression list.

If the recipient is on your account-level suppression list, you can remove individual email addresses from your Amazon SES account-level suppression list. The following is an example error message for recipient being on a account-level suppression list:

"diagnosticCode": "Amazon SES did not send the message to this address because it is on the suppression list for your account. For more information about removing addresses from the suppression list, see the Amazon SES Developer Guide at https://docs.aws.amazon.com/ses/latest/DeveloperGuide/sending-email-suppression-list.html"

Note: The suppression list feature is in place to avoid emails being sent to recipients who are known to produce bounce or complaint for your emails in past. Enabling this feature protects your reputation as a sender. Only remove the recipient when you are assured that recipients in concern will no longer produce bounce or complaint for your emails.

Scenario 3: Reject Event

SES rejects the email when it scans the email and determines that it contains a virus. The following is an example of a Reject event publishing record.

{
"eventType": "Reject",
"mail": {
"timestamp": "2016-10-14T17:38:15.211Z",
"source": "[email protected]",
"sourceArn": "arn:aws:ses:us-east-1:123456789012:identity/[email protected]",
"sendingAccountId": "123456789012",
"messageId": "EXAMPLE7c191be45-e9aedb9a-02f9-4d12-a87d-dd0099a07f8a-000000",
"destination": [
"[email protected]"
],
"headersTruncated": false,
"headers": [{
"name": "From",
"value": "[email protected]"
},
{
"name": "To",
"value": "[email protected]"
},
{
"name": "Subject",
"value": "Message sent from Amazon SES"
},
{
"name": "MIME-Version",
"value": "1.0"
},
{
"name": "Content-Type",
"value": "multipart/mixed; boundary=\"qMm9M+Fa2AknHoGS\""
},
{
"name": "X-SES-MESSAGE-TAGS",
"value": "myCustomTag1=myCustomTagValue1, myCustomTag2=myCustomTagValue2"
}
],
"commonHeaders": {
"from": [
"[email protected]"
],
"to": [
"[email protected]"
],
"messageId": "EXAMPLE7c191be45-e9aedb9a-02f9-4d12-a87d-dd0099a07f8a-000000",
"subject": "Message sent from Amazon SES"
},
"tags": {
" ses:configuration-set": [
"ConfigSet"
],
"ses:source-ip": [
"192.0.2.0"
],
"ses:from-domain": [
"example.com"
],
"ses:caller-identity": [
"ses_user"
],
"myCustomTag1": [
"myCustomTagValue1"
],
"myCustomTag2": [
"myCustomTagValue2"
]
}
},
"reject": {
"reason": "Bad content"
}
}

To fix this issue, please examine the email and make sure there is no virus in the email. You can test your email content by sending a test email to Amazon SES mailbox simulator.

Scenario 3: Emails are dropped after leaving SES
We sometimes encounter this when we successfully deliver the email to a recipient ISP, only to have the recipient ISP hold back the email and not deliver to the recipient’s inbox. This can happen for a variety of reasons, for example the recipient ISP’s anti-spam filters, or because the sender has not yet built sufficient reputation due to it being a new domain or IP and the recipient does not yet trust the sender.

Actions to troubleshoot such issues:

To identify the root cause of the issue, you need to set up Amazon SNS notification. Please refer to the prerequisite section to set this up.

When the recipient ISP accepts the email, it returns a 250 ok status code to SES. SES includes the detailed status code and response message from the recipient ISP in the SNS notification.The following is an example SNS notification for a success delivery event.

{
"notificationType":"Delivery",
"mail":{
"timestamp":"2016-01-27T14:59:38.237Z",
"messageId":"0000014644fe5ef6-9a483358-9170-4cb4-a269-f5dcdf415321-000000",
"source":"[email protected]",
"sourceArn": "arn:aws:ses:us-east-1:888888888888:identity/example.com",
"sourceIp": "127.0.3.0",
"sendingAccountId":"123456789012",
"callerIdentity": "IAM_user_or_role_name",
"destination":[
"[email protected]"
],
"headersTruncated":false,
"headers":[
{
"name":"From",
"value":"\"John Doe\" <[email protected]>"
},
{
"name":"To",
"value":"\"Jane Doe\" <[email protected]>"
},
{
"name":"Message-ID",
"value":"custom-message-ID"
},
{
"name":"Subject",
"value":"Hello"
},
{
"name":"Content-Type",
"value":"text/plain; charset=\"UTF-8\""
},
{
"name":"Content-Transfer-Encoding",
"value":"base64"
},
{
"name":"Date",
"value":"Wed, 27 Jan 2016 14:58:45 +0000"
}
],
"commonHeaders":{
"from":[
"John Doe <[email protected]>"
],
"date":"Wed, 27 Jan 2016 14:58:45 +0000",
"to":[
"Jane Doe <[email protected]>"
],
"messageId":"custom-message-ID",
"subject":"Hello"
}
},
"delivery":{
"timestamp":"2016-01-27T14:59:38.237Z",
"recipients":["[email protected]"],
"processingTimeMillis":546,
"reportingMTA":"a8-70.smtp-out.amazonses.com",
"smtpResponse":"250 ok: Message 64111812 accepted",
"remoteMtaIp":"127.0.2.0"
}
}

In this example, the SMTP response is "250 ok:Message 64111812 accepted", this confirms the recipient ISP accepted the email. At this point, the email is with the recipient ISP and SES no longer has control over the email.

ISPs use different mechanisms and algorithms to filter emails to place them in either the recipient’s inbox folder or spam folder. In such case, you will need to reach out to the recipient ISP to understand why they accepted the email but did not deliver it to the recipient inbox. Since their servers decide the placement of email, they can give a clear explanation of the cause of the issue and steps to remediate the issue.

The recipient ISP may reject the email. When this happens, the recipient ISP returns a 4xx or 5xx error code to SES. This is also called a bounce event.

There are two kinds of bounce events: Soft bounce and hard bounce. Soft bounce refers to a temporary email delivery failure, this kind of error usually can be retried. Hard bounce refers to a permanent delivery failure, and it cannot be retried. The Simple Mail Transfer Protocol (SMTP) Enhanced Status Codes Registry lists the possible error codes and reasons for email delivery failure. Generally speaking, 4xx error codes are soft bounces, 5xx error codes are hard bounces.

When the recipient ISP bounces the email, SES includes the detailed status code and response message from the recipient ISP in the SNS notification.

The following is an example SNS notification for a bounce event.

{
"notificationType":"Bounce",
"bounce":{
"bounceType":"Permanent",
"reportingMTA":"dns; email.example.com",
"bouncedRecipients":[
{
"emailAddress":"[email protected]",
"status":"5.1.1",
"action":"failed",
"diagnosticCode":"smtp; 550 5.1.1 <[email protected]> User not found"
}
],
"bounceSubType":"General",
"timestamp":"2016-01-27T14:59:38.237Z",
"feedbackId":"00000138111222aa-33322211-cccc-cccc-cccc-ddddaaaa068a-000000",
"remoteMtaIp":"127.0.2.0"
},
"mail":{
"timestamp":"2016-01-27T14:59:38.237Z",
"source":"[email protected]",
"sourceArn": "arn:aws:ses:us-east-1:888888888888:identity/example.com",
"sourceIp": "127.0.3.0",
"sendingAccountId":"123456789012",
"callerIdentity": "IAM_user_or_role_name",
"messageId":"00000138111222aa-33322211-cccc-cccc-cccc-ddddaaaa0680-000000",
"destination":[
"[email protected]",
"[email protected]",
"[email protected]"],
"headersTruncated":false,
"headers":[
{
"name":"From",
"value":"\"John Doe\" <[email protected]>"
},
{
"name":"To",
"value":"\"Jane Doe\" <[email protected]>, \"Mary Doe\" <[email protected]>, \"Richard Doe\" <[email protected]>"
},
{
"name":"Message-ID",
"value":"custom-message-ID"
},
{
"name":"Subject",
"value":"Hello"
},
{
"name":"Content-Type",
"value":"text/plain; charset=\"UTF-8\""
},
{
"name":"Content-Transfer-Encoding",
"value":"base64"
},
{
"name":"Date",
"value":"Wed, 27 Jan 2016 14:05:45 +0000"
}
],
"commonHeaders":{
"from":[
"John Doe <[email protected]>"
],
"date":"Wed, 27 Jan 2016 14:05:45 +0000",
"to":[
"Jane Doe <[email protected]>, Mary Doe <[email protected]>, Richard Doe <[email protected]>"
],
"messageId":"custom-message-ID",
"subject":"Hello"
}
}
}

In the preceding example, the recipient ISP permanently bounced the email. This is concluded from the "bounceType":"Permanent" field. Additionally, the recipient ISP provides the reason they bounced the email in the “diagnosticCode” field, and the reason is "smtp; 550 5.1.1 <[email protected]> User not found". In this case, [email protected] isn’t a valid inbox, the error can’t be resolved by retrying the request. The sender should check the spelling of the email address and confirm with the recipient that this address has a valid inbox before retry sending to this address again. You can also contact the recipient ISP to understand the reasons of the error and steps to remediate the issue when the information provided by the "diagnosticCode" isn’t sufficient or clear.

Another event that can happen is “delivery delay”. Such event happen when the email couldn’t be delivered to the recipient’s mail server because a temporary issue occurred. Delivery delays can occur, for example, when the recipient’s inbox is full, or when the receiving email server experiences a transient issue. These are usually soft bounces in which case SES is programmed to retry such deliveries several times over the next 10 hours. If you have enabled SES event publishing for this event you can review the “delivery delay” object section of SES event record to understand the cause of delay and take corrective actions accordingly. As an example, if you reach out to the recipient ISP and the recipient ISP informs you about maintenance on their side that is causing the issue then you can refrain sending to that ISP till the maintenance completes.

Conclusion:
Now that you know the 3 areas in the email sending flow where errors can occur you can begin to troubleshoot whether your problems are at your email application side, within SES, or at the recipient ISP. Make sure that you have proper logging and that you are receiving the events that provide you the detail you need to determine where your errors are occurring. Errors happening prior to the email being sent out from SES are usually solvable, while challenges in deliverability once an email has been sent from SES require more due diligence and experimentation in your content to determine why your emails are being delivered to spam, or not at all.

Feel free to post any comments or questions for us in the AWS Forum.

References:
[1]https://docs.aws.amazon.com/ses/latest/dg/send-email-concepts-process.html
[2]https://docs.aws.amazon.com/ses/latest/dg/monitor-sending-activity.html
[3]https://docs.aws.amazon.com/ses/latest/dg/monitor-using-event-publishing.html#event-publishing-how-works

Amazon SES – How to track email deliverability to domain level with CloudWatch

2023-06-21 Alaa Hammad

Post Syndicated from Alaa Hammad original https://aws.amazon.com/blogs/messaging-and-targeting/amazon-ses-how-to-track-email-deliverability-to-domain-level-with-cloudwatch/

Why is it important to track email deliverability per domain with Amazon Simple Email Service (SES)?

Amazon Simple Email Service (Amazon SES) is a scalable cloud email service provider that enables businesses to build a large-scale email solution and host multiple domains from the same SES account for different purposes ex: one domain for sending marketing emails such as special offers, another domain to send transactional emails such as order confirmations, and other types of correspondence such as newsletters.

As your product, service or solution built on Amazon SES grows and you require multiple domains verified, it is important to track email deliverability for emails you send from each domain for business continuity, billing purposes or incidents investigations. This can be useful to identify if you have low email deliverability for your business domain or if you have a domain generating high bounce or complaint rates and take proactive actions before impacting the account’s ability to send emails from any other domains.

SES offers features that automatically manage deliverability per domain through Virtual Deliverability Manager. Virtual Deliverability Manager helps enhance email deliverability and provides insights into sending and delivery data, as well as offering solutions to fix negative email sending reputation. You can learn more about Virtual Deliverability Manager here.

Solution Walkthrough

Amazon SES provides a way to monitor sender reputation metrics such as bounce and complaint rates per account or configuration sets using event publishing. This blog will discuss how you can use Amazon SES message auto-tags to monitor and publish email deliverability events (Send, Delivery, Bounce, Complaints) to CloudWatch custom metrics per domain. In addition, you will see how to create a custom CloudWatch dashboard that’s easy to access in a single view to monitor your domain metrics. This CloudWatch dashboard can help to provide guidance for your team members during operational events about how to respond to specific incidents for your sending domain.

What are Amazon SES Auto-Tags:

Message tags are a form of name/value pairs to categorize the email you are sending. For example, if you advertise books, you could name a message tag general, and assign a value of sci-fi or western, when you send an email for the associated campaign. Depending on which email sending interface you use, you can provide the message tag as a parameter to the API call (SendEmail, SendRawEmail) or as an Amazon SES-specific email header.

In addition to the message tags you add to any emails you send, Amazon SES adds a set of Auto-Tags that are automatically included in any emails you send. You don’t need to pass the parameters of the auto-tags to the API call or email headers since SES does this automatically.

The auto-tags in the list below are used to track the email deliverability for specific events ( ex: Send, Delivery, Bounce, Complaint). SES does this by using the name/value pairs of the auto-tag name as a dimension in CloudWatch metric to track the count of events of specific auto-tag. This blog post will use “ses:from-domain” auto-tag to configure event publishing for tracking and publish email deliverability events (Send, Delivery, Bounce, Complaints) you receive per domain to CloudWatch metrics and CloudWatch dashboard.

Amazon SES auto-tags added to messages you send

Prerequisites:

For this walkthrough, you should have the following prerequisites:

An AWS account.
A verified domain in Amazon SES.
Administrative Access to Amazon SES Console and Amazon CloudWatch Console.
Amazon CLI latest version installed and configured.

Configure Amazon SES to publish email deliverability events to CloudWatch destination:

To configure event publishing for tracking email deliverability events, you first need to create a configuration set. Configuration sets in SES are groups of rules, that you can apply to your verified identities. When you apply a configuration set to an email, all of the rules in that configuration set are applied to the email.

After your configuration set is created, you need to create Amazon SES event destination. Amazon SES will send all email deliverability events you intend to track to this event destination. In this blog the event destination is Amazon CloudWatch.

1. Sign in to the Amazon SES console.
2. In the navigation pane, under Configuration, choose Configuration sets. Choose Create set.
3. Enter Configuration set name, leave the rest of fields to default, scroll to the send and click on Create set.
4. Under configuration set home page click on Event destinations tab and select Add destination
6. Under Select event types, check Sends, Deliveries, Hard bounces and Complaints boxes and click Next.
8. Under Specify destination, Select Amazon CloudWatch.
10. Name – enter the name of the destination for this configuration set. The name can include letters, numbers, dashes, and hyphens. (example : Tracking_per_Domain)
11. Under Amazon CloudWatch dimensions, Select Value source: Message tag , Dimension name: ses:from-domain and Default value: example.com (you will need to add the verified domain name you want to track) as shown below:
13. Review, When you are satisfied that your entries are correct, Click Add destination to add your event destination.

Send a test email via Amazon SES mailbox simulator to trigger events in CloudWatch custom metric.

After selected Amazon CloudWatch as event destination , Amazon CloudWatch will create a custom metric with the auto-tag dimension and value you chose. For this custom metric to appear in CloudWatch Console, you must send an email to trigger each selected event. We recommend using the Amazon SES Mailbox Simulator to avoid generating real bounces or complaints that could impact your account’s reputation.

In the below section, This blog will show how to send those test emails to the following recipients manually using CLI. If you would like to use the console method to send those emails. you will need to send three separate test emails since the console will only allow one recipient per message:

Amazon SES Mailbox Simulator recipients to trigger the events in CloudWatch metrics:
[email protected]
[email protected]
[email protected]

Note: You must pass the name of the configuration set when sending an email. This can be done by either specifying the configuration set name in the headers of emails, or specifying it as a default configuration set. This can be done at the time of identity creation, or later while editing a verified identity.

The following example uses send-emailCLI command to send a formatted email to the Amazon SES simulator recipients:

Before you run any commands, set your default credentials by following Configuring the AWS CLI. The IAM user must has “ses:SendEmail” permission to send email.

Navigate to your terminal where the AWS CLI is installed and configured. Create message.json file for the message to send and add the following content:

{
"Subject": {
"Data": "Testing CW events with email simulator",
"Charset": "UTF-8"
},
"Body": {
"Text": {
"Data": "This is the message body of testing CW events with email similulator.",
"Charset": "UTF-8"
}
}
}

Create a destination.json file to add Amazon SES simulator recipients for bounces, complaints and delivery events as shown below:

{ 

"ToAddresses": ["[email protected]", "[email protected]" , "[email protected]"]

}

Send a test email using send-email CLI command to send a formatted email to the Amazon SES simulator recipients:

aws ses send-email --from [email protected] --destination file://destination.json --message file://message.json --configuration-set-name SES_Config_Set --region <AWS Region>

After the message sent, you are expected to see the following output:

{

"MessageId": "EXAMPLEf3a5efcd1-51adec81-d2a4-4e3f-9fe2-5d85c1b23783-000000"

}

Now you sent a test email to trigger the events you want to track in CloudWatch custom metrics. Lets create the CloudWatch dashboard to see those metrics.

Create CloudWatch dashboard to track the email deliverability events for my domain.

Sign in to the Amazon CloudWatch console.
In the navigation pane, choose Dashboards, and then choose Create dashboard.
In the Create new dashboard dialog box, enter a name like ‘CW_Domain_Tracking’ for the dashboard, and then choose Create dashboard.
In the Add Widget dialog box, Choose Number to add a number displaying a metric to the dashboard and then choose Next
Under Add metric graph, click on edit sign to rename the graph with your domain example.com . this will make it easy for you to select the dashboard of the domain if you have multiple domains.
In the Browse tab , Select the AWS region where you are running your SES account and in the search bar, search for “ses:from-domain”.
You will get four metrics returned with your domain name “example.com”. Select checkbox beside the four metrics and click Create widget.
Save dashboard in the top right corner of the dashboard page to save the widget settings.

After the CloudWatch dashboard created, for any email you send from example.com domain with configuration set name passed in the email header, The email deliverability events will be counted in your CloudWatch metrics and you will be able to see them in the CloudWatch dashboard.

As an additional step. You can also setup a CloudWatch alarms for this custom metrics and add a threshold for each metric. When the metric breach the threshold, the alarm goes on and send an SNS notification to you to take the necessary actions.

Cleaning Up:

This setup includes Amazon CloudWatch and Amazon SES service charges. To avoid incurring any extra charges, remember to delete any resources created manually if you no longer need them for monitoring.

Resources to delete from Amazon SES console.

In the navigation pane, under Configuration, choose Configuration sets.
Check the box beside Configuration set you created and select Delete.

Resources to delete from Amazon CloudWatch console.

In the navigation pane, choose Dashboards, and then choose the dashboard you created.
In the upper-right corner of the graph that you want to remove, choose Actions, and then choose Delete Dashboard.
Save dashboard.

Conclusion:

You have now seen how to configure Amazon SES to track email deliverability at domain level with CloudWatch dashboard. Tracking email deliverability for emails you send from each domain is essential for business continuity, billing purposes or incidents investigations. Using SES message auto-tags and CloudWatch metrics you can identify the domains that have low email deliverability quickly and take necessary actions to maximize your email deliverability and take proactive actions before impacting the account’s ability to send emails from any other domains.

About the author:

Alaa Hammad is a Senior Cloud Support Engineer at AWS and subject matter expert in Amazon Simple Email Service and AWS Backup service. She has a 10 years of diverse experience in supporting enterprise customers across different industries. She enjoys cooking and try new recipes from different cuisines.

How to send your first email on SES

2023-06-21 Dustin Taylor

Post Syndicated from Dustin Taylor original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-send-your-first-email-on-ses/

Introduction

Sending your first email on any service can be complicated. In this blog we will walk you through how to send your first email on Amazon Simple Email Service (SES) through the SES Console and to direct you to examples of how you can send email through the AWS SDK. Our public documentation includes additional information on how you can configure SES. We encourage you to read through these documents to learn about these other mechanisms in the future.

Getting Started

Getting started with sending an email on SES requires three actions which are: 1) verifying a domain or email address 2) requesting production access to SES and 3) sending your first email. Let’s walk through each of these steps and send our first email.

Verifying an Identity

To start, you will configure what email address or domain your customers will receive emails from. As part of this verification, you will need to be able to either receive a confirmation email at the email address you are trying to setup, or to publish CNAME records for your intended domain. Generally, we recommend using a domain for your email sending as this gives you the ability to set up SPF, DKIM, and DMARC alignment which will increase recipient trust in your emails. Email addresses can be used for account-specific email sending where a customer may not own a domain, but this type of use-case is prone to receiving entities having low trust in the sender and a lower probability of inbox placement. For more in-depth instructions please review our public documentation as I will briefly touch on the most important pieces to verifying a domain or email address.

To verify an identity, you can go to the SES Console and click the ‘Verified identities’ link on the left-hand side of the screen. It will then present you with a list of verified domains or email addresses currently in your account if they were previously verified. There is a yellow button that states ‘Create identity’, when you click this you will be presented with a screen to choose whether to verify an email address or domain.

Email Address Verification

To verify an email address, you will be prompted with the following dialog:

The dialog presented to a sender when they choose to verify an email address in the SES console.

To verify an email address to use as your sending identity, you will include the address in the ‘Email address’ field and then click the ‘Create identity’ button. This will trigger an automated email to the address with a verification link that will need to be clicked to verify ownership of the email address. Once verified, you can begin sending emails from your new email address identity.

Domain Verification

To start verifying a domain you will click the ‘Verified identities’ option from the ‘Configuration’ dropdown which can be found on the left side of the screen. When choosing to verify a domain, you will be presented with a series of dialogs which include:

The dialog presented to a sender that prompts a decision to verify a domain or email address.

Here you will need to include the domain you intend to use for email sending. If you are keeping to a basic configuration on SES this will be the only data you need to add to this dialog. However, it is recommended to also use a custom mail-from. A custom mail-from is a way for you to remove the amazonses.com domain from your mail-from header to ensure domain alignment throughout your headers. You can find more information about the custom mail-from addresses in our documentation.

After finishing your changes in the first dialog you will then be presented with a second dialog that looks like the following:

The dialog which allows a sender to verify the domain they intend to use to send email.

To verify the domain, you will need to utilize either the Easy DKIM feature, or to provide a DKIM authentication token if you plan to DKIM sign your own messages. In selecting the ‘Easy DKIM’ option, you will be presented with the option to use either 1024 bit or 2048 bit signing key length. We would recommend utilizing the 2048 bit signing key length for most customers as this is the more secure key.

If you use Amazon Route53 as your DNS provider, SES can automatically publish DNS records for your domain. If not, this step will require you to edit your DNS records to include three CNAME records which are used for the DKIM signature process and as a mechanism to prove domain ownership. An example of the CNAME records is as follows:

An example dialog of the CNAME records that are generated when attempting to verify an identity.

Once you have placed these DNS records SES will periodically attempt to look-up the records to change the status of your domain verification. If SES doesn’t automatically update the status, you are presented with the option to force another check to verify the records are present.

After your domain verification is successful, you are now ready to send emails from any email address for your domain.

Requesting Production Access

Now that you’ve verified an identity, the next step is to be able to send an email to an unverified identity you will need to request production access. If you only want to test to your own domain or email address you can skip this step until you are ready to send to unverified recipients.

Note: This is region-based, a request for production access is limited to the region in which you are requesting.

To begin this process, you will navigate to the SES Console and the ‘Account dashboard’ section. Once you are on this page you will be presented with the following dialog at the top of your screen.

Clicking the ‘Request production access’ button will then navigate to the ‘Request details’ page which you can reference below.

The dialog from the SES console showing that the SES account is still in the sandbox.
Fill out each section with the details of your mail-type, website URL, use case description, and then acknowledging that you have read and agree to the AWS Service Terms and Acceptable Use Policy (AUP). When filling out the use case description, provide as much detail as you can for your request as our teams will review to determine if we need more information before approving or denying your request. An example of a good use case description would look like the following:

“Example.com is the domain my company intends to use to send our transactional emails. Our recipients are all customers who have either signed up for an account, requested a new password, or have made purchases through our website. We require confirmation of opt-in for all our new accounts and if no confirmation is received, we do not attempt to send an email to that address.”

Note: SES will review your production access request and will provide feedback on your use case and whether it could pose a risk to the sending reputation of SES, our customers, or your own sending domain.

Finally, click the ‘Submit request’ button to submit your request for production access. This will create an AWS Support case and will be reviewed by our team. These requests are reviewed with a 24-hour Service Level Agreement (SLA). While you are waiting for production access you can send test emails to any of the Mailbox Simulator endpoints or to your own verified domain(s) or email address(es).

Sending Your First Email

From the Console

To send your first email from the SES Console you will need to start by clicking the ‘Verified identities’ option from the ‘Configuration’ dropdown which can be found on the left side of the screen. From here you will select the domain and/or email address you want to send your email from and then click the ‘Send test email’ button, which will open the following screen:

The message details dialog where a sender can send an email from the SES console

From here you will fill out the ‘From-address’ box with the local name (anything before the @ sign) that you want to use to send the email. If you want to test SES functionality you can choose any of the dropdown events present, or you can choose the ‘Custom’ option which will allow you to set a ‘Custom recipient’ address of your choosing. Then you will fill out the ‘Subject’ and ‘Body’ fields with the content you will use for this first test email and then click the ‘Send test email’ button.

Congratulations, you’ve sent your first email from the SES Console! Now, utilizing SES to send single emails from the console isn’t the most scalable way to send email. In the next section, I will provide you links to our documentation for the 5 programming languages supported with the AWS SDK so that you can begin building your integration with SES.

From Code

The AWS Documentation includes some code snippets on how to send an email with SES via the AWS SDK. You can find examples of how to send an email from languages such as: .NET, Java, PHP, Ruby, and Python. We highly recommend reviewing our documentation to see these introductory code snippets to get you started.

Conclusion

Hopefully this blog post has aided you in your journey to send your first email through SES. From verifying a domain, requesting production access, and finally sending an email through the console. Take this knowledge and build upon it for future success in sending email through SES. Happy sending!

How to send messages to multiple recipients with Amazon Simple Email Service (SES)

2023-06-21 Joydeep Dutta

Post Syndicated from Joydeep Dutta original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-send-messages-to-multiple-recipients-with-amazon-simple-email-service-ses/

Introduction

Customers frequently ask what is the best way to send messages to multiple recipients using Amazon Simple Email Service (SES) with the best deliverability and without exceeding the maximum recipient’s per message limit. In this blog, we will show you how to determine the best approach for sending a message to multiple recipients based on different use-cases. We will also discuss why in most situations sending messages to a single recipient at a time is the best approach.

Difference between message Header addresses and Envelope addresses

Before we dive into the use cases, let’s discuss how message addressing works in SES. When a client makes a request, SES constructs an email message compliant with the RFC 5322 Internet Message Format specification . An email comprises of a header, a body, and an RFC 5321 envelope, as described in the Email format in Amazon SES document.

The email addresses in the RFC 5322 To, Cc and Bcc headers are for display. These headers enable your email client interface to display to whom the message was addressed. These addresses do not control which recipients receive the messages; the envelope addresses do. The sending mail client provides the envelope recipient addresses to a mail server using the RFC 5321 RCPT TO commands. RCPT is an abbreviation for recipient.

An apt analogy (see diagram below) is how a physical letter within an envelope can address a person whose address is not the envelope. The address on the envelope is what the mail carrier to deliver the envelope. The postal worker should not need to open the envelope to know which address to deliver the mail.

As an example, a school district may send letters informing residents of enrollment details for their children, but they do not know all of the names of the people who live at each address. The envelope may only list the address, and the letter may just be addressed “To Resident” if the school district doesn’t have a name to address the letter. The message is delivered to the resident’s address regardless of the accuracy of the information on the letter.

To simplify, let’s summarize the differences between To & Cc header and envelope addresses:

Header To & Cc Addresses	Envelope Addresses (RCPT)
Used by email clients to display the list of recipients	Used by mail servers to deliver the email message
Not used for mail delivery	Used for mail delivery
Displayed to recipients	Not displayed to recipients

The Bcc address is different than the To and Cc headers because it is used to send a copy of the message to an additional set of recipients that are “blind” to the other recipients. Bcc addresses are only defined by envelope addresses, not as a header address. Mail servers will commonly remove a Bcc header when handling a message, but delivery to the envelope recipient address still occurs.

When to use multiple recipients in a Destination

SES supports sending messages to multiple recipients in a single SendEmail operation. The Destination argument of the SendEmail operation represents the destination of the message. A Destination consists of To, Cc, and Bcc fields which represent both the header addresses and the envelope addresses.

When multiple recipients are defined in the Destination argument to the SendEmail operation, the defining characteristic is that every recipient receives the exact same message with the same message-id. A message-id is used for event handling (bounces, complaints, etc) among other purposes. A message-id pertains to exactly one version of a particular message.

Did you know: The use cases for recipients having a message with the same message-id are limited to situations in which the recipients are expected to interact with the message as a group. For example, recipients may reply-all to the email and have a resulting email conversation. The original message-id is used by email client applications to display a “conversation” view using the References and In-Reply-To headers. This behavior may be a good fit if the use case is a mailing list or internal announcement to employees within a company.

The recipient limit in the Destination argument is 50 because that is a reasonable break-point when the “conversational” use case runs the risk of the “reply all storm“ described in the next section. Consider using a robust mailing list solution or hosted service with capabilities similar to GNU Mailman to facilitate large group email conversations.

Why bulk mail recipients should not see other recipients

For bulk sending purposes, and most transactional sending, the recipients don’t need to know that other recipients also received the message:

The recipients likely gain no value from seeing the other recipient addresses, as they may be arbitrarily segmented into batches of 50 or less, and most email client interfaces have trouble displaying more than 50 addresses.
There is a risk of a “reply-all storm“, which is when a recipient replies to all of the To and Cc addresses from the original message, and then those people reply back asking everyone to stop replying. This scenario is fun to talk about around the water cooler, but should be avoided.
If recipients are defined as Bcc recipients in the Destinations argument of the SendEmail operation, it would not contain a To address, and that can look suspicious when read by the recipients.

Note: There is no authentication mechanism protecting the To or Cc headers from spoofing, so be careful about assuming any trust placed into the values of those headers. This means that it is possible for an attacker to spoof the To or Cc headers in an email message. Therefor the only meaningful address to include in the To header is the recipient’s own address, which they know isn’t spoofed because of the fact that they are reading the message.

For bulk mail it is best practice to have each recipient see only their own name and email address in the To header of the messages they receive. This makes the messages look more personable and can improve deliverability and recipient engagement.

This approach can be achieved by sending the message to each recipient individually via the SendEmail operation. You would use a single address in the “ToAddressses” field of the “Destination” argument.

Use the ToAddress field to individual message in the SendEmail API

How email event notifications are associated with recipients

If you need email event notifications to be associated to each recipient, then you will need each recipient to receive a message with a unique message-id; one recipient per Destination.

The following event types will be associated with every recipient in the Destination:

asynchronous bounces
complaints
opens
clicks

Learn more about Amazon SES events in the documentation: how email sending works in Amazon SES

For example, if one of the recipients triggers a open engagement event, and if that recipient was in a group of 50 recipients within the Destination argument to the SendEmail operation, then all 50 of those recipients will be registered as having opened the message.

Other considerations:

If the recipients are defined by ToAddresses and CcAddresses they will all appear in the message headers, but the To and Cc headers will be truncated in the event notifications if the headers are over 10 KB. Multi-recipient Destinations may cause you to lose observability needed to troubleshoot deliverability issues.
SES Virtual Deliverability Manager only tracks metrics from emails that have one recipient. Multi-recipient Destinations are not counted in any of the Virtual Deliverability Manager dashboard metrics.
SES counts the number of envelope recipients in an email toward the account’s sending quotas. Multi-recipient Destinations is not a way to achieve higher sending limits.
SES charges for each recipient receiving a message regardless of how many recipients are included in the Destination for each API invocation. Multi-recipient Destinations is not a way to reduce costs.

For bulk sending use cases, it is best practice to have each recipient have a copy of the message with a unique message-id to achieve the highest level of observability of your email sending program. High observability leads to high deliverability. This can be achieved by sending the message to each recipient individually.

How to send Emails to multiple recipients with SES

At this point, you should understand why it is a best practice to send a message to multiple recipients by iteratively using a single recipient in the Destination argument of the SendEmail operation.

Sending a message to a single recipient at a time is the best way to get started delivering messages to multiple recipients. Sending email in this fashion ensures that your deliverability metrics are giving you the observability needed to achieve the highest engagement with your recipients.

The following example uses the SES version 2 command line interface (CLI) to send a message to a list of recipients. If you do not want to use the CLI, use SES with an AWS SDK and adapt the commands into the syntax of the SDK of your choice.

#!/bin/bash

# Replace these variables with your own values
# sender 
# - Consider not using no-reply@, and instead use SES Inbound to receive replies
# - Consider a descriptive username@; some mobile clients will display it prominently, so it should make sense to the recipient.
# - Consider using a subdomain for bulk and transactional mail. Don't use the domain used by your users.
# - Consider using a verified domain identity. Don't use an email address identity within a domain that has a DMARC policy.
sender="[email protected]"
subject="Email subject"
body="Email body"
region="us-east-1"

# List of recipients, one per line. Defaults to SES mailbox simulator addresses (https://docs.aws.amazon.com/ses/latest/dg/send-an-email-from-console.html#send-email-simulator-how-to-use)
recipients=(
  "[email protected]"
# ... 
  "[email protected]"
)

# Send an email to each recipient
# Iterate through the list of recipients.
# Invoke the AWS SES SendEmail operation with a single recipient defined in the Destination
for recipient in "${recipients[@]}"; do
  aws sesv2 send-email \
    --from-email-address "$sender" \
    --destination "ToAddresses=$recipient" \
    --content "Simple={Subject={Data='$subject',Charset='UTF-8'},Body={Text={Data='$body',Charset='UTF-8'}}}" \
    --region "$region"
done

# The output will look similar to this, with a unique MessageID associated with each recipent.
# {
#    "MessageId": "010001874edd1765-be4ea5c2-d2b1-4ffb-bfb9-46461d18d80c-000000"
# }
# ... 51 total message-ids
# {
#    "MessageId": "010001874edd1b94-468ecee9-9198-4356-9f53-a108097777e5-000000"
# }

In this example script, the SendEmail operation is invoked multiple times using the CLI to deliver the message individually to each recipient, and each recipient only sees their own address in the To header. We called the SendEmail operation 51 times and a total of 51 Message Ids were returned in the response.

How to use SendEmail for multiple recipient advanced use cases

Consider a scenario where a memo needs to be sent to an entire team, the team is large, and only a few of the recipients need to be displayed in the headers. In this use case, it is desirable to send multiple copies of an email to many recipients who all receive the same To and Cc headers.

To customize the headers, you must use the Raw field of the Content argument instead of the Simple field.

The example below will reference another internet standard called Multipurpose Internet Mail Extensions (MIME): Format of Internet Message Bodies.

What’s in a MIME object:

Headers (such as From, Subject, and Reply-to)
Body – Plain text and HTML
Attachments – Files and images

MIME extends the capabilities of RFC 5322 and is used to format most email messages to this day. There are a variety of packages that can assist in creating a MIME structured messages, which you can find by searching relevant package managers.

This is an example in Python to create a MIME formatted message for the next script.

#!/usr/bin/env python
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import base64

# You must change the 'fromAddress' variable for this example to work in your environment.
#
# When choosing a From header address:
# - Consider not using no-reply@, and instead use SES Inbound to receive replies
# - Consider a descriptive username@; some mobile clients will display it prominently, so it should make sense to the recipient.
# - Consider using a subdomain for bulk and transactional mail. Don't use the domain used by your users.
# - Consider using a verified domain identity. Don't use an email address identity within a domain that has a DMARC policy.

fromAddress = "Descriptive Name <[email protected]>"

# The To and Cc addresses here are for the email header. They are what will be displayed to the recipient.
# The actual recipient, or evelope recipient, will be set later.
toAddresses = ['Founder Name <[email protected]>']
ccAddresses = ['President <[email protected]>', 'Director <[email protected]>']
subjectTxt = "Success and Scale Bring Broad Responsibility"
messageTxt = "We started in a garage, but we’re not there anymore. We are big, we impact the world, and we are far from perfect. We must be humble and thoughtful about even the secondary effects of our actions. Our local communities, planet, and future generations need us to be better every day. We must begin each day with a determination to make better, do better, and be better for our customers, our employees, our partners, and the world at large. And we must end every day knowing we can do even more tomorrow. Leaders create more than they consume and always leave things better than how they found them."
messageHtml = "<html><body><p>" + messageTxt + "</p></body></html>"
CHARSET = "utf-8"

multiPartEmail = MIMEMultipart()
multiPartEmail['From'] = fromAddress
toAddressesJoined = ",".join(toAddresses)
multiPartEmail['To'] = toAddressesJoined
ccAddressesJoined = ",".join(ccAddresses)
multiPartEmail['Cc'] = ccAddressesJoined
multiPartEmail['Subject'] = subjectTxt
msg_body = MIMEMultipart('alternative')
textpart = MIMEText(messageTxt.encode(CHARSET), 'plain', CHARSET)
htmlpart = MIMEText(messageHtml.encode(CHARSET), 'html', CHARSET)
msg_body.attach(textpart)
msg_body.attach(htmlpart)
multiPartEmail.attach(msg_body)

print("Human readable blob:")
print((multiPartEmail.as_string()))
print("Base64 Encoded Blob:")
print(base64.b64encode(multiPartEmail.as_bytes()))
Running this script will produce output similar to the following:
Human readable blob:
Content-Type: multipart/mixed; boundary="===============0865862865556646150=="
MIME-Version: 1.0
From: [email protected]
To: [email protected]
Cc: [email protected], [email protected]
Subject: Success and Scale Bring Broad Responsibility

--===============0865862865556646150==
Content-Type: text/text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

We started in a garage, but we’re not there anymore. We are big, we impact the world, and we are far from perfect. We must be humble and thoughtful about even the secondary effects of our actions. Our local communities, planet, and future generations need us to be better every day. We must begin each day with a determination to make better, do better, and be better for our customers, our employees, our partners, and the world at large. And we must end every day knowing we can do even more tomorrow. Leaders create more than they consume and always leave things better than how they found them.

--===============0865862865556646150==--

Base64 Encoded Blob:
b'Q29udGVudC1U...TRIMMED...'

The following script has an option to divide the list into batches of 50 or fewer for each SendEmail operation and will send a Base 64 encoded MIME object to a list of recipients. The headers of the message are always the same for every recipient because the headers are defined within the MIME object, which is obtained from running the previous script With SendEmail, the Destination argument does not define the To or Cc headers.

#!/bin/bash

# Replace these variables with your own values
region="us-east-1"

# List of recipients, one per line. Defaults to SES mailbox simulator addresses (https://docs.aws.amazon.com/ses/latest/dg/send-an-email-from-console.html#send-email-simulator-how-to-use)
# These are the actual envelope recipients who will get the above email in their inbox. The To and Cc addresses set above will be displayed, not these.
recipients=(
"[email protected]"
# ...
"[email protected]"
)

# Raw message content
# Paste the base64 encoded message blob that is returned from the python script (the string within b'')
content=''

# Maximum number of recipients per batch
# Increase batch_size up to 50 if your use case requires every recipient have the same message-it. This sacrifices observability into deliverability metrics.
batch_size=1

# Send an email to batch size of 1 to 50 recipients
recipients_count=${#recipients[@]}
echo $recipients_count
for ((i=0; i<$recipients_count; i+=batch_size)); do
to_addresses="${recipients[@]:${i}:${batch_size}}"
to_addresses="${to_addresses// /,}"
aws sesv2 send-email \
--destination "ToAddresses=$to_addresses" \
--content "Raw={Data='$content'}" \
--region "$region"
done

# The output will look similar to this, with a unique MessageID associated with each send-email.
#{
# "MessageId": "010001874ee5cdca-3fe4fb4b-4d36-4ae7-b4e4-cc7fae988a42-000000"
#}
#... 51 total message-ids
#{
# "MessageId": "010001874ee5d210-9225f471-e330-4f01-9044-63a941358477-000000"
#}

Screenshot of email client, viewing email sent by the above code. The sender of the email is “Descriptive Name”, the To recipient is Founder Name, and the President and Director are displayed as Cc addresses.

Remember: If you increase the batch size to greater than 1. Every recipient in each batch will have a message with the same message-id and will all be treated the same for event processing.

Running these scripts will have the effect of each team member receiving exactly the same looking message regardless of how many recipients were defined in each SendEmail Destination. The To and CC addresses were set in the email headers, but the actual envelope recipients were set in the API operation.

SES SendEmail and SendBulkEmail APIs

The latest version of SES API (version 2) offers SendEmail and SendBulkEmail APIs.

With SendBulkEmail, you can only use a pre-defined SES template while, with SendEmail, you can send any email format including raw, text, HTML and templates.

SendEmail operation can send a single email to one Destination (50 recipients across the To:, Cc:, and Bc: fields) while the SendBulkEmail operation can send 50 unique emails to 50 Destinations by leveraging a SES template.

Both operations have the capability to send templated emails, but the SendBulkEmail operation requires less computational resources. This is due to its ability to send emails to 50 Destinations using just a single API call.

Conclusion

In this blog post we discussed how message recipient addresses are displayed by email clients, how message delivery is defined by envelope recipients, and how email sending events are associated with the recipients. Defining multiple recipients in a message destination can lead to poor observability and therefore poor deliverability and should not be used unless your use case specifically requires it

Sending messages to one recipient at a time is a best practice and leads to the highest engagement with your recipients.

About the authors

	Jesse Thompson is an Email Deliverability Manager with the Amazon Simple Email Service team. His background is in enterprise IT development and operations, with a focus on email abuse mitigation and encouragement of authenticity practices with open standard protocols. Jesse’s favorite activity outside of technology is recreational curling.
	Samuel Wallan is a Software Development Engineer at AWS Simple Email Service. Within SES, Sam works on the Digital User Experience Deliverability team. In his free time, he enjoys hanging out with friends and staying fit.
	Farnam Farshneshani is a Technical Account Manager at AWS. He specializes in AWS Simple Email service and helps customers with operational and architectural issues. In his free time, he enjoys traveling and participating in various outdoor activities.
	Joydeep Dutta is a Senior Solutions architect at AWS. Joydeep enjoys working with AWS customers to migrate their workloads to the AWS Cloud, optimize for cost and help with architectural best practices. He is passionate about enterprise architecture to help reduce cost and complexity in the enterprise. He lives in New Jersey and enjoys listening to music and spending time in the outdoors in his spare time.

Manage Incoming Emails at Scale with Amazon SES

2023-06-20 Bruno Giorgini

Post Syndicated from Bruno Giorgini original https://aws.amazon.com/blogs/messaging-and-targeting/manage-incoming-emails-with-ses/

Introduction

Are you looking for an efficient way to handle incoming emails and streamline your email processing workflows? In this blog post, we’ll guide you through setting up Amazon Simple Email Service (SES) for incoming email, focusing on the setup, monitoring, and use of receipt rules to optimize your email handling.

Amazon SES is a powerful and flexible cloud-based email service that enables you to send and receive emails at scale, while ensuring high deliverability and maintaining compliance with email best practices. By using Amazon SES for incoming email, you can customize your email processing pipeline and seamlessly integrate with other AWS services such as Amazon S3, AWS Lambda, and Amazon SNS.

We’ll start by walking you through the process of verifying your domain and setting up DomainKeys Identified Mail (DKIM) to ensure your emails are secure and authenticated. Next, we’ll explain how to create and manage receipt rule sets and add receipt rules with various actions for different processing scenarios. We’ll also cover monitoring your email processing using Amazon CloudWatch metrics.

As we progress, we’ll dive into advanced topics such as conditional receipt rules and chaining receipt rules, which can help you build complex and tailored email processing workflows, including multi-tenant scenarios. By the end of this post, you’ll have a comprehensive understanding of how to harness the power of Amazon SES for your incoming email needs.

So, let’s get started on simplifying your incoming email processing with Amazon SES!

Setting up Amazon SES for email receiving

Identifying the AWS region

For new users of the Amazon Simple Email Service (SES) inbound feature, it’s important to understand that all AWS resources used for receiving email with Amazon SES, except for Amazon S3 buckets, need to be in the same AWS Region as the Amazon SES endpoint. This means that if you are using Amazon SES in a specific region, such as US West (Oregon), any additional resources like Amazon SNS topics, AWS KMS keys, and Lambda functions also need to be created in the same US West (Oregon) Region. Additionally, to successfully receive email with Amazon SES within a particular Region, you must create an active receipt rule set specifically in that Region. By adhering to these guidelines, new users can effectively configure and utilize the inbound feature of Amazon SES, ensuring seamless email reception and efficient management of related resources. Amazon SES only supports email receiving in certain AWS Regions. For a complete list of Regions where email receiving is supported, see Amazon Simple Email Service endpoints and quotas in the AWS General Reference.

Verifying your domain

Before you can start receiving emails with Amazon SES, you must verify your domain. Domain verification is a crucial step in the setup process, as it confirms your ownership of the domain and helps prevent unauthorized use. In this section, we’ll walk you through the process of verifying your domain in the Amazon SES console.

Sign in to the AWS Management Console and open the Amazon SES console.
In the navigation pane, under Configuration, choose Verified identities.
In the list of Identities section, choose Create identity.
Under Identity details, choose Domain as the Identity type field. You must have access to the domain’s DNS settings to complete the domain verification process.
Enter the name of the domain or subdomain in the Domain field.
- (Optional) If you want to Assign a default configuration set, or Use a custom MAIL FROM domain select the appropriate check box. For more information, see Using configuration sets in Amazon SES and Using a custom MAIL FROM domain.
You must configure DKIM as part of the domain verification process. For Advanced DKIM settings, ensure that the Enabled box is checked in the DKIM signatures field.
Choose Create identity.
This will generate a list of DNS records that you need to add to your domain’s DNS configuration. These can be found in the DomainKeys Identified Mail (DKIM) container, under Publish DNS records.

Publish DNS records
Add the generated DNS records to your domain’s DNS configuration. These records include a Legacy TXT record for domain verification and CNAME records for DKIM authentication. You may need to consult your domain registrar’s documentation for instructions on adding DNS records.
- For more information on domain verification and DKIM, you can refer to the official documentation and read our recent blog post about How to set up EasyDKIM for a new domain.
Once the DNS records have been added, return to the Amazon SES console and wait for your domain’s verification status to change from “Verification pending” to “Verified.” This process may take up to 72 hours, depending on your domain registrar’s DNS propagation time.

Publishing an MX record for Amazon SES email receiving

To enable email receiving with Amazon SES, you need to publish an MX (Mail Exchange) record in your domain’s DNS configuration. The MX record directs incoming emails to Amazon SES for processing. Follow these steps to publish the MX record:

Log in to your domain registrar or DNS management console.
Locate the DNS management section for your domain.
Create a new MX record by specifying the following details:
- Host/Name/Record: Leave this field blank or enter “@” to represent the root domain.
- Value/Points to/Target: Enter the value “10 inbound-smtp.[AWS Region].amazonaws.com“, replacing [AWS Region] with the AWS region where you are using Amazon SES for email receiving. For example, if you are using US West (Oregon) region, the value should be “10 inbound-smtp.us-west-2.amazonaws.com“.
- TTL (Time to Live): Set a TTL value according to your preference or leave it as the default.
Save the MX record.

Once the MX record is published with the correct value, incoming emails addressed to your domain will be routed to Amazon SES for processing. Remember to ensure that any other email-related resources, such as SNS topics or Lambda functions, are also created in the same AWS region as your Amazon SES endpoint.

For more detailed information on publishing MX records for Amazon SES email receiving, you can refer to the official documentation.

Creating a Receipt Rule set

A receipt rule set is a collection of rules that define how Amazon SES processes incoming emails for your domain. Each rule contains one or more actions that determine the processing flow of incoming emails. In this section, we’ll guide you through the process of creating a new receipt rule set in the Amazon SES console and activating it for your domain.

Sign in to the AWS Management Console and open the Amazon SES console.
In the navigation pane, under Configuration, choose Email receiving.
- Note: if you don’t see the Email receiving option in the menu, check again that you’re in fact in a region supporting this feature.
Under the Receipt rule sets tab in the Email receiving pane, choose Create rule set.
Enter a name for your new rule set in the Rule set name field. This name should be descriptive and easy to identify, such as “MyApp-IncomingEmail.”
After entering a unique name, choose Create rule set.
To activate the newly created rule set, choose Set as active next to your rule set’s name. This action will ensure that Amazon SES uses this rule set for processing incoming emails to your domain. Your new rule set will now be listed in the Active rule set section.

For more information on creating and managing receipt rule sets, you can refer to the official documentation.

In the next section, we’ll explore adding receipt rules to your rule set, which define the specific actions to be taken for incoming emails.

Adding Receipt Rules

Receipt rules define the specific actions that Amazon SES should take when processing incoming emails for your domain. Common actions include saving the email to an Amazon S3 bucket, invoking an AWS Lambda function, or publishing a notification to an Amazon SNS topic. In this section, we’ll guide you through the process of adding receipt rules to your rule set in the Amazon SES console and provide examples of when to use each action.

Sign in to the AWS Management Console and open the Amazon SES console.
In the navigation pane, under Configuration, choose Email receiving.
Under the Email receiving pane, in the Receipt rule sets tab, select the name of your active rule set from the All rule sets section. This will navigate to the details page for that rule set.
Choose Create rule to begin creating a new receipt rule.
On the Define rule settings page, under Receipt rule details, enter a unique Rule name.
- For Status, only clear the Enabled checkbox if you don’t want to run this rule after creation.
- (Optional) For Transport Layer Security (TLS), by selecting Required you can enforce a specific TLS policy for incoming emails that match this rule. By default, Amazon SES will use the Optional policy, which means it will attempt to use TLS but will not require it.
- For Spam and virus scanning, only clear the Enabled checkbox if you don’t want Amazon SES to scan incoming messages for spam and viruses.
After entering a unique rule name, choose Next.
On the Add recipients conditions page, under Recipients conditions, use the following procedure to specify one or more recipient conditions. You can have a maximum of 100 recipient conditions per receipt rule.
- Under Recipient condition, specify the email addresses or domains that this rule should apply to. You can use wildcards to match multiple addresses or domains. For example, you can enter example.com and .example.com to apply the rule to all email addresses within the example.com domain and within all of its subdomains.
- Repeat this step for each recipient condition you want to add. When you finish adding recipient conditions, choose Next.
On the Add actions page, open the Add new action menu and select the desired action from the list, such as Deliver to S3 bucket, Invoke AWS Lambda function, or Publish to Amazon SNS topic. Configure the selected action’s settings as required.
- Deliver to S3 bucket: Choose this action if you’re expecting emails with large attachments, need to store emails for archival purposes, or plan to process emails using other AWS services that integrate with Amazon S3. You’ll need to specify the Amazon S3 bucket where the incoming emails should be stored.
- Invoke AWS Lambda function: Choose this action if you want to process incoming emails using custom logic, such as filtering, parsing, or modifying the email content. You’ll need to specify the AWS Lambda function that should be invoked when an incoming email matches this rule.
- Publish to Amazon SNS topic: Choose this action if you’re processing smaller emails or want to receive real-time notifications when an email arrives. You’ll need to specify the Amazon SNS topic where notifications should be published.
- For more information and additional actions, see the Action options section of the Developer Guide.
Once configured, choose Next to proceed to the Review page.
On the Review page, review the settings and actions of the rule. If you need to make changes, choose the Edit option.
When finished, choose Create rule to add the new receipt rule to your rule set. The rule will now be applied to incoming emails that match the specified recipient conditions.

You can create multiple receipt rules within a rule set, each with different actions and conditions. Amazon SES will apply the rules in the order they appear in the rule set. For more information on creating and managing receipt rules, you can refer to the official documentation.

Monitoring your incoming email

Configuring Amazon CloudWatch metrics

Once you have enabled email receiving in Amazon SES and created receipt rules for your emails, you can monitor and view the metrics using Amazon CloudWatch. Follow these steps to configure Amazon CloudWatch metrics for Amazon SES email receiving:

Open the Amazon CloudWatch console.
Navigate to the Metrics section and select All metrics.
In the list of available metrics, locate and select SES to view SES-related metrics.
Expand the Receipt Rule Set Metrics and Receipt Rule Metrics sections to access the specific metrics for your receipt rule sets and rules.
Under Receipt Rule Set Metrics, you will find the following metrics:
- “Received”: Indicates whether SES successfully received a message that has at least one rule applying. The metric value is always 1.
- “PublishSuccess”: Indicates whether SES successfully executed all rules within a rule set.
- “PublishFailure”: Indicates if SES encountered an error while executing rules within a rule set. The error may allow for retrying the execution.
- “PublishExpired”: Indicates that SES will no longer retry executing the rules within a rule set after four hours.

These metrics can be filtered by the dimension RuleSetName to obtain data specific to individual rule sets.

Under Receipt Rule Metrics, you will find the following metrics:
- “Received”: Indicates whether SES successfully received a message and will try to process the applied rule. The metric value is always 1.
- “PublishSuccess”: Indicates whether SES successfully executed a rule that applies to the received message.
- “PublishFailure”: Indicates if SES encountered an error while executing the actions in a rule. The error may allow for retrying the execution.
- “PublishExpired”: Indicates that SES will no longer retry executing the actions of a rule after four hours.

These metrics can be filtered by the dimension RuleName to obtain data specific to individual rules.

Note that the metrics will only appear in the CloudWatch console if you have enabled email receiving, created receipt rules, and received mail that matches any of your rules.
Keep in mind that changes made to fix your receipt rule set will only apply to emails received by Amazon SES after the update. Emails are always evaluated against the receipt rule set in place at the time of receipt.

Amazon SES also provides an Automatic Dashboard for SES in the CloudWatch console, which offers a preconfigured set of SES metrics and alarms to monitor your email sending and receiving activity. This dashboard provides a consolidated view of key metrics, making it easier to track the performance and health of your Amazon SES environment.

By configuring Amazon CloudWatch metrics, you can gain valuable insights into the performance and execution of your receipt rule sets and rules within Amazon SES. For more detailed information on viewing metrics for Amazon SES email receiving using Amazon CloudWatch, refer to the official documentation.

Using receipt rules effectively

Chaining Receipt Rules

Chaining receipt rules enable you to create sophisticated email processing workflows by linking multiple rules together, allowing each rule to apply specific actions based on the outcome of the previous rule. This advanced technique can help you achieve greater flexibility and precision in handling your incoming emails with Amazon SES. In this section, we’ll explain how to create chained receipt rules and provide examples of common use cases.

Sign in to the AWS Management Console and open the Amazon SES console.
Under the Email receiving pane, in the Receipt rule sets tab, select the name of your active rule set from the All rule sets section
Review the existing rules in your rule set and ensure that they are ordered correctly. Chaining relies on the order of the rules, as each rule’s conditions and actions are evaluated sequentially. Under the Reorder tab, the rule orders can be modified by selecting the corresponding arrow associated with each.
To chain additional rules, follow the steps previously outlined in the Adding Receipt Rules section and adjust the rule orders as necessary.

Chaining receipt rules can help you build complex email processing workflows with Amazon SES. Some common use cases include:

Executing multiple filtering criteria in an order that you specify. For example, adding a specific header value and then sending to additional AWS services such as Amazon S3, Amazon SNS, or AWS Lambda.
Creating multi-stage processing pipelines, where the output of one action (e.g., saving an email to Amazon S3) is used as the input for the next action (e.g., processing the email with AWS Lambda).
Implementing fallback actions, where the first rule in the chain attempts a specific action (e.g., saving an email to a primary S3 bucket), and if it fails, the next rule in the chain applies a different action (e.g., saving the email to a secondary S3 bucket).

The following figure shows how receipt rules, rule sets, and actions relate to each other.

SES Chaining multiple rules in a rule set

For more information on creating and managing receipt rules, you can refer to the official documentation.

Handling the 200 Receipt Rules per Rule Set limit

For each AWS account, Amazon SES imposes a limit of 200 receipt rules per receipt rule set. While this limit is sufficient for most use cases, there might be situations where you need to process a higher volume of incoming emails with more complex rule sets. These are some strategies to work around the 200 receipt rule limit using Amazon SES and other AWS services:

Utilize rule chaining: As mentioned earlier, chaining receipt rules allows you to link multiple rules together, effectively extending the number of actions you can perform for a single email. By chaining rules, you can create more complex processing workflows without exceeding the 200 rule limit.
Combine rules with actions: Instead of creating separate rules for each scenario, consider combining multiple actions within a single rule. This approach can help you reduce the total number of rules while still catering to various email processing requirements.
Use AWS Lambda for custom processing: Leverage AWS Lambda to perform custom processing on incoming emails. By incorporating Lambda functions in your receipt rules, you can handle more complex processing tasks without increasing the number of rules. This approach also allows you to offload some processing logic from Amazon SES to Lambda, providing additional flexibility.
Consolidate similar actions: If you have several rules performing similar actions, it is advisable to consolidate them into a single rule with multiple actions. This consolidation can help you reduce the total number of rules while maintaining the desired functionality.
Evaluate rule usage: Regularly review and evaluate your existing receipt rules to identify any rules that are no longer in use or can be optimized. Removing or consolidating unnecessary rules can help you stay within the 200 rule limit while still addressing your email processing requirements.

By implementing these strategies, you can effectively work around the 200 receipt rule limit in Amazon SES and build more complex email processing workflows to cater to your specific needs. Remember to monitor and optimize your rule sets regularly to make the most of the available resources and maintain efficient email processing.

For more information on the inbound quotas and limits in Amazon SES, you can refer to the official AWS documentation at Quotas related to email receiving.

Best Practices for multi-tenant scenarios

When dealing with multi-tenant scenarios in your application, it’s crucial to manage incoming emails efficiently to ensure smooth operation and a seamless experience for your users. In this section, we’ll provide best practices to handle incoming emails in multi-tenant environments using Amazon SES.

In a multi-tenant scenario, where multiple customers or tenants share a single AWS account, it’s important to consider the limit of 200 receipt rules per receipt rule set imposed by Amazon SES. To ensure compliance with this limit and maintain optimal email processing, the following practices are recommended:

Segregate tenants using email subdomains: Create unique subdomains for each tenant and route their incoming emails accordingly. This approach makes it easier to manage email processing rules and helps isolate tenants from potential issues.
Create separate rule sets for each tenant: By creating dedicated rule sets for each tenant, you can maintain better control over email processing rules and actions specific to their needs. This can simplify management and make it easier to update rules for individual tenants without affecting others.
Use tags to identify tenant-specific emails: Apply tags to incoming emails using the AddHeader action in your receipt rules. These tags can include tenant-specific identifiers, which will help you route and process emails correctly. You can later use these tags in other AWS services (e.g., AWS Lambda) to process tenant-specific emails.
Leverage conditional receipt rules: Utilize conditional receipt rules to apply tenant-specific processing based on email headers, recipients, or other criteria. This way, you can ensure that the right actions are taken for each tenant’s incoming emails.
Monitor tenant-specific metrics: Configure Amazon CloudWatch metrics and alarms for each tenant to track their email processing performance separately. This enables you to keep a close eye on individual tenants and take appropriate actions when needed.
Implement rate limiting: To prevent tenants from overwhelming your email processing pipeline, consider implementing rate limiting based on the number of incoming emails per tenant. This can help ensure fair resource allocation and prevent potential abuse.
Ensure security and privacy: Always encrypt tenant data at rest and in transit, and follow best practices for data protection and privacy. Consider using AWS Key Management Service (KMS) to manage encryption keys for each tenant.
Test and validate rule sets: Before deploying rule sets for tenants, thoroughly test and validate them to ensure they function as intended. This can help prevent unexpected behavior and maintain a high level of service quality.

By following these best practices for handling incoming emails in multi-tenant scenarios with Amazon SES, you can ensure a robust and efficient email processing pipeline that caters to each tenant’s unique requirements. As you continue to work with Amazon SES in multi-tenant environments, stay up to date with AWS documentation and best practices to further optimize your email processing workflows.

Conclusion

In this blog post, we’ve explored how to set up Amazon Simple Email Service (SES) for incoming email processing using receipt rules, rule sets, and various actions. We’ve covered domain verification, DKIM setup, creating and managing rule sets, adding receipt rules, and configuring Amazon CloudWatch metrics and alarms. We’ve also delved into advanced topics such as chaining receipt rules for more complex email processing workflows.

By following this guide, you can effectively leverage Amazon SES to process and manage your incoming emails, optimizing your email workflows, and maintaining high email deliverability standards. With Amazon SES, you can customize your email processing pipeline to meet your specific needs and seamlessly integrate with other AWS services such as Amazon S3, AWS Lambda, Amazon SNS, and Amazon CloudWatch.

In future blog posts, we will explore monitoring and alerting in more detail, providing you with additional insights on how to effectively monitor your email processing pipelines and set up alerts for critical events. Stay tuned for more information on this important aspect of managing your email infrastructure.

As you continue to work with Amazon SES and its email receiving capabilities, remember to review AWS best practices and documentation to stay up to date with new features and improvements. Don’t hesitate to experiment with different rule sets, actions, and conditions to find the perfect email processing solution for your use case.

A Guide to Maintaining a Healthy Email Database

2023-06-08 nnatri

Post Syndicated from nnatri original https://aws.amazon.com/blogs/messaging-and-targeting/guide-to-maintaining-healthy-email-database/

Introduction

In the digital age, email remains a powerful tool for businesses to communicate with their customers. Whether it’s for marketing campaigns, customer service updates, or important announcements, a well-maintained email database is crucial for ensuring that your messages reach their intended recipients. However, managing an email database is not just about storing email addresses. It involves keeping the database healthy, which means it’s up-to-date, accurate, and filled with engaged subscribers.

Amazon Simple Email Service (SES) offers robust features that help businesses manage their email environments effectively. Trusted by customers such as Amazon.com, Netflix, Duolingo and Reddit, SES helps customers deliver high-volume email campaigns of hundreds of billions of emails per year. Introduced in 2020, the list and subscription management feature of Amazon SES has added a new dimension to email database management, thereby reducing effort and time-to-value of managing a subscription list by allowing you to manage your list of contacts via its REST API, SDK or AWS CLI.

In this blog post, we will delve into the world of email database management in Amazon SES. You will explore two ways to manage your email database: building out your own email database functionality and using the built-in list and subscription management service. You will also learn the pros and cons of each approach and provide examples of customer use cases that would benefit from each approach. Regardless of the approach you ultimately decide to take, the blog will also share updated strategies for email database management to help with improving deliverability and customer engagement.

This guide is designed to help you navigate the complexities of email database management and make informed decisions that best suit your business needs. So, whether you’re new to Amazon SES or looking to optimize your existing email database management practices, this guide is for you. Let’s get started!

Email Database Management in Amazon SES

Amazon Simple Email Service (SES) offers two primary ways to manage your email database: building out your own email database functionality and using the built-in list and subscription management service. Each approach has its own set of advantages and potential drawbacks, and the best choice depends on your specific use case and business needs.

Building Out Your Email Database Functionality

When you choose to build out your own email database functionality, you have the flexibility to customize the database to suit your specific needs and leverage SES’ scalability as an email channel to send email at high volumes to your customer. Depending on the business requirement, the customizations could involve creating custom fields for subscriber data, implementing complex logic for categorizing and segmenting users, or integrating with other systems in your tech stack.

Using the Built-in List and Subscription Management Service

Alternatively, you can look at Amazon SES’s built-in list and subscription management service, which offers a ready-made solution for managing your email database. It handles tasks such as managing subscriptions to different topics and maintaining your customer email database through contact lists. Additionally, you can insert up to two links per email to the subscription preference page, which allow users to manage their topic preferences within Amazon SES.

SubscriptionPage

The non-configurable subscription page will automatically populate the customer’s current subscribed topic and allow setting of granular topic’s preferences. More information on how to configure that can be found here.

The following table should serve as a guideline to help you with deciding your approach for Email Database Management.

	Building Your Own Email Database Functionality	Using Built-in List and Subscription Management Service
Pros	Customization: Full control over the database structure and functionality, allowing for tailoring to specific needs. This includes creating custom fields for subscriber data, implementing own algorithms for handling bounces and complaints, and integrating with other systems in the tech stack. Integration: Flexible flow of data across the business due to the ability to integrate the email database with other systems in the tech stack. You’ve already built your own email database or have one in mind which supports querying, building that database external to Amazon SES would make for a more customizable implementation. Data Ownership: When you manage your own database, you have full ownership and control over your data. This can be important for businesses with strict data governance or regulatory requirements.	Ease of Use: The built-in service provides readily-available API to create, update and delete contacts. These operations are also available via REST API, AWS CLI and SDK. Once you’ve set up the subscription topics and contact lists, you can leverage the preference center to allow your customers to easily sub/unsubscribe from different topics. Cost-Effective: More cost-effective than building own functionality as it requires less time and resources. The built-in service is also available free of charge unlike building out own infrastructure which would require ongoing infrastructure service costs.
Cons	Time and Resources: Building your own email database functionality requires a significant investment of time and resources. This includes the initial setup of the database, designing the schema, setting up the servers, and configuring the database software. Additionally, you’ll need to develop the functionality for managing subscriptions, and database cleanup in upon receiving bounces and complaints. Databases require ongoing maintenance to ensure they remain operational and efficient. This includes tasks like updating the database software, managing backups, optimizing queries, and scaling the database as your subscriber base grows. Complexity: As your subscriber base grows, managing your own email database can become increasingly complex. You’ll need to handle more data, which can slow down queries and make the database more difficult to manage. You’ll also need to deal with more complex issues like data integrity, redundancy, and normalization. Additionally, as you add more features to your email database functionality, the codebase can become more complex, making it harder to maintain and debug. Security: When you manage your own email database, you’re responsible for its security. This includes protecting the data from unauthorized access, ensuring the confidentiality of your subscribers’ information, and complying with data protection regulations. You’ll need to implement security measures like encryption, access controls, and regular security audits. If your database is compromised, it could lead to data loss or a breach of your subscribers’ privacy, which could damage your reputation and potentially lead to legal consequences.	Limited Customization: The built-in service may not offer the same level of customization as building own functionality. It may not meet all needs if there are specific requirements. For example, the preference center management page cannot be customized. Dependence: Using the built-in service means you’re reliant on Amazon SES for your email database management. If the service experiences downtime or issues, it could impact your ability to manage your email database. This could potentially disrupt your email campaigns and affect your relationship with your subscribers. Furthermore, if you decide to switch to a different email service provider in the future, migrating your email database from the built-in service could be a complex and time-consuming process. Additionally, if your email database needs to be accessed or manipulated by other systems in your tech stack, this dependency on Amazon SES could complicate the integration process and limit your flexibility.
Customer Use Cases	Best suited for businesses with specific needs that aren’t met by standard list management services, or those who wish to integrate their email database with other systems. For example, a large e-commerce company might choose to build out their own email database functionality to integrate with their customer relationship management (CRM) and inventory systems.	Ideal for small to medium-sized businesses that need a straightforward, cost-effective solution for managing their email database. It’s also a good fit for businesses without the resources or technical expertise to build their own email database functionality.

Strategies for Email Database Management with Amazon Simple Email Service

Once you’ve made the decision on whether to manage your email database within Amazon SES or build your own, that’s only half of the equation. It’s important to recognize that your email databases will only work best to serve the business needs when you have processes in place to maintain them. In this section, let’s go through some of the best practices on how to do so.

Maintaining email list hygiene:
- Both Amazon SES and a custom-built email database require maintaining a healthy email list. This involves regularly cleaning your list to remove invalid email addresses, hard bounces, and unengaged subscribers. With Amazon SES, the process to handle hard bounces and complaints is automated.
- With a custom-built email database, you have more control over how and when this cleaning occurs. Rather than focusing on only email addresses that either hard bounces or complained, you can remove unengaged users. Every business will have their own definition of an un-engaged users based on business needs. Regardless, you will need to store the engagement attribute (e.g. days since last interaction). This will be simpler to architect in an external database which supports querying and bulk modification.
Managing Subscriptions:
- With Amazon SES, you can easily manage subscriptions using the built-in functionality. This includes adding new subscribers, removing unsubscribed users, and updating user topic preferences. However, you will not be able to customize the look-and-feel of your subscription preference pages.
- If you build your own email database, you’ll need to create your own system for managing subscriptions, which could require significant time and resources. The trade-off is that you can fully customize your subscription management system to showcase your branding on the subscription preference page and also handle custom logic for subscription/unsubscription.
Encouraging Engagement: Low engagement rates can indicate that your recipients are not interested in your content. To stimulate action, you can include a survey in the email, ask for feedback, or run a giveaway. You can then filter out inactive subscribers who still aren’t interacting with your emails. For engaged subscribers, you can segment these audiences into sub-groups by preference and send tailored email marketing campaigns. Before removing less active subscribers, consider what other kinds of content you could provide that might be more appealing. Unengaged subscribers can sometimes be re-engaged with the right offer, such as a free gift, a special perk, or exclusive content.
Renewing Opt-In: For your disengaged subscribers, send a re-optin campaign and remove them if they don’t re-subscribe. Be transparent! Notify inactive subscribers that you’ve noticed their lack of engagement and let them know that you don’t want to clutter their inbox if they’re not interested. Ask them if they want to continue to receive emails with a clear call-to-action button that will re-sign them up for future emails.
Making It Easy to Unsubscribe: Including an easy-to-find unsubscribe button and a one-step opt-out process won’t encourage subscribers to leave if you’re giving them a reason to stay. If recipients feel like they can’t leave, they’ll just mark your emails as spam, which counts as a big strike against your sender reputation.

Remember, effective email database management is a continuous process that requires regular attention and maintenance. By following these best practices, you can maximize the effectiveness of your email marketing efforts and build strong relationships with your subscribers.

Conclusion

In conclusion, maintaining a healthy email database is a critical aspect of successful email marketing. Whether you choose to build out your own email database functionality or use Amazon SES’s built-in list and subscription management service, it’s important to understand the pros and cons of each approach and align your decision with your business needs.

Building your own email database functionality offers the advantage of customization and integration with other systems in your tech stack. However, it requires significant time, resources, and technical expertise. On the other hand, Amazon SES’s built-in service is easy to use, cost-effective, and handles many complexities of email database management, but it may not offer the same level of customization.

Regardless of the approach you choose, following best practices for email database management is essential. This includes handling bounces and complaints, managing subscriptions, encouraging engagement, sending re-engagement email campaigns, renewing opt-ins, and making it easy to unsubscribe.

These practices will help you maintain a healthy email list, improve engagement rates, and ultimately, enhance the effectiveness of your email marketing efforts.It’s important to stay updated with the latest trends and strategies in email database management. So, keep exploring, learning, and implementing the best practices that suit your business needs.

For more information on Amazon SES and its features, visit the Amazon SES Documentation. Here, you’ll find comprehensive guides, tutorials, and API references to help you make the most of Amazon SES.

How to implement multi tenancy with Amazon SES

2023-05-25 satyaso

Post Syndicated from satyaso original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-manage-email-sending-for-multiple-end-customers-using-amazon-ses/

In this blog post, you will learn how to design multi-tenancy with Amazon SES, as well as the fundamental best practices for implementing a multi-tenant architecture that can effectively handle bulk the email sending needs of your downstream customers.

Amazon Simple Email Service (SES) is utilized by customers across various industries to send emails to their recipients. Often, they need to send emails on behalf of their downstream customers or for other business divisions. Organizations commonly refer to these use cases as “multi-tenant email sending practices. To implement email sending multi-tenancy practices (i.e. to send bulk emails on behalf of end customers), Amazon SES customers need to adopt an architecture that enables them to effectively meet the email sending needs of thousands of downstream customers while also ensuring that the email sending reputation of each customer or the tenant is isolated.

Use cases

Onboard multiple brands from different Business units (BUs) with different domains.
Separate marketing and transaction tenants.
ISV Customer’s requirement to segregate email sending reputation of their end customers.
Domain management via configuration sets.
Track individual customer’s email sending repurataion and control their email sending process.

Prerequisites

For this post, you should be familiar with the following:

Solution Overview

In the email ecosystem, domain and IP reputation are critical in getting emails delivered to the inbox. Tenants in a multi-tenant scenario might be unique businesses or an internal team (eg marketing team, customer service team and so on). Because the maturity of each tenant varies greatly, implementing a multi-tenant environment may be increasingly complicated and difficult. While one tenant may have a well-validated and highly-engaged recipient list, another tenant may have an untrusted email recipient list, and sending emails to such email addresses may result in bounces or spam, lowering the IP and domain reputation. So, organizations have to build safe guards to prevent an unsophisticated sender or a bad actor from impacting the other tenants.

To better understand multi-tenancy, let us first look at how Amazon SES sends emails. Any emails sent via Amazon SES to end users are sent using IP addresses that have been mapped within Amazon SES. Amazon SES offers two types of IP addresses: shared IP addresses and dedicated IP addresses. (Currently Amazon SES offers two kinds of dedicated IPs, which are 1/ Standard dedicated IPs, 2/ Managed dedicated IPs). Shared IPs are shared across many SES customers, and all your emails are sent using shared IP addresses by default unless you have requested for dedicated IPs. Dedicated IP addresses/addresses are designated for a single customer or tenant, where the tenant might be a business unit within the customer’s own eco system or a downstream customer of an ISV.

If a customer is using shared IPs to send email via SES and trying to achieve multi tenancy, then they can do so to segregate the business functions of multiple tenants such as tenant tagging, SES event destination routing, cost allocation for each tenant, and so on; but it won’t help to manage or isolate email sending reputation from one tenant to another. This is because; these shared IPs are mapped to an AWS region and incase one rogue tenant is trying to send spam emails then it will impact other customers in the same region who are using same set of shared IPs.

If you are an Amazon SES user and wish to separate the reputation of one end-customer from another then dedicated IPs are the ideal solution. Dedicated IP or Dedicated IPs (also known as dedicated IP pool) can be assigned to a tenant, and the email sending reputation for that tenant can be readily isolated from that of another tenant. If tenant one is a problematic sender and internet service providers (ISPs) such as Gmail, Hotmail, Yahoo and, so on, flags the respective domain or IPs, the reputation of the other tenants’ domains and IPs are unaffected since they are mutually exclusive.

Amazon SES supports multi-tenancy primarily through two constructs: 1/configuration sets, 2/Dedicate IP pools. Configuration sets are setup rules that are applicable to your verified identities, whereas dedicated IP pool is to group dedicated IPs into a pool, which can then be mapped to a configuration set, such that the respective Identity/Identities may only utilize the same IP Pool without affecting other tenants. Let’s now witness a simplified architecture view.

Multi tenancy using a single AWS account

In this architecture, if you notice tenant 1, tenant 2 and tenant 3 are using the distinct configurations with respective dedicated IPs while tenant 4 is using shared IPs. i.e. the tenants can chose which configuration sets needs to be used for their domain. This provides customers capability to achieve multi tenancy.

Amazon SES multi tenancy – best practices

Always proactively reach out to your account team or raise a support case under “service limit increase” category informing that you will be sending on behalf of tens of thousands of customers. This will help AWS in rightly setup limits within your account and be cognizant of your sending patterns.

While the architecture described above will most of the time help Amazon SES users manage multiple end customers effectively, in rare cases; Amazon SES users may receive a notification from AWS support stating that their Amazon SES account is being reviewed. This indicates that your Amazon SES account is being used to send problematic email to end recipients, or that the account has been paused (if you haven’t reacted proactively upon controlling the faulty senders within the review timeframe), which means you can’t send email from your SES account because your spam or complaint rate has exceeded a certain threshold. These type of situations occurs because, Amazon SES sanitization process is implemented at the AWS account level by default. So, even if any of the tenants using a dedicated IP or a dedicated IP pool and their spam or complaint rates exceed the approved SES limit, Amazon SES sends a notification to the account admin, flagging the concern in their account. In such cases, it is recommended to implement a process known as “automatically pausing email sending for a configuration set“. You can configure Amazon SES to export reputation metrics that are specific to emails that are sent using a specific configuration set to Amazon CloudWatch. You can then use these metrics to create CloudWatch alarms that are specific to those configuration sets. When these alarms exceed certain thresholds, you can automatically pause the sending of emails that use the specified configuration sets, without impacting the overall email sending capabilities of your Amazon SES account.

If you are an Enterprise ISV customer and you have tens of thousands of downstream customers then there is a possibility that you will hit Amazon SES provided maximum quota. In those scenarios you have two options; 1/ Ask for an exception for your AWS SES account – In this approach, you need to request AWS to increase your quota applicable for the existing account to a higher threshold and depending upon your previous usage and reputation AWS shall increase your account limit to accommodate more customers/tenants. To do this you need to raise an AWS support case under “service limit increase” and present your requirement on why you want to increase your Amazon SES account quota to a higher limit. There is no guaranty that the exception will always be granted. If your exception request is denied, you must proceed to the second option, which is to 2/ segment your customers across multiple AWS accounts. In this approach, you must calculate your customer base ahead of time and distribute your downstream customers across multiple accounts within the same AWS region in order to set up their email sending mechanism using SES. To better understand option 2, refer to the architecture diagram below.

Multi tenancy using multiple AWS account

In the above architecture various tenants are connecting to Amazon SES in different AWS accounts to implement multi tenancy. Email event responses can be taken back to a central data lake located in the same AWS region or in different region. Furthermore, as shown in the diagram above, all AWS accounts mapped to different tenants are under a Parent AWS account; this hierarchical structure is known as AWS Organizations. it is recommended to use AWS Organizations which enables you to consolidate multiple AWS accounts into an organization that you create and centrally manage. It helps in security and compliance guide lines, managing consolidated billing for all the child accounts.

Conclusion

Appropriate multi-tenancy implementation within Amazon SES not only helps you manage end-customer reputation but also aids in tracking usage of each user independently from one another. In this post, we have showcased how Amazon SES users can utilize Amazon SES to manage large number of end customer, what are the design best practices to implement multi-tenant architecture with Amazon SES.

Satyasovan Tripathy works at Amazon Web Services as a Senior Specialist Solution Architect. He is based in Bengaluru, India, and specialises on the AWS customer developer service product portfolio. He likes reading and travelling outside of work.

Choosing the Right Domain for Optimal Deliverability with Amazon SES

2023-05-18 komaio

Post Syndicated from komaio original https://aws.amazon.com/blogs/messaging-and-targeting/choosing-the-right-domain-for-optimal-deliverability-with-amazon-ses/

Choosing the Right Domain for Optimal Deliverability with Amazon SES

As a sender, selecting the right domain for the visible From header of your outbound messages is crucial for optimal deliverability. In this blog post, we will guide you through the process of choosing the best domain to use with Amazon Simple Email Service (SES)

Understanding domain selection and its impact on deliverability

With SES, you can create an identity at the domain level or you can create an email address identity. Both types of verified identities permit SES to use the email address in the From header of your outbound messages. You should only use email address identities for testing purposes, and you should use a domain identity to achieve optimal deliverability.

Choosing the right email domain is important for deliverability for the following reasons:

The domain carries a connotation to the brand associated with the content and purpose of the message.
Mail receiving organizations are moving towards domain-based reputational models; away from IP-based reputation.
Because the email address is a common target for forgery, domain owners are increasingly publishing policies to control who can and cannot use their domains.

The key takeaway from this blog is that you must be aware of the domain owner’s preference when choosing an identity to use with SES. If you do not have a relationship with the domain owner then you should plan on using your own domain for any email you send from SES.

Let’s dive deep into the technical reasons behind these recommendations.

What is DMARC?

Domain-based Message Authentication, Reporting, and Conformance (DMARC) is a domain-based protocol for authenticating outbound email and for controlling how unauthenticated outbound email should be handled by the mail receiving organization. DMARC has been around for over a decade and has been covered by this blog in the past.

DMARC permits the owner of an email author’s domain name to enable verification of the domain’s use. Mail receiving organizations can use this information when evaluating handling choices for incoming mail. You, as a sender, authenticate your email using DKIM and SPF.

DKIM works by applying a cryptographic signature to outbound messages. Mail receiving organizations will use the public key associated with the signing key that was used to verify the signature. The public key is stored in the DNS.
SPF works by defining the IP addresses permitted to send email as the MAIL FROM domain. The record of IP addresses is stored in the DNS. The MAIL FROM domain is not the same domain as the domain in the From header of messages sent via SES. It is either domain within amazonses.com or it is a custom MAIL FROM domain that is a subdomain of the verified domain identity. Read more about SPF and Amazon SES.

A message passes the domain’s DMARC policy when the evaluation DKIM or SPF indicate that the message is authenticated with an identifier that matches (or is a subdomain of) the domain in the visible From header.

How can I look up the domain’s DMARC policy?

You must be aware of the DMARC policy of the domain in which your SES identities reside. The domain owner may be using DMARC to protect the domain from forgery by unauthenticated sources. If you are the domain owner, you can use this method to confirm your domain’s current DMARC policy.

You can look up the domain’s DMARC policy in the following ways:

Perform a DNS query of type TXT against the hostname called _dmarc.<domain>. For example, you can use the ‘dig’ or ‘nslookup’ command on your computer, or make the same query using a web-based public DNS resolver, such as https://dns.google/
Use a 3rd party tool such as:

https://tools.wordtothewise.com/dmarc/
https://mxtoolbox.com/dmarc.aspx
https://dmarcian.com/dmarc-inspector/

The “p” tag in the DMARC record indicates the domain’s policy.

How does the domain’s policy affect how I can use it with SES?

This section will cover each policy scenario and provide guidance to your usage of the domain with SES.

Policy	How to Interpret	You have verified the domain identity with EasyDKIM	You have only email address identities with the domain
No DMARC record	The domain owner has not published a DMARC policy. They may not yet be aware of DMARC	There is no DMARC policy for mail receiving organizations to apply. Your messages are authenticated with DKIM, so mail receiving organization may leverage a domain-based reputational model for your email.	There is no DMARC policy for mail receiving organizations to apply. Your messages are not authenticated, so reputation remains solely based on IP.
none	The domain owner is evaluating the DMARC reports that the mail receiving organizations send to the domain owner, but has requested the mail receiving organizations not use DMARC policy logic to evaluate incoming email.	There is no DMARC policy for mail receiving organizations to apply. Your messages are authenticated with DKIM, so mail receiving organization may leverage a domain-based reputational model for your email.	There is no DMARC policy for mail receiving organizations to apply. Your messages are not authenticated, so reputation remains solely based on IP.
quarantine	The domain owner has instructed mail receiving organizations to send any non-authenticated email to a quarantine or to the Junk Mail folders of the recipients.	Your messages are authenticated with DKIM and will not be subjected to the domain’s DMARC policy.	Mail receiving organizations may not deliver your messages to the inboxes of your intended recipients.
reject	The domain owner has instructed mail receiving organizations to reject any non-authenticated email sending from the domain.	Your messages are authenticated with DKIM and will not be subjected to the domain’s DMARC policy.	Mail receiving organizations may reject these messages which will result in ‘bounce’ events within SES.

Other considerations

If the domain has a none or quarantine policy, you must be aware that the domain owner may have a plan to migrate to a more restrictive policy without consulting with you. This will affect your deliverability in the form of low inboxing/open rates, or high bounce rates. You should consult with the domain owner to determine if they recommend an alternative domain for your email use case.

Not all mail receiving organizations enforce DMARC policies. Some may use their own logic, such as quarantining messages that fail a reject policy. Some may use DMARC logic to build a domain-based reputational model based on your sending patterns even if you do not publish a policy. For example, here is a blueprint showing how you can set up custom filtering logic with SES Inbound.

If you have verified the domain identity with the legacy TXT record method, you must sign your email using a DKIM signature. The DKIM records in the DNS must be within the same domain as the domain in the From header of the messages you are signing.

If you have the domain identity verified with EasyDKIM and you also have email address identities verified within the same domain, then the email address identities will inherit the DKIM settings from the domain identity. Your email will be authenticated with DKIM and will not be subjected to the domain’s DMARC policy.

Can I use SPF instead of DKIM to align to the domain’s DMARC policy?

Messages can also pass a DMARC policy using SPF in addition to DKIM. This is enabled through the use of a custom MAIL FROM domain. The custom MAIL FROM domain needs to be a subdomain of the SES identity and the SES domain identity’s DMARC policy must not be set to strict domain alignment due to the way SES handles feedback forwarding. The domain owner enables a custom MAIL FROM domain by publishing records in the DNS. There is no way to authenticate email without publishing records in the DNS. Read Choosing a MAIL FROM domain to learn more.

The recommended approach is to use EasyDKIM primarily, and optionally enable a custom MAIL FROM domain as an additive form of authentication.

What should I do if I am not the domain owner?

The process of enabling DKIM and SPF authentication involves publishing DNS records within the domain. Only the domain owner may modify DNS for their domain. If you are not the domain owner, here are some alternative solutions.

Option 1: Segregate your email sending programs into subdomains.

This option is best for people within large or complex organizations, or vendors who are contracted to send email on behalf of an organization.

Ask the domain owner to delegate a subdomain for your use case (e.g. marketing.domain.example). Many domain owners are willing to delegate use of a subdomain because allowing for multiple use cases on a single domain becomes a very difficult management and governance challenge.

Through the use of subdomains they can segregate your email sending program from the email sent by normal mailbox users and other email sending programs. This also gives mail receiving organizations the ability to create a reputational model that is specific to your sending patterns, which means that you do not need to inherit any negative reputation incurred by others.

Option 2: Use a domain in which you are the domain owner.

This option is best if you have end-customers (or tenants) who have email addresses within domains which have domain owners that will not allow any form of delegation to you.

Use your own domain as the domain identity, and use subdomains within your domain to distinguish your end-customers from each other (e.g. tenant1.yourdomain.example, tenant2.yourdomain.example, tenant3.yourdomain.example, …). Amazon WorkMail uses this strategy for the awsapps.com domain.

This gives you complete control over the domain as well as your reputation. Use subdomains to segregate reputation between your end-customers if you have a multi-tenant business model.

Here are some additional suggestions to make your email more personable while remaining aligned to the domains’ DMARC policies.

You may format the From header of your outgoing messages so that the display name clearly reflects the name of the message author.

From: “John Doe via My App” <[email protected]>

Set the Reply-to header of your outbound messages so that when recipients reply, the return messages will go to the intended recipient.

Reply-to: [email protected]

What should I do if the domain is already being used for a different email sending program?

From a deliverability perspective, it is beneficial to compartmentalize your sending into different domains, or subdomains, for different email sending programs. That will limit the reputational blast radius if something were to go wrong with one campaign. Consider using different subdomains for each sending program. For example:

marketing.domain.example
receipts.domain.example

DMARC was designed for marketing and transactional email use cases, so it is good practice to publish ‘reject’ DMARC policies for those subdomains. Having a strong policy doesn’t give a free pass into recipient inboxes, but it allows the mail receiving organization to know what to do with messages that aren’t authenticated, which can lead to better trust. Building trust is the best way to gain a positive reputation.

If the domain is used by normal users for day-to-day correspondences, the domain owner should be very careful about publishing a DMARC policy because it is known to create interoperability issues with mailing lists and other email providers. Many of these email domains may never publish a ‘reject’ DMARC policy. For new email sending programs, you should strongly consider using a subdomain rather than any domain that is being used for user correspondences.

Conclusion

To ensure optimal deliverability with Amazon SES, it’s essential to be aware of the domain owner’s preferences and use a domain identity for outbound messages. Keep in mind that email address identities should only be used for testing purposes or with domains without DMARC policies. Domain owners can use subdomains to segregate email sending programs, making management and governance easier while allowing mail receiving organizations to build isolated reputational models.

By following the recommendations in this blog, you’ll be better prepared to align with the domain owner’s preferences, achieve higher deliverability rates for your authenticated outbound email, and be compatible with future DMARC developments.

Tracking email engagement with AWS Analytics services

2022-08-31 Justin Morris

Post Syndicated from Justin Morris original https://aws.amazon.com/blogs/messaging-and-targeting/tracking-email-engagement-with-aws-analytics-services/

Email at scale is one of the most valuable forms of reaching and engaging with customers, but creating the most engaging email content means frequent tracking, monitoring, and analyzing bounces, complaints, opens, and clicks. Managing your reputation as a sender and understanding user engagement have never been more important and play a crucial role in email deliverability and inbox placement.

Amazon Simple Email Service (Amazon SES) is a cost-effective, flexible, and scalable email service that enables customers to send mail from within any application. You can configure Amazon SES quickly to support several email use cases, including transactional, marketing, or mass email communications. An additional benefit of using Amazon SES is event publishing which enables you to track your email sending, feedback, and user engagement at a granular level. AWS also offers analytics tools like Amazon Athena and Amazon QuickSight, which can be connected to SES for even deeper insights capabilities.

In this post, we will walk you through how to create an end-to-end solution for event analysis and how to build custom dashboards to analyze SES utilization, user engagement, and sender reputation.

Prerequisites

The solution presented in this blog requires the following prerequisites to be satisfied before continuing:
An AWS Account that provides access to AWS services.
An AWS Identity and Access Management (IAM) user with the permissions to create an IAM role and policies, and create stacks in AWS CloudFormation.
An Amazon Simple Email Service (Amazon SES) verified identity. You can create and verify a sending identity by following the steps described in the documentation.
Your Amazon SES account is out of the sandbox.
Use a region where AWS Glue DataBrew is available.

Solution Overview

The Figure below describes the architecture diagram for the proposed solution.

Amazon SES publishes email sending events to Amazon Kinesis Data Firehose using a default configuration set.
Amazon Kinesis Data Firehose Delivery Stream stores event data in an Amazon Simple Storage Service (Amazon S3) bucket, known as the Destination bucket.
AWS Glue DataBrew processes and transforms event data in the Destination bucket. It applies the transformations defined in a recipe to the source dataset and stores the output using a different prefix (‘/partitioned’) within the same bucket. Output objects are stored in the Apache Parquet format and partitioned.
An AWS Lambda function copies the resulting output objects to the Aggregation bucket. The Lambda function is invoked asynchronously via Amazon S3 event notifications when objects are created in the Destination bucket.
An AWS Glue crawler runs periodically over the event data stored in the Aggregation bucket to determine its schema and update the table partitions in the AWS Glue Data Catalog.
Amazon Athena queries the event data table registered in the AWS Glue Data Catalog using standard SQL.
Amazon QuickSight dashboards allow visualizing event data in an interactive way via its integration with Amazon Athena data sources.

Figure 1. Serverless architecture for processing Amazon SES Email Sending Events at scale

Solution Deployment

After all the prerequisites listed above are met, the sequence of steps required to deploy the solution is summarized as follows:

Deploy the CloudFormation template in your desired AWS Region.
Configure your Amazon SES verified identity.
Send emails via the Amazon SES API or the Amazon SES Simple Mail Transfer Protocol (SMTP) interface.
Configure AWS Glue DataBrew to transform the source dataset as required.
Create an Amazon QuickSight dashboard to visualize email sending events.

Step 1: Deploy the CloudFormation template in your desired AWS Region

The serverless pipeline described in this post is available as a CloudFormation template at this link.

To deploy the template using the CloudFormation console:

Browse to this URL – make sure to switch to desired AWS Region using the navigation bar.
In the Stack name section, enter a name for the Stack.
Choose Create Stack.

This template will deploy the required resources to implement the serverless pipeline. The deployment takes less than 5 minutes to complete.

Step 2: Configure your Amazon SES verified identity

Amazon SES leverages configuration sets to define groups of rules that you can apply to your verified identities. In this case, the CloudFormation template deployed in the previous step defines a configuration set that uses an event destination based on Amazon Kinesis Data Firehose. In this step you are going to configure your verified identity to use the configuration set “SESConfigurationSet” by default.

Open the Amazon SES console.

In the navigation pane, under Configuration, choose Verified identities.
In the Identity column, select the verified identity that you want to edit.
On the identity’s detail page, select the Configuration set tab and then choose Edit.
In the Default configuration set page, check the Assign a default configuration set box.
For the Default configuration set dropdown, choose SESConfigurationSet.

Step 3: Send emails via the Amazon SES API or the Amazon SES Simple Mail Transfer Protocol (SMTP) interface

To test the pipeline, you can start sending emails using the Amazon SES API or the SMTP interface. During Step 2 you have associated the SESConfigurationSet as the default configuration set for the verified identity. As a result, all the emails you are going to send from the verified identity will generate email sending events that will flow through the Amazon Kinesis Data Firehose Delivery Stream and persisted in the Amazon S3 Destination bucket under the ‘/raw’ prefix.

As an alternative to the Amazon SES API or the SMTP interface, it is possible to quickly send test emails using the Amazon SES mailbox simulator:

Open the Amazon SES console.
In the navigation pane, under Configuration, choose Verified identities.
In the Identity column, select the verified identity that you want to send a test email from.
Choose Send Test Email.
On the Send test email page, for From-address enter the desired value, for example sender.
For Scenario, choose the email sending scenario that you want to simulate, for example Success Delivery.
For Subject, enter the desired email subject.
For Body, type an optional body text.
For Configuration set, you can leave the field empty if you have associated a default configuration set to this identify in Step 2. Choose the SESConfigurationSet otherwise.
Choose Send test email to send the email.

Figure 2. Sending a test email using the Amazon SES mailbox simulator

The Amazon Kinesis Data Firehose Delivery Stream has been configured to buffer data for up to 5 MiB or 60 seconds before delivering it to the Amazon S3 Destination bucket. To verify that the delivery is working as expected, wait for 60 seconds and then:

Open the Amazon S3 console.
In the navigation pane, choose Buckets.
Choose the bucket named <ACCOUNT-ID>-<REGION>-ses-events-destination (replace <ACCOUNT-ID> and <REGION> based on your actual environment).
Navigate to the ‘/raw‘ prefix and verify it contains some compressed objects (the number of objects depends on the volume of email sending events triggered).

As part of this blog post, we are providing you with sample email sending events that will be used to populate the Amazon QuickSight dashboard in Step 5. You can use AWS CloudShell in your own AWS account to run the command below and synchronize the sample events to your Amazon S3 bucket:

aws s3 sync s3://ses-blog-assets/sample-ses-email-events/ s3://<ACCOUNT-ID>-<REGION>-ses-events-destination/

Remember to replace <ACCOUNT-ID> and <REGION> based on your actual environment.

Step 4: Configure AWS Glue DataBrew to transform the source dataset as required

In this step we will go over the configuration of DataBrew, a visual data preparation tool, to clean and prepare the data for analysis in QuickSight. The CloudFormation template you deployed in previous steps already created a DataBrew dataset for you. In this step, you will import a DataBrew recipe that contains the transformations necessary to analyze the SES email sending events, and then create a DataBrew job to process the data.

Upload a DataBrew recipe
To upload a recipe, you should follow the steps below:
1. Download this JSON file that contains the recipe.
2. Navigate to the DataBrew console, on the left panel choose Recipes.
3. In the Recipes console, on the top left choose Upload recipe.
4. For Recipe name provide the following ses-clean-recipe.
5. For Upload recipe choose Upload and select the file you downloaded in Step 1.
6. Choose Create and publish recipe.
Create and Schedule a DataBrew job
Create a job to apply the recipe you created above on the dataset and configure it to output a parquet file. To do that, complete the following steps:
1. In the DataBrew console, at top right of the project editor, choose the Create job button.
2. For Job name, enter ses-event-transformed.
3. In the Job Input section, for Choose dataset choose the dataset named SESDataBrewDataset. Select the recipe imported in the previous step.
4. For output choose Amazon S3.
5. For File type, choose PARQUET and compression as GZIP.
6. For S3 location provide the following S3 Bucket that has been created for you by the CloudFormation template: ‘s3://<ACCOUNT-ID>-<REGION>-ses-events-destination/partitioned/‘ (replace <ACCOUNT-ID> and <REGION> based on your environment).
7. Choose Settings. For File output storage choose Replace output files for each job run. For Custom partition by column values choose Enabled, and on Columns to partition by add the following: year, month, day, hour (in this order). Choose Save.
8. Expand the Associated schedules section, choose Create new schedule and create a schedule that runs every hour and every day. Choose Add.
9. Under Permissions, for Role name, choose an existing role or create a new one.
10. Choose Create and run job.
11. Navigate to the Jobs page and wait for the ses-event-transformed job to complete.
12. Choose the Destination link to navigate to Amazon S3 to access the job output.

You have now created a DataBrew job that would transform your dataset based on the recipe you created above.

The SESDataBrewDataset dataset has been created to fetch only the email sending events generated in the past 1 hour to allow for a more efficient differential processing. At the end of each job execution, a Lambda function copies the latest transformed data to the Aggregation bucket named <ACCOUNT-ID>-<REGION>-ses-events-destination-aggregated. Every hour, an AWS Glue crawler scans the Aggregation bucket to update the Glue Catalog accordingly. Before moving to the next Step, either wait for the next scheduled crawler run or browse to the AWS Glue console and run the crawler called SESEventDataCrawler manually.

Step 5: Create an Amazon QuickSight dashboard to visualize email sending events

Now that your cleaned dataset is ready, you are ready to deploy your dashboard. For this blog post we are providing you with a sample QuickSight dashboard. Follow the steps below to deploy it as is. However, if you would like to have a custom dashboard with your own metrics and KPIs and want to use your BI tool of choice, you can leverage the output of the DataBrew job that is stored in S3 to build your own analysis and deploy it in a dashboard.

To deploy the sample dashboard, you first need to create a QuickSight data source, a QuickSight dataset and then a QuickSight dashboard. These resources are going to be deployed using the AWS CLI. You can use AWS CloudShell in your own AWS account to run the commands below. Remember to replace <ACCOUNT-ID> and <REGION> in the following commands based on your actual environment.

Create a QuickSight data source
1. 1. Download this JSON file which contains the data source definition.
  2. Open the file and change the following:
    1. Principal: the QuickSight users authorized to use the dataset. You can use the procedure described in the documentation to retrieve the QuickSight Principal ARN.
  3. Apply the command below to create the data source.
aws quicksight create-data-source --aws-account-id <ACCOUNT-ID> --cli-input-json file://create-data-source.json --region <REGION>
1. Record the data source ARN.
Create a QuickSight dataset
1. 1. Download this JSON file which contains the dataset definition.
  2. Open the file and change the following:
    1. DataSourceArn: data source ARN from Step a.
    2. ImportMode: if you use QuickSight Enterprise, you can use SPICE. Otherwise, use DIRECT_QUERY.
    3. Principal: the list of QuickSight users ARNs authorized to use the dataset.
  3. Apply the command below to create the data source
aws quicksight create-data-set --aws-account-id <ACCOUNT-ID> --cli-input-json file://create-dataset.json --region <REGION>
1. Record the dataset ARN
Create the Quicksight Dashboard
1. 1. Download this JSON file which contains the dataset definition.
  2. Open the file and change the following:
    1. Principal: the list of QuickSight users ARNs authorized to view the dashboard.
    2. DataSetArn: Dataset ARN from Step b.
  3. Apply the command below to create the data source.
aws quicksight create-dashboard --aws-account-id <ACCOUNT-ID> --cli-input-json file://create-dashboard.json --region <REGION>

After completing the three steps above, you will see a dashboard like the one below populated with your own data.

Figure 3. The sample QuickSight dashboard visualizing SES Email Sending Events

Cleaning up

You should have now successfully produced a first analysis of your Amazon SES email sending events. To avoid incurring any extra charges, remember to delete any resources created manually following the instructions in this blog post and delete the CloudFormation stack that you deployed.

Conclusion

In this blog post we showed you how you can collect your SES email sending events through Firehose, prepare them with DataBrew and visualize them with QuickSight. This solution provides you with a starting point that you can customize at your will to meet your organization requirements. This solution is set to run on a scheduled basis, every hour, making it easier for you to leverage incremental data processing.

You can also augment your event data with your business data by joining them through DataBrew. Finally, you can also leverage Amazon AppFlow (a SaaS integration service) with DataBrew to easily integrate data from third party solutions that you are already using.

About the Authors

Luca Iannario is a Sr. Solutions Architect at AWS within the Public Sector team in EMEA. He works with customers of all sizes across Government, Education, Healthcare and NPO verticals, helps them deploying AWS Services securely at scale and facilitates their cloud adoption journey. His goal is to build better societies by bringing the benefits of the cloud to all citizens. In his spare time, Luca enjoys traveling and watching movies.

Lotfi Mouhib

Lotfi Mouhib is a Senior Solutions Architect working for the Public Sector team with Amazon Web Services. He helps public sector customers across EMEA realize their ideas, build new services, and innovate for citizens. In his spare time, Lotfi enjoys cycling and running

Justin Morris

Justin Morris is a Email Deliverability Manager for the Simple Email Service team. With over 10 years of experience in the IT industry, he has developed a natural talent for diagnosing and resolving customer issues and continuously looks for growth opportunities to learn new technologies and services.

Opt-in to the new Amazon SES console experience

2021-02-12 Simon Poile

Post Syndicated from Simon Poile original https://aws.amazon.com/blogs/messaging-and-targeting/amazon-ses-console-opt-in/

Amazon Web Services (AWS) is pleased to announce the launch of the newly redesigned Amazon Simple Email Service (SES) console. With its streamlined look and feel, the new console makes it even easier for customers to leverage the speed, reliability, and flexibility that Amazon SES has to offer. Customers can access the new console experience via an opt-in link on the classic console.

Amazon SES now offers a new, optimized console to provide customers with a simpler, more intuitive way to create and manage their resources, collect sending activity data, and monitor reputation health. It also has a more robust set of configuration options and new features and functionality not previously available in the classic console.

Here are a few of the improvements customers can find in the new Amazon SES console:

Verified identities

Streamlines how customers manage their sender identities in Amazon SES. This is done by replacing the classic console’s identity management section with verified identities. Verified identities are a centralized place in which customers can view, create, and configure both domain and email address identities on one page. Other notable improvements include:

DKIM-based verification
DKIM-based domain verification replaces the previous verification method which was based on TXT records. DomainKeys Identified Mail (DKIM) is an email authentication mechanism that receiving mail servers use to validate email. This new verification method offers customers the added benefit of enhancing their deliverability with DKIM-compliant email providers, and helping them achieve compliance with DMARC (Domain-based Message Authentication, Reporting and Conformance).
Amazon SES mailbox simulator
The new mailbox simulator makes it significantly easier for customers to test how their applications handle different email sending scenarios. From a dropdown, customers select which scenario they’d like to simulate. Scenario options include bounces, complaints, and automatic out-of-office responses. The mailbox simulator provides customers with a safe environment in which to test their email sending capabilities.

Configuration sets

The new console makes it easier for customers to experience the benefits of using configuration sets. Configuration sets enable customers to capture and publish event data for specific segments of their email sending program. It also isolates IP reputation by segment by assigning dedicated IP pools. With a wider range of configuration options, such as reputation tracking and custom suppression options, customers get even more out of this powerful feature.

Default configuration set
One important feature to highlight is the introduction of the default configuration set. By assigning a default configuration set to an identity, customers ensure that the assigned configuration set is always applied to messages sent from that identity at the time of sending. This enables customers to associate a dedicated IP pool or set up event publishing for an identity without having to modify their email headers.

Account dashboard

There is also an account dashboard for the new SES console. This feature provides customers with fast access to key information about their account, including sending limits and restrictions, and overall account health. A visual representation of the customer’s daily email usage helps them ensure that they aren’t approaching their sending limits. Additionally, customers who use the Amazon SES SMTP interface to send emails can visit the account dashboard to obtain or update their SMTP credentials.

Reputation metrics

The new reputation metrics page provides customers with high-level insight into historic bounce and complaint rates. This is viewed at both the account level and the configuration set level. Bounce and complaint rates are two important metrics that Amazon SES considers when assessing a customer’s sender reputation, as well as the overall health of their account.

The redesigned Amazon SES console, with its easy-to-use workflows, will not only enhance the customers’ on-boarding experience, it will also change the paradigms used for their on-going usage. The Amazon SES team remains committed to investing on behalf of our customers and empowering them to be productive anywhere, anytime. We invite you to opt in to the new Amazon SES console experience and let us know what you think.