Tag Archives: messaging

Enhance Your Email Campaigns using Amazon SES SendBulkEmail APIs Inline Templates

Post Syndicated from Satya S Tripathy original https://aws.amazon.com/blogs/messaging-and-targeting/enhance-your-email-campaigns-using-amazon-ses-sendbulkemail-apis-inline-templates/

Amazon Simple Email Service (SES) is a cloud-based email sending service provided by Amazon Web Services (AWS), handling both inbound and outbound email traffic for your applications. It allows users to send and receive email using SES’s reliable and cost-effective infrastructure without having to provision email servers yourself. Customers use Amazon SES to send emails like one time passwords (OTPs), transactional emails such as order confirmation, and promotional/marketing emails.

For this post, you should be familiar with the following:

Amazon SES continues to evolve, offering new features that help you simplify and optimize your email campaigns. We’re excited to announce the addition of inline template support for both the SendEmail and SendBulkEmail APIs. This new capability allows you to include template content directly in your API requests, reducing complexity and eliminating the need to manage separate template resources in your SES account.

What are inline templates?

Inline templates allow you to provide the subject, HTML body, and text body of an email directly in the API request, along with dynamic placeholders for personalized content. Instead of creating and storing a separate email template in SES, you can define the template as part of your API call. This feature is especially useful for organizations that need flexibility in managing numerous templates or want to make quick adjustments to email content.

How inline templates simplify your workflow

Previously, Amazon SES required you to create and store email templates in your SES account, which you would then reference by name or Amazon Resource Name (ARN) when sending an email. This process adds some management overhead, particularly for organizations that frequently create new templates or exceed the limit of templates per AWS Region. With inline templates, you can reduce complexity by defining your email content directly in the API payload, avoiding the need to create and manage stored templates. This approach can improve flexibility, allowing you to quickly make changes to your email content without updating stored templates. Additionally, it can simplify your integration by providing template content directly within your application logic, making the process more seamless and efficient. When using the and SendBulkEmail API, you can include personalized content for up to 50 destinations in a single call, making large-scale communication more efficient.

How to use inline templates

To use inline templates, you simply provide the email content (subject, text, HTML) and the replacement data directly in the SendBulkEmail API request payload within TemplateContent parameter.

Inline-template API

Here’s an example for using the SendBulkEmail API with inline templates:

file://mybulkemail-inline-template-conten.json:

{
    "FromEmailAddress": "Mary Major <[email protected]>",
    "DefaultContent": {
        "Template": {
            "TemplateContent": {
                "Subject": "Greetings, {{name}}!",
                "Text": "Dear {{name}},\r\nYour favorite animal is {{favoriteanimal}}.",
                "Html": "<h1>Hello {{name}},</h1><p>Your favorite animal is {{favoriteanimal}}.</p>"
            },
            "TemplateData": "{ \"name\":\"friend\", \"favoriteanimal\":\"unknown\" }"
        }
    },
    "BulkEmailEntries": [
        {
            "Destination": {
                "ToAddresses": [
                    "[email protected]"
                ]
            },
            "ReplacementEmailContent": {
                "ReplacementTemplate": {
                    "ReplacementTemplateData": "{ \"name\":\"Anaya\", \"favoriteanimal\":\"angelfish\" }"
                }
            }
        },
        {
            "Destination": {
                "ToAddresses": [
                    "[email protected]"
                ]
            },
            "ReplacementEmailContent": {
                "ReplacementTemplate": {
                    "ReplacementTemplateData": "{ \"name\":\"Liu\", \"favoriteanimal\":\"lion\" }"
                }
            }
        }
    ],
    "ConfigurationSetName": "ConfigSet"
}

SES SendBulkEmail API call:

aws sesv2 send-bulk-email -cli-input-json file://mybulkemail-inline-template-conten.json

Output:

{
    "BulkEmailEntryResults": [
        {
            "Status": "SUCCESS",
            "MessageId": "010001xxxxxx-xxxxxxxx-xxxx-xxxx-000000"
        },
        {
            "Status": "SUCCESS",
            "MessageId": "020002xxxxxx-xxxxxxxx-xxxx-xxxx-000000"
        }
    ]
}

Backward compatibility

If you’re currently using stored templates, don’t worry – Amazon SES still supports the use of stored templates, and you can continue to use them without any changes. Inline templates are simply an additional option for customers who need more flexibility or wish to avoid managing stored templates altogether. Since inline templates only support the use of simple substitution, stored templates remain the solution for advanced personalization options such as conditional logic or complex formatting. More details in our doc link: How to use Bulk email template.

Get started today

The inline template feature is now available in all AWS Regions where Amazon SES is offered. To start using inline templates, refer to the Amazon SES Developer Guide and what’s new announcement. There are no additional charges applicable for using inline template feature.

Conclusion

The inline templating feature in SendBulkEmail allows you to avoid worrying about template management by updating / creating new email templates whenever a minor modification or alteration is required in the existing templates, as well as cleaning up unused templates on a regular basis. Therefore, if your business has a high number of email template requirements, there are no predefined rules or patterns for creating email templates, and you need to generate many templates simultaneously within Amazon SES, you must use inline templating feature of SendBulkEmail API . If you do not want to use the Inline templating capability, you can continue to use the templated SendBulkEmail API from Amazon SES.

Modernize email sending with Amazon Simple Email Service and Proofpoint SER

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/modernize-email-sending-with-amazon-simple-email-service-and-proofpoint-ser/

As organizations migrate to the cloud, managing email security and delivery becomes increasingly complex. This post explores how Amazon Simple Email Service (SES) and Proofpoint Secure Email Relay (SER) work together to provide a robust solution for modern email sending.

Shifting email workflows to the cloud

For many organizations, Simple Mail Transfer Protocol (SMTP) relay is a critical email delivery mechanism that facilitates the transmission of email messages between different domains and servers. When an email is sent to a recipient outside the sender’s domain, SMTP relay(s) ensures the message is routed correctly and delivered to the intended destination.

Traditionally, on-premises SMTP relay servers were used by messaging and security teams to manage email sending from on-premises applications on behalf of their organizations’ domains. This approach helps to:

  • Guard against brand damage and loss of sensitive data
  • Protect recipients from fraud
  • Provide straightforward control over email sending

However, as applications are modernized and migrated to the cloud, email sending has changed. Many organizations now outsource bulk email sending to cloud service providers such as Amazon SES. This shift has made it challenging for internal teams to regulate their organization’s email effectively.

Addressing modernization challenges

Amazon Web Services (AWS) is a popular choice for developing and modernizing applications, with Amazon SES offering a secure, scalable and cost effective service for applications to send email. To address the need for additional security and control over email, Proofpoint offers their popular Secure Email Relay (SER) on the AWS Marketplace.

Using Amazon SES with Proofpoint SER combines the convenience and features of Amazon SES with Proofpoint SER’s ability to:

  • Regulate and govern outbound application email
  • Support migration from on-premises relay to the cloud
  • Apply security controls to application emails

Email flow and security

SES Mail Manager allows emails to be conditionally routed from Amazon SES to Proofpoint SER. The process includes:

  1. Scanning emails with Proofpoint threat detection technologies
  2. Applying centrally managed Proofpoint SER policies
  3. Performing DKIM-signing and distributing DMARC compliant emails
  4. Sending mail to recipients

Two configuration options are available:

  1. Proofpoint SER handles distribution to final recipients (Figure 1)
  2. Emails are routed back to Amazon SES for distribution after Proofpoint SER processing (Figure 2)

In the first configuration, email is sent to SES and routed through a Mail Manager SMTP relay to Proofpoint SER where it is processed for threat detection, malware, spam, sensitive data, and more. In this configuration, Proofpoint SER distributes the email to recipients through a STMP relay or another email sending service beyond Proofpoint. Deliverability reporting, archiving, and other capabilities are left to Proofpoint or downstream providers.

SES to SER

Figure 1: Proofpoint SER applying security controls to application emails before sending.

In the second configuration, email is sent to SES which routes through a Mail Manager SMTP relay to Proofpoint SER to be processed for threat detection, malware, spam, sensitive data, and more. Proofpoint SER then sends the processed emails back to SES via Mail Manager STMP relay. Deliverability reporting, archiving and other capabilities such as Amazon Q Business can be provided by SES or other AWS services.

SES-SER-SESFigure 2. Security controls applied by Proofpoint SER before routing emails back to Amazon SES for distribution.

Conclusion

The integration of Amazon SES and Proofpoint SER offers a powerful solution for organizations looking to modernize their email sending processes while maintaining robust security. This collaboration allows businesses to leverage the scalability and convenience of cloud-based email services without compromising on control and protection.

Call to Action

Take the next step in modernizing your email infrastructure while enhancing security and control. To learn more about how Amazon SES and Proofpoint SER can benefit your organization:

How to Make Simple Email Service Resilient Across Two AWS Regions with Global Endpoints

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-make-simple-email-service-resilient-across-two-aws-regions-with-global-endpoints/

Introduction

Amazon Simple Email Service (SES) recently announced Global Endpoints, a major enhancement to its email sending features. This new capability improves the availability and reliability of SES API v2 email sending workloads by automatically distributing messages, in an active/active configuration across the Primary and Secondary AWS regions. When Global Endpoints detects degradation in either the Primary or Secondary SES region, the feature automatically shifts all traffic to the healthy region, no customer intervention is needed.

Multi-Region SES Configuration Challenges

Customers face significant difficulties in correctly implementing a multi-region setup or disaster recovery setup. The process requires careful curation of systems along the failover path and ensuring high availability of these systems. Ironically, the system designed to trigger failover can itself fail when needed most. For many SES customers, the effort required to design, build test, monitor and maintain a two-region SES system outweighs the benefits.

SES Global Endpoints eliminates the need for these complex, custom workarounds. The feature provides a straightforward solution for maintaining email sending during unexpected regional disruptions. Global Endpoints’ built-in safeguards ensure email infrastructure remains resilient when it matters most. Please note that at launch, Global Endpoints does not support SMTP or VPC endpoint access.

SES Global Endpoints: Key Technological Components

Global Endpoints utilizes four new SES components that work together to provide a seamless and reliable multi-region email sending experience:

  1. Multi-Region Endpoint (MREP) is a new type of SES endpoint that automatically distributes email traffic across two AWS regions.
  2. Deterministic Easy DKIM Identities (DEED) makes it easy to setup multi-region identities without having to make any DNS changes.
    1. Read the blog introducing SES DEED for more information.
  3. Updated AWS SES Console Tool walks you through the process and simplifies the duplication of Domain Identities, Configuration sets, and Sending limits across regions.
  4. Readiness Checks in the SES console verify uniformity between configurations of key resources in both the Parent and Secondary SES regions.

How SES Global Endpoints work

GE-MREP-healthy

Figure 1 – SES Global Endpoints with two healthy regions.

Global Endpoints are resources that distribute your SES outbound workloads across two AWS Regions. When you set up Global Endpoints, you select a Primary Region (where the actual Endpoint is created) and a Secondary Region. When configured, a new Global Endpoints resource, called “multi-region endpoint” (MREP) is created and managed. Developers will need to update their SES v2 API enabled applications and services to use their unique MREP as the entry point to SES for their email sending requests.

The Global Endpoints configuration requires that your sending domain identity(s) is verified in both the Primary and Secondary regions. SES Global Endpoints uses DEED to simplify this process. DEED is a new feature that generates consistent DKIM tokens across all AWS Regions based on a Parent Domain Identity that is configured with Easy DKIM. This consistency allows SES to automatically verify a domain identity in the Secondary region once it’s verified in the Primary region, without requiring additional DNS record updates. Customers do not need to make any additional changes other than activating the DEED identity type. When customers expand their sending workload’s geographic footprint, or reconfigure their Global Endpoints settings in the future, their DEED identities will continue to be available and managed automatically. You can learn more about DEED from this post.

Global Endpoints works alongside other SES services, such as Virtual Deliverability Manager (VDM). Once Global Endpoints are enabled, you’ll continue to see per-region data on email sending performance in VDM. If you’ve configured event destinations like CloudWatch, SNS, or Firehose, you can make use of those same monitoring and alerting tools in your second region as soon as you’re ready. As noted below, although Configuration Sets are automatically duplicated in the Secondary region, you must manually duplicate your SES event destinations in those Configuration Sets.

It is important to understand that Global Endpoints is not a failover solution for SES, it’s an active-active implementation; when no regional impairment exists, SES Global Endpoints distributes sending traffic across two AWS regions. Customers who use SES’ shared IP sending pool do not need to make any changes, Global Endpoints will utilize the SES shared IP pool in the Secondary region. Customers who use standard, dedicated IPs must manually set up equivalent number of dedicated IPs in the Secondary region. Once properly configured, Global Endpoints will keep the dedicated IPs warm in both regions as long as you use the MREP and maintain a steady sending volume.

For example, when SES’s regional health monitoring detects degradation in the Primary region (as shown in the diagram below), The MREP automatically shifts all traffic to the healthy region. This illustrates the need for matching configurations in both regions, since all traffic will be sent through a single region, in this example the Secondary region, as long as the impairment exists. When SES’s regional health monitoring detects the impaired region is back to normal, traffic is once again redistributed across both regions. Importantly, no customer intervention is needed; SES Global Endpoints automatically and dynamically monitors and manages the traffic distribution via the MREP.

GE-MREP-impaired

Figure 2 – SES Global Endpoints with one impaired region.

The key benefits of using Global Endpoints include:

  • Simplified multi-region configuration
  • Automatic routing between two regions with no delay
  • More resilient email sending

Global Endpoints: Setup and Use

Using the SES Console, the Global Endpoint setup process assists in duplicating the key artifacts and sending limits from your Primary Region to your Secondary Region. This process ensures that both regions have equivalent verified identities, configuration sets, and approved sending limits sufficient for all of the expected volume. Customers can manually duplicate these key artifacts using the SES v2API or CloudFormation, but we recommend using the SES console because these steps are simplified.

Once the Global Endpoint is ready, key resources duplicated and the application or service has been updated to use the new MREP, SES Global Endpoints automatically routes your outbound traffic evenly between your primary and secondary regions using the multi-region endpoint (MREP). If the MREP detects degraded performance in the Primary or Secondary region, it will automatically route all SES traffic to the healthy region until the impairment is resolved.

Preparing the Parent Region

The high-level steps to setup Global Endpoints using the SES Console are below.  Note – you must already have a primary SES region fully operational with at least 1 fully verified sending identity with production access before setting up Global Endpoints.

Create Global Endpoint

  1. Open the SES console in the Primary AWS region.
  2. In the navigation pane, choose Global Endpoints.
  3. Choose Create Global Endpoint.
  4. Select a Secondary Region from the dropdown menu. (Your Primary Region defaults to the region to which you signed into the console)
  5. Review the configuration and choose Create.

The creation process may take a few seconds. Once completed, the status of your Global Endpoint will change to “Ready.”

Global Endpoints "ready" status

Preparing the secondary region

Once your Global Endpoint is ready, you must now ensure that the your email sending configuration, including all its components (Domain Identities, Configuration sets, email templates, and sending limits), is consistent across the primary and secondary regions before utilizing the Global endpoint for sending emails. This alignment is crucial to avoid potential issues and ensure proper email delivery and tracking.

The new Region duplication feature in the SES console assists you by automatically duplicating resources and duplicating account-level settings from the primary to the secondary region, ensuring that both regions have equivalent configurations.

The high-level steps to setup the secondary region for use with Global Endpoints using the SES Console are below. If you’d like to use the AWS CLI to manually duplicate these resources, consult with the SES v2 API documentation.

Duplicating verified domain identities

Next you’ll use the Duplicate verified domain identities feature in the SES console to create one or more domain identities in the Secondary Region. SES will then use DEED to verify the domain identities in the Secondary Region. Note that DEED can only be used if the Primary Domain Identity is already configured with Easy DKIM. Domain identities that are verified with BYODKIM will need to be created manually in the Secondary Region, as DEED is not applicable in this case.

Important – It’s crucial that both Primary and Secondary Regions have the equivalent verified identities and configuration sets that you intend to send email with, along with matching sending limits to ensure proper functionality of the Global endpoint. Any discrepancies could cause delivery failures, diminished failover reliability, and missing metrics.

To duplicate identities from the SES console:

  1. On the Global endpoints page, choose the Global endpoint you want to duplicate by selecting it from the Name column.
  2. Under the Region duplication tab, choose Duplicate identities.
  3. Select the identities you want to duplicate followed by Confirm.

To duplicate configuration sets from the SES console:

  1. On the Global endpoints page, choose the Global endpoint you want to duplicate by selecting it from the Name column.
  2. From the Region duplication tab, choose Duplicate configuration sets.
  3. Select the configuration sets you want to duplicate followed by Confirm.

Important Notes:

  • When duplicating configuration sets across regions, Event destinations and Reputation settings require manual reconfiguration in the Secondary Region to match the Primary Region’s setup. Since SES event destinations are region-specific, you’ll need to manually recreate these configurations in each region. For cross-regional monitoring and event routing, you can refer to additional AWS documentation for services like CloudWatch (cross-region dashboards), SNS (cross-region message delivery), and EventBridge (cross-region event routing) to develop a comprehensive multi-region event strategy.
  • If you are using SES email template resources, those templates need to be manually duplicated into the Secondary Region (the console is currently unable to perform this action).
  • You must manually synchronize any changes made to the Parent Region’s configuration sets to the Secondary Region to maintain sending integrity. This includes adding or removing standard dedicated IPs to both regions to ensure either region has the required IPs to manage the expected throughput in the case of a regional event or impairment.

The Duplicate production limits feature allows you to:

  • Check if production limits are aligned between primary and secondary regions
  • Request a limit increase in the secondary region if needed

To duplicate production limits from the SES console:

  1. On the Global endpoints page, choose the Global endpoint you want to duplicate by selecting it from the Name column.
  2. From the Region duplication tab, verify the status in the Duplicate production limits card. If the status displays Sending limits not aligned, choose Duplicate production limits.
  3. The Service Quotas page opens in the secondary region where you can request increases to “Sending quota” and “Sending rate” to match the values from the primary region.

Note – SES checks if sending limits are aligned between regions and allows you to request limit increases in the secondary region if needed. We recommend that you request the maximum quota you’re eligible for in both regions. While email traffic is distributed amongst both regions under normal operating conditions, during a failover event, the full volume of email traffic will be sent to one region and its limits should be enough to handle the full volume load.

If any manual steps are required to complete the Global Endpoint creation, they will be shown in the SES Console:

manual steps warning

When the Global Endpoint is fully configured, a MREP will be created with an Endpoint ID (see below). You will use this new endpoint ID when re-configuring your SendEmail and SendBulkEmail API calls. (note – Global Endpoints MREP are only supported by SES APIv2. The feature is not available using SMTP or VPC endpoints).

Now you’re ready to send your first email through SES Global Endpoints and your MREP!

Once you’ve obtained the Endpoint ID of your Global endpoint (this is the MREP), you should update your applications’ SendEmail or SendBulkEmail API calls to include the Endpoint ID value for the -endpoint-id parameter.

Here’s an example of how to specify the Endpoint ID in a SendEmail API call using the AWS CLI (modify the from & to email addresses and the --endpoint-id accordingly):

aws sesv2 send-email \
--from-email-address "[[email protected]](mailto:[email protected])" \
--destination "ToAddresses=[[email protected]](mailto:[email protected])" \
--content "Subject={Data=Test
email,Charset=UTF-8},Body={Text={Data=This is a test email sent using Amazon SES
Global endpoints.,Charset=UTF-8}}" \
--endpoint-id "abcdef12.g3h"

The Global Endpoints console page provides summary observability metrics on the combined workload and a unified view of your email sending volume across both the primary and secondary regions. You can access these metrics through the Cross-region metrics tab on the Global endpoint details page in the SES console..

Conclusion

By using a SES Global Endpoint in their SES API v2 applications and services, SES customers benefit from uninterrupted email delivery during regional service issues. SES Global Endpoints automatically distributes sending workloads across two AWS Regions, significantly enhancing resilience against regional outages. The Global Endpoints feature maintains warmed-up dedicated IP addresses in both regions, when used, and automatically shifts traffic to the healthy region when the other is impaired, without requiring customer intervention. SES Global Endpoints eliminates the pain points typically associated with manually-built, multi-region SES sending systems.

Global Endpoint’s console tools provide quick and easy setup and includes readiness checks to identify and mitigate potential misconfigurations. These enhancements simplify the configuration and management process, making it easier for customers to maintain a robust email sending infrastructure.

Overall, SES Global Endpoints addresses customer needs for a more reliable and easily managed email sending system, automating critical processes and providing robust tools for setup and maintenance. This significant improvement to the email sending experience is expected to benefit AWS customers across various industries and use cases.

Call to Action

Get started with SES Global Endpoints today to enhance your email sending resilience!

  • Visit the AWS Console to enable this feature for your account
  • Review the comprehensive documentation for step-by-step guidance.
  • For personalized assistance, don’t hesitate to contact AWS Support or your AWS account team.

Elevate your email infrastructure now to ensure uninterrupted communication with your customers, even in the face of regional disruptions.

Simplify Multi-Region Email Sending with Simple Email Service’s Deterministic Easy DKIM

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/simplify-multi-region-email-sending-with-simple-email-services-deterministic-easy-dkim/

Introduction

Amazon Simple Email Service (SES) provides customers with a robust, scalable email solution to send large-scale, global email communications. The service offers customers many benefits, including scalability, high deliverability rates, cost effective pay-as-you go pricing, availability in over a dozen AWS regions, and tight integration with other AWS services.

We’ve heard from customers who want to synchronise domain identities across multiple AWS regions that it can be difficult to configure and maintain. These customers have shared that establishing and maintaining AWS region specific domain verifications can be confusing, time-consuming, and that it’s difficult to coordinate the many details across their users, customers and service providers.

In this post, we’ll explore the newly introduced SES feature called Deterministic Easy DKIM (aka “DEED”), which solves critical challenges in multi-region email identity management for SES customers. DEED generates consistent DKIM tokens across multiple AWS Regions based on a Parent (domain) Identity that is configured with SES Easy DKIM. DEED uses the Parent Identity’s Easy DKIM configuration to automatically provide the same DKIM signing configuration for a Replica (domain) Identity in a AWS Replica Region. With DEED, you only need to publish DNS records once for the Parent Identity. The Replica Identity will automatically use these same DNS records to verify domain ownership and manage DKIM signing. DEED streamlines multi-region email operations by simplifying DNS management and ensuring consistent DKIM signing across AWS regions, maintaining best-practice email authentication while reducing operational complexity.

Background and key challenges

SES introduced Easy DKIM over a decade ago as an innovative, streamlined solution to help customers create, verify, and manage domain identities with automated DKIM signing. Easy DKIM is simple to set-up, and works by generating pre-determined tokens that customers add to their DNS configurations. Once Easy DKIM is enabled, SES generates a public/private signing key for each domain identity and updates the verified identity’s CNAME public key. SES’ Easy DKIM simplifies email authentication by managing and rotating DKIM keys on behalf of customers.

Before DEED, customers who wanted to expand their SES email infrastructure across multiple regions faced complexity, operational challenges and a substantial administrative burden configuring and maintaining DKIM across AWS regions. This made it very difficult for customers to scale their email infrastructure efficiently, and often dissuaded customers from fully leveraging the potential of a multi-region email sending strategy. These challenges are more pronounced for Independent Software Vendors (ISVs) and email service providers using SES. Frequently these organizations don’t own or control the domains used by their end-customers, who must manually update their DNS entries each time the ISV expanded or shifted their SES sending infrastructure between AWS regions.

Solution Overview

Deterministic Easy DKIM (DEED) allows SES customers to set up email identities across multiple regions, leveraging an existing domain identity configuration in a single region without the need to make companion DNS changes in all other regions. This innovative approach eliminates the manual overhead of creating region-specific DNS entries, and provides a streamlined solution to configure and maintain global email infrastructure for organizations and ISVs alike.

The key benefits of using DEED include:

  • Simplified DNS Management – Publish DNS records once for the Parent Identity and the replica identity is automatically synced.
  • Easier Multi-Region Operations – Simplifies the process of expanding email sending operations to new AWS regions.
    Reduced Administrative Overhead – Manage DKIM configurations centrally from the Parent Identity.

Terminology:

To understand DEED, let’s explore the key terminology that underpins this innovative approach:

  • Deterministic – a process or system where the same input will always produce the same output, with no randomness or unpredictability involved. In other words, if all the starting conditions are known, the outcome can be precisely predicted or determined.
  • Parent Region – The original AWS Region where the primary email identity is initially established.
  • Parent Identity – A verified email identity configured with Easy DKIM that serves as the authoritative source for DKIM configuration across regions.
  • Replica Region – An AWS Region where an identical email identity is replicated without additional configuration.
  • Replica Identity – An identity that shares identical DNS configuration and DKIM signing configuration of a parent identity.
  • DEED Identity – Any identity that is used as either a parent identity or a replica identity. (When a new identity is created, it is initially treated as a regular (non-DEED) identity. However, once a replica is created, the identity is then considered a DEED identity.)

How DEED works

DEED flow

DEED is built on the existing Easy DKIM framework:

  1. Using Easy DKIM, SES generates a public-private key pair and automatically adds a DKIM signature to every message sent from a SES verified identity in the Parent Region.
  2. To accommodate authenticated SES sending from multiple AWS regions, DEED automatically synchronizes the signing keys from the Parent Region and Parent Identity to the Replica Region and Replica Identity.
  3. This automated process ensures that both the Parent and Replica Identities receive and use identical keys for DKIM signing, maintaining consistent authentication across different AWS regions.
  4. SES manages the complex process of key rotation across Parent Identity and Replica Identities, further simplifying email infrastructure management across multiple AWS regions.
  5. Replica Identities inherit the DKIM signing configuration of the parent identity. Because of this dependency, you cannot delete a Parent Identity until all Replica Identities are deleted.
  6. The receiving email server/service validates DKIM from DNS.

We recommend customers take advantage of DEED for single-Region sending, as this new capability is included, at no additional cost, in the base SES pricing.

Steps to setup SES DEED Replica in a second AWS region

These steps assume that you are already using SES in the Parent Region, and have a fully verified domain identity that is configured to use Easy DKIM. You can also use the AWS CLI.

Step 1 – Update the Parent Identity

  1. Login to the AWS SES Console in the Parent Region
  2. Click on the Identities link in the SES navigation panel (far left) and click on the verified identity you want to use in other AWS regions.
    1. The Parent Identity must have Easy DKIM enabled.
    2. You cannot create Replicas of Identities that use BYODKIM or self-signed identities.
  3. Click on the Authorization tab
  4. Click Create Policy and select Create custom policy from the drop-down menu.
  5. Name the policy (for example, DEED-example_com)
  6. Modify the IAM policy (below) with your AWS account ID and AWS region, and paste it into the IAM Policy document to grant permission on the Parent Identity to allow the desired Replica Region to replicate the Parent Identity’s DKIM Signing Attributes:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowDKIMReplication",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:root"
},
"Action": "ses:ReplicateEmailIdentityDKIMSigningKey",
"Resource": "arn:aws:ses:us-east-1:123456789124:identity/example.com",
"Condition": {
"ForAllValues:StringEquals": {
"ses:ReplicaRegion": ["us-west-2", "eu-west-1"]
}
}
}
]
}

Note – Use consistent IAM policies that all allow for DKIM replication across intended Replica Regions.

Step -2 – Setup DEED Replica Identity

  1. Login to the AWS SES Console in the Replica Region
  2. Click on the Identities link in the SES navigation panel (far left) and click Create identity
  3. Under Identity details, Identity type, click Domain
  4. Type the domain name exactly as it appears in the Parent Identity
  5. Under Verifying your domain, click Deterministic Easy DKIM
  6. Select the Parent Region in the Parent region drop-down
  7. Ensure DKIM Signature is Enabled
  8. Click Create identity

create DEED ID

  1. The Replica Identity in the Replica Region will now automatically synchronize DKIM with the Parent Identity in the Parent Region and your apps/services can be configured to use either Parent or Replica Region & Identity to send DKIM authenticated email.
  2. Repeat the process if you want to create additional Replica Regions.

You can verify that the replica identity was configured correctly with the parent identity’s DKIM signing configuration by using the get-email-identity command and specifying the Replica’s domain name and region:

`aws sesv2 get-email-identity --email-identity [example.com](http://example.com/) —region us-west-2`

The response will include the value of the Parent Region in the DomainSigningAttributesOrigin parameter, signifying that the Replica Identity has been successfully configured with the Parent Identity’s DKIM signing configuration:

{
  "DkimAttributes": {
    "SigningAttributesOrigin": "AWS_SES_US_EAST_1"
  }
}

Conclusion

Deterministic Easy DKIM (DEED) represents a significant leap forward in multi-region email management for Amazon SES users. By eliminating the need for manually configured region-specific DNS configurations, DEED streamlines the process of expanding email operations across multiple AWS regions. This innovation not only reduces administrative overhead but also opens up new possibilities for organizations and ISVs to implement robust, globally distributed email infrastructures. With DEED, businesses can now leverage the full potential of Amazon SES across regions, ensuring consistent authentication, improved disaster recovery, and optimal performance without the previous complexities of multi-region setup.

Call to Action

Are you ready to simplify your multi-region email strategy with Amazon SES and DEED? Take the next step in optimizing your email infrastructure:

  1. Explore the DEED feature in your Amazon SES console today.
  2. Set up a test environment to experience the seamless multi-region configuration firsthand.
  3. For more detailed information, check out our comprehensive documentation on implementing DEED.
  4. Have questions or need assistance? Reach out to our AWS support team or join the AWS community forums to connect with other users.

Don’t let regional boundaries limit your email capabilities. Embrace the power of Deterministic Easy DKIM and transform your global email strategy with Amazon SES. Start your DEED journey today and unlock new levels of efficiency and scalability in your email operations.

Enhancing Message Reach: An Omnichannel Approach Using WhatsApp, SMS, and Email with AWS

Post Syndicated from Pavlos Ioannou Katidis original https://aws.amazon.com/blogs/messaging-and-targeting/enhancing-message-reach-an-omnichannel-approach-using-whatsapp-sms-and-email-with-aws/

SMS, WhatsApp, and email are essential channels for communication, each offering distinct advantages. SMS is known for its high deliverability and broad accessibility, making it a reliable choice for reaching users even without internet access. However, SMS can be costly, and delivery failures may lead to opportunity costs, such as potential lost sales if users cannot access their account or complete payment verification. WhatsApp provides a richer user experience at a lower cost but depends on internet connectivity and users having signed up for an account, which limits its reach. Email, while cost-effective and suitable for delivering detailed content, often faces issues including spam filtering and lower open rates.

By combining intelligent fallback mechanisms with broadcast capabilities, this versatile solution addresses a wide spectrum of communication needs, from time-sensitive transactional messages to expansive marketing campaigns. It sets the stage for diverse and impactful applications across various industries. The fallback mechanism dynamically switches to a backup channel if the primary one fails, ensuring that messages reach recipients. Meanwhile, the broadcast feature allows the simultaneous delivery of messages across multiple channels, maximizing reach and engagement. Together, these approaches extend message reach, improve user experience, and ensure reliable communication across platforms.

Use Cases

This multi-channel messaging solution is highly versatile, making it suitable for various use cases:

  • OTP/Transactional Messaging: Timely and reliable delivery is critical for one-time passwords (OTP) and transactional notifications. The solution ensures these important messages are promptly delivered, utilizing fallback mechanisms across SMS, WhatsApp, and email in case of channel limitations or failures.
  • Marketing Messages: This solution optimizes message delivery by selecting the most cost-effective and reliable channel while using the broadcast feature to deliver marketing messages across multiple channels, maximizing reach and engagement.
  • Broadcast Messages: The broadcast option can be especially useful in sending high-priority or time-sensitive messages to a wide audience. Messages can be sent across multiple channels at once, increasing the likelihood that recipients will see the message in their preferred communication method.

Technical Solution

The architecture employs several AWS services to ensure reliable, cost-efficient message delivery:

  • Amazon API Gateway: Accepts incoming API requests, including message details like content, recipient, fallback channels, and whether the message is broadcast or single-channel.
  • Amazon Simple Queue Service (SQS): Manages the queuing of messages for both primary and fallback channels, as well as for broadcast messages, ensuring smooth processing and delivery.
  • AWS Lambda: The Primary Message Handler Lambda function sends messages via the selected channel (SMS, WhatsApp, or email). If the primary delivery fails, the fallback process is handled by a secondary Lambda function. For broadcast messages, the message is sent to all specified channels simultaneously.
  • Amazon DynamoDB: Stores message data and delivery statuses, enabling tracking of messages through all channels, including broadcast messages.
  • Amazon Simple Notification Service (SNS): Publishes delivery status updates and triggers the next steps based on whether the message was delivered or failed.
  • Amazon SES & End User Messaging Service: Email messages are sent via Amazon SES, while AWS End User Messaging handles SMS and WhatsApp delivery.

For a deeper technical breakdown of each component and the deployment steps, refer to the Github repository.

Architecture Diagram

omnichannel-fallback-solution

Sending an SMS, WhatsApp, or Email

When sending a message, the system first attempts delivery via the primary channel specified in the API request, whether that’s SMS, WhatsApp, or email. The message details are processed by the API Gateway, which triggers the Primary Message Handler Lambda function:

  • SMS: The message is sent via the End User Messaging Service. A delivery report is awaited to confirm whether the message was successfully delivered.
  • WhatsApp: Similarly, the message is delivered via End User Messaging. Real-time read receipts and delivery statuses are tracked and updated in DynamoDB.
  • Email: Email messages are sent through Amazon Simple Email Service (SES). The system tracks delivery confirmation events (like success or bounce) and updates the status in DynamoDB.
    In all cases, if delivery fails or confirmation isn’t received within a specified period, the system triggers the fallback mechanism specified in the API request. This mechanism attempts to deliver the message through the secondary channel, ensuring message delivery via an alternate method.

For broadcast messages, the system simultaneously delivers the message across all specified channels, ensuring the widest possible reach. For example, an emergency notification might be broadcast via SMS, WhatsApp, and email, allowing recipients to receive the message through whichever channel they prefer.

Monitoring Delivery Status

The system tracks delivery statuses for all three channels:

  • SMS: Delivery confirmations are provided by the SMS provider and published to an SNS topic. This triggers a Lambda function, which updates the message status in DynamoDB.
  • WhatsApp: Real-time delivery and read receipts from WhatsApp are received and processed similarly to SMS. SNS triggers a Lambda function to update DynamoDB.
  • Email: Delivery status for emails is tracked via events from Amazon SES (such as success, bounce, or rejection). These events are published to SNS, which triggers an update of the delivery status in DynamoDB.

The monitoring configured by this solution is solely for the fallback mechanism. It’s recommended to configure separately monitoring that you might require for storing and analyzing message engagement events.

Fallback to Secondary Channel

If the initial message delivery fails, the fallback mechanism is initiated, based on the fallback channel specified in the API request. The Primary Message Handler Lambda function places a request in the Secondary Channel Fallback Queue via SQS. The Secondary Message Handler Lambda function retrieves this request and sends the message through the secondary channel (for example, sending the message via WhatsApp if SMS delivery fails).

Deployment

Visit this GitHub repository for a detailed ReadMe and deployment guide of this solution.

Conclusion

This multi-channel messaging solution ensures reliable and efficient communication across SMS, WhatsApp, and email. By leveraging AWS services, the system guarantees message delivery through fallback mechanisms and a broadcast option. Whether sending one-time passwords, marketing communications, or emergency broadcasts, this solution adapts to different needs while optimizing costs and improving reach across multiple channels.

Build a Two-Way Email-to-SMS Service with Amazon SES and Amazon End User Messaging

Post Syndicated from Cheng Wang original https://aws.amazon.com/blogs/messaging-and-targeting/build-a-two-way-email-to-sms-service-with-amazon-ses-and-amazon-end-user-messaging/

Introduction

Businesses and organizations today struggle to effectively communicate with their customers, employees, or other stakeholders across the diverse range of digital channels they now use. One common problem arises when the requirement to exchange information quickly and reliably extends beyond traditional email. This issue challenges organizations where recipients lack immediate access to email. This applies to field workers, remote teams, or customers who prefer to communicate via text messages. It is vital to bridge this gap between email and SMS communication for timely updates, urgent notifications, and seamless collaboration. However, separate management of these disparate channels independently proves cumbersome and leads to inefficiencies.

To address this challenge, one approach is to leverage Amazon Simple Email Service (SES) and Amazon End User Messaging services to create a robust, scalable, and cost-effective messaging system. This system seamlessly bridges the gap between email and SMS, enhances the reach and delivery of your messages and streamlines your communication workflows. Ultimately, this approach delivers a superior experience for your audience, ensuring that critical information reaches recipients through their preferred channels in a timely and efficient manner.

This blog post will delve into the step-by-step process of building a solution that enables both Email-to-SMS and SMS-to-Email communication. This solution allows you to send SMS messages using email and receive any SMS replies on the same email address you used to send the original message. Furthermore, you can continue the conversation by replying to the email you receive in response. By the end, you’ll have the knowledge and tools necessary to revolutionize your communication strategy and deliver a superior experience to your audience.

Here are some of the use cases for this solution:

  • Real estate agents can use this solution to send listing updates to clients via SMS, and then receive client inquiries and responses as emails.
  • Customer service team can leverage the Email to SMS functionality to proactively reach out to customers with important notifications. Customers are able to respond directly via SMS.
  • Retailers can use this solution to send order confirmations, shipping updates, and promotional offers to customers via SMS. Customers are able to respond with inquiries or feedback that are then received as emails.
  • Medical practices and hospitals can leverage the Email to SMS functionality to quickly notify patients of appointment reminders, prescription refills, or other time-sensitive information. Patients can then respond via SMS, which gets converted to an email that the healthcare staff can access.

Solution Overview

The following figure shows the high level architecture for this solution.

Figure 1: Two-Way Email-To-SMS architecture

  1. Email Users send an email to the email address formatted as “mobile-number@verified-domain”. Amazon SES email receiving receives the email and triggers a receipt rule.
  2. The email is published to Amazon Simple Notification Service (SNS) topic (EmailToSMS) based on the receipt rule action, which triggers an AWS Lambda function (ConvertEmailToSMS). The ConvertEmailToSMS Lambda function performs the following actions:
    1. Receives the event from SNS and constructs a text message using the email body content.
    2. The constructed message is then sent to the “mobile-number” in the destination email address using the “SendTextMessage” API from AWS End User Messaging SMS. This is achieved by using a phone number in AWS End User Messaging SMS as the origination identity.
    3. The SMS message ID and source email address are stored as items in the Amazon DynamoDB table (MessageTrackingTable). This tracks email addresses for replies from SMS users.
  3. Mobile Users receive the SMS, and they have the option to reply to the phone number with two-way SMS messaging
  4. AWS End User Messaging receives the incoming SMS message from the Mobile Users. It then publishes this message to a SNS topic (SMSToEmail) for two-way SMS integration, which triggers a Lambda function (ConvertSMSToEmail). The ConvertSMSToEmail Lambda function performs the following actions:
    1. Retrieves the item from “MessageTrackingTable” using “previousPublishedMessageId” (SMS message ID) from the SNS event, and locate the corresponding email address.
    2. Sends the SMS message body to the Email Users using SES. This step uses “mobile-number@verified-domain” as the source email address, and the email address retrieved from the previous step as the destination.
  5. Email Users receive the email, and they have the option to reply to the email to continue the conversation. Mobile Users will receive the latest reply from Email Users.

Walkthrough

This section will dive into the step-by step process for the deployment. There are 4 steps to deploy this solution:

  1. Configure SES verified identity for email receiving and sending.
  2. Deploy the CloudFormation stack for the Email to SMS functionality.
  3. Deploy the CloudFormation stack for the SMS to Email functionality.
  4. Set up two-way SMS messaging in AWS End User Messaging SMS.

Note: the Lambda code for this solution is developed based on phone numbers and long code as the supported origination identity in Australia. You need to adjust the Lambda code (“format_phone_number” function) accordingly for this to work in your country.

Refer to this GitHub repository for the solution source code.

Prerequisites

Prerequisites for this walkthrough:

  1. Administrator-level access to an AWS account
  2. A domain or subdomain that you own to create SES verified identity
  3. An origination identity that supports two-way messaging, following Choosing an origination identity for two-way messaging use cases. Simulator phone numbers are available if you are in the US
  4. A mobile phone to send and receive SMS
  5. An email address to send and receive emails

Step 1: Configure SES Verified Identity

Follow the steps outlined in Creating a domain identity to create a verified identity for your domain in your AWS account. Confirm your domain identity is in the “Verified” status before proceeding to the next step:

Figure 2: Verified Identity

Step 2: Deploy Email to SMS functionality

The following steps create a CloudFormation stack to deploy the required components for Email to SMS functionality:

  1. Sign in to your AWS account.
  2. Download the email-to-sms.yaml for creating the stack.
  3. Navigate to the AWS CloudFormation page.
  4. Choose Create stack, and then choose With new resources (standard).
  5. Choose Upload a template file and upload the CloudFormation template that you downloaded earlier: email-to-sms.yaml. Then choose Next.
  6. Enter the stack name Email-To-SMS.
  7. Enter the following values for the parameters:
    • RuleName: The name of your SES Rule Set and receipt rule.
    • Recipient1: Domain name used for recipient condition in the SES Rule Set.
    • Recipient2: Domain name used for recipient condition in the SES Rule Set if you need additional recipients.
    • PhoneNumberId: Phone number ID of the phone number to send SMS messages.
  8. Choose Next, and then optionally enter tags and choose Submit. Wait for the stack creation to finish.

Now you have the required components to convert email to text, and sending it as SMS to a phone number using AWS End User Messaging SMS.

Note: if required, modify the following code in email-to-sms.yaml to format your phone numbers accordingly:

def format_phone_number(email_address):

    # Extract the local part of the email address (before @)

    local_part = email_address.split('@')[0]   

    # Remove the leading '0' and add '+61' for phone number (Australia)

    if local_part.startswith('0'):

        formatted_number = '+61' + local_part[1:]

    return formatted_number

Step 3: Deploy SMS to Email functionality

The following steps create a CloudFormation stack to deploy the required components for SMS to Email functionality:

  1. Sign in to your AWS account.
  2. Download the sms-to-email.yaml for creating the stack.
  3. Navigate to the AWS CloudFormation page.
  4. Choose Create stack, and then choose With new resources (standard).
  5. Choose Upload a template file and upload the CloudFormation template that you downloaded earlier: sms-to-email.yaml. Then choose Next.
  6. Enter the stack name SMS-To-Email.
  7. Enter the following values for the parameters:
    • EmailDomain: The email domain to use for the SMS-to-Email function
  8. Choose Next, and then optionally enter tags and choose Submit. Wait for the stack creation to finish.

Note: if required, modify the following code in sms-to-email.yaml to format your phone numbers accordingly:

def format_phone_number(phone_number):

    # Replace the '+61' with '0' from the phone number (Australia)

    formatted_number = f"0{mobile_number[3:]}"

    return formatted_number

Step 4: Set up Two-Way Messaging in AWS End User Messaging SMS

Follow the steps 1 – 5 outlined in Set up two-way SMS messaging for a phone number in AWS End User Messaging SMS.

For step 6:

  • For Destination type, choose Amazon SNS.
  • Choose Existing SNS standard topic.
  • For Incoming messages destination, choose the SNS topic created from Step 3 (default topic name is SMSToEmailTopic).
  • For Two-way channel role, choose Use SNS topic policies.
  • Choose Save changes.

This allows your origination identity (phone number) to receive incoming messages, which is then published to an SNS topic and converted into emails by Lambda.

Testing

To test the solution, send an email with the destination address of “mobile-number@verified-domain”. You should receive a SMS on your mobile with the following information:

Note: AWS End User Messaging SMS has character limit for SMS messages depending on the type of characters the message contains. This solution takes the first 160 characters of the email body by default, you can adjust the EmailToSMS Lambda function as required.

Reply directly to the SMS, you should receive an email at the same email address that sent the original email, with the following information:

  • Subject: Re:mobile-number
  • Body: SMS message content
  • Source email address: mobile-number@verified-domain

If you are not receiving the email or SMS, check the Lambda CloudWatch logs for troubleshooting.

Cleaning up

To remove unneeded resources after testing the solution, follow these steps:

  1. In the CloudFormation console, delete the Email-To-SMS stack
  2. In the CloudFormation console, delete the SMS-To-Email stack
  3. If applicable, in Amazon SES, delete the verified identities
  4. If applicable, in AWS End User Messaging, release the unused phone numbers

Additional Consideration

Conclusion

This blog has explored how organizations can leverage AWS services to build a flexible, two-way communication solution bridging the gap between email and SMS channels. By integrating Amazon SES and Amazon End User Messaging, businesses can reach their audience through multiple channels, ensuring timely and effective delivery of critical messages.

The detailed guide provided the knowledge to create a scalable, cost-effective system tailored to evolving communication needs – whether facilitating email-to-SMS or SMS-to-email exchanges. This unified approach to email and SMS capabilities empowers companies to address the common challenge of managing disparate communication platforms, streamlining workflows and enhancing responsiveness.

If you run into issues or want to submit a feature request, use the New issue button under the issues tab in the GitHub repository.

How Amazon SES Mail Manager customers can prevent EchoSpoofing

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/how-ses-mail-manager-customers-can-prevent-echospoofing/

Customers not using Amazon SES Mail Manager, or those leveraging the authenticated SMTP functionality, are not at risk of EchoSpoofing. In such cases, no further action is required.

However, customers currently using or evaluating the unauthenticated SMTP relay feature of Amazon SES Mail Manager are strongly advised to review and implement the guidance provided in this blog post.

A new type of email spoofing attack

In July 2024, the researchers at Guardio Labs disclosed a new type of email spoofing (authentication bypass) attack they called “EchoSpoofing”. The attackers successfully sent spoofed emails by redirecting them through a virtual SMTP server, Office365 Exchange Online server, and a trusted third-party SMTP relay service. This path provided the fraudulent messages a means to pass standard authentication checks. Fortunately, the Guardio Labs researchers responsibly disclosed the issue to the targeted email security provider, leading to a speedy and effective remedy.

Unfortunately, before addressing the vulnerability completely, cybercriminals executed a series of sophisticated phishing campaigns. These campaigns involved sending millions of fraudulent emails that had valid Sender Policy Framework (SPF) and DomainKeys Identified Mail (DKIM) from well-known consumer brands.

The EchoSpoofing incident reminds both providers and customers to adopt a “trust-but-verify” approach to email security. This is especially true when mail routing functions have been, or are, in the process of being decoupled from single-tenant, on-premise (or cloud) email infrastructures.

As a leading provider of global managed email infrastructure, the Amazon SES service team went to work immediately after the Guardio Labs announcement. The Amazon SES team scrutinized the announcement and remediations undertaken by both Microsoft and the third-party SMTP relay service to fully understand the EchoSpoofing exploits and devised methods to swiftly safeguard Amazon SES customers.

Although we won’t delve into Guardio Labs’ in-depth analysis, it is crucial to grasp the attack’s main elements and examine how malicious individuals could exploit supposedly “secure” email relay systems. Amazon SES has taken steps to safeguard against EchoSpoofing and similar attacks, urging its customers to do the same.

Understanding the EchoSpoofing attack

The bad actors who implemented EchoSpoofing were able to send millions of well camouflaged malicious messages through the trusted delivery path of targeted organizations by preserving the SPF and DKIM attributes of the targeted sender’s domain. This greatly increased the likelihood of recipients trusting and acting upon the fake messages.

The attacker first set up a tenant in Microsoft Office365 and then delivered a spoofed email to that tenant, falsifying the from email headers. The attacker-controlled tenant was configured to relay the email to a security relay point linked to the forged sending identity. As the forged email came from trusted IP addresses belonging to Microsoft, the security relay point signed the message and relayed to the final recipients.

The attackers had amassed a large inventory of high-profile domains, and spread the EchoSpoof campaigns out across them to smooth the traffic and avoid sending spikes from any single domain. They carried out this attack for several months undetected, sending as many as 14 million messages per day, targeting the users of the compromised domains’ email services. This made the attack easy to automate at scale, difficult to detect via automated means, and highly successful in delivering malicious emails to unsuspecting recipients.

Guardio Labs’ discovery highlights the risks associated with an insecure SMTP relay model in a trusted domain configuration. This is of particular concern when permissive security policies allow fraudulent emails to be injected into the email flow without raising alarms.

The AWS shared security model for SES

As an email sender, Amazon SES is one of the largest on the internet, operating a worldwide fleet of trusted mail relays. Amazon SES maintains high IP reputations for this large fleet by maintaining a tight focus on robust, evolving security practices.

At AWS we operate under the shared security model. For Amazon SES, this means AWS takes responsibility for securing the underlying email delivery infrastructure, including the email servers, networks, and physical data centers. Customers take responsibility for securing their configurations, email content, sender authentication, and email lists they use with Amazon SES.

To ensure we meet our obligations in the shared security model, Amazon SES has recently added new features to Mail Manager SMTP relays that provide an increased level of protection to help guard against exploits like EchoSpoofing. Theses features are live in every AWS Region where Mail Manager is accessible.

We have outlined our recommendations and updated Amazon SES Mail Manager configurations in this blog to help customers meet their obligations and strengthen their Amazon SES email infrastructure against EchoSpoofing. As noted above, authenticated SMTP relays are not subject to this exploit.

Prevent EchoSpoofing when relaying email out of MailManager

If you need to relay email to a third party system that cannot enforce SMTP authentication, our recommendation is to limit access to the IP addresses used by Mail Manager in your region.

As of this writing, Mail Manager is generally available, and has its own IP range, in six commercial regions (below). As Amazon expands SES Mail Manager availability into more regions the IP ranges will be updated in the Amazon SES documentation .

SES Mail Manager regions

SES Mail Manager IP ranges as of 10/23/2024

When unable to enforce SMTP authentication, we recommend configuring your SMTP servers, or third party software for the new MIME header"X-MAIL-MANAGER-ORIGINATOR-ORG". This new Mail Manager header is now automatically inserted into messages relayed by Mail Manager. The X-MAIL-MANAGER-ORIGINATOR-ORG will be set to the customer’s unique SMTP relay ID, which can be found via the Mail Manager console or the ListSmtpRelays API.

In addition to added security, the MIME header feature can also be used in message search and filtering behaviors for a wide range of MIME header name:value pairs.

If the original email already contains an X-MAIL-MANAGER-ORIGINATOR-ORG header, it will be replaced with the last MailManager SMTP relay ID to relay the email. Here is an example of an email relayed by MailManager with the header:

MIME-Version: 1.0
From: [email protected]
To: [email protected]
Subject: Test
X-SES-REDIRECT-MESSAGE-ID: <[email protected]>
X-MAIL-MANAGER-ORIGINATOR-ORG: rl-usmoots8mgmfgfaeijckxhqx
X-SES-Outgoing: 2024.08.26-76.223.191.14

--===============1760803815732220490==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

This is a sample message. Have a nice day.
--===============1760803815732220490==--

This approach elevates your security posture because the IP access control lists on your third party system ensures only mail from Amazon SES is accepted, and the MIME header check can be trusted and checked.

Prevent EchoSpoofing when relaying into MailManager

When relaying email from a third party into Amazon SES Mail Manager, you will similarly need to configure an IP allowlisting, and if the email comes from a shared or cloud environment, you will need an additional header check to disambiguate among the multiple tenants it hosts. Those IPs and headers are provider specific, for example, emails coming from Office365 will have a header called X-OriginatorOrg.

You can use the rule editor screen in MailManager to configure the check in Mail Manager for the IPs and 3rd party headers before executing any action.

Mail Manager Rules Detail Page

The verification of a MIME header is not necessary if the third party relaying into MailManager uses an IP dedicated for your tenant. In that case, there is no possibility that an attacker tenant injects email into MailManager using your IP.

Conclusion

While the conditions that made the EchoSpoofing exploit were highly specific, they reminded us all of the importance of taking a proactive approach to email security.

The chances of your Amazon SES Mail Manager unauthenticated SMTP relay being compromised are low, but we highly advise you follow the recommendations in this blog post promptly. You’ll find more information in the Amazon SES documentation ( here ).

If you need help with securing your Amazon SES Mail Manager SMTP relay actions against EchoSpoofing, contact AWS support, or leave us a comment in community section of the blog post.

Call to Action:

If you are using SES Mail Manager’s unauthenticated SMTP relay today, follow the guidance in this blog to secure your email infrastructure today by configuring the recommended ACLs and MIME header verification in AWS SES Mail Manager.“

Stay ahead of emerging threats by subscribing to this AWS blog where we post the latest security updates as well as new features and interesting use cases for SES.

Join the conversation and share your best practices for email security with the AWS community.

Explore the new MIME header evaluation feature in AWS SES Mail Manager and share your creative use cases with us and the SES community via the community comments section of the blog.

About the Authors

Toby Weir-Jones

Toby Weir-Jones

Toby is a Principal Product Manager for Amazon SES and WorkMail. He joined AWS in January 2021 and has significant experience in both business and consumer information security products and services. His focus on email solutions at SES is all about tackling a product that everyone uses and finding ways to bring innovation and improved performance to one of the most ubiquitous IT tools.

Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

Leandro Batista Lameiro

Leandro Batista Lameiro

Leandro is a Sr. Software Dev Engineer at AWS.

Linzhou Zhong

Linzhou Zhong

Linzhou is a software engineer at AWS.

Automate AWS End User Messaging US toll-free Number Registrations

Post Syndicated from Bruno Giorgini original https://aws.amazon.com/blogs/messaging-and-targeting/automate-us-tfn-registrations/

Introduction:

AWS End User Messaging enables customers to send SMS messages to recipients located across the globe.
If you are planning on rolling out SMS across multiple countries, we recommend reviewing this blog. When you send SMS or MMS messages using AWS End User Messaging SMS, you must use a specific origination identity that supports the sending of SMS. In the US the originators that are supported are Toll-Free(TFN), 10DLC, and short codes and each of these has its own registration process.

This blog discusses the process of programmatically registering Toll-Free numbers (TFNs), which can be used exclusively within the United States. TFNs have a max throughput of 3 Messages Per Second(MPS) and are typically used for low-volume/throughput use cases. You can learn more about TFNs here.

TFNs have a relatively simple registration process, however, for Independent Software Vendors (ISVs), SaaS providers, or large organizations with many teams or use cases, completing dozens, or hundreds of TFN registrations manually can be time-consuming, tedious, and error-prone. AWS End User Messaging APIs can be used to streamline the TFN registration process by enabling AWS customers to programmatically register TFNs in the USA. You can learn more about the TFN registration requirements and process here.

In this post, we will discuss the AWS End User Messaging APIs required for programmatically registering TFNs. We’ll explore a simple Bash script you can use to automate the registration of a single TFN, as well as a Python script that you can use to bulk register multiple TFNs. Both of these scripts help simplify the TFN registration process and can save both time and effort for businesses looking to leverage AWS End User Messaging for their communication needs.

Outline of the APIs used in the scripts:

There are seven actions that need to be taken in order for a TFN registration to be created and submitted. The two scripts in this post automate these actions:

  1. create-registration
    • Creates a new registration
    • The “RegistrationType” field controls whether this is a registration for a Toll-Free, or 10DLC or a SenderID.
  2. describe-registration-field-definitions
    • Retrieves the specified RegistrationType field definitions.
    • You can use DescribeRegistrationFieldDefinitions to view the requirements for creating, filling out, and submitting each registration type. In this post we have chosen Toll-Free and provided the values.
  3. create-registration-attachment
    • The carriers require you to provide a mockup that closely resemble the opt-in experience that your customers will complete to ensure end-users have consented to receiving SMS messages from this toll-free number. This should be a screenshot of your website or mobile application’s opt-in workflow.
    • The maximum file size is 500kb, and valid file extensions are PDF, JPEG, or PNG.
    • You need to wait until the attachment is uploaded before moving to step 4.
  4. put-registration-field-value
    • This action needs to be repeated for all required fields (retrieved in step 2)
  5. request-phone-number
    • Request an origination phone number for use in your account (needed for step 6)
  6. create-registration-association
    • This associates the registration with the origination identity (phone number) you requested in step 5.
    • You will need the PhoneNumberId for the origination identity.
  7. submit-registration-version
    1. This will submit the specified registration for review and approval
    2. IMPORTANT:
      • Make sure that all of your data is correct, especially before attempting to submit multiple registrations using the second script option below.
      • Once your script has submitted the registration, it’s initial status will be “CREATED” and should change to “REVIEWING” within 24 hours.
      • Once submitted, you will be unable to edit or delete this registration until it is approved or rejected by the third-party registrar that controls the registration process for the type of registration you are submitting (in this blog we’re registering TFNs).
      • You begin incurring costs for the TFNs as soon as you successfully submit the registration.

No matter which script you decide to use, it’s important to keep a these things in mind:

  1. The registration information you provide will be reviewed by a third-party company. This is standard, no matter which provider you use for SMS
  2. While AWS submits the registration on your behalf, we do not participate in the actual review process, nor can we influence the third-party.
  3. The review process for TFNs can take up to 15 business days and if the information provided in the registration process is incomplete, inaccurate, or the use case falls into a forbidden category (Guidelines for using toll-free numbers) , the application can be rejected.
  4. If your toll-free number registration is rejected, the status of your registration will change to “Requires Updates“. We recommend monitoring your registration status frequently to ensure that you do not miss an update and extend the time it takes to complete.

Prerequisites for either approach:

Before running either script, you’ll need:

  1. An AWS account with permission to use/provision the AWS End User Messaging service (link) in the target region.

The following information for each TFN you wish to register/request (you can retrieve this list with describe-registration-field-definitions API):

Company Info
  • Company Name
  • Website
  • Address 1
  • Address 2 (optional)
  • City
  • State / Province
  • Zip Code / Postal Code
  • Country Code
Contact Info
  • First Name
  • Last Name
  • Support Email
  • Support Phone Number
Messaging Use Case
  • Monthly Message Volume (you must use one of the options below)
    • “10”
    • “100”
    • “1,000”
    • “10,000”
    • “100,000”
    • “250,000”
    • “500,000”
    • “750,000”
    • “1,000,000”
    • “5,000,000”
    • “10,000,000+”
  • Use Case Category (you must use one of the options below)
    • “Two-factor authentication”
    • “One-time passcodes”
    • “Notifications”
    • “Polling and surveys”
    • “Info on demand”
    • “Promotions & marketing”
    • “Other”
  • Use Case Details
  • Opt-In Description – The primary purpose of the Opt-in Description is to demonstrate that the end user explicitly consents to receive text messages and understands the nature of the program. Your application is being reviewed by a 3rd party reviewer, so make sure to provide clear and thorough information about how your end-users opt-in to your SMS service and any associated fees or charges. If the reviewer cannot determine how your opt-in process works then your application will be denied and returned. If your Opt-in process requires a log-in, is not yet published publicly, is a verbal opt-in, or if it occurs on printed sources such as fliers and paper forms then make sure to thoroughly document how this process is completed by the end-user receiving messages. Provide a screenshot and host the screenshot on a publicly accessible website (like OneDrive or Google Drive) and provide the URL in the text as well as in the Opt-In Image below.
  • Opt-In Image – This is optional, but highly recommended, even if you provide a description above:
    • If your experience is not publicly available, the carriers will require you to provide one to ensure end-users have consented to receiving SMS messages from this toll-free number.
    • You are encouraged to provide a screenshot (PDF, JPEG, or PNG) of your website or mobile application’s opt-in workflow. This could also be a screenshot of a paper form, a verbal opt-in script, or an as yet published website experience in development.
    • The maximum file size is 500Kb.
Message Samples
  • Message Sample 1
  • Message Sample 2 (optional)
  • Message Sample 3 (optional)

Option 1 – Execute a script to register a single TFN

The script requires the additional prerequisites (if you use AWS CloudShell to run the script, the AWS CLI and jq are already installed).

  • AWS CLI installed and configured with appropriate AWS account credentials and region (link)
  • jq (a lightweight and flexible command-line JSON processor) installed (link)
  1. Set environment variables for each of the required fields.

Before running the registration script, you must first set the necessary variables, replacing the placeholder values with your company actual data (these values are listed as a prerequisite above). Ensure that the opt-in screenshot image file optInImage.png is in the same directory as the script, or provide the correct path to the image file in the attachment_body variable. Note – the allowed file types for the opt-in screenshot are PNG, JPG, and PDF with a max file size of 500 kb.

Run the environment variable exports:

export AWS_REGION="us-west-2"
export COMPANY_NAME="Your Company Name"
export COMPANY_WEBSITE="www.yourcompany.com"
export COMPANY_ADDRESS1="123 Main Street"
export COMPANY_ADDRESS2="Suite 200"
export COMPANY_CITY="Your City"
export COMPANY_STATE="Your State"
export COMPANY_ZIPCODE="12345"
export COMPANY_COUNTRY_CODE="US"
export CONTACT_FIRSTNAME="John"
export CONTACT_LASTNAME="Doe"
export CONTACT_EMAIL="[email protected]"
export CONTACT_PHONE="+1234567890"
export USECASE_DETAILS="Details about your use case"
export OPTIN_DESCRIPTION="Description of opt-in process"
export MONTHLY_MESSAGE_VOLUME="10"
export USECASE_CATEGORY="Other"
export MESSAGE_SAMPLE1="Sample message"
export ATTACHMENT_BODY="fileb://optInImage.png"
  1. Copy and save the below script as register_us_toll_free_number.sh into your current directory. Adjust the region variable in the script to the one you’re using to register the US TFN.
#!/bin/bash
set -euo pipefail

# Check dependencies
command -v aws >/dev/null 2>&1 || { echo "AWS CLI is required but it's not installed. Aborting."; exit 1; }
command -v jq >/dev/null 2>&1 || { echo "jq is required but it's not installed. Aborting."; exit 1; }

# Initialize variables from environment or use default values
region="${AWS_REGION:-'us-west-2'}"
companyInfo_companyName="${COMPANY_NAME:-"Default Company Name"}"
companyInfo_website="${COMPANY_WEBSITE:-"https://defaultcompany.com"}"
companyInfo_address1="${COMPANY_ADDRESS1:-"123 Default St."}"
companyInfo_address2="${COMPANY_ADDRESS2:-""}"  # Optional
companyInfo_city="${COMPANY_CITY:-"Default City"}"
companyInfo_state="${COMPANY_STATE:-"Default State"}"
companyInfo_zipCode="${COMPANY_ZIPCODE:-"00000"}"
companyInfo_isoCountryCode="${COMPANY_COUNTRY_CODE:-"US"}"
contactInfo_firstName="${CONTACT_FIRSTNAME:-"John"}"
contactInfo_lastName="${CONTACT_LASTNAME:-"Doe"}"
contactInfo_supportEmail="${CONTACT_EMAIL:-"[email protected]"}"
contactInfo_supportPhoneNumber="${CONTACT_PHONE:-"+10000000000"}"
messagingUseCase_useCaseDetails="${USECASE_DETAILS:-"Default use case details"}"
messagingUseCase_optInDescription="${OPTIN_DESCRIPTION:-"Default opt-in process description"}"
messagingUseCase_monthlyMessageVolume="${MONTHLY_MESSAGE_VOLUME:-"10"}"
messagingUseCase_useCaseCategory="${USECASE_CATEGORY:-"Other"}"
messageSamples_messageSample1="${MESSAGE_SAMPLE1:-"This is a sample message."}"
attachment_body="${ATTACHMENT_BODY:-"fileb://optInImage.png"}"  # Path to the image, can be overridden

# Log to file
log_file="registration.log"
echo "Logging to $log_file"
exec > >(tee -a "$log_file") 2>&1

# Retry configuration
max_retries=10
retry_interval=2
retry_count=0
upload_complete=false

# Helper function to put registration field values
put_registration_field_value() {
    local field_path="$1"
    local value="$2"
    aws pinpoint-sms-voice-v2 --region "$region" put-registration-field-value \
         --registration-id "$registration_id" \
         --field-path "$field_path" \
         --text-value "$value" || { echo "Failed to update $field_path"; exit 1; }
}

# Helper function to validate registration field values
validate_input() {
    local required_vars=(
        "AWS_REGION"
        "COMPANY_NAME"
        "COMPANY_WEBSITE"
        "COMPANY_ADDRESS1"
        "COMPANY_CITY"
        "COMPANY_STATE"
        "COMPANY_ZIPCODE"
        "COMPANY_COUNTRY_CODE"
        "CONTACT_FIRSTNAME"
        "CONTACT_LASTNAME"
        "CONTACT_EMAIL"
        "CONTACT_PHONE"
        "USECASE_DETAILS"
        "OPTIN_DESCRIPTION"
        "MONTHLY_MESSAGE_VOLUME"
        "USECASE_CATEGORY"
        "MESSAGE_SAMPLE1"
        "ATTACHMENT_BODY"
    )

    for var in "${required_vars[@]}"; do
        if [ -z "${!var}" ]; then
            echo "Error: $var is missing or empty" >&2
            return 1
        fi
    done

    # Validate zip code
    if [[ ! "$COMPANY_ZIPCODE" =~ ^[0-9]+$ ]]; then
        echo "Error: Invalid or missing zip code" >&2
        return 1
    fi

    # Validate monthly message volume
    local valid_volumes=("10" "100" "1,000" "10,000" "100,000" "250,000" "500,000" "750,000" "1,000,000" "5,000,000" "10,000,000+")
    local volume_valid=false
    for volume in "${valid_volumes[@]}"; do
        if [ "$MONTHLY_MESSAGE_VOLUME" = "$volume" ]; then
            volume_valid=true
            break
        fi
    done
    if [ "$volume_valid" = false ]; then
        echo "Error: Invalid or missing monthly_message_volume" >&2
        return 1
    fi

    # Validate use case category
    local valid_categories=("Two-factor authentication" "One-time passcodes" "Notifications" "Polling and surveys" "Info on demand" "Promotions & marketing" "Other")
    local category_valid=false
    for category in "${valid_categories[@]}"; do
        if [ "$USECASE_CATEGORY" = "$category" ]; then
            category_valid=true
            break
        fi
    done
    if [ "$category_valid" = false ]; then
        echo "Error: Invalid or missing use_case_category" >&2
        return 1
    fi

    # Additional validations (you can add more as needed)
    if [[ ! "$COMPANY_WEBSITE" =~ ^https?:// ]]; then
        echo "Error: COMPANY_WEBSITE must start with http:// or https://" >&2
        return 1
    fi
    
    if [[ ! "$CONTACT_EMAIL" =~ ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ ]]; then
        echo "Error: CONTACT_EMAIL is not a valid email address" >&2
        return 1
    fi
    
    if [[ ! "$CONTACT_PHONE" =~ ^\+[1-9][0-9]{10,14}$ ]]; then
        echo "Error: CONTACT_PHONE must be in E.164 format (e.g., +10000000000)" >&2
        return 1
    fi
    
    echo "All required variables are set and valid"
    return 0
}


# Step 1: Create registration
echo "Creating registration..."
create_registration_output=$(aws pinpoint-sms-voice-v2 --region "$region" create-registration \
      --registration-type 'US_TOLL_FREE_REGISTRATION' \
      --tags Key=Name,Value='Registration friendly name') || { echo "Failed to create registration"; exit 1; }

# Extract RegistrationId
registration_id=$(echo "$create_registration_output" | jq -r '.RegistrationId')
if [ -z "$registration_id" ]; then
    echo "Error: Registration ID not found!"
    exit 1
fi
echo "Registration ID: $registration_id"

# Step 2: Create registration attachment
echo "Creating registration attachment..."
create_attachment_output=$(aws pinpoint-sms-voice-v2 --region "$region" create-registration-attachment \
      --attachment-body "$attachment_body") || { echo "Failed to create registration attachment"; exit 1; }

# Extract RegistrationAttachmentId
attachment_id=$(echo "$create_attachment_output" | jq -r '.RegistrationAttachmentId')
if [ -z "$attachment_id" ]; then
    echo "Error: Attachment ID not found!"
    exit 1
fi
echo "Attachment ID: $attachment_id"

# Step 3: Wait for the attachment to be fully uploaded
echo "Waiting for attachment upload to complete..."
while [ "$upload_complete" = false ] && [ $retry_count -lt $max_retries ]; do
    # Describe the registration attachment
    describe_attachment_output=$(aws pinpoint-sms-voice-v2 --region "$region" describe-registration-attachments \
        --registration-attachment-ids "$attachment_id")
    
    # Extract the attachment status
    attachment_status=$(echo "$describe_attachment_output" | jq -r '.RegistrationAttachments[0].AttachmentStatus')

    # Check if the status is 'UPLOAD_COMPLETE'
    if [ "$attachment_status" == "UPLOAD_COMPLETE" ]; then
        echo "Attachment upload complete."
        upload_complete=true
    else
        echo "Attachment status: $attachment_status. Retrying in $retry_interval seconds... (attempt $((retry_count + 1))/$max_retries)"
        sleep "$retry_interval"
        retry_count=$((retry_count + 1))
    fi
done
if [ "$upload_complete" = false ]; then
    echo "Attachment upload did not complete within the retry limit."
    exit 1
fi
echo "Attachment upload complete."

# Step 4a: Put registration field values from environment variables
text_field_paths=(
    "companyInfo_companyName"
    "companyInfo_website"
    "companyInfo_address1"
    "companyInfo_address2"
    "companyInfo_city"
    "companyInfo_state"
    "companyInfo_zipCode"
    "companyInfo_isoCountryCode"
    "contactInfo_firstName"
    "contactInfo_lastName"
    "contactInfo_supportEmail"
    "contactInfo_supportPhoneNumber"
    "messagingUseCase_useCaseDetails"
    "messagingUseCase_optInDescription"
    "messageSamples_messageSample1"
)

for text_field_path in "${text_field_paths[@]}"; do
    echo "Putting registration field value for: $text_field_path"
    value=$(eval "echo \$$text_field_path")
    if [ -z "$value" ]; then
        echo "Error: $text_field_path is empty or not set. Skipping."
        continue
    fi
    echo "Value for $text_field_path is: '$value'"
    put_registration_field_value "${text_field_path//_/.}" "$value"
done

# Step 4b: Put other registration field values (Choice fields)
choice_field_paths=(
    "messagingUseCase_monthlyMessageVolume"
    "messagingUseCase_useCaseCategory"
)

for choice_field_path in "${choice_field_paths[@]}"; do
    echo "Putting registration field value for: $choice_field_path"
    value=$(eval "echo \$$choice_field_path")
    if [ -z "$value" ]; then
        echo "Error: $choice_field_path is empty or not set. Skipping."
        continue
    fi
    echo "Value for $choice_field_path is: '$value'"
    aws pinpoint-sms-voice-v2 --region "$region" put-registration-field-value \
        --registration-id "$registration_id" \
        --field-path "${choice_field_path//_/.}" \
        --select-choices "$value" || { echo "Failed to update $choice_field_path"; exit 1; }
done

# Step 5: Associate registration attachment with registration
echo "Associating registration attachment with registration..."
aws pinpoint-sms-voice-v2 --region "$region" put-registration-field-value \
      --registration-id "$registration_id" \
      --field-path 'messagingUseCase.optInImage' \
      --registration-attachment-id "$attachment_id" || { echo "Failed to associate registration attachment"; exit 1; }

# Step 6: Request phone number
echo "Requesting phone number..."
request_phone_number_output=$(aws pinpoint-sms-voice-v2 --region "$region" request-phone-number \
      --iso-country-code 'US' \
      --number-type 'TOLL_FREE' \
      --number-capabilities 'SMS' 'MMS' 'VOICE' \
      --message-type 'TRANSACTIONAL') || { echo "Failed to request phone number"; exit 1; }

# Extract PhoneNumberId
phone_number_id=$(echo "$request_phone_number_output" | jq -r '.PhoneNumberId')
if [ -z "$phone_number_id" ]; then
    echo "Error: Phone number ID not found!"
    exit 1
fi
echo "Phone Number ID: $phone_number_id"

# Step 7: Associate phone number with registration
echo "Associating phone number with registration..."
aws pinpoint-sms-voice-v2 --region "$region" create-registration-association \
      --registration-id "$registration_id" \
      --resource-id "$phone_number_id" || { echo "Failed to associate phone number with registration"; exit 1; }

# Step 8: Submit registration
echo "Submitting registration..."
aws pinpoint-sms-voice-v2 --region "$region" submit-registration-version \
     --registration-id "$registration_id" || { echo "Failed to submit registration"; exit 1; }

echo "Registration submitted successfully!"
  1. Run chmod command to make it executable. Run the script with the environment variable values you provided in Step 1 above.

chmod u+x register_us_toll_free_number.sh
./register_us_toll_free_number.sh
  1. The script completes with the following lines:

Submitting registration...
{
    "RegistrationArn": "arn:aws:sms-voice:us-west-2:637400000000:registration/registration-244aa3f91fcc49b598bdf1234569",
    "RegistrationId": "registration-244aa3f91fcc49b598bdf1234569",
    "VersionNumber": 1,
    "RegistrationVersionStatus": "SUBMITTED",
    "RegistrationVersionStatusHistory": {
        "DraftTimestamp": "2024-09-11T19:44:41+00:00",
        "SubmittedTimestamp": "2024-09-11T19:45:16+00:00"
    }
}
Registration submitted successfully!

Option 2 – Execute a Python script to register more than one Toll-Free number

For ISVs, SaaS providers, or large organizations registering multiple phone numbers on behalf of customers or business units, automating the process can save time and reduce errors. We’ve chosen to use Python because it’s CSV library handles file parsing complexities, and its boto3 library enables seamless interaction with AWS services, making it well-suited for programmatically managing these registrations.

  1. Below is a sample CSV file with the variables for 5 registrations, copy it and replace the sample data with your actual data (these data were listed in the prerequisites above). Save your file as data.csv:
companyInfo_companyName,companyInfo_website,companyInfo_address1,companyInfo_address2,companyInfo_city,companyInfo_state,companyInfo_zipCode,companyInfo_isoCountryCode,contactInfo_firstName,contactInfo_lastName,contactInfo_supportEmail,contactInfo_supportPhoneNumber,messagingUseCase_monthlyMessageVolume,messagingUseCase_useCaseCategory,messagingUseCase_useCaseDetails,messagingUseCase_optInDescription,messageSamples_messageSample1,attachmentFilePath
AnyCompany,https://example.com/example1,123 Any Street,Suite 200,Anytown,WA,98109,US,FirstName,LastName,[email protected],15553331234,10,One-time passcodes,Dev/Demo - Internal testing only,Internal testing only,Your AWS End User Messaging internal testing one time passcode is 123456,./companyA.png
Example Corp,https://example.net/example2,123 Main Street,,Anywhere,WA,98765,US,FirstName2,LastName2,[email protected],15553332222,10,Notifications,Dev/Demo - Internal testing only,Internal testing only,Your AWS End User Messaging internal testing one time passcode is 654321,./companyB.png
AnyDepartment,https://example.org/example3,123 Oak Avenue,Apt B,Nowhere,WA,54321,US,FirstName3,LastName3,[email protected],15553333333,10,Two-factor authentication,Dev/Demo - Internal testing only,Internal testing only,Your AWS End User Messaging internal testing one time passcode is 987654,./companyC.png
AnyOrganization,https://example.com/example4,123 Pine Blvd,,Anywhere Else,WA,12345,US,FirstName4,LastName4,[email protected],15553334444,100,One-time passcodes,Dev/Demo - Internal testing only,Internal testing only,Your AWS End User Messaging internal testing one time passcode is 654789,./companyD.png
AnyGovernment,https://example.org/example5,123 Oak St,,Nowhere Town,WA,67890,US,FirstName5,LastName5,[email protected],15553335555,"100",One-time passcodes,Dev/Demo - Internal testing only,Internal testing only,Your AWS End User Messaging internal testing one time passcode is 987654,./companyE.png
  1. Prepare opt-in screenshot image files for each registration, and upload them in the same directory as you will use for the data.csv and script files. Note: the allowed file types for the opt-in screenshot are PNG, JPG, and PDF with a max file size of 500 kb.
  2. Below is a Python script to create registrations using the data.csv file (above). Save this file as register_phone_numbers_bulk.py into your current directory.
import csv
import boto3
import sys
import argparse
import time

# Parse command line arguments
parser = argparse.ArgumentParser(description="AWS End User Messaging SMS Registration Script")
parser.add_argument("-r", "--region", default="us-west-2", help="AWS region")
parser.add_argument("-c", "--csv_file", default="data.csv", help="Path to the Comma-Separated Values file")
args = parser.parse_args()

# Initialize AWS End User Messaging SMS client
client = boto3.client('pinpoint-sms-voice-v2', region_name=args.region)

def process_csv(csv_file):
    with open(csv_file, newline='', encoding='utf-8') as f:
        reader = csv.DictReader(f)
        
        # Strip BOM from the first key (header) if it exists
        if '\ufeff' in reader.fieldnames[0]:
            reader.fieldnames[0] = reader.fieldnames[0].replace('\ufeff', '')
        
        for row in reader:
            process_row(row)

def process_row(row):
    print(f"================================================================================================")
    
    if 'companyInfo_companyName' not in row:
        print(f"Error: 'companyInfo_companyName' field is missing in the CSV file.")
        return
    
    print(f"Processing entry for: {row['companyInfo_companyName']}")
    print(f"================================================================================================")

    # Validate required fields
    required_fields = [
        'companyInfo_companyName', 'companyInfo_website', 'companyInfo_isoCountryCode', 
        'attachmentFilePath'
    ]
    for field in required_fields:
        if field not in row or not row[field]:
            print(f"Error: Required field '{field}' is missing or empty for entry: {row.get('companyInfo_companyName', 'Unknown Company')}")
            return

    # Validate field values
    if 'companyInfo_zipCode' not in row or not row['companyInfo_zipCode'].isdigit():
        print(f"Error: Invalid or missing zip code for entry: {row['companyInfo_companyName']}")
        return

    valid_volumes = ["10", "100", "1,000", "10,000", "100,000", "250,000", "500,000", "750,000", "1,000,000", "5,000,000", "10,000,000+"]
    if 'messagingUseCase_monthlyMessageVolume' not in row or row['messagingUseCase_monthlyMessageVolume'] not in valid_volumes:
        print(f"Error: Invalid or missing monthly_message_volume for entry: {row['companyInfo_companyName']}")
        return

    valid_categories = ["Two-factor authentication", "One-time passcodes", "Notifications", "Polling and surveys", "Info on demand", "Promotions & marketing", "Other"]
    if 'messagingUseCase_useCaseCategory' not in row or row['messagingUseCase_useCaseCategory'] not in valid_categories:
        print(f"Error: Invalid or missing use_case_category for entry: {row['companyInfo_companyName']}")
        return

    try:
        # Step 1: Create registration
        print("Creating registration...")
        create_registration_output = client.create_registration(
            RegistrationType='US_TOLL_FREE_REGISTRATION'
        )
        registration_id = create_registration_output['RegistrationId']

        # Step 2: Put registration field values from CSV
        print("Putting registration field values...")
        field_mappings = {
            "companyInfo.companyName": {"value": row['companyInfo_companyName'], "type": "text"},
            "companyInfo.website": {"value": row['companyInfo_website'], "type": "text"},
            "companyInfo.address1": {"value": row['companyInfo_address1'], "type": "text"},
            "companyInfo.address2": {"value": row['companyInfo_address2'], "type": "text"},            
            "companyInfo.city": {"value": row['companyInfo_city'], "type": "text"},
            "companyInfo.state": {"value": row['companyInfo_state'], "type": "text"},
            "companyInfo.zipCode": {"value": row['companyInfo_zipCode'], "type": "text"},
            "companyInfo.isoCountryCode": {"value": row['companyInfo_isoCountryCode'], "type": "text"},
            "contactInfo.firstName": {"value": row['contactInfo_firstName'], "type": "text"},
            "contactInfo.lastName": {"value": row['contactInfo_lastName'], "type": "text"},
            "contactInfo.supportEmail": {"value": row['contactInfo_supportEmail'], "type": "text"},
            "contactInfo.supportPhoneNumber": {"value": row['contactInfo_supportPhoneNumber'], "type": "text"},
            "messagingUseCase.useCaseDetails": {"value": row['messagingUseCase_useCaseDetails'], "type": "text"},
            "messagingUseCase.optInDescription": {"value": row['messagingUseCase_optInDescription'], "type": "text"},
            "messageSamples.messageSample1": {"value": row['messageSamples_messageSample1'], "type": "text"},
            "messagingUseCase.monthlyMessageVolume": {"value": row['messagingUseCase_monthlyMessageVolume'], "type": "select"},
            "messagingUseCase.useCaseCategory": {"value": row['messagingUseCase_useCaseCategory'], "type": "select"},
        }

        for field, field_data in field_mappings.items():
            if field_data['type'] == 'text' and field_data['value']:
                client.put_registration_field_value(
                    RegistrationId=registration_id,
                    FieldPath=field,
                    TextValue=field_data['value']
                )
            elif field_data['type'] == 'select' and field_data['value']:
                client.put_registration_field_value(
                    RegistrationId=registration_id,
                    FieldPath=field,
                    SelectChoices=[field_data['value']]
                )

        # Step 3: Create registration attachment
        print("Creating registration attachment...")
        with open(row['attachmentFilePath'], 'rb') as attachment_file:
            create_attachment_output = client.create_registration_attachment(
                AttachmentBody=attachment_file.read()
            )
            attachment_id = create_attachment_output['RegistrationAttachmentId']

        # Step 4: Wait for the attachment upload to finish
        print("Verifying attachment upload complete...")
        attachment_id = create_attachment_output['RegistrationAttachmentId']
        upload_complete = False
        max_retries = 10
        retry_count = 0

        while not upload_complete and retry_count < max_retries:
            try:
                describe_attachment_output = client.describe_registration_attachments(
                    RegistrationAttachmentIds=[attachment_id]
                )
                attachment_status = describe_attachment_output['RegistrationAttachments'][0]['AttachmentStatus']
                if attachment_status == 'UPLOAD_COMPLETE':
                    upload_complete = True
                else:
                    print(f"Attachment status: {attachment_status}. Waiting 2 seconds before retrying... (max {max_retries} retries)")
                    time.sleep(2)
                    retry_count += 1
            except Exception as e:
                print(f"Error occurred while checking attachment status: {str(e)}")
                time.sleep(10)
                retry_count += 1

        if not upload_complete:
            print(f"Error: Attachment upload did not complete for {row['companyInfo_companyName']}")
            return

        # Step 5: Associate registration attachment with registration
        print("Associating registration attachment with registration...")
        client.put_registration_field_value(
            RegistrationId=registration_id,
            FieldPath='messagingUseCase.optInImage',
            RegistrationAttachmentId=attachment_id
        )
        
        # Step 6: Request phone number
        print("Requesting phone number...")
        request_phone_number_output = client.request_phone_number(
            IsoCountryCode='US',
            NumberType='TOLL_FREE',
            NumberCapabilities=['SMS', 'MMS', 'VOICE'],
            MessageType='TRANSACTIONAL'
        )
        phone_number_id = request_phone_number_output['PhoneNumberId']

        # Step 7: Associate phone number with registration
        print("Associating phone number with registration...")
        client.create_registration_association(
            RegistrationId=registration_id,
            ResourceId=phone_number_id
        )
                
        # Step 8: Submit registration
        print("Submitting registration...")
        client.submit_registration_version(
            RegistrationId=registration_id
        )
        print(f"Processing completed for: {row['companyInfo_companyName']}\n")

    except Exception as e:
        print(f"Error occurred while processing {row['companyInfo_companyName']}: {str(e)}")

if __name__ == "__main__":
    process_csv(args.csv_file)
  1. Run the script: python register_phone_numbers_bulk.py -c data.csv -r us-west-2
    The 2 parameters are the name of the CSV file containing your data (shown here as data.csv) and specify the target AWS region (shown here as us-west-2) as parameters. Adjust these parameters as needed to match your environment and data location.

Check status:

Once you have successfully ran the script, you can check the status of your registration requests via AWS End User Messaging Console or with the API using the DescribeRegistrations API. If you want to use the console log in to the AWS End User Messaging Console in the target region and choose Configurations > Registrations from the left navigation. The registration status for each request will initially display “CREATED” and it will change to “REVIEWING” within 24 hours. As noted in the introduction, the actual registration review process is conducted by a third-party company and it can take up to 15 days to receive an acceptance (“COMPLETE”) or request for additional information or corrections (“Requires Update“) for each request.

Registration Status:

Registrations

Phone Number Status:

Phone Number status

Conclusion:

In this post you have learned how to automate the registration process for Toll-Free numbers with a Bash script or Python script and the AWS End User Messaging APIs. Using the API can significantly improve efficiency and reduce manual errors, but keep in mind that you will still need to wait up to 15 business days for the registrations to be approved (or not), and for the numbers to be set to “ACTIVE” status before you can begin sending messages with your TFN(s).

The AWS End User Messaging V2 API for SMS and Voice has other useful actions, and we encourage you to explore how it can further help you simplify and automate your applications.

Resources to help you plan for your SMS program:

Use this spreadsheet to plan for the countries you need to send SMS/MMS

To restrict your sending to only certain countries read this blog

Confirm the origination IDs you will need here

Check out the support tiers comparison

Leverage IAM Roles for email sending via SES from EC2 and eliminate a common credential risk

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/leverage-iam-roles-for-email-sending-via-ses-from-ec2-and-eliminate-a-common-credential-risk/

Sending automated transactional emails, such as account verifications and password resets, is a common requirement for web applications hosted on Amazon EC2 instances. Amazon SES provides multiple interfaces for sending emails, including SMTP, API, and the SES console itself. The type of SES credential you use with Amazon SES depends on the method through which you are sending the emails.

In this blog post, we describe how to leverage IAM roles for EC2 instances to securely send emails via the Amazon SES API, without the need to embed IAM credentials directly in the application code, link to a shared credentials file, or manage IAM credentials within the EC2 instance. By adopting the approach outlined in this blog, you can enhance security by eliminating the risk of credential exposure and simplify credential management for your web applications.

Solution Overview

Below we provide step-by-step instructions to configure an IAM role with SES permissions to use on your EC2 instance. This allows the EC2 hosted web application to securely send emails via Amazon SES without storing or managing IAM credentials within the EC2 instance. We present an option for running EC2 and SES in the same AWS account, as well as an option to accommodate running EC2 and SES in different AWS accounts. Both options offer a way to enhance security and simplify credential management.

Either option begins with creating an IAM role with SES permissions. Next, the IAM role is attached to your EC2 instance, providing it with the necessary permissions for SES without needing to embed IAM credentials in your application code or on a file in the EC2 instance. In option 2, we’ll add cross-account permissions that allow the code on the EC2 instance in account “A” to send email via the SES API in account “B”. We also provide a sample Python script that demonstrates how to send an email from your EC2 instance using the attached IAM role.

Option 1 – SES and EC2 are in a single AWS account

In a typical scenario where an EC2 instance is operating in the same AWS account as SES, the process of using an IAM role to send emails via SES is straightforward. In the steps below, you’ll configure and attach an IAM role to the EC2 instance. You’ll then update a sample Python script to use the permissions provided by the attached IAM role to send emails via SES. This direct access simplifies the SES sending process, as no explicit credential management is required in the code, nor do you need to include a shared credentials file on the EC2 instance.

Option_1-Single_AWS_Account

EC2 & SES in the same AWS Account

Prerequisites – single AWS account for EC2 and SES

  • A single AWS account in a region that supports SES
  • Verified domain or email identity in Amazon SES.
    • Make note of a verified sending email address here: ___________
  • EC2 instance (Linux) in running state
    • If you don’t have a EC2 instance create one (Linux)
  • Administrative Access to Amazon SES, IAM and EC2 consoles.
  • Access to a recipient email address to receive test emails from the python script.
    • Make note of a SES verified recipient email address to send test emails here: ___________

Step 1 – Create IAM Role for EC2 instance with SES Permissions

To start, create an IAM role that grants the necessary permissions to send emails using Amazon SES by following these steps:

  • Sign in to the AWS Management Console and open the IAM console.
  • In the navigation pane, choose “Roles,” and then choose “Create role.”
  • Choose the trusted entity type as “AWS service” and select “EC2” as the service that will use this role, then click ‘Next
  • Search for and select the “AmazonSESFullAccess” policy from the list (or create a custom policy with the necessary SES permissions), then click ‘Next’.
  • Provide a name for your role (e.g., EC2_SES_SendEmail_Role).
  • Click “Create role“.

Step 2 – Attach the IAM Role to EC2 instance.

Next, attach the IAM role to your EC2 instance:

  • Open the EC2 Management Console.
  • In the navigation pane, choose “Instances,” and select the running EC2 instance to which you want to attach the IAM role.
  • With the instance selected, choose “Actions,” then “Security,” and “Modify IAM role.
  • Choose the IAM role you created (EC2_SES_SendEmail_Role) from the drop-down menu and click “Update IAM role.”

Step 3 – Create a sample python script that sends emails from the EC2 instance with the attached role.

  • Now that your EC2 instance is configured with the necessary permissions, you can set up an example Python script to send emails via Amazon SES using the IAM Role. Here, we’re using the AWS SDK for Python (Boto3), a powerful and versatile library to interact with the SES API endpoint. Before running the example script, ensure that Python, pip (the package installer for Python), and the Boto3 library are installed on your EC2 instance:
    • Run the ‘python3 –version‘ command to check if Python is installed on your EC2 instance. If Python is installed, the version will be displayed, otherwise you’ll receive a ‘command not found’ or similar error message.
      • If python is not installed, run the command ‘sudo yum install python3 -y
    • Run the ‘pip3 --version‘ command to check if pip is installed on your EC2 instance. If pip3 is installed, is installed, the version will be displayed, otherwise you’ll receive a ‘command not found’ or similar error message.
      • If pip3 is not installed, run the command ‘sudo yum install python3-pip
    • Install the Boto3 Library which allows Python scripts to interact with AWS services including SES. Run the command ‘pip3 install boto3‘ to install (or update) Boto3 using pip.
  • Save the code below as a Python file named ‘sesemail.py‘ on your EC2 instance.
  • Edit 'sesemail.py‘ and replace the placeholder values of SENDER, RECIPIENT, and AWS_REGION with your values (see prerequisites). Do not modify any “” marks.

[copy]

import boto3
from botocore.exceptions import ClientError

SENDER = "[email protected]"
RECIPIENT = "[email protected]"
#CONFIGURATION_SET = "ConfigSet"
AWS_REGION = "us-west-2"
SUBJECT = "Amazon SES Test Email (SDK for Python) using IAM Role"
BODY_TEXT = ("Amazon SES Test (Python)\r\n"
             "This email was sent with Amazon SES using the "
             "AWS SDK for Python (Boto)."
            )
            
BODY_HTML = """<html>
<head></head>
<body>
  <h1>Amazon SES Test (SDK for Python) using IAM Role</h1>
  <p>This email was sent with
    <a href='https://aws.amazon.com/ses/'>Amazon SES</a> using the
    <a href='https://aws.amazon.com/sdk-for-python/'>
      AWS SDK for Python (Boto)</a>.</p>
</body>
</html>
            """            

CHARSET = "UTF-8"

client = boto3.client('ses',region_name=AWS_REGION)

try:
    response = client.send_email(
        Destination={
            'ToAddresses': [
                RECIPIENT,
            ],
        },
        Message={
            'Body': {
                'Html': {
                    'Charset': CHARSET,
                    'Data': BODY_HTML,
                },
                'Text': {
                    'Charset': CHARSET,
                    'Data': BODY_TEXT,
                },
            },
            'Subject': {
                'Charset': CHARSET,
                'Data': SUBJECT,
            },
        },
        Source=SENDER,
    )   
except ClientError as e:
    print(e.response['Error']['Message'])
else:
    print("Email sent! Message ID:"),
    print(response['MessageId'])
  • Run ‘python3 sesmail.py‘ to execute the Python script.
  • When ‘python3 sesmail.py‘ runs successfully, an email is sent to the RECIPIENT(check the inbox), and the command line will display the sent Message ID.


Option 2 – SES and EC2 are in different AWS accounts

In some scenarios, your EC2 instance might operate in a different AWS account than SES. Let’s call the EC2 AWS account “A” and SES AWS account “B”. Because the AWS resources in account A don’t automatically have permission to access AWS resources account B, we need some way to allow the code on EC2 to assume a role in the SES Account using the AWS Security Token Service (STS). This involves a method that generates temporary credentials that include an access key, secret access key, and session token, which are only valid for a limited time.

option-2

EC2 & SES in different AWS Accounts

In the steps below, you’ll configure and attach an IAM role to the EC2 instance in account “A” such that it can run an example Python script. This Python script can use the permissions provided by the attached IAM role to send emails via SES in account “B”. This approach leverages cross-account access and simplifies sending email from the EC2 in account A via SES in account B. As with Option 1, no explicit credential management is required in the code running on EC2, nor do you need to include a shared credentials file on the Ec2 instance.

Prerequisites – different AWS accounts for EC2 and SES (use cross-account access)

  • An AWS account “A” with:
    • EC2 instance (Linux) in running state. (If you don’t have a EC2 instance, create one using Amazon Linux)
    • Administrative Access to Amazon IAM and EC2 consoles.
    • Make note of your “A” AWS account ID here: ________________
  • An AWS account “B” with:
    • Verified domain (or email identity for testing only) in Amazon SES
      • Make note of a verified sending email address here: ___________
    • Administrative Access to Amazon SES and IAM consoles.
      • Make note of your “B” AWS account ID here: ________________
    • In the steps below, you will create a “SES_Role_for_account_A” role.
      • Make note of the ARN of the “SES_Role_for_account_A” role here: ___________
    • Access to a recipient email address to receive test emails from the python script.
      • Make note of a SES verified recipient email address to send test emails here: ___________

Step 1 – Create IAM Role in the SES “B” account

  • Sign in to the SES “B” account via the AWS Management Console and open the IAM console.
  • In the navigation pane, choose “Roles,” and then choose “Create role“.
  • Choose the trusted entity type as ‘AWS account’ and select ‘Another AWS account’.
  • Add the AWS account ID where your EC2 instance resides (AWS account “A” in the prerequisites) and click ‘Next’.
  • Search for and select the “AmazonSESFullAccess” policy or create a custom policy with the necessary SES permissions, then click ‘Next’.
  • Provide a name for your role (e.g., ‘SES_Role_for_account_A').
  • Click “Create role“.
  • Copy the arn for the new SES_Role_for_account_A (you’ll need the arn in the next step).

Step 2 – Create a IAM policy in the EC2 “A” account that allows this role to assume the SES_Role_for_account_A role you just created in the SES “B” Account.

  • Sign in to the EC2 “A” account via the AWS Management Console and open the IAM console.
  • In the navigation pane, choose “Policies,” and then choose “Create Policy”.
  • Choose the service as ‘EC2’ and select policy editor as JSON.
  • Copy the policy below, and in the policy editor, replace the Resource with the arn of theSES_Role_for_account_A in the SES account “B” (you created this in step 1).

[copy, paste into policy editor & replace the arn with SES_Role_for_account_A]

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::<SES_Account_ID>:role/<Role_Name>"
}
]
}

  • Click ‘Next’ and provide a name for your role (e.g., EC2_Policy_for_account_B).
  • Click ‘Create the Policy

Step 3 – Create an IAM role in the EC2 “A” account, and attach the previously created IAM policy (EC2_Policy_for_account_B) to it.

  • In the EC2 “A” account IAM console navigation pane, choose “Roles,” and then choose “Create role.”
  • Choose the trusted entity type as “AWS service” and select “EC2” as the service, then click ‘Next’.

  • Filter by type “customer managed”, search for (EC2_Policy_for_account_B) and select that policy and ‘Next’ (note – if you are using AWS Session Manger to remotely connect to your EC2 instance, you may need to add the “AmazonSSMManagedInstanceCore” policy to the role).

  • Provide a name for your role (e.g., EC2_SES_in_account_B_role).
  • Click “Create role“.

Step 4 – Attach the IAM Role (EC2_SES_in_account_B_role) to the EC2 instance in AWS account “A”.

  • Open the EC2 Management Console in AWS account “A”
  •  In the navigation pane, choose “Instances,” and select the instance to which you want to attach the EC2_SES_in_account_B_role IAM role.
  • With the instance selected, choose “Actions,” then “Security,” and “Modify IAM role.”

  • Choose the IAM role you created (EC2_SES_in_account_B_role) from the drop-down menu.
  • Click “Update IAM role.”

Step 5 – Create a sample python script that sends emails via SES in AWS account “B” from the EC2 instance in AWS account “A” using the EC2 attached role.

  1. Now that your EC2 instance is configured with the necessary permissions, you can set up an example Python script to send emails via Amazon SES in AWS Account “B” using the IAM Role on EC2 in AWS Account “A”. We’ll use the AWS SDK for Python (Boto3), a powerful and versatile library to interact with the SES API endpoint. Before running the example script, ensure that Python, pip (the package installer for Python), and the Boto3 library are installed on your EC2 instance:
    • Run the ‘python3 –version‘ command to check if Python is installed on your EC2 instance. If Python is installed, the version will be displayed, otherwise you’ll receive a ‘command not found’ or similar error message.
      • If python is not installed, run the command ‘sudo yum install python3 -y
    • Run the ‘pip3 --version‘ command to check if pip is installed on your EC2 instance. If pip3 is installed, is installed, the version will be displayed, otherwise you’ll receive a ‘command not found’ or similar error message.
      • If pip3 is not installed, run the command ‘sudo yum install python3-pip
    • Install the Boto3 Library which allows Python scripts to interact with AWS services including SES. Run the command ‘pip3 install boto3‘ to install (or update) Boto3 using pip.
  1. Save the code below as a Python file named cross_sesemail.py on your EC2 instance.
    4b. Edit cross_sesemail.py and replace the placeholder values of the ROLE_ARN with ARN of the SES_Role_for_account_A you created in SES Account “B” (see prerequisites), SENDER, RECIPIENT, and AWS_REGION with your values (see prerequisites). Do not modify any “” marks.

[copy, edit & replace the ROLE_ARN]

import boto3
from botocore.exceptions import ClientError

# Replace with your role ARN in SES Account
ROLE_ARN = "arn:aws:iam::<Account_ID>:role/<Role_Name>"

# Create an STS client
sts_client = boto3.client('sts')

# Assume the role
assumed_role = sts_client.assume_role(
    RoleArn=ROLE_ARN,
    RoleSessionName="SESSession"
)

# Extract the temporary credentials
credentials = assumed_role['Credentials']

# Create an SES client using the assumed role credentials
ses_client = boto3.client(
    'ses',
    region_name='us-west-2',
    aws_access_key_id=credentials['AccessKeyId'],
    aws_secret_access_key=credentials['SecretAccessKey'],
    aws_session_token=credentials['SessionToken']
)

# Email parameters
SENDER = "[email protected]"
RECIPIENT = "[email protected]"
SUBJECT = "Amazon SES Test (SDK for Python) using cross-account IAM Role"
BODY_TEXT = ("Amazon SES Test (Python)\r\n"
             "This email was sent with Amazon SES using the "
             "AWS SDK for Python (Boto) using IAM Role."
            )
BODY_HTML = """<html>
<head></head>
<body>
  <h1>Amazon SES Test (SDK for Python) using IAM Role</h1>
  <p>This email was sent with
    <a href='https://aws.amazon.com/ses/'>Amazon SES</a> using the
    <a href='https://aws.amazon.com/sdk-for-python/'>
      AWS SDK for Python (Boto)</a> using IAM Role.</p>
</body>
</html>
            """
CHARSET = "UTF-8"

# Send the email
try:
    response = ses_client.send_email(
        Destination={
            'ToAddresses': [RECIPIENT],
        },
        Message={
            'Body': {
                'Html': {
                    'Charset': CHARSET,
                    'Data': BODY_HTML,
                },
                'Text': {
                    'Charset': CHARSET,
                    'Data': BODY_TEXT,
                },
            },
            'Subject': {
                'Charset': CHARSET,
                'Data': SUBJECT,
            },
        },
        Source=SENDER,
    )
except ClientError as e:
    print(e.response['Error']['Message'])
else:
    print("Email sent! Message ID:"),
    print(response['MessageId'])
  • Run the python script python3 cross_sesemail.py. When the email is sent successfully, the command line output will display the message ID of the sent email, and the recipient will receive an email.


Conclusion:

By implementing IAM roles for EC2 instances with SES permissions, you can securely send emails via the SES APIs from your web applications without the need to store or manage IAM credentials within the EC2 instance or application code. This approach not only enhances security by eliminating the risk of credential exposure, but also simplifies the management of credentials. With the step-by-step guide provided in this blog post, you can easily configure IAM roles for your EC2 instances and start sending emails via the Amazon SES API in a secure and efficient manner, regardless of whether your EC2 and SES resources reside in the same or different AWS accounts.

Next Steps:

  1. Sign up for the AWS Free Tier and try out Amazon SES with IAM roles for EC2 instances as demonstrated in this blog post.
  2. Consult the AWS documentation on IAM Roles for Amazon EC2 and Amazon SES for more detailed instructions and best practices.
  3. Join the AWS Community Forums to ask questions, share experiences, and learn from other AWS users who have implemented similar solutions for secure email sending from their web applications.

About the Authors

Manas Murali M

Manas Murali M

Manas Murali M is a Cloud Support Engineer II at AWS and subject matter expert in Amazon Simple Email Service (SES) and Amazon CloudFront. With over 5 years of experience in the IT industry, he is passionate about resolving technical issues for customers. In his free time, he enjoys spending time with friends, traveling, and exploring emerging technologies.

zip

Zip

Zip is an Amazon Pinpoint and Amazon Simple Email Service Sr. Specialist Solutions Architect at AWS. Outside of work he enjoys time with his family, cooking, mountain biking and plogging.

Streamline SMS and Emailing Marketing Compliance with Amazon Comprehend

Post Syndicated from Koushik Mani original https://aws.amazon.com/blogs/messaging-and-targeting/streamline-sms-and-emailing-marketing-compliance-with-amazon-comprehend/

In today’s digital landscape, businesses heavily rely on SMS and email campaigns to engage with customers and deliver timely, relevant messages. The shift towards digital marketing has increased customer engagement, accelerated delivery, and expanded personalization options. Email and SMS marketing is essential to digital strategies according to 44% of Chief Marketing Officers and they allocate approximately 8% of their marketing towards this. Industries face stringent restrictions on the content they can send due to legal regulations and carrier filtering policies.

Messages related to the subjects listed below are considered restricted and are subject to heavy filtering or even being blocked outright. Failing to comply with these restrictions can result in severe consequences, including legal action, fines, and irreparable damage to a brand’s reputation. Marketers need a solution that will proactively scan their content used in campaigns and flag restricted content before sending it out to their customers without facing penalties and losing trust.:

  • Gambling
  • High-risk financial services
  • Debt forgiveness
  • S.H.A.F.T (Sex, Hate, Alcohol, Firearms, and Tobacco)
  • Illegal substances

In this blog, we will explore how to leverage Amazon Comprehend, Amazon S3, and AWS Lambda to proactively scan text-based marketing campaigns before publishing content . This solution enables businesses to enhance their marketing efforts while maintaining compliance with industry regulations, avoiding costly fines, and preserving their hard-earned reputation, conforming to best practices.

Solution Overview

AWS provides a robust suite of services to meet the infrastructure needs of the booming digital marketing industry, including messaging capabilities through email, SMS, push, and other channels through Amazon Simple Email Service, Amazon Simple Notification Service, or Amazon Pinpoint.

The main goal for this approach is to flag any message that contains restricted content mentioned above before distribution.

Figure 1: Architecture for proactive scanning of marketing content

Figure 1: Architecture for proactive scanning of marketing content

Following are the high-level steps:

  1. Upload documents to be scanned to the S3 bucket.
  2. Utilize Amazon Comprehend custom classification for categorizing the documents uploaded.
  3. Create an Amazon Comprehend endpoint to perform analysis.
  4. Inference output is published to the destination S3 bucket.
  5. Utilize AWS Lambda function to consume the output from the destination S3 bucket.
  6. Send the compliant messages through various messaging channels.

Solution Walkthrough

Step 1: Upload Documents to Be Scanned to S3

  1. Sign in to the AWS Management Console and open the Amazon S3 console
  2. In the navigation bar on the top of the page, choose the name of the currently displayed AWS Region. Next, choose the Region in which you want to create a bucket.
  3. In the left navigation pane, choose Buckets.
  4. Choose Create bucket.
    • The Create bucket page opens.
  5. Under General configuration, view the AWS Region where your bucket will be created.
  6. Under Bucket type, choose General purpose.
  7. For Bucket name, enter a name for your bucket.
    • The bucket name must:
      • Be unique within a partition. A partition is a grouping of Regions. AWS currently has three partitions: aws (Standard Regions), aws-cn (China Regions), and aws-us-gov (AWS GovCloud (US) Regions).
      • Be between 3 and 63 characters long.
      • Consist only of lowercase letters, numbers, dots (.), and hyphens (-). For best compatibility, we recommend that you avoid using dots (.) in bucket names, except for buckets that are used only for static website hosting.
      • Begin and end with a letter or number.
  8. In the Buckets list, choose the name of the bucket that you want to upload your folders or files to.
  9. Choose Upload.
  10. In the Upload window, do one of the following:
    • Drag and drop files and folders to the Upload window.
    • Choose Add file or Add folder, choose the files or folders to upload, and choose Open.
    • To enable versioning, under Destination, choose Enable Bucket Versioning.
    • To upload the listed files and folders without configuring additional upload options, at the bottom of the page, choose Upload.
  11. Amazon S3 uploads your objects and folders. When the upload is finished, you see a success message on the Upload: status page.

Step 2: Creating a Custom Classifiction Model

Custom Classification Model

Out-of-the-box models may not capture nuances and terminology specific to an organization’s industry or use case. Therefore, we train a custom model to identify compliant messages.

A custom classification model is a feature that allows you to train a machine learning model to classify text data based on categories that are specific to your use case or industry. It trains the model to recognize and sort different types of content which is used to power the endpoint. A custom classification model is designed to save costs and promote compliant messages and further prevent marketing companies from potential fines.

Requirements for custom classification:

  • Dataset creation
    • A CSV dataset with 1000 examples of marketing messages, each labeled as compliant (1) or non-compliant (0).
    • Designed to train a model for accurate predictions on marketing message compliance.
Figure 2: Screenshot of dataset – 20 entries of censored marketing messages

Figure 2: Screenshot of dataset – 20 entries of censored marketing messages

  • Creating a Test Data Set
    In addition to providing a dataset to power your customer classification model, a test dataset is also required to test the data that the model will be running on. Without a test dataset, Amazon Comprehend trains the model with 90 percent of the training data. It reserves 10 percent of the training data to use for testing. When using a test dataset, the test data must include at least one example for each unique label (0 or 1) in the training dataset.
  1. Upload the data set and test data set to an S3 Bucket, by following the steps in this user guide.
  2. In the AWS Console, search for Amazon Comprehend.
  3. Once selected, select custom classification on the left panel.
  4. Once there, select Create new model.
  5. Next specify model settings:
    • Model name
    • Specify the version (optional)
    • Language: EnglishModel Setting
  6. Specify the data specifications:
    • Training model type: Plain Text Documents
    • Data format: CSV File
    • Classifier Mode: Using Single-Label Mode
    • Training Dataset: Give the name of the bucket you created in step 1
    • Test Data set: Autosplit, i.e. how much of your data will be used for training and testing.
    • Data Specifications
  1. Specify the location of the model output in S3
    • Output data
  1. Create an IAM Role
    • Permissions to access: Train, Test and output data (if specified in your S3 Buckets)
    • IAM Role
  2. Once all parameters have been identified, select Create.
    • Preferences
  3. Once the model has been created, you can view it under Custom Classification. To check and verify the accuracy and F1 score, select the version number of the model. Here, you can view the model details under the Performance tab.

Step 3: Creating an Endpoint in Amazon Comprehend

Next, an endpoint needs to be created to analyze documents. To create an endpoint:

  • Select endpoint on the left panel in Amazon Comprehend.
  • Select Create endpoint in the left panel.
  • Specify Endpoints Settings :
    • Provide a name
    • Custom model type: Custom Classification
    • Choose a custom model that you want to attach to the new endpoint. From the dropdown, you can search by model name.
    • create_endpointFigure 8: Amazon Comprehend – Endpoint settings
  • Provide the number of inference units (IUs): 1
  • Once all the parameters have been provided, ensure that the Acknowledge checkbox has been selected.
  • Finally, select Create endpoint.
  • inference_units

Step 4: Scanning Text with the Custom Classification Endpoint

Once the endpoint has been successfully created, it can be used for real-time analysis or batch-processing jobs. Below is a walkthrough of how both options can be achieved.

Real-time analysis:

  • On the left panel, select Realtime Analysis.
  • Pick Analysis type: custom, to view real-time insights based on the custom models from an endpoint you’ve created
  • Select custom model type
  • Select your Endpoint
  • Insert your input text.
    • For this example, we have used a non-compliant message: Huge sale at Pine County Shooting Range 25% off for 6mm and 9mm bullets! Lazer add-ons on clearance too
  • Once inserted, click Analyze.input_data
  • Once analyzed, you will see a confidence score under Classes. Because the dataset is labeled as 0 for non-compliant and 1 for compliant. The message that was inserted was non-compliant, the result of the real-time analysis is a high confidence score for non-compliant.insights

Real-time analysis in Amazon Comprehend:

  • On the left panel in Amazon Comprend, select Analysis Jobs.
  • Select the Create Job button.
  • Configure Job settings:
    • Enter the Name
    • Analysis Type: Custom Classification
    • Classifications Model: The model you have created for your Classifier, as well as the version number of that model you would like to use for this job.
    • create_analysis
  • Enter the location of the Input Data and Output Data in the form of an S3 bucket URL.
  • input_data
  • Before creating a job the last thing, we want to do is provide the right access permission, by creating an IAM role that give access permissions to the S3 input and output locations.
  • access_premissions
  • Once the batch processing job shows a status of completed, you can view the results in the output S3 bucket which was identified earlier. The results will be in a json file where each line represents the confidence score for each marketing message.
  • json

Step 5 (optional): Publish message to communication service

The result from the batch processing is automatically uploaded to the output S3 bucket. For each json file uploaded, S3 will initiate an S3 Event Notification which will inform a Lambda function that a new S3 object has been created.

The Lambda function will evaluate the results and automatically identify the messages labeled as compliant (label 0). These compliant messages will then be published to communication services using one of the following three APIs, depending on the desired service:

To automatically trigger the AWS Lambda function, which will read the files uploaded into the S3 bucket and display the data using the Python Pandas library, we will use the boto3 API to read the files from the S3 bucket.

  1. Create an IAM Role in AWS.
  2. Create an AWS S3 bucket.
  3. Create the AWS Lambda function with S3 triggers enabled.
  4. Update the Lambda code with a Python script to read the data and send the communication to customer.

Conclusion

Proactively scanning and classifying marketing content for compliance is a critical aspect of ensuring successful digital marketing campaigns while adhering to industry regulations. Leveraging the powerful combination of Amazon Comprehend, Amazon S3, and AWS Lambda enables the automatic analysis of text-based marketing messages and flagging of any non-compliant content before sending them to your customer. Following these steps provides you with the tools and knowledge to implement proactive scanning for your marketing content. This solution will help mitigate the risks of non-compliance, avoiding costly fines and reputational damage, while freeing up time for your content creation teams to focus on ideation and crafting compelling marketing messages. Regular monitoring and fine-tuning of the custom classification model should be conducted to ensure accurate identification of non-compliant language.

To get started with proactively scanning and classifying marketing content for compliance, see Amazon Comprehend Custom Classification.

About the Authors

Caroline Des Rochers

Caroline Des Rochers

Caroline is a Solutions Architect at Amazon Web Services, based in Montreal, Canada. She works closely with customers, in both English and French, to accelerate innovation and advise them through technical challenges. In her spare time, she is a passionate skier and cyclist, always ready for new outdoor adventures.

Erika_Houde_Pearce

Erika Houde Pearce

Erika Houde-Pearce is a bilingual Solutions Architect in Montreal, guiding small and medium businesses in eastern Canada through their cloud transformations. Her expertise empowers organizations to unlock the full potential of cloud technology, accelerating their growth and agility. Away from work, she spends her spare time with her golden retriever, Scotia.

Koushik_Mani

Koushik Mani

Koushik Mani is a Solutions Architect at AWS. He had worked as a Software Engineer for two years focusing on machine learning and cloud computing use cases at Telstra. He completed his masters in computer science from University of Southern California. He is passionate about machine learning and generative AI use cases and building solutions.

Email Journaling with SES Mail Manager

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/email-journaling-with-ses-mail-manager/

Introduction to Journaling

Email journaling is the practice of preserving comprehensive records of all email communications within an organization. This approach stems from the need to maintain rigid, compliance-driven retention policies focused on auditing an entire organization’s email activities. Because journaled email messages are often required to satisfy on-demand audit and investigation requests, they must be readily searchable, making accessibility a key requirement. Reflecting legal and regulatory requirements, email journaling has historically required expensive, dedicated off-site storage and complex retrieval systems.

Amazon WorkMail is a managed business email service with flexible journaling capabilities that are configurable at both the individual mailbox and organization-wide level. With WorkMail, you can use custom rules to selectively preserve or redirect certain messages using granular journaling controls. This flexibility allows administrators to implement both traditional email journaling and configurations that you can customize to meet specific use cases.

Email journaling is used to capture and retain every email sent to and from an organization, primarily for compliance purposes. In contrast, email archiving is typically used to offload and store emails from an organization’s primary email system, often driven by inbox size limits and data backup or eDiscovery needs. While journaling focuses on preserving a consolidated record of communications separate from live mailboxes, archiving is a more selective process. Journaling is usually driven by regulatory, audit, and compliance requirements. As discussed in this blog post, you can use the Mail Manager archiving feature not only for selective email backup and optimization, but also to fulfill your email journaling requirements. You can learn more about email archiving with Mail Manager in this blog post.

Amazon Simple Email Service (SES) Mail Manager provides comprehensive tools that simplify managing large volumes of email communications within an organization. Mail Manager has a built-in archiving function which can be used as an inexpensive journaling solution for email systems like Amazon WorkMail. Mail Manager’s rules engine allows for the creation of rules that readily satisfy a wide range of email journaling requirements. Additionally, Mail Manager’s archiving capability supports multiple, concurrent archiving destinations that can be independently searched and exported on demand.

In this blog post, we discuss how Amazon WorkMail and Amazon Simple Email Service (SES) Mail Manager make email journaling easier to set up and use, more cost-effective and versatile. We’ll walk the reader through setting up email journaling for an Amazon WorkMail organization that uses SES Mail Manager’s routing, processing, and archiving features.

SES Mail Manager as Journaling Destination for WorkMail

For our purposes, we’ll assume you’ve already set up WorkMail as your mailbox provider, but the process described below will work with the journaling features of most 3rd party email solutions. If you want to explore Amazon WorkMail, visit the getting started documentation here.

In the following sections, we’ll describe how to configure WorkMail journaling to send full email journals to SES Mail Manager’s archives. We’ll define different retention periods for each archive to demonstrate how this solution can be used to meet both short and long-term retention requirements. Finally, we’ll use the AWS SES Mail Manager console to search, export, and manage the email journals and archives.

In our examples, we’ll use Amazon Route 53 to create a new domain called ‘journaling.solutions’ which we’ll configure to send all ‘@journaling.solutions’ emails to an SES Mail Manager Ingest endpoint. To begin, open the AWS Console, navigate to your WorkMail Organization’s settings, and click on the Journaling tab:

Organization settings Journaling tab

Organization settings Journaling tab

Click Edit, enable journaling, and provide a journaling email address (we’re using ‘[email protected]’) to receive journaled content. Provide a report email address, such as the admin email list, to receive journaling reports:

Provide a Journaling email address

Provide a Journaling email address

Open the AWS SES console in a new browser window, and navigate to Mail Manager’s Rule sets. Create a new rule set called ‘journaling-rule-demo’. Click Edit and create a new rule called “journal-all”, with an Archive action. Click the create an archive button and create an archive called ‘journaling-archive-demo’:

Create a new Rule Set called ‘Journaling-rule-demo’

Create a new Rule Set called ‘Journaling-rule-demo’

When creating Mail Manager archives, you have options to set the retention period from 3 months to permanent storage. You can also choose to encrypt your archived messages with your own KMS key. The configuration in our example is for permanent storage and shows the optional text field for using your own KMS key:

you can encrypt the archived messages with your own KMS key

you can encrypt the archived messages with your own KMS key

Traditional journaling calls for recording every email message to the journal, so for our ‘journal-all’ rule, we will not define filtering behaviors in the rule set. This will instruct Mail manager to send all emails for [email protected] to the journaling-archive-demo archive. It is worth noting that Mail Manager’s rule set can be configured to filter and independently process multiple recipient addresses. Consult the documentation to learn about other ways to customize Mail Manager for your use cases.

Next, create a new traffic policy, called journaling-traffic-demo, and configure it to reject any message not explicitly sent to the journaling destination address ([email protected]):

Create a new Traffic policy, called ‘Journaling-traffic-demo’

Create a new Traffic policy, called ‘Journaling-traffic-demo’

Create an open ingress endpoint called ‘journaling-demo-IG’, and select the ‘journaling-traffic-demo’ traffic policy and ‘journaling-rule-demo’ rule set:

Create an Open Ingress endpoint called ‘Journaling-demo-IG’,

Create an Open Ingress endpoint called ‘Journaling-demo-IG’,

After you press the create Ingest endpoint button, Mail Manager will create an Ingress endpoint and assign it a DNS A Record to be used in your DNS configurations to route email to Mail Manager:

Mail Manager Ingress endpoint DNS A Record to be used in your DNS configurations

Mail Manager Ingress endpoint DNS A Record to be used in your DNS configurations

From the General details page of the Ingress endpoint, copy the Ingress endpoint’s DNS A Record to your clipboard. Open a new browser window to your DNS provider’s MX configuration page (in our example below, we’re using AWS Route53). Edit the MX record for ‘journaling.solutions’ by pasting the Ingress endpoint A record. This configuration will route email sent to any address ‘@journaling.solutions’ to the Mail Manager’s Ingress endpoint for processing by the Traffic policy and Rule set:

Using AWS Route53 to edit MX record for ‘journaling.solutions’ to Ingress endpoint A record

Using AWS Route53 to edit MX record for ‘journaling.solutions’ to Ingress endpoint A record

To test your new journaling configuration, send several emails to several email addresses in your WorkMail organization (or the alternative inbox provider you configured in the first step). WorkMail (or your alternative inbox provider) will send a full record of all emails to the journaling destination address ([email protected]).

Wait a few minutes after sending the emails above, then open the AWS Mail Manager console’s archiving controls and search for messages sent in the last 12 hours:

AWS Mail Manager console’s archiving controls

AWS Mail Manager console’s archiving controls

The example above shows a search for all messages received in the “last 12 hours”, with no other filters specified. The results show every message inserted into the archive in this timeframe. You’ll see one entry where the from address is different (from toby@tegwj@…). This is an example of mail that was sent directly to the journaling destination address ([email protected]). This works because our traffic policy and rule set configurations don’t include any filters.

A cost effective solution at scale

Using Mail Manager as a journaling solution gives you more direct control over your costs than typical journaling services. While most journaling services in the market today charge a fixed rate per journaled mailbox, Mail Manager pricing is comprised of a monthly fixed fee per ingestion endpoint and consumption pricing for basic message handling, and the amount of data archived.

For example, imagine your organization has 250 mailboxes, each handling 50 messages per day. On a monthly basis this amounts to 375,000 messages. If we assume each message is 40 kilobytes in size, your organization is generating roughly 15 gigabytes of email per month. As you can see from the table below, the total cost in month 1 is about $140, or $0.56/mailbox.

|Item |Unit Price |Volume |Subtotal/Mo |
|— |— |— |— |
|Ingress Endpoint |$50/mo |1 |$50 |
|Core message processing |$0.15/1000 msgs |375 |$56.25 |
|Archive insertion/indexing |$2/GB (one-time) |15 |$30 |
|Archive storage |$0.19/GB/mo |15 |$2.85 |
|Subtotal: | | |$139.10 |
| |Monthly price per mailbox |$0.56 |

If the proposed email rate in our assumptions stays constant, the Mail Manager archive will grow by 15 gigabytes each month. After 36 months, the total monthly storage cost increases to $102.60. This results in a total monthly spend in month 36 of $238.85, or $0.96/mailbox/month.

Conclusion

In this blog post, we’ve explored how Amazon WorkMail and Amazon SES Mail Manager can provide a cost-effective and accessible solution for email journaling. By leveraging the flexible journaling capabilities of WorkMail and the archiving features of SES Mail Manager, organizations can easily satisfy rigorous compliance requirements around email retention and accessibility.

The combination of WorkMail’s journaling controls and SES Mail Manager’s rule-based archiving allows you to tailor your journaling solution to your specific needs. Whether you require short-term retention for audits or long-term preservation for legal and regulatory purposes, SES Mail Manager’s flexible archiving options have you covered with predictable and transparent costs that scale with your organization’s email volume.

If you’re looking for a modern, scalable, and cost-effective solution for your email journaling needs, we encourage you to explore the capabilities of Amazon SES Mail Manager. Get started today by visiting the AWS documentation and begin streamlining your email compliance and retention processes.

About the Authors

Toby Weir-Jones

Toby Weir-Jones

Toby is a Principal Product Manager for Amazon SES and WorkMail. He joined AWS in January 2021 and has significant experience in both business and consumer information security products and services. His focus on email solutions at SES is all about tackling a product that everyone uses and finding ways to bring innovation and improved performance to one of the most ubiquitous IT tools.

Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

Andy Wong

Andy Wong

Andy Wong is a Sr. Product Manager with the Amazon WorkMail team. He has 10 years of diverse experience in supporting enterprise customers and scaling start-up companies across different industries. Andy’s favorite activities outside of technology are soccer, tennis and free-diving.

Bruno Giorgini

Bruno Giorgini

Bruno Giorgini is a Senior Solutions Architect specializing in Pinpoint and SES. With over two decades of experience in the IT industry, Bruno has been dedicated to assisting customers of all sizes in achieving their objectives. When he is not crafting innovative solutions for clients, Bruno enjoys spending quality time with his wife and son, exploring the scenic hiking trails around the SF Bay Area.

How we improved translation experience with cost efficiency

Post Syndicated from Grab Tech original https://engineering.grab.com/improved-translation-experience-with-cost-efficiency

Introduction

As COVID restrictions were fully lifted in 2023, the number of tourists grew dramatically. People began to explore the world again, frequently using the Grab app to make bookings outside of their home country. However, we noticed that communication posed a challenge for some users. Despite our efforts to integrate an auto-translation feature in the booking chat, we received feedback about occasional missed or inaccurate translations. You can refer to this blog for a better understanding of Grab’s chat system.

An example of a bad translation. The correct translation is: ‘ok sir’.

In an effort to enhance the user experience for travellers using the Grab app, we formed an engineering squad to tackle this problem. The objectives are as follows:

  • Ensure translation is provided when it’s needed.
  • Improve the quality of translation.
  • Maintain the cost of this service within a reasonable range.

Ensure translation is provided when it’s needed

Originally, we relied on users’ device language settings to determine if translation is needed. For example, if both the passenger and driver’s language setting is set to English, translation is not needed. Interestingly, it turned out that the device language setting did not reliably indicate the language in which a user would send their messages. There were numerous cases where despite having their device language set to English, drivers sent messages in another language.

Therefore, we needed to detect the language of user messages on the fly to make sure we trigger translation when it’s needed.

Language detection

Simple as it may seem, language detection is not that straightforward a task. We were unable to find an open-source language detector library that covered all Southeast Asian languages. We looked for Golang libraries as our service was written in Golang. The closest we could find were the following:

  • Whatlang: unable to detect Malay
  • Lingua: unable to detect Burmese and Khmer

We decided to choose Lingua over Whatlang as the base detector due to the following factors:

  • Overall higher accuracy.
  • Capability to provide detection confidence level.
  • We have more users using Malay than those using Burmese or Khmer.

When a translation request comes in, our first step is to use Lingua for language detection. If the detection confidence level falls below a predefined threshold, we fall back to call the third-party translation service as it can detect all Southeast Asian languages.

You may ask, why don’t we simply use the third-party service in the first place. It’s because:

  • The third-party service only has a translate API that also does language detection, but it does not provide a standalone language detection API.
  • Using the translate API is costly, so we need to avoid calling it when it’s unnecessary. We will cover more on this in a later section.

Another challenge we’ve encountered is the difficulty of distinguishing between Malay and Indonesian languages due to their strong similarities and shared vocabulary. The identical text might convey different meanings in these two languages, which the third-party translation service struggles to accurately detect and translate.

Differentiating Malay and Indonesian is a tough problem in general. However, in our case, the detection has a very specific context, and we can make use of the context to enhance our detection accuracy.

Making use of translation context

All our translations are for the messages sent in the context of a booking or order, predominantly between passenger and driver. There are two simple facts that can aid in our language detection:

  • Booking/order happens in one single country.
  • Drivers are almost always local to that country.

So, for a booking that happens in an Indonesian city, if the driver’s message is detected as Malay, it’s highly likely that the message is actually in Bahasa Indonesia.

Improve quality of translation

Initially, we were entirely dependent on a third-party service for translating our chat messages. While overall powerful, the third-party service is not perfect, and it does generate weird translations from time to time.

An example of a weird translation from a third-party service recorded on 19 Dec 2023.

Then, it came to us that we might be able to build an in-house translation model that could translate chat messages better than the third-party service. The reasons being:

  • The scope of our chat content is highly specific. All the chats are related to bookings or orders. There would not be conversations about life or work in the chat. Maybe a small Machine Learning (ML) model would suffice to do the job.
  • The third-party service is a general translation service. It doesn’t know the context of our messages. We, however, know the whole context. Having the right context gives us a great edge on generating the right translation.

Training steps

To create our own translation model, we took the following steps:

  • Perform topic modelling on Grab chat conversations.
  • Worked with the localisation team to create a benchmark set of translations.
  • Measured existing translation solutions against benchmarks.
  • Used an open source Large Language Model (LLM) to produce synthetic training data.
  • Used synthetic data to train our lightweight translation model.

Topic modelling

In this step, our aim was to generate a dataset which is both representative of the chat messages sent by our users and diverse enough to capture all of the nuances of the conversations. To achieve this, we took a stratified sampling approach. This involved a random sample of past chat conversation messages stratified by various topics to ensure a comprehensive and balanced representation.

Developing a benchmark

For this step we engaged Grab’s localisation team to create a benchmark for translations. The intention behind this step wasn’t to create enough translation examples to fully train or even finetune a model, but rather, it was to act as a benchmark for translation quality, and also as a set of few-shot learning examples for when we generate our synthetic data.

This second point was critical! Although LLMs can generate good quality translations, LLMs are highly susceptible to their training examples. Thus, by using a set of handcrafted translation examples, we hoped to produce a set of examples that would teach the model the exact style, level of formality, and correct tone for the context in which we plan to deploy the final model.

Benchmarking

From a theoretical perspective there are two ways that one can measure the performance of a machine translation system. The first is through the computation of some sort of translation quality score such as a BLEU or CHRF++ score. The second method is via subjective evaluation. For example, you could give each translation a score from 1 to 5 or pit two translations against each other and ask someone to assess which they prefer.

Both methods have their relative strengths and weaknesses. The advantage of a subjective method is that it corresponds better with what we want, a high quality translation experience for our users. The disadvantage of this method is that it is quite laborious. The opposite is true for the computed translation quality scores, that is to say that they correspond less well to a human’s subjective experience of our translation quality, but that they are easier and faster to compute.

To overcome the inherent limitations of each method, we decided to do the following:

  1. Set a benchmark score for the translation quality of various translation services using a CHRF++ score.
  2. Train our model until its CHRF++ score is significantly better than the benchmark score.
  3. Perform a manual A/B test between the newly trained model and the existing translation service.

Synthetic data generation

To generate the training data needed to create our model, we had to rely on an open source LLM to generate the synthetic translation data. For this task, we spent considerable effort looking for a model which had both a large enough parameter count to ensure high quality outputs, but also a model which had the correct tokenizer to handle the diverse sets of languages which Grab’s customers speak. This is particularly important for languages which use non-standard character sets such as Vietnamese and Thai. We settled on using a public model from Hugging Face for this task.

We then used a subset of the previously mentioned benchmark translations to input as few-shot learning examples to our prompt. After many rounds of iteration, we were able to generate translations which were superior to the benchmark CHRF++ scores which we had attained in the previous section.

Model fine tuning

We now had one last step before we had something that was production ready! Although we had successfully engineered a prompt capable of generating high quality translations from the public Hugging Face model, there was no way we’d be able to deploy such a model. The model was far too big for us to deploy it in a cost efficient manner and within an acceptable latency. Our solution to this was to fine-tune a smaller bespoke model using the synthetic training data which was derived from the larger model.

These models were language specific (e.g. English to Indonesian) and built solely for the purpose of language translation. They are 99% smaller than the public model. With approximately 10 Million synthetic training examples, we were able to achieve performance which was 98% as effective as our larger model.

We deployed our model and ran several A/B tests with it. Our model performed pretty well overall, but we noticed a critical problem: sometimes, numbers got mutated in the translation. These numbers can be part of an address, phone number, price etc. Showing the wrong number in a translation can cause great confusion to the users. Unfortunately, an ML model’s output can never be fully controlled; therefore, we added an additional layer of programmatic check to mitigate this issue.

Post-translation quality check

Our goal is to ensure non-translatable content such as numbers, special symbols, and emojis in the original message doesn’t get mutated in the translation produced by our in-house model. We extract all the non-translatable content from the original message, count the occurrences of each, and then try to match the same in the translation. If it fails to match, we discard the in-house translation and fall back to using the third-party translation service.

Keep cost low

At Grab, we try to be as cost efficient as possible in all aspects. In the case of translation, we tried to minimise cost by avoiding unnecessary on-the-fly translations.

As you would have guessed, the first thing we did was to implement caching. A cache layer is placed before both the in-house translation model and the third-party translation. We try to serve translation from the cache first before hitting the underlying translation service. However, given that translation requests are in free text and can be quite dynamic, the impact of caching is limited. There’s more we need to do.

For context, in a booking chat, other than the users, Grab’s internal services can also send messages to the chat room. These messages are called system messages. For example,our food service always sends a message with information on the food order when an order is confirmed.

System messages are all fairly static in nature, however, we saw a very high amount of translation cost attributed to system messages. Taking a deeper look, we noticed the following:

  • Many system messages were not sent in the recipient’s language, thus requiring on-the-fly translation.
  • Many system messages, though having the same static structure, contain quite a few variants such as passenger’s name and food order item name. This makes it challenging to utilise our translation cache effectively as each message is different.

Since all system messages are manually prepared, we should be able to get them all manually translated into all the required languages, and avoid on-the-fly translations altogether.

Therefore, we launched an internal campaign, mandating all internal services that send system messages to chat rooms to get manual translations prepared, and pass in the translated contents. This alone helped us save roughly US$255K a year!

Next steps

At Grab, we firmly believe that our proprietary in-house translation models are not only more cost-effective but cater more accurately to our unique use cases compared to third-party services. We will focus on expanding these models to more languages and countries across our operating regions.

Additionally, we are exploring opportunities to apply learnings of our chat translations to other Grab content. This strategy aims to guarantee a seamless language experience for our rapidly expanding user base, especially travellers. We are enthusiastically looking forward to the opportunities this journey brings!

Join us

Grab is the leading superapp platform in Southeast Asia, providing everyday services that matter to consumers. More than just a ride-hailing and food delivery app, Grab offers a wide range of on-demand services in the region, including mobility, food, package and grocery delivery services, mobile payments, and financial services across 428 cities in eight countries.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Introducing quorum queues on Amazon MQ for RabbitMQ

Post Syndicated from Chris Munns original https://aws.amazon.com/blogs/compute/introducing-quorum-queues-on-amazon-mq-for-rabbitmq/

This post is written by Vignesh Selvam (Senior Product Manager – Amazon MQ), Simon Unge (Senior software development engineer – Amazon MQ).

Amazon MQ for RabbitMQ announced support for quorum queues, a type of replicated queue designed for higher availability and data safety. This post presents an overview of this queue type, describes when you should use it, and best practices you can follow. The post also describes how Amazon MQ has also improved quorum queues in the open-source RabbitMQ community.

Overview of quorum queues

A quorum queue is a replicated first in, first out queue type offered by open-source RabbitMQ that uses the Raft consensus algorithm to maintain data consistency. Each quorum queue has a leader and multiple followers (replicas), which ensure that messages are replicated and persisted across a majority of nodes, thus providing resilience against node failures. Quorum queues only need a majority of member nodes (a quorum) to make decisions about data. If a RabbitMQ node hosting a leader becomes unavailable, another node hosting one of the followers is automatically elected as the leader. Once the node becomes available again, the node will become a follower for the quorum queue and catch up or synchronize with the new leader. Quorum queues can detect network failures faster and recover quicker than classic mirrored queues, thus improving the resiliency of the message broker as a whole.

Quorum queues share most of the fundamental features that are key to RabbitMQ replicated queue types such as consumption, consumer acknowledgements, cancelling consumers, purging and deletion. Poison message handling is a unique feature of quorum queues which help developers manage unprocessed messages more efficiently. A poison message is a message that cannot be processed and ends up being repeatedly requeued. Quorum queues keep track of the number of unsuccessful delivery attempts and expose it in the ‘x-delivery-count’ header that is included with any redelivered message. A delivery limit can be set using a policy argument for ’delivery-limit’. If the limit is reached, the message can be dropped or put in a dead-letter queue. This feature further improves the data reliability of a quorum queue.

You can get started with quorum queues by explicitly specifying the ‘x-queue-type’ parameter as ’quorum’ on a RabbitMQ broker running version 3.13 and above. We recommend that you change the default vhost queue type to ’quorum’ to ensure that all queues are created as quorum queues by default inside a vhost.

RabbitMQ queues console

RabbitMQ queues console

When should you use quorum queues?

You should use quorum queues when you need higher availability and consistency for their messaging infrastructure. Quorum queues are ideal for scenarios where data durability and fault tolerance are critical, such as financial transaction systems, e-commerce data processing systems, or any application requiring high reliability. They are particularly beneficial in environments where node failures are more likely or where maintaining data consistency across distributed systems is essential.

When should you NOT use quorum queues?

Quorum queues are not meant to be temporary. They do not support transient or exclusive queues and are not meant to be used in scenarios with high queue churn (declaration and deletion rates). They are also not recommended for unreplicated queues.

Best practices for quorum queues

Quorum queues perform better when the queues are short. You can set the maximum queue length using a policy or queue arguments to limit the total memory usage by queues (max-length, max-length-bytes).

Add a new queue dialog

Add a new queue dialog

Amazon MQ recommends publishers to use publisher confirms and consumers to use manual acknowledgements on quorum queues. Publisher confirms will only be issued once a published message has been successfully replicated to a quorum of nodes and is considered safe within the context of the system. Publisher confirms can also serve as a form of back pressure and protect the availability of the broker during periods of high workload. Manual acknowledgements are used to ensure messages that are not processed can be returned to the queue for reprocessing.

Open-source improvements by Amazon MQ

Amazon MQ contributed multiple improvements to the open-source RabbitMQ community to improve quorum queues for operators and users.

Automatic membership reconciliation
Quorum queues depend on a majority of replicas being available for the Raft consensus algorithm. Amazon MQ identified that many users and operators would prefer to maintain a certain minimal number of replicas (generally 3 or 5) at all times to ensure a majority always exists. The quorum queue replica management was also initially available only via CLI tools. Amazon MQ engineers introduced automatic membership reconciliation to improve this experience. Now, RabbitMQ can be configured to identify any queues that are below a target group member size, and automatically grow or add a node to the queue members. Thus ensuring a certain minimum number of replicas always exist.

Voter status
RabbitMQ considers a quorum queue member node to be a full member even if the member has not caught up or fully synced to the quorum. The CLI command rabbitmq-queues check_if_node_is_ quorum_critical can provide a false positive, and indicate a node is safe to remove, even though another node has queue members that are still synchronizing to the quorum. Amazon MQ introduced a new ‘non-voter’ state for a queue member node to indicate a member that is still catching up or synchronizing to the quorum. If a queue has a member in this state, it is not considered a full member. Once the member is fully synchronized, it is automatically promoted to the voter status, and is considered a full member. The command rabbitmq-queues check_if_node_is_quorum_critical now takes this into account and correctly reports if a node can be safely terminated without any queues becoming unavailable due to a loss of majority.

Inconsistent state management
When a broker is overloaded, a quorum queue can end up in an inconsistent state, where the quorum queue membership state stored in the Raft state machine differs from the RabbitMQ internal state for the queue. Amazon MQ introduced a periodic check per quorum queue that identifies if a queue has an inconsistent state and takes action to fix it.

Default queue type
The default queue type for a RabbitMQ broker vhost was classic queues. You could declare a different queue type by explicitly stating the ’x-queue-type’ as a queue creation argument. Amazon MQ introduced a global default queue type in the configuration file (rabbit.conf) that provides the ability to define a default queue type at the broker level. Now, an operator can change the default queue type to quorum queues if not specified during creation.

Membership management permissions
RabbitMQ users are able to configure the quorum queue membership using the management API. This can interfere with automatic membership reconciliation. Amazon MQ introduced the ability for an operator to turn off the membership management permissions available through the management API. Thus, preventing customers from accidentally affecting their broker.

Conclusion

Quorum queues on RabbitMQ provide a robust solution for scenarios requiring high availability and resilience. By leveraging the Raft consensus protocol, quorum queues ensure that messages are safely stored and replicated across a quorum of nodes, making them an excellent choice for modern, distributed message queuing systems.

Amazon MQ recommends that you adopt quorum queues as the preferred replicated queue type on RabbitMQ 3.13 brokers. For more details, see Amazon MQ documentation. To know more about the open-source feature, see quorum queues.

Get started with quorum queues on Amazon MQ for RabbitMQ 3.13 with a few clicks.

Reaching your global players – message on a single, scalable communication hub

Post Syndicated from Richard Perez original https://aws.amazon.com/blogs/messaging-and-targeting/reaching-your-global-players-message-on-a-single-scalable-communication-hub/

Introduction

AWS provides solutions to address industry challenges, enabling customers to address the risk of potentially significant financial and reputational damage. This blog discusses how customers that have adopted Amazon Pinpoint have been able to deliver time-sensitive messages reliably and securely. Specifically, it can mitigate the risk associated with reaching players who have voluntarily self-excluded from marketing communications.

Regulated workloads in the betting and gaming industry refer to specific activities, systems, and processes that are subject to government regulations, oversight, and compliance requirements. The role of the gambling regulator is to ensure adherence to these requirements and maintain fairness, security, and responsible behavior within the industry. Regulators also exist to protect players, subject to their jurisdiction, by regulating the businesses that provide gambling services, imposing controls for operators, the Internet Software Vendor’s (ISVs) and players themselves. By way of example, the UK Gambling Commission (GC) is one such organization with the responsibility of regulating the individuals and businesses that provide gambling services in Great Britain. The UK GC and their counterparts across the globe regulate and license betting and gaming businesses to ensure they adhere to standards, and investigate and prosecute those that breach regulations and their conditions of licensure.

Player engagement

Player engagement in the Betting and Gaming industry refers to the level of involvement, enjoyment, and interaction that individuals experience when participating in various gambling activities. It encompasses aspects like personalized gaming experiences, social interactions, rewards, and challenges. Maintaining a strong relationship with players is vital for operators as it fosters loyalty, increases player retention, and drives engagement. Engaged players are more likely to return, deposit, and recommend the brand to others. Additionally, a positive operator-regulator relationship that promotes responsible gambling and regulatory compliance creates a trustworthy reputation, a key component for an operator’s long-term success in the industry. Player expectations and betting options are constantly changing and require operators to develop more engaging and sophisticated games to maintain participation and loyalty. At their core, they typically include:

Responsible Gaming

Proactively preventing messages being delivered to players that have self-excluded, mitigating the risk of incurring fines, and sending an SMS message for the purpose of interrupting gambling behavior or sending an email with guidance or support.

Transactional history

Sending statements of fact about the transactions that have completed and providing information to the player about the bets placed, and suspicious account activity. Send Real-time notifications that can include game results or betting odd changes. These can be delivered in multi-channel formats, subject to the player preferences.

Player Engagement

Deliver personalized notifications that are tailored to the gaming experience of the player. This can include games suggestions based on player’s preference, offering customized bonuses, or providing recommendations for in-game actions or new games. After confirming that a player is not on an exclusion list and opted to receive messaging with you can send notifications with bonuses and promotions offering attractive incentives to maintain player engagement. These can include welcome bonuses, free game access, and loyalty programs.

Account settings & Security messaging

This can be in the form of SMS to provide seamless account creation, onboarding verification and deliver one-time-passwords to accelerate player identification. Access codes can be delivered where a unique code is delivered to the player in a time sensitive window to provide access to a particular game.

Responsible Gaming

Operators face several challenges when it comes to maintaining player engagement while adhering to regulatory mandated responsible gambling standards, requiring they balance engagement and retention strategies with player well-being and safe-play requirements. A common challenge faced by operators is ingesting multiple self-exclusion lists to ensure self-excluded users do not receive engagement and retention offers. Operators do employ various activities, such as educational initiatives, such as providing players with information about responsible gambling, offering responsive customer support to address player issues and queries promptly, and use machine learning to identify problematic gambling and take proactive actions. It is also worth noting that operators collaborate with problem gambling support organizations to provide resources and assistance to affected individuals.

Players can also take decisive action themselves and are free to opt-out of all gambling notifications for a set length of time, usually between 6 months and five years. The gambling operators in that jurisdiction are required to observe this request by law or face million-dollar fines and reputational damage. With increased player participation, ensuring you have the right tools and services in place in critical, as highlighted by an AWS customer.

Betting and Gaming has become one of the fastest growing sectors in the last 20 years. Everyday millions of people use these platforms for their betting and entertainment needs. tombola recognize the importance of having a reliable and scalable outbound communication hub for engaging with and protecting our players. Reliable communication is essential for a safe, engaging and sustainable customer experience.

James Conway, Technology Director, tombola

How to build Player Engagement solutions in the Betting and Gaming industry with AWS

AWS offers robust solutions that can benefit the Betting and Gaming industry. With the challenges discussed in this blog, AWS provides the infrastructure and solutions necessary to address these challenges effectively.

In a series of blogs, we will be sharing how AWS and Amazon Pinpoint can support industry focused use cases. Other solutions in this series are;

  • How you can use AIML and GenAI to create player recommendations to improve player engagement
  • How you use Amazon Pinpoint for improved onboarding and user authentication
  • Identify player trends and next best actions using Amazon Pinpoint and AIML

The first in this four-part series is how we are supporting customers with responsible gaming, and how they can mitigate the risk of contacting players that have chosen not to receive any more communication.

Amazon Pinpoint is a key service in this solution. Pinpoint is a purpose-built, customer engagement service with multi-channel capabilities, message delivery, management, and optimization. This allows AWS customers to send personalized messages, promotions, and notifications to users via various channels, through a single console. Customers view Amazon Pinpoint as their communication hub because they can consolidate the services and tools used for their outbound channels to just one. It becomes the central service for managing and facilitating communications between different stakeholders. It streamlines and creates efficiency for front and back-office teams as they are able to concentrate on the player’s experience. Customers are looking for an easy, optimized, and consolidated solution to manage their digital communications. Customers often have different services or tools for each communication channel, causing operational complexity that can lead to system issues, inefficiencies and breaches in regulatory obligations. Operators face issues when different teams and stakeholders have to sync between their customer data repository and these services and tools. Maintaining the integrity of these systems can be cumbersome and increase the risk in the event of a sync or systems error. By maintaining a single service for outbound communications and simplifying the customer data, you are able to innovate and deliver player centric solutions. Below we have provided a selection of use cases focusing on player engagement.

Responsible Gaming – Mitigate the Risk of Targeting Self-Excluded Players on AWS

Under the UK gaming regulations, operators are required to observe a self-exclusion list, managed by GAMSTOP. Amazon Pinpoint queries GAMSTOP’s API in real time before sending the message to a customer. The above is possible with the use of Amazon Pinpoint’s Campaign Hooks and Journey Hooks. Campaign and Journey hooks can be used for other use cases that require reaching out to a 3rd party service in real time, such as players’ balance check before sending a message. Gamstop has been used in this example but can apply to any self-exclusion site or an internal exclusion list.

Tombola, who operate the largest online bingo website in Europe, successfully adopted Pinpoint across all their territories said,

‘At Tombola, we needed a communication tool that helped us deliver on one of our core values: responsible gambling. Players can opt out of marketing communications from online gambling operators by registering themselves on an independent self-exclusion scheme. We needed a system that helped deliver consistency with evolving changing requirements and ensure, in real-time, we have the controls in place not to contact the wrong people. Amazon Pinpoint allowed us to consolidate the various communication systems we had deployed across territories and allowed us to continually evaluate the content and the recipients of our communication. Only Pinpoint’s system could keep up with our shifting requirements, so we can easily connect with the players who want to hear from us, but never contact those who had self-excluded.’

About the Authors

Richard Perez

Richard Perez

Richard Perez is a Principle Go-to-Market Specialist for Communication Developer Services at AWS. He enjoys diving deep into customers’ issues and helping to deliver communication solutions. In his spare time, he enjoys watching and participating in sports, and reading historical novels.

Understanding Google Postmaster Tools (spam complaints) for Amazon SES email senders

Post Syndicated from Bruno Giorgini original https://aws.amazon.com/blogs/messaging-and-targeting/understanding-google-postmaster-tools-spam-complaints-for-amazon-ses-email-senders/

Introduction

Amazon Simple Email Service (SES) includes a robust set of built-in tools, such as the Virtual Deliverability Manager (VDM), to help senders ensure optimal email deliverability. Additionally, deliverability data from email service providers like Postmaster Tools by Google can provide invaluable insights for all sending domain owners, including those using SES for bulk or transactional email. Postmaster Tools offers detailed metrics on factors like delivery errors, spam rates, domain reputation, and recipient feedback for Gmail-hosted inboxes. Combining this external data with SES email sending events is critical for maintaining a healthy sender reputation. By leveraging both SES-native tools and resources like Postmaster Tools, senders can identify and address deliverability issues, ensuring their SES-powered emails reach intended recipients across providers.

Many, but not all, mailbox providers will send recipient feedback in the form of “complaints” that can each be attributed directly to the message that the recipient found to be objectionable. These complaints are available in the SES email sending event type “Complaint”. Gmail does not send spam complaint events because their priority is to protect the privacy of their users from the tracking techniques employed by spammers and data brokers. Gmail requires bulk senders to adopt “easy unsubscribe” mechanisms to reduce the need for their users to report messages as spam, and they will show spam complaint metrics in Postmaster Tools. This blog will show you how to maximize value in the spam complaint metric provided by Postmaster Tools.

Amazon SES now supports custom values in the Feedback-ID header in messages sent through SES. This feature provides additional details to help customers identify deliverability trends. Together with Postmaster Tools, customers can group complaints by identifiers of their choice, such as sender business unit or campaign ID. This makes it easier to track deliverability performance associated with independent workloads and campaigns, and accelerates troubleshooting when diagnosing complaint rates.

This image describes the flow for spam complaints to the email sender

Figure 1: Email Feedback Loop

This blog will guide you through implementing and using Feedback Loops within Postmaster Tools to identify email campaigns receiving high complaint volumes from Gmail users. It covers the history and background of feedback loops, the specific requirements for implementing them with Postmaster Tools, and practical examples using AWS CLI and Boto3 to send SES emails with the necessary Feedback-ID header. By the end, you’ll understand how to effectively set up and use Postmaster Tools to monitor and improve your SES email deliverability.

History and Background of FBLs

Traditional Feedback Loops (herein “FBLs”) have been a cornerstone of email deliverability for many years. Initially developed by Internet Service Providers (ISPs), FBLs serve as a mechanism for recipients to report spam complaints to the sender. This feedback is crucial for email service providers and senders to identify problematic email campaigns, take corrective actions, and maintain a healthy sender reputation.

FBLs operate by allowing recipients to mark emails as spam, which then sends a report to the sender’s email service provider. This report typically includes details about the email that triggered the complaint, enabling the sender to investigate and address any issues. By analyzing these reports, senders can refine their email lists, improve content, and ensure that their emails comply with best practices and regulatory requirements. Senders who receive a higher volume of spam complaints are more likely to be blocked or have their emails routed to the spam folder. While high spam complaints are not the sole reason for deliverability issues, they are often the underlying cause.

Postmaster Tools by Gmail is not a traditional FBL. Postmaster Tools will show complaint feedback metrics, but the complaints are not attributable to any individual recipient.

Requirements for using Postmaster Tools FBL with SES

The FBL helps identify campaigns with high complaint rates from Gmail users, specifically useful for email service providers to detect potential abuse of their services.

Note: Data in Postmaster Tools only applies to messages sent to personal Gmail accounts. A personal Gmail account is an account that ends in @gmail.comor @googlemail.com.

  • Implementation of FBL:
    • Feedback-ID Header: SES embeds a header called Feedback-ID containing parameters (Identifiers) uniquely identifying the account and SenderID (AmazonSES)
    • Header Format: The Feedback-ID header consists of four parameters, separated by colons:
      a:b:c:SenderId
      Where:
      • SenderId is a mandatory parameter that uniquely identifies the sender.
      • In the case of Amazon SES (Simple Email Service), the SenderId is always “AmazonSES” and cannot be overridden.
Header Parameter  Description
a  First parameter in the Feedback-ID header. SES users can customize through ses:feedback-id-a EmailTag
b  Second parameter in the Feedback-ID header. SES users can customize through ses:feedback-id-b EmailTag.
c  Third parameter in the Feedback-ID header. SES uses this to identify the sender account
SenderID  Fourth parameter in the Feedback-ID header. Mandatory parameter that uniquely identifies the sender. For Amazon SES, this is always “AmazonSES” and cannot be overridden.
  • Sender Data Handling:
    • DKIM signing by a sender-owned domain is required to prevent spoofing.
    • The domain must be added and verified in Postmaster Tools.
    • Complaint data is aggregated by distinct values on each of the 4 fields of Feedback-ID.
  • Feedback-ID header Requirements:
    • When sending emails through Amazon SES, users are limited to a single verified header value per traffic stream.
      • This means that the Feedback-ID header cannot contain an individualized value for each destination email address.
      • Instead, the Feedback-ID header needs to contain an identifier that can be used to match a larger campaign or batch of emails, rather than a unique value per recipient.
      • This constraint helps maintain a consistent sender reputation, improves deliverability monitoring and troubleshooting within tools like Postmaster Tools. The Feedback-ID acts as a grouping mechanism, rather than a per-message identifier
    • Identifiers must be unique and non-repetitive across fields.
  • Feedback-ID Example:
    • CampaignIDX:CustomerID2:1.us-west-2.TDQeKqHkSNfQztk25wIeVIGTuNmGDud4r1l7dUlxOio=:AmazonSES
      • Each Identifier is used to report spam percentages independently if unusual rates occur.
      • Amazon SES lets customers set the part a and part b of the Feedback-ID header using the EmailTag ses:feedback-id-a and ses:feedback-id-b
      • Amazon SES will combine these tags into a single Feedback-ID header with the format: Feedback-ID=a:b:region.accountId:AmazonSES

The next steps will cover what’s needed to leverage FBLs with SES.

Step 1 – Add Your Domain(s) To Google’s Postmaster Tools

  • In order to verify with Postmaster Tools that you’re authorized to track the feedback from your domain, you first need to register your ownership of the domain with Postmaster Tools by visiting https://gmail.com/postmaster/.
Verify a new domain with Google Postmaster Tools

Figure 2: Step 1 to verify a domain in Google Postmaster Tools

  • After entering in your domain, you’d be prompted to add a TXT record into your DNS configuration.
Step 2 to verify a domain in Google Postmaster Tools

Figure 3: Step 2 to verify a domain in Google Postmaster Tools

  • Update your sending domain(s) DNS records accordingly.
    • The example below specifies how to create the TXT record in Route53. If you’re using another DNS service provider, please refer to their documentation.
Create a new record in Route53

Figure 4: Create a new record in Route53

    • Navigate to the Route53 Console and click on Hosted zones , specify the hosted zone that contains the domain you want to verify and then Create record.
This image describe the creation of a TXT record including the value provided by Google Postmaster to verify the domain

Figure 5: Add a TXT record with the provided value for verification

    • Following the screenshot, create a TXT record type and paste the value assigned by Google for verification in step 2 here.
  • Go to Postmaster Tools and click on Verify. After successful verification of your domain in Postmaster Tools, you should see the Status column changed from Not Verified to Verified. You can verify your compliance status with the requirements in the Dashboard (2) link.
In this picture we show an example of how the domain would appear once verified in Google Postmaster Tools

Figure 6: Domain verified

  • Follow the recommendations provided in the Postmaster Tools dashboard to fully comply with the requirements (example below):
Email sender requirements

Figure 7: Email sender requirements compliance status recommendations

  • Once you have completed all the verification and configuration steps, you should see compliant checkmarks next to all available requirements (see example below):
Email sender requirements

Figure 8: Email sender requirements compliant status

Step 2 – Add Feedback-ID headers to your SES emails

  • Use this command line to send an email with Feedback-ID using the AWS CLI:
aws sesv2 send-email --from-email-address [email protected] \
   --destination '{"ToAddresses":["[email protected]"]}' \
   --content '{"Simple":{"Subject":{"Data":"Test Subject","Charset":"UTF-8"},"Body":{"Text":{"Data":"Test Data","Charset":"UTF-8"}}}}' \
   --email-tags '[{"Name": "ses:feedback-id-a","Value":"feedback-id-part-a-value"}]'

The values of ses:feedback-id-a and ses:feedback-id-b are specified using the --email-tags option.

  • Alternatively, use Boto3 to send an email with Feedback-ID with the following Python script:
import boto3
from botocore.exceptions import ClientError

def send_email(region_name):
    # Create a new SES client
    ses = boto3.client('sesv2', region_name=region_name)

    # Replace sender and recipient values
    SENDER = "Sender Name <[email protected]>"
    RECIPIENT = "[email protected]"
    CONFIGURATION_SET = "SES_Config_Set"
    SUBJECT = "Amazon SES Test (SDK for Python)"
    BODY_TEXT = "Amazon SES Test (Python)\r\nThis email was sent with Amazon SES using the AWS SDK for Python (Boto)."
    BODY_HTML = """<html>
    <head></head>
    <body>
      <h1>Amazon SES Test (SDK for Python)</h1>
      <p>This email was sent with
        <a href='https://aws.amazon.com/ses/'>Amazon SES</a> using the
        <a href='https://aws.amazon.com/sdk-for-python/'>
          AWS SDK for Python (Boto)</a>.</p>
    </body>
    </html>"""
    CHARSET = "UTF-8"

    try:
        # Send email
        response = ses.send_email(
            FromEmailAddress=SENDER,
            Destination={'ToAddresses': [RECIPIENT]},
            ConfigurationSetName=CONFIGURATION_SET,
            Content={
                "Simple": {
                    "Subject": {
                        "Charset": CHARSET,
                        "Data": SUBJECT
                    },
                    "Body": {
                        "Text": {
                            "Charset": CHARSET,
                            "Data": BODY_TEXT
                        },
                        "Html": {
                            "Charset": CHARSET,
                            "Data": BODY_HTML
                        }
                    },
                    "Headers": [
                        {
                            "Name": "List-Unsubscribe",
                            "Value": "<https://unsubscribe.example.email/[email protected]&topic=topic1>"
                        },
                        {
                            "Name": "List-Unsubscribe-Post",
                            "Value": "One-Click"
                        }
                    ]
                }
            },
            EmailTags=[
                {
                    'Name': 'ses:feedback-id-a',
                    'Value': 'campaign1'
                },
                {
                    'Name': 'ses:feedback-id-b',
                    'Value': 'line-of-business'
                }
            ] #the ses:feedback-id-a and ses:feedback-id-b are specified as a list using EmailTags
        )
        print("Email sent! Response:", response)
        print("Message ID:", response['MessageId'])

    except ClientError as e:
        print(e.response['Error']['Message'])

# Call the function to send the email
send_email(region_name='us-west-2')  # Specify the region here

Step 3 – Viewing FBL results in Postmaster Tools

In order to see any results in the Postmaster Tool dashboard (see examples below), you must send a substantial daily volume of email through the domain(s) you’ve registered. If you see the message “No Data to Display”, your reputation may already be too low, more likely the volume of email traffic sent since you configured the Postmaster tool is insufficient (return to the dashboard in later, after you’ve sent 1,000s of emails).

Figure 9: Feedback loop example image

Figure 9: Feedback loop example image

The image shows a section of the Postmaster Tools dashboard, specifically the Feedback Loop section. This dashboard provides insights into the spam complaint rates and the number of feedback loop identifiers flagged across a given time period, in this case, the last 120 days.

Conclusion

High-volume email senders should look to the combination of Amazon SES’ powerful framework for monitoring in concert with Postmaster Tools to improve and ensure email deliverability. Implementing the Feedback-ID header in your SES emails can significantly enhance your ability to track and troubleshoot deliverability issues. Use Postmaster Tools and the Feedback Loop via Feedback-ID headers in SES emails to gain detailed insights into complaint rates and other key metrics, enabling you to maintain a healthy sender reputation and ensure their emails reach the intended recipients.

Call to Action:

  1. Set Up Postmaster Tools for your sending domain(s)
  2. Verify Your Domain: Register and verify your domain with Postmaster Tools to access valuable insights and track your compliance status.
  3. Set Up Feedback-ID: Start embedding the Feedback-ID header in your emails sent via Amazon SES to take advantage of detailed complaint data and improve your email campaigns.
  4. Monitor and Adjust: Regularly check the Postmaster Tools dashboard to monitor your spam rates and feedback loop identifiers. Use this data to refine your email content and sending practices.
  5. Leverage AWS CLI and Boto3: Utilize the provided AWS CLI commands and Boto3 scripts to automate the process of sending emails with Feedback-ID headers, ensuring consistent and accurate tracking.

By following these steps, you can enhance your email deliverability, reduce spam complaints, and maintain a strong sender reputation. For more information on using Amazon SES and Google’s Postmaster Tools, refer to the Amazon SES Documentation and the Postmaster Tools Guide.

How to use Mail Manager to Archive Inbound Emails

Post Syndicated from Sesan Komaiya original https://aws.amazon.com/blogs/messaging-and-targeting/archiving-and-sending-to-final-smtp-server/

In today’s digital landscape, where email communication plays a vital role in business operations, Keeping your email archive secure, compliant, and retrievable is crucial for any business. However, managing the large volume of email data can lead to operational difficulties, including regulatory compliance, maintaining an audit trail, and preventing data loss. That’s where Amazon Simple Email Service (SES) Mail Manager’s email archiving feature comes in.

In this blog post, we will explore how Amazon SES Mail Manager’s email archiving and search features can improve email security and compliance. If you’re a newcomer to Mail Manager, look at this blog post on Amazon SES Mail Manager. It provides valuable information on important features, such as Ingress Endpoint, Traffic Policy, Rule Sets, and SMTP Relay.

Problem Statement:

Imagine a scenario where a critical email from a key client is buried deep within your organization’s email archives, and you need to retrieve it for an important audit. The challenge of ensuring your business remains compliant with stringent data retention policies across every email communication for thousands of employees for a certain period or permanently.

Solution explained:

Amazon SES Mail Manager Email archiving is a powerful tool that addresses many of the challenges organizations face dealing with the difficulty and expense of archiving email at scale. Compliance and regulatory frameworks like GDPR, HIPAA, and SOX often require email archiving, which is a common objective identified by customers needing to comply with those regulatory frameworks. For regulated businesses, failure to comply with email archiving regulations can result in severe financial penalties and reputational damage.

Amazon SES Mail Manager securely archives and safeguards your emails, providing easy search and export functionality. It provides full-time, enterprise-level archiving without increasing the storage requirements of your mailbox server. The feature provides a reliable and efficient solution to address most compliance requirements. By automatically archiving the types of emails you specify, the service ensures that your organization maintains a comprehensive audit trail of its communications, enabling quick retrieval and review as needed.

The email archiving feature of Mail Manager provides organizations the ability to archive email while in transit rather than archiving at the user’s mailbox. Many organizations prefer archiving in transit for email archiving to meet compliance requirements and maintain comprehensive records. If you would like to learn more about in transit archiving, visit this blog –  Email Archiving with Mail Manager: Why To Archive In Transit vs At The Mailbox.

How email flows with an Amazon SES Mail Manager Email Archiving

For instructional purposes in this blog post, we’ll focus on how you can introduce Mail Manager archiving into your existing email infrastructure. We’ll cover how to seamlessly integrate Mail Manager with reference architectures. Later in the blog, we are going to explore Mail Manager’s archiving capabilities, including search, export and retention policies.

Current setup

Our example organization has an existing mail server (it might be a on-premises Microsoft Exchange Server, Microsoft 365, Google Workspace, etc). Their DNS is configured to route all email directly to this mail server. There is currently no archiving capacity within the existing email infrastructure, when needed, archiving is handled by individual mailbox users and PST files. While this method is suitable for personal email archiving, it fails to meet the organizations’ security requirements and compliance standards.

Figure 1: Example organization’s existing inbound email workflow.

Email Archiving in transit

We are going to introduce Mail Manager into the current mail flow (see figure 1) to archive all incoming messages from our example enterprise’s email infrastructure.

Figure 2: Example organization’s proposed inbound email workflow, with Mail Manager archiving in-transit prior to delivery.

In the new architecture (see figure 2), we’ve introduced Mail Manager into the organization’s inbound email workflow. This new workflow leverages Mail Manager’s ability to archive either all inbound emails, or only those that match specified criteria. By using a Mail Manager Rule set, our example organization can selectively store and preserve emails that meet their configured criteria.

Mail Manager Email Archiving and Search and Export Capabilities

Mail Manager’s archive search capabilities are designed to be user-friendly and efficient. You can perform searches based on various criteria, such as sender, recipient, subject line, date range, or even specific keywords in the Subject line. The search results provide options to either export the search to Amazon Simple Storage Service (S3), or you can choose to download a single email.

Let’s explore using Mail Manager’s archive search to find a specific email by the sender’s address:

Figure 3: Mail Manager’s archive search interface

Once found, we can click on the results to review the email in the console:

Figure 4: Mail Manager’s archive search results

Once we’ve found the target email, it can be downloaded by clicking on “View details“. The image below shows an example message details page with information about the email, including message headers such as In-Reply-To, X-Original-Mailer and X-Mailer.

Figure 5: Mail Manager’s archived email detailed page

Mail Manager’s archive search history tab allows us to find archive searches created in the last 30 days, and view the search results, as shown in the image below:

Figure 6: Mail Manager’s archive search history

Mail Manager’s archive export history tab lists all of the archived email searches you exported to an Amazon S3 Bucket within the last 30 days.

Figure 7: Export History

Step by Step Setup:

Now that we have explained how Mail Manger can be inserted into our example organization’s email workflow to provide email archiving, let’s explore how you can implement Mail Manager’s archiving capabilities in your inbound email workflows. The following diagram (Figure 3) illustrates the overall structure and components involved in this architecture:

Figure 8: End-to-End Mail workflow

Follow the steps below to configure Mail Manager in your AWS account to implement this architecture:

  1. Log into the SES Console and select Mal Manager from the left navigation menu.
    1. Note, as of this writing, Mail Manager is generally available in the following AWS Regions: US East (N. Virginia), US West (Oregon), Europe (Ireland, Frankfurt), Asia Pacific (Tokyo, Sydney).
  2. Under Mail Manger navigation, create an archive (or multiple archives for different use cases)
    1. Enter a unique name in the Archive name field.
    2. (Optional) Select a retention period in the Retention period field to override the default retention period of 180 days.
    3. (Optional) You can encrypt your archive either by entering your own AWS KMS key into the KMS key ARN field, or by selecting Create new key.
    4. Choose Create archive.
  3. Under Mail Manger navigation, create a traffic policy to determine the email you want to block or allow.
    1. Create traffic policy.
    2. On the Create a traffic policy page, enter a unique name for your traffic policy.
    3. (Optional) If you want to discard any messages above a certain size, enter a value in bytes in the Maximum message size field.
    4. In Default action, choose whether the traffic policy is to either Allow or Deny (block) messages that fall outside of (are not addressed by) the conditions of your policy statements.
    5. Select Add new policy statement to create a statement for your traffic policy.
    6. Choose either Allow or Deny (block) for the action to be taken when the statement’s conditions are met.
    7. Build a condition by selecting an email protocol and a conditional operator for the value you enter. Select Add new condition if you want to add more conditions to this policy statement. To learn more about a condition property and its operators and valid values, see the Policy statement conditions reference.
    8. If you’re subscribed to an Email Add On, you’ll be able to select it here as an email protocol.
    9. If you want add more policy statements and conditions, repeat steps above.
    10. Select Create traffic policy.
  4. Under Mail Manger navigation, create a rule set to perform actions on the email you allow in.
    1. Create rule set and enter a unique name for your rule set.
    2. Create new rule on the edit page.
    3. In the Rule details sidebar, enter a unique name for your rule.
    4. Select Add new condition to create a condition that the message must match; or check the EXCEPT in the case of: box followed by Add new exception to create a condition that the message must not match.
    5. Build the condition or exception by selecting an email property and a conditional operator for the value you enter. Select Add new condition or Add new exception if you want to add more conditions or exceptions to this rule. To learn more about a condition property and its operators and valid values, see the Rule conditions reference.
    6. Select Add new action to define the action to be taken when the rule’s conditions are matched and/or exceptions are not matched. To add more actions to be taken, select Add new action. To learn more about actions and their parameters, see the Rule actions reference.
    7. Create an Archive rule. Save rule set
  5. Under Mail Manger navigation, Create your ingress endpoint and assign to it the traffic policy and rule set.
    1. choose Ingress endpoints under Mail Manager.
    2. On the Ingress endpoints page, select Create ingress endpoint.
    3. On the Create new ingress endpoint page, enter a unique name for your ingress endpoint.
    4. Choose whether it will be a Open or Authenticated endpoint.
    5. Select a traffic policy to determine the email you want to block or allow.
    6. Select a rule set containing the rule actions you want to perform on the email you allow in.
    7. Select Create ingress endpoint.
  6. Configure your environment to use the ingress endpoint.
    1. At the time you create an ingress endpoint, an “A” record for the endpoint will be generated and its value displayed on the ingress endpoint’s summary screen in the SES console. The way you use the value of this record depends on the type of endpoint you created and your use case.
    2. DNS providers have different procedures and interfaces for configuring email records. The key pieces of information you need to put into your DNS settings are listed in our documentation – https://docs.aws.amazon.com/ses/latest/dg/eb-ingress.html#eb-ingress-a-record
  7. Under Mail Manger navigation, create an SMTP Relay to send mail on to your existing mail server.
    1. choose SMTP relays under Mail Manager.
    2. On the SMTP relays page, select Create SMTP relay.
    3. On the Create SMTP relay page, enter a unique name for your SMTP relay.
    4. Depending on what type of SMTP Realy you want to configure, follow the respective instructions:
      1. inbound (non-authenticated)
      2. outbound (authenticated) SMTP relay
  8. Update your DNS MX records to point to your new Mail Manager’s ingress point, instead of the existing mail server.

Note: Make sure that you have tested the steps above in your development environment and that you understand the steps before deploying into your production environment.

Conclusion

Amazon SES Mail Manager’s email archiving capabilities are designed for organizations that are seeking to enhance the security, compliance, and audit-ability of their email communications. By seamlessly integrating this feature into their existing email infrastructure, organizations can now archive all inbound messages in transit, ensuring a comprehensive, tamper-proof record of their email activities. The powerful search and export functionality of Mail Manager makes it easy to quickly locate and access specific emails when needed, whether for compliance audits, legal requests, or internal investigations.

This level of email visibility and control is particularly crucial for organizations operating in highly regulated industries like government, healthcare and finance, where the stakes for non-compliance can be severe. Beyond the compliance benefits, Mail Manager’s email archiving also helps to alleviate the operational headaches and expenses associated with traditional in-house archiving systems. By offloading this responsibility to AWS, organizations can focus their resources on their core business priorities, while still maintaining the security and accessibility of their critical email data.

If you’re looking to strengthen your email security posture, simplify your compliance efforts, and improve the overall management of your email archives, we encourage you to explore how Amazon SES Mail Manager’s email archiving capabilities can be seamlessly integrated into your existing email infrastructure. Take the first step towards a more secure, compliant, and efficient email management solution by contacting us today.

About the Authors

Sesan Komaiya

Sesan Komaiya

Sesan is a Solutions Architect at Amazon Web Services. He works with a variety of customers, helping them with cloud adoption, cost optimization and emerging technologies. Sesan has over 15 year’s experience in Enterprise IT and has been at AWS for 5 years. In his free time, Sesan enjoys watching various sporting activities like Soccer, Tennis and Moto sport. He has 2 kids that also keeps him busy at home.

Alexey Kurbatsky

Alexey Kurbatsky

Alexey is a Senior Software Development Engineer at AWS, specializing in building distributed and scalable services. Outside of work, he enjoys exploring nature thru hiking as well as playing guitar.

Jesse Thompson
Jesse Thompson is an Email Deliverability Manager with the Amazon Simple Email Service team. His background is in enterprise development and operations, with a focus on email abuse mitigation and encouragement of authenticity practices with open standard protocols. Jesse’s favorite activity outside of technology is recreational curling.
Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

How to use SES Mail Manager SMTP Relay action to deliver inbound email to Google Workspace and Microsoft 365

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-use-ses-mail-manager-smtp-relay-action-to-deliver-inbound-email-to-google-workspace-and-microsoft-365/

Introduction

Customers often ask us if the Amazon Simple Email Service (SES) inbound capabilities they use with applications hosted on AWS infrastructure can also be used to process and automate employee email hosted on public services like Google Workspace and Microsoft 365. The answer has typically been “yes, but with some limitations”, as until now, SES inbound has been somewhat constrained by the fact that it didn’t support relaying messages for an existing domain. This limitation makes it very difficult to fully manage email flows across hybrid email environments.

Such conversations led the SES team to create Amazon Simple Email Service (SES) Mail Manager which offers a set of capabilities that simplify managing large volumes of email communications within an organization. Mail Manager’s rules set conditions and actions can optimize routing for improved delivery and communication flow, both for incoming and outgoing emails. Mail Manager’s email security features can be augmented by optional add-ons from industry-leading, vetted third-party providers. Flexible archiving features help organizations meet stringent compliance and record-keeping requirements.

In this blog, we position Mail Manager as a central ingress gateway for a fictitious company, Nutrition.co, that is based on real-world AWS customers. We discuss the customer challenges and explain how to configure Mail Manager’s SMTP Relay action to intercept, archive then deliver emails destined for employees’ Google Workspace hosted Gmail and Microsoft 365 hosted Exchange Online mailboxes. Similar mail flows can be used to process, automate and archive emails destined for their AWS hosted apps.

You can learn more about all of Mail Manager’s capabilities here.

Customer background and use case

Our fictitious company, Nutrition.co, is an online retail business with multiple employee departments, including administration, marketing, sales and fulfillment. The company has acquired several smaller rivals that use both Google Workspace and Microsoft 365 to host their employee inboxes, and plan to consolidate all users onto the same domain ( such as [email protected] and [email protected]). They also host several applications on Amazon Web Services (AWS) that use Amazon SES’ inbound capability to receive emails using a subdomain *customer-support*.nutrition.co, such as orders@*customer-support*.nutrition.co and returns@*customer-support*.nutrition.co.

Nutrition.co is looking for a solution to unify all their email domain routing, security and archiving processes onto one centralized management system to simplify their email infrastructure. They want an approach that provides more flexibility to control which addresses and domains are used for apps and automation as well as employee mail. They also want to enhance email compliance and governance with a flexible solution for screening, processing and archiving inbound emails to both employees and applications, before delivering those emails to recipient inboxes on Google Workspace and Microsoft 365 and applications hosted on AWS.

The SES Mail Manger based central ingress and egress gateway architecture we propose will allow Nutrition.co to manage their peer-to-peer and application-driven emails in one place, Amazon SES. It will simplify email security and management, and make it easy to unlock new cloud-enabled email use cases. The architecture can be modified to acommodate a wide variety of email infrastructure, including fully cloud hosted, on-premises, and hybrid mailbox hosting environments.

What is an Inbound SMTP Gateway?

An Inbound SMTP Gateway is an SMTP server that accepts inbound email via an Open Ingress Point, and then delivers those messages to another email environment’s inbound SMTP server. In the diagram below, Mail Manger is configured as an inbound SMTP Gateway:

Figure 1: Diagram of the inbound gateway mail flow to a mailbox hosting environment

Figure 1: Diagram of the inbound gateway mail flow to a mailbox hosting environment

“Inbound email” refers to email traffic flows where the originator of the message can be either a trusted (for example: the UK division of Nutrition.co) or an untrusted (for example: a Nutrition.co customer or vendor) entity. To send an email, the originating email system looks up the recipient domain’s MX record in the global DNS system to determine the address for the recipient’s inbound mail server. Once a connection is established on port 25, the originating server delivers the email message using the SMTP protocol typically using STARTTLS for transport level encryption. Inbound messages are typically authenticated using the SPF, DKIM, and DMARC industry standard protocols, which help ensure the messages are coming from the legitimate sender’s domain.

An Inbound SMTP gateway can act on messages, for example to process and/or archive, before passing them along to the end recipient’s email server. To learn more about archiving emails in transit, visit this blog.

Configuring Mail Manager as an Inbound SMTP Gateway

Before we can configure Mail Manager as an Inbound Gateway for Nutrition.co’s Google Workspace and Microsoft 365 hosted mailboxes, we need to “allow-list” Mail Manager in Nutrition.co’s Google Workspace and Microsoft 365 settings. Allow-listing in this context refers to configuring the hosted mailbox environments such that Mail Manager is not identified as the source of messages, but rather as an SMTP relay.

This configuration is necessary because the messages being relayed through Mail Manager originate from both trusted and untrusted senders. This mail flow will contain both wanted and, potentially, unwanted messages. Mail Manager is the intermediary, not the source of potentially unwanted email passing through Mail Manager’s Open Ingress Point before being relayed to the destination mailbox environment.

If Mail Manager is not allow-listed, inbound email that is relayed thru Mail Manager’s Open Ingress Point will fail SPF checks because the IP addresses of the intermediary server are not authorized by the domain’s SPF policy. Since DMARC relies on SPF, messages from intermediary mail servers will fail the domain’s DMARC policy if they are not signed with a domain-aligned DKIM signature.

Mailbox hosting environments and their anti-spam algorithms rely on SPF, DKIM and DMARC for authenticating different inbound mail flow configurations before making an assessment about the message’s disposition. Properly authenticated messages, if not otherwise identified as unwanted by recipients and their security administrator, are delivered to Inboxes. Messages that are not authenticated are more likely to be treated as spam. Messages from intermediary servers can sometimes be mistaken as spoofed or unwanted messages.

By allow-listing the egress IP addresses of the Mail Manager servers, Nutrition.co’s Google Workspace and Microsoft 365 hosting environments will be able to assess the correct SPF result when receiving inbound email from Mail Manager.

Note: Do not include Mail Manager’s IP addresses in the domain’s SPF policy, These IP addresses are shared by other Mail Manager customers so including them in the domain’s SPF policy can introduce a security risk.

Note: It is also possible to use DKIM and ARC for allow-listing mail streams, but Gmail and Exchange Online both support IP allow-listing.

Note: Nutrition.co’s Google Workspace and Microsoft 365 hosting environments may still make a spam assessment about the messages under the context that Mail Manager is not the original sender, but this is not common.

Figure 2: Diagram of the SES Mail Manager architecture to accept inbound email via an open Ingress endpoint and configured with a Rule set condition to relay messages with the SMTP Relay action.

Figure 2: Diagram of the SES Mail Manager architecture to accept inbound email via an open Ingress endpoint and configured with a Rule set condition to relay messages with the SMTP Relay action.

In the diagram above, the interaction points are as follows:

1. Email senders look in DNS to discover the MX record for example.com.
2. The value of the domain’s MX record is the A record of the Mail Manager Ingress endpoint. The Ingress endpoint is configured as an ‘open’ Ingress endpoint so that it can receive inbound email without requiring SMTP Auth
3. The Ingress endpoint traffic policy is configured to allow and deny traffic
4. The Rule Set conditions determine which messages are to be relayed
5. The SMTP Relay action relays messages for recipients that are SES verified identities

Configuring Mail Manager as an Inbound SMTP Gateway

Prerequisites

  • Access to the administrative console for Nutrition.co’s Google Workspace and Microsoft 365 hosted mailboxes
  • Access to the DNS zone hosting the MX records for the Nutrition.co’s domains

Step 1: Allow-list the regional Mail Manager IP addresses in Nutrition.co’s Google Workspace and Microsoft 365, and create the Mail Manager relay action(s) in AWS SES console.

  • If you do not configure the allow-list Nutrition.co’s Google Workspace and Microsoft 365 hosted, it may cause those mailbox providers to reject as spam or send to junk the emails replayed from your Mail Manager environment.

Step 1-a: Follow the instructions to allow-list Mail Manager to relay email to Nutrition.co’s Google Workspace and Microsoft 365 environments.

Step1-b: Create an SMTP relay for your mailbox hosting environment

* See Creating an SMTP relay in the SES console

Figure 3: Screenshot of an SMTP Relay rule action configured for Microsoft 365 Exchange Online inbound receiving

Figure 3: Screenshot of an SMTP Relay rule action configured for Microsoft 365 Exchange Online inbound receiving

Figure 4: Screenshot of an SMTP Relay rule action configured for Google Workspaces Gmail inbound receiving

Figure 4: Screenshot of an SMTP Relay rule action configured for Google Workspaces Gmail inbound receiving

Because Nutrition.co hosts email in both Google Workspace and Microsoft 365, we must create SMTP Relay actions for both.

Step 2: In SES console, verify Nutrition.co’s email domain, which is nutrition.co

SES needs to prove that Nutrition.co owns the domain of each of the recipient addresses before it will begin relaying inbound email. If you cannot verify ownership of the recipient email destinations, SES will not relay messages.

Follow the instructions to verify Nutrition.co’s SES domain identity for the recipient email addresses within Nutrition.co’s Google Workspace and Microsoft 365 environments. (*note that subdomains such as customer-support.nutrition.co inherit verification from the parent domain*).

Figure 5: Screenshot of a successfully verified domain in the SES console.

Figure 5: Screenshot of a successfully verified domain in the SES console.

Step 3: Configure Mail Manager with an Open Ingress Point and Rule Set Action to relay inbound email to the mailbox hosting environment.

Step 3-a: See Create a Traffic Policy to accept inbound email from the internet.

  • Default action: Allow
    (Optional) Add Policy statements, depending on your requirements. Choose the action to be taken when the filter conditions are met: Deny

    • While Nutrition.co does not want to apply additional security via the SMTP Relay gateway, Mail Manager supports both native capabilities and optional add-on subscriptions to 3rd party tools from vetted industry leaders such as Spamhaus and Abusix.
Figure 6: Screenshot of a traffic policy for accepting all email from the internet

Figure 6: Screenshot of a traffic policy for accepting all email from the internet

Step 3-b: Follow the instructions for creating rule sets and rules in the SES console.

  • Select the SMTP Relay that you created in Step 1-b and enable the **Preserve Mail From** option.
    • The ‘Preserve Mail From’ setting is necessary so that the mailbox provider can be configured to make the correct assessment of the message’s SPF policy evaluation, assuming that the allow-list configuration Step 1 is complete.
  • Add any conditions and exceptions for each rule, depending on your needs.
    • You may want to create a condition for the SMTP Relay rule so that only messages destined for recipients within your domain are relayed to the appropriate SMTP Relay action, and choose a different action for the recipients who are not hosted in your environment, such as the Archive action.
    • If you have both Google Workspace and Microsoft 365 configured as SMTP Relay destinations, you may combine the SMTP Relay actions in a single rule if the conditions are the same, or create them as separate rules if the conditions need to be different
Figure 7: A Mail Manager rule configured with an SMTP Relay action for Google Workspaces and another SMTP Relay actions for Microsoft 365

Figure 7: A Mail Manager rule configured with an SMTP Relay action for Google Workspaces and another SMTP Relay actions for Microsoft 365

Step 3-c: Follow the documentation for Creating an Ingress Point.

The Mail Manager Ingress point needs to be ‘Open“ for this use case because internet mail senders need to connect to port 25 and send without SMTP authentication for inbound mail flows.

  • Type: Open
    Traffic policy: Choose the traffic policy that you created step 3-a
    Rule set: Choose the rule set that you created in step 3-b

After saving the ingress endpoint settings, you should see something similar in the console.

Figure 8: Screenshot of an ‘open’ Mail Manager Ingress endpoint configured with a rule set and traffic policy

Figure 8: Screenshot of an ‘open’ Mail Manager Ingress endpoint configured with a rule set and traffic policy

Step 4. Verify your configuration and change your domain’s MX record

Once you have finished configuring Mail Manager with an Inbound Gateway configuration you will have:

  • An Open ingress point that does not require authentication and has an open traffic policy to allow messages from the internet.
  • A Rule set with SMTP Relay actions that will relay inbound messages to Google Workspace and/or Microsoft 365.

Step 4-a: Test your configuration

  • Ingress point: You can test that the Ingress endpoint receives email by using an SMTP capable client application, such as “openssl s_client” from a host that allows for outbound port 25 connections to the A Record of your Open Ingress Point (many ISPs and cloud infrastructure providers block port 25 by default to stop the proliferation of spam on the internet). If you get a “250 OK” response from the SMTP transaction, the Ingress point is configured correctly.
  • Rule set: You can test your Rule set by sending a message to your Ingress endpoint that has a recipient destination that is both a verified domain, and a domain that is hosted by your mailbox environment. You may want to add the Archive and/or Save to S3 rule actions to occur prior to SMTP Relay. This enables you to view message headers and diagnose issues that may occur during the SMTP relay to the mailbox hosting environments.
  • Final delivery: You can test the entire mail flow by looking at the received messages in your mailbox hosting environment.
    • How to look at received messages in a mailbox hosting environment
      • Google Workspace – From within the Gmail interface, find the message and open the message menu options.
      • Figure 9: Screenshot of Gmail’s interface for selecting message options
      • Choose “Show original”.
      • Screenshot of Gmail’s “Original message” summary showing SPF and DKIM passing and aligned with gmail.com, which was the source of the original message
      • Screenshot of the Gmail ‘Show original“ message headers. The Mail From address (also appears as the Return-path header, and envelope-from value in other headers) is preserved within the @gmail.com domain, and Gmail’s assessment of SPF correctly attributed the message as originating from 209.85.216.51 even though the message was relayed through 206.55.129.47. Since the 209.x.x.x address is in the SPF policy for gmail.com, the message passes SPF due to the allow-list configuration
      • (The Screenshot above shows the Gmail ‘Show original“ message headers. The Mail From address (also appears as the Return-path header, and envelope-from value in other headers) is preserved within the @gmail.com domain, and Gmail’s assessment of SPF correctly attributed the message as originating from 209.85.216.51 even though the message was relayed through 206.55.129.47. Since the 209.x.x.x address is in the SPF policy for gmail.com, the message passes SPF due to the allow-list configuration)
      • Microsoft 365 – From within the Outlook on the Web interface, find the message and open the message menu options.
      • Screenshot of Outlook on the Web’s interface for selecting message options
      • Choose “View message details”. You will see the message headers similar to the Gmail example above.

Step 4-b: Change the MX record for your domain.

Note: We recommend using a new subdomain so that you can test this mail flow configuration for a period of time prior to changing the MX record for the primary domain that is actively being used by end users and applications.

Once you have finished testing, you can change the MX record for the domain. The value of the MX record should be the **A Record** of the Open Ingress point along with the priority value.

Figure 13: A screenshot of an MX record configured in Amazon Route 53

Figure 13: A screenshot of an MX record configured in Amazon Route 53

Conclusion

In this blog post, we’ve explored how to leverage SES Mail Manager’s SMTP Relay action to simplify the handling of inbound email for organizations that use a mix of email hosting environments, specifically Google Workspace and Microsoft 365. By configuring Mail Manager as an inbound SMTP gateway, our fictitious customer, Nutrition.co was able to centralize the management of their email flows, enhance security through features like traffic policies and rule sets, and ensure compliance through flexible archiving.

The key steps involved setting up allow-listing in the Google Workspace and Microsoft 365 environments, creating SMTP relay configurations in Mail Manager, and updating Nutrition.co domain’s MX record to point to the Mail Manager ingress endpoint. This allowed Nutrition.co to seamlessly route inbound emails destined for both their cloud-hosted employee mailboxes and on-premises applications, processing and archiving the messages before final delivery.

The flexibility of Mail Manager’s SMTP Relay action makes it a powerful tool for organizations looking to unify their email infrastructure, especially in hybrid environments. By acting as a centralized ingress and egress gateway, Mail Manager can help streamline email management, improve security, and unlock new cloud-enabled email use cases. As email continues to be a critical communication channel, solutions like Mail Manager will become increasingly important for businesses looking to maximize the value of their email ecosystem.

Please visit AWS Re:Post to ask and find answers to questions about SES Mail Manager. Talk with your AWS account team if you are interested in exploring Mail Manager in more depth.

Additional blogs related to Mail Manager:

About the Authors

Jesse Thompson
Jesse Thompson is an Email Deliverability Manager with the Amazon Simple Email Service team. His background is in enterprise development and operations, with a focus on email abuse mitigation and encouragement of authenticity practices with open standard protocols. Jesse’s favorite activity outside of technology is recreational curling.
Alexey Kurbatsky

Alexey Kurbatsky

Alexey is a Senior Software Development Engineer at AWS, specializing in building distributed and scalable services. Outside of work, he enjoys exploring nature thru hiking as well as playing guitar.

Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

Email Archiving with Mail Manager: Why To Archive In Transit vs At The Mailbox

Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/email-archiving-with-mail-manager-why-to-archive-in-transit-vs-at-the-mailbox/

When designing Amazon Simple Email Service’s (SES) Mail Manager, we often heard from customers about the “PST-file problem” inherent with user-side mailbox-based archiving. This occurs when, for a variety of reasons, end users decide to archive their emails to local PST files or other local storage. These PST files are fragile and easily corrupted. Furthermore, they are subject to the backup practices of individual workstations. Lastly, PST files are readily are portable and can be easily copied and moved outside the visibility of the email system and your IT and IP controls.

We developed Amazon Simple Email Service (SES) Mail Manager archiving features in response to this problem, and based on additional customer feedback: the need for consistent email retention behaviors, for all email. Customers also wanted the flexibility to determine which messages to archive, where to put them, and how long to retain those messages.

To make the feature applicable to the widest set of use cases, we designed Mail Manager to be able to archive any email traversing the SES service, not just those that have already been delivered to a user’s mailbox. This added flexibility ensures organizations can maintain a complete record of exactly those email communications they wish to preserve. Rather than require external tools to search and export Mail Manager’s archives, we built these functions directly into the SES console.

In fact, the entire Media Manager archiving solution is fully managed by SES within the customer’s Mail Manager account, reducing the operational overhead traditionally associated with email archiving and compliance.

Figure 1 - Mail Manager Archiving

Figure 1 – Mail Manager Archiving

At the core of the SES Mail Manager archiving solution is the ability to capture and retain any message, regardless of its source or destination, as it flows through the service’s rules engine. This design approach ensures that every email message traversing Mail Manager can be subject to archiving and retention policies, rather than requiring organizations to manage different systems and tools for mail flowing through mail servers, internal relays and other email infrastructure. The result is a unified, comprehensive compliance solution that provides visibility and control over an organization’s email archiving.

SES also published a detailed overview of the Archiving feature, which is available here: Archiving and sending to final SMTP server.

Archiving on its own isn’t an innovation; it’s an email primitive – an essential capability that can be used to enable other, more complex solutions. Historically, retention of email was configured as a function of your on-premises mail server, where your mailboxes themselves were resident. Personally-authored emails were considered the high-value material to retain, and adding archiving as a function of mailbox configurations was the simplest approach.

In practice, we find that the mail captured at the mailbox server, or end user’s inbox, represents only a fraction of of the mail a typical enterprise generates. As organizations grow, the number of applications generating Application To Person (A2P) messages tends to increase dramatically. Similarly, as corporate environments become more complex, SaaS-based solutions that are external to the primary email infrastructure often use email to update employees along with workflow-management systems. Much of that mail eludes archiving as it bypasses individual user mailboxes.

The SES strategy for archiving is to capture mail from anywhere, to anywhere, as long as it transits an ingress endpoint as part of your Mail Manager configuration. You have two choices: you can write those messages directly to an S3 bucket you control, and then ingest it into any other tool you like. Alternately, you can send messages into a managed archive within Mail Manager, and gain access to search, export, and configurable retention features. By default, SES configures retention for 6 months, but it’s adjustable up to permanent retention for customers who require it.

Mail Manager’s archiving feature captures any message which matches your rule, or all messages traversing any ingress endpoint. You can choose to write all messages to or from your senior leadership team into one archive, or you can organize by other envelope metadata. The rules operate the same way whether the message is A2P or Person to Person (P2P), ensuring uniform policies and retention options.

With Mail Manager’s managed archives, you pay for each gigabyte ingested, indexed, and available for search, and a separate storage fee for each gigabyte retained every month. Note that the storage fee includes both the raw content of the messages, and the size of the computed index required for search and export functions.

For messages you write to your S3 buckets, you also have the option to invoke an S3 trigger action that calls an Amazon Lambda to drive various automatation workflows. Regulated industries might want to write all messages to S3 to leverage S3’s glacier storage option for very long-term storage.

You can even split your workload between Mail Manager’s managed archive, for emails you are likely to need readily discoverable, and the Write to S3 option, for content which you don’t expect to ever need to search with granularity, but still needs to be archived to “check the box” for your retention policy. In fact, AWS encourages such a builder-oriented approach, because it rewards thoughtful decisions and resource utilization, and conforms to the broad goal of consumption-based pricing, which Mail Manager embraces fully at every step.

Figure 2 - Rule Set with conditions for archiving

Figure 2 – Rule Set with conditions for archivingMail Manager provides a more comprehensive, resilient archiving approach that increases both the overall scope of mail that can be captured, and the fidelity of the archived data. You don’t need any special adapters or plugins to capture mail from any source. All email that comes through your Mail Manager Ingress Endpoint can be archived.

Figure 3 - Create archive

Figure 3 – Create archive

Why not try Mail Manager today and experience the benefits of a centralized, scalable email archiving solution? You’ll pay only for the data you ingest and retain each month, without the fragility and visibility issues of user-managed archives. Visit the SES website to start your free trial of Mail Manager and take control of your organization’s critical email records. To start with Mail Manager, visit https://aws.amazon.com/ses/, click on Mail Manager, and set up your first workload today.

If you have any questions or need further guidance, feel free to reach out to us via the SES Forums or in the comments section of this blog post. We’re here to help you navigate the evolving email landscape and unlock the full potential of your Amazon SES investment.

About the Authors

Toby Weir-Jones

Toby Weir-Jones

Toby is a Principal Product Manager for Amazon SES and WorkMail. He joined AWS in January 2021 and has significant experience in both business and consumer information security products and services. His focus on email solutions at SES is all about tackling a product that everyone uses and finding ways to bring innovation and improved performance to one of the most ubiquitous IT tools.

Brett Ezell

Brett Ezell

Brett is an Amazon Pinpoint and Amazon Simple Email Service Specialist Solutions Architect at AWS. As a Navy veteran, he joined AWS in 2020 through an AWS technical military apprenticeship program. When he isn’t deep diving into solutions for customer challenges, Brett spends his time collecting vinyl, attending live music, and training at the gym. An admitted comic book nerd, he feeds his addiction every Wednesday by combing through his local shop for new books.

Zip

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.

Lower Your Risk of SMS Fraud with Country Level Blocking and Amazon Pinpoint

Post Syndicated from Brett Ezell original https://aws.amazon.com/blogs/messaging-and-targeting/lower-your-risk-of-sms-fraud-with-country-level-blocking-and-amazon-pinpoint/

As a developer, marketing professional, or someone in the communications space, you’re likely familiar with the importance of SMS messaging in engaging customers and driving valuable interactions. However, you may have also encountered the growing challenge of artificially inflated traffic (AIT), also known as SMS pumping. A new report co-authored by
Enea revealed that AIT is so widespread within the SMS ecosystem that 19.8B-35.7B fraudulent messages were sent by bad actors in 2023, incurring substantial costs of over $1 billion. In this blog post, we’ll explore how you can use
Protect configurations, a powerful set of capabilities within
Amazon Pinpoint SMS, that provides granular control over which destination countries your SMS, MMS, and voice messages can be sent to.
<img decoding=” width=”1252″ height=”889″>

What is SMS Pumping, aka Artificially Inflated Traffic (AIT)?

AIT, or SMS pumping, is a type of fraud where bad actors use bots to generate large volumes of fake SMS traffic. These bots target businesses’ whose websites, apps, and other assets have forms or other mechanisms that trigger SMS being sent out. Common use cases for SMS such as one-time passwords (OTPs), app download links, promotion signups, etc. are all targets for these bad actors to “pump” SMS and send out fraudulent messages. The goal is to artificially inflate the number of SMS messages a business sends, resulting in increased costs and a negative impact on the sender’s reputation. In the realm of SMS-based artificially inflated traffic (AIT), the prevalent method for bad actors to profit involves colluding with parties in the SMS delivery chain to receive a portion of the revenue generated from each fraudulent message sent.

<img decoding=” width=”1280″ height=”720″>

AIT poses several challenges for businesses:

  1. Overspending: The fake SMS traffic generated by AIT bots results in businesses paying for messages that yield no actual results.

  2. Interrupted service: Large volumes of AIT can force businesses to temporarily halt SMS services, disrupting legitimate customer communications.

  3. Diverted focus: Dealing with AIT can shift businesses’ attention away from core operations and priorities.

  4. Reduced deliverability rates due to the messages never being interacted with and/or large volumes of SMS being sent quickly.

How does Protect mitigate AIT?

Amazon Pinpoint’s Protect feature allows you to control which countries you can send messages to. This is beneficial if your customers are located in specific countries.

With Protect, you can create a list of country rules that either allow or block messages to each destination country. These country rules can be applied to SMS, MMS, and voice messages sent from your AWS account. The Protect configurations you create enable precise control over which destination countries your messages can be sent to. This helps mitigate the impact of AIT by allowing you to tailor where you do or do not send.

Protect offers flexibility in how the country rules are applied. You can apply them at the account level, the configuration set level, or the individual message level. This enables you to customize your AIT mitigation strategy to best fit your business needs and messaging patterns.

By leveraging Protect within Amazon Pinpoint, you can help ensure the integrity and cost-effectiveness of your SMS, MMS, and voice communications.

Account-level Protect Configuration

The simplest approach is to create a Protect configuration and associate it as the account default. This means the allow/block rules defined in that configuration will be applied across all messages sent from your account, unless overridden. This is an effective option if you only need one set of country rules applied universally.

Configuration set-specific Protect configuration

You can associate a Protect configuration with one or more of your Pinpoint SMS configuration sets. This allows you to apply different country rules to distinct messaging flows or use cases within your application without changing your existing code if you are already using Config Sets. It also enables more detailed logging and monitoring of the Protect configuration’s impact, such as:

  1. Error Logs: Logging of any errors or issues encountered when messages are sent, providing insights into how the Protect configuration is affecting message delivery.
  2. Audit Logs: Records of all configuration changes, access attempts, and other relevant activities related to the Protect configuration, allowing for comprehensive auditing and monitoring.
  3. Usage Metrics: Tracking of usage statistics, such as the number of messages sent to different countries, the impact of the Protect configuration on message volumes, and other usage-related data.
  4. Compliance and Policy Enforcement Logs: Documentation of how the Protect configuration is enforcing compliance with messaging policies and regulations, including any instances where messages are blocked or allowed based on the configuration rules.

Dynamic Protect configuration per message

If your needs are even more specific, you can create a Protect configuration without any association, and then specify its ID when sending messages via the Pinpoint APIs (e.g. SendMediaMessage, SendTextMessage, SendVoiceMessage). This gives you the ability to dynamically choose the Protect configuration to apply for each individual message, providing the ultimate flexibility.

Regardless of the approach, the core benefit of Protect configurations is the ability to precisely control which destination countries your messages may be sent to. Blocking countries where you don’t have a presence or where SMS pricing is high eliminates your exposure to fraudulent AIT traffic originating from those regions. This helps protect your messaging budget, maintain service continuity, and focus your efforts on legitimate customer interactions.

Who should use Protect configurations?

Protect configurations are designed to benefit a wide range of AWS customers, particularly those who:

  1. Send SMS messages to a limited number of countries: If your business primarily operates in a few specific countries, Protect configurations can help you easily block SMS messages to countries where you don’t have a presence, reducing the risk of AIT.
  2. Have experienced AIT issues in the past: If your business has been a target of SMS pumping, Protect configurations can help you regain control over your SMS communications and prevent future AIT attacks.
  3. Want to proactively protect their SMS messaging: Even if you haven’t encountered AIT issues yet, Protect configurations can help you stay ahead of the curve and maintain the integrity of your SMS communications.

How to create a country rules list with Protect configuration

To begin building a list of country rules that allow or block messages to specific destination countries, you start by creating a new Protect configuration. There are two ways to accomplish this, either by using the console, or the API.

Option 1 – Using the AWS Console

Console Scenario: My account is out of the sandbox and I only want to send to 1 country – United Kingdom (iso:GB) using the SenderID “DEMOTP”.

At a high level, we will follow the three steps outlined below for each method. In our examples, we used a SenderID as our Originator. However, it should be noted that the same process can be achieved using any originator you’d like. i.e. SenderID, Phone pool, Phone number, 10DLC, short code, etc.

  1. Create SenderID (Optional if you already have one)
  2. Create Protect Configuration
  3. Send Test Message via console

Using the AWS Console

1) Create SenderID for United Kingdom (GB)

  • Pinpoint SMS Console – Request Originator
    • Select United Kingdom (GB) and follow the prompts for a SenderID. DO NOT select Two-way SMS Messaging
    • Enter Sender ID – Example: DEMOTP
    • Confirm and Request

2) Create default Protect Configuration

<img decoding=” width=”863″ height=”521″>

    • Search for Country=United Kingdom then deselect United Kingdom

<img decoding=” width=”865″ height=”582″>

    • Set as Account Default and select Create protect configuration

<img decoding=” width=”1497″ height=”1173″>

3) Send a test message with SMS simulator

Note: The Pinpoint SMS Simulator provides special phone numbers you can use to send test text messages and receive realistic event records, all within the confines of the Amazon Pinpoint service. These simulator phone numbers are designed to stay entirely within the Pinpoint SMS ecosystem, ensuring your test messages don’t get sent over the carrier network.

You can use these simulator phone numbers to send both SMS and MMS messages, allowing you to thoroughly validate your message content, workflow, and event handling. The responses you receive back will mimic either success or fail depending on which destination simulator number you send to.

  • From the Pinpoint SMS Console SMS Simulator page,
    • For Originator, Choose Sender ID, and select your Sender ID created from earlier.
    • Under Destination number, select Simulator numbers and choose United Kingdom (GB). Enter a test message in the Message body.
    • Finally, choose send test message. This should prompt a green “Success” banner at the top of your page.

<img decoding=” width=”1336″ height=”1313″>

    • Conversely, follow the previous test message steps, and instead attempt to send to anywhere other than the United Kingdom (GB). In this example, Australia (AU) 
    • As shown below in the screenshot this one is blocked since you have configured to only send to GB.

<img decoding=” width=”1333″ height=”1364″>

Option 2 – Using the V2 API and CLI

V2 API Scenario: 
My account is out of the sandbox and I want to BLOCK only 1 country – Australia (AU) while using the SenderID “DEMOTP”.

1) Create SenderID for GB

Note: before using the CLI remember to configure your access and secret key using

aws configure

Windows users should use PowerShell over cmd to test

  • Use RequestSenderId to create the same Sender Id as previously made via the console.
aws pinpoint-sms-voice-v2 request-sender-id --iso-country-code "GB" --sender-id "DEMOTP"

Response:

{
   "DeletionProtectionEnabled": False,
   "IsoCountryCode": "GB",
   "MessageTypes": [ "TRANSACTIONAL" ],
   "MonthlyLeasingPrice": "0.00",
   "Registered": False,
   "SenderId": "DEMOTP",
   "SenderIdArn": "string"
}

2) Create default Protect Configuration

aws pinpoint-sms-voice-v2 create-protect-configuration --tags Key=Name,Value=CLITESTING

Response:

{
   "AccountDefault": False,
   "CreatedTimestamp": number,
   "DeletionProtectionEnabled": False,
   "ProtectConfigurationArn": "string",
   "ProtectConfigurationId":  "string",
   "Tags": [ 
      { 
         "Key": "Name",
         "Value": "CLITESTING"
      }
   ]
}

  • Add AU as BLOCKED on protect configuration.
aws pinpoint-sms-voice-v2 update-protect-configuration-country-rule-set --protect-configuration-id protect-string --number-capability 'SMS' --country-rule-set-updates '{\"AU\":{\"ProtectStatus\":\"BLOCK\"}}'

Response:

{
   "CountryRuleSet": { 
      "string" : { 
         "ProtectStatus": "ALLOW | BLOCK"
      }
   },
   "NumberCapability": "SMS",
   "ProtectConfigurationArn": "string",
   "ProtectConfigurationId": "string"
}

  • Set as account default configuration.
aws pinpoint-sms-voice-v2 set-account-default-protect-configuration --protect-configuration-id protect-string

Response:

{
   "DefaultProtectConfigurationArn": "string",
   "DefaultProtectConfigurationId": "string"
}

3) Send test message

  • Use SendTextMessage to test your Protect Configuration via the V2 API.
  • Test sending to GB Simulator Number should be successful.
aws pinpoint-sms-voice-v2 send-text-message --destination-phone-number "string" --message-body "string"

Response:

{
   "MessageId": "string"
}

  • Test sending to AU Simulator Number should be blocked.
aws pinpoint-sms-voice-v2 send-text-message --destination-phone-number "string" --message-body "string"

Response – (ConflictException):

{
An error occurred (ConflictException) when calling the 
SendTextMessage operation: Conflict Occurred - 
Reason="DESTINATION_COUNTRY_BLOCKED_BY_PROTECT_CONFIGURATION" ResourceType="protect-configuration" ResourceId="string"
}

Conclusion

As SMS messaging continues to play a crucial role in customer engagement and authentication, protecting your communications from AIT is more important than ever. Amazon Pinpoint Protect provides a powerful and user-friendly solution to help you mitigate the impact of SMS pumping, ensuring the integrity of your SMS channels and preserving your business’ reputation and resources. Whether you’re a small business or a large enterprise, Pinpoint Protect is a valuable tool to have in your arsenal as you navigate the evolving landscape of SMS messaging.

To get started with Pinpoint SMS Protect, visit the Amazon Pinpoint SMS documentation or reach out to your AWS account team. And don’t forget to let us know in the comments how Protect configurations has helped you combat AIT and strengthen your SMS communications.

A few resources to help you plan for your SMS program:

About the Author

Brett Ezell

Brett Ezell is your friendly neighborhood Solutions Architect at AWS, where he specializes in helping customers optimize their SMS and email campaigns using Amazon Pinpoint and Amazon Simple Email Service. As a former US Navy veteran, Brett brings a unique perspective to his work, ensuring customers receive tailored solutions to meet their needs. In his free time, Brett enjoys live music, collecting vinyl, and the challenges of a good workout. And, as a self-proclaimed comic book aficionado, he can often be found combing through his local shop for new books to add to his collection.