Post Syndicated from singhzwz original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-investigate-what-happened-to-the-email-that-was-sent-via-ses-but-was-never-received-in-recipient-inbox/

How to investigate what happened to the email that was sent via SES but was never received in recipient inbox

Amazon Simple Email Service (SES) is an email platform that provides an easy, cost-effective way for you to send and receive email using your own email addresses and domains.

You can use SES for two major use cases. The first use case is marketing emails, which can include things such as special offers, newsletters, and product updates. The second use case is transactional emails, which can include things such as order confirmation, One Time Password (OTP), and registration confirmation.

One common issue we hear from our customers is that emails are not delivered to the recipient’s inbox. There are three areas in the flow where this could happen:

1. Emails were not sent successfully by your email application
2. Emails get dropped within SES after being received from your email application
3. Emails are dropped after leaving SES

In this blog post, we will explore each of these three scenarios.

Prerequisites:

To investigate each scenario, you need the following:

• Understand How email sending works in Amazon SES
• Enable verbose logging on your email application
• Amazon SES – Set up notifications for bounces and complaints

Scenario 1: Emails were not sent successfully by your email application

The first step in the process is for your application to contact SES to pass it the message. This is one place where problems could occur.
Remember that for each message it successfully enqueues, SES returns a message ID that you can log on your side. An example message ID looks like 01000174f87e0fb4-81b97243-56b0-4cd1-90c7-d6987a00fdc1-000000.

Here is an example of Sendmail logs for successfully relaying the email to SES.

Jun 22 15:29:42 ip-172-12-12-12.ec2.internal sendmail[323257]: 35ABCDEF123123: to=<[email protected]>, delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=30123, relay=email-smtp.us-east-1.amazonaws.com. [3.95.123.123], dsn=2.0.0, stat=Sent (Ok 01000174f87e0fb4-81b97243-56b0-4cd1-90c7-d6987a00fdc1-000000)

To verify that your requests are successful, you can look for the message IDs. If SES never accepted your message, you can be sure it won’t be delivered! If you are able to log a message ID, it means that your email safely arrived at SES.

Actions to troubleshoot client side issue:

1. In order to test that the problem lies between your application and SES, you can try to send a test email from the SES console to an email address and domain where you are experiencing the issue. If that test email arrives on time, it’s a pretty good indicator that the error is occurring before your email even reaches SES.
2. Examine the logs at application side to confirm if the application has successfully passed the email to SES. A success response looks like the following example:

250 Ok 0000012345678e09-123a4cdc-b56c-78dd-b90e-d123be456789-000000

Some common errors include the following:

* Email address is not verified. The following identities failed the check in region region: identity1, identity2, identity3
* Account is paused
* Daily message quota exceeded
* Maximum sending rate exceeded
* Maximum SigV2 SMTP sending rate exceeded

Another common type of error is a connection timeout error. Here is an example of a Java exception that shows a connection timeout error to an SES endpoint:

Caused: com.sun.mail.util.MailConnectException: Couldn't connect to host, port: email-smtp.ap-southeast-2.amazonaws.com , 465; timeout 60000; nested exception is: java.net .ConnectException: Connection timed out (Connection timed out) at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2209 ) at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:740 )

The way that you observe this error message depends on the way that you call SES.

If you call the SES API directly, the action to interact with SES will return an error like MessageRejected or one of the errors specified in the Common Errors topic of the Amazon Simple Email Service API Reference.

If you call SES through its SMTP interface, the way that you experience the error depends on the application. Some applications might display a specific error message, and others might not. For a list of SMTP response codes that SES returns, see SMTP response codes returned by Amazon SES.

Scenario 2: Emails get dropped within SES after being received from your email application

Once SES accepts the message and provides a message ID, the next step is to process it and hand it over to the recipient ISP or the recipient mailbox provider. However, in the following scenarios SES can drop the email after receiving it:

• Rendering Failure: This can happen when you use an email template to send email. Upon accepting the email from your email application, SES validates that the template data you send includes the required variables in the template. If the template data contains non-compliant variables or is missing variables, SES is unable to construct the email and drops the email. This is called a Rendering Failure.
• Suppression list: If a recipient is on an SES suppression list, SES will drop the email to protect the sender reputation. There are two types of suppression lists: Account-level suppression list and global suppression list. To learn more about the two types of suppression lists, please refer to Managing lists and subscriptions in Amazon Simple Email Service.
•  Reject Event: SES accepted the email, but determined that it contained a virus and didn’t attempt to deliver it to the recipient’s mail server. SES drops the email and generates a Reject event.

Actions to troubleshoot such issues:

To identify the root cause of the three preceding scenarios, you need to set up Amazon SES event publishing. Please refer to the prerequisite section to set this up.

Scenario 1: Rendering Failure

Here is an example event publishing record for Rendering Failure

{
"eventType":"Rendering Failure",
 "mail":{
     "timestamp":"2018-01-22T18:43:06.197Z",
     "source":"[email protected]",
     "sourceArn":"arn:aws:ses:us-east-1:123456789012:identity/[email protected]",
     "sendingAccountId":"123456789012",
     "messageId":"EXAMPLE7c191be45-e9aedb9a-02f9-4d12-a87d-dd0099a07f8a-000000",
     "destination":[
         "[email protected]"
        ],
      "headersTruncated":false,
      "tags":{
         "ses:configuration-set":[
         "ConfigSet"
        ]
    }
  },
"failure":{
        "errorMessage":"Attribute 'attributeName' is not present in the rendering data.",
        "templateName":"MyTemplate"
    }
}

In this example, the error message tells you that there is a missing variable attributeName in the template data. To learn more about troubleshooting rendering failure, please refer to Why are emails failing to deliver when I send Amazon SES emails using the SendTemplatedEmail operation?

Scenario 2: Suppression list

SES drops the email if the recipient is on an account-level suppression list or a global suppression list. The following is an example error message for recipient being on a global suppression list:

"diagnosticCode": "Amazon SES has suppressed sending to this address because it has a recent history of bouncing as an invalid address. For more information about how to remove an address from the suppression list, see the Amazon SES Developer Guide: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/remove-from-suppressionlist.html"

The example demonstrates that the email get bounced because the recipient is on the global suppression list. We recommend enabling SES account-level suppression list because the account-level suppression list supersedes the global suppression list.

If the recipient is on your account-level suppression list, you can remove individual email addresses from your Amazon SES account-level suppression list. The following is an example error message for recipient being on a account-level suppression list:

"diagnosticCode": "Amazon SES did not send the message to this address because it is on the suppression list for your account. For more information about removing addresses from the suppression list, see the Amazon SES Developer Guide at https://docs.aws.amazon.com/ses/latest/DeveloperGuide/sending-email-suppression-list.html"

Note: The suppression list feature is in place to avoid emails being sent to recipients who are known to produce bounce or complaint for your emails in past. Enabling this feature protects your reputation as a sender. Only remove the recipient when you are assured that recipients in concern will no longer produce bounce or complaint for your emails.

Scenario 3: Reject Event

SES rejects the email when it scans the email and determines that it contains a virus. The following is an example of a Reject event publishing record.

{
"eventType": "Reject",
"mail": {
      "timestamp": "2016-10-14T17:38:15.211Z",
      "source": "[email protected]",
      "sourceArn": "arn:aws:ses:us-east-1:123456789012:identity/[email protected]",
      "sendingAccountId": "123456789012",
      "messageId": "EXAMPLE7c191be45-e9aedb9a-02f9-4d12-a87d-dd0099a07f8a-000000",
      "destination": [
          "[email protected]"
],
      "headersTruncated": false,
      "headers": [{
                  "name": "From",
                  "value": "[email protected]"
                  },
                  {
                  "name": "To",
                  "value": "[email protected]"
                  },
                  {
                  "name": "Subject",
                  "value": "Message sent from Amazon SES"
                  },
                 {
                  "name": "MIME-Version",
                  "value": "1.0"
                 },
                 {
                  "name": "Content-Type",
                  "value": "multipart/mixed; boundary=\"qMm9M+Fa2AknHoGS\""
                 },
                  {
                   "name": "X-SES-MESSAGE-TAGS",
                   "value": "myCustomTag1=myCustomTagValue1, myCustomTag2=myCustomTagValue2"
                  }
],
      "commonHeaders": {
           "from": [
              "[email protected]"
                  ],
           "to": [
              "[email protected]"
],
           "messageId": "EXAMPLE7c191be45-e9aedb9a-02f9-4d12-a87d-dd0099a07f8a-000000",
           "subject": "Message sent from Amazon SES"
},
      "tags": {
           "  ses:configuration-set": [
             "ConfigSet"
              ],
               "ses:source-ip": [
               "192.0.2.0"
               ],
                "ses:from-domain": [
                "example.com"
               ],
                "ses:caller-identity": [
                "ses_user"
               ],
                 "myCustomTag1": [
                 "myCustomTagValue1"
              ],
              "myCustomTag2": [
              "myCustomTagValue2"
]
}
},
     "reject": {
            "reason": "Bad content"
}
}

To fix this issue, please examine the email and make sure there is no virus in the email. You can test your email content by sending a test email to Amazon SES mailbox simulator.

Scenario 3: Emails are dropped after leaving SES
We sometimes encounter this when we successfully deliver the email to a recipient ISP, only to have the recipient ISP hold back the email and not deliver to the recipient’s inbox. This can happen for a variety of reasons, for example the recipient ISP’s anti-spam filters, or because the sender has not yet built sufficient reputation due to it being a new domain or IP and the recipient does not yet trust the sender.

Actions to troubleshoot such issues:

To identify the root cause of the issue, you need to set up Amazon SNS notification. Please refer to the prerequisite section to set this up.

• When the recipient ISP accepts the email, it returns a 250 ok status code to SES. SES includes the detailed status code and response message from the recipient ISP in the SNS notification.The following is an example SNS notification for a success delivery event.

{
"notificationType":"Delivery",
"mail":{
     "timestamp":"2016-01-27T14:59:38.237Z",
     "messageId":"0000014644fe5ef6-9a483358-9170-4cb4-a269-f5dcdf415321-000000",
     "source":"[email protected]",
     "sourceArn": "arn:aws:ses:us-east-1:888888888888:identity/example.com",
     "sourceIp": "127.0.3.0",
     "sendingAccountId":"123456789012",
     "callerIdentity": "IAM_user_or_role_name",
     "destination":[
        "[email protected]"
     ],
"headersTruncated":false,
"headers":[
               {
                "name":"From",
                "value":"\"John Doe\" <[email protected]>"
               },
              {
                "name":"To",
                "value":"\"Jane Doe\" <[email protected]>"
              },
              {
                "name":"Message-ID",
                "value":"custom-message-ID"
              },
              {
                "name":"Subject",
                "value":"Hello"
              },
              {
                  "name":"Content-Type",
                  "value":"text/plain; charset=\"UTF-8\""
              },
              {
                  "name":"Content-Transfer-Encoding",
                  "value":"base64"
              },
              {
                  "name":"Date",
                  "value":"Wed, 27 Jan 2016 14:58:45 +0000"
              }
  ],
"commonHeaders":{
"from":[
     "John Doe <[email protected]>"
],
"date":"Wed, 27 Jan 2016 14:58:45 +0000",
"to":[
     "Jane Doe <[email protected]>"
],
    "messageId":"custom-message-ID",
    "subject":"Hello"
}
},
"delivery":{
    "timestamp":"2016-01-27T14:59:38.237Z",
    "recipients":["[email protected]"],
    "processingTimeMillis":546,
    "reportingMTA":"a8-70.smtp-out.amazonses.com",
    "smtpResponse":"250 ok: Message 64111812 accepted",
    "remoteMtaIp":"127.0.2.0"
}
}

In this example, the SMTP response is "250 ok:Message 64111812 accepted", this confirms the recipient ISP accepted the email. At this point, the email is with the recipient ISP and SES no longer has control over the email.

ISPs use different mechanisms and algorithms to filter emails to place them in either the recipient’s inbox folder or spam folder. In such case, you will need to reach out to the recipient ISP to understand why they accepted the email but did not deliver it to the recipient inbox. Since their servers decide the placement of email, they can give a clear explanation of the cause of the issue and steps to remediate the issue.

The recipient ISP may reject the email. When this happens, the recipient ISP returns a 4xx or 5xx error code to SES. This is also called a bounce event.

There are two kinds of bounce events: Soft bounce and hard bounce. Soft bounce refers to a temporary email delivery failure, this kind of error usually can be retried. Hard bounce refers to a permanent delivery failure, and it cannot be retried. The Simple Mail Transfer Protocol (SMTP) Enhanced Status Codes Registry lists the possible error codes and reasons for email delivery failure. Generally speaking, 4xx error codes are soft bounces, 5xx error codes are hard bounces.

When the recipient ISP bounces the email, SES includes the detailed status code and response message from the recipient ISP in the SNS notification.

The following is an example SNS notification for a bounce event.

{
"notificationType":"Bounce",
"bounce":{
      "bounceType":"Permanent",
      "reportingMTA":"dns; email.example.com",
       "bouncedRecipients":[
          {
             "emailAddress":"[email protected]",
             "status":"5.1.1",
             "action":"failed",
             "diagnosticCode":"smtp; 550 5.1.1 <[email protected]> User not found"
          }
     ],
      "bounceSubType":"General",
      "timestamp":"2016-01-27T14:59:38.237Z",
       "feedbackId":"00000138111222aa-33322211-cccc-cccc-cccc-ddddaaaa068a-000000",
       "remoteMtaIp":"127.0.2.0"
},
"mail":{
      "timestamp":"2016-01-27T14:59:38.237Z",
      "source":"[email protected]",
      "sourceArn": "arn:aws:ses:us-east-1:888888888888:identity/example.com",
      "sourceIp": "127.0.3.0",
      "sendingAccountId":"123456789012",
      "callerIdentity": "IAM_user_or_role_name",
      "messageId":"00000138111222aa-33322211-cccc-cccc-cccc-ddddaaaa0680-000000",
      "destination":[
             "[email protected]",
             "[email protected]",
             "[email protected]"],
      "headersTruncated":false,
      "headers":[
            {
              "name":"From",
              "value":"\"John Doe\" <[email protected]>"
            },
            {
              "name":"To",
              "value":"\"Jane Doe\" <[email protected]>, \"Mary Doe\" <[email protected]>, \"Richard Doe\" <[email protected]>"
            },
            {
              "name":"Message-ID",
              "value":"custom-message-ID"
            },
            {
              "name":"Subject",
              "value":"Hello"
            },
            {
              "name":"Content-Type",
              "value":"text/plain; charset=\"UTF-8\""
            },
            {
              "name":"Content-Transfer-Encoding",
              "value":"base64"
            },
            {
              "name":"Date",
              "value":"Wed, 27 Jan 2016 14:05:45 +0000"
             }
],
"commonHeaders":{
     "from":[
        "John Doe <[email protected]>"
                  ],
     "date":"Wed, 27 Jan 2016 14:05:45 +0000",
      "to":[
      "Jane Doe <[email protected]>, Mary Doe <[email protected]>, Richard Doe <[email protected]>"
        ],
     "messageId":"custom-message-ID",
      "subject":"Hello"
}
}
}

In the preceding example, the recipient ISP permanently bounced the email. This is concluded from the "bounceType":"Permanent" field. Additionally, the recipient ISP provides the reason they bounced the email in the “diagnosticCode” field, and the reason is "smtp; 550 5.1.1 <[email protected]> User not found". In this case, [email protected] isn’t a valid inbox, the error can’t be resolved by retrying the request. The sender should check the spelling of the email address and confirm with the recipient that this address has a valid inbox before retry sending to this address again. You can also contact the recipient ISP to understand the reasons of the error and steps to remediate the issue when the information provided by the "diagnosticCode" isn’t sufficient or clear.

• Another event that can happen is “delivery delay”. Such event happen when the email couldn’t be delivered to the recipient’s mail server because a temporary issue occurred. Delivery delays can occur, for example, when the recipient’s inbox is full, or when the receiving email server experiences a transient issue. These are usually soft bounces in which case SES is programmed to retry such deliveries several times over the next 10 hours. If you have enabled SES event publishing for this event you can review the “delivery delay” object section of SES event record to understand the cause of delay and take corrective actions accordingly. As an example, if you reach out to the recipient ISP and the recipient ISP informs you about maintenance on their side that is causing the issue then you can refrain sending to that ISP till the maintenance completes.

Conclusion:
Now that you know the 3 areas in the email sending flow where errors can occur you can begin to troubleshoot whether your problems are at your email application side, within SES, or at the recipient ISP. Make sure that you have proper logging and that you are receiving the events that provide you the detail you need to determine where your errors are occurring. Errors happening prior to the email being sent out from SES are usually solvable, while challenges in deliverability once an email has been sent from SES require more due diligence and experimentation in your content to determine why your emails are being delivered to spam, or not at all.

Feel free to post any comments or questions for us in the AWS Forum.

References:
[1]https://docs.aws.amazon.com/ses/latest/dg/send-email-concepts-process.html
[2]https://docs.aws.amazon.com/ses/latest/dg/monitor-sending-activity.html
[3]https://docs.aws.amazon.com/ses/latest/dg/monitor-using-event-publishing.html#event-publishing-how-works