Post Syndicated from Zip Zieper original https://aws.amazon.com/blogs/messaging-and-targeting/automate-the-creation-rotation-of-amazon-simple-email-service-smtp-credentials/

[Amazon Simple Email Service] provides a secure email solution that scales with your business needs. Unfortunately, all email systems, including Amazon SES, remain the primary target for spammers and bad actors due to email’s widespread use and accessibility.

While SES offers powerful features for application-based email sending, its SMTP credentials require careful management to prevent unauthorized access. Compromised credentials enable bad actors to send malicious emails through legitimate domains, which can bypass security filters and damage sender reputation.

To protect your SES implementation, you must encrypt SMTP credentials during storage and transmission. Additionally, implementing role-based access controls helps restrict credential access to authorized personnel only. Regular credential rotation at fixed intervals, typically every 90 days, minimizes potential security breaches. Automating this rotation process eliminates human error and ensures consistent security practices across your organization.

Problem Statement

Imagine you are the administrator for a large financial institution. You recently began using Amazon SES to send email from two dozen on-premises servers. Your email servers authenticate with SES using SMTP credentials to access the SES SMTP interface. Your organization’s security policies mandate regular credential rotation, including the ability to rotate them on-demand. How can you automate SMTP credential rotation such that you can meet your organization’s security policies?

This blog post will present two solutions that automate the secure management and automatic rotation of SMTP credentials for Amazon SES. Each will help enhance email security, comply with regulations, and minimize operational overhead.

• Option 1: A fully automated version of the solution that uses AWS Lambda, AWS Secrets Manager, and AWS Systems Manager.
• Option 2: A partially automated solution using AWS Lambda, AWS Systems Manager Parameter Store, and AWS Step Functions.

Both solutions provide SES customers who use SMTP with additional tools to improve email security, ensure compliance, and reduce operational overhead. You can deploy the option that best suites your needs by following the guidance in this blog post.

If your environment supports automated rotation, AWS Systems Manager Documents (SSM Documents) can help by providing pre-defined or custom automation workflows for securely managing secrets rotation, deploy Option 1.

If your environment does not support automated rotation, you can still implement an auditable, managed rotation solution by storing your secrets in AWS Systems Manager Parameter Store by deploying Option 2.

As a pay-per-use platform, the underlying AWS services used in either deployment option will only charge you for the resources that you actually consume. You can leverage the AWS Pricing Calculator to estimate the run-time costs for your specific workload. Alternatively, you can work directly with your AWS account team to understand the pricing for these solutions.

Getting SES SMTP Credentials

To send emails through the Amazon SES SMTP interface, email servers must first authenticate with SES using dedicated SES SMTP credentials. Typically, a systems administrator logs into the AWS SES console, clicks the Create SMTP Credentials button, and navigates to the AWS Identity and Access Management] (IAM) console. There, the administrator creates an IAM user with permissions for SES. The administrator then uses the IAM user’s secret access key to generate the SES SMTP password, which they use to configure their email servers or SMTP-enabled applications for use with SES.

The SES SMTP interface authenticates requests using an SMTP credential derived from an IAM user’s access key ID and secret access key. Since temporary access keys cannot be used to derive SES SMTP credentials, you must deploy and regularly rotate a long-lived key.

While the manual process of creating SES SMTP credentials works for a small number of credentials, it becomes cumbersome for customers with numerous email servers or strict password rotation policies. These customers may find the automated credential rotation mechanisms described in the following solutions better suited to their production needs.

Option 1 – Fully Automated Credential Rotation:

The fully automated version of this solution uses a custom Lambda function to create an SMTP password, which is stored in AWS Secrets Manager. AWS Secrets Manager’s built-in rotation feature then triggers the rotation of SES SMTP credentials. AWS Systems Manager Documents utilize AWS Systems Manager Agents to automatically make the changes to the authentication configuration on email servers.

The key advantages of using AWS Systems Manager to make the email server configuration changes include:

• Ability to deploy changes to on-premises and Amazon EC2 hosts, allowing rotation of secrets across a hybrid estate.
  Customization of the document to specific email software configurations.
  Targeting the secret (SMTP credential) rotation document on all email servers based on tags.

Let’s dive deep into Option 1 – Fully Automated Credential Rotation.

How Option 1 works:

Refer to the image above for the workflow:

1. AWS Secrets Manager initiates a rotation request, either on a schedule or via an authorized user’s request, triggering the “rotation Lambda” to rotate the SES SMTP credentials.
2. The SES Secret Rotation Function Lambda (see figure x above):
   • a. Creates a new IAM secret access key for the designated SES IAM user, derives the new SES SMTP password, and stores it in AWS Secrets Manager.
   • b. Connects to SES to verify the new SMTP password can authenticate.
   • c. Initiates an AWS Systems Manager Run Command to update the new SMTP password on target email servers using a pre-configured Systems Manager Document.
   • d. (and e.) Monitors the status of the Systems Manager Document execution until all updates complete successfully
   • f. Deletes the old IAM access and secret access keys.

With this fully automated solution, SES SMTP credentials can be rotated on a schedule or triggered manually, with no impact to email service uptime.

Deploying the Fully Automated Solution in Your AWS Account (Option 1)

Prerequisites for the Fully Automated Solution

1. AWS Account Access, typically with admin-level permission to allow for the deployment.
2. Your preferred IDE with AWS CLI version 2 and named profile setup.
   • Alternatively, you can use the AWS CLI from the AWS CloudShell in your browser.
3. Clone the Github repository (for this solution, you only need the README.md and sesautomaticrotation.yaml files found in /ses-credential-rotation/automatic-rotation).
   • git clone -b ses-credential-rotation https://github.com/aws-samples/serverless-mail.git
   • Note – We follow the principles of least privilege in this solution. The CloudFormation templates we’ve supplied require you to specify an identity, or configuration-set resource to use in the SES sending operation. You can find guidance on defining these values at Actions, resources, and condition keys for Amazon SES. Additionally, we’ve limited the IAM User to the ses:SendRawEmail action, which you can adjust as appropriate).
4. Console access to your AWS SES account that is properly configured to send emails via at least one verified identity.
5. Target email server(s) properly configured to send email via SES using SES SMTP authentication.
   • The AWS Systems Manager agent(s) must be correctly installed and configured on your target email server(s) as detailed in Setting up AWS Systems Manager.
   • The target email servers must be decorated with the tag (“SSMServerTag“) and value (“SSMServerTagValue“). These values allow the Systems Manager Document to identify them.
     • We use the tag “EmailServer” and the value “True” in our example, but you can use any tag and value that you wish).
6. An email address (or list) to receive SMTP rotation notifications.
7. Console access to your AWS Secrets Manager.
8. Console access to your AWS Systems Manager.

Deployment Steps

1. Clone the GitHub repository to your IDE
   • If using AWS CloudShell, ensure you are in the same region as your AWS Systems and Secrets Manager
   • run: git clone https://github.com/aws-samples/serverless-mail.git
   • Navigate to the directory ses-credential-rotation/automatic-rotation
2. Follow the steps in README.md to
   • Create a S3 bucket to deploy the CloudFormation Template.
   • Package the Lambda functions and upload them to Amazon S3.
   • Deploy the Cloud Formation Template.
   • Update the appropriate AWS Systems Manager sample document created by the CloudFormation Template to reflect your email server environments. These can be found in the AWS Systems Manager console under Documents > Owned by me
     • The ExampleWindowsIISSMTPSESpasswordrotator sample provides an example for Microsoft Windows hosts using the runPowerShellScript action to update the server’s SMTP credentials.
     • The ExamplePostfixSESpasswordrotator sample provides an example for Linux hosts using the runShellScript action to update the server’s SMTP credentials.

Testing Option 1 – Fully Automated Credential Rotation

To test the Fully Automated Credential Rotation solution, have Secrets Manager perform an immediate rotation by following these steps:

1. Open AWS Secrets Manager console
2. Locate the secret SESSendSecret
3. Select the Rotation tab
4. Click the “Rotate Secret immediately” button.

You can track the progress of the rotation by locating the logs of the Lambda that is deployed to manage the rotation.

1. In the AWS console, go to CloudFormationStack’s Resources tab
2. Find the LogicalID = SESSecretRotationFunction
3. Click the PhysicalID link to open the Lambda
4. Under the Monitor Tab, select the “View CloudWatch logs” button in the top right
5. The logs should show the rotation flow through 4 stages below (more details of each stage are available here):
   1. create_secret
   2. set_secret
   3. test_secret
   4. finish_secret

---

Option 2 – Partially Automated Credential Rotation:

The partially automated version uses a custom AWS Lambda function to create an SMTP password, which is stored in AWS Systems Manager Parameter Store. This solution simplifies credential rotation, where manual changes must be conducted by support staff. By wrapping the manual change process with AWS Step Functions, you can ensure a robust and auditable process to regularly rotate the SES SMTP credential.

How Option 2 works:

1. The credential rotation AWS Step Function creates a new SES SMTP credential and updates it in AWS Systems Manager Parameter Store.
2. It retrieves a list of servers from an Amazon DynamoDB table and launches a manual confirmation AWS Step Function execution for each server to initiate and track the manual step.
3. The manual confirmation AWS Step Function emails the designated address, requesting support staff to arrange the rotation. The email includes a link specific to that server.
4. The person completing the manual change confirms back to the AWS Step Function via the link that the rotation is complete.
5. Once the rotation on a server is confirmed, the manual confirmation AWS Step Function for that server is marked as complete.
6. After all server rotations are complete, the credential rotation AWS Step Function continues, disabling the old SES SMTP credential and deleting it after a few days.

AWS Step Function executions can last up to 365 days, providing sufficient time for the manual rotation and confirmation.

The screenshot below shows a graphical representation of the credential rotation AWS Step Function execution status, providing a real-time view of the rotation progress.

You can also track the status of individual servers via the manual rotation step function execution list.

The partially automated solution for rotating Amazon SES SMTP credentials is illustrated and detailed below:

Refer to the image above for the option 2 workflow:

1. EventBridge Scheduler Trigger: An EventBridge scheduler rule triggers a custom Starter Function Lambda (SF Lambda) on the last day of every 3rd month (this can be adjusted to suit your needs in the CloudFormation template).
2. Credential Rotation Step Function: The Starter Function Lambda triggers the Credential Rotation AWS Step Function, providing a clearly defined name to facilitate auditing (“password-rotation-dd-mm-yy“).
3. Credential Rotation Step Function Actions:
   1. Creates a new IAM (Identity and Access Management) secret access key for the SES IAM user.
   2. Triggers the SMTP Credential Generator Lambda to derive the SES SMTP password from the newly created IAM secret access key (using the algorithm provided in the SES documentation.
   3. Stores the new SES SMTP credential in AWS Systems Manager Parameter Store.
   4. Reads a list of servers that are utilizing this credential from a DynamoDB table.
4. Manual Confirmation Step Function:
   1. For each server, a manual confirmation AWS Step Function is triggered, sending a message on the Amazon Simple Notification Service (SNS) topic.
   2. The SNS notification prompts the server administrator via email to manually rotate the SMTP credentials on the on-premises email server.
   3. The server administrator uses a link in the email to confirm the credential has been rotated and tested on the server.
   4. The link triggers the Confirmation Lambda exposed via API Gateway, which marks the ManualConfirmation step function as complete.
5. Credential Rotation Completion: The CredentialRotation step function waits until all manual confirmation step functions have completed before proceeding.
6. Old IAM Access Key Deletion: Once confirmation has been received for all servers, the step function deletes the old IAM access key.

Deployment

To deploy the partially automated solution in your AWS account, you will need the following prerequisites:

Prerequisites for the Partially Automated Solution

1. AWS Account Access, typically with admin-level permission to allow for the deployment.
2. Your preferred IDE with AWS CLI version 2 and named profile setup. Alternatively, you can use the AWS CLI from the AWS CloudShell in your browser.
3. SES enabled, configured, and properly sending emails.
4. External email server(s) currently configured to use SES with SMTP.
5. Administrator email address to receive notifications.
6. AWS Secrets Manager and AWS Systems Manager set up.
7. AWS Systems Manager agent(s) correctly installed and configured on your target email servers as detailed in Setting up AWS Systems Manager.
8. Amazon EC2 instance with Postfix configured to send emails through SES
9. Target email servers must be decorated with a tag (“SSMServerTag“) and value (“SSMServerTagValue“) that will be used to identify them by the Systems Manager Document (we used “server” and “email”)
10. AWS Parameter Store and AWS Step Functions.

Once you have the prerequisites in place, follow the instructions in the GitHub project.

Conclusion

Implementing an automated credential rotation process for Amazon SES SMTP enhances security and compliance, streamlines operations, and reduces the risk of downtime and human error. By leveraging AWS Secrets Manager and AWS Systems Manager (option 1) or AWS Systems Manager Parameter Store and Step Functions (option 2), organizations can centralize SES SMTP credential management, maintain an audit trail, and quickly update email application servers with new SMTP credentials.

Need additional guidance?

• Join the conversation and connect with other administrators and security professionals on the AWS re:Post community to share insights and learn best practices.
• Consult the Amazon SES documentation
• Connect with AWS Premium Support
• Reach out to your AWS account team