Tag Archives: email

How to test email sending and monitoring

Post Syndicated from Dustin Taylor original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-test-email-sending/

Introduction

When setting up your email sending infrastructure and connections to APIs it is necessary to ensure proper setup. It is also important to ensure that after making changes to your sending pipeline that you verify that your application is working as expected. Not only is it important to test your sending processes, but it’s also important to test your monitoring to ensure that sending event tracking is working as intended. A common pitfall for email senders is that when they attempt to test their email sending infrastructure or event monitoring they send to invalid addresses and/or test accounts that generate no, or negative, reputation as a result of these sends.

The Amazon Simple Email Service (SES) provides you with an easy-to-use mechanism to accomplish these tests. Amazon SES offers the mailbox simulator feature which enables a sender the ability to test different sending events to ensure your service is working as expected. Using the mailbox simulator you can test: delivery success, bounces, complaints, automated responses (like out of office messages), and when a recipient address is on the suppression list.

In this blog we will outline some information about the mailbox simulator and how to interact with the feature to test your email sending services.

What is the mailbox simulator?

The mailbox simulator is a feature offered to help Amazon SES senders test their sending services to verify normal operation. It provides mechanisms to test their monitoring and event notification services. This feature gives a sender the ability to test their service and email monitoring to verify that it is working as expected without the risk of negatively impacting their sending reputation. The mailbox simulator is an MTA operated by SES that is set to receive mail and to simulate different sending events based on the recipient address used.

Why use the mailbox simulator?

The mailbox simulator provides an easy-to-use mechanism to test your integration with Amazon SES. This gives senders the ability to test their sending environment without triggering actual bounces or complaints, which negatively impact their account sending reputation, as well as not counting against a sender’s email sending quotas.  It is important to test these events to ensure that event monitoring is properly setup and function. A gap in monitoring these events could lead to a decrease in sender reputation from bounces or complaint events going unnoticed. The mailbox simulator gives a sender the ability to programmatically evaluate whether their event monitoring process has been set up properly without the negative impact to their sending reputation that would occur if sending test emails to differing mailbox providers or invalid email addresses.

How do I use the mailbox simulator?

Your first step is setting up a destination for your event notifications. This can be done using Amazon Simple Notification Service (SNS) or by using event publishing depending on your use-case.  Once you have set up an event destination and configured it for your sending identity (either an email address or domain) you are ready to proceed to testing the configuration.

Using the Amazon SES mailbox simulator is simple. In practice, you will be sending an email to an Amazon SES owned mailbox. This mailbox will respond based on the event-type you want to test. Below is a map of the event types and the corresponding email addresses to test the events:

Event Type Email Destination
Delivery Success [email protected]
Bounce [email protected]
Complaint [email protected]
Suppression List [email protected]
Automatic Responses (OOTO) [email protected]

If you are using the Amazon SES console to test these events, SES has already included the addresses to simplify the testing experience and you can find these under the ‘Scenario’ dropdown.

After sending an email to one of the five destinations, you should soon receive a notification, or event, to your publishing destination. This is an example of a success event.

{
    "notificationType": "Delivery",
    "mail": {
        "timestamp": "2023-05-05T21:00:23.244Z",
        "source": "[email protected]",
        "sourceArn": "arn:aws:ses:us-west-2:012345678910:identity/example.com",
        "sourceIp": "192.168.0.1",
        "callerIdentity": "root",
        "sendingAccountId": "012345678910",
        "messageId": "01010187edb7434c-4187f4b8-3e2b-404c-a5f6-72b9b64e5d66-000000",
        "destination": ["[email protected]"]
    },
    "delivery": {
        "timestamp": "2023-05-05T21:00:24.300Z",
        "processingTimeMillis": 1056,
        "recipients": ["[email protected]"],
        "smtpResponse": "250 2.6.0 Message received",
        "remoteMtaIp": "54.165.247.113",
        "reportingMTA": "a62-102.smtp-out.us-west-2.amazonses.com"
    }
}

If you have not received confirmation of the event, it is likely there is a problem with your monitoring configuration. We recommend reviewing the documentation on SNS topic setup and/or event publishing to uncover if an error was made during initial setup.

Note: A sender may have verified an email address and a domain to use for testing. The domain may have the appropriate configuration while the email address does not. When sending an email from SES, SES will use the most specific identity (email address is used before the domain) and will use the configuration associated with that identity. This means that in this instance you can either remove the email address verification for that domain and re-test or set up the same configuration for that email address that is verified.

What next?

Now that your initial setup of event publishing is complete and you have tested your first event through the mailbox simulator, it is time to set up automated testing using the mailbox simulator. Testing email events after a successful update to your application is recommended to confirm that updates have not caused bugs in your event ingestion mechanisms.

Happy sending!

What is BIMI and how to use it with Amazon SES

Post Syndicated from Matt Strzelecki original https://aws.amazon.com/blogs/messaging-and-targeting/what-is-bimi-and-how-to-use-it-with-amazon-ses/

Introduction

In this blog post I’d like to walk you through how to implement BIMI while using Amazon SES. For your information BIMI can be best described by the following excerpt from bimigroup.org:

Brand Indicators for Message Identification or BIMI (pronounced: Bih-mee) is an emerging email specification that enables the use of brand-controlled logos within supporting email clients. BIMI leverages the work an organization has put into deploying DMARC protection, by bringing brand logos to the customer’s inbox. For the brand’s logo to be displayed, the email must pass DMARC authentication checks, ensuring that the organization’s domain has not been impersonated.

Brands continually need to protect themselves from spoofing and phishing from bad actors who can damage the trust that customers and recipients have in those brands. Brand Indicators for Message Identification (BIMI) is an email specification that enables email inboxes to display a brand’s logo next to the brand’s authenticated email messages within supporting email clients. BIMI is an email specification that’s directly connected to authentication, but it’s not a standalone email authentication protocol as it requires all your email to comply with DMARC authentication. Recipients are more likely to engage with email that displays the logo of the brand associated with the message author. Higher engagement helps deliverability and inbox placement because it indicates that the recipients trust your brand. BIMI is a great brand protector in email and provides a better user experience for the end recipients and customers.

BIMI requires that you authenticate all of your organization’s email with SPF, DKIM and DMARC. In this how-to we will be utilizing Amazon SES to authenticate the emails, Amazon S3 to host the SVG image, and Amazon Route53 to add DNS records. We will be walking through how to accomplish each step until completion.

Note: While we’re using AWS products in this how-to, it is not a requirement to use all AWS products to implement BIMI. Any hosting provider for content or domain can be used however the steps may differ based on the provider you use.

BIMI Implementation

The following are the steps needed to prepare your SES account and domain for BIMI:

Step 1

Note: If you already have SPF, DKIM, and DMARC enabled for your domain (with 100% as the rate for DMARC) you can move on to Step 2.

Enable Easy DKIM for your domain

  1. Sign in to the AWS Management Console and open the Amazon SES console at https://console.aws.amazon.com/ses/.
  2. In the navigation pane, under Configuration, choose Verified identities.
  3. In the list of identities, choose an identity where the Identity type is Domain.

Note: If you need to create or verify a domain, see Creating a domain identity.

  1. Under the Authentication tab, in the DomainKeys Identified Mail (DKIM) container, choose Edit.
  2. In the Advanced DKIM settings container, choose the Easy DKIM button in the Identity type field.
  3. In the DKIM signing key length field, choose either RSA_2048_BIT or RSA_1024_BIT.
  4. In the DKIM signatures field, check the Enabled box.
  5. Choose Save changes.
  6. Now that you’ve configured your domain identity with Easy DKIM, you must complete the verification process with your DNS provider – proceed to Verifying a DKIM domain identity with your DNS provider and follow the DNS authentication procedures for Easy DKIM.

Create a DMARC record for your domain

  1. Sign in to the AWS Management Console and open the Route 53 console at https://console.aws.amazon.com/route53/
  2. In the navigation pane, choose Hosted zones.
  3. On the Hosted zones page, choose the name of the hosted zone that you want to create records in.
  4. Choose and define the applicable routing policy and the following values:
Name Record Type Value
_dmarc.example.com TXT v=DMARC1;p=quarantine;pct=100;rua=mailto:[email protected]
  1. Choose Create records.

Note: The DMARC policy must enforce at 100% and include either a quarantine or reject policy. (i.e. p=reject or p=quarantine) to meet the DMARC authentication requirement. This may mean you will need to update your existing policy and DMARC record.

Configure a Custom Mail From for your sending domain

  1. Open the Amazon SES console at https://console.aws.amazon.com/ses/.
  2. In the left navigation pane, under Configuration, choose Verified identities.
  3. In the list of identities, choose the identity you want to configure where the Identity type is Domain and Status is Verified.

a. If the Status is Unverified, complete the procedures at Verifying a DKIM domain identity with your DNS provider to verify the email address’s domain.

  1. At the bottom of the screen in the in the Custom MAIL FROM domain pane, choose Edit .
  2. In the General details pane, do the following:

a. Select the Use a custom MAIL FROM domain checkbox.

b. For MAIL FROM domain, enter the subdomain that you want to use as the MAIL FROM domain.

c. For Behavior on MX failure, choose one of the following options:

    • Use default MAIL FROM domain – If the custom MAIL FROM domain’s MX record is not set up correctly, Amazon SES uses a subdomain of amazonses.com. The subdomain varies based on the AWS Region that you use Amazon SES in.
    • Reject message – If the custom MAIL FROM domain’s MX record is not set up correctly, Amazon SES returns a MailFromDomainNotVerified error. Emails that you attempt to send from this domain are automatically rejected. If you want to ensure that 100% of your email is BIMI compatible, then you should choose the reject message option.

d. Choose Save changes – you’ll be returned to the previous screen.

  1. Publish the MX and SPF (type TXT) records to the DNS server of the custom MAIL FROM domain:

Note: In the Custom MAIL FROM domain pane, the Publish DNS records table now displays the MX and SPF (type TXT) records in that you have to publish (add) to your domain’s DNS configuration. These records use the formats shown in the following table.

Name Record Type Value
subdomain.example.com MX 10 feedback-smtp.region.amazonses.com
subdomain.example.com TXT v=spf1 include:amazonses.com ~all

Step 2

Produce an SVG Tiny PS version of your official logo

In order to display your logo in the email it must conform to the specifications of the BIMI requirements. To meet these requirements the logo must be a Scalable Vector Graphics (SVG) image and must meet the Tiny PS Specification. Once your image meets this requirement you can move on to the next step.

Note: bimigroup.org outlines this process and includes references to software to assist with this process.

Step 3

Upload your image to an S3 bucket

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/
  2. In the Buckets list, choose the name of the bucket that you want to upload your folders or files to.
  3. Choose Upload.
  4. In the Upload window, do one of the following:
    • Drag and drop files and folders to the Upload window.
    • Choose Add file choose your SVG image to upload, and choose Open.

To configure additional object properties

  1. To change access control list permissions, choose Permissions.
  2. Under Access control list (ACL), edit the permissions.
    • You need to grant read access to your objects to the public (everyone in the world) for the SVG image you are uploading. However, we recommend not changing the default setting for your bucket to public read access.
  1. To configure other additional properties, choose Properties.
  2. To upload your objects, choose Upload.

Note: Amazon S3 uploads your object. When the upload completes, you can see a success message on the Upload: status page.

  1. Choose Exit.

Step 4

Publish a BIMI record for your domain

  1. Sign in to the AWS Management Console and open the Route 53 console at https://console.aws.amazon.com/route53/
  2. In the navigation pane, choose Hosted zones.
  3. On the Hosted zones page, choose the name of the hosted zone that you want to create records in.
  4. Choose and define the applicable routing policy and the following values with the understanding the URLs must be HTTPS:
Name Record Type Value
default._bimi.example.com TXT v=BIMI1; l=[SVG URL]; a=[PEM URL]
  1. Choose Create records.

Note: the a= tag is currently optional and will not be used in this example.

You can validate your BIMI record with a tool like the BIMI Inspector.

Conclusion

All of the steps to set up your SES account and your domain are now complete. The final component in this process is to have regular sending patterns to the mailbox providers that support BIMI logo placement. Your domain should have a regular delivery cadence and needs to have a good reputation with the mailbox providers you are sending mail. BIMI logo placement may take time to populate to mailbox providers where you don’t have an established reputation or sending cadence. The time spent implementing BIMI is well worth it as it will strengthen your sender reputation and create a better and more trusted customer experience for your end recipients.

You can find more information about the BIMI specification here.

Email Link Isolation: your safety net for the latest phishing attacks

Post Syndicated from Joao Sousa Botto original https://blog.cloudflare.com/area1-eli-ga/

Email Link Isolation: your safety net for the latest phishing attacks

Email Link Isolation: your safety net for the latest phishing attacks

Email is one of the most ubiquitous and also most exploited tools that businesses use every single day. Baiting users into clicking malicious links within an email has been a particularly long-standing tactic for the vast majority of bad actors, from the most sophisticated criminal organizations to the least experienced attackers.

Even though this is a commonly known approach to gain account access or commit fraud, users are still being tricked into clicking malicious links that, in many cases, lead to exploitation. The reason is simple: even the best trained users (and security solutions) cannot always distinguish a good link from a bad link.

On top of that, securing employees’ mailboxes often results in multiple vendors, complex deployments, and a huge drain of resources.

Email Link Isolation turns Cloudflare Area 1 into the most comprehensive email security solution when it comes to protecting against phishing attacks. It rewrites links that could be exploited, keeps users vigilant by alerting them of the uncertainty around the website they’re about to visit, and protects against malware and vulnerabilities through the user-friendly Cloudflare Browser Isolation service. Also, in true Cloudflare fashion,  it’s a one-click deployment.

With more than a couple dozen customers in beta and over one million links protected (so far), we can now clearly see the significant value and potential that this solution can deliver. To extend these benefits to more customers and continue to expand on the multitude of ways we can apply this technology, we’re making Email Link Isolation generally available (GA) starting today.

Email Link Isolation is included with Cloudflare Area 1 enterprise plan at no extra cost, and can be enabled with three clicks:

1. Log in to the Area 1 portal.

2. Go to Settings (the gear icon).

3. On Email Configuration, go to Email Policies > Link Actions.

4. Scroll to Email Link Isolation and enable it.

Email Link Isolation: your safety net for the latest phishing attacks

Defense in layers

Applying multiple layers of defense becomes ever more critical as threat actors continuously look for ways to navigate around each security measure and develop more complex attacks. One of the best examples that demonstrates these evolving techniques is a deferred phishing attack, where an embedded URL is benign when the email reaches your email security stack and eventually your users’ inbox, but is later weaponized post-delivery.

Email Link Isolation: your safety net for the latest phishing attacks

To combat evolving email-borne threats, such as malicious links, Area 1 continually updates its machine learning (ML) models to account for all potential attack vectors, and leverages post-delivery scans and retractions as additional layers of defense. And now, customers on the Enterprise plan also have access to Email Link Isolation as one last defense – a safety net.

The key to successfully adding layers of security is to use a strong Zero Trust suite, not a disjointed set of products from multiple vendors. Users need to be kept safe without disrupting their productivity – otherwise they’ll start seeing important emails being quarantined or run into a poor experience when accessing websites, and soon enough they’ll be the ones looking for ways around the company’s security measures.

Built to avoid productivity impacts

Email Link Isolation provides an additional layer of security with virtually no disruption to the user experience. It’s smart enough to decide which links are safe, which are malicious, and which are still dubious. Those dubious links are then changed (rewritten to be precise) and Email Link Isolation keeps evaluating them until it reaches a verdict with a high degree of confidence. When a user clicks on one of those rewritten links, Email Link Isolation checks for a verdict (benign or malign) and takes the corresponding action – benign links open in the local browser as if they hadn’t been changed, while malign links are prevented from opening altogether.

Most importantly, when Email Link Isolation is unable to confidently determine a verdict based on all available intelligence, an interstitial page is presented to ask the user to be extra vigilant. The interstitial page calls out that the website is suspicious, and that the user should refrain from entering any personal information and passwords unless they know and fully trust the website. Over the last few months of beta, we’ve seen that over two thirds of users don’t proceed to the website after seeing this interstitial – that’s a good thing!

For the users that still want to navigate to the website after seeing the interstitial page, Email Link Isolation uses Cloudflare Browser Isolation to automatically open the link in an isolated browser running in Cloudflare’s closest data center to the user. This delivers an experience virtually indistinguishable from using the local browser, thanks to our Network Vector Rendering (NVR) technology and Cloudflare’s expansive, low-latency network. By opening the suspicious link in an isolated browser, the user is protected against potential browser attacks (including malware, zero days, and other types of malicious code execution).

In a nutshell, the interstitial page is displayed when Email Link Isolation is uncertain about the website, and provides another layer of awareness and protection against phishing attacks. Then, Cloudflare Browser Isolation is used to protect against malicious code execution when a user decides to still proceed to such a website.

What we’ve seen in the beta

As expected, the percentage of rewritten links that users actually click is quite small (single digit percentage). That’s because the majority of such links are not delivered in messages the users are expecting, and aren’t coming from trusted colleagues or partners of theirs. So, even when a user clicks on such a link, they will often see the interstitial page and decide not to proceed any further. We see that less than half of all clicks lead to the user actually visiting the website (in Browser Isolation, to protect against malicious code that could otherwise be executing behind the scenes).

Email Link Isolation: your safety net for the latest phishing attacks
Email Link Isolation: your safety net for the latest phishing attacks

You may be wondering why we’re not seeing a larger amount of clicks on these rewritten links. The answer is quite simply that link Email Link Isolation is indeed that last layer of protection against attack vectors that may have evaded other lines of defense. Virtually all the well crafted phishing attacks that try and trick users into clicking malicious links are already being stopped by the Area 1 email security, and such messages don’t reach users’ inboxes.

The balance is very positive. From all the customers using Email Link Isolation beta in production, some Fortune 500, we received no negative feedback on the user experience. That means that we’re meeting one of the most challenging goals – to provide additional security without negatively affecting users and without adding the burden of tuning/administration to the SOC and IT teams.

One interesting thing we uncover is how valuable our customers are finding our click-time inspection of link shorteners. The fact that a shortened URL (e.g. bit.ly) can be modified at any time to point to a different website has been making some of our customers anxious. Email Link Isolation inspects the link at time-of-click, evaluates the actual website that it’s going to open, and proceeds to open locally, block or present the interstitial page as adequate. We’re now working on full link shortener coverage through Email Link Isolation.

All built on Cloudflare

Cloudflare’s intelligence is driving the decisions of what gets rewritten. We have earlier signals than others.

Email Link Isolation has been built on Cloudflare’s unique capabilities in many areas.

First, because Cloudflare sees enough Internet traffic for us to confidently identify new/low confidence and potentially dangerous domains earlier than anyone else – leveraging the Cloudflare intelligence for this early signal is key to the user experience, to not add speed bumps to legitimate websites that are part of our users’ daily routines. Next, we’re using Cloudflare Workers to process this data and serve the interstitial without introducing frustrating delays to the user. And finally, only Cloudflare Browser Isolation can protect against malicious code with a low-latency experience that is invisible to end users and feels like a local browser.

If you’re not yet a Cloudflare Area 1 customer, start your free trial and phishing risk assessment here.

Email Routing leaves Beta

Post Syndicated from Celso Martinho original https://blog.cloudflare.com/email-routing-leaves-beta/

Email Routing leaves Beta

Email Routing leaves Beta

Email Routing was announced during Birthday Week in 2021 and has been available for free to every Cloudflare customer since early this year. When we launched in beta, we set out to make a difference and provide the most uncomplicated, more powerful email forwarding service on the Internet for all our customers, for free.

We feel we’ve met and surpassed our goals for the first year. Cloudflare Email Routing is now one of our most popular features and a top leading email provider. We are processing email traffic for more than 550,000 inboxes and forwarding an average of two million messages daily, and still growing month to month.

In February, we also announced that we were acquiring Area1. Merging their team, products, and know-how with Cloudflare was a significant step in strengthening our Email Security capabilities.

All this is good, but what about more features, you ask?

The team has been working hard to enhance Email Routing over the last few months. Today Email Routing leaves beta.

Also, we feel that this could be a good time to give you an update on all the new things we’ve been adding to the service, including behind-the-scenes and not-so-visible improvements.

Let’s get started.

Public API and Terraform

Cloudflare has a strong API-first philosophy. All of our services expose their primitives in our vast API catalog and gateway, which we then “dogfood” extensively. For instance, our customer’s configuration dashboard is built entirely on top of these APIs.

The Email Routing APIs didn’t quite make it to this catalog on day one and were kept private and undocumented for a while. This summer we made those APIs available on the public Cloudflare API catalog. You can programmatically use them to manage your destination emails, rules, and other Email Routing settings. The methods’ definitions and parameters are documented, and we provide curl examples if you want to get your hands dirty quickly.

Even better, if you’re an infrastructure as code type of user and use Terraform to configure your systems automatically, we have you covered too. The latest releases of Cloudflare’s Terraform provider now incorporate the Email Routing API resources, which you can use with HCL.

Email Routing leaves Beta

IPv6 egress

IPv6 adoption is on a sustained growth path. Our latest IPv6 adoption report shows that we’re nearing the 30% penetration figure globally, with some countries, where mobile usage is prevalent, exceeding the 50% mark. Cloudflare has offered full IPv6 support since 2011 as it aligns entirely with our mission to help build a better Internet.

We are IPv6-ready across the board in our network and our products, and Email Routing has had IPv6 ingress support since day one.

➜  ~ dig celso.io MX +noall +answer
celso.io.		300	IN	MX	91 isaac.mx.cloudflare.net.
celso.io.		300	IN	MX	2 linda.mx.cloudflare.net.
celso.io.		300	IN	MX	2 amir.mx.cloudflare.net.
➜  ~ dig linda.mx.cloudflare.net AAAA +noall +answer
linda.mx.cloudflare.net. 300	IN	AAAA	2606:4700:f5::b
linda.mx.cloudflare.net. 300	IN	AAAA	2606:4700:f5::c
linda.mx.cloudflare.net. 300	IN	AAAA	2606:4700:f5::d

More recently, we closed the loop and added egress IPv6 as well. Now we also use IPv6 when sending emails to upstream servers. If the MX server to which an email is being forwarded supports IPv6, then we will try to use it. Gmail is one good example of a high traffic destination that has IPv6 MX records.

➜  ~ dig gmail.com MX +noall +answer
gmail.com.		3362	IN	MX	30 alt3.gmail-smtp-in.l.google.com.
gmail.com.		3362	IN	MX	5 gmail-smtp-in.l.google.com.
gmail.com.		3362	IN	MX	10 alt1.gmail-smtp-in.l.google.com.
gmail.com.		3362	IN	MX	20 alt2.gmail-smtp-in.l.google.com.
gmail.com.		3362	IN	MX	40 alt4.gmail-smtp-in.l.google.com.
➜  ~ dig gmail-smtp-in.l.google.com AAAA +noall +answer
gmail-smtp-in.l.google.com. 116	IN	AAAA	2a00:1450:400c:c03::1a

We’re happy to report that we’re now delivering most of our email to upstreams using IPv6.

Email Routing leaves Beta

Observability

Email Routing is effectively another system that sits in the middle of the life of an email message. No one likes to navigate blindly, especially when using and depending on critical services like email, so it’s our responsibility to provide as much observability as possible about what’s going on when messages are transiting through our network.

End to end email deliverability is a complex topic and often challenging to troubleshoot due to the nature of the protocol and the number of systems and hops involved. We added two widgets, Analytics and Detailed Logs, which will hopefully provide the needed insights and help increase visibility.

The Analytics section of Email Routing shows general statistics about the number of emails received during the selected timeframe, how they got handled to the upstream destination addresses, and a convenient time-series chart.

Email Routing leaves Beta

On the Activity Log, you can get detailed information about what happened to each individual message that was received and then delivered to the destination. That information includes the sender and the custom address used, the timestamp, and the delivery attempt result. It also has the details of our SPF, DMARC, and DKIM validations. We also provide filters to help you find what you’re looking for in case your message volume is higher.

More recently, the Activity Log now also shows bounces. A bounce message happens when the upstream SMTP server accepts the delivery, but then, for any reason (exceeded quota, virus checks, forged messages, or other issues), the recipient inbox decides to reject it and return a new message back with an error to the latest MTA in the chain, read from the Return-Path headers, which is us.

Email Routing leaves Beta

Audit Logs

Audit Logs are available on all plan types and summarize the history of events, like login and logout actions, or zone configuration changes, made within your Cloudflare account. Accounts with multiple members or companies that must comply with regulatory obligations rely on Audit logs for tracking and evidence reasons.

Email Routing now integrates with Audit Logs and records all configuration changes, like adding a new address, changing a rule, or editing the catch-all address. You can find the Audit Logs on the dashboard under “Manage Account” or use our API to download the list.

Email Routing leaves Beta

Anti-spam

Unsolicited and malicious messages plague the world of email and are a big problem for end users. They affect the user experience and efficiency of email, and often carry security risks that can lead to scams, identity theft, and manipulation.

Since day one, we have supported and validated SPF (Sender Policy Framework) records,  DKIM (DomainKeys Identified Mail) signatures, and DMARC (Domain-based Message Authentication) policies in incoming messages. These steps are important and mitigate some risks associated with authenticating the origin of an email from a specific legitimate domain, but they don’t solve the problem completely. You can still have bad actors generating spam or phishing Attacks from other domains who ignore SPF or DKIM completely.

Anti-spam techniques today are often based on blocking emails whose origin (the IP address of the client trying to deliver the message) confidence score isn’t great. This is commonly known in the industry as IP reputation. Other companies specialize in maintaining reputation lists for IPs and email domains, also known as RBL lists, which are then shared across providers and used widely.

Simply put, an IP or a domain gets a bad reputation when it starts sending unsolicited or malicious emails. If your IP or domain has a bad reputation, you’ll have a hard time delivering Emails from them to any major email provider. A bad reputation goes away when the IP or domain stops acting bad.

Cloudflare is a security company that knows a few things about IP threat scores and reputation. Working with the Area1 team and learning from them, we added support to flag and block emails received from what we consider bad IPs at the SMTP level. Our approach uses a combination of heuristics and reputation databases, including some RBL lists, which we constantly update.

This measure benefits not only those customers that receive a lot of spam, who will now get another layer of protection and filtering, but also everyone else using Email Routing. The reputation of our own IP space and forwarding domain, which we use to deliver messages to other email providers, will improve, and with it, so will our deliverability success rate.

IDN support

Internationalized domain names, or IDNs for short, are domains that contain at least one non-ASCII character. To accommodate backward compatibility with older Internet protocols and applications, the IETF approved the IDNA protocol (Internationalized Domain Names in Applications), which was then adopted by many browsers, top-level domain registrars and other service providers.

Cloudflare was one of the first platforms to adopt IDNs back in 2012.  Supporting internationalized domain names on email, though, is challenging. Email uses DNS, SMTP, and other standards (like TLS and DKIM signatures) stacked on top of each other. IDNA conversions need to work end to end, or something will break.

Email Routing didn’t support IDNs until now. Starting today, Email Routing can be used with IDNs and everything will work end to end as expected.

Email Routing leaves Beta

8-bit MIME transport

The SMTP protocol supports extensions since the RFC 2821 revision. When an email client connects to an SMTP server, it announces its capabilities on the EHLO command.

➜  ~ telnet linda.mx.cloudflare.net 25
Trying 162.159.205.24...
Connected to linda.mx.cloudflare.net.
Escape character is '^]'.
220 mx.cloudflare.net Cloudflare Email ESMTP Service ready
EHLO celso.io
250-mx.cloudflare.net greets celso.io
250-STARTTLS
250-8BITMIME
250 ENHANCEDSTATUSCODES

This tells our client that we support the Secure SMTP over TLS, Enhanced Error Codes, and the 8-bit MIME Transport, our latest addition.

Most modern clients and servers support the 8BITMIME extension, making transmitting binary files easier and more efficient without additional conversions to and from 7-bit.

Email Routing now supports transmitting 8BITMIME SMTP messages end to end and handles DKIM signatures accordingly.

Other fixes

We’ve been making other smaller improvements to Email Routing too:

  • We ported our SMTP server to use BoringSSL, Cloudflare’s SSL/TLS implementation of choice, and now support more ciphers when clients connect to us using STARTTLS and when we connect to upstream servers.
  • We made a number of improvements when we added our own DKIM signatures in the messages. We keep our Rust 🦀DKIM implementation open source on GitHub, and we also contribute to Lettre, a Rust mailer library that we use.
  • When a destination address domain has multiple MX records, we now try them all in their preference value order, as described in the RFC, until we get a good delivery, or we fail.

Route to Workers update

We announced Route to Workers in May this year. Route to Workers enables everyone to programmatically process their emails and use them as triggers for any other action. In other words, you can choose to process any incoming email with a Cloudflare Worker script and then implement any logic you wish before you deliver it to a destination address or drop it. Think about it as programmable email.

The good news, though, is that we’re near completing the project. The APIs, the dashboard configuration screens, the SMTP service, and the necessary Cap’n Proto interface to Workers are mostly complete, and “all” we have left now is adding the Email Workers primitives to the runtime and testing the hell out of everything before we ship.

Thousands of users are waiting for Email Workers to start creating advanced email processing workflows, and we’re excited about the possibilities this will open. We promise we’re working hard to open the public beta as soon as possible.

Email Routing leaves Beta

What’s next?

We keep looking at ways to improve email and will add more features and support to emerging protocols and extensions. Two examples are ARC (Authenticated Received Chain), a new signature-based authentication system designed with email forwarding services in mind, and EAI (Email Address Internationalization), which we will be supporting soon.

In the meantime, you can start using Email Routing with your own domain if you haven’t yet, it only takes a few minutes to set up, and it’s free. Our Developers Documentation page has details on how to get started, troubleshooting, and technical information.

Ping us on our Discord server, community forum, or Twitter if you have suggestions or questions, the team is listening.

Click Here! (safely): Automagical Browser Isolation for potentially unsafe links in email

Post Syndicated from Joao Sousa Botto original https://blog.cloudflare.com/safe-email-links/

Click Here! (safely): Automagical Browser Isolation for potentially unsafe links in email

Click Here! (safely): Automagical Browser Isolation for potentially unsafe links in email

We’re often told not to click on ‘odd’ links in email, but what choice do we really have? With the volume of emails and the myriad of SaaS products that companies use, it’s inevitable that employees find it almost impossible to distinguish a good link before clicking on it. And that’s before attackers go about making links harder to inspect and hiding their URLs behind tempting “Confirm” and “Unsubscribe” buttons.

We need to let end users click on links and have a safety net for when they unwittingly click on something malicious — let’s be honest, it’s bound to happen even if you do it by mistake. That safety net is Cloudflare’s Email Link Isolation.

With Email Link Isolation, when a user clicks on a suspicious link — one that email security hasn’t identified as ‘bad’, but is still not 100% sure it’s ‘good’ — they won’t immediately be taken to that website. Instead, the user first sees an interstitial page recommending extra caution with the website they’ll visit, especially if asked for passwords or personal details.

Click Here! (safely): Automagical Browser Isolation for potentially unsafe links in email

From there, one may choose to not visit the webpage or to proceed and open it in a remote isolated browser that runs on Cloudflare’s global network and not on the user’s local machine. This helps protect the user and the company.

The user experience in our isolated browser is virtually indistinguishable from using one’s local browser (we’ll talk about why below), but untrusted and potentially malicious payloads will execute away from the user’s computer and your corporate network.

In summary, this solution:

  • Keeps users alert to prevent credential theft and account takeover
  • Automatically blocks dangerous downloads
  • Prevents malicious scripts from executing on the user’s device
  • Protects against zero-day exploits on the browser

How can I try it

Area 1 is Cloudflare’s email security solution. It protects organizations from the full range of email attack types (URLs, payloads, BEC), vectors (email, web, network), and attack channels (external, internal, trusted partners) by enforcing multiple layers of protection before, during, and after the email hits the inbox. Today it adds Email Link Isolation to the protections it offers.

If you are a Cloudflare Area 1 customer you can request access to the Email Link Isolation beta today. We have had Email Link Isolation deployed to all Cloudflare employees for the last four weeks and are ready to start onboarding customers.

During the beta it will be available for free on all plans. After the beta it will still be included at no extra cost with our PhishGuard plan.

Under the hood

To create Email Link Isolation we used a few ingredients that are quite special to Cloudflare. It may seem complicated and, in a sense, the protection is complex, but we designed this so that the user experience is fast, safe, and with clear options on how to proceed.

1. Find potentially unsafe domains

First, we have created a constantly updating list of domains that the Cloudflare’s DNS resolver recently saw for the first time, or that are somehow potentially unsafe (leveraging classifiers from the Cloudflare Gateway and other products). These are domains that would be too disruptive for the organization to block outright, but that should still be navigated with extra caution.

For example, people acquire domains and create new businesses every day. There’s nothing wrong with that – quite the opposite. However, attackers often set up or acquire websites serving legitimate content and, days or weeks later, send a link to intended targets. The emails flow through as benign and the attacker weaponizes the website when emails are already sitting on people’s inboxes. Blocking all emails with links to new websites would cause users to surely miss important communications, and delivering the emails while making links safe to click on is a much better suited approach.

There is also hosting infrastructure from large cloud providers, such as Microsoft or Google, that prevent crawling and scanning. These are used on our day-to-day business, but attackers may deploy malicious content there. You wouldn’t want to fully block emails with links to Microsoft SharePoint, for example, but it’s certainly safer to use Email Link Isolation on them if they link to outside your organization.

Attackers are constantly experimenting with new ways of looking legitimate to their targets, and that’s why relying on the early signals that Cloudflare sees makes such a big difference.

The second ingredient we want to highlight is that, as Cloudflare Area 1 processes and inspects emails for security concerns, it also checks the domain of every link against the suspicious list. If an email contains a link to a suspicious domain, Cloudflare Area 1 automatically changes it (rewrites) so that the interstitial page is shown, and the link opens with Cloudflare Browser Isolation by default.

Note: Rewriting email links is only possible when emails are processed inline, which is one of the options for deploying Area 1. One of the big disadvantages of any email security solution deployed as API-only is that closing this last mile gap through link rewriting isn’t a possibility.

3. Opens remotely but feels local

When a user clicks on one of these rewritten links, instead of directly accessing a potential threat, our systems will first check their current classification (benign, suspicious, malicious). Then, if it’s malicious, the user will be blocked from continuing to the website and see an interstitial page informing them why. No further action is required.

If the link is suspicious, the user is offered the option to open it in an isolated browser. What happens next? The link is opened with Cloudflare Browser Isolation in a nearby Cloudflare data center (globally within 50 milliseconds of 95% of the Internet-connect population). To ensure website compatibility and security, the target website is entirely executed in a sandboxed Chromium-based browser. Finally, the website is instantly streamed back to the user as vector instructions consumed by a lightweight HTML5-compatible remoting client in the user’s preferred web browser. These safety precautions happen with no perceivable latency to the end user.

Cloudflare Browser Isolation is an extremely secure remote browsing experience that feels just like local browsing. And delivering this is only possible by serving isolated browsers on a low latency, global network with our unique vector based streaming technology. This architecture is different from legacy remote browser isolation solutions that rely on fragile and insecure DOM-scrubbing, or are bandwidth intensive and high latency pixel pushing techniques hosted in a few high latency data centers.

4. Reassess (always learning)

Last but not least, another ingredient that makes Email Link Isolation particularly effective is that behind the scenes our services are constantly reevaluating domains and updating their reputation in Cloudflare’s systems.

When a domain on our suspicious list is confirmed to be benign, all links to it can automatically start opening with the user’s local browser instead of with Cloudflare Browser Isolation.

Similarly, if a domain on the suspicious list is identified as malign, all links to that domain can be immediately blocked from opening. So, our services are constantly learning and acting accordingly.

It’s been four weeks since we deployed Email Link Isolation to all our 3,000+ Cloudflare employees, here’s what we saw:

  • 100,000 link rewrites per week on Spam and Malicious emails. Such emails were already blocked server side by Area 1 and users never see them. It’s still safer to rewrite these as they may be released from quarantine on user request.
  • 2,500 link rewrites per week on Bulk emails. Mostly graymail, which are commercial/bulk communications the user opted into. They may end up in the users’ spam folder.
  • 1,000 link rewrites per week on emails that do not fit any of the categories above — these are the ones that normally reach the user’s inboxes. These are almost certainly benign, but there’s still enough doubt to warrant a link rewrite.
  • 25 clicks on rewritten links per week (up to six per day).
Click Here! (safely): Automagical Browser Isolation for potentially unsafe links in email

As a testament to the efficacy of Cloudflare Area 1, 25 suspicious link clicks per week for a universe of over 3,000 employees is a very low number. Thanks to Email Link Isolation, users were protected against exploits.

Better together with Cloudflare Zero Trust

In future iterations, administrators will be able to connect Cloudflare Area 1 to their Cloudflare Zero Trust account and apply isolation policies, DLP (Data Loss Protection) controls and in-line CASB (a cloud access security broker) to email link isolated traffic.

We are starting our beta today. If you’re interested in trying Email Link Isolation and start to feel safer with your email experience, you should sign up here.

Cloudflare Area 1 – how the best Email Security keeps getting better

Post Syndicated from Joao Sousa Botto original https://blog.cloudflare.com/email-security/

Cloudflare Area 1 - how the best Email Security keeps getting better

Cloudflare Area 1 - how the best Email Security keeps getting better

On February 23, 2022, after being a customer for two years and seeing phishing attacks virtually disappear from our employee’s mailboxes, Cloudflare announced the acquisition of Area 1 Security.

Thanks to its unique technology (more on that below) Cloudflare Area 1 can proactively identify and protect against phishing campaigns before they happen, and potentially prevent the 90%+ of all cyberattacks that Deloitte research identified as starting with an email. All with little to no impact on employee productivity.

But preventing 90% of the attacks is not enough, and that’s why Cloudflare Area 1 email security is part of our Zero Trust platform. Here’s what’s new.

Email Security on your Cloudflare Dashboard

Starting today you will find a dedicated Email Security section on your Cloudflare dashboard. That’s the easiest way for any Cloudflare customer to get familiar with and start using Cloudflare Area 1 Email Security.

From there you can easily request a trial, which gives you access to the full product for 30 days.

Our team will guide you through the setup, which will take just a few minutes. That’s the beauty of not having to install and tune a Secure Email Gateway (SEG). You can simply configure Area 1 inline or connect through the API, journaling, or other connectors – none of these options disrupt mail flow or the end user experience. And you don’t need any new hardware, appliances or agents.

Once the trial starts, you’ll be able to review detection metrics and forensics in real time, and will receive real-time updates from the Area 1 team on incidents that require immediate attention.

At the end of the trial you will also have a Phishing Risk Assessment where our team will walk you through the impact of the mitigated attacks and answer your questions.

Cloudflare Area 1 - how the best Email Security keeps getting better

Another option you’ll see on the Email Security section of the Cloudflare Dashboard is to explore the Area 1 demo.

At the click of a button you’ll enter the Area 1 portal of a fictitious company where you can see the product in action. You can interact with the full product, including our advanced message classifiers, the BEC protections, real time view of spoofed domains, and our unique message search and trace capabilities.

Cloudflare Area 1 - how the best Email Security keeps getting better

Product Expansions

Being cloud-native has allowed us to develop some unique capabilities. Most notably, we scan the Internet for attacker infrastructure, sources and delivery mechanisms to stop phishing attacks days before they hit an inbox. These are state of the art machine-learning models using the threat intelligence data that Area 1 has accumulated since it was founded nine years ago, and now they also incorporate data from the 124 billion cyber threats that Cloudflare blocks each day and its 1.7 trillion daily DNS queries.

Since the product is cloud-based and no local appliances are involved, these unique datasets and models benefit every customer immediately and apply to the full range of email attack types (URLs, payloads, BEC), vectors (email, web, network), and attack channels (external, internal, trusted partners). Additionally, the threat datasets, observables and Indicators of Compromise (IOC) are now additional signals to Cloudflare Gateway (part of Zero Trust), extending protection beyond email and giving Cloudflare customers the industry’s utmost protection against converged or blended threats.

The expertise Area 1 gained through this relentless focus on Threat Research and Threat Operations (i.e., disrupting actors once identified) is also leading to a new large scale initiative to make every Cloudflare customer, and the broader Internet, safer – Cloudforce One.

The Cloudforce One team is composed of analysts assigned to five subteams: Malware Analysis, Threat Analysis, Active Mitigation and Countermeasures, Intelligence Analysis, and Intelligence Sharing. Collectively, they have tracked many of the most sophisticated cyber criminals on the Internet while at the National Security Agency (NSA), USCYBERCOM, and Area 1 Security, and have worked closely with similar organizations and governments to disrupt these threat actors. They’ve also been prolific in publishing “finished intel” reports on security topics of significant geopolitical importance, such as targeted attacks against governments, technology companies, the energy sector, and law firms, and have regularly briefed top organizations around the world on their efforts.

The team will help protect all Cloudflare customers by working closely with our existing product, engineering, and security teams to improve our products based on tactics, techniques, and procedures (TTPs) observed in the wild. Customers will get better protection without having to take any action.

Additionally, customers can purchase a subscription to Cloudforce One (now generally available), and get access to threat data and briefings, dedicated security tools, and the ability to make requests for information (RFIs) to the team’s threat operations staff. RFIs can be on any security topic of interest, and will be analyzed and responded to in a timely manner. For example, the Cloudforce One Malware Analysis team can accept uploads of possible malware and provide a technical analysis of the submitted resource.

Lastly, SPF/DKIM/DMARC policies are another tool that can be used to prevent Email Spoofing and have always been a critical part of Area 1’s threat models. Cloudflare Area 1 customers receive weekly DMARC sender reports to understand the efficacy of their configuration, but customers have also asked for help in setting up SPF/DKIM/DMARC records for their own domains.

It was only logical to make Cloudflare’s Email Security DNS Wizard part of our Email Security stack to guide customers through their initial SPF, DKIM and DMARC configuration. The wizard is now available to all customers using Cloudflare DNS, and will soon be available to Cloudflare Area 1 customers using a third party DNS. Getting SPF/DKIM/DMARC right can be complex, but it is a necessary and vital part of making the Internet safer, and this solution will help you build a solid foundation.

You’ll be hearing from us very soon regarding more expansions to the Area 1 feature set. In the meantime, if you want to experience Area 1 first-hand sign up for a Phishing Risk Assessment here or explore the interactive demo through the Email section of your Cloudflare Dashboard.

Route to Workers, automate your email processing

Post Syndicated from Joao Sousa Botto original https://blog.cloudflare.com/announcing-route-to-workers/

Route to Workers, automate your email processing

Route to Workers, automate your email processing

Cloudflare Email Routing has quickly grown to a few hundred thousand users, and we’re incredibly excited with the number of feature requests that reach our product team every week. We hear you, we love the feedback, and we want to give you all that you’ve been asking for. What we don’t like is making you wait, or making you feel like your needs are too unique to be addressed.

That’s why we’re taking a different approach – we’re giving you the power tools that you need to implement any logic you can dream of to process your emails in the fastest, most scalable way possible.

Today we’re announcing Route to Workers, for which we’ll start a closed beta soon. You can join the waitlist today.

How this works

When using Route to Workers your Email Routing rules can have a Worker process the messages reaching any of your custom Email addresses.

Route to Workers, automate your email processing

Even if you haven’t used Cloudflare Workers before, we are making onboarding as easy as can be. You can start creating Workers straight from the Email Routing dashboard, with just one click.

Route to Workers, automate your email processing

After clicking Create, you will be able to choose a starter that allows you to get up and running with minimal effort. Starters are templates that pre-populate your Worker with the code you would write for popular use cases such as creating a blocklist or allowlist, content based filtering, tagging messages, pinging you on Slack for urgent emails, etc.

Route to Workers, automate your email processing

You can then use the code editor to make your new Worker process emails in exactly the way you want it to – the options are endless.

Route to Workers, automate your email processing

And for those of you that prefer to jump right into writing their own code, you can go straight to the editor without using a starter. You can write Workers with a language you likely already know. Cloudflare built Workers to execute JavaScript and WebAssembly and has continuously added support for new languages.

The Workers you’ll use for processing emails are just regular Workers that listen to incoming events, implement some logic, and reply accordingly. You can use all the features that a normal Worker would.

The main difference being that instead of:

export default {
  async fetch(request, env, ctx) {
    handleRequest(request);
  }
}

You’ll have:

export default {
  async email(message, env, ctx) {
    handleEmail(message);
  }
}

The new `email` event will provide you with the “from”, “to” fields, the full headers, and the raw body of the message. You can then use them in any way that fits your use case, including calling other APIs and orchestrating complex decision workflows. In the end, you can decide what action to take, including rejecting or forwarding the email to one of your Email Routing destination addresses.

With these capabilities you can easily create logic that, for example, only accepts messages coming from one specific address and, when one matches the criteria, forwards to one or more of your verified destination addresses while also immediately alerting you on Slack. Code for such feature could be as simple as this:

export default {
   async email(message, env, ctx) {
       switch (message.to) {
           case "[email protected]":
               await fetch("https://webhook.slack/notification", {
                   body: `Got a marketing email from ${ message.from }, subject: ${ message.headers.get("subject") }`,
               });
               sendEmail(message, [
                   "[email protected]",
                   "[email protected]",
               ]);
               break;

           default:
               message.reject("Unknown address");
       }
   },
};

Route to Workers enables everyone to programmatically process their emails and use them as triggers for any other action. We think this is pretty powerful.

Process up to 100,000 emails/day for free

The first 100,000 Worker requests (or Email Triggers) each day are free, and paid plans start at just $5 per 10 million requests. You will be able to keep track of your Email Workers usage right from the Email Routing dashboard.

Route to Workers, automate your email processing

Join the Waitlist

You can join the waitlist today by going to the Email section of your dashboard, navigating to the Email Workers tab, and clicking the Join Waitlist button.

Route to Workers, automate your email processing

We are expecting to start the closed beta in just a few weeks, and can’t wait to hear about what you’ll build with it!

As usual, if you have any questions or feedback about Email Routing, please come see us in the Cloudflare Community and the Cloudflare Discord.

Email Routing Insights

Post Syndicated from Joao Sousa Botto original https://blog.cloudflare.com/email-routing-insights/

Email Routing Insights

Email Routing Insights

Have you ever wanted to try a new email service but worried it might lead to you missing any emails? If you have, you’re definitely not alone. Some of us email ourselves to make sure it reaches the correct destination, others don’t rely on a new address for anything serious until they’ve seen it work for a few days. In any case, emails often contain important information, and we need to trust that our emails won’t get lost for any reason.

To help reduce these worries about whether emails are being received and forwarded – and for troubleshooting if needed – we are rolling out a new Overview page to Email Routing. On the Overview tab people now have full visibility into our service and can see exactly how we are routing emails on their behalf.

Routing Status and Metrics

The first thing you will see in the new tab is an at a glance view of the service. This includes the routing status (to know if the service is configured and running), whether the necessary DNS records are configured correctly, and the number of custom and destination addresses on the zone.

Email Routing Insights

Below the configuration summary, you will see more advanced statistics about the number of messages received on your custom addresses, and what happened to those messages. You will see information about the number of emails forwarded or dropped by Email Routing (based on the rules you created), and the number that fall under other scenarios such as being rejected by Email Routing (due to errors, not passing security checks or being considered spam) or rejected by your destination mailbox. You now have the exact counts and a chart, so that you can track these metrics over time.

Email Routing Insights

Activity Log

On the Cloudflare Email Routing tab you’ll also see the Activity Log, where you can drill deeper into specific behaviors. These logs show you details about the email messages that reached one of the custom addresses you have configured on your Cloudflare zone.

For each message the logs will show you the Message ID, Sender, Custom Address, when Cloudflare Email Routing received it, and the action that was taken. You can also expand the row to see the SPF, DMARC, and DKIM status of that message along with any relevant error messaging.

And we know looking at every message can be overwhelming, especially when you might be resorting to the logs for troubleshooting purposes, so you have a few options for filtering:

  • Search for specific people (email addresses) that have messaged you.
  • Filter to show only one of your custom addresses.
  • Filter to show only messages where a specific action was taken.
Email Routing Insights

Routes and Settings

Next to the Overview tab, you will find the Routes tab with the configuration UI that is likely already familiar to you. That’s where you create custom addresses, add and verify destination addresses, and create rules with the relationships between the custom and destination addresses.

Email Routing Insights

Lastly the Settings tab includes less common actions such as the DNS configuration and the options for off boarding from Email Routing.

We hope you enjoy this update. And if you have any questions or feedback about this product, please come see us in the Cloudflare Community and the Cloudflare Discord.

Democratizing email security: protecting individuals and businesses of all sizes from phishing and malware attacks

Post Syndicated from Patrick R. Donahue original https://blog.cloudflare.com/democratizing-email-security/

Democratizing email security: protecting individuals and businesses of all sizes from phishing and malware attacks

Democratizing email security: protecting individuals and businesses of all sizes from phishing and malware attacks

Since our founding, Cloudflare has been on a mission to take expensive, complex security solutions typically only available to the largest companies and make them easy to use and accessible to everyone. In 2011 and 2015 we did this for the web application firewall and SSL/TLS markets, simplifying the process of protecting websites from application vulnerabilities and encrypting HTTP requests down to single clicks; in 2020, during the start of the COVID-19 pandemic, we made our Zero Trust suite available to everyone; and today—in the face of heightened phishing attacks—we’re doing the same for the email security market.

Once the acquisition of Area 1 closes, as we expect early in the second quarter of 2022, we plan to give all paid self-serve plans access to their email security technology at no additional charge. Control, customization, and visibility via analytics will vary with plan level, and the highest flexibility and support levels will be available to Enterprise customers for purchase.

All self-serve users will also get access to a more feature-packed version of the Zero Trust solution we made available to everyone in 2020. Zero Trust services are incomplete without an email security solution, and CISA’s recent report makes that clearer than ever: over 90% of successful cyber attacks start with a phishing email, so we expect that over time analysts will have no choice but to include email in their definitions of secure access and zero edges.

If you’re interested in reserving your place in line, register your interest by logging into your Cloudflare account at dash.cloudflare.com, selecting your domain, clicking Email, and then “Join Waitlist” at the top of the page; we’ll reach out after the Area 1 acquisition is completed, and the integration is ready, in the order we received your request.

One-click deployment

If you’re already managing your authoritative DNS with Cloudflare, as nearly 100% of non-Enterprise plans are, there will just be a single click to get started. Once clicked, we’ll start returning different MX records to anyone trying to send email to your domain. This change will attract all emails destined for your domain, during which they’ll be run through Area 1’s models and potentially be quarantined or flagged. Customers of Microsoft Office 365 will also be able to take advantage of APIs for an even deeper integration and capabilities like post-delivery message redaction.

Democratizing email security: protecting individuals and businesses of all sizes from phishing and malware attacks

In addition to routing and filtering email, we’ll also automagically take care of your DNS email security records such as SPF, DKIM, DMARC, etc. We launched a tool to help with this last year, and soon we’ll be making it even more comprehensive and easier to use.

Integration with other Zero Trust products

As we wrote in the acquisition announcement post on this blog, we’re excited to integrate email security with other products in our Zero Trust suite. For customers of Gateway and Remote Browser Isolation (RBI), we’ll automatically route potentially suspicious domains and links through these protective layers. Our built-in data loss prevention (DLP) technology will also be wired into Area 1’s technology in deployments where visibility into outbound email is available.

Improving threat intelligence with new data sources

In addition to integrating directly with Zero Trust products, we’re excited about connecting threat data sources from Area 1 into existing Cloudflare products and vice versa. For example, phishing infrastructure identified during Area 1’s Internet-wide scans will be displayed within the recently launched Cloudflare Security Center, and 1.1.1.1’s trillions of queries per month will help Area 1 identify new domains that may be threats. Domains that are newly registered, or registered with slight variations of legitimate domains, are often warning signs of an upcoming phishing attack.

Getting started

Cloudflare has been a happy customer of Area 1’s technology for years, and we’re excited to open it up to all of our customers as soon as possible. If you’re excited as we are about being able to use this in your Pro or Business plan, reserve your place in line today within the Email tab for your domain. Or if you’re an Enterprise customer and want to get started immediately, fill out this form or contact your Customer Success Manager.

Email Routing is now in open beta, available to everyone

Post Syndicated from Joao Sousa Botto original https://blog.cloudflare.com/email-routing-open-beta/

Email Routing is now in open beta, available to everyone

Email Routing is now in open beta, available to everyone

I won’t beat around the bush: we’ve moved Cloudflare Email Routing from closed beta to open beta 🎉

What does this mean? It means that there’s no waitlist anymore; every zone* in every Cloudflare account has Email Routing available to them.

To get started just open one of the zones in your Cloudflare Dashboard and click on Email in the navigation pane.

Email Routing is now in open beta, available to everyone

Our journey so far

Back in September 2021, during Cloudflare’s Birthday Week, we introduced Email Routing as the simplest solution for creating custom email addresses for your domains without the hassle of managing multiple mailboxes.

Many of us at Cloudflare saw a need for this type of product, and we’ve been using it since before it had a UI. After Birthday Week, we started gradually opening it to Cloudflare customers that requested access through the wait list; starting with just a few users per week and gradually ramping up access as we found and fixed edge cases.

Most recently, with users wanting to set up Email Routing for more of their domains and with some of G Suite legacy users looking for an alternative to starting a subscription, we have been onboarding tens of thousands of new zones every day into the closed beta. We’re loving the adoption and the feedback!

Needless to say that with hundreds of thousands of zones from around the world in the Email Routing beta we uncovered many new use cases and a few limitations, a few of which still exist. But these few months of closed beta gave us the confidence to move to the next stage – open beta – which now makes Cloudflare Email Routing available to everyone, including free zones.

Thank you to all of you that were part of the closed beta and provided feedback. We couldn’t be more excited to welcome everyone else!

If you have any questions or feedback about this product, please come see us in the Cloudflare Community and the Cloudflare Discord.

___

*we do have a few limitations, such as not currently supporting Internationalized Domain Names (IDNs) and subdomains. Known limitations are listed in the documentation.

How to configure an incoming email security gateway with Amazon WorkMail

Post Syndicated from Jesse Thompson original https://aws.amazon.com/blogs/security/how-to-configure-an-incoming-email-security-gateway-with-amazon-workmail/

This blog post will walk you through the steps needed to integrate Amazon WorkMail with an email security gateway. Configuring WorkMail this way can provide a versatile defense strategy for inbound email threats.

Amazon WorkMail is a secure, managed business email and calendar service. WorkMail leverages the email receiving capabilities of Amazon Simple Email Service (Amazon SES) to scan all incoming and outgoing email for spam, malware, and viruses to help protect your users from harmful email. AWS Lambda for Amazon WorkMail functions can tap into the capabilities of other AWS services to accomplish additional business objectives, such as controlling message delivery or message modification.

For many organizations, existing features and integrations with Amazon SES are sufficient for their spam, malware, and virus detection. Other organizations may need either a dedicated on-premise security solution, or have other reasons to use an additional inspection point in the overall mail flow. A number of commercial and community-supported tools include features like special encryption capabilities, data loss prevention (DLP) content inspection engines, and embedded-hyperlink transformation features to protect end-user mailboxes.

Prerequisites

To implement this solution, you need:

  • A domain name and permission to alter domain name system (DNS) records in Amazon Route 53 or your existing DNS provider. This could be your organization’s existing domain (such as example.org), a new domain (such as example.net), or a subdomain (such as sub.example.org).
  • Access to an AWS account so you can configure WorkMail and Amazon SES. Optionally, you may also need the ability to create AWS Lambda functions to integrate with WorkMail.
  • Access to configure the email security gateway of your choosing.

How email flows with an email security gateway

Email security gateways function by handling the initial ingress of email via the Simple Mail Transport Protocol (SMTP). When email servers send messages to your domain’s email addresses, they look at your domain’s mail exchange (MX) record in the DNS. After processing an email message, the email security gateway delivers it to the downstream mailbox hosting service, such as WorkMail, by means of Amazon SES via SMTP. You can also optionally configure an AWS Lambda for Amazon WorkMail function to synchronously deliver messages into end-user junk email folders, or to take other actions. 

Figure 1. Interaction points while architecting an email security gateway

Figure 1. Interaction points while architecting an email security gateway

The interaction points are as follows:

  1. The email sender looks up the mail exchange (MX) record for the domain hosted by WorkMail. The domain name system (DNS) domain may be hosted in Route 53, or by another DNS hosting provider. The value of the MX record contains the internet protocol (IP) address of the email security gateway.
  2. The email sender connects to the email security gateway, and sends the message using the Simple Mail Transfer Protocol (SMTP)
  3. The email security gateway accepts, processes, and then delivers the message to the ingress SMTP endpoint for WorkMail. Amazon Simple Email Service (Amazon SES) handles inbound email receiving for WorkMail.
  4. Optionally, an AWS Lambda for Amazon WorkMail function can synchronously process messages before delivery to WorkMail.
  5. WorkMail receives the message for final delivery to the end-user.

The gateway assumes responsibility for inspecting incoming email, because the initial point of ingress is an important component of a multi-layer defense strategy against email-borne threats. The gateway could refuse or quarantine risky messages, it could modify the email subjects and body to add warnings visible to recipients, or it could append metadata to the email headers for downstream processing by an AWS Lambda function.

Why point of ingress email authentication is important

What is email authentication

SMTP was built at a time when networking was less reliable than it is today, and consequently, it was designed to be able to allow any domain to store and later forward messages on behalf of other domains to mitigate connection problems. While that helped at the time, today it presents real problems in authenticating who truly sent a message: the owner of the domain, or just someone else claiming to be the owner? To overcome this issue, the messaging industry has adopted three protocols to help verify the authenticity of a message: SPF, DKIM, and DMARC. These protocols aren’t perfect, but understanding how to use them is important when adding new steps to your message processing workflow, because they can affect how you receive inbound mail.

Sender Policy Framework

Sender Policy Framework (SPF) permits domain owners to declare which SMTP servers are allowed to send email messages claiming to be from their domain. This establishes an identity relationship between the owner of the domain and the authorized party that controls the SMTP server. When SPF is used, a message can only be handed off directly from an authorized SMTP server; it cannot be relayed through a second, unauthorized server without changing the originating address.

DomainKeys Identified Mail (DKIM)

DomainKeys Identified Mail (DKIM) permits domain owners to advertise a public key that a mail recipient’s system can use to verify the sender’s digital signature. This allows SMTP servers and other downstream applications to check the validity of the digital signature against the public key of the domain which had the matching private key to create the signature. DKIM signatures attached to messages can remain intact through intermediary SMTP servers, but if message contents (email body or email headers) are modified by intermediary servers, the final destination will find that the signature is no longer valid.

Domain-based Message Authentication, Reporting and Conformance (DMARC)

Domain-based Message Authentication, Reporting and Conformance (DMARC) permits domain owners to publish a policy telling receiving servers what to do when SPF or DKIM are not valid, such as if a message originated from an unauthorized server, or if it was tampered with after being sent. DMARC checks if a message matches what it knows about the sender via SPF and DKIM, a process known as alignment. The recipient’s server can then enforce the DMARC policy, for example by rejecting or quarantining non-aligned messages.

Tying it all together

Amazon WorkMail normally performs DMARC enforcement for inbound messages, based on their alignment when they were received by Amazon SES. But when an email security gateway acts as an intermediary SMTP server between the original sender and WorkMail, that breaks the relationship with the SMTP servers authorized by SPF, and if the gateway modifies the message, that invalidates the DKIM signature. This is why it’s necessary for the SMTP server at the point of ingress to perform the evaluation of SPF, DKIM, and DMARC. The email security gateway at the border should be made responsible for enforcing DMARC on messages that don’t align, and WorkMail DMARC enforcement should be disabled.

Figure 2. Diagram of SPF policy enforcement process. The full details of the interaction points are outlined below.

Figure 2. Diagram of SPF policy enforcement process. The full details of the interaction points are outlined below.

The interaction points for SPF policy enforcement are as follows:

  1. The email sender delivers the message to the email security gateway with a MAIL FROM address within a domain the sender owns.
  2. The email security gateway looks up the sender’s domain’s Sender Permitted From (SPF) policy in DNS to determine if the sending mail server’s IP address is authorized.
  3. The email security gateway delivers the message to Amazon SES with the same MAIL FROM address. The email security gateway has a different IP address than the original sending email server.
  4. When Amazon SES looks up the MAIL FROM domain’s SPF, it will not find the email security gateway’s IP address as authorized. From the perspective of Amazon SES, and the resulting logs in Amazon Cloudwatch, the message will appear to be unauthorized by the SPF policy. This result is ignored by disabling DMARC checks in the WorkMail organization configuration.
  5. The message continues delivery to WorkMail with an optional integration with AWS Lambda for Amazon WorkMail synchronous run configuration, which can analyze message headers to get a more complete picture of the message’s authenticity.

Choosing an email security gateway

Many email security vendors offer software as a service (SaaS) solutions. This offloads all management responsibilities to the software vendor’s platform. These solutions work as long as they support the email gateway features necessary for this solution as depicted in Figure 2 and described in the Why point of ingress email authentication is important section above.

If you wish to build and maintain your own email security gateway, you may deploy one available from the AWS Marketplace or add an open source application into your Amazon Virtual Private Cloud (Amazon VPC). You will need to remove port 25 restriction from your EC2 instance for the email security gateway within your Amazon VPC to send email to Amazon WorkMail.

How to configure Amazon WorkMail

Follow this procedure to configure your WorkMail organization and Amazon SES IP address filters to allow the email security gateway to process inbound email receiving.

To configure Amazon WorkMail

  1. From the WorkMail console, select your organization, navigate to Organization settings, and select Advanced. Edit the Inbound DMARC Settings and set Enforcement enabled to Off. This ensures that WorkMail does not re-evaluate DMARC.

    Figure 3. Picture of the AWS WorkMail console showing the advanced configuration depicting Inbound DMARC enforcement disabled

    Figure 3. Picture of the AWS WorkMail console showing the advanced configuration depicting Inbound DMARC enforcement disabled

  2. From the Amazon SES console, navigate to Email receiving and create IP address filters to allow the IP address or IP address range of the gateway(s).
  3. Add another rule to block 0.0.0.0/0. This prevents malicious actors from bypassing your email security gateway.

    Figure 4. Picture of the Amazon SES console showing example IP address filters to allow the email security gateway IP addresses and blocking every other IP Address

    Figure 4. Picture of the Amazon SES console showing example IP address filters to allow the email security gateway IP addresses and blocking every other IP Address

  4. From the Route 53 console, navigate to Hosted zones, select the domain and edit the MX record to the IP address or hostname of the gateway. This causes all email senders to deliver to your gateway, instead of delivering directly to WorkMail.
    Follow the instructions of your DNS provider if the domain’s DNS is hosted elsewhere.

    Figure 5. Picture of the Amazon Route 53 console showing that the DMS MX record for the domain needs to be configured with the IP address of the email security gateway

    Figure 5. Picture of the Amazon Route 53 console showing that the DMS MX record for the domain needs to be configured with the IP address of the email security gateway

  5. From the WorkMail console, navigate to Domains, select your domain to show the Domain verification status page.
  6. Copy the host name from the value of the MX record type as depicted in Figure 6.

    Figure 6. Picture showing the section of the MX record value to copy from the WorkMail console.

    Figure 6. Picture showing the section of the MX record value to copy from the WorkMail console.

  7. Configure your email security gateway with the value that you just copied (e.g. inbound-smtp.us-east-1.amazonaws.com) to send inbound messages to your WorkMail organization. Instructions for doing this will vary depending on which email security gateway you are using.

Some specifics of this configuration depend on which gateway you are using, how it is designed for high availability, and the type of features configured. Test your WorkMail integration with a non-production domain before changing your production domain.

It is normal for Amazon CloudWatch logs for WorkMail, as well as the individual message headers, to show that SPF fails for all messages which traverse the gateway. Configure the email security gateway to record its SPF evaluation in the message headers so that it remains available for troubleshooting and further processing.

Junk E-Mail folder integration

WorkMail normally moves spam messages into the Junk E-mail folder within each recipient’s mailbox. To replicate this behavior for spam messages identified by the email security gateway, use AWS Lambda for Amazon WorkMail with a synchronous run configuration to run a function for every inbound message to a group of recipients.

To configure an AWS Lambda function for every inbound message (optional)

  1. Configure the email security gateway to include a spam verdict in the message headers for all incoming mail.
  2. Create a synchronous run configuration using AWS Lambda for Amazon WorkMail  to interpret the message headers and return a response to WorkMail with type: BYPASS_SPAM_CHECK or MOVE_TO_JUNK. A sample Amazon WorkMail Upstream Gateway Filter solution is available in the AWS Serverless Application Repository.

Conclusion

By integrating an email security gateway and leveraging AWS Lambda for Amazon WorkMail, you will gain additional security and management control over your organization’s inbound email. To learn more, read the Amazon WorkMail FAQs and Amazon WorkMail documentation.

 
If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security news? Follow us on Twitter.

Jesse Thompson

Jesse Thompson

Jesse is a worldwide public sector senior solutions architect at AWS. His background is in public sector enterprise IT development and operations, with a focus on abuse mitigation and encouragement of authenticity practices with open standard protocols.

Amazon Simple Email Service Celebrates 50 Years of Email

Post Syndicated from Matt Strzelecki original https://aws.amazon.com/blogs/messaging-and-targeting/amazon-simple-email-service-celebrates-50-years-of-email/

Email as we know it turns 50 years old this month (October 2021). The first email sent over a network — the beginning of email as we use it today — was sent in October 1971, by MIT graduate Ray Tomlinson (April 23, 1941–March 5, 2016). Tomlinson was the first to use the @ symbol to identify a message recipient on a remote computer system. Using this address format, he became the first person to send an email between two computers. That first email traveled 10 feet between two computers in Cambridge, Massachusetts. Tomlinson stated when interviewed that the first email was “something like QWERTYUIOP”.

Tomlinson leveraged existing software at the time, including SNDMSG and CPYNET, which allowed people to send messages to others who used the same computer, to send the first email over a network – back then multiple users would share computers, rather than having their own dedicated computers. His work enabled the exchange of messages between computers for the first time. Creating email was a side project at work for Tomlinson, and when he showed his work to another employee for the first time, he reportedly said: “Don’t tell anyone! This isn’t what we’re supposed to be working on.”

Ray Tomlinson was inducted into the Internet Hall of Fame in 2012, and his work is ranked fourth in Boston Globe’s top 150 MIT-related “Ideas, Inventions, and Innovators”.

According to the Guinness Book of Records, the first unsolicited email was sent in May 1978 to 397 recipients advertising an upcoming a product demonstration of computers. That’s right—spam is almost as old as email itself! In 1991, the first email was sent from space by astronauts on the NASA shuttle Atlantis. That message began with “Hello Earth!” and was delivered to Mission Control at the Johnson Space Center in Houston, Texas.

Over the past 50 years, there’s been a lot of firsts in email. For us at Amazon Simple Email Service (Amazon SES), our email first was when we launched our service back in January 2011. We initially started as a service that delivered email for Amazon.com, and grew over time into launching as a public service in Amazon Web Services (AWS).

Customers told us that building large-scale email solutions to send marketing and transactional messages was often a complex and costly challenge for businesses. Amazon SES eliminates these challenges and enables businesses to benefit from the years of experience and sophisticated email infrastructure Amazon.com has built to serve its own large-scale customer base. With Amazon.com being our first customer, from day one – scalability, reliability, and deliverability have been our highest priorities. This same service has also powered the email sending capabilities of Amazon Pinpoint since 2017, as well as email-related features in several other AWS services.

Today, Amazon SES is a cost-effective, flexible, and scalable email service that enables developers to send mail from within any application – supporting multiple email use cases, including transactional, marketing, or mass email communications, as well as inbound email.

We encourage our readers to share their own stories of their email firsts, or any other interesting email anecdotes. #QWERTYUIOP #50yrsofemail

Easily creating and routing email addresses with Cloudflare Email Routing

Post Syndicated from Joao Sousa Botto original https://blog.cloudflare.com/introducing-email-routing/

Easily creating and routing email addresses with Cloudflare Email Routing

Easily creating and routing email addresses with Cloudflare Email Routing

Over four billion people — or half of the world’s population — have at least one email address, many of whom use it as an essential tool to stay on top of their personal and professional matters. More than 300 billion emails are sent and received every day, but seeing email as just a communications tool wouldn’t do it justice. Its impact in our lives goes far beyond being a vessel for messages — its use cases also cover being a common way of validating one’s identity online, and serving as the gateway for other communication platforms.

Today, most people use their email for sensitive purposes, such as logging in to their bank account, or communicating with governmental entities. At the same time, they will use that email to sign up for a 10% off coupon they found online, which will surely spam them for months to come. Despite these two use cases being polar opposites in relation to importance and security, people take the risk, usually for the sake of conveniently managing one account.

Much in the same way, businesses want to have different email addresses for different types of inquiries, such as sales and support, but often find it cumbersome to control who receives these emails. And as the business evolves, matters that were handled by the owner often need to be handed off to other people. But for small businesses it’s not usually straightforward to configure mailboxes and aliases.

And then there are countless individuals and families that juggle multiple mailboxes to handle the Internet identities that they use, to represent their various online presences.

We understand these challenges, and that’s why we’re launching Cloudflare Email Routing, the most straightforward way to create any number of email addresses that are redirected to the mailbox you, your family or your team are already using.

Cloudflare Email Routing

Cloudflare Email Routing is designed to simplify the way you create and manage email addresses, without needing to keep an eye on additional mailboxes.

The process is simple:

  1. You enter the email address you want to create on your domain
  2. You enter the email address you want it forwarded to
Easily creating and routing email addresses with Cloudflare Email Routing

As some of you already know, email consists of the envelope, the header, and its body.

The envelope is part of the SMTP transport protocol and tells the servers where the email is coming from and where it’s supposed to be delivered.

The headers contain structured information like the message traveled path, date, the sender and recipients’ addresses, subject, and other technical metadata like SPF pass results, DKIM signatures, and anti-spam scores. Every time the message goes through a server, it can add new headers or modify the existing ones until it reaches the final inbox.

And finally, there’s the body of the message, where the actual content resides. The body can be plain text, rich HTML, it can contain file attachments, and in some cases, it can be signed or even encrypted.

Here’s a simplified diagram of how the SMTP protocol works and how the three steps of an Email message fit together:

Easily creating and routing email addresses with Cloudflare Email Routing

Cloudflare Email Routing service acts as an intelligent router at the transport layer, handling and modifying the SMTP envelope to deliver the message at its final destination but preserving the original headers and keeping the body intact. This approach ensures that things like SPF, DKIM, and other security or anti-spam protocols don’t break and the recipient stays protected.

Furthermore, following the same privacy-first approach we use in other products, we don’t look into, queue, or store emails at any point. Messages are received, handled according to the configured rules, and delivered to their final destinations in real-time.

Private Beta access

Email Forwarding is now in private beta, and you can save your place in line through this sign-up form.

Then, to start using Email Routing, all you need to do is to add your domain to Cloudflare DNS.

If you don’t currently own a domain, you can buy one right here on our registrar.

Step-by-Step Configuration

Then there are only a few steps to creating a new email address and setting up forwarding:

  1. Go to the email page on the Cloudflare dashboard.
  2. Select Configure.
  3. Enter the email address you want to create (remember, this is for your domain, so you can pick anything you like). Alternatively, you can choose to use a catch-all so that all possible emails addresses in your domain are considered valid and forwarded.
  4. The DNS configuration step is automatic if you don’t have email configured for your domain. Otherwise, we provide straightforward guidance on how to best configure it for your needs.
  5. Lastly, you just need to validate that the destination email belongs to you. Super simple, and exactly the same as you’ve done a million times before.

We did say we made it straightforward!

With efficiency and simplicity in mind, we’re launching Email Routing with support for multiple rules and message forwarding to any upstream inbox of your choice.

However, we feel like the email scene has been long-dormant, and we have exciting new features coming soon that advantage of the Cloudflare platform, resources, and know-how.

We’re also listening. If you have questions, suggestions, or new ideas, share them in the community forum. We’ll be around.

To start using Cloudflare Email Routing just join the waitlist today through this form. We will be opening up this service to more users on a daily basis, and promise the short wait will be worth it!

Forwarding emails automatically based on content with Amazon Simple Email Service

Post Syndicated from Murat Balkan original https://aws.amazon.com/blogs/messaging-and-targeting/forwarding-emails-automatically-based-on-content-with-amazon-simple-email-service/

Introduction

Email is one of the most popular channels consumers use to interact with support organizations. In its most basic form, consumers will send their email to a catch-all email address where it is further dispatched to the correct support group. Often, this requires a person to inspect content manually. Some IT organizations even have a dedicated support group that handles triaging the incoming emails before assigning them to specialized support teams. Triaging each email can be challenging, and delays in email routing and support processes can reduce customer satisfaction. By utilizing Amazon Simple Email Service’s deep integration with Amazon S3, AWS Lambda, and other AWS services, the task of categorizing and routing emails is automated. This automation results in increased operational efficiencies and reduced costs.

This blog post shows you how a serverless application will receive emails with Amazon SES and deliver them to an Amazon S3 bucket. The application uses Amazon Comprehend to identify the dominant language from the message body.  It then looks it up in an Amazon DynamoDB table to find the support group’s email address specializing in the email subject. As the last step, it forwards the email via Amazon SES to its destination. Archiving incoming emails to Amazon S3 also enables further processing or auditing.

Architecture

By completing the steps in this post, you will create a system that uses the architecture illustrated in the following image:

Architecture showing how to forward emails by content using Amazon SES

The flow of events starts when a customer sends an email to the generic support email address like [email protected]. This email is listened to by Amazon SES via a recipient rule. As per the rule, incoming messages are written to a specified Amazon S3 bucket with a given prefix.

This bucket and prefix are configured with S3 Events to trigger a Lambda function on object creation events. The Lambda function reads the email object, parses the contents, and sends them to Amazon Comprehend for language detection.

Amazon DynamoDB looks up the detected language code from an Amazon DynamoDB table, which includes the mappings between language codes and support group email addresses for these languages. One support group could answer English emails, while another support group answers French emails. The Lambda function determines the destination address and re-sends the same email address by performing an email forward operation. Suppose the lookup does not return any destination address, or the language was not be detected. In that case, the email is forwarded to a catch-all email address specified during the application deployment.

In this example, Amazon SES hosts the destination email addresses used for forwarding, but this is not a requirement. External email servers will also receive the forwarded emails.

Prerequisites

To use Amazon SES for receiving email messages, you need to verify a domain that you own. Refer to the documentation to verify your domain with Amazon SES console. If you do not have a domain name, you will register one from Amazon Route 53.

Deploying the Sample Application

Clone this GitHub repository to your local machine and install and configure AWS SAM with a test AWS Identity and Access Management (IAM) user.

You will use AWS SAM to deploy the remaining parts of this serverless architecture.

The AWS SAM template creates the following resources:

  • An Amazon DynamoDB mapping table (language-lookup) contains information about language codes and associates them with destination email addresses.
  • An AWS Lambda function (BlogEmailForwarder) that reads the email content parses it, detects the language, looks up the forwarding destination email address, and sends it.
  • An Amazon S3 bucket, which will store the incoming emails.
  • IAM roles and policies.

To start the AWS SAM deployment, navigate to the root directory of the repository you downloaded and where the template.yaml AWS SAM template resides. AWS SAM also requires you to specify an Amazon Simple Storage Service (Amazon S3) bucket to hold the deployment artifacts. If you haven’t already created a bucket for this purpose, create one now. You will refer to the documentation to learn how to create an Amazon S3 bucket. The bucket should have read and write access by an AWS Identity and Access Management (IAM) user.

At the command line, enter the following command to package the application:

sam package --template template.yaml --output-template-file output_template.yaml --s3-bucket BUCKET_NAME_HERE

In the preceding command, replace BUCKET_NAME_HERE with the name of the Amazon S3 bucket that should hold the deployment artifacts.

AWS SAM packages the application and copies it into this Amazon S3 bucket.

When the AWS SAM package command finishes running, enter the following command to deploy the package:

sam deploy --template-file output_template.yaml --stack-name blogstack --capabilities CAPABILITY_IAM --parameter-overrides FromEmailAddress=info@ YOUR_DOMAIN_NAME_HERE CatchAllEmailAddress=catchall@ YOUR_DOMAIN_NAME_HERE

In the preceding command, change the YOUR_DOMAIN_NAME_HERE with the domain name you validated with Amazon SES. This domain also applies to other commands and configurations that will be introduced later.

This example uses “blogstack” as the stack name, you will change this to any other name you want. When you run this command, AWS SAM shows the progress of the deployment.

Configure the Sample Application

Now that you have deployed the application, you will configure it.

Configuring Receipt Rules

To deliver incoming messages to Amazon S3 bucket, you need to create a Rule Set and a Receipt rule under it.

Note: This blog uses Amazon SES console to create the rule sets. To create the rule sets with AWS CloudFormation, refer to the documentation.

  1. Navigate to the Amazon SES console. From the left navigation choose Rule Sets.
  2. Choose Create a Receipt Rule button at the right pane.
  3. Add info@YOUR_DOMAIN_NAME_HERE as the first recipient addresses by entering it into the text box and choosing Add Recipient.

 

 

Choose the Next Step button to move on to the next step.

  1. On the Actions page, select S3 from the Add action drop-down to reveal S3 action’s details. Select the S3 bucket that was created by the AWS SAM template. It is in the format of your_stack_name-inboxbucket-randomstring. You will find the exact name in the outputs section of the AWS SAM deployment under the key name InboxBucket or by visiting the AWS CloudFormation console. Set the Object key prefix to info/. This tells Amazon SES to add this prefix to all messages destined to this recipient address. This way, you will re-use the same bucket for different recipients.

Choose the Next Step button to move on to the next step.

In the Rule Details page, give this rule a name at the Rule name field. This example uses the name info-recipient-rule. Leave the rest of the fields with their default values.

Choose the Next Step button to move on to the next step.

  1. Review your settings on the Review page and finalize rule creation by choosing Create Rule

  1. In this example, you will be hosting the destination email addresses in Amazon SES rather than forwarding the messages to an external email server. This way, you will be able to see the forwarded messages in your Amazon S3 bucket under different prefixes. To host the destination email addresses, you need to create different rules under the default rule set. Create three additional rules for catchall@YOUR_DOMAIN_NAME_HERE , english@ YOUR_DOMAIN_NAME_HERE and french@YOUR_DOMAIN_NAME_HERE email addresses by repeating the steps 2 to 5. For Amazon S3 prefixes, use catchall/, english/, and french/ respectively.

 

Configuring Amazon DynamoDB Table

To configure the Amazon DynamoDB table that is used by the sample application

  1. Navigate to Amazon DynamoDB console and reach the tables view. Inspect the table created by the AWS SAM application.

language-lookup table is the table where languages and their support group mappings are kept. You need to create an item for each language, and an item that will hold the default destination email address that will be used in case no language match is found. Amazon Comprehend supports more than 60 different languages. You will visit the documentation for the supported languages and add their language codes to this lookup table to enhance this application.

  1. To start inserting items, choose the language-lookup table to open table overview page.
  2. Select the Items tab and choose the Create item From the dropdown, select Text. Add the following JSON content and choose Save to create your first mapping object. While adding the following object, replace Destination attribute’s value with an email address you own. The email messages will be forwarded to that address.

{

  “language”: “en”,

  “destination”: “[email protected]_DOMAIN_NAME_HERE”

}

Lastly, create an item for French language support.

{

  “language”: “fr”,

  “destination”: “[email protected]_DOMAIN_NAME_HERE”

}

Testing

Now that the application is deployed and configured, you will test it.

  1. Use your favorite email client to send the following email to the domain name info@ email address.

Subject: I need help

Body:

Hello, I’d like to return the shoes I bought from your online store. How can I do this?

After the email is sent, navigate to the Amazon S3 console to inspect the contents of the Amazon S3 bucket that is backing the Amazon SES Rule Sets. You will also see the AWS Lambda logs from the Amazon CloudWatch console to confirm that the Lambda function is triggered and run successfully. You should receive an email with the same content at the address you defined for the English language.

  1. Next, send another email with the same content, this time in French language.

Subject: j’ai besoin d’aide

Body:

Bonjour, je souhaite retourner les chaussures que j’ai achetées dans votre boutique en ligne. Comment puis-je faire ceci?

 

Suppose a message is not matched to a language in the lookup table. In that case, the Lambda function will forward it to the catchall email address that you provided during the AWS SAM deployment.

You will inspect the new email objects under english/, french/ and catchall/ prefixes to observe the forwarding behavior.

Continue experimenting with the sample application by sending different email contents to info@ YOUR_DOMAIN_NAME_HERE address or adding other language codes and email address combinations into the mapping table. You will find the available languages and their codes in the documentation. When adding a new language support, don’t forget to associate a new email address and Amazon S3 bucket prefix by defining a new rule.

Cleanup

To clean up the resources you used in your account,

  1. Navigate to the Amazon S3 console and delete the inbox bucket’s contents. You will find the name of this bucket in the outputs section of the AWS SAM deployment under the key name InboxBucket or by visiting the AWS CloudFormation console.
  2. Navigate to AWS CloudFormation console and delete the stack named “blogstack”.
  3. After the stack is deleted, remove the domain from Amazon SES. To do this, navigate to the Amazon SES Console and choose Domains from the left navigation. Select the domain you want to remove and choose Remove button to remove it from Amazon SES.
  4. From the Amazon SES Console, navigate to the Rule Sets from the left navigation. On the Active Rule Set section, choose View Active Rule Set button and delete all the rules you have created, by selecting the rule and choosing Action, Delete.
  5. On the Rule Sets page choose Disable Active Rule Set button to disable listening for incoming email messages.
  6. On the Rule Sets page, Inactive Rule Sets section, delete the only rule set, by selecting the rule set and choosing Action, Delete.
  7. Navigate to CloudWatch console and from the left navigation choose Logs, Log groups. Find the log group that belongs to the BlogEmailForwarderFunction resource and delete it by selecting it and choosing Actions, Delete log group(s).
  8. You will also delete the Amazon S3 bucket you used for packaging and deploying the AWS SAM application.

 

Conclusion

This solution shows how to use Amazon SES to classify email messages by the dominant content language and forward them to respective support groups. You will use the same techniques to implement similar scenarios. You will forward emails based on custom key entities, like product codes, or you will remove PII information from emails before forwarding with Amazon Comprehend.

With its native integrations with AWS services, Amazon SES allows you to enhance your email applications with different AWS Cloud capabilities easily.

To learn more about email forwarding with Amazon SES, you will visit documentation and AWS blogs.

Opt-in to the new Amazon SES console experience

Post Syndicated from Simon Poile original https://aws.amazon.com/blogs/messaging-and-targeting/amazon-ses-console-opt-in/

Amazon Web Services (AWS) is pleased to announce the launch of the newly redesigned Amazon Simple Email Service (SES) console. With its streamlined look and feel, the new console makes it even easier for customers to leverage the speed, reliability, and flexibility that Amazon SES has to offer. Customers can access the new console experience via an opt-in link on the classic console.

Amazon SES now offers a new, optimized console to provide customers with a simpler, more intuitive way to create and manage their resources, collect sending activity data, and monitor reputation health. It also has a more robust set of configuration options and new features and functionality not previously available in the classic console.

Here are a few of the improvements customers can find in the new Amazon SES console:

Verified identities

Streamlines how customers manage their sender identities in Amazon SES. This is done by replacing the classic console’s identity management section with verified identities. Verified identities are a centralized place in which customers can view, create, and configure both domain and email address identities on one page. Other notable improvements include:

  • DKIM-based verification
    DKIM-based domain verification replaces the previous verification method which was based on TXT records. DomainKeys Identified Mail (DKIM) is an email authentication mechanism that receiving mail servers use to validate email. This new verification method offers customers the added benefit of enhancing their deliverability with DKIM-compliant email providers, and helping them achieve compliance with DMARC (Domain-based Message Authentication, Reporting and Conformance).
  • Amazon SES mailbox simulator
    The new mailbox simulator makes it significantly easier for customers to test how their applications handle different email sending scenarios. From a dropdown, customers select which scenario they’d like to simulate. Scenario options include bounces, complaints, and automatic out-of-office responses. The mailbox simulator provides customers with a safe environment in which to test their email sending capabilities.

Configuration sets

The new console makes it easier for customers to experience the benefits of using configuration sets. Configuration sets enable customers to capture and publish event data for specific segments of their email sending program. It also isolates IP reputation by segment by assigning dedicated IP pools. With a wider range of configuration options, such as reputation tracking and custom suppression options, customers get even more out of this powerful feature.

  • Default configuration set
    One important feature to highlight is the introduction of the default configuration set. By assigning a default configuration set to an identity, customers ensure that the assigned configuration set is always applied to messages sent from that identity at the time of sending. This enables customers to associate a dedicated IP pool or set up event publishing for an identity without having to modify their email headers.

Account dashboard

There is also an account dashboard for the new SES console. This feature provides customers with fast access to key information about their account, including sending limits and restrictions, and overall account health. A visual representation of the customer’s daily email usage helps them ensure that they aren’t approaching their sending limits. Additionally, customers who use the Amazon SES SMTP interface to send emails can visit the account dashboard to obtain or update their SMTP credentials.

Reputation metrics

The new reputation metrics page provides customers with high-level insight into historic bounce and complaint rates. This is viewed at both the account level and the configuration set level. Bounce and complaint rates are two important metrics that Amazon SES considers when assessing a customer’s sender reputation, as well as the overall health of their account.

The redesigned Amazon SES console, with its easy-to-use workflows, will not only enhance the customers’ on-boarding experience, it will also change the paradigms used for their on-going usage. The Amazon SES team remains committed to investing on behalf of our customers and empowering them to be productive anywhere, anytime. We invite you to opt in to the new Amazon SES console experience and let us know what you think.

Amazon SES celebrates 10 years of email sending and deliverability

Post Syndicated from Simon Poile original https://aws.amazon.com/blogs/messaging-and-targeting/amazon-ses-celebrates-10-years-of-email-sending-and-deliverability/

Amazon Simple Email Service (Amazon SES) turns 10 years old today. Back on January 25th 2011, Amazon Web Services (AWS) had only 15 services. Today, AWS has grown to over 180 services. Jeff Barr launched Amazon SES as part of his web evangelist blog. Much of what he wrote about then is still true today. Even 10 years later, email is an important channel for customer communications. Developers still want to rely on a trusted global partner to deliver email at scale. However, mailbox providers are even more protective of their end users’ security. They actively work to ensure that any perceived, unwanted email doesn’t make it to the inbox.

Inbox providers use several factors to determine the legitimacy of email traffic. Over the last decade, we have worked diligently to measure many of those factors in Amazon SES to help our customers achieve great deliverability. The focus for much of that work has been a combination of investments into reputation, engagement, and trust. I want to outline what we’ve accomplished to improve your email sending over the last 10 years.

Reputation

Reputation is the measurement mailbox providers use to determine how closely you follow their sending standards. Amazon SES measures perceived reputation through metrics such as bounce rate or complaint rate in the reputation dashboard. The reputation dashboard also shares overall Amazon SES account sending status like “Healthy” or “Under Review.” Some Inbox providers, or ISPs, also provide feedback to help us measure the effectiveness of a specific IP or domain in sending trustworthy traffic.

You can influence reputation in Amazon SES through:

  • Setting up dedicated IPs: Set up IPs in Amazon SES for your own specific sending with appropriate warm-up plans. Split IPs out by use case such as separating password resets from marketing messages.
  • Customer owned IPs (New in 2020): You can now transition IPs you’ve invested in through your own data center or with another ESP to Amazon SES without interruption.
  • Following sending volume best practices: Nothing can flag your IP addresses faster than non-predictable sending patterns. We help you manage this through sending quotas.
  • Use our SES email simulator: Test your application sending without messages leaving the sandbox.

 

Engagement

Engagement is the rate by which customers are interacting with your content. Amazon SES helps you measure engagement through conversion rates (such as open or click-through) and unsubscribe rates. These are measured in the event publishing click stream. This area is more of an art in our deliverability calculus because success varies by industry and use case.

You can influence engagement in Amazon SES through:

  • Customizing content as much as possible, but follow content best practices to avoid setting off content filters. Mailbox providers often utilize behavioral content filtering using AI to determine if your content is relevant based on engagement behavior.
  • Use consent and list management (New in 2020) with customized topics and opt-out pages. It’s important to offer recipients a way to select what emails they want to receive from you and give them an option to opt-out. This is a great new feature that we’ve added based on customer feedback.
  • Remove emails that are not engaging from your lists. Some customers have a time limit, for example, 60 days, before they are automatically removed from an active email list.

 

Trust

Earning trust on email sending is done through the adherence to proper sending behavior, as measured by both individual ISPs as well as industry watch-groups. Trust is closely related to reputation.  We measure trust through messages in the reputation dashboard based on feedback loops, Real-time Blocklists (RBLs), and spam-traps. You can also see the complaint rate associated to your sending in the complaint area of the reputation dashboard. It has statuses like healthy or under review.

You can influence trust in Amazon SES through:

 

Deliverability is a multi-dimensional part of email sending, beyond just setting up an SMTP (Simple Mail Transfer Protocol) endpoint, with constant complexities. But, we’re here to help. In addition to these investments in deliverability, we’ve also expanded Amazon SES to 18 regions, including the government cloud. It’s been an exciting time at AWS, and we look forward to supporting all of our customers in the years to come with Amazon SES.

 

 

 

 

How Amazon Simple Email Service supported the growth of email in 2020

Post Syndicated from Simon Poile original https://aws.amazon.com/blogs/messaging-and-targeting/how-amazon-simple-email-service-supported-the-growth-of-email-in-2020/

Over the last 12 months, organizations of all types have increasingly needed to stay connected to their customers. With the move to virtual interactions accelerating across industries, email has remained a trusted channel for customer communications. Amazon Simple Email Service (SES) has seen record outbound email traffic in 2020, supporting critical customer communications during COVID and commercial moments like Black Friday and Cyber Monday.

The importance of email during COVID

Unlike real-time communications like voice or live chat, email is asynchronous. It can be read and consumed at the customer’s leisure. In some geographies like North America, email also represents an individual’s unique identity, persisting longer than mobile phone numbers or social networking accounts. Even with the importance of email established before 2020, it was important to most organizations to send only the right messages during the COVID crisis.

Many organizations chose to decrease promotional or marketing emails during the pandemic voluntarily. This decrease in sending was to recognize the increased stress most individuals were facing in their personal lives. However, even with the drop in marketing emails across organizational types, there was an increased need to communicate and maintain customer engagement. Most organizations went through three distinct customer communication phases with email in 2020: React, Respond, and Reimagine.

  • React – These were the initial emails sent to acknowledge the COVID crisis, occurring early in 2020. These emails included messages reinforcing commitment to customer health, employee safety, or communicating new cleaning protocols.
  • Respond – These messages often included communication on the status of the business or event. Most businesses needed to communicate their transition to remote work, temporary closures, and many in-person events canceled.
  • Reimagine – Throughout the crisis, organizations were reimagining how to do business. Healthcare started operating video consultations, and restaurants shifted to pick up/take out only. Email communication was vital to take customers on the journey into this “new normal,” even as some businesses started to reopen.

To send these customer communications at scale, many organizations worked with Amazon SES.

How Amazon SES scaled and supported customers in 2020

Amazon SES saw several sending spikes that aligned with organizations working to communicate with their customers during COVID. Nine times in 2020, transactions per second (TPS) in Amazon SES exceeding 150% of the previous record held by 2019 Black Friday. This over 150% TPS spike also occurred on 2020 Black Friday and Cyber Monday.

In addition to supporting those upsurges in throughput, the Amazon SES team also responded to customer feedback on increasing the global footprint of Amazon SES. Since January, Amazon SES increased the total number of regions supported from 7 to 14, including the US government cloud. These additional regions were deployed during 2020 as the team worked remotely. This regional expansion enabled customers to adhere to local data sovereignty requirements for email sending while also improving performance.

Customers also told us they needed tools to help them manage compliance with important governance laws like CAN-SPAM and GDPR. Amazon SES released list management to help organizations manage their customer’s contact information and preferences.

Looking forward

As we move into 2021, email will remain at the forefront of customer communication channels. Enterprise customers like Netflix and Duolingo rely on Amazon SES to deliver their email at scale. For more information on how you can use Amazon SES, visit our website.

Analyze and improve email campaigns with Amazon Simple Email Service and Amazon QuickSight

Post Syndicated from Apoorv Gakhar original https://aws.amazon.com/blogs/messaging-and-targeting/analyze-and-improve-email-campaigns-with-amazon-simple-email-service-and-amazon-quicksight/

Email is a popular channel for applications, used in both marketing campaigns and other outbound customer communications. The challenge with email is that it can become increasingly complex to manage for companies that must send large quantities of messages per month. This complexity is especially true when companies need to measure detailed email engagement metrics to track campaign success.

As a marketer, you want to monitor several metrics, including open rates, click-through rates, bounce rates, and delivery rates. If you do not track your email results, you could potentially be wasting your campaign resources. Monitoring and interpreting your sending results can help you deliver the best content possible to your subscribers’ inboxes, and it can also ensure that your IP reputation stays high. Mailbox providers prioritize inbox placement for senders that deliver relevant content. As a business professional, tracking your emails can also help you stay on top of hot leads and important clients. For example, if someone has opened your email multiple times in one day, it might be a good idea to send out another follow-up email to touch base.

Building a large-scale email solution is a complex and expensive challenge for any business. You would need to build infrastructure, assemble your network, and warm up your IP addresses. Alternatively, working with some third-party email solutions require contract negotiations and upfront costs.

Fortunately, Amazon Simple Email Service (SES) has a highly scalable and reliable backend infrastructure to reduce the preceding challenges. It has improved content filtering techniques, reputation management features, and a vast array of analytics and reporting functions. These features help email senders reach their audiences and make it easier to manage email channels across applications. Amazon SES also provides API operations to monitor your sending activities through simple API calls. You can publish these events to Amazon CloudWatch, Amazon Kinesis Data Firehose, or by using Amazon Simple Notification Service (SNS).

In this post, you learn how to build and automate a serverless architecture that analyzes email events. We explore how to track important metrics such as open and click rate of the emails.

Solution overview

 

The metrics that you can measure using Amazon SES are referred to as email sending events. You can use Amazon CloudWatch to retrieve Amazon SES event data. You can also use Amazon SNS to interpret Amazon SES event data. However, in this post, we are going to use Amazon Kinesis Data Firehose to monitor our user sending activity.

Enable Amazon SES configuration sets with open and click metrics and publish email sending events to Amazon Kinesis Data Firehose as JSON records. A Lambda function is used to parse the JSON records and publish the content in the Amazon S3 bucket.

Ingested data lands in an Amazon S3 bucket that we refer to as the raw zone. To make that data available, you have to catalog its schema in the AWS Glue data catalog. You create and run the AWS Glue crawler that crawls your data sources and construct your Data Catalog. The Data Catalog uses pre-built classifiers for many popular source formats and data types, including JSON, CSV, and Parquet.

When the crawler is finished creating the table definition and schema, you analyze the data using Amazon Athena. It is an interactive query service that makes it easy to analyze data in Amazon S3 using SQL. Point to your data in Amazon S3, define the schema, and start querying using standard SQL, with most results delivered in seconds.

Now you can build visualizations, perform ad hoc analysis, and quickly get business insights from the Amazon SES event data using Amazon QuickSight. You can easily run SQL queries using Amazon Athena on data stored in Amazon S3, and build business dashboards within Amazon QuickSight.

 

Deploying the architecture:

Configuring Amazon Kinesis Data Firehose to write to Amazon S3:

  1. Navigate to the Amazon Kinesis in the AWS Management Console. Choose Kinesis Data Firehose and create a delivery stream.
  2. Enter delivery stream name as “SES_Firehose_Demo”.
  3. Under the source category, select “Direct Put or other sources”.
  4. On the next page, make sure to enable Data Transformation of source records with AWS Lambda. We use AWS Lambda to parse the notification contents that we only process the required information as per the use case.
  5. Click the “Create New” Lambda function.
  6. Click on “General Kinesis Data FirehoseProcessing” Lambda blueprint and this opens up the Lambda console. Enter following values in Lambda
    • Name: SES-Firehose-Json-Parser
    • Execution role: Create a new role with basic Lambda permissions.
  7. Click “Create Function”. Now replace the Lambda code with the following provided code and save the function. 
    • 'use strict';
console.log('Loading function');
exports.handler = (event, context, callback) => {
   /* Process the list of records and transform them */
    const output = event.records.map((record) => {
        console.log(record.recordId);
        const payload =JSON.parse((Buffer.from(record.data, 'base64').toString()))
        console.log("payload : " + payload);
        
        if (payload.eventType == "Click") {
        const resultPayLoadClick = {
                eventType : payload.eventType,
                destinationEmailId : payload.mail.destination[0],
                sourceIp : payload.click.ipAddress,
            };
        console.log("resultPayLoad : " + resultPayLoadClick.eventType + resultPayLoadClick.destinationEmailId + resultPayLoadClick.sourceIp);
        
        //const parsed = resultPayLoad[0];
        //console.log("parsed : " + (Buffer.from(JSON.stringify(resultPayLoad))).toString('base64'));
        
        
        return{
            recordId: record.recordId,
            result: 'Ok',
            data: (Buffer.from(JSON.stringify(resultPayLoadClick))).toString('base64'),
        };
        }
        else {
            const resultPayLoadOpen = {
                eventType : payload.eventType,
                destinationEmailId : payload.mail.destination[0],
                sourceIp : payload.open.ipAddress,
            };
        console.log("resultPayLoad : " + resultPayLoadOpen.eventType + resultPayLoadOpen.destinationEmailId + resultPayLoadOpen.sourceIp);
        
        //const parsed = resultPayLoad[0];
        //console.log("parsed : " + (Buffer.from(JSON.stringify(resultPayLoad))).toString('base64'));
        
        
        return{
            recordId: record.recordId,
            result: 'Ok',
            data: (Buffer.from(JSON.stringify(resultPayLoadOpen))).toString('base64'),
        };
        }
    });
    console.log("Output : " + output.data);
    console.log(`Processing completed.  Successful records ${output.length}.`);
    callback(null, { records: output });
};

      Please note:

      For this blog, we are only filtering out three fields i.e. Eventname, destination_Email, and SourceIP. If you want to store other parameters you can modify your code accordingly. For the list of information that we receive in notifications, you may check out the following document.

      https://docs.aws.amazon.com/ses/latest/DeveloperGuide/event-publishing-retrieving-firehose-examples.html

  8. Now, navigate back to your Amazon Kinesis Data Firehose console and choose the newly created Lambda function.
  9. Keep the convert record format disabled and click “Next”.
  10. In the destination, choose Amazon S3 and select a target Amazon S3 bucket. Create a new bucket if you do not want to use the existing bucket.
  11. Enter the following values for Amazon S3 Prefix and Error Prefix. When event data is published.
    • Prefix:
      fhbase/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/
    • Error Prefix:
      fherroroutputbase/!{firehose:random-string}/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd}/
  12. You may utilize the above values in the Amazon S3 prefix and error prefix. If you use your own prefixes make sure to accordingly update the target values in AWS Glue which you will see in further process.
  13. Keep the Amazon S3 backup option disabled and click “Next”.
  14. On the next page, under the Permissions section, select create a new role. This opens up a new tab and then click “Allow” to create the role.
  15. Navigate back to the Amazon Kinesis Data Firehose console and click “Next”.
  16. Review the changes and click on “Create delivery stream”.

Configure Amazon SES to publish event data to Kinesis Data Firehose:

  1. Navigate to Amazon SES console and select “Email Addresses” from the left side.
  2. Click on “Verify a New Email Address” on the top. Enter your email address to which you send a test email.
  3. Go to your email inbox and click on the verify link. Navigate back to the Amazon SES console and you will see verified status on the email address provided.
  4. Open the Amazon SES console and select Configuration set from the left side.
  5. Create a new configuration set. Enter “SES_Firehose_Demo”  as the configuration set name and click “Create”.
  6. Choose Kinesis Data Firehose as the destination and provide the following details.
    • Name: OpenClick
    • Event Types: Open and Click
  7. In the IAM Role field, select ‘Let SES make a new role’. This allows SES to create a new role and add sufficient permissions for this use case in that role.
  8. Click “Save”.

Sending a Test email:

  1. Navigate to Amazon SES console, click on “Email Addresses” on the left side.
  2. Select your verified email address and click on “Send a Test email”.
  3. Make sure you select the raw email format. You may use the following format to send out a test email from the console. Make sure you send out this email to a recipient inbox to which you have the access. 
    • X-SES-CONFIGURATION-SET: SES_Firehose_Demo
X-SES-MESSAGE-TAGS: Email=NULL
From: [email protected]
To: [email protected]
Subject: Test email
Content-Type: multipart/alternative;
    		boundary="----=_boundary"

------=_boundary
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit
This is a test email.

<a href="https://aws.amazon.com/">Amazon Web Services</a>
------=_boundary
  4. Once the email is received in the recipient’s inbox, open the email and click the link present in the same. This generates a click and open event and send the response back to SES.

Creating Glue Crawler:

  1. Navigate to the AWS Glue console, select “crawler” from the left side, and then click on “Add crawler” on the top.
  2. Enter the crawler name as “SES_Firehose_Crawler” and click “Next”.
  3. Under Crawler source type, select “Data stores” and click “Next”.
  4. Select Amazon S3 as the data source and prove the required path. Include the path until the “fhbase” folder.
  5. Select “no” under Add another data source section.
  6. In the IAM role, select the option to ‘Create an IAM role’. Enter the name as “SES_Firehose-Crawler”. This provides the necessary permissions automatically to the newly created role.
  7. In the frequency section, select run on demand and click “Next”. You may choose this value as per your use case.
  8. Click on add Database and provide the name as “ses_firehose_glue_db”. Click on create and then click “Next”.
  9. Review your Glue crawler setting and click on “Finish”.
  10. Run the above-created crawler. This crawls the data from the specified Amazon S3 bucket and create a catalog and table definition.
  11. Now navigate to “tables” on the left, and verify a “fhbase” table is created after you run the crawler.

If you want to analyze the data stored until now, you can use Amazon Athena and test the queries. If not, you can move to the Amazon Quicksight directly.

Analyzing the data using Amazon Athena:

  1. Open Athena console and select the database, which is created using AWS Glue
  2. Click on “setup a query result location in Amazon S3” as shown in the following screenshot.
  3. Navigate to the Amazon S3 bucket created in earlier steps and create a folder called “AthenaQueryResult”. We store our Athena query result in this bucket.
  4. Now navigate back to Amazon Athena and select the Amazon S3 bucket with the folder location as shown in the following screenshot and click “Save”.
  5. Run the following query to test the sample output and accordingly modify your SQL query to get the desired output.
    • Select * from “ses_firehose_glue_db”.”fhbase”

Note: If you want to track the opened emails by unique Ip addresses then you can modify your SQL query accordingly. This is because every time an email gets opened, you will receive a notification even if the same email was previously opened.

 

Visualizing the data in Amazon QuickSight dashboards:

  1. Now, let’s analyze this data using Amazon Athena via Amazon Quicksight.
  2. Log into Amazon Quicksight and choose Manage data, New dataset. Choose Amazon Athena as a new data source.
  3. Enter the data source name as “SES-Demo” and click on “Create the data source”.
  4. Select your database from the drop-down as “ses_firehose_glue_db” and table “fhbase” that you have created in AWS Glue.
  5. And add a custom SQL based on your use case and click on “Confirm query”. Refer to the example below.
  6. You can perform ad hoc analysis and modify your query according to your business needs as shown in the following image. Click “Save & Visualize”.
  7. You can now visualize your event data on Amazon Quicksight dashboard. You can use various graphs to represent your data. For this demo, the default graph is used and two fields are selected to populate on the graph, as shown below.

 

Conclusion:

This architecture shows how to track your email sending activity at a granular level. You set up Amazon SES to publish event data to Amazon Kinesis Data Firehose based on fine-grained email characteristics that you define. You can also track several types of email sending events, including sends, deliveries, bounces, complaints, rejections, rendering failures, and delivery delays. This information can be useful for operational and analytical purposes.

To get started with Amazon SES, follow this quick start guide and you can learn more about monitoring sending activity here.

About the Authors

Chirag Oswal is a solutions architect and AR/VR specialist working with the public sector India. He works with AWS customers to help them adopt the cloud operating model on a large scale.

Apoorv Gakhar is a Cloud Support Engineer and an Amazon SES Expert. He is working with AWS to help the customers integrate their applications with various AWS Services.

 

Additional Resources:

Amazon SES Dedicated IP Pools

Amazon Personalize optimizer using Amazon Pinpoint events

Template Personalization using Amazon Pinpoint