Post Syndicated from André Cruz original https://blog.cloudflare.com/how-we-built-dmarc-management/

What are DMARC reports

DMARC stands for Domain-based Message Authentication, Reporting, and Conformance. It’s an email authentication protocol that helps protect against email phishing and spoofing.

When an email is sent, DMARC allows the domain owner to set up a DNS record that specifies which authentication methods, such as SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail), are used to verify the email’s authenticity. When the email fails these authentication checks DMARC instructs the recipient’s email provider on how to handle the message, either by quarantining it or rejecting it outright.

DMARC has become increasingly important in today’s Internet, where email phishing and spoofing attacks are becoming more sophisticated and prevalent. By implementing DMARC, domain owners can protect their brand and their customers from the negative impacts of these attacks, including loss of trust, reputation damage, and financial loss.

In addition to protecting against phishing and spoofing attacks, DMARC also provides reporting capabilities. Domain owners can receive reports on email authentication activity, including which messages passed and failed DMARC checks, as well as where these messages originated from.

DMARC management involves the configuration and maintenance of DMARC policies for a domain. Effective DMARC management requires ongoing monitoring and analysis of email authentication activity, as well as the ability to make adjustments and updates to DMARC policies as needed.

Some key components of effective DMARC management include:

• Setting up DMARC policies: This involves configuring the domain’s DMARC record to specify the appropriate authentication methods and policies for handling messages that fail authentication checks. Here’s what a DMARC DNS record looks like:

v=DMARC1; p=reject; rua=mailto:[email protected]

This specifies that we are going to use DMARC version 1, our policy is to reject emails if they fail the DMARC checks, and the email address to which providers should send DMARC reports.

• Monitoring email authentication activity: DMARC reports are an important tool for domain owners to ensure email security and deliverability, as well as compliance with industry standards and regulations. By regularly monitoring and analyzing DMARC reports, domain owners can identify email threats, optimize email campaigns, and improve overall email authentication.
• Making adjustments as needed: Based on analysis of DMARC reports, domain owners may need to make adjustments to DMARC policies or authentication methods to ensure that email messages are properly authenticated and protected from phishing and spoofing attacks.
• Working with email providers and third-party vendors: Effective DMARC management may require collaboration with email providers and third-party vendors to ensure that DMARC policies are being properly implemented and enforced.

Today we launched DMARC management. This is how we built it.

How we built it

As a leading provider of cloud-based security and performance solutions, we at Cloudflare take a specific approach to test our products. We “dogfood” our own tools and services, which means we use them to run our business. This helps us identify any issues or bugs before they affect our customers.

We use our own products internally, such as Cloudflare Workers, a serverless platform that allows developers to run their code on our global network. Since its launch in 2017, the Workers ecosystem has grown significantly. Today, there are thousands of developers building and deploying applications on the platform. The power of the Workers ecosystem lies in its ability to enable developers to build sophisticated applications that were previously impossible or impractical to run so close to clients. Workers can be used to build APIs, generate dynamic content, optimize images, perform real-time processing, and much more. The possibilities are virtually endless. We used Workers to power services like Radar 2.0, or software packages like Wildebeest.

Recently our Email Routing product joined forces with Workers, enabling processing incoming emails via Workers scripts. As the documentation states: “With Email Workers you can leverage the power of Cloudflare Workers to implement any logic you need to process your emails and create complex rules. These rules determine what happens when you receive an email.” Rules and verified addresses can all be configured via our API.

Here’s how a simple Email Worker looks like:

export default {
  async email(message, env, ctx) {
    const allowList = ["[email protected]", "[email protected]"];
    if (allowList.indexOf(message.headers.get("from")) == -1) {
      message.setReject("Address not allowed");
    } else {
      await message.forward("inbox@corp");
    }
  }
}

Pretty straightforward, right?

With the ability to programmatically process incoming emails in place, it seemed like the perfect way to handle incoming DMARC report emails in a scalable and efficient manner, letting Email Routing and Workers do the heavy lifting of receiving an unbound number of emails from across the globe. A high level description of what we needed is:

1. Receive email and extract report
2. Publish relevant details to analytics platform
3. Store the raw report

Email Workers enable us to do #1 easily. We just need to create a worker with an email() handler. This handler will receive the SMTP envelope elements, a pre-parsed version of the email headers, and a stream to read the entire raw email.

For #2 we can also look into the Workers platform, and we will find the Workers Analytics Engine. We just need to define an appropriate schema, which depends both on what’s present in the reports and the queries we plan to do later. Afterwards we can query the data using either the GraphQL or SQL API.

For #3 we don’t need to look further than our R2 object storage. It is trivial to access R2 from a Worker. After extracting the reports from the email we will store them in R2 for posterity.

We built this as a managed service that you can enable on your zone, and added a dashboard interface for convenience, but in reality all the tools are available for you to deploy your own DMARC reports processor on top of Cloudflare Workers, in your own account, without having to worry about servers, scalability or performance.

Architecture

Email Workers is a feature of our Email Routing product. The Email Routing component runs in all our nodes, so any one of them is able to process incoming mail, which is important because we announce the Email ingress BGP prefix from all our datacenters. Sending emails to an Email Worker is as easy as setting a rule in the Email Routing dashboard.

When the Email Routing component receives an email that matches a rule to be delivered to a Worker, it will contact our internal version of the recently open-sourced workerd runtime, which also runs on all nodes. The RPC schema that governs this interaction is defined in a Capnproto schema, and allows the body of the email to be streamed to Edgeworker as it’s read. If the worker script decides to forward this email, Edgeworker will contact Email Routing using a capability sent in the original request.

jsg::Promise<void> ForwardableEmailMessage::forward(kj::String rcptTo, jsg::Optional<jsg::Ref<Headers>> maybeHeaders) {
  auto req = emailFwdr->forwardEmailRequest();
  req.setRcptTo(rcptTo);

  auto sendP = req.send().then(
      [](capnp::Response<rpc::EmailMetadata::EmailFwdr::ForwardEmailResults> res) mutable {
    auto result = res.getResponse().getResult();
    JSG_REQUIRE(result.isOk(), Error, result.getError());
  });
  auto& context = IoContext::current();
  return context.awaitIo(kj::mv(sendP));
}

In the context of DMARC reports this is how we handle the incoming emails:

1. Fetch the recipient of the email being processed, this is the RUA that was used. RUA is a DMARC configuration parameter that indicates where aggregate DMARC processing feedback should be reported pertaining to a certain domain. This recipient can be found in the “to” attribute of the message.

const ruaID = message.to

2. Since we handle DMARC reports for an unbounded number of domains, we use Workers KV to store some information about each one and key this information on the RUA. This also lets us know if we should be receiving these reports.

const accountInfoRaw = await env.KV_DMARC_REPORTS.get(dmarc:${ruaID})

3. At this point, we want to read the entire email into an arrayBuffer in order to parse it. Depending on the size of the report we may run into the limits of the free Workers plan. If this happens, we recommend that you switch to the Workers Unbound resource model which does not have this issue.

const rawEmail = new Response(message.raw)
const arrayBuffer = await rawEmail.arrayBuffer()

4. Parsing the raw email involves, among other things, parsing its MIME parts. There are multiple libraries available that allow one to do this. For example, you could use postal-mime:

const parser = new PostalMime.default()
const email = await parser.parse(arrayBuffer)

5. Having parsed the email we now have access to its attachments. These attachments are the DMARC reports themselves and they can be compressed. The first thing we want to do is store them in their compressed form in R2 for long-term storage. They can be useful later on for re-processing or investigating interesting reports. Doing this is as simple as calling put() on the R2 binding. In order to facilitate retrieval later we recommend that you spread the report files across directories based on the current time.

await env.R2_DMARC_REPORTS.put(
    `${date.getUTCFullYear()}/${date.getUTCMonth() + 1}/${attachment.filename}`,
    attachment.content
  )

6. We now need to look into the attachment mime type. The raw form of DMARC reports is XML, but they can be compressed. In this case we need to decompress them first. DMARC reporter files can use multiple compression algorithms. We use the MIME type to know which one to use. For Zlib compressed reports pako can be used while for ZIP compressed reports unzipit is a good choice.

7. Having obtained the raw XML form of the report, fast-xml-parser has worked well for us in parsing them. Here’s how the DMARC report XML looks:

<feedback>
  <report_metadata>
    <org_name>example.com</org_name>
    <[email protected]</email>
   <extra_contact_info>http://example.com/dmarc/support</extra_contact_info>
    <report_id>9391651994964116463</report_id>
    <date_range>
      <begin>1335521200</begin>
      <end>1335652599</end>
    </date_range>
  </report_metadata>
  <policy_published>
    <domain>business.example</domain>
    <adkim>r</adkim>
    <aspf>r</aspf>
    <p>none</p>
    <sp>none</sp>
    <pct>100</pct>
  </policy_published>
  <record>
    <row>
      <source_ip>192.0.2.1</source_ip>
      <count>2</count>
      <policy_evaluated>
        <disposition>none</disposition>
        <dkim>fail</dkim>
        <spf>pass</spf>
      </policy_evaluated>
    </row>
    <identifiers>
      <header_from>business.example</header_from>
    </identifiers>
    <auth_results>
      <dkim>
        <domain>business.example</domain>
        <result>fail</result>
        <human_result></human_result>
      </dkim>
      <spf>
        <domain>business.example</domain>
        <result>pass</result>
      </spf>
    </auth_results>
  </record>
</feedback>

8. We now have all the data in the report at our fingertips. What we do from here on depends a lot on how we want to present the data. For us, the goal was to display meaningful data extracted from them in our Dashboard. Therefore we needed an Analytics platform where we could push the enriched data. Enter, Workers Analytics Engine. The Analytics engine is perfect for this task since it allows us to send data to it from a worker, and exposes a GraphQL API to interact with the data afterwards. This is how we obtain the data to show in our dashboard.

In the future, we are also considering integrating Queues in the workflow to asynchronously process the report and avoid waiting for the client to complete it.

We managed to implement this project end-to-end relying only on the Workers infrastructure, proving that it’s possible, and advantageous, to build non-trivial apps without having to worry about scalability, performance, storage and security issues.

Open sourcing

As we mentioned before, we built a managed service that you can enable and use, and we will manage it for you. But, everything we did can also be deployed by you, in your account, so that you can manage your own DMARC reports. It’s easy, and free. To help you with that, we are releasing an open-source version of a Worker that processes DMARC reports in the way described above: https://github.com/cloudflare/dmarc-email-worker

If you don’t have a dashboard where to show the data, you can also query the Analytics Engine from a Worker. Or, if you want to store them in a relational database, then there’s D1 to the rescue. The possibilities are endless and we are excited to find out what you’ll build with these tools.

Please contribute, make your own, we’ll be listening.

Final words

We hope that this post has furthered your understanding of the Workers platform. Today Cloudflare takes advantage of this platform to build most of our services, and we think you should too.

Feel free to contribute to our open-source version and show us what you can do with it.

The Email Routing is also working on expanding the Email Workers API more functionally, but that deserves another blog soon.

Post Syndicated from Joao Sousa Botto original https://blog.cloudflare.com/dmarc-management/

At the end of 2021 Cloudflare launched Security Center, a unified solution that brings together our suite of security products and unique Internet intelligence. It enables security teams to quickly identify potential security risks and threats to their organizations, map their attack surface and mitigate these risks with just a few clicks. While Security Center initially focused on application security, we are now adding crucial zero trust insights to further enhance its capabilities.

When your brand is loved and trusted, customers and prospects are looking forward to the emails you send them. Now picture them receiving an email from you: it has your brand, the subject is exciting, it has a link to register for something unique — how can they resist that opportunity?

But what if that email didn’t come from you? What if clicking on that link is a scam that takes them down the path of fraud or identity theft? And what if they think you did it? The truth is, even security minded people occasionally fall for well crafted spoof emails.

That poses a risk to your business and reputation. A risk you don’t want to take – no one does. Brand impersonation is a significant problem for organizations globally, and that’s why we’ve built DMARC Management – available in Beta today.

With DMARC Management you have full insight on who is sending emails on your behalf. You can one-click approve each source that is a legitimate sender for your domain, and then set your DMARC policy to reject any emails sent from unapproved clients.

When the survey platform your company uses is sending emails from your domain, there’s nothing to worry about – you configured it that way. But if an unknown mail service from a remote country is sending emails using your domain that can be quite scary, and something you’ll want to address. Let’s see how.

Anti-spoofing mechanisms

Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM) and Domain-based Message Authentication Reporting and Conformance (DMARC) are three common email authentication methods. Together, they help prevent spammers, phishers, and other unauthorized parties from sending emails on behalf of a domain they do not own.

SPF is a way for a domain to list all the servers the company sends emails from. Think of it like a publicly available employee directory that helps someone to confirm if an employee works for an organization. SPF records list all the IP addresses of all the servers that are allowed to send emails from the domain.

DKIM enables domain owners to automatically “sign” emails from their domain. Specifically, DKIM uses public key cryptography:

1. A DKIM record stores the domain’s public key, and mail servers receiving emails from the domain can check this record to obtain the public key.
2. The private key is kept secret by the sender, who signs the email’s header with this key.
3. Mail servers receiving the email can verify that the sender’s private key was used by applying the public key. This also guarantees that the email was not tampered with while in transit.

DMARC tells a receiving email server what to do after evaluating the SPF and DKIM results. A domain’s DMARC policy can be set in a variety of ways — it can instruct mail servers to quarantine emails that fail SPF or DKIM (or both), to reject such emails, or to deliver them.

It’s not trivial to configure and maintain SPF and DMARC, though. If your configuration is too strict, legitimate emails will be dropped or marked as spam. If it’s too relaxed, your domain might be misused for email spoofing. The proof is that these authentication mechanisms (SPF / DKIM / DMARC) have existed for over 10 years and still, there are still less than 6 million active DMARC records.

DMARC reports can help, and a full solution like DMARC Management reduces the burden of creating and maintaining a proper configuration.

DMARC reports

All DMARC-compliant mailbox providers support sending DMARC aggregated reports to an email address of your choice. Those reports list the services that have sent emails from your domain and the percentage of messages that passed DMARC, SPF and DKIM. They are extremely important because they give administrators the information they need to decide how to adjust their DMARC policies — for instance, that’s how administrators know if their legitimate emails are failing SPF and DKIM, or if a spammer is trying to send illegitimate emails.

But beware, you probably don’t want to send DMARC reports to a human-monitored email address, as these come in fast and furious from virtually every email provider your organization sends messages to, and are delivered in XML format. Typically, administrators set up reports to be sent to a service like our DMARC Management, that boils them down to a more digestible form. Note: These reports do not contain personal identifiable information (PII).

DMARC Management automatically creates an email address for those reports to be sent to, and adds the corresponding RUA record to your Cloudflare DNS to announce to mailbox providers where to send reports to. And yes, if you’re curious, these email addresses are being created using Cloudflare Email Routing.

Note: Today, Cloudflare DNS is a requirement for DMARC Management. Cloudflare Area 1 customers will soon also be able to see DMARC reports even if they’re using third-party DNS services.

As reports are received in this dedicated email address, they are processed by a Worker that extracts the relevant data, parses it and sends it over to our analytics solution. And you guessed again, that’s implemented using Email Workers. You can read more about the technical implementation here.

Taking action

Now that reports are coming in, you can review the data and take action.

Note: It may take up to 24 hours for mailbox providers to start sending reports and for these analytics to be available to you.

At the top of DMARC Management you have an at-a-glance view of the outbound security configuration for your domain, more specifically DMARC, DKIM, and SPF. DMARC Management will soon start reporting on inbound email security as well, which includes STARTTLS, MTA-STS, DANE, and TLS reporting.

The middle section shows the email volume over time, with individual lines showing those that pass DMARC and those that fail.

Below, you have additional details that include the number of email messages sent by each source (per the DMARC reports), and the corresponding DMARC, SPF and DKIM statistics. You can approve (that is, include in SPF) any of these sources by clicking on “…”, and you can easily spot applications that may not have DKIM correctly configured.

Clicking on any source gives you the same DMARC, SPF and DKIM statistics per IP address of that source. This is how you identify if there’s an additional IP address you might need to include in your SPF record, for example.

The ones that fail are the ones you’ll want to take action on, as they will need to either be approved (which technically means including in the SPF record) if legitimate, or stay unapproved and be rejected by the receiving server when the DMARC policy is configured with p=reject.

Getting to a DMARC reject policy is the goal, but you don’t want to apply such a restrictive policy until you have high confidence that all legitimate sending services are accounted for in SPF (and DKIM, if appropriate). That may take a few weeks, depending on the number of services you have sending messages from your domain, but with DMARC Management you will quickly grasp when you’re ready to go.

What else is needed

Once you have approved all your authorized email senders (sources) and configured DMARC to quarantine or reject, you should be confident that your brand and organization are much safer. From then on, keeping an eye on your approved sources list is a very lightweight operation that doesn’t take more than a few minutes per month from your team. Ideally, when new applications that send emails from your domain are deployed in your company, you would proactively include the corresponding IP addresses in your SPF record.

But even if you don’t, you will find new unapproved senders notices on your Security Center, under the Security Insights tab, alongside other important security issues you can review and manage.

Or you can check the unapproved list on DMARC Management every few weeks.

Whenever you see a legitimate sender source show up as unapproved, you know what to do — click “…” and mark them as approved!

What’s coming next

DMARC Management takes email security to the next level, and this is only the beginning.

We’re excited to demonstrate our investments in features that provide customers even more insight into their security. Up next we’ll be connecting security analytics from Cloudflare’s Cloud Access Security Broker (CASB) into the Security Center.

This product integration will provide customers a way to understand the status of their wider SaaS security at a glance. By surfacing the makeup of CASB Findings (or security issues identified in popular SaaS apps) by severity, health of the SaaS integration, and the number of hidden issues, IT and security administrators will have a way to understand the status of their wider security surface area from a single source.

Stay tuned for more news on CASB in Security Center. In the meantime you can join the waitlist for DMARC Management beta for free today and, if you haven’t yet, we recommend you also check out Cloudflare Area 1 and request a Phishing Risk Assessment to block phishing, spoof and spam emails from coming into your environment.