Post Syndicated from Let's Encrypt original https://letsencrypt.org/2022/09/07/new-life-for-crls.html

This month, Let’s Encrypt is turning on new infrastructure to support revoking certificates via Certificate Revocation Lists. Despite having been largely supplanted by the Online Certificate Status Protocol for over a decade now, CRLs are gaining new life with recent browser updates. By collecting and summarizing CRLs for their users, browsers are making reliable revocation of certificates a reality, improving both security and privacy on the web. Let’s talk about exactly what this new infrastructure does, and why it’s important.

A Brief History of Revocation

When a certificate becomes untrustworthy (for instance because its private key was compromised), that certificate must be revoked and that information publicized so that no one relies upon it in the future. However, it’s a well-worn adage in the world of the Web Public Key Infrastructure (the Web PKI) that revocation is broken. Over the history of the Web PKI, there have been two primary mechanisms for declaring that a TLS/SSL certificate should no longer be trusted: Certificate Revocation Lists (CRLs) and the Online Certificate Status Protocol (OCSP). Unfortunately, both have major drawbacks.

CRLs are basically just lists of all of the certificates that a given Certificate Authority (CA) has issued which have been revoked. This means that they’re often very large – easily the size of a whole movie. It’s inefficient for your browser to download a giant list of revoked certificates just to check if the single certificate for the site you’re visiting right now is revoked. These slow downloads and checks made web page loads slow, so OCSP was developed as an alternative.

OCSP is sort of like “what if there were a separate CRL for every single certificate”: when you want to check whether a given certificate has been revoked, your browser can check the status for just that one certificate by contacting the CA’s OCSP service. But because OCSP infrastructure has to be running constantly and can suffer downtime just like any other web service, most browsers treat getting no response at all as equivalent to getting a “not revoked” response. This means that attackers can prevent you from discovering that a certificate has been revoked simply by blocking all of your requests for OCSP information. To help reduce load on a CA’s OCSP services, OCSP responses are valid and can be cached for about a week. But this means that clients don’t retrieve updates very frequently, and often continue to trust certificates for a week after they’re revoked. And perhaps worst of all: because your browser makes an OCSP request for every website you visit, a malicious (or legally compelled) CA could track your browsing behavior by keeping track of what sites you request OCSP for.

So both of the existing solutions don’t really work: CRLs are so inefficient that most browsers don’t check them, and OCSP is so unreliable that most browsers don’t check it. We need something better.

Browser-Summarized CRLs

One possible solution that has been making headway recently is the idea of proprietary, browser-specific CRLs. Although different browsers are implementing this differently (e.g. Mozilla calls theirs CRLite, and Chrome’s are CRLSets), the basic idea is the same.

Rather than having each user’s browser download large CRLs when they want to check revocation, the browser vendor downloads the CRLs centrally. They process the CRLs into a smaller format such as a Bloom filter, then push the new compressed object to all of the installed browser instances using pre-existing rapid update mechanisms. Firefox, for example, is pushing updates as quickly as every 6 hours.

This means that browsers can download revocation lists ahead of time, keeping page loads fast and mitigating the worst problems of vanilla CRLs. It keeps revocation checks local, and the pushed updates can take immediate effect without waiting for a potentially days-long OCSP cache to expire, preventing all of the worst problems with OCSP.

Thanks to the promise of these browser-summarized CRLs, both the Apple and Mozilla root programs are requiring that all CAs begin issuing CRLs before October 1st, 2022. Specifically, they are requiring that CAs begin issuing one or more CRLs which together cover all certificates issued by that CA, and that the list of URLs pointing to those CRLs be disclosed in the Common CA Database (CCADB). This will allow Safari and Firefox to switch to using browser-summarized CRL checking for revocation.

Our New Infrastructure

When Let’s Encrypt was founded, we made an explicit decision to only support OCSP and not produce CRLs at all. This was because the root program requirements at the time only mandated OCSP, and maintaining both revocation mechanisms would have increased the number of places where a bug could lead to a compliance incident.

When we set out to develop CRL infrastructure, we knew we needed to build for scale, and do so in a way that reflects our emphasis on efficiency and simplicity. Over the last few months we have developed a few new pieces of infrastructure to enable us to publish CRLs in compliance with the upcoming requirements. Each component is lightweight, dedicated to doing a single task and doing it well, and will be able to scale well past our current needs.

Let’s Encrypt currently has over 200 million active certificates on any given day. If we had an incident where we needed to revoke every single one of those certificates at the same time, the resulting CRL would be over 8 gigabytes. In order to make things less unwieldy, we will be dividing our CRLs into 128 shards, each topping out at a worst-case maximum of 70 megabytes. We use some carefully constructed math to ensure that – as long as the number of shards doesn’t change – all certificates will remain within their same shards when the CRLs are re-issued, so that each shard can be treated as a mini-CRL with a consistent scope.

In line with the same best practices that we follow for our certificate issuance, all of our CRLs will be checked for compliance with RFC 5280 and the Baseline Requirements before they are signed by our issuing intermediates. Although the popular linting library zlint does not yet support linting CRLs, we have written our own collection of checks and hope to upstream them to zlint in the future. These checks will help prevent compliance incidents and ensure a seamless issuance and renewal cycle.

As part of developing these new capabilities, we have also made several improvements to the Go standard library’s implementation of CRL generation and parsing. We look forward to contributing more improvements as we and the rest of the Go community work with CRLs more frequently in the future.

Although we will be producing CRLs which cover all certificates that we issue, we will not be including those URLs in the CRL Distribution Point extension of our certificates. For now, as required by the Baseline Requirements, our certificates will continue to include an OCSP URL which can be used by anyone to obtain revocation information for each certificate. Our new CRL URLs will be disclosed only in CCADB, so that the Apple and Mozilla root programs can consume them without exposing them to potentially large download traffic from the rest of the internet at large.

The Future of Revocation

There’s still a long way to go before revocation in the Web PKI is truly fixed. The privacy concerns around OCSP will only be mitigated once all clients have stopped relying on it, and we still need to develop good ways for non-browser clients to reliably check revocation information.

We look forward to continuing to work with the rest of the Web PKI community to make revocation checking private, reliable, and efficient for everyone.

If you’re excited about our work developing more robust and private revocation mechanisms, you can support us with a donation, or encourage your company or organization to sponsor our work. As a nonprofit project, 100% of our funding comes from contributions from our community and supporters, and we depend on your support.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2022/05/19/database-to-app-tls.html

Let’s Encrypt has been running a Certificate Transparency (CT) log since 2019 as part of our commitment to keeping the Web PKI ecosystem healthy. CT logs have become important infrastructure for an encrypted Web ¹, but have a well-deserved reputation for being difficult to operate at high levels of trust: Only 5 organizations run logs that are currently considered to be “qualified.” ²

Our Oak log is the only qualified CT log that runs on an entirely open source stack ³. In the interest of lowering the barrier for other organizations to join the CT ecosystem, we want to cover a few recent changes to Oak that might be helpful to anyone else planning to launch a log based on Google’s Trillian backed by MariaDB:

• The disk I/O workload of Trillian atop MariaDB is easily mediated by front-end rate limits, and

• It’s worth the complexity to split each new annual CT log into its own Trillian/MariaDB stack.

This post will update some of the information from the previous post How Let’s Encrypt Runs CT Logs.

Growing Oak While Staying Open Source

Oak runs on a free and open source stack: Google’s Trillian data store, backed by MariaDB, running at Amazon Web Services (AWS) via Amazon’s Relational Database Service (RDS). To our knowledge, Oak is the only trusted CT log without closed-source components ³.

Other operators of Trillian have opted to use different databases which segment data differently, but the provided MySQL-compatible datastore has successfully kept up with Let’s Encrypt’s CT log volume (currently above 400 GB per month). The story for scaling Oak atop MariaDB is quite typical for any relational database, though the performance requirements are stringent.

Keeping Oak Qualified

The policies that Certificate Transparency Log operators follow require there to be no significant downtime, in addition to the more absolute and difficult requirement that the logs themselves make no mistakes: Given the append-only nature of Certificate Transparency, seemingly minor data corruption prompts permanent disqualification of the log ⁴. To minimize the impacts of corruption, as well as for scalability reasons, it’s become normal for CT logs to distribute the certificates they contain in different, smaller individual CT logs, called shards.

Splitting Many Years Of Data Among Many Trees

The Let’s Encrypt Oak CT log is actually made up of many individual CT log shards each named after a period of time: Oak 2020 contains certificates which expired in 2020; Oak 2022 contains certificates which expire in 2022. For ease of reference, we refer to these as “temporal log shards,” though in truth each is an individual CT log sharing the Oak family name.

It is straightforward to configure a single Trillian installation to support multiple CT log shards. Each log shard is allocated storage within the backing database, and the Trillian Log Server can then service requests for all configured logs.

The Trillian database schema is quite compact and easy to understand:

• Each configured log gets a Tree ID, with metadata in several tables.

• All log entries – certificates in our case – get a row in LeafData.

• Entries that haven’t been sequenced yet get a row in the table Unsequenced, which is normally kept empty by the Trillian Log Signer service.

• Once sequenced, entries are removed from the Unsequenced table and added as a row in SequencedLeafData.

In a nutshell: No matter how many different certificate transparency trees and subtrees you set up for a given copy of Trillian, all of them will store the lion’s share of their data, particularly the DER-encoded certificates themselves, interwoven into the one LeafData table. Since Trillian Log Server can only be configured with a single MySQL connection URI, limiting it to a single database, that single table can get quite big.

For Oak, the database currently grows at a rate of about 400 GB per month; that rate is ever-increasing as the use of TLS grows and more Certificate Authorities submit their certificates to our logs.

Amazon RDS Size Limitations

In March 2021 we discovered that Amazon RDS has a 16TB limit per tablespace when RDS is configured to use one file-per-table, as we were doing for all of our CT log shards. Luckily, we reached this limit first in our testing environment, the Testflume log.

Part of Testflume’s purpose was to grow ahead of the production logs in total size, as well as test growth with more aggressive configuration options than the production Oak log had, and in these ways it was highly successful.

Revisiting Database Design

In our blog post, How Let’s Encrypt Runs CT Logs, we wrote that each year we planned “to freeze the previous year’s shard and move it to a less expensive serving infrastructure, reclaiming its storage for our live shards.” However, that is not practical while continuing to serve traffic from the same database instance. Deleting terabytes of rows from an InnoDB table that is in-use is not feasible. Trillian’s MySQL-compatible storage backend agrees: as implemented, Trillian’s built-in Tree Deletion mechanism marks a tree as “soft deleted," and leaves the removal of data from the LeafData table (and others) as an exercise for the administrator.

Since Trillian’s MySQL-compatible backend does not support splitting the LeafData among multiple tables by itself, and since deleting stale data from those tables yields slow performance across the whole database server, to continue to scale the Oak CT log we have to instead prune out the prior seasons’ data another way.

Single RDS Instance with Distinct Schema per Log Shard

We considered adding new database schemas to our existing MariaDB-backed Amazon RDS instance. In this design, we would run a Trillian CT Front-End (CTFE) instance per temporal log shard, each pointing to individual Trillian Log Server and Log Signer instances, which themselves point to a specific temporally-identified database schema name and tablespace. This is cost-effective, and it gives us ample room to avoid the 16 TB limit.

However, if heavy maintenance is required on any part of the underlying database, it would affect every log shard contained within. In particular, we know from using MariaDB with InnoDB inside the Let’s Encrypt CA infrastructure that truncating and deleting a multi-terabyte table causes performance issues for the whole database while the operation runs. Inside the CA infrastructure we mitigate that performance issue by deleting table data only on database replicas; this is more complicated in a more hands-off managed hosting environment like RDS.

Since we wish to clear out old data regularly as a matter of data hygiene, and the performance requirements for a CT log are strict, this option wasn’t feasible.

Distinct RDS Instance per Log Shard

While it increases the number of managed system components, it is much cleaner to give each temporal log shard its own database instance. Like the Distinct Schema per Log Shard model, we now run Trillian CTFE, Log Server, and Log Signer instances for each temporal log shard. However, each log shard gets its own RDS instance for the active life of the log ⁵. At log shutdown, the RDS instance is simply deprovisioned.

With the original specifications for the Oak log, this would require allocating a significant amount of data I/O resources. However, years of experience running the Testflume log showed that Trillian in AWS did not require the highest possible disk performance.

Tuning IOPS

We launched Oak using the highest performance AWS Elastic Block Storage available at the time: Provisioned IOPS SSDs (type io1). Because of the strict performance requirements on CT logs, we worried that without the best possible performance for disk I/O that latency issues might crop up that could lead to disqualification. As we called out in our blog post How Let’s Encrypt Runs CT Logs, we hoped that we could use a simpler storage type in the future.

To test that, we used General Purpose SSD storage type (type gp2) for our testing CT log, Testflume, and obtained nominal results over the lifespan of the log. In practice higher performance was unnecessary because Trillian makes good use of database indices. Downloading the whole log tree from the first leaf entry is the most significant demand of disk I/O, and that manner of operation is easily managed via rate limits at the load balancer layer.

Our 2022 and 2023 Oak shards now use type gp2 storage and are performing well.

Synergistically, the earlier change to run a distinct RDS instance for each temporal log shard has also further reduced Trillian’s I/O load: A larger percentage of the trimmed-down data fits in MariaDB’s in-memory buffer pool.

More Future Improvements

It’s clear that CT logs will continue to accelerate their rate of growth. Eventually, if we remain on this architecture, even a single year’s CT log will exceed the 16 TB table size limit. In advance of that, we’ll have to take further actions. Some of those might be:

• Change our temporal log sharding strategy to shorter-than-year intervals, perhaps every 3 or 6 months.

• Reduce the absolute storage requirements for Trillian’s MySQL-compatible storage backend by de-duplicating intermediate certificates.

• Contribute a patch to add table sharding to Trillian’s MySQL-compatible storage backend.

• Change storage backends entirely, perhaps to a sharding-aware middleware, or another more horizontally-scalable open-source system.

We’ve also uprooted our current Testflume CT log and brought online a replacement which we’ve named Sapling. As before, this test-only log will evaluate more aggressive configurations that might bear fruit in the future.

As Always, Scaling Data Is The Hard Part

Though the performance requirements for CT logs are strict, the bulk of the scalability difficulty has to do with the large amount of data and the high and ever-increasing rate of growth; this is the way of relational databases. Horizontal scaling continues to be the solution, and is straightforward to apply to the open source Trillian and MariaDB stack.

Supporting Let’s Encrypt

As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your
company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. If you can support us with a donation, we ask that you make an individual contribution.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2022/05/19/nurturing-ct-log-growth.html

Let’s Encrypt has been running a Certificate Transparency (CT) log since 2019 as part of our commitment to keeping the Web PKI ecosystem healthy. CT logs have become important infrastructure for an encrypted Web ¹, but have a well-deserved reputation for being difficult to operate at high levels of trust: Only 6 organizations run logs that are currently considered to be “qualified.” ²

Our Oak log is the only qualified CT log that runs on an entirely open source stack ³. In the interest of lowering the barrier for other organizations to join the CT ecosystem, we want to cover a few recent changes to Oak that might be helpful to anyone else planning to launch a log based on Google’s Trillian backed by MariaDB:

• The disk I/O workload of Trillian atop MariaDB is easily mediated by front-end rate limits, and

• It’s worth the complexity to split each new annual CT log into its own Trillian/MariaDB stack.

This post will update some of the information from the previous post How Let’s Encrypt Runs CT Logs.

Growing Oak While Staying Open Source

Oak runs on a free and open source stack: Google’s Trillian data store, backed by MariaDB, running at Amazon Web Services (AWS) via Amazon’s Relational Database Service (RDS). To our knowledge, Oak is the only trusted CT log without closed-source components ³.

Other operators of Trillian have opted to use different databases which segment data differently, but the provided MySQL-compatible datastore has successfully kept up with Let’s Encrypt’s CT log volume (currently above 400 GB per month). The story for scaling Oak atop MariaDB is quite typical for any relational database, though the performance requirements are stringent.

Keeping Oak Qualified

The policies that Certificate Transparency Log operators follow require there to be no significant downtime, in addition to the more absolute and difficult requirement that the logs themselves make no mistakes: Given the append-only nature of Certificate Transparency, seemingly minor data corruption prompts permanent disqualification of the log ⁴. To minimize the impacts of corruption, as well as for scalability reasons, it’s become normal for CT logs to distribute the certificates they contain in different, smaller individual CT logs, called shards.

Splitting Many Years Of Data Among Many Trees

The Let’s Encrypt Oak CT log is actually made up of many individual CT log shards each named after a period of time: Oak 2020 contains certificates which expired in 2020; Oak 2022 contains certificates which expire in 2022. For ease of reference, we refer to these as “temporal log shards,” though in truth each is an individual CT log sharing the Oak family name.

It is straightforward to configure a single Trillian installation to support multiple CT log shards. Each log shard is allocated storage within the backing database, and the Trillian Log Server can then service requests for all configured logs.

The Trillian database schema is quite compact and easy to understand:

• Each configured log gets a Tree ID, with metadata in several tables.

• All log entries – certificates in our case – get a row in LeafData.

• Entries that haven’t been sequenced yet get a row in the table Unsequenced, which is normally kept empty by the Trillian Log Signer service.

• Once sequenced, entries are removed from the Unsequenced table and added as a row in SequencedLeafData.

In a nutshell: No matter how many different certificate transparency trees and subtrees you set up for a given copy of Trillian, all of them will store the lion’s share of their data, particularly the DER-encoded certificates themselves, interwoven into the one LeafData table. Since Trillian Log Server can only be configured with a single MySQL connection URI, limiting it to a single database, that single table can get quite big.

For Oak, the database currently grows at a rate of about 400 GB per month; that rate is ever-increasing as the use of TLS grows and more Certificate Authorities submit their certificates to our logs.

Amazon RDS Size Limitations

In March 2021 we discovered that Amazon RDS has a 16TB limit per tablespace when RDS is configured to use one file-per-table, as we were doing for all of our CT log shards. Luckily, we reached this limit first in our testing environment, the Testflume log.

Part of Testflume’s purpose was to grow ahead of the production logs in total size, as well as test growth with more aggressive configuration options than the production Oak log had, and in these ways it was highly successful.

Revisiting Database Design

In our blog post, How Let’s Encrypt Runs CT Logs, we wrote that each year we planned “to freeze the previous year’s shard and move it to a less expensive serving infrastructure, reclaiming its storage for our live shards.” However, that is not practical while continuing to serve traffic from the same database instance. Deleting terabytes of rows from an InnoDB table that is in-use is not feasible. Trillian’s MySQL-compatible storage backend agrees: as implemented, Trillian’s built-in Tree Deletion mechanism marks a tree as “soft deleted," and leaves the removal of data from the LeafData table (and others) as an exercise for the administrator.

Since Trillian’s MySQL-compatible backend does not support splitting the LeafData among multiple tables by itself, and since deleting stale data from those tables yields slow performance across the whole database server, to continue to scale the Oak CT log we have to instead prune out the prior seasons’ data another way.

Single RDS Instance with Distinct Schema per Log Shard

We considered adding new database schemas to our existing MariaDB-backed Amazon RDS instance. In this design, we would run a Trillian CT Front-End (CTFE) instance per temporal log shard, each pointing to individual Trillian Log Server and Log Signer instances, which themselves point to a specific temporally-identified database schema name and tablespace. This is cost-effective, and it gives us ample room to avoid the 16 TB limit.

However, if heavy maintenance is required on any part of the underlying database, it would affect every log shard contained within. In particular, we know from using MariaDB with InnoDB inside the Let’s Encrypt CA infrastructure that truncating and deleting a multi-terabyte table causes performance issues for the whole database while the operation runs. Inside the CA infrastructure we mitigate that performance issue by deleting table data only on database replicas; this is more complicated in a more hands-off managed hosting environment like RDS.

Since we wish to clear out old data regularly as a matter of data hygiene, and the performance requirements for a CT log are strict, this option wasn’t feasible.

Distinct RDS Instance per Log Shard

While it increases the number of managed system components, it is much cleaner to give each temporal log shard its own database instance. Like the Distinct Schema per Log Shard model, we now run Trillian CTFE, Log Server, and Log Signer instances for each temporal log shard. However, each log shard gets its own RDS instance for the active life of the log ⁵. At log shutdown, the RDS instance is simply deprovisioned.

With the original specifications for the Oak log, this would require allocating a significant amount of data I/O resources. However, years of experience running the Testflume log showed that Trillian in AWS did not require the highest possible disk performance.

Tuning IOPS

We launched Oak using the highest performance AWS Elastic Block Storage available at the time: Provisioned IOPS SSDs (type io1). Because of the strict performance requirements on CT logs, we worried that without the best possible performance for disk I/O that latency issues might crop up that could lead to disqualification. As we called out in our blog post How Let’s Encrypt Runs CT Logs, we hoped that we could use a simpler storage type in the future.

To test that, we used General Purpose SSD storage type (type gp2) for our testing CT log, Testflume, and obtained nominal results over the lifespan of the log. In practice higher performance was unnecessary because Trillian makes good use of database indices. Downloading the whole log tree from the first leaf entry is the most significant demand of disk I/O, and that manner of operation is easily managed via rate limits at the load balancer layer.

Our 2022 and 2023 Oak shards now use type gp2 storage and are performing well.

Synergistically, the earlier change to run a distinct RDS instance for each temporal log shard has also further reduced Trillian’s I/O load: A larger percentage of the trimmed-down data fits in MariaDB’s in-memory buffer pool.

More Future Improvements

It’s clear that CT logs will continue to accelerate their rate of growth. Eventually, if we remain on this architecture, even a single year’s CT log will exceed the 16 TB table size limit. In advance of that, we’ll have to take further actions. Some of those might be:

• Change our temporal log sharding strategy to shorter-than-year intervals, perhaps every 3 or 6 months.

• Reduce the absolute storage requirements for Trillian’s MySQL-compatible storage backend by de-duplicating intermediate certificates.

• Contribute a patch to add table sharding to Trillian’s MySQL-compatible storage backend.

• Change storage backends entirely, perhaps to a sharding-aware middleware, or another more horizontally-scalable open-source system.

We’ve also uprooted our current Testflume CT log and brought online a replacement which we’ve named Sapling. As before, this test-only log will evaluate more aggressive configurations that might bear fruit in the future.

As Always, Scaling Data Is The Hard Part

Though the performance requirements for CT logs are strict, the bulk of the scalability difficulty has to do with the large amount of data and the high and ever-increasing rate of growth; this is the way of relational databases. Horizontal scaling continues to be the solution, and is straightforward to apply to the open source Trillian and MariaDB stack.

Supporting Let’s Encrypt

As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your
company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. If you can support us with a donation, we ask that you make an individual contribution.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2022/04/28/database-to-app-tls.html

Most of the time, people think about using Let’s Encrypt certificates to encrypt the communication between a website and server. But connections that need TLS are everywhere! In order for us to have an Internet that is 100% encrypted, we need to think beyond the website.

MongoDB’s managed multicloud database service, called Atlas, uses Let’s Encrypt certificates to secure the connection between customers’ applications and MongoDB databases, and between service points inside the platform. We spoke with Kenn White, Security Principal at MongoDB, about how his team uses Let’s Encrypt certificates for over two million databases, across 200 datacenters and three cloud providers.

"Let’s Encrypt has become a core part of our infrastructure stack," said Kenn. Interestingly, our relationship didn’t start out that way. MongoDB became a financial sponsor of Let’s Encrypt years earlier simply to support our mission to pursue security and privacy. MongoDB Atlas began to take off and it became clear that TLS would continue to be a priority as they brought on customers like currency exchanges, treasury platforms and retail payment networks. "The whole notion of high automation and no human touch all really appealed to us," said Kenn of MongoDB’s decision to use Let’s Encrypt.

MongoDB’s diverse customer roster means they support a wide variety of languages, libraries, and operating systems. Consequently, their monitoring is quite robust. Over the years, MongoDB has become a helpful resource for Let’s Encrypt engineers to identify edge case implementation bugs. Their ability to accurately identify issues early helps us respond efficiently; this is a benefit that ripples out across our diverse subscribers all over the Web.

The open sharing of information is a core part of how Let’s Encrypt operates. In fact, "transparency" is one of our key operating principles. The ability to see and understand how Let’s Encrypt is changing helped MongoDB gain trust and confidence in our operations. "I don’t think you can really put a price on the experience we’ve had working with the Let’s Encrypt engineering team," said Kenn. "One thing that I appreciate about Let’s Encrypt is that you’ve always been extremely transparent on your priorities and your roadmap vision. In terms of the technology and your telemetry, this is an evolution; where you are today is far better than where you were two years ago. And two years ago you were already head and shoulders above almost every peer in the industry."

Check out other blog posts in this series about how other large subscribers use Let’s Encrypt certificates.

TLS Simply and Automatically for Europe’s Largest Cloud Customers

Speed at scale: Let’s Encrypt serving Shopify’s 4.5 million domains

Supporting Let’s Encrypt

As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your
company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. If you can support us with a donation, we ask that you make an individual contribution.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2022/04/13/receiving-the-levchin-prize.html

On April 13, 2022, the Real World Crypto steering committee presented the
Max Levchin Prize for Real-World Cryptography to Let’s Encrypt. The
following is the speech delivered by our Executive Director, Josh Aas upon
receiving the award. We’d like to
thank our community for supporting us and invite you to join us
in making the Internet more secure and privacy-respecting for everyone.

Thank you to the
Real World Crypto steering committee
and to Max Levchin for this
recognition. I couldn’t be more proud of what our team has accomplished
since we started working on Let’s Encrypt back in 2013.

My first temptation is to name some names, but there are so many people who
have given a significant portion of their lives to this work over the years
that the list would be too long. You know who you are. I hope you’re as
proud as I am at this moment.

Let’s Encrypt is currently used by more than
280 million websites,
issuing between two and three million certificates per day. I often think about how we got here, looking for some nugget of wisdom
that might be useful to others. I’m not sure I’ve really come up with
anything particularly profound, but I’m going to give you my thoughts
anyway. Generally speaking: we started with a pretty good idea, built a
strong team, stayed focused on what’s important, and kept ease of use in
mind every step of the way.

Let’s Encrypt ultimately came from a group of people thinking about a pretty
daunting challenge. The billions of people living increasingly large
portions of their lives online deserved better privacy and security, but in
order to do that we needed to convince hundreds of millions of websites to
switch to HTTPS. Not only did we want them to make that change, we wanted
most of them to make the change within the next three to five years.

We thought through a lot of options but in the end we just didn’t see any
other way than to build what became Let’s Encrypt. In hindsight building
Let’s Encrypt seems like it was a good and rewarding idea, but at the time
it was a frustrating conclusion in many ways. It’s not an easy solution to
commit to. It meant standing up a new organization, hiring at least a dozen
people, understanding a lot of details about how to operate a CA, building
some fairly intense technical systems, and setting all of it up to operate
for decades. Many of us wanted to work on this interesting problem for a
bit, solve it or at least put a big dent in it, and then move on to other
interesting problems. I don’t know about you, but I certainly didn’t dream
about building and operating a CA when I was younger.

It needed to be done though, so we got to work. We built a great team that
initially consisted of mostly volunteers and very few staff. Over time that
ratio reversed itself such that most people working on Let’s Encrypt on a
daily basis are staff, but we’re fortunate to continue to have a vibrant
community of volunteers who do work ranging from translating our website and
providing assistance on our community forums, to maintaining the dozens
(maybe hundreds?) of client software options out there.

Today there are just 11 engineers working on Let’s Encrypt, as well as a
small team handling fundraising, communication, and administrative tasks.
That’s not a lot of people for an organization serving hundreds of millions
of websites in every country on the globe, subject to a fairly intense set
of industry rules, audits, and high expectations for security and
reliability. The team is preparing to serve as many as 1 billion websites.
When that day comes to pass the team will be larger, but probably not much
larger. Efficiency is important to us, for a couple of reasons. The first is
principle – we believe it’s our obligation to do the most good we can with
every dollar entrusted to us. The second reason is necessity – it’s not easy
to raise money, and we need to do our best to accomplish our mission with
what’s available to us.

It probably doesn’t come as a surprise to anyone here at Real World Crypto
that ease of use was critical to any success we’ve had in applying
cryptography more widely. Let’s Encrypt has a fair amount of internal
complexity, but we expose users to as little of that as possible. Ideally
it’s a fully automated and forgettable background task even to the people
running servers.

The fact that Let’s Encrypt is free is a huge factor in ease of use. It
isn’t even about how much money people might be willing or able to pay, but
any financial transaction requirement would make it impossible to fully
automate our service. At some point someone would have to get a credit card
and manage payment information. That task ranges in complexity from finding
your wallet to obtaining corporate approval. The existence of a payment in
any amount would also greatly limit our geographic availability because of
sanctions and financial logistics.

All of these factors led to the decision to form
ISRG, a nonprofit entity to
support Let’s Encrypt. Our ability to provide this global, reliable service
is all thanks to the people and companies who believe in TLS everywhere and
have supported us financially. I’m so grateful to all of our contributors
for helping us.

Our service is pretty easy to use under normal circumstances, but we’re not
done yet. We can be better about handling exceptional circumstances such as
large revocation events. Resiliency is good. Automated, smooth resiliency is
even better. That’s why I’m so excited about the
ACME Renewal Info
work we’re doing in the IETF now, which will go into production over the
next year.

Everyone here has heard it before, but I’ll say it again because we can’t
afford to let it slip our minds. Ease of use is critical for widespread
adoption of real world cryptography. As we look toward the future of ISRG,
our new projects will have ease of use at their core. In fact, you can learn
about our newest project related to privacy-preserving measurement at two of
this afternoon’s sessions! Getting ease of use right is not just about the
software though. It’s a sort of pas de trois, a dance for three, between
software, legal, and finance, in order to achieve a great outcome.

Thank you again. This recognition means so much to us.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2022/02/25/ford-foundation.html

The Internet has considerable potential to help build a more just, equitable, and sustainable world for all people. Yet for everyone online—and indeed the billions not yet online—barriers to secure and privacy-respecting communication remain pervasive.

ISRG was founded in 2013 to find and eliminate these barriers. Today, we’re proud to announce a $1M grant from the Ford Foundation to continue our efforts.

Our first project, Let’s Encrypt, leverages technology whose foundation has existed for nearly three decades—TLS certificates for securely communicating information via HTTP. Yet even for people well-versed in technology, adopting TLS proved daunting.

Before Let’s Encrypt, the growth rate for HTTPS page loads merely puttered along. As recently as 2013, just 25% of websites used HTTPS. In order for the Internet to reach its full potential, this glaring risk to peoples’ security and privacy needed to be mitigated.

Let’s Encrypt changed the paradigm. Today 81% of website page loads use HTTPS. That means that you and the other 4.9 billion people online can leverage the Internet for your own pursuits with a greater degree of security and privacy than ever before.

But TLS adoption was just one hurdle. Much can be done to further improve the Internet’s most critical pieces of technology to be more secure; much can be done to further improve the privacy of everyone using the Internet today.

Building our efforts thanks to transformational support

Ford Foundation’s commitment recognizes that the Internet can be a technological tool to build a more just, equitable, and sustainable world, but that it will take organizations like ISRG to help build it.

“Ford Foundation is one of the most respected grantmaking institutions in the world,” Josh Aas, ISRG Executive Director, said. “We are proud that Ford believes in the impact we’ve created and the potential of our efforts to continue benefiting everyone using the Internet.”

This support, which began in 2021, will help ISRG continue to invest in Let’s Encrypt and our other projects, Prossimo and Divvi Up.

Launched in late 2020, Prossimo intends to move the Internet’s most critical security-sensitive software infrastructure to memory safe code. Society pays the price for these vulnerabilities with privacy violations, staggering financial losses, denial of public services (e.g., hospitals, power grids), and human rights violations. Meaningful effort will be required to bring about such change, but the Internet will be around for a long time. There is time for ambitious efforts to pay off.

Divvi Up is a system for privacy-preserving metrics analysis. With Divvi Up, organizations can analyze and share data to further their aims without sacrificing their users’ privacy. Divvi Up is currently used for COVID-19 Exposure Notification apps and has processed over 14 billion metrics to aid Public Health Authorities to better hone their app to be responsive to their local populations.

"ISRG’s pragmatic, public-interest approach to Internet security has fundamentally changed the web at an astonishing scale and pace,” Michael Brennan of the Ford Foundation said. "I believe their new projects have the same potential and I am eager to see what they turn their sights to next."

We’re grateful to Ford for their support of our efforts, and to all of you who have contributed time and resources to our projects. For more information on ISRG and our projects, take a read through our 2021 Annual Report. 100% of ISRG’s funding comes from contributed sources. If you or your organization are interested in helping advance our mission, consider becoming a sponsor, making a one-time contribution, or reaching out with your idea on how you can help financially support our mission at [email protected].

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2021/12/16/ed-letter-2021.html

This letter was originally published in our 2021 annual report.

We can do a lot to improve security and privacy on the Internet by taking existing ideas and applying them in ways that benefit the general public at scale. Our work certainly does involve some research, as our name implies, but the success that we’ve had in pursuing our mission largely comes from our ability to go from ideas to implementations that improve the lives of billions of people around the world.

Our first major project, Let’s Encrypt, now helps to protect more than 260 million websites by offering free and fully automated TLS certificate issuance and management. Since it launched in 2015, encrypted page loads have gone from under 40% to 92% in the U.S. and 83% globally.

We didn’t invent certificate authorities. We didn’t invent automated issuance and management. We refined those ideas and applied them in ways that benefit the general public at scale.

We launched our Prossimo project in late 2020. Our hope is that this project will greatly improve security and privacy on the Internet by making memory safety vulnerabilities in the Internet’s most critical a thing of the past. We’re bringing a healthy dose of ambition to the table and we’re backing it up with effective strategies and strong partnerships.

Again, we didn’t invent any memory safe languages or techniques, and we certainly didn’t invent memory safety itself. We’re simply taking existing ideas and applying them in ways that benefit the general public at scale. We’re getting the work done.

With our latest project, Divvi Up for Privacy Preserving Metrics (PPM), the core ideas are a bit newer than the ideas behind our other projects, but we didn’t invent them either. Over the past decade or so some bright people have come up with a way to resolve the tension between wanting to collect metrics about populations and needing to collect data about individuals.

We believe those ideas have matured enough that it’s time to deploy them to the public’s benefit. We started by building and deploying a PPM service for Covid-19 Exposure Notification applications in late 2020, in partnership with Apple, Google, the Bill & Melinda Gates Foundation and the Linux Foundation. We’re expanding that service so any application can collect metrics in a privacy-preserving way.

Being ready to bring ideas to life means a few different things.

We need to have an excellent engineering team that knows how to build services at scale. It’s not enough to just build something that works – the quality and reliability of our work needs to inspire confidence. People need to be able to rely on us.

We also need to have the experience, perspective, and capacity to effectively consider ideas. We are not an organization that “throws things at the wall to see what sticks.” Between our staff, our board of directors, our partners, and our community, we’re able to do a great job evaluating opportunities to understand technical feasibility, potential impact, and alignment with our public benefit mission—to reduce financial, technological, and educational barriers to secure communication over the Internet.

Administrative and communications capabilities are essential. From fundraising and accounting to legal and social media, our administrative teams exist in order to support and amplify the critical work that we do. We’re proud to run a financially efficient organization that provides services for billions of people on only a few million dollars each year.

Finally, it means having the financial resources we need to function. As a nonprofit, 100% of our funding comes from charitable contributions from people like you and organizations around the world. But global impact doesn’t necessarily require million dollar checks: since 2015 tens of thousands of people have given to our work. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to $17M that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and support us financially if you can.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2021/10/28/tls-simply-and-automatically.html

OVHcloud, the largest hosting provider in Europe, has used Let’s Encrypt for TLS certificates since 2016. Since then, they’ve provisioned tens of millions of certificates for their shared hosting customers. We often get asked about how large integrations work and their best practices so this will be the first in a series of blog posts we’ll publish on the topic.

OVHcloud first started looking into using Let’s Encrypt certificates because the team saw a need for the protection provided by TLS for every customer (remember, way back five years ago, when that wasn’t just a thing everybody did?). “Our goal was to deliver TLS simply. We didn’t want to have to write a tutorial for our customers to upload a cert, but instead just click and it works,” said Guillaume Marchand, OVHcloud’s Technical Team Lead.

They considered building their own CA but determined the cost and complexity of doing so would be impractical. Instead, they build an ACME client to prepare for using Let’s Encrypt. It took about six months, “we simply followed the RFC and did a bit of reverse engineering of Certbot,” said Guillaume. In addition to a custom client, OVHcloud automated their Certificate Signing Request (CSR) process and certificate installation process.

Getting a TLS certificate is on the critical path to onboarding a shared hosting client, so monitoring is a big part of OVHcloud’s success with Let’s Encrypt. They set up monitoring at every step in the delivery process: requesting the certificate, asking for challenges, waiting for validation, and requesting certificate creation. They also keep an eye on how long it takes to get a certificate (“it’s really fast”). OVHcloud also monitors our status page to stay apprised of our operational status.

Over 10,000 certificates are issued from Let’s Encrypt to OVHcloud every day. As the company continues to expand into North America, they predict that number will grow. The initial and ongoing work done by the OVHcloud team ensures that TLS will be a simple and reliable aspect of their service.

OVHcloud is a longtime sponsor of ISRG so we’d like to close by thanking them for not just being great technical collaborators, but also financial supporters.

Check out our blog post about how Shopify uses Let’s Encrypt certificates for another example of how our certificates are used in the enterprise.

Supporting Let’s Encrypt

As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. If you can support us with a donation, we ask that you make an individual contribution.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2021/10/21/celebrating-encryption-globally.html

The Internet Society has supported our work toward a 100% encrypted Web since before we’d even issued our first certificate. Their commitment to helping us execute our vision has been a substantial help over the years. Today, I’m excited to invite Christine Runnegar, Senior Director at The Internet Society and member of ISRG’s Board of Directors, to share her thoughts.

-Josh Aas, Executive Director, ISRG & Let’s Encrypt

Today, across the world, communities, organizations, and individuals are celebrating Global Encryption Day. Organized by the Global Encryption Coalition (GEC), it’s a day to take stock of the crucial role that encryption plays in securing our communications on the Internet.

The Internet Society is a GEC Steering Committee member because access to encryption is a key tool for us to realize our mission of keeping the Internet a force for good. That’s why the Internet Society is also a proud financial sponsor of Internet Security Research Group (ISRG), which founded and operates Let’s Encrypt. Let’s Encrypt provides digital certificates to more than 260 million websites, making a more secure and privacy-respecting Web for users all over the world. In just five years, the percentage of Web pages loaded over HTTPS has risen from under 50% to more than 85% and climbing, principally because of the community that has coalesced around the importance of encryption everywhere. Encrypted Web traffic protects the confidentiality and integrity of information users share with, or learn from, websites. It makes us all safer online.

Let’s Encrypt is a great success story, and an outstanding example of how supporting public interest infrastructure, such as a certificate authority operated for the public’s benefit, helps ensure everyone has access to the benefits of encryption.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2021/10/01/cert-chaining-help.html

As planned, the DST Root CA X3 has expired and we’re now using our own ISRG Root X1 for trust. We used a cross-sign with DST Root CA X3 to gain broad trust for our certificates when we were just starting out. Now our own root is widely trusted.

For most websites, it was just another day on the Internet, but inevitably with such a big change some sites and configurations have issues. Our overview of the planned expiration is here. You can read about what we’ve done to make the process smoother. Most problems can be solved by updating the software on the machine that is having trouble.

You may also find these links helpful:

Our certificate compatibility page.

Workarounds for OpenSSL 1.0.2.

Whenever there is a significant change to our API, we post in the API Announcements category in our community forum. Sign in and click the bell for notifications to be sent to your email! If you want to hear even more from Let’s Encrypt and the nonprofit team behind it, subscribe to our newsletter. You’ll only receive a handful of emails each year.

We (and our community) are here for you! If you have any questions about this change, search on our community forum or post on the thread we have to help you with this very topic.

Supporting Let’s Encrypt
As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. If you can support us with a donation, we ask that you make an individual contribution.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2021/09/14/speed-at-scale-shopify.html

What does it take to manage TLS certificates at a leading e-commerce company? Before Let’s Encrypt, it took the security team at Shopify weeks to manually obtain certificates for their websites. Doing this once is unpleasant enough, but if an incident were to happen that necessitated renewing all of their certificates, Shopify estimated it would take them 100+ days without automated issuance and management.

Today, Let’s Encrypt provides TLS for 4.5 million Shopify domains. We sat down with Charles Barbier, Development Manager at Shopify, to hear why Let’s Encrypt is their choice for reliable, free, and automated TLS at scale.

“In 2016, the TLS team started transitioning all of our merchants’ stores to HTTPS through Let’s Encrypt,” Charles said. “And when we started exploring the concept a few years earlier, it was a daunting task.” Implementing TLS for 680,000+ domains wasn’t just daunting, Charles and the team needed automated management, something that simply didn’t exist. “We didn’t want to have TLS be the merchant’s responsibility,” Charles said.

Back in 2016, although Let’s Encrypt had been making noise, it wasn’t Shopify’s first choice for a CA. “We ended up going with a different option that turned out to be problematic because the API was so slow,” Charles said. “We did some napkin math and realized it was going to take us around 100 days to provision all of our certs for our merchants. If this solution had been just for regular issuance, it would have been fine, but an emergency would be very problematic.”

That realization led Charles and the team to give Let’s Encrypt a try, making them one of the first single Let’s Encrypt subscribers to request and provision certs at a X00,000 scale. “We were able to roll out all of our domains in a couple of hours,” Charles said. “And to be frank, I think it was our ordering process that caused issuance to take even that long. It was very encouraging.”

The speed of Let’s Encrypt helped Shopify realize their goal of provisioning certs for all of their domains and automating management. Since Let’s Encrypt uses the IETF-standardized ACME protocol, Shopify felt confident that if they needed to, they could roll over to a different ACME CA. “We knew in the future, if things went well with the ACME standard, we’d be able to add a different ACME provider with the exact same implementation,” Charles said.

Of course, “things going well” doesn’t just mean technically. It means ensuring the nonprofit behind Let’s Encrypt is sound as well—which is why Shopify has financially supported Let’s Encrypt since they began using it in 2016. This year, they increased their support. “For us, using Let’s Encrypt has been a great experience,” Charles said.

Today, Let’s Encrypt certificates cover 4.5 million domains for Shopify. That means a more secure and privacy-respecting Web for all of Shopify’s merchants who, in 2020, created $307 billion in economic impact around the world. And it means a more secure Web for everyone visiting and engaging with a Shopify merchant.

We’re proud to serve Shopify with a reliable, speedy, and free service, and grateful for their longtime support of our work by being a sponsor. Together, we’re helping bolster a Web that’s free, open and more secure for everyone, everywhere.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2020/02/27/one-billion-certs.html

We issued our billionth certificate on February 27, 2020. We’re going to use this big round number as an opportunity to reflect on what has changed for us, and for the Internet, leading up to this event. In particular, we want to talk about what has happened since the last time we talked about a big round number of certificates – one hundred million.

One thing that’s different now is that the Web is much more encrypted than it was. In June of 2017 approximately 58% of page loads used HTTPS globally, 64% in the United States. Today 81% of page loads use HTTPS globally, and we’re at 91% in the United States! This is an incredible achievement. That’s a lot more privacy and security for everybody.

Another thing that’s different is that our organization has grown a bit, but not by much! In June of 2017 we were serving approximately 46M websites, and we did so with 11 full time staff and an annual budget of $2.61M. Today we serve nearly 192M websites with 13 full time staff and an annual budget of approximately $3.35M. This means we’re serving more than 4x the websites with only two additional staff and a 28% increase in budget. The additional staff and budget did more than just improve our ability to scale though – we’ve made improvements across the board to provide even more secure and reliable service.

Nothing drives adoption like ease of use, and the foundation for ease of use in the certificate space is our ACME protocol. ACME allows for extensive automation, which means computers can do most of the work. It was also standardized as RFC 8555 in 2019, which allows the Web community to confidently build an even richer ecosystem of software around it. Today, thanks to our incredible community, there is an ACME client for just about every deployment environment. Certbot is one of our favorites, and they’ve been working hard to make it even easier for people to use.

When you combine ease of use with incentives, that’s when adoption really takes off. Since 2017 browsers have started requiring HTTPS for more features, and they’ve greatly improved the ways in which they communicate to their users about the risks of not using HTTPS. When websites put their users at risk by not using HTTPS, major browsers now show stronger warnings. Many sites have responded by deploying HTTPS.

Thanks for taking the time to reflect on this milestone with us. As a community we’ve done incredible things to protect people on the Web. Having issued one billion certificates is affirmation of all the progress we’ve made as a community, and we’re excited to keep working with you to create an even more secure and privacy-respecting Web for everyone.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2020/02/19/multi-perspective-validation.html

At Let’s Encrypt we’re always looking for ways to improve the security and integrity of the Web PKI. We’re proud to launch multi-perspective domain validation today because we believe it’s an important step forward for the domain validation process. To our knowledge we are the first CA to deploy multi-perspective validation at scale.

Domain validation is a process that all CAs use to ensure that a certificate applicant actually controls the domain they want a certificate for. Typically the domain validation process involves asking the applicant to place a particular file or token at a controlled location for the domain, such as a particular path or a DNS entry. Then the CA will check that the applicant was able to do so. Historically it looks something like this:

A potential issue with this process is that if a network attacker can hijack or redirect network traffic along the validation path (for the challenge request, or associated DNS queries), then the attacker can trick a CA into incorrectly issuing a certificate. This is precisely what a research team from Princeton demonstrated can be done with an attack on BGP. Such attacks are rare today, but we are concerned that these attacks will become more numerous in the future.

The Border Gateway Protocol (BGP) and most deployments of it are not secure. While there are ongoing efforts to secure BGP, such as RPKI and BGPsec, it may be a long time until BGP hijacking is a thing of the past. We don’t want to wait until we can depend on BGP being secure, so we’ve worked with the research team from Princeton to devise a way to make such attacks more difficult. Instead of validating from one network perspective, we now validate from multiple perspectives as well as from our own data centers:

Today we are validating from multiple regions within a single cloud provider. We plan to diversify network perspectives to other cloud providers in the future.

This makes the kind of attack described earlier more difficult because an attacker must successfully compromise three different network paths at the same time (the primary path from our data center, and at least two of the three remote paths). It also increases the likelihood that such an attack will be detected by the Internet topology community.

We’d like to thank the research groups of Prof. Prateek Mittal and Prof. Jennifer Rexford at Princeton University for their partnership in developing this work. We will continue to work with them to refine the effectiveness of our multiple perspective validation design and implementation. We’d also like to thank Open Technology Fund for supporting this work.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2019/11/20/how-le-runs-ct-logs.html

Let’s Encrypt launched a Certificate Transparency (CT) log this past spring. We’re excited to share how we built it in hopes that others can learn from what we did. CT has quickly become an important piece of Internet security infrastructure, but unfortunately it’s not trivial to run a good log. The more the CT community can share about what has been done, the better the ecosystem will be.

Sectigo and Amazon Web Services have generously provided support to cover a significant portion of the cost of running our CT log. “Sectigo is proud to sponsor the Let’s Encrypt CT Log. We believe this initiative will provide much-needed reinforcement of the CT ecosystem,” said Ed Giaquinto, Sectigo’s CIO.

For more background information about CT and how it works, we recommend reading “How Certificate Transparency Works.”

If you have questions about any of what we’ve written here, feel free to ask on our community forums.

Objectives

1. Scale: Let’s Encrypt issues over 1 million certificates per day, and that number grows each month. We want our log to consume our certificates as well as those from other CAs, so we need to be able to handle as many as 2 million or more certificates per day. To support this ever-increasing number of certificates, CT software and infrastructure need to be architected for scale.
2. Stability and Compliance: We target 99% uptime, with no outage lasting longer than 24 hours, in compliance with the Chromium and Apple CT policies.
3. Sharding: Best practice for a CT log is to break it into several temporal shards. For more information on temporal sharding, check out these blog posts.
4. Low Maintenance: Staff time is expensive, we want to minimize the amount of time spent maintaining infrastructure.

System Architecture

Staging and Production Logs

We run two equivalent logs, one for staging and one for production. Any changes we plan to make to the production log are first deployed to the staging log. This is critical for making sure that updates and upgrades don’t cause problems before being deployed to production. You can find access details for these logs in our documentation.

We keep the staging log continually under production-level load so that any scale-related problems manifest there first. We also use the staging CT log to submit certificates from our staging CA environment, and make it available for use by other CAs’ staging environments.

As a point of clarification, we consider a log to be comprised of several temporal shards. While each shard is technically a separate log, it makes sense to conceptualize the shards as belonging to a single log.

Amazon Web Services (AWS)

We decided to run our CT logs on AWS for two reasons.

One consideration for us was cloud provider diversity. Since there are relatively few trusted logs in the ecosystem, we don’t want multiple logs to go down due to a single cloud provider outage. At the time we made the decision there were logs running on Google and Digital Ocean infrastructure, as well as self-hosted. We were not aware of any on AWS (in hindsight we may have missed the fact that Digicert had started using AWS for logs). If you’re thinking about setting up a trusted log for CAs to use, please consider cloud provider diversity.

Additionally, AWS provides a solid set of features and our team has experience using it for other purposes. We had little doubt that AWS was up to the task.

Terraform

Let’s Encrypt uses Hashicorp Terraform for a number of cloud-based projects. We were able to bootstrap our CT log infrastructure by reusing our existing Terraform code. There are roughly 50 components in our CT deployments; including EC2, RDS, EKS, IAM, security groups, and routing. Centrally managing this code allows our small team to reproduce a CT infrastructure in any Amazon region of the globe, prevent configuration drift, and easily test infrastructure changes.

Database

We chose to use MariaDB for our CT log database because we have extensive experience using it to run our certificate authority. MariaDB has scaled well on our journey to becoming the largest publicly trusted certificate authority.

We chose to have our MariaDB instances managed by Amazon RDS because RDS provides synchronous writes to standby cluster members. This allows for automatic database failover and ensures database consistency. Synchronous writes to database replicas are essential for a CT log. One missed write during a database failover can mean a certificate was not included as promised, and could lead to the log being disqualified. Having RDS manage this for us reduces complexity and saves staff time. We are still responsible for managing the database performance, tuning, and monitoring.

It’s important to calculate the necessary amount of storage for a CT log database carefully. Too little storage can result in needing to undertake time-consuming and potentially risky storage migrations. Too much storage can result in unnecessarily high costs.

A back of the napkin storage estimation is 1TB per 100 million entries. We expect to need to store 1 billion certificates and precertificates per annual temporal shard, for which we would need 10TB. We considered having separate database storage per annual temporal shard, with approximately 10TB allocated to each, but that was cost prohibitive. We decided to create a 12TB storage block per log (10TB plus some breathing room), which is duplicated for redundancy by RDS. Each year we plan to freeze the previous year’s shard and move it to a less expensive serving infrastructure, reclaiming its storage for our live shards.

We use 2x db.r5.4xlarge instances for RDS for each CT log. Each of these instances contains 8 CPU cores and 128GB of RAM.

Kubernetes

After trying a few different strategies for managing application instances, we decided to use Kubernetes. There is a hefty learning curve for Kubernetes and the decision was not made lightly. This was our first project making use of Kubernetes, and part of the reason we went with it was to gain experience and possibly apply that knowledge to other parts of our infrastructure in the future.

Kubernetes provides abstractions for operators such as deployments, scaling, and service discovery that we would not have to build ourselves. We utilized the example Kubernetes deployment manifests in the Trillian repository to assist with our deployment.

A Kubernetes cluster is comprised of two main components: the control plane which handles the Kubernetes APIs, and worker nodes where containerized applications run. We chose to have Amazon EKS manage our Kubernetes control plane.

We use 4x c5.2xlarge EC2 instances for the worker node pool for each CT log. Each of these instances contains 8 CPU cores and 16GB of RAM.

Application Software

There are three main CT components that we run in a Kubernetes cluster.

The certificate transparency front end, or CTFE, provides RFC 6962 endpoints and translates them to gRPC API requests for the Trillian backend.

Trillian describes itself as a “transparent, highly scalable and cryptographically verifiable data store.” Essentially, Trillian implements a generalized verifiable data store via a Merkle tree that can be used as the back-end for a CT log via the CTFE. Trillian consists of two components; the log signer and log server. The log signer’s function is to periodically process incoming leaf data (certificates in the case of CT) and incorporate them into a Merkle tree. The log server retrieves objects from a Merkle tree in order to fulfill CT API monitoring requests.

Load Balancing

Traffic enters the CT log through an Amazon ELB which is mapped to a Kubernetes Nginx ingress service. The ingress service balances traffic amongst multiple Nginx pods. The Nginx pods proxy traffic to the CTFE service which balance that traffic to CTFE pods.

We employ IP and user agent based rate limiting at this Nginx layer.

Logging and Monitoring

Trillian and the CTFE expose Prometheus metrics which we transform into monitoring dashboards and alerts. It is essential to set a Service Level Objective for the CT log endpoints above the 99% uptime dictated by CT policy to ensure that your log is trusted. A FluentD pod running in a DaemonSet ships logs to centralized storage for further analysis.

We developed a free and open source tool named ct-woodpecker that is used to monitor various aspects of log stability and correctness. This tool is an important part of how we ensure we’re meeting our service level objectives. Each ct-woodpecker instance runs externally from Amazon VPCs containing CT logs.

Future Efficiency Improvements

Here are some ways we may be able to improve the efficiency of our system in the future:

• Trillian stores a copy of each certificate chain, including many duplicate copies of the same intermediate certificates. Being able to de-duplicate these in Trillian would significantly reduce storage costs. We’re planning to look into whether this is possible and reasonable.
• See if we can successfully use a cheaper form of storage than IO1 block storage and provisioned IOPs.
• See if we can reduce the Kubernetes worker EC2 instance size or use fewer EC2 instances.

Support Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization is interested in learning more about sponsorship, please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2019/10/09/onboarding-your-customers-with-lets-encrypt-and-acme.html

If you work at a hosting provider or CDN, ACME’s DNS-01 validation
method can make it a lot easier to onboard new customers who have an
existing HTTPS website at another provider. Before your new customer
points their domain name at your servers, you need to have a certificate
already installed for them. Otherwise visitors to the customer’s site
will see an outage for a few minutes while you issue and install a
certificate. To fix this, you and your new customer should use the
DNS-01 validation method to issue a certificate before the customer
switches over DNS for their site.

How the DNS Validation Method Works

The DNS-01 validation method works like
this: to prove that you control
www.example.com, you create a TXT record at
_acme-challenge.www.example.com with a “digest value” as specified by
ACME (your ACME client should take care of creating this digest value
for you). When the TXT record is ready, your ACME client informs the ACME server (for
instance, Let’s Encrypt) that the domain is ready for validation. The
ACME server looks up the TXT record, compares it to the expected digest
value, and if the result is correct, considers your account authorized
to issue for www.example.com. Your new customer can set up this TXT
record (or a CNAME) without interfering with normal website operations.

The Advantages of a CNAME

There’s an additional trick that I recommend for hosting providers and
CDNs: Instead of giving the digest value to your new customer and
telling them to make a TXT record with it, tell your customer to
configure a CNAME from _acme-challenge.www.example.com to a domain
name that you control and that is unique to the domain being validated.
For instance, you might use www.example.com.validationserver.example.net.
Then, once your
software has verified that this CNAME is set up (accounting for
propagation delay and anycast), your ACME client should
begin the validation process for www.example.com, provisioning a TXT
record at www.example.com.validationserver.example.net. Because the
ACME server’s TXT lookup follows CNAMEs (as do all DNS lookups), it will
see the value you provisioned, and consider your account authorized.

This approach is preferable to handing your customers a raw digest value
for a few reasons. First, it gives your customers all the time they need to set
up the CNAME. If you create a pending authorization up front and give
your customer a digest value to deploy themselves, it has a fixed
lifetime before it expires (for Let’s Encrypt this lifetime is 7 days).
If your customer doesn’t complete the process in that time,
you’ll have to create a new pending authorization and give
your customer a new digest value. That’s annoying and time consuming for
both you and your customer. The CNAME method means even if it
takes your new customer a month to make the needed changes to their DNS,
you can get things up and running as soon as they do.

Another reason to prefer the CNAME method over having new customers
directly provision their TXT records is to support the best practice of
periodically rotating your ACME account key. Because the digest value
used for DNS-01 validation is computed based on your current ACME
account key, it will change whenever you rotate your account key. If you
asked customers to provision their TXT record manually, that means
notifying potential new customers that the value you asked them to put
in DNS isn’t valid anymore, and they need to use a different one. That’s pretty
inconvenient! If you use the CNAME method instead, there’s only one
ACME-related value you’ll ever need to have your new customers put in
DNS, and it won’t change as you change your account key.

Cleaning Up Unused CNAMES

One last note: This is a good way to onboard customers, but you also
need to detect when customers offboard themselves. They may simply
change their A records to point at a different CDN, without telling you
that their plans have changed. You should monitor for this situation and
stop attempting to issue certificates. If the customer has left behind a
CNAMEd _acme-challenge subdomain that points at you, you should
contact that and remind them to delete it. The CNAMEd subdomain
represents a delegated authorization to issue certificates, and cleaning
up that delegation improves both the customer’s security posture and
your own. Similarly, if a customer sets up the CNAME and you issue a
certificate on their behalf, but they never point their A records at
your servers, you should not reissue new certificates indefinitely
without further intervention from the customer.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2019/05/15/introducing-oak-ct-log.html

Update: Feb. 5 2020

The Let’s Encrypt CT logs are now included in approved log lists and are usable by all publicly-trusted certificate authorities.

Today we are announcing a new Certificate Transparency log called Oak. The Oak log will be operated by Let’s Encrypt and all publicly trusted certificate authorities will be welcome to submit certificates.

Sectigo generously provided funding to cover a significant portion of our costs to run our CT log. “Sectigo is proud to sponsor the Let’s Encrypt CT Log. We believe this initiative will provide much-needed reinforcement of the CT ecosystem,” said Ed Giaquinto, Sectigo’s CIO. We thank them for their collaboration to improve Internet security.

Certificate Transparency (CT) is a system for logging and monitoring certificate issuance. It greatly enhances everyone’s ability to monitor and study certificate issuance, and these capabilities have led to numerous improvements to the CA ecosystem and Web security. As a result, it is rapidly becoming critical Internet infrastructure. Let’s Encrypt accelerated the adoption of CT by logging every certificate since we started issuing in 2015 – approximately half a billion certificates at this point.

We decided to create and operate a CT log for a few reasons. First, operating a log is consistent with our mission to create a more secure and privacy-respecting Web. We believe transparency increases security and empowers people to make well-informed decisions. Second, operating a log helps us take control of our destiny. Google Chrome requires all new certificates to be submitted to two separate logs, so multiple log options are imperative to our operation. Finally, Let’s Encrypt often issues more than 1M certificates each day, so we wanted to design a CT log that is optimized for high volume. We’ve designed our log to be able to handle submissions from all other publicly trusted Certificate Authorities so they can use Oak to fulfill their logging requirements as well.

Our log uses Google’s Trillian software running on AWS infrastructure. We use Kubernetes for container orchestration and job scheduling and AWS RDS for database management.

We are submitting our log for inclusion in the approved log lists for Google Chrome and Apple Safari. Following 90 days of successful monitoring, we anticipate our log will be added to these trusted lists and that change will propagate to people’s browsers with subsequent browser version releases.

Continuing the forest theme, we are also announcing the launch of our open source CT monitoring tool, CT Woodpecker. We use it to monitor and ensure compliance for our log and we’ve made it open source so others in the CT ecosystem can use it as well.

We’d like to thank Google, Sectigo, Cloudflare, and DigiCert for also running open logs, and we look forward to contributing to better transparency in Web security!

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2019/04/15/transitioning-to-isrg-root.html

Update, September 17, 2020

Due to concerns about insufficient ISRG root propagation on Android devices we have decided to move the date on which we will start serving a chain to our own root to January 11, 2021. We had originally delayed this change until September 29, 2020.

Update, June 11, 2020

In an effort to provide more time for our community to prepare for this transition, we have moved back the date on which we will start serving a chain to our own root to September 29, 2020.

On January 11, 2021, we will change the default intermediate certificate we provide via ACME. Most subscribers don’t need to do anything. Subscribers who support very old TLS/SSL clients may want to manually configure the older intermediate to increase backwards compatibility.

Since Let’s Encrypt launched, our certificates have been trusted by browsers via a cross-signature from another Certificate Authority (CA) named IdenTrust. A cross-signature from IdenTrust was necessary because our own root was not yet widely trusted. It takes time for a new CA to demonstrate that it is trustworthy, then it takes more time for trusted status to propagate via software updates.

Now that our own root, ISRG Root X1, is widely trusted by browsers we’d like to transition our subscribers to using our root directly, without a cross-sign.

On January 11, 2021, Let’s Encrypt will start serving a certificate chain via the ACME protocol which leads directly to our root, with no cross-signature. Most subscribers don’t need to take any action because their ACME client will handle everything automatically. Subscribers who need to support very old TLS/SSL clients may wish to manually configure their servers to continue using the cross-signature from IdenTrust. You can test whether a given client will work with the newer intermediate by accessing our test site.

Our current cross-signature from IdenTrust expires on March 17, 2021. The IdenTrust root that we are cross-signed from expires on September 30, 2021. Within the next year we will obtain a new cross-signature that is valid until September 29, 2021. This means that our subscribers will have the option to manually configure a certificate chain that uses IdenTrust until September 29, 2021.

We’d like to thank IdenTrust for providing a cross-signature while we worked to get our own root trusted. They have been wonderful partners. IdenTrust believed in our mission to encrypt the entire Web when it seemed like a long-term dream. Together, in less than five years, we have helped to raise the percentage of encrypted page loads on the Web from 39% to 78%.

Let’s Encrypt is currently providing certificates for more than 160 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2019/03/11/acme-protocol-ietf-standard.html

It has long been a dream of ours for there to be a standardized protocol for certificate issuance and management. That dream has become a reality now that the IETF has standardized the ACME protocol as RFC 8555. I’d like to thank everyone involved in that effort, including Let’s Encrypt staff and other IETF contributors.

Having a standardized protocol for certificate issuance and management is important for two reasons. First, it improves the quality of the software ecosystem because developers can focus on developing great software for a single protocol, instead of having many pieces of less well maintained software for bespoke APIs. Second, a standardized protocol makes switching from one CA to another easier by minimizing technical dependency lock-in.

We consider the standardized version of the ACME protocol to be the second major version of ACME, so we refer to it as ACMEv2. The first version, which we call ACMEv1, is the version of ACME that Let’s Encrypt has used since our launch in 2015. Now that ACMEv2 is standardized, we are announcing an end-of-life plan for our ACMEv1 support.

Let’s Encrypt is currently providing certificates for more than 150 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2019/02/12/facebook-expands-support-for-letsencrypt.html

We’re excited that Facebook is supporting our work through a three-year Platinum sponsorship! We asked them to share their thoughts on HTTPS adoption here. Please join us in thanking Facebook for their support of Let’s Encrypt and our mission to encrypt the Web! – Josh Aas, Executive Director, ISRG / Let’s Encrypt

If the web is more secure, everybody wins. A key technology for making this happen is HTTPS, which enables encrypted connections between people and the websites that they visit. Among its many benefits, HTTPS helps to prevent sensitive data from leaking over the network, and from connections being censored or otherwise maliciously manipulated. The more widely it is deployed, the more secure and private the web becomes for everyone.

We have long worked to protect Facebook users from spammy or malicious content when navigating away from our platform, and last year we extended this protection to upgrading outbound HTTP links to HTTPS where possible. In this way we can help improve people’s security and privacy as they leave our platform. While we take these steps to improve the security and safety of Facebook users, ultimately we hope to see more websites allowing HTTPS connections.

Enabling HTTPS was historically a non-trivial task for any site. It required investment in buying and installing a TLS certificate, which verifies control over the website so that HTTPS can work. The technical difficulty and cost used to serve as barriers to expanding the use of HTTPS across the web. However, things have recently started to change, largely thanks to Let’s Encrypt, a non-profit certificate authority, launched in 2015.

Let’s Encrypt provides free TLS certificates, which are often installed using a tool maintained by the Electronic Frontier Foundation, to massively simplify enabling HTTPS. With that, Let’s Encrypt is effectively upgrading the security and privacy of the web, at no cost to over 150 million websites, including those frequented by Facebook users.

We’re excited to see the continuous increase in HTTPS adoption across the internet. More websites are choosing to enable secure connections which provide the security and privacy benefits and enable a better browsing experience. For example, navigating from Facebook to another site can be faster over encrypted connections than HTTP, and an increasing number of browser features will only work when sites use HTTPS.

We have sponsored Let’s Encrypt from the start, and are proud to share that we are increasing that support as a platinum sponsor. We believe that Let’s Encrypt has played a significant and important role in bringing encryption into the mainstream and raising the number of secure sites across the internet.

As we automatically crawl web content on Facebook (for example, to generate link previews), about 38% of HTTPS domains we observe use Let’s Encrypt, making it the top certificate authority. Over 19% of outbound clicks from Facebook to HTTPS-enabled websites go to sites that use certificates from Let’s Encrypt. Overall, more than 72% of outbound clicks from Facebook are now destined for HTTPS-enabled websites, including the links that we upgrade to HTTPS in real time.

We’re proud to continue to collaborate with Let’s Encrypt on helping to improve web security. To any website owners who haven’t yet enabled encryption, we strongly encourage you to use Let’s Encrypt to protect your users and allow HTTPS connections.

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2018/12/31/looking-forward-to-2019.html

Let’s Encrypt had a great year in 2018. We’re now serving more than 150 million websites while maintaining a stellar security and compliance track record.

Most importantly though, the Web went from 67% encrypted page loads to 77% in 2018, according to statistics from Mozilla. This is an incredible rate of change!

We’d like to thank all of the people and organizations who worked hard to create a more secure and privacy-respecting Web.

This year we created a new website for the legal entity behind Let’s Encrypt, Internet Security Research Group (ISRG), because we believe there will be other instances beyond Let’s Encrypt in which ISRG might be able to help to build, or improve access to, a better Internet.

While we’re proud of what we accomplished in 2018, we spend most of our time looking forward rather than back. As we wrap up our own planning process for 2019, we’d like to share some of our plans with you, including both the things we’re excited about and the challenges we’ll face. We’ll cover service growth, new features, infrastructure, and finances.

Service Growth

Let’s Encrypt helps to drive HTTPS adoption by offering a free, easy to use, and globally available option for obtaining the certificates required to enable HTTPS. HTTPS adoption on the Web took off at an unprecedented rate from the day Let’s Encrypt launched to the public.

The number of certificates and unique domains we support continues to grow rapidly:

We expect strong growth again in 2019, likely up to 120M active certificates and 215M fully qualified domains. You can view our recently revamped stats page for more information.

One of the reasons Let’s Encrypt is so easy to use is that our community has done great work making client software that works well for a wide variety of platforms. We’d like to thank everyone involved in the development of more than 85 client software options for Let’s Encrypt. Support for our protocol, ACME, is built in to Apache and we’re hoping 2019 will be the year that it comes to Nginx.

Other organizations and communities are also doing great work to promote HTTPS adoption, and thus stimulate demand for our services. For example, browsers are starting to make their users more aware of the risks associated with unencrypted HTTP (e.g. Firefox, Chrome). Many hosting providers and CDNs are making it easier than ever for all of their customers to use HTTPS. Government agencies are waking up to the need for stronger security to protect constituents. The media community is working to Secure the News.

New Features

In 2018 we introduced several new features, including ACMEv2 support and wildcard certificates. We’ve got some exciting features planned for 2019.

The feature we’re most excited about is multi-perspective validation. Currently, when a subscriber requests a certificate, we validate domain control from a single network perspective. This is standard practice for CAs. If an attacker along the network path for the validation check can interfere with traffic they can potentially cause certificates to be issued that should not be issued. We’re most concerned about this happening via BGP hijacking, and since BGP is not going to be secured any time soon, we needed to find another mitigation. The solution we intend to deploy in 2019 is multi-perspective validation, in which we will check from multiple network perspectives (distinct Autonomous Systems). This means that potential BGP hijackers would need to hijack multiple routes at the same time in order to pull off a successful attack, which is significantly more difficult than hijacking a single route. We are working with a talented research team at Princeton to design the most effective multi-perspective validation system we can, and have already turned parts of this feature on in our staging environment.

We are also planning to introduce a Certificate Transparency (CT) log in 2019. All certificate authorities like Let’s Encrypt are required to submit certificates to CT logs but there are not enough stable logs in the ecosystem. As such, we are moving forward with plans to run a log which all CAs will be able to submit to.

We had planned to add ECDSA root and intermediate certificates in 2018 but other priorities ultimately took precedence. We hope to do this in 2019. ECDSA is generally considered to be the future of digital signature algorithms on the Web due to the fact that it is more efficient than RSA. Let’s Encrypt will currently sign ECDSA keys from subscribers, but we sign with the RSA key from one of our intermediate certificates. Once we have an ECDSA root and intermediates, our subscribers will be able to deploy certificate chains which are entirely ECDSA.

Infrastructure

Our CA infrastructure is capable of issuing millions of certificates per day with redundancy for stability and a wide variety of security safeguards, both physical and logical. Our infrastructure also generates and signs around 40 million OCSP responses daily, and serves those responses approximately 5.5 billion times per day. We expect these numbers to grow approximately 40% in 2019.

Our physical CA infrastructure currently occupies approximately 55 units of rack space, split between two datacenters, consisting primarily of compute servers, storage, HSMs, switches, and firewalls. When we issue more certificates it puts the most stress on storage for our databases. We regularly invest in more and faster storage for our database servers, and that will continue in 2019.

All of our infrastructure is managed by our Site Reliability Engineering (SRE) team, which is comprised of six people. SRE staff are responsible for building and maintaining all physical and logical CA infrastructure. These staff are largely responsible for our high standards for security and compliance. The team also manages a 24/7/365 on-call schedule and they are primary participants in both security and compliance audits.

Finances

We pride ourselves on being an efficient organization. In 2019 Let’s Encrypt will secure a massive portion of the Web with a budget of only $3.6M. We believe this represents an incredible value and that contributing to Let’s Encrypt is one of the most effective ways to help create a more secure and privacy-respecting Web.

Our 2019 fundraising efforts are off to a strong start with Platinum sponsorships from Cisco, OVH, Mozilla, Google Chrome, Electronic Frontier Foundation, and Internet Society, as well as many other Gold and Silver sponsors. The Ford Foundation has renewed their grant to Let’s Encrypt as well. We are seeking additional sponsorship and grant assistance to meet our full needs for 2019.

Support Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

We’re grateful for the industry and community support that we receive, and we look forward to continuing to create a more secure and privacy-respecting Web!