All posts by Let's Encrypt - Free SSL/TLS Certificates

Preparing to Issue 200 Million Certificates in 24 Hours

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2021/02/10/200m-certs-24hrs.html

On a normal day Let’s Encrypt issues nearly two million certificates. When we think about what essential infrastructure for the Internet needs to be prepared for though, we’re not thinking about normal days. We want to be prepared to respond as best we can to the most difficult situations that might arise. In some of the worst scenarios, we might want to re-issue all of our certificates in a 24 hour period in order to avoid widespread disruptions. That means being prepared to issue 200 million certificates in a day, something no publicly trusted CA has ever done.

We recently completed most of the work and investments needed to issue 200 million certificates in a day and we thought we’d let people know what was involved. All of this was made possible by our sponsors and funders, including major hardware contributions from Cisco, Thales, and Fortinet.

Scenarios

Security and compliance events are a certainty in our industry. We obviously try to minimize them, but since we anticipate they will happen, we spend a lot of time preparing to respond in the best ways possible.

In February of 2020 a bug affecting our compliance caused us to need to revoke and replace three million active certificates. That was approximately 2.6% of all active certificates.

What if that bug had affected all of our certificates? That’s more than 150 million certificates covering more than 240 million domains. What if it had also been a more serious bug, requiring us to revoke and replace all certificates within 24 hours? That’s the kind of worst case scenario we need to be prepared for.

To make matters worse, during an incident it takes some time to evaluate the problem and make decisions, so we would be starting revocation and reissuance somewhat after the beginning of the 24-hour clock. That means we would actually have less time once the momentous decision has been made to revoke and replace 150 million or more certificates.

Infrastructure Improvements

After reviewing our systems, we determined that there were four primary bottlenecks that would prevent us from replacing 200 million certificates in a day. Database performance, internal networking speed, cryptographic signing module (HSM) performance, and bandwidth.

Database Performance

Let’s Encrypt has a primary certificate authority database at the heart of the service we offer. This tracks the status of certificate issuance and the associated accounts. It is write heavy, with plenty of reads as well. At any given time a single database server is the writer, with some reads directed at identical machines with replicas. The single writer non-clustered strategy helps with consistency and reduces complexity, but it means that writes and some reads must operate within the performance constraints of a single machine.

Our previous generation of database servers had dual Intel Xeon E5-2650 v4 CPUs, 24 physical cores in total. They had 1TB memory with 24 3.8TB SSDs connected via SATA in a RAID 10 configuration. These worked fine for daily issuance but would not be able to handle replacing all of our certificates in a single day.

We have replaced them with a new generation of database server from Dell featuring dual AMD EPYC 7542 CPUs, 64 physical cores in total. These machines have 2TB of faster RAM. Much faster CPUs and double the memory is great, but the really interesting thing about these machines is that the EPYC CPUs provide 128 PCIe4 lanes each. This means we could pack in 24 6.4TB NVME drives for massive I/O performance. There is no viable hardware RAID for NVME, so we’ve switched to ZFS to provide the data protection we need.

Internal Networking

Let’s Encrypt infrastructure was originally built with 1G copper networking. Between bonding multiple connections for 2G performance, and use of the very limited number of 10G ports available on our switches, it served us pretty well until 2020.

By 2020 the volume of data we were moving around internally was too much to handle efficiently with 1G copper. Some normal operations were significantly slower than we’d like (e.g. backups, replicas), and during incidents the 1G network could cause significant delays in our response.

We originally looked into upgrading to 10G, but learned that upgrading to 25G fiber wasn’t much more expensive. Cisco ended up generously donating most of the switches and equipment we needed for this upgrade, and after replacing a lot of server NICs Let’s Encrypt is now running on a 25G fiber network!

Funny story – back in 2014 Cisco had donated really nice 10G fiber switches to use in building the original Let’s Encrypt infrastructure. Our cabinets at the time had an unusually short depth though, and the 10G switches didn’t physically fit. We had to return them for physically smaller 1G switches. 1G seemed like plenty at the time. We have since moved to new cabinets with standard depth!

HSM Performance

Each Let’s Encrypt data center has a pair of Luna HSMs that sign all certificates and their OCSP responses. If we want to revoke and reissue 200 million certificates, we need the Luna HSMs to perform the following cryptographic signing operations:

  • 200 million OCSP response signatures for revocation
  • 200 million certificate signatures for replacement certificates
  • 200 million OCSP response signatures for the new certificates

That means we need to perform 600 million cryptographic signatures in 24 hours or less, with some performance overhead to account for request clustering.

We need to assume that during incidents we might only have one data center online, so when we’re thinking about HSM performance we’re thinking about what we can do with just one pair. Our previous HSMs could perform approximately 1,100 signing operations each per second, 2,200 between the pair. That’s 190,080,000 signatures in 24 hours, working at full capacity. It’s under half of what we would need, and take more time than we might have.

In order to get to where we need to be Thales generously donated new HSMs with about 10x the performance – approximately 10,000 signing operations per second, 20,000 between the pair. That means we can now perform 864,000,000 signing operations in 24 hours from a single data center.

Bandwidth

Issuing a certificate is not particularly bandwidth intensive, but during an incident we typically use a lot more bandwidth for systems recovery and analysis. We move large volumes of logs and other files between data centers and the cloud for analysis and forensic purposes. We may need to synchronize large databases. We create copies and additional back-ups. If the connections in our data centers are slow, it slows down our response.

After determining that data center connection speed could add significantly to our response time, we increased bandwidth accordingly. Fortinet helped provide hardware that helps us protect and manage these higher capacity connections.

API Extension

In order to get all those certificates replaced, we need an efficient and automated way to notify ACME clients that they should perform early renewal. Normally ACME clients renew their certificates when one third of their lifetime is remaining, and don’t contact our servers otherwise. We published a draft extension to ACME last year that describes a way for clients to regularly poll ACME servers to find out about early-renewal events. We plan to polish up that draft, implement, and collaborate with clients and large integrators to get it implemented on the client side.

Supporting Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

The Next Gen Database Servers Powering Let’s Encrypt

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2021/01/21/next-gen-database-servers.html

Let’s Encrypt helps to protect a huge portion of the Web by providing TLS certificates to more than 235 million websites. A database is at the heart of how Let’s Encrypt manages certificate issuance. If this database isn’t performing well enough, it can cause API errors and timeouts for our subscribers. Database performance is the single most critical factor in our ability to scale while meeting service level objectives. In late 2020, we upgraded our database servers and we’ve been very happy with the results.

What exactly are we doing with these servers?

Our CA software, Boulder, uses MySQL-style schemas and queries to manage subscriber accounts and the entire certificate issuance process. It’s designed to work with a single MySQL, MariaDB, or Percona database. We currently use MariaDB, with the InnoDB database engine.

We run the CA against a single database in order to minimize complexity. Minimizing complexity is good for security, reliability, and reducing maintenance burden. We have a number of replicas of the database active at any given time, and we direct some read operations to replica database servers to reduce load on the primary.

One consequence of this design is that our database machines need to be pretty powerful. Eventually we may need to shard or break the single database into multiple databases, but hardware advancements have allowed us to avoid that so far.

Hardware Specifications

The previous generation of database hardware was powerful but it was regularly being pushed to its limits. For the next generation, we wanted to more than double almost every performance metric in the same 2U form factor. In order to pull that off, we needed AMD EPYC chips and Dell’s PowerEdge R7525 was ideal. Here are the specifications:

Previous Generation Next Generation
CPU 2x Intel Xeon E5-2650
Total 24 cores / 48 threads
2x AMD EPYC 7542
Total 64 cores / 128 threads
Memory 1TB 2400MT/s 2TB 3200MT/s
Storage 24x 3.8TB Samsung PM883
SATA SSD
560/540 MB/s read/write
24x 6.4TB Intel P4610
NVMe SSD
3200/3200 MB/s read/write


Dell PowerEdge R7525 Chassis
Dell PowerEdge R7525 internals. The two silver rectangles in the middle are the CPUs. The RAM sticks, each 64GB, are above and below the CPUs. The 24x NVMe drives are in the front of the server, on the far right.

By going with AMD EPYC, we were able to get 64 physical CPU cores while keeping clock speeds high: 2.9GHz base with 3.4GHz boost. More importantly, EPYC provides 128 PCIe v4.0 lanes, which allows us to put 24 NVMe drives in a single machine. NVMe is incredibly fast (~5.7x faster than the SATA SSDs in our previous-gen database servers) because it uses PCIe instead of SATA. However, PCIe lanes are typically very limited: modern consumer chips typically have only 16 lanes, and Intel’s Xeon chips have 48. By providing 128 PCI lanes per chip (v4.0, no less), AMD EPYC has made it possible to pack large numbers of NVMe drives into a single machine. We’ll talk more about NVMe later.

Performance Impact

We’ll start by looking at our median time to process a request because it best reflects subscribers’ experience. Before the upgrade, we turned around the median API request in ~90 ms. The upgrade decimated that metric to ~9 ms!

API Latency

We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from /proc/stat) averaged over 90%:

CPU Usage Before Upgrade

The new AMD EPYC CPUs sit at about 25%. You can see in this graph where we promoted the new database server from replica (read-only) to primary (read/write) on September 15.

CPU Usage After Upgrade

The upgrade greatly reduced our overall database latency. The average query response time (from INFORMATION_SCHEMA) used to be ~0.45ms.

Database Latency Before Upgrade

Queries now average three times faster, about 0.15ms.

Database Latency After Upgrade

OpenZFS and NVMe

NVMe drives are becoming increasingly popular because of their incredible performance. Up until recently, though, it was nearly impossible to get many of them in a single machine because NVMe uses PCIe lanes. Those were very limited: Intel’s Xeon processors come with just 48 PCIe v3 lanes, and a number of those are used up by the chipset and add-on cards such as network adapters and GPUs. You can’t fit many NVMe drives in the remaining lanes.

AMD’s latest generation of EPYC processors come with 128 PCIe lanes – more than double what Intel offers – and they’re PCIe v4! This is enough to pack a 2U server full of NVMe drives (24 in our case).

Once you have a server full of NVMe drives, you have to decide how to manage them. Our previous generation of database servers used hardware RAID in a RAID-10 configuration, but there is no effective hardware RAID for NVMe, so we needed another solution. One option was software RAID (Linux mdraid), but we got several recommendations for OpenZFS and decided to give it a shot. We’ve been very happy with it!

There wasn’t a lot of information out there about how best to set up and optimize OpenZFS for a pool of NVMe drives and a database workload, so we want to share what we learned. You can find detailed information about our setup in this GitHub repository.

Conclusion

This database upgrade was necessary as more people rely on Let’s Encrypt for the security and privacy that TLS/SSL provides. The equipment is quite expensive and it was a sizable undertaking for our SRE team to plan and execute the transition, but we gained a lot through the process.

Support Let’s Encrypt

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

A Year-End Letter from the Executive Director of Let’s Encrypt and ISRG

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/12/28/executive-director-letter.html

This letter was originally published in our 2020 annual report.

ISRG’s first project, Let’s Encrypt, has been wildly successful. We’re now helping to secure more than 225 million websites and the Web is making great progress towards 100% HTTPS. We’ve put in a lot of hard work and dealt with some challenges along the way, but at a high level the outlook is quite sunny. I’m incredibly proud to share some of what our organization has accomplished in 2020.

While I’m deeply appreciative of being in this position today, I don’t let it distract me, or our fantastic Board of Directors, from thinking diligently about the risks on the road ahead. A big part of our job is to look into the future, see threats and challenges, and prepare to face them as best we can. I’m sometimes asked what I view as the biggest threat to our organization and our ability to pursue our mission and my answer is simple: being taken for granted.

When digital security and privacy is your goal, ease of use has to be your focus. When we examine why real world systems aren’t secure, it usually isn’t because we don’t have the technological means to secure them. The problem is almost always that the solutions are not easy enough to use, either for implementers or consumers.

HTTPS has been around since the mid-90s but uptake was abysmally slow because SSL/TLS certificates weren’t easy to get or manage. Let’s Encrypt made getting and managing certificates easy and as a result HTTPS adoption rates shot up. Critically, the answer wasn’t to get people to think more about their certificates—we needed to make it possible for people to spend much less time thinking about certificates. Ideally we’d be invisible—server software should just get and manage certificates automatically.

Our next project after Let’s Encrypt is going live shortly: ISRG Prio Services. It’s a system for collecting digital metrics that allows organizations to collect the information they need without any entity having the ability to access any individual user’s data. Much like Let’s Encrypt, it protects people without them having to know anything about it.

Despite 2020 being a year of unprecedented, global challenges, ISRG is well positioned for the years ahead. Our current momentum is possible through new major in-kind donations, nearly 90% of our existing sponsors renewing their support for 2020, funding from the Ford Foundation and the Bill & Melinda Gates Foundation, and by welcoming new major sponsors, including AWS, Thales, and Avast.

When your strategy as a nonprofit is to get out of the way, to offer services that people don’t need to think about, you’re running a real risk that you’ll eventually be taken for granted. There is a tension between wanting your work to be invisible and the need for recognition of its value. If people aren’t aware of how valuable our services are then we may not get the support we need to continue providing them.

How are we going to mitigate this risk? The most important thing we can do is continue to communicate effectively with people who are in a position to understand our work and support it. The most important things you can do as a supporter include being an advocate for your company sponsoring us, making an individual donation, or going over this annual report with a few people that you think should know more about us.

On behalf of the hundreds of millions of people benefiting from Let’s Encrypt around the world and our team of sixteen dedicated to this work, thank you for your support.

Extending Android Device Compatibility for Let’s Encrypt Certificates

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/12/21/extending-android-compatibility.html

We’re happy to announce that we have developed a way for older Android devices to retain their ability to visit sites that use Let’s Encrypt certificates after our cross-signed intermediates expire. We are no longer planning any changes in January that may cause compatibility issues for Let’s Encrypt subscribers.

A recurring theme in our posts about our upcoming chain switch has been our concern over the effects on users of Android operating systems prior to 7.1.1, whose devices don’t trust our ISRG Root X1. Thanks to some innovative thinking from our community and our wonderful partners at IdenTrust, we now have a solution that allows us to maintain wide compatibility. Critical to our mission as a nonprofit is to help create a more secure and privacy-respecting Web for as many people as possible. This work brings us closer to that goal.

IdenTrust has agreed to issue a 3-year cross-sign for our ISRG Root X1 from their DST Root CA X3. The new cross-sign will be somewhat novel because it extends beyond the expiration of DST Root CA X3. This solution works because Android intentionally does not enforce the expiration dates of certificates used as trust anchors. ISRG and IdenTrust reached out to our auditors and root programs to review this plan and ensure there weren’t any compliance concerns.

As such, we will be able to provide subscribers with a chain which contains both ISRG Root X1 and DST Root CA X3, ensuring uninterrupted service to all users and avoiding the potential breakage we have been concerned about.

We will not be performing our previously-planned chain switch on January 11th, 2021. Instead, we will be switching to provide this new chain by default in late January or early February. The transition should have no impact on Let’s Encrypt subscribers, much like our switch to our R3 intermediate earlier this month.

Let's Encrypt Issuance Chains

Some additional technical details follow.

I’m an ACME client developer, what do I need to do? If your client handled the X3 to R3 transition smoothly, then you shouldn’t need to take action. Ensure that your client correctly uses the intermediate certificate provided by the ACME API at the end of issuance, and doesn’t retrieve intermediates by other means (e.g. hardcoding them, reusing what is on disk already, or fetching from AIA URLs).

I’m a Let’s Encrypt subscriber, what do I need to do? In the vast majority of cases, nothing. If you want to double check, please ensure that your ACME client is up-to-date.

I use an old Android device, what do I need to do? Nothing! We’re trying to ensure that this change is completely invisible to end-users.

What exactly is changing? Today, when a subscriber fetches their certificate from our API, we provide the following chain by default:
Subscriber Certificate < – R3 < – DST Root CA X3
After this change, we will instead provide this chain by default:
Subscriber Certificate < – R3 < – ISRG Root X1 < – DST Root CA X3
This does mean that the default chain we provide to subscribers will be larger than before, because it contains two intermediates instead of just one. This will make TLS handshakes larger and slightly less efficient, but the tradeoff is worthwhile for the extra compatibility.

But isn’t DST Root CA X3 expiring? The self-signed certificate which represents the DST Root CA X3 keypair is expiring. But browser and OS root stores don’t contain certificates per se, they contain “trust anchors”, and the standards for verifying certificates allow implementations to choose whether or not to use fields on trust anchors. Android has intentionally chosen not to use the notAfter field of trust anchors. Just as our ISRG Root X1 hasn’t been added to older Android trust stores, DST Root CA X3 hasn’t been removed. So it can issue a cross-sign whose validity extends beyond the expiration of its own self-signed certificate without any issues.

What happens when the new cross-sign expires? This new cross-sign will expire in early 2024. Prior to that, perhaps as early as June 2021, we will be making a similar change to what we intended to make this January. When we make that change, subscribers will have the option to continue using DST Root CA X3 by configuring their ACME client to specifically request it. If you are a Let’s Encrypt subscriber who is already working to mitigate the chain switch (e.g. by configuring alternate chains or encouraging your users on old Android devices to install Firefox), continue that work so you’ll be well-positioned in the future.

What about the alternate chain? Today, some ACME clients are able to instead request an alternate chain, if their user has configured it. We currently provide the option of getting the chain:
Subscriber Certificate < – R3 < – ISRG Root X1
We will continue to offer this same chain as an alternate. However, note that most ACME clients don’t yet have a way to select this alternate chain (for example, Certbot selects chains by looking to see if they contain a given Issuer Name, but this chain doesn’t contain any Issuer Names which the high compatibility chain above doesn’t). We’ll be working with ACME client developers to create more flexible chain selection mechanisms going forward.

What does this have to do with the upcoming ECDSA chain? This change is unrelated to our upcoming issuance from the ECDSA-based ISRG Root X2 and E1 intermediate. We look forward to using those to provide smaller, more efficient certificates and chains, but that future offering is not related to this change.

I have more questions, how can I get them answered? Please refer your questions to our community forum.

In the future we hope to share more technical details about how and why this plan works, demonstrate why the expiration of DST Root CA X3 doesn’t affect the security of the web, and what this plan means long-term for our subscribers and end-users on the web.

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Standing on Our Own Two Feet

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/11/11/own-two-feet.html

When a new Certificate Authority (CA) comes on the scene, it faces a conundrum: In order to be useful to people, it needs its root certificate to be trusted by a wide variety of operating systems (OSes) and browsers. However, it can take years for the OSes and browsers to accept the new root certificate, and even longer for people to upgrade their devices to the newer versions that include that change. The common solution: a new CA will often ask an existing, trusted CA for a cross-signature, to quickly get it into being trusted by lots of devices.

Five years ago, when Let’s Encrypt launched, that’s exactly what we did. We got a cross-signature from IdenTrust. Their “DST Root X3” had been around for a long time, and all the major software platforms trusted it already: Windows, Firefox, macOS, Android, iOS, and a variety of Linux distributions. That cross-signature allowed us to start issuing certificates right away, and have them be useful to a lot of people. Without IdenTrust, Let’s Encrypt may have never happened and we are grateful to them for their partnership. Meanwhile, we issued our own root certificate (“ISRG Root X1”) and applied for it to be trusted by the major software platforms.

Now, those software platforms have trusted our root certificate for years. And the DST Root X3 root certificate that we relied on to get us off the ground is going to expire – on September 1, 2021. Fortunately, we’re ready to stand on our own, and rely solely on our own root certificate.

However, this does introduce some compatibility woes. Some software that hasn’t been updated since 2016 (approximately when our root was accepted to many root programs) still doesn’t trust our root certificate, ISRG Root X1. Most notably, this includes versions of Android prior to 7.1.1. That means those older versions of Android will no longer trust certificates issued by Let’s Encrypt.

Android has a long-standing and well known issue with operating system updates. There are lots of Android devices in the world running out-of-date operating systems. The causes are complex and hard to fix: for each phone, the core Android operating system is commonly modified by both the manufacturer and a mobile carrier before an end-user receives it. When there’s an update to Android, both the manufacturer and the mobile carrier have to incorporate those changes into their customized version before sending it out. Often manufacturers decide that’s not worth the effort. The result is bad for the people who buy these devices: many are stuck on operating systems that are years out of date.

Google no longer provides version numbers on its Distribution Dashboard, but you can still get some data by downloading Android Studio. Here’s what the numbers looked like as of September 2020:

Android Version Distribution as of September 2020

Currently, 66.2% of Android devices are running version 7.1 or above. The remaining 33.8% of Android devices will eventually start getting certificate errors when users visit sites that have a Let’s Encrypt certificate. In our communications with large integrators, we have found that this represents around 1-5% of traffic to their sites. Hopefully these numbers will be lower by the time DST Root X3 expires next year, but the change may not be very significant.

What can we do about this? Well, while we’d love to improve the Android update situation, there’s not much we can do there. We also can’t afford to buy the world a new phone. Can we get another cross-signature? We’ve explored this option and it seems unlikely. It’s a big risk for a CA to cross-sign another CA’s certificate, since they become responsible for everything that CA does. That also means the recipient of the cross-signature has to follow all the procedures laid out by the cross-signing CA. It’s important for us to be able to stand on our own. Also, the Android update problem doesn’t seem to be going away. If we commit ourselves to supporting old Android versions, we would commit ourselves to seeking cross-signatures from other CAs indefinitely.

It’s quite a bind. We’re committed to everybody on the planet having secure and privacy-respecting communications. And we know that the people most affected by the Android update problem are those we most want to help – people who may not be able to buy a new phone every four years. Unfortunately, we don’t expect the Android usage numbers to change much prior to ISRG Root X1’s expiration. By raising awareness of this change now, we hope to help our community to find the best path forward.

If You Are a Site Owner

As of January 11, 2021, we’re planning to make a change to our API so that ACME clients will, by default, serve a certificate chain that leads to ISRG Root X1. However, it will also be possible to serve an alternate certificate chain for the same certificate that leads to DST Root X3 and offers broader compatibility. This is implemented via the ACME “alternate” link relation. This is supported by Certbot from version 1.6.0 onwards. If you use a different ACME client, please check your client’s documentation to see if the “alternate” link relation is supported.

There will be site owners that receive complaints from users and we are empathetic to that being not ideal. We’re working hard to alert site owners so you can plan and prepare. We encourage site owners to deploy a temporary fix (switching to the alternate certificate chain) to keep your site working while you evaluate what you need for a long-term solution: whether you need to run a banner asking your Android users on older OSes to install Firefox, stop supporting older Android versions, drop back to HTTP for older Android versions, or switch to a CA that is installed on those older versions.

If You Get Let’s Encrypt Certificates Through Your Hosting Provider

Your hosting provider may be serving the DST Root X3 until September 2021, or they may decide to switch to the certificate chain that leads to ISRG Root X1 after January 11, 2021. Please contact them if you have any questions!

If You Use an Older Version of Android

If you’re on an older version of Android, we recommend you install Firefox Mobile, which supports Android 5.0 and above as of the time of writing.

Why does installing Firefox help? For an Android phone’s built-in browser, the list of trusted root certificates comes from the operating system – which is out of date on these older phones. However, Firefox is currently unique among browsers – it ships with its own list of trusted root certificates. So anyone who installs the latest Firefox version gets the benefit of an up-to-date list of trusted certificate authorities, even if their operating system is out of date.

We appreciate your understanding and support both now and over the years as we continue to grow as a CA, making sure people everywhere have access to encryption. We will provide any future updates on how this root transition affects Android devices via our community forum post. Our community is always ready to help should you have any questions about this change: community.letsencrypt.org.

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

If You Are an App Developer

If you develop an Android app, you can ship an update that adds ISRG Root X1 as a trusted root within the context of your app. There is a discussion about ways to do so in this forum thread, and this GitHub issue (on a third-party repository).

Standing on Our Own Two Feet

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/11/06/own-two-feet.html

When a new Certificate Authority (CA) comes on the scene, it faces a conundrum: In order to be useful to people, it needs its root certificate to be trusted by a wide variety of operating systems (OSes) and browsers. However, it can take years for the OSes and browsers to accept the new root certificate, and even longer for people to upgrade their devices to the newer versions that include that change. The common solution: a new CA will often ask an existing, trusted CA for a cross-signature, to quickly get it into being trusted by lots of devices.

Five years ago, when Let’s Encrypt launched, that’s exactly what we did. We got a cross-signature from IdenTrust. Their “DST Root X3” had been around for a long time, and all the major software platforms trusted it already: Windows, Firefox, macOS, Android, iOS, and a variety of Linux distributions. That cross-signature allowed us to start issuing certificates right away, and have them be useful to a lot of people. Without IdenTrust, Let’s Encrypt may have never happened and we are grateful to them for their partnership. Meanwhile, we issued our own root certificate (“ISRG Root X1”) and applied for it to be trusted by the major software platforms.

Now, those software platforms have trusted our root certificate for years. And the DST Root X3 root certificate that we relied on to get us off the ground is going to expire – on September 1, 2021. Fortunately, we’re ready to stand on our own, and rely solely on our own root certificate.

However, this does introduce some compatibility woes. Some software that hasn’t been updated since 2016 (approximately when our root was accepted to many root programs) still doesn’t trust our root certificate, ISRG Root X1. Most notably, this includes versions of Android prior to 7.1.1. That means those older versions of Android will no longer trust certificates issued by Let’s Encrypt.

Android has a long-standing and well known issue with operating system updates. There are lots of Android devices in the world running out-of-date operating systems. The causes are complex and hard to fix: for each phone, the core Android operating system is commonly modified by both the manufacturer and a mobile carrier before an end-user receives it. When there’s an update to Android, both the manufacturer and the mobile carrier have to incorporate those changes into their customized version before sending it out. Often manufacturers decide that’s not worth the effort. The result is bad for the people who buy these devices: many are stuck on operating systems that are years out of date.

Google no longer provides version numbers on its Distribution Dashboard, but you can still get some data by downloading Android Studio. Here’s what the numbers looked like as of September 2020:

Android Version Distribution as of September 2020

Currently, 66.2% of Android devices are running version 7.1 or above. The remaining 33.8% of Android devices will eventually start getting certificate errors when users visit sites that have a Let’s Encrypt certificate. In our communications with large integrators, we have found that this represents around 1-5% of traffic to their sites. Hopefully these numbers will be lower by the time DST Root X3 expires next year, but the change may not be very significant.

What can we do about this? Well, while we’d love to improve the Android update situation, there’s not much we can do there. We also can’t afford to buy the world a new phone. Can we get another cross-signature? We’ve explored this option and it seems unlikely. It’s a big risk for a CA to cross-sign another CA’s certificate, since they become responsible for everything that CA does. That also means the recipient of the cross-signature has to follow all the procedures laid out by the cross-signing CA. It’s important for us to be able to stand on our own. Also, the Android update problem doesn’t seem to be going away. If we commit ourselves to supporting old Android versions, we would commit ourselves to seeking cross-signatures from other CAs indefinitely.

It’s quite a bind. We’re committed to everybody on the planet having secure and privacy-respecting communications. And we know that the people most affected by the Android update problem are those we most want to help – people who may not be able to buy a new phone every four years. Unfortunately, we don’t expect the Android usage numbers to change much prior to ISRG Root X1’s expiration. By raising awareness of this change now, we hope to help our community to find the best path forward.

If You Are a Site Owner

As of January 11, 2021, we’re planning to make a change to our API so that ACME clients will, by default, serve a certificate chain that leads to ISRG Root X1. However, it will also be possible to serve an alternate certificate chain for the same certificate that leads to DST Root X3 and offers broader compatibility. This is implemented via the ACME “alternate” link relation. This is supported by Certbot from version 1.6.0 onwards. If you use a different ACME client, please check your client’s documentation to see if the “alternate” link relation is supported.

There will be site owners that receive complaints from users and we are empathetic to that being not ideal. We’re working hard to alert site owners so you can plan and prepare. We encourage site owners to deploy a temporary fix (switching to the alternate certificate chain) to keep your site working while you evaluate what you need for a long-term solution: whether you need to run a banner asking your Android users on older OSes to install Firefox, stop supporting older Android versions, drop back to HTTP for older Android versions, or switch to a CA that is installed on those older versions.

If You Get Let’s Encrypt Certificates Through Your Hosting Provider

Your hosting provider may be serving the DST Root X3 until September 2021, or they may decide to switch to the certificate chain that leads to ISRG Root X1 after January 11, 2021. Please contact them if you have any questions!

If You Use an Older Version of Android

If you’re on an older version of Android, we recommend you install Firefox Mobile, which supports Android 5.0 and above as of the time of writing.

Why does installing Firefox help? For an Android phone’s built-in browser, the list of trusted root certificates comes from the operating system – which is out of date on these older phones. However, Firefox is currently unique among browsers – it ships with its own list of trusted root certificates. So anyone who installs the latest Firefox version gets the benefit of an up-to-date list of trusted certificate authorities, even if their operating system is out of date.

We appreciate your understanding and support both now and over the years as we continue to grow as a CA, making sure people everywhere have access to encryption. We will provide any future updates on how this root transition affects Android devices via our community forum post. Our community is always ready to help should you have any questions about this change: community.letsencrypt.org.

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt’s New Root and Intermediate Certificates

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/09/17/new-root-and-intermediates.html

On Thursday, September 3rd, 2020, Let’s Encrypt issued six new certificates:
one root, four intermediates, and one cross-sign. These new certificates are
part of our larger plan to improve privacy on the web, by making ECDSA
end-entity certificates widely available, and by making certificates smaller.

Given that we issue 1.5 million certificates every day,
what makes these ones special? Why did we issue them? How did we issue them?
Let’s answer these questions, and in the process take a tour of how
Certificate Authorities think and work.

The Backstory

Every publicly-trusted Certificate Authority (such as Let’s Encrypt) has at
least one root certificate which is incorporated into various browser and OS
vendors’ (e.g. Mozilla, Google) trusted root stores. This is what allows
users who receive a certificate from a website to confirm that the
certificate was issued by an organization that their browser trusts. But root
certificates, by virtue of their widespread trust and long lives, must have
their corresponding private key carefully protected and stored offline, and
therefore can’t be used to sign things all the time. So every Certificate
Authority (CA) also has some number of “intermediates”, certificates which
are able to issue additional certificates but are not roots, which they use
for day-to-day issuance.

For the last five years,
Let’s Encrypt has had one root: the ISRG Root X1,
which has a 4096-bit RSA key and is valid until 2035.

Over that same time, we’ve had four intermediates: the Let’s Encrypt
Authorities X1, X2,
X3, and X4. The
first two were issued when Let’s Encrypt first began operations in 2015, and
were valid for 5 years. The latter two were issued about a year later, in
2016, and are also valid for 5 years, expiring about this time next year. All
of these intermediates use 2048-bit RSA keys. In addition,
all of these intermediates are cross-signed
by IdenTrust’s DST Root CA X3, another root certificate controlled by a
different certificate authority which is trusted by most root stores.

Finally, we also have the ISRG Root OCSP X1
certificate. This one is a little different – it doesn’t issue any
certificates. Instead, it signs Online Certificate Status Protocol (OCSP)
responses that indicate the intermediate certificates have not been revoked.
This is important because the only other thing capable of signing such
statements is our root itself, and as mentioned above, the root needs to stay
offline and safely secured.

Let’s Encrypt’s hierarchy as of August 2020

The New Certificates

For starters, we’ve issued two new 2048-bit RSA intermediates which we’re
calling R3 and
R4. These are both issued by ISRG Root X1, and
have 5-year lifetimes. They will also be cross-signed by IdenTrust. They’re
basically direct replacements for our current X3 and X4, which are expiring
in a year. We expect to switch our primary issuance pipeline to use R3 later
this year, which won’t have any real effect on issuance or renewal.

The other new certificates are more interesting. First up, we have the new
ISRG Root X2, which has an ECDSA P-384 key
instead of RSA, and is valid until 2040. Issued from that, we have two new
intermediates, E1 and
E2, which are both also ECDSA and are valid
for 5 years.

Notably, these ECDSA intermediates are not cross-signed by IdenTrust’s DST
Root CA X3. Instead, the ISRG Root X2 itself is
cross-signed by our existing ISRG Root X1.
An astute observer might also notice that we have not issued an OCSP Signing
Certificate from ISRG Root X2.

Let’s Encrypt’s hierarchy as of September 2020

Now that we have the technical details out of the way, let’s dive in to why
the new hierarchy looks the way it does.

Why We Issued an ECDSA Root and Intermediates

There are lots of other articles
you can read about the benefits of ECDSA (smaller key sizes for the same
level of security; correspondingly faster encryption, decryption, signing,
and verification operations; and more). But for us, the big benefit comes
from their smaller certificate sizes.

Every connection to a remote domain over https:// requires a TLS handshake.
Every TLS handshake requires that the server provide its certificate.
Validating that certificate requires a certificate chain (the list of all
intermediates up to but not including a trusted root), which is also usually
provided by the server. This means that every connection – and a page
covered in ads and tracking pixels might have dozens or hundreds – ends up
transmitting a large amount of certificate data. And every certificate
contains both its own public key and a signature provided by its issuer.

While a 2048-bit RSA public key is about 256 bytes long, an ECDSA P-384
public key is only about 48 bytes. Similarly, the RSA signature will be
another 256 bytes, while the ECDSA signature will only be 96 bytes. Factoring
in some additional overhead, that’s a savings of nearly 400 bytes per
certificate. Multiply that by how many certificates are in your chain, and
how many connections you get in a day, and the bandwidth savings add up fast.

These savings are a public benefit both for our subscribers – who can be
sites for which bandwidth can be a meaningful cost every month – and for
end-users, who may have limited or metered connections. Bringing privacy to
the whole Web doesn’t just mean making certificates available, it means
making them efficient, too.

As an aside: since we’re concerned about certificate sizes, we’ve also taken
a few other measures to save bytes in our new certificates. We’ve shortened
their Subject Common Names from “Let’s Encrypt Authority X3” to just “R3”,
relying on the previously-redundant Organization Name field to supply the
words “Let’s Encrypt”. We’ve shortened their Authority Information Access
Issuer and CRL Distribution Point URLs, and we’ve dropped their CPS and OCSP
urls entirely. All of this adds up to another approximately 120 bytes of
savings without making any substantive change to the useful information in
the certificate.

Why We Cross-Signed the ECDSA Root

Cross-signing is an important step, bridging the gap between when a new root
certificate is issued and when that root is incorporated into various trust
stores. We know that it is going to take 5 years or so for our new ISRG Root
X2 to be widely trusted itself, so in order for certificates issued by the E1
intermediate to be trusted, there needs to be a cross-sign somewhere in the
chain.

We had basically two options: we could cross-sign the new ISRG Root X2 from
our existing ISRG Root X1, or we could cross-sign the new E1 and E2
intermediates from ISRG Root X1. Let’s examine the pros and cons of each.

Cross-signing the new ISRG Root X2 certificate means that, if a user has ISRG
Root X2 in their trust store, then their full certificate chain will be 100%
ECDSA, giving them fast validation, as discussed above. And over the next few
years, as ISRG Root X2 is incorporated into more and more trust stores,
validation of ECDSA end-entity certificates will get faster without users or
websites having to change anything. The tradeoff though is that, as long as
X2 isn’t in trust stores, user agents will have to validate a chain with two
intermediates: both E1 and X2 chaining up to the X1 root. This takes more
time during certificate validation.

Cross-signing the intermediates directly has the opposite tradeoff. On the
one hand, all of our chains will be the same length, with just one
intermediate between the subscriber certificate and the widely-trusted ISRG
Root X1. But on the other hand, when the ISRG Root X2 does become widely
trusted, we’d have to go through another chain switch
in order for anyone to gain the benefits of an all-ECDSA chain.

In the end, we decided that providing the option of all-ECDSA chains was more
important, and so opted to go with the first option, and cross-sign the ISRG
Root X2 itself.

Why We Didn’t Issue an OCSP Responder

The Online Certificate Status Protocol is a way for user agents to discover,
in real time, whether or not a certificate they’re validating has been
revoked. Whenever a browser wants to know if a certificate is still valid, it
can simply hit a URL contained within the certificate itself and get a yes or
no response, which is signed by another certificate and can be similarly
validated. This is great for end-entity certificates, because the responses
are small and fast, and any given user might care about (and therefore have
to fetch) the validity of wildly different sets of certificates, depending on
what sites they visit.

But intermediate certificates are a tiny subset of all certificates in the
wild, are generally well-known, and are rarely revoked. Because of this, it
can be much more efficient to simply maintain a Certificate Revocation List
(CRL) containing validity information for all well-known intermediates. Our
intermediate certificates all contain a URL from which a browser can fetch
their CRL, and in fact some browsers even aggregate these into their own CRLs
which they distribute with each update. This means that checking the
revocation status of intermediates doesn’t require an extra network round
trip before you can load a site, resulting in a better experience for
everyone.

In fact, a recent change (ballot SC31)
to the Baseline Requirements, which govern CAs, has made it so intermediate
certificates are no longer required to include an OCSP URL; they can now have
their revocation status served solely by CRL. And as noted above, we have
removed the OCSP URL from our new intermediates. As a result, we didn’t need
to issue an OCSP responder signed by ISRG Root X2.

Putting It All Together

Now that we’ve shared our new certificates look the way they do, there’s one
last thing we’d like to mention: how we actually went about issuing them.

The creation of new root and intermediate certificates is a big deal, since
their contents are so regulated and their private keys have to be so
carefully protected. So much so that the act of issuing new ones is called a
“ceremony”. Let’s Encrypt believes strongly in automation,
so we wanted our ceremony to require as little human involvement as possible.

Over the last few months we’ve built a ceremony tool
which, given appropriate configuration, can produce all of the desired keys,
certificates, and requests for cross-signs. We also built a
demo of our ceremony,
showing what our configuration files would be, and allowing anyone to run it
themselves and examine the resulting output. Our SREs put together a replica
network, complete with Hardware Security Modules, and practiced the ceremony
multiple times to ensure it would work flawlessly. We shared this demo with
our technical advisory board, our community, and various mailing lists, and
in the process received valuable feedback that actually influenced some of
the decisions we’ve talked about above! Finally, on September 3rd, our
Executive Director met with SREs at a secure datacenter to execute the whole
ceremony, and record it for future audits.

And now the ceremony is complete. We’ve updated our certificates page
to include details about all of our new certificates, and are beginning the
process of requesting that our new root be incorporated into various trust
stores. We intend to begin issuing with our new intermediates over the coming
weeks, and will post further announcements in our community forum
when we do.

We hope that this has been an interesting and informative tour around our
hierarchy, and we look forward to continuing to improve the internet one
certificate at a time. We’d like to thank IdenTrust for their early and
ongoing support of our vision to change security on the Web for the better.

We depend on contributions from our community of users and supporters in
order to provide our services. If your company or organization would like to
sponsor Let’s Encrypt please
email us at [email protected]. We ask that you make an
individual contribution if it is within your
means.