All posts by Let's Encrypt - Free SSL/TLS Certificates

Preparing to Issue 200 Million Certificates in 24 Hours

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2021/02/10/200m-certs-24hrs.html

On a normal day Let’s Encrypt issues nearly two million certificates. When we think about what essential infrastructure for the Internet needs to be prepared for though, we’re not thinking about normal days. We want to be prepared to respond as best we can to the most difficult situations that might arise. In some of the worst scenarios, we might want to re-issue all of our certificates in a 24 hour period in order to avoid widespread disruptions. That means being prepared to issue 200 million certificates in a day, something no publicly trusted CA has ever done.

We recently completed most of the work and investments needed to issue 200 million certificates in a day and we thought we’d let people know what was involved. All of this was made possible by our sponsors and funders, including major hardware contributions from Cisco, Thales, and Fortinet.

Scenarios

Security and compliance events are a certainty in our industry. We obviously try to minimize them, but since we anticipate they will happen, we spend a lot of time preparing to respond in the best ways possible.

In February of 2020 a bug affecting our compliance caused us to need to revoke and replace three million active certificates. That was approximately 2.6% of all active certificates.

What if that bug had affected all of our certificates? That’s more than 150 million certificates covering more than 240 million domains. What if it had also been a more serious bug, requiring us to revoke and replace all certificates within 24 hours? That’s the kind of worst case scenario we need to be prepared for.

To make matters worse, during an incident it takes some time to evaluate the problem and make decisions, so we would be starting revocation and reissuance somewhat after the beginning of the 24-hour clock. That means we would actually have less time once the momentous decision has been made to revoke and replace 150 million or more certificates.

Infrastructure Improvements

After reviewing our systems, we determined that there were four primary bottlenecks that would prevent us from replacing 200 million certificates in a day. Database performance, internal networking speed, cryptographic signing module (HSM) performance, and bandwidth.

Database Performance

Let’s Encrypt has a primary certificate authority database at the heart of the service we offer. This tracks the status of certificate issuance and the associated accounts. It is write heavy, with plenty of reads as well. At any given time a single database server is the writer, with some reads directed at identical machines with replicas. The single writer non-clustered strategy helps with consistency and reduces complexity, but it means that writes and some reads must operate within the performance constraints of a single machine.

Our previous generation of database servers had dual Intel Xeon E5-2650 v4 CPUs, 24 physical cores in total. They had 1TB memory with 24 3.8TB SSDs connected via SATA in a RAID 10 configuration. These worked fine for daily issuance but would not be able to handle replacing all of our certificates in a single day.

We have replaced them with a new generation of database server from Dell featuring dual AMD EPYC 7542 CPUs, 64 physical cores in total. These machines have 2TB of faster RAM. Much faster CPUs and double the memory is great, but the really interesting thing about these machines is that the EPYC CPUs provide 128 PCIe4 lanes each. This means we could pack in 24 6.4TB NVME drives for massive I/O performance. There is no viable hardware RAID for NVME, so we’ve switched to ZFS to provide the data protection we need.

Internal Networking

Let’s Encrypt infrastructure was originally built with 1G copper networking. Between bonding multiple connections for 2G performance, and use of the very limited number of 10G ports available on our switches, it served us pretty well until 2020.

By 2020 the volume of data we were moving around internally was too much to handle efficiently with 1G copper. Some normal operations were significantly slower than we’d like (e.g. backups, replicas), and during incidents the 1G network could cause significant delays in our response.

We originally looked into upgrading to 10G, but learned that upgrading to 25G fiber wasn’t much more expensive. Cisco ended up generously donating most of the switches and equipment we needed for this upgrade, and after replacing a lot of server NICs Let’s Encrypt is now running on a 25G fiber network!

Funny story – back in 2014 Cisco had donated really nice 10G fiber switches to use in building the original Let’s Encrypt infrastructure. Our cabinets at the time had an unusually short depth though, and the 10G switches didn’t physically fit. We had to return them for physically smaller 1G switches. 1G seemed like plenty at the time. We have since moved to new cabinets with standard depth!

HSM Performance

Each Let’s Encrypt data center has a pair of Luna HSMs that sign all certificates and their OCSP responses. If we want to revoke and reissue 200 million certificates, we need the Luna HSMs to perform the following cryptographic signing operations:

  • 200 million OCSP response signatures for revocation
  • 200 million certificate signatures for replacement certificates
  • 200 million OCSP response signatures for the new certificates

That means we need to perform 600 million cryptographic signatures in 24 hours or less, with some performance overhead to account for request clustering.

We need to assume that during incidents we might only have one data center online, so when we’re thinking about HSM performance we’re thinking about what we can do with just one pair. Our previous HSMs could perform approximately 1,100 signing operations each per second, 2,200 between the pair. That’s 190,080,000 signatures in 24 hours, working at full capacity. It’s under half of what we would need, and take more time than we might have.

In order to get to where we need to be Thales generously donated new HSMs with about 10x the performance – approximately 10,000 signing operations per second, 20,000 between the pair. That means we can now perform 864,000,000 signing operations in 24 hours from a single data center.

Bandwidth

Issuing a certificate is not particularly bandwidth intensive, but during an incident we typically use a lot more bandwidth for systems recovery and analysis. We move large volumes of logs and other files between data centers and the cloud for analysis and forensic purposes. We may need to synchronize large databases. We create copies and additional back-ups. If the connections in our data centers are slow, it slows down our response.

After determining that data center connection speed could add significantly to our response time, we increased bandwidth accordingly. Fortinet helped provide hardware that helps us protect and manage these higher capacity connections.

API Extension

In order to get all those certificates replaced, we need an efficient and automated way to notify ACME clients that they should perform early renewal. Normally ACME clients renew their certificates when one third of their lifetime is remaining, and don’t contact our servers otherwise. We published a draft extension to ACME last year that describes a way for clients to regularly poll ACME servers to find out about early-renewal events. We plan to polish up that draft, implement, and collaborate with clients and large integrators to get it implemented on the client side.

Supporting Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

The Next Gen Database Servers Powering Let’s Encrypt

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2021/01/21/next-gen-database-servers.html

Let’s Encrypt helps to protect a huge portion of the Web by providing TLS certificates to more than 235 million websites. A database is at the heart of how Let’s Encrypt manages certificate issuance. If this database isn’t performing well enough, it can cause API errors and timeouts for our subscribers. Database performance is the single most critical factor in our ability to scale while meeting service level objectives. In late 2020, we upgraded our database servers and we’ve been very happy with the results.

What exactly are we doing with these servers?

Our CA software, Boulder, uses MySQL-style schemas and queries to manage subscriber accounts and the entire certificate issuance process. It’s designed to work with a single MySQL, MariaDB, or Percona database. We currently use MariaDB, with the InnoDB database engine.

We run the CA against a single database in order to minimize complexity. Minimizing complexity is good for security, reliability, and reducing maintenance burden. We have a number of replicas of the database active at any given time, and we direct some read operations to replica database servers to reduce load on the primary.

One consequence of this design is that our database machines need to be pretty powerful. Eventually we may need to shard or break the single database into multiple databases, but hardware advancements have allowed us to avoid that so far.

Hardware Specifications

The previous generation of database hardware was powerful but it was regularly being pushed to its limits. For the next generation, we wanted to more than double almost every performance metric in the same 2U form factor. In order to pull that off, we needed AMD EPYC chips and Dell’s PowerEdge R7525 was ideal. Here are the specifications:

Previous Generation Next Generation
CPU 2x Intel Xeon E5-2650
Total 24 cores / 48 threads
2x AMD EPYC 7542
Total 64 cores / 128 threads
Memory 1TB 2400MT/s 2TB 3200MT/s
Storage 24x 3.8TB Samsung PM883
SATA SSD
560/540 MB/s read/write
24x 6.4TB Intel P4610
NVMe SSD
3200/3200 MB/s read/write


Dell PowerEdge R7525 Chassis
Dell PowerEdge R7525 internals. The two silver rectangles in the middle are the CPUs. The RAM sticks, each 64GB, are above and below the CPUs. The 24x NVMe drives are in the front of the server, on the far right.

By going with AMD EPYC, we were able to get 64 physical CPU cores while keeping clock speeds high: 2.9GHz base with 3.4GHz boost. More importantly, EPYC provides 128 PCIe v4.0 lanes, which allows us to put 24 NVMe drives in a single machine. NVMe is incredibly fast (~5.7x faster than the SATA SSDs in our previous-gen database servers) because it uses PCIe instead of SATA. However, PCIe lanes are typically very limited: modern consumer chips typically have only 16 lanes, and Intel’s Xeon chips have 48. By providing 128 PCI lanes per chip (v4.0, no less), AMD EPYC has made it possible to pack large numbers of NVMe drives into a single machine. We’ll talk more about NVMe later.

Performance Impact

We’ll start by looking at our median time to process a request because it best reflects subscribers’ experience. Before the upgrade, we turned around the median API request in ~90 ms. The upgrade decimated that metric to ~9 ms!

API Latency

We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from /proc/stat) averaged over 90%:

CPU Usage Before Upgrade

The new AMD EPYC CPUs sit at about 25%. You can see in this graph where we promoted the new database server from replica (read-only) to primary (read/write) on September 15.

CPU Usage After Upgrade

The upgrade greatly reduced our overall database latency. The average query response time (from INFORMATION_SCHEMA) used to be ~0.45ms.

Database Latency Before Upgrade

Queries now average three times faster, about 0.15ms.

Database Latency After Upgrade

OpenZFS and NVMe

NVMe drives are becoming increasingly popular because of their incredible performance. Up until recently, though, it was nearly impossible to get many of them in a single machine because NVMe uses PCIe lanes. Those were very limited: Intel’s Xeon processors come with just 48 PCIe v3 lanes, and a number of those are used up by the chipset and add-on cards such as network adapters and GPUs. You can’t fit many NVMe drives in the remaining lanes.

AMD’s latest generation of EPYC processors come with 128 PCIe lanes – more than double what Intel offers – and they’re PCIe v4! This is enough to pack a 2U server full of NVMe drives (24 in our case).

Once you have a server full of NVMe drives, you have to decide how to manage them. Our previous generation of database servers used hardware RAID in a RAID-10 configuration, but there is no effective hardware RAID for NVMe, so we needed another solution. One option was software RAID (Linux mdraid), but we got several recommendations for OpenZFS and decided to give it a shot. We’ve been very happy with it!

There wasn’t a lot of information out there about how best to set up and optimize OpenZFS for a pool of NVMe drives and a database workload, so we want to share what we learned. You can find detailed information about our setup in this GitHub repository.

Conclusion

This database upgrade was necessary as more people rely on Let’s Encrypt for the security and privacy that TLS/SSL provides. The equipment is quite expensive and it was a sizable undertaking for our SRE team to plan and execute the transition, but we gained a lot through the process.

Support Let’s Encrypt

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

A Year-End Letter from the Executive Director of Let’s Encrypt and ISRG

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/12/28/executive-director-letter.html

This letter was originally published in our 2020 annual report.

ISRG’s first project, Let’s Encrypt, has been wildly successful. We’re now helping to secure more than 225 million websites and the Web is making great progress towards 100% HTTPS. We’ve put in a lot of hard work and dealt with some challenges along the way, but at a high level the outlook is quite sunny. I’m incredibly proud to share some of what our organization has accomplished in 2020.

While I’m deeply appreciative of being in this position today, I don’t let it distract me, or our fantastic Board of Directors, from thinking diligently about the risks on the road ahead. A big part of our job is to look into the future, see threats and challenges, and prepare to face them as best we can. I’m sometimes asked what I view as the biggest threat to our organization and our ability to pursue our mission and my answer is simple: being taken for granted.

When digital security and privacy is your goal, ease of use has to be your focus. When we examine why real world systems aren’t secure, it usually isn’t because we don’t have the technological means to secure them. The problem is almost always that the solutions are not easy enough to use, either for implementers or consumers.

HTTPS has been around since the mid-90s but uptake was abysmally slow because SSL/TLS certificates weren’t easy to get or manage. Let’s Encrypt made getting and managing certificates easy and as a result HTTPS adoption rates shot up. Critically, the answer wasn’t to get people to think more about their certificates—we needed to make it possible for people to spend much less time thinking about certificates. Ideally we’d be invisible—server software should just get and manage certificates automatically.

Our next project after Let’s Encrypt is going live shortly: ISRG Prio Services. It’s a system for collecting digital metrics that allows organizations to collect the information they need without any entity having the ability to access any individual user’s data. Much like Let’s Encrypt, it protects people without them having to know anything about it.

Despite 2020 being a year of unprecedented, global challenges, ISRG is well positioned for the years ahead. Our current momentum is possible through new major in-kind donations, nearly 90% of our existing sponsors renewing their support for 2020, funding from the Ford Foundation and the Bill & Melinda Gates Foundation, and by welcoming new major sponsors, including AWS, Thales, and Avast.

When your strategy as a nonprofit is to get out of the way, to offer services that people don’t need to think about, you’re running a real risk that you’ll eventually be taken for granted. There is a tension between wanting your work to be invisible and the need for recognition of its value. If people aren’t aware of how valuable our services are then we may not get the support we need to continue providing them.

How are we going to mitigate this risk? The most important thing we can do is continue to communicate effectively with people who are in a position to understand our work and support it. The most important things you can do as a supporter include being an advocate for your company sponsoring us, making an individual donation, or going over this annual report with a few people that you think should know more about us.

On behalf of the hundreds of millions of people benefiting from Let’s Encrypt around the world and our team of sixteen dedicated to this work, thank you for your support.

Extending Android Device Compatibility for Let’s Encrypt Certificates

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/12/21/extending-android-compatibility.html

We’re happy to announce that we have developed a way for older Android devices to retain their ability to visit sites that use Let’s Encrypt certificates after our cross-signed intermediates expire. We are no longer planning any changes in January that may cause compatibility issues for Let’s Encrypt subscribers.

A recurring theme in our posts about our upcoming chain switch has been our concern over the effects on users of Android operating systems prior to 7.1.1, whose devices don’t trust our ISRG Root X1. Thanks to some innovative thinking from our community and our wonderful partners at IdenTrust, we now have a solution that allows us to maintain wide compatibility. Critical to our mission as a nonprofit is to help create a more secure and privacy-respecting Web for as many people as possible. This work brings us closer to that goal.

IdenTrust has agreed to issue a 3-year cross-sign for our ISRG Root X1 from their DST Root CA X3. The new cross-sign will be somewhat novel because it extends beyond the expiration of DST Root CA X3. This solution works because Android intentionally does not enforce the expiration dates of certificates used as trust anchors. ISRG and IdenTrust reached out to our auditors and root programs to review this plan and ensure there weren’t any compliance concerns.

As such, we will be able to provide subscribers with a chain which contains both ISRG Root X1 and DST Root CA X3, ensuring uninterrupted service to all users and avoiding the potential breakage we have been concerned about.

We will not be performing our previously-planned chain switch on January 11th, 2021. Instead, we will be switching to provide this new chain by default in late January or early February. The transition should have no impact on Let’s Encrypt subscribers, much like our switch to our R3 intermediate earlier this month.

Let's Encrypt Issuance Chains

Some additional technical details follow.

I’m an ACME client developer, what do I need to do? If your client handled the X3 to R3 transition smoothly, then you shouldn’t need to take action. Ensure that your client correctly uses the intermediate certificate provided by the ACME API at the end of issuance, and doesn’t retrieve intermediates by other means (e.g. hardcoding them, reusing what is on disk already, or fetching from AIA URLs).

I’m a Let’s Encrypt subscriber, what do I need to do? In the vast majority of cases, nothing. If you want to double check, please ensure that your ACME client is up-to-date.

I use an old Android device, what do I need to do? Nothing! We’re trying to ensure that this change is completely invisible to end-users.

What exactly is changing? Today, when a subscriber fetches their certificate from our API, we provide the following chain by default:
Subscriber Certificate < – R3 < – DST Root CA X3
After this change, we will instead provide this chain by default:
Subscriber Certificate < – R3 < – ISRG Root X1 < – DST Root CA X3
This does mean that the default chain we provide to subscribers will be larger than before, because it contains two intermediates instead of just one. This will make TLS handshakes larger and slightly less efficient, but the tradeoff is worthwhile for the extra compatibility.

But isn’t DST Root CA X3 expiring? The self-signed certificate which represents the DST Root CA X3 keypair is expiring. But browser and OS root stores don’t contain certificates per se, they contain “trust anchors”, and the standards for verifying certificates allow implementations to choose whether or not to use fields on trust anchors. Android has intentionally chosen not to use the notAfter field of trust anchors. Just as our ISRG Root X1 hasn’t been added to older Android trust stores, DST Root CA X3 hasn’t been removed. So it can issue a cross-sign whose validity extends beyond the expiration of its own self-signed certificate without any issues.

What happens when the new cross-sign expires? This new cross-sign will expire in early 2024. Prior to that, perhaps as early as June 2021, we will be making a similar change to what we intended to make this January. When we make that change, subscribers will have the option to continue using DST Root CA X3 by configuring their ACME client to specifically request it. If you are a Let’s Encrypt subscriber who is already working to mitigate the chain switch (e.g. by configuring alternate chains or encouraging your users on old Android devices to install Firefox), continue that work so you’ll be well-positioned in the future.

What about the alternate chain? Today, some ACME clients are able to instead request an alternate chain, if their user has configured it. We currently provide the option of getting the chain:
Subscriber Certificate < – R3 < – ISRG Root X1
We will continue to offer this same chain as an alternate. However, note that most ACME clients don’t yet have a way to select this alternate chain (for example, Certbot selects chains by looking to see if they contain a given Issuer Name, but this chain doesn’t contain any Issuer Names which the high compatibility chain above doesn’t). We’ll be working with ACME client developers to create more flexible chain selection mechanisms going forward.

What does this have to do with the upcoming ECDSA chain? This change is unrelated to our upcoming issuance from the ECDSA-based ISRG Root X2 and E1 intermediate. We look forward to using those to provide smaller, more efficient certificates and chains, but that future offering is not related to this change.

I have more questions, how can I get them answered? Please refer your questions to our community forum.

In the future we hope to share more technical details about how and why this plan works, demonstrate why the expiration of DST Root CA X3 doesn’t affect the security of the web, and what this plan means long-term for our subscribers and end-users on the web.

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Standing on Our Own Two Feet

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/11/11/own-two-feet.html

When a new Certificate Authority (CA) comes on the scene, it faces a conundrum: In order to be useful to people, it needs its root certificate to be trusted by a wide variety of operating systems (OSes) and browsers. However, it can take years for the OSes and browsers to accept the new root certificate, and even longer for people to upgrade their devices to the newer versions that include that change. The common solution: a new CA will often ask an existing, trusted CA for a cross-signature, to quickly get it into being trusted by lots of devices.

Five years ago, when Let’s Encrypt launched, that’s exactly what we did. We got a cross-signature from IdenTrust. Their “DST Root X3” had been around for a long time, and all the major software platforms trusted it already: Windows, Firefox, macOS, Android, iOS, and a variety of Linux distributions. That cross-signature allowed us to start issuing certificates right away, and have them be useful to a lot of people. Without IdenTrust, Let’s Encrypt may have never happened and we are grateful to them for their partnership. Meanwhile, we issued our own root certificate (“ISRG Root X1”) and applied for it to be trusted by the major software platforms.

Now, those software platforms have trusted our root certificate for years. And the DST Root X3 root certificate that we relied on to get us off the ground is going to expire – on September 1, 2021. Fortunately, we’re ready to stand on our own, and rely solely on our own root certificate.

However, this does introduce some compatibility woes. Some software that hasn’t been updated since 2016 (approximately when our root was accepted to many root programs) still doesn’t trust our root certificate, ISRG Root X1. Most notably, this includes versions of Android prior to 7.1.1. That means those older versions of Android will no longer trust certificates issued by Let’s Encrypt.

Android has a long-standing and well known issue with operating system updates. There are lots of Android devices in the world running out-of-date operating systems. The causes are complex and hard to fix: for each phone, the core Android operating system is commonly modified by both the manufacturer and a mobile carrier before an end-user receives it. When there’s an update to Android, both the manufacturer and the mobile carrier have to incorporate those changes into their customized version before sending it out. Often manufacturers decide that’s not worth the effort. The result is bad for the people who buy these devices: many are stuck on operating systems that are years out of date.

Google no longer provides version numbers on its Distribution Dashboard, but you can still get some data by downloading Android Studio. Here’s what the numbers looked like as of September 2020:

Android Version Distribution as of September 2020

Currently, 66.2% of Android devices are running version 7.1 or above. The remaining 33.8% of Android devices will eventually start getting certificate errors when users visit sites that have a Let’s Encrypt certificate. In our communications with large integrators, we have found that this represents around 1-5% of traffic to their sites. Hopefully these numbers will be lower by the time DST Root X3 expires next year, but the change may not be very significant.

What can we do about this? Well, while we’d love to improve the Android update situation, there’s not much we can do there. We also can’t afford to buy the world a new phone. Can we get another cross-signature? We’ve explored this option and it seems unlikely. It’s a big risk for a CA to cross-sign another CA’s certificate, since they become responsible for everything that CA does. That also means the recipient of the cross-signature has to follow all the procedures laid out by the cross-signing CA. It’s important for us to be able to stand on our own. Also, the Android update problem doesn’t seem to be going away. If we commit ourselves to supporting old Android versions, we would commit ourselves to seeking cross-signatures from other CAs indefinitely.

It’s quite a bind. We’re committed to everybody on the planet having secure and privacy-respecting communications. And we know that the people most affected by the Android update problem are those we most want to help – people who may not be able to buy a new phone every four years. Unfortunately, we don’t expect the Android usage numbers to change much prior to ISRG Root X1’s expiration. By raising awareness of this change now, we hope to help our community to find the best path forward.

If You Are a Site Owner

As of January 11, 2021, we’re planning to make a change to our API so that ACME clients will, by default, serve a certificate chain that leads to ISRG Root X1. However, it will also be possible to serve an alternate certificate chain for the same certificate that leads to DST Root X3 and offers broader compatibility. This is implemented via the ACME “alternate” link relation. This is supported by Certbot from version 1.6.0 onwards. If you use a different ACME client, please check your client’s documentation to see if the “alternate” link relation is supported.

There will be site owners that receive complaints from users and we are empathetic to that being not ideal. We’re working hard to alert site owners so you can plan and prepare. We encourage site owners to deploy a temporary fix (switching to the alternate certificate chain) to keep your site working while you evaluate what you need for a long-term solution: whether you need to run a banner asking your Android users on older OSes to install Firefox, stop supporting older Android versions, drop back to HTTP for older Android versions, or switch to a CA that is installed on those older versions.

If You Get Let’s Encrypt Certificates Through Your Hosting Provider

Your hosting provider may be serving the DST Root X3 until September 2021, or they may decide to switch to the certificate chain that leads to ISRG Root X1 after January 11, 2021. Please contact them if you have any questions!

If You Use an Older Version of Android

If you’re on an older version of Android, we recommend you install Firefox Mobile, which supports Android 5.0 and above as of the time of writing.

Why does installing Firefox help? For an Android phone’s built-in browser, the list of trusted root certificates comes from the operating system – which is out of date on these older phones. However, Firefox is currently unique among browsers – it ships with its own list of trusted root certificates. So anyone who installs the latest Firefox version gets the benefit of an up-to-date list of trusted certificate authorities, even if their operating system is out of date.

We appreciate your understanding and support both now and over the years as we continue to grow as a CA, making sure people everywhere have access to encryption. We will provide any future updates on how this root transition affects Android devices via our community forum post. Our community is always ready to help should you have any questions about this change: community.letsencrypt.org.

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

If You Are an App Developer

If you develop an Android app, you can ship an update that adds ISRG Root X1 as a trusted root within the context of your app. There is a discussion about ways to do so in this forum thread, and this GitHub issue (on a third-party repository).

Standing on Our Own Two Feet

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/11/06/own-two-feet.html

When a new Certificate Authority (CA) comes on the scene, it faces a conundrum: In order to be useful to people, it needs its root certificate to be trusted by a wide variety of operating systems (OSes) and browsers. However, it can take years for the OSes and browsers to accept the new root certificate, and even longer for people to upgrade their devices to the newer versions that include that change. The common solution: a new CA will often ask an existing, trusted CA for a cross-signature, to quickly get it into being trusted by lots of devices.

Five years ago, when Let’s Encrypt launched, that’s exactly what we did. We got a cross-signature from IdenTrust. Their “DST Root X3” had been around for a long time, and all the major software platforms trusted it already: Windows, Firefox, macOS, Android, iOS, and a variety of Linux distributions. That cross-signature allowed us to start issuing certificates right away, and have them be useful to a lot of people. Without IdenTrust, Let’s Encrypt may have never happened and we are grateful to them for their partnership. Meanwhile, we issued our own root certificate (“ISRG Root X1”) and applied for it to be trusted by the major software platforms.

Now, those software platforms have trusted our root certificate for years. And the DST Root X3 root certificate that we relied on to get us off the ground is going to expire – on September 1, 2021. Fortunately, we’re ready to stand on our own, and rely solely on our own root certificate.

However, this does introduce some compatibility woes. Some software that hasn’t been updated since 2016 (approximately when our root was accepted to many root programs) still doesn’t trust our root certificate, ISRG Root X1. Most notably, this includes versions of Android prior to 7.1.1. That means those older versions of Android will no longer trust certificates issued by Let’s Encrypt.

Android has a long-standing and well known issue with operating system updates. There are lots of Android devices in the world running out-of-date operating systems. The causes are complex and hard to fix: for each phone, the core Android operating system is commonly modified by both the manufacturer and a mobile carrier before an end-user receives it. When there’s an update to Android, both the manufacturer and the mobile carrier have to incorporate those changes into their customized version before sending it out. Often manufacturers decide that’s not worth the effort. The result is bad for the people who buy these devices: many are stuck on operating systems that are years out of date.

Google no longer provides version numbers on its Distribution Dashboard, but you can still get some data by downloading Android Studio. Here’s what the numbers looked like as of September 2020:

Android Version Distribution as of September 2020

Currently, 66.2% of Android devices are running version 7.1 or above. The remaining 33.8% of Android devices will eventually start getting certificate errors when users visit sites that have a Let’s Encrypt certificate. In our communications with large integrators, we have found that this represents around 1-5% of traffic to their sites. Hopefully these numbers will be lower by the time DST Root X3 expires next year, but the change may not be very significant.

What can we do about this? Well, while we’d love to improve the Android update situation, there’s not much we can do there. We also can’t afford to buy the world a new phone. Can we get another cross-signature? We’ve explored this option and it seems unlikely. It’s a big risk for a CA to cross-sign another CA’s certificate, since they become responsible for everything that CA does. That also means the recipient of the cross-signature has to follow all the procedures laid out by the cross-signing CA. It’s important for us to be able to stand on our own. Also, the Android update problem doesn’t seem to be going away. If we commit ourselves to supporting old Android versions, we would commit ourselves to seeking cross-signatures from other CAs indefinitely.

It’s quite a bind. We’re committed to everybody on the planet having secure and privacy-respecting communications. And we know that the people most affected by the Android update problem are those we most want to help – people who may not be able to buy a new phone every four years. Unfortunately, we don’t expect the Android usage numbers to change much prior to ISRG Root X1’s expiration. By raising awareness of this change now, we hope to help our community to find the best path forward.

If You Are a Site Owner

As of January 11, 2021, we’re planning to make a change to our API so that ACME clients will, by default, serve a certificate chain that leads to ISRG Root X1. However, it will also be possible to serve an alternate certificate chain for the same certificate that leads to DST Root X3 and offers broader compatibility. This is implemented via the ACME “alternate” link relation. This is supported by Certbot from version 1.6.0 onwards. If you use a different ACME client, please check your client’s documentation to see if the “alternate” link relation is supported.

There will be site owners that receive complaints from users and we are empathetic to that being not ideal. We’re working hard to alert site owners so you can plan and prepare. We encourage site owners to deploy a temporary fix (switching to the alternate certificate chain) to keep your site working while you evaluate what you need for a long-term solution: whether you need to run a banner asking your Android users on older OSes to install Firefox, stop supporting older Android versions, drop back to HTTP for older Android versions, or switch to a CA that is installed on those older versions.

If You Get Let’s Encrypt Certificates Through Your Hosting Provider

Your hosting provider may be serving the DST Root X3 until September 2021, or they may decide to switch to the certificate chain that leads to ISRG Root X1 after January 11, 2021. Please contact them if you have any questions!

If You Use an Older Version of Android

If you’re on an older version of Android, we recommend you install Firefox Mobile, which supports Android 5.0 and above as of the time of writing.

Why does installing Firefox help? For an Android phone’s built-in browser, the list of trusted root certificates comes from the operating system – which is out of date on these older phones. However, Firefox is currently unique among browsers – it ships with its own list of trusted root certificates. So anyone who installs the latest Firefox version gets the benefit of an up-to-date list of trusted certificate authorities, even if their operating system is out of date.

We appreciate your understanding and support both now and over the years as we continue to grow as a CA, making sure people everywhere have access to encryption. We will provide any future updates on how this root transition affects Android devices via our community forum post. Our community is always ready to help should you have any questions about this change: community.letsencrypt.org.

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt’s New Root and Intermediate Certificates

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/09/17/new-root-and-intermediates.html

On Thursday, September 3rd, 2020, Let’s Encrypt issued six new certificates:
one root, four intermediates, and one cross-sign. These new certificates are
part of our larger plan to improve privacy on the web, by making ECDSA
end-entity certificates widely available, and by making certificates smaller.

Given that we issue 1.5 million certificates every day,
what makes these ones special? Why did we issue them? How did we issue them?
Let’s answer these questions, and in the process take a tour of how
Certificate Authorities think and work.

The Backstory

Every publicly-trusted Certificate Authority (such as Let’s Encrypt) has at
least one root certificate which is incorporated into various browser and OS
vendors’ (e.g. Mozilla, Google) trusted root stores. This is what allows
users who receive a certificate from a website to confirm that the
certificate was issued by an organization that their browser trusts. But root
certificates, by virtue of their widespread trust and long lives, must have
their corresponding private key carefully protected and stored offline, and
therefore can’t be used to sign things all the time. So every Certificate
Authority (CA) also has some number of “intermediates”, certificates which
are able to issue additional certificates but are not roots, which they use
for day-to-day issuance.

For the last five years,
Let’s Encrypt has had one root: the ISRG Root X1,
which has a 4096-bit RSA key and is valid until 2035.

Over that same time, we’ve had four intermediates: the Let’s Encrypt
Authorities X1, X2,
X3, and X4. The
first two were issued when Let’s Encrypt first began operations in 2015, and
were valid for 5 years. The latter two were issued about a year later, in
2016, and are also valid for 5 years, expiring about this time next year. All
of these intermediates use 2048-bit RSA keys. In addition,
all of these intermediates are cross-signed
by IdenTrust’s DST Root CA X3, another root certificate controlled by a
different certificate authority which is trusted by most root stores.

Finally, we also have the ISRG Root OCSP X1
certificate. This one is a little different – it doesn’t issue any
certificates. Instead, it signs Online Certificate Status Protocol (OCSP)
responses that indicate the intermediate certificates have not been revoked.
This is important because the only other thing capable of signing such
statements is our root itself, and as mentioned above, the root needs to stay
offline and safely secured.

Let’s Encrypt’s hierarchy as of August 2020

The New Certificates

For starters, we’ve issued two new 2048-bit RSA intermediates which we’re
calling R3 and
R4. These are both issued by ISRG Root X1, and
have 5-year lifetimes. They will also be cross-signed by IdenTrust. They’re
basically direct replacements for our current X3 and X4, which are expiring
in a year. We expect to switch our primary issuance pipeline to use R3 later
this year, which won’t have any real effect on issuance or renewal.

The other new certificates are more interesting. First up, we have the new
ISRG Root X2, which has an ECDSA P-384 key
instead of RSA, and is valid until 2040. Issued from that, we have two new
intermediates, E1 and
E2, which are both also ECDSA and are valid
for 5 years.

Notably, these ECDSA intermediates are not cross-signed by IdenTrust’s DST
Root CA X3. Instead, the ISRG Root X2 itself is
cross-signed by our existing ISRG Root X1.
An astute observer might also notice that we have not issued an OCSP Signing
Certificate from ISRG Root X2.

Let’s Encrypt’s hierarchy as of September 2020

Now that we have the technical details out of the way, let’s dive in to why
the new hierarchy looks the way it does.

Why We Issued an ECDSA Root and Intermediates

There are lots of other articles
you can read about the benefits of ECDSA (smaller key sizes for the same
level of security; correspondingly faster encryption, decryption, signing,
and verification operations; and more). But for us, the big benefit comes
from their smaller certificate sizes.

Every connection to a remote domain over https:// requires a TLS handshake.
Every TLS handshake requires that the server provide its certificate.
Validating that certificate requires a certificate chain (the list of all
intermediates up to but not including a trusted root), which is also usually
provided by the server. This means that every connection – and a page
covered in ads and tracking pixels might have dozens or hundreds – ends up
transmitting a large amount of certificate data. And every certificate
contains both its own public key and a signature provided by its issuer.

While a 2048-bit RSA public key is about 256 bytes long, an ECDSA P-384
public key is only about 48 bytes. Similarly, the RSA signature will be
another 256 bytes, while the ECDSA signature will only be 96 bytes. Factoring
in some additional overhead, that’s a savings of nearly 400 bytes per
certificate. Multiply that by how many certificates are in your chain, and
how many connections you get in a day, and the bandwidth savings add up fast.

These savings are a public benefit both for our subscribers – who can be
sites for which bandwidth can be a meaningful cost every month – and for
end-users, who may have limited or metered connections. Bringing privacy to
the whole Web doesn’t just mean making certificates available, it means
making them efficient, too.

As an aside: since we’re concerned about certificate sizes, we’ve also taken
a few other measures to save bytes in our new certificates. We’ve shortened
their Subject Common Names from “Let’s Encrypt Authority X3” to just “R3”,
relying on the previously-redundant Organization Name field to supply the
words “Let’s Encrypt”. We’ve shortened their Authority Information Access
Issuer and CRL Distribution Point URLs, and we’ve dropped their CPS and OCSP
urls entirely. All of this adds up to another approximately 120 bytes of
savings without making any substantive change to the useful information in
the certificate.

Why We Cross-Signed the ECDSA Root

Cross-signing is an important step, bridging the gap between when a new root
certificate is issued and when that root is incorporated into various trust
stores. We know that it is going to take 5 years or so for our new ISRG Root
X2 to be widely trusted itself, so in order for certificates issued by the E1
intermediate to be trusted, there needs to be a cross-sign somewhere in the
chain.

We had basically two options: we could cross-sign the new ISRG Root X2 from
our existing ISRG Root X1, or we could cross-sign the new E1 and E2
intermediates from ISRG Root X1. Let’s examine the pros and cons of each.

Cross-signing the new ISRG Root X2 certificate means that, if a user has ISRG
Root X2 in their trust store, then their full certificate chain will be 100%
ECDSA, giving them fast validation, as discussed above. And over the next few
years, as ISRG Root X2 is incorporated into more and more trust stores,
validation of ECDSA end-entity certificates will get faster without users or
websites having to change anything. The tradeoff though is that, as long as
X2 isn’t in trust stores, user agents will have to validate a chain with two
intermediates: both E1 and X2 chaining up to the X1 root. This takes more
time during certificate validation.

Cross-signing the intermediates directly has the opposite tradeoff. On the
one hand, all of our chains will be the same length, with just one
intermediate between the subscriber certificate and the widely-trusted ISRG
Root X1. But on the other hand, when the ISRG Root X2 does become widely
trusted, we’d have to go through another chain switch
in order for anyone to gain the benefits of an all-ECDSA chain.

In the end, we decided that providing the option of all-ECDSA chains was more
important, and so opted to go with the first option, and cross-sign the ISRG
Root X2 itself.

Why We Didn’t Issue an OCSP Responder

The Online Certificate Status Protocol is a way for user agents to discover,
in real time, whether or not a certificate they’re validating has been
revoked. Whenever a browser wants to know if a certificate is still valid, it
can simply hit a URL contained within the certificate itself and get a yes or
no response, which is signed by another certificate and can be similarly
validated. This is great for end-entity certificates, because the responses
are small and fast, and any given user might care about (and therefore have
to fetch) the validity of wildly different sets of certificates, depending on
what sites they visit.

But intermediate certificates are a tiny subset of all certificates in the
wild, are generally well-known, and are rarely revoked. Because of this, it
can be much more efficient to simply maintain a Certificate Revocation List
(CRL) containing validity information for all well-known intermediates. Our
intermediate certificates all contain a URL from which a browser can fetch
their CRL, and in fact some browsers even aggregate these into their own CRLs
which they distribute with each update. This means that checking the
revocation status of intermediates doesn’t require an extra network round
trip before you can load a site, resulting in a better experience for
everyone.

In fact, a recent change (ballot SC31)
to the Baseline Requirements, which govern CAs, has made it so intermediate
certificates are no longer required to include an OCSP URL; they can now have
their revocation status served solely by CRL. And as noted above, we have
removed the OCSP URL from our new intermediates. As a result, we didn’t need
to issue an OCSP responder signed by ISRG Root X2.

Putting It All Together

Now that we’ve shared our new certificates look the way they do, there’s one
last thing we’d like to mention: how we actually went about issuing them.

The creation of new root and intermediate certificates is a big deal, since
their contents are so regulated and their private keys have to be so
carefully protected. So much so that the act of issuing new ones is called a
“ceremony”. Let’s Encrypt believes strongly in automation,
so we wanted our ceremony to require as little human involvement as possible.

Over the last few months we’ve built a ceremony tool
which, given appropriate configuration, can produce all of the desired keys,
certificates, and requests for cross-signs. We also built a
demo of our ceremony,
showing what our configuration files would be, and allowing anyone to run it
themselves and examine the resulting output. Our SREs put together a replica
network, complete with Hardware Security Modules, and practiced the ceremony
multiple times to ensure it would work flawlessly. We shared this demo with
our technical advisory board, our community, and various mailing lists, and
in the process received valuable feedback that actually influenced some of
the decisions we’ve talked about above! Finally, on September 3rd, our
Executive Director met with SREs at a secure datacenter to execute the whole
ceremony, and record it for future audits.

And now the ceremony is complete. We’ve updated our certificates page
to include details about all of our new certificates, and are beginning the
process of requesting that our new root be incorporated into various trust
stores. We intend to begin issuing with our new intermediates over the coming
weeks, and will post further announcements in our community forum
when we do.

We hope that this has been an interesting and informative tour around our
hierarchy, and we look forward to continuing to improve the internet one
certificate at a time. We’d like to thank IdenTrust for their early and
ongoing support of our vision to change security on the Web for the better.

We depend on contributions from our community of users and supporters in
order to provide our services. If your company or organization would like to
sponsor Let’s Encrypt please
email us at [email protected]. We ask that you make an
individual contribution if it is within your
means.

Let’s Encrypt Has Issued a Billion Certificates

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/02/27/one-billion-certs.html

We issued our billionth certificate on February 27, 2020. We’re going to use this big round number as an opportunity to reflect on what has changed for us, and for the Internet, leading up to this event. In particular, we want to talk about what has happened since the last time we talked about a big round number of certificates – one hundred million.

One thing that’s different now is that the Web is much more encrypted than it was. In June of 2017 approximately 58% of page loads used HTTPS globally, 64% in the United States. Today 81% of page loads use HTTPS globally, and we’re at 91% in the United States! This is an incredible achievement. That’s a lot more privacy and security for everybody.

Another thing that’s different is that our organization has grown a bit, but not by much! In June of 2017 we were serving approximately 46M websites, and we did so with 11 full time staff and an annual budget of $2.61M. Today we serve nearly 192M websites with 13 full time staff and an annual budget of approximately $3.35M. This means we’re serving more than 4x the websites with only two additional staff and a 28% increase in budget. The additional staff and budget did more than just improve our ability to scale though – we’ve made improvements across the board to provide even more secure and reliable service.

Nothing drives adoption like ease of use, and the foundation for ease of use in the certificate space is our ACME protocol. ACME allows for extensive automation, which means computers can do most of the work. It was also standardized as RFC 8555 in 2019, which allows the Web community to confidently build an even richer ecosystem of software around it. Today, thanks to our incredible community, there is an ACME client for just about every deployment environment. Certbot is one of our favorites, and they’ve been working hard to make it even easier for people to use.

When you combine ease of use with incentives, that’s when adoption really takes off. Since 2017 browsers have started requiring HTTPS for more features, and they’ve greatly improved the ways in which they communicate to their users about the risks of not using HTTPS. When websites put their users at risk by not using HTTPS, major browsers now show stronger warnings. Many sites have responded by deploying HTTPS.

Thanks for taking the time to reflect on this milestone with us. As a community we’ve done incredible things to protect people on the Web. Having issued one billion certificates is affirmation of all the progress we’ve made as a community, and we’re excited to keep working with you to create an even more secure and privacy-respecting Web for everyone.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Multi-Perspective Validation Improves Domain Validation Security

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2020/02/19/multi-perspective-validation.html

At Let’s Encrypt we’re always looking for ways to improve the security and integrity of the Web PKI. We’re proud to launch multi-perspective domain validation today because we believe it’s an important step forward for the domain validation process. To our knowledge we are the first CA to announce multi-perspective validation deployment at scale.

Domain validation is a process that all CAs use to ensure that a certificate applicant actually controls the domain they want a certificate for. Typically the domain validation process involves asking the applicant to place a particular file or token at a controlled location for the domain, such as a particular path or a DNS entry. Then the CA will check that the applicant was able to do so. Historically it looks something like this:

System Architecture Diagram

A potential issue with this process is that if a network attacker can hijack or redirect network traffic along the validation path (for the challenge request, or associated DNS queries), then the attacker can trick a CA into incorrectly issuing a certificate. This is precisely what a research team from Princeton demonstrated can be done with an attack on BGP. Such attacks are rare today, but we are concerned that these attacks will become more numerous in the future.

The Border Gateway Protocol (BGP) and most deployments of it are not secure. While there are ongoing efforts to secure BGP, such as RPKI and BGPsec, it may be a long time until BGP hijacking is a thing of the past. We don’t want to wait until we can depend on BGP being secure, so we’ve worked with the research team from Princeton to devise a way to make such attacks more difficult. Instead of validating from one network perspective, we now validate from multiple perspectives as well as from our own data centers:

System Architecture Diagram

Today we are validating from multiple regions within a single cloud provider. We plan to diversify network perspectives to other cloud providers in the future.

This makes the kind of attack described earlier more difficult because an attacker must successfully compromise three different network paths at the same time (the primary path from our data center, and at least two of the three remote paths). It also increases the likelihood that such an attack will be detected by the Internet topology community.

We’d like to thank the research groups of Prof. Prateek Mittal and Prof. Jennifer Rexford at Princeton University for their partnership in developing this work. We will continue to work with them to refine the effectiveness of our multiple perspective validation design and implementation. We’d also like to thank Open Technology Fund for supporting this work.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

How Let’s Encrypt Runs CT Logs

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2019/11/20/how-le-runs-ct-logs.html

Let’s Encrypt launched a Certificate Transparency (CT) log this past spring. We’re excited to share how we built it in hopes that others can learn from what we did. CT has quickly become an important piece of Internet security infrastructure, but unfortunately it’s not trivial to run a good log. The more the CT community can share about what has been done, the better the ecosystem will be.

Sectigo and Amazon Web Services have generously provided support to cover a significant portion of the cost of running our CT log. “Sectigo is proud to sponsor the Let’s Encrypt CT Log. We believe this initiative will provide much-needed reinforcement of the CT ecosystem,” said Ed Giaquinto, Sectigo’s CIO.

For more background information about CT and how it works, we recommend reading “How Certificate Transparency Works.”

If you have questions about any of what we’ve written here, feel free to ask on our community forums.

Objectives

  1. Scale: Let’s Encrypt issues over 1 million certificates per day, and that number grows each month. We want our log to consume our certificates as well as those from other CAs, so we need to be able to handle as many as 2 million or more certificates per day. To support this ever-increasing number of certificates, CT software and infrastructure need to be architected for scale.
  2. Stability and Compliance: We target 99% uptime, with no outage lasting longer than 24 hours, in compliance with the Chromium and Apple CT policies.
  3. Sharding: Best practice for a CT log is to break it into several temporal shards. For more information on temporal sharding, check out these blog posts.
  4. Low Maintenance: Staff time is expensive, we want to minimize the amount of time spent maintaining infrastructure.

System Architecture

System Architecture Diagram

Staging and Production Logs

We run two equivalent logs, one for staging and one for production. Any changes we plan to make to the production log are first deployed to the staging log. This is critical for making sure that updates and upgrades don’t cause problems before being deployed to production. You can find access details for these logs in our documentation.

We keep the staging log continually under production-level load so that any scale-related problems manifest there first. We also use the staging CT log to submit certificates from our staging CA environment, and make it available for use by other CAs’ staging environments.

As a point of clarification, we consider a log to be comprised of several temporal shards. While each shard is technically a separate log, it makes sense to conceptualize the shards as belonging to a single log.

Amazon Web Services (AWS)

We decided to run our CT logs on AWS for two reasons.

One consideration for us was cloud provider diversity. Since there are relatively few trusted logs in the ecosystem, we don’t want multiple logs to go down due to a single cloud provider outage. At the time we made the decision there were logs running on Google and Digital Ocean infrastructure, as well as self-hosted. We were not aware of any on AWS (in hindsight we may have missed the fact that Digicert had started using AWS for logs). If you’re thinking about setting up a trusted log for CAs to use, please consider cloud provider diversity.

Additionally, AWS provides a solid set of features and our team has experience using it for other purposes. We had little doubt that AWS was up to the task.

Terraform

Let’s Encrypt uses Hashicorp Terraform for a number of cloud-based projects. We were able to bootstrap our CT log infrastructure by reusing our existing Terraform code. There are roughly 50 components in our CT deployments; including EC2, RDS, EKS, IAM, security groups, and routing. Centrally managing this code allows our small team to reproduce a CT infrastructure in any Amazon region of the globe, prevent configuration drift, and easily test infrastructure changes.

Database

We chose to use MariaDB for our CT log database because we have extensive experience using it to run our certificate authority. MariaDB has scaled well on our journey to becoming the largest publicly trusted certificate authority.

We chose to have our MariaDB instances managed by Amazon RDS because RDS provides synchronous writes to standby cluster members. This allows for automatic database failover and ensures database consistency. Synchronous writes to database replicas are essential for a CT log. One missed write during a database failover can mean a certificate was not included as promised, and could lead to the log being disqualified. Having RDS manage this for us reduces complexity and saves staff time. We are still responsible for managing the database performance, tuning, and monitoring.

It’s important to calculate the necessary amount of storage for a CT log database carefully. Too little storage can result in needing to undertake time-consuming and potentially risky storage migrations. Too much storage can result in unnecessarily high costs.

A back of the napkin storage estimation is 1TB per 100 million entries. We expect to need to store 1 billion certificates and precertificates per annual temporal shard, for which we would need 10TB. We considered having separate database storage per annual temporal shard, with approximately 10TB allocated to each, but that was cost prohibitive. We decided to create a 12TB storage block per log (10TB plus some breathing room), which is duplicated for redundancy by RDS. Each year we plan to freeze the previous year’s shard and move it to a less expensive serving infrastructure, reclaiming its storage for our live shards.

We use 2x db.r5.4xlarge instances for RDS for each CT log. Each of these instances contains 8 CPU cores and 128GB of RAM.

Kubernetes

After trying a few different strategies for managing application instances, we decided to use Kubernetes. There is a hefty learning curve for Kubernetes and the decision was not made lightly. This was our first project making use of Kubernetes, and part of the reason we went with it was to gain experience and possibly apply that knowledge to other parts of our infrastructure in the future.

Kubernetes provides abstractions for operators such as deployments, scaling, and service discovery that we would not have to build ourselves. We utilized the example Kubernetes deployment manifests in the Trillian repository to assist with our deployment.

A Kubernetes cluster is comprised of two main components: the control plane which handles the Kubernetes APIs, and worker nodes where containerized applications run. We chose to have Amazon EKS manage our Kubernetes control plane.

We use 4x c5.2xlarge EC2 instances for the worker node pool for each CT log. Each of these instances contains 8 CPU cores and 16GB of RAM.

Application Software

There are three main CT components that we run in a Kubernetes cluster.

The certificate transparency front end, or CTFE, provides RFC 6962 endpoints and translates them to gRPC API requests for the Trillian backend.

Trillian describes itself as a “transparent, highly scalable and cryptographically verifiable data store.” Essentially, Trillian implements a generalized verifiable data store via a Merkle tree that can be used as the back-end for a CT log via the CTFE. Trillian consists of two components; the log signer and log server. The log signer’s function is to periodically process incoming leaf data (certificates in the case of CT) and incorporate them into a Merkle tree. The log server retrieves objects from a Merkle tree in order to fulfill CT API monitoring requests.

Load Balancing

Traffic enters the CT log through an Amazon ELB which is mapped to a Kubernetes Nginx ingress service. The ingress service balances traffic amongst multiple Nginx pods. The Nginx pods proxy traffic to the CTFE service which balance that traffic to CTFE pods.

We employ IP and user agent based rate limiting at this Nginx layer.

Logging and Monitoring

Trillian and the CTFE expose Prometheus metrics which we transform into monitoring dashboards and alerts. It is essential to set a Service Level Objective for the CT log endpoints above the 99% uptime dictated by CT policy to ensure that your log is trusted. A FluentD pod running in a DaemonSet ships logs to centralized storage for further analysis.

We developed a free and open source tool named ct-woodpecker that is used to monitor various aspects of log stability and correctness. This tool is an important part of how we ensure we’re meeting our service level objectives. Each ct-woodpecker instance runs externally from Amazon VPCs containing CT logs.

Future Efficiency Improvements

Here are some ways we may be able to improve the efficiency of our system in the future:

  • Trillian stores a copy of each certificate chain, including many duplicate copies of the same intermediate certificates. Being able to de-duplicate these in Trillian would significantly reduce storage costs. We’re planning to look into whether this is possible and reasonable.
  • See if we can successfully use a cheaper form of storage than IO1 block storage and provisioned IOPs.
  • See if we can reduce the Kubernetes worker EC2 instance size or use fewer EC2 instances.

Support Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization is interested in learning more about sponsorship, please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Onboarding Your Customers with Let’s Encrypt and ACME

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2019/10/09/onboarding-your-customers-with-lets-encrypt-and-acme.html

If you work at a hosting provider or CDN, ACME’s DNS-01 validation
method can make it a lot easier to onboard new customers who have an
existing HTTPS website at another provider. Before your new customer
points their domain name at your servers, you need to have a certificate
already installed for them. Otherwise visitors to the customer’s site
will see an outage for a few minutes while you issue and install a
certificate. To fix this, you and your new customer should use the
DNS-01 validation method to issue a certificate before the customer
switches over DNS for their site.

How the DNS Validation Method Works

The DNS-01 validation method works like
this
: to prove that you control
www.example.com, you create a TXT record at
_acme-challenge.www.example.com with a “digest value” as specified by
ACME (your ACME client should take care of creating this digest value
for you). When the TXT record is ready, your ACME client informs the ACME server (for
instance, Let’s Encrypt) that the domain is ready for validation. The
ACME server looks up the TXT record, compares it to the expected digest
value, and if the result is correct, considers your account authorized
to issue for www.example.com. Your new customer can set up this TXT
record (or a CNAME) without interfering with normal website operations.

The Advantages of a CNAME

There’s an additional trick that I recommend for hosting providers and
CDNs: Instead of giving the digest value to your new customer and
telling them to make a TXT record with it, tell your customer to
configure a CNAME from _acme-challenge.www.example.com to a domain
name that you control and that is unique to the domain being validated.
For instance, you might use www.example.com.validationserver.example.net.
Then, once your
software has verified that this CNAME is set up (accounting for
propagation delay and anycast), your ACME client should
begin the validation process for www.example.com, provisioning a TXT
record at www.example.com.validationserver.example.net. Because the
ACME server’s TXT lookup follows CNAMEs (as do all DNS lookups), it will
see the value you provisioned, and consider your account authorized.

This approach is preferable to handing your customers a raw digest value
for a few reasons. First, it gives your customers all the time they need to set
up the CNAME. If you create a pending authorization up front and give
your customer a digest value to deploy themselves, it has a fixed
lifetime before it expires (for Let’s Encrypt this lifetime is 7 days).
If your customer doesn’t complete the process in that time,
you’ll have to create a new pending authorization and give
your customer a new digest value. That’s annoying and time consuming for
both you and your customer. The CNAME method means even if it
takes your new customer a month to make the needed changes to their DNS,
you can get things up and running as soon as they do.

Another reason to prefer the CNAME method over having new customers
directly provision their TXT records is to support the best practice of
periodically rotating your ACME account key. Because the digest value
used for DNS-01 validation is computed based on your current ACME
account key, it will change whenever you rotate your account key. If you
asked customers to provision their TXT record manually , that means
notifying potential new customers that the value you asked them to put
in DNS isn’t valid anymore, and they need to use a different one. That’s pretty
inconvenient! If you use the CNAME method instead, there’s only one
ACME-related value you’ll ever need to have your new customers put in
DNS, and it won’t change as you change your account key.

Cleaning Up Unused CNAMES

One last note: This is a good way to onboard customers, but you also
need to detect when customers offboard themselves. They may simply
change their A records to point at a different CDN, without telling you
that their plans have changed. You should monitor for this situation and
stop attempting to issue certificates. If the customer has left behind a
CNAMEd _acme-challenge subdomain that points at you, you should
contact that and remind them to delete it. The CNAMEd subdomain
represents a delegated authorization to issue certificates, and cleaning
up that delegation improves both the customer’s security posture and
your own. Similarly, if a customer sets up the CNAME and you issue a
certificate on their behalf, but they never point their A records at
your servers, you should not reissue new certificates indefinitely
without further intervention from the customer.

Introducing Oak, a Free and Open Certificate Transparency Log

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2019/05/15/introducing-oak-ct-log.html

Today we are announcing a new Certificate Transparency log called Oak. The Oak log will be operated by Let’s Encrypt and all publicly trusted certificate authorities will be welcome to submit certificates.

Sectigo generously provided funding to cover a significant portion of our costs to run our CT log. “Sectigo is proud to sponsor the Let’s Encrypt CT Log. We believe this initiative will provide much-needed reinforcement of the CT ecosystem,” said Ed Giaquinto, Sectigo’s CIO. We thank them for their collaboration to improve Internet security.

Certificate Transparency (CT) is a system for logging and monitoring certificate issuance. It greatly enhances everyone’s ability to monitor and study certificate issuance, and these capabilities have led to numerous improvements to the CA ecosystem and Web security. As a result, it is rapidly becoming critical Internet infrastructure. Let’s Encrypt accelerated the adoption of CT by logging every certificate since we started issuing in 2015 – approximately half a billion certificates at this point.

We decided to create and operate a CT log for a few reasons. First, operating a log is consistent with our mission to create a more secure and privacy-respecting Web. We believe transparency increases security and empowers people to make well-informed decisions. Second, operating a log helps us take control of our destiny. Google Chrome requires all new certificates to be submitted to two separate logs, so multiple log options are imperative to our operation. Finally, Let’s Encrypt often issues more than 1M certificates each day, so we wanted to design a CT log that is optimized for high volume. We’ve designed our log to be able to handle submissions from all other publicly trusted Certificate Authorities so they can use Oak to fulfill their logging requirements as well.

Our log uses Google’s Trillian software running on AWS infrastructure. We use Kubernetes for container orchestration and job scheduling and AWS RDS for database management.

We are submitting our log for inclusion in the approved log lists for Google Chrome and Apple Safari. Following 90 days of successful monitoring, we anticipate our log will be added to these trusted lists and that change will propagate to people’s browsers with subsequent browser version releases.

Continuing the forest theme, we are also announcing the launch of our open source CT monitoring tool, CT Woodpecker. We use it to monitor and ensure compliance for our log and we’ve made it open source so others in the CT ecosystem can use it as well.

We’d like to thank Google, Sectigo, Cloudflare, and DigiCert for also running open logs, and we look forward to contributing to better transparency in Web security!

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

Transitioning to ISRG’s Root

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2019/04/15/transitioning-to-isrg-root.html

On July 8, 2019, we will change the default intermediate certificate we provide via ACME. Most subscribers don’t need to do anything. Subscribers who support very old TLS/SSL clients may want to manually configure the older intermediate to increase backwards compatibility.

Since Let’s Encrypt launched, our certificates have been trusted by browsers via a cross-signature from another Certificate Authority (CA) named IdenTrust. A cross-signature from IdenTrust was necessary because our own root was not yet widely trusted. It takes time for a new CA to demonstrate that it is trustworthy, then it takes more time for trusted status to propagate via software updates.

Now that our own root, ISRG Root X1, is widely trusted by browsers we’d like to transition our subscribers to using our root directly, without a cross-sign.

On July 8, 2019, Let’s Encrypt will start serving a certificate chain via the ACME protocol which leads directly to our root, with no cross-signature. Most subscribers don’t need to take any action because their ACME client will handle everything automatically. Subscribers who need to support very old TLS/SSL clients may wish to manually configure their servers to continue using the cross-signature from IdenTrust. You can test whether a given client will work with the newer intermediate by accessing our test site.

Our current cross-signature from IdenTrust expires on March 17, 2021. The IdenTrust root that we are cross-signed from expires on September 30, 2021. Within the next year we will obtain a new cross-signature that is valid until September 29, 2021. This means that our subscribers will have the option to manually configure a certificate chain that uses IdenTrust until September 29, 2021.

We’d like to thank IdenTrust for providing a cross-signature while we worked to get our own root trusted. They have been wonderful partners. IdenTrust believed in our mission to encrypt the entire Web when it seemed like a long-term dream. Together, in less than five years, we have helped to raise the percentage of encrypted page loads on the Web from 39% to 78%.

Let’s Encrypt is currently providing certificates for more than 160 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

The ACME Protocol is an IETF Standard

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2019/03/11/acme-protocol-ietf-standard.html

It has long been a dream of ours for there to be a standardized protocol for certificate issuance and management. That dream has become a reality now that the IETF has standardized the ACME protocol as RFC 8555. I’d like to thank everyone involved in that effort, including Let’s Encrypt staff and other IETF contributors.

Having a standardized protocol for certificate issuance and management is important for two reasons. First, it improves the quality of the software ecosystem because developers can focus on developing great software for a single protocol, instead of having many pieces of less well maintained software for bespoke APIs. Second, a standardized protocol makes switching from one CA to another easier by minimizing technical dependency lock-in.

We consider the standardized version of the ACME protocol to be the second major version of ACME, so we refer to it as ACMEv2. The first version, which we call ACMEv1, is the version of ACME that Let’s Encrypt has used since our launch in 2015. Now that ACMEv2 is standardized, we are announcing an end-of-life plan for our ACMEv1 support.

Let’s Encrypt is currently providing certificates for more than 150 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

Facebook expands support for Let’s Encrypt

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2019/02/12/facebook-expands-support-for-letsencrypt.html

We’re excited that Facebook is supporting our work through a three-year Platinum sponsorship! We asked them to share their thoughts on HTTPS adoption here. Please join us in thanking Facebook for their support of Let’s Encrypt and our mission to encrypt the Web!
– Josh Aas, Executive Director, ISRG / Let’s Encrypt

If the web is more secure, everybody wins. A key technology for making this happen is HTTPS, which enables encrypted connections between people and the websites that they visit. Among its many benefits, HTTPS helps to prevent sensitive data from leaking over the network, and from connections being censored or otherwise maliciously manipulated. The more widely it is deployed, the more secure and private the web becomes for everyone.

We have long worked to protect Facebook users from spammy or malicious content when navigating away from our platform, and last year we extended this protection to upgrading outbound HTTP links to HTTPS where possible. In this way we can help improve people’s security and privacy as they leave our platform. While we take these steps to improve the security and safety of Facebook users, ultimately we hope to see more websites allowing HTTPS connections.

Enabling HTTPS was historically a non-trivial task for any site. It required investment in buying and installing a TLS certificate, which verifies control over the website so that HTTPS can work. The technical difficulty and cost used to serve as barriers to expanding the use of HTTPS across the web. However, things have recently started to change, largely thanks to Let’s Encrypt, a non-profit certificate authority, launched in 2015.

Let’s Encrypt provides free TLS certificates, which are often installed using a tool maintained by the Electronic Frontier Foundation, to massively simplify enabling HTTPS. With that, Let’s Encrypt is effectively upgrading the security and privacy of the web, at no cost to over 150 million websites, including those frequented by Facebook users.

We’re excited to see the continuous increase in HTTPS adoption across the internet. More websites are choosing to enable secure connections which provide the security and privacy benefits and enable a better browsing experience. For example, navigating from Facebook to another site can be faster over encrypted connections than HTTP, and an increasing number of browser features will only work when sites use HTTPS.

We have sponsored Let’s Encrypt from the start, and are proud to share that we are increasing that support as a platinum sponsor. We believe that Let’s Encrypt has played a significant and important role in bringing encryption into the mainstream and raising the number of secure sites across the internet.

As we automatically crawl web content on Facebook (for example, to generate link previews), about 38% of HTTPS domains we observe use Let’s Encrypt, making it the top certificate authority. Over 19% of outbound clicks from Facebook to HTTPS-enabled websites go to sites that use certificates from Let’s Encrypt. Overall, more than 72% of outbound clicks from Facebook are now destined for HTTPS-enabled websites, including the links that we upgrade to HTTPS in real time.

We’re proud to continue to collaborate with Let’s Encrypt on helping to improve web security. To any website owners who haven’t yet enabled encryption, we strongly encourage you to use Let’s Encrypt to protect your users and allow HTTPS connections.

Looking Forward to 2019

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2018/12/31/looking-forward-to-2019.html

Let’s Encrypt had a great year in 2018. We’re now serving more than 150 million websites while maintaining a stellar security and compliance track record.

Most importantly though, the Web went from 67% encrypted page loads to 77% in 2018, according to statistics from Mozilla. This is an incredible rate of change!

We’d like to thank all of the people and organizations who worked hard to create a more secure and privacy-respecting Web.

This year we created a new website for the legal entity behind Let’s Encrypt, Internet Security Research Group (ISRG), because we believe there will be other instances beyond Let’s Encrypt in which ISRG might be able to help to build, or improve access to, a better Internet.

While we’re proud of what we accomplished in 2018, we spend most of our time looking forward rather than back. As we wrap up our own planning process for 2019, I’d like to share some of our plans with you, including both the things we’re excited about and the challenges we’ll face. We’ll cover service growth, new features, infrastructure, and finances.

Service Growth

Let’s Encrypt helps to drive HTTPS adoption by offering a free, easy to use, and globally available option for obtaining the certificates required to enable HTTPS. HTTPS adoption on the Web took off at an unprecedented rate from the day Let’s Encrypt launched to the public.

The number of certificates and unique domains we support continues to grow rapidly:


We expect strong growth again in 2019, likely up to 120M active certificates and 215M fully qualified domains. You can view our recently revamped stats page for more information.

One of the reasons Let’s Encrypt is so easy to use is that our community has done great work making client software that works well for a wide variety of platforms. We’d like to thank everyone involved in the development of more than 85 client software options for Let’s Encrypt. Support for our protocol, ACME, is built in to Apache and we’re hoping 2019 will be the year that it comes to Nginx.

Other organizations and communities are also doing great work to promote HTTPS adoption, and thus stimulate demand for our services. For example, browsers are starting to make their users more aware of the risks associated with unencrypted HTTP (e.g. Firefox, Chrome). Many hosting providers and CDNs are making it easier than ever for all of their customers to use HTTPS. Government agencies are waking up to the need for stronger security to protect constituents. The media community is working to Secure the News.

New Features

In 2018 we introduced several new features, including ACMEv2 support and wildcard certificates. We’ve got some exciting features planned for 2019.

The feature we’re most excited about is multi-perspective validation. Currently, when a subscriber requests a certificate, we validate domain control from a single network perspective. This is standard practice for CAs. If an attacker along the network path for the validation check can interfere with traffic they can potentially cause certificates to be issued that should not be issued. We’re most concerned about this happening via BGP hijacking, and since BGP is not going to be secured any time soon, we needed to find another mitigation. The solution we intend to deploy in 2019 is multi-perspective validation, in which we will check from multiple network perspectives (distinct Autonomous Systems). This means that potential BGP hijackers would need to hijack multiple routes at the same time in order to pull off a successful attack, which is significantly more difficult than hijacking a single route. We are working with a talented research team at Princeton to design the most effective multi-perspective validation system we can, and have already turned parts of this feature on in our staging environment.

We are also planning to introduce a Certificate Transparency (CT) log in 2019. All certificate authorities like Let’s Encrypt are required to submit certificates to CT logs but there are not enough stable logs in the ecosystem. As such, we are moving forward with plans to run a log which all CAs will be able to submit to.

We had planned to add ECDSA root and intermediate certificates in 2018 but other priorities ultimately took precedence. We hope to do this in 2019. ECDSA is generally considered to be the future of digital signature algorithms on the Web due to the fact that it is more efficient than RSA. Let’s Encrypt will currently sign ECDSA keys from subscribers, but we sign with the RSA key from one of our intermediate certificates. Once we have an ECDSA root and intermediates, our subscribers will be able to deploy certificate chains which are entirely ECDSA.

Infrastructure

Our CA infrastructure is capable of issuing millions of certificates per day with redundancy for stability and a wide variety of security safeguards, both physical and logical. Our infrastructure also generates and signs around 40 million OCSP responses daily, and serves those responses approximately 5.5 billion times per day. We expect these numbers to grow approximately 40% in 2019.

Our physical CA infrastructure currently occupies approximately 55 units of rack space, split between two datacenters, consisting primarily of compute servers, storage, HSMs, switches, and firewalls. When we issue more certificates it puts the most stress on storage for our databases. We regularly invest in more and faster storage for our database servers, and that will continue in 2019.

All of our infrastructure is managed by our Site Reliability Engineering (SRE) team, which is comprised of six people. SRE staff are responsible for building and maintaining all physical and logical CA infrastructure. These staff are largely responsible for our high standards for security and compliance. The team also manages a 24/7/365 on-call schedule and they are primary participants in both security and compliance audits.

Finances

We pride ourselves on being an efficient organization. In 2019 Let’s Encrypt will secure a massive portion of the Web with a budget of only $3.6M. We believe this represents an incredible value and that contributing to Let’s Encrypt is one of the most effective ways to help create a more secure and privacy-respecting Web.

Our 2019 fundraising efforts are off to a strong start with Platinum sponsorships from Cisco, OVH, Mozilla, Google Chrome, Electronic Frontier Foundation, and Internet Society, as well as many other Gold and Silver sponsors. The Ford Foundation has renewed their grant to Let’s Encrypt as well. We are seeking additional sponsorship and grant assistance to meet our full needs for 2019.

Support Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

We’re grateful for the industry and community support that we receive, and we look forward to continuing to create a more secure and privacy-respecting Web!

Let’s Encrypt Root Trusted By All Major Root Programs

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org/2018/08/06/trusted-by-all-major-root-programs.html

As of the end of July 2018, the Let’s Encrypt root, ISRG Root X1, is directly trusted by Microsoft products. Our root is now trusted by all major root programs, including Microsoft, Google, Apple, Mozilla, Oracle, and Blackberry.

Today’s announcement that we’re trusted by all major root programs represents a major milestone for us, but it’s not the conclusion of our journey towards being directly trusted everywhere.

Certificates from Let’s Encrypt have been widely trusted since our first issuance because of a cross-signature from another CA called IdenTrust. Browsers and operating systems have not, by default, directly trusted Let’s Encrypt certificates, but they trust IdenTrust, and IdenTrust trusts us, so we are trusted indirectly. IdenTrust is a critical partner in our effort to secure the Web, as they have allowed us to provide widely trusted certificates from day one.

While Let’s Encrypt is now directly trusted by almost all newer versions of operating systems, browsers, and devices, there are still many older versions in the world that do not directly trust Let’s Encrypt. Some of those older systems will eventually be updated to trust Let’s Encrypt directly. Some will not, and we’ll need to wait for the vast majority of those to cycle out of the Web ecosystem. We expect this will take at least five more years, so we plan to use a cross signature until then.

As a subscriber of Let’s Encrypt, today’s milestone does not require any action on your part. Just continue to use best practices, including making sure that your ACME client (e.g. Certbot or an alternative) is regularly receiving software updates.

Let’s Encrypt is currently providing certificates for more than 115 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

Looking Forward to 2018

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org//2017/12/07/looking-forward-to-2018.html

Let’s Encrypt had a great year in 2017. We more than doubled the number of active (unexpired) certificates we service to 46 million, we just about tripled the number of unique domains we service to 61 million, and we did it all while maintaining a stellar security and compliance track record. Most importantly though, the Web went from 46% encrypted page loads to 67% according to statistics from Mozilla – a gain of 21% in a single year – incredible. We’re proud to have contributed to that, and we’d like to thank all of the other people and organizations who also worked hard to create a more secure and privacy-respecting Web.

While we’re proud of what we accomplished in 2017, we are spending most of the final quarter of the year looking forward rather than back. As we wrap up our own planning process for 2018, I’d like to share some of our plans with you, including both the things we’re excited about and the challenges we’ll face. We’ll cover service growth, new features, infrastructure, and finances.

Service Growth

We are planning to double the number of active certificates and unique domains we service in 2018, to 90 million and 120 million, respectively. This anticipated growth is due to continuing high expectations for HTTPS growth in general in 2018.

Let’s Encrypt helps to drive HTTPS adoption by offering a free, easy to use, and globally available option for obtaining the certificates required to enable HTTPS. HTTPS adoption on the Web took off at an unprecedented rate from the day Let’s Encrypt launched to the public.

One of the reasons Let’s Encrypt is so easy to use is that our community has done great work making client software that works well for a wide variety of platforms. We’d like to thank everyone involved in the development of over 60 client software options for Let’s Encrypt. We’re particularly excited that support for the ACME protocol and Let’s Encrypt is being added to the Apache httpd server.

Other organizations and communities are also doing great work to promote HTTPS adoption, and thus stimulate demand for our services. For example, browsers are starting to make their users more aware of the risks associated with unencrypted HTTP (e.g. Firefox, Chrome). Many hosting providers and CDNs are making it easier than ever for all of their customers to use HTTPS. Government agencies are waking up to the need for stronger security to protect constituents. The media community is working to Secure the News.

New Features

We’ve got some exciting features planned for 2018.

First, we’re planning to introduce an ACME v2 protocol API endpoint and support for wildcard certificates along with it. Wildcard certificates will be free and available globally just like our other certificates. We are planning to have a public test API endpoint up by January 4, and we’ve set a date for the full launch: Tuesday, February 27.

Later in 2018 we plan to introduce ECDSA root and intermediate certificates. ECDSA is generally considered to be the future of digital signature algorithms on the Web due to the fact that it is more efficient than RSA. Let’s Encrypt will currently sign ECDSA keys from subscribers, but we sign with the RSA key from one of our intermediate certificates. Once we have an ECDSA root and intermediates, our subscribers will be able to deploy certificate chains which are entirely ECDSA.

Infrastructure

Our CA infrastructure is capable of issuing millions of certificates per day with multiple redundancy for stability and a wide variety of security safeguards, both physical and logical. Our infrastructure also generates and signs nearly 20 million OCSP responses daily, and serves those responses nearly 2 billion times per day. We expect issuance and OCSP numbers to double in 2018.

Our physical CA infrastructure currently occupies approximately 70 units of rack space, split between two datacenters, consisting primarily of compute servers, storage, HSMs, switches, and firewalls.

When we issue more certificates it puts the most stress on storage for our databases. We regularly invest in more and faster storage for our database servers, and that will continue in 2018.

We’ll need to add a few additional compute servers in 2018, and we’ll also start aging out hardware in 2018 for the first time since we launched. We’ll age out about ten 2u compute servers and replace them with new 1u servers, which will save space and be more energy efficient while providing better reliability and performance.

We’ll also add another infrastructure operations staff member, bringing that team to a total of six people. This is necessary in order to make sure we can keep up with demand while maintaining a high standard for security and compliance. Infrastructure operations staff are systems administrators responsible for building and maintaining all physical and logical CA infrastructure. The team also manages a 24/7/365 on-call schedule and they are primary participants in both security and compliance audits.

Finances

We pride ourselves on being an efficient organization. In 2018 Let’s Encrypt will secure a large portion of the Web with a budget of only $3.0M. For an overall increase in our budget of only 13%, we will be able to issue and service twice as many certificates as we did in 2017. We believe this represents an incredible value and that contributing to Let’s Encrypt is one of the most effective ways to help create a more secure and privacy-respecting Web.

Our 2018 fundraising efforts are off to a strong start with Platinum sponsorships from Mozilla, Akamai, OVH, Cisco, Google Chrome and the Electronic Frontier Foundation. The Ford Foundation has renewed their grant to Let’s Encrypt as well. We are seeking additional sponsorship and grant assistance to meet our full needs for 2018.

We had originally budgeted $2.91M for 2017 but we’ll likely come in under budget for the year at around $2.65M. The difference between our 2017 expenses of $2.65M and the 2018 budget of $3.0M consists primarily of the additional infrastructure operations costs previously mentioned.

Support Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at [email protected]. We ask that you make an individual contribution if it is within your means.

We’re grateful for the industry and community support that we receive, and we look forward to continuing to create a more secure and privacy-respecting Web!

ACME Support in Apache HTTP Server Project

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org//2017/10/17/acme-support-in-apache-httpd.html

We’re excited that support for getting and managing TLS certificates via the ACME protocol is coming to the Apache HTTP Server Project (httpd). ACME is the protocol used by Let’s Encrypt, and hopefully other Certificate Authorities in the future. We anticipate this feature will significantly aid the adoption of HTTPS for new and existing websites.

We created Let’s Encrypt in order to make getting and managing TLS certificates as simple as possible. For Let’s Encrypt subscribers, this usually means obtaining an ACME client and executing some simple commands. Ultimately though, we’d like for most Let’s Encrypt subscribers to have ACME clients built in to their server software so that obtaining an additional piece of software is not necessary. The less work people have to do to deploy HTTPS the better!

ACME support being built in to one of the world’s most popular Web servers, Apache httpd, is great because it means that deploying HTTPS will be even easier for millions of websites. It’s a huge step towards delivering the ideal certificate issuance and management experience to as many people as possible.

The Apache httpd ACME module is called mod_md. It’s currently in the development version of httpd and a plan is being formulated to backport it to an httpd 2.4.x stable release. The mod_md code is also available on GitHub.

It’s also worth mentioning that the development version of Apache httpd now includes support for an SSLPolicy directive. Properly configuring TLS has traditionally involved making a large number of complex choices. With the SSLPolicy directive, admins simply select a modern, intermediate, or old TLS configuration, and sensible choices will be made for them.

Development of mod_md and the SSLPolicy directive has been funded by Mozilla and carried out primarily by Stefan Eissing of greenbytes. Thank you Mozilla and Stefan!

Let’s Encrypt is currently providing certificates for more than 55 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

Wildcard Certificates Coming January 2018

Post Syndicated from Let's Encrypt - Free SSL/TLS Certificates original https://letsencrypt.org//2017/07/06/wildcard-certificates-coming-jan-2018.html

Let’s Encrypt will begin issuing wildcard certificates in January of 2018. Wildcard certificates are a commonly requested feature and we understand that there are some use cases where they make HTTPS deployment easier. Our hope is that offering wildcards will help to accelerate the Web’s progress towards 100% HTTPS.

Let’s Encrypt is currently securing 47 million domains via our fully automated DV certificate issuance and management API. This has contributed heavily to the Web going from 40% to 58% encrypted page loads since Let’s Encrypt’s service became available in December 2015. If you’re excited about wildcard availability and our mission to get to a 100% encrypted Web, we ask that you contribute to our summer fundraising campaign.

A wildcard certificate can secure any number of subdomains of a base domain (e.g. *.example.com). This allows administrators to use a single certificate and key pair for a domain and all of its subdomains, which can make HTTPS deployment significantly easier.

Wildcard certificates will be offered free of charge via our upcoming ACME v2 API endpoint. We will initially only support base domain validation via DNS for wildcard certificates, but may explore additional validation options over time. We encourage people to ask any questions they might have about wildcard certificate support on our community forums.

We decided to announce this exciting development during our summer fundraising campaign because we are a nonprofit that exists thanks to the generous support of the community that uses our services. If you’d like to support a more secure and privacy-respecting Web, donate today!

We’d like to thank our community and our sponsors for making everything we’ve done possible. If your company or organization is able to sponsor Let’s Encrypt please email us at [email protected].