Following Russia’s unjustified and tragic invasion of Ukraine in late February, the world has watched closely as Russian troops attempted to advance across Ukraine, only to be resisted and repelled by the Ukrainian people. Similarly, we’ve seen a significant amount of cyber attack activity in the region. We continue to work to protect an increasing number of Ukrainian government, media, financial, and nonprofit websites, and we protected the Ukrainian top level domain (.ua) to help keep Ukraine’s presence on the Internet operational.
At the same time, we’ve closely watched significant and unprecedented activity on the Internet in Russia. The Russian government has taken steps to tighten its control over both the technical components and the content of the Russian Internet. For their part, the people in Russia are doing something very different. They have been adopting tools to maintain access to the global Internet, and they have been seeking out non-Russian media sources. This blog post outlines what we’ve observed.
The Russian Government asserts control over the Internet
Over the last five years, the Russian government has taken steps to tighten its control of a sovereign Internet within Russia’s borders, including laws requiring Russian ISPs to install equipment allowing the government to monitor and block Internet activity, and requiring the establishment of an exclusively Russian DNS (outside ICANN). And it created mechanisms for the Russian government to control how Russia was connected to the global Internet, so they could pull the plug if they wanted.
Since the Russian invasion of Ukraine, the Russian government has made a series of announcements related to implementation of its sovereign Internet laws. Russian government agencies were instructed to switch to Russian DNS servers, move public resources to Russian hosting services, and take a number of other steps designed to reduce reliance on non-Russian providers. Although some took these initiatives as an announcement that Russia intended to disconnect from the global Internet, so far Russia does not appear to have leveraged the tools it has to disconnect itself entirely from the global Internet. We continue to see connections processing successfully in Russia through non-Russia infrastructure.
In the meantime, authorities in Russia have implemented a series of targeted blocking actions against websites and operators that they find objectionable. Initially, officials targeted popular social media sites like Facebook, Instagram, Twitter, and YouTube, as well as Russian language outlets based outside of the country.
We can see the effect of some of those blocks on traffic from Russian users to different news websites in Russia and Ukraine before and after blocks were implemented.
In each case, these news sites saw exponential growth in their traffic in the days around the February 24th invasion of Ukraine. But that increase was met within a matter of days by actions to block traffic to those sites. The blocks had varying degrees of success over the first few weeks, though each of them seem to have been eventually successful in denying access to those sources of news through traditional Internet channels.
But that is only half the story. As the Russian government took steps to control traditional channels for Internet access, there were shifts in the ways many Russians used the Internet.
Russian citizens turning to tools to gain access to the open Internet
Russians have been adopting applications and tools that allow them to engage with the Internet privately and avoid some of the mechanisms that the Russian government is using to control and monitor access to the Internet. Whereas the most popular applications in the Apple App Store in most of the world in March continue to relate to social media and games, the leaderboard in Russia looked very different:
All of the top apps in Russia in March were for private and secure Internet access or encrypted messaging apps, including the most downloaded app – Cloudflare’s own WARP / 1.1.1.1 (a privacy-based recursive DNS resolver). This list of popular apps is a stunning contrast with every other country in the world.
Because of the significant and important popularity of WARP (1.1.1.1), we’ve had some detailed insight into exactly how this has played out. If we look back to the beginning of February we see that Cloudflare’s WARP tool was little used in Russia. Its use took off from the first weekend of the war, and peaked two weeks ago. Later, after this virtual migration to such secure tools became apparent, we saw attempts to block access to the tools used to access the Internet securely.
While levels have receded from their peak, a large number of Russians continue to use Cloudflare WARP in Russia at massively higher levels than pre-war.
In addition to the ways Russians are using the Internet increasingly relying on private and encrypted communications, we’ve also seen a shift in what they are trying to access. Here’s a chart of DNS requests from Russian users for a well known US newspaper. Recent DNS traffic for the site has quintupled compared to pre-war levels, indicating Russians are trying to access that news source.
And here’s DNS traffic for a large French news source. Again, DNS lookups have grown enormously as Russians try to access it.
And here’s a British newspaper.
The picture is clear from these three charts. Russians want access to non-Russian news sources and based on the popularity of private Internet access tools and VPNs, they are willing to work to get it.
A front line against cyberattack
In addition to the services we’ve been able to provide average citizens in Russia, our servers at the edge of the Internet in-country have also permitted us to detect and block attacks originating there. When attacks are mitigated inside Russia, they never travel outside Russian borders. That’s always been part of the proposition of Cloudflare’s distributed network – to identify and block cyber attacks (especially DDoS attacks) locally and before they can ever get off the ground.
Here’s what DDoS activity originating inside Russia and blocked there by Cloudflare has looked like since the beginning of February. Normal DDoS activity originating from Russian networks and blocked by Cloudflare’s servers there is relatively low throughout February but then grows massively in the middle of March.
To be clear, being able to identify where cyber attack traffic originates is not the same as being able to attribute where the attacker is located. Attributing cyber attacks is difficult, and now is a time to be particularly careful with attribution. It is relatively common for cyber attackers to launch attacks from remote locations around the world. This often happens when they are able to hijack devices in other countries through things like IoT (Internet of Things) corruptions.
But even with such subterfuge, we’ve still seen a significant increase in the number of blocked attacks that are hitting our servers inside Russia.
A few weeks ago, as the invasion of Ukraine was in its early stages, I noted that “Russia needs more Internet, not less.” At a time of unprecedented economic sanctions by the United States and Europe, there have been calls for all foreign companies to go further and exit Russia completely, including calls for Internet providers to disconnect Russia. To be clear, Cloudflare has minimal sales and commercial activity in Russia – we’ve never had a corporate entity, an office, or employees there – and we’ve taken steps to ensure that we’re not paying taxes or fees to the Russian government. But given the significant impact of our services on the availability and security of the Internet, we believe removing our services from Russia altogether would do more harm than good.
While we deeply appreciate the motivation of the calls for companies to exit Russia, this withdrawal by Internet companies can have the unintended effect of advancing and entrenching the interests of the Russian government to control the Internet in Russia. Efforts to have Russia cut off from the global Internet through ICANN and RIPE will only cut off the Russian people from information about the war in Ukraine that the Russian government doesn’t want them to access. After a number of U.S.-based certificate authorities stopped issuing SSL certificates for Russian websites, Russia responded in early March by encouraging Russian citizens to download a Russian Root Certificate Authority instead. As observed by EFF, “the Russian state’s stopgap measure to keep its services running also enables spying on Russians, now and in the future.”
This is why there has been near universal agreement by experts that it is imperative the Russian Internet stay as open as possible for the Russian people. Dozens of civil society groups have urged governments to work to counteract authoritarian actions “and ensure that sanctions and other steps meant to repudiate the Russian government’s illegal actions do not backfire, by reinforcing Putin’s efforts to assert information control.” Russian digital rights activists have pleaded with service providers to offer Russians free VPN access so they are not left isolated from global news sources. Even the U.S. State Department has made clear, “It is critical to maintain the flow of information to the people of Russia to the fullest extent possible.”
Supporting our mission to help build a better Internet, it’s been a busy six weeks for our team monitoring these developments and working around the clock to make sure Ukrainian web properties are defended and that ordinary Russians can access the global Internet. We remain in awe of the brave Ukrainians standing up in defense of their homeland, and continue to hope that peace will prevail.
Today, in partnership with CrowdStrike and Ping Identity, Cloudflare is launching the Critical Infrastructure Defense Project (CriticalInfrastructureDefense.org). The Project was born out of conversations with cybersecurity and government experts concerned about potential retaliation to the sanctions that resulted from the Russian invasion of Ukraine.
In particular, there is a fear that critical United States infrastructure will be targeted with cyber attacks. While these attacks may target any industry, the experts we consulted with were particularly concerned about three areas that were often underprepared and could cause significant disruption: hospitals, energy, and water.
To help address that need, Cloudflare, CrowdStrike, and Ping Identity have committed under the Critical Infrastructure Defense Project to offer a broad suite of our products for free for at least the next four months to any United States-based hospital, or energy or water utility. You can learn more at: www.CriticalInfrastructureDefense.org.
We are not powerless against hackers. Organizations that have adopted a Zero Trust approach to security have been successful at mitigating even determined attacks. There are three core components to any Zero Trust security approach: 1) Network Security, 2) Endpoint Security; and 3) Identity.
Cloudflare, CrowdStrike, and Ping Identity are three of the leading Zero Trust security companies securing each of these components. Cloudflare’s Zero Trust network security offers a broad set of services that organizations can easily implement to ensure their connections are protected no matter where users access the network. CrowdStrike provides a broad set of end point security services to ensure that laptops, phones, and servers are not compromised. And Ping Identity provides identity solutions, including multi-factor authentication, that are foundational to any organization’s posture.
Each of us is great at what we do on our own. Together, we provide an integrated solution that is unrivaled and proven to stand up to even the most sophisticated nation state cyber attacks.
And this is what we think is required, because the current threat is significantly higher than what we have seen since any of our companies was founded. We all built our companies relying on the nation’s infrastructure, and we believe it is incumbent on us to provide our technology in order to protect that infrastructure when it is threatened. For this period of heightened risk, we are all providing our services at no cost to organizations in these most vulnerable sectors.
We’ve also worked together to ensure our products function in harmony and are easy to implement. We don’t want short-staffed IT teams, long requisition processes, or limited budgets to stand in the way of getting the protection that’s needed in place immediately. We’ve taken a cue from hospitals to triage the risks through a recommended list showing organizations that may be short of IT staff how they can proceed: suggesting what they should prioritize over the next day, over the next week, and over the next month.
You can download the recommended security triage program here. We know that not every organization will be able to implement every recommendation. But every step you get through on the list will help your organization be incrementally better prepared for whatever is to come.
Our teams are also committed to working directly with organizations in these sectors to make onboarding as quick and painless as possible. We will onboard customers under this project with the same level of service as if they were our largest paying customers. We believe it is our duty to help ensure that the nation’s critical infrastructure remains online and available through this challenging time.
We anticipate that, based on what we learn over the days ahead, the Critical Infrastructure Defense Project may expand to additional sectors and countries. We hope the predictions of retaliatory cyberattacks don’t come true. But, if they do, we know our solutions can mitigate the risk, and we stand ready to fully deploy them to protect our most critical infrastructure.
At Cloudflare, we’ve watched in horror the Russian invasion of Ukraine. As the possibility of war looked more likely, we began to carefully monitor the situation on the ground, with the goal of keeping our employees, our customers, and our network safe.
Helping protect Ukraine against cyberattacks
Attacks against the Internet in Ukraine began even before the start of the invasion. Those attacks—and the steady stream of DDoS attacks we’ve seen in the days since—prompted us to extend our services to Ukrainian government and telecom organizations at no cost in order to ensure they can continue to operate and deliver critical information to their citizens as well as to the rest of the world about what is happening to them.
Going beyond that, under Project Galileo, we are expediting onboarding of any Ukrainian entities for our full suite of protections. We are currently assisting more than sixty organizations in Ukraine and the region—with about 25% of those organizations coming aboard during the current crisis. Many of the new organizations are groups coming together to assist refugees, share vital information, or members of the Ukrainian diaspora in nearby countries looking to organize and help. Any Ukrainian organizations that are facing attack can apply for free protection under Project Galileo by visiting www.cloudflare.com/galileo, and we will expedite their review and approval.
Securing our customers’ data during the conflict
In order to preserve the integrity of customer data, we moved customer encryption key material out of our data centers in Ukraine, Russia, and Belarus. Our services continued to operate in the regions using our Keyless SSL technology, which allows encryption sessions to be terminated in a secure data center away from where there may be a risk of compromise.
If any of our facilities or servers in Ukraine, Belarus, or Russia lose power or connectivity to the Internet, we have configured them to brick themselves. All data on disk is encrypted with keys that are not stored on site. Bricked machines will not be able to be booted unless a secure, machine-specific key that is not stored on site is entered.
Monitoring Internet availability in Ukraine
Our team continues to monitor Internet patterns across Ukraine. While usage across the country has declined over the last 10 days, we are thankful that in most locations the Internet is still accessible.
We are taking steps to ensure that, as long as there is connectivity out of the country, our services will continue to operate.
Staying ahead of the threat globally
Cyber threats to Ukrainian customers and telecoms is only part of the broader story of potential cyberattacks. Governments around the world have emphasized that organizations must be prepared to respond to disruptive cyber activity. The US Cybersecurity and Infrastructure Security Agency (CISA), for example, has recommended that all organizations—large and small—go “Shields Up” to protect themselves from attack. The UK’s National Cyber Security Centre has encouraged organizations to improve their cyber resilience.
This is where careful monitoring of the attacks in Ukraine is so important. It doesn’t just help our customers in Ukraine — it helps us learn and improve our products so that we can protect all of our customers globally. When wiper malware was identified in Ukraine, for example, we adapted our Zero Trust products to make sure our customers were protected.
We’ve long believed that everyone should have access to cybersecurity tools to protect themselves, regardless of their size or resources. But during this time of heightened threat, access to cybersecurity services is particularly critical. We have a number of free services available to protect you online — and we encourage you to take advantage of them.
Providing services in Russia
Since the invasion, providing any services in Russia is understandably fraught. Governments have been united in imposing a stream of new sanctions and there have even been some calls to disconnect Russia from the global Internet. As discussed by ICANN, the Internet Society, the Electronic Frontier Foundation, and Techdirt, among others, the consequences of such a shutdown would be profound.
The scope of new sanctions issued in the last few weeks have been unprecedented in their reach, frequency, and the number of different governments involved. Governments have issued sweeping new sanctions designed to impose severe costs against those who supported the invasion of Ukraine, including government entities and officials in Russia and Belarus. Sanctions have been imposed against Russia’s top financial institutions, including Russia’s two largest banks, fundamentally altering the ability of Russians to access capital. The entire break away territories of Donetsk and Luhansk, including all of the residents of those regions, are subject to comprehensive sanctions. We’ve seen sanctions on state-owned enterprises, elite Russian families, and the leaders of intelligence-directed disinformation outlets.
These sanctions are intended to make sure that those who supported the invasion are held to account. And Cloudflare has taken action to comply. Over the past several years, Cloudflare has developed a robust and comprehensive sanctions compliance program that allows us to track and take immediate steps to comply with new sanctions regulations as they are implemented. In addition to an internal compliance team and outside counsel, we employ third party tools to flag potential matches or partial ownership by sanctioned parties, and we review reports from third-parties about potential connections. We have also worked with government experts inside and outside of the United States to identify when there is a connection between a sanctioned entity and a Cloudflare account.
Over the past week, our team has ensured that we are complying with these new sanctions as they are announced. We have closed off paid access to our network and systems in the new comprehensively-sanctioned regions. And we have terminated any customers we have identified as tied to sanctions, including those related to Russian financial institutions, Russian influence campaigns, and the Russian-affiliated Donetsk and Luhansk governments. We expect additional sanctions are likely to come from governments as they determine additional steps are appropriate, and we will continue to move quickly to comply with those requirements as they are announced.
Beyond this, we have received several calls to terminate all of Cloudflare’s services inside Russia. We have carefully considered these requests and discussed them with government and civil society experts. Our conclusion, in consultation with those experts, is that Russia needs more Internet access, not less.
As the conflict has continued, we’ve seen a dramatic increase in requests from Russian networks to worldwide media, reflecting a desire by ordinary Russian citizens to see world news beyond that provided within Russia.
We’ve also seen an increase in Russian blocking and throttling efforts, combined with Russian efforts to control the content of the media operating inside Russia with a new “fake news” law.
The Russian government itself, over the last several years, has threatened repeatedly to block certain Cloudflare services and customers. Indiscriminately terminating service would do little to harm the Russian government, but would both limit access to information outside the country, and make significantly more vulnerable those who have used us to shield themselves as they have criticized the government.
In fact, we believe the Russian government would celebrate us shutting down Cloudflare’s services in Russia. We absolutely appreciate the spirit of many Ukrainians making requests across the tech sector for companies to terminate services in Russia. However, when what Cloudflare is fundamentally providing is a more open, private, and secure Internet, we believe that shutting down Cloudflare’s services entirely in Russia would be a mistake.
Our thoughts are with the people of Ukraine and the entire team at Cloudflare prays for a peaceful resolution as soon as possible.
Today we’re excited to announce that Cloudflare has acquired Zaraz. The Zaraz value proposition aligns with Cloudflare’s mission. They aim to make the web more secure, more reliable, and faster. And they built their solution on Cloudflare Workers. In other words, it was a no-brainer that we invite them to join our team.
Be Careful Who Takes Out the Trash
To understand Zaraz’s value proposition, you need to understand one of the biggest risks to most websites that people aren’t paying enough attention to. And, to understand that, let me use an analogy.
Imagine you run a business. Imagine that business is, I don’t know, a pharmacy. You have employees. They have a process and way they do things. They’re under contract, and you conduct background checks before you hire them. They do their jobs well and you trust them. One day, however, you realize that no one is emptying the trash. So you ask your team to find someone to empty the trash regularly.
Your team is busy and no one has the time to add this to their regular duties. But one plucky employee has an idea. He goes out on the street and hails down a relative stranger. “Hey,” your employee says to the stranger. “I’ve seen you walking by this way every day. Would you mind stopping in and taking out the trash when you do?”
“Uh”, the stranger says. “Sure?!”
“Great,” your employee says. “Here’s a badge that will let you into the building. The trash is behind the secure area of the pharmacy, but, don’t worry, just use the badge, and you can get back there. You look trustworthy. This will work out great!!”
And for a while it does. The stranger swings by every day. Takes out the trash. Behaves exactly as hoped. And no one thinks much about the trash again.
But one day you walk in, and the pharmacy has been robbed. Drugs stolen, patient records missing. Logs indicate that it was the stranger’s badge that had been used to access the pharmacy. You track down the stranger, and he says “Hey, that sucks, but it wasn’t me”. I handed off that trash responsibility to someone else long ago when I stopped walking past the pharmacy every day.”
And you never track down the person who used the privileged access to violate your trust.
The Keys to the Kingdom
Now, of course, this is crazy. No one would go pick a random stranger off the street and give them access to their physical store. And yet, in the virtual world, a version of this happens all the time.
Every day, front end developers, marketers, and even security teams embed third-party scripts directly on their web pages. These scripts perform basic tasks — the metaphorical equivalent of taking out the trash. When performing correctly, they can be valuable at bringing advanced functionality to sites, helping track marketing conversions, providing analytics, or stopping fraud. But, if they ever go bad, they can cause significant problems and even steal data.
At the most mundane, poorly configured scripts can slow down the rendering pages. While there are ways to make scripts non-blocking, the unfortunate reality is that their developers don’t always follow the best practices. Often when we see slow websites, the biggest cause of slowness is all the third-party scripts that have been embedded.
But it can be worse. Much worse. At Cloudflare, we’ve seen this first hand. Back in 2019 a hacker compromised a third-party service that Cloudflare used and modified the third-party JavaScript that was loaded into a page on cloudflare.com. Their aim was to steal login cookies, usernames and passwords. They went so far as to automatically create username and password fields that would autocomplete.
Luckily, this attack caused minimal damage because it was caught very quickly by the team, but it highlights the very real danger of third-party JavaScript. Why should code designed to count clicks even be allowed to create a password field?
Put simply, third-party JavaScript is a security nightmare for the web. What looks like a simple one-line change (“just add this JavaScript to get free page view tracking!”) opens a door to malicious code that you simply don’t control.
And worse is that third-party JavaScript can and does load other JavaScript from other unknown parties. Even if you trust the company whose code you’ve chosen to embed, you probably don’t trust (or even know about!) what they choose to include.
And even worse these scripts can change any time. Security threats can come and go. The attacker who went after Cloudflare compromised the third-party and modified their service to only attack Cloudflare and included anti-debugging features to try to stop developers spotting the hack. If you’re a CIO and this doesn’t freak you out already, ask your web development team how many third-party scripts are on your websites. Do you trust them all?
The practice of adding third-party scripts to handle simple tasks is the literal equivalent of pulling a random stranger off the street, giving them physical access to your office, and asking them to stop by once a day to empty the trash. It’s completely crazy in the physical world, and yet it’s common practice in web development.
Sandboxing the Strangers
At Cloudflare, our solution was draconian. We ordered that all third-party scripts be stripped from our websites. Different teams at Cloudflare were concerned. Especially our marketing team, who used these scripts to assess whether the campaigns they were running were successful. But we made the decision that it was more important to protect the integrity of our service than to have visibility into things like marketing campaigns.
It was around this time that we met the team behind Zaraz. They argued there didn’t need to be such a drastic choice. What if, instead, you could strictly control what the scripts that you insert on your page did. Make sure if ever they were compromised they wouldn’t have access to anything they weren’t authorized to see. Ensure that if they failed or were slow they wouldn’t keep a page from rendering.
We’ve spent the last half year testing Zaraz, and it’s magical. It gives you the best of the flexible, extensible web while ensuring that CIOs and CISOs can sleep well at night knowing that even if a third-party script provider is compromised, it won’t result in a security incident.
To put a fine point on it, had Cloudflare been running Zaraz then the threat from the compromised script we saw in 2019 would have been completely and automatically eliminated. There’s no way for the attacker to create those username and password fields, no access to cookies that are stored in the user’s browser. The attack surface would have been completely removed.
We’ve published two other posts today outlining how Zaraz works as well as examples of how companies are using it to ensure their web presence is secure, reliable, and fast. We are making Zaraz available to our Enterprise customers immediately, and all other customers can access a free beta version on their dashboard starting today.
If you’re a third-party script developer, be on notice that if you’re not properly securing your scripts, then as Zaraz rolls out across more of the web your scripts will stop working. Today, Cloudflare sits in front of nearly 20% of all websites and, before long, we expect Zaraz’s technology will help protect all of them. We want to make sure all scripts running on our customers’ sites meet modern security, reliability, and performance standards. If you need help getting there, please reach out, and we’ll be standing ready to help: [email protected].
In the meantime, we encourage you to read about how the Zaraz technology works and how customers like Instacart are using it to build a better web presence.
It’s terrific to have Zaraz on board, furthering Cloudflare’s mission to help build a better Internet. Welcome to the team. And in that vein: we’d like to welcome you to Zaraz! We’re excited for you to get your hands on this piece of technology that makes the web better.
This week we celebrate Cloudflare’s birthday. We launched the company 11 years ago tomorrow: September 27, 2010. It has been our tradition, since our first birthday, to use this week to launch innovative new products that we think of as our gift back to the Internet.
Since going public, it’s also been an opportunity for us to update our Annual Founders’ Letter and share what’s on our mind. Recently we’ve been thinking about three things: team, the Internet, and innovation.
Team
When anyone asks us the key to Cloudflare’s success, we always say the same thing: the team we’ve been able to attract to help us achieve our mission of helping build a better Internet. In the last year we’ve had more than 250,000 people apply to work for us and extended offers to less than one half of one percent of them. We continue to attract great people.
It’s incredible to realize that more than half of Cloudflare’s team today started since March 13, 2020, when we closed all our physical offices due to the pandemic. In the last several months, as we’ve started to see a light at the end of the COVID tunnel, we’ve been hosting what we called Summer Socials with our team. Getting together outside, often over a picnic lunch, it’s been fun to meet face-to-face people we’d only video conferenced with before. And even more fun to watch people from across the team get to know each other outside the confines of a Brady Bunch-like on-screen box.
As a company that was very much a work-from-office culture before the pandemic, we were terrified of what would happen to our culture when we switched to fully remote work. Eighteen months into this forced experiment on a new way of working we’re happy to report: it’s working. Really well.
It turns out what we all suspected is in fact true. Culture has little to do with fun offices, plentiful snacks, or adjustable desks. Instead, for us, it starts with hiring people who are relentlessly curious and, at the same time, empathetic. Curious people want to learn. Empathetic people love to teach. And if you put a group of them together, whether in a swanky office or on Zoom, great things will happen.
As we come out the other side of COVID, we have an opportunity to help build a better way to work. It would be naive to insist that we go back to the way we did things before. We’ve been more productive, and on average our team has been happier in their jobs, than any time in the company’s history. At the same time, we know there can be considerable value in coming together in person to solve hard problems, brainstorm about the future, and build relationships that make the company stronger.
We don’t have all the answers on what the future of work looks like, but we’ve begun to formulate a place to start our experiments as people come back. We hope we can use the times we get together as ways to better collaborate and learn. But, at the same time, give our team the flexibility to work how and wherever they are the most productive.
The Internet
Cloudflare’s mission is to help build a better Internet. We always capitalize the I in Internet, in spite of what the AP style guide has said since 2016, because it’s a proper noun, we believe there is and only should be one, and we have an enduring respect for what a miracle it is that it exists.
Right around the same time that the AP started to say that you needn’t capitalize the I in Internet anymore, something seemed to change. The world shifted from seeing the Internet and what it enabled as an irreproachable good to a source of great danger.
We’ve watched the same thing. Since 2016 it’s often felt like a connection to the Internet only brings cyberattacks, toxic social media, threats to democracy, increasing polarization, and a declining disdainful discourse.
We have real challenges ahead as some of the technologies that ride on top of the Internet have broken down traditional gatekeepers without sufficient concern for addressing the harms they previously protected against. But, at the same time, the Internet itself remains a miracle.
A mere 11 years before Cloudflare’s founding, long distance phone calls still cost a fortune, sharing a photograph with someone in another country took weeks, and the idea that you could access the sum total of human knowledge from a device in your pocket was beyond even the fantasies of science fiction.
The last 18 months of the pandemic have reaffirmed our faith in the miracle that is the Internet. Imagine just how much worse it would have been had the pandemic happened just 11 years ago, let alone 22. The Internet allowed many of us to continue to work, connect with our loved ones, exercise our creativity, and stay connected to the world.
We’re proud of what we’ve done to live up to our mission and help build a better Internet during this time. And, as we come out the other side, we will continue to engage with policy makers to address the new harms an interconnected world has brought while preserving the miracle that is the Internet itself.
Innovation
The Internet may seem static, but it is not. 11 years ago, watching a video online was an exercise in frustration. Today, it seems almost automatic that you can push play on your TV and access nearly any movie ever made instantly. That’s possible because the Internet isn’t static; it gets better through innovation.
At Cloudflare, we’re optimized to catalyze exactly that innovation. It starts with our mission: to help build a better Internet. The word “help” is important, because we know we can’t do it alone. So, wherever we can, we work with others across the Internet ecosystem to push it forward and make it better.
Sometimes people outside the company are surprised by the products we build. In fact, predicting our roadmap is pretty easy. We look at all the steps that are required to load a web page, send an email, stream a video, login to a workstation, or anything else you do online and ask: can we make that more secure, more reliable, or faster?
What’s exciting is that the pace at which the Internet is getting better is accelerating. And, in turn, the pace at which we are able to launch innovative new products is accelerating along with it. As the Internet grows and acquires more capabilities, we believe we will continue to grow with it. An investment in Cloudflare is, fundamentally, we feel an investment in the Internet itself.
And so, this week, we have an incredible series of announcements that are designed to help build a better Internet. We’re entering a new area to close one of the last network security risks that we haven’t historically protected our customers from, driving down costs of core cloud services, pushing the boundary of our network to our customers’ doorsteps, and investing in new technologies that may someday disrupt the web as we know it today.
Thank you to our team, our customers, and our investors. Happy 11th birthday to Cloudflare. And, even as we pick up steam, we continue to believe: we’re just getting started.
This is the story of how we decided to work with Google to build Signed Exchanges support at Cloudflare. But, more generally, it’s also a story of how Cloudflare thinks about building disruptive new products and how we’ve built an organization designed around continuous innovation and long-term thinking.
A Threat to the Open Web?
The story starts with me pretty freaked out. In May 2015, Facebook had announced a new format for the web called Instant Articles. The format allowed publishers to package up their pages and serve them directly from Facebook’s infrastructure. This was a threat to Google, so the company responded in October with Accelerated Mobile Pages (AMP). The idea was generally the same as Facebook’s but using Google’s infrastructure.
As a general Internet user, if these initiatives were successful they were pretty scary. The end game was that the entirety of the web would effectively be slurped into Facebook and Google’s infrastructure.
But as the cofounder and CEO of Cloudflare, this presented an even more immediate risk. If everyone moved their infrastructure to Facebook and Google, there wasn’t much left for us to do. Our mission is to help build a better Internet, but we’ve always assumed there would be an Internet. If Facebook and Google were successful, there was real risk there would just be Facebook and Google.
That said, the rationale behind these initiatives was compelling. While they ended with giving Facebook and Google much more control, they started by trying to solve a real problem. The web was designed with the assumption that the devices connecting to it would be on a fixed, wired connection. As more of the web moved to being accessed over wireless, battery-powered, relatively low-power devices, many of the assumptions of the web were holding back its performance.
This is particularly true in the developing world. While a failed connection can happen anywhere, the further you get from where content is hosted, the more likely it is to happen. Facebook and Google both reasoned that if they could package up the web and serve complete copies of pages from their infrastructure, which spanned the developing world, they could significantly increase the usability of the web in areas where there was still an opportunity for Internet usage to grow. Again, this is a laudable goal. But, if successful, the results would have been dreadful for the Internet as we know it.
Seeds of Disruption
So that’s why I was freaked out. In our management meetings at Cloudflare I’d walk through how this was a risk to the Internet and our business, and we needed to come up with a strategy to address it. Everyone on our team listened and agreed but ultimately and reasonably said: that’s in the future, and we have immediate priorities of things our customers need, so we’ll need to wait until next quarter to prioritize it.
That’s all correct, and probably the right decision if you are forced to make one, but it’s also how companies end up getting disrupted. So, in 2016, we decided to fund a small team led by Dane Knecht, Cloudflare’s founding product manager, to set up a sort of skunkworks team in Austin, TX. The idea was to give the team space away from headquarters, so it could work on strategic projects with a long payoff time horizon.
Today, Dane’s team is known as the Emerging Technologies & Incubation (ETI) team. It was where products like Cloudflare for Teams, 1.1.1.1, and Workers were first dreamed up and prototyped. And it remains critical to how Cloudflare continues to be so innovative. Austin, since 2016, has also grown from a small skunkworks outpost to what will, before the end of this year, be our largest office. That office now houses members from every Cloudflare team, not just ETI. But, in some ways, it all started with trying to figure out how we should respond to Instant Articles and AMP.
We met with both Facebook and Google. Facebook’s view of the world was entirely centered around their app, and didn’t leave much room for partners. Google, on the other hand, was born out of the open web and still ultimately wanted to foster it. While there has been a lot of criticism of AMP, much of which we discussed with them directly, it’s important to acknowledge that it started from a noble goal: to make the web faster and easier to use for those with limited Internet resources.
We built a number of products to extend the AMP ecosystem and make it more open. Viewed on their own, those products have not been successes. But they catalyzed a number of other innovations. For instance, building a third party AMP cache on Cloudflare required a more programmable network. That directly resulted in us prototyping a number of different serverless computing strategies and finally settling on Workers. In fact, many of the AMP products we built were the first products built using Workers.
Part of the magic of our ETI team is that they are constantly trying new things. They’re set up differently, in order to take lots of “shots on goal.” Some won’t work, in which case we want them to fail fast. And, even for those that don’t, we are always learning, collaborating, and innovating. That’s how you create a culture of innovation that produces products at the rate we do at Cloudflare.
Signed Exchanges: Helping Build a Better Internet
Importantly also, working with the AMP team at Google helped us better collaborate on ideas around Internet performance. Cloudflare’s mission is to “help build a better Internet.” It’s not to “build a better Internet.” The word “help” is essential and something I’ll always correct if I hear someone leave it out. The Internet is inherently a collection of networks, and also a collection of work from a number of people and organizations. Innovation doesn’t happen in a vacuum but is catalyzed by collaboration and open standards. Working with other great companies who are aligned with democratizing performance optimization technology and speeding up the Internet is how we believe we can make significant and meaningful leaps in terms of performance.
And that’s what Signed Exchanges have the opportunity to be. They take the best parts of AMP — in terms of allowing pages to be preloaded to render almost instantly — but give back control over the content to the individual publishers. They don’t require you to exclusively use Google’s infrastructure and are extensible well beyond just traffic originating from search results. And they make the web incredibly fast and more accessible even in those areas where Internet access is slow or expensive.
We’re proud of the part we played in bringing this new technology to the Internet. We’re excited to see how people use it to build faster services available more broadly. And the ETI team is back at work looking over the innovation horizon and continuously asking the question: what’s next?
During Impact Week, we’ve shared how Cloudflare is providing tools for our customers to minimize their environmental impact as well as what we, as a company, are doing to helpsocietyatlarge. But some critical stakeholders we haven’t talked much about yet are Cloudflare’s more than 2,000 employees: who build our services, support and educate our customers, keep our finances in order, work through difficult policy issues, and empower us to accomplish everything we have.
Over the last year and a half, we’ve all challenged a lot of the assumptions about what it means to “work.” Prior to the start of the pandemic, Cloudflare was very much a work-from-office culture. And so when, on March 13, 2020, we closed all our offices and asked everyone to work from home, the two of us were extremely nervous.
And then something unexpected happened: a lot of things got better.
As a company, productivity increased — when measured by our success selling our products, our pace of shipping new products, and even things like the time it takes for our finance team to close our books.
Other day-to-day things got better, too. We noticed a marked increase in participation in meetings by women, team members from whom English wasn’t their first language, junior team members, and other traditionally underrepresented groups. It turns out, putting everyone in a Brady-bunch like box on a screen smooths out some of the other social cues that, when in-person, make some people less comfortable, willing, or able to fully participate.
Virtually More Inclusive
It’s not unreasonable to speculate that the increase in productivity was driven, in no small part, by the increase in overall participation by people who previously felt reluctant to do so. And this further aligned with job surveys that we conducted over the last year and a half which showed that while the things people wanted us to improve remained the same, overall satisfaction with jobs increased.
We also noticed that the diversity of the candidates that were applying to work for us increased as we allowed people to work remotely. We were now an option for people who did not live in, or could not move to, the cities we had offices in. At Cloudflare, we’ve always believed in having a diverse team. Not to look good in a government report, but because it’s the right business strategy: more diverse teams win.
We all have different perspectives formed by our experiences that inherently give us insights and blind spots. If everyone on a team has the same insights and blind spots then there will be less unique and creative solutions proposed to whatever problems we face. Just as it’s important to have genetic diversity in a species, having diversity on every dimension in hiring makes us a stronger, more creative company. Prioritizing a diverse team is the right strategy if you’re optimizing for innovation, like we are at Cloudflare.
But not everything got better when we switched to remote; some things definitely got worse. We’re social creatures. We thrive through human interaction that is still difficult to replicate virtually. Even with improvements in video conferencing, online interactions still mute some of the social cues and make misunderstanding more likely. The osmosis for our team of learning by watching others is harder, especially for team members early in their career. And, unfortunately, for some the office is a refuge from difficult situations at home and so not having it as a place to get away can amplify those challenges.
What We’ve Learned… So Far
So we’ve been thinking a lot about what the future of work looks like at Cloudflare and wanted to share publicly what we’ve been talking about for some time internally. Here are some things we think we know.
First, we don’t know what the long term future of work will be like and so we’ve been hesitant to lay down broad proclamations. Instead, we expect that as we get past the pandemic and are able to work in-person safely again, we will do what Cloudflare has always done: run a number of experiments ourselves, watch what our peers are doing, and figure out what works for us. The one thing we feel pretty sure of is that wherever we start the experiment is highly unlikely to be exactly the place where we end up. The future of work won’t be set in stone sometime in the coming months, but evolve over the coming years.
Second, no matter what, the future of work will be more flexible. There’s no way we are putting the genie of remote work back in the bottle. Why would we want to if we’ve learned that we’ve been more productive and more satisfied with their jobs while we’ve been remote? Flexibility is the number one requested work benefit, and one of the silver linings of the pandemic for us has been that we ran a forced experiment that proved we could make it work.
Third, we are incredibly reluctant to impose arbitrary rules. Requiring team members to come in every Monday, Tuesday, and Thursday begs the question: why those days? Saying you need to come in if you’re below a certain seniority level also seems weirdly arbitrary. Instead of rules, we’re much more likely to start with general standards outlining what success as a member of the team at Cloudflare looks like and giving guidelines. We may need rules at some point, but we want to develop those rules over time based on what we learn.
Fourth, just opening offices and hoping for the best doesn’t work. What we’ve seen ourselves, and confirmed with others, is that what makes working from an office great is getting to work side-by-side with your colleagues. But if Alyssa comes in on Monday, and Blake comes in on Tuesday, and Carlos comes in on Wednesday, and Deeksha comes in on Thursday, and Ellen comes in on Friday, and they all hoped that they would get to connect, then none of them has a good experience and none of them come in the following week. If in-person work is going to work, there needs to be some deliberate structure and planning.
Fifth, we believe more in carrots than sticks. We’d rather we create an environment where people want to come in than where they have to come in. Based on our internal surveys, about 10% of our team wants to come in every day. We want to make the environment such that 100% of our team wants to come in at least some days.
Sixth, a more flexible way of working will require a more flexible physical space. The base “lego brick” we used to design all our offices pre-pandemic was the 6-person conference room. And, while none of our offices started this way, they all evolved into a sea of white, adjustable desks in neat rows as we found spots for our growing team. That already feels anachronistic. We think we need to redesign spaces to accommodate teams coming together to collaborate as well as individuals looking for a quiet spot for heads-down work.
Seventh, mixed meetings suck. When some people are in-person and some people are virtual the experience is bad for everyone. Part of why we think the last year and a half has worked is because everyone is in the same boat. We believe part of the reason why hybrid work environments have traditionally not worked is because they, left to their own devices, will tend to devolve to an experience that’s bad for everyone. The future of flexible work needs to acknowledge that most hybrid work experiments in the past haven’t worked.
Eighth, we’re a very global company. We have team members in countries around the world and need to operate our business around the clock. One of the benefits of being fully remote over the last year and a half is that it made all our offices feel like they were on equal footing. That’s something we believe is important for us to maintain.
Navigating Through the Fog
So what’s our plan? Again, we don’t pretend to have all the answers. Instead, we expect that we’ll start somewhere and experiment. So we’re starting by being more flexible about where we hire people. We still believe that people will tend to cluster in hubs around cities where we have physical offices, but we are now open to hiring for nearly all of our roles in any location where we have a legal entity setup that allows us to hire.
We are tearing apart our offices in San Francisco and London to remake them into flexible work spaces. We’re designing them to allow for teams of 10, 20, or 30 employees to get together and collaborate. We’re also creating “Zoom villages” with one-person spaces and high quality AV equipment to let people jump on conference calls.
One of the few rules that we plan on starting with is that in meetings if any person is remote then everyone in the meeting is remote. We know that will create some awkward situations where some of our team will literally be sitting next to each other at desks talking on a video conference call. But we believe this is a rule worth having, in spite of our hesitation to impose strict rules, to help keep the playing field level for all our colleagues, wherever they’re working.
We’re going to rethink the purpose of the offices as spaces where teams can come together to collaborate. Internally, we’re calling these “on-site off-sites” — though everyone agrees we need a better name. The idea being that teams can call an in-person meeting and reserve space in any of our offices to come together. We expect different teams will set different cadences of these meetings, but expect most people to have at least some time in an office at least once a quarter.
We’re planning for what we’ve termed a “Czar of Serendipity” who will coordinate cross-group lunches and other activities to help facilitate teams who may not work directly together to have the opportunity if they want to meet colleagues they may not otherwise know. They’ll also help arrange in-person speakers and other activities aligned with whatever teams or groups are physically in the office each week.
And we’re hunting for carrots to encourage our team, and especially members who are earlier in their career, to come in. One we’re working on is what we’re calling Orange Card. We hope to turn every team member’s ID into a charge card. The card will only activate after someone badges in for the day and will only work to purchase food at restaurants that are within a 10-minute walk from the office with pre-tax dollars.
It’s in Cloudflare’s interest to encourage people to come in physically to work. Across the industry, however, we think jobs that require in-person work will look increasingly anachronistic. We also believe that, rather than operating private cafeterias inside our own spaces, it’s important for us to support local businesses near our offices — especially as so many of them were hit hard during COVID. If with Orange Card we can do this and find a way to let employees pay for lunches when they’re in the office at an effective discount, then it will check both boxes: giving employees a reason to come in and also supporting the local community.
We don’t know how many of these things will work, but it’s a sense of the experiments we intend to run as we try and find the future of work that works for our team.
In many ways we were fortunate that Cloudflare’s product could be of specific help during an incredibly difficult time for the world. The superheros of the last year and a half have been the medical professionals and scientists who have taken care of the sick and looked for cures for this disease. But the Internet has been the faithful sidekick that has helped many continue to work, stay connected with loved ones, and keep ourselves entertained through this trying time. As one of the defenders of the Internet, our work at Cloudflare has been incredibly rewarding. We hope we can create a future of work that remains incredibly rewarding even long past the pandemic.
The thoughts above are just a starting place. We expect that we’re going to learn a lot not only from our own experiments, but also from what we learn works (and doesn’t work) at peer companies. We would have never tried this experiment in remote work but for the pandemic. Now, having realized that we can continue to execute in a more flexible work environment, we don’t plan to forget the lessons we learned. We’re hopeful that we, along with our peer companies, will continue to run experiments and, over time, develop a new future of work that is more flexible, more inclusive, and more productive.
When we started Cloudflare, we weren’t thinking about minimizing the environmental impact of the Internet. Frankly, I didn’t really think of the Internet as having much of an environmental impact. It was just this magical resource that gave access to information and services from anywhere.
But that was before I started racking servers in hyper-cooled data centers. Before Cloudflare started paying the bills to keep those servers powered up and cooled down. Before we became obsessed with maximizing the number of requests we could process per watt of power. And long before we started buying directly from renewable power suppliers to drive down the cost of electricity across our network.
Today, I have a very good understanding of how much power it takes to run the Internet. It therefore wasn’t surprising to read the Boston Consulting Group study which found that 2% of all carbon output, about 1 billion metric tons per year, is attributable to the Internet. That’s the equivalent of the entire aviation industry.
Cloudflare: Accidentally Environmentally Friendly By Design
While we didn’t set out to reduce the environmental impact of the Internet, Cloudflare has always had efficiency at its core. It comes from our ongoing fight with an old nemesis: the speed of light.
Because we knew we couldn’t beat the speed of light, in order to make our network fast we needed to get close to where Internet users were. In order to do that, we needed to partner directly with ISPs around the world so they’d allow us to install our gear directly inside their networks. In order to do that, we needed to make our gear as low power as possible. And we needed to invent network technology to spread load around our network to deal with spikes of traffic — whether because of a cyber attack or a sale on an exclusive new sneaker line — and to efficiently use all available capacity.
Fighting for Efficiency
When back in December 2012, just two years after we launched, I traveled to Intel’s Oregon Research Center to talk to their senior engineering team about how we needed server chips with more cores per watt, I wasn’t thinking we needed it to save the environment. Instead, I was trying to figure out how we could build equipment that was power efficient enough that ISPs wouldn’t object to installing it. Unfortunately, Intel told me that I was worrying about the wrong thing. So that’s when we started looking for alternatives, including the very power-efficient Arm.
But, it turns out, our obsession with efficiency has made Cloudflare the environmental choice in cloud computing. A 2015 study by Anders S. G. Andrae and Tomas Edler estimated the average cost of processing a byte of information online. Even accounting for the efficiency gains across the industry, based on the study’s data our best estimates are that Cloudflare data processing is more than 19 times more efficient.
Serve Local
The imperfect analogy that I like is buying from the local farmers’ market versus the big box retailer. By serving requests locally, and not backhauling them around the world to massive data centers, Cloudflare is able to reduce the environmental impact of our customers on the Internet. In 2020, we estimate that our customers reduced their carbon output by 550,000 metric tons versus if they had not used our services. That’s the equivalent of eliminating 635 million miles driven by passenger cars last year.
We’re proud of that, but it’s still a tiny percentage of the overall impact the Internet still has on the environment. As we thought about Impact Week, we set out to make reducing the environmental impact of the Internet a top priority. Given today more than 1 in 6 websites uses Cloudflare, we’re in a position where changes we make can have a meaningful impact.
We Can Do More
Starting today, we’re announcing four major initiatives to reduce Cloudflare’s environmental impact and help the Internet as a whole be more environmentally friendly.
First, we’re committing to be carbon neutral by 2022. We already extensively use renewable energy to power our global network, but we’re going to expand that usage to cover 100% of our energy use. But we’re going a step further. We’re going to look back over the 11 years since Cloudflare launched and purchase offsets to zero out all of Cloudflare’s historical carbon output from powering our global network. It’s not enough that we have less impact than others, we want to make sure Cloudflare since our beginning has been a net positive for the planet.
Second, we are ramping up our deployment of a new class of hyper-efficient servers. Based on Arm technology, these servers can perform the same amount of work while using half the energy. We are hopeful that by prioritizing energy efficiency in the server market we can help catalyze more chip manufacturers to release more efficient designs.
Third, we’re releasing a new option for Cloudflare Workers and Pages, our computing platform and JAMStack offering, which allows developers to choose to run their workloads in the most energy efficient data centers. We believe we are the first major cloud computing vendor to offer developers a way to optimize for the environment. The Green Workers option won’t cost anymore. The tradeoff will be that workloads may incur a bit of additional network latency, but we believe for many developers that’s a tradeoff they’ll be willing to make.
New Standards and Partnerships to Eliminate Excessive Emissions
Finally, and maybe most ambitiously, we’re working with a number of the leading search and crawl companies to introduce an open standard to minimize the amount of load from excessive crawl as possible. Nearly half of all Internet traffic is automated. The majority of that is malicious, and Cloudflare is designed to stop that as efficiently as possible.
But more than 5% of all Internet traffic is generated by legitimate crawlers which index the web in order to power services we all rely on like search. The problem is, more than half of that legitimate crawl traffic is redundant — reindexing pages that haven’t changed. If we can eliminate redundant crawl, it’d be the equivalent of planting a new 30 million acres of forest. That’s a goal worth striving for.
When we started Cloudflare we weren’t thinking about how we could reduce the Internet’s environmental impact. But that’s changed. Cloudflare’s mission is to help build a better Internet. And a better Internet is clearly a more environmentally friendly Internet.
If I’m completely honest, Cloudflare didn’t start out as a mission-driven company. When Lee, Michelle, and I first started thinking about starting a company in 2009 we saw an opportunity as the world was shifting from on-premise hardware and software to services in the cloud. It seemed inevitable to us that the same shift would come to security, performance, and reliability services. And, getting ahead of that trend, we could build a great business.
Matthew Prince, Michelle Zatlyn, and Lee Holloway, Cloudflare’s cofounders, in 2009.
One problem we had was that we knew in order to have a great business we needed to win large organizations with big IT budgets as customers. And, in order to do that, we needed to have the data to build a service that would keep them safe. But we only could get data on security threats once we had customers. So we had a chicken and egg problem.
Our solution was to provide a basic version of Cloudflare’s services for free. We reasoned that individual developers and small businesses would sign up for the free service. We’d learn a lot about security threats and performance and reliability opportunities based on their traffic data. And, from that, we would build a service we could sell to large businesses.
And, generally, Cloudflare’s business model made sense. We found that, for the most part, small companies got a low volume of cyber attacks, and so we could charge them a relatively small amount. Large businesses faced more attacks, so we could charge them more.
But what surprised us, and we only discovered because we were providing a free version of our service, was that there was a certain set of small organizations with very limited resources that received very large attacks. Servicing them was what made Cloudflare the mission-driven company we are today.
The Committee to Protect Journalists
If you ever want to be depressed, sign up for the newsletter of the Committee to Protect Journalists (CPJ). They’re the organization that, when a journalist is kidnapped or killed anywhere in the world, negotiates their release or, far too often, recovers their body.
I’d met the director of the organization at an event in early 2012. Not long after, he called me and asked if I wanted to meet three Cloudflare customers who were in town. I didn’t, I have to confess, but Michelle pushed me to take the meeting.
On a rainy San Francisco afternoon the director of CPJ brought three African journalists to our office. All three of them hugged me. One was from Ethiopia, another was from Angola, and the third they wouldn’t tell us his name or where he was from because he was “currently being hunted by death squads.”
For the next 90 minutes, I listened to stories of how the journalists were covering corruption in their home countries, how their work put them constantly in harm’s way, how powerful forces worked to silence them, how cyberattacks had been a constant struggle, and how, today, they depended on Cloudflare’s free service to keep their work online. That last bit hit me like a ton of bricks.
After our meeting finished, and we saw the journalists out, with Cloudflare T-shirts and other swag in hand, I turned to Michelle and said, “Whoa. What have we gotten ourselves into?”
Becoming Mission Driven
I’ve thought about that meeting often since. It was the moment I realized that Cloudflare had a mission beyond just being a good business. The Internet was a critically important resource for those three journalists and many others like them. At the same time, forces that sought to limit their work would use cyberattacks to shut them down. While we hadn’t set out to ensure everyone had world-class cybersecurity, regardless of their ability to pay, now it seemed critically important.
With that realization, Cloudflare’s mission came naturally: we aim to help build a better Internet. One where you don’t need to be a giant company to be fast and reliable. And where even a journalist, working on their own against daunting odds, can be secure online.
This is why we’ve prioritized projects that give back to the Internet. We launched Project Galileo, which provides our enterprise-grade services to organizations performing politically or artistically important work. We launched the Athenian Project to help protect elections against cyber attacks. We launched Project Fair Shot to make sure the organizations distributing the COVID-19 vaccine had the technical resources they needed to do so equitably.
And, even on the technical side, we work hard to make the Internet better even when there’s no clear economic benefit to us, or even when it’s against our economic benefit. We don’t monetize user data because it seems clear to us that a better Internet is a more private Internet. We enabled encryption for everyone even though, when we did it, it was the biggest differentiator between our free and paid plans and the number one reason people upgraded. But clearly a better Internet was an encrypted Internet, and it seemed silly that someone should have to pay extra for a little bit of math.
Our First Impact Week
This week we kick off Cloudflare’s first Impact Week. We originally conceived the idea of the week as a way to highlight some of the things we were doing as a company around our environmental, social, and governance (ESG) initiatives. But, as is the nature of innovation weeks at Cloudflare, as soon as we announced it internally our team started proposing new products and features to take some of our existing initiatives even further.
So, over the course of the week, in addition to talking about how we’ve built our network to consume less power we’ll also be demonstrating how we’re increasingly using hyper power-efficient Arm-based servers to achieve even higher levels of efficiency in order to lessen the environmental impact of running the Internet. We’ll launch a new Workers option for developers who want to be more environmentally conscious. And we’ll announce an initiative in partnership with other leading Internet companies that we hope, if broadly adopted, could cut down as much as 25% of global web traffic and the corresponding energy wasted to serve it.
We’ll also focus on how we can bring the Internet to more people. While broadband has been a revolution where it’s available, rural and underserved-urban communities around the world still suffer from slow Internet speeds and limited ISP choice. We can’t completely solve that problem (yet) but we’ll be announcing an initiative that will help with some critical aspects.
Finally, as Cloudflare becomes a larger part of the Internet, we’ll be announcing programs both to monitor the network’s health, affirm our commitments to human rights, and extend our protections of critical societal functions like protecting elections.
When I first was trying to convince Michelle that we should start a business together, I pitched her a bunch of ideas. Most of them involved finding a clever way to extract rents from some group or another, often for not much benefit to society at large. Sitting in an Ethiopian restaurant in Central Square, I remember so clearly her saying to me, “Matthew, those are all great business ideas. But they’re not for me. I want to do something where I can be proud of the work we’re doing and the positive impact we’ve made.”
That sentence made me go back to the drawing board. The next business idea I pitched to her turned out to be Cloudflare. Today, Cloudflare’s mission remains helping build a better Internet. And, as we kick off Impact Week, we are proud to continue to live that mission in everything we do.
When web hosting services first emerged in the mid-1990s, you paid for everything on a separate meter: bandwidth, storage, CPU, and memory. Over time, customers grew to hate the nickel-and-dime nature of these fees. The market evolved to a fixed-fee model. Then came Amazon Web Services.
AWS was a huge step forward in terms of flexibility and scalability, but a massive step backward in terms of pricing. Nowhere is that more apparent than with their data transfer (bandwidth) pricing. If you look at the (ironically named) AWS Simple Monthly Calculator you can calculate the price they charge for bandwidth for their typical customer. The price varies by region, which shouldn’t surprise you because the cost of transit is dramatically different in different parts of the world.
Charging for Stocks, Paying for Flows
AWS charges customers based on the amount of data delivered — 1 terabyte (TB) per month, for example. To visualize that, imagine data is water. AWS fills a bucket full of water and then charges you based on how much water is in the bucket. This is known as charging based on “stocks.”
On the other hand, AWS pays for bandwidth based on the capacity of their network. The base unit of wholesale bandwidth is priced as one Megabit per second per month (1 Mbps). Typically, a provider like AWS, will pay for bandwidth on a monthly fee based on the number of Mbps that their network uses at its peak capacity. So, extending the analogy, AWS doesn’t pay for the amount of water that ends up in their customers’ buckets, but rather the capacity based on the diameter of the “hose” that is used to fill them. This is known as paying for “flows.”
Translating Flows to Stocks
You can translate between flow and stock pricing by knowing that a 1 Mbps connection (think of it as the "hose") can transfer 0.3285 TB (328GB) if utilized to its fullest capacity over the course of a month (think of it as running the "hose" at full capacity to fill the "bucket" for a month).1 AWS obviously has more than 1 Mbps of capacity — they can certainly transfer more than 0.3285 TB per month — but you can use this as the base unit of their bandwidth costs, and compare it against what they charge a customer to deliver 1 Terabyte (1TB), in order to figure out the AWS bandwidth markup.
One more subtlety to be as accurate as possible. Wholesale bandwidth is also billed at the 95th percentile. That effectively cuts off the peak hour or so of use every day. That means a 1 Mbps connection running at 100% can actually likely transfer closer to 0.3458 TB (346GB) per month.
Two more factors are important: utilization and regional costs. AWS can’t run all their connections at 100% utilization 24×7 for a month. Instead, they’ll have some average utilization per transit connection in any month. It’s reasonable to estimate that they likely run at between 20% and 40% average utilization. That would be a typical average utilization range for the industry. The higher their utilization, the more efficient they are, the lower their costs, and the higher their effective customer markup will be.
To be conservative, we’ve assumed that AWS’s average utilization is the bottom of that range (20%), but you can download the raw data and adjust the assumptions however you think makes sense.
We have a good sense of the wholesale prices of bandwidth in different regions around the world based on what Cloudflare sees in the market when we buy bandwidth ourselves. We’d imagine AWS gets at least as good of pricing as we do. We’ve included a rough estimate of these prices in the calculation, rounding up on the wholesale price wherever there was a question (which makes AWS look better).
Massive Markups
Based on these assumptions, here’s our best estimate of AWS’s effective markup for egress bandwidth on a per-region basis.
Don’t rest easy, South Korea with your merely 357% markup. The general rule of thumb appears to be that the older a market is, the more Amazon wrings from its customers in egregious egress markups — and the Seoul availability zone is only a bit over four years old. Winter, unfortunately, inevitably seems to come to AWS customers.
AWS Stands Alone In Not Passing On Savings to Customers
Remember, this is for the transit bandwidth that AWS is paying for. For the bandwidth that they exchange with a network like Cloudflare, where they are directly connected (settlement-free peered) over a private network interface (PNI), there are no meaningful incremental costs and their effective margins are nearly infinite. Add in the effect of rebates Amazon collects from colocation providers who charge cross connect fees to customers, and the effective markup is likely even higher.
Some other cloud providers take into account that their costs are lower when passing over peering connections. Both Microsoft Azure and Google Cloud will substantially discount egress charges for their mutual Cloudflare customers. Members of the Bandwidth Alliance — Alibaba, Automattic, Backblaze, Cherry Servers, Dataspace, DNS Networks, DreamHost, HEFICED, Kingsoft Cloud, Liquid Web, Scalway, Tencent, Vapor, Vultr, Wasabi, and Zenlayer — waive bandwidth charges for mutual Cloudflare customers.
At this point, the majority of hosting providers in the industry either substantially discount or entirely waive egress fees when sending traffic from their network to a peer like Cloudflare. AWS is the notable exception in the industry. It’s worth noting that we invited AWS to be a part of the Bandwidth Alliance, and they politely declined.
It seems like a no-brainer that if we’re not paying for the bandwidth costs, and the hosting provider isn’t paying for the bandwidth costs, customers shouldn’t be charged for the bandwidth costs at the same rate as if the traffic was being sent over the public Internet. Unfortunately, Amazon’s supposed obsession over doing the right thing for customers doesn’t extend to egress charges.
Artificially Held High
Amazon’s mission statement is: “We strive to offer our customers the lowest possible prices, the best available selection, and the utmost convenience.” And yet, when it comes to egress, their prices are far from the lowest possible.
During the last ten years, industry wholesale transit prices have fallen an average of 23% annually. Compounded over that time, wholesale bandwidth is 93% less expensive than 10 years ago. However, AWS’s egress fees over that same period have fallen by only 25%.
And, since 2018, the egress fees AWS charges in North America and Europe have not dropped a penny even as wholesale prices in those markets over the same time period have fallen by more than half.
AWS’s Hotel California Pricing
Another oddity of AWS’s pricing is that they charge for data transferred out of their network but not for data transferred into their network. If the only time you’ve paid for bandwidth is with your residential Internet connection, then this may make some sense. Because of some technical limitations of the cable network, download bandwidth is typically higher than upload bandwidth on cable modem connections. But that’s not how wholesale bandwidth is bought or sold.
Wholesale bandwidth isn’t like your home cable connection. Instead, it’s symmetrical. That means that if you purchase a 1 Mbps (1 Megabit per second) connection, then you have the capacity to send 1 Megabit out and receive another 1 Megabit in every second. If you receive 1 Mbps in and simultaneously 1 Mbps out, you pay the same price as if you receive 1 Mbps in and 0 Mbps out or 0 Mbps in and 1 Mbps out. In other words, ingress (data sent to AWS) doesn’t cost them any more or less than egress (data sent from AWS). And yet, they charge customers more to take data out than put it in. It’s a head scratcher.
We’ve tried to be charitable in trying to understand why AWS would charge this way. Disappointingly, there just doesn’t seem to be an innocent explanation. As we dug in, even things like writes versus reads and the wear they put on storage media, as well as the challenges of capacity planning for storage capacity, suggest that AWS should charge less for egress than ingress.
But they don’t.
The only rationale we can reasonably come up with for AWS’s egress pricing: locking customers into their cloud, and making it prohibitively expensive to get customer data back out. So much for being customer-first.
But… But… But…
AWS may object that this doesn’t take into account the cost of things like metro dark fiber between data centers, amortized optical and other networking equipment, and cross connects. In our experience, those costs amount to a rounding error of less than one cent per Mbps when operating at AWS-like scale. And these prices have been falling at a similar rate to the decline in the price of bandwidth over the past 10 years. Yet AWS’s egress prices have barely budged.
All the data above is derived from what’s published on AWS’s simple pricing calculator. There’s no doubt that some large customers are able to negotiate lower prices. But these are the prices charged to small businesses and startups by default. And, when we’ve reviewed pricing even with large AWS customers, the egress fees remain egregious.
It’s Not Too Late!
We have a lot of mutual customers who use Cloudflare and AWS. They’re a great service, and we want to support our mutual customers and provide services in a way that meets their needs and is always as secure, fast, reliable, and efficient as possible. We remain hopeful that AWS will do the right thing, lower their egress fees, join the Bandwidth Alliance — following the lead of the majority of the rest of the hosting industry — and pass along savings from peering with Cloudflare and other networks to all their customers.
……. 1Here’s the calculation to convert a 1 Mbps flow into TB stocks: 1 Mbps @ 100% for 1 month = (1 million bits per second) * (60 seconds / minute) * (60 minutes / hour) * (730 hours on average/month) divided by (eight bits / byte) divided by 10^12 (to convert bytes to Terabytes) = 0.3285 TB/month.
Around the world government and medical organizations are struggling with one of the most difficult logistics challenges in history: equitably and efficiently distributing the COVID-19 vaccine. There are challenges around communicating who is eligible to be vaccinated, registering those who are eligible for appointments, ensuring they show up for their appointments, transporting the vaccine under the required handling conditions, ensuring that there are trained personnel to administer the vaccine, and then doing it all over again as most of the vaccines require two doses.
Cloudflare can’t help with most of that problem, but there is one key part that we realized we could help facilitate: ensuring that registration websites don’t crash under load when they first begin scheduling vaccine appointments. Project Fair Shot provides Cloudflare’s new Waiting Room service for free for any government, municipality, hospital, pharmacy, or other organization responsible for distributing COVID-19 vaccines. It is open to eligible organizations around the world and will remain free until at least July 1, 2021 or longer if there is still more demand for appointments for the vaccine than there is supply.
Crashing Registration Websites
The problem of vaccine scheduling registration websites crashing under load isn’t theoretical: it is happening over and over as organizations attempt to schedule the administration of the vaccine. This hit home at Cloudflare last weekend. The wife of one of our senior team members was trying to register her parents to receive the vaccine. They met all the criteria and the municipality where they lived was scheduled to open appointments at noon.
When the time came for the site to open, it immediately crashed. The cause wasn’t hackers or malicious activity. It was merely that so many people were trying to access the site at once. “Why doesn’t Cloudflare build a service that organizes a queue into an orderly fashion so these sites don’t get overwhelmed?” she asked her husband.
A Virtual Waiting Room
Turns out, we were already working on such a feature, but not for this use case. The problem of fairly distributing something where there is more demand than supply comes up with several of our clients. Whether selling tickets to a hot concert, the latest new sneaker, or access to popular national park hikes it is a difficult challenge to ensure that everyone eligible has a fair chance.
The solution is to open registration to acquire the scarce item ahead of the actual sale. Anyone who visits the site ahead of time can be put into a queue. The moment before the sale opens, the order of the queue can be randomly (and fairly) shuffled. People can then be let in in order of their new, random position in the queue — allowing only so many at any time as the backend of the site can handle.
At Cloudflare, we were building this functionality for our customers as a feature called Waiting Room. (You can learn more about the technical details of Waiting Room in this post by Brian Batraski who helped build it.) The technology is powerful because it can be used in front of any existing web registration site without needing any code changes or hardware installation. Simply deploy Cloudflare through a simple DNS change and then configure Waiting Room to ensure any transactional site, no matter how meagerly resourced, can keep up with demand.
Recognizing a Critical Need; Moving Up the Launch
We planned to release it in February. Then, when we saw vaccine sites crashing under load and frustration of people eligible for the vaccine building, we realized we needed to move the launch up and offer the service for free to organizations struggling to fairly distribute the vaccine. With that, Project Fair Shot was born.
Government, municipal, hospital, pharmacy, clinic, and any other organizations charged with scheduling appointments to distribute the vaccine can apply to participate in Project Fair Shot by visiting: projectfairshot.org
Giving Front Line Organizations the Technical Resources They Need
The service will be free for qualified organizations at least until July 1, 2021 or longer if there is still more demand for appointments for the vaccine than there is supply. We are not experts in medical cold storage and I get squeamish at the sight of needles, so we can’t help with many of the logistical challenges of distributing the vaccine. But, seeing how we could support this aspect, our team knew we needed to do all we could to help.
The superheroes of this crisis are the medical professionals who are taking care of the sick and the scientists who so quickly invented these miraculous vaccines. We’re proud of the supporting role Cloudflare has played helping ensure the Internet has continued to function well when the world needed it most. Project Fair Shot is one more way we are living up to our mission of helping build a better Internet.
We wanted to close out Privacy & Compliance Week by talking about something universal and certain: taxes. Businesses worldwide pay employment taxes based on where their employees do work. For most businesses and in normal times, where employees do work has been relatively easy to determine: it’s where they come into the office. But 2020 has made everything more complicated, even taxes.
As businesses worldwide have shifted to remote work, employees have been working from “home” — wherever that may be. Some employees have taken this opportunity to venture further from where they usually are, sometimes crossing state and national borders.
In a lot of ways, it’s gone better than expected. We’re proud of helping provide technology solutions like Cloudflare for Teams that allow employees to work from anywhere and ensure they still have a fast, secure connection to their corporate resources. But increasingly we’ve been hearing from the heads of the finance, legal, and HR departments of our customers with a concern: “If I don’t know where my employees are, I have no idea where I need to pay taxes.”
Today we’re announcing the beta of a new feature for Cloudflare for Teams to help solve this problem: Workplace Records. Cloudflare for Teams uses Access and Gateway logs to provide the state and country from which employees are working. Workplace Records can be used to help finance, legal, and HR departments determine where payroll taxes are due and provide a record to defend those decisions.
Every location became a potential workplace
Before 2020, employees who frequently traveled could manage tax jurisdiction reporting by gathering plane tickets or keeping manual logs of where they spent time. It was tedious, for employees and our payroll team, but manageable.
The COVID pandemic transformed that chore into a significant challenge for our finance, legal, and HR teams. Our entire organization was suddenly forced to work remotely. If we couldn’t get comfortable that we knew where people were working, we worried we may be forced to impose somewhat draconian rules requiring employees to check-in. That didn’t seem very Cloudflare-y.
The challenge impacts individual team members as well. Reporting mistakes can lead to tax penalties for employees or amendments during filing season. Our legal team started to field questions from employees stuck in new regions because of travel restrictions. Our payroll team prepared for a backlog of amendments.
Logging jurisdiction without manual reporting
When team members open their corporate laptops and start a workday, they log in to Cloudflare Access — our Zero Trust tool that protects applications and data. Cloudflare Access checks their identity and other signals like multi-factor methods to determine if they can proceed. Importantly, the process also logs their region so we can enforce country-specific rules.
Our finance, legal, and HR teams worked with our engineering teams to use that model to create Workplace Records. We now have the confidence to know we can meet our payroll tax obligations without imposing onerous limitations on team members. We’re able to prepare and adjust, in real-time, while confidentially supporting our employees as they work remotely for wherever is most comfortable and productive for them.
Respecting team member privacy
Workplace Records only provides resolution within a taxable jurisdiction, not a specific address. The goal is to give only the information that finance, legal, and HR departments need to ensure they can meet their compliance obligations.
The system also generates these reports by capturing team member logins to work applications on corporate devices. We use the location of that login to determine “this was a workday from Texas”. If a corporate laptop is closed or stored away for the weekend, we aren’t capturing location logs. We’d rather team members enjoy time off without connecting.
Two clicks to enforce regional compliance
Workplace Records can also help ensure company policy compliance for a company’s teams. For instance, companies may have policies about engineering teams only creating intellectual property in countries in which transfer agreements are in place. Workplace Records can help ensure that engineering work isn’t being done in countries that may put the intellectual property at risk.
Administrators can build rules in Cloudflare Access to require that team members connect to internal or SaaS applications only from countries where they operate. Cloudflare’s network will check every request both for identity and the region from which they’re connecting.
We also heard from our own accounting teams that some regions enforce strict tax penalties when employees work without an incorporated office or entity. In the same way that you can require users to work only from certain countries, you can also block users from connecting to your applications from specific regions.
No deciphering required
When we started planning Workplace Records, our payroll team asked us to please not send raw data that added more work on them to triage and sort.
Available today, you can view the country of each login to internal systems on a per-user basis. You can export this data to an external SIEM and you can build rules that control access to systems by country.
Launching today in beta is a new UI that summarizes the working days spent in specific regions for each user. Workplace Records will add a company-wide report early in Q1. The service is available as a report for free to all Cloudflare for Teams customers.
Going forward, we plan to work with Human Capital Management (HCM), Human Resource Information Systems (HRIS), Human Resource Management Systems (HRMS), and Payroll providers to automatically integrate Workplace Records.
What’s next?
At Cloudflare, we know even after the pandemic we are going to be more tolerant of remote work than before. The more that we can allow our team to work remotely and ensure we are meeting our regulatory, compliance, and tax obligations, the more flexibility we will be able to provide.
Cloudflare for Teams with Workplace Records is helping solve a challenge for our finance, legal, and HR teams. Now with the launch of the beta, we hope we can help enable a more flexible and compliant work environment for all our Cloudflare for Teams customers. This feature will be available to all Cloudflare for Teams subscribers early next week. You can start using Cloudflare for Teams today at no cost for up to 50 users, including the Workplace Records feature.
Tomorrow kicks off Cloudflare’s Privacy & Compliance Week. Over the course of the week, we’ll be announcing ways that our customers can use our service to ensure they are in compliance with an increasingly complicated set of rules and laws around the world.
Early in Cloudflare’s history, when Michelle, Lee, and I were talking about the business we wanted to build, we kept coming back to the word trust. We realized early on that if we were not trustworthy then no one would ever choose to route their Internet traffic through us. Above all else, we are in the trust business.
Every employee at Cloudflare goes through orientation. I teach one of the sessions titled “What Is Cloudflare?” I fill several white boards with notes and diagrams talking about where we fit in to the market. But I leave one for the end so I can write the word TRUST, in capital letters, and underline it three times. Trust is the foundation of our business.
Standing Up For Our Customers from Our Early Days
That’s why we’ve made decisions that other companies may not have. In January 2013 the FBI showed up at our door with a National Security Letter requesting information on a customer. It was incredibly scary.
We had fewer than 30 employees at the time. The agents, while professional, were incredibly intimidating. And the letter ordered us to turn over information and forbid us from discussing it with anyone other than our attorneys.
There’s a proper role for law enforcement, but National Security Letters, which at the time had almost no oversight, could be written and enforced by a single branch of the US government, and gagged recipients from talking about them indefinitely, ran counter to the foundational principles of due process. So we decided to sue the United States government.
I am thankful for Cloudflare’s Board for encouraging us to always fight for our principles. I am also thankful for the Electronic Frontier Foundation, who served as our attorneys in the case. It took several years, and we were gagged from talking about it until 2017, but ultimately the FBI withdrew the letter and Congress has taken steps to reform the law and ensure better oversight. There is a proper role for law enforcement, but when it crosses a line and infringes on basic principles of due process, then we believe it’s important to challenge it.
It’s all about trust.
Recognizing It’s Not Our Data
The same is true for the commercial side of our business. As soon as Cloudflare took off, the ad tech companies came knocking: “Do you have any idea how much you could make if you just let us cookie and retarget individuals passing through your network?” I took a lot of those meetings in our early days, but always came away feeling uneasy. Talking through it with Michelle she concisely expressed why we would never be in the advertising business: “It’s not our data.”
And that’s right. For our customers who do run ads on their sites, if we sold the data then we’d effectively be undercutting them. And, more fundamentally, if we were some invisible service that tracked you online without your knowledge then that would fail the creepiness test. While we believe there can be good ad-supported businesses, Cloudflare will never be one.
As a result, we’ve always seen any personally identifiable information that passes through our network as a toxic asset and purged it as quickly as possible. That can be a tension because we are a security company and part of security requires us to be able to know, for instance, if a particular IP address is sending DDoS traffic. But we’ve invested in implementing or inventing technologies — like Universal SSL, Privacy Pass, Encrypted DNS, and ESNI — that keep your private data private, including from us.
Again, it’s all about trust.
Privacy In Our DNA
While Cloudflare started in California, we have had a global perspective from our earliest days. Today, nearly half of our C-level executives are Europeans, including our CTO, CIO, and CFO. Michelle, my co-founder and Cloudflare’s COO, is Canadian, a country that shares many of Europe’s values around privacy. We have offices around the world and far more engineers working outside of Silicon Valley than inside of it.
I wrote the first version of our Privacy Policy back in 2010. It included from the first draft this clear statement: “Cloudflare will not sell, rent, or give away any of your personal information without your consent. It is our overriding privacy principle that any personal information you provide to us is just that: private.” That is still true today. While other tech companies have made their policies more flexible over time, we’ve made ours stricter, including committing to a list of things we have never done and will fight like hell to never do:
Cloudflare has never turned over our encryption or authentication keys or our customers’ encryption or authentication keys to anyone.
Cloudflare has never installed any law enforcement software or equipment anywhere on our network.
Cloudflare has never provided any law enforcement organization a feed of our customers’ content transiting our network.
Cloudflare has never modified customer content at the request of law enforcement or another third party.
Cloudflare has never modified the intended destination of DNS responses at the request of law enforcement or another third party.
Cloudflare has never weakened, compromised, or subverted any of its encryption at the request of law enforcement or another third party.
While many tech companies struggled to comply with privacy regulations such as GDPR, at Cloudflare it was relatively easy because the principles it imposed were at our core from our very outset. We don’t have a business if we don’t have trust, and being transparent, principled, and respecting the sanctity of personal data is critical to us continuously earning that trust.
Improving the Privacy of Our Service
But we’re not done; we can do more. There are things that have irked me about our service for a long time. For instance, from our earliest days we’ve used the _cfduid cookie to help with some of our security functions. That has meant that if you used Cloudflare you couldn’t be completely cookieless. John Graham-Cumming and I challenged the team earlier this year to see if we could kill it. Our team rose to the challenge and this week we’re announcing its deprecation. To my mind, that announcement alone is worth an entire week of celebrations.
We have multiple data centers around the world that aggregate and process data in order to display logs and provide features. While having geographic redundancy helps with availability, some customers want to make sure their data never leaves a particular region. This week we’ll be giving users a lot more control over what data is processed where.
And, like we have during Privacy and Encryption weeks in years past, we will continue to invest in technologies to enable better encryption and more private use of core Internet services like DNS. Wouldn’t it be cool if, for example, we could ensure that no DNS provider could ever see both who is using their service and also where on the Internet those users are going? Stay tuned!
Helping Customers With Increasingly Complex Compliance Challenges
While we continue to invest in ensuring Cloudflare leads the way on privacy, more and more of our customers are also looking for solutions to be more private themselves. This month we expect that the EU’s new Digital Services Act will be proposed. We expect that it will continue to raise the bar on how companies doing business in Europe have to handle customers’ data. While the Internet giants will have the resources to comply with these heightened requirements, for everyone else they will create new challenges.
To that end, this week we’re announcing the Cloudflare Data Localization Suite. It provides our customers with a powerful set of tools to ensure they have control over how and where their data is processed in order to help comply with increasingly complex local data processing requirements. This includes enhancements to Workers, our edge computing and storage platform, to help modern applications get built such that users’ data never leaves their own country or region.
It’s clear to us that the model of sending all your customer data back to a data center in Ashburn, VA, regardless of where those customers are located in the world, will look as antiquated in an increasingly privacy-conscious world as carrying a stack of punch cards to a central mainframe would today. In the not too distant future, regulations are inevitably going to force data storage and processing to be local. And, with a network that today already spans more than 100 countries, Cloudflare stands ready to help our customers enable that more private future.
Stay Tuned
Stay tuned this week to our blog for a series of announcements. Since these are topics that are so important in Europe right now, we’ll be simultaneously publishing most of them in French, Italian, Spanish, Portuguese, and German as well as English. Also check out Cloudflare TV where we’ll be interviewing a series of people whose views on privacy and compliance we respect and have learned from.
Cloudflare’s mission is to help build a better Internet. And there is no doubt that a better Internet is a more private Internet. With that in mind, welcome to Privacy & Compliance Week.
There is significant global attention around the upcoming United States election. Through the Athenian Project and Cloudflare for Campaigns, Cloudflare is providing free protection from cyber attacks to a significant number of state and local elections’ websites, as well as those of federal campaigns.
One of the bedrocks of a democracy is that people need to be able to get access to relevant information to make a choice about the future of their country. This includes information about the candidates up for election; learning about how to register, and how to cast a vote; and obtaining accurate information on the results.
A question that I’ve been increasingly asked these past few months: are cyberattacks going to impact these resources leading up to and on election day?
Internally, we have been closely monitoring attacks on the broader elections and campaign websites and have a team standing by 24×7 to help our current customers as well as state and local governments and eligible political campaigns to protect them at no cost from any cyberattacks they may see.
The good news is that, so far, cyberattacks have not been impacting the websites of campaigns and elections officials we are monitoring and protecting. While we do see some background noise of attacks, they have not interfered in the process so far. The attack traffic is below what we saw in 2016 and below what is typical in elections we have observed in other countries.
But there are still nearly two weeks before election day so our guard is up. We thought it was important to provide a view into how overall traffic to campaign and elections sites is trending as well as a view into the cyberattacks we’re observing. To that end, today we’re sharing data from our internal monitoring systems publicly through Cloudflare Radar. You can access the special “Election 2020” Radar dashboard here:
The dashboard is updated continuously with information we’re tracking on traffic to elections-related sites, both legitimate and from cyberattacks. It is normal to see fluctuations in this traffic depending on the time of day as well as when there will be occasional cyberattacks. So far, nothing here surprises us.
It’s important to note that Cloudflare does not see everything. We do not, for instance, have any view into misinformation campaigns that may be on social media. We also do not protect every state and local government or every campaign.
That said, we have Athenian Project participants in more than half of US states — including so-called red states, blue states, purple states, and several of the battleground states. We also have hundreds of federal campaigns that are using us ranging across the political spectrum. While we may not see a targeted cyberattack, given the critical role the web now plays to the election process, we believe we would likely see any wide-spread attacks attempting to disrupt the US elections.
So far, we are not seeing anything that suggests such an attack has impacted the election to date.
Our team will continue to monitor the situation. If any state or local elections agency or campaigns comes under attack, we stand ready to help at no cost through the Athenian Project and Cloudflare for Campaigns.
We could not have built Cloudflare into the company it is today without a stable, functional government. In the United States, that process depends on democracy and fair elections not tainted by outside influence like cyberattacks. We believe it is our duty to provide our technology where we can to help ensure this election runs smoothly.
Today we’re announcing Cloudflare One™. It is the culmination of engineering and technical development guided by conversations with thousands of customers about the future of the corporate network. It provides secure, fast, reliable, cost-effective network services, integrated with leading identity management and endpoint security providers.
Over the course of this week, we’ll be rolling out the components that enable Cloudflare One, including our WARP Gateway Clients for desktop and mobile, our Access for SaaS solution, our browser isolation product, and our next generation network firewall and intrusion detection system.
The old model of the corporate network has been made obsolete by mobile, SaaS, and the public cloud. The events of 2020 have only accelerated the need for a new model. Zero Trust networking is the future and we are proud to be enabling that future. Having worked on the components of what is Cloudflare One for the last two years, we’re excited to unveil today how they’ve come together into a robust SASE solution and share how customers are already using it to deliver the more secure and productive future of the corporate network.
What Is Cloudflare One? Secure, Optimized Global Networking
Cloudflare One is a comprehensive, cloud-based network-as-a-service solution that is designed to be secure, fast, reliable and define the future of the corporate network. It replaces a patchwork of appliances and WAN technologies with a single network that provides cloud-based security, performance, and control through one user interface.
Cloudflare One brings together how users connect, on ramps for branch offices, secure connectivity for applications, and controlled access to SaaS into a single platform.
Cloudflare One reflects the complex nature of corporate networking today: mobile and remote users, SaaS applications, a mix of applications hosted in private data centers and public cloud, as well the challenge of employees using the broader Internet securely from their corporate and personal devices.
Whether you call this SASE or simply the new reality, today’s enterprise needs flexibility at every layer of the network and application stack. Secure and authenticated access is needed for users wherever they are: at the office, on a mobile device or working from home. Corporate network architectures need to reflect the state of modern computing that requires secure, filtered Internet access to get to SaaS or public cloud, secure application connectivity to protect against hackers and DDoS, and fast, reliable branch and home office access.
And the new corporate network needs to be global. No matter where applications are hosted, or employees reside, connectivity needs to be secure and fast. With Cloudflare’s massive global presence, traffic is secured, routed, and filtered over an optimized backbone that uses real time Internet intelligence to protect against the latest threats and route traffic around bad Internet weather and outages.
However, you’re only as strong as your weakest link. It doesn’t matter how secure your network is if you allow the wrong people access, or your end user’s devices are compromised. That is why we’re incredibly excited to announce that Cloudflare One takes the power of Cloudflare’s network and combines it with best-of-breed identity management and device integrity to create a complete solution that encompasses the entire corporate network of today and tomorrow.
Partner ecosystem: Identity Management
Most organizations already have one or more identity management systems. Rather than requiring them to change, we are integrating with all the major providers. This week we’re announcing partnerships with Okta, Ping Identity, and OneLogin. We support nearly all the other leading identity providers including Microsoft Active Directory and Google Workspace, as well as broadly adopted consumer and developer identity platforms like Github, LinkedIn, and Facebook.
Powerfully, Cloudflare One does not require you to standardize on just one identity provider. We see multiple companies that may have one identity provider for full-time employees and another for contractors. Or one they chose themselves and another they inherited from an acquisition. Cloudflare One will integrate with one or more identity providers and allow you to then set consistent policies across all your applications.
The metaphor that makes sense to me is that the identity provider issues passports and Cloudflare One is the border agent that checks that they’re valid. At any particular moment, different passports from different providers may be allowed or forbidden to enter just by updating the instructions the border agent follows.
Partner ecosystem: Device Integrity
In addition to identity, device integrity and endpoint security are an important part of a zero trust solution. This week we’re announcing partnerships with CrowdStrike, VMware Carbon Black, SenitnelOne, and Tanium. These providers run on devices and ensure that they haven’t been compromised. Again, organizations can centralize around a single vendor for device integrity or can mix and match with Cloudflare One providing a consistent control plane.
Extending the border control analogy, it’s like having a temperature screening and COVID-19 test when you enter a country. Even if you have a valid passport, if you’re not healthy then you will be turned away. By partnering with the leading identity and device integrity providers, Cloudflare One provides a robust identity and access management solution that fully delivers on the promise of Zero Trust.
We’re thrilled to partner with these leading identity management and endpoint security companies to make Cloudflare One flexible and robust.
With this as an introduction to Cloudflare One, I wanted to provide some context on why the existing paradigm doesn’t work, what the future of the enterprise network looks like, and where we go from here. In order to understand the power of Cloudflare One, you first have to understand the way we used to build and secure corporate networks and how the transition to mobile, cloud, and remote work have all forced this fundamental change in the paradigm.
The Middle(box) Ages: How Corporate Security Used to Work
The Internet was designed to be a massive, decentralized network. Any computer could connect to that network and route data from one location to another. The model provided resiliency, but did not guarantee fast or available connections. The early Internet also lacked a framework for security.
As a result, enterprises did not trust the Internet as a platform for their businesses. To keep employees productive, network connections had to be fast and available. Those connections also had to be secure. So, businesses built their own shadow versions of the Internet:
Companies purchased dedicated, private connections between offices and across their data centers in the form of expensive MPLS links.
IT teams managed complex routing across offices, VPN hardware, and clients.
Security teams deployed physical firewall boxes and DDoS appliances to keep the private network safe.
When employees had to use the Internet, security teams backhauled traffic through a central location to filter outbound connections with yet more hardware: Internet gateways.
Legacy corporate security followed a castle and moat approach. You put all your sensitive applications and data in the castle, you required all your employees to come to work in the castle every day, and then you built a metaphorical moat around the castle using firewalls, DDoS appliances, gateways and more: an unmanageable mess of devices and vendors.
The Middle(box) Ages Are Long Gone
While smarter attackers finding ways to breach moats were always a concern for the castle and moat approach, ultimately they weren’t what caused the approach to fail. Instead the change came from transformation of the technical landscape. Smartphones made workers increasingly mobile, letting them venture outside the moat. SaaS and the public cloud moved data and corporate applications out of the metaphorical castle.
And, in 2020, COVID-19 changed everything by forcing everyone who could to work remotely. If the employees weren’t coming to work in the castle anymore, the whole paradigm completely breaks down. This transition was happening already, but this year poured gasoline on the already smoldering fire. Increasingly companies are realizing that the only way forward is to embrace the fact that employees, servers and applications are now “on the Internet” and not “in the castle.” This new paradigm is known as “Zero Trust.”
Google’s seminal paper, “BeyondCorp: A New Approach to Enterprise Security,” published in 2014, brought the idea of Zero Trust security into the mainstream. Google’s insight in 2014 was that you could solve the challenges of every employee and application being on the Internet by ensuring that every application would inherently distrust every connection. If there was zero trust inherent to what network you were on, then every user of every application would be continuously authenticated. Powerfully, that would simultaneously enhance security while enabling more use of cloud applications as well as mobile and remote work.
The Future LAN: A Secure WAN
What we realized talking to customers was that even the analyst and competitor framing of the future corporate network didn’t fully recognize some challenges that come with a Zero Trust model. One of the benefits of embracing a Zero Trust model is that it makes enabling branch and home offices easier and less expensive. Rather than having to lease expensive MPLS circuits to connect branch offices — something that is literally impossible as people work from home — you instead require every use of every application to be authenticated.
This lines up with something else we’ve heard from our customers over the last six months: “maybe the Internet is almost good enough.” Like physical offices, many MPLS or SD-WAN deployments are currently sitting idle. And yet, employees continue to be productive. If users could move to a model that runs on the Internet, and one that improves the Internet, teams can stop spending money on legacy routing. Rather than trying to build more private networks, the corporate network of the future leverages the Internet but with heightened security, performance, and reliability.
That sounds great, but it opens a whole new can of worms. Inherently to do this you need to expose more of your applications to the Internet. While they may be safe from unauthorized use if you’ve properly implemented Zero Trust, that opens them to many less sophisticated, but highly disruptive challenges.
At the end of 2019 we saw a disturbing new trend begin to emerge. DDoS attackers shifted their focus from embarrassing companies by knocking their websites offline to increasingly targeting internal applications and networks. Unfortunately, we’ve seen more of these attacks launched throughout the pandemic.
It’s not a coincidence. It’s the direct result of companies being forced to expose more of their internal applications to the Internet in order to support remote work. To our surprise, it has turned out that while we anticipated Access and Gateway being the natural pairing of products, equally often customers looking to move to a Zero Trust model are bundling Cloudflare’s DDoS and WAF products.
It makes sense. If you are exposing more of your applications to the Internet, then the problems that Internet-facing applications have had to deal with in the past now become the problems of your internal applications as well. It’s become clear to us that the future of a SASE or Zero Trust network needs to also include DDoS mitigation and WAF as well.
Making the Internet Secure and Reliable Enough for the Enterprise
We agree with the customers we’ve talked to who say that the Internet is almost good enough to replace a corporate network. We’ve been building products to fill in the gaps where it needs to be better. Virtual appliances in regional public cloud providers are not sufficient. Enterprises need a global, distributed network that accelerates traffic in any location.
We’ve spent the last decade building Cloudflare’s network; bringing the Internet closer to users around the world and supporting incredible scale. According to W3Techs, more than 14% of the web already relies on our network. We can also use that to constantly measure the Internet at scale and find faster routes. That scale allows us to deliver Cloudflare One to any organization, no matter where they are located or how global their workforce, and ensure their network and applications are secure, fast, and reliable.
Foreshadowing Cloudflare One
The same lessons we’ve learned handling traffic for the websites on our network can be applied to how enterprises connect to everything else. We started that journey last year when we launched Cloudflare WARP, a consumer product that routes all connections leaving a personal device through Cloudflare’s network, where we can encrypt and accelerate it. This week, we’ll show how the WARP Client is now one of the on-ramps to get employee traffic onto Cloudflare One.
We launched WARP on mobile devices because we knew they would prove to be the most difficult to get right. Traditionally, VPN clients are clunky battery sucks designed for desktops and, if they have mobile versions at all, they’ve been clumsily ported over. We set out to build WARP to work great on mobile, not burning battery life or slowing connections down, because we knew if we could pull that off then it would be easy to port it to the less limited constraints of the desktop.
We also launched it for consumers first because they are the best QA team you could ever assemble. More than 10 million consumers have been putting WARP through its paces for the last year. We’ve seen edge cases from every corner of the Internet and used them to iron the bugs out. We knew that if we could make the WARP Client something that consumers loved to use then it would be a stark contrast to every other enterprise solution in the market.
Meanwhile, we built products to deliver the same improvements to data centers and offices. We announced Magic Transit last year to provide secure, performant, and reliable IP connectivity to the Internet. Earlier this year, we expanded that model when we launched Cloudflare Network Interconnect (CNI) to allow our customers to interconnect branch offices and data centers directly with Cloudflare.
Cloudflare Access starts by introducing identity into Cloudflare’s network. We apply filters based on identity and context to both inbound and outbound connections. Every login, request, and response proxies through Cloudflare’s network regardless of the location of the server or user.
Cloudflare Gateway keeps connections to the rest of the Internet safe. By routing all traffic through Cloudflare’s network first, customers can deprecrate on-premise firewalls eliminating Internet backhaul requirements that slow down users.
Pulling the Pieces Together
We think about the products in Cloudflare One in two categories:
On-ramps: the products that connect a user, device, or location to Cloudflare’s edge. WARP for endpoints, Magic Transit and CNI for networks, Argo Smart Routing to accelerate traffic.
Filters: the products that shield networks from attacks, inspect traffic for threats, and apply least privilege rules to data and applications. Access for Zero Trust rules, Gateway for traffic filtering, Magic Firewall for network filtering.
Most competitors in this space focus on one area, which loses out on the efficiencies of combining them in a single solution. Cloudflare One brings those together on our network. By integrating both sides of the challenge, we can give administrators a single place to manage and secure their network.
What Differentiates Cloudflare One
Easy to Deploy, Manage, and Use
We’ve always offered free and pay-as-you-go plans that teams of any size could sign up for with a credit card. Those customers lack the systems integrators or IT departments of large enterprises. To serve those teams, we had to build a control plane and dashboard that was accessible and easy to use.
The products in Cloudflare One follow that same approach; comprehensive enough for enterprises but easy to use to make these products accessible to any team. We’ve also extended that to end users; the client application that powers Gateway is built on what we learned creating Cloudflare WARP for consumer users.
Unified Solution
Cloudflare One puts the entire corporate network behind a single pane of glass. By integrating with leading identity providers and endpoint security solutions, Cloudflare One enables companies to enforce a consistent set of policies across all their applications. Since the network is the common denominator of all applications, by building control into the network Cloudflare One ensures consistent policies whether an application is new or legacy, run on-premise or in the cloud, and delivered from your own infrastructure or a multi-tenant SaaS provider.
Cloudflare One also helps rationalize complicated deployments. While it would be great if every app and every employee and contractor used the same identity provider, for example, that isn’t always possible. Acquisitions, skunkworks projects, and internal disagreements can cause multiple different solutions to be present inside one company. Cloudflare One allows you to plug different providers into one unified network control plane to ensure consistent policies.
Significant ROI
Our core tenet of serving the entire Internet has always forced us to obsess over costs. Efficiency is in the DNA of Cloudflare and we use our efficiency to pass along customer-friendly, fixed-rate pricing. Cloudflare One builds on that experience to deliver a platform that is more cost-effective than combining point solution vendors. The differences are especially apparent versus other providers who have tried to build on top of public cloud platforms and inherit their cost and inconsistent network performance.
To achieve the level of efficiency needed to compete with hardware appliances required us to invent a new type of platform. That platform needed to be built our own network where we could drive costs down and ensure the highest level of performance. It needed to be architected so any server in any city that made up Cloudflare’s network could run every one of our services. That means that Cloudflare One runs across Cloudflare’s global network spanning more than 200 cities worldwide. Even your farthest flung branch offices and remote workers are likely within milliseconds of servers powering Cloudflare One, ensuring our service works well wherever your team works.
Leverages Cloudflare’s Scale
Cloudflare already sits in front of a huge portion of the Internet. That allows us to see and respond to new security threats continuously. It also means that Cloudflare One customers’ traffic can be more efficiently routed, even when going to applications that would appear to be on the public Internet.
For instance, an employee behind Cloudflare One who is catching up on holiday shopping during their lunch break can have their traffic routed from a corporate branch office, across Cloudflare’s Magic Transit, over Cloudflare’s global backbone, across Cloudflare’s Network Interconnect, and to the ecommerce provider. Because Cloudflare handles the packets end-to-end, we can ensure they are encrypted, optimally routed, and efficiently delivered. As more of the Internet uses Cloudflare, the experience of surfing the Internet for Cloudflare One customers will grow even more exceptional.
What Does Cloudflare One Replace?
Instead of expensive MPLS links or complex SD-WAN deployments, Cloudflare One provides two on-ramps to your applications and the entire Internet: WARP and Magic Transit. WARP connects employees from any device, and any location, to Cloudflare’s network. Magic Transit allows broad deployments across whole offices or data centers.
Cloudflare Access replaces private-networks-as-security with Zero Trust controls. Later this week, we’ll announce how you can extend Access to any application, including SaaS applications.
Finally, Cloudflare One eliminates traditional network firewalls and web gateways. Cloudflare Gateway inspects traffic leaving any device in your organization to block threats on the Internet and prevent data from leaving. Magic Firewall will give your networks the same security, filtering traffic at the transport layer to replace the top-of-rack firewalls that block data exfiltration or attacks from unsecure network protocols.
What Comes Next?
Your team can start using Cloudflare One today. Add Zero Trust control to your applications with Cloudflare Access and secure DNS queries with Cloudflare Gateway. Keep networks safe from DDoS attacks with Magic Transit and connect your applications through Cloudflare with Argo Tunnel.
Over the course of the week, we’ll be launching new features and products to start to complete this vision. On Tuesday, we’ll extend the Zero Trust security of Cloudflare Access to all of your applications. Starting Wednesday, teams will be able to use Cloudflare WARP to proxy all employee traffic to Cloudflare where Gateway will now secure more than just DNS queries. You’ll be invited to sign up for Cloudflare’s browser isolation beta on Thursday and we’ll wrap the week with new APIs to control how Magic Transit secures your network.
It’s going to be a busy week, but we’re just getting started. Replacing a corporate network should not also mean you lose control over how that network operates. Magic WAN is our solution to complex SD-WAN deployments.
Security for that entire network should also work in both directions. Magic Firewall is our alternative to the clunky “next-generation firewall” appliances that secure outbound traffic. Data loss prevention (DLP) is another space that has lacked innovation and where we plan to extend Cloudflare One.
Finally, you should have visibility into that network. We’ll be launching new tools to detect and mitigate intrusion attempts that happen anywhere on your network, including unauthorized access to any SaaS applications you use. Now that we’ve built the on-ramps onto Cloudflare One, we’re excited to continue to innovate to provide more functionality and control to solve our customers biggest network security, performance, and reliability challenges.
Delivering the Network Customers Need Today
Over the last 10 years, Cloudflare has built one of the fastest, most reliable, most secure networks in the world. We’ve seen the power of using that network internally to enable our own teams to innovate quickly and securely. With the launch of Cloudflare One, we’re extending the power of Cloudflare’s network to meet the challenges of any company. The move to Zero Trust is a paradigm shift but the changes to how we work we believe has made it inevitable for every company. We’re proud of how we’ve been able to help some of Cloudflare One’s first customers reinvent their corporate networks. It makes sense to close with their own words.
“JetBlue Travel Products needed a way to give crew-members secure and simple access to internally-managed benefit apps. Cloudflare gave us all that and more — a much more efficient way to connect business partners and crew-members to critical internal tools.” — Vitaliy Faida, General Manager, Data/DevSecOps at JetBlue Travel Products.
“OneTrust relies on Cloudflare to maintain our network perimeter, so we can focus on delivering technology that helps our customers be more trusted. “With Cloudflare, we can easily build context-aware Zero Trust policies for secure access to our developer tools. Employees can connect to the tools they need so simply teams don’t even know Cloudflare is powering the backend. It just works.” — Blake Brannon, CTO of OneTrust.
“Discord is where the world builds relationships. Cloudflare helps us deliver on that mission, connecting our internal engineering team to the tools they need. With Cloudflare, we can rest easy knowing every request to our critical apps is evaluated for identity and context — a true Zero Trust approach.” — Mark Smith, Director of Infrastructure at Discord.
“When you’re a fast-growing, security-focused company like Area 1, anything that slows development down is the enemy. With Cloudflare, we’ve found a simpler, more secure way to connect our employees to the tools they need to keep us growing – and the experience is lightning-fast.” — Blake Darché, CSO at Area 1 Security.
“We launched quickly in April 2020 to bring remote learning to children throughout the UK during the coronavirus pandemic, Cloudflare Access made it fast and simple to authenticate a huge network of teachers and developers into our production sites and we set it up in literally less than an hour. Cloudflare’s WAF helped ensure the security and resilience of our public-facing website from day one.” — John Roberts, Technology Director at Oak National Academy.
“With Cloudflare, we’ve been able to reduce our dependence on VPNs and IP allow-listing for development environments. Our developers and testers aren’t required to login from specific locations, and we’ve been able to deploy an SSO solution to simplify the login process. Access is easier to manage than VPNs and other remote access solutions, which has removed pressure from our IT teams. They can focus on internal projects instead of spending time managing remote access.” — Alexandre Papadopoulos, Director of Cyber Security, INSEAD.
Cloudflare launched on September 27, 2010 — 10 years ago today. Stopping to look back over the last 10 years is challenging in some ways because so much of who we are has changed radically. A decade ago when we launched we had a few thousand websites using us, our tiny office was above a nail salon in Palo Alto, our team could be counted on less than two hands, and our data center locations on one hand.
Outside our first office in Palo Alto in 2010. Photo by Ray Rothrock.
As the company grew, it would have been easy to stick with accelerating and protecting developers and small business websites and not see the broader picture. But, as this year has shown with crystal clarity, we all depend on the Internet for many aspects of our lives: for access to public information and services, to getting work done, for staying in touch with friends and loved ones, and, increasingly, for educating our children, ordering groceries, learning the latest dance moves, and so many other things. The Internet underpins much of what we do every day, and Cloudflare’s mission to help build a better Internet seems more and more important every day.
Over time Cloudflare has gone from an idea on a piece of paper to one of the largest networks in the world that powers millions of customers. Because we made our network to be flexible and programmable, what we’ve been able to do with it has expanded over time as well. Today we secure the Internet end-to-end — from companies’ infrastructure to individuals seeking a faster, more secure, more private connection. Our programmable, global network is at the core of everything we have been able to achieve so far.
Updating Our Annual Founders’ Letter
This is also the approximate one-year anniversary of Cloudflare going public. At the time, we wrote our first founders’ letter to the potential investors. We thought it made sense on this day, which we think of as our birthday, to reflect on the last year, as well as the last 10 years, and start a tradition of updating our original founders’ letter on September 27th every year.
Ringing the bell to go public on the NYSE on September 13, 2019.
It’s been quite a year for our business. Since our IPO, we’ve seen record expansion of new customers. That growth has come both from expanding our existing customers as well as winning new business from new customers.
The percentage of the Fortune 1,000 that pay for one or more of Cloudflare’s services rose from 10% when we went public to more than 16% today. Across the web as a whole, according to W3Techs’ data, over the last year Cloudflare has grown from 10.1% of the top 10 million websites using our services to 14.5% using them today. (Amazon CloudFront, in second place based on the number of websites they serve, grew from 0.8% to 0.9% over the same period.)
Every year to celebrate our birthday we’ve made it a tradition to launch products that surprise the market with new ways to expand how anyone can use our network. We think of them as gifts back to the Internet. Three years ago, for instance, we launched our edge computing platform called Workers. Today, just three years later, hundreds of thousands of developers are using Workers to build applications, many of which we believe would be impossible to build on any other platform.
It is impossible to reflect on the last year and not see the impact of the COVID-19 pandemic on our business, our customers, our employees, as well our friends, colleagues, and loved ones in the greater community. It’s heartening to think that for more than half of Cloudflare’s life as a public company our team has worked remote.
2020 was meant to be an Olympic year, but COVID-19 stopped that, like much else, from happening. Eight years ago, when Cloudflare was just two, the creator of the World Wide Web, Tim Berners-Lee, sent a message from the opening ceremony of the 2012 Olympics. That message read “This is for everyone” and the idea that the Internet is for all of us continues to be a key part of Cloudflare’s ethos today.
When we started Cloudflare we wanted to democratize what we thought were technologies only available to the richest and most Internet-focused organizations. We saw an opportunity to make available to everyone — from individual developers to small businesses to large corporations — the sorts of speed, protection, and reliability that, at the time, only the likes of Google, Amazon, and Facebook could afford.
Giving Back to the Internet
Over 10 years we’ve consistently rolled out the latest technologies, typically ahead of the rest of the industry, to everyone. And in doing so we’ve attracted employees, individuals, developers, customers to our platform. The Internet is for everyone and we’ve shown that a business can be very successful when we aim to serve everyone — large and small.
Something Steve Jobs said back in 1988 still resonates: “If you want to make a revolution, you’ve got to raise the lowest common denominator in every single machine.” Although we aren’t selling machines, we think that’s right: democratizing features matters.
Just look at the scourge of DDoS attacks. Why should DDoS attack mitigation be expensive when it’s a plague on companies large and small? It shouldn’t, and we optimized our business to make it inexpensive for us and passed that on to our customers through Unmetered DDoS Mitigation — another feature we rolled out to celebrate our Birthday Week three years ago.
In 2014, also during Birthday Week, we launched Universal SSL, making encryption — something that had been expensive and difficult — free for all Cloudflare customers. The week we launched it we doubled the size of the encrypted web. Let’s Encrypt followed shortly after and, together, we’ve brought encryption to more than 90% of the web and made the little padlock in your browser something everyone can afford and should expect.
In January of this year, we rolled out Cloudflare for Teams. The product was designed to replace the legacy VPNs and firewalls that were increasingly anachronistic as work moved to the cloud. Little did we know how much COVID-19 would accelerate their obsolescence and make Cloudflare for Teams essential.
Both of us sat on call after call in mid-March with at first small, then increasingly mid-sized, and eventually large and even governmental organizations who reached out to us looking for a way to survive as their teams shifted to working from home and their legacy hardware couldn’t keep up. We made the decision to sacrifice short term profits in order to help businesses large and small get through this crisis by making Cloudflare for Teams free through September.
As we said during our Q1 earnings call, the superheros of this crisis are the medical professionals and scientists who are taking care of the sick and looking for a cure to the disease. But the faithful sidekick throughout has been the Internet. And, as one of the guardians of the Internet, we’re proud of helping ensure it was fast, secure, and reliable around the world when it was needed most. We are proud of how Cloudflare’s products could help the businesses continue to get work done during this unprecedented time by leaning even more on the Internet.
Meeting the Challenges Ahead
Giving back to the Internet is core to who we are, and we do not shy away from a challenge. And there are many challenges ahead. In a little over a month, the United States will hold elections. After the 2016 elections we, along with the rest of the world, were concerned to see technology intended to bring people together instead be used to subvert the democratic process. We decided we needed to do something to help prevent that from happening again.
Three and a half years ago, we launched the Athenian Project to provide free cybersecurity resources to any local, state, or federal officials helping administer elections in the United States. We couldn’t have built Cloudflare into the company it is today without a stable government as a foundational platform. And, when that foundation is challenged, we believe it is our duty to lend our resources to defend it.
Today, we’re helping secure election infrastructure in more than half of the states in the United States. And, over these last weeks before the election, our team is working around the clock to help ensure the process is fair and not disrupted by cyber attacks.
More challenges lie ahead and we won’t shy away from them. Well intentioned governments around the world are increasingly seeking to regulate the Internet to protect their citizens. While the aims are noble, the risk is creating a patchwork of laws that only the Internet giants can successfully navigate. We believe it is critical for us to engage in the conversations around these regulations and work to help ensure as operating online becomes more complex, we can continue to make the opportunities of the Internet created for us when we started Cloudflare available to future startups and entrepreneurs.
Fighting for the Internet
Over the last 10 years, it’s been sad to watch some of the optimism around technology seem to fade. The perception of technology companies shifted from their being able to do no wrong to, today, their being able to do no right. And, as we’ve watched the industry develop, we’ve sympathized with that shift. Too many tech companies have abused customer data, ignored rules, violated privacy, and not been good citizens to the communities in which they operate and serve.
But we continue to believe what we started Cloudflare believing 10 years ago: the Internet itself is a force for good worth fighting to defend. We need to keep striving to make the Internet itself better — always on, always fast, always secure, always private, and available to everyone.
It’s striking to think how much more disruptive the COVID-19 crisis could have been had it struck in 2010 not 2020. The difference today is a better Internet. We’re proud of the role we’ve played in helping build that better Internet.
Today CenturyLink/Level(3), a major ISP and Internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare’s customers as well as a significant number of other services and providers across the Internet. While we’re waiting for a post mortem from CenturyLink/Level(3), I wanted to write up the timeline of what we saw, how Cloudflare’s systems routed around the problem, why some of our customers were still impacted in spite of our mitigations, and what appears to be the likely root cause of the issue.
Increase In Errors
At 10:03 UTC our monitoring systems started to observe an increased number of errors reaching our customers’ origin servers. These show up as “522 Errors” and indicate that there is an issue connecting from Cloudflare’s network to wherever our customers’ applications are hosted.
Cloudflare is connected to CenturyLink/Level(3) among a large and diverse set of network providers. When we see an increase in errors from one network provider, our systems automatically attempt to reach customers’ applications across alternative providers. Given the number of providers we have access to, we are generally able to continue to route traffic even when one provider has an issue.
In this case, beginning within seconds of the increase in 522 errors, our systems automatically rerouted traffic from CenturyLink/Level(3) to alternate network providers we connect to including Cogent, NTT, GTT, Telia, and Tata.
Our Network Operations Center was also alerted and our team began taking additional steps to mitigate any issues our automated systems weren’t automatically able to address beginning at 10:09 UTC. We were successful in keeping traffic flowing across our network for most customers and end users even with the loss of CenturyLink/Level(3) as one of our network providers.
Dashboard Cloudflare’s automated systems recognizing the damage to the Internet caused by the CenturyLink/Level(3) failure and automatically routing around it.
The graph below shows traffic between Cloudflare’s network and six major tier-1 networks that are among the network providers we connect to. The red portion shows CenturyLink/Level(3) traffic, which dropped to near-zero during the incident. You can also see how we automatically shifted traffic to other network providers during the incident to mitigate the impact and ensure traffic continued to flow.
Traffic across six major tier-1 networks that are among the network providers Cloudflare connects to. CenturyLink/Level(3) in red.
The following graph shows 522 errors (indicating our inability to reach customers’ applications) across our network during the time of the incident.
The sharp spike up at 10:03 UTC was the CenturyLink/Level(3) network failing. Our automated systems immediately kicked in to attempt to reroute and rebalance traffic across alternative network providers, causing the errors to drop in half immediately and then fall to approximately 25 percent of the peak as those paths were automatically optimized.
Between 10:03 UTC and 10:11 UTC our systems automatically disabled CenturyLink/Level(3) in the 48 cities where we’re connected to them and rerouted traffic across alternate network providers. Our systems take into account capacity on other providers before shifting out traffic in order to prevent cascading failures. This is why the failover, while automatic, isn’t instantaneous in all locations. Our team was able to apply additional manual mitigations to reduce the number of errors another 5 percent.
Why Did the Errors Not Drop to Zero?
Unfortunately, there were still an elevated number of errors indicating we were still unable to reach some customers. CenturyLink/Level(3) is among the largest network providers in the world. As a result, many hosting providers only have single-homed connectivity to the Internet through their network.
To use the old Internet as a “superhighway” analogy, that’s like only having a single offramp to a town. If the offramp is blocked, then there’s no way to reach the town. This was exacerbated in some cases because CenturyLink/Level(3)’s network was not honoring route withdrawals and continued to advertise routes to networks like Cloudflare’s even after they’d been withdrawn. In the case of customers whose only connectivity to the Internet is via CenturyLink/Level(3), or if CenturyLink/Leve(3) continued to announce bad routes after they’d been withdrawn, there was no way for us to reach their applications and they continued to see 522 errors until CenturyLink/Level(3) resolved their issue around 14:30 UTC.
The same was a problem on the other (“eyeball”) side of the network. Individuals need to have an onramp onto the Internet’s superhighway. An onramp to the Internet is essentially what your ISP provides. CenturyLink is one of the largest ISPs in the United States.
Because this outage appeared to take all of the CenturyLink/Level(3) network offline, individuals who are CenturyLink customers would not have been able to reach Cloudflare or any other Internet provider until the issue was resolved. Globally, we saw a 3.5% drop in global traffic during the outage, nearly all of which was due to a nearly complete outage of CenturyLink’s ISP service across the United States.
So What Likely Happened Here?
While we will not know exactly what happened until CenturyLink/Level(3) issue a post mortem, we can see clues from BGP announcements and how they propagated across the Internet during the outage. BGP is the Border Gateway Protocol. It is how routers on the Internet announce to each other what IPs sit behind them and therefore what traffic they should receive.
Starting at 10:04 UTC, there were a significant number of BGP updates. A BGP update is the signal a router makes to say that a route has changed or is no longer available. Under normal conditions, the Internet sees about 1.5MBs – 2MBs of BGP updates every 15 minutes. At the start of the incident, the number of BGP updates spiked to more than 26MBs of BGP updates per 15 minute period and stayed elevated throughout the incident.
These updates show the instability of BGP routes inside the CenturyLink/Level(3) backbone. The question is what would have caused this instability. The CenturyLink/Level(3) status update offers some hints and points at a flowspec update as the root cause.
What’s Flowspec?
In CenturyLink/Level(3)’s update they mention that a bad Flowspec rule caused the issue. So what is Flowspec? Flowspec is an extension to BGP, which allows firewall rules to be easily distributed across a network, or even between networks, using BGP. Flowspec is a powerful tool. It allows you to efficiently push rules across an entire network almost instantly. It is great when you are trying to quickly respond to something like an attack, but it can be dangerous if you make a mistake.
At Cloudflare, early in our history, we used to use Flowspec ourselves to push out firewall rules in order to, for instance, mitigate large network-layer DDoS attacks. We suffered our own Flowspec-induced outage more than 7 years ago. We no longer use Flowspec ourselves, but it remains a common protocol for pushing out network firewall rules.
We can only speculate what happened at CenturyLink/Level(3), but one plausible scenario is that they issued a Flowspec command to try to block an attack or other abuse directed at their network. The status report indicates that the Flowspec rule prevented BGP itself from being announced. We have no way of knowing what that Flowspec rule was, but here’s one in Juniper’s format that would have blocked all BGP communications across their network.
route DISCARD-BGP {
match {
protocol tcp;
destination-port 179;
}
then discard;
}
Why So Many Updates?
A mystery remains, however, why global BGP updates stayed elevated throughout the incident. If the rule blocked BGP then you would expect to see an increase in BGP announcements initially and then they would fall back to normal.
One possible explanation is that the offending Flowspec rule came near the end of a long list of BGP updates. If that were the case, what may have happened is that every router in CenturyLink/Level(3)’s network would receive the Flowspec rule. They would then block BGP. That would cause them to stop receiving the rule. They would start back up again, working their way through all the BGP rules until they got to the offending Flowspec rule again. BGP would be dropped again. The Flowspec rule would no longer be received. And the loop would continue, over and over.
One challenge of this is that on every cycle, the queue of BGP updates would continue to increase within CenturyLink/Level(3)’s network. This may have gotten to a point where the memory and CPU of their routers was overloaded, causing an additional set of challenges to getting their network back online.
Why Did It Take So Long to Fix?
This was a significant global Internet outage and, undoubtedly, the CenturyLink/Level(3) team received immediate alerts. They are a very sophisticated network operator with a world class Network Operations Center (NOC). So why did it take more than four hours to resolve?
Again, we can only speculate. First, it may have been that the Flowspec rule and the significant load that large number of BGP updates imposed on their routers made it difficult for them to login to their own interfaces. Several of the other tier-1 providers took action, it appears at CenturyLink/Level(3)’s request, to de-peer their networks. This would have limited the number of BGP announcements being received by the CenturyLink/Level(3) network and helped give it time to catch up.
Second, it also may have been that the Flowspec rule was not issued by CenturyLink/Level(3) themselves but rather by one of their customers. Many network providers will allow Flowspec peering. This can be a powerful tool for downstream customers wishing to block attack traffic, but can make it much more difficult to track down an offending Flowspec rule when something goes wrong.
Finally, it never helps when these issues occur early on a Sunday morning. Networks the size and scale of CenturyLink/Level(3)’s are extremely complicated. Incidents happen. We appreciate their team keeping us informed with what was going on throughout the incident. #hugops
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
