All posts by Matthew Prince

Cloudflare’s Annual Founders’ Letter

Post Syndicated from Matthew Prince original

Cloudflare’s Annual Founders’ Letter

Cloudflare’s Annual Founders’ Letter

This week we celebrate Cloudflare’s birthday. We launched the company 11 years ago tomorrow: September 27, 2010. It has been our tradition, since our first birthday, to use this week to launch innovative new products that we think of as our gift back to the Internet.

Since going public, it’s also been an opportunity for us to update our Annual Founders’ Letter and share what’s on our mind. Recently we’ve been thinking about three things: team, the Internet, and innovation.


When anyone asks us the key to Cloudflare’s success, we always say the same thing: the team we’ve been able to attract to help us achieve our mission of helping build a better Internet. In the last year we’ve had more than 250,000 people apply to work for us and extended offers to less than one half of one percent of them. We continue to attract great people.

It’s incredible to realize that more than half of Cloudflare’s team today started since March 13, 2020, when we closed all our physical offices due to the pandemic. In the last several months, as we’ve started to see a light at the end of the COVID tunnel, we’ve been hosting what we called Summer Socials with our team. Getting together outside, often over a picnic lunch, it’s been fun to meet face-to-face people we’d only video conferenced with before. And even more fun to watch people from across the team get to know each other outside the confines of a Brady Bunch-like on-screen box.

Cloudflare’s Annual Founders’ Letter

As a company that was very much a work-from-office culture before the pandemic, we were terrified of what would happen to our culture when we switched to fully remote work. Eighteen months into this forced experiment on a new way of working we’re happy to report: it’s working. Really well.

It turns out what we all suspected is in fact true. Culture has little to do with fun offices, plentiful snacks, or adjustable desks. Instead, for us, it starts with hiring people who are relentlessly curious and, at the same time, empathetic. Curious people want to learn. Empathetic people love to teach. And if you put a group of them together, whether in a swanky office or on Zoom, great things will happen.

As we come out the other side of COVID, we have an opportunity to help build a better way to work. It would be naive to insist that we go back to the way we did things before. We’ve been more productive, and on average our team has been happier in their jobs, than any time in the company’s history. At the same time, we know there can be considerable value in coming together in person to solve hard problems, brainstorm about the future, and build relationships that make the company stronger.

We don’t have all the answers on what the future of work looks like, but we’ve begun to formulate a place to start our experiments as people come back. We hope we can use the times we get together as ways to better collaborate and learn. But, at the same time, give our team the flexibility to work how and wherever they are the most productive.

The Internet

Cloudflare’s mission is to help build a better Internet. We always capitalize the I in Internet, in spite of what the AP style guide has said since 2016, because it’s a proper noun, we believe there is and only should be one, and we have an enduring respect for what a miracle it is that it exists.

Right around the same time that the AP started to say that you needn’t capitalize the I in Internet anymore, something seemed to change. The world shifted from seeing the Internet and what it enabled as an irreproachable good to a source of great danger.

We’ve watched the same thing. Since 2016 it’s often felt like a connection to the Internet only brings cyberattacks, toxic social media, threats to democracy, increasing polarization, and a declining disdainful discourse.

We have real challenges ahead as some of the technologies that ride on top of the Internet have broken down traditional gatekeepers without sufficient concern for addressing the harms they previously protected against. But, at the same time, the Internet itself remains a miracle.

A mere 11 years before Cloudflare’s founding, long distance phone calls still cost a fortune, sharing a photograph with someone in another country took weeks, and the idea that you could access the sum total of human knowledge from a device in your pocket was beyond even the fantasies of science fiction.

Cloudflare’s Annual Founders’ Letter

The last 18 months of the pandemic have reaffirmed our faith in the miracle that is the Internet. Imagine just how much worse it would have been had the pandemic happened just 11 years ago, let alone 22. The Internet allowed many of us to continue to work, connect with our loved ones, exercise our creativity, and stay connected to the world.

We’re proud of what we’ve done to live up to our mission and help build a better Internet during this time. And, as we come out the other side, we will continue to engage with policy makers to address the new harms an interconnected world has brought while preserving the miracle that is the Internet itself.


The Internet may seem static, but it is not. 11 years ago, watching a video online was an exercise in frustration. Today, it seems almost automatic that you can push play on your TV and access nearly any movie ever made instantly. That’s possible because the Internet isn’t static; it gets better through innovation.

Cloudflare’s Annual Founders’ Letter

At Cloudflare, we’re optimized to catalyze exactly that innovation. It starts with our mission: to help build a better Internet. The word “help” is important, because we know we can’t do it alone. So, wherever we can, we work with others across the Internet ecosystem to push it forward and make it better.

Sometimes people outside the company are surprised by the products we build. In fact, predicting our roadmap is pretty easy. We look at all the steps that are required to load a web page, send an email, stream a video, login to a workstation, or anything else you do online and ask: can we make that more secure, more reliable, or faster?

What’s exciting is that the pace at which the Internet is getting better is accelerating. And, in turn, the pace at which we are able to launch innovative new products is accelerating along with it. As the Internet grows and acquires more capabilities, we believe we will continue to grow with it. An investment in Cloudflare is, fundamentally, we feel an investment in the Internet itself.

Cloudflare’s Annual Founders’ Letter

And so, this week, we have an incredible series of announcements that are designed to help build a better Internet. We’re entering a new area to close one of the last network security risks that we haven’t historically protected our customers from, driving down costs of core cloud services, pushing the boundary of our network to our customers’ doorsteps, and investing in new technologies that may someday disrupt the web as we know it today.

Thank you to our team, our customers, and our investors. Happy 11th birthday to Cloudflare. And, even as we pick up steam, we continue to believe: we’re just getting started.

Cloudflare’s Annual Founders’ Letter

From AMP to Signed Exchanges, Or How Innovation Happens at Cloudflare

Post Syndicated from Matthew Prince original

From AMP to Signed Exchanges, Or How Innovation Happens at Cloudflare

From AMP to Signed Exchanges, Or How Innovation Happens at Cloudflare

This is the story of how we decided to work with Google to build Signed Exchanges support at Cloudflare. But, more generally, it’s also a story of how Cloudflare thinks about building disruptive new products and how we’ve built an organization designed around continuous innovation and long-term thinking.

A Threat to the Open Web?

The story starts with me pretty freaked out. In May 2015, Facebook had announced a new format for the web called Instant Articles. The format allowed publishers to package up their pages and serve them directly from Facebook’s infrastructure. This was a threat to Google, so the company responded in October with Accelerated Mobile Pages (AMP). The idea was generally the same as Facebook’s but using Google’s infrastructure.

As a general Internet user, if these initiatives were successful they were pretty scary. The end game was that the entirety of the web would effectively be slurped into Facebook and Google’s infrastructure.

But as the cofounder and CEO of Cloudflare, this presented an even more immediate risk. If everyone moved their infrastructure to Facebook and Google, there wasn’t much left for us to do. Our mission is to help build a better Internet, but we’ve always assumed there would be an Internet. If Facebook and Google were successful, there was real risk there would just be Facebook and Google.

That said, the rationale behind these initiatives was compelling. While they ended with giving Facebook and Google much more control, they started by trying to solve a real problem. The web was designed with the assumption that the devices connecting to it would be on a fixed, wired connection. As more of the web moved to being accessed over wireless, battery-powered, relatively low-power devices, many of the assumptions of the web were holding back its performance.

This is particularly true in the developing world. While a failed connection can happen anywhere, the further you get from where content is hosted, the more likely it is to happen. Facebook and Google both reasoned that if they could package up the web and serve complete copies of pages from their infrastructure, which spanned the developing world, they could significantly increase the usability of the web in areas where there was still an opportunity for Internet usage to grow. Again, this is a laudable goal. But, if successful, the results would have been dreadful for the Internet as we know it.

Seeds of Disruption

So that’s why I was freaked out. In our management meetings at Cloudflare I’d walk through how this was a risk to the Internet and our business, and we needed to come up with a strategy to address it. Everyone on our team listened and agreed but ultimately and reasonably said: that’s in the future, and we have immediate priorities of things our customers need, so we’ll need to wait until next quarter to prioritize it.

That’s all correct, and probably the right decision if you are forced to make one, but it’s also how companies end up getting disrupted. So, in 2016, we decided to fund a small team led by Dane Knecht, Cloudflare’s founding product manager, to set up a sort of skunkworks team in Austin, TX. The idea was to give the team space away from headquarters, so it could work on strategic projects with a long payoff time horizon.

Today, Dane’s team is known as the Emerging Technologies & Incubation (ETI) team. It was where products like Cloudflare for Teams,, and Workers were first dreamed up and prototyped. And it remains critical to how Cloudflare continues to be so innovative. Austin, since 2016, has also grown from a small skunkworks outpost to what will, before the end of this year, be our largest office. That office now houses members from every Cloudflare team, not just ETI. But, in some ways, it all started with trying to figure out how we should respond to Instant Articles and AMP.

We met with both Facebook and Google. Facebook’s view of the world was entirely centered around their app, and didn’t leave much room for partners. Google, on the other hand, was born out of the open web and still ultimately wanted to foster it. While there has been a lot of criticism of AMP, much of which we discussed with them directly, it’s important to acknowledge that it started from a noble goal: to make the web faster and easier to use for those with limited Internet resources.

We built a number of products to extend the AMP ecosystem and make it more open. Viewed on their own, those products have not been successes. But they catalyzed a number of other innovations. For instance, building a third party AMP cache on Cloudflare required a more programmable network. That directly resulted in us prototyping a number of different serverless computing strategies and finally settling on Workers. In fact, many of the AMP products we built were the first products built using Workers.

Part of the magic of our ETI team is that they are constantly trying new things. They’re set up differently, in order to take lots of “shots on goal.” Some won’t work, in which case we want them to fail fast. And, even for those that don’t, we are always learning, collaborating, and innovating. That’s how you create a culture of innovation that produces products at the rate we do at Cloudflare.

Signed Exchanges: Helping Build a Better Internet

Importantly also, working with the AMP team at Google helped us better collaborate on ideas around Internet performance. Cloudflare’s mission is to “help build a better Internet.” It’s not to “build a better Internet.” The word “help” is essential and something I’ll always correct if I hear someone leave it out. The Internet is inherently a collection of networks, and also a collection of work from a number of people and organizations. Innovation doesn’t happen in a vacuum but is catalyzed by collaboration and open standards. Working with other great companies who are aligned with democratizing performance optimization technology and speeding up the Internet is how we believe we can make significant and meaningful leaps in terms of performance.

From AMP to Signed Exchanges, Or How Innovation Happens at Cloudflare

And that’s what Signed Exchanges have the opportunity to be. They take the best parts of AMP — in terms of allowing pages to be preloaded to render almost instantly — but give back control over the content to the individual publishers. They don’t require you to exclusively use Google’s infrastructure and are extensible well beyond just traffic originating from search results. And they make the web incredibly fast and more accessible even in those areas where Internet access is slow or expensive.

We’re proud of the part we played in bringing this new technology to the Internet. We’re excited to see how people use it to build faster services available more broadly. And the ETI team is back at work looking over the innovation horizon and continuously asking the question: what’s next?

From AMP to Signed Exchanges, Or How Innovation Happens at Cloudflare

The Future of Work at Cloudflare

Post Syndicated from Matthew Prince original

The Future of Work at Cloudflare

The Future of Work at Cloudflare

During Impact Week, we’ve shared how Cloudflare is providing tools for our customers to minimize their environmental impact as well as what we, as a company, are doing to help society at large. But some critical stakeholders we haven’t talked much about yet are Cloudflare’s more than 2,000 employees: who build our services, support and educate our customers, keep our finances in order, work through difficult policy issues, and empower us to accomplish everything we have.

Over the last year and a half, we’ve all challenged a lot of the assumptions about what it means to “work.” Prior to the start of the pandemic, Cloudflare was very much a work-from-office culture. And so when, on March 13, 2020, we closed all our offices and asked everyone to work from home, the two of us were extremely nervous.

And then something unexpected happened: a lot of things got better.

As a company, productivity increased — when measured by our success selling our products, our pace of shipping new products, and even things like the time it takes for our finance team to close our books.

Other day-to-day things got better, too. We noticed a marked increase in participation in meetings by women, team members from whom English wasn’t their first language, junior team members, and other traditionally underrepresented groups. It turns out, putting everyone in a Brady-bunch like box on a screen smooths out some of the other social cues that, when in-person, make some people less comfortable, willing, or able to fully participate.

Virtually More Inclusive

It’s not unreasonable to speculate that the increase in productivity was driven, in no small part, by the increase in overall participation by people who previously felt reluctant to do so. And this further aligned with job surveys that we conducted over the last year and a half which showed that while the things people wanted us to improve remained the same, overall satisfaction with jobs increased.

We also noticed that the diversity of the candidates that were applying to work for us increased as we allowed people to work remotely. We were now an option for people who did not live in, or could not move to, the cities we had offices in. At Cloudflare, we’ve always believed in having a diverse team. Not to look good in a government report, but because it’s the right business strategy: more diverse teams win.

We all have different perspectives formed by our experiences that inherently give us insights and blind spots. If everyone on a team has the same insights and blind spots then there will be less unique and creative solutions proposed to whatever problems we face. Just as it’s important to have genetic diversity in a species, having diversity on every dimension in hiring makes us a stronger, more creative company. Prioritizing a diverse team is the right strategy if you’re optimizing for innovation, like we are at Cloudflare.

But not everything got better when we switched to remote; some things definitely got worse. We’re social creatures. We thrive through human interaction that is still difficult to replicate virtually. Even with improvements in video conferencing, online interactions still mute some of the social cues and make misunderstanding more likely. The osmosis for our team of learning by watching others is harder, especially for team members early in their career. And, unfortunately, for some the office is a refuge from difficult situations at home and so not having it as a place to get away can amplify those challenges.

What We’ve Learned… So Far

So we’ve been thinking a lot about what the future of work looks like at Cloudflare and wanted to share publicly what we’ve been talking about for some time internally. Here are some things we think we know.

First, we don’t know what the long term future of work will be like and so we’ve been hesitant to lay down broad proclamations. Instead, we expect that as we get past the pandemic and are able to work in-person safely again, we will do what Cloudflare has always done: run a number of experiments ourselves, watch what our peers are doing, and figure out what works for us. The one thing we feel pretty sure of is that wherever we start the experiment is highly unlikely to be exactly the place where we end up. The future of work won’t be set in stone sometime in the coming months, but evolve over the coming years.

Second, no matter what, the future of work will be more flexible. There’s no way we are putting the genie of remote work back in the bottle. Why would we want to if we’ve learned that we’ve been more productive and more satisfied with their jobs while we’ve been remote? Flexibility is the number one requested work benefit, and one of the silver linings of the pandemic for us has been that we ran a forced experiment that proved we could make it work.

Third, we are incredibly reluctant to impose arbitrary rules. Requiring team members to come in every Monday, Tuesday, and Thursday begs the question: why those days? Saying you need to come in if you’re below a certain seniority level also seems weirdly arbitrary. Instead of rules, we’re much more likely to start with general standards outlining what success as a member of the team at Cloudflare looks like and giving guidelines. We may need rules at some point, but we want to develop those rules over time based on what we learn.

Fourth, just opening offices and hoping for the best doesn’t work. What we’ve seen ourselves, and confirmed with others, is that what makes working from an office great is getting to work side-by-side with your colleagues. But if Alyssa comes in on Monday, and Blake comes in on Tuesday, and Carlos comes in on Wednesday, and Deeksha comes in on Thursday, and Ellen comes in on Friday, and they all hoped that they would get to connect, then none of them has a good experience and none of them come in the following week. If in-person work is going to work, there needs to be some deliberate structure and planning.

Fifth, we believe more in carrots than sticks. We’d rather we create an environment where people want to come in than where they have to come in. Based on our internal surveys, about 10% of our team wants to come in every day. We want to make the environment such that 100% of our team wants to come in at least some days.

Sixth, a more flexible way of working will require a more flexible physical space. The base “lego brick” we used to design all our offices pre-pandemic was the 6-person conference room. And, while none of our offices started this way, they all evolved into a sea of white, adjustable desks in neat rows as we found spots for our growing team. That already feels anachronistic. We think we need to redesign spaces to accommodate teams coming together to collaborate as well as individuals looking for a quiet spot for heads-down work.

Seventh, mixed meetings suck. When some people are in-person and some people are virtual the experience is bad for everyone. Part of why we think the last year and a half has worked is because everyone is in the same boat. We believe part of the reason why hybrid work environments have traditionally not worked is because they, left to their own devices, will tend to devolve to an experience that’s bad for everyone. The future of flexible work needs to acknowledge that most hybrid work experiments in the past haven’t worked.

Eighth, we’re a very global company. We have team members in countries around the world and need to operate our business around the clock. One of the benefits of being fully remote over the last year and a half is that it made all our offices feel like they were on equal footing. That’s something we believe is important for us to maintain.

So what’s our plan? Again, we don’t pretend to have all the answers. Instead, we expect that we’ll start somewhere and experiment. So we’re starting by being more flexible about where we hire people. We still believe that people will tend to cluster in hubs around cities where we have physical offices, but we are now open to hiring for nearly all of our roles in any location where we have a legal entity setup that allows us to hire.

We are tearing apart our offices in San Francisco and London to remake them into flexible work spaces. We’re designing them to allow for teams of 10, 20, or 30 employees to get together and collaborate. We’re also creating “Zoom villages” with one-person spaces and high quality AV equipment to let people jump on conference calls.

One of the few rules that we plan on starting with is that in meetings if any person is remote then everyone in the meeting is remote. We know that will create some awkward situations where some of our team will literally be sitting next to each other at desks talking on a video conference call. But we believe this is a rule worth having, in spite of our hesitation to impose strict rules, to help keep the playing field level for all our colleagues, wherever they’re working.

We’re going to rethink the purpose of the offices as spaces where teams can come together to collaborate. Internally, we’re calling these “on-site off-sites” — though everyone agrees we need a better name. The idea being that teams can call an in-person meeting and reserve space in any of our offices to come together. We expect different teams will set different cadences of these meetings, but expect most people to have at least some time in an office at least once a quarter.

The Future of Work at Cloudflare

We’re planning for what we’ve termed a “Czar of Serendipity” who will coordinate cross-group lunches and other activities to help facilitate teams who may not work directly together to have the opportunity if they want to meet colleagues they may not otherwise know. They’ll also help arrange in-person speakers and other activities aligned with whatever teams or groups are physically in the office each week.

And we’re hunting for carrots to encourage our team, and especially members who are earlier in their career, to come in. One we’re working on is what we’re calling Orange Card. We hope to turn every team member’s ID into a charge card. The card will only activate after someone badges in for the day and will only work to purchase food at restaurants that are within a 10-minute walk from the office with pre-tax dollars.

The Future of Work at Cloudflare

It’s in Cloudflare’s interest to encourage people to come in physically to work. Across the industry, however, we think jobs that require in-person work will look increasingly anachronistic. We also believe that, rather than operating private cafeterias inside our own spaces, it’s important for us to support local businesses near our offices — especially as so many of them were hit hard during COVID. If with Orange Card we can do this and find a way to let employees pay for lunches when they’re in the office at an effective discount, then it will check both boxes: giving employees a reason to come in and also supporting the local community.

We don’t know how many of these things will work, but it’s a sense of the experiments we intend to run as we try and find the future of work that works for our team.

In many ways we were fortunate that Cloudflare’s product could be of specific help during an incredibly difficult time for the world. The superheros of the last year and a half have been the medical professionals and scientists who have taken care of the sick and looked for cures for this disease. But the Internet has been the faithful sidekick that has helped many continue to work, stay connected with loved ones, and keep ourselves entertained through this trying time. As one of the defenders of the Internet, our work at Cloudflare has been incredibly rewarding. We hope we can create a future of work that remains incredibly rewarding even long past the pandemic.

The thoughts above are just a starting place. We expect that we’re going to learn a lot not only from our own experiments, but also from what we learn works (and doesn’t work) at peer companies. We would have never tried this experiment in remote work but for the pandemic. Now, having realized that we can continue to execute in a more flexible work environment, we don’t plan to forget the lessons we learned. We’re hopeful that we, along with our peer companies, will continue to run experiments and, over time, develop a new future of work that is more flexible, more inclusive, and more productive.

PS – We’re hiring.

Helping build a green Internet

Post Syndicated from Matthew Prince original

Helping build a green Internet

Helping build a green Internet

When we started Cloudflare, we weren’t thinking about minimizing the environmental impact of the Internet. Frankly, I didn’t really think of the Internet as having much of an environmental impact. It was just this magical resource that gave access to information and services from anywhere.

But that was before I started racking servers in hyper-cooled data centers. Before Cloudflare started paying the bills to keep those servers powered up and cooled down. Before we became obsessed with maximizing the number of requests we could process per watt of power. And long before we started buying directly from renewable power suppliers to drive down the cost of electricity across our network.

Today, I have a very good understanding of how much power it takes to run the Internet. It therefore wasn’t surprising to read the Boston Consulting Group study which found that 2% of all carbon output, about 1 billion metric tons per year, is attributable to the Internet. That’s the equivalent of the entire aviation industry.

Cloudflare: Accidentally Environmentally Friendly By Design

While we didn’t set out to reduce the environmental impact of the Internet, Cloudflare has always had efficiency at its core. It comes from our ongoing fight with an old nemesis: the speed of light.

Because we knew we couldn’t beat the speed of light, in order to make our network fast we needed to get close to where Internet users were. In order to do that, we needed to partner directly with ISPs around the world so they’d allow us to install our gear directly inside their networks. In order to do that, we needed to make our gear as low power as possible. And we needed to invent network technology to spread load around our network to deal with spikes of traffic — whether because of a cyber attack or a sale on an exclusive new sneaker line — and to efficiently use all available capacity.

Fighting for Efficiency

When back in December 2012, just two years after we launched, I traveled to Intel’s Oregon Research Center to talk to their senior engineering team about how we needed server chips with more cores per watt, I wasn’t thinking we needed it to save the environment. Instead, I was trying to figure out how we could build equipment that was power efficient enough that ISPs wouldn’t object to installing it. Unfortunately, Intel told me that I was worrying about the wrong thing. So that’s when we started looking for alternatives, including the very power-efficient Arm.

But, it turns out, our obsession with efficiency has made Cloudflare the environmental choice in cloud computing. A 2015 study by Anders S. G. Andrae and Tomas Edler estimated the average cost of processing a byte of information online. Even accounting for the efficiency gains across the industry, based on the study’s data our best estimates are that Cloudflare data processing is more than 19 times more efficient.

Serve Local

The imperfect analogy that I like is buying from the local farmers’ market versus the big box retailer. By serving requests locally, and not backhauling them around the world to massive data centers, Cloudflare is able to reduce the environmental impact of our customers on the Internet. In 2020, we estimate that our customers reduced their carbon output by 550,000 metric tons versus if they had not used our services. That’s the equivalent of eliminating 635 million miles driven by passenger cars last year.

Helping build a green Internet

We’re proud of that, but it’s still a tiny percentage of the overall impact the Internet still has on the environment. As we thought about Impact Week, we set out to make reducing the environmental impact of the Internet a top priority. Given today more than 1 in 6 websites uses Cloudflare, we’re in a position where changes we make can have a meaningful impact.

We Can Do More

Starting today, we’re announcing four major initiatives to reduce Cloudflare’s environmental impact and help the Internet as a whole be more environmentally friendly.

First, we’re committing to be carbon neutral by 2022. We already extensively use renewable energy to power our global network, but we’re going to expand that usage to cover 100% of our energy use. But we’re going a step further. We’re going to look back over the 11 years since Cloudflare launched and purchase offsets to zero out all of Cloudflare’s historical carbon output from powering our global network. It’s not enough that we have less impact than others, we want to make sure Cloudflare since our beginning has been a net positive for the planet.

Second, we are ramping up our deployment of a new class of hyper-efficient servers. Based on Arm technology, these servers can perform the same amount of work while using half the energy. We are hopeful that by prioritizing energy efficiency in the server market we can help catalyze more chip manufacturers to release more efficient designs.

Third, we’re releasing a new option for Cloudflare Workers and Pages, our computing platform and JAMStack offering, which allows developers to choose to run their workloads in the most energy efficient data centers. We believe we are the first major cloud computing vendor to offer developers a way to optimize for the environment. The Green Workers option won’t cost anymore. The tradeoff will be that workloads may incur a bit of additional network latency, but we believe for many developers that’s a tradeoff they’ll be willing to make.

New Standards and Partnerships to Eliminate Excessive Emissions

Finally, and maybe most ambitiously, we’re working with a number of the leading search and crawl companies to introduce an open standard to minimize the amount of load from excessive crawl as possible. Nearly half of all Internet traffic is automated. The majority of that is malicious, and Cloudflare is designed to stop that as efficiently as possible.

But more than 5% of all Internet traffic is generated by legitimate crawlers which index the web in order to power services we all rely on like search. The problem is, more than half of that legitimate crawl traffic is redundant — reindexing pages that haven’t changed. If we can eliminate redundant crawl, it’d be the equivalent of planting a new 30 million acres of forest. That’s a goal worth striving for.

When we started Cloudflare we weren’t thinking about how we could reduce the Internet’s environmental impact. But that’s changed. Cloudflare’s mission is to help build a better Internet. And a better Internet is clearly a more environmentally friendly Internet.

Welcome to Cloudflare Impact Week

Post Syndicated from Matthew Prince original

Welcome to Cloudflare Impact Week

Welcome to Cloudflare Impact Week

If I’m completely honest, Cloudflare didn’t start out as a mission-driven company. When Lee, Michelle, and I first started thinking about starting a company in 2009 we saw an opportunity as the world was shifting from on-premise hardware and software to services in the cloud. It seemed inevitable to us that the same shift would come to security, performance, and reliability services. And, getting ahead of that trend, we could build a great business.

Welcome to Cloudflare Impact Week
Matthew Prince, Michelle Zatlyn, and Lee Holloway, Cloudflare’s cofounders, in 2009.

One problem we had was that we knew in order to have a great business we needed to win large organizations with big IT budgets as customers. And, in order to do that, we needed to have the data to build a service that would keep them safe. But we only could get data on security threats once we had customers. So we had a chicken and egg problem.

Our solution was to provide a basic version of Cloudflare’s services for free. We reasoned that individual developers and small businesses would sign up for the free service. We’d learn a lot about security threats and performance and reliability opportunities based on their traffic data. And, from that, we would build a service we could sell to large businesses.

And, generally, Cloudflare’s business model made sense. We found that, for the most part, small companies got a low volume of cyber attacks, and so we could charge them a relatively small amount. Large businesses faced more attacks, so we could charge them more.

But what surprised us, and we only discovered because we were providing a free version of our service, was that there was a certain set of small organizations with very limited resources that received very large attacks. Servicing them was what made Cloudflare the mission-driven company we are today.

The Committee to Protect Journalists

If you ever want to be depressed, sign up for the newsletter of the Committee to Protect Journalists (CPJ). They’re the organization that, when a journalist is kidnapped or killed anywhere in the world, negotiates their release or, far too often, recovers their body.

I’d met the director of the organization at an event in early 2012. Not long after, he called me and asked if I wanted to meet three Cloudflare customers who were in town. I didn’t, I have to confess, but Michelle pushed me to take the meeting.

On a rainy San Francisco afternoon the director of CPJ brought three African journalists to our office. All three of them hugged me. One was from Ethiopia, another was from Angola, and the third they wouldn’t tell us his name or where he was from because he was “currently being hunted by death squads.”

For the next 90 minutes, I listened to stories of how the journalists were covering corruption in their home countries, how their work put them constantly in harm’s way, how powerful forces worked to silence them, how cyberattacks had been a constant struggle, and how, today, they depended on Cloudflare’s free service to keep their work online. That last bit hit me like a ton of bricks.

After our meeting finished, and we saw the journalists out, with Cloudflare T-shirts and other swag in hand, I turned to Michelle and said, “Whoa. What have we gotten ourselves into?”

Becoming Mission Driven

I’ve thought about that meeting often since. It was the moment I realized that Cloudflare had a mission beyond just being a good business. The Internet was a critically important resource for those three journalists and many others like them. At the same time, forces that sought to limit their work would use cyberattacks to shut them down. While we hadn’t set out to ensure everyone had world-class cybersecurity, regardless of their ability to pay, now it seemed critically important.

With that realization, Cloudflare’s mission came naturally: we aim to help build a better Internet. One where you don’t need to be a giant company to be fast and reliable. And where even a journalist, working on their own against daunting odds, can be secure online.

This is why we’ve prioritized projects that give back to the Internet. We launched Project Galileo, which provides our enterprise-grade services to organizations performing politically or artistically important work. We launched the Athenian Project to help protect elections against cyber attacks. We launched Project Fair Shot to make sure the organizations distributing the COVID-19 vaccine had the technical resources they needed to do so equitably.

Welcome to Cloudflare Impact Week

And, even on the technical side, we work hard to make the Internet better even when there’s no clear economic benefit to us, or even when it’s against our economic benefit. We don’t monetize user data because it seems clear to us that a better Internet is a more private Internet. We enabled encryption for everyone even though, when we did it, it was the biggest differentiator between our free and paid plans and the number one reason people upgraded. But clearly a better Internet was an encrypted Internet, and it seemed silly that someone should have to pay extra for a little bit of math.

Our First Impact Week

This week we kick off Cloudflare’s first Impact Week. We originally conceived the idea of the week as a way to highlight some of the things we were doing as a company around our environmental, social, and governance (ESG) initiatives. But, as is the nature of innovation weeks at Cloudflare, as soon as we announced it internally our team started proposing new products and features to take some of our existing initiatives even further.

So, over the course of the week, in addition to talking about how we’ve built our network to consume less power we’ll also be demonstrating how we’re increasingly using hyper power-efficient Arm-based servers to achieve even higher levels of efficiency in order to lessen the environmental impact of running the Internet. We’ll launch a new Workers option for developers who want to be more environmentally conscious. And we’ll announce an initiative in partnership with other leading Internet companies that we hope, if broadly adopted, could cut down as much as 25% of global web traffic and the corresponding energy wasted to serve it.

We’ll also focus on how we can bring the Internet to more people. While broadband has been a revolution where it’s available, rural and underserved-urban communities around the world still suffer from slow Internet speeds and limited ISP choice. We can’t completely solve that problem (yet) but we’ll be announcing an initiative that will help with some critical aspects.

Finally, as Cloudflare becomes a larger part of the Internet, we’ll be announcing programs both to monitor the network’s health, affirm our commitments to human rights, and extend our protections of critical societal functions like protecting elections.

When I first was trying to convince Michelle that we should start a business together, I pitched her a bunch of ideas. Most of them involved finding a clever way to extract rents from some group or another, often for not much benefit to society at large. Sitting in an Ethiopian restaurant in Central Square, I remember so clearly her saying to me, “Matthew, those are all great business ideas. But they’re not for me. I want to do something where I can be proud of the work we’re doing and the positive impact we’ve made.”

That sentence made me go back to the drawing board. The next business idea I pitched to her turned out to be Cloudflare. Today, Cloudflare’s mission remains helping build a better Internet. And, as we kick off Impact Week, we are proud to continue to live that mission in everything we do.

AWS’s Egregious Egress

Post Syndicated from Matthew Prince original

AWS’s Egregious Egress

AWS’s Egregious Egress

When web hosting services first emerged in the mid-1990s, you paid for everything on a separate meter: bandwidth, storage, CPU, and memory. Over time, customers grew to hate the nickel-and-dime nature of these fees. The market evolved to a fixed-fee model. Then came Amazon Web Services.

AWS was a huge step forward in terms of flexibility and scalability, but a massive step backward in terms of pricing. Nowhere is that more apparent than with their data transfer (bandwidth) pricing. If you look at the (ironically named) AWS Simple Monthly Calculator you can calculate the price they charge for bandwidth for their typical customer. The price varies by region, which shouldn’t surprise you because the cost of transit is dramatically different in different parts of the world.

Charging for Stocks, Paying for Flows

AWS charges customers based on the amount of data delivered — 1 terabyte (TB) per month, for example. To visualize that, imagine data is water. AWS fills a bucket full of water and then charges you based on how much water is in the bucket. This is known as charging based on “stocks.”

On the other hand, AWS pays for bandwidth based on the capacity of their network. The base unit of wholesale bandwidth is priced as one Megabit per second per month (1 Mbps). Typically, a provider like AWS, will pay for bandwidth on a monthly fee based on the number of Mbps that their network uses at its peak capacity. So, extending the analogy, AWS doesn’t pay for the amount of water that ends up in their customers’ buckets, but rather the capacity based on the diameter of the “hose” that is used to fill them. This is known as paying for “flows.”

Translating Flows to Stocks

You can translate between flow and stock pricing by knowing that a 1 Mbps connection (think of it as the "hose") can transfer 0.3285 TB (328GB) if utilized to its fullest capacity over the course of a month (think of it as running the "hose" at full capacity to fill the "bucket" for a month).1 AWS obviously has more than 1 Mbps of capacity — they can certainly transfer more than 0.3285 TB per month — but you can use this as the base unit of their bandwidth costs, and compare it against what they charge a customer to deliver 1 Terabyte (1TB), in order to figure out the AWS bandwidth markup.

One more subtlety to be as accurate as possible. Wholesale bandwidth is also billed at the 95th percentile. That effectively cuts off the peak hour or so of use every day. That means a 1 Mbps connection running at 100% can actually likely transfer closer to 0.3458 TB (346GB) per month.

Two more factors are important: utilization and regional costs. AWS can’t run all their connections at 100% utilization 24×7 for a month. Instead, they’ll have some average utilization per transit connection in any month. It’s reasonable to estimate that they likely run at between 20% and 40% average utilization. That would be a typical average utilization range for the industry. The higher their utilization, the more efficient they are, the lower their costs, and the higher their effective customer markup will be.

To be conservative, we’ve assumed that AWS’s average utilization is the bottom of that range (20%), but you can download the raw data and adjust the assumptions however you think makes sense.

We have a good sense of the wholesale prices of bandwidth in different regions around the world based on what Cloudflare sees in the market when we buy bandwidth ourselves. We’d imagine AWS gets at least as good of pricing as we do. We’ve included a rough estimate of these prices in the calculation, rounding up on the wholesale price wherever there was a question (which makes AWS look better).

Massive Markups

Based on these assumptions, here’s our best estimate of AWS’s effective markup for egress bandwidth on a per-region basis.

AWS’s Egregious Egress
AWS’s Egregious Egress

Don’t rest easy, South Korea with your merely 357% markup. The general rule of thumb appears to be that the older a market is, the more Amazon wrings from its customers in egregious egress markups — and the Seoul availability zone is only a bit over four years old. Winter, unfortunately, inevitably seems to come to AWS customers.

AWS Stands Alone In Not Passing On Savings to Customers

Remember, this is for the transit bandwidth that AWS is paying for. For the bandwidth that they exchange with a network like Cloudflare, where they are directly connected (settlement-free peered) over a private network interface (PNI), there are no meaningful incremental costs and their effective margins are nearly infinite. Add in the effect of rebates Amazon collects from colocation providers who charge cross connect fees to customers, and the effective markup is likely even higher.

Some other cloud providers take into account that their costs are lower when passing over peering connections. Both Microsoft Azure and Google Cloud will substantially discount egress charges for their mutual Cloudflare customers. Members of the Bandwidth Alliance — Alibaba, Automattic, Backblaze, Cherry Servers, Dataspace, DNS Networks, DreamHost, HEFICED, Kingsoft Cloud, Liquid Web, Scalway, Tencent, Vapor, Vultr, Wasabi, and Zenlayer — waive bandwidth charges for mutual Cloudflare customers.

AWS’s Egregious Egress

At this point, the majority of hosting providers in the industry either substantially discount or entirely waive egress fees when sending traffic from their network to a peer like Cloudflare. AWS is the notable exception in the industry. It’s worth noting that we invited AWS to be a part of the Bandwidth Alliance, and they politely declined.

It seems like a no-brainer that if we’re not paying for the bandwidth costs, and the hosting provider isn’t paying for the bandwidth costs, customers shouldn’t be charged for the bandwidth costs at the same rate as if the traffic was being sent over the public Internet. Unfortunately, Amazon’s supposed obsession over doing the right thing for customers doesn’t extend to egress charges.

Artificially Held High

Amazon’s mission statement is: “We strive to offer our customers the lowest possible prices, the best available selection, and the utmost convenience.” And yet, when it comes to egress, their prices are far from the lowest possible.

During the last ten years, industry wholesale transit prices have fallen an average of 23% annually. Compounded over that time, wholesale bandwidth is 93% less expensive than 10 years ago. However, AWS’s egress fees over that same period have fallen by only 25%.

And, since 2018, the egress fees AWS charges in North America and Europe have not dropped a penny even as wholesale prices in those markets over the same time period have fallen by more than half.

AWS’s Hotel California Pricing

Another oddity of AWS’s pricing is that they charge for data transferred out of their network but not for data transferred into their network. If the only time you’ve paid for bandwidth is with your residential Internet connection, then this may make some sense. Because of some technical limitations of the cable network, download bandwidth is typically higher than upload bandwidth on cable modem connections. But that’s not how wholesale bandwidth is bought or sold.

AWS’s Egregious Egress

Wholesale bandwidth isn’t like your home cable connection. Instead, it’s symmetrical. That means that if you purchase a 1 Mbps (1 Megabit per second) connection, then you have the capacity to send 1 Megabit out and receive another 1 Megabit in every second. If you receive 1 Mbps in and simultaneously 1 Mbps out, you pay the same price as if you receive 1 Mbps in and 0 Mbps out or 0 Mbps in and 1 Mbps out. In other words, ingress (data sent to AWS) doesn’t cost them any more or less than egress (data sent from AWS). And yet, they charge customers more to take data out than put it in. It’s a head scratcher.

We’ve tried to be charitable in trying to understand why AWS would charge this way. Disappointingly, there just doesn’t seem to be an innocent explanation. As we dug in, even things like writes versus reads and the wear they put on storage media, as well as the challenges of capacity planning for storage capacity, suggest that AWS should charge less for egress than ingress.

But they don’t.

The only rationale we can reasonably come up with for AWS’s egress pricing: locking customers into their cloud, and making it prohibitively expensive to get customer data back out. So much for being customer-first.

But… But… But…

AWS may object that this doesn’t take into account the cost of things like metro dark fiber between data centers, amortized optical and other networking equipment, and cross connects. In our experience, those costs amount to a rounding error of less than one cent per Mbps when operating at AWS-like scale. And these prices have been falling at a similar rate to the decline in the price of bandwidth over the past 10 years. Yet AWS’s egress prices have barely budged.

All the data above is derived from what’s published on AWS’s simple pricing calculator. There’s no doubt that some large customers are able to negotiate lower prices. But these are the prices charged to small businesses and startups by default. And, when we’ve reviewed pricing even with large AWS customers, the egress fees remain egregious.

It’s Not Too Late!

We have a lot of mutual customers who use Cloudflare and AWS. They’re a great service, and we want to support our mutual customers and provide services in a way that meets their needs and is always as secure, fast, reliable, and efficient as possible. We remain hopeful that AWS will do the right thing, lower their egress fees, join the Bandwidth Alliance — following the lead of the majority of the rest of the hosting industry — and pass along savings from peering with Cloudflare and other networks to all their customers.

AWS’s Egregious Egress

1Here’s the calculation to convert a 1 Mbps flow into TB stocks: 1 Mbps @ 100% for 1 month = (1 million bits per second) * (60 seconds / minute) * (60 minutes / hour) * (730 hours on average/month) divided by (eight bits / byte) divided by 10^12 (to convert bytes to Terabytes) = 0.3285 TB/month.

Introducing Project Fair Shot: Ensuring COVID-19 Vaccine Registration Sites Can Keep Up With Demand

Post Syndicated from Matthew Prince original

Introducing Project Fair Shot: Ensuring COVID-19 Vaccine Registration Sites Can Keep Up With Demand

Introducing Project Fair Shot: Ensuring COVID-19 Vaccine Registration Sites Can Keep Up With Demand

Around the world government and medical organizations are struggling with one of the most difficult logistics challenges in history: equitably and efficiently distributing the COVID-19 vaccine. There are challenges around communicating who is eligible to be vaccinated, registering those who are eligible for appointments, ensuring they show up for their appointments, transporting the vaccine under the required handling conditions, ensuring that there are trained personnel to administer the vaccine, and then doing it all over again as most of the vaccines require two doses.

Cloudflare can’t help with most of that problem, but there is one key part that we realized we could help facilitate: ensuring that registration websites don’t crash under load when they first begin scheduling vaccine appointments. Project Fair Shot provides Cloudflare’s new Waiting Room service for free for any government, municipality, hospital, pharmacy, or other organization responsible for distributing COVID-19 vaccines. It is open to eligible organizations around the world and will remain free until at least July 1, 2021 or longer if there is still more demand for appointments for the vaccine than there is supply.

Crashing Registration Websites

The problem of vaccine scheduling registration websites crashing under load isn’t theoretical: it is happening over and over as organizations attempt to schedule the administration of the vaccine. This hit home at Cloudflare last weekend. The wife of one of our senior team members was trying to register her parents to receive the vaccine. They met all the criteria and the municipality where they lived was scheduled to open appointments at noon.

When the time came for the site to open, it immediately crashed. The cause wasn’t hackers or malicious activity. It was merely that so many people were trying to access the site at once. “Why doesn’t Cloudflare build a service that organizes a queue into an orderly fashion so these sites don’t get overwhelmed?” she asked her husband.

A Virtual Waiting Room

Turns out, we were already working on such a feature, but not for this use case. The problem of fairly distributing something where there is more demand than supply comes up with several of our clients. Whether selling tickets to a hot concert, the latest new sneaker, or access to popular national park hikes it is a difficult challenge to ensure that everyone eligible has a fair chance.

The solution is to open registration to acquire the scarce item ahead of the actual sale. Anyone who visits the site ahead of time can be put into a queue. The moment before the sale opens, the order of the queue can be randomly (and fairly) shuffled. People can then be let in in order of their new, random position in the queue — allowing only so many at any time as the backend of the site can handle.

At Cloudflare, we were building this functionality for our customers as a feature called Waiting Room. (You can learn more about the technical details of Waiting Room in this post by Brian Batraski who helped build it.) The technology is powerful because it can be used in front of any existing web registration site without needing any code changes or hardware installation. Simply deploy Cloudflare through a simple DNS change and then configure Waiting Room to ensure any transactional site, no matter how meagerly resourced, can keep up with demand.

Recognizing a Critical Need; Moving Up the Launch

We planned to release it in February. Then, when we saw vaccine sites crashing under load and frustration of people eligible for the vaccine building, we realized we needed to move the launch up and offer the service for free to organizations struggling to fairly distribute the vaccine. With that, Project Fair Shot was born.

Government, municipal, hospital, pharmacy, clinic, and any other organizations charged with scheduling appointments to distribute the vaccine can apply to participate in Project Fair Shot by visiting:

Giving Front Line Organizations the Technical Resources They Need

The service will be free for qualified organizations at least until July 1, 2021 or longer if there is still more demand for appointments for the vaccine than there is supply. We are not experts in medical cold storage and I get squeamish at the sight of needles, so we can’t help with many of the logistical challenges of distributing the vaccine. But, seeing how we could support this aspect, our team knew we needed to do all we could to help.

The superheroes of this crisis are the medical professionals who are taking care of the sick and the scientists who so quickly invented these miraculous vaccines. We’re proud of the supporting role Cloudflare has played helping ensure the Internet has continued to function well when the world needed it most. Project Fair Shot is one more way we are living up to our mission of helping build a better Internet.

Announcing Workplace Records for Cloudflare for Teams

Post Syndicated from Matthew Prince original

Announcing Workplace Records for Cloudflare for Teams

We wanted to close out Privacy & Compliance Week by talking about something universal and certain: taxes. Businesses worldwide pay employment taxes based on where their employees do work. For most businesses and in normal times, where employees do work has been relatively easy to determine: it’s where they come into the office. But 2020 has made everything more complicated, even taxes.

As businesses worldwide have shifted to remote work, employees have been working from “home” — wherever that may be. Some employees have taken this opportunity to venture further from where they usually are, sometimes crossing state and national borders.

Announcing Workplace Records for Cloudflare for Teams

In a lot of ways, it’s gone better than expected. We’re proud of helping provide technology solutions like Cloudflare for Teams that allow employees to work from anywhere and ensure they still have a fast, secure connection to their corporate resources. But increasingly we’ve been hearing from the heads of the finance, legal, and HR departments of our customers with a concern: “If I don’t know where my employees are, I have no idea where I need to pay taxes.”

Today we’re announcing the beta of a new feature for Cloudflare for Teams to help solve this problem: Workplace Records. Cloudflare for Teams uses Access and Gateway logs to provide the state and country from which employees are working. Workplace Records can be used to help finance, legal, and HR departments determine where payroll taxes are due and provide a record to defend those decisions.

Every location became a potential workplace

Before 2020, employees who frequently traveled could manage tax jurisdiction reporting by gathering plane tickets or keeping manual logs of where they spent time. It was tedious, for employees and our payroll team, but manageable.

The COVID pandemic transformed that chore into a significant challenge for our finance, legal, and HR teams. Our entire organization was suddenly forced to work remotely. If we couldn’t get comfortable that we knew where people were working, we worried we may be forced to impose somewhat draconian rules requiring employees to check-in. That didn’t seem very Cloudflare-y.

The challenge impacts individual team members as well. Reporting mistakes can lead to tax penalties for employees or amendments during filing season. Our legal team started to field questions from employees stuck in new regions because of travel restrictions. Our payroll team prepared for a backlog of amendments.

Announcing Workplace Records for Cloudflare for Teams

Logging jurisdiction without manual reporting

When team members open their corporate laptops and start a workday, they log in to Cloudflare Access — our Zero Trust tool that protects applications and data. Cloudflare Access checks their identity and other signals like multi-factor methods to determine if they can proceed. Importantly, the process also logs their region so we can enforce country-specific rules.

Our finance, legal, and HR teams worked with our engineering teams to use that model to create Workplace Records. We now have the confidence to know we can meet our payroll tax obligations without imposing onerous limitations on team members. We’re able to prepare and adjust, in real-time, while confidentially supporting our employees as they work remotely for wherever is most comfortable and productive for them.

Announcing Workplace Records for Cloudflare for Teams

Respecting team member privacy

Workplace Records only provides resolution within a taxable jurisdiction, not a specific address. The goal is to give only the information that finance, legal, and HR departments need to ensure they can meet their compliance obligations.

The system also generates these reports by capturing team member logins to work applications on corporate devices. We use the location of that login to determine “this was a workday from Texas”. If a corporate laptop is closed or stored away for the weekend, we aren’t capturing location logs. We’d rather team members enjoy time off without connecting.

Two clicks to enforce regional compliance

Workplace Records can also help ensure company policy compliance for a company’s teams. For instance, companies may have policies about engineering teams only creating intellectual property in countries in which transfer agreements are in place. Workplace Records can help ensure that engineering work isn’t being done in countries that may put the intellectual property at risk.

Announcing Workplace Records for Cloudflare for Teams

Administrators can build rules in Cloudflare Access to require that team members connect to internal or SaaS applications only from countries where they operate. Cloudflare’s network will check every request both for identity and the region from which they’re connecting.

We also heard from our own accounting teams that some regions enforce strict tax penalties when employees work without an incorporated office or entity. In the same way that you can require users to work only from certain countries, you can also block users from connecting to your applications from specific regions.

No deciphering required

When we started planning Workplace Records, our payroll team asked us to please not send raw data that added more work on them to triage and sort.

Available today, you can view the country of each login to internal systems on a per-user basis. You can export this data to an external SIEM and you can build rules that control access to systems by country.

Launching today in beta is a new UI that summarizes the working days spent in specific regions for each user. Workplace Records will add a company-wide report early in Q1. The service is available as a report for free to all Cloudflare for Teams customers.

Announcing Workplace Records for Cloudflare for Teams

Going forward, we plan to work with Human Capital Management (HCM), Human Resource Information Systems (HRIS), Human Resource Management Systems (HRMS), and Payroll providers to automatically integrate Workplace Records.

What’s next?

At Cloudflare, we know even after the pandemic we are going to be more tolerant of remote work than before. The more that we can allow our team to work remotely and ensure we are meeting our regulatory, compliance, and tax obligations, the more flexibility we will be able to provide.

Cloudflare for Teams with Workplace Records is helping solve a challenge for our finance, legal, and HR teams. Now with the launch of the beta, we hope we can help enable a more flexible and compliant work environment for all our Cloudflare for Teams customers.
This feature will be available to all Cloudflare for Teams subscribers early next week. You can start using Cloudflare for Teams today at no cost for up to 50 users, including the Workplace Records feature.

Announcing Workplace Records for Cloudflare for Teams

Welcome to Privacy & Compliance Week

Post Syndicated from Matthew Prince original

Welcome to Privacy & Compliance Week

Welcome to Privacy & Compliance Week

Tomorrow kicks off Cloudflare’s Privacy & Compliance Week. Over the course of the week, we’ll be announcing ways that our customers can use our service to ensure they are in compliance with an increasingly complicated set of rules and laws around the world.

Early in Cloudflare’s history, when Michelle, Lee, and I were talking about the business we wanted to build, we kept coming back to the word trust. We realized early on that if we were not trustworthy then no one would ever choose to route their Internet traffic through us. Above all else, we are in the trust business.

Every employee at Cloudflare goes through orientation. I teach one of the sessions titled “What Is Cloudflare?” I fill several white boards with notes and diagrams talking about where we fit in to the market. But I leave one for the end so I can write the word TRUST, in capital letters, and underline it three times. Trust is the foundation of our business.

Standing Up For Our Customers from Our Early Days

That’s why we’ve made decisions that other companies may not have. In January 2013 the FBI showed up at our door with a National Security Letter requesting information on a customer. It was incredibly scary.

We had fewer than 30 employees at the time. The agents, while professional, were incredibly intimidating. And the letter ordered us to turn over information and forbid us from discussing it with anyone other than our attorneys.

Welcome to Privacy & Compliance Week

There’s a proper role for law enforcement, but National Security Letters, which at the time had almost no oversight, could be written and enforced by a single branch of the US government, and gagged recipients from talking about them indefinitely, ran counter to the foundational principles of due process. So we decided to sue the United States government.

I am thankful for Cloudflare’s Board for encouraging us to always fight for our principles. I am also thankful for the Electronic Frontier Foundation, who served as our attorneys in the case. It took several years, and we were gagged from talking about it until 2017, but ultimately the FBI withdrew the letter and Congress has taken steps to reform the law and ensure better oversight. There is a proper role for law enforcement, but when it crosses a line and infringes on basic principles of due process, then we believe it’s important to challenge it.

It’s all about trust.

Recognizing It’s Not Our Data

The same is true for the commercial side of our business. As soon as Cloudflare took off, the ad tech companies came knocking: “Do you have any idea how much you could make if you just let us cookie and retarget individuals passing through your network?” I took a lot of those meetings in our early days, but always came away feeling uneasy. Talking through it with Michelle she concisely expressed why we would never be in the advertising business: “It’s not our data.”

And that’s right. For our customers who do run ads on their sites, if we sold the data then we’d effectively be undercutting them. And, more fundamentally, if we were some invisible service that tracked you online without your knowledge then that would fail the creepiness test. While we believe there can be good ad-supported businesses, Cloudflare will never be one.

Welcome to Privacy & Compliance Week

As a result, we’ve always seen any personally identifiable information that passes through our network as a toxic asset and purged it as quickly as possible. That can be a tension because we are a security company and part of security requires us to be able to know, for instance, if a particular IP address is sending DDoS traffic. But we’ve invested in implementing or inventing technologies — like Universal SSL, Privacy Pass, Encrypted DNS, and ESNI — that keep your private data private, including from us.

Again, it’s all about trust.

Privacy In Our DNA

While Cloudflare started in California, we have had a global perspective from our earliest days. Today, nearly half of our C-level executives are Europeans, including our CTO, CIO, and CFO. Michelle, my co-founder and Cloudflare’s COO, is Canadian, a country that shares many of Europe’s values around privacy. We have offices around the world and far more engineers working outside of Silicon Valley than inside of it.

I wrote the first version of our Privacy Policy back in 2010. It included from the first draft this clear statement: “Cloudflare will not sell, rent, or give away any of your personal information without your consent. It is our overriding privacy principle that any personal information you provide to us is just that: private.” That is still true today. While other tech companies have made their policies more flexible over time, we’ve made ours stricter, including committing to a list of things we have never done and will fight like hell to never do:

  • Cloudflare has never turned over our encryption or authentication keys or our customers’ encryption or authentication keys to anyone.
  • Cloudflare has never installed any law enforcement software or equipment anywhere on our network.
  • Cloudflare has never provided any law enforcement organization a feed of our customers’ content transiting our network.
  • Cloudflare has never modified customer content at the request of law enforcement or another third party.
  • Cloudflare has never modified the intended destination of DNS responses at the request of law enforcement or another third party.
  • Cloudflare has never weakened, compromised, or subverted any of its encryption at the request of law enforcement or another third party.
Welcome to Privacy & Compliance Week

While many tech companies struggled to comply with privacy regulations such as GDPR, at Cloudflare it was relatively easy because the principles it imposed were at our core from our very outset. We don’t have a business if we don’t have trust, and being transparent, principled, and respecting the sanctity of personal data is critical to us continuously earning that trust.

Improving the Privacy of Our Service

But we’re not done; we can do more. There are things that have irked me about our service for a long time. For instance, from our earliest days we’ve used the _cfduid cookie to help with some of our security functions. That has meant that if you used Cloudflare you couldn’t be completely cookieless. John Graham-Cumming and I challenged the team earlier this year to see if we could kill it. Our team rose to the challenge and this week we’re announcing its deprecation. To my mind, that announcement alone is worth an entire week of celebrations.

Welcome to Privacy & Compliance Week

We have multiple data centers around the world that aggregate and process data in order to display logs and provide features. While having geographic redundancy helps with availability, some customers want to make sure their data never leaves a particular region. This week we’ll be giving users a lot more control over what data is processed where.

And, like we have during Privacy and Encryption weeks in years past, we will continue to invest in technologies to enable better encryption and more private use of core Internet services like DNS. Wouldn’t it be cool if, for example, we could ensure that no DNS provider could ever see both who is using their service and also where on the Internet those users are going? Stay tuned!

Helping Customers With Increasingly Complex Compliance Challenges

While we continue to invest in ensuring Cloudflare leads the way on privacy, more and more of our customers are also looking for solutions to be more private themselves. This month we expect that the EU’s new Digital Services Act will be proposed. We expect that it will continue to raise the bar on how companies doing business in Europe have to handle customers’ data. While the Internet giants will have the resources to comply with these heightened requirements, for everyone else they will create new challenges.

To that end, this week we’re announcing the Cloudflare Data Localization Suite. It provides our customers with a powerful set of tools to ensure they have control over how and where their data is processed in order to help comply with increasingly complex local data processing requirements. This includes enhancements to Workers, our edge computing and storage platform, to help modern applications get built such that users’ data never leaves their own country or region.

Welcome to Privacy & Compliance Week

It’s clear to us that the model of sending all your customer data back to a data center in Ashburn, VA, regardless of where those customers are located in the world, will look as antiquated in an increasingly privacy-conscious world as carrying a stack of punch cards to a central mainframe would today. In the not too distant future, regulations are inevitably going to force data storage and processing to be local. And, with a network that today already spans more than 100 countries, Cloudflare stands ready to help our customers enable that more private future.

Stay Tuned

Stay tuned this week to our blog for a series of announcements. Since these are topics that are so important in Europe right now, we’ll be simultaneously publishing most of them in French, Italian, Spanish, Portuguese, and German as well as English. Also check out Cloudflare TV where we’ll be interviewing a series of people whose views on privacy and compliance we respect and have learned from.

Cloudflare’s mission is to help build a better Internet. And there is no doubt that a better Internet is a more private Internet. With that in mind, welcome to Privacy & Compliance Week.

The Cloudflare Radar 2020 Elections Dashboard

Post Syndicated from Matthew Prince original

The Cloudflare Radar 2020 Elections Dashboard

The Cloudflare Radar 2020 Elections Dashboard

There is significant global attention around the upcoming United States election. Through the Athenian Project and Cloudflare for Campaigns, Cloudflare is providing free protection from cyber attacks to a significant number of state and local elections’ websites, as well as those of federal campaigns.

One of the bedrocks of a democracy is that people need to be able to get access to relevant information to make a choice about the future of their country. This includes information about the candidates up for election; learning about how to register, and how to cast a vote; and obtaining accurate information on the results.

A question that I’ve been increasingly asked these past few months: are cyberattacks going to impact these resources leading up to and on election day?

Internally, we have been closely monitoring attacks on the broader elections and campaign websites and have a team standing by 24×7 to help our current customers as well as state and local governments and eligible political campaigns to protect them at no cost from any cyberattacks they may see.

The good news is that, so far, cyberattacks have not been impacting the websites of campaigns and elections officials we are monitoring and protecting. While we do see some background noise of attacks, they have not interfered in the process so far. The attack traffic is below what we saw in 2016 and below what is typical in elections we have observed in other countries.

But there are still nearly two weeks before election day so our guard is up. We thought it was important to provide a view into how overall traffic to campaign and elections sites is trending as well as a view into the cyberattacks we’re observing. To that end, today we’re sharing data from our internal monitoring systems publicly through Cloudflare Radar. You can access the special “Election 2020” Radar dashboard here:

The dashboard is updated continuously with information we’re tracking on traffic to elections-related sites, both legitimate and from cyberattacks. It is normal to see fluctuations in this traffic depending on the time of day as well as when there will be occasional cyberattacks. So far, nothing here surprises us.

It’s important to note that Cloudflare does not see everything. We do not, for instance, have any view into misinformation campaigns that may be on social media. We also do not protect every state and local government or every campaign.

That said, we have Athenian Project participants in more than half of US states — including so-called red states, blue states, purple states, and several of the battleground states. We also have hundreds of federal campaigns that are using us ranging across the political spectrum. While we may not see a targeted cyberattack, given the critical role the web now plays to the election process, we believe we would likely see any wide-spread attacks attempting to disrupt the US elections.

So far, we are not seeing anything that suggests such an attack has impacted the election to date.

Our team will continue to monitor the situation. If any state or local elections agency or campaigns comes under attack, we stand ready to help at no cost through the Athenian Project and Cloudflare for Campaigns.

We could not have built Cloudflare into the company it is today without a stable, functional government. In the United States, that process depends on democracy and fair elections not tainted by outside influence like cyberattacks. We believe it is our duty to provide our technology where we can to help ensure this election runs smoothly.

Introducing Cloudflare One

Post Syndicated from Matthew Prince original

Introducing Cloudflare One

Introducing Cloudflare One

Today we’re announcing Cloudflare One™. It is the culmination of engineering and technical development guided by conversations with thousands of customers about the future of the corporate network. It provides secure, fast, reliable, cost-effective network services, integrated with leading identity management and endpoint security providers.

Over the course of this week, we’ll be rolling out the components that enable Cloudflare One, including our WARP Gateway Clients for desktop and mobile, our Access for SaaS solution, our browser isolation product, and our next generation network firewall and intrusion detection system.

The old model of the corporate network has been made obsolete by mobile, SaaS, and the public cloud. The events of 2020 have only accelerated the need for a new model. Zero Trust networking is the future and we are proud to be enabling that future. Having worked on the components of what is Cloudflare One for the last two years, we’re excited to unveil today how they’ve come together into a robust SASE solution and share how customers are already using it to deliver the more secure and productive future of the corporate network.

What Is Cloudflare One? Secure, Optimized Global Networking

Cloudflare One is a comprehensive, cloud-based network-as-a-service solution that is designed to be secure, fast, reliable and define the future of the corporate network. It replaces a patchwork of appliances and WAN technologies with a single network that provides cloud-based security, performance, and control through one user interface.

Cloudflare One brings together how users connect, on ramps for branch offices, secure connectivity for applications, and controlled access to SaaS into a single platform.

Cloudflare One reflects the complex nature of corporate networking today: mobile and remote users, SaaS applications, a mix of applications hosted in private data centers and public cloud, as well the challenge of employees using the broader Internet securely from their corporate and personal devices.

Introducing Cloudflare One

Whether you call this SASE or simply the new reality, today’s enterprise needs flexibility at every layer of the network and application stack. Secure and authenticated access is needed for users wherever they are: at the office, on a mobile device or working from home. Corporate network architectures need to reflect the state of modern computing that requires secure, filtered Internet access to get to SaaS or public cloud, secure application connectivity to protect against hackers and DDoS, and fast, reliable branch and home office access.

And the new corporate network needs to be global. No matter where applications are hosted, or employees reside, connectivity needs to be secure and fast. With Cloudflare’s massive global presence, traffic is secured, routed, and filtered over an optimized backbone that uses real time Internet intelligence to protect against the latest threats and route traffic around bad Internet weather and outages.

However, you’re only as strong as your weakest link. It doesn’t matter how secure your network is if you allow the wrong people access, or your end user’s devices are compromised. That is why we’re incredibly excited to announce that Cloudflare One takes the power of Cloudflare’s network and combines it with best-of-breed identity management and device integrity to create a complete solution that encompasses the entire corporate network of today and tomorrow.

Partner ecosystem: Identity Management

Most organizations already have one or more identity management systems. Rather than requiring them to change, we are integrating with all the major providers. This week we’re announcing partnerships with Okta, Ping Identity, and OneLogin. We support nearly all the other leading identity providers including Microsoft Active Directory and Google Workspace, as well as broadly adopted consumer and developer identity platforms like Github, LinkedIn, and Facebook.

Introducing Cloudflare One

Powerfully, Cloudflare One does not require you to standardize on just one identity provider. We see multiple companies that may have one identity provider for full-time employees and another for contractors. Or one they chose themselves and another they inherited from an acquisition. Cloudflare One will integrate with one or more identity providers and allow you to then set consistent policies across all your applications.

The metaphor that makes sense to me is that the identity provider issues passports and Cloudflare One is the border agent that checks that they’re valid. At any particular moment, different passports from different providers may be allowed or forbidden to enter just by updating the instructions the border agent follows.

Partner ecosystem: Device Integrity

In addition to identity, device integrity and endpoint security are an important part of a zero trust solution. This week we’re announcing partnerships with CrowdStrike, VMware Carbon Black, SenitnelOne, and Tanium. These providers run on devices and ensure that they haven’t been compromised. Again, organizations can centralize around a single vendor for device integrity or can mix and match with Cloudflare One providing a consistent control plane.

Introducing Cloudflare One

Extending the border control analogy, it’s like having a temperature screening and COVID-19 test when you enter a country. Even if you have a valid passport, if you’re not healthy then you will be turned away. By partnering with the leading identity and device integrity providers, Cloudflare One provides a robust identity and access management solution that fully delivers on the promise of Zero Trust.

We’re thrilled to partner with these leading identity management and endpoint security companies to make Cloudflare One flexible and robust.

With this as an introduction to Cloudflare One, I wanted to provide some context on why the existing paradigm doesn’t work, what the future of the enterprise network looks like, and where we go from here. In order to understand the power of Cloudflare One, you first have to understand the way we used to build and secure corporate networks and how the transition to mobile, cloud, and remote work have all forced this fundamental change in the paradigm.

The Middle(box) Ages: How Corporate Security Used to Work

The Internet was designed to be a massive, decentralized network. Any computer could connect to that network and route data from one location to another. The model provided resiliency, but did not guarantee fast or available connections. The early Internet also lacked a framework for security.

Introducing Cloudflare One

As a result, enterprises did not trust the Internet as a platform for their businesses. To keep employees productive, network connections had to be fast and available. Those connections also had to be secure. So, businesses built their own shadow versions of the Internet:

  • Companies purchased dedicated, private connections between offices and across their data centers in the form of expensive MPLS links.
  • IT teams managed complex routing across offices, VPN hardware, and clients.
  • Security teams deployed physical firewall boxes and DDoS appliances to keep the private network safe.
  • When employees had to use the Internet, security teams backhauled traffic through a central location to filter outbound connections with yet more hardware: Internet gateways.

Legacy corporate security followed a castle and moat approach. You put all your sensitive applications and data in the castle, you required all your employees to come to work in the castle every day, and then you built a metaphorical moat around the castle using firewalls, DDoS appliances, gateways and more: an unmanageable mess of devices and vendors.

The Middle(box) Ages Are Long Gone

While smarter attackers finding ways to breach moats were always a concern for the castle and moat approach, ultimately they weren’t what caused the approach to fail. Instead the change came from transformation of the technical landscape. Smartphones made workers increasingly mobile, letting them venture outside the moat. SaaS and the public cloud moved data and corporate applications out of the metaphorical castle.

Introducing Cloudflare One

And, in 2020, COVID-19 changed everything by forcing everyone who could to work remotely. If the employees weren’t coming to work in the castle anymore, the whole paradigm completely breaks down. This transition was happening already, but this year poured gasoline on the already smoldering fire. Increasingly companies are realizing that the only way forward is to embrace the fact that employees, servers and applications are now “on the Internet” and not “in the castle.” This new paradigm is known as “Zero Trust.”

Google’s seminal paper, “BeyondCorp: A New Approach to Enterprise Security,” published in 2014, brought the idea of Zero Trust security into the mainstream. Google’s insight in 2014 was that you could solve the challenges of every employee and application being on the Internet by ensuring that every application would inherently distrust every connection. If there was zero trust inherent to what network you were on, then every user of every application would be continuously authenticated. Powerfully, that would simultaneously enhance security while enabling more use of cloud applications as well as mobile and remote work.

The Future LAN: A Secure WAN

What we realized talking to customers was that even the analyst and competitor framing of the future corporate network didn’t fully recognize some challenges that come with a Zero Trust model. One of the benefits of embracing a Zero Trust model is that it makes enabling branch and home offices easier and less expensive. Rather than having to lease expensive MPLS circuits to connect branch offices — something that is literally impossible as people work from home — you instead require every use of every application to be authenticated.

Introducing Cloudflare One

This lines up with something else we’ve heard from our customers over the last six months: “maybe the Internet is almost good enough.” Like physical offices, many MPLS or SD-WAN deployments are currently sitting idle. And yet, employees continue to be productive. If users could move to a model that runs on the Internet, and one that improves the Internet, teams can stop spending money on legacy routing. Rather than trying to build more private networks, the corporate network of the future leverages the Internet but with heightened security, performance, and reliability.

That sounds great, but it opens a whole new can of worms. Inherently to do this you need to expose more of your applications to the Internet. While they may be safe from unauthorized use if you’ve properly implemented Zero Trust, that opens them to many less sophisticated, but highly disruptive challenges.

At the end of 2019 we saw a disturbing new trend begin to emerge. DDoS attackers shifted their focus from embarrassing companies by knocking their websites offline to increasingly targeting internal applications and networks. Unfortunately, we’ve seen more of these attacks launched throughout the pandemic.

It’s not a coincidence. It’s the direct result of companies being forced to expose more of their internal applications to the Internet in order to support remote work. To our surprise, it has turned out that while we anticipated Access and Gateway being the natural pairing of products, equally often customers looking to move to a Zero Trust model are bundling Cloudflare’s DDoS and WAF products.

It makes sense. If you are exposing more of your applications to the Internet, then the problems that Internet-facing applications have had to deal with in the past now become the problems of your internal applications as well. It’s become clear to us that the future of a SASE or Zero Trust network needs to also include DDoS mitigation and WAF as well.

Making the Internet Secure and Reliable Enough for the Enterprise

We agree with the customers we’ve talked to who say that the Internet is almost good enough to replace a corporate network. We’ve been building products to fill in the gaps where it needs to be better. Virtual appliances in regional public cloud providers are not sufficient. Enterprises need a global, distributed network that accelerates traffic in any location.

Introducing Cloudflare One

We’ve spent the last decade building Cloudflare’s network; bringing the Internet closer to users around the world and supporting incredible scale. According to W3Techs, more than 14% of the web already relies on our network. We can also use that to constantly measure the Internet at scale and find faster routes. That scale allows us to deliver Cloudflare One to any organization, no matter where they are located or how global their workforce, and ensure their network and applications are secure, fast, and reliable.

Foreshadowing Cloudflare One

The same lessons we’ve learned handling traffic for the websites on our network can be applied to how enterprises connect to everything else. We started that journey last year when we launched Cloudflare WARP, a consumer product that routes all connections leaving a personal device through Cloudflare’s network, where we can encrypt and accelerate it. This week, we’ll show how the WARP Client is now one of the on-ramps to get employee traffic onto Cloudflare One.

Introducing Cloudflare One

We launched WARP on mobile devices because we knew they would prove to be the most difficult to get right. Traditionally, VPN clients are clunky battery sucks designed for desktops and, if they have mobile versions at all, they’ve been clumsily ported over. We set out to build WARP to work great on mobile, not burning battery life or slowing connections down, because we knew if we could pull that off then it would be easy to port it to the less limited constraints of the desktop.

We also launched it for consumers first because they are the best QA team you could ever assemble. More than 10 million consumers have been putting WARP through its paces for the last year. We’ve seen edge cases from every corner of the Internet and used them to iron the bugs out. We knew that if we could make the WARP Client something that consumers loved to use then it would be a stark contrast to every other enterprise solution in the market.

Meanwhile, we built products to deliver the same improvements to data centers and offices. We announced Magic Transit last year to provide secure, performant, and reliable IP connectivity to the Internet. Earlier this year, we expanded that model when we launched Cloudflare Network Interconnect (CNI) to allow our customers to interconnect branch offices and data centers directly with Cloudflare.

Cloudflare Access starts by introducing identity into Cloudflare’s network. We apply filters based on identity and context to both inbound and outbound connections. Every login, request, and response proxies through Cloudflare’s network regardless of the location of the server or user.

Cloudflare Gateway keeps connections to the rest of the Internet safe. By routing all traffic through Cloudflare’s network first, customers can deprecrate on-premise firewalls eliminating Internet backhaul requirements that slow down users.

Introducing Cloudflare One

Pulling the Pieces Together

We think about the products in Cloudflare One in two categories:

  • On-ramps: the products that connect a user, device, or location to Cloudflare’s edge. WARP for endpoints, Magic Transit and CNI for networks, Argo Smart Routing to accelerate traffic.
  • Filters: the products that shield networks from attacks, inspect traffic for threats, and apply least privilege rules to data and applications. Access for Zero Trust rules, Gateway for traffic filtering, Magic Firewall for network filtering.

Most competitors in this space focus on one area, which loses out on the efficiencies of combining them in a single solution. Cloudflare One brings those together on our network. By integrating both sides of the challenge, we can give administrators a single place to manage and secure their network.

Introducing Cloudflare One

What Differentiates Cloudflare One

Easy to Deploy, Manage, and Use

We’ve always offered free and pay-as-you-go plans that teams of any size could sign up for with a credit card. Those customers lack the systems integrators or IT departments of large enterprises. To serve those teams, we had to build a control plane and dashboard that was accessible and easy to use.

The products in Cloudflare One follow that same approach; comprehensive enough for enterprises but easy to use to make these products accessible to any team. We’ve also extended that to end users; the client application that powers Gateway is built on what we learned creating Cloudflare WARP for consumer users.

Unified Solution

Cloudflare One puts the entire corporate network behind a single pane of glass. By integrating with leading identity providers and endpoint security solutions, Cloudflare One enables companies to enforce a consistent set of policies across all their applications. Since the network is the common denominator of all applications, by building control into the network Cloudflare One ensures consistent policies whether an application is new or legacy, run on-premise or in the cloud, and delivered from your own infrastructure or a multi-tenant SaaS provider.

Cloudflare One also helps rationalize complicated deployments. While it would be great if every app and every employee and contractor used the same identity provider, for example, that isn’t always possible. Acquisitions, skunkworks projects, and internal disagreements can cause multiple different solutions to be present inside one company. Cloudflare One allows you to plug different providers into one unified network control plane to ensure consistent policies.

Significant ROI

Our core tenet of serving the entire Internet has always forced us to obsess over costs. Efficiency is in the DNA of Cloudflare and we use our efficiency to pass along customer-friendly, fixed-rate pricing. Cloudflare One builds on that experience to deliver a platform that is more cost-effective than combining point solution vendors. The differences are especially apparent versus other providers who have tried to build on top of public cloud platforms and inherit their cost and inconsistent network performance.

To achieve the level of efficiency needed to compete with hardware appliances required us to invent a new type of platform. That platform needed to be built our own network where we could drive costs down and ensure the highest level of performance. It needed to be architected so any server in any city that made up Cloudflare’s network could run every one of our services. That means that Cloudflare One runs across Cloudflare’s global network spanning more than 200 cities worldwide. Even your farthest flung branch offices and remote workers are likely within milliseconds of servers powering Cloudflare One, ensuring our service works well wherever your team works.

Leverages Cloudflare’s Scale

Cloudflare already sits in front of a huge portion of the Internet. That allows us to see and respond to new security threats continuously. It also means that Cloudflare One customers’ traffic can be more efficiently routed, even when going to applications that would appear to be on the public Internet.

For instance, an employee behind Cloudflare One who is catching up on holiday shopping during their lunch break can have their traffic routed from a corporate branch office, across Cloudflare’s Magic Transit, over Cloudflare’s global backbone, across Cloudflare’s Network Interconnect, and to the ecommerce provider. Because Cloudflare handles the packets end-to-end, we can ensure they are encrypted, optimally routed, and efficiently delivered. As more of the Internet uses Cloudflare, the experience of surfing the Internet for Cloudflare One customers will grow even more exceptional.

What Does Cloudflare One Replace?

Instead of expensive MPLS links or complex SD-WAN deployments, Cloudflare One provides two on-ramps to your applications and the entire Internet: WARP and Magic Transit. WARP connects employees from any device, and any location, to Cloudflare’s network. Magic Transit allows broad deployments across whole offices or data centers.

Cloudflare Access replaces private-networks-as-security with Zero Trust controls. Later this week, we’ll announce how you can extend Access to any application, including SaaS applications.

Finally, Cloudflare One eliminates traditional network firewalls and web gateways. Cloudflare Gateway inspects traffic leaving any device in your organization to block threats on the Internet and prevent data from leaving. Magic Firewall will give your networks the same security, filtering traffic at the transport layer to replace the top-of-rack firewalls that block data exfiltration or attacks from unsecure network protocols.

Introducing Cloudflare One

What Comes Next?

Your team can start using Cloudflare One today. Add Zero Trust control to your applications with Cloudflare Access and secure DNS queries with Cloudflare Gateway. Keep networks safe from DDoS attacks with Magic Transit and connect your applications through Cloudflare with Argo Tunnel.

Over the course of the week, we’ll be launching new features and products to start to complete this vision. On Tuesday, we’ll extend the Zero Trust security of Cloudflare Access to all of your applications. Starting Wednesday, teams will be able to use Cloudflare WARP to proxy all employee traffic to Cloudflare where Gateway will now secure more than just DNS queries. You’ll be invited to sign up for Cloudflare’s browser isolation beta on Thursday and we’ll wrap the week with new APIs to control how Magic Transit secures your network.

It’s going to be a busy week, but we’re just getting started. Replacing a corporate network should not also mean you lose control over how that network operates. Magic WAN is our solution to complex SD-WAN deployments.

Security for that entire network should also work in both directions. Magic Firewall is our alternative to the clunky “next-generation firewall” appliances that secure outbound traffic. Data loss prevention (DLP) is another space that has lacked innovation and where we plan to extend Cloudflare One.

Introducing Cloudflare One

Finally, you should have visibility into that network. We’ll be launching new tools to detect and mitigate intrusion attempts that happen anywhere on your network, including unauthorized access to any SaaS applications you use. Now that we’ve built the on-ramps onto Cloudflare One, we’re excited to continue to innovate to provide more functionality and control to solve our customers biggest network security, performance, and reliability challenges.

Delivering the Network Customers Need Today

Over the last 10 years, Cloudflare has built one of the fastest, most reliable, most secure networks in the world. We’ve seen the power of using that network internally to enable our own teams to innovate quickly and securely. With the launch of Cloudflare One, we’re extending the power of Cloudflare’s network to meet the challenges of any company. The move to Zero Trust is a paradigm shift but the changes to how we work we believe has made it inevitable for every company. We’re proud of how we’ve been able to help some of Cloudflare One’s first customers reinvent their corporate networks. It makes sense to close with their own words.

Introducing Cloudflare One

“JetBlue Travel Products needed a way to give crew-members secure and simple access to internally-managed benefit apps. Cloudflare gave us all that and more — a much more efficient way to connect business partners and crew-members to critical internal tools.” — Vitaliy Faida, General Manager, Data/DevSecOps at JetBlue Travel Products.

Introducing Cloudflare One

“OneTrust relies on Cloudflare to maintain our network perimeter, so we can focus on delivering technology that helps our customers be more trusted. “With Cloudflare, we can easily build context-aware Zero Trust policies for secure access to our developer tools. Employees can connect to the tools they need so simply teams don’t even know Cloudflare is powering the backend. It just works.” — Blake Brannon, CTO of OneTrust.

Introducing Cloudflare One

“Discord is where the world builds relationships. Cloudflare helps us deliver on that mission, connecting our internal engineering team to the tools they need. With Cloudflare, we can rest easy knowing every request to our critical apps is evaluated for identity and context — a true Zero Trust approach.” — Mark Smith, Director of Infrastructure at Discord.

Introducing Cloudflare One

“When you’re a fast-growing, security-focused company like Area 1, anything that slows development down is the enemy. With Cloudflare, we’ve found a simpler, more secure way to connect our employees to the tools they need to keep us growing – and the experience is lightning-fast.” — Blake Darché, CSO at Area 1 Security.

Introducing Cloudflare One

“We launched quickly in April 2020 to bring remote learning to children throughout the UK during the coronavirus pandemic, Cloudflare Access made it fast and simple to authenticate a huge network of teachers and developers into our production sites and we set it up in literally less than an hour. Cloudflare’s WAF helped ensure the security and resilience of our public-facing website from day one.” — John Roberts, Technology Director at Oak National Academy.

Introducing Cloudflare One

“With Cloudflare, we’ve been able to reduce our dependence on VPNs and IP allow-listing for development environments. Our developers and testers aren’t required to login from specific locations, and we’ve been able to deploy an SSO solution to simplify the login process. Access is easier to manage than VPNs and other remote access solutions, which has removed pressure from our IT teams. They can focus on internal projects instead of spending time managing remote access.” — Alexandre Papadopoulos, Director of Cyber Security, INSEAD.

A letter from Cloudflare’s founders (2020)

Post Syndicated from Matthew Prince original

A letter from Cloudflare’s founders (2020)

To our stakeholders:

Cloudflare launched on September 27, 2010 — 10 years ago today. Stopping to look back over the last 10 years is challenging in some ways because so much of who we are has changed radically. A decade ago when we launched we had a few thousand websites using us, our tiny office was above a nail salon in Palo Alto, our team could be counted on less than two hands, and our data center locations on one hand.

A letter from Cloudflare’s founders (2020)
Outside our first office in Palo Alto in 2010. Photo by Ray Rothrock.

As the company grew, it would have been easy to stick with accelerating and protecting developers and small business websites and not see the broader picture. But, as this year has shown with crystal clarity, we all depend on the Internet for many aspects of our lives: for access to public information and services, to getting work done, for staying in touch with friends and loved ones, and, increasingly, for educating our children, ordering groceries, learning the latest dance moves, and so many other things. The Internet underpins much of what we do every day, and Cloudflare’s mission to help build a better Internet seems more and more important every day.

Over time Cloudflare has gone from an idea on a piece of paper to one of the largest networks in the world that powers millions of customers. Because we made our network to be flexible and programmable, what we’ve been able to do with it has expanded over time as well. Today we secure the Internet end-to-end — from companies’ infrastructure to individuals seeking a faster, more secure, more private connection. Our programmable, global network is at the core of everything we have been able to achieve so far.

Updating Our Annual Founders’ Letter

This is also the approximate one-year anniversary of Cloudflare going public. At the time, we wrote our first founders’ letter to the potential investors. We thought it made sense on this day, which we think of as our birthday, to reflect on the last year, as well as the last 10 years, and start a tradition of updating our original founders’ letter on September 27th every year.

A letter from Cloudflare’s founders (2020)
Ringing the bell to go public on the NYSE on September 13, 2019.

It’s been quite a year for our business. Since our IPO, we’ve seen record expansion of new customers. That growth has come both from expanding our existing customers as well as winning new business from new customers.

The percentage of the Fortune 1,000 that pay for one or more of Cloudflare’s services rose from 10% when we went public to more than 16% today. Across the web as a whole, according to W3Techs’ data, over the last year Cloudflare has grown from 10.1% of the top 10 million websites using our services to 14.5% using them today. (Amazon CloudFront, in second place based on the number of websites they serve, grew from 0.8% to 0.9% over the same period.)

Every year to celebrate our birthday we’ve made it a tradition to launch products that surprise the market with new ways to expand how anyone can use our network. We think of them as gifts back to the Internet. Three years ago, for instance, we launched our edge computing platform called Workers. Today, just three years later, hundreds of thousands of developers are using Workers to build applications, many of which we believe would be impossible to build on any other platform.

This year we’re once again launching a series of products to extend Cloudflare’s capabilities and hopefully surprise and delight the Internet. One that we’re especially excited about brings a new data model to Workers, allowing even more sophisticated applications to be built on the platform.

A letter from Cloudflare’s founders (2020)

The Year of COVID

It is impossible to reflect on the last year and not see the impact of the COVID-19 pandemic on our business, our customers, our employees, as well our friends, colleagues, and loved ones in the greater community. It’s heartening to think that for more than half of Cloudflare’s life as a public company our team has worked remote.

2020 was meant to be an Olympic year, but COVID-19 stopped that, like much else, from happening. Eight years ago, when Cloudflare was just two, the creator of the World Wide Web, Tim Berners-Lee, sent a message from the opening ceremony of the 2012 Olympics. That message read “This is for everyone” and the idea that the Internet is for all of us continues to be a key part of Cloudflare’s ethos today.

When we started Cloudflare we wanted to democratize what we thought were technologies only available to the richest and most Internet-focused organizations. We saw an opportunity to make available to everyone — from individual developers to small businesses to large corporations — the sorts of speed, protection, and reliability that, at the time, only the likes of Google, Amazon, and Facebook could afford.

Giving Back to the Internet

Over 10 years we’ve consistently rolled out the latest technologies, typically ahead of the rest of the industry, to everyone. And in doing so we’ve attracted employees, individuals, developers, customers to our platform. The Internet is for everyone and we’ve shown that a business can be very successful when we aim to serve everyone — large and small.

Something Steve Jobs said back in 1988 still resonates: “If you want to make a revolution, you’ve got to raise the lowest common denominator in every single machine.” Although we aren’t selling machines, we think that’s right: democratizing features matters.

Just look at the scourge of DDoS attacks. Why should DDoS attack mitigation be expensive when it’s a plague on companies large and small? It shouldn’t, and we optimized our business to make it inexpensive for us and passed that on to our customers through Unmetered DDoS Mitigation — another feature we rolled out to celebrate our Birthday Week three years ago.

A letter from Cloudflare’s founders (2020)

In 2014, also during Birthday Week, we launched Universal SSL, making encryption — something that had been expensive and difficult — free for all Cloudflare customers. The week we launched it we doubled the size of the encrypted web. Let’s Encrypt followed shortly after and, together, we’ve brought encryption to more than 90% of the web and made the little padlock in your browser something everyone can afford and should expect.

A letter from Cloudflare’s founders (2020)
Percent of the web served over HTTPS as reported by Google.

Helping Customers During Their Time of Need

In January of this year, we rolled out Cloudflare for Teams. The product was designed to replace the legacy VPNs and firewalls that were increasingly anachronistic as work moved to the cloud. Little did we know how much COVID-19 would accelerate their obsolescence and make Cloudflare for Teams essential.

Both of us sat on call after call in mid-March with at first small, then increasingly mid-sized, and eventually large and even governmental organizations who reached out to us looking for a way to survive as their teams shifted to working from home and their legacy hardware couldn’t keep up. We made the decision to sacrifice short term profits in order to help businesses large and small get through this crisis by making Cloudflare for Teams free through September.

A letter from Cloudflare’s founders (2020)

As we said during our Q1 earnings call, the superheros of this crisis are the medical professionals and scientists who are taking care of the sick and looking for a cure to the disease. But the faithful sidekick throughout has been the Internet. And, as one of the guardians of the Internet, we’re proud of helping ensure it was fast, secure, and reliable around the world when it was needed most. We are proud of how Cloudflare’s products could help the businesses continue to get work done during this unprecedented time by leaning even more on the Internet.

Meeting the Challenges Ahead

Giving back to the Internet is core to who we are, and we do not shy away from a challenge. And there are many challenges ahead. In a little over a month, the United States will hold elections. After the 2016 elections we, along with the rest of the world, were concerned to see technology intended to bring people together instead be used to subvert the democratic process. We decided we needed to do something to help prevent that from happening again.

A letter from Cloudflare’s founders (2020)

Three and a half years ago, we launched the Athenian Project to provide free cybersecurity resources to any local, state, or federal officials helping administer elections in the United States. We couldn’t have built Cloudflare into the company it is today without a stable government as a foundational platform. And, when that foundation is challenged, we believe it is our duty to lend our resources to defend it.

Today, we’re helping secure election infrastructure in more than half of the states in the United States. And, over these last weeks before the election, our team is working around the clock to help ensure the process is fair and not disrupted by cyber attacks.

More challenges lie ahead and we won’t shy away from them. Well intentioned governments around the world are increasingly seeking to regulate the Internet to protect their citizens. While the aims are noble, the risk is creating a patchwork of laws that only the Internet giants can successfully navigate. We believe it is critical for us to engage in the conversations around these regulations and work to help ensure as operating online becomes more complex, we can continue to make the opportunities of the Internet created for us when we started Cloudflare available to future startups and entrepreneurs.

Fighting for the Internet

Over the last 10 years, it’s been sad to watch some of the optimism around technology seem to fade. The perception of technology companies shifted from their being able to do no wrong to, today, their being able to do no right. And, as we’ve watched the industry develop, we’ve sympathized with that shift. Too many tech companies have abused customer data, ignored rules, violated privacy, and not been good citizens to the communities in which they operate and serve.

But we continue to believe what we started Cloudflare believing 10 years ago: the Internet itself is a force for good worth fighting to defend. We need to keep striving to make the Internet itself better — always on, always fast, always secure, always private, and available to everyone.

It’s striking to think how much more disruptive the COVID-19 crisis could have been had it struck in 2010 not 2020. The difference today is a better Internet. We’re proud of the role we’ve played in helping build that better Internet.

And, ten years in, we’re just getting started.

A letter from Cloudflare’s founders (2020)

Analysis of Today’s CenturyLink/Level(3) Outage

Post Syndicated from Matthew Prince original

Analysis of Today's CenturyLink/Level(3) Outage

Today CenturyLink/Level(3), a major ISP and Internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare’s customers as well as a significant number of other services and providers across the Internet. While we’re waiting for a post mortem from CenturyLink/Level(3), I wanted to write up the timeline of what we saw, how Cloudflare’s systems routed around the problem, why some of our customers were still impacted in spite of our mitigations, and what  appears to be the likely root cause of the issue.

Increase In Errors

At 10:03 UTC our monitoring systems started to observe an increased number of errors reaching our customers’ origin servers. These show up as “522 Errors” and indicate that there is an issue connecting from Cloudflare’s network to wherever our customers’ applications are hosted.

Cloudflare is connected to CenturyLink/Level(3) among a large and diverse set of network providers. When we see an increase in errors from one network provider, our systems automatically attempt to reach customers’ applications across alternative providers. Given the number of providers we have access to, we are generally able to continue to route traffic even when one provider has an issue.

Analysis of Today's CenturyLink/Level(3) Outage
The diverse set of network providers Cloudflare connects to. Source:‌‌

Automatic Mitigations

In this case, beginning within seconds of the increase in 522 errors, our systems automatically rerouted traffic from CenturyLink/Level(3) to alternate network providers we connect to including Cogent, NTT, GTT, Telia, and Tata.

Our Network Operations Center was also alerted and our team began taking additional steps to mitigate any issues our automated systems weren’t automatically able to address beginning at 10:09 UTC. We were successful in keeping traffic flowing across our network for most customers and end users even with the loss of CenturyLink/Level(3) as one of our network providers.

Analysis of Today's CenturyLink/Level(3) Outage
Dashboard Cloudflare’s automated systems recognizing the damage to the Internet caused by the CenturyLink/Level(3) failure and automatically routing around it.

The graph below shows traffic between Cloudflare’s network and six major tier-1 networks that are among the network providers we connect to. The red portion shows CenturyLink/Level(3) traffic, which dropped to near-zero during the incident. You can also see how we automatically shifted traffic to other network providers during the incident to mitigate the impact and ensure traffic continued to flow.

Analysis of Today's CenturyLink/Level(3) Outage
Traffic across six major tier-1 networks that are among the network providers Cloudflare connects to. CenturyLink/Level(3) in red.

The following graph shows 522 errors (indicating our inability to reach customers’ applications) across our network during the time of the incident.

Analysis of Today's CenturyLink/Level(3) Outage

The sharp spike up at 10:03 UTC was the CenturyLink/Level(3) network failing. Our automated systems immediately kicked in to attempt to reroute and rebalance traffic across alternative network providers, causing the errors to drop in half immediately and then fall to approximately 25 percent of the peak as those paths were automatically optimized.

Between 10:03 UTC and 10:11 UTC our systems automatically disabled CenturyLink/Level(3) in the 48 cities where we’re connected to them and rerouted traffic across alternate network providers. Our systems take into account capacity on other providers before shifting out traffic in order to prevent cascading failures. This is why the failover, while automatic, isn’t instantaneous in all locations. Our team was able to apply additional manual mitigations to reduce the number of errors another 5 percent.

Why Did the Errors Not Drop to Zero?

Unfortunately, there were still an elevated number of errors indicating we were still unable to reach some customers. CenturyLink/Level(3) is among the largest network providers in the world. As a result, many hosting providers only have single-homed connectivity to the Internet through their network.

To use the old Internet as a “superhighway” analogy, that’s like only having a single offramp to a town. If the offramp is blocked, then there’s no way to reach the town. This was exacerbated in some cases because CenturyLink/Level(3)’s network was not honoring route withdrawals and continued to advertise routes to networks like Cloudflare’s even after they’d been withdrawn. In the case of customers whose only connectivity to the Internet is via CenturyLink/Level(3), or if CenturyLink/Leve(3) continued to announce bad routes after they’d been withdrawn, there was no way for us to reach their applications and they continued to see 522 errors until CenturyLink/Level(3) resolved their issue around 14:30 UTC.

The same was a problem on the other (“eyeball”) side of the network. Individuals need to have an onramp onto the Internet’s superhighway. An onramp to the Internet is essentially what your ISP provides. CenturyLink is one of the largest ISPs in the United States.

Analysis of Today's CenturyLink/Level(3) Outage

Because this outage appeared to take all of the CenturyLink/Level(3) network offline, individuals who are CenturyLink customers would not have been able to reach Cloudflare or any other Internet provider until the issue was resolved. Globally, we saw a 3.5% drop in global traffic during the outage, nearly all of which was due to a nearly complete outage of CenturyLink’s ISP service across the United States.

So What Likely Happened Here?

While we will not know exactly what happened until CenturyLink/Level(3) issue a post mortem, we can see clues from BGP announcements and how they propagated across the Internet during the outage. BGP is the Border Gateway Protocol. It is how routers on the Internet announce to each other what IPs sit behind them and therefore what traffic they should receive.

Starting at 10:04 UTC, there were a significant number of BGP updates. A BGP update is the signal a router makes to say that a route has changed or is no longer available. Under normal conditions, the Internet sees about 1.5MBs – 2MBs of BGP updates every 15 minutes. At the start of the incident, the number of BGP updates spiked to more than 26MBs of BGP updates per 15 minute period and stayed elevated throughout the incident.

Analysis of Today's CenturyLink/Level(3) Outage

These updates show the instability of BGP routes inside the CenturyLink/Level(3) backbone. The question is what would have caused this instability. The CenturyLink/Level(3) status update offers some hints and points at a flowspec update as the root cause.

Analysis of Today's CenturyLink/Level(3) Outage

What’s Flowspec?

In CenturyLink/Level(3)’s update they mention that a bad Flowspec rule caused the issue. So what is Flowspec? Flowspec is an extension to BGP, which allows firewall rules to be easily distributed across a network, or even between networks, using BGP. Flowspec is a powerful tool. It allows you to efficiently push rules across an entire network almost instantly. It is great when you are trying to quickly respond to something like an attack, but it can be dangerous if you make a mistake.

At Cloudflare, early in our history, we used to use Flowspec ourselves to push out firewall rules in order to, for instance, mitigate large network-layer DDoS attacks. We suffered our own Flowspec-induced outage more than 7 years ago. We no longer use Flowspec ourselves, but it remains a common protocol for pushing out network firewall rules.

We can only speculate what happened at CenturyLink/Level(3), but one plausible scenario is that they issued a Flowspec command to try to block an attack or other abuse directed at their network. The status report indicates that the Flowspec rule prevented BGP itself from being announced. We have no way of knowing what that Flowspec rule was, but here’s one in Juniper’s format that would have blocked all BGP communications across their network.

   match {
      protocol tcp;
      destination-port 179;
 then discard;

Why So Many Updates?

A mystery remains, however, why global BGP updates stayed elevated throughout the incident. If the rule blocked BGP then you would expect to see an increase in BGP announcements initially and then they would fall back to normal.

One possible explanation is that the offending Flowspec rule came near the end of a long list of BGP updates. If that were the case, what may have happened is that every router in CenturyLink/Level(3)’s network would receive the Flowspec rule. They would then block BGP. That would cause them to stop receiving the rule. They would start back up again, working their way through all the BGP rules until they got to the offending Flowspec rule again. BGP would be dropped again. The Flowspec rule would no longer be received. And the loop would continue, over and over.

One challenge of this is that on every cycle, the queue of BGP updates would continue to increase within CenturyLink/Level(3)’s network. This may have gotten to a point where the memory and CPU of their routers was overloaded, causing an additional set of challenges to getting their network back online.

Why Did It Take So Long to Fix?

This was a significant global Internet outage and, undoubtedly, the CenturyLink/Level(3) team received immediate alerts. They are a very sophisticated network operator with a world class Network Operations Center (NOC). So why did it take more than four hours to resolve?

Again, we can only speculate. First, it may have been that the Flowspec rule and the significant load that large number of BGP updates imposed on their routers made it difficult for them to login to their own interfaces. Several of the other tier-1 providers took action, it appears at CenturyLink/Level(3)’s request, to de-peer their networks. This would have limited the number of BGP announcements being received by the CenturyLink/Level(3) network and helped give it time to catch up.

Second, it also may have been that the Flowspec rule was not issued by CenturyLink/Level(3) themselves but rather by one of their customers. Many network providers will allow Flowspec peering. This can be a powerful tool for downstream customers wishing to block attack traffic, but can make it much more difficult to track down an offending Flowspec rule when something goes wrong.

Finally, it never helps when these issues occur early on a Sunday morning. Networks the size and scale of CenturyLink/Level(3)’s are extremely complicated. Incidents happen. We appreciate their team keeping us informed with what was going on throughout the incident. #hugops

The Edge Computing Opportunity: It’s Not What You Think

Post Syndicated from Matthew Prince original

The Edge Computing Opportunity: It’s Not What You Think

The Edge Computing Opportunity: It’s Not What You Think

Cloudflare Workers® is one of the largest, most widely used edge computing platforms. We announced Cloudflare Workers nearly three years ago and it’s been generally available for the last two years. Over that time, we’ve seen hundreds of thousands of developers write tens of millions of lines of code that now run across Cloudflare’s network.

Just last quarter, 20,000 developers deployed for the first time a new application using Cloudflare Workers. More than 10% of all requests flowing through our network today use Cloudflare Workers. And, among our largest customers, approximately 20% are adopting Cloudflare Workers as part of their deployments. It’s been incredible to watch the platform grow.

Over the course of the coming week, which we’re calling Serverless Week, we’re going to be announcing a series of enhancements to the Cloudflare Workers platform to allow you to build much more complicated applications, lower your serverless computing bills, make your applications even faster, and prove that the Workers platform is secure to its core.

Matthew’s Hierarchy of Developers’ Needs

Before the week begins, I wanted to step back and talk a bit about what we’ve learned about edge computing over the course of the last three years. When we launched Cloudflare Workers we thought the killer feature was speed. Workers run across the Cloudflare network, closer to end users, so they inherently have faster response times than legacy, centralized serverless platforms.

However, we’ve learned by watching developers use Cloudflare Workers that there are a number of attributes to a development platform that are far more important than just speed. Speed is the icing on the cake, but it’s not, for most applications, an initial requirement. Focusing only on it is a mistake that will doom edge computing platforms to obscurity.

Today, almost everyone who talks about the benefits of edge computing still focuses on speed. So did Akamai, which launched their Java- and .NET-based EdgeComputing platform in 2002, only to shut it down in 2009 after failing to find enough customers where a bit less network latency alone justified the additional cost and complexity of running code at the edge. That’s a cautionary tale much of the industry has forgotten.

Today, I’m convinced that we were wrong when we launched Cloudflare Workers to think of speed as the killer feature of edge computing, and much of the rest of the industry’s focus remains largely misplaced and risks missing a much larger opportunity.

The Edge Computing Opportunity: It’s Not What You Think

I’d propose instead that what developers on any platform need, from least to most important, is actually: Speed < Consistency < Cost < Ease of Use < Compliance. Call it: Matthew’s Hierarchy of Developers’ Needs. While nearly everyone talking about edge computing has focused on speed, I’d argue that consistency, cost, ease of use, and especially compliance will ultimately be far more important. In fact, I predict the real killer feature of edge computing over the next three years will have to do with the relatively unsexy but foundationally important: regulatory compliance.

Speed As the Killer Feature?

Don’t get me wrong, speed is great. Making an application fast is the self-actualization of a developer’s experience. And we built Workers to be extremely fast. By moving computing workloads closer to where an application’s users are we can, effectively, overcome the limitations imposed by the speed of light. Cloudflare’s network spans more than 200 cities in more than 100 countries globally. We continue to build that network out to be a few milliseconds from every human on earth.

The Edge Computing Opportunity: It’s Not What You Think

Since we’re unlikely to make the speed of light any faster, the ability for any developer to write code and have it run across our entire network means we will always have a performance advantage over legacy, centralized computing solutions — even those that run in the “cloud.” If you have to pick an “availability zone” for where to run your application, you’re always going to be at a performance disadvantage to an application built on a platform like Workers that runs everywhere Cloudflare’s network extends.

We believe Cloudflare Workers is already the fastest serverless platform and we’ll continue to build out our network to ensure it remains so.

Speed Alone Is Niche

But let’s be real a second. Only a limited set of applications are sensitive to network latency of a few hundred milliseconds. That’s not to say under the model of a modern major serverless platform network latency doesn’t matter, it’s just that the applications that require that extra performance are niche.

Applications like credit card processing, ad delivery, gaming, and human-computer interactions can be very latency sensitive. Amazon’s Alexa and Google Home, for instance, are better than many of their competitors in part because they can take advantage of their corporate parents’ edge networks to handle voice processing and therefore have lower latency and feel more responsive.

But after applications like that, it gets pretty “hand wavy.” People who talk a lot about edge computing quickly start talking about IoT and driverless cars. Embarrassingly, when we first launched the Workers platform, I caught myself doing that all the time. Pro tip: when you’re talking to an edge computing evangelist, you can win Buzzword BINGO every time so long as you ensure you have “IoT” and “driverless cars” on your BINGO card.

The Edge Computing Opportunity: It’s Not What You Think

Donald Knuth, the famed Stanford Computer Science professor, (along with Tony Hoare, Edsgar Dijkstra, and many others) said something to the effect of “premature optimization is the root of all evil in programming.” It shouldn’t be surprising, then, that speed alone isn’t a compelling enough reason for most developers to choose to use an edge computing platform. Doing so for most applications is premature optimization, aka. the “root of all evil.” So what’s more important than speed?


While minimizing network latency is not enough to get most developers to move to a new platform, there is one source of latency that is endemic to nearly all serverless platforms: cold start time. A cold start is how long it takes to run an application the first time it executes on a particular server. Cold starts hurt because they make an application unpredictable and inconsistent. Sometimes a serverless application can be fast, if it’s hitting a server where the code is hot, but other times it’s slow when a container on a new server needs to be spun up and code loaded from disk into memory. Unpredictability really hurts user experience; turns out humans love consistency more than they love speed.

The problem of cold starts is not unique to edge computing platforms. Inconsistency from cold starts are the bane of all serverless platforms. They are the tax you pay for not having to maintain and deploy your own instances. But edge computing platforms can actually make the cold start problem worse because they spread the computing workload across more servers in more locations. As a result, it’s less likely that code will be “warm” on any particular server when a request arrives.

In other words, the more distributed a platform is, the more likely it is to have a cold start problem. And to work around that on most serverless platforms, developers have to create horrible hacks like performing idle requests to their own application from around the world so that their code stays hot. Adding insult to injury, the legacy cloud providers charge for those throw-away requests, or charge even more for their own hacky pre-warming/”reserved” solutions. It’s absurd!

Zero Nanosecond Cold Starts

We knew cold starts were important, so, from the beginning, we worked to ensure that cold starts with Workers were under 5 milliseconds. That compares extremely favorably to other serverless platforms like AWS Lambda where cold starts can take as long as 5 seconds (1,000x slower than Workers).

But we wanted to do better. So, this week, we’ll be announcing that Workers now supports zero nanosecond cold starts. Since, unless someone invents a time machine, it’s impossible to take less time than that, we’re confident that Workers now has the fastest cold starts of any serverless platform. This makes Cloudflare Workers the consistency king beating even the legacy, centralized serverless platforms.

The Edge Computing Opportunity: It’s Not What You Think

But, again, in Matthew’s Hierarchy of Developers’ Needs, while consistency is more important than speed, there are other factors that are even more important than consistency when choosing a computing platform.


If you have to choose between a platform that is fast or one that is cheap, all else being equal, most developers will choose cheap. Developers are only willing to start paying extra for speed when they see user experience being harmed to the point of costing them even more than what a speed upgrade would cost. Until then, cheap beats fast.

For the most part, edge computing platforms charge a premium for being faster. For instance, a request processed via AWS’s [email protected] costs approximately three times more than a request processed via AWS Lambda; and basic Lambda is already outrageously expensive. That may seem to make sense in some ways — we all assume we need to pay more to be faster — but it’s a pricing rationale that will always make edge computing a niche product servicing only those limited applications extremely sensitive to network latency.

The Edge Computing Opportunity: It’s Not What You Think

But edge computing doesn’t necessarily need to be more expensive. In fact, it can be cheaper. To understand, look at the cost of delivering services from the edge. If you’re well-peered with local ISPs, like Cloudflare’s network is, it can be less expensive to deliver bandwidth locally than it is to backhaul it around the world. There can be additional savings on the cost of power and colocation when running at the edge. Those are savings that we can use to help keep the price of the Cloudflare Workers platform low.

More Efficient Architecture Means Lower Costs

But the real cost win comes from a more efficient architecture. Back in the early-90s when I was a network administrator at my college, when we wanted to add a new application it meant ordering a new server. (We bought servers from Gateway; I thought their cardboard shipping boxes with the cow print were fun.) Then virtual machines (VMs) came along and you could run multiple applications on the same server. Effectively, the overhead per application went down because you needed fewer physical servers per application.

The Edge Computing Opportunity: It’s Not What You Think

VMs gave rise to the first public clouds. Quickly, however, cloud providers looked for ways to reduce their overhead further. Containers provided a lighter weight option to run multiple customers’ workloads on the same machine, with dotCloud, which went on to become Docker, leading the way and nearly everyone else eventually following. Again, the win with containers over VMs was reducing the overhead per application.

At Cloudflare, we knew history doesn’t stop, so as we started building Workers we asked ourselves: what comes after containers? The answer was isolates. Isolates are the sandboxing technology that your browser uses to keep processes separate. They are extremely fast and lightweight. It’s why, when you visit a website, your browser can take code it’s never seen before and execute it almost instantly.

By using isolates, rather than containers or virtual machines, we’re able to keep computation overhead much lower than traditional serverless platforms. That allows us to much more efficiently handle compute workloads. We, in turn, can pass the savings from that efficiency on to our customers. We aim not to be less expensive than [email protected], it’s to be less expensive than Lambda. Much less expensive.

From Limits to Limitless

Originally, we wanted Workers’ pricing to be very simple and cost effective. Instead of charging for requests, CPU time, and bandwidth, like other serverless providers, we just charged per request. Simple. The tradeoff was that we were forced to impose maximum CPU, memory, and application size restrictions. What we’ve seen over the last three years is developers want to build more complicated, sophisticated applications using Workers — some of which pushed the boundaries of these limits. So this week we’re taking the limits off.

Tomorrow we’ll announce a new Workers option that allows you to run much more complicated computer workloads following the same pricing model that other serverless providers use, but at much more compelling rates. We’ll continue to support our simplified option for users who can live within the previous limits. I’m especially excited to see how developers will be able to harness our technology to build new applications, all at a lower cost and better performance than other legacy, centralized serverless platforms.

Faster, more consistent, and cheaper are great, but even together those alone aren’t enough to win over most developers workloads. So what’s more important than cost?

Ease of Use

Developers are lazy. I know firsthand because when I need to write a program I still reach for a trusty language I know like Perl (don’t judge me) even if it’s slower and more costly. I am not alone.

That’s why with Cloudflare Workers we knew we needed to meet developers where they were already comfortable. That starts with supporting the languages that developers know and love. We’ve previously announced support for JavaScript, C, C++, Rust, Go, and even COBOL. This week we’ll be announcing support for Python, Scala, and Kotlin. We want to make sure you don’t have to learn a new language and a new platform to get the benefits of Cloudflare Workers. (I’m still pushing for Perl support.)

Ease also means spending less time on things like technical operations. That’s where serverless platforms have excelled. Being able to simply deploy code and allow the platform to scale up and down with load is magical. We’ve seen this with long-time users of Cloudflare Workers like Discord, which has experienced several thousand percent usage growth over the last three years and the Workers platform has automatically scaled to meet their needs.

The Edge Computing Opportunity: It’s Not What You Think

One challenge, however, of serverless platforms is debugging. Since, as a developer, it can be difficult to replicate the entire serverless platform locally, debugging your applications can be more difficult. This is compounded when deploying code to a platform takes as long as 5 minutes, as it can with AWS’s [email protected]. If you’re a developer, you know how painful waiting for your code to be deployed and testable can be. That’s why it was critical to us that code changes be deployed globally to our entire network across more than 200 cities in less than 15 seconds.

The Bezos Rule

One of the most important decisions we made internally was to implement what we call the Bezos Rule. It requires two things: 1) that new features Cloudflare engineers build for ourselves must be built using Workers if at all possible; and 2) that any APIs or tools we build for ourselves must be made available to third party Workers developers.

The Edge Computing Opportunity: It’s Not What You Think

Building a robust testing and debugging framework requires input from developers. Over the last three years, Cloudflare Workers’ development toolkit has matured significantly based on feedback from the hundreds of thousands of developers using our platform, including our own team who have used Workers to quickly build innovative new features like Cloudflare Access and Gateway. History has shown that the first, best customer of any platform needs to be the development team at the company building the platform.

Wrangler, the command-line tool to provision, deploy, and debug your Cloudflare Workers, has developed into a robust developer experience based on extensive feedback from our own team. In addition to being the fastest, most consistent, and most affordable, I’m excited that given the momentum behind Cloudflare Workers it is quickly becoming the easiest serverless platform to use.

Generally, whatever platform is the easiest to use wins. But there is one thing that trumps even ease of use, and that, I predict, will prove to be edge computing’s actual killer feature.


If you’re an individual developer, you may not think a lot about regulatory compliance. However, if you work as a developer at a big bank, or insurance company, or health care company, or any other company that touches sensitive data at meaningful scale, then you think about compliance a lot. You may want to use a particular platform because it’s fast, consistent, cheap, and easy to use, but if your CIO, CTO, CISO, or General Counsel says “no” then it’s back to the drawing board.

Most computing resources that run on cloud computing platforms, including serverless platforms, are created by developers who work at companies where compliance is a foundational requirement. And, up until to now, that’s meant ensuring that platforms follow government regulations like GDPR (European privacy guidelines) or have certifications providing that they follow industry regulations such as PCI DSS (required if you accept credit cards), FedRamp (US government procurement requirements), ISO27001 (security risk management), SOC 1/2/3 (Security, Confidentiality, and Availability controls), and many more.

The Coming Era of Data Sovereignty

But there’s a looming new risk of regulatory requirements that legacy cloud computing solutions are ill-equipped to satisfy. Increasingly, countries are pursuing regulations that ensure that their laws apply to their citizens’ personal data. One way to ensure you’re in compliance with these laws is to store and process  data of a country’s citizens entirely within the country’s borders.

The EU, India, and Brazil are all major markets that have or are currently considering regulations that assert legal sovereignty over their citizens’ personal data. China has already imposed data localization regulations on many types of data. Whether you think that regulations that appear to require local data storage and processing are a good idea or not — and I personally think they are bad policies that will stifle innovation — my sense is the momentum behind them is significant enough that they are, at this point, likely inevitable. And, once a few countries begin requiring data sovereignty, it will be hard to stop nearly every country from following suit.

The Edge Computing Opportunity: It’s Not What You Think

The risk is that such regulations could cost developers much of the efficiency gains serverless computing has achieved. If whole teams are required to coordinate between different cloud platforms in different jurisdictions to ensure compliance, it will be a nightmare.

Edge Computing to the Rescue

Herein lies the killer feature of edge computing. As governments impose new data sovereignty regulations, having a network that, with a single platform, spans every regulated geography will be critical for companies seeking to keep and process locally to comply with these new laws while remaining efficient.

While the regulations are just beginning to emerge, Cloudflare Workers already can run locally in more than 100 countries worldwide. That positions us to help developers meet data sovereignty requirements as they see fit. And we’ll continue to build tools that give developers options for satisfying their compliance obligations, without having to sacrifice the efficiencies the cloud has enabled.

The Edge Computing Opportunity: It’s Not What You Think

The ultimate promise of serverless has been to allow any developer to say “I don’t care where my code runs, just make it scale.” Increasingly, another promise will need to be “I do care where my code runs, and I need more control to satisfy my compliance department.” Cloudflare Workers allows you the best of both worlds, with instant scaling, locations that span more than 100 countries around the world, and the granularity to choose exactly what you need.

Serverless Week

The best part? We’re just getting started. Over the coming week, we’ll discuss our vision for serverless and show you how we’re building Cloudflare Workers into the fastest, most cost effective, secure, flexible, robust, easy to use serverless platform. We’ll also highlight use cases from customers who are using Cloudflare Workers to build and scale applications in a way that was previously impossible. And we’ll outline enhancements we’ve made to the platform to make it even better for developers going forward.

We’ve truly come a long way over the last three years of building out this platform, and I can’t wait to see all the new applications developers build with Cloudflare Workers. You can get started for free right now by visiting:

The Edge Computing Opportunity: It’s Not What You Think

Ladies and Gentlemen… Cloudflare TV!

Post Syndicated from Matthew Prince original

Ladies and Gentlemen… Cloudflare TV!

Ladies and Gentlemen… Cloudflare TV!

I’m excited to announce the upcoming launch of Cloudflare TV. A 24×7 live television broadcast, streamed globally via the Cloudflare network. You can tune in to the pre-broadcast station and check out the upcoming schedule at:

I’m kicking off the first live broadcast starting at 12:00pm Pacific (1900 UTC) on Monday, June 8 with a conversation with Chris Young (add to calendar). Chris was most recently the CEO of McAfee and has had a career defining the cyber security industry, from his own startup Cyveillance in the 1990s, to leadership positions at AOL, RSA, VMWare, Cisco, and Intel. I hope you’ll tune in and then stay tuned for all the content our team has in store.

Ladies and Gentlemen… Cloudflare TV!

Which leaves the question: why on earth is Cloudflare launching a 24×7 television station?

The Uniting Power of Television and Tech Conferences

I was born in the 70’s, am a child of the 80’s, and got started in my career in the 90’s. In the background, throughout much of it, was linear television we watched together. Over the last few months I’ve learned that Michelle Zatlyn, my co-founder and Cloudflare’s COO, and I shared a love of Children’s Television Network’s education program “3-2-1 Contact.” John Graham-Cumming, Cloudflare’s CTO, and I spent much of the late-90’s and early-00’s watching programs like “Call for Help” and “The Screensavers” on TechTV. Talking to many people across the Cloudflare team, we all shared common touchstones in our lives geeking out on nerdy programming about technical topics.

Ladies and Gentlemen… Cloudflare TV!

In some ways, a bit of that shared experience has been replaced by the modern tech conference. Tech conferences have four key features that keep people coming back in, I think, the following order of importance (from least to most): 1. Keynotes from interesting people; 2. Product and new feature demos; 3. Conversations with technical experts; and 4. Social interactions with peers.

RSA and CES Ain’t Workin… But That’s the Way You Do It?

Tech conferences have a lot of downsides too. Everyone complains about going to the Consumer Electronic Show or RSA because, in a lot of ways, spending a lot of money to be away from families, stay in difficult to book hotels, eat unhealthy food, and get herded around like cattle is awful. And yet, the value of the four things above is enough that we all continue to attend.

Or… we did. But, given the current pandemic, it feels like a long time before we’re going to be attending tech conferences again. So, at Cloudflare, we started to wonder, is there a way to replicate their best features (and not suffer their worst) in a COVID-19 world?

COVID Changes Everything

Cloudflare has offices in Asia, and we source the hardware for our equipment from the region, and so, since the virus hit those parts of the world first, we were aware of its impacts early. We adjusted our work and travel policies in our Asian offices early. Then we did something else: call around to companies in the impacted regions to ask them what they were doing that was still working even when everyone was working from home.

The answer that stood out from multiple companies was empowering more of their team to experiment with new ideas to reach customers. I remember a conversation with a cosmetics company that, prior to the conference, had used in-person events to sell most of their products. They were forced to invent new strategies as soon as the pandemic hit. They didn’t know what would work — no one did. So they empowered everyone on their sales team to run experiments. “Some of them proved so successful,” the head of marketing for this firm told me, “that I don’t think we’ll ever go back to in-person events even after this time of the virus has passed.”

So, as we shut down conferences and travel, and transitioned to remote work globally, I went to Jake Anderson, who runs Cloudflare’s marketing team, and asked him to pivot to empower everyone on our sales and marketing team to be able to run experiments. That’s the sort of thing that makes a ton of sense to a founder-CEO like me. It’s the sort of thing that strikes abject terror in the mind of any experienced marketing professional like Jake.

Jake took a deep breath and explained the danger of that strategy. Hundreds of different people on our team each running their own experiments may result in some good things, but also potentially a lot of chaos along the way. “Your last company was an anti-spam company, right?” he asked, rhetorically. “Do you really want the risk of everyone on the team thinking sending out mass emails is a good idea?” And now you understand why Jake runs our marketing team.

TV as the New Tech Conference

But he was a good sport and didn’t fully nix the idea. Instead we started brainstorming whether there could be a way to let our team run experiments — let them even be zany, crazy ideas — but do it in a way that had some structure and a framework and where any missteps could be contained. And that’s how the idea for Cloudflare TV was born.

If you read the history, it’s actually not that different from how MTV was born. It was an experiment. No one knew if the format would work. Early hosts were given a lot of leeway to try new things. And, out of it, many incredible things emerged. And, in the process, it brought a community and a generation together.

Introducing Cloudflare TV: A Platform to Experiment

Cloudflare is unlikely to morph into a television network. But I am excited to give our team a place to experiment and connect with the Cloudflare community, even while we’re locked down. And that community shares common interests in topics like web performance, Internet security, edge computing, and network reliability. And, with more than 2.8 million Cloudflare customers as part of our community, that’s more than 4 times MTV’s 2018 viewership. So who knows!

What can you expect? We’ll have some regular weekly programs. I plan on hosting a show featuring conversations with fellow entrepreneurs and business leaders I admire (add to calendar). Michelle Zatlyn, Cloudflare co-founder and COO, is doing a weekly series called “Yes We Can” highlighting women entrepreneurs and debunking the myth that there are no women in tech (add to calendar). John Graham-Cumming, our CTO, is doing a program called “This Week in Net” — looking at interesting trends we’re seeing from traffic patterns across Cloudflare’s network (add to calendar).

Ladies and Gentlemen… Cloudflare TV!

Nick Sullivan, who leads our research team, is planning a fireside chat format with heavyweights in computer science research in areas such as cryptography, artificial intelligence, databases, and more (add to calendar). Chris Scharff, on our Solutions Engineering team, will be our own Alex Trebek hosting weekly “Online Team Trivia” (add to calendar). Chaat Butsunturn on our sales team and Watson Ladd on our crypto team are hosting “Cooking with Cloudflare,” combining their favorite technical and edible recipes (add to calendar). And Dan Hollinger on our partnerships team hosting “Silicon Valley Squares,” a send up of the old game show “Hollywood Squares” (add to calendar).

We’ll also host some special programming. This week, Rita Kozlov, who is a Product Manager on our Workers team, is interviewing the people behind the COVID-19 response program Mask a Hero NY (add to calendar). Junade Ali, on our technical support operations team, is talking about the privacy-ensuring design of Pwned Passwords (add to calendar). And Bethany Sonefeld, on our Product Design team, is doing a program on dark patterns, bottomless feeds, and other manipulative software (add to calendar). And much, much more!

Highlighting Diverse Voices in Tech

We were originally scheduled to launch Cloudflare TV last week. Given the horrific violence targeting Black communities in the United States, we decided to delay the launch by a week. We’ve been inspired by the peaceful protests around the world, but we’re under no illusion that the systemic problems that inspired them have been fixed. We all have a lot of work to do.

What we have done over the course of the last week was add more content to Cloudflare TV highlighting the importance of diversity on our team. We’ve always believed that diverse teams with people who have different perspectives are more likely to find the best and most creative solutions to fulfill our mission of helping build a better Internet. I’m looking forward to sessions like those led by Cloudflare’s Black community, Afroflare, discussing their career paths and experiences (add to calendar), “Spotlight on Latino Excellence” interviewing Latino members of Cloudflare’s team (add to calendar), and “Everyone at the Table” looking at the topics of the day from a diverse set of perspectives (add to calendar).

We as a company, and the tech community overall, have a lot of work to do in order to bring in more diversity. I’m hopeful that Cloudflare TV can provide one forum to highlight the incredible professionals from communities that have been underrepresented in tech on our team and at other organizations we admire. I’m looking forward to kicking off that conversation with Chris Young on Monday (add to calendar).

Tune In, Geek Out

We’re aiming to make as much of the content interactive as possible. We’ll be interviewing existing customers and partners. We’re hosting as much of the content live so that the hosts of many programs will respond to questions from the audience. If you tune in, you’ll get to hear from the product managers and engineers that are building Cloudflare, ask them questions, and get responses live.

There will be lots of snafus. None of us have ever been television producers before. In MTV’s early days, they regularly failed to a blank screen. The production quality was low. And some experiments didn’t work. If we’re lucky, this won’t be any different. But, if we’re really lucky, hopefully some great things will also come out of it. I keep reminding our team that if we’re trying to follow in the footsteps of MTV — and its greatest success was “Jersey Shore” — then the bar is pretty low.

So hope you’ll tune in, geek out, feel part of our community, and learn more about Cloudflare and the people who are building it. And, if it works, maybe none of us will ever need to go to RSA again.

Check out the Cloudflare TV Guide:

Ladies and Gentlemen… Cloudflare TV!

Cyberattacks since the murder of George Floyd

Post Syndicated from Matthew Prince original

Cyberattacks since the murder of George Floyd

As we’ve often seen in the past, real world protest and violence is usually accompanied by attacks on the Internet. This past week has been no exception. The shocking murder of George Floyd on May 25 was followed, over the weekend of May 30/31, by widespread protests and violence in the US. At the same time, Cloudflare saw a large uptick in cyberattacks, particularly cyberattacks on advocacy organizations fighting racism.

This chart shows the number of cyberattack HTTP requests blocked by Cloudflare over the last week (blue line) compared to the corresponding week in April a month before (green line). Cloudflare’s scale means that we are blocking attacks in the many 10s of billions per day, but even with that scale it’s clear that during the last week there have been even more attacks than before. And those attacks grew over the weekend.

Cyberattacks since the murder of George Floyd

Digging in a little deeper we can compare the attacks over this past weekend with a corresponding weekend a month before. Over the weekend of April 25/26, Cloudflare blocked a total of 116,317,347,341 (a little over 116 billion cyberattack HTTP requests performing DDoS or trying to break into websites, apps or APIs were blocked).

Since 116,317,347,341 can be a little hard to comprehend, here’s another way of looking at it. 116,317,347,341 cyberattack HTTP requests over a two day period is more than 670,000 blocked requests per second. Google reportedly sees approximately 63,000 search queries per second, so the number of attacks we mitigated during this period was more than 10x Google’s entire search volume.

A month later, over the weekend of May 30/31, Cloudflare blocked 135,535,554,303 cyberattack HTTP requests. That represents a month-on-month increase of 17%: an extra 19,218,206,962 (19 billion) cyberattack HTTP requests were blocked (an extra 110,000 blocked requests per second).

Cyberattacks since the murder of George Floyd

Sunday, May 31 had the largest increase with 26% more cyberattacks than the same Sunday a month prior.

Digging into the categories of Internet properties that were attacked, we see a striking difference between the two weekends in April and May. The category with the biggest increase in cyberattacks was Advocacy Groups with a staggering increase of 1,120x.

Cyberattacks since the murder of George Floyd

In fact, those groups went from having almost no attacks at all in April, to attacks peaking at 20 thousand requests per second on a single site.

One particular attacker, likely using a hacked server in France, was especially persistent and kept up an attack continuously hitting an advocacy group continuously for over a day. We blocked those malicious HTTP requests and kept the site online.

Cyberattacks since the murder of George Floyd

We’ve also seen cyberattacks against other categories of Internet property change significantly between April and May. Attacks on Government websites (including police and fire departments) are up 1.8x and attacks on Military websites are up 3.8x month on month.

Since the murder of George Floyd there’s also been a large increase in attacks on US government web sites.

Cyberattacks since the murder of George Floyd

Project Galileo

Nearly six years ago, Cloudflare founded Project Galileo because we noticed a disturbing trend of disproportionate attacks against at-risk organizations and individuals that were advocating for marginalized groups. Project Galileo was set up to provide protection from cyberattacks for vulnerable targets, like artistic groups, humanitarian organizations, and the voices of political dissent. In our six years of protecting organizations under Project Galileo, we have often seen online attacks used in combination with physical violence and threats.

There are many organizations fighting racism who participate in Project Galileo. Over the last week we’ve seen a dramatic increase in the number of cyberattacks against them.

Cyberattacks since the murder of George Floyd

The whole Cloudflare community is deeply disturbed by the murder of George Floyd, and the shocking images of racial injustice playing out in our cities. We have been listening carefully to those who have taken to the streets in protest to demand justice and an end to structural racism, and believe that their powerful stories can serve as catalysts for real change. But that requires them to be heard. Unfortunately, if recent history is any guide, those who speak out against oppression will continue to face cyberattacks that attempt to silence them.

Cloudflare remains committed to making sure that they can continue to function in the face of these attacks, regardless of their resources or the size of the attack. If you know of an organization or group helping to fight racism that needs Project Galileo’s protection, please let them know we’re here and ready to help.

Moving from reCAPTCHA to hCAPTCHA

Post Syndicated from Matthew Prince original

Moving from reCAPTCHA to hCAPTCHA

Moving from reCAPTCHA to hCAPTCHA

We recently migrated the CAPTCHA provider we use from Google’s reCAPTCHA to a service provided by the independent hCAPTCHA. We’re excited about this change because it helps address a privacy concern inherent to relying on a Google service that we’ve had for some time and also gives us more flexibility to customize the CAPTCHAs we show. Since this change potentially impacts all Cloudflare customers, we wanted to walk through the rationale in more detail.

CAPTCHAs at Cloudflare

Moving from reCAPTCHA to hCAPTCHA

One of the services Cloudflare provides is a way to block malicious automated (“bot”) traffic. We use a number of techniques to accomplish that. When we are confident something is malicious bot activity we block it entirely. When we are confident it’s good human traffic (or a good bot like a search engine crawler) then we let it through. But, sometimes, when we’re not 100% sure if something is malicious or good we issue it a “challenge”.

We have different types of challenges, some are entirely automatic, but one requires human intervention. Those challenges are known as CAPTCHAs. That’s an acronym for Completely Automated Public Turing Test to Tell Computers and Humans Apart (a few Ts are dropped otherwise it’d be CAPTTTCHA). These are the prompts to enter squiggly letters in a box or identify traffic lights or cross walks. Generally, they’re supposed to be something that’s easy for humans to do but hard for machines.

Moving from reCAPTCHA to hCAPTCHA

Since Cloudflare’s earliest days, we have used Google’s reCAPTCHA service. ReCAPTCHA started as a research project out of Carnegie Mellon University in 2007. Google acquired the project in 2009, around the same time that Cloudflare was first getting started. Google provided reCAPTCHA for free in exchange for data from the service being used to train its visual identification systems. When we were looking for a CAPTCHA for Cloudflare, we chose reCAPTCHA because it was effective, could scale, and was offered for free — which was important since so many of Cloudflare’s customers use our free service.

Privacy and Blocking Concerns

Since those early days, some customers have expressed concerns about using a Google service to serve CAPTCHAs. Google’s business is targeting users with advertising. Cloudflare’s is not. We have strict privacy commitments. We were able to get comfortable with the Privacy Policy around reCAPTCHA, but understood why some of our customers were concerned about feeding more data to Google.

We also had issues in some regions, such as China, where Google’s services are intermittently blocked. China alone accounts for 25 percent of all Internet users. Given that some subset of those could not access Cloudflare’s customers if they triggered a CAPTCHA was always concerning to us.

Over the years, the privacy and blocking concerns were enough to cause us to think about switching from reCAPTCHA. But, like most technology companies, it was difficult to prioritize removing something that was largely working instead of brand new features and functionality for our customers.

Google’s Changing Business Model

Earlier this year, Google informed us that they were going to begin charging for reCAPTCHA. That is entirely within their right. Cloudflare, given our volume, no doubt imposed significant costs on the reCAPTCHA service, even for Google.

Again, this is entirely rational for Google. If the value of the image classification training did not exceed those costs, it makes perfect sense for Google to ask for payment for the service they provide. In our case, that would have added millions of dollars in annual costs just to continue to use reCAPTCHA for our free users. That was finally enough of an impetus for us to look for a better alternative.


We evaluated a number of CAPTCHA vendors as well as building a system ourselves. In the end, hCAPTCHA emerged as the best alternative to reCAPTCHA. We liked a number of things about the hCAPTCHA solutions: 1) they don’t sell personal data; they collect only minimum necessary personal data, they are transparent in describing the info they collect and how they use and/or disclose it, and they agreed to only use such data to provide the hCAPTCHA service to Cloudflare; 2) performance (both in speed and solve rates) was as good as or better than expected during our A/B testing; 3) it has a robust solution for visually impaired and other users with accessibility challenges; 4) it supported Privacy Pass to reduce the frequency of CAPTCHAs; 5) it worked in regions where Google was blocked; and 6) the hCAPTCHA team was nimble and responsive in a way that was refreshing.

The standard hCAPTCHA business model was similar to how reCAPTCHA started. They planned to charge customers that needed image classification data and pay publishers to install their CAPTCHA on their sites. Sounded great to us, but, unfortunately, while that may work well for most publishers, it doesn’t at Cloudflare’s scale.

We worked with hCAPTCHA in two ways. First, we are in the process of leveraging our Workers platform to bear much of the technical load of the CAPTCHAs and, in doing so, reduce their costs. And, second, we proposed that rather than them paying us we pay them. This ensured they had the resources to scale their service to meet our needs. While that has imposed some additional costs, those costs were a fraction of what reCAPTCHA would have. And, in exchange, we have a much more flexible CAPTCHA platform and a much more responsive team.

When do Customers Serve CAPTCHAs?

When we first started working on this project, the assumption was that Cloudflare Bot Management and Firewall Rules would be by far the largest consumer of CAPTCHAs. This was somewhat correct. While Firewall/Bots was the #1 consumer, it only was a bit over 50% of our CAPTCHAs served.

These are the breakdowns of when Cloudflare customers asked us to serve a CAPTCHA on their zones, by total CAPTCHAs served.

Firewall and Bot Rules 54.8%
IP Firewall 18.6%
Security Level 16.8%
DDoS 6.3%
Rate Limiting 1.7%
WAF Rules 1.5%
Other 0.3%

Our Firewall and Bot Rules are at the top and are the majority of the CAPTCHAs served by Cloudflare. These are rules written by our customers that specifically throw a CAPTCHA when the rule is matched. Examples of these include firing a Captcha if a Bot Management score is below a threshold where you believe it is likely that the connection is automated, but the score is above a threshold where you are not certain. Another common rule in this bucket is to CAPTCHA 100% of all traffic behind a site or specific endpoint. Customers may be doing this to limit connections to their origins, or to slow down automated systems from doing something like credential stuffing on a login page or polluting registration data. This leads to some sites on Cloudflare serving hundreds of millions of CAPTCHAs per day.

The second most popular is our IP Firewall. This is similar to the Firewall and Bot Rules, but much less granular at the IP, ASN, or country level. The majority of CAPTCHAs for this category are done for rules written at the ASN or country level. Presumably our customers find some level of distrust with a particular ASN (for example, why would there be supposed user traffic coming from a cloud infrastructure provider?) or are being attacked from specific countries.

Next comes Security Levels. Security levels have two distinct use cases: 1) as a blunt tool for IP address reputation and 2) I’m Under Attack Mode. While we recommend to customers that they only use I’m Under Attack Mode while under an active denial of service attack, some customers leave the feature on 100% of the time as a rudimentary form of rate limiting and filtering.

The final major use of CAPTCHA is through one of our automated systems: recently our Denial of Service protection engineering team taught Gatebot to use CAPTCHAs to mitigate small floods in specific scenarios. Gatebot can now write temporary rules that result in CAPTCHAs being shown to attackers.

Lastly, there were also some customers selecting it as an override action for their Rate Limiting or Managed WAF rulesets.

We also took a look at which types of customers were serving the CAPTCHAs. Over a week’s period of time (normalizing for attacks), our free customers configured their zones to serve roughly 40-60% of the total CAPTCHAs served by Cloudflare. Of our paying customers, there is a generally even split between our pay-as-you-go and our enterprise customers. Overall, we have measured that Cloudflare will show multiple millions of CAPTCHAs per second when one or more of our customers are under attack.

Fixing Challenges

Whenever we change any part of Cloudflare’s systems, it makes things significantly better for some, but temporarily worse for others. We and the hCAPTCHA team are committed to addressing any problems that come up. If you or your users see issues with the new hCAPTCHA implementation, please post on the forum or open a Support ticket with as much detail as possible.

Whenever possible, please include the RayID that usually appears on the footer of the CAPTCHA page so we can track down what went wrong.

Moving from reCAPTCHA to hCAPTCHA

Over time, we believe visual (and audio) CAPTCHAs are an imperfect answer to a number of difficult problems. Cloudflare is continuing work to minimize and hopefully eventually eliminate altogether the number of CAPTCHAs we issue and we will be excited to share the results of that work in this blog as we move along. The name of our internal chat room for the team making this change isn’t New CAPTCHA, it’s (No)CAPTCHA.

Cloudflare Doubling Size of 2020 Summer Intern Class

Post Syndicated from Matthew Prince original

Cloudflare Doubling Size of 2020 Summer Intern Class

Cloudflare Doubling Size of 2020 Summer Intern Class

We are living through extraordinary times. Around the world, the Coronavirus has caused disruptions to nearly everyone’s work and personal lives. It’s been especially hard to watch as friends and colleagues outside Cloudflare are losing jobs and businesses struggle through this crisis.

We have been extremely fortunate at Cloudflare. The super heroes of this crisis are clearly the medical professionals at the front lines saving people’s lives and the scientists searching for a cure. But the faithful sidekick that’s helping us get through this crisis — still connected to our friends, loved ones, and, for those of us fortunate enough to be able to continue work from home, our jobs — is the Internet. As we all need it more than ever, we’re proud of our role in helping ensure that the Internet continues to work securely and reliably for all our customers.

We plan to invest through this crisis. We are continuing to hire across all teams at Cloudflare and do not foresee any need for layoffs. I appreciate the flexibility of our team and new hires to adapt what was our well-oiled, in-person orientation process to something virtual we’re continuing to refine weekly as new people join us.

Summer Internships

One group that has been significantly impacted by this crisis are students who were expecting internships over the summer. Many are, unfortunately, getting notice that the experiences they were counting on have been cancelled. These internships are not only a significant part of these students’ education, but in many cases provide an income that helps them get through the school year.

Cloudflare is not cancelling any of our summer internships. We anticipate that many of our internships will need to be remote to comply with public health recommendations around travel and social distancing. We also understand that some students may prefer a remote internship even if we do begin to return to the office so they can take care of their families and avoid travel during this time. We stand by every internship offer we have extended and are committed to making each internship a terrific experience whether remote, in person, or some mix of both.

Doubling the Size of the 2020 Internship Class

But, seeing how many great students were losing their internships at other companies, we wanted to do more. Today we are announcing that we will double the size of Cloudflare’s summer 2020 internship class. Most of the internships we offer are in our product, security, research and engineering organizations, but we also have some positions in our marketing and legal teams. We are reopening the internship application process and are committed to making decisions quickly so students can plan their summers. You can find newly open internships posted at the link below.

Internships are jobs, and we believe people should be paid for the jobs they do, so every internship at Cloudflare is paid. That doesn’t change with these new internship positions we’re creating: they will all be paid.

Highlighting Other Companies with Opportunities

Even when we double the size of our internship class we expect that we will receive far more qualified applicants than we will be able to accommodate. We hope that other companies that are in a fortunate position to be able to weather this crisis will consider expanding their internship classes as well. We plan to work with peer organizations and will highlight those that also have summer internship openings. If your company still has available internship positions, please let us know by emailing so we can point students your way: [email protected]

Opportunity During Crisis

Cloudflare was born out of a time of crisis. Michelle and I were in school when the global financial crisis hit in 2008. Michelle had spent that summer at an internship at Google. That was the one year Google decided to extend no full-time offers to summer interns. So, in the spring of 2009, we were both still trying to figure out what we were going to do after school.

It didn’t feel great at the time, but had we not been in the midst of that crisis I’m not sure we ever would have started Cloudflare. Michelle and I remember the stress of that time very clearly. The recognition of the importance of planning for rainy days has been part of what has made Cloudflare so resilient. And it’s why, when we realized we could play a small part in ensuring some students who had lost the internships they thought they had could still have a rewarding experience, we knew it was the right decision.

Together, we can get through this. And, when we do, we will all be stronger.

The Mistake that Caused to Block LGBTQIA+ Sites Today

Post Syndicated from Matthew Prince original

The Mistake that Caused to Block LGBTQIA+ Sites Today

Today we made a mistake. The mistake caused a number of LGBTQIA+ sites to inadvertently be blocked by the new for Families service. I wanted to walk through what happened, why, and what we’ve done to fix it.

As is our tradition for the last three years, we roll out new products for the general public that uses the Internet on April 1. This year, one of those products was a filtered DNS service, for Families. The service allows anyone who chooses to use it to restrict certain categories of sites.

Filtered vs Unfiltered DNS

Nothing about our new filtered DNS service changes the unfiltered nature of our original service. However, we recognized that some people want a way to control what content is in their home. For instance, I block social media sites from resolving while I am trying to get work done because it makes me more productive. The number one request from users of was that we create a version of the service for home use to block certain categories of sites. And so, earlier today, we launched for Families.

Over time, we’ll provide the ability for users of for Families to customize exactly what categories they block (e.g., do what I do with social media sites to stay productive). But, initially, we created two default settings that were the most requested types of content people wanted to block: Malware (which you can block by setting and as your DNS resolvers) and Malware + Adult Content (which you can block by setting and as your DNS resolvers).

Licensed Categorization Data

To get data for for Families  we licensed feeds from multiple different providers who specialize in site categorization. We spent the last several months reviewing classification providers to choose the ones that had the highest accuracy and lowest false positives.

Malware, encompassing a range of widely agreed upon cyber security threats, was the easier of the two categories to define. For Adult Content, we aimed to mirror the Google SafeSearch criteria. Google has been thoughtful in this area and their SafeSearch tool is designed to limit search results for “sexually explicit content.” The definition is focused on pornography and largely follows the requirements of the US Children’s Internet Protection Act (CIPA), which schools and libraries in the United States are required to follow.

Because it was the default for the service, and because we planned in the future to allow individuals to set their own specifications beyond the default, we intended the Adult Content category to be narrow. What we did not intend to include in the Adult Content category was LGBTQIA+ content. And yet, when it launched, we were horrified to receive reports that those sites were being filtered.

Choosing the Wrong Feed

So what went wrong? The data providers that we license content from have different categorizations; those categorizations do not line up perfectly between different providers. One of the providers has multiple “Adult Content” categories. One “Adult Content” category includes content that mirrors the Google SafeSearch/CIPA definition. Another “Adult Content” content category includes a broader set of topics, including LGBTQIA+ sites.

While we had specifically reviewed the Adult Content category to ensure that it was narrowly tailored to mirror the Google SafeSearch/CIPA definition, when we released the production version this morning we included the wrong “Adult Content” category from the provider in the build. As a result, the first users who tried saw a broader set of sites being filtered than was intended, including LGBTQIA+ content. We immediately worked to fix the issue.

Slow to Update Data Structures

In order to distribute the list of sites quickly to all our data centers we use a compact data structure. The upside is that we can replicate the data structure worldwide very efficiently. The downside is that generating a new version of the data structure takes several hours. The minute we saw that we’d made a mistake we pulled the incorrect data provider and began recreating the new data structure.

While the new data structure replicated across our network we pushed individual sites to an allow list immediately. We began compiling lists both from user reports as well as from other LGBTQIA+ resources. These updates went out instantly. We continuously added sites to the allow list as they were reported or we discovered them.

By 16:51 UTC, approximately two hours after we’d received the first report of the mistaken blocking, the data structure with the intended definition of Adult Content had been generated and we pushed it out live. The only users that would have seen over-broad blocking are those that had already switched to the service. Users of — which will remain unfiltered — and would not have experienced this inadvertent blocking.

As of now, the filtering provided by the default setting of is what we intended it to be, and should roughly match what you find if you use Google SafeSearch and LGBTQIA+ sites are not being blocked. If you see site being blocked that should not be, please report them to us here.

Protections for the Future

Going forward, we’ve set up a number of checks of known sites that should fall outside the intended categories, including many that we mistakenly listed today. Before defaults are updated in the future, our build system will confirm that none of these sites are listed. We hope this will help catch mistakes like this in the future.

I’m sorry for the error. While I understand how it happened, it should never have happened. I appreciate our team responding quickly to fix the mistake we made.

Introducing for Families

Post Syndicated from Matthew Prince original

Introducing for Families

Two years ago today we announced, a secure, fast, privacy-first DNS resolver free for anyone to use. In those two years, has grown beyond our wildest imagination. Today, we process more than 200 billion DNS requests per day making us the second largest public DNS resolver in the world behind only Google.

Introducing for Families

Yesterday, we announced the results of the privacy examination. Cloudflare’s business has never involved selling user data or targeted advertising, so it was easy for us to commit to strong privacy protections for We’ve also led the way supporting encrypted DNS technologies including DNS over TLS and DNS over HTTPS. It is long past time to stop transmitting DNS in plaintext and we’re excited that we see more and more encrypted DNS traffic every day. for Families

Introducing for Families

Since launching, the number one request we have received is to provide a version of the product that automatically filters out bad sites. While can safeguard user privacy and optimize efficiency, it is designed for direct, fast DNS resolution, not for blocking or filtering content. The requests we’ve received largely come from home users who want to ensure that they have a measure of protection from security threats and can keep adult content from being accessed by their kids. Today, we’re happy to answer those requests.

Introducing for Families — the easiest way to add a layer of protection to your home network and protect it from malware and adult content. for Families leverages Cloudflare’s global network to ensure that it is fast and secure around the world. And it includes the same strong privacy guarantees that we committed to when we launched two years ago. And, just like, we’re providing it for free and it’s for any home anywhere in the world.

Two Flavors: (No Malware) & (No Malware or Adult Content)

Introducing for Families for Families is easy to set up and install, requiring just changing two numbers in the settings of your home devices or network router: your primary DNS and your secondary DNS. Setting up for Families usually takes less than a minute and we’ve provided instructions for common devices and routers through the installation guide. for Families has two default options: one that blocks malware and the other that blocks malware and adult content. You choose which setting you want depending on which IP address you configure.

Malware Blocking Only
Primary DNS:
Secondary DNS:

Malware and Adult Content
Primary DNS:
Secondary DNS:

Additional Configuration

Introducing for Families

In the coming months, we will provide the ability to define additional configuration settings for for Families. This will include options to create specific whitelists and blacklists of certain sites. You will be able to set the times of the day when categories, such as social media, are blocked and get reports on your household’s Internet usage. for Families is built on top of the same site categorization and filtering technology that powers Cloudflare’s Gateway product. With the success of Gateway, we wanted to provide an easy-to-use service that can help any home network be fast, reliable, secure, and protected from potentially harmful content.

Not A Joke

Most of Cloudflare’s business involves selling services to businesses. However, we’ve made it a tradition every April 1 to launch a new consumer product that leverages our network to bring more speed, reliability, and security to every Internet user. While we make money selling to businesses, the products we launch at this time of the year are close to our hearts because of the broad impact they have for every Internet user.

Introducing for Families

This year, while many of us are confined to our homes, protecting our communities from COVID-19, and relying on our home networks more than ever it seemed especially important to launch for Families. We hope during these troubled times it will help provide a bit of peace of mind for households everywhere.