Tag Archives: Project Galileo

Project Galileo: the view from the front lines

Post Syndicated from Erin Walk original https://blog.cloudflare.com/project-galileo-the-view-from-the-front-lines/

Project Galileo: the view from the front lines

Growing up in the age of technology has made it too easy for me to take the presence of the Internet for granted. It’s hard to imagine not being able to go online and connect with anyone in the world, whether I’m speaking with family members or following activists planning global rallies in support of a common cause. I find that as I forget the wonder of being connected, I become jaded. I imagine that many of you reading this blog feel the same way. I doubt you have gone a month, or even a week, this year without considering that the world might be better off without the Internet, or without parts of the Internet, or that your life would be better with a digital cleanse. Project Galileo is my antidote. For every person online who abuses their anonymity, there is an organization that literally could not fulfill their purpose without it. And they are doing amazing work.

Project Galileo: the view from the front lines

Working with Participants

As program manager for Project Galileo, Cloudflare’s initiative to provide free services to vulnerable voices on the Internet, a large portion of my time is spent interacting with the project’s participants and partners. This includes a variety of activities. In my organizational role, I reach out to our partnering organizations, such as the National Democratic Institute and the Center for Democracy and Technology, about sponsoring new recipients. I also help recipients onboard their websites and technically explain our product and how it works. Answering emails from Project Galileo recipients is my favorite part of every day. I can still remember when the sense of wonder truly set in. A few weeks into my time at Cloudflare, I received a request from a local community healthcare clinic that was under attack. I was new, I didn’t have all the permissions I have now, and I didn’t fully understand how all of our systems worked (I still don’t, but I’m much better at figuring out who does). I started reaching out to other teams, all of whom eagerly volunteered their time. Within a few hours, a website that had been down for a week was back up, and best practices were being discussed to help them stay online in the future.

About a week later I received a wonderful thank you message from the group, and made sure I sent it to those who had helped out and were invested. I treasure these little reminders in my day that what I’m doing makes a difference. In fact, I frequently question my luck in receiving all the praise for a project that functions thanks to the work of countless engineers, and other teams, who work tirelessly to make our product better. I try to find ways to pass these small moments on.

It makes me laugh when participants who joined while I’ve been working on the project email me with an introduction along the lines of “I don’t know if you remember us, but…”. It makes sense, in the abstract. I receive a lot of emails, and around half of all recipients have joined since I started organizing the project. Still, I remember almost everyone who I’ve written to. How could I forget the person who signed off all their emails with something joyful they were doing at the moment, or the one who told me that they had finally made it through a week without their website going down? In many ways, on Project Galileo I interact less with organizations and more with a set of extremely passionate people. The purpose and drive of these individuals infect me with a sense of wonder and excitement, even when our only communications are virtual.

Project Galileo: the view from the front lines
Project Galileo partners

Internal Commitment

Project Galileo doesn’t just bring out the best of the Internet through our recipients, it also brings out the best in Cloudflare. Working on Project Galileo has given me a lot of leeway to explore all aspects of the company. We don’t have a large team in DC, and most of us are on the Policy team. To do my job, I rely on being able to contact teams globally, from Support to Trust and Safety to Solutions Engineering. I’ve chatted with Support team members at 2am to fix an emergency situation, and had a Solutions Engineer on call from 11pm to 1am on a Friday night to support an organization during an event. Even when frustrating or anxiety provoking, these times make me proud to work for an organization that not only vocally supports this project, but whose members commit their time to it despite competing priorities.

At risk of being overly grandiose, there are a lot of hopes and dreams tied up in Project Galileo. There is the dream that the Internet is a place for vulnerable voices, no matter how small, to advocate for change. There is the dream that companies will use their products to help deserving groups who may not otherwise be able to afford them. As for me, I hope that every day I do something that makes the world a little better. It is an honor to carry these hopes and dreams within the company, and I strive to be a good steward.

Happy 5th Birthday, Project Galileo! Here’s to many more.

Project Galileo: the view from the front lines

Protecting Project Galileo websites from HTTP attacks

Post Syndicated from Maxime Guerreiro original https://blog.cloudflare.com/protecting-galileo-websites/

Protecting Project Galileo websites from HTTP attacks

Yesterday, we celebrated the fifth anniversary of Project Galileo. More than 550 websites are part of this program, and they have something in common: each and every one of them has been subject to attacks in the last month. In this blog post, we will look at the security events we observed between the 23 April 2019 and 23 May 2019.

Project Galileo sites are protected by the Cloudflare Firewall and Advanced DDoS Protection which contain a number of features that can be used to detect and mitigate different types of attack and suspicious traffic. The following table shows how each of these features contributed to the protection of sites on Project Galileo.

Firewall Feature

Requests Mitigated

Distinct originating IPs

Sites Affected (approx.)

Firewall
Rules

78.7M

396.5K

~ 30

Security
Level

41.7M

1.8M

~ 520

Access
Rules

24.0M

386.9K

~ 200

Browser
Integrity Check

9.4M

32.2K

~ 500

WAF

4.5M

163.8K

~ 200

User-Agent
Blocking

2.3M

1.3K

~ 15

Hotlink
Protection

2.0M

686.7K

~ 40

HTTP
DoS

1.6M

360

1

Rate
Limit

623.5K

6.6K

~ 15

Zone
Lockdown

9.7K

2.8K

~ 10

WAF (Web Application Firewall)

Although not the most impressive in terms of blocked requests, the WAF is the most interesting as it identifies and blocks malicious requests, based on heuristics and rules that are the result of seeing attacks across all of our customers and learning from those. The WAF is available to all of our paying customers, protecting them against 0-days, SQL/XSS exploits and more. For the Project Galileo customers the WAF rules blocked more than 4.5 million requests in the month that we looked at, matching over 130 WAF rules and approximately 150k requests per day.

Protecting Project Galileo websites from HTTP attacks
Heat map showing the attacks seen on customer sites (rows) per day (columns)

This heat map may initially appear confusing but reading one is easy once you know what to expect so bear with us! It is a table where each line is a website on Project Galileo and each column is a day. The color represents the number of requests triggering WAF rules – on a scale from 0 (white) to a lot (dark red). The darker the cell, the more requests were blocked on this day.

We observe malicious traffic on a daily basis for most websites we protect. The average Project Galileo site saw malicious traffic for 27 days in the 1 month observed, and for almost 60% of the sites we noticed daily events.

Fortunately, the vast majority of websites only receive a few malicious requests per day, likely from automated scanners. In some cases, we notice a net increase in attacks against some websites – and a few websites are under a constant influx of attacks.

Protecting Project Galileo websites from HTTP attacks
Heat map showing the attacks blocked for each WAF rule (rows) per day (columns)

This heat map shows the WAF rules that blocked requests by day. At first, it seems some rules are useless as they never match malicious requests, but this plot makes it obvious that some attack vectors become active all of a sudden (isolated dark cells). This is especially true for 0-days, malicious traffic starts once an exploit is published and is very active on the first few days. The dark active lines are the most common malicious requests, and these WAF rules protect against things like XSS and SQL injection attacks.

DoS (Denial of Service)

A DoS attack prevents legitimate visitors from accessing a website by flooding it with bad traffic.  Due to the way Cloudflare works, websites protected by Cloudflare are immune to many DoS vectors, out of the box. We block layer 3 and 4 attacks, which includes SYN floods and UDP amplifications. DNS nameservers, often described as the Internet’s phone book, are fully managed by Cloudflare, and protected – visitors know how to reach the websites.

Protecting Project Galileo websites from HTTP attacks
Line plot – requests per second to a website under DoS attack

Can you spot the attack?

As for layer 7 attacks (for instance, HTTP floods), we rely on Gatebot, an automated tool to detect, analyse and block DoS attacks, so you can sleep. The graph shows the requests per second we received on a zone, and whether or not it reached the origin server. As you can see, the bad traffic was identified automatically by Gatebot, and more than 1.6 million requests were blocked as a result.

Firewall Rules

For websites with specific requirements we provide tools to allow customers to block traffic to precisely fit their needs. Customers can easily implement complex logic using Firewall Rules to filter out specific chunks of traffic, block IPs / Networks / Countries using Access Rules and Project Galileo sites have done just that. Let’s see a few examples.

Firewall Rules allows website owners to challenge or block as much or as little traffic as they desire, and this can be done as a surgical tool “block just this request” or as a general tool “challenge every request”.

For instance, a well-known website used Firewall Rules to prevent twenty IPs from fetching specific pages. 3 of these IPs were then used to send a total of 4.5 million requests over a short period of time, and the following chart shows the requests seen for this website. When this happened Cloudflare, mitigated the traffic ensuring that the website remains available.

Protecting Project Galileo websites from HTTP attacks
Cumulative line plot. Requests per second to a website

Another website, built with WordPress, is using Cloudflare to cache their webpages. As POST requests are not cacheable, they always hit the origin machine and increase load on the origin server – that’s why this website is using firewall rules to block POST requests, except on their administration backend. Smart!

Website owners can also deny or challenge requests based on the visitor’s IP address, Autonomous System Number (ASN) or Country. Dubbed Access Rules, it is enforced on all pages of a website – hassle-free.

For example, a news website is using Cloudflare’s Access Rules to challenge visitors from countries outside of their geographic region who are accessing their website. We enforce the rules globally even for cached resources, and take care of GeoIP database updates for them, so they don’t have to.

The Zone Lockdown utility restricts a specific URL to specific IP addresses. This is useful to protect an internal but public path being accessed by external IP addresses. A non-profit based in the United Kingdom is using Zone Lockdown to restrict access to their WordPress’ admin panel and login page, hardening their website without relying on non official plugins. Although it does not prevent very sophisticated attacks, it shields them against automated attacks and phishing attempts – as even if their credentials are stolen, they can’t be used as easily.

Rate Limiting

Cloudflare acts as a CDN, caching resources and happily serving them, reducing bandwidth used by the origin server … and indirectly the costs. Unfortunately, not all requests can be cached and some requests are very expensive to handle. Malicious users may abuse this to increase load on the server, and website owners can rely on our Rate Limit to help them: they define thresholds, expressed in requests over a time span, and we make sure to enforce this threshold. A non-profit fighting against poverty relies on rate limits to protect their donation page, and we are glad to help!

Security Level

Last but not least, one of Cloudflare’s greatest assets is our threat intelligence. With such a wide lens of the threat landscape, Cloudflare uses our Firewall data, combined with machine learning to curate our IP Reputation databases. This data is provided to all Cloudflare customers, and is configured through our Security Level feature. Customers then may define their threshold sensitivity, ranging  from Essentially Off to I’m Under Attack. For every incoming request, we ask visitors to complete a challenge if the score is above a customer defined threshold. This system alone is responsible for 25% of the requests we mitigated: it’s extremely easy to use, and it constantly learns from the other protections.

Conclusion

When taken together, the Cloudflare Firewall features provide our Project Galileo customers comprehensive and effective security that enables them to ensure their important work is available. The majority of security events were handled automatically, and this is our strength – security that is always on, always available, always learning.

Project Galileo: Lessons from 5 years of protecting the most vulnerable online

Post Syndicated from Matthew Prince original https://blog.cloudflare.com/project-galileo-fifth-anniversary/

Project Galileo: Lessons from 5 years of protecting the most vulnerable online

Today is the 5th anniversary of Cloudflare’s Project Galileo. Through the Project, Cloudflare protects—at no cost—nearly 600 organizations around the world engaged in some of the most politically and artistically important work online. Because of their work, these organizations are attacked frequently, often with some of the fiercest cyber attacks we’ve seen.

Project Galileo: Lessons from 5 years of protecting the most vulnerable online

Since it launched in 2014, we haven’t talked about Galileo much externally because we worry that drawing more attention to these organizations may put them at increased risk. Internally, however, it’s a source of pride for our whole team and is something we dedicate significant resources to. And, for me personally, many of the moments that mark my most meaningful accomplishments were born from our work protecting Project Galileo recipients.

The promise of Project Galileo is simple: Cloudflare will provide our full set of security services to any politically or artistically important organizations at no cost so long as they are either non-profits or small commercial entities. I’m still on the distribution list that receives an email whenever someone applies to be a Project Galileo participant, and those emails remain the first I open every morning.

Project Galileo: Lessons from 5 years of protecting the most vulnerable online

The Project Galileo Backstory

Five years ago, Project Galileo was born out of a mistake we made. At the time, Cloudflare’s free service didn’t include DDoS mitigation. If a free customer came under attack, our operations team would generally stop proxying their traffic. We did this to protect our own network, which was much smaller than it is today.

Usually this wasn’t a problem. Most sites that got attacked at the time were companies or businesses that could pay for our services.

Every morning I’d receive a report of the sites that were kicked off Cloudflare the night before. One morning in late February 2014 I was reading the report as I walked to work. One of the sites listed as having been dropped stood out as familiar but I couldn’t place it.

I tried to pull up the site on my phone but it was offline, presumably because we were no longer shielding the site from attack. Still curious, I did a quick search and found a Wikipedia page describing the site. It was an independent newspaper in Ukraine and had been covering the ongoing Russian invasion of Crimea.

I felt sick.

When Nation States Attack

What we later learned was that this publication had come under a significant attack, most likely directly from the Russian government. The newspaper had turned to Cloudflare for protection. Their IT director actually tried to pay for our higher tier of service but the bank tied to the publication’s credit card had had its systems disrupted by a cyber attack as well and the payment failed. So they’d signed up for the free version of Cloudflare and, for a while, we mitigated the attack.

The attack was large enough that it triggered an alert in our Network Operations Center (NOC). A member of our Systems Reliability Engineering (SRE) team who was on call investigated and found a free customer being pummeled by a major attack. He followed our run book and triggered a FINT — which stands for “Fail Internal” — directing traffic from the site directly back to its origin rather than passing through Cloudflare’s protective edge. Instantly the site was overwhelmed by the attack and, effectively, fell off the Internet.

Broken Process

I should be clear: the SRE didn’t do anything wrong. He followed the procedures we had established at the time exactly. He was a great computer scientist, but not a political scientist, so didn’t recognize the site or understand its importance due to the situation at the time in Crimea and why a newspaper covering it may come under attack. But, the next morning, as I read the report on my walk in to work, I did.

Cloudflare’s mission is to help build a better Internet. That day we failed to live up to that mission. I knew we had to do something.

Politically or Artistically Important?

It was relatively easy for us to decide to provide Cloudflare’s security services for free to politically or artistically important non-profits and small commercial entities. We were confident that we could stand up to even the largest attacks. What we were less confident about was our ability to determine who was “politically or artistically important.”

While Cloudflare runs infrastructure all around the world, our team is largely based in San Francisco, Austin, London, and Singapore. That certainly gives us a viewpoint, but it isn’t a particularly globally representative viewpoint. We’re also a very technical organization. If we surveyed our team to determine what organizations deserved protection we’d no-doubt identify a number of worthy organizations that were close to home and close to our interests, but we’d miss many others.

We also worried that it was dangerous for an infrastructure provider like Cloudflare to start making decisions about what content was “good.” Doing so inherently would imply that we were in a position to make decisions about what content was “bad.” While moderating content and curating communities is appropriate for some more visible platforms, the deeper you go into Internet infrastructure, the less transparent, accountable, and consistent those decisions inherently become.

Turning to the Experts

So, rather than making the determination of who was politically or artistically important ourselves, we turned to civil society organizations that were experts in exactly that. Initially, we partnered with 15 organizations, including:

  • Access Now
  • American Civil Liberties Union (ACLU)
  • Center for Democracy and Technology (CDT)
  • Centre for Policy Alternatives
  • Committee to Protect Journalists (CPJ)
  • Electronic Frontier Foundation (EFF)
  • Engine Advocacy
  • Freedom of the Press Foundation
  • Meedan
  • Mozilla
  • Open Tech Fund
  • Open Technology Institute

We agreed that if any partner said that a non-profit or small commercial entity that applied for protection was “politically or artistically important” then we would extend our security services and protect them, no matter what.

With that, Project Galileo was born. Nearly 600 organizations are currently being protected under Project Galileo. We’ve never removed an organization from protection in spite of occasional political pressure as well as frequent extremely large attacks.

Organizations can apply directly through Cloudflare for Project Galileo protection or can be referred by a partner. Today, we’ve grown the list of partners to 28, adding:

  • Anti-Defamation League
  • Amnesty International
  • Business & Human Rights Resource Centre
  • Council of Europe
  • Derechos Digitales
  • Fourth Estate
  • Frontline Defenders
  • Institute for War & Peace Reporting (IWPR)
  • LION Publishers
  • National Democratic Institute (NDI)
  • Reporters Sans Frontières
  • Social Media Exchange (SMEX)
  • Sontusdatos.org
  • Tech Against Terrorism
  • World Wide Web Foundation
  • X-Lab

Cloudflare’s Mission: Help Build a Better Internet

Some companies start with a mission. Cloudflare was not one of those companies. When Michelle, Lee, and I started building Cloudflare it was because we thought we’d identified a significant business opportunity. Truth be told, I thought the idea of being “mission driven” was kind of hokum.

I clearly remember the day that changed for me. The director of one of the Project Galileo partners called me to say that he had three journalists who had received protection under Project Galileo that were visiting San Francisco and asked if it would be okay to bring them by our office. I said sure and carved out a bit of time to meet with them.

The three journalists turned out to all be covering alleged government corruption in their home countries. One was from Angola, one was from Ethiopia, and they wouldn’t tell me the name or home country of the third because he was “currently being hunted by death squads.” All three of them hugged me. One had tears in his eyes. And then they proceeded to tell me about how they couldn’t do their work as journalists without Cloudflare’s protection.

There are incredibly brave people doing important work and risking their lives around the world. Some of them use the Internet to reach their audience. Whether it’s African journalists covering alleged government corruption, LGBTQ communities in the Middle East providing support, or human rights workers in repressive regimes, unfortunately they all face the risk that the powerful forces that oppose them will use cyber attacks to silence them.

I’m proud of the work we’ve done through Project Galileo over the last five years lending the full weight of Cloudflare to protect these politically and artistically important organizations. It has defined our mission to help build a better Internet.

While we respect the confidentiality of the organizations that receive support under the Project, I’m thankful that a handful have allowed us to tell their stories. I encourage you to read about our newest recipients of the Project:

And, finally, if you know of an organization that needs Project Galileo’s protection, please let them know we’re here and happy to help.

Project Galileo: Lessons from 5 years of protecting the most vulnerable online