Tag Archives: Product News

Stream Live is now Generally Available

2022-09-21 Brendan Irvine-Broque

Post Syndicated from Brendan Irvine-Broque original https://blog.cloudflare.com/stream-live-ga/

Stream Live is now Generally Available

Today, we’re excited to announce that Stream Live is out of beta, available to everyone, and ready for production traffic at scale. Stream Live is a feature of Cloudflare Stream that allows developers to build live video features in websites and native apps.

Since its beta launch, developers have used Stream to broadcast live concerts from some of the world’s most popular artists directly to fans, build brand-new video creator platforms, operate a global 24/7 live OTT service, and more. While in beta, Stream has ingested millions of minutes of live video and delivered to viewers all over the world.

Bring your big live events, ambitious new video subscription service, or the next mobile video app with millions of users — we’re ready for it.

Streaming live video at scale is hard

Live video uses a massive amount of bandwidth. For example, a one-hour live stream at 1080p at 8Mbps is 3.6GB. At typical cloud provider egress prices, even a little egress can break the bank.

Live video must be encoded on-the-fly, in real-time. People expect to be able to watch live video on their phone, while connected to mobile networks with less bandwidth, higher latency and frequently interrupted connections. To support this, live video must be re-encoded in real-time into multiple resolutions, allowing someone’s phone to drop to a lower resolution and continue playback. This can be complex (Which bitrates? Which codecs? How many?) and costly: running a fleet of virtual machines doesn’t come cheap.

Ingest location matters — Live streaming protocols like RTMPS send video over TCP. If a single packet is dropped or lost, the entire connection is brought to a halt while the packet is found and re-transmitted. This is known as “head of line blocking”. The further away the broadcaster is from the ingest server, the more network hops, and the more likely packets will be dropped or lost, ultimately resulting in latency and buffering for viewers.

Delivery location matters — Live video must be cached and served from points of presence as close to viewers as possible. The longer the network round trips, the more likely videos will buffer or drop to a lower quality.

Broadcasting protocols are in flux — The most widely used protocol for streaming live video, RTMPS, has been abandoned since 2012, and dates back to the era of Flash video in the early 2000s. A new emerging standard, SRT, is not yet supported everywhere. And WebRTC has only recently evolved into an option for high definition one-to-many broadcasting at scale.

The old way to solve this has been to stitch together separate cloud services from different vendors. One vendor provides excellent content delivery, but no encoding. Another provides APIs or hardware to encode, but leaves you to fend for yourself and build your own storage layer. As a developer, you have to learn, manage and write a layer of glue code around the esoteric details of video streaming protocols, codecs, encoding settings and delivery pipelines.

We built Stream Live to make streaming live video easy, like adding an <img> tag to a website. Live video is now a fundamental building block of Internet content, and we think any developer should have the tools to add it to their website or native app.

With Stream, you or your users stream live video directly to Cloudflare, and Cloudflare delivers video directly to your viewers. You never have to manage internal encoding, storage, or delivery systems — it’s just live video in and live video out.

Our network, our hardware = a solution only Cloudflare can provide

We’re not the only ones building APIs for live video — but we are the only ones with our own global network and hardware that we control and optimize for video. That lets us do things that others can’t, like sub-second glass-to-glass latency using RTMPS and SRT playback at scale.

Newer video codecs require specialized hardware encoders, and while others are beholden to the hardware limitations of public cloud providers, we’re already hard at work installing the latest encoding hardware in our own racks, so that you can deliver high resolution video with even less bandwidth. Our goal is to make what is otherwise only available to video giants available directly to you — stay tuned for some exciting updates on this next week.

Most providers limit how many destinations you can restream or simulcast to. Because we operate our own network, we’ve never even worried about this, and let you restream to as many destinations as you need.

Operating our own network lets us price Stream based on minutes of video delivered — unlike others, we don’t pay someone else for bandwidth and then pass along their costs to you at a markup. The status quo of charging for bandwidth or per-GB storage penalizes you for delivering or storing high resolution content. If you ask why a few times, most of the time you’ll discover that others are pushing their own cost structures on to you.

Encoding video is compute-intensive, delivering video is bandwidth intensive, and location matters when ingesting live video. When you use Stream, you don’t need to worry about optimizing performance, finding a CDN, and/or tweaking configuration endlessly. Stream takes care of this for you.

Free your live video from the business models of big platforms

Nearly every business uses live video, whether to engage with customers, broadcast events or monetize live content. But few have the specialized engineering resources to deliver live video at scale on their own, and wire together multiple low level cloud services. To date, many of the largest content creators have been forced to depend on a shortlist of social media apps and streaming services to deliver live content at scale.

Unlike the status quo, who force you to put your live video in their apps and services and fit their business models, Stream gives you full control of your live video, on your website or app, on any device, at scale, without pushing your users to someone else’s service.

Free encoding. Free ingestion. Free analytics. Simple per-minute pricing.

	Others	Stream
Encoding	$ per minute	Free
Ingestion	$ per GB	Free
Analytics	Separate product	Free
Live recordings	Minutes or hours later	Instant
Storage	$ per GB	per minute stored
Delivery	$ per GB	per minute delivered

Other platforms charge for ingestion and encoding. Many even force you to consider where you’re streaming to and from, the bitrate and frames per second of your video, and even which of their datacenters you’re using.

With Stream, encoding and ingestion are free. Other platforms charge for delivery based on bandwidth, penalizing you for delivering high quality video to your viewers. If you stream at a high resolution, you pay more.

With Stream, you don’t pay a penalty for delivering high resolution video. Stream’s pricing is simple — minutes of video delivered and stored. Because you pay per minute, not per gigabyte, you can stream at the ideal resolution for your viewers without worrying about bandwidth costs.

Other platforms charge separately for analytics, requiring you to buy another product to get metrics about your live streams.

With Stream, analytics are free. Stream provides an API and Dashboard for both server-side and client-side analytics, that can be queried on a per-video, per-creator, per-country basis, and more. You can use analytics to identify which creators in your app have the most viewed live streams, inform how much to bill your customers for their own usage, identify where content is going viral, and more.

Other platforms tack on live recordings or DVR mode as a separate add-on feature, and recordings only become available minutes or even hours after a live stream ends.

With Stream, live recordings are a built-in feature, made available instantly after a live stream ends. Once a live stream is available, it works just like any other video uploaded to Stream, letting you seamlessly use the same APIs for managing both pre-recorded and live content.

Build live video into your website or app in minutes

Cloudflare Stream enables you or your users to go live using the same protocols and tools that broadcasters big and small use to go live to YouTube or Twitch, but gives you full control over access and presentation of live streams.

Step 1: Create a live input

Create a new live input from the Stream Dashboard or use use the Stream API:

Request

curl -X POST \
-H "Authorization: Bearer <YOUR_API_TOKEN>" \
-d "{"recording": { "mode": "automatic" } }" \
https://api.cloudflare.com/client/v4/accounts/<YOUR_CLOUDFLARE_ACCOUNT_ID>/stream/live_inputs

Response

{
"result": {
"uid": "<UID_OF_YOUR_LIVE_INPUT>",
"rtmps": {
"url": "rtmps://live.cloudflare.com:443/live/",
"streamKey": "<PRIVATE_RTMPS_STREAM_KEY>"
},
...
}
}

Step 2: Use the RTMPS key with any live broadcasting software, or in your own native app

Copy the RTMPS URL and key, and use them with your live streaming application. We recommend using Open Broadcaster Software (OBS) to get started, but any RTMPS or SRT compatible software should be able to interoperate with Stream Live.

Enter the Stream RTMPS URL and the Stream Key from Step 1:

Step 3: Preview your live stream in the Cloudflare Dashboard

In the Stream Dashboard, within seconds of going live, you will see a preview of what your viewers will see, along with the real-time connection status of your live stream.

Step 4: Add live video playback to your website or app

Stream your video using our Stream Player embed code, or use any video player that supports HLS or DASH — live streams can be played in both websites or native iOS and Android apps.

For example, on iOS, all you need to do is provide AVPlayer with the URL to the HLS manifest for your live input, which you can find via the API or in the Stream Dashboard.

import SwiftUI
import AVKit

struct MyView: View {
    // Change the url to the Cloudflare Stream HLS manifest URL
    private let player = AVPlayer(url: URL(string: "https://customer-9cbb9x7nxdw5hb57.cloudflarestream.com/8f92fe7d2c1c0983767649e065e691fc/manifest/video.m3u8")!)

    var body: some View {
        VideoPlayer(player: player)
            .onAppear() {
                player.play()
            }
    }
}

struct MyView_Previews: PreviewProvider {
    static var previews: some View {
        MyView()
    }
}

To run a complete example app in XCode, follow this guide in the Stream Developer docs.

Companies are building whole new video platforms on top of Stream

Developers want control, but most don’t have time to become video experts. And even video experts building innovative new platforms don’t want to manage live streaming infrastructure.

Switcher Studio’s whole business is live video — their iOS app allows creators and businesses to produce their own branded, multi camera live streams. Switcher uses Stream as an essential part of their live streaming infrastructure. In their own words:

“Since 2014, Switcher has helped creators connect to audiences with livestreams. Now, our users create over 100,000 streams per month. As we grew, we needed a scalable content delivery solution. Cloudflare offers secure, fast delivery, and even allowed us to offer new features, like multistreaming. Trusting Cloudflare Stream lets our team focus on the live production tools that make Switcher unique.”

While Stream Live has been in beta, we’ve worked with many customers like Switcher, where live video isn’t just one product feature, it is the core of their product. Even as experts in live video, they choose to use Stream, so that they can focus on the unique value they create for their customers, leaving the infrastructure of ingesting, encoding, recording and delivering live video to Cloudflare.

Start building live video into your website or app today

It takes just a few minutes to sign up and start your first live stream, using the Cloudflare Dashboard, with no code required to get started, but APIs for when you’re ready to start building your own live video features. Give it a try — we’re ready for you, no matter the size of your audience.

Introducing Advanced DDoS Alerts

2022-09-19 Omer Yoachimik

Post Syndicated from Omer Yoachimik original https://blog.cloudflare.com/advanced-ddos-alerts/

Introducing Advanced DDoS Alerts

We’re pleased to introduce Advanced DDoS Alerts. Advanced DDoS Alerts are customizable and provide users the flexibility they need when managing many Internet properties. Users can easily define which alerts they want to receive — for which DDoS attack sizes, protocols and for which Internet properties.

This release includes two types of Advanced DDoS Alerts:

Advanced HTTP DDoS Attack Alerts – Available to WAF/CDN customers on the Enterprise plan, who have also subscribed to the Advanced DDoS Protection service.
Advanced L3/4 DDoS Attack Alerts – Available to Magic Transit and Spectrum BYOIP customers on the Enterprise plan.

Standard DDoS Alerts are available to customers on all plans, including the Free plan. Advanced DDoS Alerts are part of Cloudflare’s Advanced DDoS service.

Why alerts?

Distributed Denial of Service attacks are cyber attacks that aim to take down your Internet properties and make them unavailable for your users. As early as 2017, Cloudflare pioneered the Unmetered DDoS Protection to provide all customers with DDoS protection, without limits, to ensure that their Internet properties remain available. We’re able to provide this level of commitment to our customers thanks to our automated DDoS protection systems. But if the systems operate automatically, why even be alerted?

Well, to put it plainly, when our DDoS protection systems kick in, they insert ephemeral rules inline to mitigate the attack. Many of our customers operate business critical applications and services. When our systems make a decision to insert a rule, customers might want to be able to verify that all the malicious traffic is mitigated, and that legitimate user traffic is not. Our DDoS alerts begin firing as soon as our systems make a mitigation decision. Therefore, by informing our customers about a decision to insert a rule in real time, they can observe and verify that their Internet properties are both protected and available.

Managing many Internet properties

The standard DDoS Alerts alert you on DDoS attacks that target any and all of your Cloudflare-protected Internet properties. However, some of our customers may manage large numbers of Internet properties ranging from hundreds to hundreds of thousands. The standard DDoS Alerts would notify users every time one of those properties would come under attack — which could become very noisy.

The Advanced DDoS Alerts address this concern by allowing users to select the specific Internet properties that they want to be notified about; zones and hostnames for WAF/CDN customers, and IP prefixes for Magic Transit and Spectrum BYOIP customers.

One (attack) size doesn’t fit all

The standard DDoS Alerts alert you on DDoS attacks of any size. Well, almost any size. We implemented minimal alert thresholds to avoid spamming our customers’ email inboxes. Those limits are very small and not customer-configurable. As we’ve seen in the recent DDoS trends report, most of the attacks are very small — another reason why the standard DDoS Alert could become noisy for customers that only care about very large attacks. On the opposite end of the spectrum, choosing not to alert may become too quiet for customers that do want to be notified about smaller attacks.

The Advanced DDoS Alerts let customers choose their own alert threshold. WAF/CDN customers can define the minimum request-per-second rate of an HTTP DDoS attack alert. Magic Transit and Spectrum BYOIP customers can define the packet-per-second and Megabit-per-second rates of a L3/4 DDoS attack alert.

Not all protocols are created equal

As part of the Advanced L3/4 DDoS Alerts, we also let our users define the protocols to be alerted on. If a Magic Transit customer manages mostly UDP applications, they may not care if TCP-based DDoS attacks target it. Similarly, if a Spectrum BYOIP customer only cares about HTTP/TCP traffic, other-protocol-based attacks could be of no concern to them.

Creating an Advanced DDoS Alert

We’ll show here how to create an Advanced HTTP DDoS Alert, but the process to create a L3/4 alert is similar. You can view a more detailed guide on our developers website.

First, click here or log in to your Cloudflare account, navigate to Notifications and click Add. Then select the Advanced HTTP DDoS Attack Alert or Advanced L3/4 DDoS Attack Alert (based on your eligibility). Give your alert a name, an optional description, add your preferred delivery method (e.g., Webhook) and click Next.

Second, select the domains you’d like to be alerted on. You can also narrow it down to specific hostnames. Define the minimum request-per-second rate to be alerted on, click Save, and voilà.

Actionable alerts for making better decisions

Cloudflare Advanced DDoS Alerts aim to provide our customers with configurable controls to make better decisions for their own environments. Customers can now be alerted on attacks based on which domain/prefix is being attacked, the size of the attack, and the protocol of the attack. We recognize that the power to configure and control DDoS attack alerts should ultimately be left up to our customers, and we are excited to announce the availability of this functionality.

Want to learn more about Advanced DDoS Alerts? Visit our developer site.

Interested in upgrading to get Advanced DDoS Alerts? Contact your account team.

New to Cloudflare? Speak to a Cloudflare expert.

Introducing Cloudflare Adaptive DDoS Protection – our new traffic profiling system for mitigating DDoS attacks

2022-09-19 Omer Yoachimik

Post Syndicated from Omer Yoachimik original https://blog.cloudflare.com/adaptive-ddos-protection/

Introducing Cloudflare Adaptive DDoS Protection - our new traffic profiling system for mitigating DDoS attacks

Every Internet property is unique, with its own traffic behaviors and patterns. For example, a website may only expect user traffic from certain geographies, and a network might only expect to see a limited set of protocols.

Understanding that the traffic patterns of each Internet property are unique is what led us to develop the Adaptive DDoS Protection system. Adaptive DDoS Protection joins our existing suite of automated DDoS defenses and takes it to the next level. The new system learns your unique traffic patterns and adapts to protect against sophisticated DDoS attacks.

Adaptive DDoS Protection is now generally available to Enterprise customers:

HTTP Adaptive DDoS Protection – available to WAF/CDN customers on the Enterprise plan, who have also subscribed to the Advanced DDoS Protection service.
L3/4 Adaptive DDoS Protection – available to Magic Transit and Spectrum customers on an Enterprise plan.

Adaptive DDoS Protection learns your traffic patterns

The Adaptive DDoS Protection system creates a traffic profile by looking at a customer’s maximal rates of traffic every day, for the past seven days. The profiles are recalculated every day using the past seven-day history. We then store the maximal traffic rates seen for every predefined dimension value. Every profile uses one dimension and these dimensions include the source country of the request, the country where the Cloudflare data center that received the IP packet is located, user agent, IP protocol, destination ports and more.

So, for example, for the profile that uses the source country as a dimension, the system will log the maximal traffic rates seen per country. e.g. 2,000 requests per second (rps) for Germany, 3,000 rps for France, 10,000 rps for Brazil, and so on. This example is for HTTP traffic, but Adaptive DDoS protection also profiles L3/4 traffic for our Magic Transit and Spectrum Enterprise customers.

Another note on the maximal rates is that we use the 95th percentile rates. This means that we take a look at the maximal rates and discard the top 5% of the highest rates. The purpose of this is to eliminate outliers from the calculations.

Calculating traffic profiles is done asynchronously — meaning that it does not induce any latency to our customers’ traffic. The system then distributes a compact profile representation across our network that can be consumed by our DDoS protection systems to be used to detect and mitigate DDoS attacks in a much more cost-efficient manner.

In addition to the traffic profiles, the Adaptive DDoS Protection also leverages Cloudflare’s Machine Learning generated Bot Scores as an additional signal to differentiate between user and automated traffic. The purpose of using these scores is to differentiate between legitimate spikes in user traffic that deviates from the traffic profile, and a spike of automated and potentially malicious traffic.

Out of the box and easy to use

Adaptive DDoS Protection just works out of the box. It automatically creates the profiles, and then customers can tweak and tune the settings as they need via DDoS Managed Rules. Customers can change the sensitivity level, leverage expression fields to create overrides (e.g. exclude this type of traffic), and change the mitigation action to tailor the behavior of the system to their specific needs and traffic patterns.

Adaptive DDoS Protection complements the existing DDoS protection systems which leverages dynamic fingerprinting to detect and mitigate DDoS attacks. The two work in tandem to protect our customers from DDoS attacks. When Cloudflare customers onboard a new Internet property to Cloudflare, the dynamic fingerprinting protects them automatically and out of the box — without requiring any user action. Once the Adaptive DDoS Protection learns their legitimate traffic patterns and creates a profile, users can turn it on to provide an extra layer of protection.

Rules included as part of the Adaptive DDoS Protection

As part of this release, we’re pleased to announce the following capabilities as part of Cloudflare’s Adaptive DDoS Protection:

Profiling Dimension	Availability
Profiling Dimension	WAF/CDN customers on the Enterprise plan with Advanced DDoS	Magic Transit & Spectrum Enterprise customers
Origin errors	✅	❌
Client IP Country & region	✅	Coming soon
User Agent (globally, not per customer*)	✅	❌
IP Protocol	❌	✅
Combination of IP Protocol and Destination Port	❌	Coming soon

^*The User-Agent-aware feature analyzes, learns and profiles all the top user agents that we see across the Cloudflare network. This feature helps us identify DDoS attacks that leverage legacy or wrongly configured user agents.

Excluding UA-aware DDoS Protection, Adaptive DDoS Protection rules are deployed in Log mode. Customers can observe the traffic that’s flagged, tweak the sensitivity if needed, and then deploy the rules in mitigation mode. You can follow the steps outlined in this guide to do so.

Making the impact of DDoS attacks a thing of the past

Our mission at Cloudflare is to help build a better Internet. The DDoS Protection team’s vision is derived from this mission: our goal is to make the impact of DDoS attacks a thing of the past. Cloudflare’s Adaptive DDoS Protection takes us one step closer to achieving that vision: making Cloudflare’s DDoS protection even more intelligent, sophisticated, and tailored to our customer’s unique traffic patterns and individual needs.

Want to learn more about Cloudflare’s Adaptive DDoS Protection? Visit our developer site.

Interested in upgrading to get access to Adaptive DDoS Protection? Contact your account team.

New to Cloudflare? Speak to a Cloudflare expert.

Using Cloudflare R2 as an apt/yum repository

2022-09-15 Sudarsan Reddy

Post Syndicated from Sudarsan Reddy original https://blog.cloudflare.com/using-cloudflare-r2-as-an-apt-yum-repository/

Using Cloudflare R2 as an apt/yum repository

In this blog post, we’re going to talk about how we use Cloudflare R2 as an apt/yum repository to bring cloudflared (the Cloudflare Tunnel daemon) to your Debian/Ubuntu and CentOS/RHEL systems and how you can do it for your own distributable in a few easy steps!

I work on Cloudflare Tunnel, a product which enables customers to quickly connect their private networks and services through the Cloudflare global network without needing to expose any public IPs or ports through their firewall. Cloudflare Tunnel is managed for users by cloudflared, a tool that runs on the same network as the private services. It proxies traffic for these services via Cloudflare, and users can then access these services securely through the Cloudflare network.

Our connector, cloudflared, was designed to be lightweight and flexible enough to be effectively deployed on a Raspberry Pi, a router, your laptop, or a server running on a data center with applications ranging from IoT control to private networking. Naturally, this means cloudflared comes built for a myriad of operating systems, architectures and package distributions: You could download the appropriate package from our GitHub releases, brew install it or apt/yum install it (https://pkg.cloudflare.com).

In the rest of this blog post, I’ll use cloudflared as an example of how to create an apt/yum repository backed by Cloudflare’s object storage service R2. Note that this can be any binary/distributable. I simply use cloudflared as an example because this is something we recently did and actually use in production.

How apt-get works

Let’s see what happens when you run something like this on your terminal.

$ apt-get install cloudflared

Let’s also assume that apt sources were already added like so:

  $ echo 'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared buster main' | sudo tee /etc/apt/sources.list.d/cloudflared.list


$ apt-get update

From the source.list above, apt first looks up the Release file (or InRelease if it’s a signed package like cloudflared, but we will ignore this for the sake of simplicity).

A Release file contains a list of supported architectures and their md5, sha1 and sha256 checksums. It looks something like this:

$ curl https://pkg.cloudflare.com/cloudflared/dists/buster/Release
Origin: cloudflared
Label: cloudflared
Codename: buster
Date: Thu, 11 Aug 2022 08:40:18 UTC
Architectures: amd64 386 arm64 arm armhf
Components: main
Description: apt repository for cloudflared - buster
MD5Sum:
 c14a4a1cbe9437d6575ae788008a1ef4 549 main/binary-amd64/Packages
 6165bff172dd91fa658ca17a9556f3c8 374 main/binary-amd64/Packages.gz
 9cd622402eabed0b1b83f086976a8e01 128 main/binary-amd64/Release
 5d2929c46648ea8dbeb1c5f695d2ef6b 545 main/binary-386/Packages
 7419d40e4c22feb19937dce49b0b5a3d 371 main/binary-386/Packages.gz
 1770db5634dddaea0a5fedb4b078e7ef 126 main/binary-386/Release
 b0f5ccfe3c3acee38ba058d9d78a8f5f 549 main/binary-arm64/Packages
 48ba719b3b7127de21807f0dfc02cc19 376 main/binary-arm64/Packages.gz
 4f95fe1d9afd0124a32923ddb9187104 128 main/binary-arm64/Release
 8c50559a267962a7da631f000afc6e20 545 main/binary-arm/Packages
 4baed71e49ae3a5d895822837bead606 372 main/binary-arm/Packages.gz
 e472c3517a0091b30ab27926587c92f9 126 main/binary-arm/Release
 bb6d18be81e52e57bc3b729cbc80c1b5 549 main/binary-armhf/Packages
 31fd71fec8acc969a6128ac1489bd8ee 371 main/binary-armhf/Packages.gz
 8fbe2ff17eb40eacd64a82c46114d9e4 128 main/binary-armhf/Release
SHA1:
…
SHA256:
…

Depending on your system’s architecture, Debian picks the appropriate Packages location. A Packages file contains metadata about the binary apt wants to install, including location and its checksum. Let’s say it’s an amd64 machine. This means we’ll go here next:

$ curl https://pkg.cloudflare.com/cloudflared/dists/buster/main/binary-amd64/Packages
Package: cloudflared
Version: 2022.8.0
License: Apache License Version 2.0
Vendor: Cloudflare
Architecture: amd64
Maintainer: Cloudflare <[email protected]>
Installed-Size: 30736
Homepage: https://github.com/cloudflare/cloudflared
Priority: extra
Section: default
Filename: pool/main/c/cloudflared/cloudflared_2022.8.0_amd64.deb
Size: 15286808
SHA256: c47ca10a3c60ccbc34aa5750ad49f9207f855032eb1034a4de2d26916258ccc3
SHA1: 1655dd22fb069b8438b88b24cb2a80d03e31baea
MD5sum: 3aca53ccf2f9b2f584f066080557c01e
Description: Cloudflare Tunnel daemon

Notice the Filename field. This is where apt gets the deb from before running a dpkg command on it. What all of this means is the apt repository (and the yum repositories too) is basically a structured file system of mostly plaintext files and a deb.

The deb file is a Debian software package that contains two things: installable data (in our case, the cloudflared binary) and metadata about the installable.

Building our own apt repository

Now that we know what happens when an apt-get install runs, let’s work our way backwards to construct the repository.

Create a deb file out of the binary

Note: It is optional but recommended one signs the packages. See the section about how apt verifies packages here.

Debian files can be built by the dpkg-buildpackage tool in a linux or Debian environment. We use a handy command line tool called fpm (https://fpm.readthedocs.io/en/v1.13.1/) to do this because it works for both rpm and deb.

$ fpm -s <dir> -t deb -C /path/to/project -name <project_name> –version <version>

This yields a .deb file.

Create plaintext files needed by apt to lookup downloads

Again, these files can be built by hand, but there are multiple tools available to generate this:

We use reprepro.

Using it is as simple as

$ reprepro buster includedeb <path/to/the/deb>

reprepro neatly creates a bunch of folders in the structure we looked into above.

Upload them to Cloudflare R2

We use Cloudflare R2 to now be the host for this structured file system. R2 lets us upload and serve objects in this structured format. This means, we just need to upload the files in the same structure reprepro created them.

Here is a copyable example of how we do it for cloudflared.

Serve them from an R2 worker

For fine-grained control, we’ll write a very lightweight Cloudflare Worker to be the service we talk to and serve as the front-end API for apt to talk to. For an apt repository, we only need it to perform the GET function.

Here’s an example on how-to do this: https://developers.cloudflare.com/r2/examples/demo-worker/

Putting it all together

Here is a handy script we use to push cloudflared to R2 ready for apt/yum downloads and includes signing and publishing the pubkey.

And that’s it! You now have your own apt/yum repo without needing a server that needs to be maintained, there are no egress fees for downloads, and it is on the Cloudflare global network, protecting it against high request volumes. You can automate many of these steps to make it part of a release process.

Today, this is how cloudflared is distributed on the apt and yum repositories: https://pkg.cloudflare.com/

Introducing thresholds in Security Event Alerting: a z-score love story

2022-08-30 Kristina Galicova

Post Syndicated from Kristina Galicova original https://blog.cloudflare.com/introducing-thresholds-in-security-event-alerting-a-z-score-love-story/

Introducing thresholds in Security Event Alerting: a z-score love story

Today we are excited to announce thresholds for our Security Event Alerts: a new and improved way of detecting anomalous spikes of security events on your Internet properties. Previously, our calculations were based on z-score methodology alone, which was able to determine most of the significant spikes. By introducing a threshold, we are able to make alerts more accurate and only notify you when it truly matters. One can think of it as a romance between the two strategies. This is the story of how they met.

Author’s note: as an intern at Cloudflare I got to work on this project from start to finish from investigation all the way to the final product.

Once upon a time

In the beginning, there were Security Event Alerts. Security Event Alerts are notifications that are sent whenever we detect a threat to your Internet property. As the name suggests, they track the number of security events, which are requests to your application that match security rules. For example, you can configure a security rule that blocks access from certain countries. Every time a user from that country tries to access your Internet property, it will log as a security event. While a security event may be harmless and fired as a result of the natural flow of traffic, it is important to alert on instances when a rule is fired more times than usual. Anomalous spikes of too many security events in a short period of time can indicate an attack. To find these anomalies and distinguish between the natural number of security events and that which poses a threat, we need a good strategy.

The lonely life of a z-score

Before a threshold entered the picture, our strategy worked only on the basis of a z-score. Z-score is a methodology that looks at the number of standard deviations a certain data point is from the mean. In our current configuration, if a spike crosses the z-score value of 3.5, we send you an alert. This value was decided on after careful analysis of our customers’ data, finding it the most effective in determining a legitimate alert. Any lower and notifications will get noisy for smaller spikes. Any higher and we may miss out on significant events. You can read more about our z-score methodology in this blog post.

The following graphs are an example of how the z-score method works. The first graph shows the number of security events over time, with a recent spike.

To determine whether this spike is significant, we calculate the z-score and check if the value is above 3.5:

As the graph shows, the deviation is above 3.5 and so an alert is triggered.

However, relying on z-score becomes tricky for domains that experience no security events for a long period of time. With many security events at zero, the mean and standard deviation depress to zero as well. When a non-zero value finally appears, it will always be infinite standard deviations away from the mean. As a result, it will always trigger an alert even on spikes that do not pose any threat to your domain, such as the below:

With five security events, you are likely going to ignore this spike, as it is too low to indicate a meaningful threat. However, the z-score in this instance will be infinite:

Since a z-score of infinity is greater than 3.5, an alert will be triggered. This means that customers with few security events would often be overwhelmed by event alerts that are not worth worrying about.

Letting go of zeros

To avoid the mean and standard deviation becoming zero and thus alerting on every non-zero spike, zero values can be ignored in the calculation. In other words, to calculate the mean and standard deviation, only data points that are higher than zero will be considered.

With those conditions, the same spike to five security events will now generate a different z-score:

Great! With the z-score at zero, it will no longer trigger an alert on the harmless spike!

But what about spikes that could be harmful? When calculations ignore zeros, we need enough non-zero data points to accurately determine the mean and standard deviation. If only one non-zero value is present, that data point determines the mean and standard deviation. As such, the mean will always be equal to the spike, z-score will always be zero and an alert will never be triggered:

For a spike of 1000 events, we can tell that there is something wrong and we should trigger an alert. However, because there is only one non-zero data point, the z-score will remain zero:

The z-score does not cross the value 3.5 and an alert will not be triggered.

So what’s better? Including zeros in our calculations can skew the results for domains with too many zero events and alert them every time a spike appears. Not including zeros is mathematically wrong and will never alert on these spikes.

Threshold, the prince charming

Clearly, a z-score is not enough on its own.

Instead, we paired up the z-score with a threshold. The threshold represents the raw number of security events an Internet property can have, below which an alert will not be sent. While z-score checks whether the spike is at least 3.5 standard deviations above the mean, the threshold makes sure it is above a certain static value. If both of these conditions are met, we will send you an alert:

The above spike crosses the threshold of 200 security events. We now have to check that the z-score is above 3.5:

The z-score value crosses 3.5 and an alert will be sent.

A threshold for the number of security events comes as the perfect complement. By itself, the threshold cannot determine whether something is a spike, and would simply alert on any value crossing it. This blog post describes in more detail why thresholds alone do not work. However, when paired with z-score, they are able to share their strengths and cover for each other’s weaknesses. If the z-score falsely detects an insignificant spike, the threshold will stop the alert from triggering. Conversely, if a value does cross the security events threshold, the z-score ensures there is a reasonable variance from the data average before allowing an alert to be sent.

The invaluable value

To foster a successful relationship between the z-score and security events threshold, we needed to determine the most effective threshold value. After careful analysis of our previous attacks on customers, we set the value to 200. This number is high enough to filter out the smaller, noisier spikes, but low enough to expose any threats.

Am I invited to the wedding?

Yes, you are! The z-score and threshold relationship is already enabled for all WAF customers, so all you need to do is sit back and relax. For enterprise customers, the threshold will be applied to each type of alert enabled on your domain.

Happily ever after

The story certainly does not end here. We are constantly iterating on our alerts, so keep an eye out for future updates on the road to make our algorithms even more personalized for your Internet properties!

Crawler Hints supports Microsoft’s IndexNow in helping users find new content

2022-08-12 Alex Krivit

Post Syndicated from Alex Krivit original https://blog.cloudflare.com/crawler-hints-supports-microsofts-indexnow-in-helping-users-find-new-content/

Crawler Hints supports Microsoft’s IndexNow in helping users find new content

The web is constantly changing. Whether it’s news or updates to your social feed, it’s a constant flow of information. As a user, that’s great. But have you ever stopped to think how search engines deal with all the change?

It turns out, they “index” the web on a regular basis — sending bots out, to constantly crawl webpages, looking for changes. Today, bot traffic accounts for about 30% of total traffic on the Internet, and given how foundational search is to using the Internet, it should come as no surprise that search engine bots make up a large proportion of that what might come as a surprise is how inefficient the model is, though: we estimate that over 50% of crawler traffic is wasted effort.

This has a huge impact. There’s all the additional capacity that owners of websites need to bake into their site to absorb the bots crawling all over it. There’s the transmission of the data. There’s the CPU cost of running the bots. And when you’re running at the scale of the Internet, all of this has a pretty big environmental footprint.

Part of the problem, though, is nobody had really stopped to ask: maybe there’s a better way?

Right now, the model for indexing websites is the same as it has been since the 1990s: a “pull” model, where the search engine sends a crawler out to a website after a predetermined amount of time. During Impact Week last year, we asked: what about flipping the model on its head? What about moving to a push model, where a website could simply ping a search engine to let it know an update had been made?

There are a heap of advantages to such a model. The website wins: it’s not dealing with unnecessary crawls. It also makes sure that as soon as there’s an update to its content, it’s reflected in the search engine — it doesn’t need to wait for the next crawl. The website owner wins because they don’t need to manage distinct search engine crawl submissions. The search engine wins, too: it saves money on crawl costs, and it can make sure it gets the latest content.

Of course, this needs work to be done on both sides of the equation. The websites need a mechanism to alert the search engines; and the search engines need a mechanism to receive the alert, so they know when to do the crawl.

Crawler Hints — Cloudflare’s Solution for Websites

Solving this problem is why we launched Crawler Hints. Cloudflare sits in a unique position on the Internet — we’re serving on average 36 million HTTP requests per second. That represents a lot of websites. It also means we’re uniquely positioned to help solve this problem: to help give crawlers hints about when they should recrawl if new content has been added or if content on a site has recently changed.

With Crawler Hints, we send signals to web indexers based on cache data and origin status codes to help them understand when content has likely changed or been added to a site. The aim is to increase the number of relevant crawls as well as drastically reduce the number of crawls that don’t find fresh content, saving bandwidth and compute for both indexers and sites alike, and improving the experience of using the search engines.

But, of course, that’s just half the equation.

IndexNow Protocol — the Search Engine Moves from Pull to Push

Websites alerting the search engine about changes is useless if the search engines aren’t listening — and they simply continue to crawl the way they always have. Of course, search engines are incredibly complicated, and changing the way they operate is no easy task.

The IndexNow Protocol is a standard developed by Microsoft, Seznam.cz and Yandex, and it represents a major shift in the way search engines operate. Using IndexNow, search engines have a mechanism by which they can receive signals from Crawler Hints. Once they have that signal, they can shift their crawlers from a pull model to a push model.

In a recent update, Microsoft has announced that millions of websites are now using IndexNow to signal to search engine crawlers when their content needs to be crawled and IndexNow was used to index/crawl about 7% of all new URLs clicked when someone is selecting from web search results.

On the Cloudflare side, since the release of Crawler Hints in October 2021, Crawler Hints has processed about six-hundred-billion signals to IndexNow.

That’s a lot of saved crawls.

How to enable Crawler Hints

By enabling Crawler Hints on your website, with the simple click of a button, Cloudflare will take care of signaling to these search engines when your content has changed via the IndexNow API. You don’t need to do anything else!

Crawler Hints is free to use and available to all Cloudflare customers. If you’d like to see how Crawler Hints can benefit how your website is indexed by the world’s biggest search engines, please feel free to opt-into the service by:

Sign in to your Cloudflare Account.
In the dashboard, navigate to the Cache tab.
Click on the Configuration section.
Locate the Crawler Hints and enable.

Upon enabling Crawler Hints, Cloudflare will share when content on your site has changed and needs to be re-crawled with search engines using the IndexNow protocol (this blog can help if you’re interested in finding out more about how the mechanism works).

What’s Next?

Going forward, because the benefits are so substantial for site owners, search operators, and the environment, we plan to start defaulting Crawler Hints on for all our customers. We’re also hopeful that Google, the world’s largest search engine and most wasteful user of Internet resources, will adopt IndexNow or a similar standard and lower the burden of search crawling on the planet.

When we think of helping to build a better Internet, this is exactly what comes to mind: creating and supporting standards that make it operate better, greener, faster. We’re really excited about the work to date, and will continue to work to improve the signaling to ensure the most valuable information is being sent to the search engines in a timely manner. This includes incorporating additional signals such as etags, last-modified headers, and content hash differences. Adding these signals will help further inform crawlers when they should reindex sites, and how often they need to return to a particular site to check if it’s been changed. This is only the beginning. We will continue testing more signals and working with industry partners so that we can help crawlers run efficiently with these hints.

And finally: if you’re on Cloudflare, and you’d like to be part of this revolution in how search engines operate on the web (it’s free!), simply follow the instructions in the section above.

35,000 new trees in Nova Scotia

2022-07-13 Patrick Day

Post Syndicated from Patrick Day original https://blog.cloudflare.com/35-000-new-trees-in-nova-scotia/

35,000 new trees in Nova Scotia

Cloudflare is proud to announce the first 35,000 trees from our commitment to help clean up bad bots (and the climate) have been planted.

Working with our partners at One Tree Planted (OTP), Cloudflare was able to support the restoration of 20 hectares of land at Victoria Park in Nova Scotia, Canada. The 130-year-old natural woodland park is located in the heart of Truro, NS, and includes over 3,000 acres of hiking and biking trails through natural gorges, rivers, and waterfalls, as well as an old-growth eastern hemlock forest.

The planting projects added red spruce, black spruce, eastern white pine, eastern larch, northern red oak, sugar maple, yellow birch, and jack pine to two areas of the park. The first area was a section of the park that recently lost a number of old conifers due to insect attacks. The second was an area previously used as a municipal dump, which has since been covered by a clay cap and topsoil.

Our tree commitment began far from the Canadian woodlands. In 2019, we launched an ambitious tool called Bot Fight Mode, which for the first time fought back against bots, targeting scrapers and other automated actors.

Our idea was simple: preoccupy bad bots with nonsense tasks, so they cannot attack real sites. Even better, make these tasks computationally expensive to engage with. This approach is effective, but it forces bad actors to consume more energy and likely emit more greenhouse gasses (GHG). So in addition to launching Bot Fight Mode, we also committed to supporting tree planting projects to account for any potential environmental impact.

What is Bot Fight Mode?

As soon as Bot Fight Mode is enabled, it immediately starts challenging bots that visit your site. It is available to all Cloudflare customers for free, regardless of plan.

When Bot Fight Mode identifies a bot, it issues a computationally expensive challenge to exhaust it (also called “tarpitting”). Our aim is to disincentivize attackers, so they have to find a new hobby altogether. When we tarpit a bot, we require a significant amount of compute time that will stall its progress and result in a hefty server bill. Sorry not sorry.

We do this because bots are leeches. They draw resources, slow down sites, and abuse online platforms. They also hack into accounts and steal personal data. Of course, we allowlist a small number of bots that are well-behaved, like Slack and Google. And Bot Fight Mode only acts on traffic from cloud and hosting providers (because that is where bots usually originate from).

Over 550,000 sites use Bot Fight Mode today! We believe this makes it the most widely deployed bot management solution in the world (though this is impossible to validate). Free customers can enable the tool from the dashboard and paid customers can use a special version, known as Super Bot Fight Mode.

How many trees? Let’s do the math 🚀

Now, the hard part: how can we translate bot challenges into a specific number of trees that should be planted? Fortunately, we can use a series of unit conversions, similar to those we use to calculate Cloudflare’s total GHG emissions.

We started with the following assumptions.

Table 1.

Measure	Quantity	Scaled	Source
Energy used by a standard server	1,760.3 kWh / year	To hours (0.2 kWh / hour)	Go Climate
Emissions factor	0.33852 kgCO2e / kWh	To grams (338.52 gCO2e / kWh)	Go Climate
CO2 absorbed by a mature tree	48 lbsCO2e / year	To kilograms (21 kgCO2e / year)	One Tree Planted

Next, we selected a high-traffic day to model the rate and duration of bot challenges on our network. On May 23, 2021, Bot Fight Mode issued 2,878,622 challenges, which lasted an average of 50 seconds each. In total, bots spent 39,981 hours engaging with our network defenses, or more than four years of challenges in a single day!

We then converted that time value into kilowatt-hours (kWh) of energy based on the rate of power consumed by our generic server listed in Table 1 above.

39,981 (hours) x .2 (kWh/hour) = 7,996 (kWh)

Once we knew the total amount of energy consumed by bad bot servers, we used an emissions factor (the amount of greenhouse gasses emitted per unit of energy consumed) to determine total emissions.

7,996 (kwh) x 338.52 (gCO2e/kwh) = 2,706,805 (gCO2e)

If you have made it this far, clearly you like to geek out like we do, so for the sake of completeness, the unit commonly used in emissions calculations is carbon dioxide equivalent (CO2e), which is a composite unit for all six GHGs listed in the Kyoto Protocol weighted by Global Warming Potential.

The last conversion we needed was from emissions to trees. Our partners at OTP found that a mature tree absorbs roughly 21 kgCO2e per year. Based on our total emissions that translates to roughly 47,000 trees per server, or 840 trees per CPU core. However, in our original post, we also noted that given the time it takes for a newly planted tree to reach maturity, we would multiply our donation by a factor of 25.

In the end, over the first two years of the program, we calculated that we would need approximately 42,000 trees to account for all the individual CPU cores engaged in Bot Fight Mode. For good measure, we rounded up to an even 50,000.

We are proud that most of these trees are already in the ground, and we look forward to providing an update when the final 15,000 are planted.

A piece of the puzzle

“Planting trees will benefit species diversity of the existing forest, animal habitat, greening of reclamation areas as well as community recreation areas, and visual benefits along popular hiking/biking trail networks.”
– Stephanie Clement, One Tree Planted, Project Manager North America

Reforestation is an important part of protecting healthy ecosystems and promoting biodiversity. Trees and forests are also a fundamental part of helping to slow the growth of global GHG emissions.

However, we recognize there is no single solution to the climate crisis. As part of our mission to help build a better, more sustainable Internet, Cloudflare is investing in renewable energy, tools that help our customers understand and mitigate their own carbon footprints on our network, and projects that will help offset or remove historical emissions associated with powering our network by 2025.

Want to be part of our bots & trees effort? Enable Bot Fight Mode today! It’s available on our free plan and takes only a few seconds. By the time we made our first donation to OTP in 2021, Bot Fight Mode had already spent more than 3,000 years distracting bots.

Help us defeat bad bots and improve our planet today!

—-
For more information on Victoria Park, please visit https://www.victoriaparktruro.ca
For more information on One Tree Planted, please visit https://onetreeplanted.org
For more information on sustainability at Cloudflare, please visit www.cloudflare.com/impact

Waiting Room Event Scheduling protects your site during online events

2022-07-12 Arielle Olache

Post Syndicated from Arielle Olache original https://blog.cloudflare.com/waiting-room-event-scheduling/

Waiting Room Event Scheduling protects your site during online events

You’ve got big plans for your ecommerce strategy in the form of online events — seasonal sales, open registration periods, product drops, ticket sales, and more. With all the hype you’ve generated, you’ll get a lot of site traffic, and that’s a good thing! With Waiting Room Event Scheduling, you can protect your servers from being overloaded during your event while delivering a user experience that is unique to the occasion and consistent with your brand. Available now to enterprise customers with an advanced Waiting Room subscription, Event Scheduling allows you to plan changes to your waiting room’s settings and custom queueing page ahead of time, ensuring flawless execution of your online event.

More than always-on protection

We launched Waiting Room to protect our customers’ servers during traffic spikes. Waiting Room sends excess visitors to a virtual queue during traffic surges, letting visitors in dynamically as spots become available on your site. By automatically queuing traffic that exceeds your site’s capacity, Waiting Room protects your origin servers and your customer experience. Additionally, the Waiting Room’s queuing page can be customized to match the look and feel of your site so that your users never feel as though they have left your application, ensuring a seamless customer experience.

While many of our customers use Waiting Room as an always-on failsafe against potential traffic spikes, some customers use Waiting Room to manage traffic during time-boxed online events. While these events can undoubtedly result in traffic spikes that Waiting Room safeguards against, they present unique challenges for our customers.

In the lifecycle of an online event, various stages of the event generally require the settings of a waiting room to be updated. While each customer’s event requirements are unique, consider the following customer use cases. To prevent a mad rush to a landing page that could overwhelm their site ahead of an event, some customers want to queue early arrivers in the days or hours leading up to the event. During an event, some customers want to impose stricter limits on how long visitors have to browse and complete transactions to ensure that as many visitors as possible get a fair chance to partake. After an event has concluded, many customers want to offload event traffic, blocking access to the event pages while informing users that the event has ended.

For each of these scenarios, our customers want to communicate expectations to their users and customize the look and feel of their queuing page to ensure a seamless, on-brand user experience. Combine all the use cases in the example above into one timeline, and you’ve got at least three event stages that would require waiting room settings and queuing pages to be updated, all with perfect timing.

While these use cases were technically feasible with Waiting Room, they required on-the-spot updates to its configuration settings. This strategy was not ideal or practical when customers needed to be absolutely sure their waiting room would update in lockstep with the timeline of their event. In short, many customers needed to schedule changes to the behavior of their waiting room ahead of time. We built Event Scheduling to give our customers flexibility and control over when and how their waiting room settings change, ensuring that these changes will happen automatically as planned.

Introducing Waiting Room Event Scheduling

With Event Scheduling, you can schedule cascading changes to your waiting room ahead of time as discrete events. For each waiting room event, you can customize traffic thresholds, session duration, queuing method, and the content and styling of your queuing page. Refer to the Create Event API documentation, for a complete list of customizable event settings.

New Queuing Methods

Giving our customers the ability to schedule changes to a waiting room’s settings is a game-changer for customers with event-based Waiting Room requirements, but we didn’t stop there. We’ve also added two new queueing methods — Reject and Passthrough — to give our customers more options for controlling user flow before, during and after their online events.

In the example where customers wanted to offload site traffic after an event, the Reject queuing method would do just that! A waiting room with the Reject queuing method configured will offload traffic from your site, presenting users with a static, fully customizable HTML page. Conversely, the Passthrough queuing method allows all visitors unrestricted access to your site. Unlike simply disabling your waiting room to achieve this, Passthrough has the advantage of being scheduled to turn on or off ahead of time and providing user traffic stats through the Waiting Room status endpoint.

If you prefer to have the waiting room in a completely passive state, having the waiting room on and configured with the Passthrough queueing method allows you to turn on Queue All quickly. Queue All places all new site visitors in a queue, which is a life-saver in the case of unexpected site downtime or any other crisis. Before deactivating Queue All, you can see how many users are waiting to enter your site. Queue All also overrides any active waiting room event, giving you authoritative, fast control in an emergency.

Event Pre-queuing

As part of an event’s configuration, you can also enable a pre-queue, a virtual holding area for visitors that arrive at your event before its start time. Pre-queues add extra protection against traffic spikes that can cause your site to crash.

To illustrate how, imagine your customer is a devoted fan trying to get tickets to their favorite band’s upcoming concert. Ticket sales open in one hour, so they visit your sales page about ten minutes before sales open. There is a static landing page on your site where the ticket sales page will be. The fan starts refreshing the page in the minutes leading up to the start time, hoping to get access as soon as sales open. Now multiply that one hopeful concert-goer by many thousands of fans, and before your sale has even begun, your site is already overwhelmed and at risk of crashing. Having a pre-queue in place protects your site from this type of user activity that has the potential to overwhelm your site at a time when stakes are very high for your brand. And with the ability to fully customize the pre-queuing page, you can still generate the same excitement you would have with your event’s landing page.

Taking it a step further, you can elect to shuffle the pre-queue, randomly assigning users who reach your application during the pre-queue period a place in line when the event starts. If your event uses the First in First Out queuing method, randomizing your pre-queue can help promote fairness, especially if your pre-queuing period spans many time zones. Like the Random queuing method, implementing a randomized pre-queue neutralizes the advantage customers in earlier time zones have to grab a place in line ahead of the event’s start time. Ultimately, fairness for your event is unique to you and your customers’ perspectives and needs. With the order of entry options available for both the pre-queue and overflow queuing during your event, you have control over managing fairness of entry to align with your unique requirements.

Creating a waiting room event

Similarly to configuring a waiting room, scheduling events with Waiting Room is incredibly easy and requires no coding or application changes. You will first need to have a baseline waiting room configured. Then, you can schedule events for this waiting room from the Waiting Room dashboard. In the event creation workflow, you’ll indicate when you would like the event to start and end and configure an optional pre-queue.

Unless specified otherwise, your event will always inherit the configuration settings of its associated waiting room. That way, you only need to update the waiting room settings that you would like to change for the duration of the event. You can optionally create a queuing page for your users that is unique to the event and preview what your event queuing page will look like in different queuing states and browsers, ensuring that your end-user’s experience doesn’t result in garbled CSS or broken looking pages!

Before saving your event, you will be able to review your waiting room and event settings side by side, making it easy to verify how the behavior of your waiting room will change for the duration of the event.

Once created, your event will appear in the Waiting Room dashboard nested under its associated waiting room. The date of each waiting room’s next event is indicated in the dashboard’s default view so that you can tell at a glance if any waiting room’s settings may change due to an upcoming event. You can expand each waiting room’s row to list upcoming events and associated durations. Additionally, if an event is live, a green dot will appear next to this date, adding extra assurance that it has kicked in.

Event Scheduling in action

Tying it all together, let’s walk through a real-world scenario to demonstrate the versatility and practicality of Event Scheduling. In this example, we have a major women’s fashion retailer, let’s call them Shopflare, with an upcoming flash sale. A flash sale is an online sales event that is different from a regular sale in that it lasts for a brief time and offers substantial discounts or limited stock. Often, retailers target a specific audience using marketing campaigns in the days leading up to a flash sale.

In our example, Shopflare’s marketing team plans to send an email campaign to a set of target customers, promoting their Spring Flash Sale, where they will be offering free shipping and 40% off on their freshest spring arrivals for one day only! How could Shopflare use Waiting Room and Event Scheduling to help this sales event go off without a hitch?

Preparing for the flash sale

One week before the sale, Shopflare’s web team creates a landing page with a countdown for their spring flash sale at example.com/sales/spring_flash_sale. They place a waiting room at this URL with a First In First Out queuing method and their desired traffic thresholds to allow traffic while ensuring their site remains performant. They then send an email campaign to their target audience directly linking to the sale’s landing page. With their baseline waiting room in place, early traffic to the URL will not overwhelm their site. Shopflare’s team also prepares for the upcoming sale by scheduling two cascading waiting room events ahead of time. Let’s review Shopflare’s flash sale requirements related to Waiting Room and review the steps they would take with Event Scheduling to satisfy them.

Pre-queueing and event overflow queuing

A few hours before the sale starts, Shopflare wants to allow shoppers to start “lining up” to secure a spot ahead of those who arrive after the event start time. They want to create a lottery for these early arrivers, randomly assigning them a place in line when the sale starts to mitigate the advantage that customers from earlier time zones have to secure a spot in line. To do this, they would create an event for the waiting room they have already configured.

The event creation workflow consists of four main steps: Details, Settings, Customization, and Review. On the Details page of the event creation workflow, they would enter their sale start and end times, set the start time of the pre-queue and enable “Shuffle at Event Start” to create a randomized pre-queue.

While the sale is in progress, Shopflare wants an overflow queue to protect their site from being overwhelmed by traffic in excess of their waiting room limits, letting these users in First in First Out when spots open up on their site. Since their underlying waiting room is already configured with the traffic thresholds they want to enforce for the duration of the event, they would simply leave the Settings page of the event creation workflow unchanged and proceed to Customization.

On the Customization step, Shopflare will create a custom queuing experience for their sale by uploading a custom HTML template that contains the HTML for both their pre-queueing page and their overflow queue.

Shopflare wants their pre-queuing page to get shoppers excited about the beginning of the sale. They ensure it is branded and unique to the flash sale while setting clear expectations for shoppers. For their overflow queue, they want the same look and feel of their pre-queueing page, with updated messaging that gives shoppers an estimated wait time and explains the reason for queuing. Check out the two sample queuing pages below to see how they create a unique and informative experience for their queued customers in both the pre-queue and overflow queue.

Sale conclusion

Once the sale has ended, Shopflare wants to allow active shoppers a five-minute grace period to complete their purchases without admitting any more new visitors. For 48 hours post-sale, they would like to present all visitors with a static page letting them know the sale has concluded while providing a redirect link back to their homepage. To achieve this, Shopflare would create another event for the baseline waiting room that starts when the previous event ends without a pre-queue enabled.

To offload all new site traffic after the sale has ended while giving active shoppers a five-minute grace period, from the Settings page of the event creation workflow, they would set session duration to five minutes, disable session renewal and select the Reject All queuing method.

Once again, on the Customization tab, they would elect to override the underlying waiting room template with a custom event template and upload their custom Reject page HTML. They would then Review and save their event and it will appear along with their previously created event in the Waiting Room dashboard.

And that’s it! With their waiting room events in place, Shopflare can rest assured that their site will be protected and that their customers have an on-brand and transparent shopping experience on the big day. Each customer and online event is unique. However, you choose to manage your user traffic for your online event, Event Scheduling for Cloudflare Waiting Rooms offers the options necessary to deliver a stellar and fair user experience while protecting your application during your online event. We can’t wait to support you in your next online event!

For more on Event Scheduling and Waiting Room, check out our developer documentation.

…
We protect entire corporate networks, help customers build Internet-scale applications efficiently, accelerate any website or Internet application, ward off DDoS attacks, keep hackers at bay, and can help you on your journey to Zero Trust.

Visit 1.1.1.1 from any device to get started with our free app that makes your Internet faster and safer.To learn more about our mission to help build a better Internet, start here. If you’re looking for a new career direction, check out our open positions.

Introducing Location-Aware DDoS Protection

2022-07-11 Omer Yoachimik

Post Syndicated from Omer Yoachimik original https://blog.cloudflare.com/location-aware-ddos-protection/

Introducing Location-Aware DDoS Protection

We’re thrilled to introduce Cloudflare’s Location-Aware DDoS Protection.

Distributed Denial of Service (DDoS) attacks are cyber attacks that aim to make your Internet property unavailable by flooding it with more traffic than it can handle. For this reason, attackers usually aim to generate as much attack traffic as they can — from as many locations as they can. With Location-Aware DDoS Protection, we take this distributed characteristic of the attack, that is thought of being advantageous for the attacker, and turn it on its back — making it into a disadvantage.

Location-Aware DDoS Protection is now available in beta for Cloudflare Enterprise customers that are subscribed to the Advanced DDoS service.

Distributed attacks lose their edge

Cloudflare’s Location-Aware DDoS Protection takes the attacker’s advantage and uses it against them. By learning where your traffic comes from, the system becomes location-aware and constantly asks “Does it make sense for your website?” when seeing new traffic.

For example, if you operate an e-commerce website that mostly serves the German consumer, then most of your traffic would most likely originate from within Germany, some from neighboring European countries, and a decreasing amount as we expand globally to other countries and geographies. If sudden spikes of traffic arrive from unexpected locations outside your main geographies, the system will flag and mitigate the unwanted traffic.

Location-Aware DDoS Protection also leverages Cloudflare’s Machine Learning models to identify traffic that is likely automated. This is used as an additional signal to provide more accurate protection.

Enabling Location-Aware Protection

Enterprise customers subscribed to the Advanced DDoS service can customize and enable the Location-Aware DDoS Protection system. By default, the system will only show what it thinks is suspicious traffic based on your last 7-day P95 rates, bucketed by client country and region (recalculated every 24 hours).

Customers can view what the system flagged in the Security Overview dashboard.

Location-Aware DDoS Protection is exposed to customers as a new HTTP DDoS Managed rule within the existing ruleset. To enable it, change the action to Managed Challenge or Block. Customers can adjust its sensitivity level to define how much tolerance you permit for traffic that deviates from your observed geographies. The lower the sensitivity, the higher the tolerance.

To learn how to view flagged traffic and how to configure the Location-Aware DDoS Protection, visit our developer docs site.

Making the impact of DDoS attacks a thing of the past

Our mission at Cloudflare is to help build a better Internet. The DDoS Protection team’s vision is derived from this mission: our goal is to make the impact of DDoS attacks a thing of the past. Location-aware protection is only the first step to making Cloudflare’s DDoS protection even more intelligent, sophisticated, and tailored to individual needs.

Not using Cloudflare yet? Start now with our Free and Pro plans to protect your websites, or contact us to learn more about the Enterprise Advanced DDoS Protection package.

New WAF intelligence feeds

2022-07-07 Daniele Molteni

Post Syndicated from Daniele Molteni original https://blog.cloudflare.com/new-waf-intelligence-feeds/

New WAF intelligence feeds

Cloudflare is expanding our WAF’s threat intelligence capabilities by adding four new managed IP lists that can be used as part of any custom firewall rule.

Managed lists are created and maintained by Cloudflare and are built based on threat intelligence feeds collected by analyzing patterns and trends observed across the Internet. Enterprise customers can already use the Open SOCKS Proxy list (launched in March 2021) and today we are adding four new IP lists: “VPNs”, “Botnets, Command and Control Servers”, “Malware” and “Anonymizers”.

Customers can reference these lists when creating a custom firewall rule or in Advanced Rate Limiting. For example, you can choose to block all traffic generated by IPs we categorize as VPNs, or rate limit traffic generated by all Anonymizers. You can simply incorporate managed IP lists in the powerful firewall rule builder. Of course, you can also use your own custom IP list.

Where do these feeds come from?

These lists are based on Cloudflare-generated threat feeds which are made available as IP lists to be easily consumed in the WAF. Each IP is categorized by combining open source data as well as by analyzing the behavior of each IP leveraging the scale and reach of Cloudflare network. After an IP has been included in one of these feeds, we verify its categorization and feed this information back into our security systems and make it available to our customers in the form of a managed IP list. The content of each list is updated multiple times a day.

In addition to generating IP classifications based on Cloudflare’s internal data, Cloudflare curates and combines several data sources that we believe provide reliable coverage of active security threats with a low false positive rate. In today’s environment, an IP belonging to a cloud provider might today be distributing malware, but tomorrow might be a critical resource for your company.

Some IP address classifications are publicly available, OSINT data, for example Tor exit nodes, and Cloudflare takes care of integrating this into our Anonymizer list so that you don’t have to manage integrating this list into every asset in your network. Other classifications are determined or vetted using a variety of DNS techniques, like lookup, PTR record lookup, and observing passive DNS from Cloudflare’s network.

Our malware and command-and-control focused lists are generated from curated partnerships, and one type of IP address we target when we select partners is data sources that identify security threats that do not have DNS records associated with them.

Our Anonymizer list encompasses several types of services that perform anonymization, including VPNs, open proxies, and Tor nodes. It is a superset of the more narrowly focused VPN list (known commercial VPN nodes), and the Cloudflare Open Proxies list (proxies that relay traffic without requiring authentication).

In dashboard IP annotations

Using these lists to deploy a preventative security policy for these IPs is great, but what about knowing if an IP that is interacting with your website or application is part of a Botnet or VPN? We first released contextual information for Anonymizers as part of Security Week 2022, but we are now closing the circle by extending this feature to cover all new lists.

As part of Cloudflare’s threat intelligence feeds, we are exposing the IP category directly into the dashboard. Say you are investigating requests that were blocked by the WAF and that looked to be probing your application for known software vulnerabilities. If the source IP of these requests is matching with one of our feeds (for example part of a VPN), contextual information will appear directly on the analytics page.

This information can help you see patterns and decide whether you need to use the managed lists to handle the traffic from these IPs in a particular way, for example by creating a rate limiting rule that reduces the amount of requests these actors can perform over a period of time.

Who gets this?

The following table summarizes what plans have access to each one of these features. Any paying plans will have access to the contextual in-dash information, while Enterprise will be able to use different managed lists. Managed lists can be used only on Enterprise zones within an Enterprise account.

	FREE	PRO	BIZ	ENT	Advanced ENT *
Annotations	x	✅	✅	✅	✅
Open Proxies	x	x	x	✅	✅
Anonymizers	x	x	x	x	✅
VPNs	x	x	x	x	✅
Botnets, command and control	x	x	x	x	✅
Malware	x	x	x	x	✅

* Contact your customer success manager to learn how to get access to these lists.

Future releases

We are working on enriching our threat feeds even further. In the next months we are going to provide more IP lists, specifically we are looking into lists for cloud providers and Carrier-grade Network Address Translation (CG-NAT).

Connect to private network services with Browser Isolation

2022-06-24 Tim Obezuk

Post Syndicated from Tim Obezuk original https://blog.cloudflare.com/browser-isolation-private-network/

Connect to private network services with Browser Isolation

If you’re working in an IT organization that has relied on virtual desktops but looking to get rid of them, we have some good news: starting today, you can connect your users to your private network via isolated remote browsers. This means you can deliver sensitive internal web applications — reducing costs without sacrificing security.

Browser Isolation with private network connectivity enables your users to securely access private web services without installing any software or agents on an endpoint device or absorbing the management and cost overhead of serving virtual desktops. What’s even better: Browser Isolation is natively integrated into Cloudflare’s Zero Trust platform, making it easy to control and monitor who can access what private services from a remote browser without sacrificing performance or security.

Deprecating virtual desktops for web apps

The presence of virtual desktops in the workplace tells an interesting story about the evolution of deploying and securing enterprise applications. Serving a full virtual desktop to end-users is an expensive decision, each user requiring a dedicated virtual machine with multiple CPU cores and gigabytes of memory to run a full operating system. This cost was offset by the benefits of streamlining desktop app distribution and the security benefits of isolating unmanaged devices from the aging application.

Then the launch of Chromium/V8 surprised everyone by demonstrating that desktop-grade applications could be built entirely in web-based technologies. Today, a vast majority of applications — either SaaS or private — exist within a web browser. With most Virtual Desktop Infrastructure (VDI) users connecting to a remote desktop just to open a web browser, VDI’s utility for distributing applications is really no longer needed and has become a tremendously expensive way to securely host a web browser.

Browser Isolation with private network connectivity enables businesses to maintain the security benefits of VDI, without the costs of hosting and operating legacy virtual desktops.

Transparent end-user experience

But it doesn’t just have a better ROI. Browser Isolation also offers a better experience for your end-users, too. Serving web applications via virtual desktops is a clunky experience. Users first need to connect to their virtual desktop (either through a desktop application or web portal), open an embedded web browser. This model requires users to context-switch between local and remote web applications which adds friction, impacting user productivity.

With Browser Isolation users simply navigate to the isolated private application in their preferred web browser and use the service as if they were directly browsing the remote web browser.

How it works

Browser Isolation with private network connectivity works by unifying our Zero Trust products: Cloudflare Access and Cloudflare Tunnels.

Cloudflare Access authorizes your users via your preferred Identity Provider and connects them to a remote browser without installing any software on their device. Cloudflare Tunnels securely connects your private network to remote browsers hosted on Cloudflare’s network without opening any inbound ports on your firewall.

Monitor third-party users on private networks

Ever needed to give a contractor or vendor access to your network to remotely manage a web UI? Simply add the user to your Clientless Web Isolation policy, and they can connect to your internal service without installing any client software on their device. All requests to private IPs are filtered, inspected, and logged through Cloudflare Gateway.

Apply data protection controls

All traffic from remote browsers into your network is inspected and filtered. Data protection controls such as disabling clipboard, printing and file upload/downloads can be granularly applied to high-risk user groups and sensitive applications.

Get started

Connect your network to Cloudflare Zero Trust

It’s ridiculously easy to connect any network with outbound Internet access.

Engineers needing a web environment to debug and test services inside a private network just need to run a single command to connect their network to Browser Isolation using Cloudflare Tunnels.

Enable Clientless Web Isolation

Clientless Web Isolation allows users to connect to a remote browser without installing any software on the endpoint device. That means company-wide deployment is seamless and transparent to end users. Follow these steps to enable Clientless Web Isolation and define what users are allowed to connect to a remote browser.

Browse private IP resources

Now that you have your network connected to Cloudflare, and your users connected to remote browsers it’s easy for a user to connect to any RFC 1918 address in a remote browser. Simply navigate to your isolation endpoint, and you’ll be connected to your private network.

For example, if you want a user to manage a router hosted at http://192.0.2.1, prefix this URL with your isolation endpoint such as

https://<authdomain>.cloudflareaccess.com/browser/http://192.0.2.1

That’s it! Users are automatically served a remote browser in a nearby Cloudflare data center.

Remote browser connected to a private web service with data loss prevention policies enabled

Define policies

At this point, your users can connect to any private resource inside your network. You may want to further control what endpoints your users can reach. To do this, navigate to Gateway → Policies → HTTP and allow / block or apply data protection controls for any private resource based on identity or destination IP address. See our developer documentation for more information.

Additionally, isolation policies can be defined to control how users can interact with the remote browser to disable the clipboard, printing or file upload / downloads. See our developer documentation for more information.

Logging and visibility

Finally, all remote browser traffic is logged by the Secure Web Gateway. Navigate to Logs → Gateway → HTTP and filter by identity or destination IP address.

What’s next?

We’re excited to learn how people use Browser Isolation to enable remote access to private networks and protect sensitive apps. Like always, we’re just getting started so stay tuned for improvements on configuring remote browsers and deeper connectivity with Access applications. Click here to get started with Browser Isolation.

Announcing Gateway + CASB

2022-06-24 Corey Mahan

Post Syndicated from Corey Mahan original https://blog.cloudflare.com/announcing-gateway-and-casb/

Announcing Gateway + CASB

This post is also available in 简体中文, 日本語, Español.

Shadow IT and managing access to sanctioned or unsanctioned SaaS applications remain one of the biggest pain points for IT administrators in the era of the cloud.

We’re excited to announce that starting today, Cloudflare’s Secure Web Gateway and our new API-driven Cloud Access Security Broker (CASB) work seamlessly together to help IT and security teams go from finding Shadow IT to fixing it in minutes.

Detect security issues within SaaS applications

Cloudflare’s API-driven CASB starts by providing comprehensive visibility into SaaS applications, so you can easily prevent data leaks and compliance violations. Setup takes just a few clicks to integrate with your organization’s SaaS services, like Google Workspace and Microsoft 365. From there, IT and security teams can see what applications and services their users are logging into and how company data is being shared.

So you’ve found the issues. But what happens next?

Identify and detect, but then what?

Customer feedback from the API-driven CASB beta has followed a similar theme: it was super easy to set up and detect all my security issues, but how do I fix this stuff?

Almost immediately after investigating the most critical issues, it makes sense to want to start taking action. Whether it be detecting an unknown application being used for Shadow IT or wanting to limit functionality, access, or behaviors to a known but unapproved application, remediation is front of mind.

This led to customers feeling like they had a bunch of useful data in front of them, but no clear action to take to get started on fixing them.

Create Gateway policies from CASB security findings

To solve this problem, we’re allowing you to easily create Gateway policies from CASB security findings. Security findings are issues detected within SaaS applications that involve users, data at rest, and settings that are assigned a Low, Medium, High or Critical severity per integration.

Using the security findings from CASB allows for fine-grained Gateway policies which prevent future unwanted behavior while still allowing usage that aligns to company security policy. This means going from viewing a CASB security issue, like the use of an unapproved SaaS application, to preventing or controlling access in minutes. This seamless cross-product experience all happens from a single, unified platform.

For example, take the CASB Google Workspace security finding around third-party apps which detects sign-ins or other permission sharing from a user’s account. In just a few clicks, you can create a Gateway policy to block some or all of the activity, like uploads or downloads, to the detected SaaS application. This policy can be applied to some or all users, based on what access has been granted to the user’s account.

By surfacing the exact behavior with CASB, you can take swift and targeted action to better protect your organization with Gateway.

Get started today with the Cloudflare One

This post highlights one of the many ways the Cloudflare One suite of solutions work seamlessly together as a unified platform to find and fix security issues across SaaS applications.

Get started now with Cloudflare’s Secure Web Gateway by signing up here. Cloudflare’s API-driven CASB is in closed beta with new customers being onboarded each week. You can request access here to try out this exciting new cross-product feature.

Early Hints update: How Cloudflare, Google, and Shopify are working together to build a faster Internet for everyone

2022-06-23 Alex Krivit

Post Syndicated from Alex Krivit original https://blog.cloudflare.com/early-hints-performance/

Early Hints update: How Cloudflare, Google, and Shopify are working together to build a faster Internet for everyone

A few months ago, we wrote a post focused on a product we were building that could vastly improve page load performance. That product, known as Early Hints, has seen wide adoption since that original post. In early benchmarking experiments with Early Hints, we saw performance improvements that were as high as 30%.

Now, with over 100,000 customers using Early Hints on Cloudflare, we are excited to talk about how much Early Hints have improved page loads for our customers in production, how customers can get the most out of Early Hints, and provide an update on the next iteration of Early Hints we’re building.

What Are Early Hints again?

As a reminder, the browser you’re using right now to read this page needed instructions for what to render and what resources (like images, fonts, and scripts) need to be fetched from somewhere else in order to complete the loading of this (or any given) web page. When you decide you want to see a page, your browser sends a request to a server and the instructions for what to load come from the server’s response. These responses are generally composed of a multitude of resources that tell the browser what content to load and how to display it to the user. The servers sending these instructions to your browser often need time to gather up all of the resources in order to compile the whole webpage. This period is known as “server think time.” Traditionally, during the “server think time” the browser would sit waiting until the server has finished gathering all the required resources and is able to return the full response.

Early Hints was designed to take advantage of this “server think time” to send instructions to the browser to begin loading readily-available resources while the server finishes compiling the full response. Concretely, the server sends two responses: the first to instruct the browser on what it can begin loading right away, and the second is the full response with the remaining information. By sending these hints to a browser before the full response is prepared, the browser can figure out what it needs to do to load the webpage faster for the end user.

Early Hints uses the HTTP status code 103 as the first response to the client. The “hints” are HTTP headers attached to the 103 response that are likely to appear in the final response, indicating (with the Link header) resources the browser should begin loading while the server prepares the final response. Sending hints on which assets to expect before the entire response is compiled allows the browser to use this “think time” (when it would otherwise have been sitting idle) to fetch needed assets, prepare parts of the displayed page, and otherwise get ready for the full response to be returned.

Early Hints on Cloudflare accomplishes performance improvements in three ways:

By sending a response where resources are directed to be preloaded by the browser. Preloaded resources direct the browser to begin loading the specified resources as they will be needed soon to load the full page. For example, if the browser needs to fetch a font resource from a third party, that fetch can happen before the full response is returned, so the font is already waiting to be used on the page when the full response returns from the server.
By using preconnect to initiate a connection to places where content will be returned from an origin server. For example, if a Shopify storefront needs content from a Shopify origin to finish loading the page, preconnect will warm up the connection which improves the performance for when the origin returns the content.
By caching and emitting Early Hints on Cloudflare, we make an efficient use of the full waiting period – not just server think time – which includes transit latency to the origin. Cloudflare sits within 50 milliseconds of 95% of the Internet-connected population globally. So while a request is routed to an origin and the final response is being compiled, Cloudflare can send an Early Hint from much closer and the browser can begin loading.

Early Hints is like multitasking across the Internet – at the same time the origin is compiling resources for the final response and making calls to databases or other servers, the browser is already beginning to load assets for the end user.

What’s new with Early Hints?

While developing Early Hints, we’ve been fortunate to work with Google and Shopify to collect data on the performance impact. Chrome provided web developers with experimental access to both preload and preconnect support for Link headers in Early Hints. Shopify worked with us to guide the development by providing test frameworks which were invaluable to getting real performance data.

Today is a big day for Early Hints. Google announced that Early Hints is available in Chrome version 103 with support for preload and preconnect to start. Previously, Early Hints was available via an origin trial so that Chrome could measure the full performance benefit (A/B test). Now that the data has been collected and analyzed, and we’ve been able to prove a substantial improvement to page load, we’re excited that Chrome’s full support of Early Hints will mean that many more requests will see the performance benefits.

That’s not the only big news coming out about Early Hints. Shopify battle-tested Cloudflare’s implementation of Early Hints during Black Friday/Cyber Monday 2021 and is sharing the performance benefits they saw during the busiest shopping time of the year:

Today, HTTP 103 Early Hints ships with Chrome 103!

Why is this important for #webperf? How did @Shopify help make all merchant sites faster? (LCP over 500ms faster at p50!) 🧵

Hint: A little collaboration w/ @cloudflare & @googlechrome pic.twitter.com/Dz7BD4Jplp

— Colin Bendell (@colinbendell) June 21, 2022

While talking to the audience at Cloudflare Connect London last week, Colin Bendell, Director, Performance Engineering at Shopify summarized it best: “when a buyer visits a website, if that first page that (they) experience is just 10% faster, on average there is a 7% increase in conversion“. The beauty of Early Hints is you can get that sort of speedup easily, and with Smart Early Hints that can be one click away.

You can see his entire talk here:

The headline here is that during a time of vast uncertainty due to the global pandemic, a time when everyone was more online than ever before, when people needed their Internet to be reliably fast — Cloudflare, Google, and Shopify all came together to build and test Early Hints so that the whole Internet would be a faster, better, and more efficient place.

So how much did Early Hints improve performance of customers’ websites?

Performance Improvement with Early Hints

In our simple tests back in September, we were able to accelerate the Largest Contentful Paint (LCP) by 20-30%. Granted, this result was on an artificial page with mostly large images where Early Hints impact could be maximized. As for Shopify, we also knew their storefronts were particularly good candidates for Early Hints. Each mom-and-pop.shop page depends on many assets served from cdn.shopify.com – speeding up a preconnect to that host should meaningfully accelerate loading those assets.

But what about other zones? We expected most origins already using Link preload and preconnect headers to see at least modest improvements if they turned on Early Hints. We wanted to assess performance impact for other uses of Early Hints beyond Shopify’s.

However, getting good data on web page performance impact can be tricky. Not every 103 response from Cloudflare will result in a subsequent request through our network. Some hints tell the browser to preload assets on important third-party origins, for example. And not every Cloudflare zone may have Browser Insights enabled to gather Real User Monitoring data.

Ultimately, we decided to do some lab testing with WebPageTest of a sample of the most popular websites (top 1,000 by request volume) using Early Hints on their URLs with preload and preconnect Link headers. WebPageTest (which we’ve written about in the past) is an excellent tool to visualize and collect metrics on web page performance across a variety of device and connectivity settings.

Lab Testing

In our earlier blog post, we were mainly focused on Largest Contentful Paint (LCP), which is the time at which the browser renders the largest visible image or text block, relative to the start of the page load. Here we’ll focus on improvements not only to LCP, but also FCP (First Contentful Paint), which is the time at which the browser first renders visible content relative to the start of the page load.

We compared test runs with Early Hints support off and on (in Chrome), across four different simulated environments: desktop with a cable connection (5Mbps download / 28ms RTT), mobile with 3G (1.6Mbps / 300ms RTT), mobile with low-latency 3G (1.6Mbps / 150ms RTT) and mobile with 4G (9Mbps / 170ms RTT). After running the tests, we cleaned the data to remove URLs with no visual completeness metrics or less than five DOM elements. (These usually indicated document fragments vs. a page a user might actually navigate to.) This gave us a final sample population of a little more than 750 URLs, each from distinct zones.

In the box plots below, we’re comparing FCP and LCP percentiles between the timing data control runs (no Early Hints) and the runs with Early Hints enabled. Our sample population represents a variety of zones, some of which load relatively quickly and some far slower, thus the long whiskers and string of outlier points climbing the y-axis. The y-axis is constrained to the max p99 of the dataset, to ensure 99% of the data are reflected in the graph while still letting us focus on the p25 / p50 / p75 differences.

The relative shift in the box plot quantiles suggest we should expect modest benefits for Early Hints for the majority of web pages. By comparing FCP / LCP percentage improvement of the web pages from their respective baselines, we can quantify what those median and p75 improvements would look like:

A couple observations:

From the p50 values, we see that for 50% of web pages on desktop, Early Hints improved FCP by more than 9.47% and LCP by more than 6.03%. For the p75, or the upper 25%, FCP improved by more than 20.4% and LCP by more than 15.97%.
The sizable improvements in First Contentful Paint suggest many hints are for render-blocking assets (such as critical but dynamic stylesheets and scripts that can’t be embedded in the HTML document itself).
We see a greater percentage impact on desktop over cable and on mobile over 4G. In theory, the impact of Early Hints is bounded by the load time of the linked asset (i.e. ideally we could preload the entire asset before the browser requires it), so we might expect the FCP / LCP reduction to increase in step with latency. Instead, it appears to be the other way around. There could be many variables at play here – for example, the extra bandwidth the 4G connection provides seems to be more influential than the decreased latency between the two 3G connection settings. Likely that wider bandwidth pipe is especially helpful for URLs we observed that preloaded larger assets such as JS bundles or font files. We also found examples of pages that performed consistently worse on lower-grade connections (see our note on “over-hinting” below).
Quite a few sample zones cached their HTML pages on Cloudflare (~15% of the sample). For CDN cache hits, we’d expect Early Hints to be less influential on the final result (because the “server think time” is drastically shorter). Filtering them out from the sample, however, yielded almost identical relative improvement metrics.

The relative distributions between control and Early Hints runs, as well as the per-site baseline improvements, show us Early Hints can be broadly beneficial for use cases beyond Shopify’s. As suggested by the p75+ values, we also still find plenty of case studies showing a more substantial potential impact to LCP (and FCP) like the one we observed from our artificial test case, as indicated from these WebPageTest waterfall diagrams:

These diagrams show the network and rendering activity on the same web page (which, bucking the trend, had some of its best results over mobile – 3G settings, shown here) for its first ten resources. Compare the WebPageTest waterfall view above (with Early Hints disabled) with the waterfall below (Early Hints enabled). The first green vertical line in each indicates First Contentful Paint. The page configures Link preload headers for a few JS / CSS assets, as well as a handful of key images. When Early Hints is on, those assets (numbered 2 through 9 below) get a significant head start from the preload hints. In this case, FCP and LCP improved by 33%!

Early Hints Best Practices and Strategies for Better Performance

The effect of Early Hints can vary widely on a case-by-case basis. We noticed particularly successful zones had one or more of the following:

Preconnect Link headers to important third-party origins (e.g. an origin hosting the pages’ assets, or Google Fonts).
Preload Link headers for a handful of critical render-blocking resources.
Scripts and stylesheets split into chunks, enumerated in preload Links.
A preload Link for the LCP asset, e.g. the featured image on a blog post.

It’s quite possible these strategies are already familiar to you if you work on web performance! Essentially the best practices that apply to using Link headers or <link> elements in the HTML <head> also apply to Early Hints. That is to say: if your web page is already using preload or preconnect Link headers, using Early Hints should amplify those benefits.

A cautionary note here: while it may be safer to aggressively send assets in Early Hints versus Server Push (as the hints won’t arbitrarily send browser-cached content the way Server Push might), it is still possible to over-hint non-critical assets and saturate network bandwidth in a similar manner to overpushing. For example, one page in our sample listed well over 50 images in its 103 response (but not one of its render-blocking JS scripts). It saw improvements over cable, but was consistently worse off in the higher latency, lower bandwidth mobile connection settings.

Google has great guidelines for configuring Link headers at your origin in their blog post. As for emitting these Links as Early Hints, Cloudflare can take care of that for you!

How to enable on Cloudflare

To enable Early Hints on Cloudflare, simply sign in to your account and select the domain you’d like to enable it on.
Navigate to the Speed Tab of the dashboard.
Enable Early Hints.

Enabling Early Hints means that we will harvest the preload and preconnect Link headers from your origin responses, cache them, and send them as 103 Early Hints for subsequent requests so that future visitors will be able to gain an even greater performance benefit.

For more information about our Early Hints feature, please refer to our announcement post or our documentation.

Smart Early Hints update

In our original blog post, we also mentioned our intention to ship a product improvement to Early Hints that would generate the 103 on your behalf.

Smart Early Hints will generate Early Hints even when there isn’t a Link header present in the origin response from which we can harvest a 103. The goal is to be a no-code/configuration experience with massive improvements to page load. Smart Early Hints will infer what assets can be preloaded or prioritized in different ways by analyzing responses coming from our customer’s origins. It will be your one-button web performance guru completely dedicated to making sure your site is loading as fast as possible.

This work is still under development, but we look forward to getting it built before the end of the year.

Try it out!

The promise Early Hints holds has only started to be explored, and we’re excited to continue to build products and features and make the web performance reliably fast.

We’ll continue to update you along our journey as we develop Early Hints and look forward to your feedback (special thanks to the Cloudflare Community members who have already been invaluable) as we move to bring Early Hints to everyone.

Verify Apple devices with no installed software

2022-06-22 Kenny Johnson

Post Syndicated from Kenny Johnson original https://blog.cloudflare.com/private-attestation-token-device-posture/

Verify Apple devices with no installed software

One of the foundations of Zero Trust is determining if a user’s device is “healthy” — that it has its operating system up-to-date with the latest security patches, that it’s not jailbroken, that it doesn’t have malware installed, and so on. Traditionally, determining this has required installing software directly onto a user’s device.

Earlier this month, Cloudflare participated in the announcement of an open source standard called a Private Attestation Token. Device manufacturers who support the standard can now supply a Private Attestation Token with any request made by one of their devices. On the IT Administration side, Private Attestation Tokens means that security teams can verify a user’s device before they access a sensitive application — without the need to install any software or collect a user’s device data.

At WWDC 2022, Apple announced Private Attestation Tokens. Today, we’re announcing that Cloudflare Access will support verifying a Private Attestation token. This means that security teams that rely on Cloudflare Access can verify a user’s Apple device before they access a sensitive application — no additional software required.

Determining a “healthy” device

There are many solutions on the market that help security teams determine if a device is “healthy” and corporately managed. What the majority of these solutions have in common is that they require software to be installed directly on the user’s machine. This comes with challenges associated with client software including compatibility issues, version management, and end user support. Many companies have dedicated Mobile Device Management (MDM) tools to manage the software installed on employee machines.

MDM is a proven model, but it is also a challenge to manage — taking a dedicated team in many cases. What’s more, installing client or MDM software is not always possible for contractors, vendors or employees using personal machines. Security teams have to resort to VDI or VPN solutions for external users to securely access corporate applications.

How Private Attestation Tokens verify a device

Private Attestation Tokens leverage the Privacy Pass Protocol, which Cloudflare authored with major device manufacturers, to attest to a device’s health and integrity.

In order for Private Attestation Tokens to work, four parties agree to work in concert with a common framework to generate and exchange anonymous, unforgeable tokens. Without all four parties in the process, PATs won’t work.

An Origin. A website, application, or API that receives requests from a client. When a website receives a request to their origin, the origin must know to look for and request a token from the client making the request. For Cloudflare customers, Cloudflare acts as the origin (on behalf of customers) and handles the requesting and processing of tokens.
A Client. Whatever tool the visitor is using to attempt to access the Origin. This will usually be a web browser or mobile application. In our example, let’s say the client is a mobile Safari Browser.
An Attester. The Attester is who the client asks to prove something (i.e that a mobile device has a valid IMEI) before a token can be issued. In our example below, the Attester is Apple, the device vendor.
An Issuer. The issuer is the only one in the process that actually generates, or issues, a token. The Attester makes an API call to whatever Issuer the Origin has chosen to trust, instructing the Issuer to produce a token. In our case, Cloudflare will also be the Issuer.

We are then able to rely on the attestation from the device manufacturer as a form of validation that a device is in a “healthy” enough state to be allowed access to a sensitive application.

Checking device health without client software

Private Attestation Tokens do not require any additional software to be installed on the user’s device. This is because the “attestation” of device health and validity is attested directly by the device operating system’s manufacturer — in this case, Apple.

This means that a security team can use Cloudflare Access and Private Attestation Tokens to verify if a user is accessing from a “healthy” Apple device before allowing access to a sensitive corporate application. Some checks as part of the attestation include:

Is the device on the latest OS version?
Is the device jailbroken?
Is the window attempting to log in, in focus?
And much more.

Over time, we are working with other device manufacturers to expand device support and what is verified as part of the device attestation process. The attributes that are attested will also continue to expand over time, which means the device verification in Access will only strengthen.

In the next few months, we will move Private Attestation Support in Cloudflare Access to a closed beta. The first version will work for iOS devices and support will expand from there. The only change required will be an updated Access policy, no software will need to be installed. If you would like to be part of the beta program, sign up here today!

Introducing Private Network Discovery

2022-06-22 Abe Carryl

Post Syndicated from Abe Carryl original https://blog.cloudflare.com/introducing-network-discovery/

Introducing Private Network Discovery

With Cloudflare One, building your private network on Cloudflare is easy. What is not so easy is maintaining the security of your private network over time. Resources are constantly being spun up and down with new users being added and removed on a daily basis, making it painful to manage over time.

That’s why today we’re opening a closed beta for our new Zero Trust network discovery tool. With Private Network Discovery, our Zero Trust platform will now start passively cataloging both the resources being accessed and the users who are accessing them without any additional configuration required. No third party tools, commands, or clicks necessary.

To get started, sign-up for early access to the closed beta and gain instant visibility into your network today. If you’re interested in learning more about how it works and what else we will be launching in the future for general availability, keep scrolling.

One of the most laborious aspects of migrating to Zero Trust is replicating the security policies which are active within your network today. Even if you do have a point-in-time understanding of your environment, networks are constantly evolving with new resources being spun up dynamically for various operations. This results in a constant cycle to discover and secure applications which creates an endless backlog of due diligence for security teams.

That’s why we built Private Network Discovery. With Private Network Discovery, organizations can easily gain complete visibility into the users and applications that live on their network without any additional effort on their part. Simply connect your private network to Cloudflare, and we will surface any unique traffic we discover on your network to allow you to seamlessly translate them into Cloudflare Access applications.

Building your private network on Cloudflare

Building out a private network has two primary components: the infrastructure side, and the client side.

The infrastructure side of the equation is powered by Cloudflare Tunnel, which simply connects your infrastructure (whether that be a single application, many applications, or an entire network segment) to Cloudflare. This is made possible by running a simple command-line daemon in your environment to establish multiple secure, outbound-only links to Cloudflare. Simply put, Tunnel is what connects your network to Cloudflare.

On the other side of this equation, you need your end users to be able to easily connect to Cloudflare and, more importantly, your network. This connection is handled by our robust device agent, Cloudflare WARP. This agent can be rolled out to your entire organization in just a few minutes using your in-house MDM tooling, and it establishes a secure connection from your users’ devices to the Cloudflare network.

Now that we have your infrastructure and your users connected to Cloudflare, it becomes easy to tag your applications and layer on Zero Trust security controls to verify both identity and device-centric rules for each and every request on your network.

How it works

As we mentioned earlier, we built this feature to help your team gain visibility into your network by passively cataloging unique traffic destined for an RFC 1918 or RFC 4193 address space. By design, this tool operates in an observability mode whereby all applications are surfaced, but are tagged with a base state of “Unreviewed.”

The Network Discovery tool surfaces all origins within your network, defined as any unique IP address, port, or protocol. You can review the details of any given origin and then create a Cloudflare Access application to control access to that origin. It’s also worth noting that Access applications may be composed of more than one origin.

Let’s take, for example, a privately hosted video conferencing service, Jitsi. I’m using this example as our team actually uses this service internally to test our new features before pushing them into production. In this scenario, we know that our self-hosted instance of Jitsi lives at 10.0.0.1:443. However, as this is a video conferencing application, it communicates on both tcp:10.0.0.1:443 and udp:10.0.0.1:10000. Here we would select one origin and assign it an application name.

As a note, during the closed beta you will not be able to view this application in the Cloudflare Access application table. For now, these application names will only be reflected in the discovered origins table of the Private Network Discovery report. You will see them reflected in the Application name column exclusively. However, when this feature goes into general availability you’ll find all the applications you have created under Zero Trust > Access > Applications as well.

After you have assigned an application name and added your first origin, tcp:10.0.0.1:443, you can then follow the same pattern to add the other origin, udp:10.0.0.1:10000, as well. This allows you to create logical groupings of origins to create a more accurate representation of the resources on your network.

By creating an application, our Network Discovery tool will automatically update the status of these individual origins from “Unreviewed” to “In-Review.” This will allow your team to easily track the origin’s status. From there, you can drill further down to review the number of unique users accessing a particular origin as well as the total number of requests each user has made. This will help equip your team with the information it needs to create identity and device-driven Zero Trust policies. Once your team is comfortable with a given application’s usage, you can then manually update the status of a given application to be either “Approved” or “Unapproved”.

What’s next

Our closed beta launch is just the beginning. While the closed beta release supports creating friendly names for your private network applications, those names do not currently appear in the Cloudflare Zero Trust policy builder.

As we move towards general availability, our top priority will be making it easier to secure your private network based on what is surfaced by the Private Network Discovery tool. With the general availability launch, you will be able to create Access applications directly from your Private Network Discovery report, reference your private network applications in Cloudflare Access and create Zero Trust security policies for those applications, all in one singular workflow.

As you can see, we have exciting plans for this tool and will continue investing in Private Network Discovery in the future. If you’re interested in gaining access to the closed beta, sign-up here and be among the first users to try it out!

Introducing Cloudforce One: our new threat operations and research team

2022-06-21 Blake Darché

Post Syndicated from Blake Darché original https://blog.cloudflare.com/introducing-cloudforce-one-threat-operations-and-threat-research/

Introducing Cloudforce One: our new threat operations and research team

This post is also available in 简体中文, 日本語, Deutsch, Français and Español.

Meet our new threat operations and research team: Cloudforce One. While this team will publish research, that’s not its reason for being. Its primary objective: track and disrupt threat actors.

The security teams we speak with tell us the same thing: they’re inundated with reports from threat intelligence and security product vendors that do little to improve their actual security. The stories are indeed interesting, but they want deeper insights into the techniques and actors targeting their industry—but even more than that, they want to be protected against these threats with minimal to no involvement. That is the mission on which Cloudforce One will deliver.

Introducing Cloudforce One: our new threat operations and research team

This team is led by me, Blake Darché, Area 1’s co-founder and former head of Threat Intelligence. Before starting Area 1, which was acquired by Cloudflare earlier this year, I was a founding member of CrowdStrike’s services organization, and before that a Computer Network Exploitation Analyst at the National Security Agency (NSA). My career has focused on identifying and disrupting sophisticated nation-state sponsored cyber threats before they compromise enterprises and governments, and I’m excited to accelerate that work at Cloudflare.

The Cloudforce One team comprises analysts assigned to Threat Research, Malware and Vulnerability Research, and Threat Operations (i.e., disrupting actors once identified). Collectively, members of the team have tracked many of the most sophisticated cyber criminals on the Internet while at the National Security Agency and Area 1 Security, and have worked closely with similar organizations and governments to disrupt these threat actors. They’ve also been prolific in publishing “finished intel” reports on security topics of significant geopolitical importance, such as targeted attacks against governments, technology companies, the energy sector, and law firms, and have regularly briefed top organizations around the world on their efforts. Oh, and we’re growing the team, so please reach out if you’re interested.

How will Cloudforce One work?

First and foremost, the team will help protect all Cloudflare customers by working closely with our existing product, engineering, and security teams to improve our products based on tactics, techniques, and procedures (TTPs) observed in the wild. Customers will get better protection without having to take any action, and will be able to read a subset of research published on our blog and within the Cloudflare Security Center.

Additionally, enterprise customers who wish to receive one-on-one live briefings from the team, submit periodic inquiries for follow-up, and obtain early access to threat research, will soon be able to sign up for our new Threat Intelligence subscription. All other enterprise customers will be invited to join periodic group briefings.

Lastly, new capabilities within Security Center, such as access to historical threat data via API and threat pivoting features, will also be introduced by the dedicated threat intel engineering team paired with Cloudforce One.

Getting started

If you’re looking to benefit from the insights uncovered by Cloudforce One, there is nothing you need to do. But if you’re interested in receiving regular briefings from Cloudforce One tailored to your industry, contact your Customer Success manager today or fill out this form and someone will be in touch. Finally, if you’re interested in joining the team, check out our open job postings here.

Launching In-Line Data Loss Prevention

2022-06-20 Noelle Gotthardt

Post Syndicated from Noelle Gotthardt original https://blog.cloudflare.com/inline-data-loss-prevention/

Launching In-Line Data Loss Prevention

Data Loss Prevention (DLP) enables you to protect your data based on its characteristics — or what it is. Today, we are very excited to announce that Data Loss Prevention is arriving as a native part of the Cloudflare One platform. If you’re interested in early access, please see the bottom of this post!

In the process of building Cloudflare One’s DLP solution, we talked to customers of all sizes and across dozens of industries. We focused on learning about their experiences, what products they are using, and what solutions they lack. The answers revealed significant customer challenges and frustrations. We are excited to deliver a product to put those problems in the past — and to do so as part of a comprehensive Zero Trust solution.

Customers are struggling to understand their data flow

Some customers have been using DLP solutions in their organizations for many years. They have deployed endpoint agents, crafted custom rulesets, and created incident response pipelines. Some built homemade tools to trace credit card numbers on the corporate network or rulesets to track hundreds of thousands of exact data match hashes.

Meanwhile, other customers are brand new to the space. They have small, scrappy teams supporting many IT and security functions. They do not have readily available resources to allocate to DLP and do not want to deprioritize other work to get started.

Still, many told the same story: the meteoric rise of SaaS tools left them unsure of where their data is moving and living. The migration of data off of corporate servers and into the cloud resulted in a loss of visibility and control. Even teams with established data protection programs strive for better visibility on the network. They are all asking the same types of questions:

Where is the data going?
Are uploads and downloads moving to and from corporate or personal SaaS instances?
What applications are storing sensitive data?
Who has access to those applications?
Can we see and block large downloads from file repositories?

Many customers seem to feel as though they have fallen behind because they haven’t solved these problems — and yet many customers are reporting the exact same story. However, these struggles do not mean anyone is behind — just that a better solution is needed. This told us that building a DLP product was the right choice, but why build it within Cloudflare One?

How Data Loss Prevention ties in to Zero Trust

A Zero Trust network architecture is fundamentally designed to secure your data. By checking every attempt to access a protected app, machine, or remote desktop, your data is protected on the basis of identity and device posture. With DNS and HTTP filtering, your data is protected based on content category and reputation. By adding an API-driven CASB, your data is protected based on your applications’ configurations, too.

With each piece of the architecture, your data is protected based on a new identifier. The identifiers above help you understand: who accessed the data, who owned the device that accessed it, where the data went, and how the destination was configured. However, what was the data that was moved?

Data Loss Prevention enables you to protect your data based on its characteristics, or what it is. For example, sensitive or confidential data can be identified a number of ways, such as keywords, patterns, or file types. These indicators help you understand the information being transmitted across or out of the network.

With DLP embedded in Cloudflare One, you can combine these identifiers to create rules catered to your organization. You get to specify the who, how, where, and what that meets your needs. We aim to deliver a comprehensive, detailed understanding of your network and your data, as well as allow you to easily implement protection.

How It Works

First: Identify the Data

DLP Profiles are being added to the Zero Trust dashboard. These profiles are where you define what data you want to protect. You will be able to add keywords and craft regexes to identify the presence of sensitive data. Profiles for common detections, such as credit card numbers, will be provided by Cloudflare.

Next: Create an HTTP Policy

After configuring a DLP Profile, you can then create a Cloudflare Gateway HTTP policy to allow or block the sensitive data from leaving your organization. Gateway will parse and scan your HTTP traffic for strings matching the keywords or regexes specified in the DLP profile.

Why Cloudflare

We know DLP is a big challenge to do comprehensively, and at scale. Those are the types of problems we excel at. Our network securely delivers traffic to 95% of the world’s Internet connected population within 50ms. It also supports our market leading products that send and protect customer traffic at unimaginable speed and scale. We are using that powerful network and our experience solving problems like this to take on Data Loss Prevention, and we’re very excited by our results

Join the waitlist

We are launching a closed beta of our Data Loss Prevention product. If you’re interested in early access, you can join the waitlist today by filling out this form.

What’s next?

We’re just getting started with DLP! We already have many plans for growth and integration with other Cloudflare One products, such as Remote Browser Isolation.

Area 1 threat indicators now available in Cloudflare Zero Trust

2022-06-20 Jesse Kipp

Post Syndicated from Jesse Kipp original https://blog.cloudflare.com/phishing-threat-indicators-in-zero-trust/

Area 1 threat indicators now available in Cloudflare Zero Trust

Over the last several years, both Area 1 and Cloudflare built pipelines for ingesting threat indicator data, for use within our products. During the acquisition process we compared notes, and we discovered that the overlap of indicators between our two respective systems was smaller than we expected. This presented us with an opportunity: as one of our first tasks in bringing the two companies together, we have started bringing Area 1’s threat indicator data into the Cloudflare suite of products. This means that all the products today that use indicator data from Cloudflare’s own pipeline now get the benefit of Area 1’s data, too.

Area 1 built a data pipeline focused on identifying new and active phishing threats, which now supplements the Phishing category available today in Gateway. If you have a policy that references this category, you’re already benefiting from this additional threat coverage.

How Cloudflare identifies potential phishing threats

Cloudflare is able to combine the data, procedures and techniques developed independently by both the Cloudflare team and the Area 1 team prior to acquisition. Customers are able to benefit from the work of both teams across the suite of Cloudflare products.

Cloudflare curates a set of data feeds both from our own network traffic, OSINT sources, and numerous partnerships, and applies custom false positive control. Customers who rely on Cloudflare are spared the software development effort as well as the operational workload to distribute and update these feeds. Cloudflare handles this automatically, with updates happening as often as every minute.

Cloudflare is able to go beyond this and work to proactively identify phishing infrastructure in multiple ways. With the Area 1 acquisition, Cloudflare is now able to apply the adversary-focused threat research approach of Area1 across our network. A team of threat researchers track state-sponsored and financially motivated threat actors, newly disclosed CVEs, and current phishing trends.

Cloudflare now operates mail exchange servers for hundreds of organizations around the world, in addition to its DNS resolvers, Zero Trust suite, and network services. Each of these products generates data that is used to enhance the security of all of Cloudflare’s products. For example, as part of mail delivery, the mail engine performs domain lookups, scores potential phishing indicators via machine learning, and fetches URLs. Data which can now be used through Cloudflare’s offerings.

How Cloudflare Area 1 identifies potential phishing threats

The Cloudflare Area 1 team operates a suite of web crawling tools designed to identify phishing pages, capture phishing kits, and highlight attacker infrastructure. In addition, Cloudflare Area 1 threat models assess campaigns based on signals gathered from threat actor campaigns; and the associated IOCs of these campaign messages are further used to enrich Cloudflare Area 1 threat data for future campaign discovery. Together these techniques give Cloudflare Area 1 a leg up on identifying the indicators of compromise for an attacker prior to their attacks against our customers. As part of this proactive approach, Cloudflare Area 1 also houses a team of threat researchers that track state-sponsored and financially motivated threat actors, newly disclosed CVEs, and current phishing trends. Through this research, analysts regularly insert phishing indicators into an extensive indicator management system that may be used for our email product or any other product that may query it.

Cloudflare Area 1 also collects information about phishing threats during our normal operation as the mail exchange server for hundreds of organizations across the world. As part of that role, the mail engine performs domain lookups, scores potential phishing indicators via machine learning, and fetches URLs. For those emails found to be malicious, the indicators associated with the email are inserted into our indicator management system as part of a feedback loop for subsequent message evaluation.

How Cloudflare data will be used to improve phishing detection

In order to support Cloudflare products, including Gateway and Page Shield, Cloudflare has a data pipeline that ingests data from partnerships, OSINT sources, as well as threat intelligence generated in-house at Cloudflare. We are always working to curate a threat intelligence data set that is relevant to our customers and actionable in the products Cloudflare supports. This is our North star: what data can we provide that enhances our customer’s security without requiring our customers to manage the complexity of data, relationships, and configuration. We offer a variety of security threat categories, but some major focus areas include:

Malware distribution
Malware and Botnet Command & Control
Phishing,
New and newly seen domains

Phishing is a threat regardless of how the potential phishing link gets entry into an organization, whether via email, SMS, calendar invite or shared document, or other means. As such, detecting and blocking phishing domains has been an area of active development for Cloudflare’s threat data team since almost its inception.

Looking forward, we will be able to incorporate that work into Cloudflare Area 1’s phishing email detection process. Cloudflare’s list of phishing domains can help identify malicious email when those domains appear in the sender, delivery headers, message body or links of an email.

Threat actors have long had an unfair advantage — and that advantage is rooted in the knowledge of their target, and the time they have to set up specific campaigns against their targets. That dimension of time allows threat actors to set up the right infrastructure, perform reconnaissance, stage campaigns, perform test probes, observe their results, iterate, improve and then launch their ‘production’ campaigns. This precise element of time gives us the opportunity to discover, assess and proactively filter out campaign infrastructure prior to campaigns reaching critical mass. But to do that effectively, we need visibility and knowledge of threat activity across the public IP space.

With Cloudflare’s extensive network and global insight into the origins of DNS, email or web traffic, combined with Cloudflare Area 1’s datasets of campaign tactics, techniques, and procedures (TTPs), seed infrastructure and threat models — we are now better positioned than ever to help organizations secure themselves against sophisticated threat actor activity, and regain the advantage that for so long has been heavily weighted towards the bad guys.

If you’d like to extend Zero Trust to your email security to block advanced threats, contact your Customer Success manager, or request a Phishing Risk Assessment here.

Introducing browser isolation for email links to stop modern phishing threats

2022-06-20 Shalabh Mohan

Post Syndicated from Shalabh Mohan original https://blog.cloudflare.com/email-link-isolation/

Introducing browser isolation for email links to stop modern phishing threats

This post is also available in 简体中文, 日本語 and Español.

There is an implicit and unearned trust we place in our email communications. This realization — that an organization can’t truly have a Zero Trust security posture without including email — was the driving force behind Cloudflare’s acquisition of Area 1 Security earlier this year. Today, we have taken our first step in this exciting journey of integrating Cloudflare Area 1 email security into our broader Cloudflare One platform. Cloudflare Secure Web Gateway customers can soon enable Remote Browser Isolation (RBI) for email links, giving them an unmatched level of protection from modern multi-channel email-based attacks.

Research from Cloudflare Area 1 found that nearly 10% of all observed malicious attacks involved credential harvesters, highlighting that victim identity is what threat actors usually seek. While commodity phishing attacks are blocked by existing security controls, modern attacks and payloads don’t have a set pattern that can reliably be matched with a block or quarantine rule. Additionally, with the growth of multi-channel phishing attacks, an effective email security solution needs the ability to detect blended campaigns spanning email and Web delivery, as well as deferred campaigns that are benign at delivery time, but weaponized at click time.

When enough “fuzzy” signals exist, isolating the destination to ensure end users are secure is the most effective solution. Now, with the integration of Cloudflare Browser Isolation into Cloudflare Area 1 email security, these attacks can now be easily detected and neutralized.

Human error is human

Why do humans still click on malicious links? It’s not because they haven’t attended enough training sessions or are not conscious about security. It’s because they have 50 unread emails in their inbox, have another Zoom meeting to get to, or are balancing a four-year old on their shoulders. They are trying their best. Anyone, including security researchers, can fall for socially engineered attacks if the adversary is well-prepared.

If we accept that human error is here to stay, developing security workflows introduces new questions and goals:

How can we reduce, rather than eliminate, the likelihood of human error?
How can we reduce the impact of human error when, not if, it happens?
How can security be embedded into an employee’s existing daily workflows?

It’s these questions that we had in mind when we reached the conclusion that email needs to be a fundamental part of any Zero Trust platform. Humans make mistakes in email just as regularly — in fact, sometimes more so — as they make mistakes surfing the Web.

To block, or not to block?

For IT teams, that is the question they wrestle with daily to balance risk mitigation with user productivity. The SOC team wants IT to block everything risky or unknown, whereas the business unit wants IT to allow everything not explicitly bad. If IT decides to block risky or unknown links, and it results in a false positive, they waste time manually adding URLs to allow lists — and perhaps the attacker later pivots those URLs to malicious content anyway. If IT decides to allow risky or unknown sites, best case they waste time reimaging infected devices and resetting login credentials — but all too common, they triage the damage from a data breach or ransomware lockdown. The operational simplicity of enabling RBI with email — also known as email link isolation — saves the IT, SOC, and business unit teams significant time.

How it works

For a Cloudflare Area 1 customer, the initial steps involve enabling RBI within your portal:

With email link isolation in place, here’s the short-lived life of an email with suspicious links:

Step 1: Cloudflare Area 1 inspects the email and determines that certain links in the messages are suspicious or on the margin

Step 2: Suspicious URLs and hyperlinks in the email get rewritten to a custom Cloudflare Area 1 prefix URL.

Step 3: The email is delivered to the intended inboxes.

Step 4: If a user clicks the link in the email, Cloudflare redirects to a remote browser via <authdomain>.cloudflareaccess.com/browser/{{url}}.

Step 5: Remote browser loads a website on a server on the Cloudflare Global Network and serves draw commands to the user’s clientless browser endpoint.

By executing the browser code and controlling user interactions on a remote server rather than a user device, any and all malware and phishing attempts are isolated, and won’t infect devices and compromise user identities. This improves both user and endpoint security when there are unknown risks and unmanaged devices, and allows users to access websites without having to connect to a VPN or having strict firewall policies.

Cloudflare’s RBI technology uses a unique patented technology called Network Vector Rendering (NVR) that utilizes headless Chromium-based browsers in the cloud, transparently intercepts draw layer output, transmits the draw commands efficiency and securely over the web, and redraws them in the windows of local HTML5 browsers. Unlike legacy Browser Isolation technologies that relied on pixel pushing or DOM reconstruction, NVR is optimized for scalability, security and end user transparency, while ensuring the broadest compatibility with websites.

A phishing attack before email link isolation

Let’s look at a specific example of a deferred phishing attack, how it slips past traditional defenses, and how email link isolation addresses the threat.

As organizations look to adopt new security principles and network architectures like Zero Trust, adversaries continually come up with techniques to bypass these controls by exploiting the most used and most vulnerable application – email. Email is a good candidate for compromise because of its ubiquity and ability to be bypassed pretty easily by a motivated attacker.

Let’s take an example of a “deferred phishing attack”, without email link isolation.

Attacker preparation: weeks before launch
The attacker sets up infrastructure for the phishing attempt to come. This may include:

Registering a domain
Encrypting it with SSL
Setting up proper email authentication (SPF, DKIM, DMARC)
Creating a benign web page

At this point, there is no evidence of an attack that can be picked up by secure email gateways, authentication-based solutions, or threat intelligence that relies solely on reputation-based signals and other deterministic detection techniques.

Attack “launch”: Sunday afternoon
The attacker sends an authentic-looking email from the newly-created domain. This email includes a link to the (still benign) webpage. There’s nothing in the email to block or flag it as suspicious. The email gets delivered to intended inboxes.

Attack launch: Sunday evening
Once the attacker is sure that all emails have reached their destination, they pivot the link to a malicious destination by changing the attacker-controlled webpage, perhaps by creating a fake login page to harvest credentials.

Attack landing: Monday morning
As employees scan their inboxes to begin their week, they see the email. Maybe not all of them click the link, but some of them do. Maybe not all of those that clicked enter their credentials, but a handful do. Without email link isolation, the attack is successful.

The consequences of the attack have also just begun – once user login credentials are obtained, attackers can compromise legitimate accounts, distribute malware to your organization’s network, steal confidential information, and cause much more downstream damage.

A phishing attack after email link isolation

The integration between Cloudflare Area 1 and Cloudflare Browser Isolation provides a critical layer of post-delivery protection that can foil attacks like the deferred phishing example described above.

If the attacker prepares for and executes the attack as stated in the previous section, our email link isolation would analyze the email link at the time of click and perform a high-level assessment on whether the user should be able to navigate to it.

Safe link – Users will be redirected to this site transparently

Malicious link – Users are blocked from navigating

Suspicious link – Users are heavily discouraged to navigating and are presented with a splash warning page encouraging them to view in the link in an isolated browser

While a splash warning page was the mitigation employed in the above example, email link isolation will also offer security administrators other customizable mitigation options as well, including putting the webpage in read-only mode, restricting the download and upload of files, and disabling keyboard input altogether within their Cloudflare Gateway consoles.

Email link isolation also fits into users’ existing workflows without impacting productivity or sapping their time with IT tickets. Because Cloudflare Browser Isolation is built and deployed on the Cloudflare network, with global locations in 270 cities, web browsing sessions are served as close to users as possible, minimizing latency. Additionally, Cloudflare Browser Isolation sends the final output of each webpage to a user instead of page scrubbing or sending a pixel stream, further reducing latency and not breaking browser-based applications such as SaaS.

How do I get started?

Existing Cloudflare Area 1 and Cloudflare Gateway customers are eligible for the beta release of email link isolation. To learn more and to express interest, sign up for our upcoming beta.

If you’d like to see what threats Cloudflare Area 1 detects on your live email traffic, request a free phishing risk assessment here. It takes five minutes to get started and does not impact mail flow.

Welcome to Cloudflare One Week

2022-06-19 Kenny Johnson

Post Syndicated from Kenny Johnson original https://blog.cloudflare.com/cloudflare-one-week-2022/

Welcome to Cloudflare One Week

This post is also available in 简体中文, 日本語, Español.

If we’d told you three years ago that a majority of your employees would no longer be in the office, you simply would not have believed it. We would not have believed it, either. The office has been a cornerstone of work in the modern era — almost an unshakeable assumption.

That assumption carried over into the way we built out IT systems, too. They were almost all predicated on us working from a consistent place.

And yet, here we are. Trends that had started out as a trickle — employees out of the office, remote work, BYOD — were transformed into a tsunami, almost overnight. Employees are anywhere, using any mobile or desktop device available to work, including personal devices. Applications exist across data centers, public clouds and SaaS hosting providers. Tasks increasingly are completed in a browser. All of this increases load on corporate networks.

While how we work has changed, the corporate networks and security models to enable this work have struggled to keep pace. They still often rely on a corporate perimeter that allows lateral network movement once a user or device is present on the network. VPNs remain a choke point in this model, tunneling their user traffic back into corporate perimeter where people rarely work; and MPLS lines and other private networking tools are still being used to extend an organization’s perimeter to… other offices, where people also rarely work.

And it’s not just that all these are expensive to set up: VPNs, MPLS lines and other perimeter solutions come with performance loss, create maintenance burden, and lack modern security tooling. Attackers know how to exploit their weaknesses. Many well known attacks over the last few years can be traced to unauthorized network access and subsequent lateral movement.

These problems are well known. Surprisingly, the answer to those challenges is also widely agreed upon at this point: shift to a Zero Trust Architecture. So what’s stopping people? As we’ve spoken to folks, it’s one thing, more than anything else: how? How do we do this? Underlying this is worry — that yes, while there are plenty of the risks and problems associated with the old world, they’d rather tackle the devil they know than the one that they don’t — the worry and change and cost associated with the lifting and shifting to Zero Trust.

This, more than anything else, is what we want to change with Cloudflare One Week.

Zero Trust doesn’t need to be hard. It can be stage-gated. You prove the benefits of the new model to your organization, while allowing it to transition at a pace it can handle. In short: Zero Trust can let your organization do more, let your organization do it better, and all this can come with cost savings.

Welcome to Cloudflare One Week.

The shifting goalposts of Zero Trust, SASE, SSE

While there is broad recognition of the limits of the perimeter model, one thing that keeps coming up in customer conversations about Zero Trust is: how do all these replacement concepts relate to one another? Which one should I be pursuing?

A big part of our efforts this week is to make the goal of a Zero Trust architecture approachable and understandable. All these terms get thrown around, sometimes interchangeably. We’ve spent the time understanding and building out the products to get a comprehensive Zero Trust solution.

But we don’t want you to just trust us.

We believe in Zero Trust Architecture so strongly that we worked with security experts to build a vendor-agnostic guide to implementing Zero Trust. Even if a business does not use Cloudflare, we believe that Zero Trust and SASE are the future for all businesses, regardless of which vendor they use. Here is a complete guide to navigating the world of Zero Trust.

Separately, we’ve also mapped all our products in this space to the concepts above — making it easy to follow along during the week to see how all the pieces fit together.

No one else delivers comprehensive security

Cloudflare was not the first in the application services space. We weren’t the first in the content delivery space; nor were we first in the web security space. But there’s a reason that analyst after analyst now recognize us as leaders there.

It is because our rate of innovation is simply unmatched.

We were not first to the Zero Trust space, either. But in the span of a few short years, in Cloudflare One, we have now built the most feature complete SASE offering on the market.

Cloudflare One’s Zero Trust offering includes Zero Trust Network Access, Secure Web Gateway, CASB, Data Loss Prevention, Remote Browser Isolation, Firewall as a Service, and Email Security. Every security control is configured through a single dashboard and can be deployed as code using our API or Terraform.

No one else does all of this. And over the course of this week, we’ll prove it to you.

And no one else can do it without slowing you down

Cloudflare One was built on top of Cloudflare’s existing global network. We spent over a decade building this network to support our global CDN and application security business. The network spans 270+ cities, 100 countries and is within 50ms of 95% of the Internet connected global population. From day one, we built our network to deploy additional technology on the same network, including Cloudflare One. This allows us to provide one of the most performant, reliable and interconnected Service Edges in the market.

The scale and scope of our network has other advantages when it comes to deploying a SASE solution, too. We make it easy to connect to Cloudflare Service Edge through a comprehensive set of on-ramps. These on-ramps allow users, devices, data centers, offices to connect to Cloudflare anywhere in the world. The on-ramps range from full scale SD-WAN to a lightweight client on user devices.

We plan on proving that we are the most performant Zero Trust provider over the course of this week, too.

Welcome to Cloudflare One Week – we’re just getting started

If you’ve been thinking about Zero Trust or SASE, Cloudflare One Week will demonstrate why Cloudflare One is one of the most complete SASE offerings in the market, with some of the best performance, and why it will only continue to improve. Over the week we will announce new features, show comparisons of competitors, and show you how easy it is to get started.