Tag Archives: Transform Rules

Managed Transforms: templated HTTP header modifications

Post Syndicated from Sam Marsh original https://blog.cloudflare.com/managed-transforms-templated-http-header-modifications/

Managed Transforms: templated HTTP header modifications

Managed Transforms: templated HTTP header modifications

Managed Transforms is the next step on a journey to make HTTP header modification a trivial task for our customers. In early 2021 the only way for Cloudflare customers to modify HTTP headers was by writing a Cloudflare Worker. We heard from numerous customers who wanted a simpler way.

In June 2021 we introduced Transform Rules, giving customers a simple UI letting them specify what the custom HTTP header’s name and value is—either a static string (i.e. X-My-CDN: Cloudflare) or a dynamically populated value (i.e. X-Bot-Score: cf.bot_management.score).

This made the job much simpler, however there is still a good amount of thought required—with a number of potential drop-off points on the user journey. For example, in order to dynamically populate the bot score into the value of an HTTP request header, the user needs to know the correct field name. To find that they’ll need to go to the documentation site, find the correct section, etc.

When we analyzed how our customers use Transform Rules we found a set of very common use cases in the data. Four of the top eight fields used were relating to bot management; customers wanting to have the bot score, JA3 hash, etc. of each request added as an HTTP header. A further three of the top 10 fields were relating to the geographic location of the visitor (their city, country and ASN). We also saw over 400 Transform Rules being used just to remove X-Powered-By. That means potentially 400 users went to the same part of the dashboard, typed the same header name, read the same documentation and selected the same action.

Much as we set out to productize the common Cloudflare Worker use case of HTTP header modification into Transform Rules, we have now set out to productize and further simplify the common Transform Rules use cases into Managed Transforms.

The intention is to continue to identify common reasons for use of a Transform Rule and where possible package them up into a single click solution.

We always want to make our user’s lives as easy as possible, and finding a way to stop hundreds of people typing the same thing and clicking the same buttons, to achieve the exact same outcome, seems a great way to continue that mission.

An even simpler solution

Managed Transforms is a dedicated section of Transform Rules offering one-click HTTP header modifications. Want to add relevant Cloudflare Bot Management information as custom HTTP headers? One click. Want to add geographic information (country, etc.) as custom HTTP headers? One click.

Managed Transforms: templated HTTP header modifications

Managed Transforms can be found in ‘Rules > Transform Rules’ and clicking on the ‘Managed Transforms’ button. To benefit from Managed Transforms, users simply toggle the appropriate settings, and we take care of the rest.

For example, to enrich every HTTP request with Cloudflare’s Bot Management information users would enable ‘Add bot protection headers’. This setting ensures we add four new HTTP request headers to every HTTP request. SIEM (Security Information and Event Management) products can then be configured to correctly collect and chart these new headers, allowing customers to see the bot score of every HTTP request, how many requests are coming from verified bots, and so on.

Another great use case is the ‘Add security headers’ toggle. On a completely standard, default zone, a user can improve their website’s security score from an F to a C in just one click. Enabling HSTS improves the score to a B (scores correct as of June 7, 2022).

Managed Transforms: templated HTTP header modifications

Adding a Content-Security-Policy (used to mitigate Cross-Site Scripting (‘XSS’) attacks) or a Permission-Policy (used to give websites the ability to allow or block the use of browser features) increases the score to an ‘A’; the addition of both improves the score to the maximum: A+.

Managed Transforms: templated HTTP header modifications

During the design of Managed Transforms we chose not to include default Content-Security-Policy and Permission-Policy HTTP response headers within the ‘Add security headers’ toggle as we found these particular headers to be very specific to each individual website. Any default policies we tried either caused incorrect loading of the website content, or were too open to be of any value. So we decided to remove them from scope.

However, users can still add these HTTP response headers and their appropriate values in a handful of clicks by creating a new Transform Rule:

Managed Transforms: templated HTTP header modifications

The HTTP response headers entered here will be added alongside the HTTP response headers added by Managed Transforms to give an A+ score.

Try it now

Managed Transforms can be used to improve operations, remove sensitive data, and increase security, amongst other common use cases.

Try out Managed Transforms yourself today.

Modifying HTTP response headers with Transform Rules

Post Syndicated from Sam Marsh original https://blog.cloudflare.com/transform-http-response-headers/

Modifying HTTP response headers with Transform Rules

Modifying HTTP response headers with Transform Rules

HTTP headers are central to how the web works. They are used for passing additional information between the client and server, such as which security permissions to apply and information about the client, allowing the correct content to be served.

Today we are announcing the immediate availability of the third action within Transform Rules, “HTTP Response Header Modification”, available for all Cloudflare plans. This new functionality provides Cloudflare users the ability to set or remove HTTP response headers as traffic returns through Cloudflare back to the client. This allows customers to enrich responses with information about how their request was handled, debugging information and even recruitment messages.

Previously, HTTP response header modification was done using a Cloudflare Worker. Today we’re introducing an easier way to do this without writing a single line of code.

Luggage tags of the World Wide Web

Modifying HTTP response headers with Transform Rules

Think of HTTP headers as the “luggage tag” attached to your bags when you check in at the airport.

Generally, you don’t need to know what those numbers and words mean. You just know they are important in getting your suitcase from the boarding desk, to the correct airplane, and back to the correct luggage carousel at your destination.

These tags contain information about the weight of the suitcase, the destination airport code, baggage tag number, airline carrier, customs handling information, and more. These attributes are all essential, not only for ensuring that your luggage arrives at the correct destination, but also it does so in the safest, most efficient manner.

HTTP headers are the luggage tags of the Internet. They are essential to ensuring the request from your browser arrives at the correct destination, and that traffic is returned to your browser using the correct settings also in the safest, most efficient manner.

How are HTTP response headers used?

HTTP headers are set on both the ‘request’ and ‘response’ interactions; ‘request’ being when the client asks for the file and ‘response’ being what the server returns as a result. The functionality announced today pertains specifically to HTTP response headers.

HTTP response headers are used to ensure the correct data is returned to the browser along with information which helps the browser handle the data correctly. Common response headers include “Content-Type” which tells the browser the type of the content returned, e.g. “Content-Type: text/html” or “Content-Type: image/png”. Another common header is “Server:” which contains information about the software used to handle the HTTP request, e.g. “Server: cloudflare”.

Outside of basic HTTP traffic handling there are many other uses for these response headers. One such example is to improve security. Security mechanisms such as Content Security Policy (CSP), Cross Origin Resource Sharing (CORS) and HTTP Strict Transport Security (HSTS) are all implemented as response headers to improve and harden security for website visitors.

For example, the primary goal of CSP is to mitigate and report Cross-Site Scripting (XSS) attacks. XSS attacks occur when a malicious script is injected into a trusted website, allowing an attacker to use an application to send malicious code such as a browser-side script to a different end user. This script can then be used to compromise the end user’s interactions with the website or application, siphoning sensitive information such as passwords to a third party.

To prevent this, CSP is added by the website administrator as a HTTP response header. The CSP response header specifies the domains that the browser should consider to be valid sources of executable scripts. A CSP compatible browser will then only execute scripts loaded in files received from those permitted domains, ignoring all other scripts.

CSP is added to the HTTP response by setting the ‘Content-Security-Policy’ header along with the policy which is contained in the value. For example, when using NGINX, a popular web server, the administrator would have a line in the config similar to:

add_header Content-Security-Policy "default-src 'self';" always;

When using Cloudflare Workers, the code would be similar to:

response.headers.set("Content-Security-Policy": "default-src 'self' example.com *.example.com",)

When the browser receives the HTTP response it will now detect the presence of the Content-Security-Policy header and act appropriately.

Dynamic modification of HTTP response headers

Ensuring these headers are present on the HTTP response is often the job of the reverse proxy — a server which sits between the client and the server whose job is, amongst many others, to enrich the HTTP response data returned to the client.

“HTTP Response Header Modification” is now available for all Cloudflare plans, within Transform Rules. It provides the ability to modify HTTP response headers before they are returned to the visitor, all within Cloudflare. This is especially important when the response is coming from an origin the administrator does not have total control over, such as a SaaS provider or other third party service.

Modifying HTTP response headers with Transform Rules

Transform Rules allows users to modify up to ten HTTP response headers per rule using one of three options:

Modifying HTTP response headers with Transform Rules

‘Set dynamic’ should be used when the value of a HTTP response header needs to be populated dynamically for each HTTP response. Examples include adding the Cloudflare Bot Management ‘bot score’ to each HTTP response, or the visitor’s country:

Modifying HTTP response headers with Transform Rules

Note: These values are calculated using the corresponding HTTP request, meaning the bot score returned in the response header will be calculated based upon the HTTP request. Similarly, the ‘ip.src.country value will be the country of the website visitor, not the origin where the response was sent from.

‘Set static’ should be used to populate the value of a header with a static, literal string. This option should be used for simple header creation such as setting the CORS or CSP policies:

Modifying HTTP response headers with Transform Rules

In both ‘set’ examples, if a header with the specified name already exists in the HTTP response, its value will be removed and replaced with the given value.

‘Remove’ is the final option, which should be used to remove all HTTP response headers with the specified name. For example, if you wanted to ensure the ‘Link’ HTTP response header was removed, you would use a rule similar to the following one:

Modifying HTTP response headers with Transform Rules

Cloudflare functions can be used within ‘set dynamic’ header modifications. These functions include:

  • concat()
  • regex_replace()
  • to_string()
  • lower()

An example where functions are commonly used is concat() and to_string() used to take a list of different data types and concatenate to form a single header value. For example, `concat(“score=”,to_string(cf.bot_management.score))` would result in a header value like `score=85`.

Note: regular expression functions are only available for customers on Business and Enterprise plans.

Optimizing for your website

One other huge benefit of moving HTTP response header modification into Cloudflare is the level of filtering provided in the rule builder. Typically, technologies like CORS and CSP are set as response headers on the entire website — or at best — on a per-directory basis.

With Transform Rules, administrators can set headers based upon a number of parameters including the visitor’s country of origin, bot score, user agent, requested filename / file extension, request method and more.

This allows administrators the ability to implement setups such as having a stricter Content Security Policy for verified bots vs unverified bots/low bot score traffic.

Try it now

HTTP Response Header Modification can be used to improve operations, remove sensitive data, and increase security, amongst many other use cases. Try out the latest Transform Rule yourself today.

Transform Rules:”Requests, Transform and Roll Out!”

Post Syndicated from Ming Xue original https://blog.cloudflare.com/transform-rules-requests-transform-and-roll-out/

Transform Rules:

Transform Rules:

Applications expect specific inputs in order to perform optimally. Techniques used to shape inputs to meet an application’s requirements might include normalizing the URLs to conform to a consistent formatting standard, rewriting the URL’s path and query based on different conditions and logic, and/or modifying headers to indicate an application’s specific information. These are expensive to run and complex to manage. Cloudflare can help you to offload the heavy lifting of modifying requests for your servers with Transform Rules. In this blog we will cover the nuts and bolts of the functionality.

Origin server🤒 : Thank you so much for offloading that for me, Cloudflare

Cloudflare edge servers😎 : No problem, buddy, I have taken care of that for you

Why do people need Transform Rules?

When it comes to modifying an HTTP/HTTPS request with normalization, rewriting the URLs, and/or modifying headers, Cloudflare users often use Cloudflare Workers, code they craft that runs on Cloudflare’s edge.

Cloudflare Workers open the door to many possibilities regarding the amount of work that can be done for your applications, close to where your end users are located. It provides a serverless execution environment that allows you to create application functionality without configuring or maintaining infrastructure. However, using a Worker to modify the request is kind of like wearing a diving suit in a kiddie pool. Therefore, a simple tool to modify requests without Workers has long been wanted.

Transform Rules:

It’s in this context that we looked at the most common request modifications that customers were making, and built out Transform Rules to cover them. Once Transform Rules were announced we anticipated they’d become the favourite tool in our customers’ tool box.

What do Transform Rules do?

  • URL Normalization: normalizes HTTP requests to a standard format which then allows you to predictably write security rule filters.
  • URL Rewrite: static and dynamic rewrites of the URL’s path and/or query string based on various attributes of the request.
  • Header Modify: add or remove static or dynamic headers (based on Cloudflare specific attributes) to requests based on different various attributes of the request.

URL Normalization

Bad actors on the Internet often encode your URLs to attack your applications because encoded URLs can bypass some security rules. URL Normalization transforms the request URL from encoded to unencoded before Cloudflare’s security features, so no one can bypass the firewall rules you configure.

For example, say you had a rate limiting rule for the path “/login” but someone sent the request as “/%6cogin”. Illustrated below:

You😤: Rate Limiting for https://www.example.com/login to avoid brute force attacks.

Attacker😈: You think you can stop me? I will issue massive requests to https://www.example.com/%6cogin to bypass your rate limiting rule.

Without URL Normalization, the request would bypass the rate limiting rule, but with URL Normalization the request is converted from the URL path /%6cogin to /login before the rule is applied.

By default, URL Normalization is enabled for all the zones on Cloudflare at Cloudflare’s edge, and disabled when going to origins. This means incoming URLs will be decoded to standard format before any Cloudflare security execution. When going back to the origins, we will use the original URL. In this way, no encoded URL can bypass security features and the origin also can see the original URL.

Transform Rules:
Transform Rules:

The default settings are flexible to adjust if you need. This FAQ page has more information about URL Normalization.

URL Rewrite

When talking about URL Rewrites, we always want to distinguish them from URL Redirects. They are like twins. Rewrites is a server-side modification of the URL before it is fully processed by the web server. This will not change what is seen in the user’s browser. Redirects forward URLs to other locations via a 301 or 302 HTTP status code. This will change what is seen in the user’s browser. You can do a URL Redirect with “Forwarding URL” in Cloudflare Pages Rules. Page Rules trigger actions whenever a request matches one of the URL patterns you define.

Transform Rules come into play when we need to use URL Rewrite. This allows you to rewrite a URL’s path and/or query string based on the logic of your applications. The rewrite can be a fixed string (which we call ‘static’) or a computed string (called ‘dynamic’) based on various attributes of a request, like the country it came from, the referrer, or parts of the path. These rewrites come before products such as Firewall Rules, Page Rules, and Cloudflare Workers.

Static URL Rewrite Example

When visiting www.example.com with a cookie of version=v1, you want to rewrite the URL to www.example.com/v1 when going to the origin server. In this case, the end-user facing URL will not change, but the content will be the /v1’s content. This is a static URL rewrite. It only does rewrites when end users visit the URL www.example.com with cookie version=v1. It can help you to do A/B testing when rolling out new content.

Transform Rules:

Dynamic URL Rewrite Example

When visiting any URL of www.example.com with a cookie of version=v1, you want to rewrite the request by adding /v1/ to the beginning of the URL for v1 content, when going to the origin server.

In this use case, when end users visit www.example.com/Literaturelibrary/book1314 with cookie version=v1, Cloudflare will rewrite the URL to www.example.com/v1/Literaturelibrary/book1314.

When end users visit www.example.com/fictionlibrary/book52/line43/universe with cookie version=v1, Cloudflare will rewrite the URL to www.example.com/v1/fictionlibrary/book52/line43/universe.

Transform Rules:

In this case, the URL visible in the client’s browser will not change, but the content returned will be from the /v1 location. This is a dynamic URL rewrite, so it applies the rewrite to all URLs when end users visit with the cookie.

Another Dynamic URL Rewrite Example

When visiting any URL of www.example.com with a cookie of version=v1 and query string of page=1 that has /global in the beginning of the URL, this rule rewrites the request by replacing /global in the beginning for the URL with /v1 and updates the query string to newpage=1, when going to the origin server.

When end users visit www.example.com/global/Literaturelibarary/book1013?page=1 with cookie of version=v1, Cloudflare will rewrite the URL to www.example.com/v1/Literaturelibarary/book1013?newpage=1.

And when end users visit www.example.com/global/fictionlibarary/book52/line43/universe?page=1 with cookie of version=v1, Cloudflare will rewrite the URL to www.example.com/v1/fictionlibarary/book52/line43/universe?newpage=1.

Transform Rules:

In this case, the end-user facing URLs will not change, but the content will be v1’s content. This is a dynamic URL rewrite, so it applies the rewrite to all URLs when end users visit with the cookie of version=v1 and a query string of page=1.

Header Modify

Adding/removing request headers of the requests when going to origin servers. This is one of the most requested features of customers using Cloudflare Workers, especially those sending the Bot Score as a request header to origin. You can use this feature to add/remove strings and non-strings, and static or dynamic request header values.

Set Static header: Adds a static header and value to the request and sends it to the origin.

For example, add a request header such as foo: bar only when the requests have the hostname of www.example.com.

Transform Rules:

With the above setting, Cloudflare appends a static header Foo: bar to your origin when this rule triggers. Here is what the origin should see.

Transform Rules:

Set Dynamic header : Adds a dynamic header value from the computed field, like the end user’s geolocation.

Transform Rules:

The dynamic request headers are added.

Transform Rules:

Set Dynamic Bot Management headers: Cloudflare Bot Management protects applications from bad bot traffic by scoring each request with a “bot score” from 1 to 99. With Bot Management enabled, we can send the bot score to the origin server as a request header via Transform Rules.

Transform Rules:

The bot score header is added.

Transform Rules:

It has never been easier

With Transform Rules, you can modify the request with URL Normalization, URL Rewrites, and HTTP Header Modification with easy settings to power your application. There’s no script required for Cloudflare to offload those duties from your origin servers. Just like Optimus Prime says “Autobots, transform and roll out!”, Cloudflare says “Requests, transform and roll out!”.

Try out the latest Transform Rules yourself today.

Modify HTTP request headers with Transform Rules

Post Syndicated from Sam Marsh original https://blog.cloudflare.com/transform-http-request-headers/

Modify HTTP request headers with Transform Rules

Modify HTTP request headers with Transform Rules

HTTP headers are central to how the web works. They are used for passing additional information between the client and server, such as which security permissions to apply and information about the client, allowing the correct content to be served.

Today we are announcing the immediate availability of the second action within Transform Rules, “HTTP Request Header Modification”, available for all Cloudflare plans. This new functionality provides Cloudflare administrators with the ability to easily set or remove HTTP request headers as traffic flows through Cloudflare. This allows customers to enrich requests with information such as the Cloudflare Bot Management ‘Bot Score’ prior to being sent to their servers. Previously, HTTP request header modification was done using a Cloudflare Worker. Today we’re introducing an easier way to do this without writing a single line of code.

Luggage tags of the World Wide Web

Modify HTTP request headers with Transform Rules
Photo by Markus Spiske on Unsplash

Think of HTTP headers as the “luggage tag” attached to your bags when you check in at the airport.

Generally, you don’t need to know what those numbers and words mean. You just know they are important in getting your suitcase from the boarding desk, to the correct airplane, and back to the correct luggage carousel at your destination.

These tags contain information about the weight of the suitcase, the destination airport code, baggage tag number, airline carrier, customs handling information, and more. These attributes are all essential, not only for ensuring that your luggage arrives at the correct destination, but also it does so in the safest, most efficient manner.

HTTP Headers are the luggage tags of the Internet. They are essential to ensuring the request from your browser arrives at the correct destination, and that traffic is returned to your browser using the correct settings also in the safest, most efficient manner.

How are HTTP request headers used?

HTTP headers are set on both the ‘request’ and ‘response’ interactions; ‘request’ being when the client asks for the file and ‘response’ being what the server returns as a result. The functionality announced today pertains specifically to HTTP request headers only.

Many organizations use HTTP request headers to ensure visitor requests are served correctly. They are used to route requests to different clusters, serve mobile-friendly content, and legacy-browser friendly content.

HTTP request headers are also used for security purposes, namely authentication and authorization. Simple examples include adding a static, pre-shared key as a custom header which adds an additional security check to all inbound HTTP requests.

Ensuring these headers are present on the HTTP request is often the job of the reverse proxy — a server which sits between the client and the server whose job is, amongst many others, to enrich the HTTP request data sent to the server.

For example, when using NGINX, a popular web server used as a reverse proxy, the administrator would have a line in the config similar to:

proxy_set_header X-Header-Name "custom";

When using  Cloudflare Workers, the code would be similar to:

request.headers.set("X-Header-Name", "custom")

Each of these lines of code would add a custom HTTP request header to the next-hop destination with a name of ‘X-Header-Name’ and a value of ‘custom’.

Dynamic modification of HTTP request headers

“HTTP Request Header Modification” is now available for all Cloudflare plans, within Transform Rules. It gives control to administrators by providing the ability to modify HTTP request headers before they’re sent to their own origin servers or third-party services such as SaaS providers.

Modify HTTP request headers with Transform Rules

Transform Rules allows users to modify up to 10 HTTP request headers per rule using one of three options:

Modify HTTP request headers with Transform Rules

‘Set dynamic’ should be used when the value of a HTTP request header needs to be populated dynamically for each HTTP request. Examples include adding the Cloudflare Bot Management ‘bot score’ to each HTTP request, or the visitor’s country:

Modify HTTP request headers with Transform Rules

‘Set static’ should be used to populate the value of a header with a static, literal string. This option should be used for simple header creation such as setting the source CDN (Cloudflare) or a shared secret:

Modify HTTP request headers with Transform Rules

In both “set” examples, if a header with the specified name already exists in the HTTP request, its value will be removed and replaced with the given value.

‘Remove’ is the final option, which should be used to remove all HTTP request headers with the specified name. For example, if you wanted to ensure the ‘cf-connecting-ip’ HTTP request header was removed, you would use a rule similar to the following one:

Modify HTTP request headers with Transform Rules

Cloudflare functions can be used within ‘set dynamic’ header modifications. These functions include:

  • concat()
  • regex_replace()
  • to_string()
  • lower()

An example where functions are commonly used is concat() and to_string() used to take a list of different data types and concatenate to form a single header value. For example, concat(“score=”,to_string(cf.bot_management.score)) would result in a header value of ‘score=85’.

Note: regular expression functions are only available for customers on Business and Enterprise plans.

Try it now

HTTP Request Header Modification can be used to improve operations, remove sensitive data, and increase security, amongst many other use cases. Try out the latest Transform Rule yourself today.