Today we are excited to announce that our Ethereum and IPFS gateways are publicly available to all Cloudflare customers for the first time. Since our announcement of our private beta last September the interest in our Eth and IPFS gateways has been overwhelming. We are humbled by the demand for these tools, and we are excited to get them into as many developers’ hands as possible. Starting today, any Cloudflare customer can log into the dashboard and configure a zone for Ethereum, IPFS, or both!
Over the last eight months of the private beta, we’ve been busy working to fully operationalize the gateways to ensure they meet the needs of our customers!
First, we have created a new API with end-to-end managed hostname deployment. This ensures the creation and management of gateways as you continue to scale remains extremely quick and easy! It is paramount to give time and focus back to developers to focus on your core product and services and leave the infrastructural components to us!
Second, we’ve added a brand new UI bringing web3 to Cloudflare’s zone-level dashboard. Now, regardless of the workflow you are used to, we have parity between our UI and API to ensure we fit into your existing processes and no time is wasted internally to have to figure out ‘how we integrate’, but rather, a quick setup and start to serve content or connect your services!
Third, we are pleased to say that you will soon have testnet support to ensure your new development can be easily tested, hardened, and deployed to your mainnet without incurring additional risk to your brand trust, product availability, or concern that something may fail silently and begin a cascade of problems throughout your network. We want to ensure that anyone leveraging our web3 gateways is able to achieve more confidence and trust that whatever changes are pushed forward do not impact end user experience. At the end of the day, the Internet is for end users and their experience and perception must always be kept within our purview at all times.
Lastly, Cloudflare loves to build on top of Cloudflare. This helps us stay resilient and also shows our commitment and belief in all the products we create! We have always used our SSL for SaaS and Workers products in the background. Building on our own services gives our customers the ability to define and control their own HTTP features on top of traffic destined for web3 gateways, including: rate limits, WAF rules, custom security filters, serving video, customer defined Workers logic, custom redirects and more!
Today thousands of different individuals, companies, and DAO’s are building new products leveraging our web3 gateways — the most reliable web3 infrastructure with the largest network
Here are just a few snippets of how people are already using our web3 Gateways, and we can’t wait to see what you build on them:
DeFi DAO’s use the Cloudflare IPFS gateway to serve their front end web applications globally without latency or cache penalties.
NFT designers use the Ethereum Gateway to effortlessly drop new offerings, and the IPFS gateway to store them in a fully decentralized system.
Large Dapp developers trust us to handle huge traffic spikes quickly and efficiently, without rate limits or overage caps. They combine all our offerings into a single pane of glass so that they don’t have to juggle multiple systems.
As part of this announcement, we will begin migrating our existing users away from the legacy gateway endpoints and onto our new API, which is easier, highly managed, and more robust. To ensure a smooth transition, you will first need to make sure you have signed up for a Cloudflare account if you did not already have one. On top of that, we have made sure to keep our free users in mind and thus our free users will continue to use the gateways at no cost with our free tier option! This includes no cap in the amount of traffic that can be pushed through our gateways along with offering the most transparent and forecastable pricing models in the market today. We are very excited about the future and look forward to sharing the next iterations of web3 at Cloudflare!
Also, if you can’t wait to start building on our gateways, check out our product documentation for more guidance.
Four years ago, Cloudflare went Interplanetary by offering a gateway to the IPFS network. This meant that if you hosted content on IPFS, we offered to make it available to every user of the Internet through HTTPS and with Cloudflare protection. IPFS allows you to choose a storage provider you are comfortable with, while providing a standard interface for Cloudflare to serve this data.
Since then, businesses have new tools to streamline web development. Cloudflare Workers, Pages, and R2 are enabling developers to bring services online in a matter of minutes, with built-in scaling, security, and analytics.
Today, we’re announcing we’re bridging the two. We will make it possible for our customers to serve their sites on the IPFS network.
In this post, we’ll learn how you will be able to build your website with Cloudflare Pages, and leverage the IPFS integration to make your content accessible and available across multiple providers.
A primer on IPFS
The InterPlanetary FileSystem (IPFS) is a peer-to-peer network for storing content on a distributed file system. It is composed of a set of computers called nodes that store and relay content using a common addressing system. In short, a set of participants agree to maintain a shared index of content the network can provide, and where to find it.
Let’s take two random participants in the network: Alice, a cat person, and Bob, a dog person.
As a participant in the network, Alice keeps connections with a subset of participants, referred to as her peers. When Alice is making her content 🐱 available on IPFS, it means she announces to her peers she has content 🐱, and these peers stored in their routing table 🐱 is provided by Alice’s node.
Each node has a routing table, and a datastore. The routing table retains a mapping of content to peers, and the datastore retains the content a given node stores. In the above case, only Alice has content, a 🐱.
When Bob wants to retrieve 🐱, he tells his peers they want 🐱. These peers point him to Alice. Alice then provides 🐱 to Bob. Bob can verify 🐱 is the content they were looking for, because the content identifier he requested is derived from the 🐱 content itself, using a secure, cryptographic hash function.
Even better, if Bob becomes a cat person, they can announce to their peers they are also providing a cat. Bob’s love for cats could be genuine, or because they have interest in providing the content, such as a contract with Alice. IPFS provides a common ground to share content, without being opinionated on how this content has to be stored and its guarantees.
How Pages websites are made available on IPFS
Content is made available as follows.
The components are:
Pages storage: Storage solution for Cloudflare Pages.
IPFS Index Proxy: Service maintaining a map between IPFS CID and location of the data. This is operated on Cloudflare Workers and using Workers KV to store the mapping.
IPFS node: Cloudflare-hosted IPFS node serving Pages content. It has an in-house datastore module, able to communicate with the IPFS Index Proxy.
IPFS network: The rest of the IPFS network.
When you opt in serving your Cloudflare Page on IPFS, a call is made to the IPFS index proxy. This call first fetches your Pages content, transforms it into a CID, then both populates IndexDB to associate the CID with the content and reaches out to Cloudflare IPFS node to tell them they are able to provide the CID.
For example, imagine your website structure is as follows:
/
index.html
static/
cats.txt
beautiful_cats.txt
To provide this website on IPFS, Cloudflare has to compute a CID for /. CIDs are built recursively. To compute the CID for a given folder /, one needs to have the CID of index.html and static/. index.html CID is derived from its binary representation, and static/ from cats.txt and beautiful_cats.txt.
This works similarly to a Merkle tree, except nodes can reference each other as long as they still form a Directed Acyclic Graph (DAG). This structure is therefore referred to as a MerkleDAG.
In our example, it’s not unlikely for cats.txt and beautiful_cats.txt to have data in common. In this case, Cloudflare can be smart in the way it builds the MerkleDAG.
This reduces the storage and bandwidth requirement when accessing the website on IPFS.
If you want to learn more about how you can model a file system on IPFS, you can check the UnixFS specification.
Cloudflare stores every CID and the content it references in IndexDB. This allows Cloudflare IPFS nodes to serve Cloudflare Pages assets when requested.
Let’s put this together
Let’s take an example: pages-on-ipfs.com is hosted on IPFS. It is built using Hugo, a static site generator, and Cloudflare Pages with the public documentation template. Its source is available on GitHub. If you have an IPFS compatible client, you can access it at ipns://pages-on-ipfs.com as well.
1. Read Cloudflare Pages deployment documentation
For the purpose of this blog, I want to create a simple Cloudflare page website. I have experience with Hugo, so I choose it as my framework for the project.
This is where I use Hugo, and your configuration might differ. In short, I use hugo new site pages-on-ipfs, create an index and two static resources, et voilà. The result is available on the source GitHub for this project.
3. Deploy using Cloudflare Pages
On the Cloudflare Dashboard, I go to Account Home > Pages > Create a project. I select the GitHub repository I created, and configure the build tool to build Hugo website. Basically, I follow what’s written on Cloudflare Pages documentation.
Upon clicking Save and Deploy, my website is deployed on pages-on-ipfs.pages.dev. I also configure it to be available at pages-on-ipfs.com
4. Serve my content on IPFS
First, I opt in my zone on Cloudflare Pages integration with IPFS. This feature is not available yet for everyone to try out.
This allows Cloudflare to index the content of my website. Once indexed, I get the CID for my deployment baf…1. I can check that my content is available at this hash on IPFS using an IPFS gateway https://baf…1.ipfs.cf-ipfs.com/.
5. Make my IPFS website available at pages-on-ipfs.com
Having one domain name to access both Cloudflare Pages and IPFS version, depending on if the client supports IPFS or not is ideal. Fortunately, the IPFS ecosystem supports such a feature via DNSLink. DNSLink is a way to specify the IPFS content a domain is associated with.
For pages-on-ipfs.com, I create a TXT record on _dnslink.pages-on-ipfs.com with value dnslink=/ipfs/baf…1. Et voilà. I can now access ipns://pages-on-ipfs.com via an IPFS client.
6. (Optional) Replicate my website
The content of my website can now easily be replicated and pinned by other IPFS nodes. This can either be done at home via an IPFS client or using a pinning service such as Pinata.
What’s next
We’ll make this service available later this year as it is being refined. We are committed to make information move freely and help build a better Internet. Cloudflare Pages work of solving developer problems continues, as developers are now able to make their site accessible to more users.
Over the years, IPFS has been used by more and more people. While Cloudflare started by making IPFS content available to web users through an HTTP interface, we now think it’s time to give back. Allowing Cloudflare assets to be served over a public distributed network extends developers and users capability on an open web.
Common questions
I am already hosting my website on IPFS. Can I pin it using Cloudflare?
No. This project aims at serving Cloudflare hosted content via IPFS. We are still investigating how to best replicate and re-provide already-existing IPFS content via Cloudflare infrastructure.
Does this make IPFS more centralized?
No. Cloudflare does not have the authority to decide who can be a node operator nor what content they provide.
Are there guarantees the content is going to be available?
Yes. As long as you choose Cloudflare to host your website on IPFS, it will be available on IPFS. Should you move to another provider, it would be your responsibility to make sure the content remains available. IPFS allows for this transition to be smooth using a pinning service.
Is IPFS private?
It depends. Generally, no. IPFS is a p2p protocol. The nodes you peer with and request content from would know the content you are looking for. The privacy depends on the trust you have in your peers to not snoop on the data you request.
Can users verify the integrity of my website?
Yes. They need to access your website through an IPFS compatible client. Ideally, IPFS content integrity is turned into a web standard, similar to subresource integrity.
To this end, we’ve developed the IPFS Gateway monitor, an observability tool that runs various IPFS scenarios on a given gateway endpoint. In this post, you’ll learn how we use this tool and go over discoveries we made along the way.
Refresher on IPFS
IPFS is a distributed system for storing and accessing files, websites, applications, and data. It’s different from a traditional centralized file system in that IPFS is completely distributed. Any participant can join and leave at any time without the loss of overall performance.
However, in order to access any file in IPFS, users cannot just use web browsers. They need to run an IPFS node to access the file from IPFS using its own protocol. IPFS Gateways play the role of enabling users to do this using only web browsers.
Cloudflare provides an IPFS gateway at https://cloudflare-ipfs.com, so anyone can just access IPFS files by using the gateway URL in their browsers.
As IPFS and the Cloudflare IPFS Gateway have become more and more popular, we need a way to know how performant it is: how much time it takes to retrieve IPFS-hosted content and how reliable it is. However, IPFS gateways are not like normal websites which only receive HTTP requests and return HTTP responses. The gateways need to run IPFS nodes internally and sometimes do content routing and peer routing to find the nodes which provide IPFS contents. They sometimes also need to resolve IPNS names to discover the contents. So, in order to measure the performance, we need to do measurements many times for many scenarios.
Enter the IPFS Gateway monitor
IPFS Gateway monitor is this tool. It allows anyone to check the performance of their gateway and export it to the Prometheus monitoring system.
This monitor is composed of three independent binaries:
ipfs-gw-measure is the tool that calls the gateway URL and does the measurement scenarios.
ipfs-gw-monitor is a way to call the measurement tool multiple times.
To interact with the IPFS network, the codebase also provides a way to start an IPFS node.
A scenario is a set of instructions a user performs given the state of our IPFS system. For instance, we want to know how fast newly uploaded content can be found by the gateway, or if popular content has a low query time. We’ll discuss more of this in the next section.
Putting this together, the system is the following.
During this experience, we have operated both the IPFS Monitor, and a test IPFS node. The IPFS node allows the monitor to provide content to the IPFS network. This allows us to be sure that the content provided is fresh, and actually hosted. Peers have not been fixed, and leverage the IPFS default peer discovery mechanism.
Cloudflare IPFS Gateway is treated as an opaque system. The monitor performs end-to-end tests by contacting the gateway via its public API, either https://cloudflare-ipfs.com or via a custom domain registered with the gateway.
The following scenarios have been run on consumer hardware in March. They are not representative of all IPFS users. All values provided below have been sourced against Cloudflare’s IPFS gateway endpoint.
First scenarios: Gateway to serve immutable IPFS contents
IPFS contents are the most primitive contents being served by the IPFS network and IPFS gateways. By their nature, they are immutable and addressable only by the hash of the content. Users can access the IPFS contents by putting the Content IDentifier (CID) in the URL path of the gateway. For example, ipfs://bafybeig7hluox6xefqdgmwcntvsguxcziw2oeogg2fbvygex2aj6qcfo64. We measure three common scenarios that users will often encounter.
The first one is when users fetch popular content which has a high probability of being found in our cache already. During our experiment, we measured a response time for such content is around 116ms.
The second one is the case where the users create and upload the content to the IPFS network, such as via the integration between Cloudflare Pages and IPFS (LINK TO BLOG). This scenario is a lot slower than the first because the content is not in our cache yet, and it takes some time to discover the content. The content that we upload during this measurement is a random piece of 32-byte content.
The last measurement is when users try to download content that does not exist. This one is the slowest. Because of the nature of content routing of IPFS, there is no indication that tells us that content doesn’t exist. So, setting the timeout is the only way to tell if the content exists or not. Currently, the Cloudflare IPFS gateway has a timeout of around five minutes.
Scenarios
Min (s)
Max (s)
Avg (s)
1
ipfs/newly-created-content
0.276
343
44.4
2
ipfs/in-cache-content
0.0825
0.539
0.116
3
ipfs/unavailable-cid
90
341
279
Second scenarios: Gateway to serve mutable IPNS named contents
Since IPFS contents are immutable, when users want to change the content, the only way to do so is to create new content and distribute the new CID to other users. Sometimes distributing the new CID is hard, and is out of scope of IPFS. The InterPlanetary Naming System (IPNS) tries to solve this. IPNS is a naming system that — instead of locating the content using the CID — allows users to locate the content using the IPNS name instead. This name is a hash of a user’s public key. Internally, IPNS maintains a section of the IPFS DHT which maps from a name to a CID. Therefore, when the users want to download the contents using names through the gateway, the gateway has to first resolve the name to get the CID, then use the CID to query the IPFS content as usual.
The example for fetching the IPNS named content is at ipns://k51qzi5uqu5diu2krtwp4jbt9u824cid3a4gbdybhgoekmcz4zhd5ivntan5ey
We measured many scenarios for IPNS as shown in the table below. Three scenarios are similar to the ones involving IPFS contents. There are two more scenarios added: the first one is measuring the response time when users query the gateway using an existing IPNS name, and the second one is measuring the response time when users query the gateway immediately after new content is published under the name.
Scenarios
Min (s)
Max (s)
Avg (s)
1
ipns/newly-created-name
5.50
110
33.7
2
ipns/existing-name
7.19
113
28.0
3
ipns/republished-name
5.62
80.4
43.8
4
ipns/in-cache-content
0.0353
0.0886
0.0503
5
ipns/unavailable-name
60.0
146
81.0
Third scenarios: Gateway to serve DNSLink websites
Even though users can use IPNS to provide others a stable address to fetch the content, the address can still be hard to remember. This is what DNSLink is for. Users can address their content using DNSLink, which is just a normal domain name in the Domain Name System (DNS). As a domain owner, you only have to create a TXT record with the value dnslink=/ipfs/baf…1, and your domain can be fetched from a gateway.
There are two ways to access the DNSLink websites using the gateway. The first way is to access the website using the domain name as a URL hostname, for example, https://ipfs.example.com. The second way is to put the domain name as a URL path, for example, https://cloudflare-ipfs.com/ipns/ipfs.example.com.
Scenarios
Min (s)
Max (s)
Avg (s)
1
dnslink/ipfs-domain-as-url-hostname
0.251
18.6
0.831
2
dnslink/ipfs-domain-as-url-path
0.148
1.70
0.346
3
dnslink/ipns-domain-as-url-hostname
7.87
44.2
21.0
4
dnslink/ipns-domain-as-url-path
6.87
72.6
19.0
What does this mean for regular IPFS users?
These measurements highlight that IPFS content is fetched best when cached. This is an order of magnitude faster than fetching new content. This result is expected as content publication on IPFS can take time for nodes to retrieve, as highlighted in previous studies. Then, when it comes to naming IPFS resources, leveraging DNSLink appears to be the faster strategy. This is likely due to the poor connection of the IPFS node used in this setup, preventing IPNS name from propagating rapidly. Overall, IPNS names would benefit from using a resolver to speed up resolution without losing the trust they provide.
As we mentioned in September, IPFS use has seen important growth. So has our tooling. The IPFS Gateway monitor can be found on GitHub, and we will keep looking at improving this first set of metrics.
At the time of writing, using IPFS via a gateway seems to provide lower retrieval times, while allowing for finer grain control over security settings in the browser context. This configuration preserves the content validity properties offered by IPFS, but reduces the number of nodes a user is peering with to one: the gateway. Ideally, we would like users to peer with Cloudflare because we’re offering the best service, while still having the possibility to retrieve content from external sources if they want to. We’ll be conducting more measurements to better understand how to best leverage Cloudflare presence in 270 cities to better serve the IPFS network.
It’s cliché to say that the Internet has undergone massive changes in the last five years. New technologies like distributed ledgers, NFTs, and cross-platform metaverses have become all the rage. Unless you happen to hang out with the Web3 community in Hong Kong, San Francisco, and London, these technologies have a high barrier to entry for the average developer. You have to understand how to run distributed nodes, set up esoteric developer environments, and keep up with the latest chains just to get your app to run. That stops today. Today you can sign up for the private beta of our Web3 product suite starting with our Ethereum and IPFS gateway.
Before we go any further, a brief introduction to blockchain (Ethereum in our example) and the InterPlanetary FileSystem (IPFS). In a Web3 setting, you can think of Ethereum as the compute layer, and IPFS as the storage layer. By leveraging decentralised ledger technology, Ethereum provides verifiable decentralised computation. Publicly available binaries, called “smart contracts”, can be instantiated by users to perform operations on an immutable set of records. This set of records is the state of the blockchain. It has to be maintained by every node on the network, so they can verify, and participate in the computation. Performing operations on a lot of data is therefore expensive. A common pattern is to use IPFS as an external storage solution. IPFS is a peer-to-peer network for storing content on a distributed file system. Content is identified by its hash, making it inexpensive to reference from a blockchain context.
Over the last four years, while we have been working to mature the technology required to provide access to Web3 services at a global scale, the idea of the Metaverse has come back into vogue. Popularized by novels like “Snowcrash,” and “Ready Player One,” the idea is a simple one. Imagine an Internet where you can hop into an app and have access to all of your favorite digital goods available for you to use regardless of where you purchased them. You could sell your work on social media without granting them a worldwide license, and the buyer could use it on their online game. The Metaverse is a place where copyright and ownership can be managed through NFTs (Non-Fungible Tokens) stored on IPFS, and accessed trustlessly through Ethereum. It is a place where everyday creators can easily monetize their content, and have it be used by everyone, regardless of platform, since content is not being stored in walled gardens but decentralised ecosystems with open standards.
This shifts the way users and content creators think about the Internet. Questions like: “Do you actually need a Model View Controller system with a server to build an application?” “What is the best way to provide consistent naming of web resources across platforms?” “Do we actually need to keep our data locked behind another company’s systems or can the end-user own their data?”. This builds different trust assumptions. Instead of trusting a single company because they are the only one to have your users’ data, trust is being built leveraging a source verifiable by all participants. This can be people you physically interact with for messaging applications, X.509 certificates logged in a public Certificate Transparency Log for websites, or public keys that interact with blockchains for distributed applications.
It’s an exciting time. Unlike the emergence of the Internet however, there are large established companies that want to control the shape and direction of Web3 and this Metaverse. We believe in a future of a decentralised and private web. An open, standards-based web independent of any one company or centralizing force. We believe that we can be one of the many technical platforms that supports Web3 and the growing Metaverse ecosystem. It’s why we are so excited to be announcing the private beta of our Ethereum and IPFS gateways. Technologies that are at the forefront of Web3 and its emerging Metaverse.
Time and time again over the last year we have been asked by our customers to support their exploration of Web3, and oftentimes their core product offering. At Cloudflare, we are committed to helping build a better Internet for everyone, regardless of their preferred tech stack. We want to be the pickaxes and shovels for everyone. We believe that Web3 and the Metaverse is not just an experiment, but an entirely new networking paradigm where many of the next multi-billion dollar businesses are going to be built. We believe that the first complete metaverse could be built entirely on Cloudflare today using systems like Ethereum, IPFS, RTC, R2 storage, and Workers. Maybe you will be the one to build it…
We are excited to be on this journey with our Web3 community members, and can’t wait to show you what else we have been working on.
Introducing the Cloudflare Web3 Gateways!
A gateway is a computer that sits between clients (such as your browser or mobile device) and a number of other systems and helps translate traffic from one protocol to another, so the systems powering an application required to handle the request can do so properly. But there are different types of gateways that exist today.
You have probably heard mention of an API gateway, which is responsible for accepting API calls inbound to an application and aggregating the appropriate services to fulfill those requests and return a proper response to the end user. You utilize gateways every time you watch Netflix! Their company leverages an API gateway to ensure the hundreds of different devices that access their streaming service can receive a successful and proper response, allowing end users to watch their shows. Gateways are a critical component of how Web3 is being enabled for every end user on the planet.
Remember that Web3 or the distributed web is a set of technologies that enables hosting of content and web applications in a serverless manner by leveraging purely distributed systems and consensus protocols. Gateways let you use these applications in your browser without having to install plugins or run separate pieces of software called nodes. The distributed web community runs into the same problem of needing a stable, reliable, and resilient method to translate HTTP requests into the correct Web3 functions or protocols.
Today, we are introducing the Cloudflare Ethereum and IPFS Gateways to help Web3 developers do what they do best, develop applications, without having to worry about also running the infrastructure required to support Ethereum (Eth) or IPFS nodes.
What’s the problem with existing Eth or IPFS Web Gateways?
Traditional web technologies such as HTTP have had decades to develop standards and best practices that make sites fast, secure, and available. These haven’t been developed on the distributed web side of the Internet, which focuses more on redundancy. We identified an opportunity to bring the optimizations and infrastructure of the web to the distributed web by building a gateway — a service that translates HTTP API calls to IPFS or Ethereum functions, while adding Cloudflare added-value services on the HTTP side. The ability for a customer to operate their entire network control layer with a single pane of glass using Cloudflare is huge. You can manage the DNS, Firewall, Load Balancing, Rate Limiting, Tunnels, and more for your marketing site, your distributed application (Dapp), and corporate security, all from one location.
For many of our customers, the existing solutions for Web3 gateway do not have a large enough network to handle the growing amount of requests within the Ethereum and IPFS networks, but more importantly do not have the degree of resilience and redundancy that businesses expect and require operating at scale. The idea of the distributed web is to do just that… stay distributed, so no single actor can control the overall market. Speed, security, and reliability are at the heart of what we do. We are excited to be part of the growing Web3 infrastructure community so that we can help Dapp developers have more choice, scalability, and reliability from their infrastructure providers.
A clear example of this is when existing gateways have an outage. With too few gateways to handle the traffic, the result of this outage is pre-process transactions falling behind the blockchain they are accessing, thus leading to increased latency for the transaction, potentially leading to it failing. Worse, when decentralised application (Dapp) developers use IPFS to power their front end, it can lead to their entire application falling over. Overall, this leads to massive amounts of frustration from businesses and end users alike — not being able to collect revenue for products or services, thus putting a portion of the business at a halt and breaking trust with end users who depend on the reliability of these services to manage their Web3 assets.
How is Cloudflare solving this problem?
We found that there was a unique opportunity in a segment of the Web3 community that closely mirrored Cloudflare’s traditional customer base: the distributed web. This segment has some major usability issues that Cloudflare could help solve around reliability, performance, and caching. Cloudflare has an advantage that no other company in this space — and very few in the industry — have: a global network. For instance, content fetched through our IPFS Gateway can be cached near users, allowing download latency in the milliseconds. Compare this with up to seconds per asset using native IPFS. This speed enables services based on IPFS to go hybrid. Content can be served over the source decentralised protocols while browsers and tools are maturing to access them, and served to regular web users through a gateway like Cloudflare. We do provide a convenient, fast and secure option to browse this distributed content.
On Ethereum, users can be categorised in two ways. Application developers that operate smart contracts, and users that want to interact with the said contracts. While smart contracts operate autonomously based on their code, users have to fetch data and send transactions. As part of the chain, smart contracts do not have to worry about the network or a user interface to be online. This is why decentralised exchanges have had the ability to operate continuously across multiple interfaces without disruptions. Users on the other hand do need to know the state of the chain, and be able to interact with it. Application developers therefore have to require the users to run an Ethereum node, or can point them to use remote nodes through a standardised JSON RPC API. This is where Cloudflare comes in. Cloudflare Ethereum gateway relies on Ethereum nodes and provides a secure and fast interface to the Ethereum network. It allows application developers to leverage Ethereum in front-facing applications. The gateway can interact with any content part of the Ethereum chain. This includes NFT contracts, DeFi exchanges, or name services like ENS.
How are the gateways doing so far?
Since our alpha release to very early customers as research experiments, we’ve seen a staggering number of customers wanting to leverage the new gateway technology and benefit from the availability, resiliency, and caching benefits of Cloudflare’s network.
Our current alpha includes companies that have raised billions of dollars in venture capital, companies that power the decentralised finance ecosystem on Ethereum, and emerging metaverses that make use of NFT technology.
In fact, we have over 2,000 customers leveraging our IPFS gateway lending to over 275TB of traffic per month. For Ethereum, we have over 200 customers transacting over 13TB, including 1.6 billion requests per month. We’ve seen extremely stable results from these customers and fully expect to see these metrics continue to ramp up as we add more customers to use this new product.
We are now very happy to announce the opening of our private beta for both the Ethereum and IPFS gateways. Sign up to participate in the private beta and our team will reach out shortly to ensure you are set up!
P.S. We are hiring for Web3! If you want to come work on it with us, check out our careers page.
Cloudflare’s journey with IPFS started in 2018 when we announced a public gateway for the distributed web. Since then, the number of infrastructure providers for the InterPlanetary FileSystem (IPFS) has grown and matured substantially. This is a huge benefit for users and application developers as they have the ability to choose their infrastructure providers.
Today, we’re excited to announce new secure filtering capabilities in IPFS. The Cloudflare IPFS module is a tool to protect users from threats like phishing and ransomware. We believe that other participants in the network should have the same ability. We are releasing that software as open source, for the benefit of the entire community.
Before we get to understand how IPFS filtering works, we need to dive a little deeper into the operation of an IPFS node.
The InterPlanetary FileSystem (IPFS) is a peer-to-peer network for storing content on a distributed file system. It is composed of a set of computers called nodes that store and relay content using a common addressing system.
Nodes communicate with each other over the Internet using a Peer-to-Peer (P2P) architecture, preventing one node from becoming a single point of failure. This is even more true given that anyone can operate a node with limited resources. This can be light hardware such as a Raspberry Pi, a server at a cloud provider, or even your web browser.
This creates a challenge since not all nodes may support the same protocols, and networks may block some types of connections. For instance, your web browser does not expose a TCP API and your home router likely doesn’t allow inbound connections. This is where libp2p comes to help.
libp2p is a modular system of protocols, specifications, and libraries that enable the development of peer-to-peer network applications – libp2p documentation
That’s exactly what four IPFS nodes need to connect to the IPFS network. From a node point of view, the architecture is the following:
Any node that we maintain a connection with is a peer. A peer that does not have 🐱 content can ask their peers, including you, they WANT🐱. If you do have it, you will provide the 🐱 to them. If you don’t have it, you can give them information about the network to help them find someone who might have it. As each node chooses the resources they store, it means some might be stored on a limited number of nodes.
For instance, everyone likes 🐱, so many nodes will dedicate resources to store it. However, 🐶 is less popular. Therefore, only a few nodes will provide it.
This assumption does not hold for public gateways like Cloudflare. A gateway is an HTTP interface to an IPFS node. On our gateway, we allow a user of the Internet to retrieve arbitrary content from IPFS. If a user asks for 🐱, we provide 🐱. If they ask for 🐶, we’ll find 🐶 for them.
Cloudflare’s IPFS gateway is simply a cache in front of IPFS. Cloudflare does not have the ability to modify or remove content from the IPFS network. However, IPFS is a decentralized and open network, so there is the possibility of users sharing threats like phishing or malware. This is content we do not want to provide to the P2P network or to our HTTP users.
In the next section, we describe how an IPFS node can protect its users from such threats.
If you would like to learn more about the inner workings of libp2p, you can go to ProtoSchool which has a great tutorial about it.
How IPFS filtering works
As we described earlier, an IPFS node provides content in two ways: to its peers through the IPFS P2P network and to its users via an HTTP gateway.
Filtering content of the HTTP interface is no different from the current protection Cloudflare already has in place. If 🐶 is considered malicious and is available at cloudflare-ipfs.com/ipfs/🐶, we can filter these requests, so the end user is kept safe.
The P2P layer is different. We cannot filter URLs because that’s not how the content is requested. IPFS is content-addressed. This means that instead of asking for a specific location such as cloudflare-ipfs.com/ipfs/🐶, peers request the content directly using its Content IDentifiers (CID), 🐶.
More precisely, 🐶 is an abstraction of the content address. A CID looks like QmXnnyufdzAWL5CqZ2RnSNgPbvCc1ALT73s6epPrRnZ1Xy (QmXnnyufdzAWL5CqZ2RnSNgPbvCc1ALT73s6epPrRnZ1Xy happens to be the hash of a .txt file containing the string “I’m trying out IPFS”). CID is a convenient way to refer to content in a cryptographically verifiable manner.
This is great, because it means that when peers ask for malicious 🐶 content, we can prevent our node from serving it. This includes both the P2P layer and the HTTP gateway.
In addition, the working of IPFS makes it, so content can easily be reused. On directories for instance, the address is a CID based on the CID of its files. This way, a file can be shared across multiple directories, and still be referred to by the same CID. It allows IPFS nodes to efficiently store content without duplicating it. This can be used to share docker container layers for example.
In the filtering use case, it means that if 🐶 content is included in other IPFS content, our node can also prevent content linking to malicious 🐶 content from being served. This results in 😿, a mix of valid and malicious content.
By now, you should have an understanding of how an IPFS node retrieves and provides content, as well as how we can protect peers and users from shared nodes accessing threats. Using this knowledge, Cloudflare went on to implement IPFS Safemode, a node protection layer on top of go-ipfs. It is up to every node operator to build their own list of threats to be blocked based on their policy.
Et voilà. You are ready to use your IPFS node, with safemode capabilities.
# alias ipfs command to make it easier to use
alias ipfs=’./cmd/ipfs/ipfs’
# run an ipfs daemon
ipfs daemon &
# understand how to use IPFS safemode
ipfs safemode --help
USAGE
ipfs safemode - Interact with IPFS Safemode to prevent certain CIDs from being provided.
...
Going further
IPFS nodes are running in a diverse set of environments and operated by parties at various scales. The same software has to accommodate configuration in which it is accessed by a single-user, and others where it is shared by thousands of participants.
At Cloudflare, we believe that decentralization is going to be the next major step for content networks, but there is still work to be done to get these technologies in the hands of everyone. Content filtering is part of this story. If the community aims at embedding a P2P node in every computer, there needs to be ways to prevent nodes from serving harmful content. Users need to be able to give consent on the content they are willing to serve, and the one they aren’t.
By providing an IPFS safemode tool, we hope to make this protection more widely available.
The Domain Name System (DNS) matches names to resources. Instead of typing 104.18.26.46 to access the Cloudflare Blog, you type blog.cloudflare.com and, using DNS, the domain name resolves to 104.18.26.46, the Cloudflare Blog IP address.
Similarly, distributed systems such as Ethereum and IPFS rely on a naming system to be usable. DNS could be used, but its resolvers’ attributes run contrary to properties valued in distributed Web (dWeb) systems. Namely, dWeb resolvers ideally provide (i) locally verifiable data, (ii) built-in history, and (iii) have no single trust anchor.
At Cloudflare Research, we have been exploring alternative ways to resolve queries to responses that align with these attributes. We are proud to announce a new resolver for the Distributed Web, where IPFS content indexed by the Ethereum Name Service (ENS) can be accessed.
To discover how it has been built, and how you can use it today, read on.
Welcome to the Distributed Web
IPFS and its addressing system
The InterPlanetary FileSystem (IPFS) is a peer-to-peer network for storing content on a distributed file system. It is composed of a set of computers called nodes that store and relay content using a common addressing system.
This addressing system relies on the use of Content IDentifiers (CID). CIDs are self-describing identifiers, because the identifier is derived from the content itself. For example, QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco is the CID version 0 (CIDv0) of the wikipedia-on ipfs homepage.
To understand why a CID is defined as self-describing, we can look at its binary representation. For QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco, the CID looks like the following:
The first is the algorithm used to generate the CID (sha2-256 in this case); then comes the length of the encoded content (32 for a sha2-256 hash), and finally the content itself. When referring to the multicodec table, it is possible to understand how the content is encoded.
Name
Code (in hexadecimal)
identity
0x00
sha1
0x11
sha2-256
0x12 = 00010010
keccak-256
0x1b
This encoding mechanism is useful, because it creates a unique and upgradable content-addressing system across multiple protocols.
Ethereum is an account-based blockchain with smart contract capabilities. Being account-based, each account is associated with addresses and these can be modified by operations grouped in blocks and sealed by Ethereum’s consensus algorithm, Proof-of-Work.
There are two categories of accounts: user accounts and contract accounts. User accounts are controlled by a private key, which is used to sign transactions from the account. Contract accounts hold bytecode, which is executed by the network when a transaction is sent to their account. A transaction can include both funds and data, allowing for rich interaction between accounts.
When a transaction is created, it gets verified by each node on the network. For a transaction between two user accounts, the verification consists of checking the origin account signature. When the transaction is between a user and a smart contract, every node runs the smart contract bytecode on the Ethereum Virtual Machine (EVM). Therefore, all nodes perform the same suite of operations and end up in the same state. If one actor is malicious, nodes will not add its contribution. Since nodes have diverse ownership, they have an incentive to not cheat.
How to access IPFS content
As you may have noticed, while a CID describes a piece of content, it doesn’t describe where to find it. In fact, the CID describes the content, but not its location on the network. The location of the file would be retrieved by a query made to an IPFS node.
An IPFS URL (Unified Resource Locator) looks like this: ipfs://QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco. Accessing this URL means retrieving QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco using the IPFS protocol, denoted by ipfs://. However, typing such a URL is quite error-prone. Also, these URLs are not very human-friendly, because there is no good way to remember such long strings. To get around this issue, you can use DNSLink. DNSLink is a way of specifying IPFS CIDs within a DNS TXT record. For instance, wikipedia on ipfs has the following TXT record
In addition, its A record points to an IPFS gateway. This means that, when you access en.wikipedia-on-ipfs.org, your request is directed to an IPFS HTTP Gateway, which then looks out for the CID using your domain TXT record, and returns the content associated to this CID using the IPFS network.
This is trading ease-of-access against security. The web browser of the user doesn’t verify the integrity of the content served. This could be because the browser does not implement IPFS or because it has no way of validating domain signature — DNSSEC. We wrote about this issue in our previous blog post on End-to-End Integrity.
Human readable identifiers
DNS simplifies referring to IP addresses, in the same way that postal addresses are a way of referring to geolocation data, and contacts in your mobile phone abstract phone numbers. All these systems provide a human-readable format and reduce the error rate of an operation.
To verify these data, the trusted anchors, or “sources of truth”, are:
The government registry for postal addresses. In the UK, addresses are handled by cities, boroughs and local councils.
When it comes to your contacts, you are the trust anchor.
Ethereum Name Service, an index for the Distributed Web
An account is identified by its address. An address starts with “0x” and is followed by 20 bytes (ref 4.1 Ethereum yellow paper), for example: 0xf10326c1c6884b094e03d616cc8c7b920e3f73e0. This is not very readable, and can be pretty scary when transactions are not reversible and one can easily mistype a single character.
A first mitigation strategy was to introduce a new notation to capitalise some letters based on the hash of the address 0xF10326C1c6884b094E03d616Cc8c7b920E3F73E0. This can help detect mistype, but it is still not readable. If I have to send a transaction to a friend, I have no way of confirming she hasn’t mistyped the address.
The Ethereum Name Service (ENS) was created to tackle this issue. It is a system capable of turning human-readable names, referred to as domains, to blockchain addresses. For instance, the domain privacy-pass.eth points to the Ethereum address 0xF10326C1c6884b094E03d616Cc8c7b920E3F73E0.
To achieve this, the system is organised in two components, registries and resolvers.
A registry is a smart contract that maintains a list of domains and some information about each domain: the domain owner and the domain resolver. The owner is the account allowed to manage the domain. They can create subdomains and change ownership of their domain, as well as modify the resolver associated with their domain.
Resolvers are responsible for keeping records. For instance, Public Resolver is a smart contract capable of associating not only a name to blockchain addresses, but also a name to an IPFS content identifier. The resolver address is stored in a registry. Users then contact the registry to retrieve the resolver associated with the name.
Consider a user, Alice, who has direct access to the Ethereum state. The flow goes as follows: Alice would like to get Privacy Pass’s Ethereum address, for which the domain is privacy-pass.eth. She looks for privacy-pass.eth in the ENS Registry and figures out the resolver for privacy-pass.eth is at 0x1234… . She now looks for the address of privacy-pass.eth at the resolver address, which turns out to be 0xf10326c….
Accessing the IPFS content identifier for privacy-pass.eth works in a similar way. The resolver is the same, only the accessed data is different — Alice calls a different method from the smart contract.
Cloudflare Distributed Web Resolver
The goal was to be able to use this new way of indexing IPFS content directly from your web browser. However, accessing the ENS registry requires access to the Ethereum state. To get access to IPFS, you would also need to access the IPFS network.
To tackle this, we are going to use Cloudflare’s Distributed Web Gateway. Cloudflare operates both an Ethereum Gateway and an IPFS Gateway, respectively available at cloudflare-eth.com and cloudflare-ipfs.com.
EthLink
The first version of EthLink was built by Jim McDonald and is operated by True Name LTD at eth.link. Starting from next week, eth.link will transition to use the Cloudflare Distributed Web Resolver. To that end, we have built EthLink on top of Cloudflare Workers. This is a proxy to IPFS. It proxies all ENS registered domains when .link is appended. For instance, privacy-pass.eth should render the Privacy Pass homepage. From your web browser, https://privacy-pass.eth.link does it.
The resolution is done at the Cloudflare edge using a Cloudflare Worker. Cloudflare Workers allows JavaScript code to be run on Cloudflare infrastructure, eliminating the need to maintain a server and increasing the reliability of the service. In addition, it follows Service Workers API, so results returned from the resolver can be checked by end users if needed.
To do this, we setup a wildcard DNS record for *.eth.link to be proxied through Cloudflare and handled by a Cloudflare Worker. When a user Alice accesses privacy-pass.eth.link, the worker first gets the CID of the CID to be retrieved from Ethereum. Then, it requests the content matching this CID to IPFS, and returns it to Alice.
All parts can be run locally. The worker can be run in a service Worker, and the Ethereum Gateway can point to both a local Ethereum node and the IPFS gateway provided by IPFS Companion. It means that while Cloudflare provides resolution-as-a-service, none of the components has to be trusted.
Final notes
So are we distributed yet? No, but we are getting closer, building bridges between emerging technologies and current web infrastructure. By providing a gateway dedicated to the distributed web, we hope to make these services more accessible to everyone.
We thank the ENS team for their support of a new resolver on expanding the distributed web. The ENS team has been running a similar service at https://eth.link. On January 18th, they will switch https://eth.link to using our new service.
These services benefit from the added speed and security of the Cloudflare Worker platform, while paving the way to run distributed protocols in browsers.
The collective thoughts of the interwebz
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.
