Post Syndicated from Sam Rhea original https://blog.cloudflare.com/argo-tunnels-that-live-forever/

Cloudflare secures your origin servers by proxying requests to your DNS records through our anycast network and to the external IP of your origin. However, external IP addresses can provide attackers with a path around Cloudflare security if they discover those destinations.

We launched Argo Tunnel as a secure way to connect your origin to Cloudflare without a publicly routable IP address. With Tunnel, you don’t send traffic to an external IP. Instead, a lightweight daemon runs in your infrastructure and creates outbound-only connections to Cloudflare’s edge. With Argo Tunnel, you can quickly deploy infrastructure in a Zero Trust model by ensuring all requests to your resources pass through Cloudflare’s security filters.

Originally, your Argo Tunnel connection corresponded to a DNS record in your account. Requests to that hostname hit Cloudflare’s network first and our edge sends those requests over the Argo Tunnel to your origin. Since these connections are outbound-only, you no longer need to poke holes in your infrastructure’s firewall. Your origins can serve traffic through Cloudflare without being vulnerable to attacks that bypass Cloudflare.

However, fitting an outbound-only connection into a reverse proxy creates some ergonomic and stability hurdles. The original Argo Tunnel architecture attempted to both manage DNS records and create connections. When connections became disrupted, Argo Tunnel would recreate the entire deployment. Additionally, Argo Tunnel connections could not be treated like regular origin servers in Cloudflare’s control plane and had to be managed directly from the server-side software.

Today, we’re introducing a new architecture that treats Argo Tunnel connections like a true origin server without the risk of exposure to the rest of the Internet. Now, when you create a Tunnel connection, you can point DNS records for any hostname in your account, or load balancer pools, to that connection from the Cloudflare dashboard. You can also run Argo Tunnel connections without the need for leaving certificates and service tokens on your servers.

Keeping persistent objects persistent

Argo Tunnel has objects that tend to stay persistent (DNS records) and objects that deliberately change and recreate (connections from `cloudflared` to Cloudflare). Argo Tunnel previously conflated the two categories, which led to some issues.

The edge vs. the control plane

Cloudflare as a whole consists of two components: the edge network and the control plane that manages the configuration of that network.

The data centers in 200 cities around the world that proxy traffic to your origin make up the edge network. These data centers are highly available and, thanks to Anycast IP routing, can gracefully handle traffic if one or more data centers go offline.

When you make a change to something in Cloudflare (whether via the UI in Cloudflare’s dashboard, or the API) our control plane receives it, authenticates it, and then pushes it to our edge.

If the control plane goes down, the edge should not be degraded – traffic will continue to be served using the most recent configuration. At launch, Argo Tunnel muddled the two in some places, which meant that control plane issues could become edge issues for Tunnel users.

Starting every Tunnel from scratch

Regardless of whether a Tunnel is connecting for the first time or the 100th, the operation repeated a series of high-level steps in the original architecture:

1. cloudflared connects to an Argo Tunnel service running in Cloudflare’s control plane. That service registers your Tunnel and its connections.
2. cloudflared creates a public DNS record for your hostname which points to a randomly generated CNAME record for load balanced Tunnels or an IPv6 for traditional Tunnels. The ephemeral CNAME record represents your Tunnel.
3. The control plane then tells Cloudflare’s edge about these DNS entries and where the CNAME or IP address should send traffic. Traffic can now be routed to cloudflared.
4. If the Tunnel disconnects, for any reason, the Argo Tunnel service unregistered the Tunnel and deleted the DNS record.

The last step is an issue. In most cases, you create an Argo Tunnel for a service meant to run indefinitely. The DNS record should stay persistent – it’s an app that you manage that should not change. However, a simple restart or disconnection meant that cloudflared had to follow every step and start itself from scratch. If any of those upstream services were degraded, the Tunnel would fail to reconnect.

This model also introduces other shortcomings. You cannot gracefully change the DNS record of a Tunnel; instead, you had to stop cloudflared and rerun the service. Visibility was limited. Load balancing introduced complications with how origins were counted.

Phase 1: Improving stability

The team started by reducing the impact of those dependencies. Over the last year, Argo Tunnel has quietly replaced single points of failure with distributed systems that are more fault tolerant.

Tunnels now live longer. Argo Tunnel has migrated to Cloudflare’s Unimog platform, which has increased the average life of a connection from minutes to days. When connections live longer, they restart less, and are then subject to fewer upstream hiccups.

Additionally, some Tunnels no longer need to follow the entire creation flow. If your Tunnel reconnects, we opportunistically try to reestablish it with the records already at our edge.

These changes have dramatically improved the stability of Argo Tunnel as a platform, but still left a couple of core problems: Tunnel reconnections were treated like new connections and managing those connections added friction.

Phase 2: Named Tunnels that outlive connections

Starting today, Argo Tunnel’s architecture distinguishes between the persistent objects (DNS records, cloudflared) from the ephemeral objects (the connections). To do that, this release introduces the concept of a permanent name that you assign to a Tunnel.

In the old model, cloudflaredcreated both the DNS record entries and established the connections from the server to Cloudflare’s network. DNS records became tied to those connections and could not be changed. Even worse, each time cloudflared restarted, we treated it like a new Tunnel and had to propagate this information into DNS and Load Balancer systems. If those had delays, the restart could become an outage.

Today’s release separates DNS creation from connection creation to make tunnels more stable and more simple to manage. In this model, you can use `cloudflared` to create an Argo Tunnel that has a persistent, stable name, that can be entirely unrelated to the hostname.

Once created, you can point DNS records in your account to a stable subdomain that relies on a UUID tied to that persistent name. Since the name and UUID do not change, your DNS record never needs to be cleaned up or recreated when Argo Tunnel restarts. In the event of a restart, the enrolled instance of cloudflared connects back to that UUID address.

You can also treat named Argo Tunnels like origin servers in this architecture – except these origins can only be connected to via a DNS record in your account. You can delete a DNS record and create a new one that points to the UUID address and traffic will be served – all without touching cloudflared.

How it works

You can begin using this new architecture today with the following steps. First, you’ll need to upgrade to the latest version of cloudflared.

1. Login to Cloudflare from `cloudflared`

Run cloudflared tunnel login and authenticate to your Cloudflare account. This step will generate a cert.pem file. That certificate contains a token that gives your instance of cloudflared the ability to create Named Tunnels in your account, as well as the ability to eventually point DNS records to them.

2. Create your Tunnel

You can now create a Tunnel that has a persistent name. Run cloudflared tunnel create <name> to do so. The name does not have to be a hostname. For example, you can assign a name that represents the application, the particular server, or the cloud environment where it runs.

cloudflared will create a Tunnel with the name that you give it and a UUID. This name will be associated with your account. Only DNS records in your account will proxy traffic to the connection. Additionally, the name will not be removed unless you actively delete it. The connections can stop and restart and will use the same name and UUID.

Creating a named Tunnel also generates a credentials file that is distinct from the cert.pem issued during the login. You only need the credentials file to run the Tunnel. If you do not want to create additional named Tunnels or DNS records from cloudflared, you can delete the cert.pem file to avoid leaving API tokens and certificates in your environment.

3. Configure Tunnel details

Configure your instance of cloudflared, including the URL that cloudflared will proxy traffic to in the configuration file. Alternatively, you can run the Tunnel in an ad hoc mode from the command line using the steps below.

4. Run your Tunnel

You can begin running the Tunnel with the command, cloudflared tunnel run <name> or cloudflared tunnel run <UUID> and it will start proxying traffic. If you are running the Tunnel without the cert.pem file and only the credentials file, you must use cloudflared tunnel run <UUID>.

5. Send traffic to your Tunnel

You can now decide how to send traffic to this persistent Tunnel. If you want to create a long-lived DNS record in the Cloudflare dashboard, you can point it to the Tunnel UUID subdomain in the format UUID.cfargotunnel.com. You can do the same in the Cloudflare Load Balancer panel to add this object to a load balanced pool where it will be treated as just one additional origin.

Alternatively, you can continue to create DNS records from cloudflared. Run the following command, cloudflared tunnel route dns <name> <hostname> or cloudflared tunnel route dns <UUID> <hostname> to associate the DNS record with the Tunnel address. You will only be able to create a DNS record from cloudflared for the zone name you selected when authenticating. Unlike the previous architecture, this DNS record will not be deleted if the Tunnel disconnects.

When this instance of cloudflared restarts, the name, UUID, and DNS record will not need to be recreated. The connection will reestablish and begin serving traffic.

[Optional] Check what Tunnels exist

You can also use this architecture to see your active Tunnels. Run cloudflared tunnel list to view the Tunnels created and their connection status. You can delete Tunnels, as well, by running cloudflared tunnel delete <name> or cloudflare tunnel delete <UUID>. To delete Tunnels, you do need the cert.pem file.

Credential and cert management

Once you have created a named Tunnel, you no longer need the cert.pem file to run that Tunnel and connect it to Cloudflare’s network. If you’re running the tunnel on a remote server or in a container, you can copy the credential file without sharing cert.pem outside your computer.

Similarly, if you want to let another person on your team run the Tunnel, you can send them the credentials file without sharing the cert.pem file as well. The cert.pem file is still required to create additional Tunnels, list existing tunnels, manage DNS records, or delete Tunnels.

The credentials file contains a secret scoped to the specific Tunnel UUID which establishes a connection from cloudflared to Cloudflare’s network. cloudflared operates like a client and establishes a TLS connection from your infrastructure to Cloudflare’s edge.

What’s next?

The new Argo Tunnel architecture is available today. You’ll need cloudflared version 2020.9.3 or later to begin using these features. The latest version of cloudflared is backwards compatible with the legacy model of Argo Tunnel. Additional documentation is available here.