All posts by Steve Welham

How Automatic Return Routing solves IP overlap

Post Syndicated from Steve Welham original https://blog.cloudflare.com/automatic-return-routing-ip-overlap/

The public Internet relies on a fundamental principle of predictable routing: a single IP address points to a logically unique destination. Even in an Anycast architecture like Cloudflare’s, where one IP is announced from hundreds of locations, every instance of that IP represents the same service. The routing table always knows exactly where a packet is intended to go.

This principle holds up because global addressing authorities assign IP space to organizations to prevent duplication or conflict. When everyone adheres to a single, authoritative registry, a routing table functions as a source of absolute truth.

On the public Internet, an IP address is like a unique, globally registered national identity card. In private networks, an IP is just a name like “John Smith”, which is perfectly fine until you have three of them in the same room trying to talk to the same person.

As we expand Cloudflare One to become the connectivity cloud for enterprise backbones, we’ve entered the messy reality of private IP address space. There are good reasons why duplication arises, and enterprises need solutions to handle these conflicts.

Today, we are introducing Automatic Return Routing (ARR) in Closed Beta. ARR is an optional tool for Cloudflare One customers that gives you the flexibility to route traffic back to where it originated, without requiring an IP route in a routing table. This capability allows overlapping networks to coexist without a single line of Network Address Translation (NAT) or complex Virtual Routing and Forwarding (VRF) configuration.

The ambiguity problem

In enterprise networking, IP overlap is a fact of life. We see it in three common scenarios that traditionally cause toil for admins:

  • Mergers & acquisitions: Two companies merge, and both use 10.0.1.0/24 for their core services.

  • Extranets: Partners, vendors or customers securely connect to your network using their own internal IP schemes, leading to unavoidable conflicts.

  • Cookie-cutter architectures: SaaS providers or retail brands use identical IP space for every branch to simplify deployment and operation.

The problem arises when these sites try to talk to the Internet or a data center through Cloudflare. If two different sites send traffic from the same source IP, the return packet hits an architectural wall. The administrator has to make a decision on how to route the traffic based on the ambiguous destination. If the administrator puts both routes into the routing table, it will be non-deterministic as to which path is taken: the correct path or the incorrect path. From the perspective of a standard routing table, there is no way to distinguish between two identical paths.


This diagram shows two branches (Site A and Site B) both using 10.0.1.0/24. They send packets to Cloudflare. The return packet from the Internet reaches the Cloudflare edge, and this return traffic is sometimes sent to the wrong site because the routing table has two identical egress options.

Why traditional fixes fail

There are numerous ways to resolve this ambiguity, and we are committed to solving them in the easiest way for our customers to manage. The traditional “industry standard” fixes are functional, but they introduce significant administrative overhead and complexity that we are committed to eliminating:

  1. Virtual Routing and Forwarding (VRF): This involves creating “virtual” routing tables to keep traffic isolated. While effective for separation, it adds administrative overhead. Managing cross-VRF communication (route leaking) is brittle and complex at scale. 

  2. Network Address Translation (NAT): You can NAT each overlapping subnet from an unmanaged IP space to a managed IP range that is unique in your network. This approach works well, but the mapping is administrative toil for each new site or partner.

Typically, the use case we hear from customers is an overlapping network needing to access the Internet or a private data center. How do we solve this without administrative overhead?

Introducing Automatic Return Routing (ARR)

We developed ARR as a “zero-touch” solution to this problem. ARR moves the intelligence from the routing table to stateful tracking.

So what is stateful tracking?

In traditional networking, a router is “forgetful” (aka “stateless”). It treats every single packet like a total stranger. Even if it just saw a packet from the exact same source going to the exact same destination a millisecond ago, it has to look at its routing table all over again to decide where to send the next one.

With stateful tracking, the system has a memory. It recognizes when a series of packets are all part of the same “flow” (that is, a network conversation between two endpoints), and remembers key information about that flow until it finishes. With ARR, we remember one extra piece of information when initializing the flow: the specific tunnel that initiated it. This allows us to send return traffic back to that same tunnel, without ever consulting a routing table!

Instead of asking the network, “Where does this IP live?” ARR asks, “Where did this specific conversation originate?”

The Logic:

  1. Ingress: A packet arrives at the Cloudflare edge from a site via a specific connection, i.e. an IPsec tunnel, GRE tunnel, or Network Interconnect.

  2. Flow Matching: The Cloudflare Virtual Network first checks (by header inspection) whether that packet matches an existing flow.

    1. Proxying: If the packet matches, that’s great! All of the decisions about this traffic have already been made and stored in our memory. All we need to do is pass that packet along already-established paths.

    2. Flow Setup: If it doesn’t match an existing flow, we decide which parts of the Cloudflare One stack to pass it through (e.g. Gateway, DLP, Firewall), as well as its ultimate destination. We store all of this state in memory. With ARR, this is when we record which tunnel initiated the flow.

  3. Symmetric Return: When return traffic arrives from the destination, the Cloudflare Virtual Network uses its existing in-memory state to proxy the traffic. Crucially, it does this without needing to examine the traffic’s destination IP, which could very well be reused across different sites. This completely bypasses the need to consult a routing table. We see the originating tunnel in the flow state and deliver the packet directly back to it.


Example of overlapping source IPs tracked by in-memory flow state, tagged with source onramp to inform return routing decision.

By remembering the originating tunnel for every flow, ARR facilitates zero-touch routing. If your site traffic is only client-to-Internet, there is no need to configure return routes at all, reducing toil when deploying new branch sites or “Coffee Shop Networking.”

Built on Unified Routing

To make ARR a reality at Cloudflare scale, we plugged into another initiative we have been working on: Unified Routing.

Historically, Cloudflare Zero Trust (users/proxies) and Cloudflare WAN (network-layer/sites) lived at different levels of the system. Cloudflare WAN relied on kernel primitives (Linux network namespaces, routes, eBPF, etc). Zero Trust lived in userspace, where proxies could perform deep inspection and application-level security. This “split-brain” approach often required complex logic to move traffic between component services, and some of this complexity became product limitations that customers might notice.



With our new Unified Routing mode, we have moved the initial routing decision from our network-layer data plane into our existing Zero Trust userspace routing logic, the same hardened software used by Cloudflare One Clients and Cloudflare Tunnel in our Zero Trust solution. This change has many benefits to how we enable our customers to use their private networks with products across the Cloudflare platform, as it fixes long-standing interoperability problems between Cloudflare WAN and Zero Trust. Unified Routing means you can use Cloudflare Mesh, Cloudflare Tunnel, and IPsec/GRE on-ramps together in the same account without a single conflict.


In September 2025, we deployed Unified Routing mode internally for all Cloudflare employees and sites. We saw immediate 3-5x performance improvements for Cloudflare One Clients, as you can see in the graph above.

When designing ARR, we knew that we needed to move away from kernel-based routing and build on our new Unified Routing framework.

When Unified Routing is enabled, all Cloudflare WAN traffic flows through Apollo, our Zero Trust hub. Unlike the Linux kernel’s standard routing table, our userspace data plane is fully programmable. We can attach metadata, like the originating Tunnel ID, directly to a flow entry in Apollo. 

Each packet is tracked by flow from the moment it hits our edge, and we no longer need to make independent, per-packet routing decisions. Instead, we can make consistent, session-aware decisions for the lifetime of the flow.

ARR is straightforward to enable on a per tunnel or interconnect basis:


Once enabled for a tunnel or interconnect, any traffic that matches an existing flow is routed back to the connection where it originated, without consulting the routing table.

Putting ARR to work

For the enterprise architect, ARR is a tool to bypass the persistent friction of IP address conflicts. Whether integrating an acquisition or onboarding a partner, the goal is to make the network invisible, so you can focus on the applications, not the plumbing.

Today, ARR is in closed beta and supports overlapping IP addresses accessing the Internet via our Secure Web Gateway. We are already extending this to support private data center access, adding mid-flow failover (pinning the flow to a primary onramp, and seamlessly detecting when that flow fails over to a backup onramp), and further investing in the architectural capabilities needed to make IP overlap a non-issue for even the most complex global deployments.

Not using Cloudflare One yet? Start now with our Free and Pay-as-you-go plans to protect and connect your users and networks, and contact us for comprehensive private WAN connectivity via IPsec and private interconnect.

Magic Cloud Networking simplifies security, connectivity, and management of public clouds

Post Syndicated from Steve Welham original https://blog.cloudflare.com/introducing-magic-cloud-networking


Today we are excited to announce Magic Cloud Networking, supercharged by Cloudflare’s recent acquisition of Nefeli Networks’ innovative technology. These new capabilities to visualize and automate cloud networks will give our customers secure, easy, and seamless connection to public cloud environments.

Public clouds offer organizations a scalable and on-demand IT infrastructure without the overhead and expense of running their own datacenter. Cloud networking is foundational to applications that have been migrated to the cloud, but is difficult to manage without automation software, especially when operating at scale across multiple cloud accounts. Magic Cloud Networking uses familiar concepts to provide a single interface that controls and unifies multiple cloud providers’ native network capabilities to create reliable, cost-effective, and secure cloud networks.

Nefeli’s approach to multi-cloud networking solves the problem of building and operating end-to-end networks within and across public clouds, allowing organizations to securely leverage applications spanning any combination of internal and external resources. Adding Nefeli’s technology will make it easier than ever for our customers to connect and protect their users, private networks and applications.

Why is cloud networking difficult?

Compared with a traditional on-premises data center network, cloud networking promises simplicity:

  • Much of the complexity of physical networking is abstracted away from users because the physical and ethernet layers are not part of the network service exposed by the cloud provider.
  • There are fewer control plane protocols; instead, the cloud providers deliver a simplified software-defined network (SDN) that is fully programmable via API.
  • There is capacity — from zero up to very large — available instantly and on-demand, only charging for what you use.

However, that promise has not yet been fully realized. Our customers have described several reasons cloud networking is difficult:

  • Poor end-to-end visibility: Cloud network visibility tools are difficult to use and silos exist even within single cloud providers that impede end-to-end monitoring and troubleshooting.
  • Faster pace: Traditional IT management approaches clash with the promise of the cloud: instant deployment available on-demand. Familiar ClickOps and CLI-driven procedures must be replaced by automation to meet the needs of the business.
  • Different technology: Established network architectures in on-premises environments do not seamlessly transition to a public cloud. The missing ethernet layer and advanced control plane protocols were critical in many network designs.
  • New cost models: The dynamic pay-as-you-go usage-based cost models of the public clouds are not compatible with established approaches built around fixed cost circuits and 5-year depreciation. Network solutions are often architected with financial constraints, and accordingly, different architectural approaches are sensible in the cloud.
  • New security risks: Securing public clouds with true zero trust and least-privilege demands mature operating processes and automation, and familiarity with cloud-specific policies and IAM controls.
  • Multi-vendor: Oftentimes enterprise networks have used single-vendor sourcing to facilitate interoperability, operational efficiency, and targeted hiring and training. Operating a network that extends beyond a single cloud, into other clouds or on-premises environments, is a multi-vendor scenario.

Nefeli considered all these problems and the tensions between different customer perspectives to identify where the problem should be solved.

Trains, planes, and automation

Consider a train system. To operate effectively it has three key layers:

  • tracks and trains
  • electronic signals
  • a company to manage the system and sell tickets.

A train system with good tracks, trains, and signals could still be operating below its full potential because its agents are unable to keep up with passenger demand. The result is that passengers cannot plan itineraries or purchase tickets.

The train company eliminates bottlenecks in process flow by simplifying the schedules, simplifying the pricing, providing agents with better booking systems, and installing automated ticket machines. Now the same fast and reliable infrastructure of tracks, trains, and signals can be used to its full potential.

Solve the right problem

In networking, there are an analogous set of three layers, called the networking planes:

  • Data Plane: the network paths that transport data (in the form of packets) from source to destination.
  • Control Plane: protocols and logic that change how packets are steered across the data plane.
  • Management Plane: the configuration and monitoring interfaces for the data plane and control plane.

In public cloud networks, these layers map to:

  • Cloud Data Plane: The underlying cables and devices are exposed to users as the Virtual Private Cloud (VPC) or Virtual Network (VNet) service that includes subnets, routing tables, security groups/ACLs and additional services such as load-balancers and VPN gateways.
  • Cloud Control Plane: In place of distributed protocols, the cloud control plane is a software defined network (SDN) that, for example, programs static route tables. (There is limited use of traditional control plane protocols, such as BGP to interface with external networks and ARP to interface with VMs.)
  • Cloud Management Plane: An administrative interface with a UI and API which allows the admin to fully configure the data and control planes. It also provides a variety of monitoring and logging capabilities that can be enabled and integrated with 3rd party systems.

Like our train example, most of the problems that our customers experience with cloud networking are in the third layer: the management plane.

Nefeli simplifies, unifies, and automates cloud network management and operations.

Avoid cost and complexity

One common approach to tackle management problems in cloud networks is introducing Virtual Network Functions (VNFs), which are virtual machines (VMs) that do packet forwarding, in place of native cloud data plane constructs. Some VNFs are routers, firewalls, or load-balancers ported from a traditional network vendor’s hardware appliances, while others are software-based proxies often built on open-source projects like NGINX or Envoy. Because VNFs mimic their physical counterparts, IT teams could continue using familiar management tooling, but VNFs have downsides:

  • VMs do not have custom network silicon and so instead rely on raw compute power. The VM is sized for the peak anticipated load and then typically runs 24x7x365. This drives a high cost of compute regardless of the actual utilization.
  • High-availability (HA) relies on fragile, costly, and complex network configuration.
  • Service insertion — the configuration to put a VNF into the packet flow — often forces packet paths that incur additional bandwidth charges.
  • VNFs are typically licensed similarly to their on-premises counterparts and are expensive.
  • VNFs lock in the enterprise and potentially exclude them benefitting from improvements in the cloud’s native data plane offerings.

For these reasons, enterprises are turning away from VNF-based solutions and increasingly looking to rely on the native network capabilities of their cloud service providers. The built-in public cloud networking is elastic, performant, robust, and priced on usage, with high-availability options integrated and backed by the cloud provider’s service level agreement.

In our train example, the tracks and trains are good. Likewise, the cloud network data plane is highly capable. Changing the data plane to solve management plane problems is the wrong approach. To make this work at scale, organizations need a solution that works together with the native network capabilities of cloud service providers.

Nefeli leverages native cloud data plane constructs rather than third party VNFs.

Introducing Magic Cloud Networking

The Nefeli team has joined Cloudflare to integrate cloud network management functionality with Cloudflare One. This capability is called Magic Cloud Networking and with it, enterprises can use the Cloudflare dashboard and API to manage their public cloud networks and connect with Cloudflare One.

End-to-end

Just as train providers are focused only on completing train journeys in their own network, cloud service providers deliver network connectivity and tools within a single cloud account. Many large enterprises have hundreds of cloud accounts across multiple cloud providers. In an end-to-end network this creates disconnected networking silos which introduce operational inefficiencies and risk.

Imagine you are trying to organize a train journey across Europe, and no single train company serves both your origin and destination. You know they all offer the same basic service: a seat on a train. However, your trip is difficult to arrange because it involves multiple trains operated by different companies with their own schedules and ticketing rates, all in different languages!

Magic Cloud Networking is like an online travel agent that aggregates multiple transportation options, books multiple tickets, facilitates changes after booking, and then delivers travel status updates.

Through the Cloudflare dashboard, you can discover all of your network resources across accounts and cloud providers and visualize your end-to-end network in a single interface. Once Magic Cloud Networking discovers your networks, you can build a scalable network through a fully automated and simple workflow.

Resource inventory shows all configuration in a single and responsive UI

Taming per-cloud complexity

Public clouds are used to deliver applications and services. Each cloud provider offers a composable stack of modular building blocks (resources) that start with the foundation of a billing account and then add on security controls. The next foundational layer, for server-based applications, is VPC networking. Additional resources are built on the VPC network foundation until you have compute, storage, and network infrastructure to host the enterprise application and data. Even relatively simple architectures can be composed of hundreds of resources.

The trouble is, these resources expose abstractions that are different from the building blocks you would use to build a service on prem, the abstractions differ between cloud providers, and they form a web of dependencies with complex rules about how configuration changes are made (rules which differ between resource types and cloud providers). For example, say I create 100 VMs, and connect them to an IP network. Can I make changes to the IP network while the VMs are using the network? The answer: it depends.

Magic Cloud Networking handles these differences and complexities for you. It configures native cloud constructs such as VPN gateways, routes, and security groups to securely connect your cloud VPC network to Cloudflare One without having to learn each cloud’s incantations for creating VPN connections and hubs.

Continuous, coordinated automation

Returning to our train system example, what if the railway maintenance staff find a dangerous fault on the railroad track? They manually set the signal to a stop light to prevent any oncoming trains using the faulty section of track. Then, what if, by unfortunate coincidence, the scheduling office is changing the signal schedule, and they set the signals remotely which clears the safety measure made by the maintenance crew? Now there is a problem that no one knows about and the root cause is that multiple authorities can change the signals via different interfaces without coordination.

The same problem exists in cloud networks: configuration changes are made by different teams using different automation and configuration interfaces across a spectrum of roles such as billing, support, security, networking, firewalls, database, and application development.

Once your network is deployed, Magic Cloud Networking monitors its configuration and health, enabling you to be confident that the security and connectivity you put in place yesterday is still in place today. It tracks the cloud resources it is responsible for, automatically reverting drift if they are changed out-of-band, while allowing you to manage other resources, like storage buckets and application servers, with other automation tools. And, as you change your network, Cloudflare takes care of route management, injecting and withdrawing routes globally across Cloudflare and all connected cloud provider networks.

Magic Cloud Networking is fully programmable via API, and can be integrated into existing automation toolchains.

The interface warns when cloud network infrastructure drifts from intent

Ready to start conquering cloud networking?

We are thrilled to introduce Magic Cloud Networking as another pivotal step to fulfilling the promise of the Connectivity Cloud. This marks our initial stride in empowering customers to seamlessly integrate Cloudflare with their public clouds to get securely connected, stay securely connected, and gain flexibility and cost savings as they go.

Join us on this journey for early access: learn more and sign up here.