All posts by Dominik Picheta

Introducing on-demand CPU and memory profiling with flamegraphs for Workers and Durable Objects

Post Syndicated from Dominik Picheta original https://blog.cloudflare.com/workers-on-demand-profiling/

Understanding why an application is using more resources than expected, whether that is CPU or memory, can be challenging. Logs and aggregate metrics can only get you so far. Thankfully there is a better way: CPU or memory profiling can show you the exact function where your application is using CPU or allocating memory.

Today we are happy to announce support for CPU and memory profiling of Workers and Durable Objects. From the Workers Observability page, you can now request an on-demand CPU or memory profile of an active Worker, inspect it as an interactive flamegraph, and download the profile file for further analysis.

This method of profiling gives you a useful perspective into what your code is doing and how you can improve it in a real-world setting. That’s because the best way to understand your application is to profile it in production.

To try this on one of your Workers, you have a choice of using the CLI or the Cloudflare Dashboard. 

To use the CLI, make sure you have the cf package installed, then simply run:

To use the dashboard, head to the Cloudflare dashboard and then access the list of Workers on your account by navigating through Build → Compute → Workers & Pages.

Select your Worker and navigate to its Observability tab. You can then use the drop down to select “Flamegraph”:

You can then request both CPU and memory profiles for your Worker on this page. The duration determines how long the profiler should run for on your Worker. You can also select different versions of your Worker to be profiled. Your Worker needs plenty of traffic to be successfully profiled, so make sure you choose a version that has enough traffic.

Once you click the Capture Profile button, the profiler will run for the duration that you have selected, and you will then see a flamegraph representing the results of the profile being rendered. Each rectangle in the flamegraph represents a function call, with the width representing the amount of CPU time or memory being used by that function. You can click on different functions to focus on and hover to see more details about them:

Don’t be afraid to capture a couple of profiles, then look for the widest functions to determine what is using the most resources in your Worker. You might also find that the table view is useful to get a quick sense of which functions are most commonly seen in the profile.

If your Worker is implemented in TypeScript, then you should make sure that you have source maps enabled for your project, otherwise your profile may show obfuscated function names that aren’t easy to understand.

We have already seen multiple teams at Cloudflare using CPU and memory profiling to identify opportunities for optimizing memory and CPU usage, fixing out-of-memory (OOM) errors, and improving performance. Below are some examples of this.

Using CPU profiling to find optimizations

Let’s look at a real example of how you can use CPU profiling to optimize your Workers. We will look at a CPU profile of the Worker that implements the R2 binding. This Worker receives a lot of traffic and so eliminating even the smallest amount of wasted CPU cycles can have a massive impact on its performance.

We set up the profiler to capture CPU traces, with a duration of 50 seconds. Once we received the rendered profile, we saw this:

As mentioned previously, the widest boxes are those which are using up most of the CPU time. Some of them aren’t surprising, for example decryptBlock is R2 decrypting object data on the way out, and fillResponse is just moving bytes. There weren’t any clear ways to optimize these functions.

As some of the boxes are less visible, it can be a good idea to take a look at the table view. Sorting by Samples points us at genericR2JsonReplacer:

This function represented over 5% of the CPU time, but what caught our attention was the function calling itself recursively. This looked like wasted work, and it was.

Even though the replacer was being called by JSON.stringify on every node in the JSON tree, the replacer itself was also walking the tree. This meant that a value nested five levels deep was processed five times. Fixing this enabled us to eliminate the duplicate work and make the genericR2JsonReplacer function 2.7x faster.

The second candidate in the profile that we looked at was a duplicate call to metrics.

When looking at the code, we saw something close to this:

The metrics call was heavy enough that just one call represented 1% of the CPU time in the profile. So storing the result of the first call in a variable and then not calling it again saved us a fair chunk of CPU time.

How we used Worker profiling internally to fix memory problems

Internally we had a Worker with a memory problem. P999 memory sat around 133 MB against the 128 MB Worker memory limit. This caused the Worker to be evicted frequently with an “Exceeded Memory” error.

The image above shows a screenshot of the “Errors by invocation status” graph, with the number of “Exceeded Memory” errors clearly dropping as a result of the fixes identified through profiling.

What was hard to work out was what was causing these errors. The errors and metrics pointed at memory as the source of the problem, but it didn’t explain which part of the code caused it.

The team took a heap profile of one of the Workers running in production and opened it with pprof. They saw that their Prometheus code accounted for roughly 66.7% of allocations in the profile. This code was supposed to be disabled, but enough of it was still running that it created a problem. Because it was instrumenting a lot of the code paths in the Worker, it ended up using a lot of its memory, even though the collected data never actually left the Worker.

This code wasn’t suspected as a culprit of the memory issues, because it was thought to have been disabled. The profiler showed that the code was only partially disabled, with the memory cost still being paid as if it was fully enabled.

The team removed the Prometheus code path completely. This quickly improved the memory usage of the Worker:

Percentile

Before (MB)

After (MB)

P50

70

54

P90

94

79

P99

113

97

P999

133

118

This gave the P999 about 10 MB of headroom below the 128 MB limit.

Following a profiling request to the edge

Workers have supported local profiling through Chrome DevTools for a while, but it has not been possible to start a profiling session on Workers running in production. While profiling a Worker locally is useful, the Worker does not receive the same kinds nor number of requests as one does in production. So how do we profile a Worker or a Durable Object running there?

One of the best features of Workers is that their requests are routed to the data center that is nearest to the requesting client. This reduces latency, but it does mean that Workers must be replicated across different data centers and metals (physical servers). In addition, when a Worker is receiving a lot of requests in a particular data center, it can be replicated in a particular metal multiple times to handle all the traffic.

Durable Objects add another layer of indirection: they can be placed dynamically and can move.

This and a few other features of Workers means that before we can start a profiling session, we need to first answer a few questions:

  • Which version does the developer want to profile?
  • In which data center has that version run recently?
  • Is the Worker’s isolate still loaded in that data center?
  • Does the isolate belong exclusively to the requesting account?
  • For a Durable Object, where is the exact live primary actor?

A few of these questions are answered by the client making the profiling request. In most cases this will be the dashboard, but you can also send these requests yourself (or have your coding agent do it for you). An example request might look something like this:

Notably, this specifies both the script name and version that should be profiled as part of the request URL.

Right now, a new isolate is not started to produce a profile, because the goal is to observe real production execution. This means that if your Worker is receiving very little traffic, you may find that it is hard to profile. So it’s important you pick a version of your Worker that is deployed.

Profiling an isolate without stopping the Worker

A profile is useful only if the application can continue to execute while samples are collected. The Workers Runtime therefore holds the isolate lock only for profiler lifecycle operations. For a CPU profile, the process looks like this:

  1. Acquire the isolate and its lock.
  2. Create a V8 CPU profiler and begin sampling at a one millisecond interval.
  3. Release the lock so normal requests can run.
  4. Wait for the requested duration.
  5. Reacquire the lock and stop profiling.
  6. Release the lock and serialize the result outside the critical section.

Holding the lock through the duration of the profile would prevent JavaScript execution from continuing, which would make the profile useless.

Durable Objects

Durable Objects are different from Workers. Regular Workers are stateless: no one Worker is special or different from the rest, and we pick a running isolate of the requested Worker at random. Durable Objects are stateful, and they have a name. That provides us with the very helpful ability to grab a profile of a specific isolate and a specific object anywhere in the world. When profiling a Durable Object, you can choose by name which object to profile. The runtime will then route the profiling request to the right metal that owns the specific actor and provide a profile of the specific isolate running that object.

What’s next?

On-demand profiling is just the start. It enables you to explore your Workers performance in ways that have not been possible before. But it does have some limitations worth noting:

  • You must explicitly start the profiling session. This can mean that you miss important time periods in which your Worker is misbehaving, and where you would like to see a profile to understand what went wrong.
  • The memory profiler can only show you allocations that occurred during the profiling window. So if your Worker is allocating a lot during start-up, you will not see it.

In order to resolve this we are already working on something new: continuous profiling. The idea is that we will capture profiling samples for your Worker automatically, so that you can simply explore the profiles in the dashboard. This should make it easier to capture more rare events.

To learn more about profiling in Workers, including the feature presented in this blog post, check out our documentation.

Special thanks to the Workers Control Plane team and the Cloudflare Observability Platform team, in particular William Perron, who helped with dashboard implementation for this project.

Workers RPC now works across Python and JavaScript

Post Syndicated from Dominik Picheta original https://blog.cloudflare.com/python-workers-rpc/

Two years ago, we introduced Workers RPC, built on Cap’n Proto RPC. This made it possible for Workers to call other Workers and Durable Objects’ methods, return live objects and call their methods, return functions, streams and get all the benefits of a Remote Procedure Call (RPC) system, without defining schemas or adding any dependencies. We called it “JavaScript-native RPC” because it made using RPC feel native to the language.

Last year, we made this work between web browsers and servers, and introduced Cap’n Web.

Now we’re taking it cross-language.

Normally, getting programs written in different languages to talk to each other is complicated: developers usually have to build custom APIs or adopt language-agnostic serialization formats like protobuf, so the two systems can understand each other. The RPC system built into Workers is able to translate across JavaScript and Python without any additional work.

You can now call methods defined in a Python Worker from a JavaScript Worker and vice versa. You can share objects across Python and JavaScript, and call methods on a Python object from TypeScript. It all just works.

If you define a method add() in a Worker written in TypeScript:

…you can simply call it from Python:

There are no dependencies needed. All you need to configure is a Service binding:

So, what can you do with it?

This RPC system allows you to build a complex multi-language system as if you are using a library. Here are some features of cross-language RPC.

  • Cross-language RPC calls behave like ordinary function calls that return promises in JavaScript/TypeScript and futures in Python. Exceptions are propagated and are thrown at the call site of the RPC method.
  • You can pass any Structured Cloneable types as the parameters or a return value of an RPC call. These get converted to the appropriate types in Python: for example, a JS Date is converted to a Python datetime
  • You can pass JavaScript functions to a Python Worker and return them, and vice versa. When the other side calls the function passed to it, they make a new RPC back for you.
  • Typically, RPC to another Worker does not cross a network. The other Worker usually runs in the same thread as the caller. There is near-zero performance overhead compared to running code in the same Worker.
  • The implementation is fully open source as part of workerd and workers-runtime-sdk.

But wait, how do you convert types across languages? 

The main hurdle for making RPC seamless across the JavaScript and Python Workers is bridging their distinct type systems. JavaScript developers expect to work with native JavaScript types, and Python developers expect the same for Python. Bridging two distinct languages with their own type systems required a careful, deliberate type conversion strategy.

Consider how each language handles function arguments. A typical way to define a complex function in JavaScript is passing an Object as an argument:

In contrast, a Python developer would typically define the same function using keyword arguments:

Our goal was to make cross-language RPC completely transparent. Developers should feel like they are writing code for a single-language application without needing to worry about the underlying translation layer. We achieved this by combining Pyodide’s Foreign Function Interface (FFI) with a custom type-conversion layer for Python Workers.

Pyodide FFI already translates between Python and JavaScript types

Pyodide is the CPython interpreter compiled to WebAssembly, and it has powered Python Workers from the start. It includes a robust FFI that automatically translates types between JavaScript and Python.

When a Python Worker communicates with a JavaScript Worker via Service bindings, Pyodide’s FFI transparently converts objects during the RPC call. Developers on either side don’t need to know which language the other Worker is written in, and everything is handled under the hood.

Pyodide maps native types between both environments out of the box:

When direct translation isn’t possible (such as with custom classes or functions), Pyodide creates a Proxy object. This proxy forwards attribute accesses and method calls across the boundary, enabling patterns like passing a Python function directly as a callback to JavaScript handlers.

Pyodide FFI also maps Python’s keyword arguments directly to JavaScript’s object-style parameters. For example, imagine a JavaScript Worker with a method that takes an optional options object:

When calling this JavaScript Worker from Python, you could pass a Python dictionary to represent the JavaScript object:

However, you can also use native Python keyword arguments:

Pyodide FFI translates both calls into the exact structure the JavaScript Worker expects, giving Python developers a clean, natural API experience.

To explore type translation in more detail, check out the Pyodide documentation.

Handling Cloudflare Workers objects

While Pyodide FFI seamlessly converts standard built-in types, it doesn’t automatically understand Web API objects such as Request, Response, Blob, or File. They are commonly used in Cloudflare Workers, but there is no direct built-in equivalent in Python.

As explained in the previous section, Pyodide, by default, treats these non-standard objects as JavaScript Proxies. Rather than converting them into Python objects, it creates a passthrough proxy for attribute lookups and method calls. While functional, this approach leaks underlying JavaScript implementation details into Python. Python developers would have to constantly remember they are interacting with JavaScript proxies, adding unnecessary mental overhead.

To fix this, we introduced the workers-runtime-sdk Python package. This acts as a thin conversion layer built specifically to handle custom Workers types over RPC. When you deploy a Python Worker using uv run pywrangler deploy, this package is included by default. In fact, if you import from the workers namespace, you’re already using it:

Behind the scenes, this SDK wraps the RPC stubs provided by the bindings. It intercepts objects crossing the language boundary and translates them into native forms that both JavaScript and Python Workers can work with naturally.

As a result, Python developers can work with familiar, idiomatic Python objects, making cross-language execution feel completely invisible.

Use Python packages from your JavaScript Worker

Have you ever wanted to use a great Python package, but your app is written in JavaScript? You can do this with Python Workers. Let’s look at an example.

Pygments is a popular syntax highlighting package, written in Python. To use it from JavaScript, you just need to expose a method from a Python Worker that calls the Pygments package.

We can call this method in our JavaScript by accessing the request’s env:

Now on the Python side, we define a Python Worker with this method like so:

Now all that’s left is to write the necessary code to do the highlighting in Python. A simplified version of this looks like so:

The JavaScript lives in its own Worker that is separate from the Python Worker. So you also need to define the Service bindings to ensure they can communicate. You can do so by putting this in the JavaScript Worker’s wrangler.jsonc file:

The name of the service needs to match the name of your Python Worker here.

To test these, you can run npx wrangler dev in the JavaScript Worker’s directory and uv run pywrangler dev in the Python Worker’s directory in two separate terminals.

A full example is available on GitHub. You can run it directly by using the following commands:

Try it now

In addition to those above, there are far more examples and information about RPC in our documentation.

Python Workers redux: fast cold starts, packages, and a uv-first workflow

Post Syndicated from Dominik Picheta original https://blog.cloudflare.com/python-workers-advancements/

Last year we announced basic support for Python Workers, allowing Python developers to ship Python to region: Earth in a single command and take advantage of the Workers platform.

Since then, we’ve been hard at work making the Python experience on Workers feel great. We’ve focused on bringing package support to the platform, a reality that’s now here — with exceptionally fast cold starts and a Python-native developer experience.

This means a change in how packages are incorporated into a Python Worker. Instead of offering a limited set of built-in packages, we now support any package supported by Pyodide, the WebAssembly runtime powering Python Workers. This includes all pure Python packages, as well as many packages that rely on dynamic libraries. We also built tooling around uv to make package installation easy.

We’ve also implemented dedicated memory snapshots to reduce cold start times. These snapshots result in serious speed improvements over other serverless Python vendors. In cold start tests using common packages, Cloudflare Workers start over 2.4x faster than AWS Lambda and 3x faster than Google Cloud Run.

In this blog post, we’ll explain what makes Python Workers unique and share some of the technical details of how we’ve achieved the wins described above. But first, for those who may not be familiar with Workers or serverless platforms – and especially those coming from a Python background — let us share why you might want to use Workers at all.

Deploying Python globally in 2 minutes

Part of the magic of Workers is simple code and easy global deployments. Let’s start by showing how you can deploy a FastAPI app across the world with fast cold starts in less than two minutes.

A simple Worker using FastAPI can be implemented in a handful of lines:

from fastapi import FastAPI
from workers import WorkerEntrypoint
import asgi

app = FastAPI()

@app.get("/")
async def root():
   return {"message": "This is FastAPI on Workers"}

class Default(WorkerEntrypoint):
   async def fetch(self, request):
       return await asgi.fetch(app, request.js_object, self.env)

To deploy something similar, just make sure you have uv and npm installed, then run the following:

$ uv tool install workers-py
$ pywrangler init --template \
    https://github.com/cloudflare/python-workers-examples/03-fastapi
$ pywrangler deploy

With just a little code and a pywrangler deploy, you’ve now deployed your application across Cloudflare’s edge network that extends to 330 locations across 125 countries. No worrying about infrastructure or scaling.

And for many use cases, Python Workers are completely free. Our free tier offers 100,000 requests per day and 10ms CPU time per invocation. For more information, check out the pricing page in our documentation.

For more examples, check out the repo in GitHub. And read on to find out more about Python Workers.

So what can you do with Python Workers?

Now that you’ve got a Worker, just about anything is possible. You write the code, so you get to decide. Your Python Worker receives HTTP requests and can make requests to any server on the public Internet.

You can set up cron triggers, so your Worker runs on a regular schedule. Plus, if you have more complex requirements, you can make use of Workflows for Python Workers, or even long-running WebSocket servers and clients using Durable Objects.

Here are more examples of the sorts of things you can do using Python Workers:

Faster package cold starts than Lambda and Cloud Run

Serverless platforms like Workers save you money by only running your code when it’s necessary to do so. This means that if your Worker isn’t receiving requests, it may be shut down and will need to be restarted once a new request comes in. This typically incurs a resource overhead we refer to as the “cold start.” It’s important to keep these as short as possible to minimize latency for end users.

In standard Python, booting the runtime is expensive, and our initial implementation of Python Workers focused on making the runtime boot fast. However, we quickly realized that this wasn’t enough. Even if the Python runtime boots quickly, in real-world scenarios the initial startup usually includes loading modules from packages, and unfortunately, in Python many popular packages can take several seconds to load.

We set out to make cold starts fast, regardless of whether packages were loaded.

To measure realistic cold start performance, we set up a benchmark that imports common packages, as well as a benchmark running a “hello world” using a bare Python runtime. While Lambda is able to start just the runtime quickly, once you need to import packages, the cold start times shoot up.

Here are the average cold start times when loading three common packages (httpx, fastapi and pydantic):

Platform

Mean Cold Start (secs)

Cloudflare Python Workers

1.027

AWS Lambda

2.502

Google Cloud Run

3.069

In this case, Cloudflare Python Workers have 2.4x faster cold starts than AWS Lambda and 3x faster cold starts than Google Cloud Run. We achieved these low cold start numbers by using memory snapshots, and in a later section we explain how we did so.

We are regularly running these benchmarks. Go here for up-to-date data and more info on our testing methodology.

We’re architecturally different from these other platforms — namely, Workers is isolate-based. Because of that, our aims are high, and we are planning for a zero cold start future.

Package tooling integrated with uv

The diverse package ecosystem is a large part of what makes Python so amazing. That’s why we’ve been hard at work ensuring that using packages in Workers is as easy as possible.

We realised that working with the existing Python tooling is the best path towards a great development experience. So we picked the uv package and project manager, as it’s fast, mature, and gaining momentum in the Python ecosystem.

We built our own tooling around uv called pywrangler. This tool essentially performs the following actions:

  • Reads your Worker’s pyproject.toml file to determine the dependencies specified in it

  • Includes your dependencies in a python_modules folder that lives in your Worker

Pywrangler calls out to uv to install the dependencies in a way that is compatible with Python Workers, and calls out to wrangler when developing locally or deploying Workers. 

Effectively this means that you just need to run pywrangler dev and pywrangler deploy to test your Worker locally and deploy it. 

Type hints

You can generate type hints for all of the bindings defined in your wrangler config using pywrangler types. These type hints will work with Pylance or with recent versions of mypy.

To generate the types, we use wrangler types to create typescript type hints, then we use the typescript compiler to generate an abstract syntax tree for the types. Finally, we use the TypeScript hints — such as whether a JS object has an iterator field — to generate mypy type hints that work with the Pyodide foreign function interface.

Decreasing cold start duration using snapshots

Python startup is generally quite slow and importing a Python module can trigger a large amount of work. We avoid running Python startup during a cold start using memory snapshots.

When a Worker is deployed, we execute the Worker’s top-level scope and then take a memory snapshot and store it alongside your Worker. Whenever we are starting a new isolate for the Worker, we restore the memory snapshot and the Worker is ready to handle requests, with no need to execute any Python code in preparation. This improves cold start times considerably. For instance, starting a Worker that imports fastapi, httpx and pydantic without snapshots takes around 10 seconds. With snapshots, it takes 1 second.

The fact that Pyodide is built on WebAssembly enables this. We can easily capture the full linear memory of the runtime and restore it. 

Memory snapshots and Entropy

WebAssembly runtimes do not require features like address space layout randomization for security, so most of the difficulties with memory snapshots on a modern operating system do not arise. Just like with native memory snapshots, we still have to carefully handle entropy at startup to avoid using the XKCD random number generator (we’re very into actual randomness).


By snapshotting memory, we might inadvertently lock in a seed value for randomness. In this case, future calls for “random” numbers would consistently return the same sequence of values across many requests.

Avoiding this is particularly challenging because Python uses a lot of entropy at startup. These include the libc functions getentropy() and getrandom() and also reading from /dev/random and /dev/urandom. All of these functions share the same implementation in terms of the JavaScript crypto.getRandomValues() function.

In Cloudflare Workers, crypto.getRandomValues() has always been disabled at startup in order to allow us to switch to using memory snapshots in the future. Unfortunately, the Python interpreter cannot bootstrap without calling this function. And many packages also require entropy at startup time. There are essentially two purposes for this entropy:

  • Hash seeds for hash randomization

  • Seeds for pseudorandom number generators

Hash randomization we do at startup time and accept the cost that each specific Worker has a fixed hash seed. Python has no mechanism to allow replacing the hash seed after startup.

For pseudorandom number generators (PRNG), we take the following approach:

At deploy time:

  1. Seed the PRNG with a fixed “poison seed”, then record the PRNG state.

  2. Replace all APIs that call into the PRNG with an overlay that fails the deployment with a user error.

  3. Execute the top level scope of user code.

  4. Capture the snapshot.

At run time:

  1. Assert that the PRNG state is unchanged. If it changed, we forgot the overlay for some method. Fail the deployment with an internal error.

  2. After restoring the snapshot, reseed the random number generator before executing any handlers.

With this, we can ensure that PRNGs can be used while the Worker is running, but stop Workers from using them during initialization and pre-snapshot.

Memory snapshots and WebAssembly state

An additional difficulty arises when creating memory snapshots on WebAssembly: The memory snapshot we are saving consists only of the WebAssembly linear memory, but the full state of the Pyodide WebAssembly instance is not contained in the linear memory. 

There are two tables outside of this memory.

One table holds the values of function pointers. Traditional computers use a “Von Neumann” architecture, which means that code exists in the same memory space as data, so that calling a function pointer is a jump to some memory address. WebAssembly has a “Harvard architecture” where code lives in a separate address space. This is key to most of the security guarantees of WebAssembly and in particular why WebAssembly does not need address space layout randomization. A function pointer in WebAssembly is an index into the function pointer table.

A second table holds all JavaScript objects referenced from Python. JavaScript objects cannot be directly stored into memory because the JavaScript virtual machine forbids directly obtaining a pointer to a JavaScript object. Instead, they are stored into a table and represented in WebAssembly as an index into the table.

We need to ensure that both of these tables are in exactly the same state after we restore a snapshot as they were when we captured the snapshot.

The function pointer table is always in the same state when the WebAssembly instance is initialized and is updated by the dynamic loader when we load dynamic libraries — native Python packages like numpy. 

To handle dynamic loading:

  1. When taking the snapshot, we patch the loader to record the load order of dynamic libraries, the address in memory where the metadata for each library is allocated, and the function pointer table base address for relocations. 

  2. When restoring the snapshot, we reload the dynamic libraries in the same order, and we use a patched memory allocator to place the metadata in the same locations. We assert that the current size of the function pointer table matches the function pointer table base we recorded for the dynamic library.

All of this ensures that each function pointer has the same meaning after we’ve restored the snapshot as it had when we took the snapshot.

To handle the JavaScript references, we implemented a fairly limited system. If a JavaScript object is accessible from globalThis by a series of property accesses, we record those property accesses and replay them when restoring the snapshot. If any reference exists to a JavaScript object that is not accessible in this way, we fail deployment of the Worker. This is good enough to deal with all the existing Python packages with Pyodide support, which do top level imports like:

from js import fetch

Reducing cold start frequency using sharding

Another important characteristic of our performance strategy for Python Workers is sharding. There is a very detailed description of what went into its implementation here. In short, we now route requests to existing Worker instances, whereas before we might have chosen to start a new instance.

Sharding was actually enabled for Python Workers first and proved to be a great test bed for it. A cold start is far more expensive in Python than in JavaScript, so ensuring requests are routed to an already-running isolate is especially important.

Where do we go from here?

This is just the start. We have many plans to make Python Workers better:

  • More developer-friendly tooling

  • Even faster cold starts by utilising our isolate architecture

  • Support for more packages

  • Support for native TCP sockets, native WebSockets, and more bindings

To learn more about Python Workers, check out the documentation available here. To get help, be sure to join our Discord.

A Socket API that works across JavaScript runtimes — announcing a WinterCG spec and Node.js implementation of connect()

Post Syndicated from Dominik Picheta original http://blog.cloudflare.com/socket-api-works-javascript-runtimes-wintercg-polyfill-connect/

A Socket API that works across JavaScript runtimes — announcing a WinterCG spec and Node.js implementation of connect()

A Socket API that works across JavaScript runtimes — announcing a WinterCG spec and Node.js implementation of connect()

Earlier this year, we announced a new API for creating outbound TCP sockets — connect(). From day one, we’ve been working with the Web-interoperable Runtimes Community Group (WinterCG) community to chart a course toward making this API a standard, available across all runtimes and platforms — including Node.js.

Today, we’re sharing that we’ve reached a new milestone in the path to making this API available across runtimes — engineers from Cloudflare and Vercel have published a draft specification of the connect() sockets API for review by the community, along with a Node.js compatible implementation of the connect() API that developers can start using today.

This implementation helps both application developers and maintainers of libraries and frameworks:

  1. Maintainers of existing libraries that use the node:net and node:tls APIs can use it to more easily add support for runtimes where node:net and node:tls are not available.
  2. JavaScript frameworks can use it to make connect() available in local development, making it easier for application developers to target runtimes that provide connect().

Why create a new standard? Why connect()?

As we described when we first announced connect(), to-date there has not been a standard API across JavaScript runtimes for creating and working with TCP or UDP sockets. This makes it harder for maintainers of open-source libraries to ensure compatibility across runtimes, and ultimately creates friction for application developers who have to navigate which libraries work on which platforms.

While Node.js provides the node:net and node:tls APIs, these APIs were designed over 10 years ago in the very early days of the Node.js project and remain callback-based. As a result, they can be hard to work with, and expose configuration in ways that don’t fit serverless platforms or web browsers.

The connect() API fills this gap by incorporating the best parts of existing socket APIs and prior proposed standards, based on feedback from the JavaScript community — including contributors to Node.js. Libraries like pg (node-postgres on Github) are already using the connect() API.

The connect() specification

At time of writing, the draft specification of the Sockets API defines the following API:

dictionary SocketAddress {
  DOMString hostname;
  unsigned short port;
};

typedef (DOMString or SocketAddress) AnySocketAddress;

enum SecureTransportKind { "off", "on", "starttls" };

[Exposed=*]
dictionary SocketOptions {
  SecureTransportKind secureTransport = "off";
  boolean allowHalfOpen = false;
};

[Exposed=*]
interface Connect {
  Socket connect(AnySocketAddress address, optional SocketOptions opts);
};

interface Socket {
  readonly attribute ReadableStream readable;
  readonly attribute WritableStream writable;

  readonly attribute Promise<undefined> closed;
  Promise<undefined> close();

  Socket startTls();
};

The proposed API is Promise-based and reuses existing standards whenever possible. For example, ReadableStream and WritableStream are used for the read and write ends of the socket. This makes it easy to pipe data from a TCP socket to any other library or existing code that accepts a ReadableStream as input, or to write to a TCP socket via a WritableStream.

The entrypoint of the API is the connect() function, which takes a string containing both the hostname and port separated by a colon, or an object with discrete hostname and port fields. It returns a Socket object which represents a socket connection. An instance of this object exposes attributes and methods for working with the connection.

A connection can be established in plain-text or TLS mode, as well as a special “starttls” mode which allows the socket to be easily upgraded to TLS after some period of plain-text data transfer, by calling the startTls() method on the Socket object. No need to create a new socket or switch to using a separate set of APIs once the socket is upgraded to use TLS.

For example, to upgrade a socket using the startTLS pattern, you might do something like this:

import { connect } from "@arrowood.dev/socket"

const options = { secureTransport: "starttls" };
const socket = connect("address:port", options);
const secureSocket = socket.startTls();
// The socket is immediately writable
// Relies on web standard WritableStream
const writer = secureSocket.writable.getWriter();
const encoder = new TextEncoder();
const encoded = encoder.encode("hello");
await writer.write(encoded);

Equivalent code using the node:net and node:tls APIs:

import net from 'node:net'
import tls from 'node:tls'

const socket = new net.Socket(HOST, PORT);
socket.once('connect', () => {
  const options = { socket };
  const secureSocket = tls.connect(options, () => {
    // The socket can only be written to once the
    // connection is established.
    // Polymorphic API, uses Node.js streams
    secureSocket.write('hello');
  }
})

Use the Node.js implementation of connect() in your library

To make it easier for open-source library maintainers to adopt the connect() API, we’ve published an implementation of connect() in Node.js that allows you to publish your library such that it works across JavaScript runtimes, without having to maintain any runtime-specific code.

To get started, install it as a dependency:

npm install --save @arrowood.dev/socket

And import it in your library or application:

import { connect } from "@arrowood.dev/socket"

What’s next for connect()?

The wintercg/proposal-sockets-api is published as a draft, and the next step is to solicit and incorporate feedback. We’d love your feedback, particularly if you maintain an open-source library or make direct use of the node:net or node:tls APIs.

Once feedback has been incorporated, engineers from Cloudflare, Vercel and beyond will be continuing to work towards contributing an implementation of the API directly to Node.js as a built-in API.