Post Syndicated from James M Snell original https://blog.cloudflare.com/more-npm-packages-on-cloudflare-workers-combining-polyfills-and-native-code
Today, we are excited to announce a preview of improved Node.js compatibility for Workers and Pages. Broader compatibility lets you use more NPM packages and take advantage of the JavaScript ecosystem when writing your Workers.
Our newest version of Node.js compatibility combines the best features of our previous efforts. Cloudflare Workers have supported Node.js in some form for quite a while. We first announced polyfill support in 2021, and later built-in support for parts of the Node.js API that has expanded over time.
The latest changes make it even better:
-
You can use far more NPM packages on Workers.
-
You can use packages that do not use the
node: prefix to import Node.js APIs -
You can use more Node.js APIs on Workers, including most methods on
async_hooks,buffer,dns,os, andevents. Many more, such asfsorprocessare importable with mocked methods.
To give it a try, add the following flag to wrangler.toml, and deploy your Worker with Wrangler:
compatibility_flags = ["nodejs_compat_v2"]
Packages that could not be imported with nodejs_compat, even as a dependency of another package, will now load. This includes popular packages such as body-parser, jsonwebtoken, pg, got, passport, md5, mongodb, knex, mailparser, csv-stringify, cookie-signature, stream-slice, and many more.
This behavior will soon become the default for all Workers with the existing nodejs_compat compatibility flag enabled, and a compatibility date of 2024-09-23 or later. As you experiment with improved Node.js compatibility, share your feedback by opening an issue on GitHub.
Workerd is not Node.js
To understand the latest changes, let’s start with a brief overview of how the Workers runtime differs from Node.js.
Node.js was built primarily for services run directly on a host OS and pioneered server-side JavaScript. Because of this, it includes functionality necessary to interact with the host machine, such as process or fs, and a variety of utility modules, such as crypto.
Cloudflare Workers run on an open source JavaScript/Wasm runtime called workerd. While both Node.js and workerd are built on V8, workerd is designed to run untrusted code in shared processes, exposes bindings for interoperability with other Cloudflare services, including JavaScript-native RPC, and uses web-standard APIs whenever possible.
Cloudflare helped establish WinterCG, the Web-interoperable Runtimes Community Group to improve interoperability of JavaScript runtimes, both with each other and with the web platform. You can build many applications using only web-standard APIs, but what about when you want to import dependencies from NPM that rely on Node.js APIs?
For example, if you attempt to import pg, a PostgreSQL driver, without Node.js compatibility turned on…
import pg from 'pg'You will see the following error when you run wrangler dev to build your Worker:
✘ [ERROR] Could not resolve "events"
../node_modules/.pnpm/[email protected]/node_modules/pg-cloudflare/dist/index.js:1:29:
1 │ import { EventEmitter } from 'events';
╵ ~~~~~~~~
The package "events" wasn't found on the file system but is built into node.This happens because the pg package imports the events module from Node.js, which is not provided by workerd by default.
How can we enable this?
Our first approach – build-time polyfills
Polyfills are code that add functionality to a runtime that does not natively support it. They are often added to provide modern JavaScript functionality to older browsers, but can be used for server-side runtimes as well.
In 2022, we added functionality to Wrangler that injected polyfill implementations of some Node.js APIs into your Worker if you set node_compat = true in your wrangler.toml. For instance, the following code would work with this flag, but not without:
import EventEmitter from 'events';
import { inherits } from 'util';These polyfills are essentially just additional JavaScript code added to your Worker by Wrangler when deploying the Worker. This behavior is enabled by @esbuild-plugins/node-globals-polyfill which in itself uses rollup-plugin-node-polyfills.
This allows you to import and use some NPM packages, such as pg. However, many modules cannot be polyfilled with fast enough code or cannot be polyfilled at all.
For instance, Buffer is a common Node.js API used to handle binary data. Polyfills exist for it, but JavaScript is often not optimized for the operations it performs under the hood, such as copy, concat, substring searches, or transcoding. While it is possible to implement in pure JavaScript, it could be far faster if the underlying runtime could use primitives from different languages. Similar limitations exist for other popular APIs such as Crypto, AsyncLocalStorage, and Stream.
Our second approach – native support for some Node.js APIs in the Workers runtime
In 2023, we started adding a subset of Node.js APIs directly to the Workers runtime. You can enable these APIs by adding the nodejs_compat compatibility flag to your Worker, but you cannot use polyfills with node_compat = true at the same time.
Also, when importing Node.js APIs, you must use the node: prefix:
import { Buffer } from 'node:buffer';Since these Node.js APIs are built directly into the Workers runtime, they can be written in C++, which allows them to be faster than JavaScript polyfills. APIs like AsyncLocalStorage, which cannot be polyfilled without safety or performance issues, can be provided natively.
Requiring the node: prefix made imports more explicit and aligns with modern Node.js conventions. Unfortunately, existing NPM packages may import modules without node:. For instance, revisiting the example above, if you import the popular package pg in a Worker with the nodejs_compat flag, you still see the following error:
✘ [ERROR] Could not resolve "events"
../node_modules/.pnpm/[email protected]/node_modules/pg-cloudflare/dist/index.js:1:29:
1 │ import { EventEmitter } from 'events';
╵ ~~~~~~~~
The package "events" wasn't found on the file system but is built into node.Many NPM packages still didn’t work in Workers, even if you enabled the nodejs_compat compatibility flag. You had to choose between a smaller set of performant APIs, exposed in a way that many NPM packages couldn’t access, or a larger set of incomplete and less performant APIs. And APIs like process that are exposed as globals in Node.js could still only be accessed by importing them as modules.
The new approach: a hybrid model
What if we could have the best of both worlds, and it just worked?
-
A subset of Node.js APIs implemented directly in the Workers Runtime
-
Polyfills for the majority of other Node.js APIs
-
No
node: prefix required -
One simple way to opt-in
Improved Node.js compatibility does just that.
Let’s take a look at two lines of code that look similar, but now act differently under the hood when nodejs_compat_v2 is enabled:
import { Buffer } from 'buffer'; // natively implemented
import { isIP } from 'net'; // polyfilledThe first line imports Buffer from a JavaScript module in workerd that is backed by C++ code. Various other Node.js modules are similarly implemented in a combination of Typescript and C++, including AsyncLocalStorage and Crypto. This allows for highly performant code that matches Node.js behavior.
Note that the node: prefix is not needed when importing buffer, but the code would also work with node:buffer.
The second line imports net which Wrangler automatically polyfills using a library called unenv. Polyfills and built-in runtime APIs now work together.
Previously, when you set node_compat = true, Wrangler added polyfills for every Node.js API that it was able to, even if neither your Worker nor its dependencies used that API. When you enable the nodejs_compat_v2 compatibility flag, Wrangler only adds polyfills for Node.js APIs that your Worker or its dependencies actually use. This results in small Worker sizes, even with polyfills.
For some Node.js APIs, there is not yet native support in the Workers runtime nor a polyfill implementation. In these cases, unenv “mocks” the interface. This means it adds the module and its methods to your Worker, but calling methods of the module will either do nothing or will throw an error with a message like:
[unenv] <method name> is not implemented yet!
This is more important than it might seem. Because if a Node.js API is “mocked”, NPM packages that depend on it can still be imported. Consider the following code:
// Package name: my-module
import fs from "fs";
export function foo(path) {
const data = fs.readFileSync(path, 'utf8');
return data;
}
export function bar() {
return "baz";
}
import { bar } from "my-module"
bar(); // returns "baz"
foo(); // throws readFileSync is not implemented yet!
Previously, even with the existing nodejs_compat compatibility flag enabled, attempting to import my-module would fail at build time, because the fs module could not be resolved. Now, the fs module can be resolved, methods that do not rely on an unimplemented Node.js API work, and methods that do throw a more specific error – a runtime error that a specific Node.js API method is not yet supported, rather than a build-time error that the module could not be resolved.
This is what enables some packages to transition from “doesn’t even load on Workers” to, “loads, but with some unsupported methods”.
Still missing an API from Node.js? Module aliasing to the rescue
Let’s say you need an NPM package to work on Workers that relies on a Node.js API that isn’t yet implemented in the Workers runtime or as a polyfill in unenv. You can use module aliasing to implement just enough of that API to make things work.
For example, let’s say the NPM package you need to work calls fs.readFile. You can alias the fs module by adding the following to your Worker’s wrangler.toml:
[alias]
"fs" = "./fs-polyfill"
Then, in the fs-polyfill.js file, you can define your own implementation of any methods of the fs module:
export function readFile() {
console.log("readFile was called");
// ...
}
Now, the following code, which previously threw the error message “[unenv] readFile is not implemented yet!”, runs without errors:
import { readFile } from 'fs';
export default {
async fetch(request, env, ctx) {
readFile();
return new Response('Hello World!');
},
};
You can also use module aliasing to provide an implementation of an NPM package that does not work on Workers, even if you only rely on that NPM package indirectly, as a dependency of one of your Worker’s dependencies.
For example, some NPM packages, such as cross-fetch, depend on node-fetch, a package that provided a polyfill of the fetch() API before it was built into Node.js. The node-fetch package isn’t needed in Workers, because the fetch() API is provided by the Workers runtime. And node-fetch doesn’t work on Workers, because it relies on currently unsupported Node.js APIs from the http and https modules.
You can alias all imports of node-fetch to instead point directly to the fetch() API that is built into the Workers runtime using the popular nolyfill package:
[alias]
"node-fetch" = "./fetch-nolyfill"
All your replacement module needs to do in this case is to re-export the fetch API that is built into the Workers runtime:
export default fetch;Contributing back to unenv
Cloudflare is actively contributing to unenv. We think unenv is solving the problem of cross-runtime compatibility the right way — it adds only the necessary polyfills to your application, based on what APIs you use and what runtime you target. The project supports a variety of runtimes beyond workerd and is already used by other popular projects including Nuxt and Nitro. We want to thank Pooya Parsa and the unenv maintainers and encourage others in the ecosystem to adopt or contribute.
The path forward
Currently, you can enable improved Node.js compatibility by setting the nodejs_compat_v2 flag in wrangler.toml. We plan to make the new behavior the default when using the nodejs_compat flag on September 23rd. This will require updating your compatibility_date.
We are excited about the changes coming to Node.js compatibility, and encourage you to try it today. See the documentation on how to opt-in for your Workers, and please send feedback and report bugs by opening an issue. Doing so will help us identify any gaps in support and ensure that as much of the Node.js ecosystem as possible runs on Workers.
Rana Dutt is a Principal Solutions Architect at Amazon Web Services. He has a background in architecting scalable software platforms for financial services, healthcare, and telecom companies, and is passionate about helping customers build on AWS.
Amar Surjit is a Senior Solutions Architect at Amazon Web Services (AWS), where he specializes in data analytics and streaming services. He advises AWS customers on architectural best practices, helping them design reliable, secure, efficient, and cost-effective real-time analytics data systems. Amar works closely with customers to create innovative cloud-based solutions that address their unique business challenges and accelerate their transformation journeys.
Diego Soares is a Principal Solutions Architect at AWS with over 20 years of experience in the IT industry. He has a background in infrastructure, security, and networking. Prior to joining AWS in 2021, Diego worked for Cisco, supporting financial services customers for over 15 years. He works with large financial institutions to help them achieve their business goals with AWS. Diego is passionate about how technology solves business challenges and provides beneficial outcomes by developing complex solution architectures.
