Post Syndicated from Dimitri Mitropoulos original https://blog.cloudflare.com/forge-open-source-generation-pipeline/

Today we’re introducing Forge, a fresh approach to generating SDKs, CLIs, docs, and libraries. Forge is an open source, pluggable generation pipeline that anyone can deploy and run for free.

Forge is early in its life, but already generates the output required for the cf CLI, and over the next few months will power Cloudflare’s API documentation, SDKs, and much more.

We built Forge because we needed it ourselves in order to treat agents as our customers. Now, we’re open sourcing it because we think everyone should be able to generate all the surfaces that agents need. It used to be that only developer products needed CLIs, API SDKs, MCP servers, all with great corresponding docs. Now these are table stakes for every product.

Our API outgrew our generators

Cloudflare’s API has over 3,500 operations, and the hundreds of services that power these APIs are written in many languages, including Rust, Go, TypeScript, and Python. As we embarked on building a CLI for the entire Cloudflare API, including our SDKs and API docs, we needed a code generation pipeline that could handle this scale. That pipeline needs to be flexible enough to work across languages and the ways each of our engineering teams operate.

We needed a way to reduce coordination overhead between teams. When a Cloudflare product team makes an API change, they need to be able to use a preview build of the Cloudflare-wide CLI, SDK, and docs site that will be generated, before merging that change and shipping to customers. We needed a way to ensure they didn’t inadvertently break the generation pipeline. And we needed a system that we could extend to generate more than just an SDK, from Cap‘n Web to MCP and beyond.

We’ve tried several hosted products that attempt to solve this, and relied on some in production. None of them solved this problem for us, and some have shut down entirely. One team would merge a change that inadvertently would break the generation pipeline, another team would discover this at release time, and we spent too much time swimming upstream through hosted tools we couldn’t control, coordinating changes between teams and vendors.

That’s how we started building Forge.

Forge seeks to fix all these problems: it runs in CI, on each team’s API repos, just like our AI code reviewer and test pipelines. It lints every change, and then generates preview builds of the CLI, docs, and SDKs with just your changes highlighted that you can install to test. It’s the same premise as Workers Previews: a full preview build for every change, but applied to SDK generation at scale, including when the API surface is distributed across hundreds of services and repositories. That’s what Forge seeks to deliver.

Forge transformers can generate anything, including Cap’n Web

Cloudflare has more reasons than most to want a generator that can go well beyond the normal language targets. Cap’n Web is Cloudflare’s RPC system that lets TypeScript call a remote API as if it were calling a local method:

Forge makes it possible to take an OpenAPI spec and generate Cap’n Web directly. This opens the door to generating bindings from Workers to other APIs. After all, bindings in the Workers runtime are implemented as Workers that expose RPC methods.

This isn’t specific to Cap’n Web: other popular tools you may already rely on need the same thing. If you use TanStack Query, you’d ideally want to be able to generate TanStack Query bindings for your application, built directly from your API itself. Always up to date, always validated against your real API. The same is true of generating Zod or Valibot schemas, MCP servers, or anything else that makes it easier to consume your API.

This is possible because Forge code generators are flexible. They’re built for flowing information from one output to another.

Forge transformers can be chained: generate outputs from other outputs

We’ve designed Forge to be pluggable, and support many input and output types. Forge provides CLI, SDK and docs generators, but there’s nothing stopping you from adding a transformer that generates a library-specific package or even a full dashboard or application. Forge supports OpenAPI as an input type today, but we’ve designed it to allow AsyncAPI, GraphQL, Cap’n Proto, Protobuf, or other input formats in the future.

This is about more than just compatibility: it lets you chain targets, using one target output to produce others. This is common in other generators where the CLI and Terraform targets are produced from the Go SDK. But what’s missing, and what Forge provides, is a way for the user to control this chaining system themselves.

We needed a solution for this ourselves, because our own cf CLI is written in TypeScript, which other SDK generators don't generally chain from for CLIs. But our own situation made us recognize the deeper problem: why should any SDK generator tool make this decision for you? Maybe you’re a Python shop, and you want the CLI to be in Python.

If you’re thinking “Well, but who cares if it’s in Python or not? The code is automatically generated,” it’s because CLIs are different. CLIs often introduce local-only behaviors that wouldn’t make sense in an SDK. Behaviors that you write by hand since they’re inherently not something backed by any API call. For example, the cf CLI has commands like cf dev and cf build that are added on top of the rest of the generated output. These commands need to call TypeScript APIs from other packages like Vite. 

Now let’s add docs to the mix. If you’re generating your CLI and your docs purely from your OpenAPI spec, how do you feed those handwritten commands back into your docs, so they can be documented alongside the rest?

We couldn’t find an existing tool that does this today, and yet this is exactly what we need for cf. So we’re building it into Forge.

Change your API without breaking users

Forge is also setting us up for better API versioning. Cloudflare’s v4 API has been the one major version of our API for 10 years. Since then, it appears like we haven’t launched any new major versions, but by SemVer definitions we’ve made quite a few changes worthy of a new major version. At the same time, several operations across our API feature internal ‘v2’ tags or ‘beta’ identifiers that have long outlived that part of the product’s lifecycle.

After so many years of our v4 API, we’re keenly aware that a big new v5 would leave a lot of our customers behind. That’s why, with Forge releasing artifacts along the way, we’re working on an API versioning approach that allows us to release new major API versions without breaking old clients or SDKs.

We’ll have more on our SDKs very soon, including TypeScript, Rust, Python, Go, PHP, and Terraform. Especially Terraform. We know that upgrading any Terraform provider comes with its own set of rigor, and we’re going to put extra special care into the Terraform transition.

Critical tools should be open to all

We believe that building tools for APIs is a core part of the Internet, and you should be able to do that without needing a SaaS product. You should own your SDKs, CLIs, and docs. And if you generate them, then you should be able to do whatever you want, wherever you want, for free.

That’s why we’re making Forge available open source under the permissive Apache 2.0 license. We want people to join in on this journey with us, and contribute.

Or not? Maybe you want to keep everything to yourself. Go for it! You can run Forge on your own for any purpose, with custom modifications, for free, in private.

Acknowledgements: This project was also made possible by the design and implementation efforts of Dan Carter, Steven Chong, Krishna Paritala, and Shelley Jones.