Announcing `js2nix` - scale your Node.js project builds with Nix

Hi everyone, I am happy to announce that we (at Canva) made the js2nix project public! The js2nix project is yet another lang2nix solution. We have been using it for some of our systems for more than a year and now we make it public.

Key features

  • Make every npm package the first-class citizen Nix derivation.
  • Codify against a full dependency tree.
  • Resolve dependency cycles.
  • Handle Life-cycle scripts of an npm package.
  • Build npm package from local sources, ie. support workspaces.
  • Expose Nix primitive, don’t force code structure.

Where?
Please follow this link GitHub - canva-public/js2nix: Node.js modules installation using Nix.

Why it was created?
It’s a known issue when Node.js project builds struggle with making the node modules installation phase scalable. We run millions of installs of node modules per workday at our CI agents and identified that the main issue is the granularity and flakiness. That is, js2nix was created to address these issues.

Why not just use what is already on the market?
In short, we haven’t found a project that can resolve all the corner cases so we had to create our own. I have to say that I was highly inspired by this project GitHub - Profpatsch/yarn2nix: Build and deploy node packages with nix from yarn.lock files.. But at the moment of the initial research, it was struggling with resolving some critical issues, like, dependency cycles and life-scripts execution. So we created an in-house version that serves our needs well.

We would love to hear your feedback!

12 Likes

Just wondering if you have had a look at dream2nix, and if yes, for what reason did you decide against it.
According to the problems you list (granular builds, handling of cyclic deps), dream2nix covers them as well.
I’m interested, because I’d like to understand which things we need to improve.

4 Likes

Hi @DavHau,
At the moment of creating js2nix, the dream2nix was at the beginning of its journey, AFAIK, so I haven’t considered it as a possible solution for as, maybe I just missed something. Also, it heavily used flakes and this fact blocks us from using it because we don’t use flakes, deliberately. TL;DR: flakes heavily depend on git performance which is awfully slow for monorepos with millions of LOCs, as we have.

Thanks for getting back to me!

Oleg

2 Likes

OK, I understand.

BTW, you can import flakes like any other nix expression, without using the flakes feature via flake-compat

The git performance issue with flakes should be resolved at some point by Source tree abstraction by edolstra · Pull Request #6530 · NixOS/nix · GitHub

5 Likes

It would be nice to have a comparison to:

Hopefully at some point all the nodejs efforts could be consolidated. I’m not sure how the current implementations differ (js2nix and dream2nix, nix-npm-buildpackage, npmlock2nix).

There were some discussions here a while back:

1 Like

I feel honored :blush:

2 Likes

Thanks for the great documentation in https://github.com/canva-public/js2nix/blob/161797c3f3f5b778a50ffa00d76bbeb026353e28/docs/implementation_details.md

2 Likes

How does this compare to the newly merged buildNpmPackage?

3 Likes

There is additional context in Wire things up in a flake · Issue #4 · canva-public/js2nix · GitHub (seems like a trick needs to be employed to avoid the copying, even with flake-compat).

For those who’re wondering, here is a couple of similar examples of how you can use the tool:

1 Like

Hello, it seems that the newly merged buildNpmPackage function behaves similarly to buildRustPackage by constructing the entire dependency tree during the fetch operation as a fixed-output derivation, resulting in a non-granular approach. Any change to a single package that is a transitive dependency requires refetching and rebuilding the entire project, which is not optimal. Existing projects such as poetry2nix, dream2nix, crate2nix, yarn2nix, and of course js2nix, are addressing this issue by dividing the dependency closure into individual Nix packages, allowing them to be cached, incrementally installed, and individually overridden.

1 Like

I guess there use case of functions like build*Package and the use case of the *2nix tooling are different:

  1. build*Package are used in nixpkgs and other package sets to package source code from other sources. The package source code will only be changed on upgrade, which happen infrequently (order of tens of days), and it will change substantially when that happens. Building the package incrementally based on a previous build of the package is not an important feature.
  2. *2nix are used in projects that use nix to build packages inline. The package source code will change constantly (order of seconds), and it will change very little between builds. Building the package incrementally based on a previous build of the package is an important feature.
5 Likes