Hi everyone, I am happy to announce that we (at Canva) made the js2nix project public! The js2nix project is yet another lang2nix solution. We have been using it for some of our systems for more than a year and now we make it public.
Key features
Make every npm package the first-class citizen Nix derivation.
Codify against a full dependency tree.
Resolve dependency cycles.
Handle Life-cycle scripts of an npm package.
Build npm package from local sources, ie. support workspaces.
Expose Nix primitive, don’t force code structure.
Why it was created?
It’s a known issue when Node.js project builds struggle with making the node modules installation phase scalable. We run millions of installs of node modules per workday at our CI agents and identified that the main issue is the granularity and flakiness. That is, js2nix was created to address these issues.
Why not just use what is already on the market?
In short, we haven’t found a project that can resolve all the corner cases so we had to create our own. I have to say that I was highly inspired by this project GitHub - Profpatsch/yarn2nix: Build and deploy node packages with nix from yarn.lock files.. But at the moment of the initial research, it was struggling with resolving some critical issues, like, dependency cycles and life-scripts execution. So we created an in-house version that serves our needs well.
Just wondering if you have had a look at dream2nix, and if yes, for what reason did you decide against it.
According to the problems you list (granular builds, handling of cyclic deps), dream2nix covers them as well.
I’m interested, because I’d like to understand which things we need to improve.
Hi @DavHau,
At the moment of creating js2nix, the dream2nix was at the beginning of its journey, AFAIK, so I haven’t considered it as a possible solution for as, maybe I just missed something. Also, it heavily used flakes and this fact blocks us from using it because we don’t use flakes, deliberately. TL;DR: flakes heavily depend on git performance which is awfully slow for monorepos with millions of LOCs, as we have.
Hopefully at some point all the nodejs efforts could be consolidated. I’m not sure how the current implementations differ (js2nix and dream2nix, nix-npm-buildpackage, npmlock2nix).
Hello, it seems that the newly merged buildNpmPackage function behaves similarly to buildRustPackage by constructing the entire dependency tree during the fetch operation as a fixed-output derivation, resulting in a non-granular approach. Any change to a single package that is a transitive dependency requires refetching and rebuilding the entire project, which is not optimal. Existing projects such as poetry2nix, dream2nix, crate2nix, yarn2nix, and of course js2nix, are addressing this issue by dividing the dependency closure into individual Nix packages, allowing them to be cached, incrementally installed, and individually overridden.
I guess there use case of functions like build*Package and the use case of the *2nix tooling are different:
build*Package are used in nixpkgs and other package sets to package source code from other sources. The package source code will only be changed on upgrade, which happen infrequently (order of tens of days), and it will change substantially when that happens. Building the package incrementally based on a previous build of the package is not an important feature.
*2nix are used in projects that use nix to build packages inline. The package source code will change constantly (order of seconds), and it will change very little between builds. Building the package incrementally based on a previous build of the package is an important feature.
