Announcing `js2nix` - scale your Node.js project builds with Nix

Hi everyone, I am happy to announce that we (at Canva) made the js2nix project public! The js2nix project is yet another lang2nix solution. We have been using it for some of our systems for more than a year and now we make it public.

Key features

  • Make every npm package the first-class citizen Nix derivation.
  • Codify against a full dependency tree.
  • Resolve dependency cycles.
  • Handle Life-cycle scripts of an npm package.
  • Build npm package from local sources, ie. support workspaces.
  • Expose Nix primitive, don’t force code structure.

Where?
Please follow this link GitHub - canva-public/js2nix: Node.js modules installation using Nix.

Why it was created?
It’s a known issue when Node.js project builds struggle with making the node modules installation phase scalable. We run millions of installs of node modules per workday at our CI agents and identified that the main issue is the granularity and flakiness. That is, js2nix was created to address these issues.

Why not just use what is already on the market?
In short, we haven’t found a project that can resolve all the corner cases so we had to create our own. I have to say that I was highly inspired by this project GitHub - Profpatsch/yarn2nix: Build and deploy node packages with nix from yarn.lock files.. But at the moment of the initial research, it was struggling with resolving some critical issues, like, dependency cycles and life-scripts execution. So we created an in-house version that serves our needs well.

We would love to hear your feedback!

8 Likes

Just wondering if you have had a look at dream2nix, and if yes, for what reason did you decide against it.
According to the problems you list (granular builds, handling of cyclic deps), dream2nix covers them as well.
I’m interested, because I’d like to understand which things we need to improve.

1 Like

Hi @DavHau,
At the moment of creating js2nix, the dream2nix was at the beginning of its journey, AFAIK, so I haven’t considered it as a possible solution for as, maybe I just missed something. Also, it heavily used flakes and this fact blocks us from using it because we don’t use flakes, deliberately. TL;DR: flakes heavily depend on git performance which is awfully slow for monorepos with millions of LOCs, as we have.

Thanks for getting back to me!

Oleg

1 Like

OK, I understand.

BTW, you can import flakes like any other nix expression, without using the flakes feature via flake-compat

The git performance issue with flakes should be resolved at some point by Source tree abstraction by edolstra · Pull Request #6530 · NixOS/nix · GitHub

3 Likes

It would be nice to have a comparison to:

Hopefully at some point all the nodejs efforts could be consolidated. I’m not sure how the current implementations differ (js2nix and dream2nix, nix-npm-buildpackage, npmlock2nix).

There were some discussions here a while back:

I feel honored :blush:

1 Like

Thanks for the great documentation in js2nix/implementation_details.md at 161797c3f3f5b778a50ffa00d76bbeb026353e28 · canva-public/js2nix · GitHub

1 Like

How does this compare to the newly merged buildNpmPackage?

1 Like

There is additional context in Wire things up in a flake · Issue #4 · canva-public/js2nix · GitHub (seems like a trick needs to be employed to avoid the copying, even with flake-compat).

Hosted by Flying Circus.