There are already some solutions out there for packaging Node.js projects (like node2nix for npm, or yarn2nix for Yarn v1), but I don’t believe there are any for Yarn v2 projects, so I decided to start one.
Compared to other solutions, this uses the plugin system in Yarn v2, which lends itself really well to this sort of thing. Especially because, once installed, it can keep your Nix expression up-to-date automatically as you make changes to dependencies, without having to run separate commands.
At the time of writing, it requires a Yarn build from master, and the readme tells you how to get that set up. Hopefully soon we’ll have a Yarn 2.1.2 or 2.2.0 release that’ll work out-of-the-box.
Despite this being an old thread, I’m interested whether Yarn v2 (and v3) provides a cleaner experience with Nix than npm / pnpm / Yarn v1. For instance, does Yarn v2 enforce “pure” builds? Ideally we would want to build every JavaScript package separately to produce its own Nix package.
Also, are you using NixOS, and if so, how did you install Yarn v2? There still isn’t a Yarn v2/v3 package on NixPkgs.
I’m on Mac, but the situation is similar. Yarn v2+ typically installs a local (compressed) copy of itself in the project. I believe you can use a global install, but I’ve not seen it used in practice. The global Yarn is still the Yarn v1, but if your working directory is a Yarn v2 project it simply delegates to that. (In yarn-plugin-nixify, I do some magic to skip the global Yarn v1 with a shell alias.)
The idea is that the package manager itself becomes a pinned dependency of the project. You can optionally go all the way for a ‘zero-install’ setup where you also include all dependency code in VCS, and a clean checkout may be enough to start working from right away. (No install step, hence ‘zero-install’.)
You mention building packages separately. There’s an extra difficulty here in that a package can have peer dependencies, which change the dependency tree depending on what the parent does. Normally, if your tree mentions the same package version twice, it can be shared, but peer dependencies interfere with this. Yarn v2 creates ‘virtual’ entries in the tree for this, and may do a build twice if necessary. (In yarn-plugin-nixify, you can opt-in to isolated builds, which breaks this rule and builds the dependency once, without any peer dependency context.)
So Yarn is accomplishing determinism in language package management in a different way than Nix. I’m not sure if goals align; Yarn has more a focus on package management for projects than OSes. I don’t yet really see how we’d use it in Nixpkgs, for example. Could be possible, but it’d require more thought and work, (but then again, projects like node2nix are no small effort either.)
I personally use yarn-plugin-nixify for building projects that are already flakes. I also use it in my nix-darwin config as part of a single build that installs all my CLI tools. I recently added a installNixBinariesForDependencies option for that, which means you can create an empty project with dependencies just to get a nix build that provides the binaries for those dependencies in $out/bin. Works well enough for me.
Thanks for the info. Indeed, also on Linux, Yarn v1 redirects to Yarn v2/v3 when it is installed in the project, and I managed to install and use Yarn v3.
Zero-install seems wasteful with disk space and I don’t see what’s so great about it.
Wasn’t familiar with peer dependencies (I’m definitely not a NodeJS expert). Read about it, and the way I understand it, it seems an impurity by design: Suppose a package A declares a peer dependency on B (e.g. “2.x”). Now the version of B that will be installed depends entirely on the context.
Nevertheless, I think we can make this pure (and compatible with Nix): When a package Z depends on A, Z should tell A what version of B to depend on (through a parameter). If packages Y and Z depend on the same version of A, but on a different version of B, that must lead to different versions of A as well, for instance, in the Nix store, we can have the following build outputs:
B-2.3.2
B-2.4.0
A-1.3.2-depending-on-B-2.3.2
A-1.3.2-depending-on-B-2.4.0
Y-depending-on-A-1.3.2-depending-on-B-2.3.2
Z-depending-on-A-1.3.2-depending-on-B-2.4.0
(Of course the actual build outputs will contains unique hashes, making it unnecessary to state their dependencies in natural language )
Since “A-1.3.2-depending-on-B-2.3.2” and “A-1.3.2-depending-on-B-2.4.0” contain mostly the same stuff, we can optimize this by moving duplicated stuff (=just the files) to a separate package:
A-1.3.2-src
B-2.3.2-src
B-2.4.0-src
A-1.3.2-depending-on-B-2.3.2 (tiny package containing only a symlink to A-1.3.2-src, and setting NODEPATH to include B-2.3.2-src)
A-1.3.2-depending-on-B-2.4.0 (tiny package containing only a symlink to A-1.3.2-src, and setting NODEPATH to include B-2.4.0-src)
Y-depending-on-A-1.3.2-depending-on-B-2.3.2
Z-depending-on-A-1.3.2-depending-on-B-2.4.0
Do you think this would be a good strategy? This could be similar to what Yarn v2/v3 accomplishes with “virtual packages” (don’t know Yarn well enough).
I think that’s pretty close to what Yarn does, yes. It can work. We can probably package Yarn v3 and use it like a global install. Yarn also provides APIs that can be used to walk the package tree and maybe other stuff we need.
I feel like ironing out details in discussions is difficult, though. There’s a lot of maybes, so perhaps it takes some prodding at different implementation ideas by someone motivated. There are probably multiple ways to accomplish it too, maybe also using APIs from npm or pnpm (no idea).
It specifically does not do build steps yet. That requires a bit more work, and this was already quite the effort. Despite it being only ~500 SLOC, there was a lot of fiddling to get it to work.
@zwiek yarn-plugin-nixify is for building a project by itself using Nix, while yarn-nixpkgs is for distribution packaging. The way I see it, a good comparison outside Nix would be Docker builds and Debian packaging. I find the two have incompatible goals, or at least combining them in a single tool feels like it’d just make things more complex.
But it’s also gut feeling. I find it difficult to describe actual differences. Maybe the biggest difference is simply the way you use both tools? Yarn-plugin-nixify is supposed to be very much part of the developer flow, while yarn-nixpkgs is supposed to automate creating expressions for Nixpkgs (or overlays).
