Node_modules.nix

Hi guys

I just created a proof-of-concept nix library called “node_modules.nix”. It also aims at packaging Node.js applications with Nix like node2nix, yarn2nix or pnpm2nix. Like pnpm2nix it’s just a nix-library and not a tool that needs to be run everytime you change your package.json file. But it works directly with NPM.

Check it out here: GitHub - VanCoding/node_modules.nix: A nix library to turn your package-lock.json into a fully populated node_modules directory

I’d be interested in some comments on the idea or suggestions on how to continue.

Thanks :slight_smile:

8 Likes

Looking good! I like the simplicity of it.

The base64 conversion could be removed if you can assume a recent-enough version of nix. In that case, it’s possible to use fetchurl { url = ...; hash = integrity; } where integrity contains the value of the SRI-hash.

Have you tried the tool with node-sass yet? Some packages ship with pre-built binaries that need to be patched during the installation. Some other tools want to download them on the Internet during the installation. These are generally the cases that start complexifying the library. I would recommend adding autoPatchelfHook to the build as it does a good job at detecting and reporting what libraries are missing.

1 Like

Thanks for this useful feedback :slight_smile:

Yes, I’m currently experimenting with native modules. My current goal is to bring a gatsby project in our company to work, that uses a lot of native modules for image processing and stuff. That already forced me a tarball-patching feature to library. Wit a few hacks, it’s currently somewhat possible to build the project, but there are still a lot of problems. One of them is that some packages write into the node_modules folder, which is read-only in this case.

Out of curiosity I tried bringing node-sass to work, and it surprisingly worked pretty well. I added an example for it to the repo.

I’ll add an example where a tarball gets patches soon as well.

A common approach is to maintain a list of overrides that map to package names. For example poetry2nix does that. The gem-config for ruby packages in nixpkgs, … That way other users can benefit from the patches and also add their own overrides that they found. Typically that’s used to inject the C dependencies as package managers tend to not handle those.

Nice to see this! I’ve worked on something similar for some time:
GitHub - andir/npmlock2nix based on this blog post I wrote a while ago: ranz2nix - A new approach to packaging node packages with Nix · Andreas Rammhold

I would also like to know how you deal with binaries and NPM packages that try to access the internet during the install stage. I’ve been trying to use npm ci as well but those wouldn’t execute any of the NPM hooks. I used those hooks to patchup files (shebangs,symlinks, …) while npm install is running as some package might already depend on the scripts/binaries before I get a chance to do.

@zimbatm
This is a good idea! This way, a community could grow around it to add support for packages.

@andir WOW! This takes almost exactly the same approach as my library but already looks way more polished! And there seems to be even someone from Tweag hacking on it. Although it’s quite fun working on node_modules.nix, it’ll probably lead to me contributing to npmlock2nix in the end :slight_smile:

Today I’ve added a sharp example where I needed to patch a tarball. I think this one takes a bit of a different approach to npmlock2nix, in that it takes the base tarball and modifies it using a provided script. I could be wrong though :sweat_smile:

That thing with the pre-install hook is also very clever. Are there other differences between npm ci and npm install --offline? I didn’t have the case yet where a shebang had to be patched during installation, though.

That would be neat! Not sure who is continuing the work there but I am keen to work together on improving it! I mostly started adding patches to my fork since I no longer have permissions for the repo in the Tweag Org. One of the experimental features I’ve been working on in Yarn v1 support. That is already working for very simple projects but on some inputs my parser is still tripping over some of the custom format specifics…

Yeah. I tried to avoid adding patching support to npmlock2nix before actually running into a use case for it. I had it in my mind while working on it but never had the need to use it.

Outside of a Nix sanbox npm ci is supposedly better because it should not do network I/O ever. As we have the nix sandbox we do not have to deal with that.

This sounds pretty similar to napalm and nix-npm-buildpackage. I think node ecosystem might be the one with most tools now.

Here is a list of the ones I know of:

Is my assumption correct? It would be nice if the efforts could be concentrated :slight_smile:

1 Like

@terlar Nice list. Here some comments on these:

node2nix
I already mentioned the drawbacks with this one.

yarn2nix
I think it has the same problems as node2nix, but for yarn.

napalm
It looks like it starts a local npm registry and installs the packages from there. I didn’t even know that it’s possible to access the local network in the nix sandbox. But since npm supports the cache directory, I think this is unneccessary.

nixfromnpm
This tries to solve the problem, that packages are not shared between apps, which is cool. But at the same time, it has the same problem as node2nix and yarn2nix: Before a package can be built, nixfromnpm to generate nix or something.

nix-npm-buildpackage
Of the mentioned packages, this is the only one that has the same concept as npmlock2nix and node_modules.nix. And it looks pretty promising! It also does some clever things to solve problems already mentioned. For example, when running npm ci, it uses the --script-shell parameter to use a shell, where shebangs are patched before execution. It also has support for yarn, which is nice! One feature it seems to miss, though, is creating a nix-shell containing the node_modules only.

As I see it, nix-npm-buildpackage, npmlock2nix and node_modules.nix could be combined into one package, and all 3 packages would benefit from it. Given there is interest, of course.

One additional package that has my special interest is pnpm2nix. It uses the brilliant pnpm package manager, which deduplicates packages (like nixfrompnpm). But pnpm currently lacks package hashes, so it’s not there yet.

2 Likes

Another viable strategy for handling difficult-to-build packages might be to just re-use the node packages from nixpkgs. Even if the lock file wants another package version, you can still take the nixpkgs derivation, override the src and version attributes and so inherit all the patches and fixes made there.

That way, you avoid building up you own ecosystem of overrides in which you basically just duplicate the work already done in nixpkgs.
Your own collection of overrides will probably never reach the quality of nixpkgs.