Explaining Nix to a NPM user

Hi,

I have tried to get warm with Nix a few times, but it never quite clicked for me.
My use case is the following: I’m a computer science master’s students and I want to do reproducible projects. For me, this means:

a) When we publish something and its code, the readers should be able to run this without having to figure out the necessary software and dependencies themselves. Ideally, they would need to install one “installation framework” (Nix?), and then run a single command, and this command would install all the necessary software and libraries (at exactly the same versions that we used while creating the code) into the project directory, so that the readers can easily verify the results and play with the code. There would be no complications occurring on what software and libraries they have installed globally on their computer. We can assume that they are using a Unix-based operating system.

b) My team members who I collaborate with on the code together should be able to use exactly the same software on their devices. If a) works, this should also work. Moreover, they should also be able to easily install software into the project themselves, which we share via Git.

So, my first question is: Is Nix really suited to this? My understanding is that while the focus of Nix is to have reproducible operating system states, Nix does aim at use cases like mine, is this correct? Or is Nix less suitable for this task than, for example, Docker?

The package managers for Node Javascript, NPM and Yarn do fulfil the above requirements, within the scope of Javascript development; and Virtualenv, Pipenv, and Poetry have equivalent commands to some extent, for Python development. While these tools are nice, it would be ideal to have some package manager that is (a) not restricted to managing libraries, but also able to install other software and repositories; and (b) is not restricted to one single programming language. My understanding is that Nix does this, but I may be wrong.

I will briefly describe my workflow with tools like NPM, and then my question will be: (How) can I imitate this workflow with Nix?

When working with NPM, the following commands are crucial for me:

  1. npm init: This creates a file with boilerplate. As a consequence, I never need to create or modify files myself, I can do everything conveniently on the command line.

  2. npm install mypackage: This command figures out the latest version of the package and adds an entry with the exact version to a lock file within the project directory. It also figures out the versions of all the indirect dependencies and adds them to this file. There is some way of differentiating between direct and indirect dependencies. The lock file can be shared via version control and is the basis for collaboration with the same software.

  3. npm install: This checks the lock file in the project directory, and installs the software; either to a subdirectory within the project directory that is excluded from version control, or to a global location. Where it is actually installed does not matter to me as a user.

  4. npx mypackage is an easy way of running the package (if it is executable). In the Python package managers, I would have the choice between starting a shell with a virtual environment (poetry shell), or using a command such as poetry run mypackage.

  5. npm-check-updates checks whether there are (potentially breaking) updates to my packages, and lets me choose whether I want to change to them. This command is less important than the other ones for me.

Is there a way to transfer this workflow to Nix?

When I tried Nix, I found a relatively convenient way to install packages at the level of an operating system user (with nix-shell, as far as I remember) – but the only way to do things at the level of a project was to create Nix files by hand, which I struggled with and which it would be very hard to convince team members to learn.

Looking forward to your ideas!

3 Likes

You can use it for this task. If you pin a channel in your nix file they will have the same versions as you tested with. That being said if the pinned version is to old they might need to build stuff if it got collected from hydra.

My understanding is that while the focus of Nix is to have reproducible operating system states

Nix is just a general purpose language specialized for the nixpkgs/nixos project. nixpkgs is just the package collection that can be used by NixOS or any other Unix like OS.

Or is Nix less suitable for this task than, for example, Docker?

nixpkgs has even functions for building Docker Images.

(How) can I imitate this workflow with Nix?

There are projects which can create nix files and lockfiles from npm, yarn and poetry files. If you re-use those the overhead work should be minimal and you can just put those files into a subdirectory.

When I tried Nix, I found a relatively convenient way to install packages at the level of an operating system user (with nix-shell , as far as I remember)

nix-shell just opens a bash with the projects defined in the shell.nix file in your PATH. To install programs you either use NixOS’s configuration.nix, home.nix for home-manager or nix-env.

An example shell.nix can be as easy as the following:

with (import <nixpkgs> {});
mkShell {
  buildInputs = [
    black
    gnumake
    (
      python3.withPackages (
        ps: with ps; [
          jinja2
          pandas
        ]
      )
    )
  ];
}
1 Like

I also struggle at replacing NPM with Nix and haven’t yet found way that works well for me to manage a NPM project entirely with Nix.

I generally compromise and use a shell.nix to select the version of packages I need for a project (ex: including node-12_x, and any other binaries I might need for various npm scripts (or that my code execs directly); then continue to use npm or yarn to install dependencies and manage the project.

That said, there are a few Nix tools on my radar that could help nudge me closer to 100% Nix for npm projects:
GitHub - svanderburg/node2nix: Generate Nix expressions to build NPM packages ( nodePackages.node2nix in nixpkgs )
mkYarnPackage ( mkYarnPackage in nixpkgs ), which often works out of the box to package an existing yarn-managed project.

In my ideal world I would use node2nix and Nix to replace NPM with a similar workflow:

  1. npm init would possibly stay to boilerplate package.json
  2. npm install could be replaced by node2nix. This would handle dependency resolution for everything in package.json, adding deps would be done by putting them in package.json and running node2nix
  3. npm install for building the package would become nix-build from the expressions node2nix generates
  4. npx mypackage could be replaced by nix-shell
  5. npm-check-updates would probably just be running node2nix again

I hope this gives you some ideas!
I’m looking forward to hearing other’s ideas on the subject, thanks for asking!

1 Like

That won’t work. Most tools just integrate the eco system into nix and don’t fully replace it. You will still need the other tools to manage some parts like lock file generation for go or rust and they are always used internally by the tool.

I figured in this hypothetical node2nix would be the lock file generator and the ‘node-packages.nix’ it creates is the lock-file. It’s not a fully nix managed workflow though.

I guess node2nix wouldn’t be much different to just using yarn for everything and mkYarnPackage to package the service or command for integration the rest of the nix ecosystem. :man_shrugging:

1 Like

nix believes that code added to the system should be done in a declarative way.

NPM believes that code should be dynamically downloaded from the internet in a imperative fashion , when needed.

‘one of these ways has a future, but the other one does not, Mr Anderson.’

nix tries to bring order and reproducibility to the chaotic nature of dependency and package management. NPM and other dependency management systems are in there nature chaotic and unreproducible… . You can equate it to is the shifting sands of the desert, aka ‘It works on my machine’, to a well designed architectural building, with detailed blueprints and structural plans. aka ‘my machine can be your machine’.

Nix solves some problems, but gives you some new ones (maybe nicer ones), but your constantly fighting against package management system of your language…, some of these ‘build systems’ are fundamentally broken beyond repair having been a build engineer for quite a long time.

However, stick with it, there is a lot of help around here… If nix cant be an answer to your problems , it can at least teach you the inherit insanity of package management solutions currently out there.

2 Likes

Don’t forget that nix is a package manager. It’s main goal is to capture variance, while deliver software. It’s not meant to replace ecosystem specific toolchains. For development, I would recommend just using the tooling provided by the language or ecosystem; you can use nix to introduced the toolchain (e.g. shell.nix), but it’s not likely to replace every usecase that the original toolchain provides.

7 Likes