Nix vs Language Package manager

I’m new to Nix. I’m confused about the interaction between Nix and package managers for programming languages like npm, cabal, pip, etc. My current work flow is to use ‘apt’ to install the base language system, then use the language’s package manager to install libraries on demand. Is this the same for Nix?

The alternative is to use Nix to also install the libraries. However, I don’t see how this plays nicely with the package manager. For example, if I clone a python repo, I’d like to use pip to install a requirements.txt or run the setup.py file. Neither will interact with Nix.

Thanks.

1 Like

For python specifically there are approaches like poetry2nix and mach-python (correction: mach-nix) that are capable of using poetry.lock/pyproject.toml and requirements.txt respectively.

When using nix, you need to get rid of the idea to mix different tools for package management. Nix doesn’t play together with them nicely. Nix is all about reproducibility. If you would start mixing it with other tools, it breaks the concept and therefore the benefits of nix.
If you think about it from another perspective, it is a disaster that people are used to use 10 different package management tools on their system. The reason you needed them is because every single one of them lacks important features that makes them incapable of handling the “whole” thing. Nix is the one tool to replace them all.
The reason you needed pip is because apt isn’t flexible enough to include all that python sources and be able to handle their build system. Nix is.
The reason you needed venv is because neither apt nor pip are capable of installing different versions of the same package alongside on your system and aren’t capable of creating a temporary environment out of a selection of packages. Nix is.

The downside with this one-and-only approach is, that packages which have been created for these other non-nix package universes, first need to be translated to nix. This is usually done by either writing a nix expression manually for that package or set of packages, or you use a tool that does this translation for you.

For most of the common languages such tools exist. In the case of python, take a look at:

  • poetry2nix
  • mach-nix (not mach-python btw. :wink: . I released this recently. It doesn’t support all python libraries yet, but hopefully soon)
  • pypi2nix (Doesn’t automatically resolve external dependencies, which can make the process painful sometimes, depending on your requirements.)
5 Likes

I feel like the answer here depends a lot on the context.

One context is “using software”. For example, you want to use a tool, written in Python, and the tool has import request inside and somehow assumes that the request Python package is available. In this cases, the software should packaged the nix-way, using a nix expression, and probably upstreamed to GitHub - NixOS/nixpkgs: Nix Packages collection & NixOS as well. You do want system-wide reproducibility and isolation for system-wide applications. Admittedly, the process of wrapping software which is not already in nixpkgs is annoying (as nixpkgs really wants to be a monorepo of all the things), but hopefully things will become easier in the flakes world.

Another context is “developing software”. If you are a Python developer, you might be working on three different projects, and each project has its own set of dependencies, etc. Here, I am pretty sure that language-specific build tool/package manager is the way to go, if the language has good tooling. I work with rust primarily, and rustup and cargo are perfect for me to hack on my or other people’s projects. I don’t see how nix can add anything here. There’s also a fundamental tension here that, when you develop software, you need to handle both variability and reproducibility. For Rust, in Cargo.toml I specify effectively a range of versions of dependencies my project works with, but then I have a separate Cargo.lock which pins dependencies exactly. In this context, it’s important to make sure not only that the single point, Cargo.lock, is valid, but that the whole range works (using testing with deliberately older versions of libraries). The way I see it, nix is a very good tool to handle Cargo.lock aspect of this workflow, but doesn’t have vocabulary for Cargo.toml aspect. That said, when I need to use software written in rust (ripgrep or exa), I install it using nix, as that falls into the first context.

The specific case of Python is different though – I haven’t been active in Python community for a while, but my understanding is that building & dependencies story doesn’t have a clear cut definitive answer. Virtual environments were added not from the start, and undergone several iterations, pip didn’t have (and still hasn’t?) lockfiles, etc. In this situation, it might be plausible that using nix as a substitute for the language’s native infra would be beneficial. Though, I’d try to stick to default tooling nonetheless.

Finally, the third context is when you are developing software in a specific language, but it also depends on some system libraries. This is most typically manifested by having apt-get install libfoo libXbar in project’s readme. For such cases, using nix to manage the extra dependencies is the right answer, because nix allows you to have such global deps isolated per project. When I need to work with such project, I either use direnv and a project-specific shell.nix, or (if this is one off thing) I use my default shell.nix which is just a grab bag of common system dependencies.

3 Likes

One nice benefit is of buildRustCrate is that compiled crates are shared through the Nix store, reducing compilation times across projects. Add a binary cache and you have the same benefits across machines. Some of the dependencies that my projects have, have long compile times, so it can really pay off.

I have started using crate2nix’s import-from-derivation (IFD) support in some projects to build them with Nix, with the dependencies locked in Cargo.lock.

That said, I completely agree that building most Rust project using Cargo is not a problem. Typically a shell.nix file with non-Rust library dependencies is enough.

1 Like

@DavHau My mistake! I shouldn’t be late night discoursing. I’m really excited to see multiple emerging tools for the python ecosystem and the OP should check out the discussion about mach-nix here Mach-nix: Create python environments quick and easy - #44 by DavHau

This explanation helps a lot. To summarize, I would use Nix to install system specific libraries and to package my software for distribution. However, I would still use the language-specific tools for development. When development is complete, I can use the language->nix conversion tools to build the Nix configs for distribution.

There is a gap here that Nix could help with. When doing development, I will often use a standard suite of libraries. If I can package that up in Nix, other developers can quickly use the same suite. It would need to play nice with language tools however.

Thanks!

1 Like