I’ve been venturing into the nix world of things bit by bit and am currently moving from a docker based development setup to a nix shell based one. For my current project I need Go and NodeJS and noticed that when I initialize my shell for the first time, those packages get built from source.
Building from source takes a bit more time than just downloading prebuilt binaries and this made me wonder… would it be a good idea to write overlays that replace said packages with ones that use the official tarballs that contain prebuilt binaries? (there seem to be no recent builds on hydra for aarch64-darwin, nor for x86_64-darwin)
I’ve been trying to do just that but I’m having some troubles and figured that I’d better ask the opinion of the community, along with some pointers. I’d rather not waste my time on something that’s not considered a best practice.
The first question is, where would you be getting these prebuilt binaries? If they aren’t prebuilt by nix (or another build system that manages builds reproducibly) you’re throwing out most of the beneifts.
Furthermore, replacing the software with random pre-built binaries from e.g. github, or even debian, would throw out the integration work upstream has done.
Both of these things might be less of a problem with nodejs, since most node packages actually don’t need any building (but at that point you get no speed benefits anyway). Go, with its statically compiled binaries, probably cares less about the second part.
Repackaging binaries is done, but almost only on proprietary sofrware where building isn’t an option. There are instructions here: Packaging/Binaries - NixOS Wiki
With all of that said, the idiomatic way of doing this is to set up a binary cache, with a dedicated build machine that builds the packages you care about.
Well, I was thinking about using those that can be downloaded from Download | Node.js and Downloads - The Go Programming Language. Both pages provide sha256 hashes so I figured that would guarantee the reproducible part. You’re right regarding the integration work from upstream though! I failed to take that one into consideration.
To give you an idea what I’m actually trying to achieve, this is the flake file I’ve currently set up:
Probably! The problem in your case is probably just that you clean your cache too frequently. Normally when you enter a shell, nix should reuse an already compiled binary - it should only need to do that once after all, unless you clean your nix store after you leave the shell, since nix will normally clean up things without garbage collection roots, which your shells don’t have.
There’s another project that does this, which has quite a few users, but I don’t recall it off the top of my head.
I thought you meant npm/go libraries As you see in the packaging wiki, this isn’t trivial, but yes in theory doable. Completely unnecessary if you can afford one compilation cycle and keep the result cached though.
You’d think so, but I’d be surprised if anyone could rebuild those binaries from source and get the exact same result. It’s hard to prove that they are made from the same source.
Pragmatically, this is often not a problem for individuals, but in the wider view of the software “supply chain” it matters. And because I’m more of an idealist I will always recommend using binaries you can reproduce from source.
True, I’ve been garbage collecting quite frequently lately, mainly because I’ve been experimenting a lot Now that everything has more or less settled it’s much less of an issue.
It looks like there’s no copy in the binary cache for my system (darwin aarch64)
Anyway, based on the feedback that I’ve received in this thread I’ve decided to keep things as is. If the occasional compile keeps bothering me I’ll look into the issue a bit more, looking for ways to cache my builds or something like that. Thanks for weighing in on this one!
I’ve just tried nix-shell -p nodejs and nix-shell -p nodejs-16_x and both were copied from cache (along with some other binaries, but that’s to be expected I guess). This means that my initial problem is solved. However, now I’m confused as to why nothing was copied from cache two weeks ago Is there a way to check if a binary is cached?