Good idea to override packages with prebuilt binaries?

Help
1

Hi there!

I’ve been venturing into the nix world of things bit by bit and am currently moving from a docker based development setup to a nix shell based one. For my current project I need Go and NodeJS and noticed that when I initialize my shell for the first time, those packages get built from source.

Building from source takes a bit more time than just downloading prebuilt binaries and this made me wonder… would it be a good idea to write overlays that replace said packages with ones that use the official tarballs that contain prebuilt binaries? (there seem to be no recent builds on hydra for aarch64-darwin, nor for x86_64-darwin)

I’ve been trying to do just that but I’m having some troubles and figured that I’d better ask the opinion of the community, along with some pointers. I’d rather not waste my time on something that’s not considered a best practice.

Cheers,
P

1 Like
2

The first question is, where would you be getting these prebuilt binaries? If they aren’t prebuilt by nix (or another build system that manages builds reproducibly) you’re throwing out most of the beneifts.

Furthermore, replacing the software with random pre-built binaries from e.g. github, or even debian, would throw out the integration work upstream has done.

Both of these things might be less of a problem with nodejs, since most node packages actually don’t need any building (but at that point you get no speed benefits anyway). Go, with its statically compiled binaries, probably cares less about the second part.

Repackaging binaries is done, but almost only on proprietary sofrware where building isn’t an option. There are instructions here: Packaging/Binaries - NixOS Wiki

With all of that said, the idiomatic way of doing this is to set up a binary cache, with a dedicated build machine that builds the packages you care about.

Depending on your actual needs you could consider donating a build machine to do this for nixpkgs. You can also do this in p2p fashion if you need something simple for in-home use: GitHub - cid-chan/peerix: Peer2Peer Nix-Binary-Cache

This is a bit harder to judge without knowing what exactly you’re doing :wink:

2 Likes
3

Well, I was thinking about using those that can be downloaded from Download | Node.js and Downloads - The Go Programming Language. Both pages provide sha256 hashes so I figured that would guarantee the reproducible part. You’re right regarding the integration work from upstream though! I failed to take that one into consideration.

To give you an idea what I’m actually trying to achieve, this is the flake file I’ve currently set up:

{
  description = "Dev shell";

  inputs = {
      nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
      flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};

        postgres_setup = ''
          export PGHOST=$PWD/.nix/postgresql
          export PGDATA=$PWD/.nix/postgresql/data
          export PGDATABASE=cheddar
          mkdir -p $PGHOST
        '';

        nodejs_setup = ''
          export NODE_PATH=$PWD/.nix/nodejs
          export NPM_CONFIG_PREFIX=$PWD/.nix/nodejs
          export PATH=$NODE_PATH/bin:$PATH
          mkdir -p $NODE_PATH
        '';

        go_setup = ''
          export GOPATH=$PWD/.nix/go
          mkdir -p $GOPATH
        '';

        devup = (pkgs.writeScriptBin "devup" (builtins.readFile ./.nix/scripts/devup.sh)).overrideAttrs(old: {
          buildCommand = "${old.buildCommand}\n patchShebangs $out";
        });

        devdown = (pkgs.writeScriptBin "devdown" (builtins.readFile ./.nix/scripts/devdown.sh)).overrideAttrs(old: {
          buildCommand = "${old.buildCommand}\n patchShebangs $out";
        });
      in
      {
        formatter = pkgs.nixpkgs-fmt;

        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
              go_1_19
              nodejs-16_x
              postgresql_14

              devup
              devdown
          ];

          shellHook = ''
            ${nodejs_setup}
            ${go_setup}
            ${postgres_setup}
          '';
        };
      }
    );
}

This gives me an environment in which I can hack on my little project, with the only downside that initial setup takes 20 to 25 minutes.

Going from your reply, I’m going to do some research into setting up a binary cache although in my case, I feel like that might be a bit overkill :sweat_smile:

4

Probably! The problem in your case is probably just that you clean your cache too frequently. Normally when you enter a shell, nix should reuse an already compiled binary - it should only need to do that once after all, unless you clean your nix store after you leave the shell, since nix will normally clean up things without garbage collection roots, which your shells don’t have.

You can prevent that by creating garbage collection roots for the packages in your shell. Direnv with nix-direnv makes those automatically, for example: Development environment with nix-shell - NixOS Wiki

There’s another project that does this, which has quite a few users, but I don’t recall it off the top of my head.

I thought you meant npm/go libraries :wink: As you see in the packaging wiki, this isn’t trivial, but yes in theory doable. Completely unnecessary if you can afford one compilation cycle and keep the result cached though.

You’d think so, but I’d be surprised if anyone could rebuild those binaries from source and get the exact same result. It’s hard to prove that they are made from the same source.

Pragmatically, this is often not a problem for individuals, but in the wider view of the software “supply chain” it matters. And because I’m more of an idealist I will always recommend using binaries you can reproduce from source.

1 Like
5

Why are those packages being built from source anyways? They should be in the binary cache; ready for download.

2 Likes
6

True, I’ve been garbage collecting quite frequently lately, mainly because I’ve been experimenting a lot :grinning_face_with_smiling_eyes: Now that everything has more or less settled it’s much less of an issue.

It looks like there’s no copy in the binary cache for my system (darwin aarch64) :man_shrugging:

Anyway, based on the feedback that I’ve received in this thread I’ve decided to keep things as is. If the occasional compile keeps bothering me I’ll look into the issue a bit more, looking for ways to cache my builds or something like that. Thanks for weighing in on this one!

7

I just ran nix-shell -p go nodejs on my aarch64-darwin machine and everything was cached.

Could you run nix-shell -p nix-info --run "nix-info -m"?

Do you use any overlays, overrides or the like?

2 Likes
8

Running nix-info -m returns the following output:

 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.6.0, macOS 12.6`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.11.0`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

I’ve just tried nix-shell -p nodejs and nix-shell -p nodejs-16_x and both were copied from cache (along with some other binaries, but that’s to be expected I guess). This means that my initial problem is solved. However, now I’m confused as to why nothing was copied from cache two weeks ago :stuck_out_tongue: Is there a way to check if a binary is cached?

9

You can query cache.nixos.org for a given output path hash:

curl https://cache.nixos.org/nx12g0hvhajddcz93zzh4m4lxf1pzsri.narinfo

It will return 404 when the output path is not cached.

Another way is to check the build status on hydra:

https://hydra.nixos.org/job/nixpkgs/nixpkgs-unstable-aarch64-darwin/go.aarch64-darwin

1 Like