Comparing system derivations yields 17k lines of `The builders do not match` logs

plan9better · March 21, 2026, 4:14pm

I have a remote x86_64-linux server I rebuild remotely from my mac using the following command:

nixos-rebuild build --flake ".#remote" --target-host "operator@remote" --build-host "operator@remote" --use-remote-sudo --fast

I then compare the derivations using nix-diff to check exactly what changed before applying / switching to the new config. Today while looking at the nix-diff output it was 17k lines of these types of logs:

• The input derivation named `iptables-1.8.11` differs
          - /nix/store/ssfr95ll5ivkbckm4zw6hhbw57pbr1ci-iptables-1.8.11.drv:{out}
          + /nix/store/0sbi7kjkinka3xavsccz4ncmgq0nlnp6-iptables-1.8.11.drv:{out}
          • The builders do not match
              - /nix/store/cfqbabpc7xwg8akbcchqbq3cai6qq2vs-bash-5.2p37/bin/bash
              + /nix/store/l9k32vj2aczxw62134j1x0dsh569jz2l-bash-5.2p37/bin/bash
          • The input derivation named `autoreconf-hook` differs
            • These two derivations have already been compared
          • The input derivation named `bash-5.2p37` differs
            • These two derivations have already been compared
          • The input derivation named `bison-3.8.2` differs
            • These two derivations have already been compared
          • The input derivation named `flex-2.6.4` differs
            • These two derivations have already been compared
          • The input derivation named `iptables-1.8.11.tar.xz` differs
            • These two derivations have already been compared
          • The input derivation named `libmnl-1.0.5` differs
            • These two derivations have already been compared
          • The input derivation named `libnetfilter_conntrack-1.1.0` differs
            • These two derivations have already been compared
          • The input derivation named `libnfnetlink-1.0.2` differs
            • These two derivations have already been compared
          • The input derivation named `libnftnl-1.2.9` differs
            • These two derivations have already been compared
          • The input derivation named `libpcap-1.10.5` differs
            • These two derivations have already been compared
          • The input derivation named `pkg-config-wrapper-0.29.2` differs
            • These two derivations have already been compared
          • The input derivation named `prune-libtool-files` differs
            • These two derivations have already been compared
          • The input derivation named `stdenv-linux` differs
            • These two derivations have already been compared
          • Skipping environment comparison
        • The input derivation named `make-shell-wrapper-hook` differs
          • These two derivations have already been compared
        • The input derivation named `procps-4.0.4` differs
          - /nix/store/plawfkpihcgy5z190qfygrl0kxr5mq06-procps-4.0.4.drv:{out}
          + /nix/store/pqk8sr6zsf3f3w0dgkdjrfamvzxn10r1-procps-4.0.4.drv:{out}
          • The builders do not match
              - /nix/store/cfqbabpc7xwg8akbcchqbq3cai6qq2vs-bash-5.2p37/bin/bash
              + /nix/store/l9k32vj2aczxw62134j1x0dsh569jz2l-bash-5.2p37/bin/bash
          • The set of input derivation names do not match:
              - systemd-257.9
              + systemd-257.10
          • The input derivation named `autoreconf-hook` differs
            • These two derivations have already been compared
          • The input derivation named `bash-5.2p37` differs
            • These two derivations have already been compared
          • The input derivation named `ncurses-6.5` differs
            • These two derivations have already been compared
          • The input derivation named `pkg-config-wrapper-0.29.2` differs
            • These two derivations have already been compared
          • The input derivation named `procps-ng-4.0.4.tar.xz` differs
            - /nix/store/k3r1rrkslic94z789g9hayh4w4lhclay-procps-ng-4.0.4.tar.xz.drv:{out}
            + /nix/store/rir2aisy6idhcpxhcjkrs0z0cfc071pn-procps-ng-4.0.4.tar.xz.drv:{out}
            • The builders do not match
                - /nix/store/cfqbabpc7xwg8akbcchqbq3cai6qq2vs-bash-5.2p37/bin/bash
                + /nix/store/l9k32vj2aczxw62134j1x0dsh569jz2l-bash-5.2p37/bin/bash
            • The input derivation named `bash-5.2p37` differs
              • These two derivations have already been compared
            • The input derivation named `curl-8.14.1` differs
              • These two derivations have already been compared
            • The input derivation named `mirrors-list` differs
              • These two derivations have already been compared
            • The input derivation named `stdenv-linux` differs
              • These two derivations have already been compared
            • Skipping environment comparison
          • The input derivation named `stdenv-linux` differs
            • These two derivations have already been compared
          • Skipping environment comparison
        • The input derivation named `shadow-4.17.4` differs
          - /nix/store/84ddwq3arian73j3qm8gf6skdwwhinxg-shadow-4.17.4.drv:{out}
          + /nix/store/x2gmzz72nqlq58pxgmhqvh80jvqyv2pi-shadow-4.17.4.drv:{out}
          • The builders do not match
              - /nix/store/cfqbabpc7xwg8akbcchqbq3cai6qq2vs-bash-5.2p37/bin/bash
              + /nix/store/l9k32vj2aczxw62134j1x0dsh569jz2l-bash-5.2p37/bin/bash

Nixpkgs version did not change, looking at the new derivation: /nix/store/x00f3a3nw2bm0f9nx6chyy9wdwikai87-nixos-system-remote-24.11.20241012.54cce65.drv

the end, 24.11.20241012.54cce65 matches both the flake.lock and output of the following:

# cat /run/current-system/nixos-version
24.11.20241012.54cce65

Now my question is how do i go about finding out what caused the mismatch? Can i do anything to see just the ‘meaningful’ changes? Is it related to some change on the local mac machine where i evaluate the config before building on the remote? Any info would be much appreciated.

magicquark · March 21, 2026, 9:50pm

I doubt that the nixpkgs is the same if this is the case. Unless you are doing something different to the norm in your configuration.

magicquark · March 21, 2026, 10:01pm

Your NixOS version says 24.11, but 24.11 has systemd version 256.10.

github.com/NixOS/nixpkgs

pkgs/os-specific/linux/systemd/default.nix

50ab79378


      
          assert withCoredump -> withCompression;
          assert withHomed -> withCryptsetup;
          assert withHomed -> withPam;
          assert withUkify -> (withEfi && withBootloader);
          assert withRepart -> withCryptsetup;
          assert withBootloader -> withEfi;
          
          let
            wantCurl = withRemote || withImportd;
            wantGcrypt = withResolved || withImportd;
            version = "256.10";
          
            # Use the command below to update `releaseTimestamp` on every (major) version
            # change. More details in the commentary at mesonFlags.
            # command:
            #  $ curl -s https://api.github.com/repos/systemd/systemd/releases/latest | \
            #     jq '.created_at|strptime("%Y-%m-%dT%H:%M:%SZ")|mktime'
            releaseTimestamp = "1720202583";
          in
          stdenv.mkDerivation (finalAttrs: {
            inherit pname version;

It is version 25.05 of NixOS that has systemd version 257.10.

github.com/NixOS/nixpkgs

pkgs/os-specific/linux/systemd/default.nix

ac62194c3


      
          assert withFido2 -> withOpenSSL;
          assert withSysupdate -> withOpenSSL;
          assert withImportd -> (withGcrypt || withOpenSSL);
          assert withUkify -> (withEfi && withBootloader);
          assert withRepart -> withCryptsetup;
          assert withBootloader -> withEfi;
          
          let
            wantCurl = withRemote || withImportd;
          
            version = "257.10";
          
            # Use the command below to update `releaseTimestamp` on every (major) version
            # change. More details in the commentary at mesonFlags.
            # command:
            #  $ curl -s https://api.github.com/repos/systemd/systemd/releases/latest | \
            #     jq '.created_at|strptime("%Y-%m-%dT%H:%M:%SZ")|mktime'
            releaseTimestamp = "1734643670";
          in
          stdenv.mkDerivation (finalAttrs: {
            inherit pname version;

So, something odd is going on.

Also, both of these versions of NixOS are no longer supported, upgrading may be wise.

magicquark · March 21, 2026, 10:07pm

If you have the closures, try running this command to see what has changed exactly.

plan9better · March 22, 2026, 7:19am

nix store diff-closures /run/current-system /nix/store/x00f3a3nw2bm0f9nx6chyy9wdwikai87-nixos-system-astor-2-24.11.20241012.54cce65.drv^out
libapparmor: 4.1.1 → 4.1.2
libarchive: 3.8.0 → 3.8.2
openssl: 3.4.2 → 3.4.3, +8.0 KiB
systemd: 257.9 → 257.10, +35.7 KiB
systemd-minimal-libs: 257.9 → 257.10
tailscale: -11.6 KiB
unbound: 1.23.1 → 1.24.1

Comparing the closures, the only change i expect is tailscale since it’s updated by passing the package = pkgsLatest.tailscale option and not a whole nixpkgs update. I do see your point about systemd version change and it is interesting but my config doesn’t mention 25.05 anywhere and i don’t , and isn’t the fact that running

$ cat /run/current-system/nixos-version
24.11.20241012.54cce65

shows the same nixpkgs commit as the new derivation evidence enough that the nixpkgs has not changed, or should i not pay much attention to that?

plan9better · March 22, 2026, 7:21am

Also by NixOS version do you mean the system.stateVersion or nixpkgs version?

magicquark · March 22, 2026, 8:04am

The nixpkgs version. The system.stateVersion should in the vast majority of cases be set to the version of NixOS at first install and not changed thereafter.

magicquark · March 22, 2026, 8:09am

This makes more sense now. You probably have pkgsLatest set to NixOS 25.05 because at the time of NixOS 24.11, that was the latest version.

How are you defining pkgsLatest in your config?

I presume that your input for pkgsLatest has been updated causing these version bumps that you are seeing.

magicquark · March 22, 2026, 8:11am

plan9better:

and isn’t the fact that running
$ cat /run/current-system/nixos-version
24.11.20241012.54cce65
shows the same nixpkgs commit as the new derivation evidence enough that the nixpkgs has not changed

Not in your case because you are using multiple versions of nixpkgs in your configuration.

plan9better · March 22, 2026, 11:20am

Correction, it’s nixpkgs25being used for tailscale and not latest as a part of a larger flake.

by

i meant it doesn’t mention system.stateVersion = 25.05 since that’s what i wrongly assumed you meant.

inputs:

    nixpkgs25.url = "github:nixos/nixpkgs/nixos-25.05";
    nixpkgsLatest.url = "github:nixos/nixpkgs/nixos-unstable";
    nixpkgsInfra.url = "github:NixOS/nixpkgs/54cce652cb480258b12f15b2915e7a6c79ac51ae";

then nixpkgsInfra are passed as just nixpkgs to the machine configuration generation function:

      envInput = {
          clusters = clusters;
          nixpkgs = nixpkgsInfra;
          flake = self;
          config.allowUnfree = true;
          pkgs-25 = import nixpkgs25 {
            system = "x86_64-linux";
            config.allowUnfree = true;
          };
        };
      in {
        nixosConfigurations =
          (import ./hardware/infra/lib/generate-cluster.nix) envInput

which uses lib from nixpkgs so nixpkgsInfra

{clusters, ...} @ inputs: let
  lib = inputs.nixpkgs.lib;
...

and that lib is used with lib.nixosSystem. The other pkgs are passed around and used only when needed for overriding specific package versions like:

  services.tailscale = {
    enable = true;
    package = pkgs-25.tailscale;
  }

magicquark · March 22, 2026, 11:52am

Did you run nix flake update?

plan9better · March 22, 2026, 12:14pm

Yeah i just found it 16k of the 17k lines in the diff were all about tailscale changing. I found it once i realised that the systemd version mentioned in the update is not the one used by the system. diff mentions 257.9 → 257.10 but output of

# systemctl --version
systemd 256 (256.4)

shows that’s not the one used.

Output format of the diff led me to believe all / many packages on the system changed the way they are being built while it was just tailscale.

Thank you for your help