Coreutils 9.0 won't build

noblman · August 24, 2022, 11:40pm

Just tried a home-manager switch and I get this:

============================================================================
Testsuite summary for GNU coreutils 9.0
============================================================================
# TOTAL: 632
# PASS:  478
# SKIP:  153
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./tests/test-suite.log
Please report to bug-coreutils@gnu.org
============================================================================
make[4]: *** [Makefile:13580: tests/test-suite.log] Error 1
make[4]: Leaving directory '/build/coreutils-9.0'
make[3]: *** [Makefile:13688: check-TESTS] Error 2
make[3]: Leaving directory '/build/coreutils-9.0'
make[2]: *** [Makefile:13948: check-am] Error 2
make[2]: Leaving directory '/build/coreutils-9.0'
make[1]: *** [Makefile:13465: check-recursive] Error 1
make[1]: Leaving directory '/build/coreutils-9.0'
make: *** [Makefile:13950: check] Error 2
error: builder for '/nix/store/jn1b2kqxap814a8h7fw27fmg34aq6605-coreutils-9.0.drv' failed with exit code 2;
       last 10 log lines:
       > ============================================================================
       > make[4]: *** [Makefile:13580: tests/test-suite.log] Error 1
       > make[4]: Leaving directory '/build/coreutils-9.0'
       > make[3]: *** [Makefile:13688: check-TESTS] Error 2
       > make[3]: Leaving directory '/build/coreutils-9.0'
       > make[2]: *** [Makefile:13948: check-am] Error 2
       > make[2]: Leaving directory '/build/coreutils-9.0'
       > make[1]: *** [Makefile:13465: check-recursive] Error 1
       > make[1]: Leaving directory '/build/coreutils-9.0'
       > make: *** [Makefile:13950: check] Error 2
       For full logs, run 'nix log /nix/store/jn1b2kqxap814a8h7fw27fmg34aq6605-coreutils-9.0.drv'.

Looks like an issue with coreutils-9.0. Any thoughts?

knedlsepp · August 25, 2022, 6:29am

From the logs you can see one of the tests of coreutils is failing.
Here you can see that there are already some known cases of tests that don’t properly work on certain filesystems.
It could be that you just discovered a new test that is sensitive to that and needs to be disabled in nixpkgs.

vcunat · August 30, 2022, 6:45am

If you want to debug this, run with --keep-failed and inspect tests/test-suite.log. (I suppose you don’t want to look into why you even need to rebuild coreutils yourself.)

noblman · September 7, 2022, 9:18pm

Right. coreutils seems like something that should work, or it broken, be fixed quickly. Is there a work-around?

noblman · September 7, 2022, 9:19pm

I am running btrfs, so maybe that is my problem. Any ideas how to get around this issue?

knedlsepp · September 8, 2022, 12:04am

Since I’m also running btrfs I just tried to nix build nixpkgs#coreutils --rebuild -L and could reproduce your issue:

...
coreutils> PASS: tests/cp/preserve-slink-time.sh
coreutils> PASS: tests/cp/reflink-perm.sh
coreutils> FAIL: tests/cp/reflink-auto.sh
coreutils> PASS: tests/cp/slink-2-slink.sh
coreutils> PASS: tests/cp/src-base-dot.sh
coreutils> PASS: tests/cp/sparse-to-pipe.sh
...

So here we have the culprit! I did create coreutils: Skip another test that fails on btrfs by knedlsepp · Pull Request #190211 · NixOS/nixpkgs · GitHub to address the issue.

You now have a couple of options:

Wait until this gets merged
Build your home-manager profile (specifically coreutils) on another non-btrfs machine (VM?) and copy it back
- You could do that by locally specifying a remote builder, or
- You could run the same home-manager commands (with the same channel) on another machine and then use nix copy to copy these paths back to your btrfs-machine.
Add a local overlay with an equivalent patch e.g. in ~/.config/nixpkgs/overlays.nix (or somwhat similarly to your configuration.nix if you run NixOS):

[
  (self: super: {
    coreutils = super.coreutils.overrideAttrs (o: {
      postPatch = ''
          sed '2i echo Skipping cp reflink-auto test && exit 77' -i ./tests/cp/reflink-auto.sh
      '' + o.postPatch;
    });
  })
]

However the question by @vcunat still stands

I suppose you don’t want to look into why you even need to rebuild coreutils yourself

To give a bit more background:
Normally you would never run into the issue you’re seeing as nix would always first look up if the build is already available in the cache. So normally you would hit the cache and nix would just download the precompiled coreutils and you would never notice that the tests actually don’t run on your machine. However there is something causing this build to not be available in the cache. Maybe you are using a custom fork of nixpkgs or you have already something in your ~/.config/nixpkgs/overlays.nix, or your home-manager config somehow needs a tweaked coreutils. You might want to address that instead as this could save you some time compiling all packages depending on coreutils as well as some headaches with similarly flaky test suites.

noblman · September 8, 2022, 12:50am

Ah… I think this must be my issue. I have created some overlays in an attempt to have some shell scripts available to me. Not sure if I did it correctly… This is all quite hard to understand.

Here is an example overlay I am using:

self: super: {
  rofi-bluetooth = super.writeShellApplication {
    name = "rofi-bluetooth";
    runtimeInputs = [ super.rofi super.bluez super.coreutils super.gnugrep ];
    checkPhase = ''
    	runHook preCheck
    	runHook postCheck
    '';
    text = builtins.readFile ./rofi-bluetooth.sh;
  };
}

Any assistance would be greatly appreciated.

knedlsepp · September 8, 2022, 1:42am

Here is an example overlay I am using:

This one looks rather innocent as it is only adding a custom package on top. The overlay would only be the cause of a rebuild if you were to modify a dependency of coreutils or coreutils itself.

noblman · September 10, 2022, 10:11pm

How do I troubleshoot this? Why is coreutils being built? I don’t think I have “modified” a dependency. I’m not even sure what that means, or what that would look like.

noblman · September 10, 2022, 11:33pm

More specifically, how do I troubleshoot this? I do not think I am using a custom fork of nixpkgs. I’m not even sure how to do that. I do not have a ~/.config/nixpkgs/overlays.nix, although I do have a ~/.config/nixpkgs/overlays/ directory. I do call coreutils from home.nix like this: ${pkgs.coreutils}/bin/...

knedlsepp · September 11, 2022, 6:53am

It’s somewhat hard to tell without seeing your configuration.
If you don’t want to share it, you could try commenting out most of it and work your way to a minimal example.

Some other potential causes for that rebuild issue:

Settings in ~/.config/nixpkgs/config.nix like gcc.arch or enableCuda
You’re on an architecture that’s not guaranteed to have builds in the cache for that channel version
You’re just unlucky and happen to be on a channel version where hydra couldn’t build your package due to some intermittent failure (unlikely as for such a package as coreutils the channel would not advance)

So essentially you need to play around with your nix-channel and your configuration. In the config it could be either some “global” settings (such as the gcc.arch or enableCuda, or similar) or it’s an overlay (you mentioned “an example overlay”, do you have more?). Both of which could result in a change of coreutils dependencies and thus the need for a local build.

knedlsepp · September 11, 2022, 7:08am

This won’t help you with your problem, but as a sidenote:
This is a “known issue” in the nix ecosystem that one cannot reproduce a nix expression on another machine when there are differences in global configuration in ~/.config/nixpkgs as well as differing channel versions between machines.
The solution for that is that flakes will enforce that these things are part of the git-versioned flake and thus cannot differ between machines, reducing moving parts.

knedlsepp · September 13, 2022, 5:33am

Here is another thread where they successfully found the source of undesired builds. Maybe it helps you track down the problem.

noblman · September 14, 2022, 6:52pm

That might have been helpful. Does this look right to you:

$ nix-channel --list
home-manager https://github.com/nix-community/home-manager/archive/release-22.05.tar.gz

$ sudo nix-channel --list
nixos https://nixos.org/channels/nixos-22.05

Update:
It didn’t look right to me, so I did a:

$ nix-channel --add nixos https://nixos.org/channels/nixos-22.05
$ nix-channel --update

But core-utils is still building (and failing) from source.

noblman · November 8, 2022, 6:28am

I am finally attempting to tackle this again. I believe my problem is related to my trying to implement some custom script files into my setup.

Specifically, I have two scripts files that allow me to use rofi to control my bluetooth and wifi connections. I want to be able to call these scripts from within my home.nix. There seems to be no easy way to achieve this. I searched and asked on this forum for help, and the best I could come up with is to use overlays (which I don’t fully understand).

This is what I did:

I created a file called rofi-bluetooth.nix in ~/.config/nixpkgs/overlays:

self: super: {
  rofi-bluetooth = super.writeShellApplication {
    name = "rofi-bluetooth";
    runtimeInputs = [ super.rofi super.bluez super.coreutils super.gnugrep ];
    checkPhase = ''
    	runHook preCheck
    	runHook postCheck
    '';
    text = builtins.readFile ./rofi-bluetooth.sh;
  };
}

Also, in ~/config/nixpkgs/overlays I created rofi-bluetooth.sh which is a large file and I don’t think is necessary to show here. It is the script I used in my pre-NixOS days.

Next, I “installed” these overlays/derivations/whatever they are called in my home.nix by doing this:

home = {
  packages = with pkgs ; [
    rofi-bluetooth
  ]

And then throughout my home.nix I call these scripts like this inside my sxhkd config:

"super + b" = "rofi-bluetooth";

I used to be able to update my config with a home-manager switch, but now when I try I get the following:

❯ home-manager switch
these 10 derivations will be built:
  /nix/store/jn1b2kqxap814a8h7fw27fmg34aq6605-coreutils-9.0.drv
  /nix/store/1zp5h9dand7xq0i1p7ybmzpm5wcc5awm-rofi-bluetooth.drv
  /nix/store/6pgkpiaf3x8fvyxgv19vqc0kawavflm5-rofi-network-manager.drv
  /nix/store/c414jyj3f3m9j1s7n87afb856ayba2hp-home-manager-path.drv
  /nix/store/2vmbaglcg5gvdvmkbbqr9mdaz53v69qd-hm_fontconfigconf.d10hmfonts.conf.drv
  /nix/store/mgg7blcd6x874sq65k3w59pvkrhxrjzs-polybar.conf.drv
  /nix/store/d167644k8pwnqpf65dla29vgkiy5yzlc-polybar.service.drv
  /nix/store/8wxry9mafxrpdjw82rc0hqx6mnlh4dqm-home-manager-files.drv
  /nix/store/vfkvpynq8cdy13rlbhbcmk9849i5afg1-activation-script.drv
  /nix/store/4rvw4503r8ziyvyz68q023ax38c4wc3g-home-manager-generation.drv
building '/nix/store/mgg7blcd6x874sq65k3w59pvkrhxrjzs-polybar.conf.drv'...
building '/nix/store/jn1b2kqxap814a8h7fw27fmg34aq6605-coreutils-9.0.drv'...
building '/nix/store/d167644k8pwnqpf65dla29vgkiy5yzlc-polybar.service.drv'...
unpacking sources
unpacking source archive /nix/store/is8fadziysa55wpq1dppzhv5yhlhhm6n-coreutils-9.0.tar.xz
source root is coreutils-9.0
setting SOURCE_DATE_EPOCH to timestamp 1632488258 of file coreutils-9.0/doc/coreutils.info
patching sources
applying patch /nix/store/2j8nv5xpgw7js9lr3dnspc2vk04jifzv-fix-chmod-exit-code.patch
patching file NEWS
Hunk #1 succeeded at 143 with fuzz 2 (offset 141 lines).
patching file src/chmod.c
patching file tests/chmod/ignore-symlink.sh
patching file tests/local.mk
applying patch /nix/store/snhl0w63mc0s3q6y386g061bgc7jdwwn-disable-seek-hole.patch
patching file src/copy.c
patching file tests/seek-data-capable
applying patch /nix/store/nqrwkrrnpc7443annivd10zgnnfcc0cm-fix-arm64-macos.patch
patching file src/uname.c
autoreconfPhase
autoreconf: export WARNINGS=
autoreconf: Entering directory '.'
autoreconf: running: autopoint --force
Copying file build-aux/config.rpath
Copying file m4/extern-inline.m4
Copying file m4/glibc2.m4
Copying file m4/glibc21.m4
...

etc. until the build eventually quits and I finally get this error:

error: builder for '/nix/store/jn1b2kqxap814a8h7fw27fmg34aq6605-coreutils-9.0.drv' failed with exit code 2;
       last 10 log lines:
       > ============================================================================
       > make[4]: *** [Makefile:13580: tests/test-suite.log] Error 1
       > make[4]: Leaving directory '/build/coreutils-9.0'
       > make[3]: *** [Makefile:13688: check-TESTS] Error 2
       > make[3]: Leaving directory '/build/coreutils-9.0'
       > make[2]: *** [Makefile:13948: check-am] Error 2
       > make[2]: Leaving directory '/build/coreutils-9.0'
       > make[1]: *** [Makefile:13465: check-recursive] Error 1
       > make[1]: Leaving directory '/build/coreutils-9.0'
       > make: *** [Makefile:13950: check] Error 2
       For full logs, run 'nix log /nix/store/jn1b2kqxap814a8h7fw27fmg34aq6605-coreutils-9.0.drv'.
error: 1 dependencies of derivation '/nix/store/1zp5h9dand7xq0i1p7ybmzpm5wcc5awm-rofi-bluetooth.drv' failed to build
error: 1 dependencies of derivation '/nix/store/c414jyj3f3m9j1s7n87afb856ayba2hp-home-manager-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/byl8h83l2h178l06rwhsavzj8vszkcw5-polybar.conf.drv' failed to build
error: 1 dependencies of derivation '/nix/store/q3r3418rgii6k55il672rvl9b2cc78jc-home-manager-generation.drv' failed to build

Hopefully this gives a bit more insight into my situation, and hopefully I can get some help. I am unable to make any changes to my system because of this.

NobbZ · November 8, 2022, 7:07am

You probably want to take the inputs from self. I am not sure though why this might cause nix to rebuild those.

I do not see why this requires an overlay, even less why it has to use impure overlay discovery…

Can’t you use a let binding in the home.nix?

Something like this:

home.packages = let
  rofi-bt = pkgs.writeShellApplication {
    name = "rofi-bluetooth";
    runtimeInputs = [ pkgs.rofi pkgs.bluz pkgs.coreutils pkgs.gnugrep ];
    text = builtins.readFile ./rofi-bluethooth.sh;
  };
  in [ rofi-bt ];

noblman · November 8, 2022, 7:49am

OK, I am going to try your suggestion of using a let binding instead of an overlay, but I have a question. Currently I have defined packages in home.nix like this:

home.packages = with pkgs; [
  feh
  ffmpeg
  ...
];

Is if possible to easily incorporate the let binding without having to add pkgs. in front of every application and without having to remove the with pkgs;?

NobbZ · November 8, 2022, 7:53am

with is something that the community is arguing about badly, and I am in favor of removing it from the language, so even though it is possible, you shouldn’t (in my opinion).

home.packages = with pkgs; let
  rofi-bt = writeShellApplication {
    name = "rofi-bluetooth";
    runtimeInputs = [ rofi bluz coreutils gnugrep ];
    text = builtins.readFile ./rofi-bluethooth.sh;
  };
  in [ rofi-bt feh ffmpeg ];

noblman · November 8, 2022, 7:59am

So you are saying I shouldn’t use your last example, and instead do this:

home.packages = let
  rofi-bt = writeShellApplication {
    name = "rofi-bluetooth";
    runtimeInputs = [ rofi bluez coreutils gnugrep ];
    text = builtins.readFile ./rofi-bluethooth.sh;
  };
  in [
    rofi-bt
    pkgs.feh
    pkgs.ffmpeg
  ];

What if I want to add a second let binding?

vcunat · November 8, 2022, 8:00am

lib.attrValues { inherit (pkgs) rofi bluez coreutils gnugrep; }

… though perhaps I partially mean that as a joke.