Debug a failed derivation with `breakpointHook` and `cntr`

Guides
1

These instructions describe how to get an interactive shell at the point in a derivation when the derivation fails.

This works in a Linux single-user installation of Nix.

Add the breakpointHook package to the nativeBuildInputs of a mkDerivation.

nativeBuildInputs = [ breakpointHook ];

Install cntr

nix-env -i cntr

Run your Nix build command, and when it fails, the breakpointHook will output
a message like this:

build failed in installPhase with exit code 1
To attach install cntr and run the following command as root:

   cntr attach -t command cntr-/nix/store/6vwxqrwq5h1fd3nw4mc61wgk7rppn2qw-jupyterlab-extended

So we run that command as root. The root user doesn’t have a PATH to the
cntr command which we installed in our Nix profile, so give it the full path to our single-user-installed cntr.

sudo /home/$USER/.nix-profile/bin/cntr attach -t command cntr-/nix/store/6vwxqrwq5h1fd3nw4mc61wgk7rppn2qw-jupyterlab-extended

And now we’re in the sleeping Nix container. From here, run cntr exec to fully load the build environment.

/home/$USER/.nix-profile/bin/cntr exec

The $TMPDIR directory is where Nix has created the temporary directory for the build.

cd $TMPDIR

Now we are fully inside the context of the build, and any commands which we
enter in the shell will be as if we had entered that command on a line
in the phase (for example, buildPhase) of our derivation when the derivation failed.

References

11 Likes
2

alternatively, I use nix-shell to see if anything odd is happening:

[05:45:36] jon@jon-desktop /home/jon/projects/nixpkgs (master)
$ nix-shell default.nix -A cmake
these paths will be fetched (8.54 MiB download, 8.55 MiB unpacked):
...
[nix-shell:/home/jon/projects/nixpkgs]$ unpackPhase
unpacking source archive /nix/store/rr3zil97zd9jmvdpw8milynchz9zjff2-cmake-3.18.0.tar.gz
source root is cmake-3.18.0
setting SOURCE_DATE_EPOCH to timestamp 1594811960 of file cmake-3.18.0/Utilities/std/cm/vector
[nix-shell:/home/jon/projects/nixpkgs]$ cd $sourceRoot
[nix-shell:/home/jon/projects/nixpkgs]$ patchPhase
...
[nix-shell:/home/jon/projects/nixpkgs]$ configurePhase
...
[nix-shell:/home/jon/projects/nixpkgs]$ buildPhase
...

since I don’t do --pure, I’m able to use tree, ripgrep, and other tools to find what I’m looking for.

And, if I do need to create a patch, I can do git init && git add . after cd $sourceRoot. Then after writing the changes I want, I can just do git diff -- <file> > my-patch.patch to generate my patch

6 Likes
3

Thanks for the howto! This approach has come in very handy for me in the past when debugging builds that worked great in nix-shell, but not in the build sandbox.

2 Likes
4

Thanks for the tricks.

@jonringer If you want to fix phases like fixupPhase, do you run nix-shell as root to be allowed to write in /nix/store? – EDIT-- Oh, I just read this great documentation, and they give the trick to setup a $out in a temporary folder like:

$ cd some_empty_folder # Important to avoid errors during unpack phase
$ export out=~/tmpdev/bc-build/out
$ source $stdenv/setup # loads the environment variable (`PATH`...) of the derivation to ensure we are not using the system variables
$ set +e # To ensure the shell does not quit on errors/Ctrl+C ($stdenv/setup runs `set -e`)
$ set -x # Optional, if you want to display all commands that are run 
$ genericBuild

or if we just want to run a few phases only:

phases="buildPhase checkPhase" genericBuild

Note that if the derivation does not use the default genericBuild system, you can obtain the exact command by using:

$ nix show-derivation $(nix-instantiate .)

and checking what is the builder (usually it’s bash) and the arguments args (usually contains the name of the script to run).

@jamesbrock Is it possible to obtain the definition of the phases from there? If I manually redefine my phase like installPhase = ''mkdir -p $out/bin; cp myfile $out/bin/myfile'';, I can do echo $installPhase (without even exec), but I can’t write:

$ declare -f buildPhase

to check the default implementation of a phase, while this work in a nix shell.

Note to myself: the list of phases is in variable $phases, and by default if this variable is empty the full list is given here or can be checked in the source code. For each phase (say installPhase for the example), the default builder will first check if there is a variable with the name $installPhase (this arrives if you overwrite the default phase) and run in that case:

[nix-shell]$ eval "$installPhase"

(You want both eval and "..." otherwise it won’t do what you expect, see here) Otherwise it will run the default function installPhase:

[nix-shell]$ installPhase

If the phase is the special unpackPhase, it also goes to the $sourceRoot directory (proof here):

cp $sourceRoot
4 Likes
5

For anyone who stumbles up this, I just want to clarify that it is never good to modify the nix/store manually as this can potentially corrupt your entire installation if something critical is changed. To prevent this, even root isn’t able to easily modify /nix/store since it is actually a read-only bind mount by default.

4 Likes