Nix-bisect -- Bisect Nix Builds

The ability to use git bisect to fix a broken build or find the source of a regression is one of the major benefits of nix and nixpkgs in my mind.

The naive

git bisect run nix build -f. attrname

is not perfect though. I have developed a tool to improve this workflow. With nix-bisect you can replace that command above with

git bisect run python3 -m nix_bisect.cli attrname

You get the following benefits out of the box:

  • nicer output, with color highlighting on bisect success/failure
  • bisect skip on dependency failures, bisect bad only if the actual attribute fails to build
  • the bisect is aborted on ctrl-c, instead of registering as a bisect bad
  • if there is some unexpected failure, like an instantiation failure, the bisect is aborted instead of registering as a bisect bad
  • it builds expressions by default, which means that you can override attributes on the fly.

In addition to that out of the box behaviour you can also use it for more complex use-cases. For example I am currently debugging a build failure of sage.doc. I’m using the following command:

git bisect run python3 -m nix_bisect.cli --try-cherry-pick e3601e1359ca340b9eda1447436c41f5aa7c5293 --max-rebuilds 500 --failure-line="TypeError: __init__()" sage.doc

In addition to the benefits above this will

  • Try to cherry-pick commit e3601e1e into the tree before starting the build. This is really useful to bisect when there are some already fixed intermediate failures. This option can be passed multiple times. When the commit fails to apply (for example because it is already applied on the current checkout), it is simply not applied. The tree is reset to the exact state it was before (including unstaged changes) once the build is finished.

  • Skip on any build that would require more than 500 local builds.

  • Register bisect bad only when the build fails and the specified line occurs in the output. If the build fails without the line the current revision is skipped.

  • Make use of cached builds and cached failures (which is only possible with --failure-line).

This is not my first attempt at writing a tool like this. The previous ones always failed because the command line was never flexible enough to capture all use-cases.

This time I actually wrote a python library instead of an application. The CLI is just a convenience wrapper for the most common use-cases. If you need more flexibility, you can easily write your own bisection script with the nix_bisect library.

As an example, here is a script I used to debug a digikam segfault. It will build digikam (transparently dealing with an change of its attrname that happend at some point), skipping through all build failures. Once a build finally succeeds, it will prompt me to manually check for a segfault and use my input to decide whether the current revision is good or bad.

Keep in mind however that this is very early stages. Barely anything is documented. I built this to scratch my own itch, and I continue developing it whenever I need some feature.

Still, I can already be quite useful for some people. It is not packaged in nixpkgs, but if you want to try it out simply add this expression to your python packages:

(python3.pkgs.buildPythonPackage rec {
  pname = "nix-bisect";
  version = "0.2.0";
  src = pkgs.fetchFromGitHub {
    owner = "timokau";
    repo = "nix-bisect";
    rev = "v${version}";
    sha256 = "0rg7ndwbn44kximipabfbvvv5jhgi6vs87r64wfs5by81iw0ivam";
  };
})

What do you think?

25 Likes

this is a lot better than doing git bisect run nix build -f. attrname over night for expensive builds and hoping I get an accurate regression. Thanks!

1 Like

That’s actually very very helpful because I know the pains of bisecting nix builds, though it’s still miles better than how it would have to be if you were bisecting debian :rofl:

Next time I bisect I will try to use this :+1:

1 Like

Please add it to nixos weekly :slight_smile:

4 Likes

I’m glad to see the good reception :slight_smile: I’ve added an entry to the next nixos-weekly content call.

One bonus benefit I originally forgot to mention in the OP: It will build nix expressions by default, which means that you can override attributes on the fly!

In the long run I can imagine adding some cookbook-like scripts for common use-cases.

3 Likes

So good!

Does it allow passing extra arguments to nix-build? I would like to run all my bisections against my --builders server farm to speed it up, and perhaps in the future even against something like nixbuild.net - nix build as a service.

Not currently, but please test https://github.com/timokau/nix-bisect/pull/3 which adds an argument to the library and a --build-option flag to the cli.

Not sure if I like this design yet, seems very ad-hoc. But maybe good enough for now, lets see which other usecases pop up.

This is pretty cool! One thing I always find myself wanting is for git bisect on nixpkgs to only choose commits while narrowing whose build results are in the cache, and I think this tool is a good place to implement something like this. I see you already have an issue open for that feature, looking forward to it!

That’s actually already supported, with the --max-rebuilds flag of the CLI. You can do similar things in the library, just have a look at the implemenation of the flag:

The only downside of that approach is that it needs to evaluate the attribute and check the caches in order to determine the rebuild count. That may take ~2 seconds per revision. The issue is only about using channel information to avoid that. Another side-effect would be that some non-channel revisions wouldn’t be used, even though they wouldn’t necessitate any rebuilds either. So I’m not sure if its even worth it.

In summary, --max-rebuilds=0 probably is what you’re looking for.