Nix-bisect -- Bisect Nix Builds

The ability to use git bisect to fix a broken build or find the source of a regression is one of the major benefits of nix and nixpkgs in my mind.

The naive

git bisect run nix build -f. attrname

is not perfect though. I have developed a tool to improve this workflow. With nix-bisect you can replace that command above with

git bisect run python3 -m nix_bisect.cli attrname

You get the following benefits out of the box:

  • nicer output, with color highlighting on bisect success/failure
  • bisect skip on dependency failures, bisect bad only if the actual attribute fails to build
  • the bisect is aborted on ctrl-c, instead of registering as a bisect bad
  • if there is some unexpected failure, like an instantiation failure, the bisect is aborted instead of registering as a bisect bad
  • it builds expressions by default, which means that you can override attributes on the fly.

In addition to that out of the box behaviour you can also use it for more complex use-cases. For example I am currently debugging a build failure of sage.doc. I’m using the following command:

git bisect run python3 -m nix_bisect.cli --try-cherry-pick e3601e1359ca340b9eda1447436c41f5aa7c5293 --max-rebuilds 500 --failure-line="TypeError: __init__()" sage.doc

In addition to the benefits above this will

  • Try to cherry-pick commit e3601e1e into the tree before starting the build. This is really useful to bisect when there are some already fixed intermediate failures. This option can be passed multiple times. When the commit fails to apply (for example because it is already applied on the current checkout), it is simply not applied. The tree is reset to the exact state it was before (including unstaged changes) once the build is finished.

  • Skip on any build that would require more than 500 local builds.

  • Register bisect bad only when the build fails and the specified line occurs in the output. If the build fails without the line the current revision is skipped.

  • Make use of cached builds and cached failures (which is only possible with --failure-line).

This is not my first attempt at writing a tool like this. The previous ones always failed because the command line was never flexible enough to capture all use-cases.

This time I actually wrote a python library instead of an application. The CLI is just a convenience wrapper for the most common use-cases. If you need more flexibility, you can easily write your own bisection script with the nix_bisect library.

As an example, here is a script I used to debug a digikam segfault. It will build digikam (transparently dealing with an change of its attrname that happend at some point), skipping through all build failures. Once a build finally succeeds, it will prompt me to manually check for a segfault and use my input to decide whether the current revision is good or bad.

Keep in mind however that this is very early stages. Barely anything is documented. I built this to scratch my own itch, and I continue developing it whenever I need some feature.

Still, I can already be quite useful for some people. It is not packaged in nixpkgs, but if you want to try it out simply add this expression to your python packages:

(python3.pkgs.buildPythonPackage rec {
  pname = "nix-bisect";
  version = "0.2.0";
  src = pkgs.fetchFromGitHub {
    owner = "timokau";
    repo = "nix-bisect";
    rev = "v${version}";
    sha256 = "0rg7ndwbn44kximipabfbvvv5jhgi6vs87r64wfs5by81iw0ivam";
  };
})

What do you think?

37 Likes

this is a lot better than doing git bisect run nix build -f. attrname over night for expensive builds and hoping I get an accurate regression. Thanks!

1 Like

That’s actually very very helpful because I know the pains of bisecting nix builds, though it’s still miles better than how it would have to be if you were bisecting debian :rofl:

Next time I bisect I will try to use this :+1:

1 Like

Please add it to nixos weekly :slight_smile:

5 Likes

I’m glad to see the good reception :slight_smile: I’ve added an entry to the next nixos-weekly content call.

One bonus benefit I originally forgot to mention in the OP: It will build nix expressions by default, which means that you can override attributes on the fly!

In the long run I can imagine adding some cookbook-like scripts for common use-cases.

3 Likes

So good!

Does it allow passing extra arguments to nix-build? I would like to run all my bisections against my --builders server farm to speed it up, and perhaps in the future even against something like nixbuild.net - nix build as a service.

Not currently, but please test Make it possible to pass options to nix-build by timokau · Pull Request #3 · timokau/nix-bisect · GitHub which adds an argument to the library and a --build-option flag to the cli.

Not sure if I like this design yet, seems very ad-hoc. But maybe good enough for now, lets see which other usecases pop up.

This is pretty cool! One thing I always find myself wanting is for git bisect on nixpkgs to only choose commits while narrowing whose build results are in the cache, and I think this tool is a good place to implement something like this. I see you already have an issue open for that feature, looking forward to it!

That’s actually already supported, with the --max-rebuilds flag of the CLI. You can do similar things in the library, just have a look at the implemenation of the flag:

The only downside of that approach is that it needs to evaluate the attribute and check the caches in order to determine the rebuild count. That may take ~2 seconds per revision. The issue is only about using channel information to avoid that. Another side-effect would be that some non-channel revisions wouldn’t be used, even though they wouldn’t necessitate any rebuilds either. So I’m not sure if its even worth it.

In summary, --max-rebuilds=0 probably is what you’re looking for.

1 Like

Check out this deliciously ugly hack I cooked up to bisect a whole nixos+home-manager system for #80936:

It involves monkey-patching nix-build in the home-manager switch script to print out the instantiation result instead.

2 Likes

I just released version 0.3.0. Do not mistake “release” for stability. The API will almost certainly change in the future, and this may eat your repo. But there are a few exciting new features. The two highlights are:

  • Enhanced caching even for dependency failures.
  • A custom bisect runner with skip-ranges and autofix functionality (fixes reasons for “skip”; for example enables bisection “through” ranges with a broken dependency).

Both these changes result in big speed boosts while the autofix functionality makes it possible to fully automatically bisect issues that would have needed human intervention before. The custom bisect runner also opens the doors for future improvements. Examples could include changing the max-rebuild count over time or parallel bisection.

Note that the bisect runner in particular is still pretty rough around the edges. It is currently stateful, cherry-picks are only remembered in RAM and not persisted in the git directory. The autofix functionality also has some room for improvement, especially in the way in which it invalidates skip-ranges when it has found a commit that unbreaks the range.

Check out the usage example in the README.

1 Like

Version 0.4.0

I have just released version 0.4.0. The main change in this release is on the UX side. I realized that nix-bisect was doing multiple relatively unrelated things:

  1. determine the status of a nix build and emit exit codes based on that status,
  2. set up an environment with cherry-picks that the build can run in,
  3. run bisects and automatically fix unrelated breakage,
  4. provide python bindings to nix.

So I decided to split it up into multiple executables. There is

nix-build-status

Determines the status of a nix build as efficiently as possible and emits configurable exit codes with sensible defaults. It only actually performs the build when necessary. If everything could be fetched from a binary cache, that fetch doesn’t actually need to be performed. The build is just registered as a success. In addition to nix’s caching, it also caches failures and dependency failures. You can mostly use this as a drop-in replacement for nix build in your bisects and it should to The Right Thing. Here’s the documentation:

$ nix-build-status --help
usage: nix-build-status [-h] [--file FILE] [--option name value]
                        [--argstr name value] [--max-rebuilds MAX_REBUILDS]
                        [--failure-line FAILURE_LINE]
                        [--on-success {good,bad,skip,skip-range,<int>}]
                        [--on-failure {good,bad,skip,skip-range,<int>}]
                        [--on-dependency-failure {good,bad,skip,skip-range,<int>}]
                        [--on-failure-without-line {good,bad,skip,skip-range,<int>}]
                        [--on-instantiation-failure {good,bad,skip,skip-range,<int>}]
                        [--on-resource-limit {good,bad,skip,skip-range,<int>}]
                        [--rebuild-blacklist REBUILD_BLACKLIST]
                        drvish

Build a package with nix, suitable for git-bisect.

positional arguments:
  drvish                Derivation or an attribute/expression that can be
                        resolved to a derivation in the context of the nix
                        file

optional arguments:
  -h, --help            show this help message and exit
  --file FILE, -f FILE  Nix file that contains the attribute
  --option name value   Set the Nix configuration option `name` to `value`.
  --argstr name value   Passed on to `nix instantiate`
  --max-rebuilds MAX_REBUILDS
                        Number of builds to allow.
  --failure-line FAILURE_LINE
                        Line required in the build logs to count as a failure.
  --on-success {good,bad,skip,skip-range,<int>}
                        Bisect action if the expression can be successfully
                        built
  --on-failure {good,bad,skip,skip-range,<int>}
                        Bisect action if the expression can be successfully
                        built
  --on-dependency-failure {good,bad,skip,skip-range,<int>}
                        Bisect action if the expression can be successfully
                        built
  --on-failure-without-line {good,bad,skip,skip-range,<int>}
                        Bisect action if the expression can be successfully
                        built
  --on-instantiation-failure {good,bad,skip,skip-range,<int>}
                        Bisect action if the expression cannot be instantiated
  --on-resource-limit {good,bad,skip,skip-range,<int>}
                        Bisect action if a resource limit like rebuild count
                        is exceeded
  --rebuild-blacklist REBUILD_BLACKLIST
                        If any derivation matching this regex needs to be
                        rebuilt, the build is skipped

The skip-range action requires the extra-bisect bisect runner. Using in in a regular bisect will abort the bisection.

bisect-env

Runs a command in a temporary git environment with some
commits cherry-picked. This is automatically used by the bisect runner.

$ bisect-env --help
usage: bisect-env [-h] [--try-pick TRY_PICK] [--pick PICK] cmd ...

Run a program with a certain environment

positional arguments:
  cmd                  Command to run
  args

optional arguments:
  -h, --help           show this help message and exit
  --try-pick TRY_PICK  Cherry pick a commit before building (only if it
                       applies without issues).
  --pick PICK          Cherry pick a commit before building, abort on failure.

extra-bisect

A replacement for git-bisect with skip range support. For example if one of the build dependencies does not build on commit A and commit B, it likely won’t build on any of the commits in between either. If commits A and B are marked as skip-range, commits within the range will not be tested and extra-bisect will attempt to resolve the issue by cherry-picking the first commit after the skip range.

This can mostly be used as a drop-in replacement for git-bisect. One significant difference is that extra-bisect skip (the normal skips, not skip range) is not quite as clever as git-bisect skip yet. Instead of using the result of a weighted random number generator to pick the next commit, it will always pick the closest commit.

extra-bisect will use bisect-env internally to unbreak skip-ranges. I’m not happy with the name yet, so that is likely to change in the future if I come up with something better.

$ extra-bisect --help
usage: extra-bisect [-h] {good,bad,skip,skip-range,env,run,start,reset} ...

git-bisect with extra features

optional arguments:
  -h, --help            show this help message and exit

subcommands:
  You can use one of the following subcommands:

  {good,bad,skip,skip-range,env,run,start,reset}
                        Each subcommand has its own `--help` page. Also see
                        `man git-bisect` since many of the commands are
                        similar.

library

In addition to the command-line entry points, nix-bisect provides python bindings to the nix CLI which can be used for more complex use cases. Those have become significantly more usable in this version:

from nix_bisect.derivation improve Derivation
# Uses caching, keeps a gcroot on `hello` for the lifetime of the python object
drv = Derivation("hello")

if not hello.can_build_deps():
	print("This is bad")
elif hello.can_build():
	print("This is good")
else:
	print("This is less bad")

Due to the rapid change and experimentation, the API is still a bit of a mess though. Its also sparsely documented.

Summary

Check out the full changelog.

While it has matured a lot since the initial release and I am much happier with the UX now, keep in mind that nix-bisect is still alpha-quality software. There are almost certainly bugs. There are rough edges. The API will change.

Despite all that, I think it makes bisecting nix builds a lot more pleasant already. I want to encourage you to play with it and provide feedback. If its a feature request, it may already be on my todo list. Still, hearing it from someone else may change my prioritization or just provide motivation :wink:

If you don’t have something to bisect right now, there’s an example in the README.

6 Likes

As is tradition, I noticed a bug right after pushing the release. Executing extra bisect bad (or any other command that attempts to automatically check out the next candidate) without at least one good and bad commit specified would hang. Please use 0.4.1 instead.

1 Like

I think too many of the options for nix-build-status have this docstring :wink:

Sort-of looking forward to using this version (but also not because the place where you need to bisect nixpkgs isn’t necessarily a good place)!

You just need to have a very flexible definition of success. The build successfully failed, as I wanted it to. (Thanks for the heads up)

It can be kind of fun to have the computer do all the hard work for you though!

1 Like

I was just about to start a thread musing about a tool like this, only to see that already exists!

What’s unclear to me from the description is whether it will cache the build result and avoid building when the derivation isn’t changed. Imagine I am git-bisecting a specific attribute in nixpkgs. There are many many commits that don’t change the derivation! As long as they build, running nix-build multiple times isn’t a problem, as the build will be quick, but for the bad commits it seems wasteful to run nix-build again on a different commit where the expression evaluates to the same derivation as a previously failed build.

Ah, the README says

Make use of cached builds and cached failures (which is only possible with --failure-line ).

although I am not sure why this would only be possible with --failure-line?

1 Like