Build fails on hydra but not locally, what's going on?

I’ve been stumbling across a build failure caused by an update of pylint from 3.0.3 to 3.1.0 a few days ago. I’m relatively sure that update caused the build failure due to the hydra build status of nixpkgs:trunk:pylint.aarch64-darwin.

The first thing that confuses me is that the PR was merged to master on 2024-03-08, but the first failed build (which is the first one including that version upgrade) happened on 2024-03-22.

Additionally, when I now, just four days later, build pylint locally from master, it completes just fine.

$ nix build --print-out-paths .#pylint
$ result/bin/pylint --version
pylint 3.1.0
astroid 3.1.0
Python 3.11.8 (main, Feb  6 2024, 21:21:21) [Clang 16.0.6 ]

And indeed, there was a fix submitted and merged, but that merge happened multiple days before the first bad build, on 2024-03-17!

Additionally, looking at the “Changes” link in that build, I can actually see that the fix was already included!

Screenshot 2024-04-09 at 16.56.51

And the git log clearly shows that no other changes to pylint/default.nix have been made:

$ git --no-pager log --oneline pkgs/development/python-modules/pylint/default.nix
76235223edeb Merge #296433: python3Packages.{python-lsp-black,pylint}: fix build
3213fbef2c7e python311Packages.pylint: disable failing test
e30c983e07fd pylint: 3.0.3 -> 3.1.0
a245b3da258d python311Packages.pylint: 3.0.2 -> 3.0.3
2683b2885565 python3Packages.pylint: 3.0.1 -> 3.0.2
7e6d1c030a39 pylint: 2.17.5 -> 3.0.1

The obvious conclusion would be that some change in some dependency of pylint fixed the build. I wanted to find out which one, and so I started bisecting, starting from the first failed build.

$ git checkout d03ae62edd9402b6ee0471cb1543e1e19585025e
Updating files: 100% (5029/5029), done.
Note: switching to 'd03ae62edd9402b6ee0471cb1543e1e19585025e'.
$ nix build --print-out-paths .#pylint
# Um, it succeeded?
$ result/bin/pylint --version 
pylint 3.1.0
astroid 3.1.0
Python 3.11.8 (main, Feb  6 2024, 21:21:21) [Clang 16.0.6 ]

The hash is different than before, but (as this is input-addressed), it’s the exact same as the ones in the build failures on hydra.

Additionally, because the hash changed, I know for sure that the new build failures on hydra aren’t cached. I also installed nix 2.19.3 from current unstable as that is closer to the nix version hydra used, but even then, nix build --rebuild .#pylint succeeds.

So, how can I interpret and further diagnose this issue? Is the evaluation of this package impure? Does hydra evaluate/build the package differently than me? What’s the difference between staging-next and trunk?

1 Like

That was a fix for *-linux builds. I didn’t really look at darwin in that case. (and I don’t have one at hand anyway)

1 Like