Nixos-rebuild fails, hangs forever

Help
1

When I run sudo nixos-rebuild switch it starts as normal but hangs at the same specific point every time. I’m including my output below.

From my best assessment it has something to do with a python package but I tried to install different versions of Python but it didn’t seem to help. At one point I waited for the rebuild to complete for over 3 hours before I stopped it.

All suggestions welcome
Thanks in advance

bash ❯ sudo nixos-rebuild switch
building Nix...
building the system configuration...
these 18 derivations will be built:
  /nix/store/0pa2xm0mfjji5m21njirbdgq0vpz9a56-python3.13-pyrate-limiter-3.9.0.drv
  /nix/store/d20bbhb7pvj2p07c6qabndzanj0jccvp-python3.13-moddb-0.12.0.drv
  /nix/store/rlp9v2clngldj63cg6pkyf56ff0ym8jn-lutris-unwrapped-0.5.19.drv
  /nix/store/wakmxbbvrjx4bp8vd6nfc2mmb2viqavh-lutris-0.5.19-fhsenv-rootfs.drv
  /nix/store/n8xa1xkfm48yhqznc78qvs3i0k58pkqk-lutris-0.5.19-bwrap.drv
  /nix/store/xwqkyfg4y2cp0v7qwjvmmdlyswybc3dv-lutris-0.5.19.drv
  /nix/store/lwsbb21dhj000lx9768j4g6sd51xvf8z-system-path.drv
  /nix/store/cs6i99n4rpdzczi6ybqnc61ynm1rfxlx-dbus-1.drv
  /nix/store/v9g0pmcbvzh3m6klxc9x5sbgqim6vcnf-X-Restart-Triggers-dbus.drv
  /nix/store/22p0y2pfyarn58l3gkzrk2n848jw17jz-unit-dbus.service.drv
  /nix/store/4s4nk6ra8af38pz0aapxc16bcpv5ghni-user-units.drv
  /nix/store/699zc9lk9ppk9p9aks1l8nk3v0x77l90-X-Restart-Triggers-polkit.drv
  /nix/store/hpbwxfsb54m31b0qyzhisbwy3xy6zmab-unit-accounts-daemon.service.drv
  /nix/store/n4ka3xqq0z1jv8w3v5iyf5csra1d6pmg-unit-polkit.service.drv
  /nix/store/rkks9ajsj1vmzpsh1r89xry9qx1q38wc-unit-dbus.service.drv
  /nix/store/hs9p1x3jz145zr0z4daczkk7ifay8gxd-system-units.drv
  /nix/store/q2qx8544dhiiwz6w59wbrfhf4r0m4lzd-etc.drv
  /nix/store/6rri6zxw3iig78i3pn3427njngi2sv42-nixos-system-Firefly-25.05.812554.6faeb062ee4c.drv
building '/nix/store/0pa2xm0mfjji5m21njirbdgq0vpz9a56-python3.13-pyrate-limiter-3.9.0.drv'...
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing pypa-build-hook
Using pypaBuildPhase
Sourcing python-runtime-deps-check-hook
Using pythonRuntimeDepsCheckHook
Sourcing pypa-install-hook
Using pypaInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing pytest-check-hook
Using pytestCheckPhase
Running phase: unpackPhase
unpacking source archive /nix/store/6gg2mcb80frz9ss8bpakqxfn48viz5as-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file "source/tests/test_others.py"
Running phase: patchPhase
Running phase: updateAutotoolsGnuConfigScriptsPhase
Running phase: configurePhase
no configure script, doing nothing
Running phase: buildPhase
Executing pypaBuildPhase
Creating a wheel...
pypa build flags: --no-isolation --outdir dist/ --wheel
* Getting build dependencies for wheel...
* Building wheel...
Successfully built pyrate_limiter-3.9.0-py3-none-any.whl
Finished creating a wheel...
Finished executing pypaBuildPhase
Running phase: pythonRuntimeDepsCheckHook
Executing pythonRuntimeDepsCheck
Checking runtime dependencies for pyrate_limiter-3.9.0-py3-none-any.whl
Finished executing pythonRuntimeDepsCheck
Running phase: installPhase
Executing pypaInstallPhase
Successfully installed pyrate_limiter-3.9.0-py3-none-any.whl
Finished executing pypaInstallPhase
Running phase: pythonOutputDistPhase
Executing pythonOutputDistPhase
Finished executing pythonOutputDistPhase
Running phase: fixupPhase
shrinking RPATHs of ELF executables and libraries in /nix/store/0mzcb0zcbxarl7cjmpc41b84q9rj1vi7-python3.13-pyrate-limiter-3.9.0
checking for references to /build/ in /nix/store/0mzcb0zcbxarl7cjmpc41b84q9rj1vi7-python3.13-pyrate-limiter-3.9.0...
patching script interpreter paths in /nix/store/0mzcb0zcbxarl7cjmpc41b84q9rj1vi7-python3.13-pyrate-limiter-3.9.0
stripping (with command strip and flags -S -p) in  /nix/store/0mzcb0zcbxarl7cjmpc41b84q9rj1vi7-python3.13-pyrate-limiter-3.9.0/lib
shrinking RPATHs of ELF executables and libraries in /nix/store/7vd3hr2paarr7lhm9z7ispv3l1f4hfa6-python3.13-pyrate-limiter-3.9.0-dist
checking for references to /build/ in /nix/store/7vd3hr2paarr7lhm9z7ispv3l1f4hfa6-python3.13-pyrate-limiter-3.9.0-dist...
patching script interpreter paths in /nix/store/7vd3hr2paarr7lhm9z7ispv3l1f4hfa6-python3.13-pyrate-limiter-3.9.0-dist
Executing pythonRemoveTestsDir
Finished executing pythonRemoveTestsDir
Running phase: installCheckPhase
no Makefile or custom installCheckPhase, doing nothing
Running phase: pythonCatchConflictsPhase
Running phase: pythonRemoveBinBytecodePhase
Running phase: pythonImportsCheckPhase
Executing pythonImportsCheckPhase
Check whether the following modules can be imported: pyrate_limiter
Running phase: pytestCheckPhase
Executing pytestCheckPhase
starting redis
waiting for redis to be ready
Could not connect to Valkey at /build/run/redis.sock: No such file or directory
pytest flags: -m pytest -k not\ \(test_limiter_01\) --numprocesses=8
============================= test session starts ==============================
platform linux -- Python 3.13.8, pytest-8.4.2, pluggy-1.6.0
rootdir: /build/source
configfile: pyproject.toml
plugins: asyncio-1.1.0, xdist-3.8.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
8 workers [613 items]  m
........................................................................ [ 11%]
........................................................................ [ 23%]
.......................................................F............F... [ 35%]
........................................................................ [ 46%]
..............................................F......................... [ 58%]
........................................................................ [ 70%]
........................................................................ [ 82%]
........................................................................ [ 93%]
^Cerror: interrupted by the user


~ took 22m3s
2

Seems like this may have been solved in Build failure: pyrate-limiter (dependency) · Issue #458830 · NixOS/nixpkgs · GitHub, but I’d wait until Breaking changes announcement for unstable - #105 by arianvp is merged for the branch you’re working before updating, just to be safe.

2 Likes
3

Judging by the log, the issue is with pyrate-limiter.

The problem is mentioned here: issue 458830
It seems the issue was fixed in PR 457990

Try updating again.

1 Like
4

Okay, that’s good information. I’ve tried updating a few more times to no avail.
I tried pinning old versions of the package and including it in my build but I’m having the same problem no matter what I try.

Mostly I’m wanting to find a workaround sooner than later so I don’t have to wait for a fix to be committed just to be able to make system builds and install software. Is there a way to rebuild without re-installing that package? Just install everything else except some selected package(s). Or is there a way to override the failing package with an older working version? That’s what I thought I was doing with this code in the snippet below but no luck.

let
  # ...
  temperary_python313Packages_pyrate_limiter = (import (builtins.fetchTarball
    "https://github.com/NixOS/nixpkgs/archive/f8918a1e3ea5e5549b2c78996842c6c3b47da2a8.tar.gz"
    ) {}).python313Packages.pyrate-limiter;
in
  # ...
  environment.systemPackages = with pkgs; [
    # ...
    temperary_python313Packages_pyrate_limiter
  ]
5

Your snippet is adding a version of pyrate-limiter to environment.systemPackages, but probably your original problem was that a package was pulling in that package as a dependency, so all your snippet accomplishes is adding yet another version of pyrate-limiter to the closure of all packages to be installed (since the pkgs.ContainingPackage still pulls in it’s version of pyrate-limiter)

Probably easiest thing is

nixos-rebuild build --show-trace |& grep 'while evaluating derivation'

to try to inspect the tree of dependencies and see which package is pulling in pyrate-limiter. You might also be able to use nix-tree on your current system closure, but there are edge cases where this won’t work.

I see three main options, but others may know more.

  • Option 1: remove that package from your systemPackages for now, and use a nix-shell with pinned package in the meantime if you need that package.
  • Option 2: Pin the old version of whatever is referring to pyrate-limiter and add that to systemPackages.
  • Option 3: Otherwise you are in the realm of overlays. Conceptually this is what you were trying to do in your snippet, but it usually causes cache misses, requiring building instead of downloading pre-built packages. If you want a somewhat worked example of doing this for python dependencies, you can see it at Having trouble overriding python dependencies but I’d still think in most cases just using an older pinned version of the package that was pulling in pyrate-limiter instead is an easier method.
1 Like
6

Yes, it is possible to install an older version of a package if you know its hash.
Typically, for this purpose, people use nix-versions.
Here’s how the author describes working with it: download-specific-package-version-with-nix.

I just checked, and indeed the current version in my system fails to build:

nix-shell -p python313Packages.pyrate-limiter 

       > FAILED tests/test_bucket_all.py::test_bucket_leak[clock0-create_sqlite_bucket] - assert 72 == 100
       > =========== 1 failed, 612 passed, 139 warnings in 260.36s (0:04:20) ============
       For full logs, run:
         nix log /nix/store/0pa2xm0mfjji5m21njirbdgq0vpz9a56-python3.13-pyrate-limiter-3.9.0.drv

But the old version builds perfectly:

nix-shell -p python313Packages.pyrate-limiter -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/e6f23dc08d3624daab7094b701aa3954923c6bbb.tar.gz

================ 613 passed, 139 warnings in 260.37s (0:04:20) =================
stopping redis
Finished executing pytestCheckPhase
pytestCheckPhase completed in 4 minutes 22 seconds
Running phase: pytestcachePhase
Running phase: pytestRemoveBytecodePhase

You could try something like this:

{ config, pkgs, ... }:

let
  # Pin only the needed package from old nixpkgs
  pyrate-limiter-fixed = (import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/e6f23dc08d3624daab7094b701aa3954923c6bbb.tar.gz";
    sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
  }) {}).python313Packages.pyrate-limiter;
in {
  environment.systemPackages = with pkgs; [
    # all other packages from the current channel
    git
    vim
    htop
    python313
    
    # pinned version
    pyrate-limiter-fixed
  ];
}

The build will fail initially, and you’ll need to replace the hash with the real one.
The traceback will show the actual hash, then you can run the build again with the correct hash.

But I don’t really like this method — besides being unreliable, it hardcodes the version and might cause other issues later.

—
From what we can tell, the root of the problem is that tests fail when building the fresh package.
It would actually be better to just disable testing for this package with doCheck = false and use the current version.

{ config, pkgs, ... }:

let
  pyrate-limiter-no-check = pkgs.python313Packages.pyrate-limiter.overridePythonAttrs (old: {
    doCheck = false;
  });
in {
  environment.systemPackages = with pkgs; [
    # other packages
    python313
    pyrate-limiter-no-check
  ];
}

I just tested this in my configuration, and it works.
I think it should work for you too:

ADDED
[A+] python3.13-pyrate-limiter 3.9.0

I tested it on

nix-shell -p nix-info --run "nix-info -m"                                                  1 ↵
 - system: `"x86_64-linux"`
 - host os: `Linux 6.12.57, NixOS, 25.11 (Xantusia), 25.11.20251105.ae814fd`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.31.2`
 - nixpkgs: `/nix/store/mdf5s0a86b7lnh38b71mkgzz5ca9f8gm-source`

UPD:
inline solution

# snip ...
  environment.systemPackages = [
    (pkgs.python313Packages.pyrate-limiter.overridePythonAttrs (old: {doCheck = false;}))
];
# snip ...
1 Like
7

Thanks for the great tips, it’s a lot to add to my arsenal for future trouble shooting. I had to go to bed last night but It appears that the issue was fixed upstream by the time that I woke up but I really appreciate y’alls help!

2 Likes