Massive rebuild of packages that use CUDA despite cuda-maintainers and nix-community caches

Help
1

My configuration: github.com/syudoer/nixos-public (It’s messy, I know. In this case main files are hosts/default/configuration.nix and modules/nixos/core/nvidia.nix)

When I try to run nixos-rebuild, Nix starts building way too many CUDA-using packages locally, even though I’m using the caches from nix-community and cuda-maintainers.

Running nix build .#nixosConfigurations.default.config.system.build.toplevel --dry-run outputs:

these 49 derivations will be built:
  /nix/store/ysb2in53kx5bahfz3p5qzmah94fkghvd-xgboost-3.0.5.drv
  /nix/store/0ff0rsxisa688zx36pz7nwlzk706g67d-python3.13-xgboost-3.0.5.drv
  /nix/store/0g1z22rq1nn8cblcrxrs9narsxg2c0g2-activation-script.drv
  /nix/store/8zab0fqng7w0rkdq09gjfxfvla7ckmjy-nix-2.28.5.drv
  /nix/store/48p61hhxiv0an5lk8y4zl9zj4ignr1fs-hercules-ci-cnix-store-0.4.0.0.drv
  /nix/store/b4p5w32ayvbj5f7nwwm6xh2gp6svca2g-cachix-1.9.1.drv
  /nix/store/kd11ypir2l9yclcydlk73gfhfa1x8d9y-python3.13-triton-3.5.0.drv
  /nix/store/lalbic1mhbchss820arc1d0xgrkca89x-magma-2.9.0.drv
  /nix/store/i8psn8avrcsv3c5m4lap3zs1ch2ds2a5-python3.13-torch-2.9.0.drv
  /nix/store/rj6m0fcmaw1k1laihirjw5q1806zw78j-python3.13-safetensors-0.6.2.drv
  /nix/store/2drx0cywi1zsxgq9df12dyxyclm6lbmk-python3.13-transformers-4.57.1.drv
  /nix/store/2hz8dyaanr1ggjxxx0yr8jljh7zixrra-whisper-cpp-1.8.2.drv
  /nix/store/ngpmw9mypi4wnvzgwkbkz9kn4dx5a1g3-ffmpeg-full-8.0.drv
  /nix/store/4wvdiywal343fwcp8rhb48ywiid4k2qi-ffmpeg-fix-path.patch.drv
  /nix/store/b1ndwzq4wnq148rih3q4hbz0xy3gsmvw-python3.13-pydub-0.25.1.drv
  /nix/store/pg5d9s26wk12jvqj8idaayh7a27xnwx2-python3.13-torchvision-0.24.0.drv
  /nix/store/dg03j6n527rb088j6b4mahlm75m2ny30-python3.13-albumentations-2.0.8.drv
  /nix/store/pibyil8p626hhb35vxyil8dy1w35gbcj-python3.13-einops-0.8.1.drv
  /nix/store/2lsgfar5v1hdq4b4yh3if7h8b2w93m5l-python3.13-plotly-6.3.1.drv
  /nix/store/iaxs22brin0i878jbf5w5jzq78y3vc33-python3.13-iopath-0.1.10.drv
  /nix/store/cjclfa7whv1gh0gj17z7zxvv9km31prm-python3.13-fvcore-0.1.5.post20221221.drv
  /nix/store/2k6fm0kgk77vdw5vrbxfcs1n57l1mmg8-python3.13-optuna-4.5.0.drv
  /nix/store/d37013gdz61176343kvb5zvv6fyisimz-python3.13-resampy-0.4.3.drv
  /nix/store/qw08wrpmgqbi8mx4mg18c8p1s6m14w5l-python3.13-librosa-0.11.0.drv
  /nix/store/p5nvyvjpmrs34yhjis0j47ijma06z8va-ffmpeg-full-6.1.3.drv
  /nix/store/y604qpg0j7npqh1h3gz8bl7ihzapb2zm-ffmpeg.drv
  /nix/store/rsb9vgdygrz419d51pj756vsdiysz54n-python3.13-torchaudio-2.9.0.drv
  /nix/store/vm89jf3p4gzy399fxxms3m0xk0v41dzl-python3-3.13.9-env.drv
  /nix/store/2azmxy5ggc5x0i6pvs5j01b2jdgkz864-system-path.drv
  /nix/store/3mnd120980f1sir6amdqzcx5rkx3sy4q-nix.conf.drv
  /nix/store/920kdy8nsiknsw30z24a5p9n4qb3mcsi-X-Restart-Triggers-nix-daemon.drv
  /nix/store/3g06va35g8fcic6ajqvljd1ykh1g04mi-unit-nix-daemon.service.drv
  /nix/store/rr60bjx85yp0mslqhjfr2nb95lmnqjn6-home-manager-generation.drv
  /nix/store/alcmlv8zqny661iklbvifp3hz7iz4pxx-unit-home-manager-user.service.drv
  /nix/store/gw3b04rnkvan2jp5vcy2ailk62nw7s2v-X-Restart-Triggers-polkit.drv
  /nix/store/fmyf81zgr1m4s82cwgd4dd6hb1qmr2dk-unit-polkit.service.drv
  /nix/store/hjx7hav7q0vfb0pwpsc320ydmvgdbksz-unit-accounts-daemon.service.drv
  /nix/store/kvnmpjkn3w1wg1mvvgwr55qrpihx705y-dbus-1.drv
  /nix/store/ln3h3l4d9smyf2h564z9pn6g852iaggk-X-Restart-Triggers-dbus.drv
  /nix/store/rl312f0w3kcibxp87ax182rw62m466g4-unit-dbus.service.drv
  /nix/store/5gzc048z9rgircy110rm7vqqz7p2h627-system-units.drv
  /nix/store/rb5dcjd4hr589wl8j9wfl7vcg9xf7qf8-set-environment.drv
  /nix/store/8f5l1lhzn2rb5yzcpp673gchj5wrz08z-etc-profile.drv
  /nix/store/9c89013fx71cdc0ayqb89dlldlb7pwp1-etc-zshenv.drv
  /nix/store/ff47jf0liin9n2d4m95aiqbpjhd0m1cz-unit-dbus.service.drv
  /nix/store/jqlx4vvarp5hvfrzz21qxafsh2i9crrw-user-units.drv
  /nix/store/lfaf2zw8fbv6bcysvpnv76gs4l6cfmqn-etc-pam-environment.drv
  /nix/store/3zg94rfql306sgv1bj12rxs2xh7cbngb-etc.drv
  /nix/store/bg4xs8hjhwdz1kx4gjs92v0f8pcvls5q-nixos-system-nixos1-25.11.20251117.89c2b23.drv

And in case you haven’t noticed, IT IS GOING TO BUILD nix, which is extremely weird.

Are others able to reproduce this, and how can this be fixed?

2

I don’t use cuda, but it looks like the cache has moved: CUDA - Official NixOS Wiki (substituter is now https://cache.nixos-cuda.org/). This other thread might help too: 'cuda' jobset deleted on hydra.nix-community.org? - #2 by waffle8946

2 Likes
3

I ran into this exact issue recently and it caused me significant frustration. My NixOS workstation would suddenly decide to compile PyTorch with CUDA support from scratch, consuming 18+ hours and often crashing with OOM errors.

The pain was so severe that I ended up building my own local cache server using Harmonia, with nightly auto-builds for my entire fleet. I wrote about the architecture here: Automating nightly NixOS builds with a local cache server

However, after setting up the whole thing, I discovered the root cause was much simpler (as replied by @dtannock): the official CUDA binary cache URL changed from cuda-maintainers.cachix.org to https://cache.nixos-cuda.org/, and information about this migration was surprisingly sparse.

Once I added the new cache URL, most of the heavy CUDA builds simply disappeared. Here’s my current cache configuration: nix.nix

The local cache server setup is still valuable for pinning nixpkgs revisions and avoiding compilation when upstream caches (Hydra or CUDA) lag behind nixos-unstable, but switching to the correct cache URL solved the immediate problem.