Cache misconfiguration

I added proxmox-nixos with the following:

proxmox.nix
{  inputs, system, ... }:
{
  imports = [ inputs.proxmox-nixos.nixosModules.proxmox-ve ];
  nixpkgs.overlays = [ inputs.proxmox-nixos.overlays.${system} ];

  services.proxmox-ve.enable = true;
  services.proxmox-ve.ipAddress = "10.0.0.10";

  nix.settings =
    let
      server = "https://cache.saumon.network/proxmox-nixos";
      pubkey = "proxmox-nixos:D9RYSWpQQC/msZUWphOY2I5RLH5Dd6yQcaHIuug7dWM=";
    in
    {
      substituters = [ server ];
      trusted-substituters = [ server ];
      trusted-public-keys = [ pubkey ];
    };
}

I first rebuilt without services.proxmox-ve to refresh the trusted substituters. Rebuilding again (via nix-output-monitor) triggers compilation of the pve dependencies:

nixos-rebuild switch
$ nh os switch

> Building NixOS configuration
these 28 derivations will be built:
  ...
  /nix/store/kx1lvizc8pgqpsm4yb9bwd2bs6lnxw8b-pve-rs-0.10.10
  ...
┏━ Dependency Graph:
...
┃       │           ┌─ ⏵ pve-rs-0.10.10 (buildPhase) ⏱ 13s
...
┣━━━ Builds           │ Downloads
┗━ ∑ ⏵ 3 │ ✔ 7 │ ⏸ 21 │ ↓ 0 │ ↓ 0 │ ⏸ 20 │ ⏱ 1m1s

Checking nix config show confirms the substituters are configured:

nix config show
$ nix config show | grep substitute

always-allow-substitutes = true
builders-use-substitutes = true
substitute = true
substituters = https://cache.nixos-cuda.org https://cache.saumon.network/proxmox-nixos https://niri.cachix.org https://cache.nixos.org/
trusted-substituters = https://cache.nixos.org?priority=1 https://cache.saumon.network/proxmox-nixos https://cachix.cachix.org?priority=4 https://nix-community.cachix.org?priority=2 https://nixpkgs.cachix.org?priority=3 https://walker-git.cachix.org https://walker.cachix.org

Checking the cache for a derivation confirms it’s available:

curl derivation
$ curl -I https://cache.saumon.network/proxmox-nixos/kx1lvizc8pgqpsm4yb9bwd2bs6lnxw8b.narinfo

HTTP/2 200
server: nginx
date: Wed, 24 Dec 2025 15:59:50 GMT
content-type: text/x-nix-narinfo
content-length: 1938
x-attic-cache-visibility: public

I can fetch the derivation from the cache into my store (I don’t know how to do it in nix3)

realizing derivation
$ nix-store -r /nix/store/kx1lvizc8pgqpsm4yb9bwd2bs6lnxw8b-pve-rs-0.10.10 --substitute --option substituters https://cache.saumon.network/proxmox-nixos

this path will be fetched (0.00 MiB download, 14.41 MiB unpacked):
  /nix/store/kx1lvizc8pgqpsm4yb9bwd2bs6lnxw8b-pve-rs-0.10.10
copying path '/nix/store/kx1lvizc8pgqpsm4yb9bwd2bs6lnxw8b-pve-rs-0.10.10' from 'https://cache.saumon.network/proxmox-nixos'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/kx1lvizc8pgqpsm4yb9bwd2bs6lnxw8b-pve-rs-0.10.10

Now if I rebuild the os it doesn’t compile this one but does for the other derivations…

final rebuild
pve-qemu> WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
┏━ Dependency Graph:
...
┃       │           ┌─ ↓ ⏸ pve-qemu-server-9.0.23
┃       │        ┌─ ↓ ⏸ perl-5.40.0-env
┃       │        ├─ ⏵ pve-yew-mobile-gui-0.6.2 (buildPhase) ⏱ 55s
┃       │        ├─ ⏵ pve-qemu-10.0.2-4 (configurePhase) ⏱ 55s
┃       │     ┌─ ↓ ⏸ pve-manager-9.0.11
┃       │  ┌─ ↓ ⏸ proxmox-ve-9.0.11
...
┣━━━ Builds            │ Downloads
┗━ ∑ ⏵ 3 │ ✔ 14 │ ⏸ 21 │ ↓ 0 │ ↓ 0 │ ⏸ 19 │ ⏱ 1m40s^C
1 Like

You’ve done a lot of due diligence, last thing I can easily think of to check is are you overriding any of their flake inputs? If no, it could be that they haven’t populated their cache yet with the latest derivations.

2 Likes

Ah yes that’s something I checked as well. I don’t override their input because they suggested not to:

nix flake metadata - no follows
$ nix flake metadata | grep proxmox --after-context=5
├───proxmox-nixos: github:SaumonNet/proxmox-nixos/3be878a84866b9ef9214b8ea6f53630f4
7f4b192?narHash=sha256-Oq4aCjczgnFQqFNWZ6Ablg6x9579CO8tyBktYDYiZEs%3D (2025-11-26 2
0:15:13)
│   ├───flake-compat: github:edolstra/flake-compat/9100a0f413b0c601e0533d1d94ffd501
ce2e7885?narHash=sha256-CIVLLkVgvHYbgI2UpXvIIBJ12HWgX%2BfjA8Xf8PUmqCY%3D (2025-05-1
2 10:39:32)
│   ├───nixpkgs-libvncserver: github:NixOS/nixpkgs/e6f23dc08d3624daab7094b701aa3954
923c6bbb?narHash=sha256-3a7Tha/RwYlzH/v3PJrG7%2BHjOj4c6YOv2K8sqdGsHVQ%3D (2025-06-1
6 22:00:31)
│   ├───nixpkgs-stable: github:NixOS/nixpkgs/481cf557888e05d3128a76f14c76397b7d7cc8
69?narHash=sha256-G/iC4t/9j/52i/nm%2B0/4ybBmAF4hzR8CNHC75qEhjHo%3D (2025-10-21 03:1
0:16)
│   ├───nixpkgs-unstable: github:NixOS/nixpkgs/01f116e4df6a15f4ccdffb1bcd41096869fb
385c?narHash=sha256-f/QCJM/YhrV/lavyCVz8iU3rlZun6d%2BdAiC3H%2BCDle4%3D (2025-10-22 
06:30:52)
│   └───utils: github:numtide/flake-utils/11707dc2f618dd54ca8739b309ec4fc024de578b?
narHash=sha256-l0KFg5HjrsfsO/JpG%2Br7fRrqm12kzFHyUHqHCVpMMbI%3D (2024-11-13 21:27:1
6)

Regarding the cache not being populated, it is for example populated with /nix/store/kx1lvizc8pgqpsm4yb9bwd2bs6lnxw8b-pve-rs-0.10.10 as you can see from my curl above, but my system compiles it anyway.

Thank you for the kind answer.

1 Like

Dang, you did mention that in your due diligence, I glossed over it. It is pretty suspect behavior that is presenting. It could be something like the derivation size is larger than the allow payload of their reverse proxy, but that would be odd, but could be the edge case answer.

I’ll try to reproduce on my end as well and see if I can build it or get a similar result.

1 Like

So, I was able to get the cache working but there’s still issues with the flake:

❯ nixos-rebuild dry-run --flake
building the system configuration...
[1/0/1 built, 0.0 MiB DL] building pve-qemu-839b53b: remote: Compressing objects: 100% (9621/9621), done.^C

A dry-run has prompted a package build, which turns out to be a known issue: Remove IFD from cluster.nix · Issue #209 · SaumonNet/proxmox-nixos · GitHub

Not the same derivation your system, and NOM shows the same derivation when I attempt to a build:

❯ nh os test
> Building NixOS configuration
pve-qemu-839b53b> building '/nix/store/9s4n86ickbmb14v5chadacb8wlaglcaz-pve-qemu-839b53b.drv'
pve-qemu-839b53b> exporting git://git.proxmox.com/git/pve-qemu.git (rev 839b53bab89fddb7a7fb3a1d722e05df932cce4e) into /nix/store/iabx6s35m5wiiwi8cf5n5il6gkna93lv-pve-qemu-839b53b
pve-qemu-839b53b> Initialized empty Git repository in /nix/store/iabx6s35m5wiiwi8cf5n5il6gkna93lv-pve-qemu-839b53b/.git/
pve-qemu-839b53b> remote: Enumerating objects: 101, done.
pve-qemu-839b53b> remote: Counting objects: 100% (101/101), done.
pve-qemu-839b53b> remote: Compressing objects: 100% (97/97), done.
pve-qemu-839b53b> remote: Total 101 (delta 3), reused 32 (delta 1), pack-reused 0
pve-qemu-839b53b> Receiving objects: 100% (101/101), 1.01 MiB | 1.82 MiB/s, done.
pve-qemu-839b53b> Resolving deltas: 100% (3/3), done.
pve-qemu-839b53b> From git://git.proxmox.com/git/pve-qemu
pve-qemu-839b53b>  * branch            839b53bab89fddb7a7fb3a1d722e05df932cce4e -> FETCH_HEAD
pve-qemu-839b53b> Switched to a new branch 'fetchgit'
pve-qemu-839b53b> Submodule 'qemu' (git://git.proxmox.com/git/mirror_qemu) registered for path 'qemu'
pve-qemu-839b53b> Cloning into '/nix/store/iabx6s35m5wiiwi8cf5n5il6gkna93lv-pve-qemu-839b53b/qemu'...
pve-qemu-839b53b> remote: Enumerating objects: 10835, done.
pve-qemu-839b53b> remote: Counting objects: 100% (10835/10835), done.
pve-qemu-839b53b> remote: Compressing objects: 100% (9621/9621), done.
┏━ Dependency Graph:
┃ ⏵ pve-qemu-839b53b ⏱ 3s
┣━━━ Builds
┗━ ∑ ⏵ 1 │ ✔ 0 │ ⏸ 0 │ ⏱ 11s^C

So, I can say that the caching issue was not reproduced, though plenty of other issues abound with their modules, I had to fix a broken option before it would evaluate.

This is a very early and rough project and on a mildly related note, you can likely replace the need for this setup if you look into the platform called Incus, which is available in Nixpkgs. It offers many of the features that proxmox does like VMs, LXCs, clustering, etc with various storage drivers and is worth considering, if nothing else than to avoid nasty pitfalls here.

This was definitely outside my knowledge, thanks for pointing out a possible problem I wouldn’t have known about. (I understand this is probably not the cause, but still it’s interesting)

I checked their GitHub page but didn’t see this; thank you. (Note to self: GitHub search doesn’t work well—always check the issues directly if they aren’t in the thousands. nixpkgs is especially difficult since many issues contain output derivation and if I search issues for a package many non-related ones, like tree-wide updates, pop up)

Sorry I’m not sure what you mean here. It seems to me that your output shows compilation instead of downloading, so you are seeing the caching issue as well. Am I wrong?

Yes, it’s unfortunate. Yesterday, after doing a flake update, I got an option error as well, still with the simplest config:

random option error
error: A definition for option `services.openssh.settings.AcceptEnv' is not of type `null or (list of string)'.

Thank you for the suggestion, incus is more than fine, but I wanted to try to connecting two proxmox nodes since I already have another already on my network (not proxmox-nixos based).


Happy holidays btw, thank you for taking the time :slight_smile:

I’ll try to open a PR, this is the issue probably: