No space left on nix store

Help
1

Hi all, I’ve been putting this off for far too long. I’m a very happy NixOS user for more than two years, except I always had some recurring problems with lack of space on /nix/store. So far, I have been able to fix it with garbage collection, optimization, removing some symlinks, etc. But now it got to the point that I’m almost paralyzed, and can’t create new dev envs or upgrade my system flake.

The weird thing is – as far as I can tell – I have plenty of space left on my /nix/store and also plenty of free inodes. I try to dump some helpful info below, I’d be happy if someone could help me solve this puzzle. Let me know if you need other outputs.

❯ df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 762M     0  762M   0% /dev
tmpfs                    7,5G   40K  7,5G   1% /dev/shm
tmpfs                    3,8G  8,3M  3,8G   1% /run
/dev/dm-1                 99G   35G   59G  37% /
efivarfs                 128K   39K   85K  32% /sys/firmware/efi/efivars
tmpfs                    1,0M     0  1,0M   0% /run/credentials/systemd-journald.service
tmpfs                    1,0M     0  1,0M   0% /run/credentials/systemd-sysctl.service
tmpfs                    1,0M     0  1,0M   0% /run/credentials/systemd-tmpfiles-setup-dev-early.service
tmpfs                    1,0M     0  1,0M   0% /run/credentials/systemd-tmpfiles-setup-dev.service
tmpfs                    7,5G  1,3M  7,5G   1% /run/wrappers
/dev/mapper/laptop-home  824G  532G  250G  69% /home
/dev/nvme0n1p1           511M   23M  489M   5% /boot
tmpfs                    1,0M     0  1,0M   0% /run/credentials/systemd-tmpfiles-setup.service
tmpfs                    1,0M     0  1,0M   0% /run/credentials/systemd-vconsole-setup.service
tmpfs                    1,0M     0  1,0M   0% /run/credentials/getty@tty1.service
tmpfs                    4,0G   92K  4,0G   1% /run/user/1000

❯ df -ih
Filesystem              Inodes IUsed IFree IUse% Mounted on
devtmpfs                  1,9M   607  1,9M    1% /dev
tmpfs                     1,9M     5  1,9M    1% /dev/shm
tmpfs                     1,9M  2,3K  1,9M    1% /run
/dev/dm-1                 6,3M  1,2M  5,1M   20% /
efivarfs                     0     0     0     - /sys/firmware/efi/efivars
tmpfs                     1,0K     1  1023    1% /run/credentials/systemd-journald.service
tmpfs                     1,0K     1  1023    1% /run/credentials/systemd-sysctl.service
tmpfs                     1,0K     1  1023    1% /run/credentials/systemd-tmpfiles-setup-dev-early.service
tmpfs                     1,0K     1  1023    1% /run/credentials/systemd-tmpfiles-setup-dev.service
tmpfs                     1,9M    23  1,9M    1% /run/wrappers
/dev/mapper/laptop-home    53M  1,6M   51M    3% /home
/dev/nvme0n1p1               0     0     0     - /boot
tmpfs                     1,0K     1  1023    1% /run/credentials/systemd-tmpfiles-setup.service
tmpfs                     1,0K     1  1023    1% /run/credentials/systemd-vconsole-setup.service
tmpfs                     1,0K     1  1023    1% /run/credentials/getty@tty1.service
tmpfs                     381K   135  381K    1% /run/user/1000

❯ cat /proc/self/mountinfo
22 30 0:5 / /dev rw,nosuid shared:9 - devtmpfs devtmpfs rw,size=779884k,nr_inodes=1947410,mode=755
23 22 0:21 / /dev/pts rw,nosuid,noexec,relatime shared:10 - devpts devpts rw,gid=3,mode=620,ptmxmode=666
24 22 0:22 / /dev/shm rw,nosuid,nodev shared:11 - tmpfs tmpfs rw
25 30 0:23 / /proc rw,nosuid,nodev,noexec,relatime shared:2 - proc proc rw
26 30 0:24 / /run rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=3899412k,mode=755
27 26 0:25 / /run/keys rw,nosuid,nodev,relatime shared:13 - ramfs ramfs rw,mode=750
28 30 0:26 / /sys rw,nosuid,nodev,noexec,relatime shared:3 - sysfs sysfs rw
30 1 254:1 / / rw,relatime shared:1 - ext4 /dev/disk/by-uuid/98c8e811-7965-443d-ab10-72d39d63376c rw,stripe=128
31 30 254:1 /nix/store /nix/store ro,relatime shared:14 - ext4 /dev/disk/by-uuid/98c8e811-7965-443d-ab10-72d39d63376c rw,stripe=128
32 28 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:4 - securityfs securityfs rw
33 28 0:28 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:5 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
34 28 0:29 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:6 - pstore pstore rw
35 28 0:30 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:7 - efivarfs efivarfs rw
36 28 0:31 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:8 - bpf bpf rw,mode=700
37 22 0:18 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:15 - mqueue mqueue rw
38 22 0:32 / /dev/hugepages rw,nosuid,nodev,relatime shared:16 - hugetlbfs hugetlbfs rw,pagesize=2M
39 28 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:17 - debugfs debugfs rw
59 26 0:33 / /run/credentials/systemd-journald.service ro,nosuid,nodev,noexec,relatime,nosymfollow shared:18 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,noswap
41 28 0:35 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:19 - configfs configfs rw
40 28 0:34 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:20 - fusectl fusectl rw
65 26 0:36 / /run/credentials/systemd-sysctl.service ro,nosuid,nodev,noexec,relatime,nosymfollow shared:21 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,noswap
67 26 0:37 / /run/credentials/systemd-tmpfiles-setup-dev-early.service ro,nosuid,nodev,noexec,relatime,nosymfollow shared:22 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,noswap
69 26 0:38 / /run/credentials/systemd-tmpfiles-setup-dev.service ro,nosuid,nodev,noexec,relatime,nosymfollow shared:23 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,noswap
43 26 0:40 / /run/wrappers rw,nodev,relatime shared:48 - tmpfs tmpfs rw,mode=755
45 30 254:2 / /home rw,relatime shared:50 - ext4 /dev/mapper/laptop-home rw,stripe=32
47 30 259:1 / /boot rw,relatime shared:52 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
107 26 0:41 / /run/credentials/systemd-tmpfiles-setup.service ro,nosuid,nodev,noexec,relatime,nosymfollow shared:54 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,noswap
216 26 0:39 / /run/credentials/systemd-vconsole-setup.service ro,nosuid,nodev,noexec,relatime,nosymfollow shared:256 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,noswap
817 26 0:69 / /run/credentials/getty@tty1.service ro,nosuid,nodev,noexec,relatime,nosymfollow shared:695 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,noswap
641 33 0:79 / /sys/fs/cgroup/net_cls rw,relatime shared:878 - cgroup net_cls rw,net_cls
813 26 0:80 / /run/user/1000 rw,nosuid,nodev,relatime shared:910 - tmpfs tmpfs rw,size=4194304k,nr_inodes=389941,mode=700,uid=1000,gid=100
955 813 0:81 / /run/user/1000/gvfs rw,nosuid,nodev,relatime shared:936 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=100
316 813 0:68 / /run/user/1000/doc rw,nosuid,nodev,relatime shared:667 - fuse.portal portal rw,user_id=1000,group_id=100

Here’s when I try to create a flake env (doesn’t matter which one, the error is the same, just with a different store path. The error is almost instantaneous, doesn’t even start building.

❯ nix shell ./flake_env#
warning: Git tree '/home/kupac/prg/R/hydra_API' is dirty
error:
       … while evaluating 'strict' to select 'drvPath' on it
         at /builtin/derivation.nix:1:552:
       … while calling the 'derivationStrict' builtin
         at /builtin/derivation.nix:1:208:
       (stack trace truncated; use '--show-trace' to show the full trace)

       error: opening file '/nix/store/nd9sj4i068qfrrxdmnv3kj383yq814jf-python3.12-pluggy-1.5.0.drv': No space left on device
2

Does running nix shell nixpkgs#hello produce same error?

Do you perhaps have some path option set not to a string but rather to a real directory, which gets copied into the store when evaluating?

3

Nope, that works.Also nix-shell -p somethingsomething usually works OK. I can do small stuff

4

Check the second guess from my post and/or post your config somewhere

5

Ah, I missed the 2nd part. No, as far as I can tell, all the paths in the flake are strings. I’ll post the config and the flake tomorrow. Thanks for looking into it.

6

I’m not sure it’s a “copying something large to the store” flake eval problem, if it happens immediately, although of course it still could be. It seems like some other error “borrowing” ENOSPC as a return.

1 Like
7

If I were you, I would attempt to reproduce with Lix, CppNix 2.18, and the latest CppNix version (pkgs.nixVersions.latest). Maybe it’s a bug with a specific Nix version.

environment.systemPackages = [ 
  (pkgs.nixos-rebuild.override { nix = pkgs.lix; # or any other nix version })
];

This should give you a version of nixos-rebuild with a different nix version for debugging.

8

Is it possible to strace it? Might show where the ENOSPC error is coming from.

Only other thing I can think of is if it’s trying to build anything it will want to write to $TMPDIR. I would expect that to be /tmp, and your df output doesn’t show anything mounted there, so unlikely to be that, but just in case…

9

That’s a brilliant idea. So much so, in fact, that I already tried it, and I’m already using lix. Unfortunately, the problem remained :frowning:

3 Likes
10

I tried, but I’m not sure what to make of it. strace nix build flake_env - Pastebin.com

11

Ah, you might need -f or similar to follow the forks (clones.) Also I guess it’s probably going to call into the daemon on a multi-user install, so you might need to attach to that with -p …

12

It does look like it’s not a nix store space issue.
I used nix store add-file to copy a ~1GB file without a problem. I also added about 5000 very small files to the nix store. It seems to be something with the evaluation.

I’ll add this particular flake somewhere. But actually, it happens with various builds and my system flake too. So I don’t think that they all have an issue that causes this.

13

OK, here’s some more confusing info. I thought: since the error comes from python3.12-pluggy-1.5.0.drv, let’s see if I can build that package! The answer is yes and no.
nix build nixpkgs#python312Packages.pluggy succeeds. But if I use the nixpkgs version from the flake_env/flake.lock, then it fails with the same error!

❯ nix build nixpkgs/ab82a9612aa45284d4adf69ee81871a389669a9e#python312Packages.pluggy
error:
       … while evaluating the attribute 'drvPath'
         at /nix/store/1d08aqinlp4zw8s0i9n71idfgiam8a5i-source/lib/customisation.nix:365:7:
          364|     in commonAttrs // {
          365|       drvPath = assert condition; drv.drvPath;
             |       ^
          366|       outPath = assert condition; drv.outPath;

       … while evaluating the attribute 'drvPath'
         at /nix/store/1d08aqinlp4zw8s0i9n71idfgiam8a5i-source/lib/customisation.nix:365:7:
          364|     in commonAttrs // {
          365|       drvPath = assert condition; drv.drvPath;
             |       ^
          366|       outPath = assert condition; drv.outPath;

       … while evaluating 'strict' to select 'drvPath' on it
         at /builtin/derivation.nix:1:552:
       (stack trace truncated; use '--show-trace' to show the full trace)

       error: opening file '/nix/store/nd9sj4i068qfrrxdmnv3kj383yq814jf-python3.12-pluggy-1.5.0.drv': No space left on device

Then something must be wrong with that nixpkgs version, right? Let’s just update the flake, and build the env. Not so fast! Now it’s the same error with a different drv: error: opening file '/nix/store/30vacm5q6h8jkap5v6526yv4nrn6fjfa-ragged2e.r67441.tar.xz.drv': No space left on device

14

I uploaded the flake here. Perhaps someone could try to build it on their own machine, to see if it works.

15 
 path = ./.;

hello, look what’s hiding in templates

16

Here’s what to use instead:

https://nix.dev/tutorials/working-with-local-files

But I don’t see the issue here, ./. is a path within the flake, it’s not going to do a massive copy.

Rather if they ran a flake based-command where the flake URL’s referring to CWD, and they happened to be in a large dir, then that’d be a massive copy. (e.g. nix flake show when run in $HOME would copy $HOME to the store, if it contained a flake.nix)

1 Like
17

My memory is sketchy since I last made this mistake quite some time ago, but I think this will follow symlinks, and I had symlinks to half the store under .direnv/ thanks to nix-direnv. Edit: I could well be conflating other issues, because it shouldn’t copy anything not added to git either.

Anyway, it stood out as a bare path when the OP said there were none, to be investigated. As noted earlier, a big copy would take time that doesn’t seem to be passing here, so I still suspect something else.

1 Like
18

:man_facepalming: ok, I don’t think it’s the problem, but will fix it, as it’s good practice anyways. I do use direnv, actually, so who knows.

1 Like
19

Nah, ./. did’t fix it. I removed the whole template part, btu didn’t help. Not so surprised, because it is not specific to this flake, but happens to more things. I’m wondering if something is broken in my /nix/store. It’s definitely machine specific, because I asked someone to build it, and it worked without an issue.

20

Maybe worth trying to monitor df’s output while you build, maybe it’ll tell you if a filesystem is getting full, and which.

I remember having to increase tmpfs size at some point, but it was a long time ago and I can’t really connect that to your symptom.