Disk usage for Nix store (speaking of bloat...)

Development
1

The earlier discussion of bloat, although IMO mostly going after the wrong targets, got me thinking: what do I consider “bloated” about NixOS as it is today? And the big thing that comes to mind is, the disk space requirements for the store.

Here’s current disk usage on a small server I run using NixOS. Note that / is ephemeral, that’s why it’s not visible here. The total size of the (virtual) disk is 24GB and 4G of it is used as swap.

# df -h | grep \^/dev
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda5       8.0G  3.5G  4.5G  45% /nix/store
/dev/vda6        12G  1.6G   11G  14% /home
/dev/vda3       704M  253M  452M  36% /var
/dev/vda2       255M  130M  125M  52% /boot

3.5G for the store is, in my opinion, too big. Now this is with a whole bunch of old generations from periodic system updates. After a nix-collect-garbage -d run, though, it only goes down to 2.0G, which is still too big. (In my opinion.)

So how big isn’t too big, in your opinion?

Obviously the space required is going to depend on what the system actually does.

This one is just a web server, serving a mix of static files and PHP scripts. There’s no database, no shell users besides root, and no other major services (however I was forced to install Postfix because nullmailer wasn’t quite capable enough).

I think a machine like this should be able to fit four or five generations of the system profile into one gigabyte. That means a single generation should require no more than about 200MB. Honestly, that still seems a bit on the large side to me, but I am an old fart from the days when a complete installation of Solaris 2.5 fit into less than fifty megs.

Taking 200MB per gen as a target, NixOS 25.11 requires a full order of magnitude more space for this installation. Why?

# cd /nix/store && du -sh * | sort -k1,1rh | head -n15
319M    kx2w0vzs6y6dj78azs329l372944f4vx-source
139M    746hvfrby87byzqlpj4jjwhdz89rsh27-linux-6.12.64-modules
104M    cdaifv92znxy5ai4sawricjl0p5b9sgf-python3-3.13.11
61M     j1scam1h5xmpnsn5ss02nbyhhyc7hwq3-perl-5.40.0
54M     yxk9smkrispxlz2ka3gxigvmzhf0fn65-systemd-258.2
47M     i6fivbql9nn479sq3s0wiihqjyzizzvm-vim-9.1.1869
44M     0ai16s30fy6br1iamqqyb5pxfckk5bkm-sops-install-secrets-0.0.1
43M     30i9xmq3i3kzvkl532c15xd38823zmhv-caddy-2.10.2
42M     2lalvc104lladjsl0iwbc8592w0p8c7j-git-minimal-2.51.2
39M     gf26b8zflmzi6vbhlfqsipykhkrsvagb-icu4c-76.1
38M     wr26k9b1ga5ln0w2ppmf6cn0sdnw4m0x-php-8.4.16
38M     y6j7ydwz23nj21xqj8x2z96ms6b4iqbg-icu4c-73.2
36M     30sxwq9fvi23blwxqpdahs83pw3hv1dw-initrd-linux-6.12.64
30M     wqfs0wh0wp6vdcbbck3wzk5v15qy17m7-glibc-2.40-66
30M     y7kk3dj7wlls924zlvpwmm13gskfw1hk-git-2.51.2

There’s probably tweaks I can make on my end to cut this down, e.g. replacing vim with nvi and making sure not to pull in both git and git-minimal; but the biggest pieces of the pie are going to require some work within NixOS and/or nixpkgs if we want them to be smaller.

319M    kx2w0vzs6y6dj78azs329l372944f4vx-source

This is the complete source code of nixpkgs. I think this is an example of what this article is getting at: Package managers keep using git as a database, it never works out | Andrew Nesbitt . There oughta be a way for me to pull only the small subset of nixpkgs that is actually relevant to the packages and NixOS modules I am using.

139M    746hvfrby87byzqlpj4jjwhdz89rsh27-linux-6.12.64-modules

I’m already tinkering with custom kernels but what do y’all think of there being a nixpkgs-supplied, minimal, monolithic kernel specifically for use as a VM guest? Almost all the drivers could be turned off.

104M    cdaifv92znxy5ai4sawricjl0p5b9sgf-python3-3.13.11

Python is needed by three system components I can’t reasonably get rid of: nixos-rebuild-ng, limine-install, and journalwatch. Almost all of its size (95MB) is lib/python3.13, i.e. the standard library. It should be possible to install this in a more compact form by leveraging zipimport; a zipfile containing all the .pyc files is 17M, and another one with all the .py files is 3.2M. I think it should also be possible to move the .py files to an output that most things don’t depend on.

61M     j1scam1h5xmpnsn5ss02nbyhhyc7hwq3-perl-5.40.0

I am not sure what needs this.

# nix why-depends /nix/store/flrcxy9gy89fnvf3h0w3nw62zq0l6zyq-nixos-system-tinka-25.11.20260110.d030887 /nix/store/j1scam1h5xmpnsn5ss02nbyhhyc7hwq3-perl-5.40.0/
/nix/store/flrcxy9gy89fnvf3h0w3nw62zq0l6zyq-nixos-system-tinka-25.11.20260110.d030887
└───/nix/store/swf3hf4h0gdlv2zgxyb79jkn10i2g2py-perl-5.40.0-env
    └───/nix/store/j1scam1h5xmpnsn5ss02nbyhhyc7hwq3-perl-5.40.0

Adding --all shows me additional paths through nixos-generate-config and git, which I think I can get rid of, but I do not know why there’s a direct dependency of the system configuration on perl-5.40.0-env.

44M     0ai16s30fy6br1iamqqyb5pxfckk5bkm-sops-install-secrets-0.0.1
43M     30i9xmq3i3kzvkl532c15xd38823zmhv-caddy-2.10.2

These are both programs written in Go, and almost all of the bulk of each package is the binary executable that the program compiles to:

-r-xr-xr-x 2 root root 44M Jan  1  1970 <hash>-sops-install-secrets-0.0.1/bin/sops-install-secrets
-r-xr-xr-x 2 root root 43M Jan  1  1970 <hash>-caddy-2.10.2/bin/caddy

sops-nix is not part of nixpkgs and I’ve reported the issue directly to the package maintainer (The `sops-install-secrets` binary is absurdly large · Issue #892 · Mic92/sops-nix · GitHub); however, caddy is part of nixpkgs, and it appears to me that the root cause of both binaries’ size is that the Go compiler is doing something silly. I don’t know enough about Go to dig into it further but I think someone should.

39M     gf26b8zflmzi6vbhlfqsipykhkrsvagb-icu4c-76.1
38M     y6j7ydwz23nj21xqj8x2z96ms6b4iqbg-icu4c-73.2

Classic case of “DLL hell”: xfsprogs-6.17.0 was built against icu4c-76.1 and php-intl-8.4.16 was built against icu4c-73.2. There should be an item on the checklist for each NixOS release for someone to go through the dependency tree of all the packages and make sure everything is built with the same version of each major library.

6 Likes
2

So how big is the closure? What is in it? And what of it do you not need?

3

Nixpkgs should be much smaller, around 40 MiB, Iirc.

Anyway, you are using flakes, disable pinning of your system input in the flake registry, many disable the registry completely.

2 Likes
4

I mean, those are reasonable suggestions for a good 300MB from just analyzing “low hanging” fruit. I don’t think this is an unreasonable rant; it’d be nice if upstream had better processes for ensuring system closure size isn’t unnecessarily large.

But yeah, tweaking your config to slim down the closure is definitely in practice what you need to do today.

2 Likes
5

So how big is the closure? What is in it? And what of it do you not need?

I intended the second half of my post (everything after “NixOS 25.11 requires a full order of magnitude more space …”) to address all of the above questions. What aspects of them are not fully answered by that text?

Sure doesn’t look like it to me…

[root@tinka:/nix/store/kx2w0vzs6y6dj78azs329l372944f4vx-source]# du -sh * | sort -k1,1rh
274M	pkgs
29M	nixos
15M	programs.sqlite
2.0M	lib
1.9M	doc
1.3M	maintainers
356K	ci
48K 	CONTRIBUTING.md
12K 	flake.nix
12K 	modules
8.0K	README.md
4.0K	COPYING
4.0K	default.nix
4.0K	shell.nix
4.0K	svn-revision
0   	nixpkgs

I have never heard of this “flake registry” before. How exactly do I do that? Also, even if I do do that, isn’t it going to need to pull the entire of nixpkgs into the store at least temporarily every time that I rebuild the system derivation? Keep in mind that my starting point for this discussion is that I want to be able to have four system generations fit in a partition no bigger than 1GB (assuming no other GC roots), and anything that’s required solely as a build dependency is dead weight cutting into that limit.

1 Like
6

I am in fact working on that right now (although I might pack it in for a while and go work on the thing I meant to work on today) but I made the post specifically because I don’t think I can cut things down as far as I want just by tweaking the closure. I actually require both xfsprogs and php-intl, for example, and thus I’m stuck with two copies of ICU.

7

That second half only says whats in your store, but doesn’t answer whats actually in your system closure. If its not in your system closure you have to find why else it doesn’t get collected.

You are right, the 40 MiB I had in mind are the tarballed archive, not the unpacked sources.

Sorry for that.

Disable the local pinning of nixpkgs is done via nixpkgs.flake.setFlakeRegistry = false, though whenever you use nixpkgs on the terminal it will pull the newest nixos-unstable from GH, if the last pull is more than an hour ago.

I do not remember though how to disable (actually empty) the registry though. But that would mean that using nixpkgs on the CLI as a flakeref would not work anymore at all.

8

Oh, I see. The only GC root on this system is the system profile, and I generated that listing immediately after a reboot and running nix-collect-garbage -d as root, so I believe that means everything in the store is in the current generation of the system closure.

(You are correct that the only thing pinning the nixpkgs source is the flake registry, but to be crystal clear, I won’t consider the space consumption of the nixpkgs source to be solved until something is in place that means the complete nixpkgs source never has to be downloaded at all.)

Is there a way to get a tree dump of the closure of some GC root? Kinda the inverse of nix why-depends. I’m not finding anything that looks like it does that in the Nix manual.

9

Reboot then run it again with sudo.

10

Already did that. See edits.

(This server has only the root account and role accounts, no normal users; and, I reiterate, the only GC root is the system profile. None of the users, not even root, has a profile.)

11

nix path-info -rsS /run/current-system gives some insights, it doesn’t give you a nice tree though.

For the tree nix-tree or so exists, and nix-du as well.

12

These reports are too long for me to post inline here, so I’ve uploaded them to, amusingly, the very same server we’re talking about:

“Everything that’s in /nix/store that’s not in the system closure” is mostly .drv files and I think all the rest of it is stuff used during builds. It’s a lot of tiny files and it does add up to 22.3M, which is enough to be a concern on the scale I want to chop things down to, but then again I’m not sure du is telling me the truth about each of these files eating 4K of disk space. It might not understand XFS’s small-file optimizations.

13 
[root@tinka:/nix/store/kx2w0vzs6y6dj78azs329l372944f4vx-source]# du -sh * | sort -k1,1rh
274M	pkgs
29M	nixos
15M	programs.sqlite
2.0M	lib
...

What’s this programs.sqlite file? :thinking:

I know this isn’t part of the original discussion, but I am very curious about what this contains, where it comes from / how it is being generated

14

I don’t know what it’s for, but I believe you get that if you use nixpkgs.url = "https://channels.nixos.org/nixos-YY.MM/nixexprs.tar.xz"; instead of nixpkgs.url = "github:nixos/nixpkgs/nixos-YY.MM", see discussion here: Questions about `inputs.nixpkgs`.

I suppose I might as well also post a link to my servers’ configuration git tree: https://git.sr.ht/~zackw/server-configs/tree/nix-and-guix/item/nix The machine we’re talking about is “tinka”.

Incidentally, the only thing I’m using flakes for is to pull in sops-nix and disko. Experimenting with npins or niv instead is on The List but it’s fairly low priority.

15

One way to save a lot of space on your deployment machines is not to use them as build machines too. Run nixos-rebuild on your build machine, let it eat the cost of having Nixpkgs in its store, and ship the closures off to the deployment machine using --target-host.

There’s a hidden setting system.disableInstallerTools (hidden because if you aren’t doing this build/deployment separation, it can get you in trouble!) that will keep nixos-rebuild and friends from being installed on the target, saving you a few more megs.

7 Likes
16

This looks like a channel copy, including the CNF database.

I am not aware of a way to get rid of that on a system that builds itself via channels. On a system that uses flakes exclusively, its of course safe to nix-channel --remove all users channels.

17

If you never want to download nixpgks at all, then you will not be able to nixos-rebuild on the VM, you won’t be able to nix run nixpkgs#random-tool.

I second @Sigmanificient that you should be building your system closure somewhere else, and pushing the result to the target machine. This is what i do with my home server, and it works very well. You can also use tools like colmena to handle the deployment for you.

There are some things you can try to shrink the generation size, some of which are outlined here:

In general, I agree NixOS is getting a bit bloaty, especially if using flakes where it’s all too easy to depend on multiple versions of nixpkgs and install a bunch of duplicate packages of the same version… Also, have you seen the size of linux-firmware these days?!?

Unfortunately it would be a lot of work to dig into some of the big offenders and split them up/strip them down, there’s always risk of breakage from someone relying on one of the files you stripped, and storage is just so CHEAP; it’s a thankless job that nobody’s keen to do.

18

It’s a minor thing, but we’ve also had success with ZFS and page dedup helping with chonky stores. Not useful for everybody, but can be quite handy if you’re in that problem domain.

19

It’s used by command-not-found to find which package contains an executable.

20

Have a look at the stuff in the perlless and bashless profiles for more chances of minification

1 Like