Hi folks,
This morning after 3 days I updated 23.05 (from 23.05.20230726.f3fbbc3 to 23.05.20230728.48e82fe) and I was surprised with the size of the update. I downloaded over 10 GiB and the new generation uses almost 10 GiB on disk.
I looked at Github commits on 23.05 branch and I didn’t see anything unusual in last few days.
Is it normal to have such large updates?
My setup is (I think) common - NVidia, ZFS, KDE Plasma and bunch of sw (the largest is Libreoffice).
The commit range includes a merge of staging-next-23.05 which is used to prepare large rebuild changes (like changes to stdenv). It contained an update of gnutar which is contained in stdenv which is used to build practically all packages in nixpkgs, consequently, all of nixpkgs is rebuilt which reflects itself for you in changed store paths you need to download. 10GB would then be the download size required to substitute your system from scratch which seems to check out.
If the large size on disk is an issue but you don’t want to delete the old system generations, you can run nix-store --optimise, nix store optimise or set the auto-optimise-store option.
These will deduplicate identical files in the store by replacing the multiple files with hardlinks to a single inode. As the nix store is read-only, this is a safe and transparent operation. Especially with changes like these, where the inputs change but the outputs probably don’t, optimise can save a lot of storage space.
You can also make the store content-addressed, which resolves issues like this automatically, though this is still very much an experimental setting.
Thank you. That makes sense. This experience also made me to realize what would be the minimum size for NixOS partition. Assuming that I’m using a small partition or a small VM in cloud and assuming that I’m OK with just two generations and assuming that I currently have only one generation and no data, I need minimum free space as the current amount of used space.
I’ve never used nix store --optimise so far. I reclaim space by unlink /nix/var/nix/profiles/system-#-link (not always the oldest one) following with nix store gc. Could you please tell me do I also have to use nix store --optimise?
Thank you.
No, if you run garbage collection after deleting your old profiles, optimising the store is not really necessary. You can still do it, and it will save a bit of space, but it doesn’t make that much of a difference if you don’t have a lot of duplicate files in your store anyway.
It saves a lot of space (multiple GiB) on my desktop. I think HM heavy use brings a lot of duplication.
yes, it’s doing something entirely different. It scans all files in the nix store, and replace every duplicated files by a single copy in /nix/store/.links and use hard links to that files to put into the derivations.
There is no downside doing that, except it takes a bit of CPU and I/O when running. This is 100% safe.
Do you happen to use different versions of nixpkgs with home-manager and NixOS? It’s easy to do accidentally, and will in fact duplicate most of (if not all) your packages.
I have two imports in flake.nix: x = import nixpkgs { system = ...; config = ...; } for stable and unstable nixpkgs and I pass the x everywhere - including HM as NixOS module.
In HM (not in system configuration) I combine stable / unstable a lot and I think a lot of software differ in a small number of files and most of assets are the same across different versions of that software.
Does it answer your question?
When you use nixpkgs-unstable and stable, you will duplicate even the “assets” that match, unless you use nix store --optimise. Even if you optimise your store, often very few assets will match, because they use different versions of gcc and some other core compilation tools, which will cause subtle differences in the outputs, and therefore create different files.
If you want to use less disk space, decide on one version of nixpkgs and stick with it. Or use both, and sacrifice a bit of disk space for convenience.
Pretty much; I don’t think just using home-manager should result in much additional disk usage that you wouldn’t have if you installed all your packages with environment.systemPackage. The delta might be a few MB of scripts. Your symptoms suggest use of stable/unstable on the same system (which is fine and common).
Yes, absolutely. I should clarify that HM doesn’t duplicate files but my use of HM does.
Absolutely. I remember hard links on UFS (Solaris) and reading about linking in 90s so I don’t want to guess how old the concept of inodes and links is…
Btw just example how amazing the Nix / NixOS project is:
ddh -b M -d /nix
1175329 Total files (with duplicates): 61502 Megabytes
372347 Total files (without duplicates): 27044 Megabytes
3734 Single instance files: 39 Megabytes
368613 Shared instance files: 27005 Megabytes (1171595 instances)
ddh -b M -d .local/share/flatpak/
101788 Total files (with duplicates): 10567 Megabytes
39908 Total files (without duplicates): 5628 Megabytes
2973 Single instance files: 1889 Megabytes
36935 Shared instance files: 3739 Megabytes (98815 instances)
In other words I could save almost 2g running fdupes against flatpak directory.
Your comparison is not fair. Yes, Nix handle deduplication with hard links alone, but you shown it can be done “manually” without using nix.
But Flatpak programs update by downloading the changed files compared to the installed version, where Nix downloads the entire package and all the kitchen sink everytime an input change (it’s by design). 
But Nix has the option to dedup, Flatpak doesn’t and I heard from many people that the only thing they don’t like on Flatpak is how much space the apps use.
I’m criticizing Flatpak because I like it. Nix project is more community driven, and therefore less paper cuts.
BTW if you want to figure out what is using a lot of space in your system, you can use nix-tree.
nix-shell -p nix-tree.out --run nix-tree