Hey there. I’ve wanted to try out NixOS for years, and this is the first time I decided to give it a shot.
Up until now all I’ve been doing is attempt to replicate everything about my current Arch Linux setup into the configuration.nix. But now I just thought about something.
I will, of course, want to replicate the daily automatic snapshots I currently have on Arch. I just take snapshots of the root and home subvolumes every day and keep them around for a week using cronjobs and a program named btrbk.
But, NixOS has, from what I understand, this special /nix folder where it stores every ‘version’ of the system after modifications done to the configuration.nix are applied to the system. Well, that means something very similar to my daily Btrfs snapshots is happening anyways just because of the package manager. And, now, I was thinking, could the Nix store just be configured as a set of Btrfs snapshots? This sounds much cheaper in terms of storage space, considering that Btrfs snapshots only store the initial version of a filesystem tree and then modifications done to it.
So, is this possible? It’d be great for system integration and for consuming less storage space. To me, it sounds like Nix does none of this by default/nothing similar in the way that it stores the Nix store and it just keeps the entirety of all the root filesystem versions.
What should I do?
You’re misunderstanding how Nix stores every version of your system. It doesn’t keep a whole copy of the “root filesystem” for every nixos-rebuild. Every package has its own path with a unique hash included in its name derived from every detail of how it was built, including those hashes from its dependencies. So any given system generation will refer to the packages that it depends on, and the hashes of those packages will determine the hash used in the name of the system generation’s path in /nix/store. For the most part, these references are made via symlinks, so the system generation’s path is extremely small.
When you make a change, say by adding a new package to systemPackages, all the old packages are reused, because they’re still the exact same ones. Only the very small hierarchy of symlinks has to be changed to now also refer to that new package, and the hash in the system generation’s path name will be a new one to reflect that.
Point being, Nix does share everything that’s in common between generations. Btrfs snapshots would create exactly zero benefit to this storage model.
2 Likes
Alright then. So what I understand from this is that I should not create snapshots that contain /nix. So, at the very least, when snapshotting / I should create a separate subvolume for /nix so that it doesn’t get included in a / snapshot.
Then, if I have a separate /home subvolume, is there any point of even taking snapshots of /? From what I understand now, and just quickly browsing the filesystem, everything in / is just a bunch of symlinks and environment variables to get the $PATH right (only thing I can get from the fact that, at a quick glance, the only binary in /, apart from /bin/sh is /usr/bin/env).
Would things break/be hard to fix if everything in / would disappear/get corrupted/whatever while /nix is perfectly fine in a separate subvolume? Just trying to judge if taking snapshots of / like I was in a traditional Linux filesystem brings any value at all
Not only that, there’s an entire blog post about purposefully erasing everything except /nix/store, and there’s a code base for assisting in doing that. The entire operating system configures the root fs as required by the booted generation at boot time by quickly placing some symlinks, so the root fs can be entirely empty and it will work just fine as long as the /nix/store fs is intact.
The reason to keep the root fs intact is because of stateful stuff like /var/lib or /var/log, which just represents data that’s created by services/applications while you use the system normally.
As for whether you should snapshot /nix, there’s not much reason to do it or to not do it. In theory, if you ever need to recover your system, all the necessary stuff in /nix should be regenerated by configuration.nix. But it also doesn’t really consume any extra space to snapshot it, except for the nix paths you’ve deleted since the last snapshot (e.g. by sudo nix-collect-garbage -d)
2 Likes
Alright then. I think that, for now, I’ll have / and /nix in separate subvolumes and snapshot neither. I have a separate subvolume for /var/{cache,log,tmp}/ anyways since I didn’t want logfiles, cache and temporary files to be restored by snapshots anyways
My (offsite) backup still includes /etc because of /etc/nixos/, don’t forget to back that up some way. 
Also big recommendation: enabling compression and running duperemove after each system update (though duperemove doesn’t give back that much in my experience. a few percent maybe).
1 Like
Right now I’m only testing in a VM, so /etc/nixos is a complication of sshfs and symlinks from my host. It’ll be a Git repo when/if I ever deploy it on bare metal