Coming from FreeBSD, I imagine the following server setup:
ZFS root
Jails (ezjail) for various services (web, mail etc.) for management and security reasons
Each jail on its own encrypted ZFS fs
Bootstrap unencrypted up to ssh, then manual code entry to bring up services
The management advantages of jails more or less come for free with Nix, I guess (flakes), but how about isolation for security? I looked at nix-containers and LXC but still not sure how to move forward. Would be nice to have all the active service config files on the encrypted filesystems as well to not expose them unnecessarily.
What threat model are you trying to defend against by having all services running on different encrypted volumes? Assuming you want multiple services running at the same time they would all be mounted unencrypted at runtime, so it wouldn’t necessarily prevent side channel access across services.
Disk encryption is typically more for preventing attacks from those with physical access to a machine. As for isolating services from each other, I think virtual machines are still the most secure isolation mechanism in reality. There has been a lot of chatter in the past warning that containerization really isn’t a bulletproof security barrier, mostly because the attack surface of a single Linux kernel is just so utterly massive.
Threat 1: Some services might be on “standby”, i.e. key not present on the system. In case of breach of another service, the standby services would be safe.
Threat 2: As you said, physical access to the machine.
NixOS containers are just environments with unshared Linux namespaces, managed through systemd-nspawn. Unlike FreeBSD jails, Linux namespaces were not intended as and are not really a security mechanism. This is what the man page used to say:
Note that even though these security precautions are taken systemd-nspawn is not suitable for secure container setups. Many of the security features may be circumvented and are hence primarily useful to avoid accidental changes to the host system from the container. The intended use of this program is debugging and testing as well as building of packages, distributions and software involved with boot and systems management.