Yes, that would work. Pick a location on the read-write volume, such as /var/root/mount_nix_password. Security of the file doesn’t really matter since the Nix volume will be mounted always, but feel free to lock it down as much as you want, just make it executable. At this location, write a shell script that runs the appropriate diskutil ap unlock command to mount the volume, hardcoding the passphrase. This would look something like /usr/sbin/diskutil ap unlock DEVICE -passphrase PASSPHRASE -mountpoint /nix, except replace DEVICE with the Nix volume’s device identifier (e.g. mine is disk2s5) and replace PASSPHRASE with the passphrase you used to encrypt the volume. Note: please test this command, I haven’t actually executed it on my machine.
Once you have this script and it works, run sudo defaults write com.apple.loginwindow LoginHook /var/root/mount_nix_password. This will set up that script as the login hook. Note that if you already have a login hook this will replace it, though I doubt you do.
CAVEAT: The login hook documentation says it’s run for every user. Please verify that the script does not error out if the volume is already mounted. I don’t actually know what diskutil ap unlock does if the volume is mounted. If this command cannot be run with a mounted volume, then set up whatever appropriate probe you want first.
CAVEAT 2: Since this is a login hook, it runs when a user logs in. This means the volume will not be available prior to login. I’m documenting this approach because it was claimed to work in the past.
An alternative approach that I don’t have proof that works is to do basically the same thing, except instead of setting up a login hook, set up a /Library/LaunchDaemons launchd plist to run the script at boot. This should actually be better than the login hook approach, but I don’t have proof it works. This approach also does not require ensuring that the script works with the volume already mounted, because it will only be run once per boot.