How to move nix store to external drive on macos?

Swordlash · June 7, 2022, 10:01pm

Hi, I have nix installed on my MacOS Monterey. Sadly I’m running out of space on the laptop and I’d like to make nix to use the external drive as its store (i.e. by symlinking). How to do it properly? I guess I’d need to copy the /nix catalogue contents into the drive, get rid of the Nix Store volume and then symlink the drive to /nix. Am I on a good track here?

abathur · June 8, 2022, 12:39am

While I haven’t tried it myself, IIUC a symlinked store can cause problems with some software. But the store volume doesn’t have to live on the same drive as /.

If starting over isn’t prohibitive, it’s probably easiest to follow the uninstall instructions (Installing a Binary Distribution - Nix Reference Manual) and then set the appropriate variables to specify a different disk before installing:

github.com

NixOS/nix/blob/b2dea231cf61d8d10cef89991571f57976b77f06/scripts/create-darwin-volume.sh#L64-L71


      
          # make it easy to play w/ 'Case-sensitive APFS'
          readonly NIX_VOLUME_FS="${NIX_VOLUME_FS:-APFS}"
          readonly NIX_VOLUME_LABEL="${NIX_VOLUME_LABEL:-Nix Store}"
          # Strongly assuming we'll make a volume on the device / is on
          # But you can override NIX_VOLUME_USE_DISK to create it on some other device
          readonly NIX_VOLUME_USE_DISK="${NIX_VOLUME_USE_DISK:-$(root_disk_identifier)}"
          NIX_VOLUME_USE_SPECIAL="${NIX_VOLUME_USE_SPECIAL:-}"
          NIX_VOLUME_USE_UUID="${NIX_VOLUME_USE_UUID:-}"

If starting over is prohibitive, it’ll be a little more DIY–but there’s inevitably some way to duplicate the existing volume to a new disk and then update /etc/fstab and /Library/LaunchDaemons/org.nixos.darwin-store.plist to specify the new volume’s UUID (fair warning: this will be more complicated if you’re using FileVault).

Swordlash · June 16, 2022, 11:04am

I’ll describe what I’ve done in case someone needs to do it on their own. The following steps seem to be working:

Install nix in a conventional way. This will create /nix entry in /etc/synthetic.conf and /nix entry in /etc/fstab.
Connect external drive. I formatted it to encrypted, case-sensitive APFS. Save the disk’s UUID (it’s shown in Disk Utility info and in diskutil apfs list). rsync the contents of /nix to this drive.
Delete the /nix disk created by the installer with sudo diskutil apfs deleteVolume /nix.
Using sudo vifs, edit /etc/fstab to point /nix to your disk’s UUID. For me,

mateusz@Mateuszs-MacBook-Pro ~ % cat /etc/fstab 
UUID=EBC0B7AA-1BD5-4F56-A677-6BDD47AC95F5 /nix apfs rw,noauto,nobrowse,suid,owners

This way, your disk will be automatically mounted to /nix when connected.

Reboot.
Connect the disk, if it’s not connected. With noauto it shouldn’t mount automatically; open Disk Utility and mount. This should automatically mount it to /nix.
Voila! Open the terminal and see whether the shortcuts to nix are sourced properly.

After first reboot, I had an issue with OS telling me that some nix software (the daemon I think) comes from an unknown source and won’t be launched. I opened security settings and allowed it from there.

abathur · June 16, 2022, 3:44pm

This is why I said to update /Library/LaunchDaemons/org.nixos.darwin-store.plist to specify the new volume’s UUID.

I’ll also reiterate that this will be more complicated for anyone trying to follow along with FileVault enabled. In that case, the installer creates an encrypted volume and stores the passphrase in your system keychain (using the volume UUID) and generates a slightly different mounting service to request this passphrase and use it to decrypt the volume.

The external disk/volume may or may not be encrypted (i.e., it is up to you if you are manually generating the volume)–and you’ll need to square these parts if you adapt an existing install. If you don’t encrypt the volume, you can use the below as a reference for how to replace the command in /Library/LaunchDaemons/org.nixos.darwin-store.plist:

github.com

NixOS/nix/blob/7e301fd74e929af916116a04d84e130f9e0112de/scripts/create-darwin-volume.sh#L287-L291


      
          case "$cmd_type" in
              encrypted)
                  cmd=(/bin/sh -c "/usr/bin/security find-generic-password -s '$volume_uuid' -w | /usr/sbin/diskutil apfs unlockVolume '$volume_uuid' -mountpoint '$mountpoint' -stdinpassphrase");;
              unencrypted)
                  cmd=(/usr/sbin/diskutil mount -mountPoint "$mountpoint" "$volume_uuid");;

If you do encrypt your disk/volume, you’ll need to update the UUIDs in the plist and put the passphrase in the system keychain. You can see how the installer encrypts the volume and stores the credential in:

github.com

NixOS/nix/blob/7e301fd74e929af916116a04d84e130f9e0112de/scripts/create-darwin-volume.sh#L703-L715


      
              _sudo "to mount your Nix volume for encrypting" \
                  /usr/sbin/diskutil mount "$volume_label"
          
          
    password="$(/usr/bin/xxd -l 32 -p -c 256 /dev/random)"
              _sudo "to add your Nix volume's password to Keychain" \
                  /usr/bin/security -i <<EOF
          add-generic-password -a "$volume_label" -s "$volume_uuid" -l "$volume_label encryption password" -D "Encrypted volume password" -j "Added automatically by the Nix installer for use by $NIX_VOLUME_MOUNTD_DEST" -w "$password" -T /System/Library/CoreServices/APFSUserAgent -T /System/Library/CoreServices/CSUserAgent -T /usr/bin/security "/Library/Keychains/System.keychain"
          EOF
              builtin printf "%s" "$password" | _sudo "to actually encrypt your Nix volume" \
                  /usr/sbin/diskutil apfs encryptVolume "$volume_label" -user disk -stdinpassphrase
          
          
    _sudo "to unmount the encrypted volume" \
                  /usr/sbin/diskutil unmount force "$volume_label"

Swordlash · June 16, 2022, 4:15pm

I have FileVault enabled, if that’s relevant.

abathur · June 16, 2022, 6:09pm

Indeed. I updated the previous response to leave more breadcrumbs.

Swordlash · June 16, 2022, 6:43pm

Thank you for the clarifications.

PlumpMath · September 6, 2024, 7:46am

I solved this issue by creating an image of the volume containing the Nix store right after the initial Nix installation, then transferring it to an external hard drive using Carbon Copy Cloner, formatted as APFS. (When formatting APFS, I chose the regular APFS option, not case-sensitive.) After that, I followed the steps you explained by replacing the entries in /etc/fstab and /Library/Launch... with the new APFS-formatted volume. Then, after rebooting and enabling the permissions for Nix and various Nix-related programs that run in the terminal (like nix sh), the issue with the daemon, file permissions, and Nix breaking was completely resolved.

During the resolution process, especially when building the system profile, it’s crucial to preemptively enable all permissions for Nix and Nix-related sh and bash processes to prevent Nix from breaking.

Thank you for writing this guide.