Using NixOS in an isolated environment

Help
19

Hello everyone!

After more than six months using NixOS in an isolated environment, let me share my experience and new questions.

Initial setup

Last november, I have created a mirror for the current nixos-19.09 channel, and nix-copied the associated store paths to build a local binary cache. This was achieved using this script:

#!/bin/sh

set -e

if [ $# -ne 3 ]; then
    echo "usage: $0 <channel> <local_path> <local_binary_cache>"
    exit 1
fi

channel=https://nixos.org/channels/$1
cache=$(curl -sSL $channel/binary-cache-url)
mirror=$2
mirror_cache=$3

printf "\n\e[32m=> Create a local channel in $mirror...\e[0m\n\n"

# Build the local channel.
mkdir -p "$mirror"
curl -L $channel/git-revision > "$mirror/git-revision"
curl -L $channel/nixexprs.tar.xz > "$mirror/nixexprs.tar.xz"
curl -L $channel/store-paths.xz > "$mirror/store-paths.xz"
printf "file://$mirror_cache" > "$mirror/binary-cache-url"

printf "\n\e[32m=> Copy the binary cache to $mirror_cache...\e[0m\n\n"

# Build the local binary cache.
mkdir -p "$mirror_cache"
xzcat "$mirror/store-paths.xz" | xargs nix copy --store $cache --to "file://$mirror_cache"

printf "\n\e[32m\e[1mThe mirror is up to date!\e[0m\n\n"

Once this mirror has been ready on my machine, I have switched my nixos channel to it and added this to my configuration.nix:

# Nix configuration for offline mode.
nix = {
  binaryCaches = lib.mkForce [ "file:///data/Mirroirs/nixpkgs/cache" ];
  gc.automatic = false;
};

Then, I rebuilt my system, and was ready to go offline for a year.

What works out of the box

Everyday software installation on an already-working system

Each time I have needed new software on my machine that is already present in nixpkgs, rebuilding my system using the local binary cache worked smoothly. I’m sure there are some edge cases where an internet access would be required, but I have not encountered any of them yet.

New server-like system setup

Once arrived in Kerguelen, I have copied my local mirror to a server in our network to make it easily available in the network. Installing a NixOS server instance for test purpose was pretty straightforward: I just needed to configure the channel and substituters in the installer, and all worked as expected.

When things are missing

Things have become more complicated since I wanted to setup machines with a desktop environment: some store paths are missing from the local binary cache. Fore some reason, Nix wants to rebuild some things from source, then need the sources and patches for OpenSSL, Gettext and other few software.

As a workaround, I use nix-serve to share my Nix store, which contains the missing paths, allowing me to deploy new desktop machines, but this feels highly hacky.

I have then a few questions regarding these missing paths:

  1. How to get a list of what store paths are needed by a configuration but are missing from a given binary cache?

  2. Why are some comonly-needed store paths missing from the nixos-xx.xx channel? I mean, a configuration with a desktop environment is quite common, and I don’t remember building anything from source when connected to the internet: that means the missing paths do exist in cache.nixos.org somehow, but are not part of a channel. Is this possible?

Before to go any further, I would like to be able to create a binary cache containing everything needed to setup new « standard » desktop machines. Currently almost everything is there, but these tiny missing things are a blocker.

Another case that needs my Nix store as a complementary binary cache is the setup of the first generation for home-manager. I can setup a local mirror for its channel so I can install it on machines without internet, but some things are missing to build the first generation. Afterwards all works OK with my incomplete binary cache.

System upgrades

I am able to get new versions of nixpkgs and my binary cache mirror every few months, when the Marion Dufresne brings us postal mail. Last time was in April, next time will be in September, but I need to prepare everything for the parcel to be shipped from Metropolitan France before July, 15th. I have access to a distant NixOS machine through our very slow and high latency VSAT link, from which I am able to build new versions of my binary cache.

Provided that my initial binary cache was incomplete, I have done several things when preparing for my first update in February:

  1. I have created a mirror for the new version of NixOS 19.09 available, as incomplete as the first one,

  2. I have built my configuration on the distant NixOS machine, using the mirror channel, and exported its output closure:

    # Add the mirror channel.
sudo nix-channel --add file:///path/to/mirror mirror

# Build and export the output closure.
nix build -f '<mirror/nixos>' system --arg configuration path/to/configuration.nix
nix-store --export $(nix-store -qR result) > path/to/export.deps

  3. To be sure I would not miss anything, I even built a full cache, including the build derivations:

    # Instantiate the configuration and build the cache.
nix-instanciate '<mirror/nixos>' -A system --arg configuration path/to/configuration.nix --add-root drv --indirect
nix-store --realise $(nix-store -qR --include-outputs drv)
nix-store --export $(nix-store -qR --include-outputs drv) > path/to/export.cache

  4. I repeated this for my home-manager configuration, and some custom default.nix files for development projects environments.

    NIX_PATH=nixpkgs=/nix/var/nix/profiles/per-user/root/channels/mirror:/nix/var/nix/profiles/per-user/root/channels HOME_MANAGER_CONFIG=$PWD/jpc.nix home-manager build
nix-store --export $(nix-store -qR result) > path/to/export.deps
nix-store --realise $(nix-store -qR --include-outputs $(nix-store -qd result))
nix-store --export $(nix-store -qR --include-outputs $(nix-store -qd result)) > path/to/export.cache

In April, when the SSD containing the new mirror and these store exports arrived in Kerguelen, I first tried to rebuild my system using only the updated local binary cache. Thing where missing, as expected, so I did a sudo nix-store --import < path/to/export.deps and everything worked as expected.

This ability to build a configuration somewhere, export it and import it in some other store kind of amazes me. This is a really powerful feature of Nix. However, this is still a workaround for my problem, and it works only when you know in advance which configurations are present in the isolated environment.

I’d prefer to build a binary cache that already contains what is needed for most desktop configurations, and try this enhanced binary cache in september to see what is still missing—hopefully only edge case things related to my specific configuration.

24 Likes