Using NixOS in an isolated environment

Help
5

:open_mouth: 71 GB sounds pretty small but those are then all xz compressed nar files. Looking at the output paths this looks like it is only final build products but no sources (which should be sufficient to install packages on a running system?).

6

I am actually in a similar situation (want to set up a local cache for a location with horrible internet connection) . I wondered if I should mirror tarballs.nixos.org as well. What would be the preferred method for that?

2 Likes
7

I think it would be good to have one indeed. The full tarballs.nixos.org must be really big though: what we need is only the tarballs for a given version of nixpkgs. For this, we want to get all the external files referenced by all the derivations we want to cache, then download them.

I haven’t worked on it lately as I was on vacation, and now I still need to do a lot of other things. I try to advance on this subject in parallel to the other parts of my preparation.

The blocking points for me right now are:

  1. For a given derivation, how do we know its external dependencies?
  2. What is the set of derivations that we want to include? For instance, many packages in texlive.combined.scheme-full are not in the binary cache, but this is typically something that would be interesting to have in the tarballs. Same for beam.packages.erlangRxx.*: not everything is pre-compiled with every erlang version available here, but I’d like to have the sources so I can compile them offline.
3 Likes
8
  1. For a given derivation, how do we know its external dependencies?

I would do a nix-store -q -R to find all the dependencies, then do nix-store -q -b outputHash to check which ones are fixed-output. Hopefully Nixpkgs checkout + all fixed-output dependencies should be enough.

  1. What is the set of derivations that we want to include? For instance, many packages in texlive.combined.scheme-full are not in the binary cache, but this is typically something that would be interesting to have in the tarballs. Same for beam.packages.erlangRxx.*: not everything is pre-compiled with every erlang version available here, but I’d like to have the sources so I can compile them offline.

I guess you would need to write a more aggressive version of release.nix… Basically, you tryEval the Nixpkgs (this should succeed) and find out all the attribute names in the package set. Then for every attribute you tryEval it and check if there is outPath (if yes: one more derivation found, most likely) and also list all its subattributes to check later. Rinse, repeat. Of course you’d better check that the data values are attribute sets before recursion.

1 Like
9

Until one will need a new package, for example to set up a VPN to get access to the network (I guess it is exactly the case of recent post of Rebuild NixOS Offline as I hit this problem few times as well).

Finding ALL possible fixed-output derivations (even not included to release.nix) looks doable. Challenging but doable.

And besides solving the problem with isolated environments it would be useful to have a bot to check if fixed-output derivations still have valid hash.
(I bet that fetchurl { url = http://download.processing.org/reference.zip; sha256 = ... } does not, it is updated monthly. And there must be many other like this. It would be useful to have a list of problem spots, published on Hydra or ryatm-bot)

3 Likes
10

Adding here to the idea here to use tryEval over nixpkgs:

  1. use nix-instantiate to then instantiate all derivations. This should give a list of all relevant derivations.
  2. iterate with nix-store -qR over all drv files to get all input/dependency paths (which includes all sources).
  3. Download all of the store paths from step 2 with nix-store --realise from the cache.

That should put all derivations in the store path to rebuild all packages offline(?)

1 Like
11

I’m told the sum total of derivations in the caches it 180TB. It’s never been garbage collected. :smiley:
Don’t forget to have a checkout of the nixpkgs repository too!

12
  1. iterate with nix-store -qR over all drv files to get all input/dependency paths (which includes all sources).
  2. Download all of the store paths from step 2 with nix-store --realise from the cache.

Erm, you really want to select only the fixed-output derivations — there are quite a few long-build packages that are also not in the cache.

Actually, it is a good question whether one wants to include the sources of previously-working broken packages…

13

I am afraid a single nix-instantiate run would not be enough, some FOD may be excluded by
if stdenv.isAarch64 then fetchpatch {}
or
if config.use_pulseaudio then fetchpatch {}

The task is closer to test coverage or fuzzers

2 Likes
14

That would address one the original questions: how do you get everything into your store so that you can re-build a random package of your choice? For that I need all FODs, right?

2 Likes
15

OK, so even this one may still miss some FODs.

16

sidenote: this could be useful in high security contexts where you don’t want to ever connect a machine to the network?

8 Likes
17

So what’s the status here currently?

1 Like
18

I’m working hard on a specific mirroring tool. It’s not ready yet and there is no 100% certainty it solves the task fully. For example, this comment makes me think it’s far from done. Currently, the README does not the reflect the current state, there were lots of changes since its last update! Also, mirroring https://tarballs.nixos.org is no more something the project is focused on.

3 Likes
19

Hello everyone!

After more than six months using NixOS in an isolated environment, let me share my experience and new questions.

Initial setup

Last november, I have created a mirror for the current nixos-19.09 channel, and nix-copied the associated store paths to build a local binary cache. This was achieved using this script:

#!/bin/sh

set -e

if [ $# -ne 3 ]; then
    echo "usage: $0 <channel> <local_path> <local_binary_cache>"
    exit 1
fi

channel=https://nixos.org/channels/$1
cache=$(curl -sSL $channel/binary-cache-url)
mirror=$2
mirror_cache=$3

printf "\n\e[32m=> Create a local channel in $mirror...\e[0m\n\n"

# Build the local channel.
mkdir -p "$mirror"
curl -L $channel/git-revision > "$mirror/git-revision"
curl -L $channel/nixexprs.tar.xz > "$mirror/nixexprs.tar.xz"
curl -L $channel/store-paths.xz > "$mirror/store-paths.xz"
printf "file://$mirror_cache" > "$mirror/binary-cache-url"

printf "\n\e[32m=> Copy the binary cache to $mirror_cache...\e[0m\n\n"

# Build the local binary cache.
mkdir -p "$mirror_cache"
xzcat "$mirror/store-paths.xz" | xargs nix copy --store $cache --to "file://$mirror_cache"

printf "\n\e[32m\e[1mThe mirror is up to date!\e[0m\n\n"

Once this mirror has been ready on my machine, I have switched my nixos channel to it and added this to my configuration.nix:

# Nix configuration for offline mode.
nix = {
  binaryCaches = lib.mkForce [ "file:///data/Mirroirs/nixpkgs/cache" ];
  gc.automatic = false;
};

Then, I rebuilt my system, and was ready to go offline for a year.

What works out of the box

Everyday software installation on an already-working system

Each time I have needed new software on my machine that is already present in nixpkgs, rebuilding my system using the local binary cache worked smoothly. I’m sure there are some edge cases where an internet access would be required, but I have not encountered any of them yet.

New server-like system setup

Once arrived in Kerguelen, I have copied my local mirror to a server in our network to make it easily available in the network. Installing a NixOS server instance for test purpose was pretty straightforward: I just needed to configure the channel and substituters in the installer, and all worked as expected.

When things are missing

Things have become more complicated since I wanted to setup machines with a desktop environment: some store paths are missing from the local binary cache. Fore some reason, Nix wants to rebuild some things from source, then need the sources and patches for OpenSSL, Gettext and other few software.

As a workaround, I use nix-serve to share my Nix store, which contains the missing paths, allowing me to deploy new desktop machines, but this feels highly hacky.

I have then a few questions regarding these missing paths:

  1. How to get a list of what store paths are needed by a configuration but are missing from a given binary cache?

  2. Why are some comonly-needed store paths missing from the nixos-xx.xx channel? I mean, a configuration with a desktop environment is quite common, and I don’t remember building anything from source when connected to the internet: that means the missing paths do exist in cache.nixos.org somehow, but are not part of a channel. Is this possible?

Before to go any further, I would like to be able to create a binary cache containing everything needed to setup new « standard » desktop machines. Currently almost everything is there, but these tiny missing things are a blocker.

Another case that needs my Nix store as a complementary binary cache is the setup of the first generation for home-manager. I can setup a local mirror for its channel so I can install it on machines without internet, but some things are missing to build the first generation. Afterwards all works OK with my incomplete binary cache.

System upgrades

I am able to get new versions of nixpkgs and my binary cache mirror every few months, when the Marion Dufresne brings us postal mail. Last time was in April, next time will be in September, but I need to prepare everything for the parcel to be shipped from Metropolitan France before July, 15th. I have access to a distant NixOS machine through our very slow and high latency VSAT link, from which I am able to build new versions of my binary cache.

Provided that my initial binary cache was incomplete, I have done several things when preparing for my first update in February:

  1. I have created a mirror for the new version of NixOS 19.09 available, as incomplete as the first one,

  2. I have built my configuration on the distant NixOS machine, using the mirror channel, and exported its output closure:

    # Add the mirror channel.
sudo nix-channel --add file:///path/to/mirror mirror

# Build and export the output closure.
nix build -f '<mirror/nixos>' system --arg configuration path/to/configuration.nix
nix-store --export $(nix-store -qR result) > path/to/export.deps

  3. To be sure I would not miss anything, I even built a full cache, including the build derivations:

    # Instantiate the configuration and build the cache.
nix-instanciate '<mirror/nixos>' -A system --arg configuration path/to/configuration.nix --add-root drv --indirect
nix-store --realise $(nix-store -qR --include-outputs drv)
nix-store --export $(nix-store -qR --include-outputs drv) > path/to/export.cache

  4. I repeated this for my home-manager configuration, and some custom default.nix files for development projects environments.

    NIX_PATH=nixpkgs=/nix/var/nix/profiles/per-user/root/channels/mirror:/nix/var/nix/profiles/per-user/root/channels HOME_MANAGER_CONFIG=$PWD/jpc.nix home-manager build
nix-store --export $(nix-store -qR result) > path/to/export.deps
nix-store --realise $(nix-store -qR --include-outputs $(nix-store -qd result))
nix-store --export $(nix-store -qR --include-outputs $(nix-store -qd result)) > path/to/export.cache

In April, when the SSD containing the new mirror and these store exports arrived in Kerguelen, I first tried to rebuild my system using only the updated local binary cache. Thing where missing, as expected, so I did a sudo nix-store --import < path/to/export.deps and everything worked as expected.

This ability to build a configuration somewhere, export it and import it in some other store kind of amazes me. This is a really powerful feature of Nix. However, this is still a workaround for my problem, and it works only when you know in advance which configurations are present in the isolated environment.

I’d prefer to build a binary cache that already contains what is needed for most desktop configurations, and try this enhanced binary cache in september to see what is still missing—hopefully only edge case things related to my specific configuration.

24 Likes
20

Ah, store-paths.xz.

As I maintain the TUNA Nix mirror (I maintain the download script, TUNA provides the server), I also need to copy entire channels. I have also used store-paths.xz but as you have seen it only includes almost every path, not really all of them. For this reason in the help page

It seems that store-paths.xz is generated by Hydra, corresponding to the jobs of a channel, and posted by this script, whereas cache.nixos.org consists of every(?) store path Hydra produces.

I haven’t dug much further than that though.

5 Likes
21

I haven’t been on discourse a while and I come back to this status report, thats pretty cool!! :smiley:

22

I used to work in an air-gapped environment, and being able to easily mirror would have been necessary to adopt nixpkgs. Thanks for exploring this domain :slight_smile: @ejpcmac

3 Likes
23

@deliciouslytyped, @jonringer You’re welcome :slight_smile: Your comments acted as a reminder to post here the last investigations I’ve done while preparing my over-the-sea upgrade in July.

Building an “enhanced” binary cache

With the idea of getting a more complete binary cache, but still using only upstream cache.nixos.org, I’ve imagined a process I’ve called “cache enhancement”.

First, I’ve split my script in two parts:

  1. nix-mirror-channel, that just fetches files from the channel:

    #!/bin/sh

set -e

if [ $# -ne 3 ]; then
    echo "usage: $0 <channel> <local_path> <local_binary_cache>"
    exit 1
fi

channel=https://nixos.org/channels/$1
mirror=$2
mirror_cache=$3

printf "\n\e[32m=> Create a local channel in $mirror...\e[0m\n\n"

# Build the local channel.
mkdir -p "$mirror"
curl -L $channel/git-revision > "$mirror/git-revision"
curl -L $channel/nixexprs.tar.xz > "$mirror/nixexprs.tar.xz"
curl -L $channel/store-paths.xz > "$mirror/store-paths.xz"
printf "file://$mirror_cache" > "$mirror/binary-cache-url"

printf "\n\e[32m\e[1mThe mirror is up to date!\e[0m\n\n"

  2. nix-build-cache, that builds a local binary cache given a store-paths.xz and an upstream binary cache:

    #!/bin/sh

set -e

if [ $# -ne 3 ]; then
    echo "usage: $0 <store-paths.xz> <upstream_binary_cache> <local_binary_cache>"
    exit 1
fi

paths=$1
upstream=$2
mirror=$3

printf "\n\e[32m=> Copy the binary cache to $mirror...\e[0m\n\n"

# Build the local binary cache.
mkdir -p "$mirror"
xzcat "$paths" | xargs nix copy --store $upstream --to "file://$mirror"

printf "\n\e[32m\e[1mThe mirror is up to date!\e[0m\n\n"

This, obviously, gives me the same incomplete binary cache. Now, let’s enhance the store-paths.xz by adding to it paths from my configuration that are also present in cache.nixos.org. I’ve written nix-add-store-paths for this:

#!/bin/sh

if [ $# -ne 4 ]; then
    echo "usage: $0 <store-paths.xz> <paths_to_add> <binary_cache> <updated-store-paths.xz>"
    exit 1
fi

store_paths=$1
paths_to_add=$2
binary_cache=$3
updated_paths=$4
tmp_dir=/tmp/nix-add-store-paths


printf "\n\e[32m=> Querying $binary_cache for new paths...\e[0m\n\n"

mkdir -p "$tmp_dir"
rm $tmp_dir/*
xzcat "$store_paths" > "$tmp_dir/store-paths"

while read path; do
    if [ $(grep $path "$tmp_dir/store-paths" | wc -l) -eq 0 ]; then
        nix path-info $path --store $binary_cache

        if [ $? -eq 0 ]; then
            echo $path >> "$tmp_dir/to_add"
        else
            # TODO: Remove when debugging is done.
            echo $path >> "$tmp_dir/missing"
        fi
    fi
done < "$paths_to_add"

sort "$tmp_dir/store-paths" "$tmp_dir/to_add" | xz > "$updated_paths"

# TODO: Remove when debugging is done.
# rm -r "$tmp_dir"

printf "\n\e[32m\e[1mPaths have been merged!\e[0m\n\n"

I was then able to run a second time nix-build-cache with this enhanced store-paths.xz.

Missing paths

I’ve purposely kept lists of added and missing paths during the “enhancement” process to investigate a bit more. Interestingly, there where far less missing paths (332) than added ones (4668).

Paths present in my store but not in cache.nixos.org

The absence of these paths is generally not a surprise, as they belong to the following categories:

  • local system and user configurations,
  • local packages,
  • proprietary software present in nixpkgs but not cached (like the Nvidia driver and unrar),
  • nmd, needed by home-manager.

Paths present in my store and in cache.nixos.org but not in the channel

These paths are however kind of a surprise: they contain a large part of TeXLive, but also lots of libraries. I think their absence is what currently prevents a completely-offline graphical installation of NixOS.

For reference, here’s a gist containing the full list: Nix paths present in both my config and cache.nixos.org but not in the channel · GitHub. There are two files: the second one is exactly the same as the first one, with TeXLive paths removed for readability.

This is strange these paths are not included in the channel. Maybe this is to help channel updates to be available more quickly, but in this case: what builds these paths on Hydra? Is there a complete path list somewhere so I can provide it to my nix-build-cache script without using the “enhancement” hack?

Next steps

The Marion Dufresne is on its way to Kerguelen and the OP will start here on Friday, bringing with it new people and the postal mail containing my SSD with the new channel and binary cache. I will then be able to try things next week.

For sure, I will not be able to upgrade my laptop with the new cache only, since it lacks the first category of paths. For them, I have an archive built by nix-store --export. However, I will try to setup a new machine from a configuration with a desktop environment and nixpkgs-only dependencies, using only my “enhanced” binary cache. I will then let you know if it works as expected.

8 Likes
24

The main TV channel here showed a bunch of documentaries about the marion Dufresnes and the Kerguelen islands. It’s a bit of a random comment but you are a lucky one ! Have fun over there (with nixos and the rest) !

2 Likes