Rebuild NixOS Offline without `system.includeBuildDependencies`

Guides
1

I found a working approach to make a flake-based NixOS configuration self-contained (being able to rebuild offline as much as possible). Comparing with expensive system.includeBuildDependencies(https://discourse.nixos.org/t/rebuild-nixos-offline/3679/11), which can make the system derivation to take up a very large amount of space, this approach preserves minimal dependencies needed to build OS derivation when there are only minor changes in the system configuration, and no new packages are introduced.

In a NixOS flake, pass all flake inputs into the NixOS module as a flake-inputs specialArg:

{
  inputs = { ... };

  outputs =
    { ... }@inputs:
    {
      nixosConfigurations.xxx = nixpkgs.lib.nixosSystem {
        ...
        specialArgs = {
          flake-inputs = inputs;
        };
      };
    };
}

Recursively collect flake inputs (and inputs of inputs, inputs of inputs of inputs, …) into allFlakeInputsClosure and add their store paths to system.extraDependencies:

{
  config,
  lib,
  pkgs,
  flake-inputs,
  ...
}:
{
  system.extraDependencies =
    let
      filterValidInputs = lib.filterAttrs (n: v: n != "self" && builtins.isAttrs v && v ? outPath);

      startSet = lib.mapAttrsToList (name: input: {
        key = input.outPath;
        value = input;
      }) (filterValidInputs flake-inputs);

      allFlakeInputsClosure = builtins.genericClosure {
        inherit startSet;
        operator =
          { value, ... }:
          lib.mapAttrsToList (name: input: {
            key = input.outPath;
            value = input;
          }) (filterValidInputs (value.inputs or { }));
      };

      allFlakeInputs = builtins.map (node: node.value) allFlakeInputsClosure;
    in
    allFlakeInputs;
}

Enable keep-derivations and keep-outputs so the GC will retain build-time derivations and their outputs that are referenced by the system derivation but not directly referenced by system closure. This prevents transient evaluation/build-time dependencies from being garbage-collected and therefore prevents unexpected refetches during an offline nixos-rebuild:

nix.settings.keep-derivations = true;
nix.settings.keep-outputs = true;

Add --offline and --option substitute false when building OS to force the rebuild to use only local store paths and to avoid contacting binary caches for substitutes:

sudo nixos-rebuild switch --offline --option substitute false

Now, with the closure of flake inputs recorded and these Nix settings enabled, the system’s build-time dependencies that were previously realized will survive GC, minor parameter tweaks won’t pull new network artifacts, which makes offline nixos-rebuild practical for the common, small-change workflows (while still requiring an initial online bootstrap or an explicit store/cache copy when new inputs are introduced).

Hope this helps.

6 Likes
2

I just implemented something very similar in my personal config for similar reasons, what are the odds? In my case, it’s because I’m deploying the flake to a server with deploy-rs. Some of the flake’s inputs aren’t publicly accessible, but I wanted the server to be able to update its own nixpkgs, so I needed a way to make sure those private inputs get uploaded along with the flake and are kept around.

I spent some more time on my version of allFlakeInputs, though. There’s an edge case that can happen with follows where two flakes can have the same outPath but different transitive inputs. Consider this flake:

{
    inputs = {
        nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
        uses-nixpkgs-1 = {
            url = "github:thisthing/uses-nixpkgs";
            inputs.nixpkgs.follows = "nixpkgs";
        };
        uses-nixpkgs-2 = {
            url = "github:thisthing/uses-nixpkgs";
            # This one uses its own nixpkgs
        };
    };
}

Contrived example, but it can happen across different locations in your graph instead of in one place. Say /uses-nixpkgs-1 happens to be the same version as /uses-nixpkgs-2. If the dependency crawler happens to process /uses-nixpkgs-1 first, it’ll register its outPath as done and proceed to crawl /uses-nixpkgs-1/nixpkgs, which is /nixpkgs. Then once it gets to /uses-nixpkgs-2 it’ll notice it already processed that outPath, and won’t notice /uses-nixpkgs-2/nixpkgs, which is not the same as /nixpkgs.

Here’s my version of the crawler:

{ lib, ... }:
let
  decodePointer = ptr: lib.splitString "/" ptr |> lib.filter (x: x != "");
  augmentPointer =
    decoded:
    lib.concatMap (x: [
      "inputs"
      x
    ]) decoded;

  nullInputSpec = {
    origin = false;
    follows = null;
    inputs = { };
  };

  mapFlake =
    path: flake: inputOverrides:
    let
      flakeSrc = import "${flake.outPath}/flake.nix";
      hereBase = { inherit path flake; };
      inputNames = builtins.attrNames (flake.inputs or { });
      hereInputs =
        inputNames
        |> map (k: {
          name = k;
          value = mapInput here flake k flakeSrc.inputs.${k} or { } inputOverrides.${k} or nullInputSpec;
        })
        |> lib.listToAttrs;
      here = hereBase // lib.optionalAttrs (flake ? inputs) { inputs = hereInputs; };
    in
    here;

  mkInputSpec = here: flakeSrcInput: implicit: {
    origin = flakeSrcInput ? url || flakeSrcInput ? type || (implicit && !flakeSrcInput ? follows);
    follows =
      if flakeSrcInput ? follows then
        lib.getAttrFromPath (decodePointer flakeSrcInput.follows |> augmentPointer) here
      else
        null;
    inputs = flakeSrcInput.inputs or { } |> lib.mapAttrs (k: v: mkInputSpec here v false);
  };

  mergeInputSpecs =
    lhs: rhs:
    let
      origin = (lhs.origin || rhs.origin) && follows == null;
      follows =
        if rhs.follows != null then
          rhs.follows
        else if rhs.origin then
          null
        else
          lhs.follows;
      inputs =
        if follows != null || (lhs.inputs == { } && rhs.inputs == { }) then
          { }
        else
          mergeInputSpecInputs lhs.inputs rhs.inputs;
    in
    {
      inherit origin follows inputs;
    };
  mergeInputSpecInputs =
    lhs: rhs:
    lhs // rhs
    |> lib.mapAttrs (k: v: mergeInputSpecs lhs.${k} or nullInputSpec rhs.${k} or nullInputSpec);

  mapInput =
    here: flake: k: flakeSrcInput: inputOverride:
    let
      inputSpec = mkInputSpec here flakeSrcInput true;
      mergedInputSpec = mergeInputSpecs inputSpec inputOverride;
    in
    (
      if mergedInputSpec.follows != null then
        mergedInputSpec.follows
      else
        mapFlake (here.path ++ [ k ]) flake.inputs.${k} mergedInputSpec.inputs
    );

  getTransitiveFlakeSources =
    root:
    builtins.genericClosure {
      startSet = [
        {
          key = [ ];
          obj = root;
        }
      ];
      operator =
        { obj, ... }:
        obj.inputs or { }
        |> lib.mapAttrsToList (
          k: v: {
            key = v.path;
            obj = v;
          }
        );
    }
    |> map (v: v.obj.flake.outPath);
in
{
  resolveFlakeGraph = flake: mapFlake [ ] flake { };
  inherit getTransitiveFlakeSources;
}

Used like this: system.extraDependencies = resolveFlakeGraph self |> getTransitiveFlakeSources;. I’m not actually sure whether my version is fully correct, but it seems to work well enough.

2 Likes