Docker.nix from NixOS/nix registers roots?

I am wondering if

registers GC roots.

The motivation comes from the PR nixosTest.gitlab.runner: improvements and refactor by gabyx · Pull Request #520953 · NixOS/nixpkgs · GitHub
where I maintain the gitlab-runner test and which serves directly as a demonstration how to do it:

  nixDaemonImageBase = pkgs.callPackage (import (inputs.nix.outPath + "/docker.nix")) {
    name = "local/nix-base";
    tag = "latest";

    bundleNixpkgs = false;
    maxLayers = 2;

    # You can add here a user with uid,gid,uname,gname etc.
    # We are using root.
    extraPkgs = jobImgs.allStoreDrv;

    nixConf = {
      cores = "0";
      experimental-features = [
        "nix-command"
        "flakes"
      ];

      secret-key-files = [ config.sops.secrets.nix-store-signing-key.path ];

      min-free = "1G"; # Triggers garbage collection.
      max-free = "100G"; # Stops garbage collection at 100G free space.

      inherit extra-trusted-public-keys extra-trusted-substituters;
    };
  };

  nixDaemonImage = pkgs.dockerTools.buildLayeredImage {
    fromImage = nixDaemonImageBase;
    name = "local/nix-daemon";
    tag = "latest";

    fakeRootCommands =
      # bash
      ''
        mkdir -m 700 -p root/.ssh
        cp "${sshConfig}/config" root/.ssh/config
        chmod 600 root/.ssh/config
        cp "${knownHosts}/known_hosts" root/.ssh/known_hosts
        chmod 600 root/.ssh/known_hosts
      '';

    config = {
      Volumes = {
        "/nix/store" = { };
        "/nix/var/nix/db" = { };
        "/nix/var/nix/daemon-socket" = { };
      };
      Labels = noPruneLabels;
    };
    maxLayers = 4;
  };

I see that when the daemon runs garbage collection, certain /nix/store paths in extraPkgs are deleted…?

Well, did you inspect the GC roots? /nix/var/nix/gcroots

The mere existence of that file does not register GC roots. A file can’t register GC roots, it’s just data - your wallpaper doesn’t register GC roots either.

If you ask nix to do something with the file, that may involve creating GC roots, depending on what exactly you ask nix to do. But you’re not telling us what you’re asking nix to do, so your question doesn’t have enough information to give you an answer.

1 Like

Actually the solution is that it does not add these roots in extraPkgs, and I needed to do symlinks. also not using extraPkgs cause it mangles everything into a pkgs.buildEnv where my packages collide… :

nixDaemonImageBase = pkgs.callPackage (import (inputs.nix.outPath + "/docker.nix")) {
    name = "local/nix-base";
    tag = "latest";

    bundleNixpkgs = false;
    maxLayers = 2;

    nixConf = {
      cores = "0";
      experimental-features = [
        "nix-command"
        "flakes"
      ];

      secret-key-files = [ config.sops.secrets.nix-store-signing-key.path ];

      min-free = "1G"; # Triggers garbage collection.
      max-free = "100G"; # Stops garbage collection at 100G free space.

      # # Reduce disk usage by discarding old derivations/outputs
      # keep-derivations = false;
      # keep-outputs = false;

      inherit extra-trusted-public-keys extra-trusted-substituters;
    };
  };

nixDaemonImage = pkgs.dockerTools.buildLayeredImage {
    fromImage = nixDaemonImageBase;
    name = "local/nix-daemon";
    tag = "latest";

    fakeRootCommands =
      # bash
      ''
        mkdir -m 700 -p root/.ssh
        cp "${sshConfig}/config" root/.ssh/config
        chmod 600 root/.ssh/config
        cp "${knownHosts}/known_hosts" root/.ssh/known_hosts
        chmod 600 root/.ssh/known_hosts

        # Add all store paths and make them GC roots, so we dont loose them.
        # NOTE: Cannot add it to `extraPkgs` cause of the profile
        #       which uses `buildEnv` which collides.
         mkdir -p nix/var/nix/gcroots/additional-pkgs
         ${lib.concatMapStringsSep "\n" (pkg: ''
           echo "Adding package '${pkg}'"
           ln -fs "${pkg}" "nix/var/nix/gcroots/additional-pkgs/"
         '') jobImgs.allStoreDrv}
      '';

    config = {
      Volumes = {
        "/nix/store" = { };
        "/nix/var/nix/db" = { };
        "/nix/var/nix/daemon-socket" = { };
      };
      Labels = noPruneLabels;
    };
    maxLayers = 4;
  };

What you’re sharing isn’t much of a solution. I suppose you’ve figured things out for yourself, but if anyone stumbles upon this in the future: tell us what your actual use case is, and we can give proper advice for solving it.

I suspect in this case a more sustainable fix would be something like devenv, but it really depends on what the problem actually is.

1 Like

Would also be interested in the use case, i.e., what the jobImgs.allStoreDrv entail, is it related to Gitlab runner with containerized daemon ?

The question arose due to a PR about the gitlab runner

which I try to maintain as an example in the nixos manual for the CI curious minds =)
review welcome!