Privacy of remote builder's /nix/store

Help
1

I am building a CI infrastructure which consists of nix remote builders aka workers. I want those workers to be shared between different projects, teams and repositories (e.q. github actions or gh self-hosted runners having remote builders configured via nix.conf).

I want worker clients to only be able to build nix derivations or download their cached build result from remote builder’s store. For example. projectFoo has built a derivation which result is located in /nix/store/<hash>-foo, then projectBar should only be able to request building its own derivation, I don’t want it to be able to download <hash>-foo derivation’s result.

I’ve built a PoC which configures remote builder on a worker host like this:

let
  # ref: https://github.com/nix-community/infra/blob/399542c1116422a0f37735ccd8c47ad8cf016762/modules/shared/remote-builder.nix
  nix-ssh-wrapper = pkgs.writeShellScript "nix-ssh-wrapper" ''
    case $SSH_ORIGINAL_COMMAND in
      "nix-daemon --stdio")
        exec ${config.nix.package}/bin/nix-daemon --stdio
        ;;
      *)
        echo "Access only allowed for using the nix remote builder" 1>&2
        exit
    esac
  '';
in
{

  users.users.remote-builder = {
    isNormalUser = true;
    openssh.authorizedKeys.keys = [
      ''restrict,command="${nix-ssh-wrapper}" ssh-ed25519 <pubkey>''
    ];
    group = "nogroup";
  };

  nix.settings.allowed-users = [ "remote-builder" ];
}

However I’m somewhat anxious that this setup would allow to poison store or traverse it. Due to lack of documentation on nix-daemon protocol I’m not sure what exactly allowed-users can and can’t do. It’s clear that they can’t make builder substitute with a substituter untrusted by the builder host but some other parts of attach surface are not clear. For example, is it possible to get nix store contents via nix-daemon --stdio and fetch whatever all other builder’s client have built?

1 Like
2

Nix store by definition is meant to be “world-readable” so it should be assumed that it can be traversed, and nothing considered secret should be put in it. As for poisoning the store - only trusted users can do that. If an untrusted user can overwrite anything in the store that’s a serious bug in nix and a security vulnerability

4

In general yes, nix store is world-readable, file permissions are usually +r for everyone and remote builders clients should be able to read /nix/store via basic ls command.
However remote builders can be restricted to only calling nix-daemon --stdio hence my question regarding nix-daemon allowing to read anything which its client doesn’t know digest/hash of.

I’ve analysed piece of nix-daemon which processes all operations requested by daemon clients: https://github.com/NixOS/nix/blob/75101396dc9bcd121ac865fcfb2b81bebe386746/src/libstore/daemon.cc#L278
Apparently every store-related operation requires client to know derivation hash. This makes me think that it should be safe to expose daemon to different clients without risk of them exposing their own derivation outputs to each other (considering that sandbox is enabled, client is allowed but not trusted). Am I missing something?

5

There’s QueryAllValidPaths which anyone can execute and which returns a list of all paths in the store. Then the worker can simply iterate through each of them and request that path from the daemon.

6

Ahh I missed this one. It means a single nix-daemon can’t be shared between mutually untrusting clients.