Questions about local / remote deps

Help
1

Hi,

I’m trying to build a Docker image with Nix on macOS + a remote builder that runs in Docker container. The Nix expression looks like this:

{
  nixpkgs ? import <nixpkgs> { system = "x86_64-linux"; }
}:

with nixpkgs;

dockerTools.buildImage {
  name = "sample";
  tag = "latest";

  contents = python36.withPackages (ps: [
    ps.arrow
    ps.requests
  ]);

  config = {
    Cmd = ["python3" ./main.py];
  };
}

When I run nix build, it seems to download all deps on macOS, copies some of them to the remote builder, builds the image there, and then copies the final output back to macOS. Why doesn’t it download all deps on remote builder to avoid unnecessary copying (and disk space waste)?

2

You can use builders-use-substitutes = true in /etc/nix/nix.conf to avoid copying at least those dependencies that can be downloaded from hydra: Distributed build - NixOS Wiki

2 Likes
3

I’ve already set builders-use-substitutes = true, but it doesn’t seem to help.

Here’s what I’ve observed:

Before the build:

macOS $ nix show-config | grep builders-use-substitutes
builders-use-substitutes = true

macOS $ du -sh /nix
3.9G    /nix

remote-builder $ du -sh /nix
259M    /nix/store

After the build:

connecting to 'ssh://remote-builder'...
building '/nix/store/drncprw8hqhjjg1fa8vn6m3nmrz9r56j-python3-3.6.6-env.drv' on 'ssh://remote-builder'...
waiting for the upload lock to 'ssh://remote-builder'...
copying dependencies to 'ssh://remote-builder'...
copying outputs from 'ssh://remote-builder'...
copying 1 paths...
copying path '/nix/store/lb34dnpw5ffmlghajypn1hw2nw6cyqqf-python3-3.6.6-env' from 'ssh://remote-builder'...
connecting to 'ssh://remote-builder'...
building '/nix/store/aphpsjwdz3lc7vx7w4y4a94mfajzibp2-docker-layer-sample.drv' on 'ssh://remote-builder'...
waiting for the upload lock to 'ssh://remote-builder'...
copying dependencies to 'ssh://remote-builder'...
[1/2/5 built, 96 copied (616.1 MiB), 134.0 MiB DL] building docker-layer-sample on ssh://remote-builder

macOS $ du -sh /nix
4.7G    /nix

remote-builder $ du -sh /nix
1.1G    /nix

All these dependencies were downloaded from binary cache - if builders-use-substitutes = true is in effect, I would expect that disk usage of /nix on macOS increases only by 47MB (the size of the final image), not 819MB. And it’s lucky that the remote builder is actually running in a local Docker container, otherwise this extra 819MB needs to be transferred on the Internet twice (download from binary cache & upload to remote builder).

Is there anything I can do about this?

4

The default mode is to pass everything through the local store. I’ve seen instead workflows with

env NIX_REMOTE='ssh-ng://my.machine?compress=true' nix-build ...

though I don’t remember if one needs to configure something specific on the remote end to make this work. This should just evaluate locally and do all the rest on the remote machine (not even copying the result back to you).

3 Likes
5

Wow, this looks great! :yum:

macOS $ nix-collect-garbage -d
remote-builder $ nix-collect-garbage -d

macOS $ NIX_REMOTE=ssh-ng://remote-builder nix build -f docker.nix
Adding layer...
Adding meta...
Cooking the image...
Finished.
[1 built]

macOS $ nix copy -v --no-recursive --no-check-sigs --from ssh://remote-builder -f docker.nix
copying 1 paths...
copying path '/nix/store/srmi8azkkvjggw90g1s668w50j0i8c2g-docker-image-sample.tar.gz' from 'ssh://remote-builder'...
[1 copied (41.7 MiB)]

# faking out-link
macOS $ ln -fns $(nix path-info -f docker.nix) ./result

macOS $ du -sh /nix
3.9G    /nix

remote-builder $ du -sh /nix
1.1G    /nix
1 Like