I’m trying to build a Docker image with Nix on macOS + a remote builder that runs in Docker container. The Nix expression looks like this:
{
nixpkgs ? import <nixpkgs> { system = "x86_64-linux"; }
}:
with nixpkgs;
dockerTools.buildImage {
name = "sample";
tag = "latest";
contents = python36.withPackages (ps: [
ps.arrow
ps.requests
]);
config = {
Cmd = ["python3" ./main.py];
};
}
When I run nix build, it seems to download all deps on macOS, copies some of them to the remote builder, builds the image there, and then copies the final output back to macOS. Why doesn’t it download all deps on remote builder to avoid unnecessary copying (and disk space waste)?
You can use builders-use-substitutes = true in /etc/nix/nix.conf to avoid copying at least those dependencies that can be downloaded from hydra: Distributed build - NixOS Wiki
connecting to 'ssh://remote-builder'...
building '/nix/store/drncprw8hqhjjg1fa8vn6m3nmrz9r56j-python3-3.6.6-env.drv' on 'ssh://remote-builder'...
waiting for the upload lock to 'ssh://remote-builder'...
copying dependencies to 'ssh://remote-builder'...
copying outputs from 'ssh://remote-builder'...
copying 1 paths...
copying path '/nix/store/lb34dnpw5ffmlghajypn1hw2nw6cyqqf-python3-3.6.6-env' from 'ssh://remote-builder'...
connecting to 'ssh://remote-builder'...
building '/nix/store/aphpsjwdz3lc7vx7w4y4a94mfajzibp2-docker-layer-sample.drv' on 'ssh://remote-builder'...
waiting for the upload lock to 'ssh://remote-builder'...
copying dependencies to 'ssh://remote-builder'...
[1/2/5 built, 96 copied (616.1 MiB), 134.0 MiB DL] building docker-layer-sample on ssh://remote-builder
macOS $ du -sh /nix
4.7G /nix
remote-builder $ du -sh /nix
1.1G /nix
All these dependencies were downloaded from binary cache - if builders-use-substitutes = true is in effect, I would expect that disk usage of /nix on macOS increases only by 47MB (the size of the final image), not 819MB. And it’s lucky that the remote builder is actually running in a local Docker container, otherwise this extra 819MB needs to be transferred on the Internet twice (download from binary cache & upload to remote builder).
though I don’t remember if one needs to configure something specific on the remote end to make this work. This should just evaluate locally and do all the rest on the remote machine (not even copying the result back to you).