Copying only certain files from a derivation into a docker image

Help
1

I’m looking to create a Docker image using nix which is based on a light image such as alpine and contains a static binary built using another nix expression. I’m using haskell.nix to create my static binary, and this results in a derivation which contains a /bin/server-prod (approx 5MB) and a /nix/store/... of >1GB.

Using the following nix expression, I’ve managed to create a docker image which contains the binary at bin/server-prod but it also contains the large nix store path, meaning the resultant image is very large.

docker-image = pkgs.dockerTools.buildImage {
    name = "project-server";
    tag = "latest";
    fromImage = alpine;
    fromImageName = "alpine";
    fromImageTag = "3.13.4";
    contents = server-prod-static;
  };

Is it possible to get Nix to evaluate the derivation first, and then copy just the resultant static binary into the image?

2

contents = runCommand "cp ${server-prod-static}/bin/hello $out/bin/hello"

Something simple as this should do. Maybe you need to adjust some pathes, and create some folders first, but you should get the idea.

3

Hey, thanks for your reply. Having spent the last couple of hours on this, the closest I’ve gotten is this:

The following expression

builtStaticExecutable = pkgs.runCommand
    "project"
    {}
    "cp -r ${server-prod-static} $out";

builds me a derivation which just contains the bin/server-prod binary.

Using this like so:

cloud-run-image = pkgs.dockerTools.buildImage {
    #...
    contents = "${builtStaticExecutable}";
};

the binary appears in the right place in the resultant docker image but all the dependency nix store paths are also present (the ones used to build the static binary).

I’ve tried fiddling with the paths in runCommand, but you have to put something in $out so each of these ends up more or less the same. The path being passed to contents must be a nix store path (or list of them), and it must be a directory.

Looking at how the buildImage function ends up using the contents argument, it appears it uses rsync to copy the paths as below (where $item is the path passed to contents).

rsync -a${if keepContentsDirlinks then "K" else "k"} --chown=0:0 $item/ layer/

This just looks like a straightforward copy of the things in $item to me, so I don’t understand why the other nix store paths are also passed.

I don’t suppose you have any ideas on this?

4

You are copying the full derivation, so everything it depends on will also be copied. Copy selectively just the files you need.

Edit

Though, if that path does indeed only hold the binary and nothing else, do you perhaps compile with debugging enabled and/or aren’t stripping those out?

That might as well lead to references to other packages that then get pulled.

Have you tried building without docker and using nix why-depends (dunno if there is a non-flake equivalent)

5

If I use the code as above, I get the following output derivation:

$ ls result -a
Permissions Size User Date Modified Name
dr-xr-xr-x     - root  1 Jan  1970  bin
$ ls result/bin -a
Permissions Size User Date Modified Name
.r-xr-xr-x   15M root  1 Jan  1970  server-prod

By using /bin within the cp command’s first argument, this becomes:

$ ls result -a
Permissions Size User Date Modified Name
.r-xr-xr-x   15M root  1 Jan  1970  server-prod

However, if I use nix-tree on the buildStaticExecutable derivation, I find that the dependencies are listed there:

┌─────────────────────────────────────────────────────────┬──────────────────────────────┬─┐
│project  1.08 GiB (1.08 GiB)│x86_64-unknown-linux-musl-sta       1.02 GiB (1010.7 MiB)│...│
│                            │musl-1.2.0-x86_64-unknown-lin      41.95 MiB (551.86 KiB)│...│
│                            │gmp-6.2.1                           39.49 MiB (39.49 MiB)│...│
│                            │numactl-2.0.13-x86_64-unknown       4.01 MiB (354.33 KiB)│...│
│                            │libffi-3.3-x86_64-unknown-lin       3.81 MiB (145.98 KiB)│...│
│                            │                                                         │...│

In case it’s relevant, the way I’m building this is just running nix-build ./nix/pkgs.nix -A builtStaticExecutable. I’m running NixOS.

6

As I said, it seems as if there are references kept in the binary despite the fact that it is statically linked.

As we do not have access to your derivation and code, we can’t do more than some guesswork.

7

Sorry, that wasn’t very helpful of me. I’ve produced a copy of my repo and just stripped out the business code, leaving all the nix files which produce the static executable and docker image from it. Running the commands in the README results in a ~1.15GB docker image containing the binary and the superfluous nix store paths.

Here’s a link to the repo. I really do appreciate you taking the time to help me with this.

8

This is how I do it, but I don’t know if it is the way to go.
But maybe it will help in the discussion
https://ersocon.medium.com/lightweight-haskell-docker-images-with-nix-51bfeea31546

2 Likes
9

Ooh that looks interesting, thanks. I’ll give it a try

10

Thanks both for your help - I’ve now come to a solution.

Following this issue in the haskell.nix repo, I found that I can enable the equivalents of those mentioned in your blog post as follows:

clouded-quays-server-prod-static =
    clouded-quays-server.projectCross.musl64.hsPkgs.clouded-quays-server.components.exes.server-prod.override {
      configureFlags = [
        "--disable-executable-dynamic"
        "--disable-shared"
        "--ghc-option=-optl=-pthread"
        "--ghc-option=-optl=-static"
        "--ghc-option=-optl=-L${pkgs.gmp6.override { withStatic = true; }}/lib"
      ];
      # == THE TWO CHANGED LINES BELOW ARE THE IMPORTANT ONES =================================================
      dontStrip = false;
      enableShared = false;
    };

I’ll update my reproduction repo to include these changes and clean it up a bit for anyone else who comes across this issue.

Cheers!

And here’s a link to my solution repo to avoid needing to scroll back up: notquiteamonad/haskell-nix-docker-reproduction

2 Likes