Multi-Arch docker image from a flake: Getting the same hash for a expression on two systems

Help
1

Hi all!

I am trying to build a docker image containing files (generated by an SSG) and a web-server. For sport, I would like to have a layered image for amd64 and arm64 that contains the same layer for the static files.

This is a minimized example of what I have:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  inputs.systems.url = "github:nix-systems/default";
  inputs.flake-utils = {
    url = "github:numtide/flake-utils";
    inputs.systems.follows = "systems";
  };

  outputs = { nixpkgs, flake-utils, self, ... }:
    let
      static_site = nixpkgs.legacyPackages.${builtins.currentSystem}.runCommand "site-files" { } ''
        mkdir $out
        echo hello > $out/index.html
      '';
    in flake-utils.lib.eachDefaultSystem (system:
      let pkgs = nixpkgs.legacyPackages.${system};
      in {
        packages.dockerImage = pkgs.dockerTools.buildLayeredImage {
          name = "static-site";
          tag = "latest";
          contents = [ static_site ];
          config = { EntryPoint = [ "${pkgs.nginx}/bin/nginx" ]; };
        };
      });
}

This can be built with nix build --impure .#dockerImage (It will not run, the nginx is not configured, it’s just there for illustration). --impure is needed as I access currentSystem.

But (of course) the resulting docker image will be different in all layers between arm64 and amd64, as the hash-input for static_site contains currentSystem.

So: is there a way I can tell nix (or prove to nix?) that the output of two expressions (aarch64-linux.runCommand ... and x86_64-linux.runCommand ...) is the same and “should” have the same hash?

Thanks!

Cheers,
Philipp

2

The experimental ca-derivations feature could get you there. Without that, the name of any derivation output is a hash of the input instructions, not the contents, so it’s impossible for the names to be the same since the input instructions do (and must) contain a specification of what system to run the build process on. Note, however, that even with ca-derivations, nix will still have to build that part twice, once on each architecture, in order to discover that they have the same content.

The other option is to make the static_site portion have to be built on amd64, even when building the arm64 image. That would make your outputs difficult to build in many environments, however.

Side note: you wouldn’t need builtins.currentSystem, and thus the --impure, if you just moved static_site into the lower let-in block where pkgs is… (and installPhase is actually just an unused variable).

1 Like
3

ca-derivations

I’ll look at that, thanks!

The other option is to make the static_site portion have to be built on amd64, even when building the arm64 image. That would make your outputs difficult to build in many environments, however.

I hoped that I could have one builder that creates the static site system agnostic and then packages the rest using cross-compilation if necessary. Now that I am writing that, it seems possible to do that. Let me experiment some more :smiley: .

(and installPhase is actually just an unused variable).

I removed that – that was a copy+paste mistake.

4

If you were planning on cross-compiling one of the systems anyway, then just using the builder’s system for the static site derivation becomes much more ergonomic and feasible.

1 Like
5

I played some more and arrived at

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  inputs.systems.url = "github:nix-systems/default";
  inputs.flake-utils = {
    url = "github:numtide/flake-utils";
    inputs.systems.follows = "systems";
  };

  outputs = { nixpkgs, flake-utils, self, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        pkgsArm = import nixpkgs {
          localSystem = system;
          crossSystem = "aarch64-linux";
        };
        pkgsAmd = import nixpkgs {
          localSystem = system;
          crossSystem = "x86_64-linux";
        };
        static_site = pkgs.runCommand "site-files" { } ''
          mkdir $out
          echo hello > $out/index.html
        '';
        buildLayeredImage = pkgsCross: tag:
          pkgs.dockerTools.buildLayeredImage {
            inherit tag;
            name = "static-site";
            contents = [ static_site ];
            config = { EntryPoint = [ "${pkgsCross.nginx}/bin/nginx" ]; };
          };
      in {
        packages.dockerImageArm = buildLayeredImage pkgsArm "latest-arm64";
        packages.dockerImageAmd = buildLayeredImage pkgsAmd "latest-amd64";
        packages.push = pkgs.writeScriptBin "push.sh" ''
          set -exu
          docker load < ${self.outputs.packages.${system}.dockerImageArm}
          docker load < ${self.outputs.packages.${system}.dockerImageAmd}

          docker tag static-site:latest-arm64 registry/test:latest-arm64
          docker tag static-site:latest-amd64 registry/test:latest-amd64

          docker push registry/test:latest-arm64
          docker push registry/test:latest-amd64

          docker manifest create registry/test:latest \
            --amend registry/test:latest-arm64 \
            --amend registry/test:latest-amd64

          docker manifest push registry/test:latest
        '';
      });
}

I am happy with this; nix run .#push does what I want.

Thank you for the pointers!

1 Like