How to build a docker image with a working Nix inside it

Help
1

Context

I already have a docker container image built using dockerTools.buildImage that I use for work. This allows me to keep my work files completely separate from personal ones including .gitconfig, ssh keys and other dotfiles and gives my system extra security while at it (no NPM supply chain attack can get to my personal files!). Some of my colleagues are taking interest in this setup but unfortunately, some of them work on other projects and require different software so I can’t just share my container image in its current form.

Goal

So far, I concluded that the best solution to current and even future problems is to build a new image that only includes:

  • things that are required for basic operation (toybox, CA certificates),
  • things that we use across all teams (bash, git, ssh),
  • and Nix, so we can use nix develop to get our dependencies.

On top of that, I would like to keep some features of the current setup, namely:

  • a /home/nobody directory with some dotfiles,
  • no way to become root.
  • no unused files or unexplained dependencies.

The problem

Unfortunately, I am unable to get Nix to work in a container, which prevents the whole idea from working.

My image definition looks something like this:

packages.default = pkgs.dockerTools.buildImage {
  # ...
  copyToRoot = pkgs.buildEnv {
    name = "image-root";
    paths = with pkgs; [
      bashInteractive
      nix
      (pkgs.writeTextDir "etc/nix/nix.conf" "experimental-features = nix-command flakes")
      # ...
    ];
    pathsToLink = [ "/bin" "/etc" ];
  };
  # ...
  config = {
    User = "nobody";
    Cmd = [ "/bin/sh" "-l" ];
  };
}

This seems to be completely broken. I am unable to execute nix develop without errors (see below) and I am even unable to run nix --help (error: executing '': No such file or directory)

error: builder for '/nix/store/8mvprq5spsl0z3b620gkyz971yk317if-nix-shell-env.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/rhvbjmcfnkg8i2dxpzr114cp1ws7f667-bash-5.2-p15/bin/bash': No such file or directory
       For full logs, run 'nix log /nix/store/8mvprq5spsl0z3b620gkyz971yk317if-nix-shell-env.drv'.

rhvbjmcfnkg8i2dxpzr114cp1ws7f667-bash-5.2-p15/bin/bash can be found in ~/.local/share/nix/root/nix/store but for some reason Nix is looking in /nix/store. This error remains the same if I specify --store manually.

This error specifically changes when running as root, it is complaining about the lack of nixbld users, but I did not investigate this because one of my goals is to run completely unprivileged.

I tried replacing dockerTools.buildImage with dockerTools.buildImageWithNixDb but this only affected the non-root behavior when executing nix develop replacing the error with error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permitted.

Alternative solution?

I am aware of the nixos/nix image on DockerHub and I am expecting something related to this to become the solution but I am not sure how to execute this approach and I am concerned about how big the source code of that image is.

I looked at the source code and expected to find maybe 200 lines to bootstrap Nix but instead I found almost 800 lines and too many things happening for how seemingly simple the end goal is. I understand that we need some nixbld users but there is a lot more code than just the setup of those.

That image also seems to expect that the user will be root, which I would really like to avoid. I don’t really see why this should not be possible with --store in an owned location.

If you are able to give me any clues on how I can progress on any of the mentioned direction or know a completely different way of solving this please let me know. I really hope I can put Nix into more people’s hands and I don’t want to let my colleagues down that I preemptively got quite excited. I hope you can learn something from this too.

1 Like
2

You can simply pick the code that builds the official nix container and base your own container on it. I have a flake that contains (partly redacted):

inputs = {
   nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
   flake-utils.url = "github:numtide/flake-utils";
   nix = {
     url = "github:NixOS/nix";
     inputs.nixpkgs.follows = "nixpkgs";
   };
  };

outputs = { self, nix, nixpkgs, flake-utils }:

    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        flakeRegistry = {
          nixos = {
            from = { type = "indirect"; id = "nixos"; };
            to = nixpkgs;
            exact = true;
          };
          nixpkgs = {
            from = { type = "indirect"; id = "nixpkgs"; };
            to = nixpkgs;
            exact = true;
          };
        };

        flakeRegistryJSON = (pkgs.formats.json { }).generate "flake-registry.json" {
          version = 2;
          flakes =
            pkgs.lib.mapAttrsToList (n: v: { inherit (v) from to exact; }) flakeRegistry;
        };

        containerInfo = {
          name = "mycontaner";
          tag = nixpkgs.rev;
        };
        flake-input-sources = pkgs.writeTextFile rec { name = "flake-inputs.log"; text = ''${nixpkgs.outPath} ${nix.outPath}''; destination = "/${name}"; };
        # use official nix container as base to get everything for flake-based building
        container-layered = pkgs.lib.throwIfNot (pkgs.stdenv.isLinux) "Docker images are only supported on Linux." (import ("${nix.outPath}" + "/docker.nix") {
          inherit (containerInfo) name tag;
          inherit pkgs;
          bundleNixpkgs=false;
          extraPkgs = [ pkgs.git pkgs.which pkgs.stdenvNoCC ] ++ [ flake-input-sources ];
          maxLayers = 111;
          nixConf = {
            experimental-features = [ "nix-command" "flakes" ];
          };
          flake-registry = flakeRegistryJSON;
        });

      in
      rec {
        devShell = pkgs.mkShell {
          buildInputs = [ run-packages dev-packages ];
          shellHook = '' echo Hello '';
        };
        packages = flake-utils.lib.flattenTree {
          container = container-layered;
        };

to create a flake-enabled Docker image that I use in CI.

3

Thank you so much! This gets me a lot closer to the goal! I tried basing off this image but the best I could figure out was to pull it from Docker Hub.

I do have some questions, however:

  • What is the difference between the official image that you use and the one from docker-nixpkgs?
  • Why is the official image so large (>500Mb) compared to Ubuntu (<80Mb) and the Nix image from docker-nixpkgs (288Mb). And why are both so large actually? I thought one of the benefits of doing containers this way was minimalism. I am able to build minimalist images using Nix but why does including Nix suddenly blow it up so much?
  • How did you figure out that you have to use import ("${nix.outPath}" + "/docker.nix"). I would like to be able to learn how to find this myself so that I don’t have to ask next time and maybe even able to help others. How did you decide to use that instead of (import nix).packages.dockerImage?
4

I have no idea. :slight_smile: I never looked at docker-nixpkgs - @zimbatm knows more about this.

The image pulls in a lot of dependencies to get the full nix flakes experience, including copies of unpacked nixpkgs-tarballs. These are quite heavy but only needed for running certain nix commands. If one builds a dedicated container that runs a pre-built software those are not required, resulting in a much smaller image.

When a flake input is given the name “x”, the source of the flake is available as x.outPath. I studied docker.nix via the GitHub UI and decided i wanted to access the definitions in this file rather than build on the final artifact.

1 Like
5

You are awesome! I tried exploring the code of docker-nixpkgs image again and actually got somewhere this time!

I think nix image from docker-nixpkgs values minimalism much more. I really like that it asks to pass all packages manually so you know exactly what software is already in the image before you layer on more.

{
  inputs = {
    docker-nixpkgs = {
      url = "github:nix-community/docker-nixpkgs";
      flake = false;
    };
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs";
  };

  outputs = { self, docker-nixpkgs, flake-utils, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };
      buildImageWithNix = import ("${docker-nixpkgs}" + "/images/nix/default.nix");
    in rec {
      packages.nixImage = buildImageWithNix {
        # All of this is required by the function
        inherit (pkgs)
          dockerTools
          bashInteractive
          cacert
          coreutils
          curl
          gnutar
          gzip
          iana-etc
          nix
          openssh
          xz;

        # We are actually going to use Git so we use the full version.
        gitReallyMinimal = pkgs.git;

        extraContents = with pkgs; [
          # Since we need git in this image let's add git-lfs right away.
          git-lfs
        ];
      };

      packages.default = pkgs.dockerTools.buildImage {
        fromImage = packages.nixImage;
        # ...
      };
    };
}

I think I am going to go with this solution because it is smaller and I can comprehend everything that is happening here (I have no idea what flake registries are and I don’t think I am ready to know yet :grin:).

I think it is clear that the answer to my original question, “How to build a docker image with a working Nix inside it” is “don’t” but another question remains: can this work without root?

With all the complexity that turned up I am ready accept that you have to be root (or use the nix daemon which, in a container, is a lot worse :laughing:). I actually already switched to rootless Docker daemon so there is no real reason for me personally to bother with this anymore but just for the sake of understanding Nix better, there really is no way to use it completely unprivileged?