How to build a docker image with a working Nix inside it

Context

I already have a docker container image built using dockerTools.buildImage that I use for work. This allows me to keep my work files completely separate from personal ones including .gitconfig, ssh keys and other dotfiles and gives my system extra security while at it (no NPM supply chain attack can get to my personal files!). Some of my colleagues are taking interest in this setup but unfortunately, some of them work on other projects and require different software so I can’t just share my container image in its current form.

Goal

So far, I concluded that the best solution to current and even future problems is to build a new image that only includes:

  • things that are required for basic operation (toybox, CA certificates),
  • things that we use across all teams (bash, git, ssh),
  • and Nix, so we can use nix develop to get our dependencies.

On top of that, I would like to keep some features of the current setup, namely:

  • a /home/nobody directory with some dotfiles,
  • no way to become root.
  • no unused files or unexplained dependencies.

The problem

Unfortunately, I am unable to get Nix to work in a container, which prevents the whole idea from working.

My image definition looks something like this:

packages.default = pkgs.dockerTools.buildImage {
  # ...
  copyToRoot = pkgs.buildEnv {
    name = "image-root";
    paths = with pkgs; [
      bashInteractive
      nix
      (pkgs.writeTextDir "etc/nix/nix.conf" "experimental-features = nix-command flakes")
      # ...
    ];
    pathsToLink = [ "/bin" "/etc" ];
  };
  # ...
  config = {
    User = "nobody";
    Cmd = [ "/bin/sh" "-l" ];
  };
}

This seems to be completely broken. I am unable to execute nix develop without errors (see below) and I am even unable to run nix --help (error: executing '': No such file or directory)

error: builder for '/nix/store/8mvprq5spsl0z3b620gkyz971yk317if-nix-shell-env.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/rhvbjmcfnkg8i2dxpzr114cp1ws7f667-bash-5.2-p15/bin/bash': No such file or directory
       For full logs, run 'nix log /nix/store/8mvprq5spsl0z3b620gkyz971yk317if-nix-shell-env.drv'.

rhvbjmcfnkg8i2dxpzr114cp1ws7f667-bash-5.2-p15/bin/bash can be found in ~/.local/share/nix/root/nix/store but for some reason Nix is looking in /nix/store. This error remains the same if I specify --store manually.

This error specifically changes when running as root, it is complaining about the lack of nixbld users, but I did not investigate this because one of my goals is to run completely unprivileged.

I tried replacing dockerTools.buildImage with dockerTools.buildImageWithNixDb but this only affected the non-root behavior when executing nix develop replacing the error with error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permitted.

Alternative solution?

I am aware of the nixos/nix image on DockerHub and I am expecting something related to this to become the solution but I am not sure how to execute this approach and I am concerned about how big the source code of that image is.

I looked at the source code and expected to find maybe 200 lines to bootstrap Nix but instead I found almost 800 lines and too many things happening for how seemingly simple the end goal is. I understand that we need some nixbld users but there is a lot more code than just the setup of those.

That image also seems to expect that the user will be root, which I would really like to avoid. I don’t really see why this should not be possible with --store in an owned location.

If you are able to give me any clues on how I can progress on any of the mentioned direction or know a completely different way of solving this please let me know. I really hope I can put Nix into more people’s hands and I don’t want to let my colleagues down that I preemptively got quite excited. I hope you can learn something from this too.

1 Like

You can simply pick the code that builds the official nix container and base your own container on it. I have a flake that contains (partly redacted):

inputs = {
   nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
   flake-utils.url = "github:numtide/flake-utils";
   nix = {
     url = "github:NixOS/nix";
     inputs.nixpkgs.follows = "nixpkgs";
   };
  };

outputs = { self, nix, nixpkgs, flake-utils }:

    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        flakeRegistry = {
          nixos = {
            from = { type = "indirect"; id = "nixos"; };
            to = nixpkgs;
            exact = true;
          };
          nixpkgs = {
            from = { type = "indirect"; id = "nixpkgs"; };
            to = nixpkgs;
            exact = true;
          };
        };

        flakeRegistryJSON = (pkgs.formats.json { }).generate "flake-registry.json" {
          version = 2;
          flakes =
            pkgs.lib.mapAttrsToList (n: v: { inherit (v) from to exact; }) flakeRegistry;
        };

        containerInfo = {
          name = "mycontaner";
          tag = nixpkgs.rev;
        };
        flake-input-sources = pkgs.writeTextFile rec { name = "flake-inputs.log"; text = ''${nixpkgs.outPath} ${nix.outPath}''; destination = "/${name}"; };
        # use official nix container as base to get everything for flake-based building
        container-layered = pkgs.lib.throwIfNot (pkgs.stdenv.isLinux) "Docker images are only supported on Linux." (import ("${nix.outPath}" + "/docker.nix") {
          inherit (containerInfo) name tag;
          inherit pkgs;
          bundleNixpkgs=false;
          extraPkgs = [ pkgs.git pkgs.which pkgs.stdenvNoCC ] ++ [ flake-input-sources ];
          maxLayers = 111;
          nixConf = {
            experimental-features = [ "nix-command" "flakes" ];
          };
          flake-registry = flakeRegistryJSON;
        });

      in
      rec {
        devShell = pkgs.mkShell {
          buildInputs = [ run-packages dev-packages ];
          shellHook = '' echo Hello '';
        };
        packages = flake-utils.lib.flattenTree {
          container = container-layered;
        };

to create a flake-enabled Docker image that I use in CI.

1 Like

Thank you so much! This gets me a lot closer to the goal! I tried basing off this image but the best I could figure out was to pull it from Docker Hub.

I do have some questions, however:

  • What is the difference between the official image that you use and the one from docker-nixpkgs?
  • Why is the official image so large (>500Mb) compared to Ubuntu (<80Mb) and the Nix image from docker-nixpkgs (288Mb). And why are both so large actually? I thought one of the benefits of doing containers this way was minimalism. I am able to build minimalist images using Nix but why does including Nix suddenly blow it up so much?
  • How did you figure out that you have to use import ("${nix.outPath}" + "/docker.nix"). I would like to be able to learn how to find this myself so that I don’t have to ask next time and maybe even able to help others. How did you decide to use that instead of (import nix).packages.dockerImage?

I have no idea. :slight_smile: I never looked at docker-nixpkgs - @zimbatm knows more about this.

The image pulls in a lot of dependencies to get the full nix flakes experience, including copies of unpacked nixpkgs-tarballs. These are quite heavy but only needed for running certain nix commands. If one builds a dedicated container that runs a pre-built software those are not required, resulting in a much smaller image.

When a flake input is given the name “x”, the source of the flake is available as x.outPath. I studied docker.nix via the GitHub UI and decided i wanted to access the definitions in this file rather than build on the final artifact.

1 Like

You are awesome! I tried exploring the code of docker-nixpkgs image again and actually got somewhere this time!

I think nix image from docker-nixpkgs values minimalism much more. I really like that it asks to pass all packages manually so you know exactly what software is already in the image before you layer on more.

{
  inputs = {
    docker-nixpkgs = {
      url = "github:nix-community/docker-nixpkgs";
      flake = false;
    };
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs";
  };

  outputs = { self, docker-nixpkgs, flake-utils, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };
      buildImageWithNix = import ("${docker-nixpkgs}" + "/images/nix/default.nix");
    in rec {
      packages.nixImage = buildImageWithNix {
        # All of this is required by the function
        inherit (pkgs)
          dockerTools
          bashInteractive
          cacert
          coreutils
          curl
          gnutar
          gzip
          iana-etc
          nix
          openssh
          xz;

        # We are actually going to use Git so we use the full version.
        gitReallyMinimal = pkgs.git;

        extraContents = with pkgs; [
          # Since we need git in this image let's add git-lfs right away.
          git-lfs
        ];
      };

      packages.default = pkgs.dockerTools.buildImage {
        fromImage = packages.nixImage;
        # ...
      };
    };
}

I think I am going to go with this solution because it is smaller and I can comprehend everything that is happening here (I have no idea what flake registries are and I don’t think I am ready to know yet :grin:).

I think it is clear that the answer to my original question, “How to build a docker image with a working Nix inside it” is “don’t” but another question remains: can this work without root?

With all the complexity that turned up I am ready accept that you have to be root (or use the nix daemon which, in a container, is a lot worse :laughing:). I actually already switched to rootless Docker daemon so there is no real reason for me personally to bother with this anymore but just for the sake of understanding Nix better, there really is no way to use it completely unprivileged?

Quite some time has passed since I settled on a solution that has been working well enough (exactly one season and one day) but I was tinkering with the image again and realized that I could, in fact, very easily start nix daemon and su into a non-root user in a small init script before starting any interactive software, and I think I can now say that my original question has been fully answered.

For those seeking the solution in the future: you should probably not attempt to make Nix work on an empty system yourself and build from an image that already has it configured instead, e.g. the Nix image by NixOS or the Nix image from docker-nixpkgs by nix-community. If you really must build it from scratch use those images to learn or refer to this Installation guide and the official install script and replicate what it does.

To use Nix as a non-root user you will have to set up what the guide refers to as the multi-user mode which configures the system to run nix daemon. Once you are able to run Nix as root this is a very easy extra step: you have to write a small init system (congrats you can now officially become a systemd denier) that starts the daemon and logs you in an a normal user, and set it as the image entrypoint. Don’t worry it can literally be a 2 line shell script, see the following code for reference:

/bin/nix daemon &> /dev/null &
su -l your-normal-user

I hope this helps whoever might find this in the future.

I am marking this post as the solution but I would not have been able to figure this out without @wamserma’s help.

1 Like

May I ask what CI environment you are working on?

For Github it seems that actions will not work with these custom made containers since mounted binary will not be patched: cannot execute: required file not found.

There are open issues to let actions use tools already installed in the container, but right now there isn’t one if you want to use actions with Nix Container. Future reader FYI, if you want to use arc-runner specifically.

My image specifically is a development system so it is designed to be ran locally, just for dependency management and isolation. If you already have Nix natively and don’t care about isolation you don’t need to do what I did there, even if you care about isolation you can probably just use another user, in my case I am trying to get Windows people to run this :smile:.

My company is not using Nix in CI (yet?).

I am not sure why you’d want to run a non-nix app in a Nix system in CI, if you built it for HFS+Linux just test/deploy in the same system you built it in or for.

If you are deploying software than you should probably add the software you are deploying to the image image, if it is already available in Nixpkgs that’s what you want instead of the binary you are trying to mount. You likely also don’t want Nix in that image you want just the software you are trying to deploy.

If you are trying to compile your software in Nix using a flake you probably should just use this image, mount an output dir and cp the result into it.

If you are trying to run the software you are developing yourself in Nix than you have to make sure your software supports the Nix+Linux target instead of patching it.