buildFHSUserEnv package inside a docker image

Help
1

I’m trying to set up a buildFHSUserEnv-powered package that I can run the same way while developing my app and then package it up in a docker container, so I can run that on my server. The local set up works perfectly. Running the docker container though crashes with a weird create_tmpdir error (full error log below).

Is that scenario even supported? Can FHS env packages work in a docker/chroot environment? Here’s a small expression that reproduces the issue with just a copy of bash.

{
  pkgs ? import <nixpkgs> {}
}:
let
  bash-env = pkgs.buildFHSUserEnv rec {
    name = "bash-env";

    runScript = "bash";

    targetPkgs = pkgs: (with pkgs; [
      bashInteractive
    ]);
  };
in
  pkgs.dockerTools.buildImage {
    name = "bash-fhs";
    tag = "latest";
    contents = bash-env;

    config = {
      Cmd = [ "/bin/bash-env" ];
      WorkingDir = "/";
    };
  }

I build that with a simple nix-build bash.nix command and import it in my local docker with docker load <./result. Running it fails with the error above:

❯ docker run -it ca97872140be /bin/bash-env

** (process:1): ERROR **: 06:06:13.081: create_tmpdir: !g_mkdtemp_full(prefix, 0755): No such file or directory

Any ideas? What am I missing here?

2

buildFHSUserEnv depends on mount namespaces (and maybe some other namespaces) which are not available inside of Docker by default. This is used to put the program in that fake Ubuntu-like environment where the filesystem looks “normal”.

I think that adding the --privileged flag to the docker command should be enough to work around that. Just be careful as this also open new holes in your production environment. If that works it’s possible to be more selective using the --cap-add CAPABILITY flag, see Docker run reference | Docker Documentation

1 Like
3

Modern docker deployments are also locked down further with seccomp so I also recommend reading up on it: Seccomp security profiles for Docker | Docker Documentation

And sometimes also with SELinux: SELinux and docker notes | Ozznotes

1 Like
4

Thinking about it a bit more. Since the only program that is going to be executed is that sandboxed one, I think the best would be if the FHS root was somehow added to the docker image root directly. That would avoid having those mount namespaces issues entirely. To do that you would have to read up on the implementation of buildFHSUserEnv to see if the root filesystem can be accessed and composed into the docker image.

1 Like
5

Thanks @zimbatm! That’s the piece I’ve been missing:

Docker noob here – I wasn’t aware bind mounts weren’t available there. :frowning:

I think I have several options:

  1. Read up on buildFHSUserEnv and maybe mimic its implementation to build something that brings “up” its root to the root of the docker image, as you suggested.
  2. Rework my expression so that it doesn’t use buildFHSUserEnv.

I’m lucky enough that option 2 above is easy in my case, and I’ll probably go for that. I was using buildFHSUserEnv because I was going for perfect :angel: reuse of a third party script relying on several root-based paths being always available. I sort of control that script and I can just patch it in my derivation so that it works with my paths instead.

6

HI, I have the same issue now, have you solved your problem? and is there anything like buildFHSUserEnvDocker?

7

As said before in the thread, buildFHSUserEnv creates a script that uses some kernel features that require permissions docker intentionally doesn’t have. You should not use buildFHSUserEnv in a docker container.

I don’t think a library function that creates a docker container with a similar mimicked environment exists yet. If you’re interested in writing one, I think using the bubblewrap user env builder as a template might be helpful: nixpkgs/default.nix at f7f0f81cf615d4b087b9974675b93d5d18236616 · NixOS/nixpkgs · GitHub

@gutalax seem to have “fixed” their problem by rewriting the application they were using so it doesn’t need buildFHSUserEnv in the first place. This is probably the best option, if at all possible :slight_smile:

2 Likes