dockerTools.buildImage and user-writable /tmp

Help
1

I’m trying to put together a Docker image using dockerTools.buildImage but I’m stuck on creating a /tmp that all users can use.

  • I need to run xvfb-run in the container, and it needs write access /tmp (it creates a folder /tmp/xvfb-run.XXXXXX for its temporary files).
  • The program run “inside” xvfb-run does not like being run as root.

I’m failing to combine these two things.

The derivation I’m trying to use is

  theImage = dockerTools.buildImage {
    name = "foo";
    tag = "latest";
    created = "now";

    contents = [thePkg
                xvfb_run
                bash
                coreutils
                (runCommand "tmp-dir" {} ''mkdir -p $out/tmp; chmod ugo=rwx $out/tmp'')
               ];

    config = {
      Cmd = ["/bin/xvfb-run" "-d" "/bin/foo"];
      WorkingDir = "/bin";
      User = "1000:1000";
      ExposedPorts = {
        "3000" = {};
        "4444-4473" = {};
      };
    };
  };

After creating the image I can start it and observe that /tmp isn’t the desired drwxrwxrwx but rather dr-xr-xr-x and the non-root user can’t create dirs there:

bash-4.4$ cd /tmp
bash-4.4$ mkdir foobar
mkdir: cannot create directory 'foobar': Permission denied

How do I create /tmp in the correct way?

1 Like
2

I have no experience with building docker images through nix, but you’re setting wrong permissions on your /tmp.
The correct permissions for /tmp are:
$ LANG=C stat /tmp
File: /tmp
Size: 520 Blocks: 0 IO Block: 4096 directory
Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root)

That small t instead of the last x is very important. Try chmod 1777 after creating your directory instead of ugo=…

3

Yes, you are correct, what I want is 1777, but it doesn’t matter, because the permission in the container is still not what I expect:

bash-4.4$ stat /tmp
  File: /tmp
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 3bh/59d Inode: 26217699    Links: 2
Access: (0555/dr-xr-xr-x)  Uid: (    0/ UNKNOWN)   Gid: (    0/ UNKNOWN)
Access: 2020-01-08 14:20:57.163131295 +0000
Modify: 1970-01-01 00:00:01.000000000 +0000
Change: 2020-01-08 14:20:56.943131297 +0000
4

It looks like the directory is created in the nix-store, which is always read-only.
I don’t know enough about the dockerTools to provide a good answer, but one work-around I can think of is defining a volume to be mounted at /tmp in the container.

5

Unfortunately that isn’t an option for me, as the container has to stand on its own – it’s to be run in AWS ECS.

6

Maybe run the following commands during the start of the container (if that’s possible)?

$ mkdir -p /tmp
$ mount -t tmpfs tmp /tmp
7

Obviously untested, but might show you what I suggest:

Cmd = pkgs.writeScript "container-init" ''
#!${stdenv.shell}
set -euo pipefail
mkdir -p /tmp
mount -t tmpfs tmp /tmp
su -c "/bin/xvfb-run -d /bin/foo" - username
'';
8

Also using ECS, we should exchange ideas/solutions.

For this, i had luck with creating an entrypoint that did a lot of the setup that would allow builds in the container. I also wanted to avoid the QEMU build step.

https://gist.github.com/tomberek/6b601e78980af88ca523ce95a5b360fa

My other approach i’m experimenting with is something like:

FROM scratch
COPY nix /bin/nix
COPY busybox /bin/busybox
COPY busybox /bin/sh
COPY busybox /usr/bin/env
COPY ca-bundle.crt /etc/ssl/certs/ca-bundle.crt
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
RUN /bin/busybox ln -sf /bin/busybox /bin/ln
RUN ln -sf /bin/busybox /bin/sh && \
    ln -sf /bin/busybox /bin/sed && \
    ln -sf /bin/busybox /bin/tr && \
    ln -sf /bin/busybox /bin/date && \
    ln -sf /bin/busybox /bin/head && \
    ln -sf /bin/busybox /bin/tar && \
    ln -sf /bin/busybox /bin/hexdump && \
    ln -sf /bin/busybox /bin/mkdir && \
    ln -sf /bin/busybox /bin/rm && \
    ln -sf /bin/busybox /bin/uname && \
    ln -sf /bin/busybox /bin/chmod && \
    ln -sf /bin/busybox /bin/ls && \
    ln -sf /bin/busybox /usr/bin/env

COPY passwd /etc/passwd
COPY group /etc/group
WORKDIR /
ENV PATH="/bin"
RUN mkdir -p /tmp

COPY extra-roots.crt extra-roots.crt
ENV CMD="python -m service"
ENV SUBSTITUTERS="https://arm.cachix.org?trusted=1"
CMD nix run $ENV_PATH --substituters "$SUBSTITUTERS" -c $CMD

where nix is the static nix from https://matthewbauer.us/nix and a pkgsStatic.busybox

This is basically a generic container, where if you feed it paths and cmds, will fetch them from any stores it knows about and can get to, then nix run's it. Kinda bypasses the normal docker mechanism. Then you can bind mount /nix to the host machine, and all the other containers on the machine share the store. (it’s why i’m making a non /nix environment with busybox)

9

If you want custom file attributes, another option is to use the extraCommands parameter like this:

 theImage = dockerTools.buildImage {
    name = "foo";
    tag = "latest";
    created = "now";

    extraCommands = "mkdir -m 0777 tmp";

    contents = [thePkg
                xvfb_run
                bash
                coreutils
               ];

    config = {
      Cmd = ["/bin/xvfb-run" "-d" "/bin/foo"];
      WorkingDir = "/bin";
      User = "1000:1000";
      ExposedPorts = {
        "3000" = {};
        "4444-4473" = {};
      };
    };
  };

EDIT: note that the file is tmp and not /tmp. All the files relative to the current folder will later be packed into the image.

5 Likes
10

Thanks @zimbatm, that answer got me reading the docs for buildImage again, and apparently another option is to use the runAsRoot parameter.

11

Under what environment is extraCommands executed? What commands are available? And if I need to use a special command do I have to use ${package}/bin/command?