Initdb (postgres) installed inside dockerTools.buildImage fails when run in docker, but works when run on host

rzetterberg · August 16, 2019, 12:14pm

I have a project that currently use the official postgres docker image that I’m changing to building my own image using dockerTools.buildImage to get more control over what is put inside the image and because I prefer using nix over Dockerfiles.

The problem I have is that the following entrypoint fails:

    #!${stdenv.shell}

    ${coreutils}/bin/id

    echo "PGDATA: $PGDATA"

    ${coreutils}/bin/ls -la /bin/postgres
    ${file}/bin/file /bin/postgres

    ${coreutils}/bin/ls -la /bin/initdb
    ${file}/bin/file /bin/initdb

    postgres -V
    initdb -U ${user}

Where contents = [ postgresql_11 ]; is set. The result is the following output:

uid=999(postgres) gid=999(postgres) groups=999(postgres)
PGDATA: /postgres/data
-r-xr-xr-x 1 root root 8182360 Jan  1  1970 /bin/postgres
/bin/postgres: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
-r-xr-xr-x 1 root root 291 Jan  1  1970 /bin/initdb
/bin/initdb: a /nix/store/cinw572b38aln37glr0zb8lxwrgaffl4-bash-4.4-p23/bin/bash -e script, ASCII text executable
postgres (PostgreSQL) 11.2
no data was returned by command ""/bin/postgres" -V"
The program "postgres" is needed by initdb but was not found in the
same directory as "/bin/initdb".

As you can see:

postgres is in the same directory as initdb
postgres is an executable file
postgres has execution permissions for anyone
When run postgres produces the correct version

The interesting thing is that if I would put the initdb command in runAsRoot instead, so that it is done while the image is being created inside the VM-based builder, I encounter the same problem.

If I use the specific package that is put inside the Docker image and run it directly on my host, it works as expected.

I can’t for the life of me understand why this happens. I even looked at the source code of initdb and strace. The only thing I see is that “postgres -V” is being run but the receiving end of the pipe is just empty.
I’ve tried Postgres 9.6 with the same result.

Here’s the specific code in initdb that does the check that fails:

I can provide a fully reproducible script if anyone is interested in recreating the problem locally.

Any ideas or suggestions are greatly appreciated!

manveru · August 16, 2019, 12:54pm

Please provide the derivation, it’d be quite helpful for figuring this out

rzetterberg · August 17, 2019, 6:56am

Thanks for the reply!

I managed to reduce the problem even more. Basically I took the offending code from initdb and built an executable that only tries to run “postgres -V” with popen. I also started reading up on how popen and similar functions work and found out that there are at least 2 more ways to run a command from C:

system
execve

So I built 3 executables that use those functions to see if there is a problem with how the command is executed.

Here are the files:

default.nix

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

let
  workingDir = "/postgres";

  entrypoint = writeScript "entrypoint.sh" ''
    #!${stdenv.shell}

    echo ">> Running \"postgres -V\" directly:"
    /bin/postgres -V

    ./test1-popen
    ./test2-system
    ./test3-execve
  '';
in
  dockerTools.buildImage {
    name = "rzetterberg/docker-problem";
    tag  = "latest";

    runAsRoot = ''
      #!${stdenv.shell}

      mkdir ${workingDir}
      cd ${workingDir}

      ${gcc}/bin/gcc -o test1-popen ${./test1-popen.c} 
      ${gcc}/bin/gcc -o test2-system ${./test2-system.c} 
      ${gcc}/bin/gcc -o test3-execve ${./test3-execve.c} 
    '';

    contents = [ postgresql_11 ];

    config = {
      Entrypoint = [ entrypoint ];

      WorkingDir = workingDir;
    };
  }

test1-popen.c

#include <stdio.h>
#include <errno.h>

#define MAX_SIZE 1024

int main() {
  fprintf(stdout, ">> Test1 popen\n");

  FILE *pgver;

  char line[MAX_SIZE];

  fflush(stdout);
  fflush(stderr);

  errno = 0;

  char cmd[] = "/bin/postgres -V";

  if ((pgver = popen(cmd, "r")) == NULL) {
    perror("popen failure");
    return -1;
  }

  errno = 0;

  if (fgets(line, sizeof(line), pgver) == NULL) {
    if (feof(pgver)) {
      fprintf(stderr, "no data was returned by command \"%s\"\n", cmd);
    } else {
      perror("fgets failure");
    }

    pclose(pgver);

    return -2;
  }

  fprintf(stdout, "OK, got output: \"%s\"\n", line);

  return 0;
}

test2-system.c

#include <stdio.h>
#include <stdlib.h>

int main() {
  fprintf(stdout, ">> Test2 system\n");

  fflush(stdout);
  fflush(stderr);

  char cmd[] = "/bin/postgres -V";

  int result = system(cmd);

  if (result != 0) {
    fprintf(stderr, "system \"%s\" failed with: %d\n", cmd, result);
    return -1;
  }

  fprintf(stdout, "OK");

  return 0;
}

test3-execve.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>

int main() {
  fprintf(stdout, ">> Test3 execve\n");

  fflush(stdout);
  fflush(stderr);

  errno = 0;

  char cmd[] = "/bin/postgres";
  char *const args[] = {"/bin/postgres", "-V", (char*)0};

  int result = execve(cmd, args, NULL);

  if (result != 0) {
    perror("execve failed");
    return -1;
  }

  fprintf(stdout, "OK");

  return 0;
}

Putting all these files in the same directory and running the following:

nix-build ./default.nix
docker load < ./result
docker run rzetterberg/docker-problem:latest

Produces the following:

Loaded image: rzetterberg/docker-problem:latest
>> Running "postgres -V" directly:
postgres (PostgreSQL) 11.3
>> Test1 popen
no data was returned by command "/bin/postgres -V"
>> Test2 system
system "/bin/postgres -V" failed with: 32512
>> Test3 execve
postgres (PostgreSQL) 11.3

So now we see that:

popen fails
system fails
execve is successful

I tried using other commands than postgres -V and I see the same behaviour there. So I believe the problem has nothing to do with postgres, but with how the shell is setup and how it runs commands.

I also tried compiling and running these executables (with the path to postgres changed to /home/rzetterberg/.nix-profile/bin/postgres) on the host where all 3 works as expected.

I also found this in the man pages for system (see " Caveats"):

Do not use system() from a privileged program (a set-user-ID or set-
group-ID program, or a program with capabilities) because strange
values for some environment variables might be used to subvert system
integrity. For example, PATH could be manipulated so that an arbi‐
trary program is executed with privilege. Use the exec(3) family of
functions instead, but not execlp(3) or execvp(3) (which also use the
PATH environment variable to search for an executable).

system() will not, in fact, work properly from programs with set-
user-ID or set-group-ID privileges on systems on which /bin/sh is
bash version 2: as a security measure, bash 2 drops privileges on
startup. (Debian uses a different shell, dash(1), which does not do
this when invoked as sh.)

Any user input that is employed as part of command should be care‐
fully sanitized, to ensure that unexpected shell commands or command
options are not executed. Such risks are especially grave when using
system() from a privileged program.

Maybe the same can be said for popen?

lewo · August 19, 2019, 7:49am

Did you try to run ${postgre}/bin/initdb instead of initdb?
You could also look in the postgre nixos module what is in the PATH.

rzetterberg · August 19, 2019, 8:20am

Thanks for the suggestions!

I have tried using ${postgres}/bin/initdb and I encountered the same problem.

I have not tried looking at how PATH is setup in dockerTools.buildImage, I just assumed everything that was put in contents ended up in /bin and that /bin was added to PATH.

I’ll try and see if adding ${postgres}/bin to PATH (same as the PostgreSQL module) makes a difference

rzetterberg · August 19, 2019, 2:23pm

I changed PATH so that it only was pointing to ${postgres}/bin and then ran the initdb command, like so:

PATH=${postgresql_11}/bin; initdb -U postgres -D ./data

But I was faced with the same problem:

no data was returned by command ""/nix/store/pbd3wxm8xxdcrwl5gs1jv7iskdnyyjp0-postgresql-11.2/bin/postgres" -V"
The program "postgres" is needed by initdb but was not found in the
same directory as "/nix/store/pbd3wxm8xxdcrwl5gs1jv7iskdnyyjp0-postgresql-11.2/bin/initdb".

Just to be sure, I ran the command directly, like I said in the previous post:

${postgresql_11}/bin/initdb -U postgres -D ./data

This also gave me the same result as last time:

no data was returned by command ""/nix/store/pbd3wxm8xxdcrwl5gs1jv7iskdnyyjp0-postgresql-11.2/bin/postgres" -V"
The program "postgres" is needed by initdb but was not found in the
same directory as "/nix/store/pbd3wxm8xxdcrwl5gs1jv7iskdnyyjp0-postgresql-11.2/bin/initdb".
Check your installation.

rzetterberg · August 19, 2019, 2:48pm

Alright, I found the solution.

Basically popen and system fails when there’s no shell installed at /bin/sh.

I looked at the official postgres:11 and postgres:11-alpine, they both have bash installed at /bin/sh.

After making the following change in default.nix:

-     contents = [ postgresql_11 ];
+     contents = [ bash postgresql_11 ];

All my test cases worked:

>> Running "postgres -V" directly:
postgres (PostgreSQL) 11.2
>> Test1 popen
OK, got output: "postgres (PostgreSQL) 11.2
"
>> Test2 system
postgres (PostgreSQL) 11.2
OK>> Test3 execve
postgres (PostgreSQL) 11.2