Podman containers always fail to start

Help
1

Hello, my containers no longer start properly and i can reproduce the problem even with a hello world container.
This is my configuration.nix

...
  virtualisation = {
    podman = {
      enable = true;
      dockerCompat = true;
    };
    oci-containers = {
      backend = "podman";
      containers = {
        hello = {
          image = "hello-world";
        };
      };
    };
  };
...

and this is the outcome:

miguel@margiehamilton nixos (git)[master] % sudo systemctl status podman-hello.service|cat                                                     ~/configs/nixos
â—Ź podman-hello.service
     Loaded: loaded (/nix/store/086bhbxgbi74pg84crar20z6s8x688dr-unit-podman-hello.service/podman-hello.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2021-03-12 01:18:33 UTC; 3min 33s ago
    Process: 956 ExecStartPre=/nix/store/5ha6p04gkdrzmz2gj435889zybqljhnl-unit-script-podman-hello-pre-start/bin/podman-hello-pre-start (code=exited, status=0/SUCCESS)
    Process: 988 ExecStart=/nix/store/9lrsm6dwa1zrh61fi6c4w4mc41dp2772-system-path/bin/podman run --rm --name=hello --log-driver=journald hello-world (code=exited, status=125)
    Process: 1019 ExecStopPost=/nix/store/kx7q1fyj58f68zslhvy12xcbhm0gyf38-unit-script-podman-hello-post-stop/bin/podman-hello-post-stop (code=exited, status=0/SUCCESS)
   Main PID: 988 (code=exited, status=125)
         IP: 0B in, 0B out
        CPU: 830ms

Mar 12 01:18:33 margiehamilton systemd[1]: podman-hello.service: Scheduled restart job, restart counter is at 5.
Mar 12 01:18:33 margiehamilton systemd[1]: Stopped podman-hello.service.
Mar 12 01:18:33 margiehamilton systemd[1]: podman-hello.service: Start request repeated too quickly.
Mar 12 01:18:33 margiehamilton systemd[1]: podman-hello.service: Failed with result 'exit-code'.
Mar 12 01:18:33 margiehamilton systemd[1]: Failed to start podman-hello.service.

Please help, I’m running out of ideas, this was running completely fine up until 2021-03-08, when I did a nixos-rebuild --upgrade switch to update my system (and no other changes…)

Thanks in advance

2

The way I’ve been dealing with this is just to run the container from the command line to see why it is failing.

In this case, run
/nix/store/9lrsm6dwa1zrh61fi6c4w4mc41dp2772-system-path/bin/podman run --rm --name=hello --log-driver=journald hello-world which I took from the ExecStart of the systemd unit file.

1 Like
3

A user in the discord today had a similar issue.

Declarative containers failed with exit code 125, but worked when started “by hand” by copying the ExecStart.

We have been able to eventually figure out that zfs binary was missing from PATH. And it seems as if adding virtualisation.podman.extraPackages = [ zfs ] to the config worked for them.

2 Likes
4

just to let you know I’ve tried this, or the corrected variant:

  virtualisation = {
    podman = {
      enable = true;
      extraPackages = [ pkgs.zfs ];
    };
...

and it still didn’t work.

but I confirm that I’m using ZFS in this system. I’ll now try podman containers on a system without ZFS to see if this can still be the problem…

additionally, my containers run w/ docker, this config can be tested by anyone:

  virtualisation = {
    podman = {
      enable = true;
      extraPackages = [ pkgs.zfs ];
    };
    docker = {
      enable = true;
    };
    oci-containers = {
      backend = "docker";  # switch to podman to reproduce failure
      containers = {
         kostal2influx = {
          image = "quay.io/msf/kostal2influx:v0.2";
          user = "nobody:nogroup";
          extraOptions = ["--network=host"];
        };
      };
    };
  };
1 Like
5

The trick we used to get better/more detailed output from podmanwas that we opened 2 terminals, one we used to systemctl cat and the other we did a export PATH=… with the PATH from the service.

The we used the Exec from the service. This way we have been able to see stdout and stderr of the podman process, which are surpressed by the systemd service by default.

2 Likes