Rootless podman and shadow / newuidmap

glenndavy · February 22, 2025, 8:59am

Hi there

I’m trying to get a rootless podman running. When I do I get the error on rebuild-switch that 'Error: command required for rootless mode with multiple IDs: exec: “newuidmap”: executable not found in path".

I’ve included shadow in the packages, done everything I can to ensure that the systemd environment has newuidmap in the path, but I can’t get past this. I suspect I’m missing something fundamental, but I have no clue what.

Any ideas? (this is on nixos, not just nixpkgs)

kfiad · February 24, 2025, 7:43pm

Got the same problem. Seems like podman does not look into /run/wrappers/bin/ where newuidmap and newgidmap wrappers with setuid are.

This:

{
  virtualisation.podman = {
    enable = true;
    // ...
    extraPackages = [
      pkgs.shadow
    ];
  };
}

does not help.

I’m still looking for the fix…

glenndavy · February 24, 2025, 7:56pm

thanks kfiad, good to know I’m not special - i explicitly forced the path to include the wrappers, but didn’t make a difference.

kfiad · February 24, 2025, 10:20pm

Yes, there are two of us, @glenndavy
While trying to find a solution/workaround I have discovered strange default PATH set by systemd for service units (nevermind the error, reformatted for readibility):

error: The option systemd.services.podman.environment.PATH has conflicting definition values:
In /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix: 
/nix/store/bgmaaz4g83qnn3is8c8z43avp8qacddg-shadow-4.16.0/bin
:/run/wrappers/bin/bin
:/nix/store/b1wvkjx96i3s7wblz38ya0zr8i93zbc5-coreutils-9.5/bin
:/nix/store/w8pnfazxqwmrqmwkb5zrz1bifsd8abxl-findutils-4.10.0/bin
:/nix/store/85amyk92rg19l4fy0qmy7wr4jmq8p5z0-gnugrep-3.11/bin
:/nix/store/vmby25zqxnx94awvj0k7zn0rsd0zyrwg-gnused-4.9/bin
:/nix/store/3abwqv1a1bdycmgaydzfw3a0qzxwk8am-systemd-256.8/bin
:/nix/store/bgmaaz4g83qnn3is8c8z43avp8qacddg-shadow-4.16.0/sbin
:/run/wrappers/bin/sbin
:/nix/store/b1wvkjx96i3s7wblz38ya0zr8i93zbc5-coreutils-9.5/sbin
:/nix/store/w8pnfazxqwmrqmwkb5zrz1bifsd8abxl-findutils-4.10.0/sbin
:/nix/store/85amyk92rg19l4fy0qmy7wr4jmq8p5z0-gnugrep-3.11/sbin
:/nix/store/vmby25zqxnx94awvj0k7zn0rsd0zyrwg-gnused-4.9/sbin
:/nix/store/3abwqv1a1bdycmgaydzfw3a0qzxwk8am-systemd-256.8/sbin

Note the /run/wrappers/bin/bin and /run/wrappers/bin/sbin entries.
I do not know where do they came from, but there are no such subdirectories in my NixOS, there is only /run/wrappers/bin.

eblechschmidt · February 25, 2025, 8:28am

This is indeed strange. The /run/wrappers is added here:

github.com

NixOS/nixpkgs/blob/04ef94c4c1582fd485bbfdb8c4a8ba250e359195/nixos/modules/virtualisation/podman/default.nix#L11


      
          { config, lib, pkgs, ... }:
          let
            cfg = config.virtualisation.podman;
            json = pkgs.formats.json { };
          
            inherit (lib) mkOption types;
          
            podmanPackage = pkgs.podman.override {
              extraPackages = cfg.extraPackages
                # setuid shadow
                ++ [ "/run/wrappers" ]
                ++ lib.optional (config.boot.supportedFilesystems.zfs or false) config.boot.zfs.package;
              extraRuntimes = [ pkgs.runc ]
                ++ lib.optionals (config.virtualisation.containers.containersConf.settings.network.default_rootless_network_cmd or "" == "slirp4netns") (with pkgs; [
                slirp4netns
              ]);
            };
          
            # Provides a fake "docker" binary mapping to podman
            dockerCompat = pkgs.runCommand "${podmanPackage.pname}-docker-compat-${podmanPackage.version}"
              {

This is added to the package here by just adding /bin to it.

github.com

NixOS/nixpkgs/blob/04ef94c4c1582fd485bbfdb8c4a8ba250e359195/pkgs/by-name/po/podman/package.nix#L53


      
          binPath = lib.makeBinPath (
            lib.optionals stdenv.hostPlatform.isLinux [
              fuse-overlayfs
              util-linux
              iptables
              iproute2
            ]
            ++ lib.optionals stdenv.hostPlatform.isDarwin [
              vfkit
            ]
            ++ extraPackages
          );
          
          helpersBin = symlinkJoin {
            name = "podman-helper-binary-wrapper";
          
            # this only works for some binaries, others may need to be added to `binPath` or in the modules
            paths =
              [
                gvproxy
              ]

There should not be a need to add any extra packages for this to work.

Are you using a different kind of package?

kfiad · February 25, 2025, 9:13am

I am using podman from 24.11 stable, no modifications.

This error happens when using podman as executor for Laminar jobs. I wonder if the source of error lies in laminar package, not podman itself.

Found it!
I’ve checked my laminar.nix system package configuration, where I declared systemd unit service for Laminar:

{
  systemd.services.laminar = {
    # (...)
    path = [
      pkgs.bash
      pkgs.wget
      pkgs.git
      pkgs.openssh
      pkgs.podman
      pkgs.rsync
      pkgs.gnumake
    ];
  };
}

Yesterday my trial-and-error way included adding "/run/wrappers/bin" to the path definitions above, which did not help.
Today I have added "/run/wrappers" (no trailing /bin) and this fixed the issue with no newuidmap executable available in $PATH.

I remember that yesterday, when reading whole Internet, I have read that adding package to systemd.services.<service>.path makes both /bin and /sbin sub-paths of this package added to the $PATH. From my point of view it is confusing that giving a string (here "/run/wrappers") follows the same procedure.

Do you think it is worth discussion with some mainstream folks?

Anyway, thank you for your interest & time.

eblechschmidt · February 25, 2025, 12:29pm

Don’t quote me on anything as I would consider myself a novice NixOS user. I think the reason is that pkgs.something will - when interpreted as a string - resemble the store path where the content of that package will be found. As a convention executables should end up in /bin of that store path hence adding this makes sense so you can use pkgs.sonething directly in those lists without the need to manually add anything.

kfiad · February 25, 2025, 2:48pm

You’re quite right.

I thought that – to prevent user’s confusion – there could be two ways of dealing with elements in path, following this pseudo-algorithm:

if ELEMENT is of pkgs.<name> type: instead of adding path to the package add two paths, one to <pkgPath>/bin and second to <pkgPath>/sbin subdirectories. Done.

if ELEMENT is of string type: use it “as-is”. Done.

Or maybe this is something not worth discussing

eblechschmidt · February 25, 2025, 5:29pm

I get your point and I also think it would be easier from a user perspective. I have no idea if that is worth discussing. One downside would be that it is not backward compatible…