I’m trying to run podman containers via podman-compose with a rootless user (with systemd).
My problem is that for some reason podman containers can’t use networks:
User config:
(normally this is handled by my "podman-wrapper.nix" which creates a systemd service)
users.users.immich = {
isNormalUser = true;
group = immich;
home = "/var/lib/containers/immich";
createHome = true;
description = "User for container immich";
subUidRanges = [ { count = 65536; startUid = 615536; } ];
subGidRanges = [ { count = 65536; startGid = 615536; } ];
linger = true;
};
# Create a group for each container
users.groups.immich = { };
Virtualisation Config:
# Enable container support
virtualisation = {
oci-containers.backend = "podman";
podman = {
enable = true;
# Create a `docker` alias for podman, to use it as a drop-in replacement
dockerCompat = true;
# Required for containers under podman-compose to be able to talk to each other.
defaultNetwork.settings.dns_enabled = true;
};
};
# add podman and podman-compose to the system
environment.systemPackages = with pkgs; [ podman podman-compose runc conmon skopeo slirp4netns fuse-overlayfs ];
execute as immich user:
> podman network create immich
immich
> podman run --network=immich ubuntu
Error: unable to find network with name or ID immich: network not found
even though:
> podman network ls
707ab6184bb2 immich bridge
On my main user this works fine though:
> podman network create immich
immich
> podman run --network=immich ubuntu
(no output)
> podman run --log-level debug --network=doesnotexist ubuntu
(...)
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading default seccomp profile
Error: unable to find network with name or ID doesnotexist: network not found
DEBU[0000] Shutting down engines
for the existing immich network:
(...)
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading default seccomp profile
DEBU[0000] Allocated lock 29 for container 78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/immich/.local/share/containers/storage+/tmp/containers-user-1002/containers]@a50ab9f167975489853cbffd2be3bcadab3a9da27faf390ac48603c60d5c59e7"
DEBU[0000] exporting opaque data as blob "sha256:a50ab9f167975489853cbffd2be3bcadab3a9da27faf390ac48603c60d5c59e7"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819"
DEBU[0000] Container "78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819" has work directory "/var/lib/containers/immich/.local/share/containers/storage/overlay-containers/78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819/userdata"
DEBU[0000] Container "78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819" has run directory "/tmp/containers-user-1002/containers/overlay-containers/78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819/userdata"
DEBU[0000] Not attaching to stdin
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/immich/.local/share/containers/storage/overlay/l/HETPDNQ2AYSBKKWROSYXF7IZYK,upperdir=/var/lib/containers/immich/.local/share/containers/storage/overlay/215c045468aa47f49708f44097ff615ba72f063921eb791ab564aa14e6630bdd/diff,workdir=/var/lib/containers/immich/.local/share/containers/storage/overlay/215c045468aa47f49708f44097ff615ba72f063921eb791ab564aa14e6630bdd/work,,userxattr
DEBU[0000] Mounted container "78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819" at "/var/lib/containers/immich/.local/share/containers/storage/overlay/215c045468aa47f49708f44097ff615ba72f063921eb791ab564aa14e6630bdd/merged"
DEBU[0000] Created root filesystem for container 78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819 at /var/lib/containers/immich/.local/share/containers/storage/overlay/215c045468aa47f49708f44097ff615ba72f063921eb791ab564aa14e6630bdd/merged
INFO[0000] Received shutdown.Stop(), terminating! PID=230820
DEBU[0000] Made network namespace at /run/user/1002/netns/netns-2955ed7e-c381-ece5-3145-5948d1b61e8c for container 78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819
DEBU[0000] The path of /etc/resolv.conf in the mount ns is "/etc/resolv.conf"
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Unmounted container "78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819"
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Cleaning up container 78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container 78ec3e42be623352dc30a49a2b9ef4ad07882c69da24adcfd21e3ef23f32d819 storage is already unmounted, skipping...
DEBU[0000] ExitCode msg: "unable to find network with name or id immich: network not found"
Error: unable to find network with name or ID immich: network not found
DEBU[0000] Shutting down engines
And for a server (running oracle linux) where this actually works:
(...)
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] Allocated lock 6 for container 1ee0041191bf322e33ea432c974170e8d3017f7d23faf5e05b39d637f2a091d4
DEBU[0000] parsed reference into "[overlay@/home/test/.local/share/containers/storage+/tmp/containers-user-1008/containers]@a50ab9f167975489853cbffd2be3bcadab3a9da27faf390ac48603c60d5c59e7"
DEBU[0000] exporting opaque data as blob "sha256:a50ab9f167975489853cbffd2be3bcadab3a9da27faf390ac48603c60d5c59e7"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "1ee0041191bf322e33ea432c974170e8d3017f7d23faf5e05b39d637f2a091d4"
DEBU[0000] Container "1ee0041191bf322e33ea432c974170e8d3017f7d23faf5e05b39d637f2a091d4" has work directory "/home/test/.local/share/containers/storage/overlay-containers/1ee0041191bf322e33ea432c974170e8d3017f7d23faf5e05b39d637f2a091d4/userdata"
DEBU[0000] Container "1ee0041191bf322e33ea432c974170e8d3017f7d23faf5e05b39d637f2a091d4" has run directory "/tmp/containers-user-1008/containers/overlay-containers/1ee0041191bf322e33ea432c974170e8d3017f7d23faf5e05b39d637f2a091d4/userdata"
DEBU[0000] Not attaching to stdin
INFO[0000] Received shutdown.Stop(), terminating! PID=827436
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/home/test/.local/share/containers/storage/overlay/l/JRUN4AHTZWVMT6BSKR2HDNV5T3,upperdir=/home/test/.local/share/containers/storage/overlay/464684a1c6f243aa8ca18def07a8de878e91ce16d98a177d6031d9172d8e0ebd/diff,workdir=/home/test/.local/share/containers/storage/overlay/464684a1c6f243aa8ca18def07a8de878e91ce16d98a177d6031d9172d8e0ebd/work,,userxattr,context="system_u:object_r:container_file_t:s0:c124,c416"
DEBU[0000] Mounted container "1ee0041191bf322e33ea432c974170e8d3017f7d23faf5e05b39d637f2a091d4" at "/home/test/.local/share/containers/storage/overlay/464684a1c6f243aa8ca18def07a8de878e91ce16d98a177d6031d9172d8e0ebd/merged"
DEBU[0000] Made network namespace at /run/user/1008/netns/netns-0b7e22cc-8c4e-fed4-5e4a-eef0bc826dbd for container 1ee0041191bf322e33ea432c974170e8d3017f7d23faf5e05b39d637f2a091d4
DEBU[0000] creating rootless network namespace with name "rootless-netns-b61e7d0397c25bfae43a"
DEBU[0000] Created root filesystem for container 1ee0041191bf322e33ea432c974170e8d3017f7d23faf5e05b39d637f2a091d4 at /home/test/.local/share/containers/storage/overlay/464684a1c6f243aa8ca18def07a8de878e91ce16d98a177d6031d9172d8e0ebd/merged
DEBU[0000] slirp4netns command: /bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -r 3 --netns-type=path /run/user/1008/netns/rootless-netns-b61e7d0397c25bfae43a tap0
DEBU[0000] The path of /etc/resolv.conf in the mount ns is "/etc/resolv.conf"
[DEBUG netavark::network::validation] "Validating network namespace..."
[DEBUG netavark::commands::setup] "Setting up..."
[INFO netavark::firewall] Using iptables firewall driver
[DEBUG netavark::network::bridge] Setup network immich
(...)