Podman and Docker fail to mount

Help
1

Hi all, unfortunately, I’m having problems similar to Podman on 20.09 fails to find fuse-overlayfs. Docker and Podman don’t want to start because of a mount problem.

$ (sudo) docker run -it --rm alpine:latest # Same error for sudo and non-sudo.
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
ba3557a56b15: Extracting  2.812MB/2.812MB
docker: failed to register layer: Failed to mount; dmesg: <6>[  127.377333] refused connection: IN=eno1 OUT= MAC=1c:69:7a:68:a1:13:e0:63:da:70:d1:e8:08:00 SRC=192.168.1.100 DST=192.168.1.105 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50222 PROTO=TCP SPT=56486 DPT=8080 WINDOW=29200 RES=0x00 SYN URGP=0
: mount /dev/mapper/docker-259:1-47710300-8dc9af1b5843f67ee72127c30b1dfc7c0710d08534420c86e83f5bcd8038fe8c:/var/lib/docker/devicemapper/mnt/8dc9af1b5843f67ee72127c30b1dfc7c0710d08534420c86e83f5bcd8038fe8c, data: nouuid: no such device.
See 'docker run --help'.

$ podman run -it --rm alpine:latest
ERRO[0000] error unmounting /home/rik/.local/share/containers/storage/overlay/b043f6e287d22a0011b673818093daa2fe392652e6be18dce29b0192c8f35ebe/merged: invalid argument
Error: error mounting storage for container 620c74f7b4996ee27fb8b138b9b1df3f2db9aea3ac87b0c9cbb4c6364d2d276c: error creating overlay mount to /home/rik/.local/share/containers/storage/overlay/b043f6e287d22a0011b673818093daa2fe392652e6be18dce29b0192c8f35ebe/merged: using mount program /nix/store/6vf40qi2r1y1zv48dcgv2kak8z85243x-fuse-overlayfs-1.1.2/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such device
: exit status 1

$ sudo podman run -it --rm alpine:latest
ERRO[0000] [graphdriver] prior storage driver overlay failed: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver

$ (sudo) modprobe fuse # Same error for sudo and non-sudo.
modprobe: ERROR: could not insert 'fuse': Operation not permitted

$ sudo modprobe overlay
modprobe: ERROR: could not insert 'overlay': Operation not permitted

$ strace -f -o /tmp/trace podman run -it alpine
ERRO[0000] error unmounting /home/rik/.local/share/containers/storage/overlay/2b389eaa8e050caf7f37b89b9765220c641ebe87cd458dd5abb7f4393affa4a4/merged: invalid argument
Error: error mounting storage for container dda2156c48169cb5cb2e11a344d4750df990a65076021cb1ee42fe4f39966e9c: error creating overlay mount to /home/rik/.local/share/containers/storage/overlay/2b389eaa8e050caf7f37b89b9765220c641ebe87cd458dd5abb7f4393affa4a4/merged: using mount program /nix/store/6vf40qi2r1y1zv48dcgv2kak8z85243x-fuse-overlayfs-1.1.2/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such device

It’s on a fairly new installation with

users.users.rik = {
  ...
  extraGroups = [ "disk" "docker" "wheel" "networkmanager" ];
};

...

environment.systemPackages = with pkgs; [
  fuse-overlayfs
];

virtualisation.docker.enable = true;
virtualisation.podman.enable = true;

system.stateVersion = "20.09";

The hardware configuration contains

boot.initrd.availableKernelModules = [ "xhci_pci" "ahci" "nvme" "usbhid" "usb_storage" "s  d_mod" "rtsx_pci_sdmmc" ];
boot.initrd.kernelModules = [ "dm-snapshot" ];
boot.kernelModules = [ "kvm-intel" ];
2

have you seen/checked Podman - NixOS Wiki

  • but for me it’s working without explicit subUidRanges and or environment.etc (but I’m on 20.03 upgraded to channel 20.09)
3

Not recently no. I see that it contains some more instructions. Should I add all those and see whether it works then? I would find it odd since those settings are a part of the mentioned PR (nixos/virtualisation.podman: Init module by adisbladis · Pull Request #85604 · NixOS/nixpkgs · GitHub).

4

For me, on 20.09 and unstable, it just requires setting virtualization.podman.enable to get podman working

5

For me now too, the problem was that I had set some security settings which interfered with Docker.

Specifically, it was one of these settings:

security.lockKernelModules = true;
security.protectKernelImage = true;

security.forcePageTableIsolation = true;

# This is required by podman to run containers in rootless mode.
security.unprivilegedUsernsClone = config.virtualisation.containers.enable;

security.apparmor.enable = true;

boot.blacklistedKernelModules = [
        # Obscure network protocols
        "ax25"
        "netrom"
        "rose"

        # Old or rare or insufficiently audited filesystems
        "adfs"
        "affs"
        "bfs"
        "befs"
        "cramfs"
        "efs"
        "erofs"
        "exofs"
        "freevxfs"
        "f2fs"
        "hfs"
        "hpfs"
        "jfs"
        "minix"
        "nilfs2"
        "qnx4"
        "qnx6"
        "sysv"
        "ufs"
]