Running OCI-Containers in a NixOS containers

Hi folks,

I have a NixOS container with some services wired up and namespaced in its own network. I was hoping to extend this container to run oci-containers with either docker or podman (either is fine) although none of those is able to start due to permissions.

Specifically, docker stops with:

dockerd[724]: failed to start daemon: Devices cgroup isn't mounted

while podman similarly with:

Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded.
Error: create keyring `<>`: Operation not permitted: OCI permission denied

The containers are for software that’s not yet available in nixpkgs and also as an experiment to see if this kind of nesting would work.

Has anyone tried this before?

I never did that but I was a bit intrigued so I made a search and I end up on this page which seems helpful: systemd-nspawn - ArchWiki

Am I reading this right and you’re trying to run docker inside a nixos container?

If so, I doubt that’s possible. The reason you’re seeing those errors are because the kernel isn’t permitting something already in a restricted namespace to create another. To my knowledge nested namespaces are only possible with a very limited scope, which isn’t enough to run docker in a namespace.

In the docker world this is sometimes done at immense security cost by mounting the docker socket inside the container, and hence allowing the container to tell the host docker instance what to do (which essentially gives it root access to the system, thereby reducing docker to a packaging format). This is inadvisable for most use cases, and simply not possible with nixos containers which don’t have a managing daemon exposed through Unix sockets.

Podman may be able to run in a more restricted environment, but generally I just don’t think this is the right approach. Why not run the OCI container directly in podman/docker?

Ah, @azazel75 actually did find a way to do it with the old cgroups implementation, which seems to luckily no longer work with cgroups2 :wink:

Thanks folks, I’m logically separating components in NixOS containers, one of which has its own static IP address that also has NFS mount permissions from another host. Having the containers on the same IP is useful, although not totally required.

Truthfully the problem I was trying to solve is the some missing or older packages from nixpkgs; podman seems to be able to run in user-mode, which sounded interesting.

It might be easier just to package those packages, and maybe even upstream them :slight_smile: