Running Nix(OS) containers directly from the store with podman

Guides
1

I’ve been building containers with Nix and wanted to avoid running podman load < result every time I did a change.
I realized you could directly run the copyToRoot derivation of buildImage rather than build a full image and then run it.
For the classic hello world container:

pkgs.dockerTools.buildImage {
    name = "hello";
    tag = "latest";
    copyToRoot = pkgs.hello;
    config = { cmd = [ "/bin/hello" ]; };
};

You can also run it using:

$ nix build nixpkgs#hello
$ podman run -ti --rm -v /nix/store:/nix/store --rootfs ./result:O /bin/hello

The Nix store is shared with the container for the dependencies, and the overlay :O option is used on --rootfs to make it writable in the container. You could share only the closure of the root instead of the whole nix store with nix path-info --recursive nixpkgs#hello.

It also works with a NixOS root, but I had issues with the /etc symlink so I made a function to copy it instead:

system-container-root = (system:
  let
    pkgs = system.pkgs;
  in (system.pkgs.runCommand "nixos-container-root" {
    nixos = system.config.system.build.toplevel;
  } ''
    export PATH=${pkgs.coreutils}/bin
    cp -r $nixos/ $out
    chmod u+w $out
    rm -f $out/etc
    mkdir $out/etc
    mkdir $out/sbin
    cp $out/init $out/sbin/init
    cp -r $nixos/etc/* $out/etc/
  '')
);

I also copy /init to /sbin/init so podman can automatically detect that it’s running under systemd.

You can then build a root for the container:

(system-container-root (nixpkgs.lib.nixosSystem {
  system = "x86_64-linux";
  modules = [
    ({ pkgs, modulesPath, ... }: {
      imports = [
        "${toString modulesPath}/virtualisation/docker-image.nix"
      ];
      boot.isContainer = true;
      services.journald.console = "/dev/console";
      services.getty.autologinUser = "root";
      services.nginx.enable = true;
    })
  ];
}));

I’ve put this example in a flake than you build with:

$ nix build .#nixos-container

And run it. You have to copy the root directory so it is owned by your user, I think due to a limitation of podman (if someone has a clue): cp -r result root, or run it as root:

$ podman run -ti --rm -p 8080:80 -v /nix/store:/nix/store --rootfs ./root:O /sbin/init
$ curl http://localhost:8080

Hope it helps, I’m curious if someone has similar tricks?

10 Likes
2

Note that this will probably break if you modify the host’s Nix store while the container is running. On Linux, an overlay filesystem whose underlying storage has been modified will exhibit undefined behaviour (inconsistent VFS state).

1 Like