Running Nix(OS) containers directly from the store with podman

Guides
1

I’ve been building containers with Nix and wanted to avoid running podman load < result every time I did a change.
I realized you could directly run the copyToRoot derivation of buildImage rather than build a full image and then run it.
For the classic hello world container:

pkgs.dockerTools.buildImage {
    name = "hello";
    tag = "latest";
    copyToRoot = pkgs.hello;
    config = { cmd = [ "/bin/hello" ]; };
};

You can also run it using:

$ nix build nixpkgs#hello
$ podman run -ti --rm -v /nix/store:/nix/store --rootfs ./result:O /bin/hello

The Nix store is shared with the container for the dependencies, and the overlay :O option is used on --rootfs to make it writable in the container. You could share only the closure of the root instead of the whole nix store with nix path-info --recursive nixpkgs#hello.

It also works with a NixOS root, but I had issues with the /etc symlink so I made a function to copy it instead:

system-container-root = (system:
  let
    pkgs = system.pkgs;
  in (system.pkgs.runCommand "nixos-container-root" {
    nixos = system.config.system.build.toplevel;
  } ''
    export PATH=${pkgs.coreutils}/bin
    cp -r $nixos/ $out
    chmod u+w $out
    rm -f $out/etc
    mkdir $out/etc
    mkdir $out/sbin
    cp $out/init $out/sbin/init
    cp -r $nixos/etc/* $out/etc/
  '')
);

I also copy /init to /sbin/init so podman can automatically detect that it’s running under systemd.

You can then build a root for the container:

(system-container-root (nixpkgs.lib.nixosSystem {
  system = "x86_64-linux";
  modules = [
    ({ pkgs, modulesPath, ... }: {
      imports = [
        "${toString modulesPath}/virtualisation/docker-image.nix"
      ];
      boot.isContainer = true;
      services.journald.console = "/dev/console";
      services.getty.autologinUser = "root";
      services.nginx.enable = true;
    })
  ];
}));

I’ve put this example in a flake than you build with:

$ nix build .#nixos-container

And run it. You have to copy the root directory so it is owned by your user, I think due to a limitation of podman (if someone has a clue): cp -r result root, or run it as root:

$ podman run -ti --rm -p 8080:80 -v /nix/store:/nix/store --rootfs ./root:O /sbin/init
$ curl http://localhost:8080

Hope it helps, I’m curious if someone has similar tricks?

9 Likes