Nvidia-container-runtime exit status 125: unknown

1

It seems nvidia + docker was broken by the upgrade to 24.05.

What are the proper configs for nvidia container runtime to work properly under docker and 24.05?

{ config, lib, pkgs, ... }:
let
in
{
  environment.systemPackages = with pkgs; [ 
    nvidia-docker
    docker_25
    nvidia-container-toolkit
  ];
  systemd.enableUnifiedCgroupHierarchy = false;
  hardware.nvidia-container-toolkit.enable = true;
  virtualisation = {
    containers = {
      enable = true;
      #cdi.dynamic.nvidia.enable = true;
    };
    docker = {
      enable = true;
      #enableNvidia = true;
      # CDI is feature-gated and only available from Docker 25 and onwards
      #package = pkgs.docker_26;
      package = pkgs.docker_25;
      #setSocketVariable = true;
      daemon.settings = {
        features.cdi = true;
        default-runtime = "nvidia";
        runtimes.nvidia.path = "${pkgs.nvidia-docker}/bin/nvidia-container-runtime";
        exec-opts = ["native.cgroupdriver=cgroupfs"];
      };
      #storageDriver = "zfs";
    };
    #oci-containers.backend = "docker";
  };
}

nvidia-container-runtime run --rm  ubuntu nvidia-smi
warning: $XDG_CONFIG_HOME=/home/thoth/.config
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: /nix/store/va74ykggqzmamwh2aj39fxlwzf8csw6s-nvidia-docker/bin/nvidia-container-runtime did not terminate successfully: exit status 125: unknown.

Of note, I have tried docker_26 docker_25 docker_24 and docker none of those seem to work.

1 Like
2

I also tried many things, but the only thing that worked for me was to downgrade the nvidia-docker to the one from 23.11

1 Like
3

Do you mind sharing how were you able to achieve this? Iā€™m struggling to setup an overlay for my flake.

Thanks :slight_smile:

4

I use flakes too.
You basically need to get the derivation from 23.11, and use an overlay to replace the one in 24.05 with the one from 23.11

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";
    nixpkgs-23_11.url = "github:nixos/nixpkgs/nixos-23.11";
  };

  outputs =
    inputs:
    let
      system = "x86_64-linux";
      pkgs-23_11 = import inputs.nixpkgs-23_11 {
        inherit system;
        config.allowUnfree = true;
      };

      pkgs = import inputs.nixpkgs {
        inherit system;
        config.allowUnfree = true;
        overlays = [ (final: prev: { nvidia-docker = pkgs-23_11.nvidia-docker; }) ];
      };
    in
    {

      nixosConfigurations.your_computer = inputs.nixpkgs-unstable.lib.nixosSystem {
        inherit system;
        specialArgs = {
          inherit pkgs;
        }; # Pass flake inputs to our config
        # > Our main nixos configuration file <
        modules = [
          # your other modules  e.g.
          ./configuration.nix
          {
            # and this some where in your configurationnix. Or just here is fine too.
            # or podman if you use podman
            virtualisation.docker = {
              enable = true;
              enableNvidia = true;
            };
          }

        ];
      };

    };
}
1 Like
5

Thank you so much!

Amazing, worked like a charm.

Note: I had to reboot my PC (restarting the docker service was not enough)