Ollama can't find libstdc++.so.6 [Resolved]

heartbeast42 · April 28, 2024, 2:43am

TL;DR:

I want to get Ollama running but it can’t find libstdc++.so.6.
– How do I get Ollama running?
– How can I go about debugging this and understanding this in the right “nixish” way?
EDIT: For context
– this issue arises when using Ollama from nixpkgs 23.11 as well as nixpkgs unstable
– I’m using NixOS 23.11 as my base OS

I had Ollama running several months ago but when I try and run it now I get the following error in the output of ollama serve

(NOTE: this is an edited excerpt with the timestamps removed for easier reading)

...
- skipping accelerated runner because num_gpu=0
- starting llama runner
- waiting for llama runner to start responding
- /tmp/ollama2915273314/llama.cpp/gguf/build/cpu/bin/ollama-runner:
    error while loading shared libraries: libstdc++.so.6: 
        cannot open shared object file: No such file or directory
- exit status 127
- error starting llama runner: llama runner process has terminated
- llama runner stopped successfully
...

I’ve researched this issue to the point of exhaustion over the last week. I’ve found a lot of posts, here and github and reddit among others, about people having issues with PROGRAM_X can’t find specifically libstdc++.so.6 under NixOS (sorry for yet another one lol). Unfortunately they’ve all been dead ends for me. I can’t help feeling like the answer is kinda staring me in the face or I’m not thinking about things in the proper “nix way” yet. I still feel like such a newbie with Nix and NixOS but I feel like I’m getting it slowly.

My first inclination is to ask some version of “How do I install libstdc++.so.6” or “what package provides libstdc++.so.6” but I think that’s kindof the wrong question no…??
Also to the best of my knowledge the library is part of the C/C++ compiler toolchain, which I have installed under environment.systemPackages in my main configuration.nix (shown below).

environment.systemPackages = with pkgs; [
  ...
  # -- c / c++
  gcc
  libgcc
  gnumake
  cmake
  extra-cmake-modules
  ...
];

Am I missing something? Have I committed some egregious C/C++ programming fopaux by omitting essential tools? For context I’m self taught regarding C/C++ and I’m very much still learning…

Thanks y’all!

samrose · April 28, 2024, 2:48am

heartbeast42 · April 28, 2024, 3:25am

Thanks for the reply.

I added the above suggested code to my configuration.nix right above environment.systemPackages as shown below.

environment = {
  sessionVariables = {
    LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
  };
};

I’m currently getting the same error when I try and run ollama but adding the above lines had an interesting result

error: The option `environment.sessionVariables.LD_LIBRARY_PATH' is defined multiple times while it's expected to be unique.

       Definition values:
       - In `/etc/nixos/configuration.nix': "/nix/store/myw67gkgayf3s2mniij7zwd79lxy8v0k-gcc-12.3.0-lib/lib"
       - In `/nix/store/bparqn3baq68j4zry66c7hldh3ddjn82-nixos-23.11.2130.d65bceaee0fb/nixos/nixos/modules/services/desktops/pipewire/pipewire.nix':
           [
             "/nix/store/ym30ihpm3sjgkphpab3f2lcg6j4mla9v-pipewire-0.3.85-jack/lib"
           ]
       Use `lib.mkForce value` or `lib.mkDefault value` to change the priority on any of these definitions.

I notic that jack audio is defining environment.sessionVariables.LD_LIBRARY_PATH so I commented out jack.enable = true; under services.pipewire = {... which got rid of the nix rebuild error but not the ollama error.

rgoulter · April 28, 2024, 3:43am

In this case, ollama is available in nixpkgs-unstable. NixOS Search

The LD_LIBRARY_PATH controls where an executable looks for its shared libraries. So, uh, setting this globally in NixOS is quite a kludge & might break other programs.

samrose · April 28, 2024, 3:49am

I agree with @rgoulter was just reading up on this and coming back here to suggest not using that setting for the reasons he mentioned.

as he mentions, since it is already packaged you can just add it to

environment.systemPackages = with pkgs; [
  ...
  # -- c / c++
  gcc
  libgcc
  gnumake
  cmake
  extra-cmake-modules
  ...
];

heartbeast42 · April 28, 2024, 4:07am

I forgot to mention that this issue arises when I try using ollama from nixpkgs either from 23.11 stable or unstable. The issue is exactly the same i.e.

...
error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
...

heartbeast42 · April 28, 2024, 4:13am

Okay, I was wondering if setting LD_LIBRARY_PATH globally was a bit heavy handed so thanks to both of y’all (@rgoulter & @samrose) for confirming that.

Please pardon me if this is obvious but is there a way to set the LD_LIBRARY_PATH for just Ollama or any other 1 single program? If so it’s not clear to me how to do that…

samrose · April 28, 2024, 4:18am

if you are using nixos, I wonder if it will work for you if you just enable the service from the contributed module (I’ve never used Ollama so didn’t know about this module until just now)?

github.com

NixOS/nixpkgs/blob/d6f6eb2a984f2ba9a366c31e4d36d65465683450/nixos/modules/services/misc/ollama.nix

{ config, lib, pkgs, ... }:
let
  inherit (lib) types;

  cfg = config.services.ollama;
  ollamaPackage = cfg.package.override {
    inherit (cfg) acceleration;
    linuxPackages = config.boot.kernelPackages // {
      nvidia_x11 = config.hardware.nvidia.package;
    };
  };
in
{
  options = {
    services.ollama = {
      enable = lib.mkEnableOption "ollama server for local large language models";
      package = lib.mkPackageOption pkgs "ollama" { };
      home = lib.mkOption {
        type = types.str;
        default = "%S/ollama";

This file has been truncated. show original

samrose · April 28, 2024, 4:20am

that would likely be

services.ollama = {
  enable = true;
  # Add any specific configuration options here
};

rgoulter · April 28, 2024, 4:21am

For specifically “I want to set LD_LIBRARY_PATH to have a libstdc++.so in it when running a binary”, running something like:

env LD_LIBRARY_PATH=$(nix path-info nixpkgs#stdenv.cc.cc.lib)/lib:$LD_LIBRARY_PATH ...

in the shell would be one way to do this.

Though, as the reply to that thread points out, packages in nixpkgs shouldn’t need such a fix, it’s an issue if they do.

jchw · April 28, 2024, 4:21am

For what it is worth, I definitely echo the sentiment that it is best to use the existing Nix-packaged version of software whenever possible. If you’re running stable NixOS and it’s not good enough, it’s relatively easy to temporarily shell things in from unstable selectively. Using the Nix packaged versions is also probably going to provide a vastly better experience if the program you are trying to run may need CUDA or ROCm to work and the Nix package/NixOS module already has that covered.

That said, just for sake of answering it… If you want to be able to run unmodified binaries on NixOS, I’ve found Nix-LD to be a good solution for this. I believe the specific attr I am bringing in for libstdc++.so.6 is stdenv.cc.cc.lib, but I’m not 100% sure.

I believe I originally copied some of my config from the unofficial NixOS wiki, but I can’t remember where. Maybe the page on AppImage? Either way, I have a fairly complete Nix-LD setup for running random binaries directly. Since most standalone binaries like this don’t care much about the FHS as long as the linker gives them their packages this actually works well. I can see that the ollama binary runs on my machine.

My exact Nix-LD config can be found on GitHub. But it’s basically like this:

{
  programs.nix-ld = {
    enable = true;
    libraries = with pkgs; [
      ...
      stdenv.cc.cc.lib
    ];
  };
}

The advantage of Nix-LD is that it doesn’t matter if LD_LIBRARY_PATH gets overridden somewhere and it handles making sure that the standard DT_INTERP path is linked to the right place. But if you almost have it working, adding ${stdenv.cc.cc.lib}/lib to LD_LIBRARY_PATH might get it working.

Note: Because Nix-LD installs into the standard DT_INTERP location, it has no impact on any “standard” Nix stuff, though it is global for standalone binaries. If you needed different per-program things, then it’d probably best to go all the way and use patch elf on them, basically as if you were maintaining a binary derivation.

samrose · April 28, 2024, 4:24am

It appears to be set in the package here

github.com

NixOS/nixpkgs/blob/91376bd301d53d0946a5dfb8fb235cfbb7104382/pkgs/tools/misc/ollama/default.nix#L186


      
          '';
          postFixup = ''
            # the app doesn't appear functional at the moment, so hide it
            mv "$out/bin/app" "$out/bin/.ollama-app"
          '' + lib.optionalString (enableRocm || enableCuda) ''
            # expose runtime libraries necessary to use the gpu
            mv "$out/bin/ollama" "$out/bin/.ollama-unwrapped"
            makeWrapper "$out/bin/.ollama-unwrapped" "$out/bin/ollama" ${
              lib.optionalString enableRocm
                ''--set-default HIP_PATH '${rocmPath}' ''} \
              --suffix LD_LIBRARY_PATH : '/run/opengl-driver/lib:${lib.makeLibraryPath runtimeLibs}'
          '';
          
          
ldflags = [
            "-s"
            "-w"
            "-X=github.com/jmorganca/ollama/version.Version=${version}"
            "-X=github.com/jmorganca/ollama/server.mode=release"
          ];
          
          
passthru.tests = {

cig0 · April 28, 2024, 4:42am

Hey @heartbeast42, I’m using Ollama from the unstable channel without issues (knocking on wood).

Running Ollama with Podman also works quite well as a last resort, so you don’t waste any more time until you can fix the issue.

heartbeast42 · April 28, 2024, 4:55am

Thanks to everyone for all the great replies! Y’all are the best of the internet. I have a lot to look over here. My wife just got home and we’ve got plans to play Stardew Valley until we pass out.

I’ll reply with any new information in the next couple days.

heartbeast42 · April 28, 2024, 5:30am

Okay, so, I fixed it and it’s embarrassing. But I feel like I owe y’all an explanation cause it’s funny and I appreciate y’all taking the time to share your knowledge and type out thoughtful replies. I legit learned a lot.

The short story is that I had a broken ollama binary installed in my Scripts folder in my home directory which I keep in my shell path…

Just as I was wrapping up the issue for the evening I commented out ollama in my configuration and rebuilt my system. After the system rebuilt I exited the ollama server and, because it costs me nothing, I ran ollama serve again after I had “uninstalled” it and the server totally ran lol. So I ran which ollama which gave me /home/MyUser/Scripts/ollama. Once I saw it I remembered installing it months ago and I forgot about it and didn’t use ollama for a while.

So here we are lol

I’m running ollama from unstable and it works as well as it ever has.

Thanks again to all of you who took some time to help me out!