Including a custom local library

Pretty new to nix, but I’ve been reading through as much as I can find, but I’m stuck.

I am wanting to include a library that I have built from source using an absolute file path.

I have a custom library I for work that I need accessible globally to other packages (IDEs, etc). The source lives in a repo (in Azure) that can only be accessed when on a VPN, and I really cannot have to sign into a work VPN every time I want to rebuild my config.

I’m currently trying to use nix-ld to set my paths, but I cannot figure it out since I don’t have a package built for it. It would be great if I could either

A) be able to set a global path to include /path/to/my/lib - or -

B) be able to create some package that I could build once while on the VPN, but then as long as I didn’t need to update it would not have it interfere with other changes while not connected

I’m on version 23.11

What am I missing, and how can I get there? Thank you, all

I think what you are suggesting should be possible somehow but I do not think it is the Nix way™. Personally, I would start by creating a Nix derivation for the library.

I would probably just use local checkout of the library repo as the source for the library derivation. Then VPN would only be needed to update the repo. If I wanted it more self-contained, I could even add the library repo as a subtree to my configuration repo.

Alternately, you could symlink the src to the library derivation output somewhere. That way, the source will not be garbage collected while the library package is accessible from some GC root. (A sort of poor man’s system.includeBuildDependencies, when that is too heavy.)

2 Likes

Thank you for the advice! I’m still struggling with the actual script to create the derivation. This is the latest iteration, but I’ve looked through examples from multiple docs and tutorials

What I have now is this, which is failing with a new error. Any idea what I’m doing wrong?:

 error: infinite recursion encountered

       at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:506:28:

          505|         builtins.addErrorContext (context name)
          506|           (args.${name} or config._module.args.${name})
             |                            ^
          507|       ) (lib.functionArgs f);

my_repo.nix

{ config, lib, stdenv, pkgs, cmake, makeWrapper, ... }:

let
  build_pkgs = with pkgs; [
    pkgs.gcc
    pkgs.gdb
    pkgs.python3
    pkgs.cmake
    pkgs.sqlite.dev
    pkgs.libtiff.dev
    pkgs.curl.dev
  ];
in
stdenv.mkDerivation {  # removed "rec" to prevent infinite recursion
  pname = "my_project";
  version = "1.0.0";

  src = /path/to/repo/.;

  nativeBuildInputs = build_pkgs;

  LD_LIBRARY_PATH = "${pkgs.lib.makeLibraryPath build_pkgs}:${pkgs.stdenv.cc.cc.lib}/lib";
  SQLITE3_LIBRARY = "${pkgs.sqlite.dev}/lib/libsqlite3.so"; #
  SQLITE3_INCLUDE_DIR = "${pkgs.sqlite.dev}/include";
  TIFF_LIBRARY = "${pkgs.libtiff.dev}/lib/libtiff.so";
  TIFF_INCLUDE_DIR = "${pkgs.libtiff.dev}/include";
  CURL_LIBRARY = "${pkgs.curl.dev}/lib";
  CURL_INCLUDE_DIR = "${pkgs.curl.dev}/include";

  # Configure phase
  configurePhase = ''
    mkdir $src/build
    cd $src/build
    cmake ..
  '';   # Does it need "-DCMAKE_INSTALL_PREFIX=$out"" instead?

  # Build phase
  buildPhase = ''
    cd $src/build
    cmake --build .
  '';

  # Install phase
  installPhase = ''
    mkdir -p $out/bin
    mv $src/bin/* $out/bin/

    mkdir -p $out/lib
    mv $src/lib/* $out/lib/
  '';

  meta = with lib; {
    description = "Customized version of proj";
    homepage = "https://example.com/my_project";
    license = licenses.mit;
    maintainers = with maintainers; [ "Names" ];
  };
}

This is a bit sparse to be able to tell anything.

How are you invoking my_repo.nix? And what is the full error with --show-trace?

I’m including the .nix file in my main configuration.nix file:

imports = [
  ./hardware-configuration.nix
  /home/myself/.dotfiles/my_repo.nix
];

Although now I’m beginning to think that’s the the correct method to include it.

The errors with --show-trace seem repeat this same pattern until it fails:


       error: infinite recursion encountered

       at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:506:28:

          505|         builtins.addErrorContext (context name)
          506|           (args.${name} or config._module.args.${name})
             |                            ^
          507|       ) (lib.functionArgs f);
building the system configuration...
error:
       … while evaluating the attribute 'config.system.build.toplevel'

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:320:9:

          319|         options = checked options;
          320|         config = checked (removeAttrs config [ "_module" ]);
             |         ^
          321|         _module = checked (config._module);

       … while calling the 'seq' builtin

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:320:18:

          319|         options = checked options;
          320|         config = checked (removeAttrs config [ "_module" ]);
             |                  ^
          321|         _module = checked (config._module);

       … while evaluating a branch condition

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:261:9:

          260|       checkUnmatched =
          261|         if config._module.check && config._module.freeformType == null && merged.unmatchedDefns != [] then
             |         ^
          262|           let

       … in the left operand of the AND (&&) operator

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:261:72:

          260|       checkUnmatched =
          261|         if config._module.check && config._module.freeformType == null && merged.unmatchedDefns != [] then
             |                                                                        ^
          262|           let

       … in the left operand of the AND (&&) operator

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:261:33:

          260|       checkUnmatched =
          261|         if config._module.check && config._module.freeformType == null && merged.unmatchedDefns != [] then
             |                                 ^
          262|           let

       … while evaluating a branch condition

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:254:12:

          253|
          254|         in if declaredConfig._module.freeformType == null then declaredConfig
             |            ^
          255|           # Because all definitions that had an associated option ended in

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:242:28:

          241|           # For definitions that have an associated option
          242|           declaredConfig = mapAttrsRecursiveCond (v: ! isOption v) (_: v: v.value) options;
             |                            ^
          243|

       … while calling 'mapAttrsRecursiveCond'

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/attrsets.nix:696:5:

          695|     # Attribute set to recursively map over.
          696|     set:
             |     ^
          697|     let

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:234:33:

          233|           ({ inherit lib options config specialArgs; } // specialArgs);
          234|         in mergeModules prefix (reverseList collected);
             |                                 ^
          235|

       … while calling 'reverseList'

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/lists.nix:510:17:

          509|   */
          510|   reverseList = xs:
             |                 ^
          511|     let l = length xs; in genList (n: elemAt xs (l - n - 1)) l;

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:229:25:

          228|       merged =
          229|         let collected = collectModules
             |                         ^
          230|           class

       … while calling anonymous lambda

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:443:37:

          442|
          443|     in modulesPath: initialModules: args:
             |                                     ^
          444|       filterModules modulesPath (collectStructuredModules unknownModule "" initialModules args);

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:444:7:

          443|     in modulesPath: initialModules: args:
          444|       filterModules modulesPath (collectStructuredModules unknownModule "" initialModules args);
             |       ^
          445|

       … while calling 'filterModules'

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:411:36:

          410|       # modules recursively. It returns the final list of unique-by-key modules
          411|       filterModules = modulesPath: { disabled, modules }:
             |                                    ^
          412|         let

       … while calling anonymous lambda

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:437:31:

          436|           disabledKeys = concatMap ({ file, disabled }: map (moduleKey file) disabled) disabled;
          437|           keyFilter = filter (attrs: ! elem attrs.key disabledKeys);
             |                               ^
          438|         in map (attrs: attrs.module) (builtins.genericClosure {

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:398:22:

          397|           let
          398|             module = checkModule (loadModule args parentFile "${parentKey}:anon-${toString n}" x);
             |                      ^
          399|             collectedImports = collectStructuredModules module._file module.key module.imports args;

       … while calling anonymous lambda

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:357:11:

          356|         then
          357|           m:
             |           ^
          358|             if m._class != null -> m._class == class

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:398:35:

          397|           let
          398|             module = checkModule (loadModule args parentFile "${parentKey}:anon-${toString n}" x);
             |                                   ^
          399|             collectedImports = collectStructuredModules module._file module.key module.imports args;

       … while calling 'loadModule'

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:334:53:

          333|       # Like unifyModuleSyntax, but also imports paths and calls functions if necessary
          334|       loadModule = args: fallbackFile: fallbackKey: m:
             |                                                     ^
          335|         if isFunction m then

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:352:14:

          351|           throw "Module imports can't be nested lists. Perhaps you meant to remove one level of lists? Definitions: ${showDefs defs}"
          352|         else unifyModuleSyntax (toString m) (toString m) (applyModuleArgsIfFunction (toString m) (import m) args);
             |              ^
          353|

       … while calling 'unifyModuleSyntax'

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:452:34:

          451|      of ‘options’, ‘config’ and ‘imports’ attributes. */
          452|   unifyModuleSyntax = file: key: m:
             |                                  ^
          453|     let

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:352:59:

          351|           throw "Module imports can't be nested lists. Perhaps you meant to remove one level of lists? Definitions: ${showDefs defs}"
          352|         else unifyModuleSyntax (toString m) (toString m) (applyModuleArgsIfFunction (toString m) (import m) args);
             |                                                           ^
          353|

       … while calling 'applyModuleArgsIfFunction'

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:486:39:

          485|
          486|   applyModuleArgsIfFunction = key: f: args@{ config, options, lib, ... }:
             |                                       ^
          487|     if isFunction f then applyModuleArgs key f args else f;

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:487:26:

          486|   applyModuleArgsIfFunction = key: f: args@{ config, options, lib, ... }:
          487|     if isFunction f then applyModuleArgs key f args else f;
             |                          ^
          488|

       … while calling 'applyModuleArgs'

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:489:29:

          488|
          489|   applyModuleArgs = key: f: args@{ config, options, lib, ... }:
             |                             ^
          490|     let

       … from call site

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:513:8:

          512|       # works.
          513|     in f (args // extraArgs);
             |        ^
          514|

       … while calling anonymous lambda

         at /home/myself/.dotfiles/my_repo.nix:1:1:

            1| { config, lib, stdenv, pkgs, cmake, makeWrapper, ... }:
             | ^
            2|

       … while calling anonymous lambda

         at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:504:44:

          503|       context = name: ''while evaluating the module argument `${name}' in "${key}":'';
          504|       extraArgs = builtins.mapAttrs (name: _:
             |                                            ^
          505|         builtins.addErrorContext (context name)

       … while evaluating the module argument `stdenv' in "/home/myself/.dotfiles/my_repo.nix":

       error: infinite recursion encountered

       at /nix/store/1xclxik35hm5agkgp9a0wh6465an2j98-nixos-23.11/nixos/lib/modules.nix:506:28:

          505|         builtins.addErrorContext (context name)
          506|           (args.${name} or config._module.args.${name})
             |                            ^
          507|       ) (lib.functionArgs f);

Ah. imports expects NixOS modules, whereas my_repo.nix is a Nixpkgs package expression.

You should be able to instantiate the package with pkgs.callPackage, which will fill in the dependency arguments.

But for now, including it to your system configuration is probably too early. You should first try to make the package itself build and you can do that directly with nix-build --expr 'let pkgs = import <nixpkgs> {}; in pkgs.callPackage ~/.dotfiles/my_repo.nix { }' and check the result/ directory symlink. That will allow for faster iteration (no need to rebuild the whole system).

After you are happy with it, you would create a package for a program that depends on the library. You should be able to build it similarly:

nix-build --expr 'let pkgs = import <nixpkgs> {}; my_library = pkgs.callPackage ~/.dotfiles/my_repo.nix { }; in pkgs.callPackage ~/.dotfiles/my_program.nix { inherit my_library; }'

Finally, when everything is ready, you can use it in your configuration.nix as follows:

{ pkgs, … }:

let
  # And now, you will be able to use my_lib variable in the rest of the file.
  my_lib = pkgs.callPackage ~/.dotfiles/my_repo.nix { };

  # For example, pass it to another package that depends on it.
  my_program = pkgs.callPackage = ~/.dotfiles/my_program.nix {
    inherit my_lib;
  };
in
{
  imports = [
    ./hardware-configuration.nix
  ];

  environment.systemPackages = [
    # You can install programs through this. Note that installing libraries globally is not generally supported:
    # https://wiki.nixos.org/wiki/FAQ#I_installed_a_library_but_my_compiler_is_not_finding_it._Why?
    my_program
  ];

  # …
}
1 Like

Thank you! I’ve gotten the package to build and install cleanly into the nix store, and is now included in my main configuration. So now I guess the lingering question is what is the correct nix way to make those libraries available to my system?

I have in my configuration:

nixpkgs.config.packageOverrides = pkgs: {
  my-repo = pkgs.callPackage /home/myself/.dotfiles/my_repo.nix { };
};

programs.nix-ld.enable = true;

programs.nix-ld.libraries = [
  pkgs.my-repo
];

But I’m still unable to locate the libraries. For example: Running code inside of Rider that needs to access that installed lib. Or running a dotnet program from the console that is looking for that library.