Custom rust package in pkg-overlay

monotux · February 7, 2025, 11:26am

Hi,

I’m trying to build a simple rust program I wrote the other day. The goal is to use this program in a service later on (technically have dnsmasq call it as a script for DHCP lease events). I’m stuck.

# shell.nix in my project repo
{ pkgs ? import <nixpkgs> {} }:
  let
    overrides = (builtins.fromTOML (builtins.readFile ./rust-toolchain.toml));
    libPath = with pkgs; lib.makeLibraryPath [
      # load external libraries that you need in your rust project here
      openssl
    ];
in
  pkgs.mkShell rec {
    nativeBuildInputs = [ pkgs.pkg-config ];
    buildInputs = with pkgs; [
      clang
      # Replace llvmPackages with llvmPackages_X, where X is the latest LLVM version (at the time of writing, 16)
      llvmPackages.bintools
      rustup
      openssl
      cmake
    ];
    RUSTC_VERSION = overrides.toolchain.channel;
    # https://github.com/rust-lang/rust-bindgen#environment-variables
    LIBCLANG_PATH = pkgs.lib.makeLibraryPath [ pkgs.llvmPackages_latest.libclang.lib ];
    shellHook = ''
      export PATH=$PATH:''${CARGO_HOME:-~/.cargo}/bin
      export PATH=$PATH:''${RUSTUP_HOME:-~/.rustup}/toolchains/$RUSTC_VERSION-x86_64-unknown-linux-gnu/bin/
      '';
    # Add precompiled library to rustc search path
    RUSTFLAGS = (builtins.map (a: ''-L ${a}/lib'') [
      # add libraries here (e.g. pkgs.libvmi)
    ]);
    LD_LIBRARY_PATH = libPath;
    # Add glibc, clang, glib, and other headers to bindgen search path
    BINDGEN_EXTRA_CLANG_ARGS =
    # Includes normal include path
    (builtins.map (a: ''-I"${a}/include"'') [
      # add dev libraries here (e.g. pkgs.libvmi.dev)
      pkgs.glibc.dev
    ])
    # Includes with special directory paths
    ++ [
      ''-I"${pkgs.llvmPackages_latest.libclang.lib}/lib/clang/${pkgs.llvmPackages_latest.libclang.version}/include"''
      ''-I"${pkgs.glib.dev}/include/glib-2.0"''
      ''-I${pkgs.glib.out}/lib/glib-2.0/include/''
    ];
  }

I can then build the program with nix-shell followed by cargo build.

But then I try to package it on another machine.

# configuration.nix
nixpkgs.overlays = [ (import ./overlay/pkgs.nix) ];

# overlay/pkgs.nix
self: super:
{
  # ddns-updater works fine!
  ddns-updater = super.callPackage ./ddns-updater/default.nix {};

  # this package doesn't work
  rust-mqtt-presence = super.callPackage ./rust-mqtt-presence/default.nix {};
}

My attempt at packaging this:

{ lib
, rustPlatform
, pkgs ? import <nixpkgs> {}
}:

rustPlatform.buildRustPackage rec {
  pname = "rust-mqtt-presence";
  version = "0.1.0";

  buildInputs = with pkgs; [
    openssl
    cmake
    rustup
    llvmPackages.bintools
    clang
  ];

  nativeBuildInputs = [
    pkgs.pkg-config
  ];

  src = builtins.fetchGit {
    url = "http://172.19.yyy.xxx:3000/oscar/${pname}";
    rev = "ce3dfbcbfaf6eeced5e2ab26482d13e0f3a05f72";
  }

    useFetchCargoVendor = true;
  cargoHas = lib.fakeHash;

  cargoLock = {
    lockFile = ./Cargo.lock;
  };

  meta = {
    description = "MQTT presence tool for dnsmasq";
    homepage = "https://gitea.spindeltax.net/oscar/${pname}";
    license = lib.licenses.unlicense;
    maintainers = [ ];
  };
}

Just getting this error message:

nix-shell -p rust-mqtt-presence
error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:34:12:
           33|
           34|   strict = derivationStrict drvAttrs;
             |            ^
           35|

       … while evaluating derivation 'shell'
         whose name attribute is located at /nix/store/ffvjf7l60zlfsyfwslw4lm35dkz8wgrl-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'buildInputs' of derivation 'shell'
         at /nix/store/ffvjf7l60zlfsyfwslw4lm35dkz8wgrl-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:383:7:
          382|       depsHostHost                = elemAt (elemAt dependencies 1) 0;
          383|       buildInputs                 = elemAt (elemAt dependencies 1) 1;
             |       ^
          384|       depsTargetTarget            = elemAt (elemAt dependencies 2) 0;

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: undefined variable 'rust-mqtt-presence'
       at «string»:1:107:
            1| {...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (rust-mqtt-presence) ]; } ""
             |

What am I missing?

Torinthiel · February 8, 2025, 11:12pm

@monotux Let me clarify a few things. I presume that you’ve put the configuration.nix snippet somewhere in your nixos configuration and the file overlay/pkgs.nix relatively to your config. Have you rebuilt and activated your system (nixos-rebuild switch or test)? Where did the fragment “My attempt at packaging this:” go? ./rust-mqtt-presence/default.nix?

The error message you’ve pasted, I’d assume it’s from running in an unrelated directory, without any shell.nix or default.nix, right?

Can you run printenv NIX_PATH and ls -lA ~/.config/nixpkgs ~/.nixpkgs

My current best guess is that even though you’ve defined your overlay, it’s not taken into account.

Nixpkgs Reference Manual says that nixpkgs.overlays does not affect non-NixOS operations, and nix-shell certainly seems to be one. Thus, next section specifies that

First, if an overlays argument to the Nixpkgs function itself is given, then that is used and no path lookup will be performed.

Otherwise, if the Nix path entry <nixpkgs-overlays> exists, we look for overlays at that path, as described below.See the section on NIX_PATH in the Nix manual for more details on how to set a value for <nixpkgs-overlays>.

If one of ~/.config/nixpkgs/overlays.nix and ~/.config/nixpkgs/overlays/ exists, then we look for overlays at that path, as described below. It is an error if both exist.

nix-shell won’t pass anyting to nixpkgs function, thus ruling out 1. If you haven’t explicitly defined nixpkgs-overlays in NIX_PATH or put someting in ~/.config/nixpkgs/overlays either as a nix file or subdir, than my (pretty limited) knowledge would tell me that there’s no way nixpkgs would know about your overlay.

monotux · February 9, 2025, 3:35pm

My packaging attempt:

# /etc/nixos/overlay/rust-mqtt-presence/default.nix
{ lib
, rustPlatform
, pkgs ? import <nixpkgs> {}
}:

rustPlatform.buildRustPackage rec {
  pname = "rust-mqtt-presence";
  version = "0.1.0";

  buildInputs = with pkgs; [
    openssl
    cmake
    rustup
    llvmPackages.bintools
    clang
  ];

  nativeBuildInputs = [
    pkgs.pkg-config
  ];

  src = builtins.fetchGit {
    url = "http://172.19.123.123:3000/oscar/${pname}";
    rev = "ce3dfbcbfaf6eeced5e2ab26482d13e0f3a05f72";
  }

    useFetchCargoVendor = true;
  cargoHas = lib.fakeHash;

  cargoLock = {
    lockFile = ./Cargo.lock;
  };

  meta = {
    description = "MQTT presence tool for dnsmasq";
    homepage = "https://gitea.example.net/oscar/${pname}";
    license = lib.licenses.unlicense;
    maintainers = [ ];
  };
}

This is then included in my overlay like so:

# /etc/nixos/overlay/pkgs.nix
self: super:
{
  # ddns-updater works afaik, used before it was packaged in nixpkgs
  ddns-updater = super.callPackage ./ddns-updater/default.nix {};
  # rust-mqtt-presence = super.callPackage ./rust-mqtt-presence/default.nix {};
  rust-mqtt-presence = builtins.trace "defining rust-mqtt-presence" (super.callPackage ./rust-mqtt-presence/default.nix {});
}

My overlay is then included in my main system configuration:

# grep overlay/pkgs configuration.nix
nixpkgs.overlays = [ (import ./overlay/pkgs.nix) ];

monotux · February 9, 2025, 3:39pm

I’ve rebuilt the system several times while troubleshooting this but doing this again as this isn’t my first time either

[root@unicorn:/etc/nixos]# nixos-rebuild build
building Nix...
building the system configuration...
unpacking 'https://github.com/ryantm/agenix/archive/main.tar.gz' into the Git cache...

[root@unicorn:/etc/nixos]# nixos-rebuild test
building Nix...
building the system configuration...
activating the configuration...
[agenix] creating new generation in /run/agenix.d/2
[agenix] decrypting secrets...
# removed some names here, all are working
[agenix] symlinking new secrets to /run/agenix (generation 2)...
[agenix] removing old secrets (generation 1)...
[agenix] chowning...
setting up /etc...
reloading user units for oscar...
restarting sysinit-reactivation.target
the following new units were started: run-credentials-systemd\x2dtmpfiles\x2dresetup.service.mount, sysinit-reactivation.target, systemd-tmpfiles-resetup.service

…and then, finally:

[root@unicorn:/etc/nixos]# nix-shell -p rust-mqtt-presence --show-trace
error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:34:12:
           33|
           34|   strict = derivationStrict drvAttrs;
             |            ^
           35|

       … while evaluating derivation 'shell'
         whose name attribute is located at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'buildInputs' of derivation 'shell'
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:383:7:
          382|       depsHostHost                = elemAt (elemAt dependencies 1) 0;
          383|       buildInputs                 = elemAt (elemAt dependencies 1) 1;
             |       ^
          384|       depsTargetTarget            = elemAt (elemAt dependencies 2) 0;

       … while calling anonymous lambda
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:312:13:
          311|       (map (drv: getDev drv.__spliced.hostHost or drv) (checkDependencyList "depsHostHost" depsHostHost))
          312|       (map (drv: getDev drv.__spliced.hostTarget or drv) (checkDependencyList "buildInputs" buildInputs'))
             |             ^
          313|     ]

       … from call site
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:312:18:
          311|       (map (drv: getDev drv.__spliced.hostHost or drv) (checkDependencyList "depsHostHost" depsHostHost))
          312|       (map (drv: getDev drv.__spliced.hostTarget or drv) (checkDependencyList "buildInputs" buildInputs'))
             |                  ^
          313|     ]

       … while calling 'getOutput'
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/lib/attrsets.nix:1796:23:
         1795|   */
         1796|   getOutput = output: pkg:
             |                       ^
         1797|     if ! pkg ? outputSpecified || ! pkg.outputSpecified

       … while evaluating a branch condition
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/lib/attrsets.nix:1797:5:
         1796|   getOutput = output: pkg:
         1797|     if ! pkg ? outputSpecified || ! pkg.outputSpecified
             |     ^
         1798|       then pkg.${output} or pkg.out or pkg

       … in the left operand of the OR (||) operator
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/lib/attrsets.nix:1797:32:
         1796|   getOutput = output: pkg:
         1797|     if ! pkg ? outputSpecified || ! pkg.outputSpecified
             |                                ^
         1798|       then pkg.${output} or pkg.out or pkg

       … in the argument of the not operator
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/lib/attrsets.nix:1797:10:
         1796|   getOutput = output: pkg:
         1797|     if ! pkg ? outputSpecified || ! pkg.outputSpecified
             |          ^
         1798|       then pkg.${output} or pkg.out or pkg

       … from call site
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:312:25:
          311|       (map (drv: getDev drv.__spliced.hostHost or drv) (checkDependencyList "depsHostHost" depsHostHost))
          312|       (map (drv: getDev drv.__spliced.hostTarget or drv) (checkDependencyList "buildInputs" buildInputs'))
             |                         ^
          313|     ]

       … while calling anonymous lambda
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/lib/lists.nix:335:29:
          334|   */
          335|   imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
             |                             ^
          336|

       … from call site
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/lib/lists.nix:335:32:
          334|   */
          335|   imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
             |                                ^
          336|

       … while calling anonymous lambda
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:281:15:
          280|     imap1
          281|       (index: dep:
             |               ^
          282|         if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep

       … while evaluating a branch condition
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:282:9:
          281|       (index: dep:
          282|         if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep
             |         ^
          283|         else if isList dep then checkDependencyList' ([index] ++ positions) name dep

       … in the left operand of the OR (||) operator
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:282:69:
          281|       (index: dep:
          282|         if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep
             |                                                                     ^
          283|         else if isList dep then checkDependencyList' ([index] ++ positions) name dep

       … in the left operand of the OR (||) operator
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:282:44:
          281|       (index: dep:
          282|         if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep
             |                                            ^
          283|         else if isList dep then checkDependencyList' ([index] ++ positions) name dep

       … in the left operand of the OR (||) operator
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:282:29:
          281|       (index: dep:
          282|         if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep
             |                             ^
          283|         else if isList dep then checkDependencyList' ([index] ++ positions) name dep

       … from call site
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/pkgs/stdenv/generic/make-derivation.nix:282:12:
          281|       (index: dep:
          282|         if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep
             |            ^
          283|         else if isList dep then checkDependencyList' ([index] ++ positions) name dep

       … while calling 'isDerivation'
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/lib/attrsets.nix:1282:5:
         1281|   isDerivation =
         1282|     value: value.type or null == "derivation";
             |     ^
         1283|

       … while calling the 'elemAt' builtin
         at /nix/store/2hw1z4svghmvp4aya6s59rd24bjj7yh9-nixos-24.11-small/nixos/lib/lists.nix:335:43:
          334|   */
          335|   imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
             |                                           ^
          336|

       error: undefined variable 'rust-mqtt-presence'
       at «string»:1:107:
            1| {...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (rust-mqtt-presence) ]; } ""
             |

If I try with my older overlayed package ddns-updater:

[root@unicorn:/etc/nixos]# nix-shell -p ddns-updater

[nix-shell:/etc/nixos]# ddns-updater -h
========================================
========================================
============= ddns-updater =============
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================

Running version unknown built on an unknown date (commit unknown)

🔧 Need help? https://github.com/qdm12/ddns-updater/discussions/new
🐛 Bug? https://github.com/qdm12/ddns-updater/issues/new
✨ New feature? https://github.com/qdm12/ddns-updater/issues/new
☕ Discussion? https://github.com/qdm12/ddns-updater/discussions/new
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2025-02-09T16:38:46+01:00 INFO Settings summary:
├── HTTP client