Cannot find library on statically-linked builds

Help
1

Hi there. I tried hard to put together a “stitched” flake to cross-compile rust projects for Raspberry Pi:

{
  description = "The Rust cross-compiling devShell for Raspberry Pi 3/4/5";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    rust-overlay = {
      url = "github:oxalica/rust-overlay";
      inputs = {
        nixpkgs.follows = "nixpkgs";
      };
    };
  };

  outputs = { self, nixpkgs, rust-overlay }: let
    system = "x86_64-linux";
    pkgs = import nixpkgs {
      inherit system;
      overlays = [ (import rust-overlay) ];
      crossSystem.config = "aarch64-unknown-linux-gnu";
    };
    rustToolchain = pkgs.pkgsBuildHost.rust-bin.fromRustupToolchainFile ./rust-toolchain.toml;

    nativeBuildInputs = with pkgs; [
      rustToolchain 
      pkg-config ];

    buildInputs = with pkgs; [
      #glibc.static
      protobuf 
      openssl 
      libiio
    ];

    in with pkgs; {
      devShells.x86_64-linux.default = callPackage (
        { mkShell, pkg-config }:
        mkShell {
          inherit buildInputs nativeBuildInputs;
        }
      ) {};
    };
}

This works fine when building dynamically linked binaries, but if I uncomment the glibc.static package and try to build statically linked binaries (with rustflags = ["-C", "target-feature=+crt-static"]), it can’t find libiio:

/nix/store/<digest>-aarch64-unknown-linux-gnu-binutils-2.40/bin/aarch64-unknown-linux-gnu-ld: cannot find -liio: No such file or directory

Digging further into the build log, -liio is populated, but not -L${libdir}.

I don’t know how to check if packages are set up properly because pkg-config throws me an error message: cannot execute binary file: Exec format error. Nevertheless, I do find the package under /nix/store/<digest>-libiio-aarch64-unknown-linux-gnu-0.24-dev/lib/pkgconfig/libiio.pc, and it says:

prefix=/nix/store/bjs21s0av4qzpmqyfw8xgfmaa2440ia7-libiio-aarch64-unknown-linux-gnu-0.24
exec_prefix=${prefix}
libdir=/nix/store/gpp73c39nmimlqx60q3jvm6na727zy91-libiio-aarch64-unknown-linux-gnu-0.24-lib/lib
includedir=/nix/store/6d0q5ma128fcxmv6mb2190b3p186gali-libiio-aarch64-unknown-linux-gnu-0.24-dev/include

Name: libiio
Description: Library for interfacing IIO devices
Version: 0.24

Requires:
Libs: -L${libdir} -liio
Cflags: -I${includedir}

My gut feeling is that adding the library path manually is a bad idea because the digest will change over time. Please advise the proper practice to set up libraries with flakes.

Thanks.

2

I’m answering since I don’t see any other answer… but I’m not an expert at all on cross compiling. But I’m wondering if it could be the case that libiio is poorly configured for cross compiling: indeed, the NixOs wiki mentions

      # we want to hack on SDL, don't want to hack on those. Some even don't cross-compile
[…]
      # those shouldn't be neither pkgsCross, nor pkgsArm
      # because those trigger
      #     cannot execute binary file: Exec format error
      # in this case it was enough to just use buildPackages variants
      # but in general there may be problems

Which corresponds exactly to the error you have. So you might prefer to use instead the regular aarch64 for libiio taken directly from the aarch64 NixOs cache instead of cross-compiling it? The wiki mentions something like:

let
    # this will use aarch64 binaries from binary cache, so no need to build those
    pkgsArm = import <nixpkgs> {
        config = {};
        overlays = [];
        system = "aarch64-linux";
    };

    # these will be your cross packages
    pkgsCross = import <nixpkgs> {

       overlays = [(self: super: {

         # we want to hack on SDL, don't want to hack on those. Some even don't cross-compile
         inherit (pkgsArm)
           xorg libpulseaudio libGL guile systemd libxkbcommon
         ;
 
       })];
       crossSystem = {
         config = "aarch64-unknown-linux-gnu";
       };
     };

in pkgsCross.SDL2.override { 
      # those shouldn't be neither pkgsCross, nor pkgsArm
      # because those trigger
      #     cannot execute binary file: Exec format error
      # in this case it was enough to just use buildPackages variants
      # but in general there may be problems
      inherit (pkgsCross.buildPackages) 
         wayland wayland-protocols
      ;
   }

So I guess in your case it might be enough to replace the pkgs = …; line with:

    pkgsArm = import nixpkgs {
        config = {};
        overlays = [];
        system = "aarch64-linux";
    };
    pkgs = import nixpkgs {
      inherit system;
      overlays = [
        (import rust-overlay)
        (self: super: { inherit (pkgsArm) libiio; }) # <-- see this added line
      ];
      crossSystem.config = "aarch64-unknown-linux-gnu";
    };

but I can’t test it since you do not provide ./rust-toolchain.toml.

3

@tobiasBora Big thanks for your insight. I followed your suggestions with Nix Wiki and made the same changes as you suggested. This is my current flake.nix to ensure we are on the same page.

rust-toolchain.toml:

[toolchain]
channel = "stable"
components = [ "rustfmt", "rustc-dev" ]
targets = [
       "x86_64-unknown-linux-gnu",
       "aarch64-unknown-linux-gnu",
]
profile = "complete"

.envrc:

#!/bin/bash

if ! has nix_direnv_version || ! nix_direnv_version 2.3.0; then
    source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/2.3.0/direnvrc" "sha256-Dmd+j63L84wuzgyjITIfSxSD57Tx7v51DMxVZOsiUD8="
fi

nix_direnv_watch_file rust-toolchain.toml
use flake

Unfortunately, I’m running into the same issue with the hack. After checking the environment variables, I found that it is pointing to the wrong architecture of libiio. Both NIX_CFLAGS_COMPILE and NIX_LDFLAGS are pointing to /nix/store/<digest>-libiio-0.24-dev.

$ export | grep iio
NIX_CFLAGS_COMPILE=' -frandom-seed=1v9ij4f9c8 -isystem /nix/store/sv94xjhaw6h1v490vhk3h93zw0jinzkc-protobuf-aarch64-unknown-linux-gnu-24.4/include -isystem /nix/store/234mpzladw7a2rr3lir7w2x1x2bwia80-abseil-cpp-aarch64-unknown-linux-gnu-20230125.3/include -isystem /nix/store/m2qqnaacng2dnxiqpxwqm8s2vvqw6ckk-openssl-aarch64-unknown-linux-gnu-3.0.12-dev/include -isystem /nix/store/8frjmww7r7i5l257fvbgj07wxaiw6gmd-libiio-0.24-dev/include'
NIX_LDFLAGS='-rpath /home/hftsai/CytoRepo/pifds/outputs/out/lib  -L/nix/store/65wgx7xdqx2dkds2qmv356ih3rnn80cm-glibc-aarch64-unknown-linux-gnu-2.38-27-static/lib -L/nix/store/sv94xjhaw6h1v490vhk3h93zw0jinzkc-protobuf-aarch64-unknown-linux-gnu-24.4/lib -L/nix/store/234mpzladw7a2rr3lir7w2x1x2bwia80-abseil-cpp-aarch64-unknown-linux-gnu-20230125.3/lib -L/nix/store/b39nrlb4v11pfyvwmk53jw00isf9xjas-openssl-aarch64-unknown-linux-gnu-3.0.12/lib -L/nix/store/hll0zwkabpcg8dg90l9w6hci05y0agxx-libiio-0.24-lib/lib'
PKG_CONFIG_PATH=/nix/store/sv94xjhaw6h1v490vhk3h93zw0jinzkc-protobuf-aarch64-unknown-linux-gnu-24.4/lib/pkgconfig:/nix/store/234mpzladw7a2rr3lir7w2x1x2bwia80-abseil-cpp-aarch64-unknown-linux-gnu-20230125.3/lib/pkgconfig:/nix/store/m2qqnaacng2dnxiqpxwqm8s2vvqw6ckk-openssl-aarch64-unknown-linux-gnu-3.0.12-dev/lib/pkgconfig:/nix/store/8frjmww7r7i5l257fvbgj07wxaiw6gmd-libiio-0.24-dev/lib/pkgconfig
buildInputs='/nix/store/65wgx7xdqx2dkds2qmv356ih3rnn80cm-glibc-aarch64-unknown-linux-gnu-2.38-27-static /nix/store/sv94xjhaw6h1v490vhk3h93zw0jinzkc-protobuf-aarch64-unknown-linux-gnu-24.4 /nix/store/m2qqnaacng2dnxiqpxwqm8s2vvqw6ckk-openssl-aarch64-unknown-linux-gnu-3.0.12-dev /nix/store/8frjmww7r7i5l257fvbgj07wxaiw6gmd-libiio-0.24-dev'

Is there anything I can do on my end? Much appreciated.

4

Thanks a lot. Sorry, again random guess (hopefully someone qualified will jump in at some points ^^), but after a quick inspection of the libiio derivation, I found this patch:

$ cat pkgs/development/libraries/libiio/cmake-fix-libxml2-find-package.patch
diff --color -ur a/CMakeLists.txt b/CMakeLists.txt
--- a/CMakeLists.txt    2022-06-02 02:57:01.503340155 +0300
+++ b/CMakeLists.txt    2022-06-02 02:54:33.726941188 +0300
@@ -378,7 +378,7 @@
        # So, try first to find the CMake module provided by libxml2 package, then fallback
        # on the CMake's FindLibXml2.cmake module (which can lack some definition, especially
        # in static build case).
-       find_package(LibXml2 QUIET NO_MODULE)
+       find_package(LibXml2 QUIET MODULE)
        if(DEFINED LIBXML2_VERSION_STRING)
                set(LIBXML2_FOUND ON)
                set(LIBXML2_INCLUDE_DIR ${LIBXML2_INCLUDE_DIRS})
Seulement dans b: good.patch

which clearly mentions that things could go wrong in the static build case (which is exactly where things break down in your case) since it uses instead the CMake version of FindLibXml instead of the libxml2 package, which, apparently, contains less stuff. The derivation say that this patch could be removed once libxml2.dev: cmake module doesn't have a correct LIBXML2_LIBRARY_DIR · Issue #125008 · NixOS/nixpkgs · GitHub is properly fixed.

That being sayed, I’m not so sure why you get the error message cannot find -liio, and what is this /home/hftsai/CytoRepo/pifds/target/aarch64-unknown-linux-gnu/debug/deps/liblibiio_sys-dcd8fe422efbc76a.rlib (why is it not coming from /nix/store?). This seems to be related to the static version of iio…

Also, if you want to force a -L, you can add options using something like:

NIX_LDFLAGS = "-L${lib.getLib json_c}/lib"

so maybe setting:

NIX_LDFLAGS = "-L${lib.getLib libiio}/lib"

or something like:

PKG_CONFIG_PATH = "${pkgs.libiio.dev}/lib/pkgconfig";

could help nix to find the proper libraries… But to be honest I don’t know if it will help.

Also, why do you say that it is the wrong architecture? It is not clear from the path you provide since when using the hack, you use now the aarch64 package which is not cross compiled.

Also, I tried to run your example, I can enter into a shell with:

$ nix develop

but then if I create a file:

fn main() {
    // Statements here are executed when the compiled binary is called.

    // Print text to the console.
    println!("Hello World!");
}

and compile statically, I get a different error:


[leo@bestos:/tmp/tt]$ rustc -C target-feature=+crt-static hello.rs 
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/nix/store/s1mgys5x5rxm28801yby886a8byk231z-rust-complete-1.74.0/lib/rustlib/x86_64-unknown-linux-gnu/bin:/nix/store/90h6k8ylkgn81k10190v5c9ldyjpzgl9-gcc-wrapper-12.3.0/bin:/nix/store/hf2gy3km07d5m0p1lwmja0rg9wlnmyr7-gcc-12.3.0/bin:/nix/store/cx01qk0qyylvkgisbwc7d3pk8sliccgh-glibc-2.38-27-bin/bin:/nix/store/bblyj5b3ii8n6v4ra0nb37cmi3lf8rz9-coreutils-9.3/bin:/nix/store/1alqjnr40dsk7cl15l5sn5y2zdxidc1v-binutils-wrapper-2.40/bin:/nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin:/nix/store/s1mgys5x5rxm28801yby886a8byk231z-rust-complete-1.74.0/bin:/nix/store/gwvk9jj6ygnq8b0xnysrd18rb8kc25jj-aarch64-unknown-li
[…]
  = note: /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: cannot find -lutil: No such file or directory
          /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: cannot find -lrt: No such file or directory
          /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: cannot find -lpthread: No such file or directory
          /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: cannot find -lm: No such file or directory
          /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: cannot find -ldl: No such file or directory
          /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: cannot find -lc: No such file or directory
          /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: cannot find -lc: No such file or directory
          collect2: error: ld returned 1 exit status

it it the procedure you followed?

Sorry for not being more useful…

5

Just, I tried a different approach: I just created via

$ cargo new helloworld

a new project (containing by default a basic hello world), then I created a default.nix containing:

let pkgs = import <nixpkgs> {}; in
# Uncomment to compile normally:
# pkgs.callPackage ./derivation.nix {}
# Uncomment to compile for aarch64, dynamically linked
# pkgs.pkgsCross.aarch64-multiplatform.callPackage ./derivation.nix {}
# Uncomment to compile for aarch64, statically linked:
pkgs.pkgsCross.aarch64-multiplatform.pkgsStatic.callPackage ./derivation.nix {}

(note that I load the static version of the libraries using the .pkgsStatic) and a derivation.nix containing:

{  rustPlatform }:
rustPlatform.buildRustPackage rec {
  pname = "myproject";
  version = "1.0.0";

  src = ./helloworld;
  
  cargoLock = {
    lockFile = ./helloworld/Cargo.lock;
  };
  
}

and then nix-build does build a statically linked, cross-compiled image (see above comments to get the dynamically linked version).

$ file result/bin/helloworld
result/bin/helloworld: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped

Is this solution working for you as well? If yes, why is it not satisfactory?

6

You are very welcome and thanks for all the efforts. At least we crossed out a few possible causes.

The local library liblibiio_sys-dcd8fe422efbc76a.rlib is the intermediate FFI dependency during the build process, so it comes from the local build directory.

and compile statically, I get a different error:

I remember I ran into those glibc linking issues if I forgot to bring up glibc.static in buildInputs.

Also, why do you say that it is the wrong architecture? It is not clear from the path you provide since when using the hack, you use now the aarch64 package which is not cross compiled.

You are absolutely correct. I didn’t realize the library is aarch64 and was confused because the folder does not contain aarch64.

❯ file /nix/store/hll0zwkabpcg8dg90l9w6hci05y0agxx-libiio-0.24-lib/lib/libiio.so.0.24
/nix/store/hll0zwkabpcg8dg90l9w6hci05y0agxx-libiio-0.24-lib/lib/libiio.so.0.24: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, not stripped

This path does appear in NIX_LDFLAGS. It is weird because rustc can find other required libraries during static builds.

The bare minimum hello-world works on my end as well but it does not invoke libiio. I may play around with it for a few days to see if that can help set up libiio.

Thanks again.