Rust: cross compiling no_std with custom target

I am trying to follow A Minimal Rust Kernel | Writing an OS in Rust and build the target using buildRustPackage. In particular, the code I’m trying to build is this: GitHub - phil-opp/blog_os at post-02

As far as I can tell, this package was part of a NixOS test before, but has been removed because it was broken. I have not figured out what exactly broke: tests.rustCustomSysroot: remove unmaintained and broken by upstream test · NixOS/nixpkgs@c4a06db · GitHub

I am now following the NixOS guide for cross complation. I have tried many different things; from using crossSystem, to rust-overlay, with custom toolchains and without; none seemed to have really worked well.

I am looking for a derivation that spits out my target; I’m not looking to just set up a shell and use rustup - I’m sure that works, but I don’t want that.

I currently have the following flake:

 {
  outputs = inputs@{ nixpkgs, flake-utils, ...}:
    flake-utils.lib.eachDefaultSystem(system: 
      let
        pkgs = import nixpkgs {
          inherit system;
        };

        # The book uses rust-lld for linking, but rust-lld is not currently packaged for NixOS.
        # The justification in the book for using rust-lld suggests that gcc can still be used for testing:
        # > Instead of using the platform's default linker (which might not support Linux targets),
        # > we use the cross platform LLD linker that is shipped with Rust for linking our kernel.
        # https://github.com/phil-opp/blog_os/blame/7212ffaa8383122b1eb07fe1854814f99d2e1af4/blog/content/second-edition/posts/02-minimal-rust-kernel/index.md#L157
        targetSpec = {
          llvm-target = "x86_64-unknown-none";
          data-layout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128";
          arch = "x86_64";
          target-endian = "little";
          target-pointer-width = "64";
          target-c-int-width = "32";
          # target-family = ["unix"];
          os = "none";
          executables = true;
          linker-flavor = "gcc"; #linker-flavor = "ld.lld"; #linker = "rust-lld";
          panic-strategy = "abort";
          disable-redzone = true;
          features = "-mmx,-sse,+soft-float";
        };

        pkgsCross = import nixpkgs {
          inherit system;
          crossSystem = {
            inherit system;
           
            rust.rustcTarget = "x86_64-smolos";
            rust.platform = targetSpec;
          };
        };

        kernel = pkgsCross.rustPlatform.buildRustPackage {
          name = "smol-kernel";

          src = pkgs.fetchFromGitHub {
            owner = "phil-opp";
            repo = "blog_os";
            rev = "post-02";
            hash = "sha256-xS6d3DwhUJmbLt4vgb3xoLV7Jpue0M2Zqs5blI4Ud08=";
          };
          cargoHash = "";

          doCheck = false;
        };
      in {
        packages = { inherit kernel; };
      }
    );
}

Building this derivation fails with the following error:

error: builder for '/nix/store/qi1033fdgibsxhzl5p3x4y64lgp8dy5m-x86_64-unknown-linux-gnu-rustc-1.73.0.drv' failed with exit code 1;
       last 10 log lines:
       > 100 |         for (desc, result, duration, stdout) in std::mem::take(&mut self.results) {
       >     |                                                 ^^^^^^^^^^^^^^
       >     |
       >     = help: add `#![feature(restricted_std)]` to the crate attributes to enable
       >
       > For more information about this error, try `rustc --explain E0658`.
       > warning: `test` (lib) generated 1 warning (1 duplicate)
       > error: could not compile `test` (lib) due to 11 previous errors; 1 warning emitted
       > warning: build failed, waiting for other jobs to finish...
       > Build completed unsuccessfully in 0:00:02
       For full logs, run 'nix log /nix/store/qi1033fdgibsxhzl5p3x4y64lgp8dy5m-x86_64-unknown-linux-gnu-rustc-1.73.0.drv'.
error: 1 dependencies of derivation '/nix/store/lff1shjxgs9yrp7q5c3h3rnf0slrngw1-smol-kernel-x86_64-unknown-linux-gnu.drv' failed to build

(The full logs here: https://pastebin.com/raw/hpKZuW5S )

I strongly suspect that already setting up rustPlatform with the custom target fails to build - given by the fact that test is failing to build during building x86_64-unknown-linux-gnu-rustc-1.73.0.drv and it does not complain that cargoHash is empty.

I feel like creating a custom target like that should be easily possible; what am I doing wrong?

1 Like

Ah ha;

Looking at rustc: fix build for no_std targets by newAM · Pull Request #190697 · NixOS/nixpkgs · GitHub it seems rust.rustcTarget must include -none or some other substring which indicates that we don’t want to build std. This is quite surprising to me but seems to get me past the current build error.

So something like rust.rustcTarget = "x86_64-smolos-none" seems to get me a step further.

This is what I currently have, which is sadly not working, still:

{
  outputs = inputs@{ nixpkgs, flake-utils, ...}:
    flake-utils.lib.eachDefaultSystem(system:
      let
        pkgs = import nixpkgs {
          inherit system;
        };

        targetSpec = {
          llvm-target = "x86_64-unknown-none";
          data-layout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128";
          arch = "x86_64";
          target-endian = "little";
          target-pointer-width = "64";
          target-c-int-width = "32";
          os = "none";
          executables = true;
          linker-flavor = "ld.lld";
          linker = "ldd";
          panic-strategy = "abort";
          disable-redzone = true;
          features = "-mmx,-sse,+soft-float";
        };

        pkgsCross = import nixpkgs {
          inherit system;
          crossSystem = {
            inherit system;

            rust.rustcTarget = "x86_64-smolos-none";
            rust.platform = targetSpec;
          };
        };

        kernel = pkgsCross.rustPlatform.buildRustPackage {
          name = "smol-kernel";

          nativeBuildInput = [ pkgs.lld ];

          src = pkgs.fetchFromGitHub {
            owner = "phil-opp";
            repo = "blog_os";
            rev = "post-02";
            hash = "sha256-xS6d3DwhUJmbLt4vgb3xoLV7Jpue0M2Zqs5blI4Ud08=";
          };
          cargoHash = "sha256-XsPILFbPdV59Bk10sF6N2r+AefHOEipAljbTa9VjS/g=";

          doCheck = false;
          auditable = false;
        };
      in {
        packages = { inherit kernel; };
      }
    );
}

With nixpkgs nixos-unstable I get:

       > Executing cargoSetupPostPatchHook
       > Validating consistency between /build/source/Cargo.lock and /build/smol-kernel-vendor.tar.gz/Cargo.lock
       > Finished cargoSetupPostPatchHook
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > Running phase: buildPhase
       > Executing cargoBuildHook
       > ++ env CC_0I57D34NDXCQAHRD43I5HYFFZPK16P79_X86_64_SMOLOS_NONE=/nix/store/qnd7d05bz809fls608f5fic4ysqbblpr-x86_64-unknown-linux-gnu-gcc-wrapper-13.2.0/bin/x86_64-unknown-linux-gnu-cc CXX_0I57D34NDXCQAHRD43I5HYFFZPK16P79_X86_64_SMOLOS_NONE=/nix/store/qnd7d05bz809fls608f5fic4ysqbblpr-x86_64-unknown-linux-gnu-gcc-wrapper-13.2.0/bin/x86_64-unknown-linux-gnu-c++ CARGO_TARGET_0I57D34NDXCQAHRD43I5HYFFZPK16P79_X86_64_SMOLOS_NONE_LINKER=/nix/store/qnd7d05bz809fls608f5fic4ysqbblpr-x86_64-unknown-linux-gnu-gcc-wrapper-13.2.0/bin/x86_64-unknown-linux-gnu-cc CC_X86_64_UNKNOWN_LINUX_GNU=/nix/store/khkhbch4p1wjfl1g89gw1mszvvr7bzv0-gcc-wrapper-13.2.0/bin/cc CXX_X86_64_UNKNOWN_LINUX_GNU=/nix/store/khkhbch4p1wjfl1g89gw1mszvvr7bzv0-gcc-wrapper-13.2.0/bin/c++ CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=/nix/store/khkhbch4p1wjfl1g89gw1mszvvr7bzv0-gcc-wrapper-13.2.0/bin/cc CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu HOST_CC=/nix/store/khkhbch4p1wjfl1g89gw1mszvvr7bzv0-gcc-wrapper-13.2.0/bin/cc HOST_CXX=/nix/store/khkhbch4p1wjfl1g89gw1mszvvr7bzv0-gcc-wrapper-13.2.0/bin/c++ cargo build -j 16 --target /nix/store/0i57d34ndxcqahrd43i5hyffzpk16p79-x86_64-smolos-none.json --frozen --profile release
       > error: "/nix/store/zab4yxcrllb3rrv6zr44ng6i2vklsbda-x86_64-unknown-linux-gnu-rustc-1.75.0/lib/rustlib/src/rust/Cargo.lock" does not exist, unable to build with the standard library, try:
       >         rustup component add rust-src
       For full logs, run 'nix log /nix/store/ys8rjxl6rlqndgiar5kskp3j8jw6b9x6-smol-kernel-x86_64-unknown-linux-gnu.drv'.

and with nixos-23.11 I get:

       > Running phase: buildPhase
       > Executing cargoBuildHook
       > ++ env CC_0I57D34NDXCQAHRD43I5HYFFZPK16P79_X86_64_SMOLOS_NONE=/nix/store/sjqxrqajawk7ynyzmy2spp8sbcbriwvr-x86_64-unknown-linux-gnu-gcc-wrapper-12.3.0/bin/x86_64-unknown-linux-gnu-cc CXX_0I57D34NDXCQAHRD43I5HYFFZPK16P79_X86_64_SMOLOS_NONE=/nix/store/sjqxrqajawk7ynyzmy2spp8sbcbriwvr-x86_64-unknown-linux-gnu-gcc-wrapper-12.3.0/bin/x86_64-unknown-linux-gnu-c++ CARGO_TARGET_0I57D34NDXCQAHRD43I5HYFFZPK16P79_X86_64_SMOLOS_NONE_LINKER=/nix/store/sjqxrqajawk7ynyzmy2spp8sbcbriwvr-x86_64-unknown-linux-gnu-gcc-wrapper-12.3.0/bin/x86_64-unknown-linux-gnu-cc CC_X86_64_UNKNOWN_LINUX_GNU=/nix/store/vylmp73qymnv4siaqn1kl2hghj07hrj8-gcc-wrapper-12.3.0/bin/cc CXX_X86_64_UNKNOWN_LINUX_GNU=/nix/store/vylmp73qymnv4siaqn1kl2hghj07hrj8-gcc-wrapper-12.3.0/bin/c++ CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=/nix/store/vylmp73qymnv4siaqn1kl2hghj07hrj8-gcc-wrapper-12.3.0/bin/cc CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu HOST_CC=/nix/store/vylmp73qymnv4siaqn1kl2hghj07hrj8-gcc-wrapper-12.3.0/bin/cc HOST_CXX=/nix/store/vylmp73qymnv4siaqn1kl2hghj07hrj8-gcc-wrapper-12.3.0/bin/c++ cargo build -j 16 --target /nix/store/0i57d34ndxcqahrd43i5hyffzpk16p79-x86_64-smolos-none.json --frozen --profile release
       > error: failed to run `rustc` to learn about target-specific information
       >
       > Caused by:
       >   process didn't exit successfully: `rustc - --crate-name ___ --print=file-names --sysroot /nix/store/gjcxxwxn9nyjmsbgyyvhaggv79qnr78b-custom-sysroot-x86_64-unknown-linux-gnu --target /nix/store/0i57d34ndxcqahrd43i5hyffzpk16p79-x86_64-smolos-none.json --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 1)
       >   --- stderr
       >   error: Option 'sysroot' given more than once
       >
       For full logs, run 'nix log /nix/store/5dzy1pi5bh69dk3rwhqyp6flq842mjsa-smol-kernel-x86_64-unknown-linux-gnu.drv'.

The latter I believe was fixed with wrapRustcWith: allow --sysroot to be overridden by alyssais · Pull Request #274853 · NixOS/nixpkgs · GitHub but even with that, I don’t get a successful build.

What’s kind of irritating is also that any change in the target spec leads me to both recompile rustc, as well as sometimes gcc. I’m sure I’m doing something wrong, so any advice would be apprechiated.

So I gave up in the end and just opted to use rustup for now. It’s overall the way better experience because it doesn’t involve recompling any rustc. It’s unfortunate but oh well…

{
  outputs = inputs@{ nixpkgs, flake-utils, ...}:
    flake-utils.lib.eachDefaultSystem(system:
      let
        pkgs = import nixpkgs {
          inherit system;
        };

        target-spec = pkgs.writeText "bare-target.json" (builtins.readFile ./bare-target.json);
      in {
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            clang
            llvmPackages.bintools
            rustup

            cargo-bootimage
            qemu
          ];

          RUSTC_VERSION = "nightly-2024-02-01";
          LIBCLANG_PATH = pkgs.lib.makeLibraryPath [ pkgs.llvmPackages_latest.libclang.lib ];
          shellHook = ''
            export PATH=$PATH:''${CARGO_HOME:-~/.cargo}/bin
            export PATH=$PATH:''${RUSTUP_HOME:-~/.rustup}/toolchains/$RUSTC_VERSION-x86_64-unknown-linux-gnu/bin/
          '';
          
          # Add precompiled library to rustc search path
          RUSTFLAGS = (builtins.map (a: ''-L ${a}/lib'') [
            # add libraries here (e.g. pkgs.libvmi)
          ]);

          # Add glibc, clang, glib and other headers to bindgen search path
          BINDGEN_EXTRA_CLANG_ARGS = 
            # Includes with normal include path
            (builtins.map (a: ''-I"${a}/include"'') [
              # add dev libraries here (e.g. pkgs.libvmi.dev)
              pkgs.glibc.dev 
            ])
            # Includes with special directory paths
            ++ [
              ''-I"${pkgs.llvmPackages_latest.libclang.lib}/lib/clang/${pkgs.llvmPackages_latest.libclang.version}/include"''
              ''-I"${pkgs.glib.dev}/include/glib-2.0"''
              ''-I${pkgs.glib.out}/lib/glib-2.0/include/''
            ];
        };
      }
    );
}

with bare-target.json:

{
  "llvm-target": "x86_64-unknown-none",
  "data-layout": "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128",
  "arch": "x86_64",
  "target-endian": "little",
  "target-pointer-width": "64",
  "target-c-int-width": "32",
  "os": "none",
  "executables": true,
  "linker-flavor": "ld.lld",
  "linker": "rust-lld",
  "panic-strategy": "abort",
  "disable-redzone": true,
  "features": "-mmx,-sse,+soft-float"
}