Proposal: a dedicated cross-compiling team?

Hello! I’ve discovered that cross-compiling in Nixpkgs needs a bit more love <3 - so I decided to provide it and I want to propose to form a team dedicated to testing and fixing cross-compilation problems in Nixpkgs.

Currently I’m filing PRs/issues as I go, building a replacement SD card for my old Raspberry Pi 2. I’m pretty sure I’m not the only one with this particular board or similar armv7l boards, and I’m pretty sure I’m not the only one who wants to see NixOS on them.

The last big armv7l effort was made by @Dezgeg who hosted a binary cache with native armv7l packages, but it hasn’t been updated in months, and as I quickly found out, a single RPi2 can’t handle some big packages like GCC without swap-space. So right now, cross-compiling may be the best option.

Here’s what I’ve currently found:

nixpkgs = {
  crossSystem = {
    system = "armv7l-linux";
  };
  localSystem = {
    system = "x86_64-linux";
  };
};

This snippet theoretically should allow for using nixos-rebuild with cross-compiled packages, with the exception that packages should be built specifically on x86_64 hosts - this is a limitation of Nix, because it requires for a clear dependency chain, and a cross-stdenv isn’t considered equal to a native stdenv. (Thankfully, x86_64 is common in today’s desktops and laptops). You also need to configure a remote x86_64 builder in configuration.nix if you want to use nixos-rebuild directly on the armv7l machine!

Second, some packages don’t cross-compile and right now I don’t know why. Maybe someone with more knowledge can help?

  • apr (filed as #67259)
    • Prevents apacheHttpd from compiling and PHP from being used as an Apache module
    • Gentoo somehow solved it by manually configuring stuff with undocumented(?) variables and patching
  • clucene_core_2 (filed as #67505)
    • Is a dependency of Dovecot, optional in upstream but hardcoded in Nixpkgs
    • Provides full-text search I suppose?
    • According to upstream documentation, not very good at it, but the alternative is Solr, which is AFAIK, big
  • git (may be solved by PR#61552)
    • fails to link to libcurl
    • lack of curl-config when cross-compiling poses a problem
    • could be simple to solve
  • perlPackages.moduleBuild (filed as #66741 by @lopsided98)
    • Provides Perl support in git
    • Seems to only be needed in git-instaweb?
  • uwsgi.override { plugins = ["python3"]; } (not yet filed as an issue)
    • Right when its built, it tries to run armv7l python executable, I can’t figure out where the call is made
    • Vanilla uwsgi seems to build fine, according to my Hydra instance
  • judy (filed as #67619)
    • Tries to run native binary compiled from sources when cross-compiling
    • Is a dependency of mariadb
  • redis (filed as #51517 by @furrycatherder but unclear and old, maybe refile?)
    • Fails to find jemalloc
    • Hardcoded unprefixed ar (probably a good thing to fix upstream?)

Also, I plan on creating a binary cache based on the configuration.nix snippet above, so people would have an easier time compiling stuff. Right now I’m not sure if I’ll be able to keep up with master, so I’m manually launching builds. Also, the cache and Hydra instance aren’t yet public (but I plan to fix this!).

11 Likes

Yeah, this is kind of a tricky thing to solve. It would be cool if we could tell Nix that we don’t care about where this was built on, just what it is built for, but I can’t think of a good way to solve it without introducing some magic. Maybe fully reproducible builds will solve this, though, I don’t think anyone has put work into making things reproducible /between/ build architectures before.

Using remote builders with a cross-compiled Raspberry Pi works surprisingly well for now. It’s a weird setup, but you probably don’t want to compile that much stuff on those things anyway.

Thanks for opening those issues though! It is definitely helpful in tracking these things down. I think the git one is solved by @nh2 in git: Don't depend `curl-config` when cross-compiling. by nh2 · Pull Request #61552 · NixOS/nixpkgs · GitHub.

1 Like

Yes, indeed, it should be solved by this. I’ll pull this branch into my local checkout and test it.

I think one can solve it by somehow telling Nix that “this derivation produces equal stuff to this derivation”, so Nix could substitute its hash with the hash of native stdenv.

Except I think it’s not a great idea, because cross-compiling may introduce bugs that are not seen with native compiling. We sometimes have to change flags, and that could also contaminate a package. This will NOT contribute to reproducibility…

1 Like

It would be cool if we could tell Nix that we don’t care about where this was built on, just what it is built for, but I can’t think of a good way to solve it without introducing some magic.

There are two ideas I have on this topic.

Either the completely-ugly-and-hackish one I hope we don’t need: just
have a cross-compiling hydra spoof its build parameters so that it
builds by cross-compiling but places the built artifact under the hash
of the native-compiled derivation.

Or the not-as-ugly-as-the-previous-one solution, which would work by
having nix have a way to say “I want either this derivation if it can be
found in a binary cache, or null” via some builtin. With this scheme
it would be possible to have nix code attempt downloading derivations
from cross-compiling hydras and, if it fails, by native-building them
locally.

However, the second solution would have to trigger network traffic
during evaluation, which would also be an issue.

I can’t find any better idea, unfortunately…

1 Like

So the second solution is specifying compatible hashes. Basically telling Nix that binaries produced by /nix/store/-stdenv-armv7l-unknown-linux-gnueabihf.drv and /nix/store/-stdenv.drv are equivalent and these are interchangable. This should probably be specified as an attribute in Nixpkgs, to prevent exploitation - meaning that Nix won’t accept just anything under a certain hash. To preserve tracing of hashes, we could make Nix include hashes of ALL variants of interchangable derivation in the hash - this would close possible attack vectors from rogue binary caches.

Filed some more issues and found some old relevant issues.

When I first started looking at cross-compiling in Nix, I had a similar desire to be able to mix native and cross-compiled packages, but after seeing all the special cases required to make many packages cross-compile, I realized that solutions such as those mentioned in this thread would lead to all sorts of subtle bugs of the kind that Nix was created to avoid. The only sane way I can see to make this work would be cross/native reproducibility and the intensional store.

Personally, I think the best use of people’s efforts is to create, review and test cross-compiling related PRs so we can fix the last few broken packages required to cross-compile a practically useful NixOS system. Most of the necessary PRs have already been submitted, but need a little bit more work or have stalled for lack of reviewers. @samueldr showed that this is already possible with a couple of configuration tweaks and package overrides, but I would like to remove the need for any configuration changes.

We also need to prevent regressions, ideally by having ofborg and Hydra cross-compile packages. For cross-compiled NixOS to be practically useful, we need to fix a number of closure size issues, usually due to build arch references. For example, [WIP] gcc: put target-specific libs in lib output by illegalprime · Pull Request #58606 · NixOS/nixpkgs · GitHub fixes the issue where almost every cross-compiled package ends up depending on gcc. Many people do not check for build references when making cross-compiling PRs, so there are quite a few packages to fix.

I have a branch that consolidates a number of cross related patches (along with the other downstream patches that I use), but it still is not enough to cross-compile any of my systems.

The cross-compiling effort might benefit from some clear goals and organization, but I think the main issue is that there are relatively few people who have the skills and interest in working on cross-compiling, so things tend to move slowly.

6 Likes

Yeah, a full NixOS cross compiled ISO in Hydra would be nice. We have a Hydra jobset for cross here: https://hydra.nixos.org/eval/1539009, which I use to track major regressions. Nothing there makes a full NixOS system though.

I think using the cross-compilation label would help organize these things a little bit. I updated some labels in GitHub to use it. You can see some of the backlogged prs here:

https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+is%3Apr+label%3A"6.topic%3A+cross-compilation"

2 Likes

I have actually created a binary cache and a Hydra instance to track stuff! I called it Project Glovebox (because arms wear gloves :3) and it’s here: https://glovebox.fireburn.ru/

It tracks mostly packages I’m interested in along with some common stuff. The repo it’s pulling some very crude and hacky fixes that are not for upstreaming and release.nix from is there: https://git.fireburn.ru/nixos-armv7l.git

There is no GitHub mirror, sadly, so you’ll need to use git request-pull and send patches to vika@fireburn.ru if you want to contribute. I might think about setting up a GitHub mirror if this gets traction. For now it’s easier for me to host it on my machine.

Please note that the repo includes some cases of using family-inappropriate language since debugging cross-compiling bugs is sometimes a huge pain in the you-know-where.

1 Like

While trying to cross-compile to aarch64, I found several packages which explicitly forbid cross-compiling.

Usually those try to compile and run some stuff during configure phase. They use native compiler, but compile for target platform, so executables won’t run on native.

Is there any way to fix this issue except making ./configure patches? Even then, what is a common way to do that? Are there examples of existing packages which do that?


The workaround I use now is to:

let pkgs = import ./nixpkgs {
        config = {};
        overlays = [];
    };
    pkgsArm = import ./nixpkgs {
        config = {};
        overlays = [];
        system = "aarch64-linux";
    };
    pkgsCross = import ./nixpkgs {
        config = {};
        overlays = [
            (self: super: {
                hdf5 = pkgsArm.hdf5;
                enca = pkgsArm.enca;
            })
        ];
        crossSystem = pkgs.lib.systems.examples.aarch64-multiplatform;
    };
in {
  ... use pkgsCross or pkgs
}

This works because my ./nixpkgs is on published channel commit. So instead of building, I fetch results from Hydra.

Maybe we should create meta.crossBroken for such packages?


I also found one case when the naming of target platform is an issue. It is named aarch64-unknown-linux-gnu-gcc, however configure script expects smth like aarch64-gnu-gcc (pkgs.libvpx). Renaming target platform to aarch64_unknown_linux-gnu-gcc would fix an issue, anybody knows other packages which have problems like this?

3 Likes