Remote cross-compile for NixOS

Help
1

I have a variety of ARMv7 and ARM64 devices with very slow storage, and mass-rebuilds on these devices are quite painful. I also have a mostly spare AMD64 machine with an SSD and lots of RAM, and would very much be interested in using this horsepower for building everything for the ARM machines.

Is this possible?

I’ve been trying to piece together the cross-compile story from the GitHub Issue tracker, but I’m finding it hard to get a foot-hold. I know a lot of progress has been made lately, but I can’t tell if what I want is feasible.

5 Likes
2

Documentation is a bit sparse. Checkout Cross Compiling - NixOS Wiki
You can also ask in #nixos-aarch64 on IRC. We also have a beefy community server for nixos: GitHub - NixOS/aarch64-build-box: Config for the Community aarch64 NixOS box [maintainer=@grahamc]

3

Hi @Mic92, thanks for replying.

Interesting! Looking through aarch64-build-box a little, it seems nix.distributedBuilds and nix.buildMachines looks super close to what I want, except (I think) for the cross-compile part. I had completely missed them before.

I guess what I’m really after is basically just this, except that the builders will automatically spin up a cross-compiler whenever I request a build from a machine of a different architecture.

I… presume this is not currently how this works (although, what is the system field of the buildmachines entries for?), but would be very happy to learn I’m wrong!

4

I am not sure, if you can use distributedBuilds for cross compiling yet,
but if you request access to the aarch64-build-box it should have enough horse power.

5

My approach is to have a Hydra instance and to have several ARMv7 build slaves. This allows me to simultaneously build for both architectures and everything is cached on the big server in EC2. You should be able to set up something similar. Let me know if you’d like help setting up Hydra or want to collaborate (perhaps via cachix?) to deduplicate ARMv7 efforts.

6

Cross builds and native builds have different DRV hashes, and thus look unrelated to Nix, so transparent mixing of native and cross builds is not possible. It would be possible to add some support for this with e.g. “derivation equivalence classes”, but it would be unsafe because there’s no guarantee that the native and cross derivations actually produce the same thing.

3 Likes
7

Ouch. Excellent point.

Then, would you say it is possible/most feasible to just run a QEMU-based ARM VM on my big machine with its own whole NixOS/store, and just use that as a remote builder? That sounds relatively simple to me… I think? (Guidelines welcome, especially if I can configure it declaratively, from /etc/nixos/configuration.nix!)

8

The VM performance penalty is pretty high. Even without Hydra you can set up remote builders for nix in nix.conf, then just set the system parameter for each build you want to do. I’ve found that easier than trying the QEMU route.

9

I am interested in anyone’s experience with a cross compiling setup, but even that ended up harder than just putting a few SBC build slaves on a network.

10

Cross builds and native builds have different DRV hashes, and thus
look unrelated to Nix, so transparent mixing of native and cross builds
is not possible.

I think this is only due to our current stdenv. If the default builder
was similar to this (pseudo-code):

curSystem="$(uname -m)"
if [ "$curSystem" = "$targetSystem" ]; then
    # normal build
else
    # cross-compiling build
    # (with optionally additional checks depending on
    # curSystem / targetSystem to figure out which cross-compiler
    # to use, etc.)
fi

Then it should work naturally (I hope?). Then I’m not up-to-date at all
on what’s the current state of cross-compilation in nixpkgs, so I may be
telling nonsense :slight_smile:

That said this is still risky (I’d rather avoid using “unsafe” without a
threat model defined) if the two build scripts get out-of-sync, but I
don’t think it would require any change to nix itself, and I think it
could be done with nixpkgs changes only (and thus would not change
anything from a local privilege escalation or similar attacks).

11

Even without Hydra you can set up remote builders for nix in nix.conf, then just set the system parameter for each build you want to do. I’ve found that easier than trying the QEMU route.

But, am I right in thinking this wouldn’t let me use my actual (x86) hardware? My situation is that I have ARM machines that are painfully slow, and a spare x86 machine that is fast. With a Hydra instance running somewhere as you describe, I’d still need actual ARM hardware somewhere to build anything for ARM, right?

Maybe, though. The biggest bottleneck on my ARM hardware is actually the SD-based root fs. They also don’t have enough RAM (2GiB, fixed) to put /tmp on a tmpfs the way I do on the x86 machine, so that card gets absolutely hammered. I guess in the worst case I could get an additional ARM board, set it up with an SSD and dedicate it to building…

12

The VM performance penalty is pretty high.

Also, how high are we talking? 50%? I’m happy to soak up quite a bit of loss if I can make use of this otherwise-idle hardware.

1 Like
13

I suppose that would be possible, but it would mean duplicating the nativeBuildInputs for every possible platform.

14

tl;dr: skip to the last section delimited by “-----”, which looks like
the least worse idea I could come up with

(note to self: nativeBuildInputs is depsBuildHost)

Hmm… you’re talking about duplicating the nativeBuildInputs for every
build platform, right? If so then that’s true indeed, I hadn’t thought
about that… maybe something like [1] in the sandbox would solve the
issue? That said it requires changes in nix once again…

15

Ugh, looks like the mail interface cut off part of my message. Let’s try
again:

tl;dr: skip to the last section delimited by “=====”, which looks like
the least worse idea I could come up with

(note to self: nativeBuildInputs is depsBuildHost)

Hmm… you’re talking about duplicating the nativeBuildInputs for every
build platform, right? If so then that’s true indeed, I hadn’t thought
about that… maybe something like [1] in the sandbox would solve the
issue? That said it requires changes in nix once again…

16

Discourse… OK, third time lucky?

tl;dr: skip to the last section delimited by # Hashing functions which
looks like the least worse idea I could come up with

(note to self: nativeBuildInputs is depsBuildHost)

Hmm… you’re talking about duplicating the nativeBuildInputs for every
build platform, right? If so then that’s true indeed, I hadn’t thought
about that… maybe something like [1] in the sandbox would solve the
issue? That said it requires changes in nix once again…

builder = [(condition, builder script)]

An alternative could be to have nix take a list of (condition, builder
script), and use the first builder script for which the condition
passes. It’s almost the same as the “derivation equivalence class” idea
you mentioned above, but avoids the problem of safety by just putting
both the cross- and the non-cross-build in the “trusted package”.
However, it still requires a way to hash the [(cond, build)] list, but
AFAIR that can be done without ever actually fetching the dependencies,
just by computing the dependency graph of every nativeBuildInputs for
every build platform.

Pros and cons of first-class cross-compiling

Even though this may make evaluating a package slower, I must say that,
to me, this’d be a killer app of nix: afaiu, distcc and the like, though
they theoretically support cross-compiling, actually are already way too
hard to make work in non-cross-compiling setups for anyone to actually
try to use them cross-compiling.

Then, the net loss is potentially greater than the net benefit for the
average user who doesn’t cross-compile? If so, a flag “Use
cross-compile-friendly build” could make sense, that’d change the build
scripts to require a version of nativeBuildInputs for every possible
platform.

Objective

Basically, what I’m hoping is a way for RPi’s to fetch binaries from a
reasonably efficient hydra (so not from a RPi build farm :innocent:), while
still being able to locally build packages that are not on the hydra. In
order to have this, the only solution I could think of is to have the
build script be invariant of the builder.

Unfortunately, as you mention, this seems to imply at best computing the
hash of the dependency graph of all possible nativeBuildInputs (in order
to not allow an untrusted user to insert a maliciously-built package
into the store), and at worst downloading/building the nativeBuildInputs
for every possible build blatform.

Hashing functions

But while writing this I noticed, there may be a third solution. This
third solution would be to have the builder script be parameterized by
the build platform.

However, this would require a completely reworked hashing scheme (being
able to hash a function, and not only a string), and most likely large
architectural changes in nix.

Then, current builder scripts would mean _: "current builder script",
and there would be an easy path forwards: actually turn the builder
script into a function of the build, that would pass it to the relevant
depended-upon derivations.

And then, there would need to be no duplication of the depsBuildHost,
nor duplication of the computation of the dependency graph. In exchange
for hashing over functions instead of hashing over strings.

Before I raise this on the nix issue tracker, what do you think of this
idea? Is it as flawed as the previous one without my noticing?

[1]
https://github.com/edolstra/nix/commit/a9cbd67f90d15751108c4da128bc542ce9daf25e

17

i think that in a real sense native build might differ significantly anyway; at least some of the tests can be run only on a native platform…

18

That would also hurt reproducibility right? Transparent cross-compiling sounds very useful, but should be opt-in.

19

Well, for instance with clang, I don’t think it would hurt
reproducibility: it’s the same codebase that would generate the two
executables anyway. (well, so long as the builder script is actually
written in a sensible way)

With gcc, however, I don’t really know, gcc’s cross-compilation
story is… not a great success story, so it might hurt reproducibility
indeed.

20

Well, for instance with clang, I don’t think it would hurt
reproducibility: it’s the same codebase that would generate the two
executables anyway. (well, so long as the builder script is actually
written in a sensible way)

And even then: the optional-dependency checks may work differently in
the native case and in the cross-compilation case, data file generation
may work differently, etc.