System-units & etc take too long when using binfmt qemu

Development Deployments
1

I’m deploying a Go service from my computer (x86_64-linux) to a remote server (aarch64-linux) using nixos-rebuild like this:

nixos-rebuild switch --flake .#server --target-host root@server.example.com

Right now I’m only cross-compiling the Go service and the SQLite lib it needs, the rest of the packages are not cross-compiled, just using the default aarch64 ones by enabling qemu binfmt on my computer:

boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

It works great, it’s reusing a lot of the aarch64 binary packages from the cache but there’s a little thing I want to see if I can optimize.

Problem is when deploying to the server there are 2 steps (system-units & etc) that take too long to execute (10-20 seconds each). It’s not too much, but I want to check if it’s possible to do something about them, I guess maybe cross-compile them?

I suppose qemu it’s doing something on those steps and I know it is quite slow when doing things like compiling…

Any pointers/suggestions?

Example output showing the culprits:

these 5 derivations will be built:
  /nix/store/q759xy0q6c32mrwx18yds9dc06397s4x-myGoService-aarch64-unknown-linux-gnu-1.0.0.drv
  /nix/store/lq1ip0jm6z8iymfykd20bwk68zf7ag6w-unit-myGoService.service.drv
  /nix/store/gfkp9ipwz9p4hpljc926drqzcckh1zd7-system-units.drv
  /nix/store/wbizm15hyj0sfabfm5d4g7gdfwdmwi3k-etc.drv
  /nix/store/y6hxfa5kxaiaqb9qv9vm4w5a53hjxr7k-nixos-system-nixos-23.11.20240301.79baff8.drv
building '/nix/store/q759xy0q6c32mrwx18yds9dc06397s4x-myGoService-aarch64-unknown-linux-gnu-1.0.0.drv'...
building '/nix/store/lq1ip0jm6z8iymfykd20bwk68zf7ag6w-unit-myGoService.service.drv'...
building '/nix/store/gfkp9ipwz9p4hpljc926drqzcckh1zd7-system-units.drv'... <<<<<< THIS ONE IS SLOW
building '/nix/store/wbizm15hyj0sfabfm5d4g7gdfwdmwi3k-etc.drv'... <<<<<< THIS ONE TOO
building '/nix/store/y6hxfa5kxaiaqb9qv9vm4w5a53hjxr7k-nixos-system-nixos-23.11.20240301.79baff8.drv'...
Running nix-copy-closure with these NIX_SSHOPTS:  -o ControlMaster=auto -o ControlPath=/tmp/nix-shell.uJaEHv/nixos-rebuild.hByWpm/ssh-%n -o ControlPersist=60
2

You’re using binfmt qemu emulation and complaining about 10-20 second delays? Sorry, that just comes with the territory. Emulating an architecture is slow work, and utilizing that for something as slow as shell scripting only exacerbates it.

1 Like
3

Yep, that I know :cry:

My question is more about if there is some kind of way to cross-compile these 2 derivations the same way I’m cross-compiling my Go service (without cross-compiling everything, ofc)

4

It’s theoretically possible to cross compile systemd units and mix them in a native compiled system, but for correctness reason, this sort of dangerous mix up is not easy to enable.

You would have to pave your way in the nixpkgs machinery to have such mechanisms. I fear it is much easier to accept the aarch64 situation or obtain an native aarch64 builder.

Oracle Cloud offers free aarch64 VM with 4 cores, should be enough for that.

1 Like
5

Ok, thanks for the answer!