I have a small Haskell application which I used to build and deploy with Nix. This ended up being too much of a maintenance burden and I wanted to switch to a Docker container that I can deploy with Fly.io. I also own both a MacBook Pro and a desktop computer, the latter running NixOS.
What I didn’t know, although it makes a lot of sense in retrospect, is that, even when using dockerImage, the application is still built through Nix, outside of Docker. As a result, I get a MachO binary, which doesn’t work with the Linux Docker container that I want to deploy. So I need to cross compile things. When I tried doing that I ran into so many errors that I first wanted to see if I can cross compile something smaller. Then I remembered that I already asked a question about Docker once and so I just took the repository I created for this issue and tried building it again, after running nix flake update. The repo is here.
If I run nix build in that repo, after a long, long time of rebuilding the world, I get:
builder for '/nix/store/ijc5viqyvbr5fwqzx8zqx2c94c47872d-x86_64-unknown-linux-gnu-stage-final-gcc-11.2.0.drv' failed with exit code 2; last 10 log lines:
checking the number of available CPUs... 8
checking whether the target supports __sync_*_compare_and_swap... yes
checking for CET support... yes
configure: updating cache ./config.cache
checking that generated files are newer than configure... done
configure: error: conditional "ENABLE_DARWIN_AT_RPATH" was never defined.
Usually this means the macro was only invoked conditionally.
make[1]: *** [Makefile:15634: configure-target-libgomp] Error 1
make[1]: Leaving directory '/private/tmp/nix-build-x86_64-unknown-linux-gnu-stage-final-gcc-11.2.0.drv-0/build'
make: *** [Makefile:974: all] Error 2
cannot build derivation '/nix/store/hxlx40wzpwm60xyq132bki5gh79wrbap-x86_64-unknown-linux-gnu-stage-final-gcc-wrapper-11.2.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/n89s4cdb54yvq6z4rjr9qy7qb4pnbjr4-stdenv-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/qyzvaaylkz2h8pack4i8lism6hlfwkmn-hello-x86_64-unknown-linux-gnu-2.12.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/fl1wz6rm77w94s76f5b6knv8b9hf3mv4-foobar-config.json.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8y1301aps5xnmgip3kqkfhn5xccngs35-foobar-config.json.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/hr4h67ckkipflb9spg8d49q0d4jj97fd-docker-image-foobar.tar.gz.drv': 1 dependencies couldn't be built
error: build of '/nix/store/hr4h67ckkipflb9spg8d49q0d4jj97fd-docker-image-foobar.tar.gz.drv' failed
I’d be willing to debug this a bit further before giving up and ditching Nix for building, but I don’t have any experience with cross compilation.
I have had similar experiences with building python (specifically a python module that needed rust/musl to compile) docker images on darwin and it seems like even cross compiling is enough, nix wanted a linux builder, though a substituter worked for trivial stuff.
I think the most promising solution would be to set up a remote builder running Linux and let nix automatically forward the build to that machine.
For this you’ve got a broad range of options. Either you’ve got a Linux machine already running somewhere (for example your desktop computer running NixOS) or you could set up a virtual machine locally (See e.g. GitHub - nix-community/linuxkit-nix: An easy to use Linux builder for macOS [maintainer=@nicknovitski]) or if you’ve got some budget for this you could use a service such as https://nixbuild.net/.
I was able to spend some more time on this topic. I won’t use any paid solution since we’re talking about really small projects I do just for fun. I then looked at GitHub - nix-community/linuxkit-nix: An easy to use Linux builder for macOS [maintainer=@nicknovitski] but it honestly seems like a lot of complexity. Considering my alternative is to ditch Nix and just use the default Haskell Docker image, it seems like adding yet more layers isn’t something I want to do.
Then I looked at GitHub - LnL7/nix-docker: Docker images for the Nix package manager and used it to craft my own remote builder using the official Nix Docker image. To rule out mistakes on my part, I tried to use the LnL7 remote builder too. But in both cases I get “unable to load seccomp BPF program: Invalid argument”:
$ nix build .#packages.x86_64-linux.docker --builders 'ssh://nix-docker x86_64-linux'
error: build of '/nix/store/yaf6fy6wdjfkd2qka9ifgjsasy9zj2q1-build-spago-style.drv' on 'ssh://nix-docker' failed: while setting up the build environment: unable to load seccomp BPF program: Invalid argument
error: builder for '/nix/store/yaf6fy6wdjfkd2qka9ifgjsasy9zj2q1-build-spago-style.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/jcaq8l839jcv1w5vb7mli4rkyy9pkc38-lions-client.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0xw07gaii4r1syn28fkn8ays101z2r45-lions-all-client-assets.drv' failed to build
error: 1 dependencies of derivation '/nix/store/jhf2ka90dpbjfq6ajyv7mkn1l7jmj92y-lions-website.drv' failed to build
error: 1 dependencies of derivation '/nix/store/dpg0qb0valcxppm2l5f5g41kfn66isbk-server-config.json.drv' failed to build
error: 1 dependencies of derivation '/nix/store/niqlffyvy725j844dwxivys13pb84kn0-docker-image-server.tar.gz.drv' failed to build
I thought it might be related to the sandboxing difference between the official Docker image (disabled) and the default install (enabled). So I enabled it in the container and ran it with --privileged but same issue.
So I now resorted to filter-syscalls = false. The documentation is vague and just says:
Whether to prevent certain dangerous system calls, such as creation of setuid/setgid files or adding ACLs or extended attributes. Only disable this if you’re aware of the security implications.
I have no idea what the security implications are. I am inclined to say it’s a local builder that I control. On the other hand I’m building packages I don’t control so they might make syscalls I don’t like. On the other hand the whole thing happens in a container. So I guess… it’s fine? But it feels like I shouldn’t have to do any of these things.
I have to say the divide between “Haskell Cross Compilation is best-in-class on Nix” and the reality of “everything is more or less broken” is pretty extreme.
Luckily there’s a fix for linked issue about cross compilation (the fix which worked on a toy example and I’ll try this with the actual application next.