Getting started with Nix on ppc64le

Hi everyone,

I’m a big fan of NixOS and would love to use it on a Raptor Talos II. I know that the NixOS distro as a whole is not ready for ppc64le, but I’m hoping it’s possible to start running the standalone package manager. @r-burns seems to have done a lot of great work on this front recently.

How do I get started running Nix on top of Debian ppc64le? The install script simply says there are no bootstrap binaries available for my architecture.

Edit: I do have an amd64 NixOS machine available for cross-compiling, I’m just not sure what to invoke.

Awesome! I would say the first step is to build the Nix package manager from source for ppc64le. In the past I’ve done this natively on the target machine by hand using dependencies from APT. Alternatively, it may be easier to try cross-compiling. For example, on your amd64 machine, nix-build '<nixpkgs>' -A pkgsCross.powernv.nix && nix-copy-closure --to=user@talos result/ (substituting your username and hostname for user@talos). pkgsCross.powernv is the special crossPkgs set for typical ppc64le machines, so you can also use that for any other packages you wish to cross-compile. Some subset of packages are regularly cross-compiled on Hydra (Hydra - Evaluations of jobset nixpkgs:cross-trunk) so you will hopefully be fetching from the Hydra cache rather than building locally.

Once you have a working Nix on your ppc64le machine, you will need to make some modifications to Nixpkgs to build packages natively. For now, I’d take a look at my ppc64le branch (GitHub - r-burns/nixpkgs at ppc64le) which is volatile but at least tested to work for stdenv and some common packages. This should take care of most trivial modifications to stdenv/bootstrap-tools, and work around some more difficult issues such as Cross bootstrap tools have broken stackprotector support · Issue #113977 · NixOS/nixpkgs · GitHub.

AFAIK @CrystalGamma also owns a Talos and was the one who originally added support for ppc64, so may have additional hardware-specific tips.

1 Like

Hi … so yeah, you basically have 2 ways to get bootstrapped: doing the cross-build, as described by @r-burns above, or using the native (“impure”) stdenv, which I haven’t done in ages, so I don’t remember the exact process.

I also have a personal branch I keep rebasing on top of nixpkgs master: GitHub - CrystalGamma/nixpkgs at local1 Compare and contrast with @r-burns 's branch (in fact, I’ll be yoinking that s2n patch, up to now I was just disabling the PQ code altogether). Keep in mind 2 of the commits just add the bootstrap blobs to the repo. Unless you trust me with building them (and github not to manipulate them), might want to build them yourself.

I daily drive my Talos with a KDE desktop, so I wouldn’t say it’s not ready (though upstream clearly is). In addition to that nixpkgs branch I use an overlay that patches qtwebengine and a configuration setting that lets Xorg use the right GPU (no designated ‘default’ GPU on Power, though if there’s exactly one on the system, I’d have expected it to just use that one by default). If you want to run a desktop on your Talos, I can share those with you too (would be a bit more work, because I don’t feel comfortable sharing the entire repo they’re in).

2 Likes

Thanks for the helpful answers! Cross-compiling worked beautifully, but nix-copy-closure requires nix to already be present on the other end. I’m trying to figure out how to manually install by cross referencing the official install script, but that seems to assume nix-2.3.11-$system.tar.xz contains a script file install, which I’m not seeing in pkgsCross.powernv.nix.

I am interested in using the Talos as a KDE workstation… @CrystalGamma which distro are you running? Nix on top of Debian, or have you actually managed to get NixOS installed?

I’m running actual NixOS, though the only thing I remember about the bootstrapping process is that it was complicated and longwinded (though I probably took a very roundabout approach).

You should be able to nix-copy-closure by mounting the drive in your cross-builder system. You’ll probably need to manually write a bootloader configuration to start it on the Talos though, since I assume the activation script will call ppc64le binaries. My branch has a patch that changes the syslinux configuration generator to work with petitboot (not sure if that’s even necessary anymore, haven’t checked in a long while). Anyway the bootloader configuration needs to set init=/nix/store/$your_system_derivation_path/init in the kernel command line; you can also find the kernel and initrd in the system derivation path. Hope this helps.

1 Like

Thanks, it’s helpful to even know that someone has pulled off the bootstrap! Mounting the boot drive on the x64 machine is a good idea. I’ve been booting off a USB3 SSD anyway since I opted out of the SATA controller for my Talos.

When I get this working I will update the thread with my steps.

I am not familiar with cross-building, but maybe you could tar up the closure with something like

tar czf nix.tar.gz $(nix-store -qR result)

and then copy over the tarball and unpack it?

1 Like

This proved to be a tricky problem, but I’ve finally got Nix running on top of Debian ppc64le! The missing step involves a file called .reginfo, which the installer requires to find store paths from an adjacent folder (rather than installed into /nix/store with an active database).

Googling for .reginfo was difficult. First I found references to pkgs/build-support/kernel/paths-from-graph.pl, but I couldn’t get it to work and it appears to be deprecated in favor of pkgs/build-support/closure-info.nix, which I also could not figure out how to invoke. Ultimately, all I had to use was nix-store --dump-db.

From start to finish, here’s what I executed to get this working:

# Get r-burns' ppc64le branch and build cross (this takes a long time)
git clone -b ppc64le --depth 1 https://github.com/r-burns/nixpkgs.git
nix-build './nixpkgs' -A pkgsCross.powernv.nix

# Get a donor copy of the installer
wget https://releases.nixos.org/nix/nix-2.4/nix-2.4-x86_64-linux.tar.xz
tar xf nix-2.4-x86_64-linux.tar.xz

# Keep the cert package
mkdir -p nix-ppc64le-linux/store
cp nix-2.4-x86_64-linux/store/*nss-cacert* nix-ppc64le-linux/store/

# Toss other packages, keep the scripts
rm -rf nix-2.4-x86_64-linux/store
mv nix-2.4-x86_64-linux/* nix-ppc64le-linux/

# Add ppc64le packages
cp -r $(nix-store -qR result) nix-ppc64le-linux/

# Generate a .reginfo for those ppc64le packages
rm nix-ppc64le-linux/.reginfo
nix-store --dump-db $(nix-store -qR result > nix-ppc64le/.reginfo

Then scp to the other machine, extract, and execute install.

Hosted by Flying Circus.