I’m a big fan of NixOS and would love to use it on a Raptor Talos II. I know that the NixOS distro as a whole is not ready for ppc64le, but I’m hoping it’s possible to start running the standalone package manager. @r-burns seems to have done a lot of great work on this front recently.
How do I get started running Nix on top of Debian ppc64le? The install script simply says there are no bootstrap binaries available for my architecture.
Edit: I do have an amd64 NixOS machine available for cross-compiling, I’m just not sure what to invoke.
Awesome! I would say the first step is to build the Nix package manager from source for ppc64le. In the past I’ve done this natively on the target machine by hand using dependencies from APT. Alternatively, it may be easier to try cross-compiling. For example, on your amd64 machine, nix-build '<nixpkgs>' -A pkgsCross.powernv.nix && nix-copy-closure --to=user@talos result/ (substituting your username and hostname for user@talos). pkgsCross.powernv is the special crossPkgs set for typical ppc64le machines, so you can also use that for any other packages you wish to cross-compile. Some subset of packages are regularly cross-compiled on Hydra (Hydra - Evaluations of jobset nixpkgs:cross-trunk) so you will hopefully be fetching from the Hydra cache rather than building locally.
Once you have a working Nix on your ppc64le machine, you will need to make some modifications to Nixpkgs to build packages natively. For now, I’d take a look at my ppc64le branch (GitHub - r-burns/nixpkgs at ppc64le) which is volatile but at least tested to work for stdenv and some common packages. This should take care of most trivial modifications to stdenv/bootstrap-tools, and work around some more difficult issues such as Cross bootstrap tools have broken stackprotector support · Issue #113977 · NixOS/nixpkgs · GitHub.
AFAIK @CrystalGamma also owns a Talos and was the one who originally added support for ppc64, so may have additional hardware-specific tips.
Hi … so yeah, you basically have 2 ways to get bootstrapped: doing the cross-build, as described by @r-burns above, or using the native (“impure”) stdenv, which I haven’t done in ages, so I don’t remember the exact process.
I also have a personal branch I keep rebasing on top of nixpkgs master: GitHub - CrystalGamma/nixpkgs at local1 Compare and contrast with @r-burns 's branch (in fact, I’ll be yoinking that s2n patch, up to now I was just disabling the PQ code altogether). Keep in mind 2 of the commits just add the bootstrap blobs to the repo. Unless you trust me with building them (and github not to manipulate them), might want to build them yourself.
I daily drive my Talos with a KDE desktop, so I wouldn’t say it’s not ready (though upstream clearly is). In addition to that nixpkgs branch I use an overlay that patches qtwebengine and a configuration setting that lets Xorg use the right GPU (no designated ‘default’ GPU on Power, though if there’s exactly one on the system, I’d have expected it to just use that one by default). If you want to run a desktop on your Talos, I can share those with you too (would be a bit more work, because I don’t feel comfortable sharing the entire repo they’re in).
Thanks for the helpful answers! Cross-compiling worked beautifully, but nix-copy-closure requires nix to already be present on the other end. I’m trying to figure out how to manually install by cross referencing the official install script, but that seems to assume nix-2.3.11-$system.tar.xz contains a script file install, which I’m not seeing in pkgsCross.powernv.nix.
I am interested in using the Talos as a KDE workstation… @CrystalGamma which distro are you running? Nix on top of Debian, or have you actually managed to get NixOS installed?
I’m running actual NixOS, though the only thing I remember about the bootstrapping process is that it was complicated and longwinded (though I probably took a very roundabout approach).
You should be able to nix-copy-closure by mounting the drive in your cross-builder system. You’ll probably need to manually write a bootloader configuration to start it on the Talos though, since I assume the activation script will call ppc64le binaries. My branch has a patch that changes the syslinux configuration generator to work with petitboot (not sure if that’s even necessary anymore, haven’t checked in a long while). Anyway the bootloader configuration needs to set init=/nix/store/$your_system_derivation_path/init in the kernel command line; you can also find the kernel and initrd in the system derivation path. Hope this helps.
Thanks, it’s helpful to even know that someone has pulled off the bootstrap! Mounting the boot drive on the x64 machine is a good idea. I’ve been booting off a USB3 SSD anyway since I opted out of the SATA controller for my Talos.
When I get this working I will update the thread with my steps.
This proved to be a tricky problem, but I’ve finally got Nix running on top of Debian ppc64le! The missing step involves a file called .reginfo, which the installer requires to find store paths from an adjacent folder (rather than installed into /nix/store with an active database).
Googling for .reginfo was difficult. First I found references to pkgs/build-support/kernel/paths-from-graph.pl, but I couldn’t get it to work and it appears to be deprecated in favor of pkgs/build-support/closure-info.nix, which I also could not figure out how to invoke. Ultimately, all I had to use was nix-store --dump-db.
From start to finish, here’s what I executed to get this working:
# Get r-burns' ppc64le branch and build cross (this takes a long time)
git clone -b ppc64le --depth 1 https://github.com/r-burns/nixpkgs.git
nix-build './nixpkgs' -A pkgsCross.powernv.nix
# Get a donor copy of the installer
wget https://releases.nixos.org/nix/nix-2.4/nix-2.4-x86_64-linux.tar.xz
tar xf nix-2.4-x86_64-linux.tar.xz
# Keep the cert package
mkdir -p nix-ppc64le-linux/store
cp nix-2.4-x86_64-linux/store/*nss-cacert* nix-ppc64le-linux/store/
# Toss other packages, keep the scripts
rm -rf nix-2.4-x86_64-linux/store
mv nix-2.4-x86_64-linux/* nix-ppc64le-linux/
# Add ppc64le packages
cp -r $(nix-store -qR result) nix-ppc64le-linux/
# Generate a .reginfo for those ppc64le packages
rm nix-ppc64le-linux/.reginfo
nix-store --dump-db $(nix-store -qR result > nix-ppc64le/.reginfo
Then scp to the other machine, extract, and execute install.
It turns out that we can build binaryTarballCross target in NixOS/nix to get the tarball. Patch flake.nix to add powerpc64l-linux to crossSystems and then build the tarball using nix build .#hydraJobs.binaryTarballCross.x86_64-linux.powerpc64le-linux. Thanks @NickCao for the method.
So, the problem I’ve having with the .#hydraJobs.binaryTarballCross approach is that there doesn’t seem to be a working stdenv.
I’m able to install nix-2.12.0pre20221203_3086739-powerpc64le-linux.tar.xz, but building packages on the ppc64le machine fails. nix-shell -p bash hits this error:
building '/nix/store/z4hwf527nfx1a5xi3d69nqxcz4j59fjl-gnum4-1.4.19.drv'...
unpacking sources
unpacking source archive /nix/store/mcayf992diwks293vs12iwx1ywgm4ijg-m4-1.4.19.tar.bz2
source root is m4-1.4.19
setting SOURCE_DATE_EPOCH to timestamp 1622228112 of file m4-1.4.19/ChangeLog
patching sources
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/axjnwvrj4fa2n25wch3yazjwjgj29wzl-gnum4-1.4.19 --with-syscmd-shell=/nix/store/ljbjpmgpchd81qyy0rblhrh404vy5jrg-bootstrap-tools/bin/bash
/nix/store/d9phvfv7044fmy4rr2dw15ra5ldjd03r-bootstrap-stage1-stdenv-linux/setup: ./configure: /bin/sh: bad interpreter: No such file or directory
error: builder for '/nix/store/z4hwf527nfx1a5xi3d69nqxcz4j59fjl-gnum4-1.4.19.drv' failed with exit code 126
I think the way forward is to cross-reference r-burns/nixpkgs/ppc64le and CrystalGamma/nixpkgs/local1 branches, and try to bring that stdenv work forward?
I would continue to use the older branches, but they’re on nix 2.3 whereas I’d like to use flakes.
Starting from a working Nix 2.3 environment using the steps above, I was able to further bootstrap the nixos-22.05-small branch of nixpkgs (including Nix 2.8, and flake support!)
I’m generally finding it easier to work from the nixpkgs repo rather than the nix repo, since that’s where the necessary changes are needed. I’ve cherry-picked some fixes into this 22.05 branch sufficient to enable the packages I want to build. The 22.11 branch is not working for me yet.
I was a bit surprised that it’s no longer necessary to update the bootstrap binaries manually. It must be pulling artifacts from Hydra these days? But here are the steps in case it seems necessary for some reason:
nix-build ./pkgs/stdenv/linux/make-bootstrap-tools.nix -A bootstrapFiles
cp result ./pkgs/stdenv/linux/bootstrap-files/bootstrap-tools.tar.xz
cp result-2 ./pkgs/stdenv/linux/bootstrap-files/busybox
then edit ./pkgs/stdenv/linux/bootstrap-files/default.nix to refer to those two artifacts.
I’m new to Nix, and also trying to get it to work on my Raptor Talos II.
I was able to compile Nix on an x86_64 machine, following these instructions (adding “powerpc64le-linux” to flake.nix). And I was able to install it on Debian ppc64le. No problems there.
But actually trying to use Nix, I get errors like this:
The installer said to test with this command nix-shell -p nix-info --run "nix-info -m".
I get:
error:
… <borked>
at «none»:0: (source not available)
… while calling the 'import' builtin
at «string»:1:18:
1| {...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (nix-info) ]; } ""
| ^
(stack trace truncated; use '--show-trace' to show the full trace)
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
at «none»:0: (source not available)
I did relaunch the terminal, so the changes should have taken effect, as per the instructions.
I found a Github issue where someone mentioned using this: env NIX_PATH=$HOME/.nix-defexpr/channels, so I tried env NIX_PATH=$HOME/.nix-defexpr/channels nix-shell -p 'bash'.
And I get:
warning: Nix search path entry '/home/rjzak/.nix-defexpr/channels' does not exist, ignoring
error:
… <borked>
at «none»:0: (source not available)
… while calling the 'import' builtin
at «string»:1:18:
1| {...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (bash) ]; } ""
| ^
(stack trace truncated; use '--show-trace' to show the full trace)
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
at «none»:0: (source not available)
$ uname -a
Linux behemoth 6.1.0-3-powerpc64le #1 SMP Debian 6.1.8-1 (2023-01-29) ppc64le GNU/Linux
$ nix --version
nix (Nix) 2.15.0pre20230303_dirty
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
You tried editing flake.nix of the Nix repo itself? I had no luck with that approach either. What worked for me was cross-compiling the Nix target from a ppc64le-enabled branch of the Nixpkgs repo.
In this post I got a working Nix 2.4 cross-compiled from @r-burns’ nixpkgs branch.
@jiegec fixed part of my script here but I’m not sure which nixpkgs branch they built from.
In my last post I was able to natively compile a newer Nix starting from my cross-compiled Nix 2.4.
I don’t think you need to start with an old Nix anymore now that I’ve pulled necessary changes into my nixos-22.05-small-ppc64le branch.
Here’s a mishmash of all the best steps so far, which should get you to Nix 2.12 in one step. (I’ve successfully done this flow but I’m writing from memory, so let me know if there’s a bug here)
# Get a ppc64le-compatible branch and build cross (this takes a long time)
git clone https://github.com/ChadSki/nixpkgs.git -b nixos-22.05-small-ppc64le
nix-build './nixpkgs' -A pkgsCross.powernv.nix
# Get a donor copy of the installer
wget https://releases.nixos.org/nix/nix-2.12.0/nix-2.12.0-x86_64-linux.tar.xz
tar xf nix-2.12.0-x86_64-linux.tar.xz
# Copy the cert package to our new installer
mkdir -p nix-ppc64le-linux/store
cp nix-2.12.0-x86_64-linux/store/*nss-cacert* nix-ppc64le-linux/store/
# Toss other packages, but keep the scripts
rm -rf nix-2.12.0-x86_64-linux/store
mv nix-2.12.0-x86_64-linux/* nix-ppc64le-linux/
# Add ppc64le packages
cp -r $(nix-store -qR result) nix-ppc64le-linux/
# Generate a .reginfo for those ppc64le packages
rm nix-ppc64le-linux/.reginfo
nix-store --dump-db $(nix-store -qR result > nix-ppc64le/.reginfo
# Replace NIX_INSTALLED_NIX in install script
export NIX_INSTALLED_NIX=$(readlink result)
sed -i "s#NIX_INSTALLED_NIX=\".*\"#NIX_INSTALLED_NIX=\"$NIX_INSTALLED_NIX\"#" nix-ppc64le-linux/install-multi-user
echo "Done! Copy nix-ppc64le-linux to ppc64le machine and run ./install --daemon"
Note that you’re probably going to want to build & install packages from that ppc64le-enabled branch as well. This post demonstrates that process with a few different command-line programs.
So I tested my steps above and there are a couple minor errors. (cp missing the -r flag, used the wrong version of Nix as a donor copy, etc.) Unfortunately the post has passed the edit window or I’d fix them in place.
I fixed up those problems in this gist and just verified that the whole bootstrapping path works. You end up with a working Nix 2.8.1.
The installer suggests running nix-shell -p nix-info --run "nix-info -m" to verify the installation, which took a while to build but eventually printed this output:
Support is getting better and better, although it’s still inconvenient that prebuilt Nix installers are not available for ppc64le.
There must be a “correct” way to build those installer tarballs (the hydra job I guess?), but I still haven’t pulled it off. So here’s the updated hacky approach which replaces the guts of an x64 installer with ppc64le binaries and definitely works.
A custom nixpkgs branch is no longer necessary: nixos-23.11 appears to have pretty decent ppc64le support, although I’m not sure how to bootstrap GHC Haskell anymore since they switched to their new Hadrian build system.
Anyway, starting on an x64 machine:
# Get the 23.11 branch (currently commit 8bf65f17d807)
git clone --depth 1 https://github.com/NixOS/nixpkgs.git -b nixos-23.11
# Cross-build Nix (this takes a long time)
nix-build './nixpkgs' -A pkgsCross.powernv.nix
# That built Nix 2.18.1, so get the matching x64 donor
wget https://releases.nixos.org/nix/nix-2.18.1/nix-2.18.1-x86_64-linux.tar.xz
tar xf nix-2.18.1-x86_64-linux.tar.xz
# Copy the cert package to our new installer
mkdir -p nix-2.18.1-ppc64le-linux/store
cp -r nix-2.18.1-x86_64-linux/store/*nss-cacert* nix-2.18.1-ppc64le-linux/store/
# Toss other packages, but keep the scripts
rm -rf nix-2.18.1-x86_64-linux/store
mv nix-2.18.1-x86_64-linux/* nix-2.18.1-ppc64le-linux/
# Add ppc64le packages
cp -r $(nix-store -qR result) nix-2.18.1-ppc64le-linux/store/
# Generate a .reginfo for those ppc64le packages
nix-store --dump-db $(nix-store -qR result) > nix-2.18.1-ppc64le-linux/.reginfo
# Replace NIX_INSTALLED_NIX in install script
export NIX_INSTALLED_NIX=$(readlink result)
sed -i "s#NIX_INSTALLED_NIX=\".*\"#NIX_INSTALLED_NIX=\"$NIX_INSTALLED_NIX\"#" nix-2.18.1-ppc64le-linux/install-multi-user
# Compress for transfer
tar cJvf nix-2.18.1-ppc64le-linux.tar.xz nix-2.18.1-ppc64le-linux
echo "Done! Copy nix-ppc64le-linux to ppc64le machine and run ./install --daemon"
Personally I’m using this on a Talos II workstation. I don’t know what kind of ppc64le machines other people use, but I figure I might as well detail the steps I use on the Talos.
Download the Debian 12 bookworm ppc64le server netinstall iso and flash it to a USB stick
At the petitboot menu, select “Expert Install” and run through the steps.
For partitioning, I did “Guided Partitioning” and let it automatically create a ~33MB gap at the front of the disk for GRUB. When finishing installation, I let it install GRUB to that same disk. Petitboot does not invoke GRUB but it does know how to parse the config format, and this approach is sufficient to boot. Gentoo wiki claims you can use a “delightfully simple” kboot config but I don’t know how to author one of those.
Copy over nix-2.18.1-ppc64le-linux.tar.xz and extract it.
Run ./install --daemon and respond to the prompts.
Verify installation by cloning another copy of nixpkgs and building nix-info from it:
At this point I’ve got Nix 2.18.1 on Debian 12 bookworm. Ultimately I’d still prefer to be running NixOS, but I hope to use this as a starting point.
It’s probably worth reinvestigating the .#hydraJobs.binaryTarballCross approach, and I’d also like to attempt making a ppc64le version of the Determinate Nix Installer.