How difficult would it be to port NixOS to PPC32? Any documentation on how to port NixOS to another architecture?
Background: I have a perfectly serviceable PPC Mac Mini that I would like to a run current Linux. Unfortunately, almost everyone has abandoned the PPC32 platform. I’ve built Linux from Scratch before but if I have to bootstrap Linux on the system, why not put my effort into NixOS…
As far as I’m aware there isn’t any documentation on this, but I might be able to help out. I have a colorful iMac G3, it would be fun to get NixOS running on it.
The boot sequence for PowerPC is IEEE1275 which is unlike any currently-supported NixOS platforms. That might take some experimentation to get working, so I’d recommend first installing e.g. Ubuntu or Debian as the base OS, and trying to get multi-user Nix on non-NixOS working.
For Nix on non-NixOS, we’d first want to cross-compile Nix and bootstrap tools for PPC32. This requires adding the platform definition for powerpc-unknown-linux-gnu to lib/systems/examples.nix.
That sounds like a good way to go. Although 16.04 is the last supported PPC version of Ubuntu, Debian PPC is still available (tier-2) so I don’t think we need to cross compile for architectures, only for platforms.
Thanks for the bootstrap tools link. I’ll take a look at it. (I already figured out that there needs to be a new entry in lib/systems/examples.nix.)
Time for a status update…
My G4 Mac Mini already has Debian 8 LTS installed but we will need a newer Debian, as building Nix requires newer compilers and libraries. Initially I considered installing Debian 10 but there appear to be enough problems with installing the boot loader that rather than reinstall I choose to create a chroot environment based on the 2021-02-02 ISO. Note: although the ISO says it is Debian 10, it is really Sid/Bullseye.
Make the packages from the ISO available and install using debootstrap:
$ wget https://cdimage.debian.org/cdimage/ports/snapshots/2021-02-02/debian-10.0.0-powerpc-NETINST-1.iso
$ wget https://cdimage.debian.org/cdimage/ports/snapshots/2021-02-02/SHA256SUMS
$ grep debian-10.0.0-powerpc-NETINST-1.iso SHA256SUMS | tee debian-10.0.0-powerpc-NETINST-1.iso.SHA256 | sha256sum -c # OK
$ rm SHA256SUMS
$ sudo mount -o ro,loop debian-10.0.0-powerpc-NETINST-1.iso /mnt
$ sudo apt-get install debootstrap
$ sudo debootstrap --no-check-gpg sid sid file:///mnt # not supported so no signatures
Configure apt to access the work-in-progress Debian powerpc port:
$ cat <<EOF | sudo tee sid/etc/apt/apt.conf.d/99no-recommends
APT::Install-Recommends "0";
APT::Install-Suggests "0";
EOF
$ sudo mv sid/etc/apt/sources.list sid/etc/apt/sources.list.orig
$ cat <<EOF | sudo tee sid/etc/apt/sources.list.d/deb.debian.org.list
# Have to trust; powerpc has no signatures
deb [trusted=yes] http://deb.debian.org/debian-ports sid main
EOF
Set up a chroot environment:
$ sudo mount -o bind /dev sid/dev
$ sudo mount -o bind /dev/pts sid/dev/pts
$ sudo mount -o bind /proc sid/proc
$ sudo mount -o bind /sys sid/sys
$ sudo chroot sid /bin/bash
# apt-get update --allow-unauthenticated # ignore signature warning
# /usr/lib/dpkg/methods/apt/update /var/lib/dpkg
# apt-get install locales
# dpkg-reconfigure # en_US.UTF-8
I expected to build Nix from source but because the ISO is really for Sid/Bullseye, Nix is already available:
$ sudo apt-get install nix-bin
$ nix --version
nix (Nix) 2.3.10
To summarize, I now have a G4 Mini with a Debian 11 Sid/Bullseye environment with Nix installed. That is enough for today. I need to think of what steps come next…
I am stuck making the cross compile bootstrap tools. I added entries for powerpc:
$ git diff
diff --git a/lib/systems/examples.nix b/lib/systems/examples.nix
index 389c4eebcc8..2bf62e8c310 100644
--- a/lib/systems/examples.nix
+++ b/lib/systems/examples.nix
@@ -34,6 +34,14 @@ rec {
gcc = { abi = "elfv2"; }; # for gcc configuration
};
+ ppc32-gnu = {
+ config = "powerpc-unknown-linux-gnu";
+ };
+ ppc32-musl = {
+ config = "powerpc-unknown-linux-musl";
+ };
+ ppc32 = ppc32-gnu;
+
sheevaplug = {
config = "armv5tel-unknown-linux-gnueabi";
} // platforms.sheevaplug;
diff --git a/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix b/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix
index d8ab96952b7..c0dc2d3b9dc 100644
--- a/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix
+++ b/pkgs/stdenv/linux/make-bootstrap-tools-cross.nix
@@ -17,6 +17,8 @@ in lib.mapAttrs (n: make) (with lib.systems.examples; {
armv6l-musl = muslpi;
aarch64-musl = aarch64-multiplatform-musl;
riscv64 = riscv64;
+ powerpc = ppc32;
+ powerpc-musl = ppc32-musl;
powerpc64 = ppc64;
powerpc64-musl = ppc64-musl;
powerpc64le = powernv;
which took me to the next error message:
$ nix-build ./pkgs/stdenv/linux/make-bootstrap-tools-cross.nix -A powerpc --show-trace
error: while evaluating 'wrapBintoolsWith' at /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11495:5, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:12293:14:
while evaluating 'callPackageWith' at /home/mkg/src/nixpkgs/lib/customisation.nix:117:35, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11499:7:
while evaluating 'makeOverridable' at /home/mkg/src/nixpkgs/lib/customisation.nix:67:24, called from /home/mkg/src/nixpkgs/lib/customisation.nix:121:8:
while evaluating anonymous function at /home/mkg/src/nixpkgs/pkgs/build-support/bintools-wrapper/default.nix:8:1, called from /home/mkg/src/nixpkgs/lib/customisation.nix:69:16:
while evaluating the attribute 'cc.nativeTools' at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/default.nix:166:14:
while evaluating the attribute 'gcc' at /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:10139:3:
while evaluating 'addMetaAttrs' at /home/mkg/src/nixpkgs/lib/meta.nix:15:28, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:10330:11:
while evaluating 'wrapCC' at /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11490:12, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:10330:20:
while evaluating 'wrapCCWith' at /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11464:5, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11490:16:
while evaluating 'callPackageWith' at /home/mkg/src/nixpkgs/lib/customisation.nix:117:35, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11478:7:
while evaluating 'makeOverridable' at /home/mkg/src/nixpkgs/lib/customisation.nix:67:24, called from /home/mkg/src/nixpkgs/lib/customisation.nix:121:8:
while evaluating anonymous function at /home/mkg/src/nixpkgs/pkgs/build-support/cc-wrapper/default.nix:8:1, called from /home/mkg/src/nixpkgs/lib/customisation.nix:69:16:
while evaluating the attribute 'buildInputs' of the derivation 'glibc-2.32-40-powerpc-unknown-linux-gnu' at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:203:11:
while evaluating the attribute 'depsBuildBuild' of the derivation 'linux-headers-5.11' at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:197:11:
while evaluating 'getOutput' at /home/mkg/src/nixpkgs/lib/attrsets.nix:482:23, called from undefined position:
while evaluating anonymous function at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:138:17, called from undefined position:
while evaluating the attribute 'stdenv.cc' at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/default.nix:166:14:
while evaluating the attribute 'cc' at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/default.nix:166:14:
while evaluating the attribute 'cc' at /home/mkg/src/nixpkgs/pkgs/stdenv/native/default.nix:122:5:
anonymous function at /home/mkg/src/nixpkgs/pkgs/build-support/cc-wrapper/default.nix:8:1 called without required argument 'lib', at /home/mkg/src/nixpkgs/pkgs/stdenv/native/default.nix:128:5
Adding what appears to be a missing lib attribute in the import of cc-wrapper:
$ git diff pkgs/stdenv/native/default.nix
diff --git a/pkgs/stdenv/native/default.nix b/pkgs/stdenv/native/default.nix
index b79b81253ad..2a8daaaaf82 100644
--- a/pkgs/stdenv/native/default.nix
+++ b/pkgs/stdenv/native/default.nix
@@ -129,10 +129,10 @@ in
name = "cc-native";
nativeTools = true;
nativeLibc = true;
- inherit nativePrefix;
+ inherit nativePrefix lib;
bintools = import ../../build-support/bintools-wrapper {
name = "bintools";
- inherit stdenvNoCC nativePrefix;
+ inherit stdenvNoCC nativePrefix lib;
nativeTools = true;
nativeLibc = true;
};
solved that problem. But now I’m stuck:
$ nix-build ./pkgs/stdenv/linux/make-bootstrap-tools-cross.nix -A powerpc --show-trace
error: while evaluating 'wrapBintoolsWith' at /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11495:5, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:12293:14:
while evaluating 'callPackageWith' at /home/mkg/src/nixpkgs/lib/customisation.nix:117:35, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11499:7:
while evaluating 'makeOverridable' at /home/mkg/src/nixpkgs/lib/customisation.nix:67:24, called from /home/mkg/src/nixpkgs/lib/customisation.nix:121:8:
while evaluating anonymous function at /home/mkg/src/nixpkgs/pkgs/build-support/bintools-wrapper/default.nix:8:1, called from /home/mkg/src/nixpkgs/lib/customisation.nix:69:16:
while evaluating the attribute 'cc.nativeTools' at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/default.nix:166:14:
while evaluating the attribute 'gcc' at /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:10139:3:
while evaluating 'addMetaAttrs' at /home/mkg/src/nixpkgs/lib/meta.nix:15:28, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:10330:11:
while evaluating 'wrapCC' at /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11490:12, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:10330:20:
while evaluating 'wrapCCWith' at /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11464:5, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11490:16:
while evaluating 'callPackageWith' at /home/mkg/src/nixpkgs/lib/customisation.nix:117:35, called from /home/mkg/src/nixpkgs/pkgs/top-level/all-packages.nix:11478:7:
while evaluating 'makeOverridable' at /home/mkg/src/nixpkgs/lib/customisation.nix:67:24, called from /home/mkg/src/nixpkgs/lib/customisation.nix:121:8:
while evaluating anonymous function at /home/mkg/src/nixpkgs/pkgs/build-support/cc-wrapper/default.nix:8:1, called from /home/mkg/src/nixpkgs/lib/customisation.nix:69:16:
while evaluating the attribute 'buildInputs' of the derivation 'glibc-2.32-40-powerpc-unknown-linux-gnu' at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:203:11:
while evaluating the attribute 'depsBuildBuild' of the derivation 'linux-headers-5.11' at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:197:11:
while evaluating the derivation attribute 'name' at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:197:11:
cannot coerce null to a string, at /home/mkg/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:197:19
Apparently, either attrs.pname or attrs.version are null in line 197 of make-derivation.nix but I have no idea why and can’t find where they would be set anyway.
From your traceback it looks like stdenv is trying to use stagesNative for bootstrapping, but you want stagesLinux if you’re building bootstrap tools. Try adding powerpc-linux = stagesLinux to stdenv/default.nix here: https://github.com/NixOS/nixpkgs/blob/a8f6369dda06b3e47299f7760b47c6806c03fb95/pkgs/stdenv/default.nix#L42-L63
(AFAIR the stagesNative issue has been occurring since the stdenv.lib attr was removed, but you don’t need to fix that for your purposes)
It remembers me the PowerPC Notebook Project! Maybe it can interest them!
I tried a different tack and made some progress. Following the cross compiling wiki article:
$ mkdir ppc32
$ cd ppc32
$ git clone https://github.com/NixOS/nixpkgs/
$ git diff
index c6d90ba9850..b0b5fe5e9a0 100644
--- a/lib/systems/doubles.nix
+++ b/lib/systems/doubles.nix
@@ -26,8 +26,9 @@ let
# Linux
"aarch64-linux" "armv5tel-linux" "armv6l-linux" "armv7a-linux"
- "armv7l-linux" "i686-linux" "mipsel-linux" "powerpc64-linux"
- "powerpc64le-linux" "riscv32-linux" "riscv64-linux" "x86_64-linux"
+ "armv7l-linux" "i686-linux" "mipsel-linux" "powerpc-linux"
+ "powerpc64-linux" "powerpc64le-linux" "riscv32-linux"
+ "riscv64-linux" "x86_64-linux"
# MMIXware
"mmix-mmixware"
diff --git a/lib/systems/examples.nix b/lib/systems/examples.nix
index 6a8f4e091aa..2dbecd3d7de 100644
--- a/lib/systems/examples.nix
+++ b/lib/systems/examples.nix
@@ -21,6 +21,13 @@ rec {
config = "powerpc64le-unknown-linux-musl";
};
+ powerpc = {
+ config = "powerpc-unknown-linux-gnu";
+ };
+ powerpc-musl = {
+ config = "powerpc-unknown-linux-musl";
+ };
+
ppc64 = {
config = "powerpc64-unknown-linux-gnu";
gcc = { abi = "elfv2"; }; # for gcc configuration
Now set up an empty cross compilation environment:
$ cat <<EOF > crossShell.nix
with import <nixpkgs> {
crossSystem = {
config = "powerpc-unknown-linux-gnu";
};
};
mkShell {
buildInputs = [
];
}
EOF
$ export NIX_PATH=nixpkgs=$(pwd)/nixpkgs
$ time nix-shell crossShell.nix # 44:28.17 (on i5-2500K)
$ powerpc-unknown-linux-gnu-cc --version # powerpc-unknown-linux-gnu-gcc (GCC) 10.3.0
$ nix build -f ./nixpkgs --arg crossSystem '(import <nixpkgs>{}).lib.systems.examples.powerpc' hello
$ file result/bin/hello
result/bin/hello: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), dynamically linked, interpreter /nix/store/i5sqay18wybbghf8ncp6gqr0b3xlf7lv-glibc-powerpc-unknown-linux-gnu-2.32-46/lib/ld.so.1, for GNU/Linux 2.6.32, not stripped
Definitely progress. I need to do some thinking of what to do next. Suggestions are welcome, of course.
I’m interested in having a NixOS for PowerPC 32bit in the long run for work. I’m right now upstreaming some changes needed to have a fully working PPC64 NixOS image, but PPC32 is also on my radar.
If hello works, I’d try to build a minimalist NixOS image, and fix things as they come.
An example config:
{ lib, modulesPath, ... }:
{
imports = [
(modulesPath + "/profiles/minimal.nix")
];
boot.initrd.includeDefaultModules = false;
systemd.shutdownRamfs.enable = false;
xdg.mime.enable = false;
services.udisks2.enable = false;
nixpkgs.crossSystem = lib.systems.examples.powerpc;
}
Another possibility would be to try and compile the kernel. You can look at lib/systems/platforms.nix to take inspiration if your kernel needs a special defconfig, or other tweaking.
Feel free to hit me up on matrix @Minijackson:matrix.org
Have there been any more work on the ppc32. I would interested in this. First I only need Cross compile, but in the long run kernel would be nice.
I have a full NixOS image that I’m using for work, which is cross-compiled from x86_64-linux to ppc32 ppc64be.
I try to upstream every fix that I can, but there might still be some missing.
I’m not using the mainline kernel in my case, I’m using the NXP one, but it was kind of a mess to integrate.
I hope that someday we will redo the Nix infrastructure for building the Linux kernel.
What made it a "mess to integrate’?
The kernel build infra is quite solid IME.
So here’s what my current config looks like (with some kernel config removed):
{
buildLinux,
fetchFromGitHub,
fetchurl,
lib,
...
} @ args: let
ifc14xx_ess_defconfig = fetchurl {
url = "https://raw.githubusercontent.com/icshwi/yocto-ess/e3f166c64465d79a4ae2eb8acdbe53ee3a743ad2/sources/meta-ess/recipes-kernel/linux/linux-qoriq-4.14/ifc14xx_4_14_defconfig";
hash = "sha256-7odc0Lpyw62thN17TqV3QUTxxJXKJ2jNXo0K/o0IJK4=";
};
in
lib.overrideDerivation (buildLinux (args
// {
version = "5.10.72-ifc14xx";
src = fetchFromGitHub {
owner = "nxp-qoriq";
repo = "linux";
rev = "lf-5.10.72-2.2.0";
hash = "sha256-VC+vq2u+dl2ENME/88dZU6/YmdVA7i4SE4+X7MwBDHU=";
postFetch = ''
cp ${ifc14xx_ess_defconfig} $out/arch/powerpc/configs/ifc1410_defconfig
'';
};
kernelPatches = [];
features = {
efiBootStub = false;
iwlwifi = false;
needsCifsUtils = false;
};
structuredExtraConfig = with lib.kernel; {
# Needed for netboot
# ==================
# squashfs is used for delivering the computed content of the /nix/store
# It is compressed, and is read when needed, which is useful for saving
# RAM
SQUASHFS = module;
# overlayfs is used so that we can have a writable /nix/store:
# modifications will be written in a tmpfs mounted over the squashfs
# /nix/store
OVERLAY_FS = module;
AUTOFS_FS = module;
# Firewall support
# ================
# In NixOS right now, we are in a transition from iptables to its
# successor nf_tables.
# We are using a compatibility version of iptables, whose command-line
# is compatible with iptables, but who uses nf_tables as a backend.
# This might explain the list of chosen modules here.
NETFILTER = yes;
NETFILTER_XTABLES = module;
NETFILTER_XT_MATCH_CONNTRACK = module;
NETFILTER_XT_MATCH_PKTTYPE = module;
NETFILTER_XT_TARGET_LOG = module;
# So that iptables works by using the nf_tables backend
NFT_COMPAT = module;
# Every iptables rules by default have a counter in nf_tables
NFT_COUNTER = module;
IP6_NF_FILTER = module;
IP6_NF_IPTABLES = module;
IP_NF_FILTER = module;
IP_NF_IPTABLES = module;
# Needed if the NixOS option networking.firewall.checkReversePath is
# set, which is true by default.
IP6_NF_MATCH_RPFILTER = module;
IP_NF_MATCH_RPFILTER = module;
# Needed for the reverse path filter
IP6_NF_MANGLE = module;
IP6_NF_RAW = module;
IP_NF_MANGLE = module;
IP_NF_RAW = module;
NF_CONNTRACK = module;
NF_LOG_IPV4 = module;
NF_LOG_IPV6 = module;
NF_TABLES = module;
# Some other stuff...
};
}))
(old: {
patches = old.patches ++ [
# Backport of fix from 5.15.66
# https://lore.kernel.org/lkml/20220906132826.180891759@linuxfoundation.org/
./pahole-skip_encoding_btf_enum64.patch
];
postPatch =
(old.postPatch or "")
+ ''
cp ${./ifc14xx-nixos.dts} arch/powerpc/boot/dts/fsl/ifc14xx-nixos.dts
'';
postInstall =
''
rm $out/vmlinux
cp arch/powerpc/boot/uImage $out/
install -m 0644 -D -t $out/dtbs arch/powerpc/boot/dts/fsl/ifc14xx-nixos.dtb
''
+ old.postInstall;
buildFlags = old.buildFlags ++ ["fsl/ifc14xx-nixos.dtb"];
})
ifc1410 = {
system = "powerpc64-linux";
linux-kernel = {
target = "uImage";
installTarget = "install";
# Provided by the kernel specified in ./kernel.nix
baseConfig = "ifc1410_defconfig";
autoModules = false;
name = "ifc1410";
DTB = true;
};
gcc = {
# Version of the Application Binary Interface[1] for PPC64 BE.
#
# Contains for example the calling convention, i.e. the way functions are
# called, with which CPU registers, etc.
#
# Probably for historical reasons, GCC normally defaults to ELFv1 for
# PPC64 Big-Endian (and ELFv2 for Little-Endian, as it is younger). ELFv1
# has been deprecated since a while ago, and some packages won't compile
# with ELFv1. We choose the sane option here, even if it leads to some
# patches to apply here and there. See ./hack.nix
#
# As a reference, Void Linux uses ELFv2 with PPC64 BE.
#
# [1]: https://en.wikipedia.org/wiki/Application_binary_interface
abi = "elfv2";
cpu = "e6500";
};
};
I think there’s some patterns that feels unintuitive, and some that didn’t work when applied to the ppc64 architecture. If there’s some things that I’ve missed, I’d be very interested to know.
The things that surprised me:
buildLinux (args // { ... }) is only specific to buildLinux? I haven’t seen it anywhere else.overrideAttrs to apply additional patches, and customize phases, but I think I remember it now working, and I had to use lib.overrideDerivation, which I haven’t seen used anywhere else eitherstructuredExtraConfig was a lot of trial and error_defconfig file, but haven’t found a proper way to add it, so I just copied the file in postFetch in the src
.dts file, I copied it in postPatch
For the thing that didn’t work for ppc:
dbts and dtbs_install for that architecture (feels weird that there’s no consistency between arm and ppc, but hey), so I had to explicitly build the dtb in buildFlags and install it in postInstall
All in all, a lot of trial and errors, some of it could be alleviated with more extensive documentation (I couldn’t find the time to contribute it, sorry…), and also analyzing what Buildroot and Yocto do, if we want the kernel infra to be more suitable for embedded systems. I’d also love to see autokernel integrated into the build, which could really help building a config.