Hi there, i was unsure whether this is a nix or nixpkgs issue but I’m running into an issue where a flake i need to cross compile in CI will OOM on the server it’s running on (the CI kills it when its ram usage exceeds 10gb from what i can tell).
I made a small reproduction flake that showcases the issue
{
inputs.nixpkgs.url = "github:nixos/nixpkgs?rev=c374d94f1536013ca8e92341b540eba4c22f9c62";
outputs = {
self,
nixpkgs,
}: let
# pkgsSet is written in a way that it also works as a non-flake
# default.nix without modifications, issue persists in non flake context
pkgsSet = {
nixpkgs ? <nixpkgs>,
localSystem ? builtins.currentSystem,
}: let
# ensure we're cross compiling
crossSystem = {
"x86_64-linux" = "aarch64-linux";
"aarch64-linux" = "x86_64-linux";
}.${localSystem};
pkgs = import nixpkgs {
system = localSystem;
};
pkgsCross = import nixpkgs {
inherit crossSystem localSystem;
};
mkLargePython = pkgs:
pkgs.python3.withPackages (ps:
builtins.attrValues {
inherit (ps) numpy matplotlib requests pandas;
});
in {
python = mkLargePython pkgs; # native, no problem
hello = pkgsCross.hello; # small cross, no problem
pythonCross = mkLargePython pkgsCross; # big cross, ooms during evaluation
};
in {
packages = nixpkgs.lib.genAttrs ["x86_64-linux" "aarch64-linux"] (system:
pkgsSet {
inherit nixpkgs;
localSystem = system;
});
};
}
Evaluating (not building!) the native python package takes very little memory but pythonCross will OOM on my machine. Trying it on a device with 32GB ram shows that it will indeed complete if it can take around 12gb for the evaluation, which doesnt seem to be right. Also since one of the packages i need to cross compile is linux which also takes quite a bit of ram i fear i dont have a machine with enough ram to run this build anymore especially given that nix from what i can tell is not freeing up the eval memory until its done with the builds.
I wonder if I’m just doing something wrong but I’ve looked around and despite the information around flake cross compiling being generally sparse it doesnt seem like im doing something different than what should work.
I’ve tested this on a x86_64-linux laptop and a aarch64-linux fedora install to make sure it affects cross compiling in general not just a specific architecture, both run nix version 2.18.4
1 Like
Mic92
August 24, 2024, 1:24am
3
What seems weird is that the very same derivation seems to get instantiated many many times:
% command time -v nix eval -vvv .#pythonCross 2>&1 | grep python3.12-flit-core-3.9.0.drv
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
command time -v nix eval -vvv .#pythonCross 2>&1 11.68s user 4.18s system 93% cpu 16.973 total
grep --binary-files=without-match --directories=skip --color=auto 0.05s user 0.11s system 0% cpu 16.972 total
But even without cross compilation I am wondering why this happens so often:
% command time -v nix eval -vvv .#python 2>&1 | grep python3.12-flit-core-3.9.0.drv
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
instantiated 'python3.12-flit-core-3.9.0' -> '/nix/store/ni6f0980j4flhk1m6l6l69jqi1npfdkr-python3.12-flit-core-3.9.0.drv'
command time -v nix eval -vvv .#pythonCross 2>&1 11.68s user 4.18s system 93% cpu 16.973 total
grep --binary-files=without-match --directories=skip --color=auto 0.05s user 0.11s system 0% cpu 16.972 total
Mic92
August 24, 2024, 8:57am
4
I think it’s known that cross-compiling evaluates slower. However I didn’t knew it was that much slower.
# The `splicedPackages' package set, and its use by `callPackage`
#
# The `buildPackages` pkg set is a new concept, and the vast majority package
# expression (the other *.nix files) are not designed with it in mind. This
# presents us with a problem with how to get the right version (build-time vs
# run-time) of a package to a consumer that isn't used to thinking so cleverly.
#
# The solution is to splice the package sets together as we do below, so every
# `callPackage`d expression in fact gets both versions. Each derivation (and
# each derivation's outputs) consists of the run-time version, augmented with
# a `__spliced.buildHost` field for the build-time version, and
# `__spliced.hostTarget` field for the run-time version.
#
# For performance reasons, rather than uniformally splice in all cases, we only
# do so when `pkgs` and `buildPackages` are distinct. The `actuallySplice`
# parameter there the boolean value of that equality check.
lib: pkgs: actuallySplice:
let
show original
“Splicing” select host and build dependencies in a hacky way. You can read about more complaints here: Frustrations about splicing
1 Like
Ah, i heard about splicing here and there before but wasn’t aware its that arcane and problematic.
Also if i’m understanding the situation right it’s not something i can fix/workaround for now, that’s a bummer