I/O & CPU scheduling, jobs & cores... and performance, baby

Help
1

hello everyone!!! :sweat_smile:

im back at yall again with yet another question! :crazy_face:

closing in my nix misadventures by finally setting up a remote builder… ive begun to think about performance. i mean, i have both shitty and insane computers, and it is only now that ive suddenly started to care about SPEED. :red_car: :wind_face:

so, ive seen(1) some(2) posts(3) about these fairly new little options:

nix.daemonIOSchedClass
nix.daemonCPUSchedPolicy
nix.daemonIOSchedPriority
nix.settings.cores
nix.settings.max-jobs

and it involves math. OBVIOUSLY. but the more i started reading about it, the less it made sense to me - “should i even change anything”? also, the words are too technical, maybe its too early for me to try this? anyway, i think people are giving different advice to the same thing!

BEFORE i had a dedicated computer on which i update/compile/build stuff (aka a remote builder), i was doing everything on a laptop (well, actually a steam deck, but thats besides the point, but if youre curious, it was surprisingly cool with nixOS, ha, get it?) - watching youtube, gaming and rebuilding! i did NOT have any performance loss DURING any builds, though… HOWEVER, now that i know that those options exist - am i missing out (should i even care about these options and who are these for, actually)?

should i, in the case of having only A-not-so-necessarily-“desktop”-computer, where i do absolutely everything (i.e. a regular ass user), where i do not care about battery or other bits and bobs, but do care about maximum performance, speed, FAST… set every nix scheduler to LOW, potentially sacrificing building speed, or should i set it to HIGH, minimising the amount of time it takes to finish, but rendering the computer utterly useless at the same time? hmm, now that i think about it… it really depends, is it not? what if i want to update as FAST as possible, giving away ALL of the computer’s resources?

ugh. it sounds so stupid, i know, i apologise. the way these options are explained are extremely, excrutiatingly difficult for someone with autism ("best-effort", "idle", "other", "batch", etc.). but bear with me: then, what about the opposite?

i have a HANDHELD device, i care about its battery, longevity, health. but it is a GAMING device (!). in this case, is it valid to keep the settings as default? as in, “it only matters DURING the compilation, and you are completely mistaken, you should completely ditch this idea and ignore those options forever, besides you shouldve at least attempted to calculate the ratio of jobs to cores!!!”

BUT! THIRD CASE: finally, what if i have a remote builder? if the build-process-daemon-thingie has nothing to do with this machine, do i have to adjust its scheduler or not? i think that sounds like a good question.

just wanted to ask this for people in the future, cos lots of folks have slowdowns, hangs and whatnot everytime something is updating. well, actually, i just wanna know if itll have any noticeable impact on OVERALL performance OUTSIDE of building/compiling/updating.

so, for 4 cores, 8 threads, i have calculated that the best values are…

nix.settings.cores = 4
nix.settings.max-jobs = 1

wait. that cant be right, can it? did i miss something, or…? do they also count towards threads or only logical cores? so… i should be having just 1 simultaneous build task at ONCE? but… right now, with the default settings, i have 4? the buildPhase thing, right? i have 4 of these at the same time, which is pretty slow for me… one buildPhase at once would fit me better, so, once again, i am very confused: what values should those settings be (not just cores and max-jobs, but also the scheduling thingies)?

EDIT: oh btw speaking of concurrent tasks. is it possible somehow to change the amount of SIMULTANEOUS DOWNLOADS per build/update? because with my network, the fastest way to download things is to NOT “parallel” it, but rather to download them one by one, and that’d sort it out, i think! thanks in advance!!! (edit2: for example with nh helper tool, is there a way?)

its so embarrassing. perhaps, especially since im trying to achieve something impossible here, right?

thank you guys for any help in advance :sob: :sob:

i love asking embarrassing questions! :yum:

2 Likes
2

To save everyone else the trouble of looking: there is a page at Nix Evaluation Performance - NixOS Wiki but it has nothing relevant!

1 Like
3

finally, a page that describes what it is like to have ADHD. might as well scroll through it, youre gonna get @#$% all outta that bwahahaha :crazy_face:

anyway, lets get serious a bit. LOTS OF QUESTIONS. and rambling. you love it.

in the manual,

https://nixos.org/manual/nixos/unstable/options#opt-nix.daemonCPUSchedPolicy

there is nothing new about them! it’s the same information that’s in the nixpkgs search engine:

https://search.nixos.org/options?channel=unstable&show=nix.daemonCPUSchedPolicy&from=0&sort=relevance&type=packages&query=daemonCPUSchedPolicy

which usually shouldnt be the case. there should be more explainy things in the manual, RIGHT??? it shouldnt be just a copy paste, is that right?

btw, the “declared in” links are dead, lol (in the manual)

i can go deeper into each and every other sentence that is poorly and not-very-understandably phrased but… i dont know how to say this, but i do know that all of this, nixOS, is always developing and there were (are) problems with documentation, which is COOL (get it? cos flakes… ok ill stop), but… i thought buiilding and declaring were always here since day one? like, you know what i mean? one can use nixOS without even really getting into learning nix language, and if there’s a person, okay, who likes the whole philosophy of nixOS, installed nixOS on a crappy laptop… and it lags! so, theyre gonna search for “nixos build freezes”, and theyre gonna eventually find that they should change the way nixOS allocates CPU cores to the building process. and they find that… there is a massive skill issue (in the usual sense), right? would you, as an utter newbie, just imagine, go through all of this:

https://www.man7.org/linux/man-pages/man7/sched.7.html

https://www.man7.org/linux/man-pages/man2/ioprio_set.2.html

https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html

(and the manuals), just to fix a system freeze that is caused by default settings of nixOS? ONCE AGAIN, i dont have these issues, im just kinda speaking out for others. …i just, i dont understand all of that. both the pages and the fact that they couldnt just rephrase it a bit simpler, like:

nix.daemonCPUSchedPolicy = "idle";
nix.daemonIOSchedClass = "idle";
worsens building speed, improves system performance

nix.daemonCPUSchedPolicy = "other"; (default)
nix.daemonIOSchedClass = "best-effort"; (default)
nix.daemonIOSchedPriority = 0; (doesnt work with “idle”)
improves building speed, worsens system performance

at least I THINK this is what they do! :sweat_smile:

in the ioprio_set(2), they say this:

The highest real-time priority level is 0; the lowest is 7.

but nix.daemonIOSchedPriority explains it like this:

best-effort supports values in the range 0 (high) to 7 (low).

guys, perhaps im too silly, too stupid for serious topics like this, but… this is very difficult for someone with autism, i think ive said that before already. i dont like THAT. the way it was explained. OKAY OKAY, yes, sure, i can help, i can fix that… but i would rather start over with like a SIMPLE english (and dutch) manual, completely devoid of technifficult words, you know! but thats just me, maybe.

ANYWAY, what about these:

nix.settings.cores
nix.settings.max-jobs

well, once again, according to this page:

https://nix.dev/manual/nix/2.24/advanced-topics/cores-vs-jobs

and the manual (sorry, i am not aware of any other proper source of information… the https://wiki.nixos.org, which is NOT to be confused with https://nixos.wiki/, though i still confuse them A LOT, i think, are not very reliable… as the person above me wrote :sweat_smile:)

https://nixos.org/manual/nixos/unstable/options#opt-nix.settings.cores

https://nixos.org/manual/nixos/unstable/options#opt-nix.settings.max-jobs

(btw holy shit those pages are so laggy, why?)

…i think those options are just a matter of preference, is it not?

by default, nix.settings.cores is 0, which is to use all available LOGICAL (not threads) cores, but… then it says this:

Some builds may become non-deterministic with this option; use with care!

what does THAT mean? … you mean, if the value is odd? if a laptop has 7 cores, that’d make it non-deterministic by design, by manufacture? THERE IS 0 EXPLANATION. i love sentences like these, actually, just kinda ominously out there, yknow, but no one cares about them, i think. has anyone had this problem with non-determinism in a declarative system? okay, maybe someone have, oh well, anyway…

Packages will only be affected if enableParallelBuilding is set for them.

no idea what this is. do they mean that this option will work ONLY IF the package, that is NOT in the binary cache, will use the specified amount of cores, which has enableParallelBuilding inside of it? uhm, why would it not have that? im just curious, please do tell! :sob:

but what’s nix.settings.max-jobs?

This option defines the maximum number of jobs that Nix will try to build in parallel. The default is auto, which means it will use all available logical cores. It is recommend to set it to the total number of logical cores in your system (e.g., 16 for two CPUs with 4 cores each and hyper-threading).

wait, didnt we just read that already? no, no, no wait. is that right? this is about THE parallel (simultaneous) build tasks, correct? so, why are they talking about logical cores? didnt we literally JUST set that in nix.settings.cores, no?

anyway, lets completely ignore those warnings and before things get any more confusing, quickly write down what you have in mind thusly:

for a system that has 4 CORES, 8 THREADS, a typical Steam Deck should have:

nix.settings.cores = 4;
nix.settings.max-jobs = 1; (buildPhase?)

but, according to this person,

cores and max-jobs multiply together? so, i will eventually get 4 parallel processes? im too lazy to make another post to ask just one question, so, WHAT IS A PARALLEL PROCESS???

is it the buildPhase thing, the ACTUAL building process, or is it the downloading from binary cache, unpacking, whatever else is happening when you rebuild, all of the above?

i just gotta ask this, guys. i WANNA KNOW. gotta feel stupid now to be knowledgeable later.

by the way, the link to that post, where the person explains that they multiply, the OP clearly has an 8 core CPU, and 16 threads. it’s just a typical AMD thing, you know? the name of the CPU gives it away. so im not sure what is going on anymore… are we talking threads or logical cores, guys… :triumph:

im just gonna hit “reply” and hope for the best :crazy_face:

EDIT: i feel like i have to clarify: i love nixOS. im never distrohopping again. its intuitive, its declarative, its THE BEST. but once you wanna know something - THATS IT. muhahaha, you shall never know the secrets! and like, ive seen many people use “ai” to explain some options to them. well, that “ai” is taking information from someplace that already exists, yknow… so it’s also wrong and outdated bwahahaha. anyway, yeah, i wish i knew about nixOS earlier, cos its just soooo cool. but it got me thinking: was it worse? back then? the documentation? …

4

You got this one backwards. A hyper-threaded processor, advertised as, say, having 8 cores and 16 threads (2 threads per core) has 8 physical cores, but 16 logical cores, i.e. “hardware threads”. A Linux system will mostly report this as “16 cores”, without distinction. htop will show individual load for 16 cores, /proc/cpuinfo will list 16 cores. See also this question.

My understanding is that this refers to the fact that certain compilers or compiling certain codebases may not produce bit-for-bit identical output, or compilation may outright fail when you allow them to compile several parts of the code in parallel, but will produce a consistent result if you compile stuff sequentially.

enableParallelBuilding seems to be the default for any package that uses stdenv (which is the majority of packages in nixpkgs). Notably, this issue lists a bunch of issues that were previously present when building certain packages in parallel. You can also use GitHub code search to find packages where parallel building is still explicitly disabled due to issues.

I think the documentation you linked is pretty clear. max-jobs defines how many derivations will be built in parallel (at the same time) - the Nix daemon itself takes care of this. cores defines how many “cores” may be utilized while building each one of those derivations, and it is up to the derivation to respect this (e.g. by running make -j$NIX_BUILD_CORES - the cores setting is exposed to the derivation as the NIX_BUILD_CORES environment variable).

The funny thing I realized is that NixOS sets both max-jobs and cores to auto or 0, meaning they will both default default to the number of logical cores. So for a CPU with 16 threads you will potentially have 16 * 16 = 256 build processes. There’s been discussion about improving this, but there’s no consensus yet, it seems.

Looks like max-substitution-jobs is what you need. I don’t blame you for not immediately realizing that substitute roughly means “download from a cache” in Nix land.

2 Likes
5

It’s not. But as https://nixos.org/manual/nixpkgs/unstable/#var-stdenv-enableParallelBuilding mentions:

Unless set to false , some build systems with good support for parallel building including cmake , meson , and qmake will set it to true .

So it’s usually parallelised anyway.

1 Like
6

wowie!!! :exploding_head: :exploding_head: thanks heaps and bunches!!! really, a lot. i much prefer talking to real people… at least i think youre real. anyway, thats just mind boggling… knowledge… okay… mhm. ohhhhhhhhhh thats right, yeah…

sorry, i just realised im typing while reading. uhm, but what about the other 3?

nix.daemonIOSchedClass
nix.daemonCPUSchedPolicy
nix.daemonIOSchedPriority

did i get that right, when i said

is this how they work? SPECIFICALLY DURING the buildPhase? so they DO NOT affect unpacking, copying, everything I/O, as written literally in the name, orrr?

:frog:

OH! ALSO. wait wait wait, one more thing

what about nix.daemonCPUSchedPolicy’s

This policy propagates to build processes. other is the default scheduling policy for regular tasks. The batch policy is similar to other, but optimised for non-interactive tasks.

what exactly are regular tasks and non-interactive tasks? THANKS IN ADVANCE!!!

:snowflake:

7

That’s because building is very different from evaluation.

1 Like