I noticed that most of my issues with nixpkgs that require a significant modification (warranting a PR or a local copy in my flake) are related to “optional” functionality. In many cases these are actual bugs for ubiquitous use cases like wayland systems that have not been solved due to the relatively new state of adoption.
There are many packages that absolutely need optional “switches” for compile-time choice of functionality (the alternative would be to impose huge bloat on users), though the granularity of “Gentoo-style USE flags” has been all but ruled out by the community for good reason (too many variants lead to explosion of the binary cache for instance, and discoverability suffers too).
Currently, nixpkgs is rife with ad-hoc switches like withX, withWayland, noXlibs, enableSonos, etc., even many different names for the same functionality.
If we want to improve nixpkgs, I’d argue it is a must to tackle this issue sooner rather than later, i.e. we need standardised boolean switches for derivations.
How? (Ideas in arbitrary order, not mutually exclusive):
low hanging fruit:
rename existing switches with the same function to the same name (search/replace in nixpkgs)
make a list of needed switches per category (sound, graphics, dbase?) somewhere prominently in the documentation / as README.md in nixpkgs
could there be a way to enforce this somehow? (attribute set of switches as a default argument, with sensible default values?)
how to “start” such a development in terms of organisation? RFC?
should we have a (more or less) maximum count of switches to strive for to limit the number of variants? How many?
what process to follow to “request an additional switch”? RFC?
I personally think we need to first improve the whole mkDerivation situation, I’m an advocate for module system derivations personally. If we had that it would immediately, make things more discoverable and structured. I do want to poijt out that limiting the number of switches is I dont think a good idea, limiting anything is imo a bad idea, though scrutenizing how many different variants we build is a good idea imo.
Could you give an example of a “module system derivation”, and how we could (start to) improve the “mkDerivation situation”?
As for “limiting switches”, I guess indeed a better mental model would be somehow a “standardised set”, though I’m not sure how we could avoid unnecessary deviations without some form of “enforcement”. I guess if one would have something like the set of standard, agreed switches in an attribute set (say, featureSwitches = { withX = true; withWayland = true; with ... }), one would still have the freedom to have an arbitrary argument to a package function in case it’s needed. But maybe there’s a smarter way to handle the concept entirely.
One could even conceive of a function handling this, e.g. if there is a boolean argument to a package expression (or any argument that is not an existing pkg?) which is not part of a predefined set of standard switches, a warning could be thrown?
Thanks for the pointers, that was one of the things I was hoping for. (Yes indeed I have already been about this topic, since it’s a thing that often pops up when fixing packages that have oiptional features. Actually the trigger was my renewed interest in deploying a headless calbre server, you might even vaguely remember the related PR (as yet unresolved) ).
I hope we’ll have at least one potential solution (I didn’t have time yet to study the details of RFC169 and nixpkgs#312432 yet).
Probably it doesn’t even have to be an all or nothing solution from the start, any de-duplication of same-use-different-name flags could constitute a win, even if nothing else would change. So specific PRs for improvement could coexist with (the IMO very much needed) development of a more fundamental solution.
I personally think that having a bunch of switch combinations that don’t get built on Hydra isn’t great, because they will almost inevitably bitrot. Every switch increases the combinatorial explosion of possible configurations and makes it more likely that there are combinations that simply don’t work. Ones we need to switch around in‐tree are okay, and for some especially big flags or dependencies there is an argument for exposing them to users (e.g. minimal versions of packages with very large dependency closures or security implications), but I’d personally like to have fewer exposed package options than we do today. The cost of supporting an option frequently outweighs the cost of whatever dependencies it gates. People who want to extensively customize a package can always fork a local copy.
In principle I agree with limiting the number of switches, that’s why it features prominently in my first considerations. So maybe the ones we could “standardise” should work toward implementing a select few “major use-case profiles”, such as “withWayland”, “withX” (where “headless” could then e.g. be the implicit “! (withWayland || withX)”, and maybe a few others (related to the supported sound system or OpenGL, or GPU family (NV/AMD/Intel).
The nice thing is that the defaults for some of the above could be taken from the nixos module config (OS feature switches for graphics and sound).
If the number of switches thus remains small enough, one could probably afford to build them on Hydra too? The nice thing is that the “officially supported” permutations of nixpkgs would then (logically) be the ones that have the same value for occuring switches throughout the dependency tree, which should keep the number of variants low enough to be practical. (The situation for per-package overrides would then remain as it is now, i.e. they would be locally built on the user’s system).
A rephrased argument in favour of really needing the (at least minimum/most important) feature flags/switches: after system these are the second most influential parameters in terms of dependencies for OS deployments and package configuration (build-time options) throughout nixpkgs, that affect the dependency tree to such an extent that choice/options “at the user’s fingertips” (i.e. without needing forks) arguably matter significantly.
I think a simple boolean “headless” option is a more valuable toggle than Wayland and X11 separately. For instance, packages that support Wayland and X11 on Linux might use Cocoa instead on macOS, and the GUI dependencies can be several layers of dependencies deep. The advantage of explicitly toggling off only one of Wayland or X11 is minimal; basically just closure size. Even then I think it’s only worth it where it significantly affects closure size (e.g. I think FFmpeg qualifies here).
Defaulting options for the whole package set based on NixOS configuration has the problem that we either get Hydra to build a huge number of packages twice or the toggle is essentially useless for anyone who doesn’t want to spend hours building packages on every update, yes. If the number of packages that have toggles like that are tightly limited then it should be fine (and indeed Hydra does already build e.g. the headless variants of FFmpeg), but I wouldn’t want to encourage people to add options to packages that really don’t need them.
There are probably many others that qualify on that criterion that are not directly obvious, point in case this PR I referred to above.
But the idea that toggles appear in a package’s arguments only if it is actually useful for the package (in terms of “bloat”/closure size and/or other wastefulness) was assumed all along, so in total (applied to current nixpkgs) the expected outcome would actually be more lean than the current situation:
first step: define major use-case profiles and their associated flags
second step: harmonise names => no structural change but
better discoverability
reduction in actual flags (e.g. withXorg, !noXlibs, withX11, withX, withXwayland => withX11 (just very quickly perusing nixpkgs yielded 5 switches for the same feature!))
feasibility to actually build (in Hydra) dependency chains based on certain major use-case profiles
Third step: reconsider existing (legacy) flags on a per-package basis (async/per maintainer/PR), based on a “top level agreement” (per RFC?)
Regarding wayland specifically, it just so happens that I stumbled on “many” (mostly qt5) packages that only started in xwayland or crashed. The logical fix in such cases was to optionally build for wayland using a switch. But I agree, if the inclusion of e.g. qtwayland only marginally affects closure size, we might as well go for just “headless = true/false”. Against this argument speaks the desire we might have to phase out X11 alltogether (for headfull systems), so it would be a clear objective of we could build our complete deployment without any X-dependencies, which arguably could constitute significant “savings”?
I think I pretty much agree with @jtojnar on that PR. To be honest noXlibs is a bad UX even now (in ways I’m not sure this proposal would address) and NixOS is inherently a really bad choice if you’re very concerned about disk space. Things breaking on Wayland is bad but we should fix it in ways other than exposing a toggle users have to manually flip.
My main point in the X11 example was merely that the proposal would at least reduce all X11 switches currently in nixpkgs from about 5 to 1, which is not bad for low hanging fruit.
Generally I strongly believe that we need at least a switch that allows us to do “headless” in a consistent way (i.e. throughout nixpkgs). The PR was a kind of “short-cut” towards making that behaviour consistent for a framework (gtk3) that apparently can be compiled/used without depending on a huge chain of dependencies (by selecting the compiled-in rendering backends consistently with the top level switches). TBH the PR itself is probably only tangential to the “trigger” for it (calibre-server headless), because its main dependency is in fact Qt. But conceptually I strongly believe that if a large and ubiquitous framework is used by (potentially many?) programs/packages that can be built in headless mode, and offers build processes for headless itself out of the box, it is very bad UX to force a user to pull in a huge chain of desktop dependencies that can be easily avoided by a switch.
My main point is that to optimally satisfy major use cases is a top-level requirement, and it should be the job of the package system and derivations to satisfy this in the best way possible.
The customer is King
Specifically: if there is a way to make this happen/improve this without significant disadvantages at an acceptable cost (note I’m advocating for an incremental and non-intrusive approach), it should be done IMO.
).