A (sad?) state of Nixpkgs

Development Nixpkgs Architecture
1

:wave: Hello fellow Nix users!

I’m starting this conversation to better understand the state of nixpkgs and the way the community as a whole approaches the project.

Just to be clear - I love what Nix ecosystem has to offer and I do really want to stay here in the long term. Yes, maybe Nix language didn’t need to exist and we could have gotten away with an existing (slimmed down) language. Yes, flakes vs no flakes is the endless debate. But I can deal with all of this and be quite successful and productive.

What I am concerned (for the lack of a better word) about is the state of the nixpkgs repository. The package selection is great, but everything is always built differently, following different standards and expectations. For example, you can’t apply a custom patch to Chromium right now. Further to that, quite often there isn’t a way to build a package with exact flags and configuration I would like, simply because these flags aren’t exposed through Nix derivations.

To put things into perspective here, I am coming from Gentoo Linux, where I spent over a decade. What’s worth knowing about Gentoo is that it is a source-based distribution and the maintainers and community are obsessed with providing as much flexibility and control over the dependencies and build options as practically possible. This means that if there is a flag that can be enabled or disabled, Gentoo ebuild would most likely expose it and allow users to toggle it. But they don’t stop there. Quite often, packages have “automagic” dependencies, meaning the dependency will be pulled in if it is presently installed on the system, and will not be pulled in if it is not. This obviously is bad for reproducibility. So when encountered, Gentoo folks would try and work with the package maintainers to expose a proper configuration flag. If that doesn’t work for any reason, they would go and create a patch that will add necessary flags and turn things off as required and will keep maintaining it themselves. If all else fails, as a last resort, they will just give up and mark the package as a hard dependency. This is quite a challenge for the Gentoo ecosystem, however…

…this problem does not exist in Nix. We have full control over what dependencies are exposed to the package at the build time and during runtime, so we can neatly control any hard, soft or automagical dependencies and achieve a much greater degree of control. Awesome, right? Well, not really, because currently nixpkgs is severely lacking in this aspect.

A few recent examples I ran into (in addition to not being able to patch Chromium as I mentioned before):

  • Nautilus (Gnome File Browser) has an automagic dependency on gnome-user-share. This pulls in the Apache HTTP server and stuff. For security reasons, I’d prefer not to have randomly sprinkled HTTP server on my machine, thank you very much;
  • pipewire exposes a gstreamerSupport flag, which is great, but is currently broken;
  • Also pipewire breaks with bluezSupport=false flag;

I find this to be a massive letdown - having such a powerful tool and severely underusing it, missing such a big opportunity. Am I the only one thinking this? How do people generally approach problems like this in Nix? I wouldn’t want to fork every single package I use and expose more flags, is that what everyone else is doing?

Any input is appreciated.
Thanks in advance,
Garry :saluting_face:

11 Likes
2

Pipewire maintainer here: most of those options are completely unnecessary, and should be removed. We do not support controlling dependencies to the extent Gentoo does, and that’s a feature, not a bug. You can still override anything you want manually, but providing flags to do all of this will always make it possible to have configurations that break, and this is just not something anyone wants to deal with.

18 Likes
3

Hey @K900! Thanks a lot for the prompt reply.

I’ve noticed that, and hence I am posting this discussion. A few questions:

  • Why not? Seems like Nix is perfect for this.
  • If not, then what are my options? Just fork and maintain my own copy? Are you firmly against enabling this use case in principle and I should fork and maintain my own copy?

I am totally fine with things breaking on updates and changes - such is the life of the software. I am happy to report the issues and contribute fixes.

1 Like
4
  • Why not? Seems like Nix is perfect for this.

Because it creates a lot of work for maintainers.

  • If not, then what are my options? Just fork and maintain my own copy? Are you firmly against enabling this use case in principle and I should fork and maintain my own copy?

You can use Nixpkgs overlays and things like overrideAttrs to do all kinds of local customizations, including these.

I am totally fine with things breaking on updates and changes - such is the life of the software. I am happy to report the issues and contribute fixes.

We already have extremely limited review bandwidth, we should not be creating even more workload targeting ultra niche use cases.

19 Likes
5

That one is easily override-able:

nautilus = pkgs.nautilus.override {
  gnome-user-share = null;
};

I too support the concept of maintaining such flags, and I try to do my best in some of my own packages were these flags are meaningful in my opinion. However, indeed that can become hard for some packages, so in your case I can only recommend to you to try to fix those issues yourself:).

Also relevant:

Given such a module system laid out in Nixpkgs, it would be interesting to explore the automation of testing combinations of such flags via a passthru.tests by using lib.mapCartesianProduct.

6 Likes
6

But that’s the problem. The combinatorial explosion. Flags, platforms, etc. And if you’re overriding a package that’s depended on by other package(s), it gets even worse. Encouraging such overrides leads to a state where you have less sharing between people, which is bad for caching but also for debugging and maintenance.

12 Likes
7

Of course this presents the same difficulties with nix vs whatever other build system. It seems like Gentoo users have far more free time to maintain patches for every combination of flags. Nix is not a magic sauce to prevent such maintenance effort. So it’s a question of do nixpkgs maintainers take on that responsibility or do end-users? If it’s down to niche usecases, it should be the latter, as exposing some flags implies that nixpkgs (to some degree) supports the use of said flags.

If you want to customise packages further locally, look into the overriding section of the manual. If you instead want to forcibly replace a dependency throughout a nixpkgs instance, overlays are better suited.

Of course, note that use of overlays implies that you will rebuild the entire reverse dependency tree, since hydra would not have cached it.

2 Likes
8

Well apparently it’s not “totally fine” as you seem to be confused as to why some flags don’t work calling it a “massive letdown”. These flags likely worked at some point but stopped working some time later because nobody cared for them.

I agree that we should strive to make packages configurable but only to a point where it doesn’t present significant maintenance burden but there should be no expectation that even just the subset of configuration options that do make sense together work correctly.

I’d love if Gentoo and Nix could join forces in making packages meaningfully configurable on a fine-grained level via a USE-flags like mechanism but I also see it as a fringe use-case. The extent to which I care that i.e. an unused apache is in my closure is that it takes up a few megabytes of storage which is to say: Not very much.

Keep in mind that a “package” being present in the Nix store as part of your closure because one package you care about transitively depends on it is not even close to being as significant of a problem as a package being installed on an FHS distro because we have very local scoping while an FHS distro is, by design, always globally scoped.

5 Likes
9

Also, the way package rebuilds work in Nix, CI for a combinatorial explosion is even more complicated or expensive or both to set up than elsewhere. Given that CI complexity and review bandwidth are both bottlenecked… work in this direction is not going to happen anytime soon.

10

Thanks a lot for your input everyone.

Fair. Although I was happy to open a PR to fix it and keep it up to date, because I am using this flag, but I do understand that at an overall scale we are not there yet.

It is clearly a skill issue on my end, so can you please give me an example, how can I toggle the build (configure/meson/ninja) flag using something like overrideAttrs, because I have been unable to achieve this.

I did not realize that, but you are actually right, because we don’t use gnome-user-share in any way, apart from it being present. Thanks for this tip. It won’t work for every instance, but for simple instances like this - makes total sense!

Even though I don’t disagree with this, I personally don’t care much about the caching and happy to build things from the source. Debugging indeed can be harder, but we can say “we support defaults and if you tinker - you are kind of on your own to get yourself out of the trouble”.

Perhaps I wasn’t clear on what I mean here. Because Nix has full control of the build environment and which packages / header files are exposed at compile time, we don’t have to patch anything, we can just expose or hide the given package and automagic dependency will or won’t link. This is impossible on any other distros, because they rely on system-wide libraries.

That’s what I was looking at, but some things are not possible, like Chromium patching.

Sorry, maybe I wasn’t clear on that, but what I am “upset” about is that I found a broken thing and offered help to fix it, but instead the response was “let’s instead remove it”. Not quite what I was expecting, especially given how much opportunity there is in Nix for this stuff.

The way I see this, and the whole point of something like a “publicShare” flag in Nix that will enable multiple configure options and pull in dependent packages is that it serves both as a documentation and a cohesive item that can be enabled or disabled by users who need it.

It very much depends on your use-case, circumstances and the risk profile. I understand that majority of people don’t care about the number of dependencies and happy to pull half of the internet onto their system, but given latest situations with xz-utils and polykill, I’d like to err on the safer side.

1 Like
11 
pipewire.overrideAttrs(old: {
    mesonFlags = old.mesonFlags ++ [ "-Dfoo=disabled" ];
});

That’s just an implementation issue, and Chromium is one of the most complicated packages in the entire tree.

This was not a response to you stepping up to fix it, it was a response to it being broken, and clearly not being used enough, because you are the first person to even notice it being broken in a very long time.

Those dependencies being somewhere on your system does not make them part of your attack surface.

3 Likes
12

Depends on the dependency and the attack surface. XZ attack was sophisticated enought to selectively enable itself if it is loaded specifically into the sshd binary. I don’t want to try and convince you, we just see things differently and that’s okay. I appreciate your help and engagement with this topic.

Thank you. I’ve tried something like this before, but it didn’t do what I was expecting. Let me tinker around a bit more with this.

2 Likes
13

IIRC, Nautilus only uses schemas from the package. It might be possible to split them out into a separate output and then you would not depend on Apache at all.

Edit: Looks like it is just using the presence of the schemas as a proxy for gnome-user-share being available, so it will produce false positives on systems without gnome-user-share. We should probably change it to detection of presence of the org.gnome.user-share.webdav service on the D-Bus session bus.

6 Likes
14

this adds ridiculous levels of complexity. why? because not only supporting these options is a lot of work by itself, but then there are not only other packages, but also configuration options that may depend on some feature that the package can be built without, and cross-validating all this is gonna be a nightmare

1 Like
15

Just in case this is not clear and while this might not feel optimal, overlays can not only update the existing recipe, they can also replace existing one:

self: super:

{
  chromium = prev.callPackage ./pkgs/my-chromium.nix {};
}

Thus, you can always clone the expression you are looking to update locally, if patching it externally feels too complicated at first.

Then you would have better examples of what you want to achieved that you failed to do with the various override functions, and you will probably find people to help you at doing it, if this is possible.

Being able to tune Nixpkgs is part of the features that made me stick to NixOS since 2007.
I wish you good luck in your journey :wink:

10 Likes
16

Yes, I am aware of this, thank you!

This is what I meant by ^ replacing packages with a completely custom derivation.

17

I’m sure everyone has their own way of going about it.
What I personally do is maintain my own nixpkgs fork, which

  • allows me a lot more control over what I change
  • isn’t extra work (as I already fork nixpkgs for the purpose of contributing back)
  • allows me to avoid the performance penalty of extra overlays
3 Likes
18

I just use .override and .overrideAttrs in very limited scopes (so not with an overlay) to try and stay as close to hydra as possible.

If I already maintained a fork I’d probably do what @waffle8946 does, though.

I also tend to find that derivations are pretty easy to modify. There aren’t very many like chromium.

3 Likes
19

Note that “maintaining your own fork” is not nearly as much work as you might think. The way I update my system nixpkgs works by updating my fork, so I just git pull Nixpkgs to update and thereby also keep my fork up to date. Conflicts are usually easy to resolve and I don’t have many conflicts to begin with.

7 Likes
20

In practice, those options work that people use and care about and help make work.

If you find an option that doesn’t work, and you help fix and maintain it, chances are quite high that it’ll stay.

The one thing I wish existed to make non-default options better tested:

Something that makes it easy to set up your own CI, which automatically tracks nixpkgs (e.g. nixos-24.05 and nixos-unstable) and builds things with custom options.

And the ability to post such “community CI” results automatically on Github.

That way, people can contribute CI time for things they care about which official nixpkgs/ofborg/Hydra don’t cover.

For example, I want to be able to just buy a few Hetzner servers that track nixpkgs and automatically tell people and me when their upstream nixpkgs PR breaks musl builds.

The PR authors have no obilgation to fix that, but people who care about musl support want to be alerted so they can fix it.

8 Likes