Where did you get stuck in the Nix ecosystem? Tell me your story

[doronbehar]

@turion

Finding builtin and standard library functions. I used to full-text search through the corresponding parts of the nixpkgs and nix manuals, often ending up simply not completing my task because it was so complicated to find the right function. Luckily there is noogle.dev now.

Wow this website is so helpful! Thanks for sharing, I wonder whether the Nix ecosystem will agree at some point to point to it instead of the official docs.

[nessdoor]

Being someone who has issues with discerning between useful work and waste of time, my point of view on Nix and NixOS is, at most, unreliable, as what I consider small hiccups are unbearable by most Real Engineers™, as @MattRixman calls them.

Therefore, I will say what an acquaintance of mine has said on the matter. He tried NixOS out because it was one of the few distros where a certain package was said to be working without major hindrances.

Although I spent hours reading various docs, I lost enthusiasm

And that’s a point that was brought up by many others, here.

Then I came to the conclusion that a declarative build is cool, but with too much overhead to manage

Yes, because we lack any sort of tooling that actually eases writing Nix, and the API documentation for NixOS is lacking at best. In addition, Nix as a language is kind of ugly, don’t you all think? I wish a working group existed to work just on making Nix a better language, as it definitely has room for improvement.

Home-manager and flakes explained like ****

And here we go again. Flakes need to happen, it seems that they are stagnating while absorbing huge amounts of developer energies. Maybe they don’t have to be perfect from day 1?

I lost it and now I’m playing ******* on Windows
Moral: Windows works and I’m tired of pretending it doesn’t

And that, folks, is how we lost not only a NixOS user, but a Linux user altogether.

I would like to add that we act as if everyone who uses Nix is either also part of the people who open issues or PRs on the Nixpkgs repo, browse this forum, or chat with us over at Matrix. That’s not true. Most of the people simply cannot be bothered. Some of us seem to be looking at Guix with admiration, but let’s not forget that that, too, is an incredibly niche system, especially if you factor in the complicated relationship that mainstream computer people seem to have with the foundation behind them.

So, where is our strength? What is it that we do better than our competitors? How do we march on to become the declarative package manager/distro of the future? Is there any exclusive functionality inherent to the way Nix handles things on which we can capitalize?

[NobbZ]

I thinked back and forth about shall I write or shall I not, and at the end I decided to just dump some random experiences and thoughts here.

1. Even though I got the commit bit, there was basically no introduction about what the rules are. After having asked in the matrix, it boiled down to “apply common sense”
2. I want to review and/or merge much more often, though whenever I decide to do some reviews I pick random ones from the first GH page and look at them. Any non-trivial PR is already commented on, and I do not have anything to add or am completely opposite opinion. Quite often I just let minor formatting issues slip, as I do not consider the PR the place to discuss that, especially as there is still no formally written formatting standard to apply.
3. In general there is to much opinion and to little reasoning in reviews
4. there are opposing standards between different sub-ecosystems (can’t give an example, as this is mostly hearsay)
5. The GH repository is huge… To be able to contribute an initial clone of 3+ GiB is required, leading more and more poeple asking for a way to contribute via GH-Web-UI, which doesn’t give the required amount of control for manipulating the history as required by the contribution guide.
6. There are PRs open for many years, that do not have any chance for getting merged without a major conflict handling overhaul or might even been partially or completely superseeded by other PRs already merged.

[Patrick]

To be able to contribute an initial clone of 3+ GiB is required…

In fairness, git clone https://github.com/NixOS/nixpkgs --depth=1 right now results in pulling 44.26MiB over the network, for a total space on disk of about 198MiB. Which is still vast, but an order of magnitude less vast.

[Alper-Celik]

--depth=1 works on git fetch too, you can use it like
git fetch origin --depth=1 <some revision> to just download the rev you want without the history

[NobbZ]

Last time I tried, I was unable to create PRs from a shallow clone

[doronbehar]

@nessdoor

Then I came to the conclusion that a declarative build is cool, but with too much overhead to manage

Yes, because we lack any sort of tooling that actually eases writing Nix, […]

I agree that the docs could be improved, but I don’t think that it will ever reach a point where it will be easier to manage Nix system in comparison to a regular distribution, because you are managing something much more complex, but with features that worth the efforts.

@nessdoor

So, where is our strength? What is it that we do better than our competitors? How do we march on to become the declarative package manager/distro of the future? Is there any exclusive functionality inherent to the way Nix handles things on which we can capitalize?

I am sure you are aware of these strengths (see also this thread). My point is that NixOS strengths come with a price, and if you are not ready to pay the price because you don’t appreciate the product, that’s OK :), that’s why it’s a niche system.

@nessdoor

Nix as a language is kind of ugly, don’t you all think?

Maybe Python is less ugly, but have you ever read Guix’ packages’ source code? Nix is much easier to understand then Guile.

[iFreilicht]

@nessdoor

In addition, Nix as a language is kind of ugly, don’t you all think? I wish a working group existed to work just on making Nix a better language, as it definitely has room for improvement.

There is! It’s called Nickel, and nickel-nix is an experimental implementation that integrates it into Nix.

And while that is a welcome development, I can’t help but feel there is just too few people doing too many things. A lot of work in the Nix ecosystem is still conceptual, there is divided attention between the old and new CLI, way more tickets are being opened than completed, same for PRs.

And let’s not forget, almost all contributors to Nix offer consulting on how to integrate Nix as far as I understand, so the amount of stuff that has to be done is just completely overwhelming compared to the amount of stuff that can be done.

I mean just look at one of the core team’s weekly meeting notes, the issue tracker and the PR List;
8 Issues/PRs discussed. 21 Issues opened. 11 PRs opened. All within one week. Maybe I’m pessimistic, but it seems to me that there is a severe lack of manpower. Most issues aren’t even triaged, ever.

[nessdoor]

@doronbehar

I don’t think that it will ever reach a point where it will be easier to manage Nix system in comparison to a regular distribution

I dream of a future where I can specify which modules to include in my system via a two-columns menu-based UI that lets me select a flake -> module pair to add, and then present me with a graphical shell on which I can set all the parameters of the module (although in a simplified form without let bindings and ad-hoc functions).

But that’s a dream of mine.

@doronbehar

I am sure you are aware of these strengths […].

There’s a reason why I’m stuck with NixOS since September 2020 and contribute stuff to Nixpkgs whenever I can. My point is that I am not sure we are selling ourselves well enough to the wider audience, considering that people nowadays are totally in love with containers and are more willing to take that as an approach to manage complexity and reproducibility.
Heck, then there is this friend of mine who writes Ansible scripts for a living, a ton of them, and I begin to wonder how that is seen as sustainable rather than writing Nix expressions to solve deployment and management problems in a more robust way.

@doronbehar

[…] have you ever read Guix’ packages’ source code? Nix is much easier to understand then Guile.

I did a little, and, maybe due to personal inclinations, I find them much more clearer than Nix expressions. Maybe it’s because I prefer the syntactic simplicity of Lisp to the vaguely haskelly syntax of Nix with pragmatic sugar stuck on it (which is far from the clarity of Haskell code, in my opinion).
In addition, Guile is a proper programming language, made to express logic, while sometimes I find it difficult to do things programmatically in Nix because it lacks the facilities of other programming languages (maybe because I cannot remember properly what is in the standard and the Nixpkgs library).
Let’s not even talk about the fact that Nix is a completely new language, while Guile is a scheme, so you cannot apply much transfer learning and you have to teach people its syntax from scratch, and busy people are averse to learning new syntax.

@iFreilicht

There is! It’s called Nickel , and nickel-nix is an experimental implementation that integrates it into Nix.

I heard about Nickel as a generic configuration language, but I never looked into that, so I can’t express any opinion on its syntax. I’ll be checking it back as soon as it comes out of its experimental phase, as I am still not sure what its niche is, and how it solves the little syntax issues I find in Nix.

@iFreilicht

And while that is a welcome development, I can’t help but feel there is just too few people doing too many things.

Absolutely, it’s a massive project that I want to see succeed and that’s why I really want to get more involved into its development.

@iFreilicht

[…] the amount of stuff that has to be done is just completely overwhelming compared to the amount of stuff that can be done.

I think there are also problems of efficiency here and there that must be addressed both at the technological and organizational level, but I am in no position to offer suggestions, as for me The Team feels like a very distant entity of which I know almost nothing about.

@iFreilicht

Maybe I’m pessimistic, but it seems to me that there is a severe lack of manpower.

We know that the solution to scarce manpower is automation, and I am happy to see efforts going in that direction. I wonder what the strongest limiting factors are, though.

[MattRixman]

@nessdoor

So, where is our strength? What is it that we do better than our competitors?

There are a lot of other ways to think about the question of who our competitors are. Depending on who you are, you might instead think our competitors are:

• standalone package managers: guix, homebrew, pip, cargo, cabal, maven, choco…
• config tools: ansible, puppet, saltstack…
• operating sytems: debian, redhat, gentoo…

It’s hard to be so many different things to so many different people. You’re never going to make them all happy at once. But if you have so many different kinds of people participating in your ecosystem–that’s an advantage.

I think of it as analogous to Arch and Manjaro. Arch is where the wizards and warriors do battle with hordes of bugs from the bleeding edge, and Manjaro is where the newbies get to live in the peace paid for by that fight.

Every time I contribute to nixpkgs, I’m amazed by the volume of activity on that repo. I can set something down for a week and rebase thousands of commits when I pick it back up. It’s group coordination on a scale rarely seen.

@nessdoor

I would like to add that we act as if everyone who uses Nix is either also part of the people who open issues or PRs on the Nixpkgs repo

I was a Ubuntu user for a decade, and in that time I built only a single .deb and didn’t publish it. I’ve been a NixOS user for two years and I’m already making (small, so far) contributions to nixpkgs, with a growing to-do list of more contributions because it’s just going to be easier to reference this stuff by name instead of pasting scripts through slack to my colleagues.

That, I think, is our superpower. Yes, it’s hard to configure your system with nix. But from there, you’re only a small step away from configuring anyone’s system with the same code.

Yeah, we lose users because we have such a thin wall between user and package maintainer. Let’s get better about that. But we also have a lot of package maintainers.

(disclaimer: I’m new here, I don’t think I speak for this community, these are just my impressions)

[MattRixman]

@DianaOlympos

Impossibility to find someone that understand wtf the beamPackages wrapper do. This has limited a lot of progress over cleaning up how nixpkgs handle all the Erlang stuff

I’ve definitely been feeling that kind of pain.

Each language ecosystem appears to take on its own quirks when it gets nixified, and aside from reading the code in nixpkgs it’s really unclear where a person should go when they want to get up to date with those quirks.

Once I know nix better I intend to write a number of blog posts for the languages I use (“Python on Nix”, “Nim on Nix”, etc), but it feels like it’ll be a while before I’m confident enough to do that.

In the meantime, I wonder if something can be done to ensure that the right people cross paths. It’s nice to be part of the Nix community, but maybe it would be more helpful to be part of the Nix-Erlang community. Seems like language-focused subforums (subfora?) might help address this as well:

@DianaOlympos

But it makes it hard to find places and ways to express my needs. I simply do not see where and how to express them.

Erlang is on my list. If there ever emerges a place for that community to gather, I’ll probably be lurking there in the future.

Good luck with that packages wrapper. A kind soul recently taught me a lesson about the python one (thanks @anund), so it would seem that making mistakes in public and then making noise on discourse about it is a decent strategy (even though it doesn’t feel quite right).

[shamrocklee]

The documentation about porting Nixpkgs and Nix to a new platform is absent. I tried to figure it out once, but it turns out that we could only find (fragmented) information inside PRs and commits. This is a pity as Nixpkgs is advertised for its ability of cross-platform build and cross-compilation.

Nixpkgs porting guide

Nixpkgs is well-known for its ability to cross-compile on many platforms and compile natively on many of them. However, despite the continuous efforts and dedicated maintainer teams working to port Nixpkgs and Nix onto new platforms, there is still a lack of documentation about the process of porting. It could be a good way to collect the domain knowledge during successful porting and encourage new contributions. I opened a GitHub issue to keep track of this need: https://github.com/NixOS/nix…

[AndersonTorres]

@doronbehar

Maybe Python is less ugly, but have you ever read Guix’ packages’ source code? Nix is much easier to understand then Guile.

Because Guile, or more exactly Scheme, is too terse.
A fat language like Common Lisp would be way easier to understand

[asymmetric]

@fzakaria

The bureaucracy burden was too overwhelming so I stopped pushing on it.

If you’re willing to share something more specific about the bureaucracy you encountered, I think that would be valuable, as yours was a contribution that did make it into other distros, so it could be interesting to see why it didn’t make it into nixpkgs.

[woobilicious]

Bureaucracy / maintenance issues:

• From a real basic perspective, most “starting to use nix” guides do not layout or format derivations in the same style as nixpkgs (callPackage is hidden inside all-packages.nix)
  This leads to the problem that you end up calling ugly code like: nix-build -E 'with {}…"
  Once you realise you can put the callPackage in default.nix, the code is explicity not compatible with nixpkgs, you can’t just copy-paste it in to it. It nix3 /fakes intheory fixes this…nixpkgs isn’t layed out like that

• WHY IS all-packages.nix 41k LINES OF CODE??? WHY is it not organized based on the folder structure??? WHY NIL AND RNIX-LSP FREEZE EMACS??

• meta.maintainers is missing from core features, and modules, because most core features are heavily modified derivations they seem to be treated differently. A few times I’ve opened up a Issue and didn’t know who to @. for example the “default” less implementation is broken, it fails on some files because “strings” is missing, I had no clue why, after a lot of digging I figured out why, I’ll leave this as an exercise to prove my point, hint, the official maintainer hasn’t made any commits since 2016, and it’s not the stand linux less command.

• nix 3 needs to be enabled, but give a warning, flakes should just work.

• I think there’s a lack of “leadership” or clear defined hierarchy in the sense that no one is providing a direction or holding maintainers to some sort of standard, which is why it’s simultaneously impossible to get things PR’d and also why stupid decisions like hard-coding a badly maintained less package got merged in to the core definition of nixos.

• secret management sucks, especially for builders.

• Its hard to communicate the difference between nixpkgs the repo and nixpkgs the package set pkgs = import <nixpkgs> {}, for example nixos modules are in nixpkgs, but not the “pkgs” output.

I would propose that:

• all-packages.nix is split out in to their respective folders.
• each folder should have a meta.maintainers definition.
• modules should also have a maintainers definition somewhere.
• the maintainer reviews PR for their respcetiev folder or delagates to trusted maintainers in the respective subfolder.
• A linter needs to check for missing maintainers list, many Linux kernels don’t have them for example.

One thing I do love about nixpkgs is just the share amount of pkgs available, it’s so much easier to just use overrideAttr to bump old packages than try and deal with the likes of aur. And I would hate if that changed tho.

[Infinisil]

• WHY IS all-packages.nix 41k LINES OF CODE??? WHY is it not organized based on the folder structure??? WHY NIL AND RNIX-LSP FREEZE EMACS??

I have some great news for you! https://github.com/NixOS/rfcs/pull/140

[TLATER]

That’s beautiful Hopefully eventually there’s also a solution to variants/arguments/aliases, since that’s what I find most of my reason to actually read all-packages.nix to be. A lot of hidden implementation lives in that file.

Still, this will be an incredible improvement, an 81% reduction sounds enough for editors and language servers to actually open the file.

[woobilicious]

I guess that changes the hierarchy of management I was suggesting, but I definitely support the improvements otherwise, I’m happily surprised that package.nix is being used, I was considering the same thing when I write my own ad-hoc stuff

[fzakaria]

I don’t think anything not mentioned in this thread already.
Nixpkgs seems to try and solve for everyone which makes getting changes in a bit harder via committee.
I’ve noticed the other alternatives (spack and guix) have a bit more of a BDFL approach which makes some large issues easier. (democracy vs dictatorship lol).

tbh, I just didn’t have the personal time to devote needed to steer it all the way through.
I am impressed with everyone who puts effort into the RFC, comments, comments, comments, issues and finally to the implementation.

[mounty]

Here. It was just too much trouble and I’m too busy.