Where did you get stuck in the Nix ecosystem? Tell me your story

Development
41

Parts of the nixpkgs documention is generated from code, but I always have trouble finding things.

There is an RFC about it:

too many unspoken rules for nixpkgs PRs

I am writing a FAQ about my personal preferences. But many of them were found by routinely bumping on them.

barely any linting → “Please remove this newline”, “this function doesn’t use the proper case”

Another RFC:

fields in mkDerivation.meta have minimal descriptions and it isn’t clear when to use the optional fields

How precisely you say the descriptions are minimal?
https://ryantm.github.io/nixpkgs/stdenv/meta/

Also, you can just ignore optional fields if you are in doubt.

package naming conventions aren’t clear. fresh, still, wrapped, unwrapped, bin … what?

Wrapped packages are there because many packages are not so useful without being wrapped first. E.g. X16 emulator needs a ROM that is distributed separately. The wrapper puts both together.

-bin is because sometimes we provide both a source-based package and a binary one. Firefox is such an example.

Others are mostly because the upstream uses a similar naming.

42

Thanks everybody for participating in this survey. This gives me good qualitative data to work with. And if you see other areas that haven´t been talked about, please feel free to pile on, or send me DMs.

And if you want to start discussing some of the points, I would encourage creating new threads that reference this one.

11 Likes
43

Start with Ryan … GitHub - ryan4yin/nixos-and-flakes-book: :hammer_and_wrench: Want to know…

Brilliant man.
I am using Helix to get some rubber on the road

44

After finding a vulnerability in SUID wrappers, I wanted to make them simpler so that a similar thing cannot happen again by construction. Alas, it’s unclear whether being able to build a new SUID wrapper without access to the C++ toolchain is a requirement (and, if so, what’s the exact evaluatable shape of that requirement). The PR is sitting in that state for a year now. I probably could have been more engaging with everyone, but this is exactly the kind of interpersonal interaction I dislike.

2 Likes
45

It strikes me as incredibly weird that given how SMALL the Nix community is, bureaucracy is one of the main complaints. That should be a big warning sign that there’s cultural and process issues.

I can co-sign a lot of what has already been said. Every now and then I’ll have some idea or ask why something “can’t” be done, and the answer is very frequently :man_shrugging: or “that would never get through review” or “idk who would review it.”

9 Likes
46

You’re probably way further ahead than this video, but I thought I should mention it:

“Nix derivations explained” - Vimjoyer

3 Likes
47

Ok i am coming late but eh. Time off matters. Multiple things have blocked me and still block me. Some technical, some less, in general a mix of both because it is hard to separate in real systems

  1. Impossibility to find someone that understand wtf the beamPackages wrapper do. This has limited a lot of progress over cleaning up how nixpkgs handle all the Erlang stuff

  2. No api to hook existing language package managers to nix, who already maintain a hash validated store. No x2nix are no answer there, except if shipped with the tool, because the x2nix need to follow the tools versionning for their internal private behavior. And if we want package managers to ship an adapter it means they need a spec to follow

  3. Got stumped doing proper bootstraped cross compiling of otp/erlang with nixpkgs. Could not understand how the splicing was failing and every time i asked in public, noone seemed to have any idea how this worked

  4. Handling filesystems and mount points. This is honestly not nix specific but a general problem i have with linux. How the heck does any of this kess make sense to anyone. Where is the documentation that makes sense…

  5. Being able to setup a way to deploy config change to machine without needing a pet machine with a stateful file system. Couldn’t find a tool to do it, so we dropped using nix to deploy things. The only way we found was a channel and ansible to trigger a channel update. Flakes do not help here.

  6. Never figured out how to make an overlay work.

  7. The tutorial and docs are mostly useless. I am not building a package in the wild in general. Or i update something existing deeeeep in nixpkgs. Or i am writing a shell.nix that need to work cross platform to support a developer environment

  8. Afaict glibc Unicode archive are still broken in nixpkgs with Erlang, but i never figured out wtf happened. I just have a magical line to add and it works.

  9. Finding options and libraries function in the docs, both for nixpkgs and home manager. They desperately need some ToC with collapseable header, more organisation and section description. See the elixir stdlib documentation for ideas. Do not provide a search. To search i need to know what to search. A structured doc allow me to discover what is needed.

  10. Flakes and the new cli. I honestly try really hard to not touch them. Everytime i end up, by mistake, using some of it, i lose a day of clean up. Flakes are not cross platform ruining the main interest for devenv. The new commands tend to not do what i want, generating backward incompatible change which wreck the main interest of nix, aka rollbacks. It also use languages that i have no idea what it means. And why the heck would i want lockfiles which block me from getting automated security patches? If this stuff handled semver maybe, but that is not how nixpkgs work. So we get none of the benefit of a lockfile and all the inconveniences.

  11. A feeling that my usecase is totally orthogonal to the one of the core team and people designing nix and nixpkgs. Some of the things coming out of @domenkozar work shows he seems to work in that direction, but all the rest of the community seems to pull for other use cases. Which is fair! But it makes it hard to find places and ways to express my needs. I simply do not see where and how to express them.

This is by definition a rather bleak list. Don’t be fooled. I still use nix in all my projects. My machines are home-managed. I think the path Nix offers is a ray of hope for the future. But i am starting to think it will need a completely new tools that learned from it, as Nix will not be able to get us far further on that path

10 Likes
48
  • I took a long time adapting to flakes. It was a big leap, the official docs took a long time, and still I can’t find docs for many of the options I need.
  • Finding builtin and standard library functions. I used to full-text search through the corresponding parts of the nixpkgs and nix manuals, often ending up simply not completing my task because it was so complicated to find the right function. Luckily there is noogle.dev now.
  • Some big packages cannot be properly built in nixpkgs, see e.g. https://github.com/NixOS/nixpkgs/pull/152626 and related PRs for many roadblocks we encountered while trying to build cockroach from source. We tried many times with different contributors, getting stuck at different points. I even went to an in-person meetup to find people to help us with that, but we couldn’t make progress.
  • In the past I found it hard to get my PRs merged. I discovered PRs ready for review _ and related threads, and at some time even got my own commit bit, but I think it’s still hard for others.
  • I paused reviewing PRs on nixpkgs for the time being, because they are too many. Yes, that’s counterintuitive, but that’s how it is. Half of the PRs are minor version bumps which I consider, frankly, a waste of my time to review, because I don’t know what I can do with them other than simply merging if all ofborg jobs are green. A bot could do that so why not automatise it instead. The other half of the PRs (the manually created ones) are nearly always on a topic that I don’t understand, so I’m hoping that someone more knowledgeable will come around to reviewing them. The tiny fraction of PRs where I know my way around are surprisingly hard to find. I used to dig for those, but for one where I could add value there were 20 to click through where I couldn’t, which was time and energy consuming, so I stopped.
  • Surprisingly often, nix-related tools (see e.g. https://github.com/ryantm/nixpkgs-update/issues/265) are not packaged in nixpkgs. This may have diverse technical reasons though.
  • The “green hydra debate” is not resolved to many people’s satisfaction. I would like to see much more automation in nixpkgs, burdening humans only with interesting nontrivial questions. (I.e. let minor package bumps be merged automatically.) But every time someone proposes a change in that direction it seems like it is being shouted down very quickly.
  • We’re using nix at work to provide a dev environment on some laptops. There are a lot of pitfalls for beginners. Usually if something doesn’t work, nix/nixpkgs is always one of the primary suspects. This makes any kind of larger adoption in our team hard.
12 Likes
49

although I’m not a NixOS developer, and answering this from a user-perspective, please: make flake official.

It is experimental yet the Internet is full of references to it. But the official docs do not because it is disabled in NixOS stable. Should I go for the old way (nix-channel, nix-env, shell.nix default.nix) or the new way (flake.nix). And what does experimental mean anyway, that flake can disappear any moment because it is just an experiment?

My plea, please stop the limbo, adopt flake as the way to go, kill off nix-channel, update the official NixOS stable docs as if the old way has never existed. Flake all the way!

3 Likes
50

Definitely feeling this too. Using nix / nixpkgs without NixOS / nix-darwin has all sorts of issues on both linux (opengl, x11, targetPlatform has to be NixOS) and darwin (cacert), but there’s no clear vision / roadmap around that use case, so it’s hard to understand which things were never meant to work, happen to work by coincidence, are broken but are supposed to work, and it’s hard to have meaningful discussions on how nix / nixpkgs should evolve to support this use case.

51

Wow this website is so helpful! Thanks for sharing, I wonder whether the Nix ecosystem will agree at some point to point to it instead of the official docs.

52

Being someone who has issues with discerning between useful work and waste of time, my point of view on Nix and NixOS is, at most, unreliable, as what I consider small hiccups are unbearable by most Real Engineers™, as @MattRixman calls them.

Therefore, I will say what an acquaintance of mine has said on the matter. He tried NixOS out because it was one of the few distros where a certain package was said to be working without major hindrances.

Although I spent hours reading various docs, I lost enthusiasm

And that’s a point that was brought up by many others, here.

Then I came to the conclusion that a declarative build is cool, but with too much overhead to manage

Yes, because we lack any sort of tooling that actually eases writing Nix, and the API documentation for NixOS is lacking at best. In addition, Nix as a language is kind of ugly, don’t you all think? I wish a working group existed to work just on making Nix a better language, as it definitely has room for improvement.

Home-manager and flakes explained like ****

And here we go again. Flakes need to happen, it seems that they are stagnating while absorbing huge amounts of developer energies. Maybe they don’t have to be perfect from day 1?

I lost it and now I’m playing ******* on Windows
Moral: Windows works and I’m tired of pretending it doesn’t

And that, folks, is how we lost not only a NixOS user, but a Linux user altogether.

I would like to add that we act as if everyone who uses Nix is either also part of the people who open issues or PRs on the Nixpkgs repo, browse this forum, or chat with us over at Matrix. That’s not true. Most of the people simply cannot be bothered. Some of us seem to be looking at Guix with admiration, but let’s not forget that that, too, is an incredibly niche system, especially if you factor in the complicated relationship that mainstream computer people seem to have with the foundation behind them.

So, where is our strength? What is it that we do better than our competitors? How do we march on to become the declarative package manager/distro of the future? Is there any exclusive functionality inherent to the way Nix handles things on which we can capitalize?

2 Likes
53

I thinked back and forth about shall I write or shall I not, and at the end I decided to just dump some random experiences and thoughts here.

  1. Even though I got the commit bit, there was basically no introduction about what the rules are. After having asked in the matrix, it boiled down to “apply common sense”
  2. I want to review and/or merge much more often, though whenever I decide to do some reviews I pick random ones from the first GH page and look at them. Any non-trivial PR is already commented on, and I do not have anything to add or am completely opposite opinion. Quite often I just let minor formatting issues slip, as I do not consider the PR the place to discuss that, especially as there is still no formally written formatting standard to apply.
  3. In general there is to much opinion and to little reasoning in reviews
  4. there are opposing standards between different sub-ecosystems (can’t give an example, as this is mostly hearsay)
  5. The GH repository is huge… To be able to contribute an initial clone of 3+ GiB is required, leading more and more poeple asking for a way to contribute via GH-Web-UI, which doesn’t give the required amount of control for manipulating the history as required by the contribution guide.
  6. There are PRs open for many years, that do not have any chance for getting merged without a major conflict handling overhaul or might even been partially or completely superseeded by other PRs already merged.
15 Likes
54

To be able to contribute an initial clone of 3+ GiB is required…

In fairness, git clone https://github.com/NixOS/nixpkgs --depth=1 right now results in pulling 44.26MiB over the network, for a total space on disk of about 198MiB. Which is still vast, but an order of magnitude less vast.

6 Likes
55

--depth=1 works on git fetch too, you can use it like
git fetch origin --depth=1 <some revision> to just download the rev you want without the history

3 Likes
56

Last time I tried, I was unable to create PRs from a shallow clone

3 Likes
57

I agree that the docs could be improved, but I don’t think that it will ever reach a point where it will be easier to manage Nix system in comparison to a regular distribution, because you are managing something much more complex, but with features that worth the efforts.

I am sure you are aware of these strengths (see also this thread). My point is that NixOS strengths come with a price, and if you are not ready to pay the price because you don’t appreciate the product, that’s OK :), that’s why it’s a niche system.

Maybe Python is less ugly, but have you ever read Guix’ packages’ source code? Nix is much easier to understand then Guile.

4 Likes
58

There is! It’s called Nickel, and nickel-nix is an experimental implementation that integrates it into Nix.

And while that is a welcome development, I can’t help but feel there is just too few people doing too many things. A lot of work in the Nix ecosystem is still conceptual, there is divided attention between the old and new CLI, way more tickets are being opened than completed, same for PRs.

And let’s not forget, almost all contributors to Nix offer consulting on how to integrate Nix as far as I understand, so the amount of stuff that has to be done is just completely overwhelming compared to the amount of stuff that can be done.

I mean just look at one of the core team’s weekly meeting notes, the issue tracker and the PR List;
8 Issues/PRs discussed. 21 Issues opened. 11 PRs opened. All within one week. Maybe I’m pessimistic, but it seems to me that there is a severe lack of manpower. Most issues aren’t even triaged, ever.

3 Likes
59

I dream of a future where I can specify which modules to include in my system via a two-columns menu-based UI that lets me select a flake -> module pair to add, and then present me with a graphical shell on which I can set all the parameters of the module (although in a simplified form without let bindings and ad-hoc functions).

But that’s a dream of mine.

There’s a reason why I’m stuck with NixOS since September 2020 and contribute stuff to Nixpkgs whenever I can. My point is that I am not sure we are selling ourselves well enough to the wider audience, considering that people nowadays are totally in love with containers and are more willing to take that as an approach to manage complexity and reproducibility.
Heck, then there is this friend of mine who writes Ansible scripts for a living, a ton of them, and I begin to wonder how that is seen as sustainable rather than writing Nix expressions to solve deployment and management problems in a more robust way.

I did a little, and, maybe due to personal inclinations, I find them much more clearer than Nix expressions. Maybe it’s because I prefer the syntactic simplicity of Lisp to the vaguely haskelly syntax of Nix with pragmatic sugar stuck on it (which is far from the clarity of Haskell code, in my opinion).
In addition, Guile is a proper programming language, made to express logic, while sometimes I find it difficult to do things programmatically in Nix because it lacks the facilities of other programming languages (maybe because I cannot remember properly what is in the standard and the Nixpkgs library).
Let’s not even talk about the fact that Nix is a completely new language, while Guile is a scheme, so you cannot apply much transfer learning and you have to teach people its syntax from scratch, and busy people are averse to learning new syntax.

I heard about Nickel as a generic configuration language, but I never looked into that, so I can’t express any opinion on its syntax. I’ll be checking it back as soon as it comes out of its experimental phase, as I am still not sure what its niche is, and how it solves the little syntax issues I find in Nix.

Absolutely, it’s a massive project that I want to see succeed and that’s why I really want to get more involved into its development.

I think there are also problems of efficiency here and there that must be addressed both at the technological and organizational level, but I am in no position to offer suggestions, as for me The Team feels like a very distant entity of which I know almost nothing about.

We know that the solution to scarce manpower is automation, and I am happy to see efforts going in that direction. I wonder what the strongest limiting factors are, though.

2 Likes
61

There are a lot of other ways to think about the question of who our competitors are. Depending on who you are, you might instead think our competitors are:

  • standalone package managers: guix, homebrew, pip, cargo, cabal, maven, choco…
  • config tools: ansible, puppet, saltstack…
  • operating sytems: debian, redhat, gentoo…

It’s hard to be so many different things to so many different people. You’re never going to make them all happy at once. But if you have so many different kinds of people participating in your ecosystem–that’s an advantage.

I think of it as analogous to Arch and Manjaro. Arch is where the wizards and warriors do battle with hordes of bugs from the bleeding edge, and Manjaro is where the newbies get to live in the peace paid for by that fight.

Every time I contribute to nixpkgs, I’m amazed by the volume of activity on that repo. I can set something down for a week and rebase thousands of commits when I pick it back up. It’s group coordination on a scale rarely seen.

I was a Ubuntu user for a decade, and in that time I built only a single .deb and didn’t publish it. I’ve been a NixOS user for two years and I’m already making (small, so far) contributions to nixpkgs, with a growing to-do list of more contributions because it’s just going to be easier to reference this stuff by name instead of pasting scripts through slack to my colleagues.

That, I think, is our superpower. Yes, it’s hard to configure your system with nix. But from there, you’re only a small step away from configuring anyone’s system with the same code.

Yeah, we lose users because we have such a thin wall between user and package maintainer. Let’s get better about that. But we also have a lot of package maintainers.

(disclaimer: I’m new here, I don’t think I speak for this community, these are just my impressions)

9 Likes