Where did you get stuck in the Nix ecosystem? Tell me your story

I don’t have much interesting to add regarding the general nature of the Nix ecosystem Ugh field – I think it’s mostly well-grasped by now and is something in the intersection of incoherent overall vision, lack of resources, and unclear hierarchy.

I do observe a couple of things that are currently handled manually and look like they could be (mostly) solved by technical means:

  • It seems that most PRs are package bumps. Can a clear policy be formulated along the lines of “a bump for a leaf package with no patches, performed by the bundled update script (or by a human but that looks like it could be automated i.e. only increases the version string and changes the source hash), and which passes automation, Just Gets Effing Merged”? If not, why? I suppose this touches on the deeper question of “does Nixpkgs want to just package or does it want to curate too”, but the answer to that one should be pretty clear.

  • Regarding other aspects of contribution flow, what would be great is a way for PRs stuck in limbo to bubble up into reviewer/committer attention that does not involve prodding people via side channels (Matrix, Discord, DMs, etc.). The current situation is clearly miserable for contributors and must cause reviewer/committer burn-out.

12 Likes

Related to this: I really struggle with writing flakes because of how unopinionated the ecosystem is and how tricky it is to debug problems. Flexibility is a double edged sword:

  • If people can do anything it’s hard to have everyone agree on consistent patterns and it’s easy to get lost.
  • you can do anything so it’s very powerful and flexible.
1 Like

The Setting:

In this corner of the internet, we have a lot of Real Engineers™ who will learn a new language if its the right tool for the job. Anyone else would never stick with nix long enough to see the light. So that’s like, a bias we have. Most days, I consider myself a Real Engineer.

But there’s a whole world out there (most notably Science, Finance, and “Data” if that can be called a sector–where I work we think so). A world where code is not king, but instead is just a minor noble. No matter how good it would be for them, they’re never going to learn Haskell. For better or worse, Python is perceived as “good enough”. And you know, it used to be Microsoft Excel that was “good enough”, so perhaps we should be proud of them.

Python is home for me. Its packaging system is an imperative mess, but I still like writing it. Also, I’m a bit tired of writing code that solves code problems which solves code problems which… I want to be nearer a real problem. So my current focus is to find a way to be a bridge between these worlds. How can I help the people for whom Python is good enough, particularly ones that are solving Real Problems™?

Finally, one last bit of context: the threat landscape. The bad guys are getting better and better at being bad. The good guys are getting better too, but not as quickly. That imperative mess that is the python packaging landscape is a great big rug under which people sweep packaging complexity and never look at it again. It’s full of opportunities to bundle malware into otherwise useful packages and not get caught for a good long time.

Us Real Engineers, we’ll be fine. But the others… they brought an umbrella when they needed a tent. The mosquitos are coming and they’re going to be eaten alive–I want to help them. So that’s me (before I got stuck). Perhaps arrogantly, I was going to be the hero that Python needed. I was going to learn nix so well that I could build a bridge for them to someplace safer. (That’s also how I get motivated, if I can’t convince myself that I’m saving the world, I have a hard time getting started.)

The Stuck Part:

I convinced my colleagues to let me maintain a flake.nix in my work repo–it’s not like they have to use it. And for a while it was going well. I found bugs involving dependency on different versions of grep, and other such things. I was making a good case for nix. People would encounter problems that I didn’t have because they invoked homebrew in the wrong way or whatever.

But I never got that flake to look good enough to present to the company in a “we should do this everywhere” kind of way. Two things prevent it:

A seasoned nix user would just avoid these tools, but the people I’m trying to convince that they need nix… they’re not seasoned nix users. If I want to convince them, I need to make these two work.

The Cherry on Top:

While attempting to fix the second bug listed above, I reached out to the package maintainer and asked for some guidance. The way I saw it, I was helping bring his work to a wider audience, I was helping him out. He politely said:

No. I don’t recommend using nix.

I’m not trying to call him out here. He was well within his rights to say no. I can’t fault him for disagreeing with me about how cool nix is. That’s fine. But where I was in my journey, it sounded like:

we don’t want to be saved, thanks

He’s not exactly Guido Van Rossum, but he’s a stronger voice in the Python community than I am. His package is incredibly popular. Maybe he’s right, maybe I’m barking up the wrong tree.

Plus it’s kind of eerie that I broke this incredibly popular package in nixpkgs and haven’t gotten much noise on that issue. It’s got me feeling like I’ve been fighting a lost cause. I’m still learning nix so I can apply it in my own endeavors, but the interaction has sort of took the wind out of my sails when it comes to making a difference in the community.

14 Likes

Joachim Schiele is one of the people I see as ‘Heros’.
Kind of like the French resistance…

https://nixcloud.io/main/en/

1 Like

I’ve almost given up a dozen times due to cryptic backtraces and difficulty in tracing expressions.

13 Likes

Honestly, it’s the nixos and nixpkgs manual. Frankly, I’m going to sound like many of the other people who started out with nix, hit a wall called documentation, and had to climb it: the documentation is neither here nor there.

More specifically, for devs / package maintainers these are my major gripes:

documentation generation from code

doxygen, rustdoc, javadoc, sphinx, docco, and other tools exist, but nothing has been made of the same caliber. Parts of the nixpkgs documention is generated from code, but I always have trouble finding things.
Want a quick reminder of the options available to you in mkDerivation? “Ctrl+F” is your only friend and you have to hit F3 a few times to get somewhere and then manually scroll to places you’re looking for.

too many unspoken rules for nixpkgs PRs

  • barely any linting → “Please remove this newline”, “this function doesn’t use the proper case”
  • fields in mkDerivation.meta have minimal descriptions and it isn’t clear when to use the optional fields
  • package naming conventions aren’t clear. fresh, still, wrapped, unwrapped, bin … what?

I’m missing more, but it’s a pity that PRs get stuck because they have to go through multiple rounds issues that could’ve been checked by a linter.


As a user:

should this be an option?

When I was a newbie, the fact that options existed and simply installed nginx wouldn’t start an nginx service was completely unclear. That’s a simple case though and nowadays I still run into the problem of not know whether a package is actually managed by an option. Some options have a package field which defaults to some package, but there are known, unlisted alternatives which aren’t found by the search.

It would be great to have a warning or field generated in search.nixos.org like usedInOptions.

package descriptions can be confusing

See libreoffice. Yes there’s a description but it’s the same for about 8 packages and I still don’t know which one to install. I went with libreoffice-qt, but I have no idea if that’s the right one. It works, yes, but am I missing something?

This happened a few times and for newbies is probably more frustrating as they most likely don’t want to click on “source”, find out it’s a function that spits out mkDerivation depending on certain input, trace where that input comes from and hope to stumble upon a comment explaining why this input what chosen or not.

the nixos website is unclear

Made an issue for it that hasn’t moved for 2 years
https://github.com/NixOS/nixos-homepage/issues/734

Oh boy… I’ve been using nixos for 2 years already :open_mouth:

11 Likes

One thing for me as a beginner is/was the documentation. I think this kind of stems from Nix(OS) being so fundamentally different to most other distros. I have tried to set up Neovim and/or Intellij for Rust development and just failed, because I didn’t even know where to start (there were different problems for both problems). Which in turn made me switch back to another distro. I am definitely willing to give it another try, but Nix just seems really daunting to get into.

Parts of the nixpkgs documention is generated from code, but I always have trouble finding things.

There is an RFC about it:

https://github.com/NixOS/rfcs/pull/145

too many unspoken rules for nixpkgs PRs

I am writing a FAQ about my personal preferences. But many of them were found by routinely bumping on them.

barely any linting → “Please remove this newline”, “this function doesn’t use the proper case”

Another RFC:

https://github.com/NixOS/rfcs/pull/101

fields in mkDerivation.meta have minimal descriptions and it isn’t clear when to use the optional fields

How precisely you say the descriptions are minimal?
https://ryantm.github.io/nixpkgs/stdenv/meta/

Also, you can just ignore optional fields if you are in doubt.

package naming conventions aren’t clear. fresh, still, wrapped, unwrapped, bin … what?

Wrapped packages are there because many packages are not so useful without being wrapped first. E.g. X16 emulator needs a ROM that is distributed separately. The wrapper puts both together.

-bin is because sometimes we provide both a source-based package and a binary one. Firefox is such an example.

Others are mostly because the upstream uses a similar naming.

Thanks everybody for participating in this survey. This gives me good qualitative data to work with. And if you see other areas that haven´t been talked about, please feel free to pile on, or send me DMs.

And if you want to start discussing some of the points, I would encourage creating new threads that reference this one.

11 Likes

Start with Ryan … GitHub - ryan4yin/nixos-and-flakes-book: :hammer_and_wrench: Want to know…

Brilliant man.
I am using Helix to get some rubber on the road

After finding a vulnerability in SUID wrappers, I wanted to make them simpler so that a similar thing cannot happen again by construction. Alas, it’s unclear whether being able to build a new SUID wrapper without access to the C++ toolchain is a requirement (and, if so, what’s the exact evaluatable shape of that requirement). The PR is sitting in that state for a year now. I probably could have been more engaging with everyone, but this is exactly the kind of interpersonal interaction I dislike.

2 Likes

It strikes me as incredibly weird that given how SMALL the Nix community is, bureaucracy is one of the main complaints. That should be a big warning sign that there’s cultural and process issues.

I can co-sign a lot of what has already been said. Every now and then I’ll have some idea or ask why something “can’t” be done, and the answer is very frequently :man_shrugging: or “that would never get through review” or “idk who would review it.”

9 Likes

You’re probably way further ahead than this video, but I thought I should mention it:

“Nix derivations explained” - Vimjoyer

3 Likes

Ok i am coming late but eh. Time off matters. Multiple things have blocked me and still block me. Some technical, some less, in general a mix of both because it is hard to separate in real systems

  1. Impossibility to find someone that understand wtf the beamPackages wrapper do. This has limited a lot of progress over cleaning up how nixpkgs handle all the Erlang stuff

  2. No api to hook existing language package managers to nix, who already maintain a hash validated store. No x2nix are no answer there, except if shipped with the tool, because the x2nix need to follow the tools versionning for their internal private behavior. And if we want package managers to ship an adapter it means they need a spec to follow

  3. Got stumped doing proper bootstraped cross compiling of otp/erlang with nixpkgs. Could not understand how the splicing was failing and every time i asked in public, noone seemed to have any idea how this worked

  4. Handling filesystems and mount points. This is honestly not nix specific but a general problem i have with linux. How the heck does any of this kess make sense to anyone. Where is the documentation that makes sense…

  5. Being able to setup a way to deploy config change to machine without needing a pet machine with a stateful file system. Couldn’t find a tool to do it, so we dropped using nix to deploy things. The only way we found was a channel and ansible to trigger a channel update. Flakes do not help here.

  6. Never figured out how to make an overlay work.

  7. The tutorial and docs are mostly useless. I am not building a package in the wild in general. Or i update something existing deeeeep in nixpkgs. Or i am writing a shell.nix that need to work cross platform to support a developer environment

  8. Afaict glibc Unicode archive are still broken in nixpkgs with Erlang, but i never figured out wtf happened. I just have a magical line to add and it works.

  9. Finding options and libraries function in the docs, both for nixpkgs and home manager. They desperately need some ToC with collapseable header, more organisation and section description. See the elixir stdlib documentation for ideas. Do not provide a search. To search i need to know what to search. A structured doc allow me to discover what is needed.

  10. Flakes and the new cli. I honestly try really hard to not touch them. Everytime i end up, by mistake, using some of it, i lose a day of clean up. Flakes are not cross platform ruining the main interest for devenv. The new commands tend to not do what i want, generating backward incompatible change which wreck the main interest of nix, aka rollbacks. It also use languages that i have no idea what it means. And why the heck would i want lockfiles which block me from getting automated security patches? If this stuff handled semver maybe, but that is not how nixpkgs work. So we get none of the benefit of a lockfile and all the inconveniences.

  11. A feeling that my usecase is totally orthogonal to the one of the core team and people designing nix and nixpkgs. Some of the things coming out of @domenkozar work shows he seems to work in that direction, but all the rest of the community seems to pull for other use cases. Which is fair! But it makes it hard to find places and ways to express my needs. I simply do not see where and how to express them.

This is by definition a rather bleak list. Don’t be fooled. I still use nix in all my projects. My machines are home-managed. I think the path Nix offers is a ray of hope for the future. But i am starting to think it will need a completely new tools that learned from it, as Nix will not be able to get us far further on that path

10 Likes
  • I took a long time adapting to flakes. It was a big leap, the official docs took a long time, and still I can’t find docs for many of the options I need.
  • Finding builtin and standard library functions. I used to full-text search through the corresponding parts of the nixpkgs and nix manuals, often ending up simply not completing my task because it was so complicated to find the right function. Luckily there is noogle.dev now.
  • Some big packages cannot be properly built in nixpkgs, see e.g. https://github.com/NixOS/nixpkgs/pull/152626 and related PRs for many roadblocks we encountered while trying to build cockroach from source. We tried many times with different contributors, getting stuck at different points. I even went to an in-person meetup to find people to help us with that, but we couldn’t make progress.
  • In the past I found it hard to get my PRs merged. I discovered PRs ready for review _ and related threads, and at some time even got my own commit bit, but I think it’s still hard for others.
  • I paused reviewing PRs on nixpkgs for the time being, because they are too many. Yes, that’s counterintuitive, but that’s how it is. Half of the PRs are minor version bumps which I consider, frankly, a waste of my time to review, because I don’t know what I can do with them other than simply merging if all ofborg jobs are green. A bot could do that so why not automatise it instead. The other half of the PRs (the manually created ones) are nearly always on a topic that I don’t understand, so I’m hoping that someone more knowledgeable will come around to reviewing them. The tiny fraction of PRs where I know my way around are surprisingly hard to find. I used to dig for those, but for one where I could add value there were 20 to click through where I couldn’t, which was time and energy consuming, so I stopped.
  • Surprisingly often, nix-related tools (see e.g. https://github.com/ryantm/nixpkgs-update/issues/265) are not packaged in nixpkgs. This may have diverse technical reasons though.
  • The “green hydra debate” is not resolved to many people’s satisfaction. I would like to see much more automation in nixpkgs, burdening humans only with interesting nontrivial questions. (I.e. let minor package bumps be merged automatically.) But every time someone proposes a change in that direction it seems like it is being shouted down very quickly.
  • We’re using nix at work to provide a dev environment on some laptops. There are a lot of pitfalls for beginners. Usually if something doesn’t work, nix/nixpkgs is always one of the primary suspects. This makes any kind of larger adoption in our team hard.
12 Likes

although I’m not a NixOS developer, and answering this from a user-perspective, please: make flake official.

It is experimental yet the Internet is full of references to it. But the official docs do not because it is disabled in NixOS stable. Should I go for the old way (nix-channel, nix-env, shell.nix default.nix) or the new way (flake.nix). And what does experimental mean anyway, that flake can disappear any moment because it is just an experiment?

My plea, please stop the limbo, adopt flake as the way to go, kill off nix-channel, update the official NixOS stable docs as if the old way has never existed. Flake all the way!

3 Likes

Definitely feeling this too. Using nix / nixpkgs without NixOS / nix-darwin has all sorts of issues on both linux (opengl, x11, targetPlatform has to be NixOS) and darwin (cacert), but there’s no clear vision / roadmap around that use case, so it’s hard to understand which things were never meant to work, happen to work by coincidence, are broken but are supposed to work, and it’s hard to have meaningful discussions on how nix / nixpkgs should evolve to support this use case.

Wow this website is so helpful! Thanks for sharing, I wonder whether the Nix ecosystem will agree at some point to point to it instead of the official docs.

Being someone who has issues with discerning between useful work and waste of time, my point of view on Nix and NixOS is, at most, unreliable, as what I consider small hiccups are unbearable by most Real Engineers™, as @MattRixman calls them.

Therefore, I will say what an acquaintance of mine has said on the matter. He tried NixOS out because it was one of the few distros where a certain package was said to be working without major hindrances.

Although I spent hours reading various docs, I lost enthusiasm

And that’s a point that was brought up by many others, here.

Then I came to the conclusion that a declarative build is cool, but with too much overhead to manage

Yes, because we lack any sort of tooling that actually eases writing Nix, and the API documentation for NixOS is lacking at best. In addition, Nix as a language is kind of ugly, don’t you all think? I wish a working group existed to work just on making Nix a better language, as it definitely has room for improvement.

Home-manager and flakes explained like ****

And here we go again. Flakes need to happen, it seems that they are stagnating while absorbing huge amounts of developer energies. Maybe they don’t have to be perfect from day 1?

I lost it and now I’m playing ******* on Windows
Moral: Windows works and I’m tired of pretending it doesn’t

And that, folks, is how we lost not only a NixOS user, but a Linux user altogether.

I would like to add that we act as if everyone who uses Nix is either also part of the people who open issues or PRs on the Nixpkgs repo, browse this forum, or chat with us over at Matrix. That’s not true. Most of the people simply cannot be bothered. Some of us seem to be looking at Guix with admiration, but let’s not forget that that, too, is an incredibly niche system, especially if you factor in the complicated relationship that mainstream computer people seem to have with the foundation behind them.

So, where is our strength? What is it that we do better than our competitors? How do we march on to become the declarative package manager/distro of the future? Is there any exclusive functionality inherent to the way Nix handles things on which we can capitalize?

2 Likes

I thinked back and forth about shall I write or shall I not, and at the end I decided to just dump some random experiences and thoughts here.

  1. Even though I got the commit bit, there was basically no introduction about what the rules are. After having asked in the matrix, it boiled down to “apply common sense”
  2. I want to review and/or merge much more often, though whenever I decide to do some reviews I pick random ones from the first GH page and look at them. Any non-trivial PR is already commented on, and I do not have anything to add or am completely opposite opinion. Quite often I just let minor formatting issues slip, as I do not consider the PR the place to discuss that, especially as there is still no formally written formatting standard to apply.
  3. In general there is to much opinion and to little reasoning in reviews
  4. there are opposing standards between different sub-ecosystems (can’t give an example, as this is mostly hearsay)
  5. The GH repository is huge… To be able to contribute an initial clone of 3+ GiB is required, leading more and more poeple asking for a way to contribute via GH-Web-UI, which doesn’t give the required amount of control for manipulating the history as required by the contribution guide.
  6. There are PRs open for many years, that do not have any chance for getting merged without a major conflict handling overhaul or might even been partially or completely superseeded by other PRs already merged.
15 Likes