Marketing Team: Can we present Nix/NixOS better?

https://github.com/NixOS/rfcs/pull/74

I’m committed to evolving this idea, waiting currently for some feedback from @garbas & @samueldr. Or anyone who’s got a great idea how this idea can be put to use.

Or …

… I might just ask for becoming part of the marketing team :stuck_out_tongue_winking_eye:

For my part, I am mainly interested in the process of developing a system config from scratch. My current config works and is pretty solid, but it’s built up on a bunch of ideas and structures that aren’t mine. I stole useful looking stuff from a bunch of other configs and sort of just jammed it all together until it did the stuff I wanted. This got me over part of the learning curve, but it leaves me in the awkward position of basically understanding the components, but not really being able to explain the “why” of all those choices.

For instance, I’d love to replace my Emacs overlay with emacs-overlay or rework my Haskell overlay to use haskell.nix. I think I know enough to do the former with some thought, but I have no idea if it’s sensible or even possible to use haskell.nix for a system-wide GHC or if it’s really just for projects, and I don’t know where to look to determine that.

I decided to just start reading the manuals and the pills end to end and taking notes to start tying ideas together, but the primary target of the Nix and nixpkgs manuals definitely appears to be people writing packages. This is fine and sensible, but it leaves me a bit adrift in answering the question “how do I, starting from nothing, write a quality system config that I can version control?”

The extra sticky bit is that I primarily use Darwin, so most of the NixOS documentation that might go toward answering this question is of varying usefulness. Most blog posts are about how to package such and such type of software or a recapitulation of the “how to package GNU hello” exercise, which tells me little about how to actually configure my machine the way I want it.

All of this is to say, I would be extremely interested in seeing (and even helping to write) a “Nix book” that isn’t just a manual.

Example: This is a really slick config, with a really straightforward installation and support for multiple machines and all that cool stuff. But! I have no idea what the assumptions made in this config are, what structural choices are good or just interesting, how I could write my own config like this other than copying it and tweaking. It even installs nix-darwin as part of its setup, which would remove one more manual step from applying my config on a new machine.

What is the gap in my knowledge that lets me read this config, basically understand what it’s doing, but not be at all certain where to start to make my own?

Parts of these could (and perhaps should) be addressed in the documentation of the projects themselves, but some kind of unified “Configure your machine with Nix” or even just a “Nix config cookbook” would be huge for me (and, again, I’d be happy to help). Figuring out and documenting these patterns centrally may also encourage their conversion to modules, so that users can say “yes, I want to interact with x in this way, so I just import the module and set some variables” or “no, I want to use this differently, but now I have the mental tools to write my own module to manage this”.

From a consumer perspective, it’s also a whole lot easier for me to pitch Nix to my team if I can say “just read this book and you’ll know a bunch of things you need to know” rather than “go read the manual to learn the vocabulary, now go read these 15 blog posts, but also ignore about a third of that content because Nix and nixpkgs have improved a bunch in the last year”.

To draw an analogy, there are a handful of books on how to use any given Linux build system, and a dozen “how to use Linux to do stuff” books. On the other hand, there a dozen books worth of content on the Nix build system, but only a handful of non-source-code resources for “how to use Nix to do stuff” (where “stuff” is “anything but building packages”).

4 Likes

You should take a look at GitHub - LnL7/nix-darwin: nix modules for darwin which is the macos equivalent.

I agree, it would be nice to have more of a narrative with nix/nixpkgs/nixos.

I started doing some videos https://www.youtube.com/channel/UC-cY3DcYladGdFQWIKL90SQ
I was going to do a more polished (slides, final voice over, graphics, maybe transitions) video series, after the 20.09 release. Maybe have them as the “unofficial” NixOS series.

Yea… unfortunately a lot of the conventions for nixpkgs all exist in the nixpkgs repo, which means they are subject to change over time. Nix allows for a pretty large breadth of expressiveness, and this means that there’s usually no idiomatic way to do something. Certain aspects are “too painful to change” (e.g. stdenv usage), but almost everything else could be changed.

3 Likes

I do use nix-darwin, though it’s one of those things that, as you say, could use more of a narrative element to its docs. The videos look cool, though! Happy to have another resource to pitch to curious coworkers!

1 Like

Something that just came to my mind: When I started using NixOS I was always a little surprised that the releases always seemed to be delayed. It’s the same for NixOS 20.09. By end of September it was still undergoing quite some big fixes and people start wondering “when will it finally be ready?”
Don’t get me wrong here, I see how much effort actually goes into the stabilization phase where a lot of bugs are being fixed.

I’m just wondering: Should we maybe consider not naming the release by the date it was actually branched from, but instead give it a name of when it is actually being officially “released”?

I think the current naming scheme just gives the impression that we’re late with our promises when in fact the release managers were very well aware of the fact that stabilization will very likely take a good month.

4 Likes

I think it is a bit worse than that. As an example, I remember when I tried to compile some Mate apps, but I never understood the makeWrapper mechanics. It just segfaulted and I could do nothing to fix or understand it. My frustration can be summed up as Hey, I can install it from sources on Archlinux or even Slackbuilds but I can’t do the same in Nixpkgs! (and the answer is “because they install everything in /usr and everyone sees anyone, it makes the things way easier!”)

Even if the Nix language doesn’t impose limits on the creativity of our package maintainers, I think there could be such a thing as a “cookbook” for such annoyances.

I think using the YY.03 and YY.09 is fine. It’s just our preparation for the branchoff is awkward. In unstable essentially we want everything to be the latest, however, that’s not how a lot of ecosystems work. Most notably for this release: plasma.

We should really stabilize “core” packages around the ecosystems which support them. The systemd 246 bump systemd: 245.6 -> 246 by flokli · Pull Request #94354 · NixOS/nixpkgs · GitHub caused two pretty serious regressions for the 20.09 release: NixOS Tests broken in master and staging · Issue #96197 · NixOS/nixpkgs · GitHub and KDE "Switch User" menu item and lock screen button no longer present after systemd 246 update · Issue #98141 · NixOS/nixpkgs · GitHub. We should select the major blocking ecosystems (plasma, gnome) and work backwards in selecting their dependencies for qt, systemd, and other more “critical” packages. Since we’re shipping plasma 5.18.5, and gnome 3.36, both which were released early this year; we should have kept the version of systemd to something that would have been developed against. Not having to deal with the staging cycles around fixing systemd regressions would have saved 2-3 weeks.

We could “delay” the release to align with plasma and gnome schedules (release something in october), but this would give the plasma and gnome maintainers essentially no time to package and polish those ecosystems during stabilization.

For me personally, I took off quite a bit of time from work in September to help do ZHF and stabilize the release. And now that the schedule slipped into october, and I was expecting the release to have been cut, I don’t have as much free time to dedicate to nixpkgs. Also, I got burned out from reviewing PRs and fixing builds.

Anyway, this is getting off-topic. I’ll probably have a constructive thread on how we can align schedules of staging-next PRs to avoid cramming 500+ staging commits a few days before branch off Staging next by FRidh · Pull Request #97146 · NixOS/nixpkgs · GitHub Staging next by FRidh · Pull Request #96437 · NixOS/nixpkgs · GitHub Staging next by FRidh · Pull Request #96280 · NixOS/nixpkgs · GitHub

This post wasn’t meant to be disparaging against anyone. Everyone in the nixos community is here by choice, and I don’t think I’ve seen another community with such high caliber of technical skill and passion. I’m coming from a place where I want to see NixOS succeed, and I think being accountable for timelines is necessary for larger adoption. However, we kind of set ourselves up for failure in doing stable releases, and the release date always suffers. Like I said above, I’ll have a separate thread on improving the timeline for doing a release, so that we have reasonable expectations.

13 Likes

sounds somewhat similar to the goal of https://nix.dev/

Would it make sense to align with Ubuntu and Fedora releases (or perhaps a little later) and perhaps also sync the versions of major components to theirs for a release? That way we could benefit from fixes and testing done in their ecosystems as well.

1 Like

It may, but remember that Canonical and Redhat both have teams dedicated to releases. We can’t have the same expectation of hours that they can put behind their release process. Most of these distros only “ship” with one Desktop manager, meanwhile we support 17 Desktop managers, and 34 window managers. Arch experiences similar pain points, but they also don’t do stable releases, so there’s no broken expectations. Our advantages are the declarative configuration management of nix, and passion of individuals.

Another option would to be more “relaxed” about backporting changes from the latest plasma and gnome ecosystems. So we could have a release at the end of march/september and then those ecosystems would just get a DE bump a month in. However, there would be significant divergence between master and the release, and there may be a lot of non-obvious brokeness (e.g. plasma+kdeApplications+qt)

EDIT:
I also mentioned the drawback of delaying the release:

We could “delay” the release to align with plasma and gnome schedules (release something in october), but this would give the plasma and gnome maintainers essentially no time to package and polish those ecosystems during stabilization.

If we did go this route, we should probably push it back to november

Pehaps we could try to keep up with their development branches between releases. If we had those polished up a month before release, the actual release branches shouldn’t be too big of a change.

I would go the other way: rather than delaying NixOS, just delay the Plasma/Gnome upgrades to the next release. It doesn’t seem fair to NixOS release managers to have to scramble at the last minute to package and polish new software for stable release, compared to just having users wait another 6 months for a smooth desktop env update.

4 Likes

I think this is normally the case, really it was systemd+plasma which caused the issues. Unfortunately systemd has a huge api surface area, and in this case, a non-obvious assumption was broken.

The original udev issue could have been caught with more testing around staging-next. But the printer dialog issue couldn’t have.

I have a feeling (but I am not going to use releases anytime soon, so it is just a slightly curious observer point of view) that the way to release NixOS half a month to a month earlier is to do branch-off half a month earlier, with backports having to justify they are low enough impact from that point on.

I think it was the wrong call from our side to bump systemd.

I think perhaps a good rule of thumb for next time is to bump a systemd major release only once every two cycles. Instead of bumping it each release. Such that we have a bit more stable interface to talk to that doesn’t change every release. Most security fixes get backported to systemd-stable branches anyway. This way we have a bit of a more stable interface.

OTOH I’d like nixpkgs-unstable to be on a newer branch ASAP to actually find these kind of issues early.

Now the thing is; that unstable is what is used as the basis of the next release branch; whilst sometimes certain bumps just need a bit more baking in the oven because their API surface is large; and it is used by many packages.

Other distro’s like The container optimized OS | The Fedora Project have a next branch and a stable branch; and the next branch contains experimental changes that don’t end up in the stable branches per se. E.g. CoreOS now has systemd 245 on stable; but next already has 246; but this doesn’t mean per se that stable will have 246 for the next release.

Yeah, with systemd, it’s a bit more complicated.

systemd-stable is a nice way to receive (security) fixes and minor bumps, which I’m feeling generally okay to backport to NixOS stable too (if it’s only a minor bump) - but I’m actually not sure how many major versions back still receive these fixes - stable shouldn’t run on an already unsupported release.

In any case, there needs to be some place receiving major systemd bumps quickly, uncovering issues with higher-level tooling and potentially fixing any fallouts, and I feel like this is what only nixos-unstable can be - but this shouldn’t delay/risk stable releases.

Maybe we need a longer stabilization period / earlier branchoff from nixos-unstable, and be less afraid to revert things there if it looks like regressions can’t be fixed in time?

Also, it might make sense to align release schedules to those of other distros, so they also run into these issues during their stabilization periods - and we’re not the first ones running into these issues alone :wink:

4 Likes

I think there’s two main issues from coming from this discussion: lack of a good way to verify changes coming from staging/staging-next PRs, and our release period being awkwardly right before the releases of plasma and gnome.

I have broken this out into two different topics to avoid cluttering this thread with off-topic posts:

1 Like

Fine. Now we need an article explaining the raison d’être of wrapProgram’s :confused:

The best marketing I can think of would be native Windows support. On the 2020 Stack Overflow survey, developers are 45% on Windows compared to 27% on Mac and 25% on Linux. I realize tools like WSL are available, but those workarounds can be awkward.

In addition to the individual developers who would use Nix, I think upstream projects would be more likely to use Nix as their main packaging tool if it natively supported most of their users.

6 Likes

To play a bit of devils advocate here are some of my thoughts

Most Loved, Dreaded, and Wanted Platforms

Linux remains the most loved platform.
Linux: 76.9%
Windows 57.5%

Platforms

Linux and Windows maintain the top spots for most popular platforms, with over half of the respondents reporting that they have done development work with them this year.
Linux: 55.0%
Windows: 53.1%
(with the percentage more skewed towards Linux when selecting Professional Developers)


I’d be interested to know if the respondents took the question “Primary Operating Systems” as what OS do you use most and if they have a Windows Desktop they use for personal use and gaming etc and a Windows Laptop for family use and viewing media etc.
Or even if they just primarily use windows as their host OS but heavily develop within WSL or a VM, or if they build source for Linux & Docker etc then most of their packaging requirements would be within Linux/Unix like systems


Supporting more platforms would nice but it does require extra effort

There are a couple packages that cover windows or cygwin