Thoughts from a new NixOS user

Help
1

Hi all,

Apologies in advance if this is not the right category for this post. Feel free to move it.

Also please excuse any mistakes caused by my lack of knowledge.

I’m a first time NixOS user (just a week in) and I love it so far. I like tinkering with slightly experimental features (my previous OS was ArchLinux) so I’m currently using ‘unstable’ (nixpkgs-channels master branch).

I thought you may want to hear which parts of NixOS and it’s ecosystem work really well and what caused me a bit of trouble along the way (from a newcomer’s perspective).

So here it goes, a list of things that I really like about Nix:

  1. The manuals.
    I can’t stress this enough but the manuals (Nix, NixOS, Nixpkgs, all three of them) are really comprehensive. I thought that ArchLinux install guide was in-depth but this is really on the next level. The availability of the manual offline on install USB system is a big plus.
  2. Reproducible system set-up that can be (in principle) configured entirely in a single file.
    I guess I’m just re-stating one of the main points of NixOS but here’s why it’s important to me:
    My biggest pain point when managing my ArchLinux installation was the need to reconfigure everything whenever changing machines. Rarely it would be enough to just install the base system, reinstall the packages, copy over /etc and /home and call it done. Besides that I really hated the fact that 99% of that /etc I had no idea what it does. Maybe I’m a purist but what really annoys me is when a program installs a config file with some defaults that you can then modify, but you get stuck with old defaults if they change because your config file never gets overwritten. Nix doesn’t do that (at least for system configuration, I’ll get to user config later) which I really like because each deviation from default config line is explicitly stated by myself in /etc/nixos/configuration.nix (or included files).
  3. Atomic system upgrades and easy rollbacks (no need for huge backups of the whole system).
    This is another big issue I had (especially with my tendency to live on the edge). Many times I had to rescue my system from ArchLinux live USB stick after a system upgrade (I’ll keep that USB stick anyway since it is a good Linux rescue image overall). I even went to the point of having btrfs subvolumes regularly snapshotted and sent incrementally (using btrfs send) to a backup drive from which I could simply restore every subvolume. I had separate ones for /usr, /opt, /home/, and / (it was quite small as it mostly included /var (excluding /var/cache), /etc, and /root). I guess I’ll keep backing up /home/, /etc/nixos, and perhaps /var but now I can stop worrying about /usr which saves a good few hundreds of GBs of backup space. Haven’t gotten to setting it up yet though :frowning:.
  4. Per user profiles; being able to install any program as non-root without awkwardly forcing it to use ~/bin etc. and also being able to use different version of a program per-user and even separate environments (virtualenv anyone?).
  5. The nix package manager.
    I really like how the directory structure in /nix/ is really simple and understandable. No magic databases (sqlite is no magic), caches, multi-level hash directory structures (de/adbeef), etc. except for /nix/var/log/.
  6. Nix pills.
    I don’t like using software that I don’t understand how it works (at least at a high level) and without these it would be much harder to understand how exactly nix-env and nixos-rebuild work.

And here are a few things that I didn’t like:

  1. There’s still no official way to declaratively manage user packages and config.
    Go team! I really hope we can come up with something amazing in this space. For now I’m rolling something of my own to manage my user profile. Just to get more fluent in Nix.
  2. Umm, how big is the Nix community?
    It seems like the community is quite split:
    • Repology lists 991 maintainers, my guess is this covers the discussions that happen on GitHub pull requests.
    • Discourse seems to be the preferred(?) communication medium outside of pull requests, or at least it’s listed the highest on the community page. It has 765 total users, less than the number of maintainers (!).
    • There’s the IRC channel. To be honest I have never heard that name (“IRC”) ever in my life, only read about it and used it once. That doesn’t say anything about other users though.
    • Buried deep in the community page there’s a link to /r/NixOS with 3k subscribers. I don’t understand how is it more popular if it’s lower than discourse?
  3. The NixOS manual section 11.6. Wireless Networks would have been useful somewhere around chapter 2. Installing NixOS. Also manual could mention that it’s possible to store hashed psk in /etc/nixos/configuration.nix by using wpa-passphrase and pskRaw config field. I’ll update that and make a pull request when I have the time.
  4. The systemd/logind module has config for HandleLidSwitch and HandleLidSwitchDocked but not HandleLidSwitchExternalPower. Yeah. I know this is a minor thing but I had to use extraConfig :face_vomiting:. I’ll add support for this and make a pull request when I have the time.
  5. I really wanted to have global /bin/bash available (sorry… :cry:) and there was no option to do that. In fact /bin/sh and /usr/bin/env (which are the only globally installed binaries) share no common implementation of that installation process (even though it’s almost identical: /bin/sh, /usr/bin/env). There was no system.extraGlobalBinaries to help, I had to copy-paste the install script. Let me know if it would be useful for someone else, then we can discuss if it’s worth having such a config available.
Nitpicks

Finally a bunch of nitpicks:

  1. The links in the Table Of Contents of NixOS manual all point to https://nixos.org/nixos/manual/index.html#<section_name> even if you loaded https://nixos.org/nixos/manual/, this causes a reload of a page when navigating to such link. Other manuals don’t have that problem.
  2. The nix-* and nixos-* commands should really be merged into single commands (nix and nixos) or exploded into separate commands for each subcommand. For consistency. I totally understand that they are this way due to history but it is possible to deprecate the old way and remove it when most users have upgraded.
  3. The above applies to builtins in nix language too. Most of them are builtins.* but there’s a bunch of names that are in the global namespace. Some of them make sense to be global (like abort) but some of them really stick out like a sore thumb (baseNameOf, dirOf, fetchTarball).

Don’t worry about these though, they are nitpicks after all…

17 Likes
2

Thanks @grahamc :grin:

image
image1046×183 18.6 KB

3 Likes
3

Thanks for this post; it’s great to see the perspective of a new user.

Our Discourse instance was started in April 2018, and it wasn’t until August 2018 that it officially replaced the mailing list. It’s a lot easier to subscribe to a sub-Reddit than to sign up for a specific Discourse instance.

4

What a nice thread.

Having been used Nixos for more than a year now, I can second most of your points. My background is not in understanding Linuxes and I won’t claim I understand how an operating system works, but here’s why Nixos has been great for me:

I’m a statistician by trade, and I mostly use computers to program something applied mathematics. Previously I have used Ubuntu for this. I feel that because nothing is ever quite finished in the Linux world, some really obscure tools I have to use sometimes require quite complex setup. Most troubles I encounter are solved by executing some bash code lines some stranger tells on Stackoverflow, hoping that my issue is the same as in the question. These little tinkerings add up and in the end I’ve got no clue as what I have done to my machine. Weird bugs end up happening. The worst was when months after a successful CUDA installation my graphics driver just stopped working. Not understanding how Linuxes work, it was quite painful trying to solve it when the whole computer crashed before login screen. I ended up installing my / again as I had my /home in a different partition.

Nixos is – in principle – a collection of working installation scripts. I don’t have to worry about those much, they just work. When I switched to Nixos little more than a year ago, I was warned that it would require tinkering, but in my experience, Ubuntu required it more. Sure, some things are not packaged, but the packaging is so dead simple, that I’ve managed to do some myself.

Now for the cons.

I still have trouble understanding what will I find in the Nix manual, and what in the Nixpkgs manual. When I try to make my own Nix expressions, some things are in Nix manual, and some in Nixpkgs manual.

The manuals are comprehensive, but I’d like to have a proper API documentation (akin to Haddocks or Javadocs) for all Nix functions commonly used in packaging. It’s really hard to find a complete specification on what miracles mkDerivation does under the hood. Also, it’s hard to even find that some handy functions exist in the first place.

5 Likes
5

Thanks a lot for taking time and effort to write down your first impression — it’s valuable information that is usually lost (memories about the first impression quickly start evolving)!

Atomic system upgrades and easy rollbacks (no need for huge backups of the whole system).

Please note that there are ways to change the system in a way that is not trivial to rollback. And atomicity is a goal but there are exceptions where full switch needs a reboot for an unexpected reason.

Of course, in the worst case your fall back to the previous point —

Reproducible system set-up

(but if you ever actually doubt whether NixOS can rollback something rare and global, chances are — manual work will be needed)

multi-level hash directory structures

That may get worse with time, as scaling becomes more pressing than keeping the same formats. Logs were just cheapest to change. Of course, humans are not expecting to navigate these manually except when debugging something unusual — multilevel trees are understandable, just annoying, and something like nix log should navigate them for you.

There’s still no official way to declaratively manage user packages and config.
come up with something amazing

We have, a few times too many times. The problem is of choice… for some ways, even integration with top-level system-wide configuration is loadable.

how big is the Nix community

Note that all the measures you list are about different things, too.

There are Nixpkgs committers (I think between 70 and 150 now, depending on how you filter for activity), there are people listed as maintainers for at least one package, there are people with at least one accepted patch/commit.

Discourse seems to be used for project-wide discussions and user support (not always for easy questions, though) and user support becoming a project-wide policy discussion (which means that for some single-package maintainers it is quite natural to not feel a pressing need to subscribe).

IRC… I think the #nixos channel/chat/room is now mostly user support, #nixos-dev is development discussions and then there are a few more niche channels. If you heard of Matrix you can use the bridge, and I think there is also a separate non-bridged Matrix-only room… yeah, our IM chat options are also fragmented.

Reddit might have the largest user-to-developer ratio (and possibly lower developer count).

Yes, we strive to be better than others in technical consistency, but community structure consistency is not something we think much about. Yet?

has config for … but not

This is a constant discussion — should we try to cover everything by verified options? Should leaf options be added via extraConfig? (There are evaluation performance costs for everyone if we cover every possible option of every module…)

I really wanted to have global /bin/bash available

Argh, #!/usr/bin/env bash is the only true way (it works on FreeBSD, too).

If you really want /bin/bash with the current system structure you can just ln -s /run/current-system/sw/bin/bash /bin/bash once, no? If you are leaving one true way anyway…

The nix-* and nixos-* commands should really be merged into single commands

The development of a consistent UI for nix command is still ongoing. It still cannot fully replace the old nix-* stuff for some goals. Once it has all the necessary functionality, what you say becomes more realistic.

builtins

And then you can access builtins via __-prefixing of the name.

6

ilikeavocadoes via Nix community nixos1@discoursemail.com writes:

What a nice thread.

I agree :slight_smile:

These little tinkerings add up and in the end I’ve got no clue as what
I have done to my machine.

Yes, I think this is really important to keep track of for scientific
reproducibility, and Nix/Guix seem to be getting a little traction here.
Definitely superior to a typical Docker script, which runs things like
“apt-get update” whose results will change over time. Not quite as
foolproof as storing a whole VM image, but also takes less storage space
(e.g. easily tracked in git) is more explicit about what’s been done,
and provides convenient hooks to build things differently.

I still have trouble understanding what will I find in the Nix manual,
and what in the Nixpkgs manual. When I try to make my own Nix
expressions, some things are in Nix manual, and some in Nixpkgs
manual.

The flippant answer is that the Nix manual describes the contents of
builtins, whilst the Nixpkgs manual describes the contents of
import <nixpkgs>. More usefully, the rule of thumb is that Nixpkgs
mostly contains things that we can’t be bothered to write ourselves,
whilst Nix (builtins) contains things that we usually couldn’t write
ourselves, since it needs to be done by the interpreter (e.g. typeOf).
The nix repl is also invaluable, e.g. builtins ? foo prints true
if foo is in builtins, and hence is in the Nix manual.

The manuals are comprehensive, but I’d like to have a proper API
documentation (akin to Haddocks or Javadocs) for all Nix functions
commonly used in packaging. It’s really hard to find a complete
specification on what miracles mkDerivation does under the
hood. Also, it’s hard to even find that some handy functions exist in
the first place.

I 100% agree with this. Especially as Nixpkgs evolves, I’ve found that
functions have been added which I’ve previously been reimplementing
myself (e.g. callHackage and haskellSrc2nix)

7

Thanks for such a detailed post! Your perspective is really valuable.

It seems the community is quite split:

My perspective as a committer is this:

  • Specific code changes and discussion around them happens on GitHub, as do RFCs.
  • IRC is the primary place where discussion happens, both support and development. #nixos has 727 people right now, and lots of others just drop in when they have a question.
  • Discourse is used for things that need more discussion than a quick real-time back and forth on IRC.
  • I’ve never used the subreddit, nor found any information I needed on it. I suspect it is mostly users rather than developers, just because it never really seems to come up in conversations I have.

I really wanted to have global /bin/bash available

I think you’d have a hard time making the case for an option to do this, because it’s really antithetical to the goals of the distribution. The only reason for /bin/sh is that POSIX requires it, and /usr/bin/env is there to give shebangs some supported way of finding binaries.

As it stands, it is possible to do have this (though very much not recommended). You can ln -s /run/current-system/sw/bin/bash /bin/bash, and if you want Nix to make sure that’s there every time you rebuild you can put it in an activation script (add -f in this case). But, I strongly encourage you not to do it.

2 Likes
8

Slightly off-topic, but Pacman includes a tool called pacdiff which makes it easy to deal with this: it lists the .pacnew files one at a time, and gives the chance to delete/overwrite/compare & merge with a diffing tool of your choice. I understand there are similar tools in the RPM world to deal with .rpmnew files.

I used to run this once a week for exactly this reason: it kept the default values ‘up to date’ (and clued me into what was changing) while also letting me keep my customisations, at the expense of a little bit of time and effort. It also helped to keep /etc ‘tidy’.

Of course, I haven’t needed to even think about this since switching to NixOS :slightly_smiling_face:.

2 Likes
9

Hi @matix, Regarding /bin/bash, you might be interested in this discussion.

10

So if this is the case, why is it that:

/bin/sh --version
GNU bash, version 4.4.23(1)-release (x86_64-unknown-linux-gnu)

I mean either we are strict or we are not strict but not this Schrödinger stuff.
I feel kind of embarrassed right now, after I wasted myself and others time to do this:

11

I don’t understand what you mean. /bin/sh is required by POSIX, so what exactly are you complaining about? The fact that this requirement is being satisfied by bash? Doing anything else would require installing a whole new shell just to use for /bin/sh.

12

@spriobel I haven’t looked at your thread in detail but usually the best approach is to help and convince upstream to use #!/usr/bin/env bash instead of #!/bin/bash. This will work both on Ubuntu and NixOS and not require to rewrite the script to POSIX shell which is quite cumbersome to use.

2 Likes
13

So we cant have /bin/bash because POSIX does not demand that, but we are okay with /bin/sh not actually running sh but bash. Even ubuntu is more strictly complying with this by linking /bin/sh to dash.

dash is around 200Kb.

I am uncomfortable doing this. I dont want to waste peoples time for no benefit. Big distros like ubuntu have /bin/bash around. What is common practice is as good as a standard like POSIX. Especially as this is not a sweeping change. I just linked /bin/bash /bin/sh. I guess that many scripts in nixos with /bin/sh at the top are actually depending on bash.

1 Like
14

The goal of NixOS is not to be POSIX compatible. We need /bin/sh because of POSIX. Because a lot of programs assume that /bin/sh exists.

The goal of Nix, nixpkgs and NixOS is to re-think how a system is composed together. Instead of installing programs in /usr/bin, install them in /nix/store/<hash>/bin. And if we capture all the build and runtime dependencies accurately the programs will become more reproducible and work in more environments. In that sense, /bin/sh is a missing runtime dependencies that should be patched to /nix/store/<hash>/bin/sh. We already went to the trouble of changing the ELF interpreter path and many other things but /bin/sh is tricky. Hopefully one day we’ll be able to even remove that.

The main reason not to link /bin/bash on NixOS is to make sure that programs fail if the package didn’t capture bash as a runtime dependency. To continue on our quest to map all the dependencies. And we haven’t used dash as our /bin/sh because this isn’t part of our mission. It doesn’t mean it’s a bad idea and it should be evaluated on it’s own.

I hope this gives you a bit of historical context on why things are as they are today.

Regarding Discourse, I recommend submitting a patch upstream to use #!/usr/bin/env bash in their scripts. It’s the best thing to do as bash is not guaranteed to live at /bin/bash. If anyone wants to run Discourse on a BSD distribution they will be facing the same problem. Or convert the scripts to POSIX shell and then use /bin/sh as the interpreter. Either will work fine.

11 Likes
15

so how does submiting the #!/usr/bin/env patch to discourse gets us closer to this goal?
To me this whole /bin/bash vs /usr/bin/env bash thing sounds a lot like bikeshedding.
I actually went down this rabbit hole a bit. Here is some of the stuff I came up with besides the stackoverflow answers:
https://www.cyberciti.biz/tips/finding-bash-perl-python-portably-using-env.html
I dont know if its true what the comments say about env creating problems, but it clearly seems like a hack that does not bring any clear benefits in regards to achieving the goals that you stated. I am also not using BSD so I dont care if BSD users need to symlink stuff. (They are probably used to this anyway).
So what if we just added an option to the bash nixpkg to symlink in /bin (that defaults to true)?
I think the nix ideals are good. But they are ideals. In reality stuff is messy and I dont see how having /bin/bash makes it more messy. I would even say it makes it less messy. (In how many cases does something depend on a different special version of bash?)
I have the suspicion that probably the same is true for the ELF linker. Sure its nice to have tools like patchelf, but how often do you really need to use a different linker? wouldnt it be better to just use it in this case and not all the time? It just seems like unnecessary pain.
Imagine we wrote a linker plugin, that links the right libraries(defined in the nixpkg) by looking at the path of the binary. So we can do away with all the binary patching and just declare the dependencies in the derivation. This would save so much resources while the edge cases where a different linker need to be used can still be binary patched with patchelf.
Realistically the only way to truly get rid of /bin/sh is to hook the open syscall for all binaries and tell them this way where /bin/sh really is. You can’t patch the /bin/sh out of all shell scripts everywhere. It also opens the door to let the projects keep their idiosyncrasies while still being able to manage the dependencies. You could just define in the derivation that /bin/sh is for some project /bin/mylovelylovelyshell if the projects owner likes it better this way. Instead of trying to convince this guy to accept your patches you could just move on with your day and use his lovelylovely software.

16

Because bash is then loaded from the PATH and it works on NixOS?

If you swap bash by python it will become more apparent why it’s a desirable thing to have that in development environments. Especially combined with nix-shell it allows to select precisely which version should be used.

Then once installed, nix patches the shebangs and they become absolute paths again.

5 Likes
17

(Somewhat off-topic)

I was going to do exactly this myself but was put off by their soul-selling CLA…

If anyone does try to submit this, be careful and thoroughly read what they’re asking you to sign so you can be sure you’re okay with it.

1 Like