General questions for a popularization article in France

Still writing. I think that I’ve eventually found a friendly way to explain the core principles of NixOS that I really like.

I still have couple of questions though :

  • /run/current-system/sw/ folder seems willingly to stick to the FHS. Is it needed for something ? Maybe for the Linux startup process ? Thinking about GoboLinux’s folder hierarchy makes me wonder if it could be otherwise.
  • Furthermore, /run/current-system/sw is a symlink to a folder in the store. I guess that, during startup, selecting an older generation corresponds to editing this link, is it correct ?
  • Overall /run/current-system/sw is kind of equivalent to profile/generation concepts used for users nix-envs and root nix-env. I am wrong ?
  1. Correct, see congruent/convergent deployments.

  2. Nixpkgs-unstable is rolling release. However we rarely do “state” migration. Almost never. We don’t have tests that state is migrated correctly between updates, that’s why stateVersion is fixed. Only a few services are affected here.

  3. Technically, yes. In practice, those images are often larger than alpine (but smaller than Ubuntu’s).

Yes, only runtime deps, except when due to some bug build time deps are retained.

Yes, it’s much easier to build with Nix, rather than with dockerfiles!

1 Like
    1. It is implementation detail. NixOS is a package, it is mapped to that /sw path. I’m sure there can be another fork of NixOS without that folder, but just as you mentioned - it is easy to switch generations with current layout.
  1. Kinda yes. It is a copy of /nix/var/nix/profiles/system. And yes, nixos-rebuild uses nix-env under the hood!

1 Like

Thanks @danbst, very useful answers.

What do you call “state” migration ? Is it the state of a NixOS database ? If yes, I would be really interested to know what kind of data is stored and which services depends on it ? If migrations are hard to apply I also wonder about the sustainability of the database.

First of all, there is no NixOS database!

  • there is Nix sqlite database, but it doesn’t contain anything system-specific (AFAIR). You can remove /nix directory, reinstall Nix with new sqlite DB and rebuild your system from config file (don’t try that on running system!)
  • there is NixOps database file, and it indeed contains bits of system information. And it has internal version code. But this is NixOps, which isn’t quite NixOS.

What do you call “state” migration ?

For example, we have postgresql_9_6 and postgresql_11. However, if you build a system with

services.postgresql.enable = true;
services.postgresql.package = pkgs.postgresql_9_6;

and later change it to

...
services.postgresql.package = pkgs.postgresql_11;

then NixOS won’t change database files on disk to match new PG version (and service will fail). This may be strange, because NixOS upgrade can “upgrade” (i.e. change) configuration files and default package versions. So, NixOS is selective in what to upgrade and what not. Experienced devs find that obvious, but newcomers may face service errors (because they believe NixOS is fully declarative).

So, for stateful services, NixOS isn’t fully declarative, but nevertheless, it is highly reproducible.

2 Likes

Thanks for this information. It was counter-intuitive since configuration.nix uses system.stateVersion = "19.03". Why is the version of NixOS specified here rather than Nix or NixOps ?

Ok. Is it impossible to upgrade the DB file due to postgresql lack of migration between one version of postgresql and another ?

Is it the same for all database files, whatever the db manager (postgresql, mysql, …) ?

IIUC, the services mentioned here are the database servers, correct ?

While upgrading, are packages of database servers (postgresql, mysql, …) pinned ?

It would be great if someone could translate the finished article to english. Sadly there are not many good articles that explain what makes NixOS special.

Will it be openly licensed, for example under CC-BY-SA?

Just a precision : it seems that /nix/var/nix/profiles/system points in fact to a folder that contains the symlink sw.

IIUC, NixOS is a package at /nix/store/2wqdx...-nixos-system-nixos-19.09pre188... pointed by /run/current-system, which is itself a copy of /nix/var/nix/profiles/system. Thus, I imagine that during startup, selecting an older generation edits /run/current-system (instead of /run/current-system/sw).

It would be great if someone could translate the finished article to english. Sadly there are not many good articles that explain what makes NixOS special.

If needed I can translate, my English is not that good but enough,
after someone can proofread the English part for grammar/syntax
error…

– Ingmar

There is https://nixos.org/nixos/about.html. I often wonder why I don’t just translate this in french… Hopefully, I will manage to explain, in a nutshell, how NixOS features are achieved, without loosing the reader.

For me, it was important to know how NixOS was solving the hell dependency using the store. Usually, it is far easier to trust and use something when you understand how it works. I don’t know if it is me, but I also tend to forget a feature if I don’t know why this feature is possible. (same for mathematical equations…)

Sadly many parts of the page are probably outdated and need better explanation. I think most people in the NixOS community have a technical background, so we develop great solutions, but we lack people with UX, technical writing or marketing background. So the onboarding process and explanations/documentation is not great. New users might get frustrated quickly if not patient.



A good article can help us find good ways to explain what NixOS is and why it is amazing!

1 Like

Why is the version of NixOS specified here rather than Nix or NixOps ?

Let me show you how is this system.stateVersion used.

Now you can predict what happens when you change system.stateVersion:

  • if you change it to 20.03, nothing will change
  • if you change it to 17.09, still nothing will change
  • if you change it to 16.03, PG package will be downgraded to 9.5, but in the same time data directory will be changed to /var/db/postgresql. This will cause fresh DB initialization (because there is no DB), and service will work using PG 9.5. You suddenly discover all your tables are lost, but in fact not, it is just they reside in a different dataDir. Rollback will fix issues.
  • if you change it now to 15.09, you’ll get another PG downgrade (to 9.4), but this time it will be incompatible with dataDir (given you had rebuilt with 16.03 stateVersion previously). Service will fail

And the other way round. My current system has stateVersion 18.03. But I actually run of 19.09pre (some checkout of master). Just to not have those problems above when changing stateVersion, I don’t change that. When PG default version will be changed to 11.0, my stateVersion will protect me from getting this upgrade.

Ok. Is it impossible to upgrade the DB file due to postgresql lack of migration between one version of postgresql and another ?

It is possible to use pg_upgrade (or dump/restore) to make the automatic migration. It is just NixOS doesn’t even try to do that (for the good).

Is it the same for all database files, whatever the db manager (postgresql, mysql, …) ?

I think so. Use this search tool for other stateVersion usecases: https://github.com/NixOS/nixpkgs/search?q=system.stateVersion&unscoped_q=system.stateVersion

While upgrading, are packages of database servers (postgresql, mysql, …) pinned ?

exactly

1 Like

BTW, in the article you can mention how far does NixOS gets when testing things.

The nixos/tests is the most awesome collection of integration tests I’ve ever seen. Go ahead and try to figure out how are Ubuntu services tested. :man_shrugging: But in NixOS integration test is just a Nix package! Nix package, which launches arbitrary VMs with declarative NixOS specs and clearly defined test steps.

  1. One of the coolest examples is LetsEncrypt test. It starts an ACME server, a website and a client. Website then does checkup with test ACME server and obtains a certificate, clients does TLS curl to website to check that certificates are properly installed.

  2. So you want to try to build etcd cluster in NixOS. But internet has no examples. Don’t worry, just check up the integration test! It contains working configs, which is enough to start from.

  3. Want to test some GUI stuff? Here’s how Firefox is tested - an X server is started on virtual framebuffer, and FF is controlled using xdotool.

  4. Remote printing test!

  5. Another kind of graphical tests, looks simple but look at the line:

    $machine->waitForText(qr/Link your phone to Signal Desktop/);
    

    It means, do a screenshot, perform OCR over that screenshot and check if this text line is present. Does Ubuntu do automated OCR-backed tests?

  6. And one of the most important tests - installer test. This tests start a VM (an installation device) with a few virtual disk drives attached, install nixos using nixos-install on one of drives, then (inside VM) run another VM to test that installed NixOS actually boots and works.

    And this test is performed for: MBR+ext3, GPT/UEFI, UEFI/Grub, with /boot partition on another disk, with fat32 /boot, with ZFS, with LVM, with LUKS, with RAID, with few variants of BTRFS…

    This installer test sometimes fails. Which is great, as it fails on Hydra, not on someone’s machine!

There is a paper published (as for Nix and NixOS), it is really worth to mention that.

5 Likes

@davidak : It might be too early to plan an english translation since substantial work remains to finish the article.
Concerning the license, I planed to choose the less restrictive article’s license of the magazine : Creative Commons license BY-NC-ND. It’s type B. Unfortunately, this license forbids modification and I would rather choose CC-BY-SA if it was available. I will ask the editor for any translation rights after publication.

I agree.

@danbst : Thank you for your explanations. Now I do understand in details how postgresql versions and dataDir are managed by system.stateVersion. Though I still wonder why it works this way. Next sentence is not obvious for me

My misunderstanding is certainly due to my lack of sustainable management of databases : I fear that I’m not an “experienced dev” on this point.

Wow, I 100% agree, these integration tests are awesome :no_mouth:. A lot of smart abstractions here :brain:. That is very professional. I didn’t know about it and I also believe that these important tests could be displayed more. For sure I will mention that :+1:.

1 Like

sure, start when published and wait with release some days, so the magazine has it’s exclusive story for some days.

Would they get exclusive publishing rights or could you publish your text also on your personal blog under CC-BY-SA?

I asked the editor about translation. Publishing the article, untouched, on a personal blog is ok AFAIK.

Editor’s answer for the CC BY-NC-ND licence

“Pour faire simple, nous souhaitions proposer un contenu exclusif à nos lecteurs et protéger le rédactionnel qui a été rédigé afin de pouvoir éventuellement le proposer par la suite sur d’autres supports.
En même temps nous voulions permettre aux auteurs de pouvoir repartager leur contenu, d’où ce choix du CC BY-NC-ND qui est très rare dans le monde de la presse.”

In english it would be something like :

To cut a long story short, we wanted to offer an exclusive content to our readers and to protect the written content that has been written in the aim of possibly been published afterwards on other media.
We also wanted to allow authors to share themselves their content, hence the choice of the CC BY-NC-ND, which is a very rare choice in the press world.

In the mail, they have been clear about it : the ND (No Derivative) does forbid any translation. I’m really sad about this @davidak.

Anyway, I’m not legally bound to this editor and I can still switch to another media with a less restrictive license. However, I don’t know any equivalent editor in France, dealing with the Linux universe.
French NixOS users, do you have suggestions ? do you know any fitting media/plateforms ? linuxfr.org is interesting concerning license but @juliendehos has already published many articles about NixOS there; IMO it might be time to publish somewhere else.

2 Likes

Thanks for clarification. When publishing a translation is forbidden, everyone can translate it for personal use. DeepL works way better than Google Translate, but sadly is not open source. Apertium is open source, but don’t offer French to English or German (the languages i understand).

1 Like

Reading Nix Pill’s section about composability, I wonder if the difference between a derivation and a package is mostly the composability feature of derivations compared to packages, that cannot be composed.
IIUC the composability of a Nix derivation mostly means the ability to edit a derivation with chains of functions, each taking an input derivation and morphing it into an edited derivation.
Please correct me if this is wrong.

EDIT : Based on a stackoverflow answer, it seems to me more general and more accurate to think that : composability of Nix derivations means that several derivations can be combined in a certain way to produce another derivation.

A draft of the article is written and it is being reviewed by two members that offered help. Mostly waiting for reviews now before sending to the editor. Depending on how well reviewers like the draft it could be quick or take some time.

Some questions came during the review discussion :

  1. What does mean sw in /run/current-system/sw ? The wiki states that “System-wide packages are in /run/current-system/sw/” so I thought it means “system-wide” but sw may also mean “software”, since sw is a common acronym alongside hd for “hardware”.
  2. Is it ok to write that NixOS does something when this action is actually done by the packet manager Nix. For instance, is it ok to write that NixOS updates generations/profiles links when it is actually performed by Nix ? Another example, for the moment, in the article it is written that NixOS allows each user to manage the installation of packets in their user space, through profiles and generations. Should it be written that it is because of Nix, or is it ok like this for popularization purpose ?
2 Likes