What's the replacement for NIXOS_LUSTRATE?

Development Documentation
1

Reading Installing from another Linux distribution (from nixos manual) step 9 says:

The lustrate process will not work if the boot.initrd.systemd.enable option is set to true, which is now the default. Setting this to false is deprecated and scheduled for removal in NixOS 26.11, along with NIXOS_LUSTRATE. Other installation methods, such as the one outlined above, or installing from kexec, are recommended instead.

Following to the kexec link only explains setting up and starting the kexec, not what should come after.
As opposed to the lustrate tutorial which is very step-by-step and ends with a complete NixOS system.

Given kexec puts you into a linux system which is on a seperate partition (since it is in ram), I’m assuming the logical next steps is to follow a different branch from the same section:
Start of the section:

You can […] Install NixOS on another partition, from your existing Linux distribution (without the use of a USB or optical device!)

Step 4:

The following 5 steps are only for installing NixOS to another partition.

But this guide expects the current system is not on NixOS (which the kexec is), and so steps 1-3 are not necessary.

Overall it’s very confusing to follow, which is especially a problem in this case since we’re dealing with formatting drives and the likes, which can make a system unrecoverable.
And so small uncertainties quickly become huge doubts !

As for the solution, I think there should be a step-by-step guide in the same style as for NIXOS_LUSTRATE for the kexec way, from starting on a distro without nix, all the way to having in-place replaced the OS with NixOS
(and probably have small notes on how to keep ssh connectivity throughout, since usually if you can’t use a boot media, it’s because you’re connecting remotely)

I’m open to contributing this guide myself, but do not yet have a full understanding of everything involved (myself trying to follow this guide), and as such would need guidance to do so.

2

Note:
I ended up on this tutorial in the nixos manual through:

which links to:

in which:

Note: If you don’t need to change the filesystem, you can follow the new section in the nixos manual.

3

Sorry, I can’t get into the weeds of what you’re up against, but, perhaps if you take a step back and share what you’re trying to achieve there’d be an easier way?

4

Discussion of this change and possible alternatives seems to be at nixos/lustrate: init module, for systemd initrd by dramforever · Pull Request #509036 · NixOS/nixpkgs · GitHub

5

Since the kexec boot is NixOS, I would rather think the guide to follow for installation after kexec is the normal installation guide, not the “install from a different distribution” guide. Since (I see) Lustrate is deprecated, and the kexec guide is specifically titled to indicate that it is just a way of booting NixOS, I’m not sure what improvements could be made to the docs here. The intended flow is to just use the normal / primary installation guide, which instructs you to boot from an external installation drive. If one were to decide to try to avoid having to use an external drive, the option is there to use kexec, in which case you would swop out only the “booting into the installation environment” part, and then continue with the installation as normal. But since this is more advanced, it makes sense that it is not mentioned in the normal installation guide.

6

You could probably use whitequark/nixos-bite: A script for automatically converting a Debian VPS into a NixOS VPS - Codeberg.org . It works even on 26.11.

2 Likes
7

My goal here is not really to figure out a solution to my personal problem, I’m more trying to communicate an issue in documentation
(that I of course experienced trying to solve a problem)
That is why I put this under Documentation - NixOS Discourse as opposed to “help”

Maybe I should have presented it more clearly, but I don’t have all the facts yet
Here is my attempt:

  1. The manual says that boot.initrd.systemd.enable and NIXOS_LUSTRATE are both deprecated and scheduled for removal, but offers no way of getting additional information. (Things like why was this decision taken, as well as more details onto the set of ramifications.)
    In particular the link to boot.initrd.systemd.enable leads to a page which does not even mention it is deprecated.
  2. The kexec guide is only “here is how you do this”, a “here is what this allows you to do” part I think would be nice, could simply consist of links to other sections of the manual
  3. The official NixOS wiki recommends using this deprecated (and potentially fragile?) method of installing NixOS
  4. The guide in the manual for “Install[ing] NixOS on the same partition […], from your existing non-NixOS Linux distribution” uses the deprecated NIXOS_LUSTRATE, in other words, there is not guide for doing so without using deprecated tools.
  5. (somewhat redundant with previous points) the guide for “Install[ing] NixOS on the same partition” tells you “don’t use lustrate, use kexec instead” (paraphrase), and links you to a page which just shows you how to do kexec, as opposed to using it for the task the guide is about.

Thank you !
That’s the kind of thing that would make sense for point 1

That does make sense, I guess my thinking was that I started (before kexec) on a different distribution, so that is where I should keep going from.
(A mistake in hindsight, but maybe illustrative of the kind of mistakes users make, and which the documentation should strive to nudge against ?)

Maybe this betrays my recent dive into OSes, but it’s not obvious to me what to do with something that boots my desired OS, but in a temporary manner.

This frames it as more of a choice than it is, as far as I know, I can’t use an external drive, since this is on a VPS.

My point (5) is that a similarly complex option was deprecated, as a result there is a regression in the documentation.
(in the software testing sense)

I’m not sure what you mean, 26.11 is not out yet ?
And your comment on the above PR mention that lustration underpins the current nixos-bite.
(and both nixos-bite and lustrate work on 26.05)

8

nixos-bite includes its own copy of lustration logic so it is not dependent on it being available in upstream NixOS.

1 Like
9

I think it’s both the curse of knowledge and also the fact that NixOS is not aimed at beginners that the docs don’t explain what a live-boot environment is, and that it is the normal way to install most Linux distributions. If someone’s got that down, then assuming that kexec basically uses RAM instead of a USB is pretty straightforward.

The other problem is that there isn’t really a single officially supported/endorsed way of doing this, which is why there are several projects that address this problem. nixos-infect is one, @whitequark’s nixos-bite is similar, nixos-anywhere is related but different, etc.

If we were to write an official guide on how to use kexec specifically, the method would have to be thoroughly tested. As far as I’m aware, it’s a bit hacky or unstable.

For all these reasons, in my opinion, it’s wise of the manual to not recommend this as an officially supported path for installing NixOS. But maybe it’s worth putting on the to-do list.

10

More information is available in the release notes. The release notes don’t mention lustration because I thought it was sufficient that the section in the manual about lustration informed users that it was deprecated. A user seeking information about lustration is more likely to encounter the lustration documentation than the 26.05 release notes. But I suppose the lustration documentation could have linked to the release notes itself.

There’s not much point in taking space in the documentation to explain the motivation behind a change, and doing so would have made the release note less useful. The more noise users have to sift through to see what’s relevant to them, the worse the documentation is. That’s why things have been stated closest to where they will actually be encountered. For instance:

This would not actually help anyone. Once you set that option to false, your config raises an evaluation warning that tells you it’s deprecated. This warning is more effective than any option description ever could be; it ensures the user of the deprecated option sees that it is deprecated, because using it in the first place generates the warning. It’s possible that, for this specific option, the added noise in the option description would be negligible, but the M.O. (putting information where it will be encountered by those who need it) did not demand it. Putting the deprecation warning in the option description is neither sufficient (because it only reaches those who read the description) nor necessary (because all users who need to be informed are definitionally informed by the eval warning), so it just didn’t make sense to put it in the option description.

I spent a lot of time trying to decide what to put in the release note and had a lot of input from others on the matter. I could have written pages and pages about all the things that are different and better about systemd stage 1, but no one would have read that, and the information would have been impenetrable. So it made more sense to reserve the release note for only the information that would not be provided to the user in some better way. For instance, the release note mentions cryptsetup-askpass being gone because there’s not really a better way for that information to be conveyed; but there’s also this assertion as a separate thing about cryptsetup-askpass. That assertion applies to a more specific instance (users setting the initrd shell so that an SSH client can only remote in for the purpose of password entry), and provides extra information about what to do instead for that use case. That is additional information that would add noise to the release note, so that assertion is the better place for it. A user who needs that information will encounter it by nature of the assertion. Similarly, the phase scripting options like boot.initrd.postDeviceCommands are not mentioned in the release note because there is an assertion for it that informs any users who need to know.

A user seeking to use lustration will look primarily at the documentation about lustration, so that’s where it makes sense to inform users about the deprecation. So I don’t think this was the wrong approach. That said, yes, it should probably have links to a couple things, including the release notes. But at the time, I thought that the link to the kexec guide was all it would need. On that subject:

I think part of the confusion here is that the kexec guide doesn’t quite spell out that it’s intended to be a NixOS installation environment. I thought it was clear enough on its own, because it’s in the “Additional installation notes” subsection of the “Installing NixOS” section, and because the first thing it tells you is that something you might be using instead of it is an installation image:

In some cases, your system might already be booted into/preinstalled with another Linux distribution, and booting NixOS by attaching an installation image is quite a manual process.

But I think it would be reasonable for it to more plainly spell out “This can be used to install NixOS using normal NixOS installation instructions, as though you booted the ISO”. Titles of higher sections aren’t necessarily visually obvious, and the line I quoted is passive about this point.

Unfortunately a wiki just can’t be considered documentation. When a change is made in nixpkgs, the author of the change can do a reasonable job of making sure everything related to that change within nixpkgs is updated. Extending that to the wiki is not reasonable, because the wiki is freely editable by anyone without review. There’s a lot of information in the wiki, and it is not curated by maintainers, or in step with that maintenance. If it were, it would just be documentation, not a wiki. So the whole point of a wiki I think is to be something very different from documentation.

Again, I think you’re just looking for the kexec guide, and maybe it could more explicitly state that that is what it is.

Overall, I think the main actionable improvements we can make for these issues are:

  • Add some motivation to the kexec section to inform the reader that it is useful as a standard NixOS installation environment in the first couple sentences.
  • Add a link in the lustration section pointing to the release notes.

Does that seem reasonable to you?

2 Likes
11

There’s also nixos-anywhere which, if it detects the target system is not running a nixos installer already, will kexec for you and automate the install. This is my personal preference for bootstrapping NixOS installs, I personally use it for both systems where I can boot a NixOS ISO, and those where I can’t do that.

The downside is the system requirements are a little heavy, so it’s not suitable for micro VPSes, but for anything else it should work just fine.

1 Like
12

I totally agree, I’m not saying this note-callout-thing should have been 3 paragraphs long !
A couple of links to useful sources would suffice !

Yes exactly !

I think it would have been good nonetheless, consequences like this are a lot easier to miss that one might think.
(As an example, in my experience that’s what makes the C/C++ documentation hard to read, it plainly tells you that A does B, but never even hints that very importantly, B does C, so A does C as well)

This seems a bit baffling to me ^^’
I get the idea, if you use it, you’ll see the deprecation warning.
But isn’t the point of documentation to document this kind of thing ?
I get that with nix “execution is free”, but I don’t know. It feels weird to me that if I wanted to reason about code before writing it and look at the documentation, I would miss something as vital as “is this deprecated”.

And so I would disagree with

That is true and cool, and I don’t think we should stop.

Note that this sentence tells me “it can be used to install nixos”, but I was left to figure out “how to intall nixos using it”

Yes exactly !

I understand your point from a dev pov, but from a user’s pov, it’s documentation !
Maybe I’ve missed something better, but the wiki has consistently held answers to my questions, whereas other pages have left me more confused.
The wiki tells you things like “btw, this is also a good idea to do”, “here’s what to do if this doesn’t work”. (things IMO good documentation does)
My experience with the rest of documentation is “this, it does that”, with often not even a link to what “that” is

I most likely agree, my point is that this guide does not (yet) work as a guide for "Install[ing] NixOS on the same partition […], from your existing non-NixOS Linux distribution”.

Yes, it seems very good !

For the kexec section, I would maybe put the link at the end, currently it ends a bit abruptly:

Once you finished copying, execute kexec-boot on the destination, and after some seconds, the machine should be booting into an (ephemeral) NixOS installation medium.

or

Make sure your configuration.nix does still import netboot-minimal.nix (or netboot-base.nix).

depending on where you consider the end to be

Something like that would be nice:

[…] the machine should be booting into an (ephemeral) NixOS installation medium. Once there, you can install NixOS on your system proper by following the same steps as when manually installing from a boot media. Since “kexec” runs the OS entirely in ram, the drives are not mounted, and thus can be formatted.

(This is a WIP, but hopefully illustrative of what I’m thinking of)

This section also assumes knowledge of what kexec is, so I think an external link to what kexec is would be useful, for example changing:

In these cases, it might be easier to use kexec to “jump into NixOS” from the running system, which only assumes bash and kexec to be installed on the machine.

to

In these cases, it might be easier to use kexec to “jump into NixOS” from the running system, which only assumes bash and kexec to be installed on the machine.

As for why would a user would read the kexec section when they don’t even know what kexec is, I hope this post shows an example of why that happens
(To spell it out: I didn’t know kexec [tool] was a thing, I just wanted to install NixOS on a VPS [task], and I got linked to this section when looking for how to do that task)

P.S: @ElvishJerricco Thank you for putting so much thought into documentation, and for replying to my comments!

1 Like
13

(aside) I’ve managed to complete installation! In the end it was a very round-about way to do the steps described here in the wiki (end aside)

With knowledge in hand, I went back to update the wiki with the information that would have helped me (lustrate is deprecated, clarify “install like normal”)

I think I’ll try to rewrite the “Building the system” section in a way which:

  1. Does not depend on “clever’s kexec config” since it is outdated, and contains things which we later override anyways. So relying much more closely on the manual’s kexec.
  2. Shares as much configuration as possible between the kexec image and the final nixos-installed image (if possible, they would be identical). This would allow a lot more confidence that being able to ssh into the kexec translates to being able to ssh into the final install.

(I’ll put it on the wiki, but I’d gladly open a PR to the manual as well if you tell me it would be useful)

For additional feedback on the documentation, I think the Manual Installation section is a bit awkward in that it assumes you are in the install medium, but that ends up basically only being relevant for a few paragraphs at the start (and one at the very end). As a result, the best place to link to for a non-install-medium manual installation ends up being the sub-section Partitioning and formatting

1 Like
14

The pivot from kexec’s temp environment to the actual new install is where the docs stop. That’s exactly the moment you’d want spelled out when you’re remote—especially with SSH in the mix.

2 Likes