RFC: Varlink API for switch-to-configuration-ng

Development
1

TL;DR: Extend existing switch-to-configuration script with a new flag --varlink-stdio which exposes an Varlink API over stdout/stdin.

Audience

This is a design discussion for people who work on the activation pipeline, Varlink or any deployment tool interacting with NixOS. Drive-by comments are not helpful there.

Motivation

switch-to-configuration (abbreviated stc here) is one of the most well-known “public” APIs of the NixOS project, derivatives tend to reproduce similar concepts to enable the activation process of their generations (if they have generations). It only applies to store-based NixOS, image-based NixOS do not make use of this concept.

After a recent rewrite in Rust, stc remains a program that many pieces of the ecosystem interact with via free-text stderr and exit codes.

Structured APIs in Linux distributions have always existed via D-Bus but certain programs cannot easily expose a D-Bus API due to the requirement of a centralized broker. Recently, systemd has started to adopt Varlink, a broker-free approach to system IPCs and has converted a significant amount of their APIs to it (varlinkctl can be used to play with this).

The fact that stc is not a structured API cause at least three problems:

  1. No machine-readable progress, e.g. it is only possible to stream back the contents of stc.
  2. Deployment tools (generic term to refer to things like nixos-rebuild but also out-of-tree deployment tools) reimplements their ways of knowing how to parse the results of a dry-run and so on.
  3. Passing more structured information to tools that stc calls itself, e.g. the bootloader installation phase or the switch inhibitor hook.

This RFC proposes that we explore a Varlink stdio interface so that deployers can consume the same “binary/script” that the toplevel’s stc consumes, instead of screen scraping it.

Goals

  • Add an optional Varlink entrypoint to switch-to-configuration-ng, without removing the existing CLI surface.
  • Provide a typed, streamable public API contract for Switch, DryActivate, Boot, Check with structured Progress and Result and errors

Non-goals

  • No long-lived or socket-activated org.nixos.SwitchToConfiguration Varlink service (See the “Why not a socket?” below).
  • No fancy auth/authnz for allowing unprivileged stc.
  • No persistent Plan that can be replayed.
  • No change to the activation script contract. $out/activate remains a shell script.

Why not a socket?

A long-lived (or even socket-activated) Varlink service would mean that switching to a new NixOS system would reuse the code of an arbitrary older generation (exactly the one that the previous stc has put in place). The design of stc is one that you use the newer stc to stc into a newer generation, therefore, it’s not possible to start a stc server in systemd and have a mechanism to reexec remotely the stc server via the newer generation stc client.

Additionally, stc has requirements to operate in systemdless environments for recovery reasons. For this, a stdio transport is the simplest.

If people wants to be able to observe ongoing stc at a global level for desktop-level or fleet-level observability, it should either be done at the deployment tool level or stc can register a standard event socket location which talks a standard event vocabulary, this is out of scope for this proposal.

Expected approx. implementation plan

  1. Move most of the main.rs into a lib.rs
  2. Add a dependency on zlink to stc-ng (is it a problem? If so, we can have our own impl of Varlink “low-tech” easily).
  3. Add --varlink-stdio flag or subcommand + an IDL file shipped as part of the output. The CLI remains untouched.
  4. Have nixos-rebuild-ng consume the Varlink output.
  5. Have the stc client consume the Varlink stdio.

cc @arianvp @flokli @nikstur @ElvishJerricco @Ramses (cannot find his @) for the systemd maintainers.
cc @jmbaur for the author of stc-ng (thank you!)
cc @k0kada for the author of nixos-rebuild-ng (thank you!)
cc @lewo for comin.

If there’s no objection in 2 weeks, I will start with the trivial refactors and open PRs in nixpkgs.

30 Likes
2

As someone who suspects that we should also be using for Varlink for builder process ↔ Nix, I like this. Thanks for proposing this.

Stdio also seems like a good choice. There is no need to dirty the file system with a listening socket for a short-lived process. If we ever were to need to pass FDs, we could use a single socketpair rather than two pipes.

Do you intend it to be a “server serving a single connection with many requests?” Or something like “mini Varlink” that takes a single request, services a single response, and exits? I’m fine either way.

1 Like
3

It’s been in the back of my mind for a while that the interaction between s-t-c and nixos-rebuild is really not that great, and I have some custom tooling on my machines for automatic updates that also does that kind of stderr processing.

So yeah, the proposal makes sense to me, to have a structured format for the communication between s-t-c and its callers. I wonder if this could also get us a proper fix to switch-to-configuration-ng: fails when stopping services disconnects the fds · Issue #462179 · NixOS/nixpkgs · GitHub , if we are anyway restructuring the whole way s-t-c logs its output.

1 Like
4

It’s explained why a listening socket is not a workable option and we may still have an inode in the filesystem for general discoverability of high level events.

As written in the proposal above, the API is meant to have streaming, therefore, it cannot be a single shot request.

As I have written it in the audience section, please be mindful of drive-by comments.

5

Yeah, what I imagine is that we can have a bag of options to enable all sorts of differing behavior, including how we run the stc-ng inside a systemd-run and what expectations to have on its survival across the system upgrade transaction.

With an API, I expect it also to be easier to perform more aggressive testing of such scenarios.

cc @raboof I guess who is probably interesting in this topic then.

6

@arianvp who is not accessing his Discourse account suggest rightfully we could use fd3 instead of stdin/stdout, which I agree with.

1 Like
7

I agree with your reasons too, I am just adding an additional reason in support. :slight_smile:

I agree with you that the use-case here is short-lived operations not a long lived service, and I am adding that for short-term operations it also is better to work with pipes (or maybe someday socketpair) because they are anonymous and just referenced via file descriptor.

In contrast, temporary socket files on the file system can be left behind if something goes wrong, and this just one more failure mode / thing to clean up. (And, as you say, we can’t be relying on systemd in all cases, so by extension we cannot be relying on systemd to clean up the socket file.)

Your design removes the socket files junk/conflict failure mode, and that is good.

I am not sure what you are referring to? Do you mean the IDL file? That is definitely fine to me and doesn’t count to me as “dirtying the file system” because it is static immutable metadata rather than temporary state.

I was not quite clear; my apologies. By “single response” and “single request” I meant high-level requests, not individual JSON messages. It would still be possible to accept a single streaming / (multi-message) request, return a single streaming (multi-message) response, and then exit.

I am also realizing this isn’t so impactful on ergonomics because if the server exits when the standard streams are closed, then there need extra “shut down the server after I am done talking to it step” for casual use-cases.

One thing I want people to realize is that Varlink is simple enough that with @RaitoBezarius’s design some lighter-weight scripting like

printf '%s\0' '{"method":"...","parameters":{...}}' \
  | switch-to-configuration \
  | tr -d '\0' \
  | jq '...jq-script-to-handle-response(s)....'

should, in fact, work. (And with Feature request: support for streaming input delimited by null characters · Issue #2659 · jqlang/jq · GitHub it gets even simpler.) Varlink says it is much more lighter-weight than other RPC systems and that is correct.

Since this is a single fd which must be bidirectional, that is a socket (from socketpair) not a pipe?

3 Likes
8

I’m referring to this which is written in the initial proposal.

My proposal is to implement a Varlink server, Varlink does not codify what is the method for graceful shutdowns. It is left up to the protocol or an out-of-band signal. Whether it’s standard streams being closed, in-protocol shutdown message, process-directed kill signals is an implementation detail at this point and I’d like to propose to focus on the merits of that protocol to decide what is the most appropriate method for graceful shutdowns if they are needed at all.

Rather than recommending jq, any NixOS user can already use varlinkctl for scripting. Please do not hijack this thread into a lesson on Varlink, this is neither the audience neither the intent of this thread. I don’t mind the enthusiasm but this is the second time I have to tell you to be mindful of drive-by comments.

I never said anything about pipes, can we focus on what do we want out of the activation protocol?

1 Like
9

@RaitoBezarius you are being rude in repeatedly trying to paint me as a thread-derailing dilettante here. I am trying to “bury the hatchet” from prior interactions between us because I saw an idea from you that I genuinely like a lot, and my technical questions are not, in fact, vacuous. It would serve you well to also adopt a “burying the hatchet” mindset. This is the only off-topic aside I’ll make on this matter — I do in fact not want to derail your thread.

When the proposal is to not use sockets, but to use standard streams (note the plural) for inter-process communication, this almost always means using two pipes.

My proposal is to implement a Varlink server, Varlink does not codify what is the method for graceful shutdowns. It is left up to the protocol or an out-of-band signal. Whether it’s standard streams being closed, in-protocol shutdown message, process-directed kill signals is an implementation detail at this point and I’d like to propose to focus on the merits of that protocol to decide what is the most appropriate method for graceful shutdowns if they are needed at all.

Typically a server serves multiple connections. They do not shut down based on a client closing one connection because more clients may try to connect after.

When a process serves a single connection, which is how stdio communication works, there is no need to stick around after those open files (in the abstract sense, could be pipes or a dup’d socket) are closed on the other end, because no more communication is possible.

I am aware of varlinkctl, but I’ll grant that I was not aware of its exec: and ssh-exec: modes — I thought all systemd varlink usage was with listening sockets / “bona fide” multiple-connection-serving servers.

If we want varlinkctl to work — and I think we should because it codifies idiomatic systemd Varlink usage in a number of areas beyond what the official varlink tool does — that answers many of the outstanding questions in this thread!

  • Add --varlink-stdio flag or subcommand

    varlinkctl’s exec: takes an absolute path only — CLI arguments are not supported. Therefore, instead of using a CLI flag, I think we should ship a second executable (or do argv[0] tricks to just get away with a symlink in bin).

    (FWIW ssh-exec: does support them, but poorly, for example mangling quoting. This might thus be an accident more than an intended feature. I don’t think it changes the answer to the above.)

  • @arianvp who is not accessing his Discourse account suggest rightfully we could use fd3 instead of stdin/stdout, which I agree with.

    exec: does a modified version of the systemd socket activation protocol: a single connected socket on fd 3 rather than a listening one, plus LISTEN_FDS=1 and LISTEN_FDNAMES=varlink.

    We should respect the protocol: require that LISTEN_FDS is 1, LISTEN_FDNAMES is exactly varlink (an fd named otherwise is not ours; note a varlink-named one may legitimately be a listening socket, in the normal server case — SO_ACCEPTCONN is what tells the two apart), and — per the usual socket activation contract — that LISTEN_PID is our own PID.

    ssh-exec: I would have guessed has a little remote-side helper to do the same thing, but actually it works a completely different way: stdin/stdout pipes, signaled by SYSTEMD_VARLINK_LISTEN=-.

    I think being able to do remote switch-to-configuration with varlinkctl is a good feature to have — remote deployments are after all a core feature of NixOS — so I think the varlink executable (or the same executable with the aforementioned argv[0] trick) should also support this mode of operation.

By your own prior interaction in Introspection doesn't support `varlinkctl` · Issue #233 · z-galaxy/zlink · GitHub, it seems that the zlink author is amenable to fixing problems that arise in the systemd / varlinkctl context. While there are many existing crates supporting the general systemd socket-activation workflow, the varlink server protocol adds a few steps. I therefore think it would be great if all of the above could be handled in some zlink-systemd crate, and then we can say for switch-to-configuration that we are just making a new executable (or, again, the argv[0] + symlink hack to use the same executable) that has an entry point defined using the zlink-systemd-provided trampoline, and thus we don’t have to think about any of this stuff because zlink-systemd makes all the decisions for us :).

4 Likes
10

I mentioned repeatedly that technical details are not the crux of this proposal, you keep following up with those. This is the third time.

Please stop engaging with this project.

1 Like
11

I do not understand why you are claiming that technical details are off-topic in a thread about technical details. You didn’t mind @arianvp’s point about using fd3, for example.

My high-level point with the above deep dive was that it should work with varlinkctl. I think you agree with that.

I’ll happily admit that so long as this remains a high-level goal of the project:

User can interact with switch-to-configuration using varlinkctl exec:... and varlinkctl ssh-exec:...

the details will get sorted out in the implementation PR.

If you can’t stand me I’ll leave this thread alone then. I’ll keep my eyes pealed for the implementation PR as I, personally, do not use nixos-rebuild and do use switch-to-configuration directly, and I would love to switch to using it with varlinkctl instead.

Thank you again for working on this project. Ciao.

5 Likes
12

That is not true in a “socket activated” context. The preferred way to do socket activation in a systemd unit is to use Accept=yes in the socket unit, which intrinsically means that the service unit must be a template that will be instantiated separately for every connection. The service does not handle multiple connections; it receives a connected socket FD and talks to one client until that session is done.

varlinkctl does that because varlinkctl is not the server with the varlink interface, it’s the client and it needs to start its server program, which is obviously not our use case. The varlink servers in systemd just do the same binary as the CLI, which is able to recognize contextually whether it’s being invoked as a varlink server. I think they do it by just checking if they’re being socket-activated.

Is it even “modified”? AFAIK it just is the socket activation protocol in its Accept=yes form. But that’s not a varlink thing, that’s a systemd thing. Granted, you are right that we probably should do this systemd-ism, but the reason for that is primarily so that varlinkctl can invoke it trivially.

I agree with raito that much of this conversation hasn’t seemed useful. I think the point of the post was to ask what blockers or requirements haven’t been anticipated in the original post’s description, and whether the design in general is desirable. The discussion so far seems to have been mostly about how varlink / socket activation works and about implementation details, neither of which were really the intended subject. “Back seat developing”, if you will. The only real point of substance has been your valid point that this should be able to be invoked by varlinkctl.

My input: @RaitoBezarius I like the proposal. I agree with @arianvp that we should probably pass a connected socket in. And I agree with @Ericson2314 that we should probably do so via the socket activation protocol so that it can be invoked via varlinkctl; AFAICT varlinkctl only does the stdio trick for the ssh-exec mode, not the exec mode, which, ugh, does that mean we have to do both? Whatever, you’ll figure it out :stuck_out_tongue: (EDIT: probably not; for users that want to invoke with varlinkctl over SSH, they can just use the remote’s varlinkctl and do JSON over stdio)

2 Likes
13

As a sidenote, I’ll mention that I think it’s likely that we’ll want another varlink interface for this purpose at some point, like a nixos-rebuild.socket sort of thing. This is future work, but I’m mentioning it in case it implies any constraints on the design of the stc protocol. Basically all the stuff that nixos-rebuild-ng does over SSH IMO should instead be done over remote varlink calls. If nothing else, that eliminates the difference in code paths for how binaries are executed, would serve as a better basis of authentication than the current elevation mechanisms, and should even allow the target system to decide for itself whether to trust the incoming nix closure (i.e. ensure it trusts the nix signatures).

But importantly for this discussion, it eliminates the need for the client to run an arbitrary executable themselves; rather than remote’ing in and starting the next generation’s stc directly, it would instruct the remote nixos-rebuild service to do that and hand back the connection to the resulting stc instance. I don’t think this imposes any constraints; the nixos-rebuild protocol should be able to either upgrade/replace itself on the connection with the stc protocol, or tunnel the stc protocol, or do some kind of second connection + cookie thing to move the connection. @RaitoBezarius Do you think that’s correct, or do you think the stc varlink interface will likely need to be designed with this use case in mind?

14

If I understand you well (there’s many things to be said about a nixos rebuild service that would go beyond this thread’s scope and I have thoughts on this and I believe that network boundary crossing requires capnp FWIW and mixing capnp and varlink is a problem I have thought about), the reply is no. There’s only a constraint.

The stc-ng Varlink interface should not introduce connection scoped handles (an handle), opaque references (an handle) or correlation tokens (an handle), if it doesn’t do that, an org.nixos.Rebuild can do whatever it wants with connection handover, tunnelling or second connections + cookie.

Here’s why:

  1. If you have connection scoping, you have a statement about “this invocation of stc-ng API is the same object as the one of five minutes ago”, for this to be true across conn handover, the handle to the connection has to survive handover, so the new connection needs to inherit the handle’s identity, which means the handle needs to be serializable in the rebuild protocol, which means that we need to define a serialization in stc-ng design, which means we need to define what is an identity in stc-ng, which means that we need to pick a wire format of identity and stick to it forever. That’s too early, so no handles, no identity.
  2. If you have a handle, you have a capability. Defining a handle means also defining who can present it and in our context, naturally, it’s going to be “anyone with a file descriptor”, “anyone who can connect to the socket”, “anyone who can present the cookie” and those are 3 different trust models. Not having a handle means that the transport decides who can open the connection.
  3. Finally, a cookie is already a handle and you find yourself with 2 identities in the system, meh.

That being said, if we do a Switch interface with no commitment to identity, lifetime and trust. The interface can push this concern to the transport again, so I feel like the solution is going to look like this:

  1. A facade for org.nixos.Rebuild local to the system
  2. An arbitrary program that makes use of that interface and pick an identity system (mTLS/OIDC/SSH keys/etc.), a lifetime (continuous, discrete, event based, etc.), a trust (objcap/ABAC/RBAC/HMAC-based claims/)

If we split things this way, we never promise something we will hardly keep true if we get it wrong even in presence of an upgrade system. The upgrade is possible iff the switch API is self sufficient, otherwise, we need to solve and do choices on serialization of handles, a “second connection + cookies” is a trick because it throws away the previous handle to have a new handle. I would favor a solution which cost already nothing. :smiley:

1 Like