SystemD user services fail to run on login

I’m using sway from home-manager (but have also enabled it in nixos to have the session entry) and during login almost all the user services fail with cannot open display

From my understanding they start before sway or before sway populates the WAYLAND_DISPLAY variable.

For example network-manager-applet fails to start during login but doing a restart fixes it:

╰─❯ systemctl status --user network-manager-applet.service
× network-manager-applet.service - Network Manager applet
     Loaded: loaded (/home/dzervas/.config/systemd/user/network-manager-applet.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Mon 2024-07-22 20:15:46 EEST; 1h 19min ago
   Duration: 50ms
   Main PID: 2590 (code=exited, status=1/FAILURE)
        CPU: 29ms

Jul 22 20:15:46 desktop systemd[2474]: Started Network Manager applet.
Jul 22 20:15:46 desktop .nm-applet-wrap[2590]: cannot open display:
Jul 22 20:15:46 desktop systemd[2474]: network-manager-applet.service: Main process exited, code=exited, status=1/FAILU>
Jul 22 20:15:46 desktop systemd[2474]: network-manager-applet.service: Failed with result 'exit-code'.
╰─❯ systemctl restart --user network-manager-applet.service
╰─❯ systemctl status --user network-manager-applet.service
● network-manager-applet.service - Network Manager applet
     Loaded: loaded (/home/dzervas/.config/systemd/user/network-manager-applet.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-07-22 22:12:10 EEST; 1s ago
   Main PID: 64358 (.nm-applet-wrap)
      Tasks: 6 (limit: 38334)
     Memory: 9.5M (peak: 10.5M)
        CPU: 96ms
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/network-manager-applet.service
             └─64358 /nix/store/jsv79icwlajx0la8r0dz4pm3yhrn8jsq-network-manager-applet-1.36.0/bin/nm-applet

Jul 22 22:12:10 desktop systemd[2474]: Started Network Manager applet.

Context:

I’m not sure why they fail to start, but I remember that i had sometbing similar with my hand rolled services, I had to set after and partOf properly as seen here. Hopefully this helps debugging.

EDIT: These appear to be already set, home-manager/modules/services/network-manager-applet.nix at 635563f245309ef5320f80c7ebcb89b2398d2949 · nix-community/home-manager · GitHub so not sure from here.

1 Like

that was my first hunch as well (something to do with lifetimes) but from what I see network-manager-applet already has an After=graphical-session-pre.target - home-manager code

EDIT: Maybe it needs After=tray.target?

You need to remember to also enable the sway systemd support, or it won’t import the variables/start the target: Appendix A. Home Manager Configuration Options

Also means you need to use home-manager to configure your sway.

I do both:

  • systemd is enabled here (although it defaults to true anyway)
  • I include the module in home-manager
1 Like

Wayland compositors and systemd just do not seem to mix well (I have similar issues with Hyprland). Even with the systemd integration stuff in Home Manager modules, I still get start-up-order problems with swayidle, swaylock, etc, even if the dependent services have things like After=graphical-session.target in their systemd service files.

I have 2 ways that somewhat work for my machines. I have not actually tested any of the below code with your NixOS config, but hopefully they can point you in the right directions.

1. Systemd activation all the way down

(The harder, more complex route, but perhaps neater because systemd handles all the process launching)

From what I can see, systemd activation of services that depend on long-running processes (such as Wayland compositors) requires some form of inter-process communication originating from the long-running process that signals when the process is ready, at which point dependent processes (such as Network Manager applets) can then launch. Otherwise, systemd has no way of knowing the correct time to launch the dependent processes – The list of systemd’s available activation Types summarizes how systemd determines when a service is considered “finished”. For Wayland compositors that must launch, be ready before other processes begin, and then stay running, the only appropriate types are notify and notify-reload, but these types require internal support from the compositor.

The best way for compositors to fully support systemd activation is by implementing systemd’s sd_notify function and sending a ready signal. This mean the compositor should be started with its own systemd service file, with Type=notify. However, Sway’s developer(s) appears to explicitly avoid implementing sd_notify or some equivalent.

(On a different but related track: having ConditionEnvironment=WAYLAND_DISPLAY in the systemd service file does nothing for resolving start-up-ordering issues. If the environment variable is not set when systemd decides to launch the service, the service simply fails.)

But, at least for the start-up-ordering problem, you can try sticking in sd_notify functionality yourself, using the shell command systemd-notify in the Sway config file.

Unfortunately, sd_notify does not solve the other important issue about propagating environment variables. You may need to insert systemctl --user import-environment <various env vars> (possibly among other annoying rituals) in the right places for this method to work fully.

(Untested) Nix code for all the above:

# https://github.com/swaywm/sway/pull/3486#issuecomment-456292899
wayland.windowManager.sway.extraconfig = ''
  exec systemd-notify --ready || true
''

Add your own systemd service file for Sway in your Home Manager module:

systemd.user.services.sway = {
  Unit = {
    Description = "Sway Wayland Compositor";
    BindsTo = ["sway-session.target"];
  };
  Service = {
    # You may need an `Environment=` line to propagate environment variables to child processes
    Type = "notify";
    NotifyAccess = "all"; # Probably need this because a child process actually sends the notify signal
    ExecStart="${pkgs.sway}/bin/sway"; # Unforunately, this is IFD (import from derivation)
  };
  Install = {
    # This line causes systemd, by itself, to launch Sway at machine boot. Ignore this line if Sway should be started by a login manager or some other way.
    WantedBy = [ "multi-user.target" ]; 
  };
}

Finally, get systemd to launch Network Manager applet strictly after Sway has sent the notification (Note: this does not define the complete network-manager-applet.service but modifies the existing service file defined in Home Manager’s module):

systemd.user.services.network-manager-applet = {
  After = [ "sway.service" ];
};

2. Launch dependent processes through Sway’s config file and not systemd

(The easier route, and probably the one you want if you like to stay sane)

Using Network Manager applet as an example:

Do not start up Network Manager applet directly through systemd (the Home Manager module does this) and instead launch it through Sway.

In the Home Manager part of your NixOS configuration:

systemd.user.services.network-manager-applet.Install = {};

And in the Sway config file:

exec systemctl --user network-manager-applet.service

You may also need systemctl --user import-environment somewhere too, maybe right before the systemctl --user line in the Sway config, to deal with environment variable issues.


Hopefully this mess of notes on the nightmare that is running Wayland compositors on machines with systemd might be helpful.

1 Like

FWIW, I’ve not had any issues with this. As long as the compositor is in charge of starting the .target, there should be no issues that can occur.

Maybe something is starting the target before your WM starts?

Wow, what a writeup, thanks a ton, it would take me years to dig out all that knowledge!

I imagined the second way is a possible fix but let’s leave it aside cause I’ll have to manually disable a bunch of services

what I don’t get for the systemd-notify solution is how adding a new service makes the sway-session.target to wait for the ready notification. Is the BindsTo the magic?

also I’m using a login manager that already calls sway through the regular desktop session mechanism (regreet) and I don’t get how the network-manager-applet will wait for the user service you provided to be ready and not jump the gun and crash. I guess in my case there’s a way to tinker with the desktop session entry to wait for the ready notification?

I’d suspect that’s the underlying issue, personally. It’s worth digging into the startup scripts to see if any of them already start graphical-session.target before sway starts.

An alternative would be renaming the graphical session target, and making sure that any services that depend on it depend on the new name. Then you can be certain that only sway starts the session.

It’s not easy at all to dig into the startup scripts - they’re hidden within the nix store

I’d like to stay away from renaming the graphical session target since I’d had to override every single startup service After

For solution (1) in my post, sway-session.target is almost totally replaced with the new sway.service (a new service unit file that you would create yourself, with a name that could be something else). The BindsTo entry is mostly for compatibility with other services that you installed that need Sway started but is less strict about starting after Sway (Apparently, network-manager-applet.service is not one of these). (Documentation on BindsTo is here. Yes, systemd has dazzlingly many rules governing service inter-dependencies)

The “magic” is simply with adding a After=sway.service entry to network-manager-applet.service (which I forgot to include in the first post. I’ll edit the post. The additional piece is systemd.user.services.network-manager-applet.includes = [ "sway.service" ];

Yes, there could be no issues if the compositor is responsible for starting the target file (same idea as (2) in my post). Those issues I mentioned arise when I tried to go with using systemd to start everything (Solution (1) in my post).

I do know that the Sway project (and Hyprland) do not recommend using systemd to start the compositor, as I described in (1), for the very good reason of avoiding difficulties with propagating environment variables. However, as we see with the issue with network-manager-applet.service, that can cause start-up-order issues with other programs that need to start strictly after the compositor is ready.

Well, the point is that home-manager’s “systemd support” option just adds a line like:

exec systemctl --user start graphical-session.target

to your sway/hyprland config (see here - note that sway-session.target also starts graphical-session.target).

In other words, your configuration was following your approach 2) to begin with, but making systemd figure out what services need to be started based on their config instead of hard-coding the whole list. It also ensures that services still work when you restart sway/hyprland.

sway/hyprland do not have native support for any of this, there’s no other way to do this except your approach 2).

The underlying issue is probably that something else also starts that target, before sway/hyprland have launched. This causes your race condition - my suspicion is that both of you use a display manager or such which is either configured wrong or just doesn’t play nice with wayland because it expects an X11 session where it would be able to run a script in-between X and the WM starting.

Hence my suggestion to change the name of the target to confirm that suspicion - doing it even for just one service would be enough (and you could do so quite easily with an option).

After that we can go dig into what display manager y’all are using and ultimately which startup script needs fixing upstream, rather than all these manual workarounds.

Admittedly, (1) is a significantly different approach from what the upstream Wayland compositors support and what the Nix and Home Manager modules are designed for. It requires ripping out much of the upstream nixpkgs and Home Manager functionality and I just touched the heinous environment variable issues. It really is a different topic than what @dzervas needs to solve his race condition.

@dzervas appears to be using regreet which implies greetd. I don’t see much customization of regreet in GitHub - dzervas/dotfiles so maybe the upstream Nix module could be improved? I do not use display managers and are not familiar with any.

Ok crazy amounts of info and for the past 2 weeks I’ve been digging around nixos (I love it! it really makes sense!). @TLATER wow you’re almost in every forum topic I read in :stuck_out_tongue:

about the topic: from what I understand the root cause is the greeter being marked as the actual graphic target and sway not letting systemd know that it actually started

So what if I completely remove the greeter part an my graphical target is bound to sway? (I’ll test it soon)

If I’m to use greeter I could define a function to apply patches to all the used services? something like:

{ config, ...}: let
  swayTargetPatch = service: service // {
    After = [ "sway.service" ];
  };
in {
  systemd.user.services = map swayTargetPatch (with config.systemd.user.services; [
    network-manager-applet
    blueman-applet
  ]);
}

or maybe I could iterate over all services that have an after set to graphical target?

Kinda, yeah. Regreet tries to support X11 with the whole xinit stack, it’s possible there are some crossed wires there when wayland comes into play. This is still speculation on my end, though.

I think the simplest test would be:

  1. Switch to a tty after a reboot and before signing in
  2. Sign in on the tty
  3. Assert the graphical-session.target is not running yet with systemctl --user status graphical-session.target
  4. Run sway manually without the greeter
  5. Check if systemctl --user status graphical-session.target is now active, and get journalctl -x --user --boot while you’re at it

If that all works fine, maybe repeat the process, except using the greeter to log in this time - that should tell us without a doubt where the issue lies. After that we can look at solutions - either patching systemd units, fixing upstream modules, etc.

Wow just manually running sway fixes everything!

Before executing sway (so regreet/greetd just waiting) all of graphical-target, graphical-target-pre, tray (home-manager’s sway intermediate target) and sway-session targets are not active/running

Exiting sway and re-running it leaves all services exited and on failure and in general is in a semi-broken state

So creating a custom sway service (not target) that binds to sway-session.target and doing the systemd-notify thingy seems like the way to go

thank you both a ton for your info, I’m marking @testplayername 's first answer since the first solution is what I’m describing above

EDIT: Ok using the systemd service provided didn’t work, sway starts but it can’t find any other binary (rofi, wayland, etc.) which is super weird. sway command runs correctly

You are seeing the problem with environment variable propagation with using the (1) approach. Specifically, the PATH that Sway sees does not contain the other commands that you have installed in NixOS.

My somewhat-hacky solution is using Environment= in the systemd service file for Sway as hinted at in the comment, and explicitly listing the packages containing the commands you want to launch directly from Sway. Below is the same service file for Sway from (1) from my first post, with two main additions: a list of packages, each of which contain the aforementioned commands; and an Environment= line in the Sway service file that sets the PATH that Sway sees that includes the paths to the commands.

let 
  dependencies = [
    pkgs.rofi
    # Put in other packages that Sway needs to know about
  ];

in

  # ... Other Sway configuration...

systemd.user.services.sway = {
  Unit = {
    Description = "Sway Wayland Compositor";
    BindsTo = ["sway-session.target"];
  };
  Service = {
    # Hacky solution to the problem with propagating environment variables
    Environment = "PATH=/run/wrappers/bin:${pkg.lib.makeBinPath dependencies}";
    Type = "notify";
    NotifyAccess = "all"; # Probably need this because a child process actually sends the notify signal
    ExecStart="${pkgs.sway}/bin/sway"; # Unforunately, this is IFD (import from derivation)
  };
  Install = {
    # This line causes systemd, by itself, to launch Sway at machine boot. Ignore this line if Sway should be started by a login manager or some other way.
    WantedBy = [ "multi-user.target" ]; 
  };
}

I thought that the dbus hack that home-manager does would be enough for the env vars:

dbus-update-activation-environment --systemd DISPLAY WAYLAND_DISPLAY SWAYSOCK XDG_CURRENT_DESKTOP XDG_SESSION_TYPE NIXOS_OZONE_WL XCURSOR_THEME XCURSOR_SIZE; systemctl --user reset-failed && systemctl --user start sway-session.target && swaymsg -mt subscribe '[]' || true && systemctl --user stop sway-session.target

defined here

is there a way to populate the env vars during startup? I don’t want to have to define all the dependencies one by one

Yeah, that’s why launching WMs with systemd never took off, you’re missing out on all the rcfiles your system runs because those don’t happen in systemd services. You could manually trawl through all possible environment variables and add an import to your sway startup script, but you’d need to write a custom session and enumerate all variables that could possibly affect anything. If you really want to do this, you’ll at least need $PATH in there, but this will probably cause other subtle issues because you’re basically skipping system init.

Just running a login shell to launch sway in that can work too, but the subtle differences in environment will probably come back to bite you eventually, and by then it’ll be very hard to understand what’s wrong.

Well, fair enough, that means your DM is in fact not set up correctly. This isn’t that surprising, it’s the only greetd implementation that tries to support session files. It isn’t really compliant with the X11 standards, and wayland standards simply don’t exist, so there will be problems. There will be some upstream work necessary to fix this, we should file an issue. Did you ever capture logs from a regreet session? Those would be helpful when filing, and later debugging.

Meanwhile, personally, regreet’s mild brokenness is why I use gtkgreet with a hand-written script.

Yeah, this one is a pretty obvious bug, I’d never noticed because I don’t re-enter my session very often.

sway-session.target stays active after you quit sway, so when you start it back up the target is already running and systemd doesn’t think any services need to be restarted.

You can fix this manually by just running systemctl --user stop sway-session.target (which will also stop graphical-session.target) after sway exits. Can even add that as a band-aid to any scripts you run with non-regreet greetd implementations. This is probably pretty trivial to fix upstream, I’ll file an issue and figure it out sometime this weekend.

So… The issues are ultimately caused by upstream bugs. The workarounds suggested here so far at best tape over the problems, and will cause subtle integration issues down the line, as workarounds always do. I’d definitely prefer we just fix the bugs instead, but you’re of course free to decide what works best for you.

1 Like