Able to run program from shell, systemd unit fails

Help
1

Hi all. I’m relatively new to NixOS and I’ve run into this problem I cannot figure out on my own:

What are the differences when running a command from the shell vs running it from a systemd unit?

I’m trying to run a Pleroma instance. I am able to run it on the shell when logged in as the pleroma user. However, when I try to run it via a ported systemd unit, I am getting errors.

Here is the relevant section of /etc/nixos/configuration.nix:

    systemd.services.pleroma_backend = {
    enable = true;
    description = "Pleroma Backend";
    after = [ "network.target" "postgresql.service" ];
    environment = {
      HOME = "/home/pleroma";
      MIX_ENV = "prod";
    };
    serviceConfig = {
      User = "pleroma";
      WorkingDirectory = "/home/pleroma/pleroma";
      ExecStart = "${pkgs.elixir}/bin/elixir -S ${pkgs.elixir}/bin/mix phx.server";
      ExecReload = "/run/current-system/sw/bin/kill $MAINPID";
      KillMode = "process";
      Restart = "on-failure";
    };
    wantedBy = [ "multi-user.target" ];
  };

And the output of systemctl cat pleroma_backend:

# /nix/store/pa1q1cwbgsxl9ylc8al38gpp1klc4qcj-unit-pleroma_backend.service/pleroma_backend.service
[Unit]
After=network.target postgresql.service
Description=Pleroma Backend

[Service]
Environment="HOME=/home/pleroma"
Environment="LOCALE_ARCHIVE=/nix/store/vg0s4sl74f5ik64wrrx0q9n6m48vvmgs-glibc-locales-2.26-131/lib/locale/locale-archive"
Environment="MIX_ENV=prod"
Environment="PATH=/nix/store/cb3slv3szhp46xkrczqw7mscy5mnk64l-coreutils-8.29/bin:/nix/store/364b5gkvgrm87bh1scxm5h8shp975n0r-findutils-4.6.0/bin:/nix/store/s63b2myh6rxfl4aqwi9yxd6rq66djk33-gnugrep-3.1/bin:/nix/store/navldm477k3ar6cy0zlw9rk43i459g69-gnused-4.4/bin:/nix/store/f9dbl8y4zjgr81hs3y3zf187rqv83apz-systemd-237/bin:/nix/store/cb3slv3szhp46xkrczqw7mscy5mnk64l-coreutils-8.29/sbin:/nix/store/364b5gkvgrm87bh1scxm5h8shp975n0r-findutils-4.6.0/sbin:/nix/store/s63b2myh6rxfl4aqwi9yxd6rq66djk33-gnugrep-3.1/sbin:/nix/store/navldm477k3ar6cy0zlw9rk43i459g69-gnused-4.4/sbin:/nix/store/f9dbl8y4zjgr81hs3y3zf187rqv83apz-systemd-237/sbin"
Environment="TZDIR=/nix/store/brib029xs79az5vhjd5nhixp1l39ni31-tzdata-2017c/share/zoneinfo"

ExecReload=/run/current-system/sw/bin/kill $MAINPID
ExecStart=/nix/store/l1i8bfi23psnrh9jsj9707drlw2w17lj-elixir-1.6.2/bin/elixir -S /nix/store/l1i8bfi23psnrh9jsj9707drlw2w17lj-elixir-1.6.2/bin/mix phx.server
KillMode=process
Restart=on-failure
User=pleroma
WorkingDirectory=/home/pleroma

And the truncated output from journalctl -u pleroma_backend

Jul 16 18:31:31 cloud systemd[1]: Started Pleroma Backend.
Jul 16 18:31:31 cloud elixir[21905]: ** (Mix.Config.LoadError) could not load config config/config.exs
Jul 16 18:31:31 cloud elixir[21905]:     ** (ErlangError) Erlang error: :enoent
Jul 16 18:31:31 cloud elixir[21905]:     (elixir) lib/system.ex:622: System.cmd("git", ["rev-parse", "HEAD"], [])
Jul 16 18:31:31 cloud elixir[21905]:     (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
Jul 16 18:31:31 cloud elixir[21905]:     (stdlib) erl_eval.erl:269: :erl_eval.expr/5
Jul 16 18:31:31 cloud elixir[21905]:     (stdlib) erl_eval.erl:438: :erl_eval.expr/5
Jul 16 18:31:31 cloud elixir[21905]:     (stdlib) erl_eval.erl:122: :erl_eval.exprs/5
Jul 16 18:31:31 cloud elixir[21905]:     (elixir) lib/code.ex:192: Code.eval_string/3
Jul 16 18:31:31 cloud elixir[21905]:     (mix) lib/mix/config.ex:188: Mix.Config.read!/2
Jul 16 18:31:31 cloud systemd[1]: pleroma_backend.service: Main process exited, code=exited, status=1/FAILURE
Jul 16 18:31:31 cloud systemd[1]: pleroma_backend.service: Failed with result 'exit-code'.
Jul 16 18:31:31 cloud systemd[1]: pleroma_backend.service: Service hold-off time over, scheduling restart.
Jul 16 18:31:31 cloud systemd[1]: pleroma_backend.service: Scheduled restart job, restart counter is at 5.
Jul 16 18:31:31 cloud systemd[1]: Stopped Pleroma Backend.
Jul 16 18:31:31 cloud systemd[1]: pleroma_backend.service: Start request repeated too quickly.
Jul 16 18:31:31 cloud systemd[1]: pleroma_backend.service: Failed with result 'exit-code'.
Jul 16 18:31:31 cloud systemd[1]: Failed to start Pleroma Backend.

I know the output points to a non-existent config/config.exs file, but the file is definitely there:

[pleroma@cloud:~]$ ls -la /home/pleroma/pleroma/config/config.exs
-rw-r--r-- 1 pleroma users 2761 Jul 16 08:52 /home/pleroma/pleroma/config/config.exs

and like I mentioned, the program works when run from the shell:

[pleroma@cloud:~/pleroma]$ MIX_ENV=prod /nix/store/l1i8bfi23psnrh9jsj9707drlw2w17lj-elixir-1.6.2/bin/elixir -S /nix/store/l1i8bfi23psnrh9jsj9707drlw2w17lj-elixir-1.6.2/bin/mix phx.server
19:47:16.447 [info] Running Pleroma.Web.Endpoint with Cowboy using http://0.0.0.0:4000
19:47:16.494 [info] Gopher server disabled

Maybe I’m approaching “porting” the unit wrong way? I have also tried this configuration:

/etc/nixos/configuration.nix:

  systemd.services.pleroma_backend = {
    enable = true;
    description = "Pleroma Backend";
    after = [ "network.target" "postgresql.service" ];
    environment = {
      HOME = "/home/pleroma";
      MIX_ENV = "prod";
    };
    path = [ pkgs.elixir ];
    script = "mix phx.server";
    serviceConfig = {
      User = "pleroma";
      WorkingDirectory = "/home/pleroma/pleroma";
      ExecReload = "/run/current-system/sw/bin/kill $MAINPID";
      KillMode = "process";
      Restart = "on-failure";
    };
    wantedBy = [ "multi-user.target" ];
  };

systemctl cat pleroma_backend:

# /nix/store/48dgyp8hcm6rikg8r0b2qq39mls9n82x-unit-pleroma_backend.service/pleroma_backend.service
[Unit]
After=network.target postgresql.service
Description=Pleroma Backend

[Service]
Environment="HOME=/home/pleroma"
Environment="LOCALE_ARCHIVE=/nix/store/vg0s4sl74f5ik64wrrx0q9n6m48vvmgs-glibc-locales-2.26-131/lib/locale/locale-archive"
Environment="MIX_ENV=prod"
Environment="PATH=/nix/store/cb3slv3szhp46xkrczqw7mscy5mnk64l-coreutils-8.29/bin:/nix/store/364b5gkvgrm87bh1scxm5h8shp975n0r-findutils-4.6.0/bin:/nix/store/s63b2myh6rxfl4aqwi9yxd6rq66djk33-gnugrep-3.1/bin:/nix/store/navldm477k3ar6cy0zlw9rk43i459g69-gnused-4.4/bin:/nix/store/f9dbl8y4zjgr81hs3y3zf187rqv83apz-systemd-237/bin:/nix/store/cb3slv3szhp46xkrczqw7mscy5mnk64l-coreutils-8.29/sbin:/nix/store/364b5gkvgrm87bh1scxm5h8shp975n0r-findutils-4.6.0/sbin:/nix/store/s63b2myh6rxfl4aqwi9yxd6rq66djk33-gnugrep-3.1/sbin:/nix/store/navldm477k3ar6cy0zlw9rk43i459g69-gnused-4.4/sbin:/nix/store/f9dbl8y4zjgr81hs3y3zf187rqv83apz-systemd-237/sbin"
Environment="TZDIR=/nix/store/brib029xs79az5vhjd5nhixp1l39ni31-tzdata-2017c/share/zoneinfo"

ExecReload=/run/current-system/sw/bin/kill $MAINPID
ExecStart=/nix/store/95kbgqwpfx3vckafvnnigjm94a9ryd1f-unit-script/bin/pleroma_backend-start
KillMode=process
Restart=on-failure
User=pleroma
WorkingDirectory=/home/pleroma/pleroma

cat /nix/store/95kbgqwpfx3vckafvnnigjm94a9ryd1f-unit-script/bin/pleroma_backend-start:

#! /nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/bash -e
mix phx.server

Which I would assume more closely resembles running from the shell, yet I get the exact same result.

I don’t know how else to troubleshoot the issue. Any help is greatly appreciated.

2

Going from that line, I will hazard a guess: git is needed for this service, and you have git available in your systemPackages.

systemd services have quite a strict relation with PATH[1] (and most environment variables). You will need to add the required dependencies to the .path option for your service.

1 Like
3

Double check your working dirctory. There seems to be mismatch between some of the confings.

4

Gah! That was it! Adding pkgs.git to path fixed it. Thank you so much!

My working configuration, in case it helps somebody in the future:

  systemd.services.pleroma_backend = {
    enable = true;
    description = "Pleroma Backend";
    after = [ "network.target" "postgresql.service" ];
    environment = {
      HOME = "/home/pleroma";
      MIX_ENV = "prod";
    };
    path = [ pkgs.elixir pkgs.git ];
    serviceConfig = {
      User = "pleroma";
      WorkingDirectory = "/home/pleroma/pleroma";
      ExecStart = "${pkgs.elixir}/bin/mix phx.server";
      ExecReload = "/run/current-system/sw/bin/kill $MAINPID";
      KillMode = "process";
      Restart = "on-failure";
    };
    wantedBy = [ "multi-user.target" ];
  };

Thanks again!

3 Likes
5

This wasn’t the issue, but thanks for replying.

6

Sorry for the unexpected request. I’m trying to run Pleroma on my NixOS server, but I can’t define declarative server installation. If there’s no sort of sensible information, could you please provide full config, with which you deploy Pleroma(not only systemd service, but server binaries provision)?

7

Hi @anon22412468. Unfortunately, I don’t run Pleroma anymore. I see you’ve already managed to get a hold of someone smarter than me in another thread. Sorry for the delay in relpying, and best of luck!