How to append string to existing value from a module

Help
1

Hi have the feeling I should know this after using Nix for 3 years, but I cannot find it in the documentation…

I want to update the Nitter service to extend the ExecPreStart with an extra command.
ExecPreStart is already defined in the module, I just want to concatenate a string to whatever is there. (The extra command basically sleeps the startup of the service until Twitter is actually available, otherwise Nitter does not work)

So I tried using:

  systemd.services.nitter.serviceConfig.ExecStartPre =  ''
      ${pkgs.busybox}/bin/sh -c '(while ! ${pkgs.busybox}/bin/nc -z -v -w1 twitter.com 443 2>/dev/null; do echo "Waiting for Twitter to be reachable..."; sleep 2; done); sleep 2'
    '';

But this gives “error: The option `systemd.services.nitter.serviceConfig.ExecStartPre’ has conflicting definition values”

I remember something about mkAfter and mkMerge etc. but cannot find the documentation of these functions.

How to concatenate the string?

2

If the option doesn’t have a mergeable type, which I think this one doesn’t, the only option is to completely override it, replicating the original value. That or disable the module that originally set it and import a modified version in its place.

3

Doesn’t ExecStartPre take multiple entries? So if it were a list in nix as maybe it should be this could work…

4

If all attempts to set ExecStartPre set it to a list, the lists should be concatenated, but the nitter module doesn’t set it to a list, or use systemd.services.<name>.preStart, which would also merge properly, so there’s nothing you can do.

5

Right. Which is why using a list in ExecStartPre is usually a good idea. PRs welcome?

1 Like
6

thanks for the answers.
So I conclude what I want is not possible without copying the whole module…

Based on this thread and my experiences to try to get it to work, I think there are three separate issues. I don’t use Github, if anyone with Github account can copy/paste these into an issue, that would be great!

Issue 1. Module functions not in auto-generated documentation

Some important/often used library functions that used to define a NixOS configuration are not documented in the automatically generated manuals on nixos.org

For example, mkMerge, mkAfter, etc. are defined in modules.nix, and are nicely documented in the source code, but are not rendered in the HTML manuals.

Workaround:
As far as I know, the only way a regular NixOS user can read about these functions is to find the correct source file and read the source.

Issue 2. Policy to make service definitions extendable

Make it a generic policy for all modules to make options like ExecStartPre and others always mergable.

Not sure how to describe this correctly and what the procedure it to this (RFC?). See comments from @aanderse @tejing . I’ll leave it up to actual NixOS devs to decide if this is an issue or not.

Issue 3. When nitter service starts too early, it will not work

Perhaps for @erdnaxe to decide
This is specific to the Nitter module. If the systemd service starts before the network is fully up, Nitter will not be able to fetch a token.

and you will always get this message:

Instance has been rate limited.
Use another instance or try again later.

Possible solution:

Extende the ExecStartPre with something like:

systemd.services.nitter.serviceConfig.ExecStartPre = ''
   ${pkgs.busybox}/bin/sh -c '(while ! ${pkgs.busybox}/bin/nc -z -v -w1 twitter.com 443 2>/dev/null; do
        echo "Waiting for Twitter to be reachable...";
        sleep 2;
      done); sleep 2'
''

or a nicer variant of this (a ‘Twitter-reachable’ target?)

Workaround:

systemctl restart nitter.service

Thanks everyone!

7

They’re all mentioned in the nixos manual, though not in one consolidated location.

There’s a network-online.target. An after dependency on that may do the trick. On my system that depends on dhcpcd being up, but I don’t know the details of exactly how that works. (Is it only considered “up” when you actually have an ip address? The target suggests maybe…)

8

Thank you @kvtb for the bug report, I opened Nitter is unable to fetch a token if service starts before network is fully up · Issue #208797 · NixOS/nixpkgs · GitHub to keep track of it. I never encounter such issue on my VPS, but maybe my network stack is faster to get up.

As explained at NetworkTarget, maybe we should replace network.target by network-online.target.

9

They’re all mentioned in the nixos manual, though not in one consolidated location.

do you have links, because I did use the search function but could not find it

10

I’ve tried to make network-online as the target, by overriding the default of the module, but unfortunately it did not solve the problem. Perhaps reachability of twitter is not a requirement or cause of the token issue.

Here’s the output output of journalctl -eu nitter:

Jan 08 12:52:51 laptop systemd[1]: Started Nitter (An alternative Twitter front-end).
Jan 08 12:52:51 laptop systemd[1]: Starting Nitter (An alternative Twitter front-end)...
Jan 08 12:52:52 laptop nitter[1507]: Starting Nitter at http://localhost
Jan 08 12:52:52 laptop nitter[1507]: Connected to Redis at localhost:6379
Jan 08 12:52:52 laptop nitter[1507]: fetching token failed: Resource temporarily unavailable
Jan 08 12:52:52 laptop nitter[1507]: Additional info: Temporary failure in name resolution
Jan 08 12:58:16 laptop nitter[1507]: RateLimitError: rate limited
Jan 08 12:58:24 laptop systemd[1]: nitter.service: Deactivated successfully.
Jan 08 12:58:24 laptop systemd[1]: Stopping Nitter (An alternative Twitter front-end)...
Jan 08 12:58:24 laptop systemd[1]: Stopped Nitter (An alternative Twitter front-end).
Jan 08 12:58:24 laptop systemd[1]: nitter.service: Consumed 825ms CPU time, received 9.9K IP traffic, sent 286.5K IP traffic.
Jan 08 12:58:24 laptop systemd[1]: Starting Nitter (An alternative Twitter front-end)...
Jan 08 12:58:24 laptop systemd[1]: Started Nitter (An alternative Twitter front-end).
Jan 08 12:58:24 laptop nitter[3007]: Starting Nitter at http://localhost
Jan 08 12:58:24 laptop nitter[3007]: Connected to Redis at localhost:6379
  • Jan 08 12:52 is when the laptop booted and nitter started
  • Jan 08 12:58:16 is first access to Nitter (user (=me) sees error)
  • Jan 08 12:58:24 is systemctl restart nitter.service
11

The “option definitions merging” section talks about mkMerge, the “modularity” section mentions mkBefore, and mkAfter comes up in a code snippet once, though it isn’t explained.

12

Thanks, do you think my suggestion to create an issue for this to also include it in the auto-generated docs is invalid? Because almost all other library functions are included in the docs.

(I mean, the fact that I as regular user has spent so much time on finding out how it works is maybe an indication some documentation is missing)

13

Not at all. It’s completely valid. We need a section in the documentation that presents an accurate model of what the nixos module system actually does to merge the attrsets returned by the modules. It’s nearly impossible to find that information, and pretty difficult to discover (the code is very dense). It should explain the roles of both priority (mkForce, mkDefault) and order (mkAfter, mkBefore), and how mkIf and mkMerge interact with the module system as well.

1 Like
14

I came across this issue when trying to find a solution to

Is there an effort/proposal to create a function like mkOrder that will treat Strings in different modules as mergeable?

15

It’s not a technical limitation. It would be easy to do so, but it wouldn’t be correct to do so. The general string type exists specifically for cases where merging doesn’t make sense. If merging does make sense, and the module author still used the string type, then the module author chose poorly. There are a variety of other types that can be used if merging makes sense, but in many cases, the most useful one is simply listOf str. Changing how you consume the option to encode the list in the appropriate way (separated by spaces or :, etc) generally makes more sense than inventing a new type for every special case.