Help: Network magic on boot, or systemd one time script? (docker / macvlan)

Help
1

I’m trying to configure docker macvlan - basically replicating my blog post: Docker and macvlan networking (IPv4) – Roo's View - except on Nix.

From the docker macvlan setup - I think I’m good to go. That’s just a docker persistent state / network setup.

However, there is a boot time script I used to run using cron @reboot function to massage the network into routing magic

Here is the script

# run as root - once, on boot
ip link add myNewNet-shim link enp3s0 type macvlan mode bridge
ip addr add 192.168.1.67/32 dev myNewNet-shim
ip link set myNewNet-shim up
ip route add 192.168.1.64/30 dev myNewNet-shim

I think there are two ways to make this happen on NixOS.

Path 1 - networking magic. That will perform the same function - to cause the routes to exist. This is well beyond me honestly, but I’m open to advice if someone knows how.

Path 2 - a mechanism to run the script at boot time, once. I think I can use systemd to do this, but I’m hitting path/binary visibility issues.

I’ve added this to my /etc/nixos/configuration.nix file

  systemd.services.macvlan-host-routing = {
    serviceConfig.Type = "oneshot";
    wantedBy = [ "docker.service" ];
    script = ''
      echo $PATH > /tmp/foobar;
      echo 'more than one thing' >> /tmp/foobar
    '';
  };

Which works - upon boot I see the /tmp/foobar file created. Issuing nixos-rebuilds do not appear to cause this to run again (desired state, I only need to fiddle the network once)

However, the ip command I want to run – isn’t in the $PATH. Searching for which package the ip command is in… is eluding me.

I think I want my systemd bit to look like this

  systemd.services.macvlan-host-routing = {
    serviceConfig.Type = "oneshot";
    wantedBy = [ "docker.service" ];
    script = ''
      ip link add myNewNet-shim link enp3s0 type macvlan mode bridge;
      ip addr add 192.168.1.67/32 dev myNewNet-shim;
      ip link set myNewNet-shim up;
      ip route add 192.168.1.64/30 dev myNewNet-shim;
    '';
  };

but of course – the `ip’ command cannot be found, and this fails to run.

If there is a 3rd path - I’m open to it as well.

2

It’s like rubber duck debugging - I keep answering my own questions.

$ which ip
/run/current-system/sw/bin/ip

$ ls -l /run/current-system/sw/bin/ip
lrwxrwxrwx 6 root root 66 Dec 31  1969 /run/current-system/sw/bin/ip -> /nix/store/69c1w82hwk2ki1xapdci0hdsvrhl232g-iproute2-6.14.0/bin/ip

From this I can squint and figure out that the package is iproute2

This let’s me path it in the systemd setup … thus

systemd.services.macvlan-host-routing = {
    serviceConfig.Type = "oneshot";
    wantedBy = [ "docker.service" ];
    script = ''
      ${pkgs.iproute2}/bin/ip link add myNewNet-shim link enp3s0 type macvlan mode bridge;
      ${pkgs.iproute2}/bin/ip addr add 192.168.1.67/32 dev myNewNet-shim;
      ${pkgs.iproute2}/bin/ip link set myNewNet-shim up;
      ${pkgs.iproute2}/bin/ip route add 192.168.1.64/30 dev myNewNet-shim;
    '';
  };

Will work… but is it the best approach with NixOS?

3

I would add a check if the network already exists and only then create it. It should persist reboots anyway.

You can find the package that provides ip and add it to the path of the service: NixOS Search or you could do something like:

${lib.getExe' iproute2 "ip"} …

Which retrieves the store path where the binary is found.

2 Likes
4

You can also use the options to create the network: NixOS Search

I haven’t done this but it should not be hard to do as you already know how it is done via a script.

1 Like
5

Interesting. I think - based on what I saw, this is only going to define the macvlan network - which is a 1 time operation really. It doesn’t solve the routing from the host to the containers that have IPs on the macvlan which is the script I’m trying to run at boot time.

I think what I have above will work - and is probably the right way to do it. This is really a collision of the ‘docker way’ and the ‘nix way’.

Docker macvlan networking is magic - it let’s you treat docker containers a bit like they are VMs. IMO - the industry as a whole hasn’t yet really figured out the whole software-defined-networking problem yet… tailscale and the like seem to be closest.

Thank you for your replies - I appreciate that you took time to look at my post and comment. It’s pointed me at a few things to learn more about.

1 Like
6

Yes I use a docker macvlan on my last host that I haven’t gotten around to porting to NixOS. That is still on my todo list.

I also use this one-shot systemd service philosophy to for example create a pod for podman. So yes i would agree it is the Nix way I just don’t know if there is another way because I haven’t gotten around to testing that.

7

Well - your mileage may vary - but I seem to have been successful.

Overall - I’m following my blog post on the subject: Docker and macvlan networking (IPv4) – Roo's View

Now, I’ve taken the easy path and declare my NixOS machine to have docker via the single line in my config

  virtualisation.docker.enable = true;

It is possible I could declare the macvlan network in the config - but I just cooked up a script which I ran on the machine once, resulting in a persistent docker network (my macvlan network) – oh and that script was literally 1 line – the creation of the docker macvlan network.

Then the only magic I needed, was a “one shot” systemd service to run on boot

systemd.services.macvlan-host-routing = {
    serviceConfig.Type = "oneshot";
    wantedBy = [ "docker.service" ];
    script = ''
      ${pkgs.iproute2}/bin/ip link add myNewNet-shim link enp3s0 type macvlan mode bridge;
      ${pkgs.iproute2}/bin/ip addr add 192.168.1.67/32 dev myNewNet-shim;
      ${pkgs.iproute2}/bin/ip link set myNewNet-shim up;
      ${pkgs.iproute2}/bin/ip route add 192.168.1.64/30 dev myNewNet-shim;
    '';
  };

Reboots result in my macvlan network still existing - AND with the systemd magic above, I can see my containers from the host (the container could always see the host).

8

Use networking.macvlans to create an interface.

Your script is not idempotent and not ordered after the interface it depends on, so it’s pretty much guaranteed that it will break sooner or later, like if the service is restarted or the network configuration changes.

9

I agree that the script is not idempotent, however it should run once on boot thus it shouldn’t be a problem. Still, I’ll consider how I might write some logic there to make it idempotent which I agree would be safer.

networking.macvlans I think does something totally different from my script - the script is not trying to define a macvlan at all, only provide a route from the host system into that macvlan network. In fact, it’s not clear that networking.macvlans addresses the thing I’m trying to do which is to enable macvlans in docker.

Again, I suspect the challenge is that the ‘docker way’ is interfering a bit with the ‘nixos way’.

I do appreciate the dialog here - and it does feel that I’m not doing this quite right - but I would request that you read the blog post linked at the top of this to make sure your comments actually apply to this situation. (or if someone can point me at a tutorial / article on networking.macvlans – I’ll go learn something)

10

There is a quite good example on macvlans in the tests which shows how it is supposed to work: nixpkgs/nixos/tests/containers-macvlans.nix at 62f746b630d5189b2eedc19688ddcfb8021c1c71 · NixOS/nixpkgs · GitHub

Adding routes should be as easy as: NixOS Search

11

The first command of your script is creating a macvlan interface, which is what networking.macvlans does.

The declarative way to do what your script is trying to do is something like this:

{
  networking.macvlans."myNewNet-shim" = {
    mode = "bridge";
    interface = "enp3s0";
  };

  networking.interfaces."myNewNet-shim".ipv4 = {
    addresses = [{ address = "192.168.1.67"; prefixLength = 32; }];
    routes = [{ address = "192.168.1.64"; prefixLength = 30; }];
  };

}

I haven’t read the blog and I don’t know much about Docker, but I’m the maintainer of the networking.interfaces module, so I’m pretty sure what these options do. Your script is actually pretty similar to the “scripted” networking implementation (you can check for yourself: createMacvlanDevice, configureAddrs), except that it lacks the proper dependencies and the clean up part.

3 Likes
12

Tanks @rnhmjoj for your detailed answer. It is exactly how I thought it should work. I will bookmark this for future reference.

1 Like
13

:bowing_man:

Thank you for your more detailed answer - very helpful. I’ve tagged it as the ‘solution’ but have not yet verified that this will work (but am fairly confident it will)

I’m still learning NixOS so it takes me longer to figure these things out. Hopefully this thread will contribute to the community understanding of how things work.

Thank you @rnhmjoj for your work on NixOS as well.

[Update]

I can confirm that the ‘solution’ tagged above by @rnhmjoj works. There is one small difference, but I don’t think it is a problem unless someone here spots an issue

With my ‘script’ the ip route shows this

192.168.1.64/30 dev myNewNet-shim scope link

vs. the networking.macvlans approach that gives me

192.168.1.64/30 dev myNewNet-shim proto static scope link

There is the addition of proto static - which is probably fine? I’ll run with it and see if there are any problems.

14

From ip-route(8):

protocol RTPROTO
the routing protocol identifier of this route. RTPROTO may be a number or a
string from $out/share/iproute2/rt_protos or /etc/iproute2/rt_protos (has
precedence if exists). If the routing protocol ID is not given, ip assumes
protocol boot (i.e. it assumes the route was added by someone who doesn’t
understand what they are doing). Several protocol values have a fixed
interpretation. Namely:

  • redirect - the route was installed due to an ICMP redirect.

  • kernel - the route was installed by the kernel during autoconfiguration.

  • boot - the route was installed during the bootup sequence.
    If a routing daemon starts, it will purge all of them.

  • static - the route was installed by the administrator to override dynamic
    routing. Routing daemon will respect them and, probably, even advertise
    them to its peers.

  • ra - the route was installed by Router Discovery protocol.

In practice it’s unlikely to make any difference, but you should set static
when manually inserting static routes.

1 Like
15

Thank you again for the detailed and insightful response.

I’ve created an update blog post to capture this here: NixOS + Docker with MacVLAN (IPv4) – Roo's View

1 Like