As mentioned in the NixCon 2019 lightning talks, I want to discuss some ideas around the migration to networkd in NixOS. There are some new questions that have come up since my talk about networkd at NixCon 2018 and I think it would be wise to discuss them with more people from the community before starting to implement them.
We want to host a networkd sprint for everyone that is interested in discussing, implementing some ideas or just want to test networkd to possibly discover more issues. The ideal outcome would be an RFC for a redesign of the networking module for better composition and extensibility. We also have to decide what to do with scripted networking since we will most probably include a lot more of the features available in networkd. My goal would be to have networkd as a viable option for 20.03 and make it the default in at least 20.09 (including breaking compatibility for at least some of the networking options).
It will take place on a weekend at the new Mayflower office in Munich (Landsberger Strasse 314, OSM) on Saturday 2019-11-23 and the following Sunday. You can also already visit us on Friday. The next public transport station is Laim via the S-Bahn. Food and drinks will be provided for the duration of the sprint at least.
There will also be the possibility of joining remotely so you don’t necessarily need travel to Munich but that would of course simplify communication. If you’re unsure about accommodation, we can also host you in our office since we have a shower available (please contact us at nix@mayflower.de). There is a cheap hostel just a few 100 meters away (A&O Hostel) and relatively cheap hotels like the Super 8 Munich City West that are also close and in walking distance.
Office will be open at 10:00 CET on both days. First session on Saturday starts at 12:00 CET / 11:00 UTC.
We coordinate efforts on IRC in #nixos-systemd on Freenode. Please join, especially if you’re remote.
I was hoping for some more votes. To break the tie I would choose 2019-11-23 since we have our company christmas party the day before 2019-12-14. I’ll edit the first post to that effect.
It would be awesome if anybody else who wants to attend physically apart from the dudle votes would either mention it here or contact us at nix@mayflower.de.
I was wondering if there is anywhere where the usecases you’ll be trying to support with this sprint are listed currently? Might I be able to provide some feedback on usecases for consideration in the design possibly if I don’t see something I’m currently doing or would like to do covered there?
I have what might be a somewhat unique setup to support a virtual domain across some VMs currently supported with NetworkManager and unbound. I’ve also struggled with the coffee shop/hotel wifi terms and conditions page use case.
I don’t think I’ll be able to attend virtually based on schedule so hoping to provide the feedback up front.
Thanks a lot for asking this. That would be very welcome since every use case would help us to design a networking module that just works for everyone without resorting to hacky bash scripting. If anybody else wants to chime in, please do!
It would be great if you and anybody else could sketch how you are using the networking module along the following questions.
Are you working with different routing tables or networking namespaces or do you want to?
Are you using networking.localCommands and for what?
i.e. policy routing with ip rule, creating some special interface types, adding special routes, …
If you’re using dhcp, do you have some special dhcpcd configuration?
What’s your stance on predictable interface names? Do you like specifying hardware interfaces by their predictable interface name or would you rather match them in another way i.e. via mac address?
Since systemd-resolved should also be introduced together with systemd-networkd, what dns resolving oddities are you dealing with and how you’re solving it currently?
i.e. do you have an internal domain and need split dns via a vpn, that typical free wifi problem resolving problem if you have always-on-vpn
Anything else you have additionally set up regarding networking in your NixOS configuration would be good to know, i.e. services that configure some aspect of the Linux networking stack that are not already part of the networking module
And some extra questions for some work after networkd but which we might need to consider for the networking module:
Are you using the networking.firewall module? Since we know it’s quite limited, what features or options are you missing? You can think big here.
Are you using the networking.nat module and what are your thoughts on that?
This might not be a useful data point (apologies if not) but just in case it ends up being a useful:
I’ve used the networking module to force the traffic of some users through a VPN (and drop any traffic from those users if the VPN isn’t available).
It made pretty heavy use of the networking module (everything from making routing tables to maintaining iptables chains), you can find it here, hopefully there’s something in there you can draw inspiration for some improvements from.
A bit to consider: How does networkd work together with other pieces of software which fuzz with the running network config and how does NixOS cover the whole situation:
I’m currently only using NixOS on my laptop and I think it would be nice if the following scenarios would be considered:
Hot plug support and automatic switching between wifi/cable (if this even needs consideration. I just remember having some problems in the past, but it might’ve been because of badly supported hardware)
The classic VPN routing problem with public networks
Rather simple Virtual Networks for development with kvm/virtualbox/etc.
The possibility to configure stuff very flexible and quickly. E.g.: I once had to debug a broken L3-Switch with a debian laptop. I added two virtual interfaces on one physical adapter and quickly rotated through various VLANS while staying connected to my office network through a vpn tunnel on another network interface. Having to rebuild the local config everytime I change the VLAN is not what I want, but I also still want to be connected to my office…
I don’t know if this is already given or needs some consideration, but being able to keep existing (static) network configurations while temporarily tinkering around with iproute2 tools would be really nice
I also think that having one specific networking.firewall module for rather simple configurations is a great thing and I love it. It makes the setup of things like webservers very easy.
It would be great if that module also works ‘‘out of the box’’ with other software like Docker, VPN servers, etc. (which I assume it doesn’t at moment?) but I imagine that could become a bit of a rabbit hole, as such software may add/remove network interfaces on their own…
This first usecase is somewhat predicated on the assumption that systemd-resolved will be used as it seems somewhat linked to systemd-networkd but not totally required.
I run a VM with a static IP which has a DNS server running a zone for other VMs. On nixos I currently have unbound configured with a private-domain and routes requests to the static IP of the bootstrap VM. Since NetworkManager has unbound set I think it configures resolvconf to write /etc/resolv.conf with nameserver 127.0.0.1. So requests are made to unbound and then it routes to the internal DNS server or a global one depending on the domain in question.
This seems entirely possible with resolved without having an additional service as it seems to have some private domain features built in.
I’m currently using nmcli to join various wifi networks. Adding the derived keys to the configuration didn’t seem like a great idea for static configuration as I think they’ll end up in the store and becomes world readable. Occasionally I have to join networks for a short period of time and then forget them. Mixed managed an non-managed config doesn’t seem to work but I could be wrong on that.
On the note of short term wifi, hotel/coffee shop/airport networks that have a TOS page or some sort of web login, haven’t really worked well for me. It used to be that you could go to some page and you’d get redirected but with the level of use of https that can be difficult. On macOS and Windows there is some signal maybe during dhcp negotiation that says there is a page to authenticate with. I’m not totally sure how it works, but it would be great if that could open the link in your browser when you join such a network.
Ideally the browser wouldn’t be using your user-profile but be more like a system web-view that doesn’t bring your cookies and plugins into the matter.
Lastly joining a corporate Cisco VPN is something I’ve tried but haven’t gotten to work completely with Openconnect. I’ve been able to authenticate, but incorporating the appropriate DNS servers and routes is where I got stuck. I think there is a payload of network configuration delivered right after authentication which would need to update various config. This is where I think networkd can maybe make this work.
To answer your questions:
I think I would want to in the openconnect case described above.
I am not using networking.localCommands.
I am using dhcp no special configuration but I did run into a bug there last week which I’m going to open a ticket for.
I think I’m pro predictable interface names.
Yes, internal domain for development on VMs, corporate domain via VPN, running in a coffee shop would be ideal but I don’t know if I can ask for all of that .
I run avahi for mDNS mostly for printer discovery.
I’m currently only running NixOS on my laptop at work, but I do some administration of a number of VMs on another distro. Certainly some features would make it easier to move to NixOS, such as running strongswan or an equivalent that can easily respond to re-keying events. I’m looking forward to using wireguard to encrypt traffic between VMs and ideally even authorized admins. This might be a little more than you intend to cover in the sprint but HA DNS Service discovery backed by a consensus protocol/service. (You said think big )
I am not using mainline NixOS, so feel free to disregard, but I do have some usecase observations (these are all things that have happenned to me this year).
Of course no-rebuild on-the-fly reconfiguration.
There are networks where you want to use IPv6, and there are networks where you really don’t; what are the best ways to configure it?
There are cases where I need to use ethernet connection for one subnet, but WiFi is actually better for most of the external subnets, so I want to have both cable and WiFi on with some simple routing rules.
2a. (and of course some of the traffic should be routed through a VPN)
Some networks have a rogue DHCP server that would be really nice to blacklist.
Sometimes running your own recursive resolver is actually a better idea than using the network-provided one, sometimes no; ideally one could use a local resolver for a local zone and decide per-network whether to forward most of the requests to the DHCP-provided DNS resolver.
I would be interested in participating to do some requirements work, maybe first drafts of documentation, and to learn to do Python NixOS tests (whatever comes out of the sprint, it needs some NixOS tests, right?). I will definitely not test anything systemd on my laptop. and probably not write implementation code. Am I too far out of the target audience, or could I be useful as a test rabbit who knows basic stuff about networks but nothing about networkd?
@fpletz Doodle (the link is now hidden into a spoiler) seems to imply that the get-together time on Saturday starts at 10:00; should this be a part of the visible text (probably with a quick confirmation message so that people get notifications)?
As I won’t be ale to attend remotely, I wanted to share a few answers and thoughts :
Yes, I use different routing tables, but not namespaces.
I configure everything through systemd.network. The only exception is that I use localCommands to IPv6 null-route cache.nixos.org, on every machine.
When I use DHCP, it’s mostly through Network-Manager.
I use predictable names on physical hosts, but tend to to match on MAC address in VMs, so it depends.
I often have my own resolver, so I disable resolved, and use dnsdist to handle split DNS. Avoiding VPN leaks is done using routing tables and firewall marks (using namespaces could be better).
On my personal machines, networking has usually been enough, except when I had to setup point-to-point links. On servers with complex networking needs, networking is not enough and I have to use systemd.network directly. networkd seems to fully cover the Linux networking API, while there are lots of things missing in networking.
Extra:
I only use networking.firewall in application containers and on my personal machines, it’s too cumbersome to use to write complex forwarding firewalls, and in that case I use ferm.
I never used it, as on the single machine where I have to do NAT, I’m using multiple external IPs, and want to NAT networks to specific IPs.
Additional answers:
I’m using BIRD2 to handle OSPF. It seems to work fine with NixOS and the current networking things.
I use Wireguard tunnels, but I don’t use the NixOS module for this, I configure it as a netdev in networkd, because I need to add rules and routes to specific tables. IIRC, networking.wireguard allows configuring the routing tables for routes generated by Wireguard, but does not allow configuring additional matching rules.
When using systemd.network, I’ve had a few times to resort to extraConfig because of missing options (especially for routing policy rules).
I still use networking for a few things on the servers on which I use systemd.network : enabling Wireguard (just having the wireguard interfaces in systemd.network is not enough), configuring extra hosts in /etc/hosts, and adding iproute2 table aliases.
I used systemd.network because I really wanted to avoid using custom bash scripts full of ip and iptables invocations.
Also:
I think switching to networkd is a good idea for NixOS, even if it may come at some cost. From my experience, networkd allows good access to Linux’s networking API, but suffers sometimes of some opinionated choices. It would have huge benefits, as the current networking module is somewhat limited. I’m very glad that networkd can already be configured directly through NixOS, I like the way networkd is structured (separating netdevs, interfaces and networks) and think it could serve as a good basis for a reworked networking module. One of my few gripes with networkd is that it’s sometimes way too much verbose, and I don’t how much can it be made to support on-the-fly temporary reconfiguration, which seems to be a common requirement.
To break the tie I would choose 2019-11-23 since we have our company christmas party the day before 2019-12-14. I’ll edit the first post to that effect.