Wirenix: A flake for creating wireguard mesh networks

Announcements
1

Hi everyone!
I did a thing
I’ve been enjoying NixOS for about a year now and I figure it’s about time I contribute my own project to the nix community: Wirenix. I encourage you to check out the wiki. I don’t want to spread out documentation across forum posts and chatrooms, so everything you need should be in there. If it isn’t, that’s a bug and you should raise an issue about it. You can also find the repo on sourcehut.

What is Wirenix?
Wirenix is a flake designed to take a single configuration file describing a (set of) virtual private networks, and generates the network configuration necessary on each machine to make it happen using Wiregaurd. It supports networkd, static network configurations, and generating secrets with agenix-rekey or reading them from files. Agenix automatically generates ipv6 addresses and adds them to the hosts file, but does support manual IP assignment as well. Wirenix can handle arbitrary network topologies, but is best suited for full mesh or hub and spoke networks. In addition, Wirenix is built to be modular, and you can plug in your own configuration format, output configuration generators, and secret providers if you need to.

What is not Wirenix?
Wirenix is not aiming to be a tailscale/headscale replacement. Those services already do a great job for connecting machines across NATs and Firewalls you don’t control, and are excellent choices for allowing users to connect to a network from any machine. Wirenix is instead meant to offer a simple (dare I say more robust?) setup for machines on networks you control to connect to eachother. Wirenix doesn’t handle any routing, NAT, DNS, DHCP, or anything else like that. There’s nothing stopping you from adding that on top of a Wirenix network, Wirenix won’t do it for you.

Me rambling if you’re into that
My personal motivation for this project comes from a need to secure certain parts of my home LAN (without an 802.1X capable switch):

  • I could individually configure each peer with Wiregaurd, but that wouldn’t scale well, especially if I add containerized/virtualized services to my network.
  • I could write some bespoke code for my particular configuration, but I wanted to keep my nix config simple and decoupled from any complex logic wiring peers together.
  • I could have used tailscale or headscale, but these were essential parts of my LAN network, and I didn’t want more single points of failure in the form of control nodes.
  • I could have secured all my services at a higher layer, which I do, but I use internal ACME for some services that don’t like other forms certificate distribution. As far as I can tell (I’m a bit of a novice here so don’t trust me), ACME relies on assumptions that DNS and IP addresses are secure. In the case of a home network without 802.1X, this isn’t true.

So I was left with the option of writing a nix flake to automate the tedious bits of setting up Wiregaurd. I was further convinced that writing a flake for the community to use was the right choice after asking around and hearing that many people in the nix community have solved this exact same problem in their own configs, but there was no generalized flake sufficiently decoupled from its corresponding nix configurations.
I was hoping to be further along with testing and stabilization before letting people know about this project, but the core features work, and I don’t want to hold off releasing it indefinitely just to finish that last 20% of work. If you run into any issues, which is likely, feel free to report them and I’ll make sure to give them some attention.

Thanks Everyone
As a recent member of the cult of Nix, I just wanted to thank everyone for this awesome and welcoming community. NixOS has seen some massive growth, I’m a part of that wave, and I’m sure there’s been some tension around that and difficulty maintaining the community. Even so, I’ve seen nothing but friendly, welcoming, slightly fanatical, people whenever I engage with the community (with some exceptions because it’s the internet). So here’s to everyone who enables (at least some) newbies like me to survive long enough to add something to the project. Now let the brutal unsolicited code reviews begin (because, again, it’s the internet)!

29 Likes
2

Hi, it seems like a pretty cool project but I was wondering if you did any benchmarking to see how performance differs from tailscale/zerotier?

Also, it would be fantastic if you were to use some nicer git platform like github/gitlab

3

Benchmarking this is very tricky if you want anything resembling comparable numbers. You can’t really get comparable infrastructure.

The performance differences will be more of a networking benchmark than actually covering any of what this flake does. Even if you could deploy this on the same networks, it’d be a benchmark of Linux kernel components (which should in theory be the same, and if they differ it’s not a fair comparison again) rather than a benchmark of the flake.

This flake is ultimately just a configuration deployment tool, which allows you to create a mesh network with the performance characteristics your hardware and configuration permit. It doesn’t itself do any heavy lifting.

The difference between this and tailscale/zerotier is cost, ease-of-use, commercial support, selfhosting vs SaaS, and rudimentary extra features not directly related to the actual networking.

I for one enjoy the use of a modern (and AGPL!) git forge. GitHub has shown itself to be as painfully disrespectful to FOSS projects as you would expect a M$ product to be, and GitLab is incredibly slow and clunky because of its focus on corporate deployments (which to their credit is not a bad thing by itself, they do great work, doesn’t mean I enjoy using the resulting product).

Project marketing will suffer without a GitHub repo, of course, but that’s a problem that can only be resolved by fewer projects using GitHub, not more.

4 Likes
4
  • Wirenix uses wireguard
  • Tailscale uses wireguard-go
  • Zerotier uses something similar to wireguard-go (user space VPN)

So Wirenix should be the fastest…

5

That’s only a small part of the stack though; tailscale provides firewall holes by acting as a proxy in some scenarios, which may provide better routing and therefore performance. I’ve seen significant performance differences depending on wireguard setups when used in China, for example.

My point is that benchmarking the entire configuration this flake provides and comparing it to tailscale/zerotier isn’t easy precisely because it’s more than just a small networking component. I didn’t realize the latter two are limited to user space though, which is definitely another reason to hand-roll!

1 Like
6

Not really, wireguard-go can take control completely of the network stack à la DPDK, so can achieve higher performance than the in-kernel one.

Either case, wirenix has the same performance as the kernel WireGuard, which can be benchmarked separately.

Everything on firewall / proxy, etc. is missing the point IMHO, you usually run the benchmarking without those in perfect conditions first to see if you can saturate linerate ≥ 10Gbps cards.

Then, surely, you can see what’s the impact of STUN/TURN tunnels and NAT punching or iptables firewalling. But there are solutions for those and is orthogonal to the whole project IMHO :slight_smile:.

2 Likes
7

I think the answer is: it depends

Wirenix is WireGuard so it can’t do relaying and other advance network configurations so we would be comparing hosts with reachable IPs with no FW / proxy configurations. In that case Kernel WireGuard will be faster for “normal” use.

But if you have two rootless Podman containers on two hosts, speaking to each other, wireguard-go will be faster since there will be no context switch between kernel threads and user threads.

1 Like
8

Apologies but this is still incorrect.

The userspace implementation of wireguard-go is always faster than the in-kernel above 10Gbps.

Surpassing 10Gb/s over Tailscale for reference.

I don’t understand what is the point you are making with podman or whatever, this is irrelevant to the context AFAIK. Advanced network configurations or whatever are also irrelevant, at most, this is a data plane / control plane concern to punch through *-cone NATs or to apply L3 filtering. You seems to be conflating a lot of things at the same time.

Again, WireNix could probably receive a data plane to perform more advanced network configurations, this is beyond the point ;).

2 Likes
9

I agree that wireguard-go will be faster in many cases. The point about Podman was an example where wireguard-go + slirp4netns will be always faster.

10

Hey all! Glad to see there’s interest.

About performance:

Much of the technical details are beyond me, but as far as the runtime is concerned Wirenix doesn’t exist. It’s just doing the tedious bits of writing up every possible peer combination to your wiregaurd config. If you really wanted, I see no reason you couldn’t use it to configure wiregaurd-go or wiregaurd-rs if you wrote your own configurer.

About sourcehut:

I agree that sourcehut isn’t perfect, and the user experience is a lot less polished than something like github. That being said, github has lost a lot of my trust with their usage of copilot and general business practices. If there’s a significant push for using github I might put up a read only repo there, but I’d rather leave my work with a company that I trust won’t be using AI to launder my copyleft license.
Gitlab, codeberg, etc are options, but then I’d just be switching from one platform nobody uses to another. Sourcehut has an advantage over the other platforms because you only need an email to submit issues and propose changes.
I also consider wirenix to be fairly feature complete, so I don’t expect to see a lot of incoming patches or active development. Just bug reports and additional tests I’ll mostly be writing.

About firewalls/NAT/relaying:

wirenix can’t do any of that, but you can define different endpoints depending on the connecting peer. I haven’t tested this particular scenario but I don’t see why you couldn’t forward a range of UDP ports on your router to all your other peers, and then configure their endpoints to show your public ip and corresponding port only for other peers that are outside your network (while still showing the internal IP to peers inside the network). You may need to enable NAT reflection but I don’t see why it wouldn’t work. IPV6 simplifies things further; just add a firewall rule and you’re done.
If you wanted a traditional centralized VPN, you should be able to mix and match wirenix with your own static routing configuration, I plan to do this myself at some point.
But if your network starts getting more dynamic, you should just use tailscale. It’s a good solution and if what you’re looking for is a way to connect to your home network from anywhere outside, I think it’s the best option.

3 Likes
11

Does it support VPN style forwaridngs lile 0.0.0.0/0 with networkd?

12

This feels like an apologize. You shouldn’t feel the need to apologize for your git repo hosting platform choice. :+1:

12 Likes
13

I don’t think I’ve heard of that particular term before, searching it just shows a bunch of information about port forwarding with vpns. Do you think you could elaborate? If it does support it I should mention it in the docs. If it doesn’t I should either add support or list it out as out of scope.

14

I appreciate the sentiment. I don’t particularly care about wide spread adoption or being searchable or anything like that. But I do care about accessiblillity. I am aware that sourcehut both excels in some regards (no js, no account needed to contribute) and also falls behind in others. In particular, their email based collaboration features that enable accountless contributions can be confusing and gatekeepy to people not used to the workflow (including myself). You are right though, it does feel like an apology. Thanks for reminding me I don’t need to apologize for things like that.

3 Likes
15

tl;dr?

  • Is it a NixOS Module?
  • Can it work without NixOS?
  • It has command line tool?
16

Basically my goal for a longer time is to send all my traffic over wireguard to a server to basically have a VPN. This failed in the past with networkd because it would also send the wireguard traffic itself through the wireguard tunnel. wg-quick is solving that through scripting but I couldn’t solve it so far with firewall marks.

1 Like
17

Wirenix is a nixos module. It translates a network topology specified in a configuration file into the individual nixos network configurations needed to achieve the topology using wireguard. Using the module is the same as writing that generated configuration into your nixos config.
You can connect to any wireguard peer but it won’t make any configuration for the non-wirenix peer. Connecting from a wirenix client to a non wirenix client is as simple as adding the non wirenix client into the wirenix config. Establishing the connection the other way requires finding out the generated config and adding the wirenix peer like you would any other wiregaurd peer.
There is no command line tool. But wirenix has been designed to be at least somewhat usable from the nix repl.

2 Likes
18

I don’t think wirenix will be very helpful in that case. Wirenix is designed for peer to peer VPN setups. It should be possible to do peer to site and site to site networking with a wirenix configured tunnel. But wirenix won’t do anything to make it easier. So you’d still be writing in firewall marks and static routing into your configs on your own.

1 Like
19

I want to chime in and say I’m happy with my initial use of Wirenix. I built a three-node mesh for a storage cluster and it works flawlessly. I did need one change, but @ttamttam1 has already merged it. :heart: I have another Wireguard setup that I plan on switching over to Wirenix, when the time appears…

For my use case I ended up using static addressing, so you can mark that as tested in the Wiki. I was unsure at the time how the addresses were computed and I needed known addresses. Looking through the source again, they should be predictable as they are generated from a hash of the node name.

Finally, I use agenix (not agenix-rekey), and was able to work with the acl key provider by pointing to the path of the agenix file, e.g.

{
  version = "v1";

  subnets = [
    {
      name = "storage";
      endpoints = [
        {
          port = 51820;
        }
      ];
      presharedKeyFile = "/run/agenix/wireguard-psk-storage";
    }
  ];
}

That can and should be refactored to config.age.secrets.wireguard-psk.path

2 Likes
20

There’s a new update out! This new version of wirenix introduces some bug fixes, better documentation, and also a new feature that enables multiple wireguard devices for the same subnet on the same machine. The primary use case for this is for passing netdevs into containers. Here’s an example of what the module options might look like:


      wirenix = {
        configurer = "networkd";
        devNameMethod = "hash";
        enable = true;
        aclConfig = import ./acls/double-dev-ring.nix;
        peerNames = ["peer1" "peer3"];
      };

To then get the correct device when using the “hash” method include wnlib from the wirenix flake and use wnlib.getDevName "hash" "<peer>" "<subnet>" to get the proper subnet name. See the Module Options section of the docs for more information, as well as the NixOS configuration example and ACL configuration example used for tests.

It also appears, judging by a steady flow of reactions on discourse, that this project is seeing slow but unexpected growth. As my first open source project that has more than 1 or 2 users it makes me nervous that I could change one small thing and suddenly break other people’s configurations without warning. To help remedy this, there is a new wirenix-breaking mailing list: Breaking changes, if any, will be announced there before going into the release branch. Breaking changes are unlikely and I take great care to avoid it, so hopefully it will never be needed. See the breaking changes mailing list on sourcehut or subscribe to the breaking changes mailing list via email
To avoid spam, you probably won’t see further updates in here. I am posting to increase the chances of current users learning about the new breaking changes mailing list.

3 Likes