Connecting to a server inside of a NixOS test from the browser

Help
1

Hi all,

We have encountered the following situation in a test that we are writing currently for the federated reddit clone Lemmy (https://github.com/ngi-nix/nixpkgs/blob/mob/lemmy-improvements-new/nixos/tests/lemmy.nix).

Our goal here is to setup a nixos test with a lemmy server such that we can, not only explore it in the terminal with the interactive test driver but also through our host machine browser. We have succeeded to do so in the following way, but it is conflicting a bit with how tests are typically setup:

First, it’s important that Lemmy gets served through the reverse proxy Caddy that then hands of requests to the backend, ui and other attached services. To route a request, Caddy listens on specific IP addresses and ports, and checks whether the hostname header in the http request corresponds to a pre-configured one. The hostname is passed to caddy in the lemmy module here.

As Caddy is currently setup, it is therefore not enough to just ping the correct IP address from a different hostname. E.g. if the hostname is set to http://lemmy.com, we can launch the test driver from nixpkgs root directory with $(nix-build -A nixosTests.lemmy.driverInteractive)/bin/nixos-test-driver, then start the virtual machines with start_all() in the Python interpreter, forward a host port of the server node port 80 that caddy listens to in the test executing server.forward_port(8888, 80) in the Python interpreter, and finally connect to it from outside of the test environment with curl -v -H Host:lemmy.com localhost:8888 successfully. But leaving out the -H argument from curl won’t work and it’s also not possible to connect to Lemmy from the browser with the URL localhost:8888 without installing cumbersome browser extensions that set this host header.

The inability to examine the test machines with a browser is why we starting setting the hostname to http://lemmy.localhost. After running the interactive driver and forwarding the port in the same way as before, we can now simply do curl -v localhost:8888 and also access and use lemmy from the browser by just entering lemmy.localhost:8888 in the URL bar.

However, this hostname somewhat goes against the way nixos tests are setup at the moment where the default hostname is the same as the name that is used for the nixOS virtual machine. For example, checkout this tutorial on nix.dev which says: “Between the machines defined inside the nodes attribute, hostnames are resolved based on their attribute names. In this case we have client and server .”.

We are now confronted with a few questions that we struggle to answer. The main question is, of course:

  • Is there a simple way to enable the use case of connecting with a browser to a nixOS test that is consistent with the nixpkgs conventions?
  • Are nixOS tests the right place for interactive browser testing? [I personally think it’s enormously valuable but would be happy to be convinced that an alternative is better]

Some subquestions that may lead to a solution:

  • is there a simple way (without installing extensions or editing individual http requests) to modify the hostname header in a browser (e.g. on firefox and chrome)?
  • is there a less invasive way to setup a temporary hostname on a system than editing the hosts file?
  • is it bad practice to use a *.localhost hostname in a nixOS test?
  • is it bad practice for a reverse proxy like caddy to listen to several hostnames, one of them being *.localhost and the others referring to the internal nixos test hostnames?

ps: this is a followup of this question:

2 Likes
2

@mat, I’ve somewhat worked through these sorts of problems in non-nix environments, but not so much in nix. Consequently, I’ll offer some input but not really answer your nix questions.

My first approach would be to override DNS of the test client for the test network. You suggest doing this with a hosts file, which might be the easiest way. It also can be done with a local DNS resolver such as resolved where the configuration can target a specific network (either by IP address or top-level domain). The qemu internal network could have private DNS this way.

You use of *.localhost is effectively turning localhost into a top-level-domain. I’ve never seen or tried what you are doing there, but don’t have enough experience to say it is a bad practice.

I am pretty sure that having a reverse proxy listen for multiple hostnames is normal. Assuming that caddy is like envoy, it can do layer-7 routing based on the URL, which would include routing based on different hostnames.

Another option is to use a proxy on the test client, assuming you can configure the proxy to rewrite the hostname info in the http request (and that it would be enough).

I prefer the DNS approach, but I am much more comfortable manipulating DNS than I am with http headers. I think using a private DNS server involves much less deviation from the real-world use cases you are testing.

I hope this helps.

3

Thanks @ericgundrum that’s really interesting. I am myself unfamiliar with DNS servers etc. but this does sound like a clean solution. I wonder whether one can easily set one up in the command line that temporarily maps say example.com to localhost:8888.

Another option is to use a proxy on the test client, assuming you can configure the proxy to rewrite the hostname info in the http request (and that it would be enough).

For this I actually found a solution with mitmproxy:

nix-shell -p mitmdump

mitmdump --flow-detail=2 --mode reverse:localhost:8888 --modify-headers /~q/Host/example.org

On curl -v localhost:8080 this outputs:

Proxy server listening at http://*:8080
[::1]:57874: clientconnect
[::1]:57874: GET https://localhost:8888/
    User-Agent: curl/7.76.1
    Accept: */*
    Host: example.org
 << Server connection to ('localhost', 8888) failed: Error connecting to "localhost": [Errno 111] Connection refused
[::1]:57874: clientdisconnect

This works! I don’t have anything running here on 8888 but the Host header get’s modified.

1 Like
4

Enabling resolved on nixOS is as easy as services.resolved.enabled = true in the OS config. The default settings seem fine for most environments, I think. NetworkManager will automatically detect resolved and adjust accordingly. Even glibc based apps will go through it. However, browsers (and others) configured to use DNS-over-HTTPS will bypass it unless changed.

The tricky bit for your use case to temporarily map a hostname from the command line should be doable with one of the resolvectl subcommands such as dns or domain. See man resolvectl. You should be able to make this work through shell.nix; although I am not sure how to wire-in resolvedctl revert when the shell exits. I also found useful info in the systemd-resolved ArchLinux wiki page.

I’m glad to know the proxy solution works as expected. It seems reasonable to use that if it meets the expectations of your test server.

1 Like