Nix-buildproxy, reproducible HTTP/HTTPS responder in sandboxed Nix builds

[polygon]

Nix Buildproxy - Providing reproducible HTTP/HTTPS responders to builds that just can not live without

After patching my umphteenth CMake project to work around downloads during configure/build-time, I was wondering if it is actually possible to just serve HTTP/HTTPS requests inside of sandboxed Nix build environments. Of course, this needs to happen without accessing the internet and in a reproducible manner.

After an evening of playing with scripting mitmproxy it turns out, this is actually viable and works great, at least for CMake. In the hopes that others might find this useful, too, I present:

Nix Buildproxy

HTTP requests are captured and converted to a Nix-store backed inventory during a non-sandboxed build. Part two provides a mitmproxy that responds to the original URLs with content from the Nix-store, providing the expected responses without all the patching.

This is still very much WIP and has still some weaknesses, all of which is described on the project page. I still think it’s quite useful already and wanted to put it out there to get some comments/ideas on how to maximize the utility of this. Or to be told that this is a terrible idea and I should just delete the repo, this is fine, too.

I used this when packaging gRainbow, so head over there for a non toy-example.

[drupol]

Amazing idea ! Thanks for it !

[Salsa9]

I mentioned this at the last Stockholm meetup, I love the concept. It’s a bit “evil genious” style

Have you tried setting SSL_CERT_DIR and/or SSL_CERT_FILE to the CA from mitmproxy? I don’t know which applications respect that environment variable though. (Seems to be most things).

[polygon]

Thanks for spreading the word.

I tried to set these variables and it is also picked up by, e.g., curl. But this still doesn’t seem to be working.

curl -v --proxy 127.0.0.1:8080 --cacert ../nix-buildproxy/confdir/mitmproxy-ca-cert.pem https://example.com/
*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to example.com:443
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> User-Agent: curl/8.6.0
> Proxy-Connection: Keep-Alive
> 
[21:03:34.613][127.0.0.1:38946] client connect
[21:03:34.769][127.0.0.1:38946] server connect example.com:443 (93.184.216.34:443)
< HTTP/1.1 200 Connection established
< 
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: ../nix-buildproxy/confdir/mitmproxy-ca-cert.pem
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
[21:03:35.209][127.0.0.1:38946] Client TLS handshake failed. The client does not trust the proxy's certificate for example.com (tlsv1 alert unknown ca)
[21:03:35.210][127.0.0.1:38946] client disconnect
[21:03:35.210][127.0.0.1:38946] server disconnect example.com:443 (93.184.216.34:443)     

I’m a bit at a loss right now why this happens. Does anyone have an idea what might be wrong here?

[Salsa9]

nixos [ master][$?⇡][ v3.11.8][🐚fish][as 🧙 ]
[17:24:24]❯ HTTP_PROXY=http://localhost:1234 HTTPS_PROXY=http://localhost:1234 SSL_CERT_FILE=/home/lillecarl/.mitmproxy/mitmproxy-ca.pem curl https://canhazip.com -vvv
* Uses proxy env variable HTTPS_PROXY == 'http://localhost:1234'
* Host localhost:1234 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:1234...
* connect to ::1 port 1234 from ::1 port 45746 failed: Connection refused
*   Trying 127.0.0.1:1234...
* Connected to localhost (127.0.0.1) port 1234
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to canhazip.com:443
> CONNECT canhazip.com:443 HTTP/1.1
> Host: canhazip.com:443
> User-Agent: curl/8.6.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /home/lillecarl/.mitmproxy/mitmproxy-ca.pem
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=canhazip.com; O=Cloudflare, Inc.
*  start date: Feb 26 17:24:19 2024 GMT
*  expire date: Feb 27 17:24:19 2025 GMT
*  subjectAltName: host "canhazip.com" matched cert's "canhazip.com"
*  issuer: CN=mitmproxy; O=mitmproxy
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://canhazip.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: canhazip.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.6.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: canhazip.com
> User-Agent: curl/8.6.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200 
< date: Wed, 28 Feb 2024 16:24:28 GMT
< content-type: text/plain
< content-length: 16
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 85ca048efc105f19-ARN
< alt-svc: h3=":443"; ma=86400
< 
185.183.13.37
* Connection #0 to host localhost left intact

I was able to curl canhazip.com through mitmproxy with the command specified above, does that help your case?

[winter]

Especially odd is that those flags work according to mitmproxy’s documentation.

[polygon]

Turned out to be a bug. I’m not passing the certificate confdir to mitmproxy here. Adding --set confdir=${self}/nix-buildproxy/confdir fixes the issue. Haven’t noticed that buildproxy-capture uses the wrong CA because I never tried to pass the CA-cert to applications before.

Will test some more and release an update probably tomorrow.

[polygon]

Uploaded a new version. This now allows using curl without the --insecure option and hopefully other tools that respect the SSL_CERT_FILE environment variable. Thanks again for that hint.

I also added functionality to handle redirects. Just capturing/replaying them turned out not to work since, e.g., Github creates redirects that have a very short time to live. After a few minutes, the generated proxy-content.nix will contain URLs in the fetchurl blocks that will no longer work. Right now, whenever a redirect status is received, it is resolved directly in mitmproxy so the redirect never reaches the client.

This probably has other drawbacks that I don’t see right now, but it seems to work well for everything I’ve encountered so far.

[timbertson]

May be relevant, I have an open PR to support NIX_SSL_CERT_FILE as the canonical location for ca certs, for both nix itself as well as inside derivations:

treewide: support NIX_SSL_CERT_FILE as an impureEnvVar

[polygon]

Is that for the CA that the Nix binaries are using when querying the cache or git servers, or is this also having more of an effect, like being passed down to builders?

[timbertson]

It’s for both. The nix binaries already support this envvar, but I want to extend that into impure builders (fetchers, typically).

My main use case is GitHub - timbertson/netproxrc: netrc-enabled https proxy for nix, which is a HTTP(s) proxy to inject credentials in order for nix (and fetchers) to access authenticated resources without having to teach fetchers how to authenticate.

[Atemu]

As of gradle: add setup hook by chayleaf · Pull Request #272380 · NixOS/nixpkgs · GitHub, there is now infrastructure in Nixpkgs to use a similar tool (GitHub - chayleaf/mitm-cache: MITM caching proxy) and it’s already being used to package gradle-based packages.

[SergeK]

@Atemu

there is now infrastructure in Nixpkgs to use a similar tool

I’m awed and horrified

[fzakaria]

Slick idea; this is kind of a meta language2Nix type solution that effectively creates a lockfile for any build system.

I bet this might make Bazel builds a lot simpler too.

[SergeK]

@fzakaria

I bet this might make Bazel builds a lot simpler too.

I’d not be optimistic: downloads in Bazel are often conditional (platform, cuda support, etc), and sometimes completely unpinned…

[fzakaria]

You have to provide the SHA for dependencies similar to Nix so they should all be pinned.

[Atemu]

What @SergeK meant is that problems arise when the deps the build tool attempts to fetch differ depending on which platform the tool is ran on.

A fetch recorded on e.g. Linux may not work on e.g. Darwin because the hash for the Darwin-specific files are not recorded on Linux.

[SergeK]

@fzakaria

so they should all be pinned.

But also this: The cause of reproducibility problem of bazel build in jaxlib · Issue #321920 · NixOS/nixpkgs · GitHub