More airgap questions

Help
1

I am trying to create the ability to build and use nixos environments on an airgapped network.

I originally created Building with all-tarballs - Help - NixOS Discourse and then realized I was asking the wrong questions.
Private Mirrors in Airgapped Environments - Help - NixOS Discourse explains how to get source for all of the free nix packages and also individual other packages.
I made a modification of the copy-tarballs.pl to remove the AWS upload portion and now I have 94G of nix compressed nix source in a nix store.

It seems that the topics I have seen so far skip over the next part, which makes me feel like I should already realize how to do this, but how do I redirect nixpkgs to this source rather than all of the individual urls within each package?

I have also come to realize that I don’t absolutely need source (though I prefer it). I keep seeing marsnix posted, but I can’t find any information on how it is supposed to be used. This is probably trivial for those who have been nixing for awhile, but I am still figuring out things that I am sure most of the community takes for granted, like that NIX_PATH points to a tar.xz archive which contains a particular revision of github:NixOS/nixpkgs which is why it works different than the flake which points directly to github.

That makes me hopeful that I can just point to a local version of the nixexprs.tar.xz, but then it points to source all over the place.

I would really appreciate a breakdown or a link to somewhere which will do the breakdown. I would also really appreciate a how-to marsnix. At what point is it run on the internet and at what point can it leave the internet? A step by step example would be really nice.

Thanks in advance for all of the help!

2 Likes
2

I had planned to also build a cache server on this other network. I see that substitors can be used in a nix flake to reference a cache server. Will that also work if the /nix/store entries for the source is also present on that server or does nix serve require .nar’s?

If I bring over everything, will it matter that only my substitutor server exists and the original servers do not exist?

3

Well, in principle there is e.g. nix-serve so you can serve a binary cache directly from the store, if somewhat less efficiently.

4

Does it have to be a binary? If all of the source’s were already in the store, wouldn’t it grab them like any other derivation?

5

We call it binary cache because the modal use case is serving precompiled binaries and we are often bad with properly taking into account non-modal use cases. But technically it serves any store path resulting from a build, be they fixed-output or not (so store paths that are downloaded source tarballs are served just fine)

1 Like
6

Unless I am doing something wrong, it seems that fetch won’t check store paths to see if the source is already downloaded whether the source is within the /nix/store of the current system or being served up via a nix serve.

I changed the nix-store perl command in copy-tarballs.pl to --add-root packages/$name I am hoping that if I serve the packages directory up with nginx, perhaps I can use it as source for source.

7

That sounds like different content hashes… (But it is not clear what is fetch in this context)

8

Sorry, by fetch I was referring (perhaps improperly) to the method that is used to retrieve the source code by the various packages. I assume (assume being the operative word) that the ability to redirect toward tarballs.nixos.org is based within the fetch utility. In either case, despite having the source for the packages downloaded into the store (using the copy-tarballs.pl), trying to build without the internet still fails to find the source.

Sorry for my ignorance here. I feel like I am missing something very basic.

9

As far as I understand, tarballs.nixos.org support is fetchurl-only, and there it looks at hashedMirrors in pkgs/build-support/fetchurl/mirrors.nix, literally downloading from tarballs.nixos.org specifically.

copy-tarballs.pl tries to upload specifically there, and in the local store it creates files named by the hash — but normal building only checks locally for the files with the name specified in the call to fetchurl.

I guess you could resolve tarballs.nixos.org locally to a server serving the hash-named tarballs…But then there is fetchgit which would not be covered anyway.

You can also use nix-instantiate and nix-store -qR to figure out all the dependency derivations for a package, check for the presence of fixed-output output hash in those derivations files, and nix-store -r the fixed-output derivations in advance, getting the properly named fixed-output store paths in advance.

You can also read the documentation about substituters and write a substituter script that tries to use a local hash-named path for fixed-output derivations (thus using the copy-tarballs.pl tarballs), but I am not sure if there is a ready-made one already, probably setting up a local hashedMirror is easier.

1 Like
10

I did not realize that there was a nix.conf option to set hashed-mirrors, nor did I understand the pathing that needed to be used. This sounds promising! Thanks!

11

Thanks for the patience and help on this subject. Serving the tarballs as a hashed-mirror using ngnix did work for building small packages. It seems that every time busybox became involved in the build (which is more than you would think), the source was unable to be resolved.

I switched gears and decided to try for a full binary archive.

I think I finally understand this subject enough to understand previous posts on the topic. Perhaps I have complete cognitive dissidence in regards to this subject because it is so difficult to believe this problem isn’t solved.

I have been a software engineer for more than 30 years and somehow working Nix always makes me feel completely dumb. That isn’t to be taken as a slight against Nix, but perhaps as a slight against my own inability to adjust my way of thinking. Nix has so much potential. I am absolutely aghast that it isn’t the preferred solution world wide for so many things.

All of that said, the ability to be used completely offline/isolated/air-gapped is a critical piece that does not seem to have great support at this time.

The state of Nix Isolation and Airgapping

(Please correct the next paragraph if it is wrong)
For anyone researching this in the future, it should be understood that as of this time there is no foolproof solution to binary cache and even less so a source cache. There are some promising works that have been provided on the topic, but I have yet to find a complete working solution.

1. Marsnix

I first tried marsnix, since that has some record that it has been used successfully at least once. My thought was that if I could build a marsnix server, I could nix serve its packages. I was unable to get marsnix to work. It tried nix run with the github url; check out the repository and using nix-build, nix build, nix run; and several attempts at editing it to make it work.

2. Nix in an Isolated Environment

“Nix in an Isolated Environment” is a post by a few nixers trying to achieve the same thing as me. Reading it now makes me feel irresponsible to be posting all of this again. The same questions were brought up by people who undoubtedly have more experience with Nix and they got further than I have been able to so far. I found out the hard way that it is important to read the entire thread before trying anything in the thread as you will find the problems that came out of each attempt and the proposed (if somewhat hacky) solutions. I am going to try the final solution listed there, but I was trying to use existing Ubuntu infrastructure to prepare a cache to build up NixOS and Nix packages and this solution will require me to start from a nix system first. This is not so much a problem as it requires me to rework my infrastructure a bit, meaning I will have to get a bit more lenience from my IT group to let me try this.

3. Specific Mirroring Tool

The Specific Mirroring Tool was mentioned in the “Nix In an Isolated Environment” post. Since the author was kind enough to let us know the documentation is outdated, it is difficult to know the level of completeness offered by this tool. I will be continuing to cheer on AleXoundOS, but it is unclear whether this work continues.

4. Offline Install Iso’s

pete3n generated a repo for offline nix installer isos and discusses it in this thread. It is unclear whether this dependency list can be made large enough to support an offline for an extended period, but this warrants a look.

Additional paths from reponses below:

1. A function rebuildDependencyClosure that produces an efficient closure for offline rebuilds · Issue #180529 · NixOS/nixpkgs · GitHub .

More Information Requested

If I have missed any ongoing efforts to provide isolated/offline/airgapped support for a nix mirror or nix systems, please let me know and I will try to include them in this list. If anyone has specific success with any of these, also please let me know.

5 Likes
12

Thank you for collecting these resources.

I think you are expecting an “on-the-shelf” “push button” solution because otherwise, as you can see, most people have been implementing their own thing for their own needs. The core of it remains and is a form of local binary cache.

Usually, the ecosystem is not great at providing everyone with “push button” solution because it’s rarely needed except for commercial interests / entities.

I will add extra one: A function `rebuildDependencyClosure` that produces an efficient closure for offline rebuilds · Issue #180529 · NixOS/nixpkgs · GitHub.
And I can confirm that I used many of the solutions you mentioned, including just manual mirroring, successfully.

In the end, this is mostly a documentation problem and understanding from first principles how things work because it’s so related to the core of Nix: the Nix store.

2 Likes
13

Just saw this thread, and I’d add what I did here.

I had a setup* last year that builds (not just copying the store) from bootstrap all the way up to Emacs in an airgapped environment. Here’s what I did:

  1. Use nix path-info --recursive --derivation <installable> to get the closure.
  2. Use a script to filter the list of derivations to get the fixed-output derivations (by checking whether each derivation has outputHash etc.) only.
  3. Build the fixed-output derivations on a host that has Internet access.
  4. Copy the outputs of the fixed-output derivations to a local binary cache store by nix copy --to file://<path>, which can then be transferred physically to the airgapped host to build the <installable>.

The Nix store surgery is definitely relying on the implementation details. It was doable with the tools that we already have, but I’d definitely like it to be easier. I felt quite lucky when I learned that IFD is not allowed in Nixpkgs though, because otherwise this would be a much more difficult problem.

*: The whole setup was documented in my note, but it also involves other distractions e.g. it’s musl, it’s at a custom store location, and it’s on an outdated CentOS 6 machine with kernel 2.6.32.

3 Likes
14

I’d just build a configuration on a machine with internet access, copy closure to usb drive, plug that drive to airgapped machine, copy closure from usb drive & activate it. It probably can be automated, but it depends on the environment.
Yes, no rebuilding directly on the airgapped machine, but on the other hand you don’t have to deal with setting up the binary cache mirror, which is itself is not straightforward.

15

Yes, building on a machine with Internet access is much easier than building on an airgapped machine.

In my case though, I specifically wanted to build on the airgapped machine, because the airgapped machine was an HPC system with more cores and thus builds faster. For example, the Emacs-from-bootstrap build can take several hours v.s. under 1 hour to complete.

1 Like