Best practices for air-gapped NixOS servers with untrusted users?

Help
1

Hi everyone,

I’m looking for advice on the best workflow for managing software installations on an air-gapped (offline) NixOS server shared by untrusted users.

Setup & Constraints

  • The server has no direct outbound internet access.

  • Users log in via SSH, but they are untrusted (so they cannot be added to nix.settings.trusted-users).

  • Users want to manage their own software environments (e.g., via Home Manager).

Core Idea

In theory, building a package offline only requires having all of its Fixed-Output Derivations (FODs)—such as source tarballs and Git repositories—already present in the store. Since FODs are content-addressed and verified by SHA-256 hashes, importing them shouldn’t pose a security risk, even when done by untrusted users.

However, we are struggling to make this a smooth, practical workflow for end users.

Scenario 1: Pre-downloading FODs on the user’s local machine

Is it possible for users to pre-fetch all required FODs on their local machine and transfer them to the server’s store?

  • Manually tracking down every FOD for a complex configuration is impossible. Is there an automated tool in the Nix ecosystem that can resolve, extract, and download all FODs for a given configuration?

  • What if the user’s local machine does not run Nix (e.g., they are using Windows)? Is there a standalone or universal tool that can handle this?

Scenario 2: Downloading FODs on the server via a user-space proxy

If we relax the constraints slightly: nix-daemon itself remains strictly offline, but the user has a user-space proxy (e.g., SOCKS/HTTP proxy) available in their SSH session.

  • Can the user download the FODs in user-space using their session proxy, and then hand them over to nix-daemon to continue offline compilation in the sandbox?

Has anyone built a workflow or tool for this kind of setup, or is there an established pattern we might be missing? Any insights would be greatly appreciated!

Thanks!

Update:

LLMs were used to assist in translation and optimize expression; the original text/prompt is provided here at the request of the community policy:

算了我感觉可以上 nixos 论坛问问。你帮我撰写一下英文版吧。大概就叫“在无网络情况下部署公共服务器的最佳实践是什么？” 

理论上来说，只有 FOD 部分才可能联网，因此是否可以让用户在自己的设备上预先下载完整编译所需的 FOD ，并添加到 store 中？（而 FOD 是可以通过 hash 进行验证的，这个应该不会有信任问题。）我认为这是可行的，但似乎没有找到现有的工具去完成这个事情——例如我们肯定不能要求用户手动把 home-manager 引用的 FOD 列出来，必须要有一个自动化的工具去完成这个事情。我想知道 nix 环境里是否有这样的工具。此外，如果用户本地的设备不使用 nix ，我也并不想强制他们使用，因此如果这个工具是通用的，那再好不过。

我还有一个弱化的场景——在这个服务器上，nix-daemon仍然不可访问网络。但是用户可以通过特定的用户态 PROXY 去访问。这种情况下，用户能不能再用户态完成 FOD 的下载，然后交给 nix-daemon 继续之后的编译？
5 Likes
2

A pointer on the FOD-extraction and transfer can be found here: Demonstrably Secure Software Supply Chains with Nix

2 Likes
3

@tfc

Thanks for the link! I’ve briefly read through that page. So overall, the workflow seems to be:

  1. Run nix build locally first.

  2. Run nix derivation show -r to list the full derivations, and use jq to parse out all the FODs.

  3. Use nix-store --export and nix-store --import to transfer them across devices.

However, in actual practice, I still have a few questions and might need something more engineering-oriented / robust:

  1. Do I necessarily have to run nix build locally first? This implies that my local machine must have Nix installed, and it might take a considerable amount of time to build (even though I won’t be using the final build outputs locally).

  2. Is this kind of jq processing reliable and future-proof? Do we only need to check the existence of value.outputs.out.hash? Actually, I noticed that structure might have changed, and we may need to navigate through .derivations first before applying those jq filters.

  3. Will nix-store --import reject the import because I am not a trusted-user? Or does it automatically allow FODs? Theoretically, I think this is safe, but it probably depends on the internal implementation of Nix.

Sorry, I might have rushed into replying a bit. I found that nix derivation show can actually be used directly (e.g., nix derivation show -r .#homeConfigurations.yueyinqiu.activationPackage), without necessarily having to run nix build first (though can we use nix-store --export this way?).

As for whether nix-store --import works directly, I will do some further testing on my own.

Therefore, my only remaining concerns are the reliability of the jq parsing, and whether it’s possible to get rid of Nix on the local machine entirely.

4

Hey @yueyinqiu!

1.) If your builds don’t contain any IFD etc. you don’t have to nix build everything locally first. evaluation will be enough to get the FODs!

2.) It turned out to be very reliable. At a customer we first scripted this and resulted in relatively short scripts that haven’t changed in years.

3.) We did not have the (non-)trusted-user constraint at our customers where we built that, but to my understanding FODs should be importable and then the rest would “just build”.

Let me know where reality deviates from theory. :slight_smile:

2 Likes
5

I did this a few years ago but unfortunately can’t easily find my scripts.

If you don’t already have a solution for transfering FODs over the gap, NNCP is a good tool.

The jump machine sends store paths to $HOST with something like:
nix-store --export $STORE_PATHS ... | nncp-exec $HOST nix-store-import
and the airgapped machine has a receive endpoint defined something like:

{
  nncp.settings.neigh.jumphost.exec = {
    nix-store-import = "${lib.getExe' pkgs.nix "nix-store"} --import";
  };
}

For sending paths that have a closure, like actual packages, the best solution I found was to transfer using an intermediate /nix/store on some portable storage.

1 Like
6

@tfc

:sob: I’m not sure if I did something wrong, but when I tried to import, it says:

image

I have placed my test steps here: https://github.com/yueyinqiu/TestAirgappedNixOS . I would be very grateful if you could provide further assistance.

7

@state-plumber Thank you very much for your suggestion on the FOD transfer part. It also gave me some inspiration for my previous issue! Perhaps I could have a single trusted-user to receive all these FODs. (However, the subsequent question is how to verify that they are indeed FODs.)

8

Sorry if I don’t get your threat model here, if you have any real reason to make nix-daemon offline while giving users a proxy, but you can set up a proxy for FODs too:

systemd.services.nix-daemon.environment = {
  HTTP_PROXY = "http://proxy.example:3128";
  HTTPS_PROXY = "http://proxy.example:3128";
};

Or use the configurable-impure-env experimental feature and nix.settings.impure-env = [ "HTTP_PROXY=http://proxy.example:3128" ... ];:

Actual FOD outputs should not need a trusted user, but I imagine there is something wrong with your script as nix derivation show JSON is rather unstable between nix versions and implementations. If you could pastebin the JSON that nix derivation show -r .#homeConfigurations.home.activationPackage shows on your nix, as well as nix --version, that would be helpful.

1 Like
9

@bitbloxhub

Thank you for the explanation regarding the nix-daemon proxy configuration.

In our scenario, the machine isn’t supposed to have internet connectivity at all. While users might occasionally set up a proxy via SSH remote forwarding (which is currently tolerated, even though it should violate security policy), I definitely should not configure a proxy directly for nix-daemon. And that’s why, our baseline assumption is still a completely air-gapped environment. The user-space proxy path merely as a practical workaround for users who don’t run nix locally.

Regarding the previous test, here is the Nix version being used:

[client@nixos:~/TestAirgappedNixOS]$ nix --version
nix (Nix) 2.34.8

Since the derivation is a bit long (it includes the Home Manager setup), I’ve uploaded it directly to GitHub: https://github.com/yueyinqiu/TestAirgappedNixOS/tree/main/derivation .

Looking forward to any further help or guidance you might have on this.

Update:
LLMs were used to assist in translation and optimize expression; the original text is provided here at the request of the community policy:

感谢您关于在 nix-daemon 上使用 proxy 的说明。事实上在我们的场景下，该设备本不应该具有因特网连接。但是用户有可能使用 SSH Remote Forward 构建 proxy （我知道这其实不符合安全要求。但是这个行为目前被默许了。）但是我肯定不能直接为 nix-daemon 配置任何 proxy 。因此，这个场景其实也不是我首要考虑的，最好还是可以实现完全 air-gapped 情况下的使用。

关于之前的测试，nix 版本如下
[client@nixos:~/TestAirgappedNixOS]$ nix --version
nix (Nix) 2.34.8

derivation 的结果比较长，因此我直接上传到了 GitHub https://github.com/yueyinqiu/TestAirgappedNixOS/tree/main/derivation 。
1 Like
10

I noticed that nix-store --add-fixed seems to work even for untrusted users.

Based on this, I put together an extraction tool. The current workflow is:

  1. Uses the mentioned jq rules to parse nix derivation show and identify FODs along with their hash methods (flat or nar).

  2. Calls nix-store --realise to download the targets locally.

  3. Bundles the realized output using nix-store --dump.

  4. Generates an unpacking script that, once transferred to the target server, automatically unpacks the archives (nix-store --restore) and populates the target store (nix-store --add-fixed).

Once all FOD outputs are present in the target store, the build can proceed via the standard workflow (provided inputs like nixpkgs are also uploaded).

This approach seems feasible in preliminary testing. I’ve pushed the code to a separate branch in my test repo: TestAirgappedNixOS (addfixed branch).

That said, I just realized my previous approach might have had a flaw: I think I was exporting/importing the downloader .drv itself rather than its output. That indeed seems untrusted? (I’m not completely sure yet, and I’ll keep experimenting.)

Also, the resulting size was quite surprising—the FOD outputs for just nixpkgs#hello came out to 432M, which is much larger than I expected.

That said, I just realized my previous approach might have had a flaw: I think I was exporting/importing the downloader .drv itself rather than its output. That indeed seems untrusted? (I’m not completely sure yet, and I’ll keep experimenting.)

Update: I further tried restricting it to just import/export FOD output on a new branch. It doesn’t look like it works either:

image

Update:
LLMs were used to assist in translation and optimize expression; the original text is provided here at the request of the community policy:

我发现 nix-store --add-fixed 似乎可以在 untrusted user 下工作。目前我初步实现了一个提取工具 https://github.com/yueyinqiu/NixFodExporter 。它基于 @tfc提供的 FOD 判断规则，解析 derivation show 的结果获取 FOD 及其 hash method （flat 或 nar ）；调用 nix-store --realise 将目标下载到本地；然后把它的 output （例如，下载得到的代码）使用 nix-store --dump 打包。同时生成一个解包脚本，在发送到目标服务器后，可以自动解包（通过 nix-store --restore ）并添加到目标服务器的 store 中（通过 nix-store --add-fixed）。这样一来所有 FOD 的输出都已经位于 store ，接下来按正常的流程编译即可（当然还得把 nixpkgs 之类的 inputs 上传上去）。

目前这个流程经过初步验证似乎是可行的。我将它放在先前测试仓库的另一个分支中：https://github.com/yueyinqiu/TestAirgappedNixOS/tree/addfixed

不过我突然意识到之前我的做法可能有点问题，我似乎是把下载代码的 drv 本身 export/import 了，而不是它的输出。这似乎确实是不可信的？（我不是很确定，我会继续做一些尝试。）

另外这个大小和我想象的差异很大。仅仅一个 nixpkgs#hello 的 FOD 输出就要占用 432M 。
2 Likes
11

Thanks for explaining.

It looks like the nix-store --export format does not have CA/FOD info, and your jq was returing .drvs.

With the help of ChatGPT (GPT-5.6-Sol) I got it to make this script. You may need to play around with it a little, as I have not verified the copying, although it looks OK. I have verified the jq however. It seems like nix copy should work even to a non-trusted user as long as all the paths are CA.

ChatGPT script 
set -euo pipefail

nix derivation show -r \
  .#homeConfigurations.home.activationPackage > derivation.json

jq -r '
  .derivations[]
  | . as $drv
  | .outputs
  | to_entries[]
  | select(.value.hash? != null)
  | .key as $output
  | $drv.env[$output]
' derivation.json |
  sort -u > fod.txt

mapfile -t fod < fod.txt

# Make sure all the FODs actually exist locally.
nix-store --realise "${fod[@]}"

# Copy ONLY those FOD outputs into the offline machine's store.
nix copy \
  --no-recursive \
  --to ssh-ng://server@192.168.100.10 \
  "${fod[@]}"
1 Like
13

What? why was this flagged/unlisted?

1 Like
14

:smiling_face_with_tear: It says that I used LLMs. I will find the original text and paste it here, then it should be unhidden.

1 Like
15

LLMs were used to assist in translation and optimize expression; the original text is provided here at the request of the community policy:

感谢您的帮助。也就是说整体上可以通过

nix build
nix derivation show
nix-store --export / --import

实现。但在具体实践上我其实有一些疑问：

1、我是否必须先进行 nix build ？这意味着我的本地设备必须安装 nix ，并且可能要花费相当时间进行构建（即使我不会在本地设备使用它）。

2、这样的 jq 处理是有保障的吗？我们只需验证 value.outputs.out.hash 的存在性？事实上我发现它已经过时，我们需要先 .derivations 再使用该网页提到的 jq 指令

3、nix-store --import 是否可能因为我不是 trusted-user 而拒绝？或者它会自动允许 FOD 。理论上我认为这是安全的，但这可能取决于 nix 的实现。

---

抱歉，我可能有点急于回复了。我发现 nix derivation show 似乎可以直接使用（例如 nix derivation show -r .#homeConfigurations.yueyinqiu.activationPackage），而不一定针对 nix build 的产物。

关于 nix-store --import 能否直接使用，我会自己进行进一步测试。

因此现在唯一令我担心的是 jq 的问题，以及是否有可能在本地设备上摆脱 nix 。
1 Like
17

@bitbloxhub Oh, nix copy does work! It seems to be the easiest way to copy these FODs right now (assuming SSH is available; if not, doing nix copy twice might do it as well).

Speaking of CA, if that’s the case, could I even directly upload packages downloaded from the official cache (since the server could trust the official cache)?

A practical question is: suppose I try to nix copy a fairly complex home-manager where some packages come from the official cache, some from untrusted caches (like Cachix), and some are built locally. Is there any way to distinguish between them—uploading the final cache for the former, while uploading the FODs for the latter to be compiled on the server? (Hmm… maybe I’m asking for too much, but this seems to be super useful and significantly cut down compile times on the server!)

The above text was translated with the assistance of LLM. Here is the original text.

哦 nix copy 确实可以工作！它应该是目前最简单的复制这些 FOD 方式（如果 ssh 可用。SSH不可用的话两次copy或许也可以。）。

提到 CA ，如果是这样的话，我是不是甚至可以直接上传从官方 cache 中下载的包（服务器理应信任官方的 cache ）？唯一的问题是，假设我试图 nix copy 一个相当复杂的 home-manager ，其中部分包是从官方 cache 下载的，部分包是来自其他 untrusted 的 cache （例如自己部署的 cachix ），部分包是从本地编译的。我们是否有可能把它们区分开，对前者上传最终的cache，而后者上传FOD再行编译（嗯……或许我太贪婪了，不过这可能非常有用，能够显著地减少服务器的编译时间？）

1 Like
18

You don’t even need to copy over all the FODs for this! If you trust cache.nixos.org you can just copy over the cache’s final prebuilt outputs and only copy the needed FODs/drvs/other CA paths.

I got ChatGPT (GPT-5.6-Sol) to start making a script and my coding agent (Pi with GPT-5.6-Luna) to finish it, see airgap.sh in:

I can polish it up into being a real program if you want. Mostly just needs some changes to have a real documented command line interface, and not assume defaults like ./ssh_config.

2 Likes
19

By the way, you can ask questions in Chinese here, to avoid the effort of translating: 汉语 (Chinese) - NixOS Discourse

1 Like
20

Well, I (mostly my coding agent) rewrote it in Python with a nicer CLI and found a bug in Cachix!

It also has a nicer README now. Have fun with this new premade solution! If you need to export the stuff to a file and get that 1 file across the airgap physically, just make an issue and I’ll find out how to do that!

1 Like