make nix work in restricted environments (containers, HPC, …)
be able to use the official binary cache (by simulating the /nix/store)
make it easy to distribute nix (via other package managers)
Systems confirmed working (Please add yours via PR):
CentOS 7
Debian (in docker)
NixOS
Under the hood:
the nix-portable binary is a self extracting archive, caching its contents under $HOME/.nix-portable
either bublewrap (bwrap) or proot is used to simulate the /nix/store directory which actually resides in $HOME/.nix-portable/store
a default nixpkgs channel is included and the NIX_PATH variable is set accordingly.
nix version 2.4 is used and configured to enable flakes and nix-command out of the box.
Missing Features:
managing nix profiles via nix-env
managing nix channels via nix-channel
MacOS
support other architecutres than x86_64
Container Runtimes
To simulate the /nix/store and a few other directories, nix-portable supports the following container runtimes.
bwrap (existing installation)
bwrap (shipped via nix-portable)
proot (existing installation)
proot (shipped via nix-portable)
bwrap is preferred over proot and existing installations are preferred over the nix-portable included binaries.
nix-portable will try to figure out which runtime is best for your system.
In my experience, the nix --store option is not very well tested and tends to break between releases. So wrapping the whole nix process in a chroot seems like a better option until a proper test suite is added to Nix.
There are two other related projects in the same space:
Overall, using bubblewrap or another well-tested technology seems like a good idea. I need to test that out as I wanted to solve the bootstrap problem for the devshell project.
AFAIK, the original nix supports two methods for re-locating the store.
Recompiling nix with some flags, or setting the NIX_STORE_DIR. The problem with that is, that you will not be able to use the official cache, since all your stuff has to be linked against another store path.
Using the --store flag when executing nix. That will behave similarly to nix-portable, but it requires user namespaces and won’t work on some distros where this is disabled.
Apart from that, nix-portable tries to ensure that nix really works out of the box. Using the original nix, you might need to install SSL certificates first, if you haven’t, then you have to setup your nix-channel, or write a nix configuration to enable flakes, etc. Nix-portable does all of that automatically. It doesn’t really depend on anything else than bash.
Thanks for mentioning them. Both of those projects require user namespaces, though.
Not sure what you’re talking about. --store has been required by the NixOS installer since 2018 so it’s extensively tested (NixOS cannot install without it). It’s also tested as part of the Nix test suite.
Oh nice! I assumed that bwrap also needed userns but it actually depends on setuid to setup the mount namespace. I missed that detail.
Sorry I didn’t mean to say that Nix is not well tested. There are some blind spots in the testing that appear when I try to use it. For example, try this:
If you look at the last couple of lines in buublewrap/configure.ac, you can see that there are some settings about userns and setuid. I assume, that the way bwrap is built in nix-portable is not optimal yet. I think it still requires userns. It fails for example on Centos. But that’s why I included proot as a fallback for now.
If I understand correctly, it would be optimal to build bwrap with --with-priv-mode=setuid. But the build fails if I set this flag. If anybody could help to fix that, it would be nice. Until then, proot works, but it’s usage is quite hacky and I think bwrap would be smoother.
Curious, did you also evaluate using fakechroot? It doesn’t need setuid either and seems to be a bit more focused and lower-overhead than proot, using LD_PRELOAD instead of ptrace. That does exclude statically linked executables, which shouldn’t be an issue for usual Nix derivations (except for Go programs…?). No idea how they compare on robustness.
If I use fakechroot, I could have my nix store in ./newroot/nix/store and then chroot into newroot. But then, all nix processes run totally isolated. If you are in a nix-shell, then you woudn’t have access to anything on your normal system. And any nix installed programs couldn’t access your files (since they also need to be executed in that environment).
If it is possible to do any bind mounts ontop of the chroot via fakechroot, it might be a good candidate. If you know how to achieve this, please let me know.
For MacOS, I remeber somebody created nix-bundle-macos. The same method could probably be used for nix-portable.
I’m not a MacOS user and therefore have not much motivation to spend time with that, but PR’s are welcome.
All in all I guess MacOS support should be fairly simple, because there aren’t so many different distros/kernel versions like with linux, and there is probably one single way how to do this properly.
It looks like there is an environment variable FAKECHROOT_EXCLUDE_PATH, I wonder if setting it to / makes the entire host visible? But even just fakechroot ls simply segfaults for me on NixOS, so I can’t test it.
That seems to be using binary rewriting, so it won’t work if the path of your new store is longer than /nix/store.
Yes, the fakeroot package in nixpkgs seems to be broken. But I tried on debian and in general FAKECHROOT_EXCLUDE_PATH seems to work, except with /.
But there is a problem. When I exclude /nix, the chroot environment behaves strangely:
Executing ls /nix lists the contents of /nix successfully.
BUT: Executing ls / doesn’t show the nix directory.
Proot has the exact same problem BTW. /nix not showing up when enumerating /, induces strange behavior in nix. Nix re-downloads store paths over and over again (even though they exist).
In proot this enumeration issue can be circumvented by not sharing the / directly, and instead creating a new empty root and mount all the directories like /nix, /run, /proc, /etc, /home, etc… inside that new root.
If the enumeration problem would be fixed in proot, or if nix could be patched to react less sensitive to it, then proot could be used more comfortably.
EDIT: I just realized that excluding /nix in fakechroot doesn’t bring us any further, since we obviously don’t want to map /nix to /nix but rather $HOME/.nix-portable to /nix. We could use a symlink inside the new chroot. Not sure if there are any downsides using a symlinked store. I will try
I checked again, and it really seems like there is no permissionless container technology for macos.
Assuming that docker is probably installed on most MacOS machines, we could just use docker and mount all the systems directories inside the container (like we do it with proot).
A side effect would be that every process run via nix-protable (nix-shell etc…), would actually run on linux but will feel like working on the local system.
Maybe this is even a cool feature? No more worries about macos builds in dev environments…
Folks who have been running into trouble using nix-portable might wanna try the newest release.
I have added a test pipeline to continuously test nix-portable against the following systems:
Distros:
Arch Linux
Debian 10
CentOS 7
CentOS 8
NixOS
Ubuntu 20.04
Other Environments:
Docker (debian image)
Github Action
These tests have revealed issues here and there which are fixed in the newest release.
Apart from that, there are many other improvements, mainly:
Just to resolve some confusion I might have caused with my earlier comments about bubblewrap and proot.
The situation is like this:
Bwrap supports 2 ways of creating namespaces:
setuid root
user namespaces
We cannot use the setuid mode with the bwrap that is shipped with nix-portable, since setting that flag on the users file system would require permissions which we don’t have. In case a bwrap with setuid-root is already installed on the users system, nix-portable will make use of it.
Usually the setuid root will not be available and nix-portable will fall back to using user namespaces via the bwrap that comes along with it.
Only if user namespaces are not available, nix-portable will fall back to proot, which doesn’t require any permissions. Proot is treated as last resort, since it can have noticable performance overhead.
With the latest version 008, the fallback to proot is only necessary for centos7 and debian. All other systems seem to work fine with userns where the performance penalty is much lower. I didn’t benchmark but I can see that builds are finishing significantly quicker than with proot.
.