From inside `nix shell`, many system files are owned by `nobody:nobody`, breaking host programs

Help
1

Dear NixOS/Nix community!

This post is a follow-up to my previous help request here. As a quick reminder, I want to use Nix as a package manager for an unprivileged user with no system-wide Nix installation.

I have discovered that when I use a static build of Nix (currently experimenting with the following builds: maintenance-2.32 and master), from inside the nix shell many system files are seen as owned by nodoby:nobody, which breaks some of the host programs.

More specifically, I am running a Fedora 42 host, I downloaded a static build of Nix from one of the links above (reproducible with either of them), put it into ~/.local/bin and made it executable. Then, I created ~/.config/nix/nix.conf with the following contents:

extra-experimental-features = nix-command flakes

and created a flake directory, ~/flake, with the following in ~/flake/flake.nix:

{
  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.default = with nixpkgs.legacyPackages.x86_64-linux; buildEnv {
      name = "project";
      paths = [
        yazi
        btop
      ];
    };
  };
}

When I enter the Nix shell by doing nix shell ~/flake, yazi and btop binaries become available as expected, with /nix/store/j993rn8ppsrqbyi5j4q2cgr7aqc799z9-project/bin/ in the $PATH, and /nix visible as owned by user_main:user_main (my unprivileged user that I run all commands as). /nix doesn’t exist outside of the Nix shell.

I discovered an issue when I put a similar nix shell invocation in my ~/.bashrc to ensure that all the binaries from the flake are available to me right away on every login. When I later tried to perform a git push, I ran into an SSH error, which I was able to pin down to the following:

[user_main@libvirt-fedora-experiments ~]$ ssh user_main@localhost
Bad owner or permissions on /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf

[user_main@libvirt-fedora-experiments ~]$ ls -la /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
lrwxrwxrwx. 1 nobody nobody 63 Oct 12 20:00 /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf -> ../../../usr/lib/systemd/ssh_config.d/20-systemd-ssh-proxy.conf

[user_main@libvirt-fedora-experiments ~]$ ls -la /
total 60
drwxr-xr-x.  20 user_main user_main   480 Dec  1 08:10 .
drwxr-xr-x.  20 user_main user_main   480 Dec  1 08:10 ..
dr-xr-xr-x.   2 nobody    nobody     4096 Jul 29 20:00 afs
lrwxrwxrwx.   1 user_main user_main     7 Dec  1 08:10 bin -> usr/bin
dr-xr-xr-x.   6 nobody    nobody     4096 Nov 30 08:50 boot
drwxr-xr-x.  19 nobody    nobody     3880 Dec  1 07:53 dev
drwxr-xr-x.  84 nobody    nobody     4096 Nov 30 08:50 etc
drwxr-xr-x.   3 nobody    nobody     4096 Oct 20 15:30 home
lrwxrwxrwx.   1 user_main user_main     7 Dec  1 08:10 lib -> usr/lib
lrwxrwxrwx.   1 user_main user_main     9 Dec  1 08:10 lib64 -> usr/lib64
drwx------.   2 nobody    nobody    16384 Oct 20 15:28 lost+found
drwxr-xr-x.   2 nobody    nobody     4096 Jul 29 20:00 media
drwxr-xr-x.   2 nobody    nobody     4096 Jul 29 20:00 mnt
drwxr-xr-x.   3 user_main user_main    60 Dec  1 08:10 nix
drwxr-xr-x.   2 nobody    nobody     4096 Jul 29 20:00 opt
dr-xr-xr-x. 264 nobody    nobody        0 Dec  1 07:53 proc
dr-xr-x---.   4 nobody    nobody     4096 Oct 20 16:38 root
drwxr-xr-x.  36 nobody    nobody      860 Dec  1 07:53 run
lrwxrwxrwx.   1 user_main user_main     8 Dec  1 08:10 sbin -> usr/sbin
drwxr-xr-x.   2 nobody    nobody     4096 Jul 29 20:00 srv
dr-xr-xr-x.  13 nobody    nobody        0 Dec  1 07:53 sys
drwxrwxrwt.  15 nobody    nobody      320 Dec  1 08:10 tmp
drwxr-xr-x.  11 nobody    nobody     4096 Oct 20 15:28 usr
drwxr-xr-x.  18 nobody    nobody     4096 Oct 20 16:26 var

As can be seen from the listing above, one of the SSH configuration files is visible as owned by nobody:nobody from within the Nix shell, which prevents normal SSH functioning. Apparently, this is also the case for many other system files.

If I exit the Nix shell, the permissions are back to expected:

[user_main@libvirt-fedora-experiments ~]$ exit
exit

[user_main@libvirt-fedora-experiments ~]$ ls -la /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
lrwxrwxrwx. 1 root root 63 Oct 12 20:00 /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf -> ../../../usr/lib/systemd/ssh_config.d/20-systemd-ssh-proxy.conf

[user_main@libvirt-fedora-experiments ~]$ ls -la /
total 33554504
dr-xr-xr-x.  19 root root        4096 Oct 20 16:37 .
dr-xr-xr-x.  19 root root        4096 Oct 20 16:37 ..
dr-xr-xr-x.   2 root root        4096 Jul 29 20:00 afs
lrwxrwxrwx.   1 root root           7 Jul 29 20:00 bin -> usr/bin
dr-xr-xr-x.   6 root root        4096 Nov 30 08:50 boot
drwxr-xr-x.  19 root root        3880 Dec  1 07:53 dev
drwxr-xr-x.  84 root root        4096 Nov 30 08:50 etc
drwxr-xr-x.   3 root root        4096 Oct 20 15:30 home
lrwxrwxrwx.   1 root root           7 Jul 29 20:00 lib -> usr/lib
lrwxrwxrwx.   1 root root           9 Jul 29 20:00 lib64 -> usr/lib64
drwx------.   2 root root       16384 Oct 20 15:28 lost+found
drwxr-xr-x.   2 root root        4096 Jul 29 20:00 media
drwxr-xr-x.   2 root root        4096 Jul 29 20:00 mnt
drwxr-xr-x.   2 root root        4096 Jul 29 20:00 opt
dr-xr-xr-x. 263 root root           0 Dec  1 07:53 proc
dr-xr-x---.   4 root root        4096 Oct 20 16:38 root
drwxr-xr-x.  36 root root         860 Dec  1 07:53 run
lrwxrwxrwx.   1 root root           8 Jul 29 20:00 sbin -> usr/sbin
drwxr-xr-x.   2 root root        4096 Jul 29 20:00 srv
-rw-------.   1 root root 34359738368 Oct 20 16:37 swapfile
dr-xr-xr-x.  13 root root           0 Dec  1 07:53 sys
drwxrwxrwt.  15 root root         320 Dec  1 08:10 tmp
drwxr-xr-x.  11 root root        4096 Oct 20 15:28 usr
drwxr-xr-x.  18 root root        4096 Oct 20 16:26 var

I wonder is anything can be done to achieve a similar workflow of binaries installed with Nix being readily available in $PATH, but with minimal disruptions to the rest of the system?

I have little knowledge of the inner workings of Nix, but I suspect that this may be related to the Nix sandbox and UID remapping via user namespaces. I have experimented with various nix.conf options discovered here, including:

extra-experimental-features = nix-command flakes auto-allocate-uids
auto-allocate-uids = true
sandbox = false
system-features = uid-range

but to no avail.

I would appreciate some guidance.
Thank you so much!

2

Most likely.

Rather than running your entire user session in a subshell of nix-shell, have you considered using home-manager or hejm?

3

I am not sure I understand how that would help in my case.

My end goal is to be able to use binaries installed with Nix together with system-wide binaries by simply typing their names in the shell, just like I would do if I used Conda instead.

To that end I tried to drop into nix shell from my ~/.bashrc, similarly to how I used to activate a Conda environment in the same file.

As I understand it, having Nix-installed binaries in the $PATH necessarily requires entering a Nix shell and having /nix (which is non-existent system-wide in my case) bind-mounted via user namespaces, but the missing piece then is that permissions for the wider system get changed.

4

No, an environment manager like home-manager or hejm can set up your $PATH to contain paths from your nix store instead, without entering a shell. They do this with a script you source from your shell rc file.

That way you don’t end up in a build environment for regular shell use, and end up with less unexpected results.

Particularly, no sandboxing would be involved; that said, I’m not certain the sandbox is the issue here. I suspect you’ll still have issues, but trying to use nix-shell as a general-purpose env manager long-term isn’t going to go well.

5

I was under impression that if I want to use pre-built binaries from the Nix cache (one more preference that I forgot to mention before), rather than build everything from source, everything must live under /nix, because it was built this way and relies on hard-coded paths.

Actually, maybe the preference of using cache is not very important for me, because I don’t expect to install too many packages this way.

I will experiment with Home Manager then, thank you for the suggestion! By the way, I am not using nix-shell, as there doesn’t seem to exist a static build thereof. Rather, it’s a static build of nix which has a shell sub-command. As I understand, the entire nix utility is still experimental, so I understand my goals and approaches may well be beyond the established best practices of the Nix community.

Actually, I have just thought that maybe I can simply install whatever software that is installed system-wide and fails from within a Nix shell (SSH, in this case) into that shell using the same flake. So I am going to take some time experimenting, but in the meantime I would still appreciate additional advice.

6

You have a lot of misunderstandings :smiley:

If you want to use nix-built binaries at all, regardless of whether they are cached or not, they must be used from /nix. This is how nix can achieve reproducibility and caching in the first place; building from source will (assuming no inherent reproducibility issues with the build scripts that nix cannot account for) never give you different results than using cached results. Hence it’s called a “cache”.

It’s impossible to get nix to download binaries into paths other than the nix store, so you don’t have to worry about doing that wrong. It does not matter if the path to the software is placed on $PATH by nix shell or something else, however. home-manager and hejm are two ways to put your software on $PATH designed for the type of thing you’re trying to do here, while nix-shell is intended for temporary use.

hejm is likely more like what you want - it gives you much more control over the actual result, though it requires more understanding of what goes on with shell initialization and nix in general, so you might find it a bit difficult to use. home-manager is likely easier to use for newbies.

You might want to toy around with NixOS on a VM for a while, frankly, you’re very deep into nearly-uncharted territory with far too little experience to actually navigate it by yourself.

I don’t understand what you’re trying to say here. nix-shell is also just another entrypoint into nix, it just symlinks to nix:

~> ls -l /nix/store/n5s95aq6my065mxv8nywdp24afqmk48q-nix-2.31.2/bin/nix-shell
                                 name                                      type     target
──────────────────────────────────────────────────────────────────────────────────────────
 /nix/store/n5s95aq6my065mxv8nywdp24afqmk48q-nix-2.31.2/bin/nix-shell    symlink    nix

nix checks its arg0 to see whether to behave as the new nix command or to behave as nix-shell, or another of the various traditional nix sub-binaries. It’s how old-style multiplex CLI applications distinguish between their use cases.

It is, but it also sees pretty widespread use. The two commands basically do the same thing, and I mostly use the names interchangeably to describe the same concept, sorry about the confusion.

I’d suggest to try and understand why people don’t use the term “install” very much in nix land. You’re not so much “installing” software as you are grabbing it from different prefixes ad-hoc.

Within the context of nix-shell/nix shell, the word “install” makes little sense. I suspect you’ve not fully grokked the intention of ephemeral package scopes; The way you’re trying to use it is quite horrifying.

7

Yes, this is a matter of the proot you are running the shell in. I doubt there is a way to work around that.

If you want to use static nix without a “proper” installation, you should not enter a shell unless you need it.

This means, there is no viable workaround for you to achive “global” installations. You can not even use HM or hjem, they will just create dead symlinks all over your home.

Also to get nix-* style commands, you can symlink them to nix and have the links in your PATH somewhere. nix is the same for all of the commands, and just branching on ARG0.

1 Like
8

I wonder if it’d be possible with some particularly naughty abuse of the user session slice. You can probably get systemd to put that in a special cgroup. That’d probably cause its own set of issues, of course…

But yeah, I don’t think you should attempt a project like this without first really understanding all the moving parts. It’s completely off the beaten path.

9

TLATER, @NobbZ, thank you for your comments!

Re nix-shell vs nix: I vaguely remember something along the lines of nix-shell being a utility specific to the “proper” installation of Nix, and nix being a new-generation utility, for which static builds are available. Maybe this is no longer the case, or maybe I just had the wrong idea about this from the very beginning, thank you for correcting me!

I have tried adding openssh into the flake that I have been using, and the result is somewhat unexpected: even though from within the Nix shell ssh now corresponds to the OpenSSH binary inside the Nix store, this binary still looks up the system-wide configs, for which the ownership is messed up by the sandbox/namespace mechanisms. By the way, the system-wide configs are used even if I revert all my attempts to disable the Nix sandbox. I can tell this binary to actually use configs inside the Nix store with ssh -F /nix/store/law40w42z5j8abrlz9xwir82aaf36iqb-project/etc/ssh/ssh_config, and then it works as expected. I wonder if you think this is a bug with the Nix build of openssh? I believe it should not, unless explicitly instructed, try to access system-wide configs. I can report a bug if you think this is worthy of attention.

So all these experiments lead me to believe that I should indeed abandon the approach I have been taking so far, because it is messy and, apparently, doesn’t follow the best practices.

I wonder what you would suggest me to do? The ultimate goal is that my unprivileged user is able to install the binaries that are missing system-wide (or are outdated system-wide), their versions can be locked down and easily reproduced elsewhere (100% binary reproducibility is not required), and they can be updated on-demand. All this should ideally come with minimal disruptions to the functioning of the software installed system-wide.

So far I have been using Miniforge to achieve this, and I was mostly happy with the solution. My attempts to move to Nix are mostly due to my idea that Nix has more (and more up-to-date) software and would therefore be a better choice in general, but it’s not like I lacked something when I used Miniforge. I can go back to this solution if Nix doesn’t end up working for my use-case.

Another option I see is that I continue using a Nix flake, but rather than dropping to the Nix shell on login, I create wrappers for every binary that I want to use from this flake, and put them under ~/.local/bin such that, for example, ~/.local/bin/yazi is a wrapper for nix shell ~/flake -c yazi, and so on for every binary. That way I get version lockdown and reproducibility through flake.lock, and can update all the software by removing the lock file. The downside is that I should be mindful of the sandbox, and avoid doing something like “run a file manager from the flake, drop into a shell from this file manager and then attempt to use SSH”.

I would greatly appreciate any comments about these and other possible approaches!

I would also like to politely request @twhitehead’s comments if possible. You were very helpful in the previous thread of mine, here. I wonder if you can suggest something here…?

Thank you so much!

10

The first thing you have to understand, the “sandbox” you are in, is not what we usually call the “nix sandbox”. Nix’ sandbox is for buildtime only.

The sandbox you are caught in here, is an implementation detail of static/portable nix, and just using a proot if I recall directly.

All limitations you see here, are because of userlevel namespacing limitations. They mostly come from the kernel and can not be workedaround.

Portable nix is not a tool meant for permanent package management. It is something to play with nix, learn it, or maybe use it to evaluate some expressions.

I do not see much of use for anything else.

If you want to use nix, install nix in the single user or daemon mode. If you can not do that, it is better to not use nix, you will fight a lot of things when trying to do so.

11

I see, thank you so much for this perspective!

My ultimate goal is to maintain mostly the same environment on several machines, and while on two of them I can’t install Nix “properly”, on the one where I spend most of my time working (and where I need most of the extra packages) this is possible.

It seems like on the one machine where my user has sufficient privileges I can install Nix conventionally, and use nix shell ~/flake -c MY_SHELL_BINARY to simultaneously launch the shell of interest (I use Fish) and make all the other Nix-managed binaries available in my $PATH.

On the remaining two machines, I can use a portable installation of Nix, and simple wrappers around nix shell ~/flake -c MY_BINARY, with the only drawback being that the shell of interest should be installed outside of Nix. As I understand it, the flake can be the same on all three machines, and synchronized via Git.

I think I will roll with this setup for now, but if I encounter further issues, I will not push further from the best practices, and just find another well-supported approach :slight_smile: